Failures in the Supply Chain
[This is dervied from a posting of mine to Dave Farber’s Interesting People list.]
There is an article in the October Businessweek that describes the problem of counterfeit electronic components being purchased and used in critical Defense-related products.
This is not a new threat. But first let’s reflect on the past.
Historically, the military set a number of standards (MIL-SPEC) to ensure that materials they obtained were of an appropriate level of quality, as well as interoperable with other items. The standards helped ensure a consistency for everything from food to boots to tanks to software, as well as ensuring performance standards (quality).
The standards process was not without problems, however. Among issues often mentioned were:
- Standards were sometimes not revised often enough to reflect changes in technology. The result was that the military often had to acquire and use items that were generations behind the commercial marketplace (esp. in software/computers);
- Knowing and complying with so many standards often caused companies considerable extra time and effort in supplying items, thus raising their cost well above comparable commercial equivalents;
- Incompatible standards across military agencies and services, especially when compared with commercial items used by civilian agencies, led to waste and increased cost, and lack of flexibility in implementation;
- Imposition of rigid standards cut down on innovation and rapid development/acquisition/deployment cycles;
- The rigidity and complexity of the standards effectively shut out new vendors, especially small vendors because they could not match the standards-compliance efforts of large, entrenched defense vendors.
Thus, in June of 1994, William Perry, the then Secretary of Defense, issued a set of orders that effectively provide a pathway to move away from the standards and adopt commercial standards and performance goals in their place. (cf. the Wikipedia article on MIL-SPEC). One of the rationales expressed then, especially as regarded computing software and hardware, was that the competition of the marketplace would lead to better quality products. (Ironically, the lack of vendor-neutral standards then led to a situation where we have large monocultures of software/hardware platforms throughout government, and the resultant lack of meaningful competition has almost certainly not served the goals of better quality and security.)
In some cases, the elimination of standards has indeed helped keep down costs and improve innovation. I have been told, anecdotally, that stealth technology might not have been fielded had those aircraft been forced within the old MIL-SPEC regime.
As a matter of cost and speed many MIL-SPEC standards seem to have been abandoned to choose COTS whenever possible without proper risk analysis. Only recently have policy-makers begun to realize some of the far-reaching problems that have resulted from the rush to abandon those standards.
As the Businessweek article details, counterfeit items and items with falsified (or poorly conducted) quality control have been finding their way into critical systems, including avionics and weapons control. The current nature of development means that many of those systems are assembled from components and subsystems supplied by other contractors, so a fully-reputable supplier may end up supplying a faulty system because of a component supplied by a vendor with which they have no direct relationship. One notable example of this was the “Cisco Raider” effort from a couple of years ago where counterfeit Cisco router boards were being sold in the US.
As noted in several press articles (such as the ones linked in, above) there is considerable price motive to supply less capable, “grey market” goods in large bids. The middlemen either do not know or care where the parts come from or where they are being used—the simply know they are making money. The problem is certainly not limited to Defense-related parts, of course. Fake “Rolex” watches that don’t keep time, fake designer shoes that fall apart in the rain, and fake drugs that either do nothing or actually cause harm are also part of the “gray market.” Adulteration of items or use of prohibited materials is yet another aspect of the problem: think “lead paint” and “melamine” for examples. Of course, this isn’t a US-only problem; people around the world are victimized by gray-market, adulterated and counterfeit goods.
These incidents actually illustrate some of the unanticipated future effects of abandoning strong standards. One of the principal values of MIL-SPEC standards was that it established a strict chain of accountability for products. I suspect that little thought has been given by policy-makers to the fact that there is considerable flow of items across borders from countries where manufacturing expertise and enforcement of both IP laws and consumer-protection statutes may not be very stringent. Buying goods from countries where IP violations are rampant (If there is little fear over copying DVDs, then there is little fear over stamping locally-produced items as “Cisco”), and where bribes are commonplace, is not a good strategy for uniform quality.
Of course, there are even more problems than simply quality. Not every country and group has the same political and social goals as we do in the US (or any other country—this is a general argument). As such, if they are in a position to produce and provide items that may be integrated into our defense systems or critical infrastructure, it may be in their interests to produce faulty goods—or carefully doctored goods. Software with hidden ‘features” or control components with hidden states could result in catastrophe. That isn’t fear-mongering—we know of cases where this was done, such as to the Soviets in the 1980s. Even if the host country isn’t subtly altering the components, it may not have the resources to protect the items being produced from alteration by third parties. After all, if the labor is cheaper in country X, then it will also be cheaper to bribe the technicians and workers to make changes to what they are producing.
The solution is to go back to setting high standards, require authentication of supply chain, and better evaluation of random samples. Unfortunately, this is expensive, and we’re not in a state nationally where extra expense (except to line the pockets of Big Oil and Banking) is well tolerated by government. Furthermore, this alters the model where many small vendors acting as middlemen are able to get a “piece of the action.” Their complaints to elected representatives who may not understand the technical complexities adds even further pressure against change.
In some cases, the risk posed in acquisition of items may warrant subsidizing the re-establishment of some manufacturing domestically (e.g., chip fabs). This doesn’t need to be across the board, but it does required judicious risk-analysis to determine where critical points are—or will be in the future. We must realize that the rapid changes in technology may introduce new patterns of production and acquisition that we should plan for now. For instance, once elements of nanotechnology become security-critical, we need to ensure that we have sufficient sources of controlled, quality production and testing.
I’m not going to hold my breath over change, however. Some of us have been complaining about issues such as this for decades. The usual response is that we are making a big deal out of “rare events” or are displaying xenophobia. The sheer expense frightens off many from even giving it more than a cursory thought. I know I have been dismissed as an “over-imaginative academic” more times than I can count when I point out the weaknesses.
One of the factors that allegedly led to the decline of the Roman empire was the use of lead in pipes, and lead salts to make cheap wine more palatable for the masses. The Romans knew there was a health problem associated with lead, but the vendors saw more profit from using it.
Once we have sufficiently poisoned our own infrastructure to save money and make the masses happier, how long do we last?
[If you are interested in being notified of new entries by spaf on cyber and national security policy issues, you can either subscribe to the RSS feed for this site, or subscribe to the notification list.]