The Center for Education and Research in Information Assurance and Security (CERIAS)

The Center for Education and Research in
Information Assurance and Security (CERIAS)

Failures in the Supply Chain


[This is dervied from a posting of mine to Dave Farber’s Interesting People list.]

There is an article in the October Businessweek that describes the problem of counterfeit electronic components being purchased and used in critical Defense-related products.

This is not a new threat. But first let’s reflect on the past.

Historically, the military set a number of standards (MIL-SPEC) to ensure that materials they obtained were of an appropriate level of quality, as well as interoperable with other items. The standards helped ensure a consistency for everything from food to boots to tanks to software, as well as ensuring performance standards (quality).

The standards process was not without problems, however. Among issues often mentioned were:

  • Standards were sometimes not revised often enough to reflect changes in technology. The result was that the military often had to acquire and use items that were generations behind the commercial marketplace (esp. in software/computers);
  • Knowing and complying with so many standards often caused companies considerable extra time and effort in supplying items, thus raising their cost well above comparable commercial equivalents;
  • Incompatible standards across military agencies and services, especially when compared with commercial items used by civilian agencies, led to waste and increased cost, and lack of flexibility in implementation;
  • Imposition of rigid standards cut down on innovation and rapid development/acquisition/deployment cycles;
  • The rigidity and complexity of the standards effectively shut out new vendors, especially small vendors because they could not match the standards-compliance efforts of large, entrenched defense vendors.

Thus, in June of 1994, William Perry, the then Secretary of Defense, issued a set of orders that effectively provide a pathway to move away from the standards and adopt commercial standards and performance goals in their place. (cf. the Wikipedia article on MIL-SPEC). One of the rationales expressed then, especially as regarded computing software and hardware, was that the competition of the marketplace would lead to better quality products. (Ironically, the lack of vendor-neutral standards then led to a situation where we have large monocultures of software/hardware platforms throughout government, and the resultant lack of meaningful competition has almost certainly not served the goals of better quality and security.)

In some cases, the elimination of standards has indeed helped keep down costs and improve innovation. I have been told, anecdotally, that stealth technology might not have been fielded had those aircraft been forced within the old MIL-SPEC regime.

As a matter of cost and speed many MIL-SPEC standards seem to have been abandoned to choose COTS whenever possible without proper risk analysis. Only recently have policy-makers begun to realize some of the far-reaching problems that have resulted from the rush to abandon those standards.

As the Businessweek article details, counterfeit items and items with falsified (or poorly conducted) quality control have been finding their way into critical systems, including avionics and weapons control. The current nature of development means that many of those systems are assembled from components and subsystems supplied by other contractors, so a fully-reputable supplier may end up supplying a faulty system because of a component supplied by a vendor with which they have no direct relationship. One notable example of this was the “Cisco Raider” effort from a couple of years ago where counterfeit Cisco router boards were being sold in the US.

As noted in several press articles (such as the ones linked in, above) there is considerable price motive to supply less capable, “grey market” goods in large bids. The middlemen either do not know or care where the parts come from or where they are being used—the simply know they are making money. The problem is certainly not limited to Defense-related parts, of course. Fake “Rolex” watches that don’t keep time, fake designer shoes that fall apart in the rain, and fake drugs that either do nothing or actually cause harm are also part of the “gray market.” Adulteration of items or use of prohibited materials is yet another aspect of the problem: think “lead paint” and “melamine” for examples. Of course, this isn’t a US-only problem; people around the world are victimized by gray-market, adulterated and counterfeit goods.

These incidents actually illustrate some of the unanticipated future effects of abandoning strong standards. One of the principal values of MIL-SPEC standards was that it established a strict chain of accountability for products. I suspect that little thought has been given by policy-makers to the fact that there is considerable flow of items across borders from countries where manufacturing expertise and enforcement of both IP laws and consumer-protection statutes may not be very stringent. Buying goods from countries where IP violations are rampant (If there is little fear over copying DVDs, then there is little fear over stamping locally-produced items as “Cisco”), and where bribes are commonplace, is not a good strategy for uniform quality.

Of course, there are even more problems than simply quality. Not every country and group has the same political and social goals as we do in the US (or any other country—this is a general argument). As such, if they are in a position to produce and provide items that may be integrated into our defense systems or critical infrastructure, it may be in their interests to produce faulty goods—or carefully doctored goods. Software with hidden ‘features” or control components with hidden states could result in catastrophe. That isn’t fear-mongering—we know of cases where this was done, such as to the Soviets in the 1980s. Even if the host country isn’t subtly altering the components, it may not have the resources to protect the items being produced from alteration by third parties. After all, if the labor is cheaper in country X, then it will also be cheaper to bribe the technicians and workers to make changes to what they are producing.

The solution is to go back to setting high standards, require authentication of supply chain, and better evaluation of random samples. Unfortunately, this is expensive, and we’re not in a state nationally where extra expense (except to line the pockets of Big Oil and Banking) is well tolerated by government. Furthermore, this alters the model where many small vendors acting as middlemen are able to get a “piece of the action.” Their complaints to elected representatives who may not understand the technical complexities adds even further pressure against change.

In some cases, the risk posed in acquisition of items may warrant subsidizing the re-establishment of some manufacturing domestically (e.g., chip fabs). This doesn’t need to be across the board, but it does required judicious risk-analysis to determine where critical points are—or will be in the future. We must realize that the rapid changes in technology may introduce new patterns of production and acquisition that we should plan for now. For instance, once elements of nanotechnology become security-critical, we need to ensure that we have sufficient sources of controlled, quality production and testing.

I’m not going to hold my breath over change, however. Some of us have been complaining about issues such as this for decades. The usual response is that we are making a big deal out of “rare events” or are displaying xenophobia. The sheer expense frightens off many from even giving it more than a cursory thought. I know I have been dismissed as an “over-imaginative academic” more times than I can count when I point out the weaknesses.

One of the factors that allegedly led to the decline of the Roman empire was the use of lead in pipes, and lead salts to make cheap wine more palatable for the masses. The Romans knew there was a health problem associated with lead, but the vendors saw more profit from using it.

Once we have sufficiently poisoned our own infrastructure to save money and make the masses happier, how long do we last?

[If you are interested in being notified of new entries by spaf on cyber and national security policy issues, you can either subscribe to the RSS feed for this site, or subscribe to the notification list.]



Posted by Clive Robinson
on Monday, November 17, 2008 at 12:11 PM

@ Gene,

A couple of things,

First of one of the reasons that COTS was looked at and implemented was the “10,000USD Hammer” problem.

Where ever ther is a system involving “public money” you will find somebody pushing the margins as hard as they can in their direction, until they are stopped.

Even when “whistle-blower” policies are put in place, people will work out ways of using these to their own advantage, at somebody elses expense.

Secondly is the issue of WASP Nation (US, UK, et al) “elected representatives”. Their outlook usually does not stretch much beyond the next “political dog fight” or lobbyist largess. Gives rise to “will-o’-the-wisp” policies with a large side helping of “self serving” pork…

Seperatly or together the “quick profit” and “short term outlook” mentalities always lead to problems (out sourcing, banking, environment, food production, health care to name but a few recent examples).

Some nations still think one or two generations into the future. And China once used to think in dynasties.

The simple fact is that anyone who takes a longer term view and has sufficient resources will usually beat those of a short term viewpoint.

A simple example of this is the dreaded QA process, those who put in the effort reap the rewards in the long term. Those who don’t might show short term benifit but have little or no longterm sustainability or stability.

There are solutions to these two “handicaped” mentalities of “quick profit” and “short term outlook” but you will be accused of “shooting the American dream” by those “middle men” who take out as much as they can, as quickly as they can, at such great cost to the rest of us.

There used to be the notion of “stewardship” where you did things not for yourself or those immediatly around you but for the good of all and for those to come.

To a certain extent that is what Standards try to codify, which is why I’m in favour of them.

But standards must not be set in stone and must evolve with experiance and the market place. A clasic example of standards that work are things like the European GSM standards for mobile phones.

Leave a comment

Commenting is not available in this section entry.