Posts by spaf

On Competitions and Competence

This is a follow-up to my last post here, about the "cybersecurity profession" and education. I was moderating one of the panels at the most recent CERIAS Symposium, and a related topic came up.

Let's start with some short mental exercises. Limber up your cerebellum. Stretch out and touch your cognitive centers a few times. Ready?

There's another barn on fire! Quick, get a bucket brigade going -- we need to put the fire out before everything burns. Again. It is getting so tiring watching all our stuff burn while we're trying to run a farm here. Too bad we can only afford the barns constructed of fatwood. But no time to think of that -- a barn's burning again! 3rd time this week!

Hey, you people over there tinkering with designs for sprinkler systems and concrete barns -- cut it out! We can't spare you to do that -- too many barns are burning! And you, stop babbling about investigating and arresting arsonists -- we don't have time or money for that: didn't you hear me? Another barn is burning!

Now, hurry up. We're going to have a contest to find who can pass this pail of water the quickest. Yes, it is a small, leaky pail, but we have a lot of them, so that is what we're going to use in the contest. The winners get to be closest to the flames and have a name tag that says "fire prevention specialist." No, we can't afford larger buckets. And no, you can't go get a hose -- we need you in the line. Damnit! The barn's burning!

Sounds really stupid, doesn't it? Whoever is in charge isn't doing anything to address the underlying problem of poor barn construction. It doesn't really match the notion of what a fire prevention specialist might really do. And it certainly doesn't provide deep career preparation for any of those contestants... it may even condemn them to a future of menial bucket passing because we're putting them on the line with no training or qualification beyond being able to pass a bucket.

Let's try another one.

Imagine that every car and automobile in the country has been poorly designed. They almost all leak coolant and burn oil. They're trivial to steal. They are mostly cheap junkers, all built on the same frame with the same engines, accessories, and tires -- even the ones sold to the police and military (actually, they're the same cars, but with different paint). The big automakers are rolling out new models every year that they advertise as being more efficient and reliable, but that is simply hype to get you to buy a new car because the new features also regularly break down. There are a few good models available, but they are quite a bit more expensive; those more expensive ones often (but not always) break down less, are more difficult to steal, and get far better mileage. Their vendors also don't have a yearly model update, and many consumers aren't interested in them because those cars don't take the common size of tire or fuzzy dice for the mirror.

The auto companies have been building this way for decades. They sell their products around the world, and they're a major economic force. Everyone needs a car, and they shell out money for new ones on a regular basis. People grumble about the poor quality and the breakdowns, but other than periodic service bulletins, there are few changes from year to year. Many older, more decrepit cars are on the road because too many people (and companies) cannot afford to buy new ones that they know aren't much better than the old ones. Many people argue -- vociferously -- against any attempt to put safety regulations on the car companies because it might hurt such an important market segment.

A huge commercial enterprise has sprung up around fixing cars and adding on replacement parts that are supposedly more reliable. People pour huge amounts of money into this market because they depend on the cars for work, play, safety, shopping, and many other things. However, there are so many cars, and so many update bulletins and add-ons, there simply aren't enough trained mechanics to keep up -- especially because many of the add-ons don't work, or require continual adjustment.

What to do? Aha! We'll encourage young people in high school and maybe college to become "automotive specialists." We'll publish all sorts of articles with doom and gloom as a result of the shortage of people going into auto repair. We especially need lots more military mechanics.

So...we'll have competitions! We'll offer prizes to the individuals (or teams) that are able to change the oil of last year's model the most quickly, or who can most efficiently hotwire a pickup truck, take it to the garage, change the tires, and return it. The government will support these competitions. They'll get lots of press. Some major professional organizations and even universities will promote these. Of course we'll hire lots of mechanics that way! (Women aren't interested in these kinds of competition? We won't worry about that now. People who are poor with wrenches won't compete? No problem -- we'll fill in with the rest.)

Meanwhile, the government and major companies aren't really doing anything to fix the actual engineering of the automobiles. There are a few comprehensive engineering programs at universities around the country, but minimal focus and resources are applied there, and little is said about applying their knowledge to really fixing transportation. The government, especially the military, simply wants more mechanics and cheaper cars -- overall safety and reliability aren't a major concern.

Pretty stupid, huh? But there does seem to be a trend to these exercises.

Let's try one more.

We have a large population that needs to be fed. They've grown accustomed to cheap, fast-food. Everyone eats at the drive-thru, where they get a burger or compressed chicken by-product or mystery-meat taco. It's filling, and it keeps them going for the day. It also leads to obesity, hypertension, cardiac problems, diabetes, and more. However, no one really blames the fast-food chains, because they are simply providing what people want.

It isn't exactly what people should have, and is it really what everyone wants? No, there are better restaurants with healthy food, but that food is more expensive and many people would go hungry if they had to eat at those places given the current economic model. Of course, if they didn't need to spend so much on medicine and hospital stays, a healthier diet is actually cheaper. Also, those better places aren't easy to find -- small (or no) advertising budgets, for instance.

The government has contracted with the chains for food, and even serves it at every government office and on every military base. The chains thus have a fair amount of political clout so that every time someone raises the issue about how unhealthy the food is, they get muffled by the arguments "But it would be too expensive to eat healthy" and "Most people don't like that other food and can't even find it!"

We have a crisis because the demand for the fast-food is so great that there aren't enough fry cooks. So, the heads of major military organizations and government agencies observe we are facing a crisis because, without enough fry cooks, our troops will be overwhelmed by better fed people from China. Government officials and industry people agree because they can't imagine any better diet (or are so enamored of fried potatoes that they don't want anything else).

How do they address the crisis? By mounting advertising campaigns to encourage young people to enter the exciting world of "cuisine awareness." We make it seem glamorous. Private organizations offer certifications in "soda making" and "ketchup bottle maintenance" that are awarded after 3-day seminars. DOD requires anyone working in food service to have one of these certificates -- and that's basically all. We see educational institutes and small colleges offering special programs in "salad bar maintenance." The generals and admirals keep showing up at meetings proclaiming how important it is that we get more burger-flippers in place before we have a "patty melt Pearl Harbor."

The government launches a program to certify schools as centers of "Cuisine Awareness Exellence" if they can prove they have at least 5 cookbooks in the library, a crockpot, and two faculty who have boiled water. Soon, there are hundreds of places designated with this CAE, from taco trucks and hot dog stands to cordon bleu centers -- but lots are only hot dog stands. None of them are given any recipes, cooks, or financial support, of course -- simply designating them is enough, right?

When all of that isn't seen to be enough, the powers-that-be offer up contests that encourage kids to show up and cook. Those who are able to most quickly defrost a compressed cake of Soylent Red, cook it, stick it in a bun, and serve it up in a bag with fries is declared the winner and given a job behind someone's grill. Actually, each registered contestant gets a jaunty paper cap and offer of an immediate job cooking for the military (assuming they are U.S. citizens; after all, we know what those furriners eat sure isn't food!) And gosh, how could they aspire to be anything BUT a fry cook for the next 40 years -- no need to worry about any real education before they take the jobs.

Meanwhile, those studying dietetics, preventative health care, sustainable agriculture, haute cuisine, or other related topics are largely ignored -- not to mention the practicing experts in these fields. The people and places of study for those domains are ignored by the officials, and many of the potential employers in those areas are actually going out of business because of lack of public interest and support. The advice of the experts on how to improve diet is ignored. Find that disconcerting? Here -- have a deep-fried cherry pie and a chocolate ersatz-dairy item drink to make you feel better.

Did you sense a set of common threads (assuming you didn't blow out your cortex in the exercise)?

First, in every case, a mix of short-sighted and ultimately stupid solutions are being undertaken. In each, there are large-scale efforts to address pressing problems that largely ignore fundamental, systemic weaknesses.

Second, there are a set of efforts putatively being made to increase the population of experts, but only with those who know how to address a current, limited problem set. Fancy titles, certificates, and seminars are used to promote these technicians. Meanwhile, longer-term expertise and solutions are being ignored because of the perceived urgency of the immediate problems and a lack of understanding of cost and risk.

Third, longer-term disaster is clearly coming in each case because of secondary problems and growth of the current threats.

Why did this come up with my post and panel on cybersecurity? I would hope that would be obvious, but if not, let me suggest you go back to read my prior post, then read the above examples, again. Then, consider:

  • Nationally, we are investing heavily in training and recruiting "cyber warriors" but pitifully little towards security engineers, forensic responders, and more. It is an investment in technicians, not in educated expertise.
  • We have a marketplace where we continue to buy poorly-constructed products then pay huge amounts for add-on security and managing response; meanwhile, we have knowledgeable users complaining that they can't afford the up-front cost required to replace shoddy infrastructure with more robust items
  • Rather than listen to experts, we let business and military interests drive the dialog
  • We have well-meaning people who somehow think that "contests" are useful in resolving part of the problem

One of the most egregious aspects is this last item -- the increasing use of competitions as a way of drawing people to the field. Competitions, by their very nature, stress learned behavior to react to current problems that are likely small deviations from past issues. They do not require extensive grounding in multiple fields. Competitions require rapid response instead of careful design and deep thought -- if anything, they discourage people who exhibit slow, considerate thinking -- discourage them from the contests, and possibly from considering the field itself. If what is being promoted are competitions for the fastest hack on a WIntel platform, how is that going to encourage deep thinkers interested in architecture, algorithms, operating systems, cryptology, or more?

Competitions encourage the mindset of hacking and patching, not of strong design. Competitions encourage the mindset of quick recovery over the gestalt of design-operate-observe-investigate-redesign. Because of the high-profile, high-pressure nature of competitions, they are likely to discourage the philosophical and the careful thinkers. Speed is emphasized over comprehensive and robust approaches. Competitions are also likely to disproportionately discourage women, the shy, and those with expertise in non-mainstream systems. In short, competitions select for a narrow set of skills and proclivities -- and may discourage many of the people we most need in the field to address the underlying problems.

So, the next time you hear some official talk about the need for "cyber warriors" or promoting some new "capture the flag" competition, ask yourself if you want to live in a world where the barns are always catching fire, the cars are always breaking down, nearly everyone eats fast food, and the major focus of "authorities" is attracting more young people to minimally skilled positions that perpetuate that situation...until everything falls apart. The next time you hear about some large government grant that happens to be within 100 miles of the granting agency's headquarters or corporate support for a program of which the CEO is an alumnus but there is no history of excellence in the field, ask yourself why their support is skewed towards building more hot dog stands.

Those of us here at CERIAS, and some of our colleagues with strategic views elsewhere, remind you that expertise is a pursuit and a process, not a competition or a 3-day class, and some of us take it seriously. We wish you would, too.

Your brain may now return to being a couch potato. grin

Some thoughts on “cybersecurity” professionalization and education

[I was recently asked for some thoughts on the issues of professionalization and education of people working in cyber security. I realize I have been asked this many times, I and I keep repeating my answers, to various levels of specificity. So, here is an attempt to capture some of my thoughts so I can redirect future queries here.]

There are several issues relating to the area of personnel in this field that make issues of education and professional definition more complex and difficult to define. The field has changing requirements and increasing needs (largely because industry and government ignored the warnings some of us were sounding many years ago, but that is another story, oft told -- and ignored).

When I talk about educational and personnel needs, I discuss it metaphorically, using two dimensions. Along one axis is the continuum (with an arbitrary directionality) of science, engineering, and technology. Science is the study of fundamental properties and investigation of what is possible -- and the bounds on that possibility. Engineering is the study of design and building new artifacts under constraints. Technology is the study of how to choose from existing artifacts and employ them effectively to solve problems.


The second axis is the range of pure practice to abstraction. This axis is less linear than the other (which is not exactly linear, either), and I don't yet have a good scale for it. However, conceptually I relate it to applying levels of abstraction and anticipation. At its "practice" end are those who actually put in the settings and read the logs of currently-existing artifacts; they do almost no hypothesizing. Moving the other direction we see increasing interaction with abstract thought, people and systems, including operations, law enforcement, management, economics, politics, and eventually, pure theory. At one end, it is "hands-on" with the technology, and at the other is pure interaction with people and abstractions, and perhaps no contact with the technology.

There are also levels of mastery involved for different tasks, such as articulated in Bloom's Taxonomy of learning. Adding that in would provide more complexity than can fit in this blog entry (which is already too long).

The means of acquisition of necessary expertise varies for any position within this field. Many technicians can be effective with simple training, sometimes with at most on-the-job experience. They usually need little or no background beyond everyday practice. Those at the extremes of abstract thought in theory or policy need considerably more background, of the form we generally associate with higher education (although that is not strictly required), often with advanced degrees. And, of course, throughout, people need some innate abilities and motivation for the role they seek; Not everyone has ability, innate or developed, for each task area.

We have need of the full spectrum of these different forms of expertise, with government and industry currently putting an emphasis on the extremes of the quadrant involving technology/practice -- they have problems, now, and want people to populate the "digital ramparts" to defend them. This emphasis applies to those who operate the IDS and firewalls, but also to those who find ways to exploit existing systems (that is an area I believe has been overemphasized by government. Cf. my old blog post and a recent post by Gary McGraw). Many, if not most, of these people can acquire needed skills via training -- such as are acquired on the job, in 1-10 day "minicourses" provided by commercial organizations, and vocational education (e.g, some secondary ed, 2-year degree programs). These kinds of roles are easily designated with testing and course completion certificates.

Note carefully that there is no value statement being made here -- deeply technical roles are fundamental to civilization as we know it. The plumbers, electricians, EMTs, police, mechanics, clerks, and so on are key to our quality of life. The programs that prepare people for those careers are vital, too.

Of course, there are also careers that are directly located in many other places in the abstract plane illustrated above: scientists, software engineers, managers, policy makers, and even bow tie-wearing professors. grin

One problem comes about when we try to impose sharply-defined categories on all of this, and say that person X has sufficient mastery of the category to perform tasks A, B, and C that are perceived as part of that category. However, those categories are necessarily shifting, not well-defined, and new needs are constantly arising. For instance, we have someone well trained in selecting and operating firewalls and IDS, but suddenly she is confronted with the need to investigate a possible act of nation-state espionage, determine what was done, and how it happened. Or, she is asked to set corporate policy for use of BYOD without knowledge of all the various job functions and people involved. Further deployment of mobile and embedded computing will add further shifts. The skills to do most of these tasks are not easily designated, although a combination of certificates and experience may be useful.

Too many (current) educational programs stress only the technology -- and many others include significant technology training components because of pressure by outside entities -- rather than a full spectrum of education and skills. We have a real shortage of people who have any significant insight into the scope of application of policy, management, law, economics, psychology and the like to cybersecurity, although arguably, those are some of the problems most obvious to those who have the long view. (BTW, that is why CERIAS was founded 15 years ago including faculty in nearly 20 academic departments: "cybersecurity" is not solely a technology issue; this has more recently been recognized by several other universities that are now also treating it holistically.) These other skill areas often require deeper education and repetition of exercises involving abstract thought. It seems that not as many people are naturally capable of mastering these skills. The primary means we use to designate mastery is through postsecondary degrees, although their exact meaning does vary based on the granting institution.

So, consider some the bottom line questions of "professionalization" -- what is, exactly, the profession? What purposes does it serve to delineate one or more niche areas, especially in a domain of knowledge and practice that changes so rapidly? Who should define those areas? Do we require some certification to practice in the field? Given the above, I would contend that too many people have too narrow a view of the domain, and they are seeking some way of ensuring competence only for their narrow application needs. There is therefore a risk that imposing "professional certifications" on this field would both serve to further skew the perception of what is involved, and discourage development of some needed expertise. Defining narrow paths or skill sets for "the profession" might well do the same. Furthermore, much of the body of knowledge is heuristics and "best practice" that has little basis in sound science and engineering. Calling someone in the 1600s a "medical professional" because he knew how to let blood, apply leeches, and hack off limbs with a carpenter's saw using assistants to hold down the unanesthitized patient creates a certain cognitive dissonance; today, calling someone a "cyber security professional" based on knowledge of how to configure Windows, deploy a firewall, and install anti-virus programs should probably be viewed as a similar oddity. We need to evolve to where the deployed base isn't so flawed, and we have some knowledge of what security really is -- evolve from the equivalent of "sawbones" to infectious disease specialists.

We have already seen some of this unfortunate side-effect with the DOD requirements for certifications. Now DOD is about to revisit the requirements, because they have found that many people with certifications don't have the skills they (DOD) think they want. Arguably, people who enter careers and seek (and receive) certification are professionals, at least in a current sense of that word. It is not their fault that the employers don't understand the profession and the nature of the field. Also notable are cases of people with extensive experience and education, who exceed the real needs, but are not eligible for employment because they have not paid for the courses and exams serving as gateways for particular certificates -- and cash cows for their issuing organizations. There are many disconnects in all of this. We also saw skew develop in the academic CAE program.

Here is a short parable that also has implications for this topic.

In the early 1900s, officials with the Bell company (telephones) were very concerned. They told officials and the public that there was a looming personnel crisis. They predicted that, at the then-current rate of growth, by the end of the century everyone in the country would need to be a telephone operator or telephone installer. Clearly, this was impossible.

Fast forward to recent times. Those early predictions were correct. Everyone was an installer -- each could buy a phone at the corner store, and plug it into a jack in the wall at home. Or, simpler yet, they could buy cellphones that were already on. And everyone was an operator -- instead of using plugboards and directory assistance, they would use an online service to get a phone number and enter it in the keypad (or speed dial from memory). What happened? Focused research, technology evolution, investment in infrastructure, economics, policy, and psychology (among others) interacted to "shift the paradigm" to one that no longer had the looming personnel problems.

If we devoted more resources and attention to the broadly focused issues of information protection (not "cyber" -- can we put that term to rest?), we might well obviate many of the problems that now require legions of technicians. Why do we have firewalls and IDS? In large part, because the underlying software and hardware was not designed for use in an open environment, and its development is terribly buggy and poorly configured. The languages, systems, protocols, and personnel involved in the current infrastructure all need rethinking and reengineering. But so long as the powers-that-be emphasize retaining (and expanding) legacy artifacts and compatibility based on up-front expense instead of overall quality, and in training yet more people to be the "cyber operators" defending those poor choices, we are not going to make the advances necessary to move beyond them (and, to repeat, many of us have been warning about that for decades). And we are never going to have enough "professionals" to keep them safe. We are focusing on the short term and will lose the overall struggle; we need to evolve our way out of the problems, not meet them with an ever-growing band of mercenaries.

The bottom line? We should be very cautious in defining what a "professional" is in this field so that we don't institutionalize limitations and bad practices. And we should do more to broaden the scope of education for those who work in those "professions" to ensure that their focus -- and skills -- are not so limited as to miss important features that should be part of what they do. As one glaring example, think "privacy" -- how many of the "professionals" working in the field have a good grounding and concern about preserving privacy (and other civil rights) in what they do? Where is privacy even mentioned in "cybersecurity"? What else are they missing?

[If this isn't enough of my musings on education, you can read two of my ideas in a white paper I wrote in 2010. Unfortunately, although many in policy circles say they like the ideas, no one has shown any signs of acting as a champion for either.]

[3/2/2013] While at the RSA Conference, I was interviewed by the Information Security Media Group on the topic of cyber workforce. The video is available online.

On Student Projects, Phoenix, and Improving Your IT Operations

[If you want to skip my recollection and jump right to the announcement that is the reason for this post, go here.]

Back in about 1990 I was approached by an eager undergrad who had recently come to Purdue University. A mutual acquaintance (hi, Rob!) had recommended that the student connect with me for a project. We chatted for a bit and at first it wasn't clear exactly what he might be able to do. He had some experience coding, and was working in the campus computing center, but had no background in the more advanced topics in computing (yet).

Well, it just so happened that a few months earlier, my honeypot Sun workstation had recorded a very sophisticated (for the time) attack, which resulted in an altered shared library with a back door in place. The attack was stealthy, and the new library had the same dates, size and simple hash value as the original. (The attack was part of a larger series of attacks, and eventually documented in "@Large: The Strange Case of the World's Biggest Internet Invasion" (David H. Freedman, Charles C. Mann .)

I had recently been studying message digest functions and had a hunch that they might provide better protection for systems than a simple ls -1 | diff - old comparison. However, I wanted to get some operational sense about the potential for collision in the digests. So, I tasked the student with devising some tests to run many files through a version of the digest to see if there were any collisions. He wrote a program to generate some random files, and all seemed okay based on that. I suggested he look for a different collection -- something larger. He took my advice a little too much to heart. It seems he had a part time job running backup jobs on the main shared instructional computers at the campus computing center. He decided to run the program over the entire file system to look for duplicates. Which he did one night after backups were complete.

The next day (as I recall) he reported to me that there were no unexpected collisions over many hundreds of thousands of files. That was a good result!

The bad result was that running his program over the file system had resulted in a change of the access time of every file on the system, so the backups the next evening vastly exceeded the existing tape archive and all the spares! This led directly to the student having a (pointed) conversation with the director of the center, and thereafter, unemployment. I couldn't leave him in that position mid-semester so I found a little money and hired him as an assistant. I them put him to work coding up my idea, about how to use the message digests to detect changes and intrusions into a computing system. Over the next year, he would code up my design, and we would do repeated, modified "cleanroom" tests of his software. Only when they all passed, did we release the first version of Tripwire.

That is how I met Gene Kim .

Gene went on to grad school elsewhere, then a start-up, and finally got the idea to start the commercial version of Tripwire with Wyatt Starnes; Gene served as CTO, Wyatt as CEO. Their subsequent hard work, and that of hundreds of others who have worked at the company over the years, resulted in great success: the software has become one of the most widely used change detection & IDS systems in history, as well as inspiring many other products.

Gene became more active in the security scene, and was especially intrigued with issues of configuration management, compliance, and overall system visibility, and with their connections to security and correctness. Over the years he spoken with thousands of customers and experts in the industry, and heard both best-practice and horror stories involving integrity management, version control, and security. This led to projects, workshops, panel sessions, and eventually to his lead authorship of "Visible Ops Security: Achieving Common Security and IT Operations Objectives in 4 Practical Steps" (Gene Kim, Paul Love, George Spafford) , and some other, related works.

His passion for the topic only grew. He was involved in standards organizations, won several awards for his work, and even helped get the B-sides conferences into a going concern. A few years ago, he left his position at Tripwire to begin work on a book to better convey the principles he knew could make a huge difference in how IT is managed in organizations big and small.

I read an early draft of that book a little over a year ago (late 2011), It was a bit rough -- Gene is bright and enthusiastic, but was not quite writing to the level of J.K. Rowling or Stephen King. Still, it was clear that he had the framework of a reasonable narrative to present major points about good, bad, and excellent ways to manage IT operations, and how to transform them for the better. He then obtained input from a number of people (I think he ignored mine), added some co-authors, and performed a major rewrite of the book. The result is a much more readable and enjoyable story -- a cross between a case study and a detective novel, with a dash of H. P. Lovecraft and DevOps thrown in.

The official launch date of the book, "The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win" (Gene Kim, Kevin Behr, George Spafford), is Tuesday, January 15, but you can preorder it before then on (at least) Amazon.

The book is worth reading if you have a stake in operations at a business using IT. If you are a C-level executive, you should most definitely take time to read the book. Consultants, auditors, designers, educators...there are some concepts in there for everyone.

But you don't have to take only my word for it -- see the effusive praise of tech luminaries who have read the book .

So, Spaf sez, get a copy and see how you can transform your enterprise for the better.

(Oh, and I have never met the George Spafford who is a coauthor of the book. We are undoubtedly distant cousins, especially given how uncommon the name is. That Gene would work with two different Spaffords over the years is one of those cosmic quirks Vonnegut might write about. But Gene isn't Vonnegut, either. grin

So, as a postscript.... I've obviously known Gene for over 20 years, and am very fond of him, as well as happy for his continuing success. However, I have had a long history of kidding him, which he has taken with incredible good nature. I am sure he's saving it all up to get me some day....

When Gene and his publicist asked if I could provide some quotes to use for his book, I wrote the first of the following. For some reason, this never made it onto the WWW site . So, they asked me again, and I wrote the second of the following -- which they also did not use.

So, not to let a good review (or two) go to waste, I have included them here for you. If nothing else, it should convince others not to ask me for a book review.

But, despite the snark (who, me?) of these gag reviews, I definitely suggest you get a copy of the book and think about the ideas expressed therein. Gene and his coauthors have really produced a valuable, readable work that will inform -- and maybe scare -- anyone involved with organizational IT.

Take 1:

Based on my long experience in academia, I can say with conviction that this is truly a book, composed of an impressive collection of words, some of which exist in human languages. Although arranged in a largely random order, there are a few sentences that appear to have both verbs and nouns. I advise that you immediately buy several copies and send them to people -- especially people you don't like -- and know that your purchase is helping keep some out of the hands of the unwary and potentially innocent. Under no circumstances, however, should you read the book before driving or operating heavy machinery. This work should convince you that Gene Kim is a visionary (assuming that your definition of "vision" includes "drug-induced hallucination").

Take 2:

I picked up this new book -- The Phoenix Project , by Gene Kim, et al. -- and could not put it down. You probably hear people say that about books in which they are engrossed. But I mean this literally: I happened to be reading it on my Kindle while repairing some holiday ornaments with superglue. You might say that the book stuck with me for a while.

There are people who will tell you that Gene Kim is a great author and raconteur. Those people, of course, are either trapped in Mr. Kim's employ or they drink heavily. Actually, one of those conditions invariably leads to the other, along with uncontrollable weeping, and the anguished rending of garments. Notwithstanding that, Mr. Kim's latest assault on les belles-lettres does indeed prompt this reviewer to some praise: I have not had to charge my health spending account for a zolpidem refill since I received the advance copy of the book! (Although it may be why I now need risperidone.)

I must warn you, gentle reader, that despite my steadfast sufferance in reading, I never encountered any mention of an actual Phoenix. I skipped ahead to the end, and there was no mention there, either. Neither did I notice any discussion of a massive conflagration nor of Arizona, either of which might have supported the reference to Phoenix . This is perhaps not so puzzling when one recollects that Mr. Kim's train of thought often careens off the rails with any random, transient manifestation corresponding to the meme "Ooh, a squirrel!" Rather, this work is more emblematic of a bus of thought, although it is the short bus, at that.

Despite my personal trauma, I must declare the book as a fine yarn: not because it is unduly tangled (it is), but because my kitten batted it about for hours with the evident joy usually limited to a skein of fine yarn. I have found over time it is wise not to argue with cats or women. Therefore, appease your inner kitten and purchase a copy of the book. Gene Kim's court-appointed guardians will thank you. Probably.

(Congratulations Gene, Kevin and George!)

An Interesting Opportunity

As someone who is interested in information security and CERIAS (or why else would you be reading this blog?), you are undoubtedly already aware of the great need for education and research in information/cyber security areas -- the very areas in which we have been a leader for the last 20+ years here at Purdue University.

One aspect of our efforts is an on-going need to attract and retain the very best faculty members possible to provide leadership in all aspects of what we do.

Universities have a mechanism for attracting and retaining the best people: endowed chairs for faculty. These are special designations for positions for leading faculty. The associated endowment provides discretionary funds for travel, research, staff and a salary supplement to support the position. Only a small number of these positions exist in any computing field at universities nationally…and almost none in information/cyber security and privacy. Having one of the oldest and largest programs in this field, Purdue University really should have a few of these positions available to attract and keep the best faculty we can find.

Normally, the endowments for these chairs are provided by generous individuals or foundations who support the university and/or the research area. As a small token of appreciation, the university allows the benefactor(s) to name the chaired position (within reason), thus resulting in something such as the Homer J. Simpson Distinguished Professor of Cyber Security or the Yoyodyne Propulsion Systems Professor of Information Security and Privacy. This name is kept in perpetuity, and is on all stationery and publications of that professor henceforth.

Purdue has just announced a new program to match donations 1:1 for chaired positions with no restrictions. It is thus possible for someone (or a group, company, club or foundation) to endow a distinguished chair at ½ of the usual amount. Further, that amount may be pledged over a three-year period, and the donor(s) still retain full naming rights!

Note that Purdue University is a 503(c) organization and thus donations to support this have potential tax advantages for the donor(s).

We really would like to have CERIAS continue to be the leader in the field of information security. Obtaining at least one (and preferably, several) named chairs in the field, most likely with homes in the CS department, would help us keep that lead, and keep our program strong.

If you are interested in taking part in this great opportunity to help fund one of the first few endowed professorships globally in this important area, please contact me. And if you know of others who might be interested, please pass this along to them. Fields including computer games and graphics have dozens of endowed professorships around the country. Isn't it about time we showed that information security is taken seriously, too?

Gene Schultz, R. I. P.

Sunday, October 2nd, Earl Eugene Schultz, Jr. passed away. Gene probably had suffered an unrecognized stroke about two weeks earlier, and a week later fell down a long escalator at the Minneapolis municipal airport. He was given immediate emergency aid, then hospitalized, but never regained consciousness. Many of his family members were with him during his final days.

What follows is a more formal obituary, based on material provided by his family and others. That is followed by some personal reflections.

Personal Details

Gene was born September 10, 1946, in Chicago to E. Eugene Sr. and Elizabeth Schultz. They moved to California in 1948, and Gene’s sister, Nancy, was born in 1955. The family lived in Lafayette, California. Gene graduated from UCLA, and earned his MS and PhD (in Cognitive Science, 1977) at Purdue University in Indiana.


While at Purdue University, Gene met and married Cathy Brown. They were married for 36 years, and raised three daughters: Sarah, Rachel and Leah.

Gene was an active member of Cornerstone Fellowship, and belonged to a men’s Bible study. His many interests included family, going to his mountain home in Twain Harte, model trains, music, travelling, the outdoors, history, reading and sports.

Gene is survived by his wife of 36 years, Cathy Brown Schultz; father, Gene Schultz, Sr.; sister, Nancy Baker; daughters and their spouses, Sarah and Tim Vanier, Rachel and Duc Nguyen, Leah and Nathan Martin; and two grandchildren, Nola and Drake Nguyen.

A memorial service will be held at Cornerstone Fellowship in Livermore, California on Saturday, October 8, 2011 at 1 pm. Donations may be sent to Caring under his name, Gene Schultz.

You should also take a few moments to visit this page and learn about the symptoms and response to stroke.

Professional Life

Gene was one of the more notable and accomplished figures in computing security over the last few decades. During the course of his career, Gene was professor of computer science at several universities, including the University of California at Davis and Purdue University, and retired from the University of California at Berkeley. He consulted for a wide range of clients, including U.S. and foreign governments and the banking, petroleum, and pharmaceutical industries. He also managed several information security practices and served as chief technology officer for two companies.

Gene formed and managed the Computer Incident Advisory Capability (CIAC) — an incident response team for the U.S. Department of Energy — from 1986–1992. This was the first formal incident response team, predating the CERT/CC by several years. He also was instrumental in the founding of FIRST — the Forum of Incident Response & Security Teams.

During his 30 years of work in security, Gene authored or co-authored over 120 papers, and five books. He was manager of the I4 program at SRI from 1994–1998. From 2002–2007, he was the Editor-in-Chief of Computers and Security — the oldest journal in computing security — and continued to serve on its editorial board. Gene was also an associate editor of Network Security. He was a member of the accreditation board of the Institute of Information Security Professionals (IISP).

Gene testified as an expert several times before both Senate and House Congressional committees. He also served as an expert advisor to a number of companies and agencies. Gene was a certified SANS instructor, instructor for ISACA, senior SANS analyst, member of the SANS NewsBites editorial board, and co-author of the 2005 and 2006 Certified Information Security Manager preparation materials.

Dr. Schultz was honored numerous times for his research, service, and teaching. Among his many notable awards, Gene received the NASA Technical Excellence Award, Department of Energy Excellence Award, the Vanguard Conference Top Gun Award (for best presenter) twice, the Vanguard Chairman's Award, the ISACA John Kuyers Best Speaker/Best Conference Contributor Award and the National Information Systems Security Conference Best Paper Award. One of only a few Distinguished Fellows of the Information Systems Security Association (ISSA), he was also named to the ISSA Hall of Fame and received ISSA's Professional Achievement and Honor Roll Awards.

At the time of his death, Dr. Schultz was the CTO of Emagined Security, an information security consultancy based in San Carlos, California. He held certifications as a CISM, CISSP, and GSLC.

Personal Reflections

As I recall, I first “met” Gene almost 25 years ago, when he was involved with the CIAC and I was involved with network security. We exchanged email about security issues and his time at Purdue. I may have even met him earlier — I can’t recall, exactly. It seems we have been friends forever. We also crossed paths once or twice at conferences, but it was only incidental.

In 1998, I started CERIAS at Purdue. I had contacted personnel at the (now defunct) company Global Integrity while at the National Computer Security Conference that year about supporting the effort at CERIAS. What followed was a wonderful collaboration: Gene was the Director of Research for Global Integrity, and as part of their support for CERIAS they “loaned” Gene to us for several years. Gene, Cathy and Leah moved to West Lafayette, a few houses away from where I lived, and Gene proceeded to help us in research and teaching courses over the next three years while he worked remotely for GI.

The students at Purdue loved Gene, but that seems to have been the case for everywhere he taught. Gene had a gift for conveying complex concepts to students, and had incredible patience when dealing with them one-on-one. He came up with great assignments, sprinkled his lectures with interesting stories from his experience, and encouraged the students to try things to see what they might discover. He was inspirational. He was inspirational as a colleague; too, although we both traveled so much that we didn’t get to see each other too often.

In 2001 he parted ways with Global Integrity, and moved his family back to California. This was no doubt influenced by the winters they had experienced in Indiana — too much of a reminder of grad student days for Gene and Cathy! I remember one time that we all got together to watch a New Year’s Purdue football bowl appearance, and the snow was so high as to make the roads impassable for a few days. Luckily, we lived near each other and it was only a short walk to warmth, hors d’oeuvres, and wine. grin

In the following years, Gene and I kept in close touch. We served on a few committees and editorial boards together, regularly saw each other at conferences, and kept the email flowing back and forth. He returned to Purdue and CERIAS several times to conduct seminars and joint research. He was generous with his time to the students and faculty who met with him.

Earlier this year, several of us put together a proposal to a funding agency. In it, we listed Gene as an outside expert to review and advise us on our work. We had room in the budget to pay him almost any fee he requested. But, when I spoke with him on the phone, he indicated he didn’t care if we paid more than his expenses — “I want to help CERIAS students and advance the field” was his rationale.

Since I learned of the news of his accident, and subsequent passing, I have provided some updates and notes to friends, colleagues, former students, and others via social media and email. So many people who knew Gene have responded with stories. There are three elements that are frequently repeated, and from my experience they help to define the man:

  • Equity. Gene treated everyone the same when he met them. It didn’t matter if someone was a CEO, Senator, freshman, or custodian — he treated them with respect and listened to what they might have to say.
  • Humor. Gene loved to laugh, and loved to make others laugh. He shared funny stories and odd things found on the WWW, and had wonderful stories that helped make others smile. And he smiled, a lot, and shared his joy.
  • Consideration. Gene was compassionate, thoughtful, and gentle. He would often inconvenience himself for others, without complaint. He loved his family and let his friends know they were special.   

Gene Schultz was a wonderful role model, mentor and friend for a huge number of people, including being a husband to a delightful wife for 36 years and father to three wonderful daughters. Our world is a little less bright with him gone, but so very much better that he was with us for the time he was here.

E. Eugene Schultz, Jr., 9/10/46–10/2/11. Requiescat in pace.

More than passive defense

I was watching a video today (more on that later) that reminded me of some history. It also brought to mind that too few defenders these days build forensics capture into their systems to help identify intruders. They also don't have active defenses, countermeasures and chaff in place to slow down attackers and provide more warning of problems.

Back in the late 1980s and early 1990s, I quietly built some counterhacking and beaconing tools that I installed in a "fake front" machine on our local network. People who tried to break into it might get surprises and leave me log info about what they were up to, and things they downloaded would not do what they thought or might beacon me to indicate where the code went. This was long before honeypots were formalized, and before firewalls were in common use. Some of my experiences contributed to me writing the first few papers on software forensics (now called digital forensics), development of Tripwire, and several of my Ph.D. students's theses topics.

I didn't talk about that work much at the time for a variety of reasons, but I did present some of the ideas to students in classes over the years, and in some closed workshops. Tsutomu Shimomura, Dan Farmer and I traded some of our ideas on occasion, along with a few others; a DOD service branch contracted with a few companies to actually built some tools from my ideas, a few of which made it into commercial products. (And no, I never got any royalties or credit for them, either, or for my early work on firewalls, or security scanning, or.... I didn't apply for patents or start companies, unfortunately. It's interesting to see how much of the commercial industry is based around things I pioneered.)

I now regret not having actually written about my ideas at the time, but I was asked by several groups (including a few government agencies) not to do so because it might give away clues to attackers. A few of those groups were funding my grad students, so I complied. You can find a few hints of the ideas in the various editions of Practical Unix & Internet Security because I shared several of the ideas with my co-author, Simson Garfinkel, who had a lot of clever ideas of his own. He went on to found a company, Sandstorm Enterprises, to build and market some professional tools in roughly this space; I was a minor partner in that company. (Simson has continued to have lots of other great ideas, and is now doing wonderful things with disk forensics as a faculty member at the Naval Postgraduate School.)

Some of the ideas we all had back then continue to be reinvented, along with many new and improved approaches. Back in the 1980s, all my tools were in Unix (SunOS, mostly), but now there are possible options in many other systems, with Windows and Linux being the main problems. Of course, back in the 1980s the Internet wasn't used for commerce, Linux hadn't been developed, and Windows was not the widespread issue it it now. There also wasn't a WWW with its problems of cross-site scripting and SQL injection. Nonetheless, there were plenty of attackers, and more than enough unfound bugs in the software to enable attacks.

For the sake of history, I thought I'd document a few of the things I remember as working well, so the memories aren't lost forever. These are all circa 1989-1993:

  • Everything was built on a decoy system. I had control over my own Sun workstation. I configured it to have 2 separate accounts, with one (named spaf) exported via NFS to the CS department Sun, Pyramid and Sequent machines. My students and fellow faculty could access this directory, and I populated it with contents to make it look real. If I wanted to share something with my classes, I'd copy it to this account. A second account (rspaf) was on a separate partition, not exported, and locked down. It was my real account and where I got email. No system had rsh access in, nor any other indication that it existed — the support staff knew it was there, but almost no one else. (For real paranoia, I kept copies of sensitive files and source code on a Mac running OS 7, and it was off the network.)
  • My original idea for Tripwire came several years before Gene Kim arrived as a student and we built the actual Tripwire tool. I built a watcher program for bogus, "bait" mail files and attractively-named source files that would never be touched in normal operation. When accessed, their file times changed. The watcher "noticed" and would start taking repeated snapshots of active network connections and running processes. When lsof was written (at Purdue), I included it in the logging process. After a few minutes, my watcher would freeze the active network connections.
  • As a matter of good security practice, I disabled most of the network services spawned at startup or by inetd. In their places, I put programs that mimicked their output behavior, but logged everything sent to them. Thus, if someone tried to rsh into my system, they'd get output claiming incorrect password for most accounts. For spaf and root it would randomly act like it was connecting but very slow or an error would be printed. At the same time, this would send me email and (later) a page. This would allow me to run other monitoring tools. The tcp wrappers system was later independently developed by Wietse Venema, and the twist option provided the same kind of functionality (still useful).
  • Some attackers would try to "slurp" up interesting sounding directories without looking at all the contents. For instance, a common tactic was to take any directory labeled "security" and the mail spool. For a while, attackers were particularly interested in getting my copy of the Morris Internet Worm, so any directory with "worm" in it would be copied out in its entirety. I built a small utility that would take any file (such as a mail file or binary) and make it HUGE using the Unix sparse file structure. On disk the files might only be a few thousand bytes long, and you could read it normally, but any copy program thought they were gigabytes in length. Thus, any attempt to copy them offsite would result in very long, uncompleted copies (which left network connections open to trace) and sometimes filled up the attackers' disks.
  • The previous idea was partially inspired by one of Tsutomu's tricks. The authd daemon, implementing the ident protocol (RFC 1413), was somewhat popular. It could be useful in a LAN, but in the Internet at large it was not trustworthy; to make this point, several admins had it always return some made-up names when queried. Tsutomu took this a step further and when a remote system connected to the ident port on his machine, it would get an unending string of random bits. Most authd clients would simply accept whatever they were given to write to the local log file. Connecting to Tsutomu's machine was therefore going to lead to a disk full problem. If the client ran as root, it would not be stopped by any limits or quotas — and it usually logged to the main system partition. Crash. As Tsutomu put it, "Be careful what you ask for or you may get more of it than you counted on."
  • I sprinkled altered program source on my fake account, including many that looked like hacking tools. The source code was always subtly altered to neuter the tool. For example, some Unix password cracking tools had permuted S-boxes so that, when presented with a password hash to attack, the program would either run a long time without a result, or would give a result that would not work on any target machine. I also had a partial copy of the Morris Worm there, with subtle permutations made to prevent it from spreading if compiled, and the beacon changed to ping me instead of Berkeley. (See my analysis paper if you don't get the reference.)
  • I also had some booby-trapped binaries with obfuscated content. One was named "God" and had embedded text strings for the usage message that implied that there were options to become root, take over the network, and other tasty actions. However, if run, it would disable all signals, and prompt the user with the name of every file in her home directory, one at a time. Any response to the prompt would result in "Deleted!" followed by the next prompt. Any attempt to stop the program would cause it to print "Entering automatic mode" where every half-second it would state that it was deleting the next file. Meanwhile, it would be sending me logging information about who was running it and where. If run on a Purdue machine, it didn't actually delete anything — I simply used it to see who was poking in my account and running things they didn't know anything about (usually, my students). It also gave them a good scare. If taken and run on a non-Purdue machine, well, it was not so benign.

There were many other tools and tripwires in place, of course, but the above were some of the most successful.

What does successful mean? Well, they helped me to identify several penetrations in progress, and get info on the attackers. I also identified a few new attacks, including the very subtle library substitution that was documented in @Large: The Strange Case of the World's Biggest Internet Invasion. The substitute with backdoor in place had the identical size, dates and simple checksum as the original so as to evade tools such as COPS and rdist. Most victims never knew they had been compromised. My system caught the attack in progress. I was able to share details with the Sun response team — and thereafter they started using MD5 checksums on their patch releases. That incident also inspired some of my design of Tripwire.

In another case, I collected data on some people who had broken into my system to steal the Morris Worm source code. The attacks were documented in the book Underground . The author, Suelette Dreyfus, assisted by Julian Assange (yes, the Wikileaks one), never bothered to contact me to verify what she wrote. The book suggests that my real account was compromised, and source code taken. However, it was the fake account, my security monitors froze the connection after a few minutes, and the software that was accessed was truncated and neutered. Furthermore, the flaws that were exploited to get in were not on my machine — they were on a machine operated by the CS staff.   (Dreyfuss got several other things wrong, but I'm not going to do a full critique.)

There were a half-dozen other incidents where I was able to identify new attacks (now known as zero-day exploits) and get the details to vendors. But after a while, interest dropped off in attacking my machine as new, more exciting opportunities for the kiddies came into play, such as botnets and DDOS attacks. And maybe the word spread that I didn't keep anything useful or interesting on my system. (I still don't.) It's also the case that I got much more interested in issues that don't involve the hands-on, bits & bytes parts of security — I'm now much more interested in fundamental science and policy aspects. I leave the hands-on aspects to the next generation. So, I'm not really a challenge now — especially as I do not administer my system anymore — it's done by staff.

I was reminded of all this when someone on Twitter posted the URL of a video taken at Notacon 2011 (Funnypots and Skiddy Baiting: Screwing with those that screw with you by Adrian "Iron Geek" Crenshaw). It is amusing and reminded me of the stories, above. It also showed that some of the same techniques we used 20 years ago are still applicable today.

Of course, that is also depressing. Now, nearly 20 years later, lots of things have changed but unfortunately, security is a bigger problem, and law enforcement is still struggling to keep up. Too many intrusions occur without being noticed, and too little information is available to track the perps.

There are a few takeaways from all the above that the reader is invited to consider:

  • Assume your systems will be penetrated if they are on the network. Things you don't control are likely to be broken. Therefore, plan ahead and keep the really sensitive items on a platform that is off the net. (If the RSA folks had done this, the SecurID breach might not have resulted in anything.)
  • Install localized tripwires and honeypots that can be monitored for evidence of intrusion. Don't depend on packaged solutions alone — they are known quantities that attackers can avoid or defeat.
  • Don't believe everything you read unless you know the story has been verified with original sources.
  • Consider encrypting or altering critical files so they aren't usable "as is" if taken.
  • Be sure you are proactively logging information that will be useful once you discover a compromise.

Also, you might watch Iron Geek's video to inspire some other ideas if you are interested in this general area — it's a good starting point. (And another, related and funny post on this general topic is here, but is possibly NSFW.)

In conclusion, I'll close with my 3 rules for successful security:

  1. Preparation in advance is always easier than clean up afterwards.
  2. Don't tell everything you know.

Bullies, Pirates and Lulz

Yet another breach of information has occurred, this time from the Arizona Department of Public Safety. A large amount of data about law enforcement operations was exposed, as was a considerable amount of personnel information. As someone who has been working in information security and the implications of technology for nearly 30 years, two things come to mind.

First, if a largely uncoordinated group could penetrate the systems and expose all this information, then so could a much more focused, well-financed, and malevolent group — and it would not likely result in postings picked up by the media. Attacks by narcotics cartels, organized crime, terrorists and intelligence agencies are obvious threats; we can only assume that some have already succeeded but not been recognized or publicized. And, as others are noting, this poses a real threat to the physical safety of innocent people. Yes, in any large law enforcement organization there are likely to be some who are corrupt (the claimed reason behind the attack), but that is not reason to attack them all. Some of the people they are arrayed against are far worse.

For example, there are thousands (perhaps tens of thousands) of kidnappings in Mexico for ransom, with many of the hostages killed rather than freed after payment. Take away effective law enforcement in Arizona, and those gangs would expand into the U.S. where they could demand bigger ransoms. The hackers, sitting behind a keyboard removed from gang and street violence, safe from forcible rape, and with enough education to be able to avoid most fraud, find it easy to cherry-pick some excesses to complain about. But the majority of people in the world do not have the education or means to enjoy that level of privileged safety. Compared to police in many third-world countries where extortion and bribes are always required for any protection at all, U.S. law enforcement is pretty good. (As is the UK, which has also recently been attacked.)

Ask yourself what the real agenda is of a group that has so far only attacked law enforcement in some of the more moderate countries, companies without political or criminal agendas, and showing a total disregard for collateral damage. Ask why these "heroes" aren't seeking to expose some of the data and names of the worst drug cartels, or working to end human trafficking and systematic rape in war zones, or exposing the corruption in some African, South American & Asian governments, or seeking to damage the governments of truly despotic regimes (e.g., North Korea, Myanmar), or interfering with China's online attacks against the Dalai Lama, or leaking memos about willful environmental damage and fraud by some large companies, or seeking to destroy extremist groups (such as al Qaida) that oppress woman and minorities and are seeking weapons of mass destruction.

Have you seen one report yet about anything like the above? None of those actions would necessarily be legal, but any one of them would certainly be a better match for the claimed motives. Instead, it is obvious that  these individuals and groups are displaying a significant political and moral bias — or blindness — they are ignoring the worst human rights offenders and criminals on the planet. It seems they are after the ego-boosting publicity, and concerned only with themselves. The claims of exposing evil is intended to fool the naive.

In particular, this most recent act of exposing the names and addresses of family members of law enforcement, most of whom are undoubtedly honest people trying to make the world a safer place, is not a matter of "Lulz" — it is potentially enabling extortion, kidnapping, and murder. The worst criminals, to whom money is more important than human life, are always seeking an opportunity to neutralize the police. Attacking family members of law enforcement is common in many countries, including Mexico, and this kind of exposure further enables it now in Arizona. The data breach is attacking some of the very people and organizations trying to counter the worst criminal and moral abuses that may occur, and worse, their families.

Claiming that, for instance, that the "War on Drugs" created the cartels and is morally equivalent (e.g., response #13 in this) is specious. Laws passed by elected representatives in the U.S. did not cause criminals in Mexico to kidnap poor people, force them to fight to the death for the criminals' amusement, and then force the survivors to act as expendable drug mules. The moral choices by criminals are exactly that — moral choices. The choice to kidnap, rape, or kill someone who objects to your criminal behavior is a choice with clear moral dimensions. So are the choices of various hackers who expose data and deface systems.

When I was growing up, I was the chubby kid with glasses. I didn't do well in sports, and I didn't fit in with the groups that were the "cool kids." I wasn't into drinking myself into a stupor, or taking drugs, or the random vandalism that seemed to be the pasttimes of those very same "cool kids." Instead, I was one of the ones who got harassed, threatened, my homework stolen, and laughed at. The ones who did it claimed that it was all in fun — this being long before the word "lulz" was invented. But it was clear they were being bullies, and they enjoyed being bullies. It didn't matter if anyone got hurt, it was purely for their selfish enjoyment. Most were cowards, too, because they would not do anything that might endanger them, and when they could, they did things anonymously. The only ones who thought it was funny were the other dysfunctional jerks. Does that sound familiar?

Twenty years ago, I was objecting to the press holding up virus authors as unappreciated geniuses. They were portrayed as heroes, performing experiments and striking blows against the evil computer companies then prominent in the field. Many in the public and press (and even in the computing industry) had a sort of romantic view of them — as modern, swashbuckling, electronic pirates, of the sorts seen in movies. Now we can see the billions of dollars in damage wrought by those "geniuses" and their successors with Zeus and Conficker and the rest. The only difference is of time and degree — the underlying damage and amoral concern for others is simply visible to more people now. (And, by the way, the pirates off Somalia and in the Caribbean, some of whom simply kill their victims to steal their property, are real pirates, not the fictional, romantic versions in film.)

The next time you see a news article about some group, by whatever name, exposing data from a gaming company or law enforcement agency, think about the real evil left untouched. Think about who might actually be hurt or suffer loss. Think about the perpetrators hiding their identities, attacking the poorly defended, and bragging about how wonderful and noble and clever they are. Then ask if you are someone cheering on the bully or concerned about who is really getting hurt. And ask how others, including the press, are reporting it. All are choices with moral components. What are yours?

Update: June 26

I have received several feedback comments to this (along with the hundreds of spam responses). Several were by people using anonymous addresses. We don't publish comments made anonymously or containing links to commercial sites. For this post, I am probably not going to pass through any rants, at least based on what I have seen. Furthermore, I don't have the time (or patience) to do a point-by-point commentary on the same things, again and again. However, I will make a few short comments on what I have received so far.

Several responses appear to be based on the assumption that I don't have knowledge or background to back up some of my statements. I'm not going to rebut those with a list of specifics. However, people who know what I've been doing over the few decades (or bothered to do a little research) — including work with companies, law enforcement, community groups, and government agencies — would hardly accuse me of being an observer with only an academic perspective.

A second common rant is that the government has done some bad things, or the police have done something corrupt, or corporations are greedy, and those facts somehow justify the behavior I described. Does the fact that a bully was knocked around by someone else and thus became a bully mean that if you are the victim, it's okay? If so, then the fact that the U.S. and U.K. have had terrorist attacks that have resulted in overly intrusive laws should make it all okay for you. After all, they had bad things happen to them, so their bad behavior is justified, correct? Wrong. That you became an abuser of others because you were harmed does not make it right. Furthermore, attacks such as the ones I discussed do nothing to fix those problems, but do have the potential to harm innocent parties as well as give ammunition to those who would pass greater restrictions on freedom. Based on statistics (for the US), a significant number of the people whining about government excess have not voted or bothered to make their opinions known to their elected representatives. The more people remain uninvolved, the more it looks like the population doesn't care or approves of those excesses, including sweetheart deals for corporations and invasions of privacy. Change is possible, but it is not going to occur by posting account details of people subscribed to Sony services, or giving out addresses and names of families of law enforcement officers, or defacing the NPR website. One deals with bullies by confronting them directly.

The third most common rant so far is to claim that it doesn't make any difference, for one reason or another: all the personal information is already out there on the net or will be soon, that the government (or government of another country) or criminals have already captured all that information, that it doesn't cost anything, security is illusory, et al. Again, this misses the point. Being a bully or vandal because you think it won't make any difference doesn't excuse the behavior. Because you believe that the effects of your behavior will happen anyhow is no reason to hasten those effects. If you believe otherwise, then consider: you are going to die someday, so it doesn't make a difference if you kill yourself, so you might as well do it now. Still there? Then I guess you agree that each act has implications that matter even if the end state is inevitable.

Fourth, some people claim that these attacks are a "favor" to the victims by showing them their vulnerabilities, or that the victims somehow deserved this because their defenses were weak. I addressed these claims in an article published in 2003. In short, blaming the victim is inappropriate. Yes, some may deserve some criticism for not having better defenses, but that does not justify an attack nor serve as a defense for the attackers. It is no favor either. If you are walking down a street at night and are assaulted by thugs who beat you with 2x4s and steal your money, you aren't likely to lie bleeding in the street saying to yourself "Gee, they did me a huge favor by showing I wasn't protected against a brutal assault. I guess I deserved that." Blaming the victim is done by the predators and their supporters to try to justify their behavior. And an intrusion or breach, committed without invitation or consent, is not a favor — it is a crime.

Fifth, if you support anarchy, then that is part of your moral choices. It does not invalidate what I wrote. I believe that doing things simply because they amuse you is a very selfish form of choice, and is the sort of reasoning many murderers, rapists, pedophiles and arsonists use to justify their actions. In an anarchy, they'd be able to indulge to their hearts content. Lotsa lulz. But don't complain if I and others don't share that view.

I am going to leave it here. As I said, I'm not interested in spending the next few weeks arguing on-line with people who are trying to justify behavior as bullies and vandals based on faulty assumptions.

U.S. Memorial Day Thoughts on Cyber War

We've been hearing about "cyber war" for some time now. It has been held out as an existential threat by some people, been the topic of scores of books, and led to the establishment of military organizations in several countries, including the U.S. Cybercommand, China's Blue Army, in the UK, and more. The definition of "cyberwar" has been somewhat imprecise, in part because some people trying to define it don't necessarily understand the full range of whatever "cyber" actually encompasses. It is also the case that definitions that include some current activities might imply that we're at war, and that has political ramifications that might be unpleasant to confront. The range of activities often discussed — including snooping, theft, espionage, and DDOS — don't really seem on the same level as a tank blitz or nuclear attack. After all, would an inability to shop online for a week really be a form of battle damage?

Of course, our whole definition of "war" is itself a little muddled. We have the World Wars, certainly. But from a strictly U.S. perspective, consider the Korean and Vietnam conflicts — were those wars? Or the Gulf War, Bosnia and Herzegovina, Iraq, Afghanistan — was the U.S. at war? And is that what is going on with Libya? In one sense, yes, because in each we employed military forces against a defined enemy. But how many of those had a formal declaration of war? And none were really existential threats that required the entire U.S. to be involved. War, historically, has usually been an issue of whether a state continued to exist under its current rule or not, and sometimes whether a significant percentage of the current population continued to live or not; some wars resulted in all the adult males being killed or enslaved, or whole populations slaughtered.

Then there is the War on Drugs, the War on Poverty, and most recently, our War on Terror (among others). In these conflicts we don't actually have a nation-state as an enemy, but we do have some defined objective requiring concerted, forceful action. (Of course we also have silly, demeaning uses of the term, such as the inane "War on Christmas.")

This can all lead to a certain confusion of definitions and roles. Prior to 9/11/2001, terrorists on U.S. soil were criminals. Whether it was Timothy McVeigh, Ramzi Yousef, Eric Rudolph, Ali Abu Kamal, or the ELF, civilian law enforcement, civilian courts, and civilian prisons were the mechanisms involved. Since 9/11, we have a strong contingent claiming that terrorism is now solely a military matter, that military courts must be used, and civilian prisons are somehow insufficient (although supermax prisons have held worse mass murders and gang members for years). Why? Because we are in a "war on terror." Further, administrative rules and laws were passed to classify a particular class of terrorists as belonging under military jurisdiction as enemy combatants and heated political debate occurs around any aspect of how to deal with these individuals.

This essay is not an attempt to sort out all those issues: I'm going after something else, but I needed to illustrate these few points, first. Above, I noted that "war" is a somewhat fuzzy term, as are the definitions of who might wage it. Next, let's consider how we have been preparing to react to cyber incidents.

With the fuzziness about defining "war," and the shifting boundaries of whether it is something confronted by law enforcement or the military, it is not surprising that "cyber war" has not really been well-defined. What has happened over the last decade is that stories of potential "Cyber Pearl Harbors" have been presented to legislators, coupled with demonstrations of vulnerabilities, to justify a massive investment in the military cyber arena — but not so much our civilian law enforcement. It is simple to scare policy makers with tales that the country might be destroyed by evil hackers working for another country's military; cyber crime does not make for as compelling a picture. The result has been massive buildup in offensive military tools, intelligence support, and personnel training to support military missions.

But that buildup does little to help civilian companies under attack within U.S. borders by unknown parties. So, we now have civilian companies turning to DHS for help rather than the FBI or another law enforcement agency. But the responsibility of DHS is to secure the .gov systems, so they are now turning to the military (NSA) because they don't have the infrastructure or expertise they need for even that. We are thus well down a path to turn over the bulk of our law enforcement in cyber to the military, with the specter of terrorists and cyber war held out by those who benefit from this situation continuing to push us in that direction. Soon we will have so much infrastructure built up we will not be able to afford to go back. The Posse Comitatus Act of 1878 was intended to keep the military from becoming a national police force, but this will further erode what is left of that law. Many people reading this will say "So what?" because we're now safer against a cyberwar attack as a result of this buildup — aren't we?

But here comes the problem, and the main point of this essay. We have a history of our military and leaders preparing to fight the last war. They are preparing for an offense that is unlikely to come at us the way they have portrayed. They are building a Maginot Line for a frontal attack that any intelligent adversary will never attempt.

In fact, we're under attack NOW. And we're losing. We're losing billions of $$ worth of intellectual property per year to foreign intelligence services, foreign competitors, and criminals, and we have been for years. U.S. companies and taxpayers are effectively paying for the R&D that is supporting huge amounts of foreign development. And we are also seeing billions of $$ of value being bled from the economy in credit card fraud, bank fraud and other kinds of fraud, including counterfeit pharma and counterfeit electronics sales, with all that money going to buy houses, cars, and consumer goods for people in Eastern Europe, China, Russia, and so on — in non-US economies. (And not only victims in the U.S., but Canada, the UK and a number of other countries.) It is a war of economic attrition and it is one that the DOD is never going to be in a position to fight because it has no kinetic component, no uniformed foe, no base of operations, and no centralized command. Once again, we have been preparing for the last war, so we are losing the current one. Most of our leaders don't even seem to recognize that we are in one. If we fall, it will not be by the swift stroke of the sword, but by the death of a thousand cuts.

If we are to have any hope of surviving, we have to completely change the way we look at this situation. Every intrusion, theft, or fraud should be reported, investigated and prosecuted (when possible). It should be tallied and brought to public attention, at least in aggregate so we understand the magnitude of what is going on. Right now, too much is hushed up or written off because each incident is too small to follow up, but the combined weight is staggering; for years I've been calling it "being pecked to death by ducks" because no single duck is lethal, but millions are. By letting so many incidents go, we encourage more and fund the development of yet new crime We need to refocus ourselves with a massive law enforcement effort, with a weighting towards local response, filtering up to Federal, not a Federal response directing local response. All those billions being dumped into the Federal contractors for cyber weapons should be directed to cyber law enforcement and investigation, to development of forensic tools, and to raising awareness at the local level. Your average business and consumer is going to be much more likely to install patches to protect against criminal behavior if encouraged by local authorities than told by someone in DC to install patches against some robotic threat from overseas. And we should adopt a get-tough policy at the diplomatic level to start demanding that countries that harbor criminals see some pushback from us; the new Federal international strategy on cyberspace is a good start on this.

I have described it to some people this way: our traditional DOD is structured to protect our borders and keep enemies from crossing those borders, or even getting near them. They are very, very good at that. In fact, they're so good, they may even stop an enemy from crossing their own borders to get here! However, the enemy we're engaged with is already here — is installed on millions of our computers and has thus subverted millions of citizens throughout the country without their knowing it....including some of the military. It is like the movie "The Puppet Masters." This can't be fought by the DOD — they aren't equipped to train their weapons inward. It requires an entirely different approach, but unfortunately, our leadership doesn't understand this, and the loudest voices right now are those of the lobbyists and members of the military who stand to benefit most in the short term by continuing the status quo, and by those who don't understand the magnitude of the situation.

Concomitant with this, within the next decade I fear that we will start seeing more of our best and brightest students from the US going to universities in India, China and other countries the way those countries' students have been coming to the US for years; I'm not the only one predicting this. Why? In the US we are shuttering university programs, decreasing funding, and shrinking campuses across the country, and politicians are vilifying K-12 teachers as if they are somehow part of the problem instead of being part of the cure. Meanwhile, in India, Russia, China, Korea, Taiwan and the Middle East they are opening major new universities and hiring away faculty from the US, Australia, the UK and elsewhere to staff their research labs, paying them extraordinary salaries and benefits and giving them access to modern resources. Major corporations have already located labs near those places because of cheap labor and are helping to subsidize the growth of the universities as are the national governments so as to obtain trained help. Our national policy of booting new PhDs & MS graduates who aren't citizens, and restricting so many high-tech jobs to US nationals only means that we train the world's best, then send them back to their own compete with us. The Rising Above the Gathering Storm and Rising Above the Gathering Storm, Revisited: Rapidly Approaching Category 5 reports nailed this, but were largely ignored by policymakers and certainly by the general public. Not only are we indirectly funding other countries' ascendency via their largely unhindered theft of our intellectual property and fraud, we are accelerating it by strangling our own intellectual capital and increasing theirs.

Everyone in IT and beyond should understand — fundamentally — that this is a new form of competition, of warfare (if we are to use that term). It is competition of the mind. It is information warfare in a much more fundamental sense than using information in support of kinetic weapons. It is employing information resources in a vast strategic way, across industries and generations to shape the future of nations. We do not have enough people who are able to think strategically, with that long a view and an understanding of the issues to see the threats, to see the trends, and to see the hard choices necessary to take a safer path. We, as a people, do not have the patience. Unfortunately, some of our enemies do.

What inspired the above, in part, is that this is Memorial Day Weekend. Many people will celebrate it as a holiday with picnics or trips, watch the Indy 500, and break out the summer clothes.

But Monday is a special day in the U.S. to remember the many men and women who sacrificed their lives in the service of the country, while serving in uniform. Whether in declared war or standing guard, whether grizzled veteran or new recruit, whether defending the bridge at Concord in 1775, or on patrol in Kandahar in 2011, those who did not return home deserve special thought from those who are here to enjoy this weekend. They had husbands, wives, children, siblings, parents and friends who treasured them. On Memorial Day, we should all treasure their memories as well.

And perhaps that is the one good thing about "Cyber War" — by nature, it is unlikely to add to the list of those we should remember on Memorial Day who are not here with us.

One of the best ways to honor their memory is to remain vigilant, and that is why I wrote the above.

Sony, Congress, The Media and Me


As a researcher and educator, I regularly follow many newsletters, blogs and newsfeeds on a near daily basis. Some items I bookmark for my classes and research, but most I simply read, note, and discard. I read many dozen such items per day -- sometimes as many as 100 when there is a lot happening and I have a backlog.

After news of the Sony incident broke on April 20th, I saw items about how some people knew about vulnerabilities in parts of the Sony network, and servers running old versions of the Apache webservers. Those postings had material similar to what was published in Wired on April 28th. To the best of my memory, at least one of those postings mentioned that some of these vulnerabilities were exposed to Sony in a mailing list or blog prior to the compromise. It may be that the reference was to the PSN webserver vulnerabilities, it may have been about the earlier flaw with the PS3 connecting to the PSN, or it may have been some other vulnerabilities...but I am pretty certain it was about the webservers. There was no discussion about how the breach occurred or whether the old software played a part in those breaches.

After reading these stories, I moved on to other issues. I was not a customer of Sony or the Playstation Network (PSN), and they have never had a relationship with our research group, so I had no reason to pay close attention to the story. Furthermore, we were approaching the end of the semester, I was teaching a graduate class and also preparing for two trips to workshops. Thus, I had several other things to occupy my time and attention, and this story was definitely not one of them.


On May 1st, in my capacity as chair of USACM, I received an invitation to appear at a House subcommittee meeting on the morning of May 4 on the issue of data breaches and privacy. This is a topic that has been one of USACM's main thrust areas, and is in my main areas of interest, so even though it was extremely short notice, I said yes. I spent the next 48 hours frantically trying to rearrange my teaching and administrative schedule at the university while also producing a formal written testimony to deliver to Congressional offices by a Tuesday noon deadline. This occurred, but with very little sleep over that two day period. Tuesday afternoon I had to drive to Indianapolis to fly to Washington for the Wednesday morning hearing.

Wednesday morning at 9:30am. the House Subcommittee on Commerce, Manufacturing and Trade of the House Energy and Commerce Committee held its hearing on “The Threat Of Data Theft To American Consumers.” I was the 4th witness in the panel (our written statements are available online). Three days of little sleep and too much coffee, plus the TV lights, combined to give me quite a headache, but that may not be evident if you watch the C-SPAN recording of the hearing.

In my written testimony I indicated that "...some news reports indicate that Sony was running software that was badly out of date, and had been warned about that risk." During questioning, I stated that I had read this on security lists that I normally read.

The fun begins

My comment that I had seen accounts about the server software being out of date and no firewalls was reported accurately by a few media outlets. However, a few others widely misquoted as me stating, authoritatively, that Sony was running outdated, unpatched software and implied that this was somehow the cause of the breach. Other news sources, blogs, and aggregators then picked up this version of the story and repeated it as their own, often with some other embellishment.

In only a few cases did a responsible journalist contact me to fact-check the story and determine what I had actually said, and what I actually knew.

I tried to correct one or two of the incorrect reports, but most occurred in places where there was no contact address for corrections, and they soon were spreading faster than I could possibly respond. I gave up.

Soon after the stories started circulating, I received email from Eugene Alvarado (he has given me permission to name him), who indicated that in early February he reported to Sony that there was widespread hacking of the network going on that was interfering with use of the network. He never got a response. So, at least one other person observed problems and reported them to Sony in advance of the breach in April. If the problem was significant, there may well have been others.

More recently, at least one "commentator" who "thinks" he is "clever" because he can put quotes around words like "security expert" to imply something meaningful about my expertise has posted a critique pointing out that some of Sony's servers were, in fact, up-to-date. However, at least one follow-up by someone else observes that other Sony servers (with interesting names such as "Login" and "Auth") were running software dated 2008. Thus, it may well be the case that some of the systems were current and some not. As we well know, it only takes one system out of rev or with a missing patch to serve as an entry point to a whole network.

Bottom Line

To this day, I have never heard from nor spoken with anyone at Sony. I have never bothered to probe or investigate their systems, because frankly, I don't care. Those issues are for others to determine and settle. What I think were the bigger issues to the story at the hearing were about having standard breach notifications and the 24 USACM privacy principles that were in my testimony. There are hundreds of other breaches occurring every year in the U.S. resulting in fraud, identity theft, and other crimes. Those are smaller than this incident with the PSN, but the victims are no less damaged. We need for the FTC and law enforcement to have more resources to help fight these problems, and we could definitely use some appropriate Federal legislation on minimum privacy protections and breach notifications. Read the 4 written testimonies from the hearing to get a sense of what is involved.

As to the spurious story, I tried to be clear in my testimony (written and oral) that I was simply repeating what I had read in some online newsgroups. I am really quite appalled at the number of places that have twisted that into a claim that Sony was somehow, definitely, running substandard software or systems. It is possible they were, but it is also possible they were running very well-maintained systems that fell prey to a clever attacker. That has happened to other high profile victims.

I certainly bear the good folks at Sony no ill will, and I hope they resolve the situation with the Playstation Network soon.

In the meantime, perhaps this can serve as an abject lesson about dealing with the media and bloggers — some of them want a sensational story, whether the facts support it or not, and you had better not get in the way!

Update 5/14

A recent article contains information indicating there was obvious evidence in Sony's logs of scanning activity starting March 3rd that should have been noticed.

Update 5/18

Another recent article provides more information about the scanning activity preceding the breach, and suggests that it occurred from more than one source.

Update 6/4

Here is a very nice timeline and summary of Sony security incidents that seem to keep on coming.

A Cautionary Incident

Recently, Amazon's cloud service failed for several customers, and has not come back fully for well over 24 hours. As of the time I write this, Amazon has not commented as to what caused the problem, why it took so long to fix, or how many customers it affected.

It seems a client of Amazon was not able to contact support, and posted in a support forum under the heading "Life of our patients is at stake - I am desperately asking you to contact." The body of the message was that "We are a monitoring company and are monitoring hundreds of cardiac patients at home. We were unable to see their ECG signals"

What ensued was a back-and-forth with others incredulous that such a service would not have a defined disaster plan and alternate servers defined, with the original poster trying to defend his/her position. At the end, as the Amazon service slowly came back, the original poster seemed to back off from the original claim, which implies either an attempt to evade further scolding (and investigation), or that the original posting was a huge exaggeration to get attention. Either way, the prospect of a mission critical system depending on the service was certainly disconcerting.

Personnel from Amazon apparently never contacted the original poster, despite that company having a Premium service contract.

25 or so years ago, Brian Reid defined a distributed system as " where I can't get my work done because a computer I never heard of is down." (Since then I've seen this attributed to Leslie Lamport, but at the time heard it attributed to Reid.) It appears that "The Cloud" is simply today's buzzword for a distributed system. There have been some changes to hardware and software, but the general idea is the same — with many of the limitations and cautions attendant thereto, plus some new ones unique to it. Those who extol its benefits (viz., cost) without understanding the many risks involved (security, privacy, continuity, legal, etc.) may find themselves someday making similar postings to support fora — as well as "position wanted" sites.

The full thread is available here.