[If you want to skip my recollection and jump right to the announcement that is the reason for this post, go here.]

Back in about 1990 I was approached by an eager undergrad who had recently come to Purdue University. A mutual acquaintance (hi, Rob!) had recommended that the student connect with me for a project. We chatted for a bit and at first it wasn't clear exactly what he might be able to do. He had some experience coding, and was working in the campus computing center, but had no background in the more advanced topics in computing (yet).

Well, it just so happened that a few months earlier, my honeypot Sun workstation had recorded a very sophisticated (for the time) attack, which resulted in an altered shared library with a back door in place. The attack was stealthy, and the new library had the same dates, size and simple hash value as the original. (The attack was part of a larger series of attacks, and eventually documented in "@Large: The Strange Case of the World's Biggest Internet Invasion" (David H. Freedman, Charles C. Mann .)

I had recently been studying message digest functions and had a hunch that they might provide better protection for systems than a simple ls -1 | diff - old comparison. However, I wanted to get some operational sense about the potential for collision in the digests. So, I tasked the student with devising some tests to run many files through a version of the digest to see if there were any collisions. He wrote a program to generate some random files, and all seemed okay based on that. I suggested he look for a different collection -- something larger. He took my advice a little too much to heart. It seems he had a part time job running backup jobs on the main shared instructional computers at the campus computing center. He decided to run the program over the entire file system to look for duplicates. Which he did one night after backups were complete.

The next day (as I recall) he reported to me that there were no unexpected collisions over many hundreds of thousands of files. That was a good result!

The bad result was that running his program over the file system had resulted in a change of the access time of every file on the system, so the backups the next evening vastly exceeded the existing tape archive and all the spares! This led directly to the student having a (pointed) conversation with the director of the center, and thereafter, unemployment. I couldn't leave him in that position mid-semester so I found a little money and hired him as an assistant. I them put him to work coding up my idea, about how to use the message digests to detect changes and intrusions into a computing system. Over the next year, he would code up my design, and we would do repeated, modified "cleanroom" tests of his software. Only when they all passed, did we release the first version of Tripwire.

That is how I met Gene Kim .

Gene went on to grad school elsewhere, then a start-up, and finally got the idea to start the commercial version of Tripwire with Wyatt Starnes; Gene served as CTO, Wyatt as CEO. Their subsequent hard work, and that of hundreds of others who have worked at the company over the years, resulted in great success: the software has become one of the most widely used change detection & IDS systems in history, as well as inspiring many other products.

Gene became more active in the security scene, and was especially intrigued with issues of configuration management, compliance, and overall system visibility, and with their connections to security and correctness. Over the years he spoken with thousands of customers and experts in the industry, and heard both best-practice and horror stories involving integrity management, version control, and security. This led to projects, workshops, panel sessions, and eventually to his lead authorship of "Visible Ops Security: Achieving Common Security and IT Operations Objectives in 4 Practical Steps" (Gene Kim, Paul Love, George Spafford) , and some other, related works.

His passion for the topic only grew. He was involved in standards organizations, won several awards for his work, and even helped get the B-sides conferences into a going concern. A few years ago, he left his position at Tripwire to begin work on a book to better convey the principles he knew could make a huge difference in how IT is managed in organizations big and small.

I read an early draft of that book a little over a year ago (late 2011), It was a bit rough -- Gene is bright and enthusiastic, but was not quite writing to the level of J.K. Rowling or Stephen King. Still, it was clear that he had the framework of a reasonable narrative to present major points about good, bad, and excellent ways to manage IT operations, and how to transform them for the better. He then obtained input from a number of people (I think he ignored mine), added some co-authors, and performed a major rewrite of the book. The result is a much more readable and enjoyable story -- a cross between a case study and a detective novel, with a dash of H. P. Lovecraft and DevOps thrown in.

The official launch date of the book, "The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win" (Gene Kim, Kevin Behr, George Spafford), is Tuesday, January 15, but you can preorder it before then on (at least) Amazon.

The book is worth reading if you have a stake in operations at a business using IT. If you are a C-level executive, you should most definitely take time to read the book. Consultants, auditors, designers, educators...there are some concepts in there for everyone.

But you don't have to take only my word for it -- see the effusive praise of tech luminaries who have read the book .

So, Spaf sez, get a copy and see how you can transform your enterprise for the better.

(Oh, and I have never met the George Spafford who is a coauthor of the book. We are undoubtedly distant cousins, especially given how uncommon the name is. That Gene would work with two different Spaffords over the years is one of those cosmic quirks Vonnegut might write about. But Gene isn't Vonnegut, either.

So, as a postscript.... I've obviously known Gene for over 20 years, and am very fond of him, as well as happy for his continuing success. However, I have had a long history of kidding him, which he has taken with incredible good nature. I am sure he's saving it all up to get me some day....

When Gene and his publicist asked if I could provide some quotes to use for his book, I wrote the first of the following. For some reason, this never made it onto the WWW site . So, they asked me again, and I wrote the second of the following -- which they also did not use.

So, not to let a good review (or two) go to waste, I have included them here for you. If nothing else, it should convince others not to ask me for a book review.

But, despite the snark (who, me?) of these gag reviews, I definitely suggest you get a copy of the book and think about the ideas expressed therein. Gene and his coauthors have really produced a valuable, readable work that will inform -- and maybe scare -- anyone involved with organizational IT.

Take 1:

Based on my long experience in academia, I can say with conviction that this is truly a book, composed of an impressive collection of words, some of which exist in human languages. Although arranged in a largely random order, there are a few sentences that appear to have both verbs and nouns. I advise that you immediately buy several copies and send them to people -- especially people you don't like -- and know that your purchase is helping keep some out of the hands of the unwary and potentially innocent. Under no circumstances, however, should you read the book before driving or operating heavy machinery. This work should convince you that Gene Kim is a visionary (assuming that your definition of "vision" includes "drug-induced hallucination").

Take 2:

I picked up this new book -- The Phoenix Project , by Gene Kim, et al. -- and could not put it down. You probably hear people say that about books in which they are engrossed. But I mean this literally: I happened to be reading it on my Kindle while repairing some holiday ornaments with superglue. You might say that the book stuck with me for a while.

There are people who will tell you that Gene Kim is a great author and raconteur. Those people, of course, are either trapped in Mr. Kim's employ or they drink heavily. Actually, one of those conditions invariably leads to the other, along with uncontrollable weeping, and the anguished rending of garments. Notwithstanding that, Mr. Kim's latest assault on les belles-lettres does indeed prompt this reviewer to some praise: I have not had to charge my health spending account for a zolpidem refill since I received the advance copy of the book! (Although it may be why I now need risperidone.)

I must warn you, gentle reader, that despite my steadfast sufferance in reading, I never encountered any mention of an actual Phoenix. I skipped ahead to the end, and there was no mention there, either. Neither did I notice any discussion of a massive conflagration nor of Arizona, either of which might have supported the reference to Phoenix . This is perhaps not so puzzling when one recollects that Mr. Kim's train of thought often careens off the rails with any random, transient manifestation corresponding to the meme "Ooh, a squirrel!" Rather, this work is more emblematic of a bus of thought, although it is the short bus, at that.

Despite my personal trauma, I must declare the book as a fine yarn: not because it is unduly tangled (it is), but because my kitten batted it about for hours with the evident joy usually limited to a skein of fine yarn. I have found over time it is wise not to argue with cats or women. Therefore, appease your inner kitten and purchase a copy of the book. Gene Kim's court-appointed guardians will thank you. Probably.

(Congratulations Gene, Kevin and George!)

I was watching a video today (more on that later) that reminded me of some history. It also brought to mind that too few defenders these days build forensics capture into their systems to help identify intruders. They also don't have active defenses, countermeasures and chaff in place to slow down attackers and provide more warning of problems.

Back in the late 1980s and early 1990s, I quietly built some counterhacking and beaconing tools that I installed in a "fake front" machine on our local network. People who tried to break into it might get surprises and leave me log info about what they were up to, and things they downloaded would not do what they thought or might beacon me to indicate where the code went. This was long before honeypots were formalized, and before firewalls were in common use. Some of my experiences contributed to me writing the first few papers on software forensics (now called digital forensics), development of Tripwire, and several of my Ph.D. students's theses topics.

I didn't talk about that work much at the time for a variety of reasons, but I did present some of the ideas to students in classes over the years, and in some closed workshops. Tsutomu Shimomura, Dan Farmer and I traded some of our ideas on occasion, along with a few others; a DOD service branch contracted with a few companies to actually built some tools from my ideas, a few of which made it into commercial products. (And no, I never got any royalties or credit for them, either, or for my early work on firewalls, or security scanning, or.... I didn't apply for patents or start companies, unfortunately. It's interesting to see how much of the commercial industry is based around things I pioneered.)

I now regret not having actually written about my ideas at the time, but I was asked by several groups (including a few government agencies) not to do so because it might give away clues to attackers. A few of those groups were funding my grad students, so I complied. You can find a few hints of the ideas in the various editions of Practical Unix & Internet Security because I shared several of the ideas with my co-author, Simson Garfinkel, who had a lot of clever ideas of his own. He went on to found a company, Sandstorm Enterprises, to build and market some professional tools in roughly this space; I was a minor partner in that company. (Simson has continued to have lots of other great ideas, and is now doing wonderful things with disk forensics as a faculty member at the Naval Postgraduate School.)

Some of the ideas we all had back then continue to be reinvented, along with many new and improved approaches. Back in the 1980s, all my tools were in Unix (SunOS, mostly), but now there are possible options in many other systems, with Windows and Linux being the main problems. Of course, back in the 1980s the Internet wasn't used for commerce, Linux hadn't been developed, and Windows was not the widespread issue it it now. There also wasn't a WWW with its problems of cross-site scripting and SQL injection. Nonetheless, there were plenty of attackers, and more than enough unfound bugs in the software to enable attacks.

For the sake of history, I thought I'd document a few of the things I remember as working well, so the memories aren't lost forever. These are all circa 1989-1993:

• Everything was built on a decoy system. I had control over my own Sun workstation. I configured it to have 2 separate accounts, with one (named spaf) exported via NFS to the CS department Sun, Pyramid and Sequent machines. My students and fellow faculty could access this directory, and I populated it with contents to make it look real. If I wanted to share something with my classes, I'd copy it to this account. A second account (rspaf) was on a separate partition, not exported, and locked down. It was my real account and where I got email. No system had rsh access in, nor any other indication that it existed — the support staff knew it was there, but almost no one else. (For real paranoia, I kept copies of sensitive files and source code on a Mac running OS 7, and it was off the network.)
• My original idea for Tripwire came several years before Gene Kim arrived as a student and we built the actual Tripwire tool. I built a watcher program for bogus, "bait" mail files and attractively-named source files that would never be touched in normal operation. When accessed, their file times changed. The watcher "noticed" and would start taking repeated snapshots of active network connections and running processes. When lsof was written (at Purdue), I included it in the logging process. After a few minutes, my watcher would freeze the active network connections.
• As a matter of good security practice, I disabled most of the network services spawned at startup or by inetd. In their places, I put programs that mimicked their output behavior, but logged everything sent to them. Thus, if someone tried to rsh into my system, they'd get output claiming incorrect password for most accounts. For spaf and root it would randomly act like it was connecting but very slow or an error would be printed. At the same time, this would send me email and (later) a page. This would allow me to run other monitoring tools. The tcp wrappers system was later independently developed by Wietse Venema, and the twist option provided the same kind of functionality (still useful).
• Some attackers would try to "slurp" up interesting sounding directories without looking at all the contents. For instance, a common tactic was to take any directory labeled "security" and the mail spool. For a while, attackers were particularly interested in getting my copy of the Morris Internet Worm, so any directory with "worm" in it would be copied out in its entirety. I built a small utility that would take any file (such as a mail file or binary) and make it HUGE using the Unix sparse file structure. On disk the files might only be a few thousand bytes long, and you could read it normally, but any copy program thought they were gigabytes in length. Thus, any attempt to copy them offsite would result in very long, uncompleted copies (which left network connections open to trace) and sometimes filled up the attackers' disks.
• The previous idea was partially inspired by one of Tsutomu's tricks. The authd daemon, implementing the ident protocol (RFC 1413), was somewhat popular. It could be useful in a LAN, but in the Internet at large it was not trustworthy; to make this point, several admins had it always return some made-up names when queried. Tsutomu took this a step further and when a remote system connected to the ident port on his machine, it would get an unending string of random bits. Most authd clients would simply accept whatever they were given to write to the local log file. Connecting to Tsutomu's machine was therefore going to lead to a disk full problem. If the client ran as root, it would not be stopped by any limits or quotas — and it usually logged to the main system partition. Crash. As Tsutomu put it, "Be careful what you ask for or you may get more of it than you counted on."
• I sprinkled altered program source on my fake account, including many that looked like hacking tools. The source code was always subtly altered to neuter the tool. For example, some Unix password cracking tools had permuted S-boxes so that, when presented with a password hash to attack, the program would either run a long time without a result, or would give a result that would not work on any target machine. I also had a partial copy of the Morris Worm there, with subtle permutations made to prevent it from spreading if compiled, and the beacon changed to ping me instead of Berkeley. (See my analysis paper if you don't get the reference.)
• I also had some booby-trapped binaries with obfuscated content. One was named "God" and had embedded text strings for the usage message that implied that there were options to become root, take over the network, and other tasty actions. However, if run, it would disable all signals, and prompt the user with the name of every file in her home directory, one at a time. Any response to the prompt would result in "Deleted!" followed by the next prompt. Any attempt to stop the program would cause it to print "Entering automatic mode" where every half-second it would state that it was deleting the next file. Meanwhile, it would be sending me logging information about who was running it and where. If run on a Purdue machine, it didn't actually delete anything — I simply used it to see who was poking in my account and running things they didn't know anything about (usually, my students). It also gave them a good scare. If taken and run on a non-Purdue machine, well, it was not so benign.

There were many other tools and tripwires in place, of course, but the above were some of the most successful.

What does successful mean? Well, they helped me to identify several penetrations in progress, and get info on the attackers. I also identified a few new attacks, including the very subtle library substitution that was documented in @Large: The Strange Case of the World's Biggest Internet Invasion. The substitute with backdoor in place had the identical size, dates and simple checksum as the original so as to evade tools such as COPS and rdist. Most victims never knew they had been compromised. My system caught the attack in progress. I was able to share details with the Sun response team — and thereafter they started using MD5 checksums on their patch releases. That incident also inspired some of my design of Tripwire.

In another case, I collected data on some people who had broken into my system to steal the Morris Worm source code. The attacks were documented in the book Underground . The author, Suelette Dreyfus, assisted by Julian Assange (yes, the Wikileaks one), never bothered to contact me to verify what she wrote. The book suggests that my real account was compromised, and source code taken. However, it was the fake account, my security monitors froze the connection after a few minutes, and the software that was accessed was truncated and neutered. Furthermore, the flaws that were exploited to get in were not on my machine — they were on a machine operated by the CS staff.   (Dreyfuss got several other things wrong, but I'm not going to do a full critique.)

There were a half-dozen other incidents where I was able to identify new attacks (now known as zero-day exploits) and get the details to vendors. But after a while, interest dropped off in attacking my machine as new, more exciting opportunities for the kiddies came into play, such as botnets and DDOS attacks. And maybe the word spread that I didn't keep anything useful or interesting on my system. (I still don't.) It's also the case that I got much more interested in issues that don't involve the hands-on, bits & bytes parts of security — I'm now much more interested in fundamental science and policy aspects. I leave the hands-on aspects to the next generation. So, I'm not really a challenge now — especially as I do not administer my system anymore — it's done by staff.

I was reminded of all this when someone on Twitter posted the URL of a video taken at Notacon 2011 (Funnypots and Skiddy Baiting: Screwing with those that screw with you by Adrian "Iron Geek" Crenshaw). It is amusing and reminded me of the stories, above. It also showed that some of the same techniques we used 20 years ago are still applicable today.

Of course, that is also depressing. Now, nearly 20 years later, lots of things have changed but unfortunately, security is a bigger problem, and law enforcement is still struggling to keep up. Too many intrusions occur without being noticed, and too little information is available to track the perps.

There are a few takeaways from all the above that the reader is invited to consider:

• Assume your systems will be penetrated if they are on the network. Things you don't control are likely to be broken. Therefore, plan ahead and keep the really sensitive items on a platform that is off the net. (If the RSA folks had done this, the SecurID breach might not have resulted in anything.)
• Install localized tripwires and honeypots that can be monitored for evidence of intrusion. Don't depend on packaged solutions alone — they are known quantities that attackers can avoid or defeat.
• Don't believe everything you read unless you know the story has been verified with original sources.
• Consider encrypting or altering critical files so they aren't usable "as is" if taken.
• Be sure you are proactively logging information that will be useful once you discover a compromise.

Also, you might watch Iron Geek's video to inspire some other ideas if you are interested in this general area — it's a good starting point. (And another, related and funny post on this general topic is here, but is possibly NSFW.)

In conclusion, I'll close with my 3 rules for successful security:

1. Preparation in advance is always easier than clean up afterwards.
2. Don't tell everything you know.

Yet another breach of information has occurred, this time from the Arizona Department of Public Safety. A large amount of data about law enforcement operations was exposed, as was a considerable amount of personnel information. As someone who has been working in information security and the implications of technology for nearly 30 years, two things come to mind.

First, if a largely uncoordinated group could penetrate the systems and expose all this information, then so could a much more focused, well-financed, and malevolent group — and it would not likely result in postings picked up by the media. Attacks by narcotics cartels, organized crime, terrorists and intelligence agencies are obvious threats; we can only assume that some have already succeeded but not been recognized or publicized. And, as others are noting, this poses a real threat to the physical safety of innocent people. Yes, in any large law enforcement organization there are likely to be some who are corrupt (the claimed reason behind the attack), but that is not reason to attack them all. Some of the people they are arrayed against are far worse.

For example, there are thousands (perhaps tens of thousands) of kidnappings in Mexico for ransom, with many of the hostages killed rather than freed after payment. Take away effective law enforcement in Arizona, and those gangs would expand into the U.S. where they could demand bigger ransoms. The hackers, sitting behind a keyboard removed from gang and street violence, safe from forcible rape, and with enough education to be able to avoid most fraud, find it easy to cherry-pick some excesses to complain about. But the majority of people in the world do not have the education or means to enjoy that level of privileged safety. Compared to police in many third-world countries where extortion and bribes are always required for any protection at all, U.S. law enforcement is pretty good. (As is the UK, which has also recently been attacked.)

Ask yourself what the real agenda is of a group that has so far only attacked law enforcement in some of the more moderate countries, companies without political or criminal agendas, and showing a total disregard for collateral damage. Ask why these "heroes" aren't seeking to expose some of the data and names of the worst drug cartels, or working to end human trafficking and systematic rape in war zones, or exposing the corruption in some African, South American & Asian governments, or seeking to damage the governments of truly despotic regimes (e.g., North Korea, Myanmar), or interfering with China's online attacks against the Dalai Lama, or leaking memos about willful environmental damage and fraud by some large companies, or seeking to destroy extremist groups (such as al Qaida) that oppress woman and minorities and are seeking weapons of mass destruction.

Have you seen one report yet about anything like the above? None of those actions would necessarily be legal, but any one of them would certainly be a better match for the claimed motives. Instead, it is obvious that  these individuals and groups are displaying a significant political and moral bias — or blindness — they are ignoring the worst human rights offenders and criminals on the planet. It seems they are after the ego-boosting publicity, and concerned only with themselves. The claims of exposing evil is intended to fool the naive.

In particular, this most recent act of exposing the names and addresses of family members of law enforcement, most of whom are undoubtedly honest people trying to make the world a safer place, is not a matter of "Lulz" — it is potentially enabling extortion, kidnapping, and murder. The worst criminals, to whom money is more important than human life, are always seeking an opportunity to neutralize the police. Attacking family members of law enforcement is common in many countries, including Mexico, and this kind of exposure further enables it now in Arizona. The data breach is attacking some of the very people and organizations trying to counter the worst criminal and moral abuses that may occur, and worse, their families.

Claiming that, for instance, that the "War on Drugs" created the cartels and is morally equivalent (e.g., response #13 in this) is specious. Laws passed by elected representatives in the U.S. did not cause criminals in Mexico to kidnap poor people, force them to fight to the death for the criminals' amusement, and then force the survivors to act as expendable drug mules. The moral choices by criminals are exactly that — moral choices. The choice to kidnap, rape, or kill someone who objects to your criminal behavior is a choice with clear moral dimensions. So are the choices of various hackers who expose data and deface systems.

When I was growing up, I was the chubby kid with glasses. I didn't do well in sports, and I didn't fit in with the groups that were the "cool kids." I wasn't into drinking myself into a stupor, or taking drugs, or the random vandalism that seemed to be the pasttimes of those very same "cool kids." Instead, I was one of the ones who got harassed, threatened, my homework stolen, and laughed at. The ones who did it claimed that it was all in fun — this being long before the word "lulz" was invented. But it was clear they were being bullies, and they enjoyed being bullies. It didn't matter if anyone got hurt, it was purely for their selfish enjoyment. Most were cowards, too, because they would not do anything that might endanger them, and when they could, they did things anonymously. The only ones who thought it was funny were the other dysfunctional jerks. Does that sound familiar?

Twenty years ago, I was objecting to the press holding up virus authors as unappreciated geniuses. They were portrayed as heroes, performing experiments and striking blows against the evil computer companies then prominent in the field. Many in the public and press (and even in the computing industry) had a sort of romantic view of them — as modern, swashbuckling, electronic pirates, of the sorts seen in movies. Now we can see the billions of dollars in damage wrought by those "geniuses" and their successors with Zeus and Conficker and the rest. The only difference is of time and degree — the underlying damage and amoral concern for others is simply visible to more people now. (And, by the way, the pirates off Somalia and in the Caribbean, some of whom simply kill their victims to steal their property, are real pirates, not the fictional, romantic versions in film.)

The next time you see a news article about some group, by whatever name, exposing data from a gaming company or law enforcement agency, think about the real evil left untouched. Think about who might actually be hurt or suffer loss. Think about the perpetrators hiding their identities, attacking the poorly defended, and bragging about how wonderful and noble and clever they are. Then ask if you are someone cheering on the bully or concerned about who is really getting hurt. And ask how others, including the press, are reporting it. All are choices with moral components. What are yours?

Update: June 26

I have received several feedback comments to this (along with the hundreds of spam responses). Several were by people using anonymous addresses. We don't publish comments made anonymously or containing links to commercial sites. For this post, I am probably not going to pass through any rants, at least based on what I have seen. Furthermore, I don't have the time (or patience) to do a point-by-point commentary on the same things, again and again. However, I will make a few short comments on what I have received so far.

Several responses appear to be based on the assumption that I don't have knowledge or background to back up some of my statements. I'm not going to rebut those with a list of specifics. However, people who know what I've been doing over the few decades (or bothered to do a little research) — including work with companies, law enforcement, community groups, and government agencies — would hardly accuse me of being an observer with only an academic perspective.

A second common rant is that the government has done some bad things, or the police have done something corrupt, or corporations are greedy, and those facts somehow justify the behavior I described. Does the fact that a bully was knocked around by someone else and thus became a bully mean that if you are the victim, it's okay? If so, then the fact that the U.S. and U.K. have had terrorist attacks that have resulted in overly intrusive laws should make it all okay for you. After all, they had bad things happen to them, so their bad behavior is justified, correct? Wrong. That you became an abuser of others because you were harmed does not make it right. Furthermore, attacks such as the ones I discussed do nothing to fix those problems, but do have the potential to harm innocent parties as well as give ammunition to those who would pass greater restrictions on freedom. Based on statistics (for the US), a significant number of the people whining about government excess have not voted or bothered to make their opinions known to their elected representatives. The more people remain uninvolved, the more it looks like the population doesn't care or approves of those excesses, including sweetheart deals for corporations and invasions of privacy. Change is possible, but it is not going to occur by posting account details of people subscribed to Sony services, or giving out addresses and names of families of law enforcement officers, or defacing the NPR website. One deals with bullies by confronting them directly.

The third most common rant so far is to claim that it doesn't make any difference, for one reason or another: all the personal information is already out there on the net or will be soon, that the government (or government of another country) or criminals have already captured all that information, that it doesn't cost anything, security is illusory, et al. Again, this misses the point. Being a bully or vandal because you think it won't make any difference doesn't excuse the behavior. Because you believe that the effects of your behavior will happen anyhow is no reason to hasten those effects. If you believe otherwise, then consider: you are going to die someday, so it doesn't make a difference if you kill yourself, so you might as well do it now. Still there? Then I guess you agree that each act has implications that matter even if the end state is inevitable.

Fourth, some people claim that these attacks are a "favor" to the victims by showing them their vulnerabilities, or that the victims somehow deserved this because their defenses were weak. I addressed these claims in an article published in 2003. In short, blaming the victim is inappropriate. Yes, some may deserve some criticism for not having better defenses, but that does not justify an attack nor serve as a defense for the attackers. It is no favor either. If you are walking down a street at night and are assaulted by thugs who beat you with 2x4s and steal your money, you aren't likely to lie bleeding in the street saying to yourself "Gee, they did me a huge favor by showing I wasn't protected against a brutal assault. I guess I deserved that." Blaming the victim is done by the predators and their supporters to try to justify their behavior. And an intrusion or breach, committed without invitation or consent, is not a favor — it is a crime.

Fifth, if you support anarchy, then that is part of your moral choices. It does not invalidate what I wrote. I believe that doing things simply because they amuse you is a very selfish form of choice, and is the sort of reasoning many murderers, rapists, pedophiles and arsonists use to justify their actions. In an anarchy, they'd be able to indulge to their hearts content. Lotsa lulz. But don't complain if I and others don't share that view.

I am going to leave it here. As I said, I'm not interested in spending the next few weeks arguing on-line with people who are trying to justify behavior as bullies and vandals based on faulty assumptions.

Recently, Amazon's cloud service failed for several customers, and has not come back fully for well over 24 hours. As of the time I write this, Amazon has not commented as to what caused the problem, why it took so long to fix, or how many customers it affected.

It seems a client of Amazon was not able to contact support, and posted in a support forum under the heading "Life of our patients is at stake - I am desperately asking you to contact." The body of the message was that "We are a monitoring company and are monitoring hundreds of cardiac patients at home. We were unable to see their ECG signals"

What ensued was a back-and-forth with others incredulous that such a service would not have a defined disaster plan and alternate servers defined, with the original poster trying to defend his/her position. At the end, as the Amazon service slowly came back, the original poster seemed to back off from the original claim, which implies either an attempt to evade further scolding (and investigation), or that the original posting was a huge exaggeration to get attention. Either way, the prospect of a mission critical system depending on the service was certainly disconcerting.

Personnel from Amazon apparently never contacted the original poster, despite that company having a Premium service contract.

25 or so years ago, Brian Reid defined a distributed system as "...one where I can't get my work done because a computer I never heard of is down." (Since then I've seen this attributed to Leslie Lamport, but at the time heard it attributed to Reid.) It appears that "The Cloud" is simply today's buzzword for a distributed system. There have been some changes to hardware and software, but the general idea is the same — with many of the limitations and cautions attendant thereto, plus some new ones unique to it. Those who extol its benefits (viz., cost) without understanding the many risks involved (security, privacy, continuity, legal, etc.) may find themselves someday making similar postings to support fora — as well as "position wanted" sites.

The full thread is available here.

Yes, I have been quiet (here) over the last few months, and have a number of things to comment on. This hiatus is partly because of schedule, partly because I had my laptop stolen, and partly health reasons. However, I'm going to try to start back into adding some items here that might be of interest.

To start, here is one item that I found while cleaning out some old disks: a briefing I gave at the NSA Research division in 1994. I then gave it, with minor updates, to the DOD CIO Council (or whatever their name was/is -- the CNSS group?), the Federal Infosec Research Council, and the Criticial Infrastructure Commission in 1998. In it, I spoke to what I saw as the biggest challenges in protecting government systems, and what were major research challenges of the time.

I have no software to read the 1994 version of the talk any more, but the 1998 version was successfully imported into Powerpoint. I cleaned up the fonts and gave it a different background (the old version was fugly) and that prettier version is available for download. (Interesting that back then it was "state of the art"

I won't editorialize on the content slide by slide, other than to note that I could give this same talk today and it would still be current. You will note that many of the research agenda items have been echoed in other reports over the succeeding years. I won't claim credit for that, but there may have been some influences from my work.

Nearly 16 years have passed by, largely wasted, because the attitude within government is still largely one of "with enough funding we can successfully patch the problems." But as I've quoted in other places, insanity is doing the same thing over and over again and expecting different results. So long as we believe that simple incremental changes to the existing infrastructure, and simply adding more funding for individual projects, is going to solve the problems then the problems will not get addressed -- they will get worse. It is insane to think that pouring ever more funding into attempts to "fix" current systems is going to succeed. Some of it may help, and much of it may produce some good research, but overall it will not make our infrastructure as safe as it should be.

Yesterday, Admiral (ret) Mike McConnell, the former Director of National Intelligence in the US, said in a Senate committee hearing that if there were a cyberwar today, the US would lose. That may not be quite the correct way of putting it, but we certainly would not come out of it unharmed and able to claim victory. What's more, any significant attack on the cyberinfrastructure of the US would have global repercussions because of the effects on the world's economy, communications, trade, and technology that are connected by the cyber infrastructure in the US.

As I have noted elsewhere, we need to do things differently. I have prepared and circulated a white paper among a few people in DC about one approach to changing the way we fund some of the research and education in the US in cybersecurity. I have had some of them tell me it is too radical, or too different, or doesn't fit in current funding programs. Exactly! And that is why I think we should try those things -- because doing more of the same in the current funding programs simply is not working.

But 15 years from now, I expect to run across these slides and my white paper, and sadly reflect on over three decades where we did not step up to really deal with the challenges. Of course, by then, there may be no working computers on which to read these!

Short answer: " Almost certainly, no."  

Longer answer:

The blogosphere is abuzz with comments on John Markoff's Saturday NT Times piece, Do We Need a New Internet? John got some comments from me about the topic a few weeks back. Unfortunately, I don't think a new Internet will solve the problems we are facing.

David Akin, a journalist/blogger commented on nicely John's post. In it, he quoted one of my posts to Dave Farber's IP list, which I then turned into a longer post in this blog. Basically, I noted that the Internet itself is not the biggest problem. Rather, it is the endpoints, the policies, the economics, and the legal environment that make things so difficult. It is akin to trying to blame the postal service because people manage to break into our houses by slipping their arms through the mailslots or because we leave the door unlocked "just in case" a package is going to be delivered.

Consider that some estimates of losses as a result of computer crime and fraud are in the many billions of $$ per year. (Note my recent post on a part of this.) Consider how much money is repeatedly spent on reissuing credit and debit cards because of loss of card info, restoring systems from backups, trying to remove spyware, bots, viruses, and the like. Consider how much is spent on defensive mechanisms than only work in limited cases -- anti-virus, IDS, firewalls, DLP, and whatever the latest fad might be.

What effect does that play on global finances? It is certainly a major drag on the economy. This was one of the conclusions (albeit, described as "friction") of the CSTB report Towards a Safer and More Secure Cyberspace, which did not seem to get much attention upon release.

Now, think about the solutions being put forward, such as putting all your corporate assets and sensitive records "out in the cloud" somewhere, on servers that are likely less well-protected or isolated than the ones being regularly compromised at the banks and card processors. But it will look cheaper because organizations won't need to maintain resources in-house. And it is already being hyped by companies, and seemingly being promoted by the NSF and CCC as "the future." Who can resist the future?

Next, stir in the economic conditions where any talk is going to be dismissed immediately as "crazy" if it involves replacing infrastructure with something that (initially) costs more, or that needs more than a minor change of business processes. And let's not forget that when the economy goes bad, more criminal behavior is likely as people seek value wherever they can find it.

The institutional responses from government and big vendors will be more of the same: update the patches, and apply another layer of gauze.

I have long argued that we should carefully re-examine some of the assumptions underlying what we do rather than blindly continue doing the same things. People are failing to understand that many important things have changed since we first started building computing artifacts! That means we might have better solutions if we really thought about the underlying problems from first principles.

I recently suggested this rethinking of basic assumptions to a few senior leaders in computing research (who shall remain nameless, at least within this posting) and was derided for not thinking about "new frontiers" for research. There is a belief among some in the research community (especially at the top universities) that the only way we (as a community; or perhaps more pointedly, them and their students) will get more funding for research and that we (again, the royal "we") will get premier publications is by pushing "new" ideas. This is partly a fault of the government agencies and companies, which aren't willing to support revisiting basic ideas and concepts because they want fixes to their existing systems now!

One part that makes sense from Markoff's article is about the research team making something that is effectively "plug compatible" with existing systems. That is roughly where a longer-term solution lies. If we can go back and devise more secure systems and protocols, we don't need to deploy them everywhere at once: we gradually phase them in, exactly as we do periodic refreshes of current systems. There is not necessarily an impassible divide between what we need and what we can afford.

I'm sorry to say that I don't see necessary changes occurring any time soon. It would upset too much of the status quo for too many parties. Thus, the situation isn't going to get better -- it's going to get worse -- probably much worse. When we finally get around to addressing the problems, it will be more expensive and traumatic than it needed to be.

As I noted before:

"Insanity: doing the same thing over and over again expecting different results."

Of course, my continued efforts to make this point could be branded insane.

An Aside

Over a decade ago, I gave several talks where I included the idea of having multiple "service network" layers on top of the Internet -- effectively VPNs. One such network would be governed by rules similar to those of the current Internet. A second would use cryptographic means to ensure that every packet was identified. This would be used for commercial transactions. Other such virtual networks would have different ground rules on authentication, anonymity, protocols and content. There would be contractual obligations to be followed to participate, and authorities could revoke keys and access for cause. Gateways would regulate which "networks" organizations could use. The end result would be a set of virtual networks on the Internet at large, similar to channels on a cable service. Some would be free-for-all and allow anonymous posting, but others would be much more regulated, because that is what is needed for some financial and government transactions.

I remember one audience at an early SANS conference at the time was so hostile to the idea that members began shouting objections before I could even finish my talk. I also couldn't find a venue willing to publish a speculative essay on the topic (although I admit I only tried 2-3 places before giving up). The general response was that it would somehow cut out the possibility for anonymous and experimental behavior because no one would want to use the unauthenticated channels. It was reminiscent of the controversy when I was the lead in the Usenet "Great Renamng."   

The problem, of course, is that if we try to support conflicting goals such as absolute anonymity and strong authentication on the same network we will fail at one or the other (or both). We can easily find situations where one or the other property (as simply two examples of properties at stake) is needed. So long as we continue to try to apply patches onto such a situation before reconsidering the basic assumptions, we will continue to have unhappy failures.

But as a bottom line, I simply want to note that there is more than one way to "redesign the Internet" but the biggest problems continue to be the users and their expectations, not the Internet itself.

Over the last few years, I have been involved in issues related to the use of computerization in voting. This has come about because of my concerns about computer security, privacy and reliability, and from my role as chair of the ACM U.S. Public Policy Committee (USACM). USACM has taken a strong position as regards use of computers as voting stations and voting over the internet.

Two recent items address the issue of voting over the Internet.

The first is a study released by NIST about the threats posed by internet voting. This is a well-written document describing problems that would be encountered with any online voting system. Their conclusion is that, for public elections, distribution of blank ballots (on paper) is the only reasonable improvement that we can make with current technology.

The second is a note from my colleague, Yvo Desmedt, one of the senior leaders in information security He has asked that I circulate this to a wider audience:

  IACR (the International Association for Cryptologic Research) has changed its bylaws to allow e-voting over the internet to elect its board members and other purposes. IACR will likely move towards internet e-voting. The IACR Board of Directors subcommittee on internet e-voting has published a list of requirements for such a system at: http://www.iacr.org/elections/eVoting/requirements.html This is evidently a first step and the question remains whether the system the International Association for Cryptologic Research will choose will be easy to hack or not. So, security experts should follow this development.

The problems that need to be addressed by any voting technology are mostly obvious: impersonation of the voter, impersonation of the voting system, disclosure of the ballot, multiple voting, loss of votes, denial of access, and a number of other issues. The problems are complicated by the requirements of a fair voting system, one of which is that of vote deniability—that the voter is able to deny (or claim) that her/his vote was cast a particular way. This is important to prevent vote buying, or more importantly, retribution against voters who do not cast ballots in a particular way. It isn’t difficult to find stories where voters have been beaten or killed because of how they voted (or were presumed to have intended to vote). Thus, the tried-and-true concept of providing a receipt (ala ATM machines) is not a workable solution.

My intent in making this post isn’t to discuss all the issues behind e-voting—that is well beyond the scope of a single posting, and is covered well many other places. My main goal is to give some wider circulation to Yvo’s statement. However, in light of the recent problem with certificate issuance, it is also worth noting that schemes requiring encryption to secure voting may have hidden vulnerabilities that could lead to compromise and/or failures in the future.   

In the end, it comes down to a tradeoff of risk/reward (as do all security choices): can we accurately quantify the risks with a particular approach, and are we willing to assume them? Do we have appropriate mechanisms to eliminate, mitigate or shift the risks? Are we willing to accept the risks associated with adopting a particular form of e-voting in return for the potential benefit of better access for remote voters? Or are accurate (fair) results all the time more important than complete results?

Note that one objection often raised to USACM as we argue these points is “There is no evidence there has ever been a failure or tampering with a vote.” In addition to being incorrect (there are numerous cases of computer-based voting failures), this misses two key issues:

 
• How do you tell if there is tampering if there are no safeguards that definitively disclose such tampering? That you have not detected something does not mean it has not occurred.
 
• The past does not predict the future in such a case. That no failure (accidental or otherwise) has occurred does not mean it will not occur in the future. Worse, a string of occurrences without a failure may help cloud a future discovered discrepancy!

In the case of IACR, it is obvious why this group of cryptography professionals would wish to adopt techniques that show confidence in cryptography. However, the example they set could be very damaging for other groups—and populations—if their confidence is misplaced. Given the long history of spectacular failures in cryptography—often going unannounced while being exploited—it is somewhat surprising that the IACR is not more explicit in their statement about the risks of technological failures.

Yesterday, I posted a long entry on the recent news about how some researchers obtained a “rogue” certificate from one of the Internet Certificate Authorities. There are some points I missed in the original post that should be noted.

 
• The authors of the exploit have a very readable, interesting description of what they did and why it worked. I should have included a link to it in the original posting, but forgot to edit it in. The interested reader should definitely see that article online, include the animations.
 
• There are other ways this attack can be defeated, certainly, but they are stop-gap measures. I didn’t explain them because I don’t view them as other than quick patches. However, if you are forced to continue to use MD5 and you issue certificates, then it is important to randomize the certificate serial number that is issued, and to insert a random delay interval in the validity time field. Both will introduce enough random bits so as to make this particular attack against the CA infeasible given current technology.
 
• I suggested that vendors use another hash algorithm, and have SHA-1 as an example. SHA-2 would be better, as SHA-1 has been shown to have a theoretical weakness similar to MD5, although it has proven more resistant to attack to date. Use of SHA-1 could possible result in a similar problem within a few years (or, as suggested in the final part of my post, within a few weeks if a breakthrough occurs). However, use of SHA-1 would be preferable to MD5!
 
• MD5 is not “broken” in a complete way. There are several properties of a message digest that are valuable, including collision resistance: that it is infeasible to end up with two inputs giving the same hash value. To the best of my knowledge, MD5 has only been shown to be susceptible to “weak collisions”—instances where the attacker can pick one or both inputs so as to produce identical hash values. The stronger form of preimage resistance, where there is an arbitrary hash output H and an attacker cannot form an input that also produces H, still holds for MD5. Thus, applications that depend on this property (including many signing applications and integrity tools) are apparently still okay.
 
• One of our recent PhD grads, William Speirs, worked on defining hash functions for his PhD dissertation. His dissertation, Dynamic Cryptographic Hash Functions, is available online for those interested in seeing it.

I want to reiterate that there are more fundamental issues of trust involved than what hash function is used. The whole nature of certificates is based around how much we trust the certificate authorities that issue the certificates, and the correctness of the software that verifies those certificates then shows us the results. If an authority is careless or rogue, then the certificates may be technically valid but not match our expectations for validity. If our local software (such as a WWW browser) incorrectly validates a certificate, or presents the results incorrectly, we may trust a certificate we shouldn’t. Even such mundane issues as having one’s system at the correct time/date can be important: the authors of this particular hack demonstrated that by backdating their rogue certificate.

My continuing message to the community is to not lose sight of those things we assume. Sometimes, changes in the world around us render those assumptions invalid, and everything built on them becomes open to question. If we forget those assumptions—and our chains of trust built on them—we will continue to be surprised by the outcomes.

That is perhaps fitting to state (again) on the last day of the year. Let me observe that as human beings we sometimes take things for granted in our lives. Spend a few moments today (and frequently, thereafter) to pause and think about the things in your life that you may be taking for granted: family, friends, love, health, and the wonder of the world around you. Then as is your wont, celebrate what you have.

Best wishes for a happy, prosperous, safe—and secure—2009.

[12/31/08 Addition]: Steve Bellovin has noted that transition to the SHA-2 hash algorithm in certificates (and other uses) would not be simple or quick. He has written a paper describing the difficulties and that paper is online.

There are several news stories now appearing (e.g., Security News) about a serious flaw in how certificates used in online authentication are validated. Ed Felten gives a nice summary of how this affects online WWW site authentication in his Freedom to Tinker blog posting. Brian Krebs also has his usual readable coverage of the problem in his Washington Post article. Steve Bellovin has some interesting commentary, too, about the legal climate.

Is there cause to be concerned? Yes, but not necessarily about what is being covered in the media. There are other lessons to be learned from this.

Short tutorial

First, for the non-geek reader, I’ll briefly explain certificates.

Think about how, online, I can assure myself that the party at the other end of a link is really who they claim to be. What proof can they offer, considering that I don’t have a direct link? Remember that an attacker can send any bits down the wire to me and may access to faster computers than I do.

I can’t base my decision on how the WWW pages appear, or embedded images. Phishing, for instance, succeeds because the phishers set up sites with names and graphics that look like the real banks and merchants, and users trust the visual appearance. This is a standard difficulty for people—understanding the difference between identity (claiming who I am) and authentication (proving who I am).

In the physical world, we do this by using identity tokens that are issued by trusted third parties. Drivers licenses and passports are two of the most common examples. To get one, we need to produce sufficient proof of identity to a third party to meet its standards of proof. Then, the third party issues a document that is very difficult to forge (almost nothing constructed is impossible to forge or duplicate—but some things require so much time and expenditure it isn’t worthwhile). Because the criteria for proof of identity and strength of construction of the document are known, various other parties will accept the document as “proof” of identity. Of course, other problems occur that I’m not going to address—this USACM whitepaper (of which I was principal author) touches on many of them.

Now, in the online world we cannot issue or see physical documents. Instead, we use certificates. We do this by putting together an electronic document that gives the information we want some entity to certify as true about us. The format of this certificate is generally fixed by standards, the most common one being the X.509 suite. This document is sent to an organization known as a Certificate Authority (CA), usually along with a fee. The certificate authority is presumably well-known, and performs a check (to their own standards) that the information in the document is correct, and it has the right form. The CA then calculate a digital hash value of the data, and creates a digital signature of that hash value. This is then added to the certificate and sent back to the user. This is the equivalent of putting a signature on a license and then sealing it in plastic. Any alteration of the data will change the digital hash, and a third party will find that the new hash and the hash value signed with the key of the CA don’t match. The reason this works is that the hash function and encryption algorithm used are presumed to be so computationally difficult to forge that it is basically not possible.

As an example of a certificate , if you visit “https://www.cerias.purdue.edu” you can click on the little padlock icon that appears somewhere in the browser window frame (this is browser dependent) to view details of the CERIAS SSL certificate.

You can get more details on all this by reading the referenced Wikipedia pages, and by reading chapters 5 & 7 in Web Security, Privacy and Commerce.

Back to the hack

In summary, some CAs have been negligent about updating their certificate signing mechanisms in the wake of news that MD5 is weak, published back in 2004. The result is that malicious parties can generate and obtain a certificate “authenticating” them as someone else. What makes it worse is that the root certificate of most of these CAs are “built in” to browser and application trust lists to simplify look-up of new certificates. Thus, most people using standard WWW browsers can be fooled into thinking they have connected to real, valid sites—even through they are connecting to rogue sites.

The approach is simple enough: a party constructs two certificates. One is for the false identity she wishes to claim, and the other is real. She crafts the contents of the certificate so that the MD5 hash of the two, in canonical format, is the same. She submits the real identity certificate to the authority, which verifies her bona fides, and returns the certificate with the MD5 hash signed with the CA private key. Our protagonist then copies that signature to the false certificate, which has the same MD5 hash value and thus the same digital signature, and proceeds with her impersonation!

What makes this worse is that the false key she crafts is for a secondary certificate authority. She can publish this in appropriate places, and is now able to mint as many false keys as she wishes—and they will all have signatures that verify in the chain of trust back to the issuer! She can even issue these new certificates using a stronger hash algorithm than MD5!

What makes this even worse is that it has been known for years that MD5 is weak, yet some CAs have continued to use it! Particularly unfortunate is the realization that Lenstra, Wang and de Weger described how this could be done back in 2005. Methinks that may be grounds for some negligence lawsuits if anyone gets really burned by this….

And adding to the complexity of all this is the issue of certificates in use for other purposes. For example, certificates are used with encrypted S/MIME email to digitally sign messages. Certificates are used to sign ActiveX controls for Microsoft software. Certificates are used to verify the information on many identity cards, including (I believe) government-issued Common Access Cards (CAC). Certificates also provide identification for secured instant messaging sessions (e.g., iChat). There may be many other sensitive uses because certificates are a “known” mechanism. Cloud computing services , software updates, and more may be based on these same assumptions. Some of these services may accept and/or use certificates issued by these deficient CAs.

Fixes

Fixing this is not trivial. Certainly, all CAs need to start issuing certificates based on other message digests, such as SHA-1. However, this will take time and effort, and may not take effect before this problem can be exploited by attackers. Responsible vendors will cease to issue certificates until they get this fixed, but that has an economic impact some many not wish to incur.

We can try to educate end-users about this, but the problem is so complicated with technical details, the average person won’t know how to actually make a determination about valid certificates. It might even cause more harm by leading people to distrust valid certificates by mistake!

It is not possible to simply say that all existing applications will no longer accept certificates rooted at those CAs, or will not accept certificates based on MD5: there are too many extant, valid certificates in place to do that. Eventually, those certificates will expire, and be replaced. That will eventually take care of the problem—perhaps within the space of the next 18 months or so (most certificates are issued for only a year at a time, in part for reasons such as this).

Vendors of applications, and especially WWW browsers, need to give careful thought about updates to their software to flag MD5-based certificates as deserving of special attention. This may or may not be a worthwhile approach, for the reason given above: even with a warning, too few people will be able to know what to do.

Bigger issue

We base a huge amount of trust on certificates and encryption. History has shown how easy it is to get implementations and details wrong. History has also shown how quickly things can be destabilized with advances in technology.

In particular, too many people and organizations take for granted the assumptions on which this vast certificate system is based. For instance, we assume that the hash/digest functions in use are computationally difficult to reverse or cause collisions. We also assume that certain mathematical functions underlying public/private key encryption are too difficult to reverse or “brute force.” However, all it takes is some new insight or analysis, or maybe new, affordable technology (e.g., practical quantum computing, or massively parallel computing) to violate those assumptions.

If you look at the way our systems are constructed, too little thought is given to what happens to existing infrastructure when something breaks. Designs can include compensating and recovery code, but doing so requires some cost in space or time. However, all too often people are willing to avoid the investment by putting off the danger to “if and when that happens.” Thus, we instance such as the Y2K problems and the issues here with potentially rogue CAs.

(I’ll note as an aside, that when I designed the original version of Tripwire and what became the Signacert product, I specifically included simultaneous use of several different message digest functions in different families for this very reason. I knew it was a matter of time before one or two were broken. I still believe that it is beyond reason to find files that will match multiple, different algorithms simultaneously.)

Another issue is the whole question of who we trust, and for what. As noted in the USACM whitepaper, authentication is always relative to a third party. How much do we trust those third parties? How much trust have we invested in the companies and agencies issuing certificates? Are they properly verifying identities? How good is there internal security? How do we know, and how much is at risk from our trust in those entities?

Let me leave you with a final thought. How do we know that this problem has not already been quietly exploited? The basic concept has been in the open literature for years. The general nature of this attack on certificates has been known for well over a decade, if not two. Given the technical and infrastructure resources available to national agencies and organized criminals, and given the motivation to use this hack selectively and quietly, how can we know that it is not already being used?

[Added 12/31/2008]: A follow-up post to this one is available in the blog.

I have repeatedly pointed out (e.g., this post) to people that sending Word files as attachments is a bad idea. This has been used many, many times to circulate viruses, worms, and more. People continue to push back because (basically) it is convenient for them. How often have we heard that convenience trumps good security (and good sense)?

Now comes this story of yet another attack being spread with Word documents.

There are multiple reasons why I don’t accept Word documents in email. This is simply one of the better reasons.

If you want to establish a sound security posture at your organization, one of the things you should mandate is no circulation of executable formats—either out or in. “.doc” files are in this category. I am unsure if the new “.docx” format is fully immune to these kinds of things but it seems “.rtf” is.