Posts in Secure IT Practices

Page Content

Why We Don’t Have Secure Systems Yet, Introduction

Over the past couple of months I’ve been giving an evolving talk on why we don’t yet have secure systems, despite over 50 years of work in the field. I first gave this at an NSF futures workshop, and will give it a few more times this summer and fall.

As I was last reviewing my notes, it occurred to me that many of the themes I’ve spoken about have been included in past posts here in the blog, and are things I’ve been talking about for nearly my entire career. It’s disappointing how little progress I’ve seen on so many fronts. The products on the market, and the “experts” who get paid big salaries to be corporate and government advisors and who get the excessive press coverage, also serve to depress.

My current thinking is to write a series of blog posts to summarize my thinking on this general topic. I’m not sure how many I’ll write, but I have a list of probable topics already in mind. They break out roughly into (in approximate order of presentation):

  • Definition & metrics
  • History
  • Changes in technology
  • Research & Development
  • Legacy and Inertia
  • Bad practices
  • Media & milieu focus
  • Funding
  • Law enforcement
  • National policies
  • International issues

Each of these will be of moderate length, with some references and links to material to read. If you’re interested in a preview, I recommend looking at some of my recent talks archived on YouTube, some of my past blog posts here, and oral histories of various pioneers in the field of infosec done by the Babbage Institute (including, perhaps, my own).

I’ll start with the first posting sometime in the next few days, after I get a little more caught up from my vacation. But I thought I’d make this post, first, to solicit feedback on ideas that people might like me to add to the list.

My first post will be about the definition of security — and why part of the problem is that we can’t very well fix something that we can’t reliably define and thus obviously don’t completely understand.

Telling the Future, Looking at the Past: A Few Short Items

I have continued to update my earlier post about women in cybersecurity. Recent additions include links to some scholarship opportunities offered by ACSA and the (ISC)2 Foundation. Both scholarship opportunities have deadlines in the coming weeks, so look at them soon if you are interested.

The 15th Annual Security Symposium is less than a month away! Registration is still open but filling quickly. If you register for the Symposium, or for the 9th ICCWS held immediately prior, you can get a discount on the other event. Thus, you should think about attending both and saving on the registration costs! See the link for more details.

I periodically post an item to better define my various social media presences. If you follow me (Spaf) and either wonder why I post in multiple venues, or want to read even more of my musings, then take a look at it.

I ran across one of my old entries in this blog — from October 2007 — that had predictions for the future of the field. In rereading them, I think I did pretty well, although some of the predictions were rather obvious. What do you think?

Sometime in the next week or so (assuming the polar vortex and ice giants don’t get me) I will post some of my reflections on the RSA 2014 conference. However, if you want a sneak peek at what I think about what I saw on the display floor and after listening to some of the talks, you can read another of my old blog entries — things haven’t changed much.

Happy Anniversary—Bang My Head Against A Wall

Over the last month or two I have received several invitations to go speak about cyber security. Perhaps the up-tick in invitations is because of the allegations by Edward Snowden and their implications for cyber security. Or maybe it is because news of my recent awards has caught their attention. It could be it is simply to hear about something other than the (latest) puerile behavior by too many of our representatives in Congress and I'm an alternative chosen at random. Whatever the cause, I am tempted to accept many of these invitations on the theory that if I refuse too many invitations, people will stop asking, and then I wouldn't get to meet as many interesting people.

As I've been thinking about what topics I might speak about, I've been looking back though the archive of talks I've given over the last few decades. It's a reminder of how many things we, as a field, knew about a long time ago but have been ignored by the vendors and authorities. It's also depressing to realize how little impact I, personally, have had on the practice of information security during my career. But, it has also led me to reflect on some anniversaries this year (that happens to us old folk). I'll mention three in particular here, and may use others in some future blogs.

In early November of 1988 the world awoke to news of the first major, large-scale Internet incident. Some self-propagating software had spread around the nascent Internet, causing system crashes, slow-downs, and massive uncertainty. It was really big news. Dubbed the "Internet Worm," it served as an inspiration for many malware authors and vandals, and a wake-up call for security professionals. I recall very well giving talks on the topic for the next few years to many diverse audiences about how we must begin to think about structuring systems to be resistant to such attacks.

Flash forward to today. We don't see the flashy, widespread damage of worm programs any more, such as what Nimda and Code Red caused. Instead, we have more stealthy botnets that infiltrate millions of machines and use them for spam, DDOS, and harassment. The problem has gotten larger and worse, although in a manner that hides some of its magnitude from the casual observer. However, the damage is there; don't try to tell the folks at Saudi Aramaco or Qatar's Rasgas that network malware isn't a concern any more! Worrisomely, experts working with SCADA systems around the world are increasingly warning how vulnerable they might be to similar attacks in the future.

Computer viruses and malware of all sorts first notably appeared "in the wild" in 1982. By 1988 there were about a dozen in circulation. Those of us advocating for more care in design, programming and use of computers were not heeded in the head-long rush to get computing available on every desktop (and more) at the lowest possible cost. Thus, we now have (literally) tens of millions of distinct versions of malware known to security companies, with millions more appearing every year. And unsafe practices are still commonplace -- 25 years after that Internet Worm.

For the second anniversary, consider 10 years ago. The Computing Research Association, with support from the NSF, convened a workshop of experts in security to consider some Grand Challenges in information security. It took a full 3 days, but we came up with four solid Grand Challenges (it is worth reading the full report and (possibly) watching the video):

  1. Eliminate epidemic-style attacks within 10 years
    • Viruses and worms
    • SPAM
    • Denial of Service attacks (DOS)
  2. Develop tools and principles that allow construction of large-scale systems for important societal applications that are highly trustworthy despite being attractive targets.
  3. Within 10 years, quantitative information-systems risk management will be at least as good as quantitative financial risk management.
  4. For the dynamic, pervasive computing environments of the future, give endusers security they can understand and privacy they can control.

I would argue -- without much opposition from anyone knowledgeable, I daresay -- that we have not made any measurable progress against any of these goals, and have probably lost ground in at least two.

Why is that? Largely economics, and bad understanding of what good security involves. The economics aspect is that no one really cares about security -- enough. If security was important, companies would really invest in it. However, they don't want to part with all the legacy software and systems they have, so instead they keep stumbling forward and hope someone comes up with magic fairy dust they can buy to make everything better.

The government doesn't really care about good security, either. We've seen that the government is allegedly spending quite a bit on intercepting communications and implanting backdoors into systems, which is certainly not making our systems safer. And the DOD has a history of huge investment into information warfare resources, including buying and building weapons based on unpatched, undisclosed vulnerabilities. That's offense, not defense. Funding for education and advanced research is probably two orders of magnitude below what it really should be if there was a national intent to develop a secure infrastructure.

As far as understanding security goes, too many people still think that the ability to patch systems quickly is somehow the approach to security nirvana, and that constructing layers and layers of add-on security measures is the path to enlightenment. I no longer cringe when I hear someone who is adept at crafting system exploits referred to as a "cyber security expert," but so long as that is accepted as what the field is all about there is little hope of real progress. As J.R.R. Tolkien once wrote, "He that breaks a thing to find out what it is has left the path of wisdom." So long as people think that system penetration is a necessary skill for cyber security, we will stay on that wrong path.

And that is a great segue into the last of my three anniversary recognitions. Consider this quote (one of my favorite) from 1973 -- 40 years ago -- from a USAF report, Preliminary Notes on the Design of Secure Military Computer Systems, by a then-young Roger Schell:

…From a practical standpoint the security problem will remain as long as manufacturers remain committed to current system architectures, produced without a firm requirement for security. As long as there is support for ad hoc fixes and security packages for these inadequate designs and as long as the illusory results of penetration teams are accepted as demonstrations of a computer system security, proper security will not be a reality.

That was something we knew 40 years ago. To read it today is to realize that the field of practice hasn't progressed in any appreciable way in three decades, except we are now also stressing the wrong skills in developing the next generation of expertise.

Maybe I'll rethink that whole idea of going to give a talks on security and simply send them each a video loop of me banging my head against a wall.

PS -- happy 10th annual National Cyber Security Awareness Month -- a freebie fourth anniversary! But consider: if cyber security were really important, wouldn't we be aware of that every month? The fact that we need to promote awareness of it is proof it isn't taken seriously. Thanks, DHS!

Now, where can I find I good wall that doesn't already have dents from my forehead....?

A Valuable Resource for Young People (limited time offer)

Over the years, I've gotten to know many people working in security and privacy. Too few have focused on issues relating to children and young adults. Thankfully, one of these people is Linda McCarthy. A security professional with an impressive resume that includes senior positions at Sun Microsystems and Symantec, Linda has had actual "boots-on-the-ground" experience in the practice of information protection.

Linda has written several books on security, including "Intranet Security - Stories from the Trenches," and "IT Security: Risking the Corporation." She also co-authored the recent free, quite popular, Facebook tutorial on security and privacy. I have read these, heard her speak, and worked with her on projects over the years -- Linda is thoughtful, engaging and an effective communicator on the topics of security and privacy. I'm not the only person to think so -- not too long ago she was a recipient of the prestigious Women of Influence award, presented by CSO Magazine and Alta Associates, recognizing her many achievements in security, privacy and risk management.

About a decade ago, based on some personal experiences with young adults close to her, Linda took on the cause of education about how to be safe online. Youngsters seldom have the experience (and the judgement born of experience) to make the best choices about how to protect themselves. Couple that naiveté with the lure of social contact and the lack of highly-visible controls, and toss in a dash of the opportunity to rebel against elders, and a dangerous mix results. Few people, young or old, truly grasp the extent and reach in time and space of the Internet -- postings of pictures and statements never really go away. Marketers, for one, love that depth of data to mine, but it is a nightmare that can haunt the unwary for decades to come.

Long term loss of privacy isn't the only threat, of course. Only last week news broke of yet another tragic suicide caused by cyberbullies; there is a quiet epidemic of this kind of abuse. Also, Miss Teen USA, Cassidy Wolf, spoke a few days ago about being the victim of cyberstalking and sexual extortion. These are not things kids think about when going online -- and neither do their parents. This is the complex milieu that Linda is confronting.

In 2006, Linda began to focus on writing for the younger set and produced "Own Your Space: Keep Yourself and Your Stuff Safe Online," which is a nice introduction that kids seem to appreciate. A few years ago, Linda updated it and under a Creative Commons license it is now available as a free download from Microsoft (among others). I wrote about the release of that update in this blog in 2010.

Earlier this year, Linda released a new book, "Digital Drama: Staying Safe While Being Social Online" (also available en español). This book covers a multitude of issues, including privacy, reputation, online bullying and stalking, avoiding predators, spotting scams, how to manage settings and online persona, and a wealth of other valuable insights for young people -- and therefore it is also of value to their parents, teachers, and an older audience that may not have the expertise but faces many of the same concerns. Linda's book doesn't address all the problems out there -- she doesn't address the really dark side of youth gang culture, for instance -- but this book does admirably cover many of the major issues that face kids who really want to stay out of trouble.

What makes this especially useful is a limited-time offer. In support of National Cyber Security Awareness Month, Microsoft has provided support to allow Linda to offer a free digital download of "Digital Drama" from (the Spanish version, too). Parents, teachers, teens, tweens, kids, and the young at heart can all get that free download from 12am on Tuesday, September 24th until 11:59pm on Friday, September 27 (2013; times are PDT). (If you are reading this blog after that week, you should still check out the book.)

To quote from the "About this book" section of Amazon:

Every day, millions of teens log on and make decisions that can compromise their safety, security, privacy, and future. If you are like most teens, you are already using social networking sites like Twitter and Facebook and have your smartphone super-glued to your hand. You tag your friends in photos, share your location and thoughts with friends, and post jokes online that later may be misunderstood. At the same time, you might not realize how that information can affect your reputation and safety, both online and offline. We’ve all heard the horror stories of stolen identities, cyber stalking, pedophiles on the Internet, and lost job, school, and personal opportunities. All teens need to learn how to protect themselves against malware, social networking scams, and cyberbullies. Learn crucial skills:
- Deal with cyberbullies
- Learn key social networking skills
- Protect your privacy
- Create a positive online reputation
-Protect yourself from phishing and malware scams

Spaf sez, "Check it out."

Just sayin

In the June 17, 2013 online interview with Edward Snowden, there was this exchange:


User avatar for Mathius1

Is encrypting my email any good at defeating the NSA survelielance? Id my data protected by standard encryption?


Encryption works. Properly implemented strong crypto systems are one of the few things that you can rely on. Unfortunately, endpoint security is so terrifically weak that NSA can frequently find ways around it.

I simply thought I'd point out a statement of mine that first appeared in print in 1997 on page 9 of Web Security & Commerce (1st edition, O'Reilly, 1997, S. Garfinkel & G. Spafford):

Secure web servers are the equivalent of heavy armored cars. The problem is, they are being used to transfer rolls of coins and checks written in crayon by people on park benches to merchants doing business in cardboard boxes from beneath highway bridges. Further, the roads are subject to random detours, anyone with a screwdriver can control the traffic lights, and there are no police.

I originally came up with an abbreviated version of this quote during an invited presentation at SuperComputing 95 (December of 1995) in San Diego. The quote at that time was everything up to the "Further...." and was in reference to using encryption, not secure WWW servers.

A great deal of what people are surprised about now should not be a surprise -- some of us have been lecturing about elements of it for decades. I think Cassandra was a cyber security professor....

[Added 9/10: This also reminded me of a post from a couple of years ago. The more things change....]


Some thoughts on “cybersecurity” professionalization and education

[I was recently asked for some thoughts on the issues of professionalization and education of people working in cyber security. I realize I have been asked this many times, I and I keep repeating my answers, to various levels of specificity. So, here is an attempt to capture some of my thoughts so I can redirect future queries here.]

There are several issues relating to the area of personnel in this field that make issues of education and professional definition more complex and difficult to define. The field has changing requirements and increasing needs (largely because industry and government ignored the warnings some of us were sounding many years ago, but that is another story, oft told -- and ignored).

When I talk about educational and personnel needs, I discuss it metaphorically, using two dimensions. Along one axis is the continuum (with an arbitrary directionality) of science, engineering, and technology. Science is the study of fundamental properties and investigation of what is possible -- and the bounds on that possibility. Engineering is the study of design and building new artifacts under constraints. Technology is the study of how to choose from existing artifacts and employ them effectively to solve problems.


The second axis is the range of pure practice to abstraction. This axis is less linear than the other (which is not exactly linear, either), and I don't yet have a good scale for it. However, conceptually I relate it to applying levels of abstraction and anticipation. At its "practice" end are those who actually put in the settings and read the logs of currently-existing artifacts; they do almost no hypothesizing. Moving the other direction we see increasing interaction with abstract thought, people and systems, including operations, law enforcement, management, economics, politics, and eventually, pure theory. At one end, it is "hands-on" with the technology, and at the other is pure interaction with people and abstractions, and perhaps no contact with the technology.

There are also levels of mastery involved for different tasks, such as articulated in Bloom's Taxonomy of learning. Adding that in would provide more complexity than can fit in this blog entry (which is already too long).

The means of acquisition of necessary expertise varies for any position within this field. Many technicians can be effective with simple training, sometimes with at most on-the-job experience. They usually need little or no background beyond everyday practice. Those at the extremes of abstract thought in theory or policy need considerably more background, of the form we generally associate with higher education (although that is not strictly required), often with advanced degrees. And, of course, throughout, people need some innate abilities and motivation for the role they seek; Not everyone has ability, innate or developed, for each task area.

We have need of the full spectrum of these different forms of expertise, with government and industry currently putting an emphasis on the extremes of the quadrant involving technology/practice -- they have problems, now, and want people to populate the "digital ramparts" to defend them. This emphasis applies to those who operate the IDS and firewalls, but also to those who find ways to exploit existing systems (that is an area I believe has been overemphasized by government. Cf. my old blog post and a recent post by Gary McGraw). Many, if not most, of these people can acquire needed skills via training -- such as are acquired on the job, in 1-10 day "minicourses" provided by commercial organizations, and vocational education (e.g, some secondary ed, 2-year degree programs). These kinds of roles are easily designated with testing and course completion certificates.

Note carefully that there is no value statement being made here -- deeply technical roles are fundamental to civilization as we know it. The plumbers, electricians, EMTs, police, mechanics, clerks, and so on are key to our quality of life. The programs that prepare people for those careers are vital, too.

Of course, there are also careers that are directly located in many other places in the abstract plane illustrated above: scientists, software engineers, managers, policy makers, and even bow tie-wearing professors. grin

One problem comes about when we try to impose sharply-defined categories on all of this, and say that person X has sufficient mastery of the category to perform tasks A, B, and C that are perceived as part of that category. However, those categories are necessarily shifting, not well-defined, and new needs are constantly arising. For instance, we have someone well trained in selecting and operating firewalls and IDS, but suddenly she is confronted with the need to investigate a possible act of nation-state espionage, determine what was done, and how it happened. Or, she is asked to set corporate policy for use of BYOD without knowledge of all the various job functions and people involved. Further deployment of mobile and embedded computing will add further shifts. The skills to do most of these tasks are not easily designated, although a combination of certificates and experience may be useful.

Too many (current) educational programs stress only the technology -- and many others include significant technology training components because of pressure by outside entities -- rather than a full spectrum of education and skills. We have a real shortage of people who have any significant insight into the scope of application of policy, management, law, economics, psychology and the like to cybersecurity, although arguably, those are some of the problems most obvious to those who have the long view. (BTW, that is why CERIAS was founded 15 years ago including faculty in nearly 20 academic departments: "cybersecurity" is not solely a technology issue; this has more recently been recognized by several other universities that are now also treating it holistically.) These other skill areas often require deeper education and repetition of exercises involving abstract thought. It seems that not as many people are naturally capable of mastering these skills. The primary means we use to designate mastery is through postsecondary degrees, although their exact meaning does vary based on the granting institution.

So, consider some the bottom line questions of "professionalization" -- what is, exactly, the profession? What purposes does it serve to delineate one or more niche areas, especially in a domain of knowledge and practice that changes so rapidly? Who should define those areas? Do we require some certification to practice in the field? Given the above, I would contend that too many people have too narrow a view of the domain, and they are seeking some way of ensuring competence only for their narrow application needs. There is therefore a risk that imposing "professional certifications" on this field would both serve to further skew the perception of what is involved, and discourage development of some needed expertise. Defining narrow paths or skill sets for "the profession" might well do the same. Furthermore, much of the body of knowledge is heuristics and "best practice" that has little basis in sound science and engineering. Calling someone in the 1600s a "medical professional" because he knew how to let blood, apply leeches, and hack off limbs with a carpenter's saw using assistants to hold down the unanesthitized patient creates a certain cognitive dissonance; today, calling someone a "cyber security professional" based on knowledge of how to configure Windows, deploy a firewall, and install anti-virus programs should probably be viewed as a similar oddity. We need to evolve to where the deployed base isn't so flawed, and we have some knowledge of what security really is -- evolve from the equivalent of "sawbones" to infectious disease specialists.

We have already seen some of this unfortunate side-effect with the DOD requirements for certifications. Now DOD is about to revisit the requirements, because they have found that many people with certifications don't have the skills they (DOD) think they want. Arguably, people who enter careers and seek (and receive) certification are professionals, at least in a current sense of that word. It is not their fault that the employers don't understand the profession and the nature of the field. Also notable are cases of people with extensive experience and education, who exceed the real needs, but are not eligible for employment because they have not paid for the courses and exams serving as gateways for particular certificates -- and cash cows for their issuing organizations. There are many disconnects in all of this. We also saw skew develop in the academic CAE program.

Here is a short parable that also has implications for this topic.

In the early 1900s, officials with the Bell company (telephones) were very concerned. They told officials and the public that there was a looming personnel crisis. They predicted that, at the then-current rate of growth, by the end of the century everyone in the country would need to be a telephone operator or telephone installer. Clearly, this was impossible.

Fast forward to recent times. Those early predictions were correct. Everyone was an installer -- each could buy a phone at the corner store, and plug it into a jack in the wall at home. Or, simpler yet, they could buy cellphones that were already on. And everyone was an operator -- instead of using plugboards and directory assistance, they would use an online service to get a phone number and enter it in the keypad (or speed dial from memory). What happened? Focused research, technology evolution, investment in infrastructure, economics, policy, and psychology (among others) interacted to "shift the paradigm" to one that no longer had the looming personnel problems.

If we devoted more resources and attention to the broadly focused issues of information protection (not "cyber" -- can we put that term to rest?), we might well obviate many of the problems that now require legions of technicians. Why do we have firewalls and IDS? In large part, because the underlying software and hardware was not designed for use in an open environment, and its development is terribly buggy and poorly configured. The languages, systems, protocols, and personnel involved in the current infrastructure all need rethinking and reengineering. But so long as the powers-that-be emphasize retaining (and expanding) legacy artifacts and compatibility based on up-front expense instead of overall quality, and in training yet more people to be the "cyber operators" defending those poor choices, we are not going to make the advances necessary to move beyond them (and, to repeat, many of us have been warning about that for decades). And we are never going to have enough "professionals" to keep them safe. We are focusing on the short term and will lose the overall struggle; we need to evolve our way out of the problems, not meet them with an ever-growing band of mercenaries.

The bottom line? We should be very cautious in defining what a "professional" is in this field so that we don't institutionalize limitations and bad practices. And we should do more to broaden the scope of education for those who work in those "professions" to ensure that their focus -- and skills -- are not so limited as to miss important features that should be part of what they do. As one glaring example, think "privacy" -- how many of the "professionals" working in the field have a good grounding and concern about preserving privacy (and other civil rights) in what they do? Where is privacy even mentioned in "cybersecurity"? What else are they missing?

[If this isn't enough of my musings on education, you can read two of my ideas in a white paper I wrote in 2010. Unfortunately, although many in policy circles say they like the ideas, no one has shown any signs of acting as a champion for either.]

[3/2/2013] While at the RSA Conference, I was interviewed by the Information Security Media Group on the topic of cyber workforce. The video is available online.

On Student Projects, Phoenix, and Improving Your IT Operations

[If you want to skip my recollection and jump right to the announcement that is the reason for this post, go here.]

Back in about 1990 I was approached by an eager undergrad who had recently come to Purdue University. A mutual acquaintance (hi, Rob!) had recommended that the student connect with me for a project. We chatted for a bit and at first it wasn't clear exactly what he might be able to do. He had some experience coding, and was working in the campus computing center, but had no background in the more advanced topics in computing (yet).

Well, it just so happened that a few months earlier, my honeypot Sun workstation had recorded a very sophisticated (for the time) attack, which resulted in an altered shared library with a back door in place. The attack was stealthy, and the new library had the same dates, size and simple hash value as the original. (The attack was part of a larger series of attacks, and eventually documented in "@Large: The Strange Case of the World's Biggest Internet Invasion" (David H. Freedman, Charles C. Mann .)

I had recently been studying message digest functions and had a hunch that they might provide better protection for systems than a simple ls -1 | diff - old comparison. However, I wanted to get some operational sense about the potential for collision in the digests. So, I tasked the student with devising some tests to run many files through a version of the digest to see if there were any collisions. He wrote a program to generate some random files, and all seemed okay based on that. I suggested he look for a different collection -- something larger. He took my advice a little too much to heart. It seems he had a part time job running backup jobs on the main shared instructional computers at the campus computing center. He decided to run the program over the entire file system to look for duplicates. Which he did one night after backups were complete.

The next day (as I recall) he reported to me that there were no unexpected collisions over many hundreds of thousands of files. That was a good result!

The bad result was that running his program over the file system had resulted in a change of the access time of every file on the system, so the backups the next evening vastly exceeded the existing tape archive and all the spares! This led directly to the student having a (pointed) conversation with the director of the center, and thereafter, unemployment. I couldn't leave him in that position mid-semester so I found a little money and hired him as an assistant. I them put him to work coding up my idea, about how to use the message digests to detect changes and intrusions into a computing system. Over the next year, he would code up my design, and we would do repeated, modified "cleanroom" tests of his software. Only when they all passed, did we release the first version of Tripwire.

That is how I met Gene Kim .

Gene went on to grad school elsewhere, then a start-up, and finally got the idea to start the commercial version of Tripwire with Wyatt Starnes; Gene served as CTO, Wyatt as CEO. Their subsequent hard work, and that of hundreds of others who have worked at the company over the years, resulted in great success: the software has become one of the most widely used change detection & IDS systems in history, as well as inspiring many other products.

Gene became more active in the security scene, and was especially intrigued with issues of configuration management, compliance, and overall system visibility, and with their connections to security and correctness. Over the years he spoken with thousands of customers and experts in the industry, and heard both best-practice and horror stories involving integrity management, version control, and security. This led to projects, workshops, panel sessions, and eventually to his lead authorship of "Visible Ops Security: Achieving Common Security and IT Operations Objectives in 4 Practical Steps" (Gene Kim, Paul Love, George Spafford) , and some other, related works.

His passion for the topic only grew. He was involved in standards organizations, won several awards for his work, and even helped get the B-sides conferences into a going concern. A few years ago, he left his position at Tripwire to begin work on a book to better convey the principles he knew could make a huge difference in how IT is managed in organizations big and small.

I read an early draft of that book a little over a year ago (late 2011), It was a bit rough -- Gene is bright and enthusiastic, but was not quite writing to the level of J.K. Rowling or Stephen King. Still, it was clear that he had the framework of a reasonable narrative to present major points about good, bad, and excellent ways to manage IT operations, and how to transform them for the better. He then obtained input from a number of people (I think he ignored mine), added some co-authors, and performed a major rewrite of the book. The result is a much more readable and enjoyable story -- a cross between a case study and a detective novel, with a dash of H. P. Lovecraft and DevOps thrown in.

The official launch date of the book, "The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win" (Gene Kim, Kevin Behr, George Spafford), is Tuesday, January 15, but you can preorder it before then on (at least) Amazon.

The book is worth reading if you have a stake in operations at a business using IT. If you are a C-level executive, you should most definitely take time to read the book. Consultants, auditors, designers, educators...there are some concepts in there for everyone.

But you don't have to take only my word for it -- see the effusive praise of tech luminaries who have read the book .

So, Spaf sez, get a copy and see how you can transform your enterprise for the better.

(Oh, and I have never met the George Spafford who is a coauthor of the book. We are undoubtedly distant cousins, especially given how uncommon the name is. That Gene would work with two different Spaffords over the years is one of those cosmic quirks Vonnegut might write about. But Gene isn't Vonnegut, either. grin

So, as a postscript.... I've obviously known Gene for over 20 years, and am very fond of him, as well as happy for his continuing success. However, I have had a long history of kidding him, which he has taken with incredible good nature. I am sure he's saving it all up to get me some day....

When Gene and his publicist asked if I could provide some quotes to use for his book, I wrote the first of the following. For some reason, this never made it onto the WWW site . So, they asked me again, and I wrote the second of the following -- which they also did not use.

So, not to let a good review (or two) go to waste, I have included them here for you. If nothing else, it should convince others not to ask me for a book review.

But, despite the snark (who, me?) of these gag reviews, I definitely suggest you get a copy of the book and think about the ideas expressed therein. Gene and his coauthors have really produced a valuable, readable work that will inform -- and maybe scare -- anyone involved with organizational IT.

Take 1:

Based on my long experience in academia, I can say with conviction that this is truly a book, composed of an impressive collection of words, some of which exist in human languages. Although arranged in a largely random order, there are a few sentences that appear to have both verbs and nouns. I advise that you immediately buy several copies and send them to people -- especially people you don't like -- and know that your purchase is helping keep some out of the hands of the unwary and potentially innocent. Under no circumstances, however, should you read the book before driving or operating heavy machinery. This work should convince you that Gene Kim is a visionary (assuming that your definition of "vision" includes "drug-induced hallucination").

Take 2:

I picked up this new book -- The Phoenix Project , by Gene Kim, et al. -- and could not put it down. You probably hear people say that about books in which they are engrossed. But I mean this literally: I happened to be reading it on my Kindle while repairing some holiday ornaments with superglue. You might say that the book stuck with me for a while.

There are people who will tell you that Gene Kim is a great author and raconteur. Those people, of course, are either trapped in Mr. Kim's employ or they drink heavily. Actually, one of those conditions invariably leads to the other, along with uncontrollable weeping, and the anguished rending of garments. Notwithstanding that, Mr. Kim's latest assault on les belles-lettres does indeed prompt this reviewer to some praise: I have not had to charge my health spending account for a zolpidem refill since I received the advance copy of the book! (Although it may be why I now need risperidone.)

I must warn you, gentle reader, that despite my steadfast sufferance in reading, I never encountered any mention of an actual Phoenix. I skipped ahead to the end, and there was no mention there, either. Neither did I notice any discussion of a massive conflagration nor of Arizona, either of which might have supported the reference to Phoenix . This is perhaps not so puzzling when one recollects that Mr. Kim's train of thought often careens off the rails with any random, transient manifestation corresponding to the meme "Ooh, a squirrel!" Rather, this work is more emblematic of a bus of thought, although it is the short bus, at that.

Despite my personal trauma, I must declare the book as a fine yarn: not because it is unduly tangled (it is), but because my kitten batted it about for hours with the evident joy usually limited to a skein of fine yarn. I have found over time it is wise not to argue with cats or women. Therefore, appease your inner kitten and purchase a copy of the book. Gene Kim's court-appointed guardians will thank you. Probably.

(Congratulations Gene, Kevin and George!)

More than passive defense

I was watching a video today (more on that later) that reminded me of some history. It also brought to mind that too few defenders these days build forensics capture into their systems to help identify intruders. They also don't have active defenses, countermeasures and chaff in place to slow down attackers and provide more warning of problems.

Back in the late 1980s and early 1990s, I quietly built some counterhacking and beaconing tools that I installed in a "fake front" machine on our local network. People who tried to break into it might get surprises and leave me log info about what they were up to, and things they downloaded would not do what they thought or might beacon me to indicate where the code went. This was long before honeypots were formalized, and before firewalls were in common use. Some of my experiences contributed to me writing the first few papers on software forensics (now called digital forensics), development of Tripwire, and several of my Ph.D. students's theses topics.

I didn't talk about that work much at the time for a variety of reasons, but I did present some of the ideas to students in classes over the years, and in some closed workshops. Tsutomu Shimomura, Dan Farmer and I traded some of our ideas on occasion, along with a few others; a DOD service branch contracted with a few companies to actually built some tools from my ideas, a few of which made it into commercial products. (And no, I never got any royalties or credit for them, either, or for my early work on firewalls, or security scanning, or.... I didn't apply for patents or start companies, unfortunately. It's interesting to see how much of the commercial industry is based around things I pioneered.)

I now regret not having actually written about my ideas at the time, but I was asked by several groups (including a few government agencies) not to do so because it might give away clues to attackers. A few of those groups were funding my grad students, so I complied. You can find a few hints of the ideas in the various editions of Practical Unix & Internet Security because I shared several of the ideas with my co-author, Simson Garfinkel, who had a lot of clever ideas of his own. He went on to found a company, Sandstorm Enterprises, to build and market some professional tools in roughly this space; I was a minor partner in that company. (Simson has continued to have lots of other great ideas, and is now doing wonderful things with disk forensics as a faculty member at the Naval Postgraduate School.)

Some of the ideas we all had back then continue to be reinvented, along with many new and improved approaches. Back in the 1980s, all my tools were in Unix (SunOS, mostly), but now there are possible options in many other systems, with Windows and Linux being the main problems. Of course, back in the 1980s the Internet wasn't used for commerce, Linux hadn't been developed, and Windows was not the widespread issue it it now. There also wasn't a WWW with its problems of cross-site scripting and SQL injection. Nonetheless, there were plenty of attackers, and more than enough unfound bugs in the software to enable attacks.

For the sake of history, I thought I'd document a few of the things I remember as working well, so the memories aren't lost forever. These are all circa 1989-1993:

  • Everything was built on a decoy system. I had control over my own Sun workstation. I configured it to have 2 separate accounts, with one (named spaf) exported via NFS to the CS department Sun, Pyramid and Sequent machines. My students and fellow faculty could access this directory, and I populated it with contents to make it look real. If I wanted to share something with my classes, I'd copy it to this account. A second account (rspaf) was on a separate partition, not exported, and locked down. It was my real account and where I got email. No system had rsh access in, nor any other indication that it existed — the support staff knew it was there, but almost no one else. (For real paranoia, I kept copies of sensitive files and source code on a Mac running OS 7, and it was off the network.)
  • My original idea for Tripwire came several years before Gene Kim arrived as a student and we built the actual Tripwire tool. I built a watcher program for bogus, "bait" mail files and attractively-named source files that would never be touched in normal operation. When accessed, their file times changed. The watcher "noticed" and would start taking repeated snapshots of active network connections and running processes. When lsof was written (at Purdue), I included it in the logging process. After a few minutes, my watcher would freeze the active network connections.
  • As a matter of good security practice, I disabled most of the network services spawned at startup or by inetd. In their places, I put programs that mimicked their output behavior, but logged everything sent to them. Thus, if someone tried to rsh into my system, they'd get output claiming incorrect password for most accounts. For spaf and root it would randomly act like it was connecting but very slow or an error would be printed. At the same time, this would send me email and (later) a page. This would allow me to run other monitoring tools. The tcp wrappers system was later independently developed by Wietse Venema, and the twist option provided the same kind of functionality (still useful).
  • Some attackers would try to "slurp" up interesting sounding directories without looking at all the contents. For instance, a common tactic was to take any directory labeled "security" and the mail spool. For a while, attackers were particularly interested in getting my copy of the Morris Internet Worm, so any directory with "worm" in it would be copied out in its entirety. I built a small utility that would take any file (such as a mail file or binary) and make it HUGE using the Unix sparse file structure. On disk the files might only be a few thousand bytes long, and you could read it normally, but any copy program thought they were gigabytes in length. Thus, any attempt to copy them offsite would result in very long, uncompleted copies (which left network connections open to trace) and sometimes filled up the attackers' disks.
  • The previous idea was partially inspired by one of Tsutomu's tricks. The authd daemon, implementing the ident protocol (RFC 1413), was somewhat popular. It could be useful in a LAN, but in the Internet at large it was not trustworthy; to make this point, several admins had it always return some made-up names when queried. Tsutomu took this a step further and when a remote system connected to the ident port on his machine, it would get an unending string of random bits. Most authd clients would simply accept whatever they were given to write to the local log file. Connecting to Tsutomu's machine was therefore going to lead to a disk full problem. If the client ran as root, it would not be stopped by any limits or quotas — and it usually logged to the main system partition. Crash. As Tsutomu put it, "Be careful what you ask for or you may get more of it than you counted on."
  • I sprinkled altered program source on my fake account, including many that looked like hacking tools. The source code was always subtly altered to neuter the tool. For example, some Unix password cracking tools had permuted S-boxes so that, when presented with a password hash to attack, the program would either run a long time without a result, or would give a result that would not work on any target machine. I also had a partial copy of the Morris Worm there, with subtle permutations made to prevent it from spreading if compiled, and the beacon changed to ping me instead of Berkeley. (See my analysis paper if you don't get the reference.)
  • I also had some booby-trapped binaries with obfuscated content. One was named "God" and had embedded text strings for the usage message that implied that there were options to become root, take over the network, and other tasty actions. However, if run, it would disable all signals, and prompt the user with the name of every file in her home directory, one at a time. Any response to the prompt would result in "Deleted!" followed by the next prompt. Any attempt to stop the program would cause it to print "Entering automatic mode" where every half-second it would state that it was deleting the next file. Meanwhile, it would be sending me logging information about who was running it and where. If run on a Purdue machine, it didn't actually delete anything — I simply used it to see who was poking in my account and running things they didn't know anything about (usually, my students). It also gave them a good scare. If taken and run on a non-Purdue machine, well, it was not so benign.

There were many other tools and tripwires in place, of course, but the above were some of the most successful.

What does successful mean? Well, they helped me to identify several penetrations in progress, and get info on the attackers. I also identified a few new attacks, including the very subtle library substitution that was documented in @Large: The Strange Case of the World's Biggest Internet Invasion. The substitute with backdoor in place had the identical size, dates and simple checksum as the original so as to evade tools such as COPS and rdist. Most victims never knew they had been compromised. My system caught the attack in progress. I was able to share details with the Sun response team — and thereafter they started using MD5 checksums on their patch releases. That incident also inspired some of my design of Tripwire.

In another case, I collected data on some people who had broken into my system to steal the Morris Worm source code. The attacks were documented in the book Underground . The author, Suelette Dreyfus, assisted by Julian Assange (yes, the Wikileaks one), never bothered to contact me to verify what she wrote. The book suggests that my real account was compromised, and source code taken. However, it was the fake account, my security monitors froze the connection after a few minutes, and the software that was accessed was truncated and neutered. Furthermore, the flaws that were exploited to get in were not on my machine — they were on a machine operated by the CS staff.   (Dreyfuss got several other things wrong, but I'm not going to do a full critique.)

There were a half-dozen other incidents where I was able to identify new attacks (now known as zero-day exploits) and get the details to vendors. But after a while, interest dropped off in attacking my machine as new, more exciting opportunities for the kiddies came into play, such as botnets and DDOS attacks. And maybe the word spread that I didn't keep anything useful or interesting on my system. (I still don't.) It's also the case that I got much more interested in issues that don't involve the hands-on, bits & bytes parts of security — I'm now much more interested in fundamental science and policy aspects. I leave the hands-on aspects to the next generation. So, I'm not really a challenge now — especially as I do not administer my system anymore — it's done by staff.

I was reminded of all this when someone on Twitter posted the URL of a video taken at Notacon 2011 (Funnypots and Skiddy Baiting: Screwing with those that screw with you by Adrian "Iron Geek" Crenshaw). It is amusing and reminded me of the stories, above. It also showed that some of the same techniques we used 20 years ago are still applicable today.

Of course, that is also depressing. Now, nearly 20 years later, lots of things have changed but unfortunately, security is a bigger problem, and law enforcement is still struggling to keep up. Too many intrusions occur without being noticed, and too little information is available to track the perps.

There are a few takeaways from all the above that the reader is invited to consider:

  • Assume your systems will be penetrated if they are on the network. Things you don't control are likely to be broken. Therefore, plan ahead and keep the really sensitive items on a platform that is off the net. (If the RSA folks had done this, the SecurID breach might not have resulted in anything.)
  • Install localized tripwires and honeypots that can be monitored for evidence of intrusion. Don't depend on packaged solutions alone — they are known quantities that attackers can avoid or defeat.
  • Don't believe everything you read unless you know the story has been verified with original sources.
  • Consider encrypting or altering critical files so they aren't usable "as is" if taken.
  • Be sure you are proactively logging information that will be useful once you discover a compromise.

Also, you might watch Iron Geek's video to inspire some other ideas if you are interested in this general area — it's a good starting point. (And another, related and funny post on this general topic is here, but is possibly NSFW.)

In conclusion, I'll close with my 3 rules for successful security:

  1. Preparation in advance is always easier than clean up afterwards.
  2. Don't tell everything you know.

Bullies, Pirates and Lulz

Yet another breach of information has occurred, this time from the Arizona Department of Public Safety. A large amount of data about law enforcement operations was exposed, as was a considerable amount of personnel information. As someone who has been working in information security and the implications of technology for nearly 30 years, two things come to mind.

First, if a largely uncoordinated group could penetrate the systems and expose all this information, then so could a much more focused, well-financed, and malevolent group — and it would not likely result in postings picked up by the media. Attacks by narcotics cartels, organized crime, terrorists and intelligence agencies are obvious threats; we can only assume that some have already succeeded but not been recognized or publicized. And, as others are noting, this poses a real threat to the physical safety of innocent people. Yes, in any large law enforcement organization there are likely to be some who are corrupt (the claimed reason behind the attack), but that is not reason to attack them all. Some of the people they are arrayed against are far worse.

For example, there are thousands (perhaps tens of thousands) of kidnappings in Mexico for ransom, with many of the hostages killed rather than freed after payment. Take away effective law enforcement in Arizona, and those gangs would expand into the U.S. where they could demand bigger ransoms. The hackers, sitting behind a keyboard removed from gang and street violence, safe from forcible rape, and with enough education to be able to avoid most fraud, find it easy to cherry-pick some excesses to complain about. But the majority of people in the world do not have the education or means to enjoy that level of privileged safety. Compared to police in many third-world countries where extortion and bribes are always required for any protection at all, U.S. law enforcement is pretty good. (As is the UK, which has also recently been attacked.)

Ask yourself what the real agenda is of a group that has so far only attacked law enforcement in some of the more moderate countries, companies without political or criminal agendas, and showing a total disregard for collateral damage. Ask why these "heroes" aren't seeking to expose some of the data and names of the worst drug cartels, or working to end human trafficking and systematic rape in war zones, or exposing the corruption in some African, South American & Asian governments, or seeking to damage the governments of truly despotic regimes (e.g., North Korea, Myanmar), or interfering with China's online attacks against the Dalai Lama, or leaking memos about willful environmental damage and fraud by some large companies, or seeking to destroy extremist groups (such as al Qaida) that oppress woman and minorities and are seeking weapons of mass destruction.

Have you seen one report yet about anything like the above? None of those actions would necessarily be legal, but any one of them would certainly be a better match for the claimed motives. Instead, it is obvious that  these individuals and groups are displaying a significant political and moral bias — or blindness — they are ignoring the worst human rights offenders and criminals on the planet. It seems they are after the ego-boosting publicity, and concerned only with themselves. The claims of exposing evil is intended to fool the naive.

In particular, this most recent act of exposing the names and addresses of family members of law enforcement, most of whom are undoubtedly honest people trying to make the world a safer place, is not a matter of "Lulz" — it is potentially enabling extortion, kidnapping, and murder. The worst criminals, to whom money is more important than human life, are always seeking an opportunity to neutralize the police. Attacking family members of law enforcement is common in many countries, including Mexico, and this kind of exposure further enables it now in Arizona. The data breach is attacking some of the very people and organizations trying to counter the worst criminal and moral abuses that may occur, and worse, their families.

Claiming that, for instance, that the "War on Drugs" created the cartels and is morally equivalent (e.g., response #13 in this) is specious. Laws passed by elected representatives in the U.S. did not cause criminals in Mexico to kidnap poor people, force them to fight to the death for the criminals' amusement, and then force the survivors to act as expendable drug mules. The moral choices by criminals are exactly that — moral choices. The choice to kidnap, rape, or kill someone who objects to your criminal behavior is a choice with clear moral dimensions. So are the choices of various hackers who expose data and deface systems.

When I was growing up, I was the chubby kid with glasses. I didn't do well in sports, and I didn't fit in with the groups that were the "cool kids." I wasn't into drinking myself into a stupor, or taking drugs, or the random vandalism that seemed to be the pasttimes of those very same "cool kids." Instead, I was one of the ones who got harassed, threatened, my homework stolen, and laughed at. The ones who did it claimed that it was all in fun — this being long before the word "lulz" was invented. But it was clear they were being bullies, and they enjoyed being bullies. It didn't matter if anyone got hurt, it was purely for their selfish enjoyment. Most were cowards, too, because they would not do anything that might endanger them, and when they could, they did things anonymously. The only ones who thought it was funny were the other dysfunctional jerks. Does that sound familiar?

Twenty years ago, I was objecting to the press holding up virus authors as unappreciated geniuses. They were portrayed as heroes, performing experiments and striking blows against the evil computer companies then prominent in the field. Many in the public and press (and even in the computing industry) had a sort of romantic view of them — as modern, swashbuckling, electronic pirates, of the sorts seen in movies. Now we can see the billions of dollars in damage wrought by those "geniuses" and their successors with Zeus and Conficker and the rest. The only difference is of time and degree — the underlying damage and amoral concern for others is simply visible to more people now. (And, by the way, the pirates off Somalia and in the Caribbean, some of whom simply kill their victims to steal their property, are real pirates, not the fictional, romantic versions in film.)

The next time you see a news article about some group, by whatever name, exposing data from a gaming company or law enforcement agency, think about the real evil left untouched. Think about who might actually be hurt or suffer loss. Think about the perpetrators hiding their identities, attacking the poorly defended, and bragging about how wonderful and noble and clever they are. Then ask if you are someone cheering on the bully or concerned about who is really getting hurt. And ask how others, including the press, are reporting it. All are choices with moral components. What are yours?

Update: June 26

I have received several feedback comments to this (along with the hundreds of spam responses). Several were by people using anonymous addresses. We don't publish comments made anonymously or containing links to commercial sites. For this post, I am probably not going to pass through any rants, at least based on what I have seen. Furthermore, I don't have the time (or patience) to do a point-by-point commentary on the same things, again and again. However, I will make a few short comments on what I have received so far.

Several responses appear to be based on the assumption that I don't have knowledge or background to back up some of my statements. I'm not going to rebut those with a list of specifics. However, people who know what I've been doing over the few decades (or bothered to do a little research) — including work with companies, law enforcement, community groups, and government agencies — would hardly accuse me of being an observer with only an academic perspective.

A second common rant is that the government has done some bad things, or the police have done something corrupt, or corporations are greedy, and those facts somehow justify the behavior I described. Does the fact that a bully was knocked around by someone else and thus became a bully mean that if you are the victim, it's okay? If so, then the fact that the U.S. and U.K. have had terrorist attacks that have resulted in overly intrusive laws should make it all okay for you. After all, they had bad things happen to them, so their bad behavior is justified, correct? Wrong. That you became an abuser of others because you were harmed does not make it right. Furthermore, attacks such as the ones I discussed do nothing to fix those problems, but do have the potential to harm innocent parties as well as give ammunition to those who would pass greater restrictions on freedom. Based on statistics (for the US), a significant number of the people whining about government excess have not voted or bothered to make their opinions known to their elected representatives. The more people remain uninvolved, the more it looks like the population doesn't care or approves of those excesses, including sweetheart deals for corporations and invasions of privacy. Change is possible, but it is not going to occur by posting account details of people subscribed to Sony services, or giving out addresses and names of families of law enforcement officers, or defacing the NPR website. One deals with bullies by confronting them directly.

The third most common rant so far is to claim that it doesn't make any difference, for one reason or another: all the personal information is already out there on the net or will be soon, that the government (or government of another country) or criminals have already captured all that information, that it doesn't cost anything, security is illusory, et al. Again, this misses the point. Being a bully or vandal because you think it won't make any difference doesn't excuse the behavior. Because you believe that the effects of your behavior will happen anyhow is no reason to hasten those effects. If you believe otherwise, then consider: you are going to die someday, so it doesn't make a difference if you kill yourself, so you might as well do it now. Still there? Then I guess you agree that each act has implications that matter even if the end state is inevitable.

Fourth, some people claim that these attacks are a "favor" to the victims by showing them their vulnerabilities, or that the victims somehow deserved this because their defenses were weak. I addressed these claims in an article published in 2003. In short, blaming the victim is inappropriate. Yes, some may deserve some criticism for not having better defenses, but that does not justify an attack nor serve as a defense for the attackers. It is no favor either. If you are walking down a street at night and are assaulted by thugs who beat you with 2x4s and steal your money, you aren't likely to lie bleeding in the street saying to yourself "Gee, they did me a huge favor by showing I wasn't protected against a brutal assault. I guess I deserved that." Blaming the victim is done by the predators and their supporters to try to justify their behavior. And an intrusion or breach, committed without invitation or consent, is not a favor — it is a crime.

Fifth, if you support anarchy, then that is part of your moral choices. It does not invalidate what I wrote. I believe that doing things simply because they amuse you is a very selfish form of choice, and is the sort of reasoning many murderers, rapists, pedophiles and arsonists use to justify their actions. In an anarchy, they'd be able to indulge to their hearts content. Lotsa lulz. But don't complain if I and others don't share that view.

I am going to leave it here. As I said, I'm not interested in spending the next few weeks arguing on-line with people who are trying to justify behavior as bullies and vandals based on faulty assumptions.

A Cautionary Incident

Recently, Amazon's cloud service failed for several customers, and has not come back fully for well over 24 hours. As of the time I write this, Amazon has not commented as to what caused the problem, why it took so long to fix, or how many customers it affected.

It seems a client of Amazon was not able to contact support, and posted in a support forum under the heading "Life of our patients is at stake - I am desperately asking you to contact." The body of the message was that "We are a monitoring company and are monitoring hundreds of cardiac patients at home. We were unable to see their ECG signals"

What ensued was a back-and-forth with others incredulous that such a service would not have a defined disaster plan and alternate servers defined, with the original poster trying to defend his/her position. At the end, as the Amazon service slowly came back, the original poster seemed to back off from the original claim, which implies either an attempt to evade further scolding (and investigation), or that the original posting was a huge exaggeration to get attention. Either way, the prospect of a mission critical system depending on the service was certainly disconcerting.

Personnel from Amazon apparently never contacted the original poster, despite that company having a Premium service contract.

25 or so years ago, Brian Reid defined a distributed system as " where I can't get my work done because a computer I never heard of is down." (Since then I've seen this attributed to Leslie Lamport, but at the time heard it attributed to Reid.) It appears that "The Cloud" is simply today's buzzword for a distributed system. There have been some changes to hardware and software, but the general idea is the same — with many of the limitations and cautions attendant thereto, plus some new ones unique to it. Those who extol its benefits (viz., cost) without understanding the many risks involved (security, privacy, continuity, legal, etc.) may find themselves someday making similar postings to support fora — as well as "position wanted" sites.

The full thread is available here.