CERIAS Weblogs » Secure IT Practices

It occurred to me that virtual machine monitors (VMMs) provide similar functionality to that of operating systems. Virtualization supports functions such as these:

1. Availability
• Minimized downtime for patching OSes and applications
• Restart a crashed OS or server
2. Scalability
• More or different images as demand changes
3. Isolation and compartmentalization
4. Better hardware utilization
5. Hardware abstraction for OSes
• Support legacy platforms

Compare it to the list of operating system duties:

1. Availability
• Minimized downtime for patching applications
• Restart crashed applications
2. Scalability
• More or different processes as demand changes
3. Isolation and compartmentalization
• Protected memory
• Accounts, capabilities
4. Better hardware utilization (with processes)
5. Hardware abstraction for applications

The similarity suggests that virtualization solutions compete with operating systems. I now believe that a part of their success must be because operating systems do not satisfy these needs well enough, not taking into account the capability to run legacy operating systems or entirely different operating systems simultaneously. Typical operating systems lack security, reliability and ease of maintenance. They have drivers in kernel space; Windows Vista thankfully now has them in user space, and Linux is moving in that direction. The complexity is staggering. This is reflected in the security guidance; hardening guides and “benchmarks” (essentially an evaluation of configuration settings) are long and complex. The attempt to solve the federal IT maintenance and compliance problem created the SCAP and XCCDF standards, which are currently ambiguously specified, buggy and very complex. The result of all this is intensive, stressful and inefficient maintenance in an environment of numerous and unending vulnerability advisories and patches.

What it looks like is that we have sinking boats, so we’re putting them inside a bigger, more powerful boat, virtualization. In reality, virtualization typically depends on another, full-blown operating system.

VMWare ESX Server runs its own OS with drivers. Xen and offerings based on it have a full, general purpose OS in domain 0, in control and command of the VMM (notwithstanding disaggregation). Microsoft’s “Hyper-V” requires a full-blown Windows operating system to run it. So what we’re doing is really exchanging an untrusted OS for another, that we should trust more for some reason. This other OS also needs patches, configuration and maintenance. Now we have multiple OSes to maintain! What did we gain? We don’t trust OSes but we trust “virtualization” that depends on more OSes? At least ESX is “only” 50 MB, simpler and smaller than the others, but the number of defects/MB of binary code as measured by patches issued is not convincing.

I’m now not convinced that a virtualization solution + guest OS is significantly more secure or functional than just one well-designed OS could be, in theory. Defense in depth is good, but the extent of the spread of virtualization may be an admission that we don’t trust operating systems enough to let them stand on their own. The practice of wiping and reinstalling an OS after an application or an account is compromised, or deploying a new image by default suggests that there is little trust in the depth provided by current OSes.

As for ease of management and availability vs patching, I don’t see why operating systems would be unable to be managed in a smart manner just like ESX is, migrating applications as necessary. ESX is an operating system anyway… I believe that all the special things that a virtualization solution does for functionality and security, as well as the “new” opportunities being researched, could be done as well by a trustworthy, properly designed OS; there may be a thesis or two in figuring out how to implement them back in an operating system.

What virtualization vendors are really doing is a clever way to smoothly replace one operating system with another. This may be how an OS monopoly could be dislodged, and perhaps would explain the virtualization-unfriendly clauses in the licensing options for Vista: virtualization could become a threat to the dominance of Windows, if application developers started coding for the underlying OS instead of the guest. Of course, even with a better OS we’d still need virtualization for testbeds like ReAssure, and for legacy applications. Perhaps ReAssure could help test new, better operating systems.
(This text is the essence of my presentation in the panel on virtualization at the 2008 CERIAS symposium).

Related reading:
Heiser G et al. (2007) Towards trustworthy computing systems: Taking microkernels to the next level. ACM Operating Systems Review, 41
Tanenbaum AS, Herder JN and Bos H (2006) Can we make operating systems reliable and secure? Computer, 39

The EDUCAUSE security mailing list has yet (another) discussion on password policies. I’ve blogged about this general issue several times in the past, but maybe it is worth revisiting.

Someone on the list wrote:

Here is my question - does anyone have the data on how many times a hack (attack) has occurred associated to breaking the “launch codes” from outside of the organization? The last information I gleaned from the FBI reports (several years ago) indicated that 70 percent of hackings (attacks) were internal.

My most recent experience with intrusions has had nothing to do with a compromised password, rather an exploit of some vunerability in the OS, database, or application.

I replied:

I track these things, and I cannot recall the last time I saw any report of an incident caused by a guessed password. Most common incidents are phishing, trojans, snooping, physical theft of sensitive media, and remote exploitation of bugs.

People devote huge amounts of effort to passwords because it is one of the few things they think they can control.

Picking stronger passwords won’t stop phishing. It won’t stop users downloading trojans. It won’t stop capture of sensitive transmissions. It won’t bring back a stolen laptop (although if the laptop has proper encryption it *might* protect the data). And passwords won’t ensure that patches are in place but flaws aren’t.

Creating and forcing strong password policies is akin to being the bosun ensuring that everyone on the Titanic has locked their staterooms before they abandon ship. It doesn’t stop the ship from sinking or save any lives, but it sure does make him look like he’s doing something important…..

That isn’t to say that we should be cavalier about setting passwords. It is important to try to set strong passwords, but once reasonably good ones are set in most environments the attacks are going to come from other places — password sniffing, exploitation of bugs in the software, and implantation of trojan software.

As a field, we spend waaaaay too much time and resources on palliative measures rather than fundamental cures. In most cases, fiddling with password rules is a prime example. A few weeks ago, I blogged about a related issue.

Security should be based on sound risk assessment, and in most environments weak passwords don’t present the most significant risk.

A news story that hit the wires last week was that someone with a history of breaking into systems, who had “reformed” and acted as a security consultant, was arrested for new criminal behavior. The press and blogosphere seemed to treat this as surprising. They shouldn’t have.

I have been speaking and writing for nearly two decades on this general issue, as have others (William Hugh Murray, a pioneer and thought leader in security, is one who comes to mind). Firms that hire “reformed” hackers to audit or guard their systems are not acting prudently any more than if they hired a “reformed” pedophile to babysit their kids. First of all, the ability to hack into a system involves a skill set that is not identical to that required to design a secure system or to perform an audit. Considering how weak many systems are, and how many attack tools are available, “hackers” have not necessarily been particularly skilled. (The same is true of “experts” who discover attacks and weaknesses in existing systems and then publish exploits, by the way — that behavior does not establish the bona fides for real expertise. If anything, it establishes a disregard for the community it endangers.)

More importantly, people who demonstrate a questionable level of trustworthiness and judgement at any point by committing criminal acts present a risk later on. Certainly it is possible that they will learn the error of their ways and reform. However, it is also the case that they may slip later and revert to their old ways. Putting some of them in situations of trust with access to items of value is almost certainly too much temptation. This has been established time and again in studies of criminals of all types, especially those who commit fraud. So, why would a prudent manager take a risk when better alternatives are available?

Even worse, circulating stories of criminals who end up as highly-paid consultants are counterproductive, even if they are rarely true. That is the kind of story that may tempt some without strong ethics to commit crimes as a shortcut to fame and riches. Additionally, it is insulting to the individuals who work hard, study intently, and maintain a high standard of conduct in their careers — hiring criminals basically states that the honest, hardworking real experts are fools. Is that the message we really want to put forward?

Luckily, most responsible managers now understand, even if the press and general public don’t, that criminals are simply that — criminals. They may have served their sentences, which now makes them former criminals…but not innocent. Pursuing criminal activity is not — and should not be — a job qualification or career path in civilized society. There are many, many historical examples we can turn to for examples, including those of hiring pirates as privateers and train robbers as train guards. Some took the opportunity to go straight, but the instances of those who abused trust and made off with what they were protecting illustrate that it is a big risk to take. It also is something we have learned to avoid. We are long past the point where those of us in computing should get with the program.

So, what of the argument that there aren’t enough real experts, or they cost too much to hire? Well, what is their real value? If society wants highly-trained and trustworthy people to work in security, then society needs to devote more resources to support the development of curriculum and professional standards. And it needs to provide reasonable salaries to those people, both to encourage and reward their behavior and expertise. We’re seeing more of that now than a dozen years ago, but it is still the case that too many managers (and government officials) want security on the cheap, and then act surprised when they get hacked. I suppose they also buy their Rolex and Breitling watches for $50 from some guy in a parking lot and then act surprised and violated when the watch stops a week later. What were they really expecting?

Here’s an interesting story about what people can do if they gain access to streaming video at a poorly-protected site. If someone on the other end of the phone is really convincing, what could she get the victims to do?

FBI: Strip Or Get Bombed Threat Spreads - Local News Story - KPHO Phoenix:

The Greek Cell Phone Incident

A great story involving computers and software, even though the main hack was against cell phones:

IEEE Spectrum: The Athens Affair. From this we can learn all sorts of lessons about how to conduct a forensic investigation, retention of logs, wiretapping of phones, and more.

Now, imagine VoIP and 802.11 networking and vulnerabilities in routers and…. — the possibilities get even more interesting. I suspect that there’s a lot more eavesdropping going on than most of us imagine, and certainly more than we discover.

NRC Report Released

Last week, the National Research Council announced the release of a new report: Towards a Safer and More Secure Cyberspace. The report is notable in a number of ways, and should be read carefully by anyone interested in cyber security. I think the authors did a great job with the material, and they listened to input from many sources.

There are 2 items I specifically wish to note:

1. I really dislike the “Cyber Security Bill of Rights” listed in the report. It isn’t that I dislike the goals they represent — those are great. The problem is that I dislike the “bill of rights” notion attached to them. After all, who says they are “rights”? By what provenance are they granted? And to what extremes do we do to enforce them? I believe the goals are sound, and we should definitely work towards them, but let’s not call them “rights.”
2. Check out Appendix B. Note all the other studies that have been done in recent years pointing out that we are in for greater and greater problems unless we start making some changes. I’ve been involved with several of those efforts as an author — including the PITAC report, the Infosec Research Council Hard Problems list, and the CRA Grand Challenges. Maybe the fact that I had no hand in authoring this report means it will be taken seriously, unlike all the rest. More to the point, people who put off the pain and expense of trying to fix things because “Nothing really terrible has happened yet” do not understand history, human nature, or the increasing drag on the economy and privacy from current problems. The trends are fairly clear in this report: things are not getting better.

Evolution of Computer Crime

Speaking of my alleged expertise at augury, I noted something in the news recently that confirmed a prediction I made nearly 8 years ago at a couple of invited talks: that online criminals would begin to compete for “turf.” The evolution of online crime is such that the “neighborhood” where criminals operate overlaps with others. If you want the exclusive racket on phishing, DDOS extortion, and other such criminal behavior, you need to eliminate (or absorb) the competition in your neighborhood. But what does that imply when your “turf” is the world-wide Internet?

The next step is seeing some of this spill over into the physical world. Some of the criminal element online is backed up by more traditional organized crime in “meat space.” They will have no compunction about threatening — or disabling — the competition if they locate them in the real world. And they may well do that because they also have developed sources inside law enforcement agencies and they have financial resources at their disposal. I haven’t seen this reported in the news (yet), but I imagine it happening within the next 2-3 years.

Of course, 8 years ago, most of my audiences didn’t believe that we’d see significant crime on the net — they didn’t see the possibility. They were more worried about casual hacking and virus writing. As I said above, however, one only needs to study human nature and history, and the inevitability of some things becomes clear, even if the mechanisms aren’t yet apparent.

The Irony Department

GAO reported a little over a week ago that DHS had over 800 attacks on their computers in two years. I note that the report is of detected attacks. I had one top person in DC (who will remain nameless) refer to DHS as “A train wreck crossed with a nightmare, run by inexperienced political hacks” when referring to things like TSA, the DHS cyber operations, and other notable problems. For years I (and many others) have been telling people in government that they need to set an example for the rest of the country when it comes to cyber security. It seems they’ve been listening, and we’ve been negligent. From now on, we need to stress that they need to set a good example.

[posted with ecto]

In my last post, I wrote about the problems brought about by complexity. Clearly, one should not take the mantra of “simplification” too far, and end up with a situation where everything is uniform, simple, and (perhaps) inefficient. In particular, simplification shouldn’t be taken to the point where diversity is sacrificed for simple uniformity.

Nature penalizes monocultures in biological systems. Monocultures are devastated by disease and predators because they have insufficient diversity to resist. The irish potato famine, the emerald ash borer, and the decimation of the Aztecs by smallpox are all examples of what happens when diversity is not present. Nature naturally promotes diversity to ensure a robust population.

We all practice diversity in our everyday lives. Diversity of motor vehicles, for instance supports fitness for purpose — a Camero, is not useful for hauling dozens of large boxes of materials. For that, we use a truck. However, for one person to get from point A to point B in an economical fashion, a truck is not the best choice. It might be cheaper and require less training to use the same vehicle for everything, but there are advantages to diversity. Or tableware — we have (perhaps) too many forks and spoon types in a formal placesetting, but try eating soup with a fork and you discover that some differentiation is useful!

In computing, competition has resulted in advances in hardware and software design. Choice among products has kept different approaches moving forward. Competition for research awards from DARPA and NSF has encouraged deeper thought and more focused proposals (and resultant development). Diversity in operating systems and programming languages brought many advancements in the era 1950-2000. However, expenses and attempts to cut staff have led to widespread homogenization of OS, applications, and languages over approximately the last decade.

Despite the many clear benefits of promoting diversity, too many organizations have adopted practices that prevent diversity of software and computing platforms. For example, the OMB/DoD Common Desktop initiative is one example where the government is steering personnel towards a monoculture that is more maintainable day-to-day, but which is probably more vulnerable to zero-day attacks and malware.

Disadvantages of homogeneity:

• greater susceptibility to zero-day vulnerabilities and attacks
• “box canyon” effect of being locked into a vendor for future releases
• reduced competition to improve quality
• reduced competition to reduce price and/or improve services
• reduced number of algorithms and approaches that may be explored
• reduced fitness for particular tasks
• simpler for adversaries to map and understand networks and computer use
• increased likelihood that users will install unauthorized software/hardware from outside sources

Advantages of homogeneity:

• larger volume for purchases
• only one form of tool, training, etc needed for support
• better chance of compatible upgrade path
• interchangeability of users and admins
• more opportunities for reuse of systems

Disadvantages of heterogeneity:

• more complexity so possibly more vulnerabilities
• may not be as interoperable
• may require more training to administer
• may not be reusable to the same extent as homogeneous systems

Advantages of heterogeneity:

• when at a sufficient level greater resistance to malware
• highly unlikely that all systems will be vulnerable to a single new attack
• increased competition among vendors to improve price, quality and performance
• greater choice of algorithms and tools for particular tasks
• more emphasis on standards for interoperability
• greater likelihood of customization and optimization for particular tasking
• greater capability for replacement systems if a vendor discontinues a product or support

Reviewing the above lists makes clear that entities concerned with self-continuation and operation will promote diversity, despite some extra expense and effort. The potential disadvantages of diversity are all things that can be countered with planning or budget. The downsides of monocultures, however, cannot be so easily addressed.

Dan Geer wrote an interesting article for Queue Magazine about diversity, recently. It is worth a read.

The simplified conclusion: diversity is good to have.

It is common practice to make forms more user-friendly by giving immediate feedback on the inputs with client-side scripting. Everyone with a bit of secure programming knowledge knows, however, that the server side needs to do the final input validation. If the two validations are not equivalent, then an input that passes client-side validation may be rejected later, confusing and annoying the customer, or the client-side validation may be needlessly restrictive. Another problem is when the form stops working if JavaScript is disabled, due to the way input validation was attempted.

I was delighted to discover that the regular expression syntax in JavaScript and Ruby match, and the matching differs only in greedy vs non-greedy behavior, and not whether a match is possible or not. This means that regular expressions describing a white list of correct inputs can be used for both (this probably works for Perl, Python and PHP as well but I haven’t checked).

In the code for ReAssure, all inputs are defined by classes that create the html for forms, as well as perform input validation. This means that the regular expression can be defined in a single place, when the class is instantiated:

   def initialize(...)
      (...)
      @regexp = Regexp.new(/^\d+$/) # positive integer
   end

This regular expression can be used to perform the initial server-side input validation:

   def validate(input)
      if input == nil
         unescaped = default()
      else
         unescaped = CGI.unescapeHTML(input.to_s.strip)
      end
      unescaped.scan(@regexp) { |match|
         return @value = match.untaint
      }
      if input != ''
         raise 'Input "' + @ui_name + '" is not valid'
      end
   end

To perform client-side input validation, the onblur event is used to trigger validation when focus is lost. The idea is to make the input red and bold (for color-blind people) when validation fails, and green when it passes. The onfocus event is used to restore the input to a neutral state while editing (this is the Ruby code that generates the form html):

   def form
      $cgi.input('NAME'=>@name, 'VALUE'=>to_html(), 'onblur' => onblur(),
          'onfocus' => onfocus())
   end

   def onblur()
      return "if (this.value.search(/" + @regexp.source + "/) < 0)
          {this.className = 'bad'} else {this.className = 'good'};"
   end

   def onfocus()
      return "this.className = 'normal';"
   end

where the classes “bad”, “good” and “normal” are specified in a style sheet (CSS).
There are cases when more validation may happen later on the server side, e.g., if an integer must match an existing key in a database that the user may be allowed to reference. Could the extra validation create a mismatch? Perhaps. However, in these cases the client-side interface should probably be a pre-screened list and not a free-form input, so the client would have to be malicious to fail server-side validation. It is also possible to add range (for numbers) and size (for strings) constraints in the “onblur” JavaScript. In the case of a password field, the JavaScript contains several checks matching the rules on the server side. So, a lone regular expression may not be sufficient for complete input validation, but it is a good starting point.

Note that the form still works even if JavaScript is disabled! As you can see, it is easy to perform client-side validation without forcing everyone to turn on JavaScript

Jim Horning suggested a topic to me a few weeks ago as a result of some email I sent him.

First, as background, consider that phishing and related frauds are increasingly frequent criminal activities on the WWW. The basic mechanism is to fool someone into visiting a WWW page that looks like it belongs to a legitimate organization with which the user does business. The page has fields requesting sensitive information from the user, which is then used by the criminals to commit credit card fraud, bank fraud or identity theft.

Increasingly, we have seen that phishing email and sites are also set up to insert malware into susceptible hosts. IE on Windows is the prime target, but attacks are out there for many different browsers and systems. The malware that is dropped can be bot clients, screen scrapers (to capture keystrokes at legitimate pages), and html injectors (to modify legitimate pages to ask for additional information). It is important to try to keep from getting any of this malware onto your system. One aspect of this is to be careful clicking on URLs in your email, even if they seem to come from trusted sources because email can be spoofed, and mail can be sent by bots on known machines.

How do you check a URL? Well, there are some programs that help, but the low-tech way is to look at the raw text of a URL before you visit it, to ensure that it references the site and domain you expected.

But consider the case of short-cut URLs. There are many sites out there offering variations on this concept, with the two I have seen used most often being “TinyURL” and “SnipURL”. The idea is that if you have a very long URL that may get broken when sent in email, or that is simply too difficult to remember, you submit it to one of these services and you get a shortened URL back. With some services, you can even suggest a nickname. So, for example, short links to the top level of my blog are <http://tinyurl.com/2geym5>, <http://snipurl.com/1ms17> and <http://snurl.com/spafblog>.

So far, this is really helpful. As someone who has had URLs mangled in email, I like this functionality.

But now, let’s look at the dark side. If Jim gets email that looks like it is from me, with a message that says “Hey Jim, get a load of this!” with one of these short URLs, he cannot tell by looking at the URL whether it points somewhere safe or not. If he visits it, it could be a site that is dangerous to visit (Well, most URLs I send out are dangerous in one way or another, but I mean dangerous to his computer. :-)). The folks at TinyURL have tried to address this by adding a feature so that if you visit <http://preview.tinyurl.com/2geym5> you will get a preview of what the URL resolves to; you can set this (with cookies) as your default. That helps some.

But now step deeper into paranoia. Suppose one of these sites was founded by fraudsters with the intent of luring people into using it. Or the site gets acquired by fraudsters, or hijacked. The code could be changed so that every time someone visits one of these URLs, some code at the redirect site determines the requesting browser, captures some information about the end system, then injects some malicious javacode or ActiveX before passing the connection to the “real” site. Done correctly, this would result in largely transparent compromise of the user system. According to the SnipURL statistics page, as of midnight on May 30th there have been nearly a billion clicks on their shortened URLs. That’s a lot of potential compromises!

Of course, one of the factors to make this kind of attack work is for the victim to be running a vulnerable browser. Unfortunately, there have been many vulnerabilities found for both IE and Firefox, as well as some of the less well-known browsers. With users seeking more functionality in their browsers, and web designers seeking more latitude in what they deliver, we are likely to continue to see browser exploits. Thus, there is likely to be enough of a vulnerable population to make this worthwhile. (And what browser are you using to read this with?)

I should make it clear that I am not suggesting that any of these services really are being used maliciously or for purposes of fraud. I am a happy and frequent user of both TinyURL and SnipURL myself. I have no reason to suspect anything untoward from those sites, and I certainly don’t mean to suggest anything sinister. (But note that neither can I offer any assurances about their motives, coding, or conduct.) Caveat emptor.

This post is simply intended as commentary on security practices. Thinking about security means looking more deeply into possible attack vectors. And one of the best ways to commit such attacks is to habituate people into believing something is safe, then exploiting that implicit trust relationship for bad purposes.

Hmm, reminds me of a woman I used to date. She wasn’t what she appeared, either…. But that’s a story for a different post.

[posted with ecto]

In my last post, I ranted about a government site making documents available only in Word. A few people have said to me “Get over it — use OpenOffice instead of the Microsoft products.” The problem is that those are potentially dangerous too — there is too much functionality (some of it may be undocumented, too) in Word (and Office) documents.

Now, we have a virus specific to OpenOffice. We’ve had viruses that run in emulators, too. Trying to be compatible with something fundamentally flawed is not a security solution. That’s also my objection to virtualization as a “solution” to malware.

I don’t mean to be unduly pejorative, but as the saying goes, even if you put lipstick on a pig, it is still a pig.

Word and the other Office components are useful programs, but if MS really cared about security, they would include a transport encoding that didn’t include macros and potentially executable attachments — and encourage its use! RTF is probably that encoding for text documents, but it is not obvious to the average user that it should be used instead of .doc format for exchanging files. And what is there for Excel, Powerpoint, etc?

Earlier, I wrote about the security risks of using Microsoft Word documents as a presentation and encoding format for sending files via email (see posts here and here). Files in “.doc” format contain macros, among other things, that could be executable. They also have metadata fields that might give away sensitive information, and a lot of undocumented cruft that may be used in the process of exploiting security. It is no wonder that exotic exploits are showing up for Word documents. And only today it was revealed that the latest version of Office 2007 may not have even gotten the most recent patch set.

Want to find some vulnerabilities in Word? Then take a look at the list of US-CERT alerts on that software; my search returns almost 400 hits. Some of these are not yet patched, and there are likely many as-yet unpatched flaws still in there.

Clearly, the use of Word as a document exchange medium is Bad (that’s with a definite capital B). People who understand good security practices do not exchange Word files unless they are doing collaborative editing, and even then it is better to use RTF (if one continues to be beholden to Microsoft formats). Good security hygiene means warning others, and setting a good example.

Now, consider that DHS has released BAA07-09 to solicit research and prototypes to get fixes for current cyber infrastructure vulnerabilities. I could rant about how they claim it is for R&D but is really a BAA for further product development for fundamentally flawed software that cannot be fixed. But that isn’t the worst part. No, the BAA is only available as Word documents!

Can you say “irony”? This is the agency charged with helping guide us to a more secure infrastructure? If so, electronically KYAG.

Update: A response from Dr. Douglas Maughn at DHS points out that the site I indicated for the BAA is actually FedBizOps rather than DHS. The DHS posting site actually has it in PDF…although the FedBizOps link is the one I’ve seen in several articles (and in a posting in SANS NewsBites).

Of course, it would be great if DHS could get the folks at FedBizOps to clean up their act, but at least in this case, DHS — or rather, DHSARPA — got it right. I stand corrected.