The Center for Education and Research in Information Assurance and Security (CERIAS)

The Center for Education and Research in
Information Assurance and Security (CERIAS)

On Competitions and Competence


This is a follow-up to my last post here, about the "cybersecurity profession" and education. I was moderating one of the panels at the most recent CERIAS Symposium, and a related topic came up.

Let's start with some short mental exercises. Limber up your cerebellum. Stretch out and touch your cognitive centers a few times. Ready?

There's another barn on fire! Quick, get a bucket brigade going -- we need to put the fire out before everything burns. Again. It is getting so tiring watching all our stuff burn while we're trying to run a farm here. Too bad we can only afford the barns constructed of fatwood. But no time to think of that -- a barn's burning again! 3rd time this week!

Hey, you people over there tinkering with designs for sprinkler systems and concrete barns -- cut it out! We can't spare you to do that -- too many barns are burning! And you, stop babbling about investigating and arresting arsonists -- we don't have time or money for that: didn't you hear me? Another barn is burning!

Now, hurry up. We're going to have a contest to find who can pass this pail of water the quickest. Yes, it is a small, leaky pail, but we have a lot of them, so that is what we're going to use in the contest. The winners get to be closest to the flames and have a name tag that says "fire prevention specialist." No, we can't afford larger buckets. And no, you can't go get a hose -- we need you in the line. Damnit! The barn's burning!

Sounds really stupid, doesn't it? Whoever is in charge isn't doing anything to address the underlying problem of poor barn construction. It doesn't really match the notion of what a fire prevention specialist might really do. And it certainly doesn't provide deep career preparation for any of those contestants... it may even condemn them to a future of menial bucket passing because we're putting them on the line with no training or qualification beyond being able to pass a bucket.

Let's try another one.

Imagine that every car and automobile in the country has been poorly designed. They almost all leak coolant and burn oil. They're trivial to steal. They are mostly cheap junkers, all built on the same frame with the same engines, accessories, and tires -- even the ones sold to the police and military (actually, they're the same cars, but with different paint). The big automakers are rolling out new models every year that they advertise as being more efficient and reliable, but that is simply hype to get you to buy a new car because the new features also regularly break down. There are a few good models available, but they are quite a bit more expensive; those more expensive ones often (but not always) break down less, are more difficult to steal, and get far better mileage. Their vendors also don't have a yearly model update, and many consumers aren't interested in them because those cars don't take the common size of tire or fuzzy dice for the mirror.

The auto companies have been building this way for decades. They sell their products around the world, and they're a major economic force. Everyone needs a car, and they shell out money for new ones on a regular basis. People grumble about the poor quality and the breakdowns, but other than periodic service bulletins, there are few changes from year to year. Many older, more decrepit cars are on the road because too many people (and companies) cannot afford to buy new ones that they know aren't much better than the old ones. Many people argue -- vociferously -- against any attempt to put safety regulations on the car companies because it might hurt such an important market segment.

A huge commercial enterprise has sprung up around fixing cars and adding on replacement parts that are supposedly more reliable. People pour huge amounts of money into this market because they depend on the cars for work, play, safety, shopping, and many other things. However, there are so many cars, and so many update bulletins and add-ons, there simply aren't enough trained mechanics to keep up -- especially because many of the add-ons don't work, or require continual adjustment.

What to do? Aha! We'll encourage young people in high school and maybe college to become "automotive specialists." We'll publish all sorts of articles with doom and gloom as a result of the shortage of people going into auto repair. We especially need lots more military mechanics.

So...we'll have competitions! We'll offer prizes to the individuals (or teams) that are able to change the oil of last year's model the most quickly, or who can most efficiently hotwire a pickup truck, take it to the garage, change the tires, and return it. The government will support these competitions. They'll get lots of press. Some major professional organizations and even universities will promote these. Of course we'll hire lots of mechanics that way! (Women aren't interested in these kinds of competition? We won't worry about that now. People who are poor with wrenches won't compete? No problem -- we'll fill in with the rest.)

Meanwhile, the government and major companies aren't really doing anything to fix the actual engineering of the automobiles. There are a few comprehensive engineering programs at universities around the country, but minimal focus and resources are applied there, and little is said about applying their knowledge to really fixing transportation. The government, especially the military, simply wants more mechanics and cheaper cars -- overall safety and reliability aren't a major concern.

Pretty stupid, huh? But there does seem to be a trend to these exercises.

Let's try one more.

We have a large population that needs to be fed. They've grown accustomed to cheap, fast-food. Everyone eats at the drive-thru, where they get a burger or compressed chicken by-product or mystery-meat taco. It's filling, and it keeps them going for the day. It also leads to obesity, hypertension, cardiac problems, diabetes, and more. However, no one really blames the fast-food chains, because they are simply providing what people want.

It isn't exactly what people should have, and is it really what everyone wants? No, there are better restaurants with healthy food, but that food is more expensive and many people would go hungry if they had to eat at those places given the current economic model. Of course, if they didn't need to spend so much on medicine and hospital stays, a healthier diet is actually cheaper. Also, those better places aren't easy to find -- small (or no) advertising budgets, for instance.

The government has contracted with the chains for food, and even serves it at every government office and on every military base. The chains thus have a fair amount of political clout so that every time someone raises the issue about how unhealthy the food is, they get muffled by the arguments "But it would be too expensive to eat healthy" and "Most people don't like that other food and can't even find it!"

We have a crisis because the demand for the fast-food is so great that there aren't enough fry cooks. So, the heads of major military organizations and government agencies observe we are facing a crisis because, without enough fry cooks, our troops will be overwhelmed by better fed people from China. Government officials and industry people agree because they can't imagine any better diet (or are so enamored of fried potatoes that they don't want anything else).

How do they address the crisis? By mounting advertising campaigns to encourage young people to enter the exciting world of "cuisine awareness." We make it seem glamorous. Private organizations offer certifications in "soda making" and "ketchup bottle maintenance" that are awarded after 3-day seminars. DOD requires anyone working in food service to have one of these certificates -- and that's basically all. We see educational institutes and small colleges offering special programs in "salad bar maintenance." The generals and admirals keep showing up at meetings proclaiming how important it is that we get more burger-flippers in place before we have a "patty melt Pearl Harbor."

The government launches a program to certify schools as centers of "Cuisine Awareness Exellence" if they can prove they have at least 5 cookbooks in the library, a crockpot, and two faculty who have boiled water. Soon, there are hundreds of places designated with this CAE, from taco trucks and hot dog stands to cordon bleu centers -- but lots are only hot dog stands. None of them are given any recipes, cooks, or financial support, of course -- simply designating them is enough, right?

When all of that isn't seen to be enough, the powers-that-be offer up contests that encourage kids to show up and cook. Those who are able to most quickly defrost a compressed cake of Soylent Red, cook it, stick it in a bun, and serve it up in a bag with fries is declared the winner and given a job behind someone's grill. Actually, each registered contestant gets a jaunty paper cap and offer of an immediate job cooking for the military (assuming they are U.S. citizens; after all, we know what those furriners eat sure isn't food!) And gosh, how could they aspire to be anything BUT a fry cook for the next 40 years -- no need to worry about any real education before they take the jobs.

Meanwhile, those studying dietetics, preventative health care, sustainable agriculture, haute cuisine, or other related topics are largely ignored -- not to mention the practicing experts in these fields. The people and places of study for those domains are ignored by the officials, and many of the potential employers in those areas are actually going out of business because of lack of public interest and support. The advice of the experts on how to improve diet is ignored. Find that disconcerting? Here -- have a deep-fried cherry pie and a chocolate ersatz-dairy item drink to make you feel better.

Did you sense a set of common threads (assuming you didn't blow out your cortex in the exercise)?

First, in every case, a mix of short-sighted and ultimately stupid solutions are being undertaken. In each, there are large-scale efforts to address pressing problems that largely ignore fundamental, systemic weaknesses.

Second, there are a set of efforts putatively being made to increase the population of experts, but only with those who know how to address a current, limited problem set. Fancy titles, certificates, and seminars are used to promote these technicians. Meanwhile, longer-term expertise and solutions are being ignored because of the perceived urgency of the immediate problems and a lack of understanding of cost and risk.

Third, longer-term disaster is clearly coming in each case because of secondary problems and growth of the current threats.

Why did this come up with my post and panel on cybersecurity? I would hope that would be obvious, but if not, let me suggest you go back to read my prior post, then read the above examples, again. Then, consider:

  • Nationally, we are investing heavily in training and recruiting "cyber warriors" but pitifully little towards security engineers, forensic responders, and more. It is an investment in technicians, not in educated expertise.
  • We have a marketplace where we continue to buy poorly-constructed products then pay huge amounts for add-on security and managing response; meanwhile, we have knowledgeable users complaining that they can't afford the up-front cost required to replace shoddy infrastructure with more robust items
  • Rather than listen to experts, we let business and military interests drive the dialog
  • We have well-meaning people who somehow think that "contests" are useful in resolving part of the problem

One of the most egregious aspects is this last item -- the increasing use of competitions as a way of drawing people to the field. Competitions, by their very nature, stress learned behavior to react to current problems that are likely small deviations from past issues. They do not require extensive grounding in multiple fields. Competitions require rapid response instead of careful design and deep thought -- if anything, they discourage people who exhibit slow, considerate thinking -- discourage them from the contests, and possibly from considering the field itself. If what is being promoted are competitions for the fastest hack on a WIntel platform, how is that going to encourage deep thinkers interested in architecture, algorithms, operating systems, cryptology, or more?

Competitions encourage the mindset of hacking and patching, not of strong design. Competitions encourage the mindset of quick recovery over the gestalt of design-operate-observe-investigate-redesign. Because of the high-profile, high-pressure nature of competitions, they are likely to discourage the philosophical and the careful thinkers. Speed is emphasized over comprehensive and robust approaches. Competitions are also likely to disproportionately discourage women, the shy, and those with expertise in non-mainstream systems. In short, competitions select for a narrow set of skills and proclivities -- and may discourage many of the people we most need in the field to address the underlying problems.

So, the next time you hear some official talk about the need for "cyber warriors" or promoting some new "capture the flag" competition, ask yourself if you want to live in a world where the barns are always catching fire, the cars are always breaking down, nearly everyone eats fast food, and the major focus of "authorities" is attracting more young people to minimally skilled positions that perpetuate that situation...until everything falls apart. The next time you hear about some large government grant that happens to be within 100 miles of the granting agency's headquarters or corporate support for a program of which the CEO is an alumnus but there is no history of excellence in the field, ask yourself why their support is skewed towards building more hot dog stands.

Those of us here at CERIAS, and some of our colleagues with strategic views elsewhere, remind you that expertise is a pursuit and a process, not a competition or a 3-day class, and some of us take it seriously. We wish you would, too.

Your brain may now return to being a couch potato. grin


Posted by Doug Jacobson
on Sunday, April 7, 2013 at 04:05 PM

I agree that competitions that are nothing more than lets come in and defend some environment built by others do nothing to help teach people about cyber security.  I would however claim that if the competition starts with students designing a secure environment over the period for several weeks starting with just a scenario has a place in cyber security education.  If these competitions are supported by teaching security fundamentals they can be a great way to allow students to work in teams and to try different defense methods.  As a high school recruiting tool they can work if there is a curriculum that is provided to the high schools and the teams are coupled with professionals.  I agree that these competitions are not the path to creating the cyber security professional, but that can help keep students engaged and teach them some skills that are useful.  The most important point is that the competitions need to be designed with student learning in mind and not just a glorified video game.

Spaf sez:  quite right that it can be a valid pedagogical tool.  But making it a “contest” and promoting it renders some of those points moot.

Furthermore, there is the question of what the educational goals might be.  Configuring a PC to be resistant to some attacks shows that the students understand some basic settings.  But is that education, or training?  And what if the attacks are not very current or imaginative—what do negative results prove?

We should be very careful in using any form of black box experiment as a form of “proof” of anything—other than our ability to construct black boxes.  I know you know this, as you have been a master educator in this realm for years.  It is not so certain that others do.

Posted by HD Moore
on Tuesday, April 9, 2013 at 04:30 PM

I believe you are mixing up CTF/CCDC-style contests with what actual red teams do. The comparison is equivalent to the difference between a hackathon and professional software engineering. CTFs serve a similar role to hackathons - they provide a fast-paced environment for learning a little bit of everything in a highly contrived environment.

This has nothing to do with how most internal red teams operate. We need more qualified security people and a lot less “cyber warriors”, but writing off offensive skills entirely puts security professionals at a disadvantage. It is difficult to be a forensics investigator without having first-hand experience in how attacks work. The safest way to get that experience is to spend time on a red team.

Spaf sez:  no, I’m not mixing them up, although I didn’t explicitly distinguish them.  Competitions with red teams (“pen testing”) isn’t the best way to demonstrate competence, either, nor is it the best way to train.

I agree that understanding vulnerabilities and attacks is important—not essential, but very useful.  However, there are also many attacks that aren’t yet widespread in the wild that are seldom “practiced” that may appear before too long.  For example, I was teaching students about supply chain attacks and personnel subversion for years.  We are only now seeing a surge of these.  Race conditions and covert channels are problems that are possible but not in common use, but should be taught—and not as pen testing exercises.

Pen testing (“red teaming”) is useful, but has too much cred in the industry.  If a test succeeds, it only has value if the reasons for the success are IMMEDIATELY addressed, otherwise known problems are left exposed.  And if the test fails, what does it mean?  It means one of at least 3 things: there were no problems (unlikely), the team found a problem but is not reporting it for some reason (possible), and/or the team involved was not able to exercise or recognize a vulnerability (probable).

My basic criticism is that there is too much emphasis on surface skills, and too much of the field is being taught by people with deep knowledge of both the subject matter and sound pedagogy.  We should not accept this as “state of the art.”

Posted by Drew A
on Tuesday, April 9, 2013 at 11:12 PM

I’m way ahead of you in that I’ve been cringing at the ‘cyber’ prefix for about 15 years, so ‘cyberwarriors’ just makes my skin crawl and eyes roll.

In all seriousness, I never considered that security-related competitions might negatively affect the state of the art, but I don’t think they’re entirely flawed because they provide legitimate outlets for creative energy that might otherwise not have them.  Maybe you think that, too and I just thought some of the snark was real.

Spaf sez, most competitions are structured in ways that cause some problems as well as provide creative outlets.  The problem is that it is a “patch” approach to a real problem, and most such competitions are organized by people without sufficient background to understand the overall effects.  Competitions that stress speed rather than teamwork, prior preparation, inclusiveness, and foundation skills are not helpful.

Posted by Alex Nicoll
on Wednesday, April 10, 2013 at 07:19 AM

May I say “AMEN” Spaf?

Posted by Teodor Sommestad
on Thursday, April 11, 2013 at 04:43 AM

Thank you for a good post. I certainly agree with you that capture the flag competitions is an inefficient method when it comes to training people to solve the bigger security problems. However, my experience is that actual and serious education rarely is the primary reason for why competitions or exercises with red teams vs. blue team or with a wild live-fire format are conducted. The motives I am used to hear are one or more of the following:

a)They are the easiest way to get our super-technicians to meet each other (because they find them fun enough to make them take a trip, free their calendar, obtain travel clearance etc.).
b)They gather security-interested and/or technically skilled students (or other potential recruits) so that they are easy to find for our HR-people.
c) They are exciting and fun enough to attract students to the security field (since shooting is funnier than defending).
d)They give us a controlled environment where we can study how people perform security-relevant tasks, e.g., to identify factors that determine performance under stress.
e) They are thrilling and easy enough for the media get it, which gives headlines that support XYZ (e.g., raising awareness of the security problem in general).

So, there may be good reasons to have these events even if they are inefficient for education and training of most security-competences of relevance. In addition, most participants alse seem to like them just because they are fun.

And I cant help to take you up on the analogies and the question in the end of the post. I do want my firefighters to train on a burning barn every now and then; I would want my mechanic to be quick with the oil change; I do believe that cooking competitions on the TV spurs an interest in home cooking.

Leave a comment

Commenting is not available in this section entry.