Making the CWE Top 25, 2010 Edition
As last year, I was glad to be able to participate in the making of the CWE Top 25. The 2010 Edition has been more systematically and methodically produced than last year's. We adjusted the level of abstraction of the entries to be more consistent, precise and actionable. For that purpose, new CWE entries were created, so that we didn't have to include a high-level entry because there was no other way to discuss a particular variation of a weakness. There was a formal vote with metrics, with a debate about which metrics to use, how to vote, and how to calculate a final score. We moved the high-level CWE entries which could be described as "Didn't perform good practice X" or "Didn't follow principle Y" into a mitigations section which specifically addresses what X and Y are and why you should care about them. Those mitigations were then mapped against the top-25 CWE entries that they affected.
For the metrics, CWE entries were ranked by prevalence and importance. We used P X I to calculate scores. That makes sense to me because risk is defined as Potential loss x Probability of occurrence, so by this formula the CWE rankings are related to the risk those weaknesses pose to your software and business. Last year, the CWEs were not ranked; they instead had "champions" who argued for their inclusion in the Top-25.
I worked on creating an educational profile, with its own metrics (of course not alone; it wouldn't have happened without Steve Christey, his team at MITRE, and other CWE participants). The Top-25 now has profiles; so depending on your application and concerns, you may select a profile that ranks entries differently and appropriately. The educational profile used prevalence, importance but also emphasis. Emphasis relates to how difficult a concept is to explain and understand. Easy concepts can be learned in homeworks, labs, or are perhaps so trivial that they can be learned in the students own reading time. Harder concepts deserve more class time, provided that they are important enough. Another factor for emphasis was how much a particular CWE is helpful in learning others, and its general applicability. So, the educational profile tended to include higher-level weaknesses. Also, it considered all historical time periods for prevalence, whereas the Top-25 is more focused on data for the last 2 years. This is similar to the concept of regression testing -- we don't want problems that have been solved to reappear.
Overall, I have a good feeling about this year's work, and I hope that it will prove useful and practical. I will be looking for examples of its use and experiences with it, and of course I'd love to hear what you think of it. Tell us both the good and the bad -- I'm aware that it's not perfect, and it has some subjective elements, but perhaps comments will be useful for next year's iteration.
For the metrics, CWE entries were ranked by prevalence and importance. We used P X I to calculate scores. That makes sense to me because risk is defined as Potential loss x Probability of occurrence, so by this formula the CWE rankings are related to the risk those weaknesses pose to your software and business. Last year, the CWEs were not ranked; they instead had "champions" who argued for their inclusion in the Top-25.
I worked on creating an educational profile, with its own metrics (of course not alone; it wouldn't have happened without Steve Christey, his team at MITRE, and other CWE participants). The Top-25 now has profiles; so depending on your application and concerns, you may select a profile that ranks entries differently and appropriately. The educational profile used prevalence, importance but also emphasis. Emphasis relates to how difficult a concept is to explain and understand. Easy concepts can be learned in homeworks, labs, or are perhaps so trivial that they can be learned in the students own reading time. Harder concepts deserve more class time, provided that they are important enough. Another factor for emphasis was how much a particular CWE is helpful in learning others, and its general applicability. So, the educational profile tended to include higher-level weaknesses. Also, it considered all historical time periods for prevalence, whereas the Top-25 is more focused on data for the last 2 years. This is similar to the concept of regression testing -- we don't want problems that have been solved to reappear.
Overall, I have a good feeling about this year's work, and I hope that it will prove useful and practical. I will be looking for examples of its use and experiences with it, and of course I'd love to hear what you think of it. Tell us both the good and the bad -- I'm aware that it's not perfect, and it has some subjective elements, but perhaps comments will be useful for next year's iteration.
Comments