NIST Special Publication 500-264 Proceedings of Defining the State of the Art in Software Security Tools Workshop Paul E. Black (workshop chair) Elizabeth Fong (editor) Information Technology Laboratory National Institute of Standards & Technology Gaithersburg MD 20899 September 2005 U.S. Department of Commerce Carlos M. Gutierrez. Secretary Technology Administration Phillip Bond, Under Secretary of Commerce for Technology National Institute of Standards and Technology William Jeffrey, Director Disclaimer: Any commercial product mentioned is for information only; it does not imply recommendation or endorsement by NIST nor does it imply that the products mentioned are necessarily the best available for the purpose. Defining the State of the Art in Software Security Tools (SoftSecTools’05 Proceedings) ISBN # 1-59593-179-1/05/08 2 Proceedings of Defining the State of the Art in Software Security Tools Workshop Paul E. Black (workshop chair) Elizabeth Fong (editor) Information Technology Laboratory National Institute of Standards and Technology Gaithersburg, MD 20899 ABSTRACT This proceeding is the result of a workshop held on August 10 and 11, 2005 hosted by the Software Diagnostics and Conformance Testing Division, Information Technology Laboratory, at the National Institute of Standards and Technology. The workshop, “Defining the State of the Art in Software Security Tools,” is one of a series in the NIST Software Assurance Measurement and Tool Evaluation (SAMATE) project, which is partially funded by DHS to help identify and enhance software security assurance (SSA) tools. The goal of this workshop is to understand the state of the art of SSA tools that detect security flaws and vulnerabilities and develop a standard reference dataset of programs with known flaws. Forty-five people from outside NIST attended, including representatives from the federal government (NSF, FDA, NSA, DoD, and DHS), seven universities, more than a dozen tool vendors and service providers, and many research companies. Keywords: Software assessment tools; software assurance; software metrics; software security; reference dataset; vulnerability 3 Foreword The workshop on “Defining the State of the Art in Software Security Tools” was held 10- 11 August 2005 at the National Institute of Standards and Technology in Gaithersburg, Maryland. Forty-five people from outside NIST attended, including people from government, universities, tool vendors and service providers, and research companies. For this workshop, members of the workshop committee introduced each session with a brief orientation, then led a discussion among the attendees. The discussions ranged from whether tools and services are commercially viable to if there was enough academic interest and research to how best to encourage more secure software (in both sense) to be developed. The goals of this workshop were to • move toward a shared taxonomy of software security flaws and vulnerabilities, • outline a shared taxonomy of software security assurance (SSA) functions, • understand the state of the art of SSA tools embodying those functions, • discuss possible metrics to evaluate the effectiveness of SSA tools, and • find, collect, and develop a set of flawed and "clean" software to be a standard reference dataset. Several groups that have been developing taxonomies of security flaws and vulnerabilities exchanged notes and agreed to work together to develop standard nomenclature. Many volunteered to help work toward a standard reference dataset. These proceedings have five main parts: • introduction, agenda, etc., • position statements and background information from participants, • discussion material developed by NIST employees, • extensive notes of the discussions, including the orientations, and • other material submitted by participants. Position statements precede the associated participant's background material (the Think- Tank approach) so you can consider the position statement first, discover the source, then reconsider the statement, if warranted. We thank those who worked to organize this workshop, particularly Elizabeth Fong, who handled much of the correspondence. We are grateful to NIST, especially the Software Diagnostics and Conformance Testing division, for providing the facilities and organizers' time. Special thanks go to Dr. William Spees, FDA, for a very thorough review of the minutes. On behalf of the program committee and the whole SAMATE team, thanks to everyone for taking their time and resources to join us. Dr. Paul E. Black 4 Table of Contents Summary....................................................................................................... 6 Workshop Announcement ........................................................................ 7 Workshop Agenda ...................................................................................... 9 Attendees .................................................................................................... 10 Position Statements and Background Information............................ 13 Discussion Material.................................................................................. 46 Survey of SA Tools by Categories................................................................................ 46 The State of the Art in SA Tools and their Functions................................................... 50 Classes of Software Security Flaws and Vulnerabilities .............................................. 52 Possible Metrics to Evaluate SA Security Tools .......................................................... 58 Reference Dataset ......................................................................................................... 60 Workshop Minutes ................................................................................... 63 Welcome (Shashi Phoha).............................................................................................. 63 Scope and Introduction (Paul Black) ............................................................................ 64 Tools Survey and Categorization (Elizabeth Fong)...................................................... 67 Taxonomy of Software Assurance Functions (Michael Kass) ..................................... 71 Recommended Best Practices, or, State of the Art in SA Tools (Brad Martin) ........... 75 Software Assurance Vulnerability List &Taxonomy (Mike Koo) ............................... 79 Security metrics for Software and Tools (Paul Black) ................................................. 83 Reference Dataset (Mike Sindelar)............................................................................... 85 Next Step(s) (Paul Black) ............................................................................................. 89 Develop Consensus on Workshop Report (Paul Black) ............................................... 90 Conclusions................................................................................................................... 93 Submitted Material ....................................................................................... 94 Michael Zhivich, Tim Leek, & Richard Lippmann, “Dynamic Buffer Overflow Detection.” Kendra Kratkiewicz & Richard Lippmann, “Using a Diagnostic Corpus of C Programs to Evaluate Buffer Overflow Detection by Static Analysis Tools.” Thomas W. Reps, Tim Teitelbaum, Paul Anderson, & David Melski, “Static Analysis of Binary Executable Code.” 5 Summary This is the proceeding of the workshop on Defining the State of the Art in Software Security Tools held on August 10 and 11, 2005. It was hosted by the Software Diagnostics and Conformance Testing Division, Information Technology Laboratory, at the National Institute of Standards and Technology (NIST) in Gaithersburg, MD, USA. The workshop is one of a series in the NIST Software Assurance Measurement and Tool Evaluation (SAMATE) project, home page http://samate.nist.gov/ Before the workshop the SAMATE team posted discussion material. The workshop consisted of eight sessions: - Tools survey and categorization - Taxonomy of software assurance functions - Recommended best practices, or, state of the art in software assurance tools - Software assurance (SA) vulnerability list and taxonomy - Software assurance (SA) tool metrics - Reference dataset - Next steps - Develop consensus of workshop report Each session had a facilitator who presented orientation material. The pre-workshop discussion material, orientation material for each session, and workshop minutes are included in these proceedings. Here are summaries of the discussions: - The scope of the tools surveyed was too narrowly focused on just security coding tools and needs to expand to assurance tools which are used in the requirements and design phase of software development life cycle. - In the session on software assurance functions, discussions centered around the fact the tools do not do the same thing and it is difficult to compare tools. However, most agreed that a specification of tool functions (e.g., detecting buffer overflow) is useful. - The state of the art of software assurance tools was considered not mature due to lack of knowledge in education and lack of research in how to produce secure software. - A taxonomy of vulnerabilities is important, however, it will be difficult to achieve “a taxonomy” with properties that everyone will agree on. - Software metrics and tool metrics are difficult areas and more research is required to develop a set of attributes for measurements.