NIST Special Publication 500-209 Computer Systems Software Error Analysis Technology U.S. DEPARTMENT OF COMMERCE Wendy W. Peng Technology Administration Dolores R. Wallace National Institute of Standards and Technology NAT L INST. OF ST4ND & TECH R I.C. NISI PUBLICATIONS A111D3 TTSTll ^QC ' 100 .U57 //500-209 1993 7he National Institute of Standards and Technology was established in 1988 by Congress to "assist industry in the development of technology . needed to improve product quality, to modernize processes, to reliability . manufacturing ensure product and to facilitate rapid commercialization . , of products based on new scientific discoveries." NIST, originally founded as the National Bureau of Standards in 1901, works to strengthen U.S. industry's competitiveness; advance science and engineering; and improve public health, safety, and the environment. One of the agency's basic functions is to develop, maintain, and retain custody of the national standards of measurement, and provide the means and methods for comparing standards used in science, engineering, manufacturing, commerce, industry, and education with the standards adopted or recognized by the Federal Government. As an agency of the U.S. Commerce Department's Technology Administration, NIST conducts basic and applied research in the physical sciences and engineering and performs related services. The Institute does generic and precompetitive work on new and advanced technologies. NIST's research facilities are located at Gaithersburg, MD 20899, and at Boulder, CO 80303. Major technical operating units and their principal activities are listed below. For more information contact the Public Inquiries Desk, 301-975-3058. Technology Services Manufacturing Engineering Laboratory • Manufacturing Technology Centers Program • Precision Engineering • Standards Services • Automated Production Technology • Technology Commercialization • Robot Systems • Measurement Services • Factory Automation • Technology Evaluation and Assessment • Fabrication Technology • Information Services Materials Science and Engineering Electronics and Electrical Engineering Laboratory Laboratory • Intelligent Processing of Materials • Microelectronics • Ceramics • Law Enforcement Standards • Materials Reliability' • Electricity • Polymers • Semiconductor Electronics • Metallurgy • Electromagnetic Fields' • Reactor Radiation • Electromagnetic Technology' Building and Fire Research Laboratory Chemical Science and Technology • Structures Laboratory • Building Materials • Biotechnology • Building Environment • Chemical Engineering' • Fire Science and Engineering • Chemical Kinetics and Thermodynamics • Fire Measurement and Research • Inorganic Analytical Research • Organic Analytical Research Computer Systems Laboratory • Process Measurements • Information Systems Engineering • Surface and Microanalysis Science • Systems and Software Technology • Thermophysics^ • Computer Security • Systems and Network Architecture Physics Laboratory • Advanced Systems • Electron and Optical Physics • Atomic Physics Computing and Applied Mathematics • Molecular Physics Laboratory • Radiometric Physics • Applied and Computational Mathematics^ • Quantum Metrology • Statistical Engineering^ • Ionizing Radiation • Scientific Computing Environments^ • Time and Frequency' • Computer Services^ • Quantum Physics' • Computer Systems and Communications^ • Information Systems 'At Boulder, CO 80303. ^Some elements at Boulder, CO 80303. NIST Special Publication 500-209 Software Error Analysis Wendy W. Peng Dolores R. Wallace Systems and Software Technology Division Computer Systems Laboratory National Institute of Standards and Technology Gaithersburg, MD 20899 April 1993 U.S. DEPARTMENT OF COMMERCE Ronald H. Brown, Secretary NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY Raymond G. Kammer, Acting Director Reports on Computer Systems Technology The National Institute of Standards and Technology (NIST) has a unique responsibility for computer systems technology within the Federal government. NIST's Computer Systems Laboratory (CSL) devel- ops standards and guidelines, provides technical assistance, and conducts research for computers and related telecommunications systems to achieve more effective utilization of Federal information technol- ogy resources. CSL's responsibilities include development of technical, management, physical, and ad- ministrative standards and guidelines for the cost-effective security and privacy of sensitive unclassified information processed in Federal computers. CSL assists agencies in developing security plans and in improving computer security awareness training. This Special Publication 500 series reports CSL re- search and guidelines to Federal agencies as well as to organizations in industry, government, and academia. National Institute of Standards and Technology Special Publication 500-209 Natl. Inst. Stand. Technol. Spec. Publ. 500-209, 113 pages (Apr. 1993) CODEN: NSPUE2 U.S. GOVERNMENT PRINTING OFFICE WASHINGTON: 1993 ABSTRACT This document provides guidance on software error analysis. Software error analysis includes error detection, analysis, and resolution. Error detection techniques considered in the study are those used in software development, software quality assurance, and software verification, validation and testing activities. These techniques are those frequendy cited in technical literature and software engineering standards or those representing new approaches to support error detection. The study includes stadstical process control techniques and rslates them to their use as a software quality assurance technique for both product and process improvement. Finally, the report describes several software reliability models. KEYWORDS Data Collection; Error Detection; Error Removal; High Integrity Software; Metrics; Software Error Analysis; Software Quality Assurance; Software Verification and Validation; Stadsdcal Process Control iii EXECUTIVE SUMMARY The main purpose of this document is to provide the software engineering community with current information regarding error analysis for software, which will assist them to do the following: • Understand how error analysis can aid in improving the software development process; • Assess the quality of the software, with the aid of error detection techniques; • Analyze errors for their cause and then fix the errors; and • Provide guidelines for the evaluation of high-integrity software. The software industry is currently still young, without sufficient knowledge and adequate standards to guarantee fault-free software. Although research continues to identify better processes for error prevention, with current practices, errors will probably occur during software development and maintenance. Hence, there is the need for error analysis. Error analysis for software includes the acdvities of detecting errors, collecdng and recording error data, analyzing and removing single errors, and analyzing collective error data to remove classes of errors. The collective error data may be used with stadsdcal process control (SPC) techniques to improve the product and the processes used in developing, maintaining, and assuring the quality of software. This report provides a descripdon of error detection techniques which are cited frequently in technical literature and standards and describes the cost benefits of applying error detection early in the lifecycle. However, error detection alone is not sufficient for removal of an error. Informadon must be recorded about the error to assist in the analysis of its cause, its removal, and its relationship to the project and to other similar projects. This report provides guidance on data collecdon, analysis, and removal as well as error detecdon. This report describes how several SPC techniques can be used for software quality assurance technique and for process improvement. The report identifies metrics related to software error detection and identifies several software reliability esdmation models. Metrics are used to assess the product or process, while SPC techniques are used to monitor a project by observing trends. SPC techniques help to locate major problems in the development process, the assurance processes (e.g., software quality assurance, verification and validation), and the product itself The study of software engineering standards reported in [NUREG, NIST204] indicates that standards are beginning to associate requirements for error detection techniques with the quality requirements and problem types of the software project implementing the standard. Further examination of these documents and additional standards and guidelines for high integrity software indicates that these documents vary widely in their recommendations of specific error v techniques. Appendix B provides a summary of the error detection techniques required or recommended by these documents for the assurance of the quality of high integrity software. This report recommends collection of error data into an organizational database for use by a vendor' over several projects, and modified collections of these databases for use by government auditors (e.g., Nuclear Regulatory Commission, Environmental Protection Agency). Software organizations should consider institudonalizing mechanisms for establishing and maintaining a database of error analysis data within their organizadon. Over time, it may become apparent that some error analysis techniques are more effecdve than others with respect to a given type of problem. It may