Handbook of Software Reliability and Security Testing A CSIAC State of the Art Report CSIAC is a DoD Information Analysis Center sponsored by the Defense Technical Information Center Handbook of Software Reliability and Security Testing A CSIAC State of the Art Report CSIAC Report Number 519193 Contract FA8075-12-D-0001 Prepared for the Defense Technical Information Center Prepared By Taz Daughtrey Richard Wisniewski David Nicholls Thomas McGibbon Cyber Security and Information Systems Information Analysis Center Quanterion Solutions Inc. 100 Seymour Road, Suite C102 Utica, NY 13502-1311 Distribution Statement A Approved for public release: distribution is unlimited i REPORT DOCUMENTATION PAGE Form Approved OMB No. 0704-0188 Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing this collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number. PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS. 1. REPORT DATE (23-05-2011) 2. REPORT TYPE 3. DATES COVERED (From - To) 15-01-2013 Technical N/A 4. TITLE AND SUBTITLE 5a. CONTRACT NUMBER Handbook of Software Reliability and Security Testing FA8075-12-D-0001 5b. GRANT NUMBER 5c. PROGRAM ELEMENT NUMBER N/A 6. AUTHOR(S) 5d. PROJECT NUMBER Hugh “Taz” Daughtrey, Richard Wisniewski, David Nicholls, Thomas McGibbon N/A 5e. TASK NUMBER N/A 5f. WORK UNIT NUMBER N/A 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION REPORT Cyber Security and Information Systems Information Analysis Center NUMBER Quanterion Solutions, Inc., 100 Seymour Rd Suite C102 Utica, NY 13502-1311 DAN #519193 9. SPONSORING / MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR’S ACRONYM(S) Defense Technical Information Center DTIC DTIC/AI 11. SPONSOR/MONITOR’S REPORT 8725 John J. Kingman Rd., STE 0944 Ft. Belvoir, VA 22060 NUMBER(S) 12. DISTRIBUTION / AVAILABILITY STATEMENT Approved for Public Release, Distribution Unlimited 13. SUPPLEMENTARY NOTES 14. ABSTRACT The level of reliability achieved by a DoD system is instrumental to its level of success in today's demanding missions with increasingly limited resources. Each organization must make its own determination as to why and how much reliability is necessary in its designs, as a direct function of what its customer explicitly or implicitly demands. Factors to be considered in the commercial world include the characteristics of the marketplace, the cost of implementing or not implementing a reliability program strategy; and complete knowledge and understanding of customer expectations. As systems, and systems-of-systems, become increasingly complex, the realm of reliability necessarily extends beyond consideration of solely hardware reliability. It must expand to address significant issues pertaining to software reliability and security (the subject of this Handbook) and human factors as a function of human-machine interactions. 15. SUBJECT TERMS Software Reliability, Security, Cyber Security, Software Engineering, Software Testing 16. SECURITY CLASSIFICATION OF: 17. LIMITATION 18. NUMBER 19a. NAME OF RESPONSIBLE PERSON OF ABSTRACT OF PAGES Thomas McGibbon a. REPORT b. ABSTRACT c. THIS PAGE 19b. TELEPHONE NUMBER (include area U U U UU 340 code) 315-351-4203 Standard Form 298 (Rev. 8-98) Prescribed by ANSI Std. Z39.18 ii Table of Contents Page SECTION 1.0: NEED FOR SOFTWARE RELIABILITY................................................................................................... 1 TOPIC 1.1: HOW TO USE THIS HANDBOOK ................................................................................................................... 2 TOPIC 1.2: THE NEED FOR SOFTWARE RELIABILITY .......................................................................................................... 3 SECTION 2.0: SOFTWARE AND SYSTEM RELIABILITY .............................................................................................. 6 TOPIC 2.1: OVERVIEW OF SOFTWARE RELIABILITY ENGINEERING ........................................................................................ 7 TOPIC 2.2: COMPARISON OF HARDWARE AND SOFTWARE RELIABILITY ............................................................................... 13 TOPIC 2.3: SOFTWARE AND SYSTEM RELIABILITY .......................................................................................................... 18 TOPIC 2.4: COMPARISON OF SOFTWARE RELIABILITY AND SOFTWARE SECURITY ASSURANCE ................................................... 20 TOPIC 2.5: SOFTWARE RELIABILITY AND RISK ASSESSMENT ............................................................................................. 24 TOPIC 2.6: RELIABILITY OVER THE SYSTEM LIFE CYCLE ................................................................................................... 28 TOPIC 2.7: IDENTIFICATION OF SYSTEM NEEDS AND FEASIBILITY ANALYSIS .......................................................................... 34 SECTION 3.0: TESTING ......................................................................................................................................... 35 TOPIC 3.1: RELATIONSHIP BETWEEN POLICIES/STANDARDS/GUIDANCE AND SOFTWARE TESTING ............................................ 36 TOPIC 3.2: SYSTEM TEST REQUIREMENTS ................................................................................................................... 40 TOPIC 3.3: SYSTEM OPERATIONAL REQUIREMENTS ................................................................................................... 44 Topic 3.3.1: Operational Profiles ................................................................................................................................46 TOPIC 3.4: TEST STRATEGIES ................................................................................................................................... 51 Topic 3.4.1: Software Reliability Test Strategies ............................................................................................ 51 Topic 3.4.2: Design of Experiments (DOE) ..................................................................................................... 55 TOPIC 3.5: SOFTWARE RELIABILITY TESTING ................................................................................................................ 65 Topic 3.5.1: Overview ................................................................................................................................... 65 Topic 3.5.2: Software Test Coverage Metrics................................................................................................. 68 Topic 3.5.3: Control-Flow Testing .................................................................................................................. 71 Topic 3.5.4: Loop Testing .............................................................................................................................. 77 Topic 3.5.5: Data-Flow Testing ..................................................................................................................... 80 Topic 3.5.6: Transaction-Flow Testing ........................................................................................................... 87 Topic 3.5.7: Domain Testing ......................................................................................................................... 92 Topic 3.5.8: Finite-State Testing .................................................................................................................... 97 Topic 3.5.9: Orthogonal Array Testing ........................................................................................................ 102 Topic 3.5.10: Software Statistical Usage Testing ......................................................................................... 105 Topic 3.5.11: Operational Profile Testing .................................................................................................... 113 Topic 3.5.12: Markov Testing...................................................................................................................... 117 Topic 3.5.13: Optimal Release Time ............................................................................................................ 121 Topic 3.5.14: Security Testing ..................................................................................................................... 123 TOPIC 3.6: RELIABILITY GROWTH AND RELIABILITY DEMONSTRATION/QUALIFICATION TESTING .......................................... 128 Topic 3.6.1: Overview ................................................................................................................................. 128 Topic 3.6.2: Reliability Growth Testing ........................................................................................................ 130 Topic 3.6.2.1: Duane and Crow/AMSAA Models