Software Assurance Approaches, Considerations, and Limitations
Total Page:16
File Type:pdf, Size:1020Kb
DOT/FAA/TC-15/57 Software Assurance Approaches, Federal Aviation Administration William J. Hughes Technical Center Considerations, and Limitations: Aviation Research Division Atlantic City International Airport Final Report New Jersey 08405 October 2016 Final Report This document is available to the U.S. public through the National Technical Information Services (NTIS), Springfield, Virginia 22161. This document is also available from the Federal Aviation Administration William J. Hughes Technical Center at actlibrary.tc.faa.gov. U.S. Department of Transportation Federal Aviation Administration NOTICE This document is disseminated under the sponsorship of the U.S. Department of Transportation in the interest of information exchange. The U.S. Government assumes no liability for the contents or use thereof. The U.S. Government does not endorse products or manufacturers. Trade or manufacturers’ names appear herein solely because they are considered essential to the objective of this report. The findings and conclusions in this report are those of the author(s) and do not necessarily represent the views of the funding agency. This document does not constitute FAA policy. Consult the FAA sponsoring organization listed on the Technical Documentation page as to its use. This report is available at the Federal Aviation Administration William J. Hughes Technical Center’s Full-Text Technical Reports page: actlibrary.tc.faa.gov in Adobe Acrobat portable document format (PDF). Technical Report Documentation Page 1. Report No. 2. Government Accession No. 3. Recipient's Catalog No. DOT/FAA/TC-15/57 4. Title and Subtitle 5. Report Date SOFTWARE ASSURANCE APPROACHES, CONSIDERATIONS, AND LIMITATIONS FINAL REPORT October 2016 6. Performing Organization Code 7. Author(s) 8. Performing Organization Report No. Mats Heimdahl, University of Minnesota; Nancy Leveson, Massachusetts Institute of Technology. Julie Redler, Melanie Felton, and Grady Lee are from Safeware Engineering Corporation 9. Performing Organization Name and Address 10. Work Unit No. (TRAIS) Safeware Engineering Corporation 180 Nickerson Street, Suite 110 Seattle, WA 98109 11. Contract or Grant No. DTFACT-11-C-00004 12. Sponsoring Agency Name and Address 13. Type of Report and Period Covered U.S. Department of Transportation Final Report Federal Aviation Administration National Headquarters July 2011–November 2014 950 L'Enfant Plaza N SW Washington, DC 20024 14. Sponsoring Agency Code AIR-134 15. Supplementary Notes The Federal Aviation Administration William J. Hughes Technical Center Aviation Research Division COR was Srini Mandalapu. 16. Abstract The cost of developing software in compliance with RTCA/DO-178B/RTCA/DO-278 is generally high. Nevertheless, these standards have helped to ensure the development of software systems of high integrity with excellent operational histories. The “Alternative Approaches to Software Assurance” three-phase study was undertaken to evaluate the current state of software assurance processes and propose alternative approaches with the potential to streamline the process and reduce the assurance costs without compromising safety. Phase 1 work focused on three areas: an examination of alternative methods, a comparison of aerospace industry standards to other safety-critical industry’s standards, and a poll to query aviation industry personnel on their experience with DO-178B and DO-278. The findings from Phase 1 did not highlight any alternative approaches that could replace DO-178B or DO-278. The authors recommended looking at technical advances that could still meet the goal of the study but were not necessarily alternatives to DO-178B and DO-278. The Phase 1 findings directed the team to look at techniques that could help users of the standards to streamline the process (and realize cost benefits) by ensuring the requirements were complete and correct early in the development process. The goal of Phase 2 was to conduct an in-depth study of techniques that warranted further study from Phase 1, including: hazard analysis; human reviews; model-based specification and analysis; architectural modeling and analysis; and collection of information regarding how each approach helps in streamlining the certification process and which approaches are best used for commercial off-the-shelf and legacy software. The research from the first two phases directed the team to further focus on Systems Theoretic Process Analysis (STPA), model-based development, and formal verification in the third phase. Although these methods have been around for some time, there have been advancements in model-based development and formal verification that deemed it worthwhile to re-visit them. The Phase 3 work also highlighted how STPA can catch more system and software errors in the requirements than the traditional hazard analysis techniques, such as fault tree analysis. The analysis demonstrated how STPA could be applied to a flight guidance system and how hazard causes could be mitigated. The research also looked at cost savings that were realized by Rockwell Collins when they used model-based development and by Airbus when they used formal verification on their projects. A discussion about the pitfalls of using model-based development and formal verification was also included. 17. Key Words 18. Distribution Statement Software assurance, Alternative approaches, Requirements assurance, This document is available to the U.S. public through the National Software architecture assurance, Quality assurance, STPA, Systems, Technical Information Service (NTIS), Springfield, Virginia 22161. Hazard analysis, Model-based development, Formal verification This document is also available from the Federal Aviation Administration William J. Hughes Technical Center at actlibrary.tc.faa.gov. 19. Security Classif. (of this report) 20. Security Classif. (of this page) 21. No. of Pages 22. Price Unclassified Unclassified 175 Form DOT F 1700.7 (8-72) Reproduction of completed page authorized TABLE OF CONTENTS Page EXECUTIVE SUMMARY vii 1. INTRODUCTION 1 1.1 PHASE 1: FIND ALTERNATIVE APPROACHES TO DO-178B AND DO-278 2 1.2 PHASE 2: TECHNICAL ADVANCES TO STREAMLINE THE SOFTWARE DEVELOPMENT PROCESS 2 1.3 PHASE 3: FURTHER INVESTIGATION OF TECHNICAL ADVANCEMENTS 3 2. PHASE 1 RESEARCH 4 2.1 DOCUMENTS AND STANDARDS REVIEWED 4 2.1.1 National Academies Press Report: “Software for Dependable Systems: Sufficient Evidence?” 4 2.1.2 Evaluation of Other Industry Software Assurance Standards 9 2.2 POLL RESULTS OF ALTERNATIVE APPROACHES TO SOFTWARE ASSURANCE 12 2.2.1 Poll Summary 13 2.3 OVERVIEW OF ASSURANCE METHODS 14 2.3.1 Dynamic Analysis 14 2.3.2 Static Analysis 16 2.3.3 Quality Assurance 17 2.3.4 Verification and Validation of Non-Software Life-Cycle Products 18 2.3.5 Service History 18 2.3.6 Software Metrics and Reliability Models 18 2.3.7 Hazard Analysis and Safety Engineering 23 2.3.8 Model-Based Development and Automatic Code Generation 24 2.3.9 Assurance Cases 25 2.3.10 Incremental Integration of Components 26 2.3.11 Reverse Engineering 27 2.3.12 Phase 1 Summary 28 3. PHASE 2 RESEARCH: TECHNICAL ADVANCES TO STREAMLINE THE SOFTWARE DEVELOPMENT PROCESS 29 iii 3.1 STREAMLINING THE PROCESS: INCORPORATING SAFETY AND REDUCING COST 29 3.1.1 Requirements Assurance 30 3.1.2 Software Architecture Assurance 41 3.1.3 Implementation (coding) Assurance 42 4. PHASE 3 RESEARCH 47 4.1 TECHNICAL ADVANCES TO AID IN FINDING SAFETY CONSTRAINTS 47 4.1.1 STPA Hazard Analysis 47 4.2 CASE STUDIES ON COST SAVINGS RELATED TO MODEL-BASED DEVELOPMENT 60 4.2.1 Model-Based Development 60 4.3 FORMAL VERIFICATION 69 4.3.1 Reducing Rework and Test Effort Through Formal Verification 70 4.3.2 Reducing Testing Efforts Through Formal Verification 73 4.3.3 Formal Methods Summary and Recommendations 74 5. RESULTS AND FURTHER WORK 77 6. REFERENCES 79 APPENDICES A—POLL ON ALTERNATIVE APPROACHES TO SOFTWARE ASSURANCE B—SYSTEMS THEORETIC PROCESS ANALYSIS iv LIST OF FIGURES Figure Page 1 Example NextGen control structure diagram 33 2 Example NextGen inadequate control actions 35 3 Example flight control panel 48 4 Example PFD from speedbirdair.com 49 5 FGS control structure diagram 51 6 Interactions between the FMS and FGS 54 v LIST OF ABBREVIATIONS AND ACRONYMS AADL Architecture Analysis and Design Language ADS Air Data System ADS-B Automatic Dependent Surveillance-Broadcast ALT Altitude hold AP Autopilot APPR Approach mode CNS/ATM Communications, Navigation, Surveillance/Air Traffic Management COTS Commercial off-the-shelf CTL Computation tree logic FCP Flight control panel FCS Flight control system FD Flight director FGS Flight guidance system FHA Functional Hazard Assessment FMEA Failure modes and effects analysis FMS Flight Management System FTA Fault Tree Analysis GA Go Around GPS Global Positioning System HDG Heading Select JAXA Japan Aerospace Exploration Agency IAS Indicated airspeed LTL Linear Time Temporal Logic MC/DC Modified condition/decision coverage NAV Lateral navigation NextGen Next Generation Air Transportation System PFD Primary flight display PSA Preselected altitude RC/DC Reinforced condition/decision coverage RSML Requirements State Machine Language RSML-e Requirements State Machine Language without Events RTCA RTCA, Inc. (formerly Radio Technical Commission for Aeronautics) SCADE Safety Critical Application Development Environment SCR Software cost reduction STPA Systems Theoretic Process Analysis STAMP System Theoretic Accident Model and Processes SYNC Synchronization TCAS II Traffic Alert and Collision Avoidance System