DOT/FAA/TC-16/39 Safety Issues and Shortcomings Federal Aviation Administration William J. Hughes Technical Center With Requirements Definition, Aviation Research Division Atlantic City International Airport Validation, and Verification New Jersey 08405 Processes Final Report December 2016 Final Report This document is available to the U.S. public through the National Technical Information Services (NTIS), Springfield, Virginia 22161. This document is also available from the Federal Aviation Administration William J. Hughes Technical Center at actlibrary.tc.faa.gov. U.S. Department of Transportation Federal Aviation Administration NOTICE This document is disseminated under the sponsorship of the U.S. Department of Transportation in the interest of information exchange. The U.S. Government assumes no liability for the contents or use thereof. The U.S. Government does not endorse products or manufacturers. Trade or manufacturers’ names appear herein solely because they are considered essential to the objective of this report. The findings and conclusions in this report are those of the author(s) and do not necessarily represent the views of the funding agency. This document does not constitute FAA policy. Consult the FAA sponsoring organization listed on the Technical Documentation page as to its use. This report is available at the Federal Aviation Administration William J. Hughes Technical Center’s Full-Text Technical Reports page: actlibrary.tc.faa.gov in Adobe Acrobat portable document format (PDF). Technical Report Documentation Page 1. Report No. 2. Government Accession No. 3. Recipient's Catalog No. DOT/FAA/TC-16/39 4. Title and Subtitle 5. Report Date SAFETY ISSUES AND SHORTCOMINGS WITH REQUIREMENTS December 2016 DEFINITION, VALIDATION, AND VERIFICATION PROCESSES FINAL REPORT 6. Performing Organization Code 7. Author(s) 8. Performing Organization Report No. Peter DeSalvo and Daniel Fogarty 9. Performing Organization Name and Address 10. Work Unit No. (TRAIS) Boeing Aerospace Operations, Inc. 6001 S Air Depot Oklahoma City, OK 73135- 6601 11. Contract or Grant No. 12. Sponsoring Agency Name and Address 13. Type of Report and Period Covered FAA National Headquarters Phase 3 Final Report/DS #18 950 L’Enfant Plaza N SW 950 L’Enfant Plaza Washington, DC 20024 14. Sponsoring Agency Code AIR-134 15. Supplementary Notes This report addresses safety issues and shortcomings with requirements definition, validation, and verification processes. It was revised (Revision A) in response to FAA review and feedback. The FAA Aviation Research Division COR was John Zvanya. 16. Abstract This document presents safety issues and shortcomings with requirements definition, validation, and verification processes. System architectures and associated requirements for aerospace digital avionics systems have accelerated in complexity and integration over the last two decades. Initial generations of digital avionics automated individual functions were standalone or had limited integration with other airplane-level functions. However, today’s complex avionics architectures can be highly integrated across complex systems. This research has been initiated to identify and address problems caused by, or that contributed to, incorrect or incomplete requirements. This report builds on research completed in years 1 and 2 of this task order, which addressed safety issues with requirements definition, validation and verification processes and practices, and the root causes of requirements errors, omissions, or conflicts. Included is research based on input from subject matter experts, including recommendations to address the root causes. 17. Key Words 18. Distribution Statement Requirements, Validation, Verification, safety, Development This document is available to the U.S. public through the assurance, ARP4754A, ARP4761, DO-178B/C, DO-254, DO- National Technical Information Service (NTIS), Springfield, 297, Digital avionics systems, Systems integration, Cascading Virginia 22161. This document is also available from the failure effects, Supplier oversight, Process assurance Federal Aviation Administration William J. Hughes Technical Center at actlibrary.tc.faa.gov. 19. Security Classif. (of this report) 20. Security Classif. (of this page) 21. No. of Pages 22. Price Unclassified Unclassified 161 Form DOT F 1700.7 (8-72) Reproduction of completed page authorized ACKNOWLEDGEMENTS The authors would like to acknowledge the following FAA review team individuals for providing support to the project: Chakradhar Agava, Charles Kilgore, Srini Mandalapu, Robin Sova, and John Zvanya. iii TABLE OF CONTENTS Page EXECUTIVE SUMMARY x 1. INTRODUCTION 1 1.1 TASK BACKGROUND 1 1.2 RESEARCH SCOPE 1 1.3 RESEARCH APPROACH, ACTIVITIES, AND PRINCIPAL RESULTS 2 2. AVIONICS EVOLUTION IMPACT ON REQUIREMENTS ISSUES AND VERIFICATION AND VALIDATION 4 3. SAFETY ISSUES WITH REQUIREMENTS DEFINITION, V&V PROCESSES 8 3.1 SUMMARY OF PHASE 1 WHITE PAPERS 1–3 8 3.2 SUMMARY OF PHASE 1 FINDINGS 8 3.3 SUMMARY OF PHASE 1 RECOMMENDATIONS 9 4. PROBLEMS, ISSUES, SHORTCOMINGS, AND ROOT CAUSE DETERMINATION 10 4.1 SUMMARY OF WHITE PAPERS 4 AND 5 10 4.2 SUMMARY OF PHASE 2 FINDINGS 10 4.3 SUMMARY OF PHASE 2 RECOMMENDATIONS 11 5. QUESTIONNAIRES 12 5.1 QUESTIONNAIRE USED IN PHASE 2 13 5.1.1 Approach 13 5.1.2 Root Causes Identified for Recommendation 14 5.2 QUESTIONNAIRE USED IN PHASE 3 17 5.2.1 Approach 17 5.2.2 Recommendations to Address Root Causes 17 6. RESEARCH TO DETERMINE RECOMMENDATIONS TO ROOT CAUSES 17 6.1 APPROACH 18 6.2 FINDINGS AND RESULTS 18 6.3 RECOMMENDATIONS ON SPECIFIC CHANGES TO ADDRESS AND MITIGATE IDENTIFIED ROOT CAUSES FOR REQUIREMENTS ISSUES AND SHORTCOMINGS 27 iv 6.3.1 Establishing OEM and Supplier DA Roles and Responsibilities 27 6.3.2 Cross-Functional Systems Integration 30 6.3.3 S&MF Analyses 38 6.3.4 Change Impact Analysis 41 6.3.5 Technical Planning–Process Assurance Reviews 45 6.3.6 Supplier Oversight−Assessing Supplier Risks 48 6.3.7 Model-Based Systems Engineering 50 7. NEXT GENERATION AIR TRANSPORTATION SYSTEM DISCUSSION 52 8. FINDINGS, RESULTS, AND RECOMMENDATIONS 53 8.1 FINDINGS AND RESULTS 53 8.2 RECOMMENDATIONS FOR FURTHER RESEARCH 54 9. REFERENCES 55 APPENDICES A—WHITE PAPER 1 EXTRACTS, INCLUDING EVENTS NOT SELECTED FOR FURTHER RESEARCH WITHIN THIS TORP 1380 DELIVERY ORDER 0022 B—WHITE PAPER 2 EXTRACT C—WHITE PAPER 3 EXTRACT D—WHITE PAPER 4 EXTRACT E—WHITE PAPER 5 EXTRACT F—SCENARIO MAPPING G—PHASE 3 QUESTIONNAIRE H—PROCESS ASSURANCE REVIEW CRITERIA I—POTENTIAL FUTURE WORK CONCEPTS v LIST OF FIGURES Figure Page 1 Notional large commercial passenger transport airborne software development 6 2 Interrelationships among processes 27 3 Supplier DA determination 28 4 Systems integration analysis 31 5 V&V 31 6 Integration analyses relationships 33 7 Airplane, intersystem, and intrasystem integration 34 8 Functional integration analyses interrelationships 35 9 Intrasystem analysis 35 10 Intersystem analysis 36 11 Interfaces required for functionality 37 12 Simplified interfaces required to implement functionality 37 13 Multiple levels of interface integration 38 14 Example of cascading effects for a single failure 39 15 S&MF analyses–multi-airplane-level function assessment 39 16 System- and airplane-level change impact analysis 41 17 Evaluating intersystem effects 42 18 Commercial airplane digital network evolution 51 19 Model-based system engineering benefits 52 vi LIST OF TABLES Table Page 1 Phase 3 root causes to findings 19 2 Advisory circulars and industry guidelines 27 3 Component-/system-level change impact considerations 42 4 Cumulative airplane-level effect considerations 43 5 Criteria for safety requirement types 44 6 Non-safety requirements types 45 7 Artifacts for process assessment reviews 47 8 Supplier risk assessment 49 9 Evolution of systems architectures 50 vii LIST OF ABBREVIATIONS AND ACRONYMS AC Advisory Circular AD Airworthiness Directive ADIRU Air data inertial reference unit AEH Airborne electronic hardware AFHA Airplane functional hazard assessment AIR Aerospace Information Report AR Authorized Representative ARP Aerospace Recommended Practice ATC Air Traffic Control AVSI Aerospace Vehicle System Institute BCA Boeing Commercial Airplanes BITE Built-in Test Equipment BQN Borinquen International Airport CAS Caution Advisory System CCA Common cause analysis CIA Change impact analysis CMA Common mode analysis DA Development assurance DAL Development assurance level ECL Electronic checklist EICAS Engine instrument and crew alerting system FHA Functional hazard assessment FMEA Failure modes and effects analysis FTA Fault tree analysis IMA Integrated modular avionics IP Issue paper LRM Line replaceable module LRU Line replaceable unit MBD Model-based design MBSE Model-based systems engineering MIA Modification Impact Analysis MIT Massachusetts Institute of Technology NTSB National Transportation Safety Board NextGen Next Generation Air Transportation System OEM Original equipment manufacturer PA Process assurance PR Problem report S&MF Single and multiple failure SAVI System Architecture Virtual Integration SCD Specification control drawing SEE Single event effects SFHA System functional hazard assessment SME Subject matter expert SOS System of systems SSA System safety assessment viii T&E Test and evaluation TC Type certification UTC Universal Coordinated Time V&V Validation and verification ix EXECUTIVE SUMMARY System architectures and associated requirements for aerospace digital avionics systems have accelerated in complexity and integration over the