DEPARTMENT OF THE AIR FORCE AC-17-01 Headquarters Air Force Life Cycle Management Center (AFMC) 23 MAR 2017 Engineering Directorate Wright-Patterson AFB OH 45433-7101 AIRWORTHINESS CIRCULAR Verification Expectations for Select Section 15 Criteria PURPOSE: This Airworthiness Circular (AC) identifies expectations for showing compliance to a set of criteria found in MIL-HDBK-516C, Section 15, Computer Systems and Software (CS&S), which is used in the United States Air Force (USAF) airworthiness certification process. SCOPE: This AC applies to all USAF air systems, including those operated by the Air National Guard and USAF Reserve. ATTACHMENTS: (1) Detailed Guidance Regarding Safety Critical Function (SCF) Identification (2) Detailed Guidance Regarding SCF Thread Analysis (SCFTA) (3) Detailed Guidance Regarding System and Software Integration Methodology (4) Detailed Guidance Regarding Failure Modes and Effect Testing (FMET) (5) Detailed Guidance Regarding Safety Interlock Design Mechanization (6) Detailed Guidance Regarding CS&S Development Process and Product Attributes (7) Abbreviations, Acronyms, and Glossary of Terms CANCELATIONS: Not applicable. This is the first issuance of this AC. REFERENCED DOCUMENTS: [1] MIL-HDBK-516C, Airworthiness Certification Criteria, 12 December 2014 [2] MIL-STD-882D/E, System Safety Program, 10 February 2000/23 April 2012 [3] AWB-1011A, Airworthiness Expert Endorsement, 4 September 2014 [4] IEEE 12207, Systems and Software Engineering – Software Life Cycle Processes, 2008 [5] MIL-STD-498, Software Development and Documentation, 5 December 1994 [6] DOD-STD-2167A, Defense System Software Development, 29 February 1988 [7] DO-178B/C, Software Considerations in Airborne Systems and Equipment Certification, 1 December 1992 / 13 December 2011 [8] JSSG-2000A, Joint Service Specification Guide: Air System, 8 October 2002 [9] JSSG 2001A, Joint Service Specification Guide: Air Vehicle, 22 October 2002 [10] MIL-HDBK-61A, Configuration Management Guidance, 7 February 2001 [11] SAE EIA-649_1, Configuration Management Requirements for Defense Contracts, Nov 2014 [12] DOD Unmanned Aircraft System Airspace Integration Plan, Version 2.0, March 2011 Distribution Statement A – Approved for public release; distribution unlimited USAF Center of Excellence for Airworthiness AC-17-01 BACKGROUND: With the advent of integrated computer system architectures, air system functionality is often dependent on sensor information, data buses, subsystem processing, backplanes, output signals, data/system latency requirements, software partitioning, etc. This has led to an increased reliance on executing Safety Critical Functions (SCFs) with integrated computer system architectures. To provide the requisite safety assurance, the USAF airworthiness certification process has recognized that it is necessary to adhere to a rigorous standard of safety verification for these systems, referred to as System Processing Architectures (SPAs). The USAF airworthiness certification process utilizes MIL-HDBK-516 Section 15, Computer Systems and Software, to establish the airworthiness verification criteria for SPAs. Over the past few years, the majority of airworthiness certification activities have involved development of, or modifications to, computer systems and software. During the review of many of these efforts, certain criteria within Section 15 have frequently been found to be non-compliant. As a result, this AC has been written to clarify expectations associated with airworthiness criteria within Section 15 experiencing repeated non-compliance. The guidance in this AC elaborates on particular airworthiness certification requirements that focus on design contributions that the hardware and software must provide to the system architecture in support of Safety/Flight Critical functionality, as well as key verification activities that are needed to evaluate the safety risk associated with the system design. An overview of the verification expectations focused upon in this AC involve: a) Identification of Safety Critical Functions (SCFs), b) Identification of all hardware/software supporting SCFs, c) Redundancy as required to support the criticality, d) Fault tolerance and redundancy management to handle faults, e) Verification and validation (V&V) activity expected from lower level testing through system integration (including Failure Modes Effects Analysis/Testing (FMEA/T), f) Evaluation of interlock design mechanizations that provide de-confliction of functional modes and are used to ensure safe operation, g) Strict adherence to a robust development process and sound V&V approach, h) Full qualification of software requirements for every flight release of software supporting an SCF. Meeting the above requirements is vital for demonstrating compliance to certain Section 15 criteria. USAF Center of Excellence for Airworthiness Page 2 of 72 AC-17-01 DISCUSSION AND RECOMMENDATIONS: The following focus areas are addressed to ensure a better understanding of the design, development, integration, and V&V expectations for showing compliance to Section 15. Each focus area contains a discussion paragraph and a recommendation paragraph. a) Safety Critical Function (SCF) Identification b) SCF Thread Analysis (SCFTA) c) Integration Methodology: System, Software, & Levels of Testing d) Failure Mode and Effects Testing (FMET) e) Safety Interlock Design f) SPA and Software Development Processes g) Full Qualification of Software SCF Identification Discussion: The term Safety Critical Function is defined in both MIL-STD-882 and MIL-HDBK- 516C as: a function whose failure to operate or incorrect operation will directly result in a mishap of either Catastrophic or Critical severity. Per MIL-STD-882, SCFs are to be identified as part of the initial activity associated with the system safety process. Once identified, the SCFs are used in the Functional Hazard Analysis (FHA), which lays the foundation for identifying hazards within the system. The identification of SCFs is critical to understanding the focus area of airworthiness- oriented functionality. In Section 15 one criterion, 15.1.1, requires that SCFs have been identified for the air system. All criteria in Section 15 indirectly rely on SCF identification since the criteria are to be applied to equipment supporting SCFs. Recommendation: To demonstrate compliance with MIL-HDBK-516 Section 15 criteria 15.1.1 involving the identification of SCFs, the System Safety process (supported by functional engineering teams) should identify the system’s applicable SCFs, which should then be used as the foundation for performing SCFTAs. NOTE: Other MIL-HDBK-516 sections (4, 6, 7, 8, 9, 10, 11, 12, 13, 14, and 17) are also dependent on this information. See Attachment 1 (page 9) for Detailed Guidance Regarding SCF Identification. SCFTA Discussion: An SCF thread is defined in MIL-HDBK-516C as: the combination of elements/components within a system and the required interfacing and interaction of those elements/components whose overall contribution is necessary for the operation of a given SCF. An SCFTA’s purpose is to: USAF Center of Excellence for Airworthiness Page 3 of 72 AC-17-01 1) Identify all the elements, hardware and software components, and interfaces that are necessary for the safe execution of all identified SCFs, 2) Ensure the identified elements and components are developed at Computer System Integrity Levels (CSILs) appropriate for SCF applications, and that safety critical interfaces are identified as such, and 3) Verify that end-to-end V&V coverage is achieved by the tests used to verify the SCF functionality (includes: component level test and review; subsystem level test; through system integration test). A key purpose of the airworthiness process is to ensure the design is safe to operate within its intended envelope of operation. It stands to reason then that verifying the end-to-end functionality of the portions of the system that contribute to airworthiness safety risk (i.e., SCFs) is essential to establishing confidence in the airworthiness of the design. The SCFTA is considered to be a foundational tool for providing evidence that the end-to-end SCF functionality has been verified. Recommendation: To demonstrate compliance with the various MIL-HDBK-516 Section 15 criteria involving SCFTAs and demonstrating that the end-to-end V&V of the air system’s SCFs have been adequately achieved, the SCFTA activity should be performed as an integral process to the development and V&V activities. An SCFTA should be performed on all SCFs that are supported by computer systems and software. An SCFTA is considered to be satisfactorily completed when all the SCF threads have been fully identified (i.e., all supporting elements, components, and interfaces identified with associated CSIL) and complete test coverage of all SCF threads is verified and documented. The SCFTA should be reviewed and updated as necessary for every flight release of software supporting an SCF and every modification impacting an SCF. See Attachment 2 (page 14) for Detailed Guidance Regarding SCFTA. System and Software Integration Methodology Discussion: Within Section 15, paragraphs 15.2.3 and 15.6.1 evaluate the adequacy of the integration methodology associated with the development of computer systems supporting SCFs. Section 15 seeks to verify the system integration methodology, the software integration methodology, and assure that complete test coverage is obtained through all levels of testing. The system integration methodology