NOT FAA POLICY OR GUIDANCE LIMITED RELEASE DOCUMENT 18 October 2013 Advanced Verification Methods for DOT/FAA/A R-XX/XX Safety-Critical Airborne Electronic Office of Aviation Hardware Research and Development Washington, DC 20591 DISCLAIMER This draft document is being made available as a “Limited Release” document by the FAA Software and Digital Systems (SDS) Program and does not constitute FAA policy or guidance. This document is being distributed by permission by the Contracting Officer’s Representative (COR). The research information in this document represents only the viewpoint of its subject matter expert authors. The FAA is concerned that its research is not released to the public before full editorial review is completed. However, a Limited Release distribution does allow exchange of research knowledge in a way that will benefit the parties receiving the documentation and, at the same time, not damage perceptions about the quality of FAA research. NOT FAA POLICY OR GUIDANCE LIMITED RELEASE DOCUMENT 18 October 2013 Technical Report Documentation Page 1. Report No. 2. Government Accession No. 3. Recipient's Catalog No. DOT/FAA/(AR)-xx/xx 4. Title and Subtitle 5. Report Date Qualification of Tools for Airborne Electronic Hardware July. 1, 2013 6. Performing Organization Code 7. Author(s) 8. Performing Organization Report No. Brian Butka 9. Performing Organization Name and Address 10. Work Unit No. (TRAIS) 1 Embry Riddle Aeronautical University 600 S. Clyde Morris Blvd. Daytona Beach, FL 32114 11. Contract or Grant No. DTFACT-11-C-00007 12. Sponsoring Agency Name and Address 13. Type of Report and Period Covered U.S. Department of Transportation Federal Aviation Administration Final Report Office of Aviation Research and Development Washington, DC 20591 Aug.. 2011 – Aug. 2013 14. Sponsoring Agency Code 15. Supplementary Notes The Federal Aviation Administration Aviation Research Division COR was Srini Mandalapu, and the Alternate COR was Charles Kilgore. 16. Abstract The purpose of this research is to provide safety input to the FAA for developing policy and guidance for the verification coverage analysis of complex airborne electronic hardware (AEH), such as field programmable gate arrays (FPGAs), programmable logic devices (PLDs), and application specific integrated circuits (ASICs). The recommended verification process is below. All designs go through a design review where the design strategy is analyzed and discussed and the ability of the design to meet all of the requirements is documented. In addition to the design review process, constrained random verification is used to generate large numbers of random test cases that can detect problems with the requirements and remove human biases from the verification effort. Assertions are used throughout the design hierarchy to detect if any violations of the requirements or the designer’s intent occur at any time during the verification process. The requirements based verification required for DO-254 needs to be extended to a more robust functional verification that considers the combinations of inputs and system state. Robustness testing should explore negative compliance issues to enable the hardware to gracefully recover from unexpected conditions. The verification process recommended in this report includes three coverage metrics: code coverage, assertions and functional. Using these coverage metrics, coverage targets are proposed for DAL A, B and C hardware. In parallel to the simulation based verification, formal methods should be applied. If assertions are used, formal verification of the assertions should be performed. Formal techniques such as sequential equivalence checking and model checking should be applied on hardware that is suitable for these analysis techniques. 17. Key Words 18. Distribution Statement Programmable Logic, Tools, Coverage Metrics, Airborne Electronic Hardware 19. Security Classif. (of this report) 20. Security Classif. (of this page) 21. No. of Pages 22. Price Unclassified Unclassified Form DOT F 1700.7 (8-72) Reproduction of completed page authorized ii TABLE OF CONTENTS LIST OF ACRONYMS vii EXECUTIVE SUMMARY viii 1. INTRODUCTION. 1 1.1 Objectives. 1 1.2 Research Method. 1 1.3 Audience. 2 1.4 Document Structure. 2 2. IDENTIFY CURRENT INDUSTRY PRACTICES FOR VERIFICATION COVERAGE ANALYSIS OF CEH 3 2.1 Verification Process Overview 4 2.2 Coverage Metrics: 6 2.3 Functional coverage 8 2.4 Assertions 9 2.5 Constrained Random Verification 11 2.6 Formal verification 12 2.6.1 Sequential Equivalence Checking 12 2.6.2 Model Checking 13 2.6.3 When Are Formal Methods Effective? 14 2.6.4 Formal Verification in Practice 15 2.7 Verification Process Details 16 2.8 Robustness testing 19 2.9 Verification Methodologies 20 2.10 Hardware Based Verification 20 3. INDUSTRY SURVEYS 22 3.1 Design and Verification Language 22 3.2 Verification Methodology 22 3.3 Assertions 23 3.4 Formal Methods 23 3.5 Safety-Critical Verification 23 4. IDENTIFY KNOWN AND EMERGING OBSTACLES, PROBLEMS, OR ISSUES 23 4.1 Issues with Coverage metrics 24 4.2 Issues with Functional coverage 25 4.3 Issues with formal verification 25 4.4 Issues with Assertions 26 4.5 Issues with COTS IP 26 5. IDENTIFY POTENTIAL APPROACHES AND CRITERIA TO DEMONSTRATE SUFFICIENCY OF VERIFICATION COVERAGE ANALYSIS OF CEH LEVELS A, B, AND C 26 iii 5.1 Hardware verification Plan 27 5.2 Simulation Based Verification Plan 28 5.2.1 Create the Functional Coverage Specification 28 5.2.2 Writing and Running the Testbench 28 5.2.3 Coverage Analysis 28 5.3 Formal Verification Plan 29 6. RECOMMENDATIONS 29 7. CONCLUSIONS 30 8. REFERENCES. 30 iv LIST OF FIGURES Figure Page Figure 1. AEH Stakeholders. 2 Figure 2. Coverage closure vs the number of tests generated. 12 Figure 3. A typical hardware verification flow. 17 Figure 4. Cadence Inc’s recommended verification process. 18 Figure 5. .How hardware verification is used in industry. 21 Figure 6. Author’s survey question #1. “Which of the following tools/techniques are used in your verification flow? (Check all that apply.)” 32 Figure 7. Results of an Aldec survey question asking what verification language the respondents will use on their next design. 33 Figure 8. Results of an Aldec survey asking respondents what verification methodologies were being used. 34 Figure 9. Authors survey question #2. “How do you identify where to insert assertions?” 35 Figure 10. Author's survey question #3. “Who puts assertions into the RTL code?” 35 Figure 11. Author's Survey question #4. “Where are assertions used? (Check all that apply)” 36 Figure 12. Author's survey question #5. “Which code coverage metrics do you use? (Check all that apply.)” 36 Figure 13. Author’s survey question #6. “How often are code coverage checks run?” 37 Figure 14. Author's survey question #7. “Do you use constrained random verification?” 37 Figure 15. Author's survey question #9. "Do you use formal methods in your verification process?” 38 Figure 16. Author's survey question # 10. “How critical/essential are formal methods in your verification process?” 38 Figure 17. Testbench output of Implementation 1. 42 Figure 18. Testbench output for implementation 2. 43 Figure 19. The state machine implemented in the COTS IP. 45 Figure 20. Testbench showing that the light outputs of the COTS IP meets the requirements. 45 Figure 21. Testbench demonstrating that the crosswalk module is correct. 46 Figure 22. Testbench of the crosswalk system with integrated COTS IP showing glitches on the crosswalk output signals. 47 Figure 23. Comparison of the output glitches to intesignal next_state. 47 Figure 24. Detailed simulation of the COTS IP showing the cause of the next_state glitches. 48 v LIST OF TABLES Table Page Table 1. The top two areas needing advancement in the design and verification process. 23 Table 2. Top 3 challenges in managing IP. 24 Table 3 Table documenting the logic for the reverse_thruster_control_signal in terms of the deploy-reverse-thrusters-switch and in_air_sensor inputs. 43 Table 4 Crosswalk signal requirements 46 vi LIST OF ACRONYMS ACO Aircraft Certification Office AEH Airborne Electronic Hardware ARP Aerospace Recommended Practice ASIC Application Specific IC CDC Clock Domain Crossing CRC Cyclic Redundancy Check CRV Constrained Random Verification COTS Commercial Off-The-Shelf CPLD Complex PLD DAL Design Assurance Level DER Designated Engineering Representative FAA Federal Aviation Administration FPGA Field-Programmable Gate Array FSM Finite State Machine FTA Fault Tree Analysis FVI Formal Verification Interface GAL Generic Array Logic GCLK Global Clock Resource HDL Hardware Description Language IP Intellectual Property PAL Programmable Array Logic PLD Programmable Logic Device RBV Requirements Based Verification RTCA RTCA, Inc., (formerly Radio Technical Commission for Aeronautics) RTL Register Transfer Language VHDL VHSIC HDL vii EXECUTIVE SUMMARY The purpose of this research is to provide safety input to the FAA for developing policy and guidance for the verification coverage analysis of complex airborne electronic hardware, such as field programmable gate arrays, programmable logic devices, and application specific integrated circuits. RTCA/DO-254 [1] defines the verification process used to assure that the designed hardware meets the requirements and identifies additional verification processes that should be performed for design assurance level A and B systems. While RTCA/DO-254 identifies potential verification methods for Level A and B systems, it does not define any criteria to determine when the verification process is sufficient or complete. This research investigates advanced verification coverage methods suitable for safety-critical AEH, identifies applicable coverage metrics, and proposes verification methods and coverage targets for design assurance level A, B, and C level hardware. In addition, the need for the qualification of verification tools and the use of commercial off-the-shelf intellectual property are investigated for potential safety issues. It should be noted that, while a wide variety of COTS solutions exist for aviation systems, this research is focused on AEH, which is certified using DO-254.