Institute for Software-Integrated Systems Technical Report TR#: ISIS-15-111 Title: Software Quality Assurance for the META Toolchain Authors: Ted Bapty, Justin Knight, Zsolt Lattmann, Sandeep Neema and Jason Scott This research is supported by the Defense Advanced Research Project Agency (DARPA)’s AVM META program under award #HR0011-13-C-0041. Copyright (C) ISIS/Vanderbilt University, 2015 Table of Contents List of Figures .............................................................................................................................................. iii List of Tables ............................................................................................................................................... iv 1. Introduction ........................................................................................................................................... 1 1.1 Purpose .......................................................................................................................................... 1 1.2 Scope ............................................................................................................................................. 1 2. Applicability ......................................................................................................................................... 1 3. Applicable Documents .......................................................................................................................... 2 3.1 Contract Level Documents............................................................................................................ 2 3.2 ISIS Governing Documents .......................................................................................................... 2 3.3 Reference Documents ................................................................................................................... 2 4. Program Management, Planning and Environment .............................................................................. 3 4.1 The META SQA Plan ................................................................................................................... 3 4.2 Organization .................................................................................................................................. 3 4.3 Task Planning ................................................................................................................................ 4 4.4 Software Personnel Training ......................................................................................................... 6 4.4.1 SQA Personnel ...................................................................................................................... 6 4.4.2 Software Developer Training Certification ........................................................................... 7 4.4.3 META project management .................................................................................................. 7 4.5 Tools and Environment ................................................................................................................. 7 5 SQA Program Requirements ................................................................................................................. 9 5.1 User Threads as Requirements ...................................................................................................... 9 5.2 Innovation and Improvement ........................................................................................................ 9 5.3 Program Resources Allocation Monitoring .................................................................................. 9 5.4 Best practices of software development ........................................................................................ 9 5.5 Inspections .................................................................................................................................. 10 5.6 Test Case Management ............................................................................................................... 11 5.7 Defect Reports and Change Requests ......................................................................................... 11 5.8 Software and Project Document Deliverables ............................................................................ 11 i 5.9 Requirements Traceability .......................................................................................................... 12 5.10 Software Development Process .................................................................................................. 12 5.11 Project reviews ............................................................................................................................ 12 5.11.1 Formal Reviews .................................................................................................................. 12 5.11.2 Informal Reviews ................................................................................................................ 12 5.12 Test Benches ............................................................................................................................... 13 5.13 Software Configuration Management ......................................................................................... 14 5.14 Release Procedures and Software Configuration Management .................................................. 14 5.15 Change Control ........................................................................................................................... 15 5.16 Problem Reporting ...................................................................................................................... 15 5.17 Continuous Build ........................................................................................................................ 16 5.18 Services and Resources Provided to AVM ................................................................................. 19 5.19 Software Testing ......................................................................................................................... 21 5.19.1 Unit Test .............................................................................................................................. 21 5.19.2 Integration Test ................................................................................................................... 21 5.19.3 Alpha Testing ...................................................................................................................... 21 5.19.4 Beta Testing ........................................................................................................................ 22 5.19.5 Gamma Test ........................................................................................................................ 24 5.20 Meta Release Schedule ................................................................................................................ 26 5.21 Quality Metrics ........................................................................................................................... 26 6 Appendix ............................................................................................................................................. 28 6.1 Coding Documentation Requirements ........................................................................................ 28 6.2 Testing Requirements ................................................................................................................. 28 6.3 Inspection and Code Review Guidance ...................................................................................... 30 6.4 Sample Checklist ........................................................................................................................ 31 ii List of Figures Figure 1: META Team Organization ............................................................................................................ 4 Figure 2: Spiral Development Cycle ............................................................................................................. 5 Figure 3: Internal Development/Testing Cycle ............................................................................................. 6 Figure 4: High Level Architecture Integration View .................................................................................... 7 Figure 5: JIRA Issue Summary page for META ........................................................................................ 15 Figure 6: Beta Testing Issue Reporting (Beta.VehicleFORGE.org) ........................................................... 16 Figure 7: Jenkins interface showing the status of the Meta development branch ....................................... 17 Figure 8: Installer Acceptance Test Checklist ............................................................................................ 22 Figure 9: Core CyPhy Tools Test Checklist ............................................................................................... 25 Figure 10: Core Test Bench Functionality Checklist .................................................................................. 25 Figure 11: 2014 META Release Schedule .................................................................................................. 26 iii List of Tables Table 1: AVM Tools at the End of Gamma Testing ....................................................................................