Testing Programs for Transportation Management Systems A Technical Handbook February 2007 Notice This document is disseminated under the sponsorship of the Department of Transportation in the interest of information exchange. The United States Government assumes no liability for its contents or use thereof. This report does not constitute a standard, specification, or regulation. The United States Government does not endorse products or manufacturers. Trade and manufacturers’ names appear in this report only because they are considered essential to the object of the document. 1. Report No. 2. Government Accession No. 3. Recipient's Catalog No. FHWA-HOP-07-088 4. Title and Subtitle 5. Report Date Testing Programs for Transportation Management Systems: A February 2007 Technical Handbook 6. Performing Organization Code 7. Author(s) 8. Performing Organization Report No. Mr. Robert Rausch, P.E., TransCore David Benevelli Mort Serell 9. Performing Organization Name and Address 10. Work Unit No. (TRAIS) TransCore 11. Contract or Grant No. 192 Technology Parkway, Suite 500 DTFH61-01-C-00180 Norcross, GA 30092 12. Sponsoring Agency Name and Address 13. Type of Report and Period Covered Final Report Operations Office of Transportation Management July 2006 – February 2007 Federal Highway Administration 400 Seventh Street, SW 14. Sponsoring Agency Code Washington, DC 20590 15. Supplementary Notes FHWA Task Order Manager: Tom Stout (HOTM) 16. Abstract Testing Plans for Transportation Management Systems: A Technical Handbook to provide direction, guidance, and recommended practices for test planning, test procedures, and test execution for the acquisition, operation, and maintenance of transportation management systems and ITS devices. 17. Key Word 18. Distribution Statement test, testing, acceptance, verification, test plan, No restrictions installation, transportation management system, intelligent transportation system 19. Security Classif. (of this report) 20. Security Classif. (of this page) 21. No. of Pages 22. Price Unclassified Unclassified 170 Form DOT F 1700.7 (8-72) Reproduction of completed page authorized This page intentionally left blank. Table of Contents 1. Introduction......................................................................................................................................1 1.1 Background ..............................................................................................................................1 1.2 Objectives.................................................................................................................................2 1.3 Document Structure .................................................................................................................2 2. Testing: A System Engineering Life Cycle Process ........................................................................4 2.1 Overview ..................................................................................................................................4 2.2 The System Life Cycle .............................................................................................................4 2.3 Test Planning ...........................................................................................................................5 2.4 Summary................................................................................................................................14 3. The Role of Testing in the Project Life Cycle ................................................................................15 3.1 Overview ................................................................................................................................15 3.2 National ITS Architecture .......................................................................................................15 3.3 Transportation Management Systems Procurement..............................................................21 3.4 Project Management, Staffing and Training ...........................................................................25 3.5 Documentation .......................................................................................................................26 3.6 Configuration Management....................................................................................................27 3.7 Summary................................................................................................................................29 4. An Introduction to Testing .............................................................................................................30 4.1 Overview ................................................................................................................................30 4.2 Test Methods..........................................................................................................................30 4.3 Test Approach........................................................................................................................31 4.4 Test Types..............................................................................................................................33 4.5 Test Identification and Requirements Traceability .................................................................37 4.6 Test Descriptions....................................................................................................................37 4.7 Test Requirements and Resources........................................................................................37 4.8 Test Execution........................................................................................................................40 4.9 Summary................................................................................................................................43 5. Planning a Project Test Program ..................................................................................................44 5.1 Overview ................................................................................................................................44 5.2 A Building Block Approach.....................................................................................................44 5.3 The Product Maturity Concept................................................................................................48 5.4 Risk ........................................................................................................................................51 5.5 Summary................................................................................................................................51 6. Hardware Testing ..........................................................................................................................52 6.1 Overview ................................................................................................................................52 6.2 What Types of Testing Should Be Considered? ....................................................................52 6.3 When Should Testing Occur? ................................................................................................68 6.4 Hardware Test Phases...........................................................................................................71 6.5 Other Considerations for the Hardware Test Program...........................................................80 6.6 Summary................................................................................................................................85 7. Software Testing ...........................................................................................................................87 7.1 Overview ................................................................................................................................87 7.2 What Types of Testing Should Be Considered? ....................................................................88 Table of Contents Page iii 7.3 When Should Testing Occur? ..............................................................................................103 7.4 Software Test Phases ..........................................................................................................104 7.5 Other Considerations for a Software Test Program .............................................................106 7.6 Summary..............................................................................................................................107 8. System Level Testing ..................................................................................................................109 8.1 Overview ..............................................................................................................................109 8.2 Subsystem Testing...............................................................................................................109 8.3 Systems Testing...................................................................................................................110 8.4 Summary..............................................................................................................................111 9. Other Testing Considerations......................................................................................................112 9.1 Overview ..............................................................................................................................112