FINAL Evaluation Report Evaluation of the MAYDAY/9-1-1 Field Operational Test Prepared for: Federal Highway Administration and National Highway Traffic Safety Administration By: In Association with: The Melcher Group July 19, 2006 Quality Assurance Statement The Federal Highway Administration provides high-quality information to serve Government, industry, and the public in a manner that promotes public understanding. Standards and policies are used to ensure and maximize the quality, objectivity, utility, and integrity of its information. FHWA periodically reviews quality issues and adjusts its programs and processes to ensure continuous quality improvement. July 19, 2006 i Final Evaluation Report Evaluation of MAYDAY /9-1-1 FOT Acknowledgement The national evaluation of the MAYDAY/9-1-1 project was performed under Battelle’s contract DTFH61-02-C-00134, Task BA34010 with the United States Department of Transportation (USDOT). This evaluation was conducted in association with Mr. Jim Goerke of the Melcher Group who serves as a sub-consultant to Battelle. The evaluation supports the on-going efforts of the Intelligent Transportation System (ITS) Joint Program Office (JPO) to assess the benefits of ITS deployments in the United States. Battelle wishes to acknowledge the guidance provided to the evaluation team by Ms. Laurie Flaherty of National Highway Traffic Safety Administration (NHTSA), who serves as the Government Task Manager (GOTM), and Mr. Ted Smith of Mitretek who provides support to Ms. Flaherty. Appreciation is also extended to Ms. Linda Dodge of the JPO who provided oversight to this evaluation; Dr. Joseph Peters who manages the national evaluation program within the JPO; the MAYDAY/9-1-1 Field Operational Test (FOT) team led by Mr. Bradley Estochen of the Minnesota Department of Transportation; and Mr. Jim Beutelspacher, 9-1-1 Program Manager of the Minnesota Department of Public Safety, who provided kind support to the conduct of various evaluation activities. Battelle also wishes to acknowledge the Mayo Clinic and many Public Safety Answering Point (PSAP) staff who provided invaluable support to this evaluation, especially Mr. Pete Eggimann with the Minneapolis/St. Paul Metropolitan Emergency Services Board who kindly provided coordination assistance. July 19, 2006 ii Final Evaluation Report Evaluation of MAYDAY /9-1-1 FOT Table of Contents Page ACKNOWLEDGEMENT............................................................................................................ ii LIST OF ACRONYMS ............................................................................................................... vi EXECUTIVE SUMMARY .......................................................................................................viii 1.0 INTRODUCTION............................................................................................................. 1 1.1 Organization of this Document................................................................................1 2.0 DESCRIPTION OF MAYDAY/9-1-1 FIELD OPERATIONAL TEST...................... 2 2.1 The Problems ...........................................................................................................2 2.2 The Field Operational Test Solutions ......................................................................4 2.3 Overview of Field Operational Test Solutions ........................................................8 3.0 EVALUATION APPROACH........................................................................................ 10 3.1 Evaluation Objectives and Scope...........................................................................10 3.2 System Performance Evaluation Approach ...........................................................11 3.2.1 Voice Routing System Acceptance Testing.............................................. 11 3.2.1.1 Description of Field Acceptance Test........................................... 12 3.2.1.2 Evaluation Hypotheses and Measures of Performance................. 18 3.2.1.3 Data Collection ............................................................................. 18 3.2.1.4 Analysis......................................................................................... 19 3.2.2 Data Routing System Performance Analyses ........................................... 20 3.2.2.1 Evaluation Hypotheses and Measures of Performance................. 20 3.2.2.2 Data Collection ............................................................................. 20 3.2.2.3 Analysis of Data Transmission Logs ............................................ 21 3.3 User Acceptance and Deployment Issues Evaluation Approach...........................21 3.3.1 Objectives ................................................................................................. 21 3.3.2 Evaluation Hypotheses and Measures of Performance............................. 22 3.3.3 Data Collection......................................................................................... 22 3.3.3.1 User Acceptance ........................................................................... 22 3.3.3.2 Deployment Issues........................................................................ 27 3.3.4 Analysis..................................................................................................... 27 3.3.4.1 Assessment of User Acceptance ................................................... 27 3.3.4.2 Assessment of Deployment Issues................................................ 28 4.0 EVALUATION FINDINGS........................................................................................... 29 4.1 TSP Voice Routing Field Test ...............................................................................29 4.1.1 Test Results............................................................................................... 29 4.1.2 Summary of Findings................................................................................ 34 4.2 ACN/AACN Data Routing Evaluation..................................................................35 4.2.1 Analysis Results........................................................................................ 35 4.2.2 Summary of Findings................................................................................ 38 July 19, 2006 iii Final Evaluation Report Evaluation of MAYDAY /9-1-1 FOT Table of Contents (Continued) Page 4.3 User Acceptance and Deployment Issues Evaluation Results...............................39 4.3.1 PSAP, Medical Responder, 9-1-1 Program Manager, Traffic Operator Feedback ................................................................................................... 39 4.3.1.1 Interview Results .......................................................................... 39 4.3.1.2 Summary of Findings.................................................................... 41 4.3.2 Telematics Service Provider Feedback..................................................... 45 4.3.2.1 Interview Results .......................................................................... 45 4.3.2.2 Summary of Findings.................................................................... 45 4.3.3 Implications for NG9-1-1 ......................................................................... 46 4.3.3.1 Third Party Technical Review ...................................................... 46 4.3.3.2 Summary of Findings.................................................................... 47 4.3.3.3 Project Team Observations........................................................... 47 4.3.3.4 Summary of Findings.................................................................... 47 5.0 CONCLUSIONS ............................................................................................................. 49 5.1 Evaluation Hypotheses and Findings.....................................................................49 5.2 Conclusions............................................................................................................52 REFERENCES............................................................................................................................ 55 List of Appendices Appendix A: Sampling Theory for Voice Routing Field Test Appendix B: Data Collected for Acceptance Test of FOT Voice Routing Functionality Appendix C: Data Summaries for FOT Data Routing Performance List of Tables Table 2-1. OnStar Data in Support of Data Routing ...................................................................... 7 Table 3-1. Simulated AACN Data Sent as Part of Acceptance Sampling ................................... 17 Table 3-2. Hypotheses, Measures of Performance, Data Sources for Voice Routing Test.......... 18 Table 3-3. Hypotheses, Measures of Performance, and Data Sources for Data Routing Test..... 20 Table 3-4. Hypotheses, Measures of Performance, Data Sources for User Acceptance and Deployment Issues Evaluation.................................................................................... 23 Table 4-1. Summary Results of FOT Acceptance Testing of Voice Routing Functions ............. 30 Table 5-1. Hypotheses Addressed by the Evaluation................................................................... 49 July 19, 2006 iv Final Evaluation Report Evaluation of MAYDAY /9-1-1 FOT Table of Contents (Continued) Page List of Figures Figure E-1. OnStar 9-1-1 Call Procedure Without FOT Solution ............................................... ix Figure E-2. High-Level FOT System Functions Illustrated........................................................