. NAOMS Reference Report Concepts, Methods, and Development Roadmap Prepared by Battelle Memorial Institute for NASA Ames Research Center Contract NNA05AC07C November 30, 2007 NAOMS Reference Report Table of Contents 1. Executive Summary............................................................................................. 1 2. Background .......................................................................................................... 3 2.1. Gore Commission Report ......................................................................... 3 2.2. Preexisting Sources of Aviation Safety Data......................................... 3 3. The NAOMS Concept .......................................................................................... 6 3.1. Tapping the Knowledge of System Operators...................................... 6 3.2. Data Reliability............................................................................................ 7 3.3 Emphasis on Experience Rather Than Opinion.................................... 7 3.4. Voluntary, Confidential, and Anonymous............................................. 8 3.5. Methodological Success Measures .......................................................... 8 3.5.1. Response Rate.............................................................................. 8 3.5.2. Data Quality.................................................................................. 8 4. Development Roadmap...................................................................................... 10 4.1. Methods Development Phase................................................................... 10 4.2. Operational Phase...................................................................................... 12 4.3. Handoff Phase............................................................................................. 13 5. NAOMS Methodology.......................................................................................... 14 5.1 Overview of Methodological Issues ........................................................ 14 5.1.1. Questionnaire Content............................................................... 14 5.1.2. Questionnaire Structure............................................................. 15 5.1.3. Recall Period................................................................................. 15 5.1.4. Survey Mode................................................................................. 15 5.1.5. Sample Source and Size.............................................................. 16 5.1.6. Random versus Panel Sampling Approaches......................... 16 5.1.7. Maximizing Response Rate........................................................ 17 5.2 Resolution of Methodological Issues ..................................................... 17 5.2.1. Establishing Questionnaire Content ........................................ 17 5.2.1.1 Section A: Flight Activity Levels................................ 17 i NAOMS Reference Report 5.2.1.2 Section B: Longitudinal Safety Event Question ....... 18 5.2.1.3 Section C: Focus Questions ......................................... 20 5.2.1.4 Section D: Respondent Feedback Questions............ 21 5.2.2. Determining Questionnaire Organization .............................. 21 5.2.3. Pilot Recall Period........................................................................ 24 5.2.4. Survey Mode................................................................................. 27 5.2.5. Sample Source Size...................................................................... 32 5.2.5.1 Sample Source................................................................ 33 5.2.5.2 Sample Size .................................................................... 34 5.2.6. Choosing Between Random versus Panel Sampling Approaches................................................................................... 34 5.2.7. Maximizing Response Rate........................................................ 35 5.3. Maintaining Respondent Confidentiality............................................... 36 6. Development of a Web-based Version of NAOMS ........................................ 38 6.1. Opportunities for Cost Reduction .......................................................... 38 6.2. Web-based Data Collection Quality ........................................................ 39 6.3. Creation of the Web-based Data Collection Application.................... 40 6.4. Web-based Experiment.............................................................................. 41 6.5. Transfer of Web-based Data Collection System................................... 44 7. Summary............................................................................................................... 45 8. Annotated Bibliography..................................................................................... 46 List of Appendices Appendix 1: Contractor Research Team ........................................................ 1-1 Appendix 2: Demographics .............................................................................. 2-1 Appendix 3: Annotated Bibliography ............................................................. 3-1 Appendix 4: Memory Organization................................................................. 4-1 Appendix 5: Satisficing and Social Desirability Bias.................................... 5-1 Appendix 6: Focus Group Questions .............................................................. 6-1 ii NAOMS Reference Report Appendix 7: Event Lists Generated During Focus Groups and Interviews 7-1 Appendix 8: Workshop 1................................................................................... 8-1 Appendix 9: Field Trial Final Results.............................................................. 9-1 Appendix 10: Workshop 2................................................................................... 10-1 Appendix 11: Air Carrier Questionnaire .......................................................... 11-1 Appendix 12: General Aviation Questionnaire ............................................... 12-1 List of Tables Table 2-1. Major Sources of Aviation Safety Data in 1997 at the Time of NAOMS Conception ............................................................................. 4 Table 4-1. NAOMS Briefings .................................................................................. 12 Table 4-2. NAOMS Response Rates and Surveys Conducted.......................... 13 Table 5-1. Organization of Safety Events in Pilot Memory............................. 23 Table 5-2. Questionnaire Mean Completion Time............................................ 31 Table 5-3. Respondents Failing to Complete at Least One Question ........... 31 Table 5-4. Relationship between number of Events Experienced and Hours Flown During Field Trial ........................................................ 32 Table 5-5. Response Rates by Mode During Field Trial .................................. 32 Table 6-1. Web-survey Response and Completion Metrics ............................ 41 List of Figures Figure 4-1. NAOMS Development Timeline......................................................... 10 Figure 5-1. Confidence in Responses Declines as Recall Period (in days) Increases................................................................................................ 25 Figure 5-2. Effect of Recall Period on Estimated Event Rates ......................... 26 Figure 6-1. Respondents’ View of Ease of Navigation in the Web Survey..... 42 Figure 6-2. Respondents’ View of Time to Complete the Web Survey .......... 43 iii NANAOMS ReOMS Refferencerence Repe Repoortrt Acronyms and Abbreviations AC Survey Air Carrier Survey (NAOMS) ADF Automatic Direction Finder AFA Association of Flight Attendants AFL-CIO American Federation of Labor-Congress of Industrial Organizations AIDS Accident/Incident Data System (FAA) ALPA Air Line Pilots Association AOA Angle of Attack AOPA Aircraft Owners and Pilots Association ASIST Aviation Safety Investment Strategy Team ASMM Aviation System Modeling and Monitoring (component of AvSP) ASRS Aviation Safety Reporting System ATC Air Traffic Control ATIS Air Terminal Information System ATP Air Transport Pilot AvSP Aviation Safety Program (NASA) (succeeded by AvSSP) AvSPEC Aviation Specialty Corporation AvSSP Aviation Safety and Security Program (NASA) BTS U.S. Bureau of Transportation Statistics CAPI Computer-Assisted Personal Interview CASI Computer-Assisted Self Interview CAST Commercial Aviation Safety Team CATI Computer-Assisted Telephone Interview CFI Certified Flight Instructor CFIT Controlled Flight into Terrain CFTT Controlled Flight Toward Terrain COTS Commercial Off the Shelf CPHRE Centers for Public Health Research and Evaluation (Battelle) CRM Cockpit Resource Management DNFA Distributed National FOQA Archive DOT Department of Transportation ETOPS Extended Twin (Engine) Operations F/O First Officer FAA Federal Aviation Administration FADEC Full Authority Digital Engine Control FL Flight Level FOQA Flight Operational Quality Assurance GA General Aviation GA Survey General Aviation Survey (NAOMS) GAAAS General Aviation Avionics and Activity Survey GLM General Linear Model GPS Global Positioning System GPWS Ground Proximity Warning System iviv NAOMS Reference Report HUD Heads-Up Display ICAC In-Close Approach Changes IFR Instrument Flight Rules ILS Instrument Landing System IMC Instrument Meteorological Conditions JIMDAT Joint