DOT/FAA/AM-19/6 Office of Aerospace Medicine Washington, DC 20591 Best Practices in Pilot Selection Dana Broach David Schroeder Kevin Gildea February 13, 2019 Final Report NOTICE This document is disseminated under the sponsorship of the U.S. Department of Transportation in the interest of information exchange. The United States Government assumes no liability for the contents thereof. ___________ This publication and all Office of Aerospace Medicine technical reports are available in full-text from the Civil Aerospace Medical Institute’s publications website: http://www.faa.gov/go/oamtechreports Technical Documentation Page 1. Report No. 2. Government Accession No. 3. Recipient's Catalog No. DOT/FAA/AM-19/6 4. Title and Subtitle 5. Report Date Best Practices in Pilot Selection February 13, 2019 6. Performing Organization Code 7. Author(s) 8. Performing Organization Report No. Dana Broach, Kevin Gildea, & David J. Schroeder 9. Performing Organization Name and Address 10. Work Unit No. (TRAIS) Aerospace Human Factors Research Laboratory AAM-500 11. Contract or Grant No. P.O. Box 25082 Oklahoma City, OK 73125 12. Sponsoring Agency Name and Address 13. Type of Report and Period Covered Office of Aerospace Medicine Federal Aviation Administration Final 800 Independence Avenue, S. W. Washington, DC 20591 15. Supplementary Notes 16. Abstract Demand for new civilian pilots continues to grow as the world’s aviation system expands. U.S. mainline and regional air carriers will need about 1,900 to 4,500 new pilots per year (U.S. Government Accountability Office, 2014). Selection of the new pilots is a critical human resource management challenge for operators. We review civilian pilot selection procedures relative to a set of seven best practices: 1) Conduct a job analysis; 2) Define measurable, observable job performance metrics; 3) Identify and use reliable and valid predictors; 4) Conduct an appropriate validation study; 5) Determine cut-scores (pass/fail) on tests based on predicted job performance; 6) Evaluate the fairness of tests and cut-scores; and 7) Document the analyses. We reviewed 15 pilot test batteries and selection processes for 2 U.S. and multiple foreign airlines based on publicly available information. Overall, adherence to best practices as defined by the relevant legal and professional guidelines, standards, principles, and practices for the development, validation, and use of employee selection procedures, was inconsistent. Relatively few of the test batteries were based on a recent or current job analysis. There is evidence of psychometric reliability and useful validity for the test batteries. However, there is no evidence on the reliability or validity of the interviews used in pilot selection. The primary job performance criterion is success or failure in training. No data are available on the comparative selection (pass) rates by demographic groups on the test batteries. Technical reporting and documentation on pilot test batteries is better in Europe than in the U.S. Overall, we agree with the 2012 conclusion reached by the International Air Transport Association (IATA) that pilot selection programs worldwide and in the U.S. do not appear to have a strong scientific basis. We make two recommendations to improve the state-of-the-art in civilian pilot selection. 17. Key Words 18. Distribution Statement Pilot Document is available to the public Personnel Selection through the National Technical Job Performance Information Service, Springfield, VA 22161 Job Analysis Ability Tests 19. Security Classif. (of this report) 20. Security Classif. (of this page) 21. No. of Pages 22. Price UNCLASSIFIED UNCLASSIFIED 73 Form DOT F 1700.7 (8-72) Reproduction of completed page authorized ACKNOWLEDGMENT Research report in this paper was conducted under the Air Traffic Program Directive/Level of Effort Agreement between the Human Factors Division (ANG-C1), FAA Headquarters, and the Aerospace Human Factors Research Division of the FAA Civil Aerospace Medical Institute. i TABLE OF CONTENTS Acknowledgment ........................................................................................................................................... i Table of Contents .......................................................................................................................................... ii Table of Figures ........................................................................................................................................... iv Table of Tables ............................................................................................................................................. v Introduction ................................................................................................................................................... 1 Organization of report ............................................................................................................................... 4 Terminology .............................................................................................................................................. 4 Chapter 1: Best practices in personnel selection processes .......................................................................... 7 Conduct a job analysis .............................................................................................................................. 7 Define job performance measures ............................................................................................................. 8 Identify predictors ..................................................................................................................................... 9 Evaluate relationship of KSAO measures to job performance criteria ..................................................... 9 Set cut-scores and/or decision rules ........................................................................................................ 10 Evaluate the fairness of the selection process ......................................................................................... 10 Assess selection process utility ............................................................................................................... 11 Challenges in applying these best practices ............................................................................................ 12 Chapter 2: Pathways for Becoming a Pilot ................................................................................................. 15 Pathway 1: University-based pilot training programs ............................................................................. 15 Pathway 2: Independent flight schools/flight academies ........................................................................ 16 Pathway 3: Former military pilot ............................................................................................................ 17 Pathway 4: Multi-Crew Pilot License (MPL) ......................................................................................... 17 Pathway 5: “Flow-through” agreements ................................................................................................. 18 Chapter 3: Civilian Pilot Selection .............................................................................................................. 20 Previous reviews of (civilian) pilot selection practices........................................................................... 20 Review of pilot aptitude test batteries ..................................................................................................... 20 Best practice #1: Conduct a job analysis............................................................................................. 21 Best practice #2: Define job performance measures. .......................................................................... 27 Best practice #3: Identify predictor measures. .................................................................................... 27 Best practice #4: Evaluate relationship of KSAO measures (predictors) to job performance criteria. ............................................................................................................................................................ 27 Best practice #5: Set cut-scores and/or decision rules. ....................................................................... 28 Best practice #6: Evaluate the fairness of the selection procedures. ................................................... 30 Best practice #7: Assess selection procedure utility. .......................................................................... 30 Innovations in pilot aptitude assessments ............................................................................................... 30 Personality assessment. ....................................................................................................................... 30 Social skills or competencies. ............................................................................................................. 30 Assessment Centers. ........................................................................................................................... 32 Interviews ................................................................................................................................................ 33 Chapter 4: Screening in Other Transportation-related Occupations ..........................................................