Aptitude Testing and the Legal Profession 6 June 2011 Dr. Chris Dewberry Birkbeck, University of London 1 Acknowledgement: I wish to thank Chris Hutton for his help in compiling information in the section of this report entitled Aptitude Tests Currently Used in the Professional Services Sector, Appendix 3, and Appendix 4. 2 Contents Executive Summary ................................................................................................................... 6 The Background to this Report ................................................................................................ 12 An Introduction to Aptitude and Aptitude Tests ..................................................................... 13 A Brief History of Psychological Ability Testing ....................................................................... 16 The Origins of Testing .......................................................................................................... 16 The Birth of the Modern Ability Test ................................................................................... 17 The Introduction of Group Testing ...................................................................................... 19 The New Science of Psychological Testing .......................................................................... 20 The Growth of the Ability and Aptitude Testing Industry ................................................... 20 Aptitudes and Ability in the 20th Century ................................................................................ 22 A Single Factor of Cognitive Ability? .................................................................................... 24 Multiple Components of Cognitive Ability? ........................................................................ 24 The Sixty Year Reign of Aptitudes ....................................................................................... 25 Aptitudes and Ability in the 21st Century ................................................................................ 27 The Fall of Aptitudes and the Rise of g ................................................................................ 27 The Importance of Cognitive Ability .................................................................................... 28 Adverse Impact: Sub-Group Differences in g ...................................................................... 32 The Influence of Practice and Coaching on Aptitude Test Results ...................................... 38 The Nature of Aptitude Tests and the Influences on Aptitude Test Results ....................... 38 Criteria for Evaluating Psychometric Tests: Reliability and Validity ........................................ 41 Reliability ............................................................................................................................. 41 Face validity ......................................................................................................................... 42 Construct Validity ................................................................................................................ 43 Content Validity ................................................................................................................... 43 Criterion-related validity ..................................................................................................... 43 Concurrent validity .......................................................................................................... 44 3 Predictive validity ............................................................................................................ 45 Operational and “Corrected” Criterion-Related Validity ..................................................... 45 Incremental Criterion-Related Validity................................................................................ 46 Other Criteria for Evaluating Selection Methods .................................................................... 47 Acceptability ........................................................................................................................ 47 Usability ............................................................................................................................... 47 Generality ............................................................................................................................ 48 Fairness ................................................................................................................................ 48 Adverse impact ................................................................................................................ 49 The four-fifths rule .......................................................................................................... 51 Selection methods and differences in sub-group performance ...................................... 53 Differential Item Functioning .......................................................................................... 54 Utility ................................................................................................................................... 58 Aptitude Tests Currently Used in the Professional Service Sector .......................................... 59 Aptitude Tests in the UK and Ireland .................................................................................. 59 LNAT ................................................................................................................................ 61 Irish Aviation Authority ................................................................................................... 62 NATS ................................................................................................................................ 63 BioMedical Admissions Test (BMAT) ............................................................................... 64 United Kingdom Clinical Aptitude Test (UK CAT) ............................................................ 65 UK Aptitude Tests, Practice, and Coaching ..................................................................... 66 Clinical Problem Solving and Situational Judgement Tests ............................................. 66 Aptitude Tests in North America ......................................................................................... 67 SAT ................................................................................................................................... 67 Medical College Admissions Test (MCAT) ....................................................................... 68 Law Schools Admission Test (LSAT) ................................................................................. 69 The Development of a New Selection Test for Law Students ......................................... 70 4 Conclusions and Recommendations ....................................................................................... 71 Recommended Criteria for Evaluating Ability or Aptitude Tests in the Legal Profession ... 73 Appendix 1: Personnel Selection Methods ............................................................................. 75 Appendix 2: A Brief Introduction to Factor Analysis ............................................................... 79 Background .......................................................................................................................... 79 Factor extraction ................................................................................................................. 80 Factor rotation ..................................................................................................................... 83 Interpreting the results of the analysis ............................................................................... 85 Appendix 3: Examples of Aptitude Tests ................................................................................. 89 American College Test (ACT) ............................................................................................... 92 Armed Forces Vocational Aptitude Battery (ASVAB) .......................................................... 92 BioMedical Admissions Test (BMAT) ................................................................................... 93 Dental Admission Test (DAT) ............................................................................................... 94 Differential Aptitude Test (DAT) for Personnel and Career Assessment ............................ 94 Employee Aptitude Survey (EAS) ......................................................................................... 95 Flanagan Aptitude Classification Tests (FACT) .................................................................... 95 General Aptitude Test Battery (GATB) ................................................................................ 97 LNAT .................................................................................................................................... 97 Law School Admission Test (LSAT) ...................................................................................... 98 Medical College Admission Test (MCAT) ........................................................................... 100 Multidimensional Aptitude Battery-II ............................................................................... 101 Professional Aptitudes .....................................................................................................