A University of Sussex DPhil thesis Available online via Sussex Research Online: http://sro.sussex.ac.uk/ This thesis is protected by copyright which belongs to the author. This thesis cannot be reproduced or quoted extensively from without first obtaining permission in writing from the Author The content must not be changed in any way or sold commercially in any format or medium without the formal permission of the Author When referring to this work, full bibliographic details including the author, title, awarding institution and date of the thesis must be given Please visit Sussex Research Online for more information and further details i The use of psychometric and other assessment centre measures in predicting performance on a naval command course Ian William Beadle Doctor of Education The University of Sussex April 2011 ii DECLARATION WORK NOT SUBMITTED ELSEWHERE FOR EXAMINATION I hereby declare that this thesis has not been and will not be, submitted in whole or in part to another University for the award of any other degree. Signature I W Beadle iii SUMMARY UNIVERSITY OF SUSSEX IAN WILLIAM BEADLE DOCTOR OF EDUCATION The use of psychometric and other assessment centre measures in predicting performance on a naval command course SUMMARY The Admiralty Interview Board (AIB) is the Royal Navy‘s assessment centre whose role is to select young people for officer training. The two aims of the study were (1) to investigate the relative value of psychometric versus other assessment centre selection measures and (2) the value of these and other approaches for selecting naval commanding officers for practitioners. The AIB selection data was used to investigate the long-term prediction of some of the selection measures, particularly the psychometric tests, in predicting the outcome for students attending the Submarine Command Course. Few pieces of research have looked at the long-term prediction of a real command situation. This research examines the prediction of a practical naval command situation where the student has to make rapid decisions under pressure and where failure to make the correct decision could be costly. A literature review showed that whilst cognitive tests, personality inventories and other assessment measures can predict job performance and training successes, the meta-analytical techniques used to pool research studies have produced inconsistent findings that could confuse practitioners. The students attended the command course, on average, thirteen years after the initial AIB selection process. Selection scores were available for 93 students, 57 of whom also completed a ‗Big-Five‘ personality inventory and an Occupational Stress Indicator (OSI) at the start of the 24-week course. The average age of the students starting the course was 32. The students were assessed throughout the course and were graded as pass or fail. They were also given an A to F Course Grade. In addition, 88 students were graded on twenty aspects of performance covering eight tactical grades, three administrative grades and nine personality grades. iv The results showed that the means of the pass and fail groups on the AIB Non-verbal test were statistically significantly different with a moderate effect size. The correlation for this relationship (with the A to F Course Grade) was 0.20. The Non-verbal test score also correlated with the course instructor‘s grades on tactical performance at 0.30 and the Verbal test correlated 0.23 with the administration grades. None of the other AIB selection measures showed significant results. While this is a disappointing result, the students were a very homogeneous group and to obtain these findings for the Non-verbal and Verbal test after thirteen years shows the predictive power of these tests. Although the findings may be of theoretical interest the low correlations mean that not much variance in performance is explained. The tests would not be a useful screening device to reduce the failure rate on the course because there would be too much misclassification. None of the Big Five personality scales predicted success on the course or the other course grades but statistically significant differences were found for the means of two the OSI scales: these were for ‗Ambition‘ and a Type A Behaviour measure. Ambition was the only scale which correlated significantly with the A to F Course Grade at 0.43. Ambition was also found to correlate with the total score for the twenty performance grades, the tactical grades and the personality grades and several individual performance grades including Practical Ability, Leadership and Command Presence with correlations approaching 0.4. Further research on this aspect of behaviour may be worthwhile. However, there are lessons to be learned. The literature review shows that practitioners need to scrutinize journal articles and book chapters on the validity of selection measures extremely carefully. It may be that measures which have been shown to predict the performance of junior staff are inappropriate for the selection of more senior staff with similar job experience. [594 words] v TABLE OF CONTENTS Page Title i Declaration ii Summary iii Table of Contents v List of Tables vii List of Figures ix Acknowledgements x Acknowledgement of Third Party Copyright xi Preface xii 1 Introduction 1.1 Background 1 1.2 Aims of the Research 1 1.3 Assessment and Selection 2 1.4 Assessment Centres 3 1.5 Typical Components of an Assessment Centre 5 1.6 The Use of Meta-Analysis in Personnel Selection Research 7 1.7 Practical Aspects of Personnel Selection 8 1.8 Thesis Structure 10 1.9 Chapter Summary 10 2 Psychometric and other Assessment Centre Methods 2.1 Introduction 11 2.2 Psychometric Tests which Measure Cognitive Ability 11 2.3 Tests of Cognitive Ability as Predictors of Job Performance and 12 Training Success 2.4 How Relevant is the Historical Data used in Meta-Analysis? 15 2.5 Personality Theories and Inventories 16 2.6 Personality Inventories as Predictors of Job Performance and Training 18 Success 2.7 Issues with Personality Inventories and Prediction 22 2.8 Situational Tests 26 2.9 Scored Biodata 26 2.10 References 27 2.11 Meta-Analysis Research on Assessment Centre Methods 27 2.12 Chapter Summary 30 3 Selection in the Military 3.1 Introduction 31 3.2 Testing in the Military 31 3.3 Leadership 32 3.4 Command and Hazardous Environments 35 3.5 Stress 37 3.6 The Moderating Effects of Type A Behaviour, Locus of Control and 39 Hardiness 3.7 Chapter Summary 43 continued … vi 4 Context and Method 4.1 Introduction 44 4.2 Research Aims and Hypotheses 44 4.3 Sequence of Research Events 45 4.4 The Admiralty Interview Board Procedures 45 4.5 The Submarine Command Course 49 4.6 A Previous Study 49 4.7 The Second Phase of the Research for this Thesis 51 4.8 Research Design 52 4.9 Participants 53 4.10 Materials 54 4.10.1 The Admiralty Interview Board Measures 54 4.10.2 The personality inventory: OCEAN 56 4.10.3 The Occupational Stress Indicator 59 4.10.4 Submarine Command Course Performance Measures 60 4.11 Procedures 65 4.12 Analyses Undertaken 66 4.13 The Limitations of the Research 66 4.14 Chapter Summary 67 5 Results 5.1 Introduction 68 5.2 Analysis A: The AIB Measures and the Pass-Fail Result and the 69 Course Grade 5.3 Analysis B: The AIB Measures and Twenty Quality Grades 71 5.4 Analysis C: The Personality Inventory (OCEAN) and the Pass-Fail 75 Result and the Course Grade 5.5 Analysis D: The Personality Inventory and Twenty Quality Grades 77 5.6 Analysis E: The Occupational Stress Indicator, the Pass-Fail Result 82 and the Course Grade 5.7 Analysis F: The Occupational Stress Indicator and the Twenty Quality 84 Grades 5.8 Analysis G: Miscellaneous AIB items 88 5.9 Analysis H: Links between the OCEAN Personality Inventory and the 91 AIB Measures# 6 Discussion 6.1 Introduction 92 6.2 The Psychometric Tests and other AIB Measures 92 6.3 Biodata, Board Dimensions and Headteacher‘s Reference Report 94 6.4 The OCEAN Personality Inventory 95 6.5 The Occupational Stress Indicator 98 6.6 Links between the Personality Dimensions and the AIB Measures 102 6.7 Predicting Naval Command Performance 103 6.8 Practitioner Issues 107 6.9 Conclusion 109 7 References 111 8 Appendices 120 vii LIST OF TABLES Page Chapter 1 Table 1.1 Components of an assessment centre 6 Table 1.2 Job criteria and exercise matrix 8 Chapter 2 Table 2.1 Meta-analyses of cognitive ability validity studies from the UK and Spain 13 Table 2.2 Meta-analyses of cognitive ability validity studies from European studies 13 Table 2.3 Meta-analyses of UK cognitive ability validity studies 14 Table 2.4 Meta-analyses of UK cognitive ability validity studies for various 15 occupations Table 2.5 The Big Five personality domains and representative traits 17 Table 2.6 Validity coefficients of the Big Five for different occupational groups 20 Table 2.7 Validity coefficients from meta-analysis for various selection 28 measures Chapter 3 Table 3.1 Validity coefficients for the Big Five Personality dimensions and 34 leadership ratings Table 3.2 Validity coefficients for lower order personality traits and leadership 34 ratings Chapter 4 Table 4.1 Table of the sequence of the research. 45 Table 4.2 Timetable for AIB showing candidates‟ activities and outputs from this. 46 Table 4.3 Prediction of AIB Tests against Professional Marks at BRNC. 48 Table 4.4 Details of the research carried out. 52 Table 4.5 Ages and time elapsed relevant to participants.