The Relationship between Complex Problem Solving and Intelligence: An Analysis of Three Computer Simulated Scenarios Katherine Jane Ryan School of Psychology The University of Sydney New South Wales AUSTRALIA A thesis submitted in fulfilment of the requirements for the degree of Doctor of Philosophy 2006 ii STATEMENT OF ORIGINALITY I hereby declare that this submission is my own work, and that, to the best of my knowledge and belief, it contains no material previously published or written by another person, nor material which to a substantial extent has been accepted for the award of any other degree or diploma of a university or other institute of higher learning, except where due acknowledgement is made in the body of the text. Katherine Jane Ryan Signed: __________________________ Date: ____29 March, 2006____ iii DEDICATION I would like to thank my original supervisor, Lazar Stankov, for teaching me everything I know about ‘intelligence’ theory and for having faith in my abilities. I am both proud and humbled to be a (small) part of the Spearman, Cattell, Horn, Stankov, and Fogarty lineage of supervisors. Special thanks must go to Alan Craddock and Damian Birney who took over my supervision and provided guidance and generous support. Sincere thanks also to John Crawford for his role as a mentor throughout. Thank you to Robert Wood, Dietrich Wagener, and Heinz-Martin Süß for giving me access to their computer simulations and for teaching me the intricacies of the programs. I would especially like to thank Robert Wood for guiding my interest in ‘organisational behaviour’. Thank you to Sabina Kleitman who showed a Melbournian the ropes at Sydney University and made summer statistics camps in Canberra actually enjoyable! I also owe a huge thank you to Andrew Cartwright for imbuing me with his computer expertise and making the computerised testing for this thesis possible. Thank you to my maternal Austrian side ‘the Renz a.k.a. Renc family’, particularly my aunt Rita Miles, for assistance with the translation of much of the material for this thesis from German to English. I would also like to thank my aunt Carol and uncle/Professor Chris Harman and uncle Peter Ryan for inspiring me to follow intellectual pursuits. Most importantly, I would like to thank my parents Irene (Renc) and Terry Keating, and Doug Brimble, to whom I dedicate this thesis. Thank you for providing me with a loving and supportive environment that allowed me to fulfil my dreams. In loving memory of my father, Stephen Charles Ryan [15 March1950 – 06 January1995]. iv TABLE OF CONTENTS 1 The Measurement of Problem Solving 1 1.1 Simple Problem Solving 3 1.1.1 Gap Definition 6 1.1.2 Simple Problem Solving and Complex Problem Solving: Task 7 Comparison 1.2 Complex Problem Solving: The Task 8 1.2.1 Complexity 9 1.2.2 Connectivity 12 1.2.3 Dynamic Environment 14 1.2.4 Intransparency 15 1.2.5 Polytely 16 1.2.6 Complex Problem Solving Tasks: A Definition 17 1.2.7 Complex Problem Solving and Traditional Intelligence Tests: Task 19 Comparison 1.3 Complex Problem Solving: The Problem Solver 20 1.4 Complex Problem Solving: The Environment 23 1.5 Complex Problem Solving in the Thesis 25 2 Theories of Complex Problem Solving and Intelligence 27 2.1 Methodological Approaches to Complex Problem Solving: North 28 America 2.1.1 Computer Games and Complex Problem Solving Tasks: A 28 Comparison 2.1.2 Computerisation of Traditional Intelligence Tests 32 2.1.3 Ecological Validity of Computer Simulations 34 2.1.4 Summary: North American Research 35 2.2 Methodological Approaches to Complex Problem Solving: Europe 36 2.2.1 Dörner School vs. Broadbent School 37 2.2.2 Complex Problem Solving and Intelligence 41 2.2.3 Explicit and Implicit Complex Problem Solving 42 2.2.4 Theoretical Approach of the Thesis 43 2.3 Naturalistic Decision Making 45 2.3.1 Field Research vs. Laboratory Research 46 2.3.2 Naturalistic Decision Making vs. Complex Problem Solving 47 2.3.3 Summary: Naturalistic Decision Making Research 50 3 Hierarchical Theories of Intelligence 52 3.1 The Theory of Fluid and Crystallised Intelligence: A Review 55 v 3.2 Complex Computer Simulated Scenarios: Some Examples 61 3.2.1 Summary: Complex Computer Simulated Scenarios 62 3.3 Overview of the Thesis 63 4 Specific Components of Intelligence as Predictors of Complex 69 Problem Solving: Study 1 4.1 Introduction 69 4.1.1 Different Demands Hypothesis 69 4.1.2 Low Reliability Hypothesis 72 4.1.3 Brunswik Symmetry 74 4.1.4 Berlin Intelligence Structure Model 76 4.1.5 Specific Components of intelligence as Predictors of Complex 78 Problem Solving 4.1.6 Gender and Group Differences in Complex Problem Solving 81 4.1.7 Strategy in Complex Problem Solving 84 4.1.8 Naturalistic Decision Making in Stocks and Flows Tasks 91 4.1.9 Overview of Predictions: Study 1 94 4.2 Method 98 4.2.1 Participants 98 4.2.2 Materials 99 4.2.3 Cognitive Abilities Tests 99 4.2.4 Complex Problem Solving Tasks 107 4.2.5 Dynamic Forecasting Questionnaire 107 4.2.6 Furniture Factory 110 4.2.7 Decision Rules for Furniture Factory Simulation 116 4.2.8 Furniture Factory Measures: Performance 117 4.2.9 Furniture Factory Measures: Strategy 118 4.3 Procedure 119 4.3.1 Statistical Analyses 120 4.4 Results and Discussion 121 4.4.1 Descriptive Statistics for Cognitive Abilities Variables 121 4.4.2 Descriptive Statistics for Complex Problem Solving Variables 123 4.4.3 Correlations between Cognitive Abilities and Furniture Factory Trials 124 4.4.4 Correlations between Cognitive Abilities and Complex Problem 128 Solving Tasks 4.4.5 Regression of Dynamic Forecasting Questionnaire Performance on 132 Cognitive Abilities Factors 4.4.6 Gender Differences 133 4.4.7 Group Differences 134 4.4.8 Exploration Strategy 135 4.5 Conclusion: Study 1 137 vi 5 Introduction: Study 2 145 5.1 Scoring Issues in Complex Problem Solving Research: Brunswik 146 Symmetry 5.1.1 Goal Achievement Scores 148 5.1.2 Factor Analysis of Complex Problem Solving Performance 150 5.1.3 Relationships of Complex Problem Solving Performance across Three 151 Computer Simulations 5.1.4 Loopholes 151 5.2 Relationship of Complex Problem Solving and Intelligence 152 5.2.1 Relationship between Intelligence and Complex Problem Solving 152 Performance on Three Computer Simulations 5.2.2 Predictors of Complex Problem Solving Performance 154 5.3 Relationship between Complex Problem Solving and Non-Cognitive 155 Variables 5.3.1 Personality 155 5.3.2 Interests 155 5.4 Summary of Key Predictions 156 6 Method: Study 2 159 6.1 Participants 159 6.2 Materials 160 6.2.1 Cognitive Abilities Tests: Computerised 160 6.2.2 Cognitive Abilities Tests: Paper and Pencil 162 6.2.3 Personality 167 6.2.4 Interests 167 6.2.5 Complex Problem Solving Tasks 168 6.2.5.1 Tailorshop: The Task 169 6.2.5.2 Tailorshop: The Measures 174 6.2.5.3 Forestry System (FSYS 2.0) 176 6.2.5.4 Forestry System: Translation 176 6.2.5.5 Forestry System: The Task 179 6.2.5.6 Forestry System: The Measures 184 6.3 Development of a Consistent Complex Problem Solving Score: Goal 186 Achievement 6.3.1 Goal Achievement Score: Furniture Factory 187 6.3.2 Goal Achievement Score: Forestry System 188 6.4 Procedure 189 6.4.1 Statistical Analyses 191 7 Results and Discussion: Study 2 192 7.1 Data Screening 192 vii 7.2 Descriptive Statistics for Cognitive Abilities Variables 193 7.2.1 Correlations Among Cognitive Abilities Variables 195 7.2.2 The Structure of Cognitive Abilities Variables 197 7.3 Descriptive Statistics for Complex Problem Solving Variables 202 7.3.1 Correlations Among Complex Problem Solving Variables 205 7.4 The Relation of Complex Problem Solving and Intelligence 207 7.4.1 The Relation of Furniture Factory Performance and Intelligence 207 7.4.2 The Relation of Tailorshop Performance and Intelligence 209 7.4.3 The Relation of Forestry System Performance and Intelligence 211 7.4.4 Hierarchical Regression of Complex Problem Solving on Intelligence 213 Factors 7.5 The Relation of Complex Problem Solving and Personality Factors 217 7.6 The Relation of Complex Problem Solving and Interest Variables 221 7.7 The Relation of Complex Problem Solving and Biodata 226 7.8 Final Analyses: The Factor Structure of Complex Problem Solving 228 Variables 7.8.1 The Relation of the Furniture Factory Component with Intelligence 229 Factors 7.8.2 The Relation of the Tailorshop Component with Intelligence Factors 231 7.8.3 The Relation of the Forestry System Factors with Intelligence Factors 233 8 Conclusions: Study 2 243 8.1 Scoring Issues 243 8.1.1 Goal Achievement Scores 244 8.1.2 Factor Analysis of Complex Problem Solving Performance 244 8.1.3 Shared Variance in Complex Problem Solving Performance across 247 Three Computer Simulations 8.1.4 Loopholes 247 8.2 Relationship between Intelligence and Complex Problem Solving 248 8.2.1 Relationship between Intelligence and Complex Problem Solving 248 across Three Computer Simulations 8.2.2 Predictors of Complex Problem Solving Performance 251 8.3 Relationship of Complex Problem Solving and Non-Cognitive 251 Variables 8.3.1 Personality 251 8.3.2 Interests 252 8.4 Future Directions 253 9 References 255 viii A Appendix A: Computer simulated scenarios employed in individual 270 differences research B Appendix B: Post-test questionnaire measuring rule knowledge of 284 Furniture Factory simulation C Appendix C: Participant handout for the Furniture Factory task 286 D Appendix D: Tailorshop Simulation - Participant Reference Sheet 287 E Appendix E: Tailorshop Simulation – Directions for The Second 289 Practice Month F Appendix F: Test manual