Doctoral Thesis Submitted to the Faculty of Behavioural and Cultural Studies Heidelberg University in Partial Fulfillment of Th
Total Page:16
File Type:pdf, Size:1020Kb
Doctoral thesis submitted to the Faculty of Behavioural and Cultural Studies Heidelberg University in partial fulfillment of the requirements of the degree of Doctor of Philosophy (Dr. phil.) in Psychology Title of the thesis Assessment of Problem Solving Skills by means of Multiple Complex Systems – Validity of Finite Automata and Linear Dynamic Systems presented by Dipl.-Psych. Andreas Fischer year of submission 2015 Dean: Prof. Dr. Klaus Fiedler Advisor: Prof. Dr. Joachim Funke Assessment of Problem Solving Skills 2 Assessment of Problem Solving Skills Table of content Acknowledgements…………………………………………………………………………….. 4 Summary…………………………………………………………………………..................... 5 List of scientific publications regarding the publication-based thesis ……………………………………………………… 7 1. Introduction…………………………………………………………………………. 8 2. Conceptual Background…………………………………………………………... 9 2.1. MicroDYN – Microworlds based on dynamic linear equations………………... 15 2.2. MicroFIN – Microworlds based on finite state machines………………………. 16 2.3. Problem Solving Skills assessed by means of MicroDYN and MicroFIN……. 18 3. Research on Tools and Perspectives concerning the MCS approach………. 22 3.1. Review 1: Content Validity (cf. Fischer, Greiff, & Funke, 2012) ……….…….. 23 3.2. Review 2: Convergent Validity (cf. Greiff, Fischer, et al., 2013) ……….…….. 25 3.3. Review 3: Discriminant Validity (cf. Fischer et al., in press)..………….……… 27 4. Discussion and outlook …………………………………………………………… 29 References……………………………………………………………………………………… 37 Declarations in accordance with § 8 (1) b and § 8 (1) c of the regulations for doctoral degrees of the Faculty of Behavioural and Cultural Studies of Heidelberg University……………………………………………………………………… 43 Appendix……………………………………………………………………………………… 44 3 Assessment of Problem Solving Skills Acknowledgements First of all, with regard to this thesis, I would like to thank my supervisor Prof. Dr. Joachim Funke, who provided me the opportunity to write this thesis the way I did (in the BMBF project Fu 01JG1062). I deeply appreciate the helpful advice and the creative freedom Joachim offered me throughout all these years. Joachim was always open-minded and ready to discuss each idea objectively, which was deeply inspiring and very helpful in phases of doubt. I also would like to thank my friends and colleagues Dr. Daniel Holt and Dr. Sascha Wüstenberg, who managed to share an office with me for several years each and who contributed indirectly to this thesis by the countless fruitful discussions that helped me clarifying ideas and concepts related to this thesis. Of course this list will always be inconclusive for there are many friends and colleagues who supported me in one way or the other – nevertheless I would like to thank them all. Last but not least – this goes without saying – I am also very grateful to my beloved wife Saskia, my children and the rest of my family for supporting me, believing in me and for making all this possible. 4 Assessment of Problem Solving Skills Summary The assessment of highly domain-general problem solving skills is increasingly important as problem solving is increasingly demanded by modern workplaces (e.g., Autor, Levy, & Murnane, 2003) and increasingly present in international large-scale assessments such as the Programme for International Student Assessment (PISA, e.g., OECD, 2014). This thesis is about the computer-based assessment of problem solving skills based on Multiple Complex Systems (MCS, Greiff, Fischer, Stadler, & Wüstenberg, 2014): The main idea of the MCS approach is to present multiple computer-simulations of “minimally complex” problems (Greiff, 2012) in order to reliably assess certain problem solving skills. In each simulation, the problem solver has to interact with a problem in order to find out (a) how to adequately represent the problem, and (b) how to solve the problem. Up to now, two instances of the MCS approach have been proposed: (1) the MicroDYN approach (based on simulations of linear equation systems) and – more recently, in the second paper of this thesis – (2) the MicroFIN approach (based on simulations of finite state machines). In the current thesis I will elaborate on three research questions regarding the validity (cf. Bühner, 2006) of the MCS approach: (1) its content validity with regard to the concept of complex problem solving; (2) the convergent validity of different instances of the MCS approach; (3) the discriminant validity of the interactive problems of the MCS approach with regard to traditional static measures of reasoning and analytic problem solving skills. Each research question will be addressed in one corresponding paper: In a first paper (Fischer, Greiff, & Funke, 2012) complex problem solving is defined as the goal-oriented control of systems that contain multiple highly interrelated elements. After reviewing some of the major strands of research on complex problem solving (e.g., research on strategy selection, information reduction, intelligence, or on the interplay of implicit and explicit knowledge in the process of complex problem solving) a theoretical framework outlining the most important cognitive processes involved in solving complex problems is derived. The theoretical framework highlights both interactive knowledge acquisition (problem representation) and interactive knowledge application (problem solution) as the two major phases in the process of complex problem solving. Both phases are represented in all current instances of the MCS approach. In a second paper (Greiff, Fischer et al., 2013) the convergent validity of MicroDYN and MicroFIN is investigated (thereby introducing MicroFIN as an alternative to MicroDYN) in order to demonstrate that both instances address the same kind of problem 5 Assessment of Problem Solving Skills solving skills. Based on a multitrait-multimethod analysis of a sample of university students (N = 339) it is demonstrated that – in addition to method-specific skills – both instances assess a common set of skills (method-general traits) related to (1) representing and (2) solving different kinds of interactive problems. In a regression of science grades on reasoning and the skills assessed by the instances of the MCS approach it is demonstrated that only the method-general representation trait and reasoning have substantial unique contributions. Thus, MicroDYN and MicroFIN seem to address a common set of skills and this set of skills is relevant for explaining school grades in science classes even beyond reasoning. In a third paper (Fischer et al., in press) the discriminant validity of the interactive MicroDYN test is investigated by relating it to reasoning and traditional static measures of Analytic Problem Solving skills (APS) as they were applied in PISA 2003 (OECD, 2004). Besides a common core of problem solving skills addressed by both kinds of tasks (e.g., analyzing complex information about the information given at a certain moment in time) Fischer et al. (in press) expected to find evidence for additional skills that were related to interactive problems only (e.g., systematically generating information and interactively testing hypotheses). Results indicate that MicroDYN shares a lot of variance with APS even after controlling for reasoning in a sample of high-school students (N = 577) and the university student sample (see above). With regard to the explanation of school grades MicroDYN had an incremental value compared to reasoning and APS in the high-school student sample but not significantly so in the university student sample (whereas APS had an incremental value in both samples). Basically these findings highlight both potential and limitations of the MicroDYN approach in its current form. Current instances of the MCS approach address a small set of problem solving skills reliably, but it takes more than these skills to competently solve complex problems. Implications for future research on the assessment of problem solving skills are discussed. 6 Assessment of Problem Solving Skills List of scientific publications regarding the publication-based thesis I. Manuscript Fischer, A., Greiff, S., & Funke, J. (2012). The process of solving complex problems. Journal of Problem Solving, 4, 19-42. II. Manuscript Greiff, S., Fischer, A., Wüstenberg, S., Sonnleitner, P., Brunner, M., & Martin, R. (2013). A multitrait–multimethod study of assessment instruments for complex problem solving. Intelligence, 41, 579-596. III. Manuscript Fischer, A., Greiff, S., Wüstenberg, S., Fleischer, J., Buchwald, F., & Funke, J. (in press). Assessing analytic and interactive aspects of problem solving competency. Learning and Individual Differences, DOI:10.1016/j.lindif.2015.02.008. Additional Manuscripts Fischer, A., Greiff, S., & Funke, J. (under review). The history of complex problem solving. In B. Csapó, J. Funke, & A. Schleicher (Eds.), The nature of problem solving. Paris: OECD. Greiff, S., Fischer, A., Stadler, M., & Wüstenberg, S. (2014). Assessing complex problem- solving skills with multiple complex systems. Thinking & Reasoning (ahead-of-print), 1-27. Greiff, S. & Fischer, A. (2013). Der Nutzen einer komplexen Problemlösekompetenz: Theoretische Überlegungen und praktische Befunde. Zeitschrift für Pädagogische Psychologie, 27, 27-39. Greiff, S. & Fischer, A. (2013). Measuring complex problem solving: an educational application of psychological theories. Journal of Educational Research Online, 5, 38-58. Greiff, S., Wüstenberg, S., Molnár, g., Fischer, A., Funke, J., & Csapó, B. (2013) Complex problem solving