The “Error” in Psychology: An Analysis of Quantitative and Qualitative Approaches in the Pursuit of Accuracy by Donna Tafreshi MA, Simon Fraser University, 2014 BA, Simon Fraser University, 2011 Thesis Submitted in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy in the Department of Psychology Faculty of Arts and Social Sciences © Donna Tafreshi 2018 SIMON FRASER UNIVERSITY Fall 2018 Copyright in this work rests with the author. Please ensure that any reproduction or re-use is done in accordance with the relevant national copyright legislation. Approval Name: Donna Tafreshi Degree: Doctor of Philosophy Title: The “Error” in Psychology: An Analysis of Quantitative and Qualitative Approaches in the Pursuit of Accuracy Examining Committee: Chair: Thomas Spalek Professor Kathleen Slaney Senior Supervisor Professor Timothy Racine Supervisor Professor Susan O’Neill Supervisor Professor Barbara Mitchell Internal Examiner Professor Departments of Sociology & Gerontology Christopher Green External Examiner Professor Department of Psychology York University Date Defended/Approved: October 12, 2018 ii Abstract The concept of “error” is central to the development and use of statistical tools in psychology. Yet, little work has focused on elucidating its conceptual meanings and the potential implications for research practice. I explore the emergence of uses of the “error” concept within the field of psychology through a historical mapping of its uses from early observational astronomy, to the study of social statistics, and subsequently to its adoption under 20th century psychometrics. In so doing, I consider the philosophical foundations on which the concepts “error” and “true score” are built and the relevance of these foundations for its usages in psychology. Given the recent surge in interest in qualitative research methods in psychology, I also investigate whether a notion of “error” is relevant to qualitative research practice. In particular, I conduct a content analysis of usages of the term “reliability” within the qualitative methodological literature as a proxy for the concept of “error” within the qualitative research domain. Finally, I compare my explorations of discourse around quantitative and qualitative methods. I conclude that although researchers using methodological tools from these two traditions may hold opposing views on knowledge and truth, they also share a common aim of accuracy. Implications for research practice and education in psychology are discussed. Keywords: measurement error; true scores; accuracy; certainty; reliability; quantitative and qualitative methods iii Dedication Dedicated to the memory of my friend, Joshua Scott Patillo. iv Acknowledgements I thank Dr. Kate Slaney for her guidance and support as a supervisor, mentor, and friend. I also thank Dr. Tim Racine for his encouragement and wisdom throughout my graduate school career and I thank Dr. Susan O’Neill for her enthusiasm and knowledge on teaching qualitative methods. I thank my partner, Minilik Joseph, for putting up with yet another graduate degree. Thank you for keeping me grounded and always being there for me, even during the toughest times. I thank my family for their unconditional support and I thank my parents for the sacrifices they made to allow me to pursue my education. v Table of Contents Approval .......................................................................................................................... ii Abstract .......................................................................................................................... iii Dedication ...................................................................................................................... iv Acknowledgements ......................................................................................................... v Table of Contents ........................................................................................................... vi List of Tables ................................................................................................................. viii List of Figures................................................................................................................. ix Chapter 1. Introduction .............................................................................................. 1 Chapter 2. Error in Historical Perspective Part I: Early Uses of Error in Statistics and the Study of Social and Psychological Phenomena .................................. 5 2.1. The Astronomer’s Error Law .................................................................................. 5 2.2. Quetelet and L’Homme Moyen ............................................................................ 10 2.3. Fechner, Wundt, and the Study of Intra-Individual Variations .............................. 13 2.4. Galton, Pearson, and the Study of Inter-Individual Variations .............................. 15 Chapter 3. Error in Historical Perspective Part II: Twentieth Century Psychology and the Emergence of Test Theory .................................................................. 19 3.1. Foundational Concepts in Classical Test Theory ................................................. 19 3.1.1. Error at the Person-Level (Intra-Individual) .................................................. 20 3.1.2. Error at the Group-Level (Inter-Individual) .................................................... 21 3.1.3. Precision, standard error, and reliability. ...................................................... 22 3.2. Error under Classical Test Theory ....................................................................... 24 3.2.1. Charles Spearman: “Accidental” vs. “Systematic” Errors, Correction for Attenuation, and the “Reliability Coefficient” ............................................................... 24 3.2.2. Truman Kelley: Defining “True” Scores ........................................................ 27 3.2.3. Louis Thurstone: Summarizing Ideas in Classical Test Theory .................... 29 3.2.4. Louis Guttman: Error Based on Three Sources of Variation ......................... 30 3.2.5. Harold Gulliksen: Summarizing Ideas in Classical Test Theory ................... 31 3.3. Meanwhile in Statistics: Fisher’s Analysis of Variance ......................................... 32 3.4. Error under Item Response Theory (IRT) ............................................................ 34 3.5. Error under Generalizability Theory ..................................................................... 36 3.5.1. The Problem of “True” Scores ..................................................................... 38 3.6. Error in Historical View ........................................................................................ 40 Chapter 4. An Analysis of Uses of the “Error” Concept in Psychology and Statistics ............................................................................................................ 41 4.1. Analysis Plan ....................................................................................................... 41 4.2. Results ................................................................................................................ 43 4.2.1. Connecting Emergent Themes .................................................................... 48 4.2.1.1. Aim of Accuracy ........................................................................................ 48 vi 4.2.1.2. Error Over Repeated Measurements ......................................................... 52 4.2.1.3. Sources of Error ........................................................................................ 53 4.2.1.4. “Error” vs. “Variation” ................................................................................. 55 4.2.1.5. “Intra-” vs. “Inter-” Levels of Variation/Error ............................................... 56 4.2.2. Integrating Themes and Research Aims ...................................................... 58 4.2.2.1. Research Aim 1: Comparison of Error in Psychology and Error in Early Astronomy/Statistics ............................................................................................... 59 4.2.2.2. Research Aim 2: German vs. British Psychology Influences on 20th Century American Psychology ............................................................................................. 63 4.2.2.3. Research Aim 3: Levels of Analysis........................................................... 66 4.2.3. Conclusions ................................................................................................. 69 Chapter 5. Error, Reliability, and Qualitative Methodology ................................... 72 5.1. Qualitative Research in Psychology..................................................................... 72 5.2. The “Error” in Qualitative Research ..................................................................... 74 5.3. Reliability in Qualitative Research: A Content Analysis ........................................ 78 5.3.1. Method ........................................................................................................ 79 5.3.1.1. Search Strategy ........................................................................................ 79 5.3.1.2. Analytic Strategy ....................................................................................... 80 5.3.1.3. Reliability of the Current Analyses ............................................................