The Accuracy and Precision of Measurement: Tools for Validating Reaction Time Stimuli Thesis Presented in Partial Fulfillment of the Requirements for the Degree Master of Arts in the Graduate School of The Ohio State University By Leandro Avila Calcagnotto Graduate Program in Communication The Ohio State University 2020 Thesis Committee Gerald Kosicki, Advisor Richard Huskey 1 Copyrighted by Leandro Avila Calcagnotto 2020 2 Abstract Our ability to test and falsify theories rests on valid and reliable measurement. Minimizing noise, or measurement error, is a hurdle that researchers must overcome. Researchers use state-of-the-art tools to investigate the cognitive underpinnings of human communicative behavior and measurement error is likely to differ by instrument. It is important to determine if a given instrument is sufficiently accurate to extract signal from noise. Reaction time is commonly used to measure cognitive processes during media use. A reaction time measurement tool has been recently implemented in Asteroid Impact, an open source, naturalistic, experimental video game stimulus. We tested if reaction time latency differed across computer operating system (macOS, Windows 10, Ubuntu 18.04), software (Python 2, Python 3), and trial modality (audio, visual) in Asteroid Impact. We used an Arduino microcontroller to generate a ground truth value of average response time latency. Specifically, we measured the latency between the onset of visual and auditory trials and a response from the Arduino. The Arduino was programmed to look for changes in luminance (visual trial) or sound amplitude (audio trial) every millisecond. When a change was detected, the Arduino was programmed to issue a keypress response. Any observed latency between trial onset and the microcontroller’s response reflects measurement error in the computer hardware and software. Ideally, this latency will have a small mean and variance, which would indicate that Asteroid Impact ii is accurate in measuring reaction times. All code to fully reproduce and replicate this study is available on GitHub (https://anonymous.4open.science/r/8b618f1e-6cf4-4bdd- 9e3b-4058a37bfb53/). ANOVA results showed a significant three-way interaction F (2, 1188) = 199.95, p < .001, η2 = .25. Auditory trials using Python 3 on macOS had the lowest latency (M = 44.6, SD = 8.4). Auditory trials using Python 2 on Ubuntu 18.04 had the longest latency (M = 121, SD = 18.4). Raincloud plots of the main and two-way interaction (all p’s < .001) effects are included to further characterize the distribution of results. Which configuration of Asteroid Impact is most accurate when measuring reaction times? It depends. Auditory trials are most accurate on macOS (regardless of Python version). However, visual trials are most accurate using Python 2 on Ubuntu 18.04. Practically speaking, Windows 10 may be the best solution as it has acceptable accuracy for both visual and auditory trials (regardless of Python version). We are encouraged to see that reaction time measurements collected using Asteroid Impact are comparable to other response-latency measuring software. Moreover, these values are well below the speed at which the human visual system can detect and respond to complex visual perception tasks. Together, these results demonstrate that Asteroid Impact is a valid and reliable stimulus for measuring reaction time data. iii Dedication In dedication to my family and friends who supported me throughout this process. iv Acknowledgments My deepest gratitude to Richard Huskey and Gerald Kosicki for their unwavering guidance, support, and devotion to this project. My sincere thanks to The Ohio State University for the opportunities and life experience. v Vita 2009 ............................................................... Palos Verdes Peninsula High School 2013 ............................................................... B.A. Double Majored in Psychology and Communication, University of California, Santa Barbara 2018 to 2019 ................................................. Graduate Research Fellow, The Ohio State University 2019 to 2020 ................................................. Graduate Teaching Associate, School of Communication, The Ohio State University 2020 .............................................................. M.A. Communication, The Ohio State University Fields of Study Major Field: Communication vi Table of Contents Abstract ............................................................................................................................... ii Dedication .......................................................................................................................... iv Acknowledgments ............................................................................................................... v Vita ..................................................................................................................................... vi List of Tables ..................................................................................................................... ix List of Figures ..................................................................................................................... x Chapter 1. Introduction ....................................................................................................... 1 The Big Problem in Science ........................................................................................... 1 Identifying Instrument Noise .......................................................................................... 1 Validating Experimental Tools ....................................................................................... 2 Chapter 2. Literature Review .............................................................................................. 3 The Impetus for Reaction Time Methodology ................................................................ 4 The Variable Field of Communication ........................................................................... 6 Reaction Time Measures in Communication .................................................................. 7 Current Trends in Communication’s Empirical Philosophy ........................................... 7 Asteroid Impact as an Experimental Platform ................................................................ 8 Sources of Response Latency Noise in the Modern Era ................................................. 9 Microcontrollers as Validation Tools ........................................................................... 10 The Current Investigation ............................................................................................. 11 Chapter 3. Methods ........................................................................................................... 13 Overview and Design .................................................................................................... 13 Materials ....................................................................................................................... 14 Procedure ...................................................................................................................... 15 Chapter 4. Results ............................................................................................................. 16 Purpose of Experiment and Statistical Approach ......................................................... 16 Interpreting Response Latencies ................................................................................... 16 Overall ANOVA Model and Main Effects ................................................................... 18 Moderation Analysis for Two and Three-Way Interactions ......................................... 20 Commentary .................................................................................................................. 22 vii Chapter 5. Discussion ....................................................................................................... 25 Summary of Principle Findings .................................................................................... 25 Limitations .................................................................................................................... 26 Value of Present Research ............................................................................................ 26 Summation .................................................................................................................... 27 Ch 6. Conclusion ............................................................................................................... 28 Implications for Future Researchers ............................................................................. 28 Arduinos as Empirical Tools ........................................................................................ 29 From Method to Theory ................................................................................................ 29 Final Statement ............................................................................................................. 29 Bibliography ..................................................................................................................... 31 Appendix A: Figures ......................................................................................................... 34 Appendix B: Tables .........................................................................................................