TEST ENVIRONMENT FOR OPTIMAL PERFORMANCE IN HIGH SCHOOL STUDENTS: MEASURE DEVELOPMENT AND THE RELATIONSHIP WITH STANDARDIZED TEST SCORES A dissertation submitted to the Kent State University College of Education, Health, and Human Services in partial fulfillment of the requirements for the degree of Doctor of Philosophy By Randal J. Rair August, 2019 © Copyright, 2019 by Randal J. Rair All Rights Reserved ii A dissertation written by Randal J. Rair B.S., Youngstown State University, 1999 B.S., Youngstown State University, 2002 M.Ed., Ashland University, 2008 Ph.D., Kent State University, 2019 Approved by , Director, Doctoral Dissertation Committee Aryn C. Karpinski , Member, Doctoral Dissertation Committee Jason Schenker , Member, Doctoral Dissertation Committee Anthony Vander Horst Accepted by , Director, School of Foundations, Leadership, and Kimberly Schimmel Administration , Dean, College of Education, Health and Human James C. Hannon Services iii ABSTRACT Randal J. Rair, June 2019 EVALUATION AND MEASUREMENT TEST ENVIRONMENT FOR OPTIMAL PERFORMANCE IN HIGH SCHOOL STUDENTS: MEASURE DEVELOPMENT AND THE RELATIONSHIP WITH STANDARDIZED TEST SCORES (294 pp.) Director of Dissertation: Aryn C. Karpinski, Ph.D. The rise of mobile technology and increased school accountability has enmeshed the last two generations of students in unprecedented educational experiences. This has fostered learners who are varied in their communication and academic capabilities. Simultaneously, the number of standardized tests, both mandated and voluntary, has been steadily increasing. Few studies are available that examine the preferred environmental conditions for students during these tests. The current study consists of two parts: (1) the Pilot Study and (2) the Main Study. The Pilot Study examined the psychometric properties of a measure called the Test Environment for Optimal Performance (TEOP), which was created to explore the testing preferences of the current generation of college students (i.e., Millennials). Following this quantitative section of the Pilot Study, selections of Millennials were qualitatively interviewed on topics related to test environment preferences to explore the underlying factor structure meaning. Findings from both the quantitative and qualitative portions suggest that students have a partiality for physical (i.e., “Action”) and auditory (i.e., “Sound) activities when testing. The Main Study extended the Pilot Study by investigating current high school students’ (i.e., Generation Z) preferred standardized testing conditions and if these preferred conditions have a predictive relationship with high-stakes standardized test scores (i.e., the ACT). The Main Study analyzed data using a Confirmatory Factor Analysis (CFA) to examine the psychometric properties of the TEOP in the new population, and to provide further evidence to support the “Action” and “Sound” test environment factor structure. Following the CFA, Hierarchical Multiple Linear Regressions were conducted to examine the hypothesized relationships between the TEOP factors “Action” and “Sound” and high-stakes aptitude outcomes (i.e., ACT scores). The results warranted an additional post hoc Hierarchical Multiple Linear Regression using Grade Point Average (GPA) as the outcome. The results from the Main Study advanced the outcomes from the Pilot Study by providing additional validation support for the TEOP scores in a population of high school students. The study, therefore, provided validity and reliability evidence of the TEOP for two subpopulations (i.e., Millennial college students and Generation Z high school students). The results also showed that while the TEOP scores did not have a significant predictive relationship with ACT scores, there was a significant relationship between the TEOP scores and GPA. These findings may be useful to various stakeholders looking to address students’ test preparation and academic performance. The TEOP scores can be used to inform students, parents, and school administrators of the potential congruence or incongruence between students’ practices and preferences while studying and the actual test environments in order to maximize their performance. Results from this correlational study should be interpreted with caution; however, future research may consider how to use the TEOP as a tool for in-class assessment preparation. ACKNOWLEDGEMENTS This document is dedicated to everyone who has ever helped me, there are too many of you to name. “Those five years are going to go by whether you finish or not” – Msgr. Robert Siffrin “Those five years are going to go by whether you finish or not” are the words Msgr. Robert Siffrin told me when I mentioned to him that I was starting my PhD and that it would take approximately five years. Well it has been six months longer than my original estimate, but I have frequently repeated this simple statement as the responsibilities of completing my classes, comprehensive exams, and this dissertation while trying to fulfill my obligations to my job and family at times seemed overwhelming. I thank you for this inspiration you gave in a moment that you most likely have long forgotten. To my parents and sister, I want to thank you for a lifetime of love and support. I hope this makes up for me not getting in National Honor Society :). I am eternally grateful to my lovely wife, Sara, for her constant support and sacrifice over these last five years. In particular, thank you for fixing all my formatting SNAFU’s and my whining about why Word just won’t work. I love you. Thank you to all my teachers, the good and the bad. I have learned from all of you, but especially Matt Giambattista, Sr. Regina Rogers, and Sharon Machuga. Miss iv Machuga teaching me that the fish < eats the larger number has been particularly helpful as I matriculated through the Evaluation and Measurement program. To my committee members, Dr. Jason Schenker and Dr. Anthony Vander Horst and Graduate Faculty Representative Dr. Richard Ferdig, thank you. Thank you for putting in the time and effort and thank you for adding your expertise to my dissertation. I want to wish an additional thank you to Dr. Schenker for how well you taught the Statistics and Quantitative Research classes that formed the foundation of my success in this program. Thank you to all of my Evaluation and Measurement classmates for your support, patience, and friendship. Please know that if I can do this so can all of you. I want to thank my direct supervisors, first Dr. Nicholas Wolsonovich and then Mary Fiala for their support and understanding. I also would like to thank whoever invented the synonym feature on Microsoft Word for allowing me to appear to have a much bigger vocabulary than I do. Words cannot express how appreciative I am of what my advisor Dr. Aryn Karpinski has done for me. You were tireless as a guide and mentor, and your demand for excellence made me a better person. In the Mount Rushmore of my life, Dr. Aryn Karpniski gets a sculpture. Lastly thank you to my dogs and two best friends, Gunner and Erie, thank you for always being happy to see me, thank you for reminding me when it was time to take a break, and I am sorry I scared you every time I yelled at the computer. v TABLE OF CONTENTS ACKNOWLEDGEMENTS ............................................................................................... iv LIST OF FIGURES ........................................................................................................... xi LIST OF TABLES ............................................................................................................ xii PART I CHAPTER I: INTRODUCTION (PILOT STUDY) ............................................................................... 1 Statement of Problem ........................................................................................................ 2 Standardized Testing and Tests ........................................................................................ 3 Millennials ........................................................................................................................ 6 Millennials, Learning, and Context-Dependent Memory .............................................. 7 Millennials and Testing Preferences. ............................................................................. 8 Organization of Part I: Pilot Study.................................................................................... 9 II: LITERATURE REVIEW (PILOT STUDY) ................................................................ 10 Millennial Learning ........................................................................................................ 10 Multitasking .................................................................................................................... 12 Differentiated Instruction ................................................................................................ 14 Test Performance Factors ............................................................................................... 16 Time of Test .................................................................................................................... 21 Context-Dependent Memory .......................................................................................... 22 Measurement of Preferences ........................................................................................... 24 III: PILOT STUDY METHODOLOGY ...........................................................................