A VALIDATION OF A GAME-BASED ASSESSMENT FOR THE MEASUREMENT OF VOCATIONAL INTEREST A Thesis submitted to the faculty of San Francisco State University In partial fulfillment of the requirements for the Degree Master of Science In Psychology: Industrial/Organizational Psychology by Hope Elizabeth Wear San Francisco, California May 2018 Copyright by Hope Elizabeth Wear 2018 CERTIFICATION OF APPROVAL I certify that I have read A Validation of a Game-Based Assessment for the Measurement of Vocational Interest by Hope Elizabeth Wear, and that in my opinion this work meets the criteria for approving a thesis submitted in partial fulfillment of the requirement for the degree Master of Science in Psychology: Industrial/Organizational Psychology at San Francisco State University. Chris Wright, Ph.D. Professor A VALIDATION OF A GAME-BASED ASSESSMENT FOR THE MEASUREMENT OF VOCATIONAL INTEREST Hope Elizabeth Wear San Francisco, California 2018 Game-based assessments (GBAs) are a new type of technologically-based assessment tool which allow for traditional selection concepts to be measured from gameplay behaviors (e.g., completing levels by following game rules). GBAs use game elements to create an immersive environment which changes how assessments are traditionally measured but retains the psychometric properties within the game to assess a variety of knowledge, skills and abilities. In this study we examined the validity of a GBA for use as a measure of RIASEC vocational interests from Holland (1985). Participants played the GBA as well as completed traditional measures of RIASEC interests. We compared the scores from participants for congruence across the different measures using a multitrait-multimethod matrix (MTMM). The results from this study will be used to provide validation evidence for the use of GBA as a method to measure vocational interests and to extend the RIASEC literature to the area of technological assessments. ct representation of the content of this thesis. Date ACKNOWLEDGEMENTS First, I would like to thank my advisor, Dr. Diana Sanchez, for all her help and guidance throughout this project. I am grateful Dr. Sanchez came and joined us at San Francisco State and took on a thesis project in her first year as a faculty member. In addition, I would like to thank Dr. Chris Wright for all his helpful insights and lessons about selection and assessments throughout the program. I would also like to thank the team at Artie Shores, Lara, Liam and Kirsty, who made this project possible and for all their help throughout the process. I would also like to thank my family for all their support over the years which allowed me to get to this point. Finally, I would like to thank my cohort members who have been supportive and helpful throughout this program. It was truly great spending the last two years with you all. v TABLE OF CONTENTS List of Tables....................................................................................................................vii List of Figures..................................................................................................................viii Introduction......................................................................................................................... 1 Game-Based Assessments....................................................................................... 2 Psychometric Measures......................................................................................... 10 Cognitive Ability Tests.................................................................................... 10 Personality Tests........................ 12 Applicant Reactions.............................................................................................. 13 Method.............................................................................................................................. 18 Results............................................................................................................................... 23 Discussion.........................................................................................................................33 Reference..........................................................................................................................41 LIST OF TABLES Table Page 1. Elements of GBAs...................................................................................................2 2. Demographic Characteristics.................................................................................24 3. Correlations of Demographic and Work Outcomes..............................................26 4. MTMM Matrix for Vocational Interest Measures..................................................28 5. Congruence of RIASEC Codes............................................................................. 30 6. Congruence of Individual and Occupational Codes...............................................31 7. Congruence Correlations with Work Outcomes....................................................33 1 A Validation of a Game-Based Assessment for the Measurement of Vocational Interest Technology is changing the world of selection, hiring and assessment (Arthur, 2017; Chapman & Webster, 2003). The way HR practitioners view technological tools has shifted. In a 2016 survey, 75% of HR practitioners indicated they would consider using gamification or game-based assessment for screening and selection (Cut-E, 2017). Technology has also changed how assessments are conducted as computer and mobile based assessments increase in usage (Arthur, 2017; Lowman, 2016; Morelli, Mahan & Illingworth, 2014). Game-based methods (i.e., using games for training or evaluation) are one example of how technology is changing the nature of work. The gamification and game-based industry was predicted to be worth $5.5 billion by 2018 (Dale, 2014). Game- based assessment (GBA) in particular, is being used more by companies to evaluate a number of different aspects of the workplace, such as individual characteristics and job skills (Landers, 2014). GBAs are defined as evaluations that use game elements to create an immersive environment and allow the individual to interact with and demonstrate certain knowledge or skills (Landers, 2015). Using a game to evaluate an individual retains the psychometric properties (i.e. ability of the assessment to measure qualities such as personality characteristics) within the game to assess a variety of knowledge, skills and abilities (Montefiori, 2016). The increased usage of GBAs in the workforce increases the need for evidence that these new methods are valid and appropriate for such uses. As these assessments are new, validation evidence is needed or there could be legal repercussions for companies who use theses assessments for high-stakes sectors such as 2 such as selection. Currently the amount of research on the validity of GBAs in the workforce is limited and more research is needed (Arthur, 2017; Lowman, 2016; McCarthy et al., 2017). It is important to distinguish different methods that games, and game elements can be used for. Table 1 Elements o f GBAs Element Definition Rules Define how the game responds to the actions (or inaction) of the player Objectives Define what the player is meant to be striving for and can be more structured or vague States The environment which arises from the connections between rules and objects Note. Elements from (Mislevy et al, 2015) This is important because these methods function in different ways and research on the methods have conclusions which cannot be assumed to generalize to the different methods. For example, GBAs are distinct from gamification (Cut-E, 2017; Landers, 2015). A commonly cited definition of gamification is “the use of game design elements in nongame contexts” (Deterding, Sicart, Nacke, O’Hara, & Dixon, 2011, p. 1). While there are similarities between game-based assessment and gamification, game-based assessments are administered within the context of a game which has typically been designed to measure knowledge, skills, and abilities. (For a review of gamification see Hamari, Koivisto, & Sarsa, 2014). GBAs share components (Table 1) with traditional games such as rules, objectives, objects, and states. These shared elements are combined to create GBAs which are used by businesses to make decisions about selection, 3 development and hiring of potential employees. How these elements are developed to measure important characteristics needs to be validated to determine if the characteristics intended are in fact being measured. While GBAs have increased in popularity in the workplace, the research into the validity and reliability of these measures has not lead to conclusive evidence (Chamorro- Premuzic, Winsborough, Sherman, & Hogan, 2016; Kim & Shute, 2015; Lowman, 2016). Without clear evidence of validity, or lack thereof, the research has fallen behind the adoption in organizations. Due to the lack of conclusive evidence, it is important that more research be conducted to understand how GBAs can be used in the workplace. This study aims to add to the existing research literature by evaluating the use of GBA in a specific context (e.g. measurement of vocational interests). This research study, examined the validity of a GBA which was developed for screening employees and job applicants across a variety of characteristics including personality and cognitive ability. The current study is examined whether this GBA can also be used to assess vocational