The Development and Validation of an Automatic-Item Generation Measure
Total Page:16
File Type:pdf, Size:1020Kb
Louisiana Tech University Louisiana Tech Digital Commons Doctoral Dissertations Graduate School Spring 2017 The development and validation of an automatic- item generation measure of cognitive ability Hines Scott Follow this and additional works at: https://digitalcommons.latech.edu/dissertations Part of the Industrial and Organizational Psychology Commons, and the Organization Development Commons THE DEVELOPMENT AND VALIDATION OF AN AUTOMATIC-ITEM- GENERATION MEASURE OF COGNITIVE ABILITY by Scott Hines, M.A., B.A. A Dissertation Presented in Partial Fulfillment Of the Requirements for the Degree Doctor of Philosophy COLLEGE OF EDUCATION LOUISIANA TECH UNIVERSITY May 2017 ProQuest Number: 10612791 All rights reserved INFORMATION TO ALL USERS The quality of this reproduction is dependent upon the quality of the copy submitted. In the unlikely event that the author did not send a complete manuscript and there are missing pages, these will be noted. Also, if material had to be removed, a note will indicate the deletion. ProQuestQue ProQuest 10612791 Published by ProQuest LLC(2017). Copyright of the Dissertation is held by the Author. All rights reserved. This work is protected against unauthorized copying under Title 17, United States Code. Microform Edition © ProQuest LLC. ProQuest LLC 789 East Eisenhower Parkway P.O. Box 1346 Ann Arbor, Ml 48106-1346 LOUISIANA TECH UNIVERSITY THE GRADUATE SCHOOL FEBRUARY 22,2017 Date We hereby recommend that the dissertation prepared under our supervision by Scott Hines___________________________________________________________________ entitled_______________________________________________________________________ THE DEVELOPMENT AND VALIDATION OF AN AUTOMATIC-ITEM-_______ GENERATION MEASURE OF COGNITIVE ABILITY_________________________ be accepted in partial fulfillment of the requirements for the Degree of Doctor of Philosophy_________________________________________________________ Supervisor o f Dissertation Research Head of Department Psychology and Behavioral Sciences Department Recommendation concurred in: Advisory Committee Approved: Approved: ean o f the Graduate SchoolDirector o f Graduate Studies A ean o f the Graduate SchoolDirector Dean of the College ABSTRACT Cognitive ability is perhaps the most studied individual difference available to researchers, being measured quickly and effectively while demonstrating a predictable influence on many life outcomes. Historically, the evolution of the psychometric study of cognitive abilities has pivoted on the development of new and better methodologies allowing for a more complete and efficient capture of intellect. For instance, recent advances in computer and Internet technology have largely replaced traditional pencil-and-paper methods, allowing for innovative item development and presentation. However, concerns regarding the potential adverse impact and test security of online measures of cognitive ability, particularly in unproctored situations, are well documented and have limited the use of such measures in organizational settings. Methods, such as the use of multiple test forms and computer adaptive testing coupled with item exposure algorithms, have addressed some test-security concerns. However, these methods require the costly and tedious development of extensive item pools. The burgeoning area of automatic item generation potentially addresses many of the test-security and item-development concerns through the creation of assessment items based solely on an item model and a computer algorithm. Moreover, once the elements that contribute to item difficulty are calibrated, the psychometric properties of the items are known, meaning that little to no human review of the items is required before their use. The purpose of the current study was to develop an experimental non-verbal measure of cognitive ability through automatic item generation, using an innovative item type. Using a sample of 333 adults, the results of the current analysis support the proposed cognitive model’s ability to explain item difficulty. Likewise, the temporal stability and predictive validity of the experimental measure are supported. In doing so, the experimental measure answers some of the test-security and item-generation concerns that are associated with the development and administration of cognitive-ability measures in organizational settings. APPROVAL FOR SCHOLARLY DISSEMINATION The author grants to the Prescott Memorial Library of Louisiana Tech University the right to reproduce, by appropriate methods, upon request, any or all portions o f this Dissertation. It is understood that “proper request” consists of the agreement, on the part of the requesting party, that said reproduction is for his personal use and that subsequent reproduction will not occur without written approval o f the author of this Dissertation. Further, any portions of the Dissertation used in books, papers, and other works must be appropriately referenced to this Dissertation. Finally, the author o f this Dissertation reserves the right to publish freely, in the literature, at any time, any or all portions of this Dissertation. Author Date 2 ~ Z Z GS Form 14 (5/03) TABLE OF CONTENTS ABSTRACT .............................................................................................................................. iii LIST OF TABLES........................................................................................................................ xi LIST OF FIGURES.....................................................................................................................xii CHAPTER ONE INTRODUCTION........................................................................................... 1 Cognitive Ability ......................................................................................................................... 6 Factor Analytic Theories ................................................................................................. 7 Two-Factor Theory.....................................................................................................8 Primary Mental Abilities ..........................................................................................11 Structure-of-Intellect .................................................................................................12 Gf-Gc.......................................................................................................................... 13 Cattell-Horn ............................................................................................................... 14 Three-Stratum Theory.............................................................................................. 15 Cattell-Hom-Carroll..................................................................................................16 Theoretical Approaches to Intelligence ....................................................................... 19 Successful Intelligence ............................................................................................. 19 Multiple Intelligences ...............................................................................................22 Planning, Attention, Simultaneous, and Successive ............................................24 The Nature of g and Gf .................................................................................................. 25 The Power of Intelligence ..............................................................................................29 vii Intelligence in the Workplace .......................................................................................32 Why Does Cognitive Ability Affect Performance? ...................................................34 Adverse Impact ...............................................................................................................35 The Measurement of Cognitive Ability ....................................................................................37 The History of Testing and Measurement .................................................................. 38 Darwin ....................................................................................................................... 38 Galton ........................................................................................................................ 39 Cattell.........................................................................................................................41 Binet ...........................................................................................................................42 Yerkes........................................................................................................................42 Cognitive Ability Testing in the Modern Era .............................................................43 Computer-Based Testing ............................................................................................................46 Test Security Specific to Technology-Based Testing ................................................48 Combating Test-Security Issues...................................................................................50 Automatic Item Generation ....................................................................................................... 53 Item Model Development ............................................................................................. 56 Item modeling ...........................................................................................................57