Reasoning with Pseudowords: How Properties of Novel Verbal Stimuli Influence Item Difficulty and Linguistic-Group Score Differences on Cognitive Ability Assessments
Total Page:16
File Type:pdf, Size:1020Kb
City University of New York (CUNY) CUNY Academic Works All Dissertations, Theses, and Capstone Projects Dissertations, Theses, and Capstone Projects 2-2018 Reasoning with Pseudowords: How Properties of Novel Verbal Stimuli Influence Item Difficulty and Linguistic-Group Score Differences on Cognitive Ability Assessments Paul Agnello The Graduate Center, City University of New York How does access to this work benefit ou?y Let us know! More information about this work at: https://academicworks.cuny.edu/gc_etds/2505 Discover additional works at: https://academicworks.cuny.edu This work is made publicly available by the City University of New York (CUNY). Contact: [email protected] REASONING WITH PSEUDOWORDS: HOW PROPERTIES OF NOVEL VERBAL STIMULI INFLUENCE ITEM DIFFICULTY AND LINGUISTIC-GROUP SCORE DIFFERENCES ON COGNITIVE ABILITY ASSESSMENTS by PAUL AGNELLO A dissertation submitted to the Graduate Faculty of Psychology in partial fulfillment of the requirements of the degree of Doctor of Philosophy, The City University of New York 2018 ii © 2018 Paul Agnello All Rights Reserved iii REASONING WITH PSEUDOWORDS: HOW PROPERTIES OF NOVEL VERBAL STIMULI INFLUENCE ITEM DIFFICULTY AND LINGUISTIC-GROUP SCORE DIFFERENCES ON COGNITIVE ABILITY ASSESSMENTS by PAUL AGNELLO This manuscript has been read and accepted for the Graduate Faculty in Psychology to satisfy the dissertation requirement for the degree of Doctor of Philosophy. Charles A. Scherbaum__________________________ _____________________ ____________________________________________ Date Chair of Examining Committee Richard Bodnar_______________________________ _____________________ ____________________________________________ Date Executive Officer _Charles A. Scherbaum______________________________ _Harold Goldstein__________________________________ _Erin Eatough_____________________________________ _Logan Watts______________________________________ _Kenneth Yusko____________________________________ Supervisory Committee THE CITY UNIVERSITY OF NEW YORK iv Abstract REASONING WITH PSEUDOWORDS: HOW PROPERTIES OF NOVEL VERBAL STIMULI INFLUENCE ITEM DIFFICULTY AND LINGUISTIC-GROUP SCORE DIFFERENCES ON COGNITIVE ABILITY ASSESSMENTS by Paul Agnello Advisor: Dr. Charles A. Scherbaum Pseudowords (words that are not real but resemble real words in a language) have been used increasingly as a technique to reduce contamination due to construct-irrelevant variance in assessments of verbal fluid reasoning (Gf). However, despite pseudowords being researched heavily in other psychology sub-disciplines, they have received little attention in cognitive ability testing contexts. Thus, there has been an assumption that all pseudowords work equally and work equally well for all test-takers. The current research examined three objectives with the first being whether changes to the pseudoword properties of length and wordlikeness (how much a pseudoword resembles a typical or common word in English) led to changes in item difficulty on verbal Gf items. The second objective was whether boundary conditions existed such that changes to pseudoword properties would differentially impact two linguistic sub-groups of participants – those who have English as their dominant language and those who do not have English as their dominant language. The last objective was to index and explore performance on these verbal Gf items when pseudowords were replaced with real words. Hypotheses predicting how pseudoword properties influenced item difficulty, how stimulie type – pseudoword or real word, impacted performance across linguistic sub-groups, and how linguistic sub-group status interacted with pseudoword properties were tested. Four sets of pseudowords were developed – short and wordlike, long and wordlike, short and un-wordlike, and long and un-wordlike, as well v as two sets of real words – short and wordlike, and long and un-wordlike. Sixteen verbal Gf items, adapted from the LSAT, were developed to accommodate the pseudowords or real words and explore these three objectives. While none of the hypotheses were statistically significant, the results did indicate further areas of exploration. Specifically, verbal Gf items were easier when they featured longer pseudowords and more difficult when they featured un-wordlike pseudowords. Additionally, while performance of English-non-dominant participants was fairly balanced across real and pseudoword sets, English-dominant participants performed better on items featuring real words. Similarly, linguistic status interacted with wordlikeness such that English-dominant participants featured a decrease in performance as pseudowords moved from wordlike to un-wordlike. A full discussion of the findings, their implications, limitations of the current study, and directions for future research are included. vi Acknowledgements There are numerous individuals to whom I am grateful for helping me complete this research. First and foremost, none of this would be possible without my advisor, Charles Scherbaum who was a constant source of guidance, patience, optimism, and camaraderie throughout the entire graduate school process. I would also like to thank Harold Goldstein and Erin Eatough for serving on my committee as well as Logan Watts and Kenneth Yusko for serving as outside readers. Due to your questions, comments, and contributions, this research was greatly refined and helped create a perfectly cromulent dissertation. This work also would not have been possible without my cohort, fellow graduate students, family, and friends who always provided support and much-needed relief from the grind of completing a dissertation. Lastly, I would like to thank my Mom and Dad for everything they provided for me leading up to and through graduate school. Their enthusiasm throughout my efforts was a perpetual source of motivation. vii Table of Contents Chapter 1: Introduction ................................................................................................................... 1 Chapter 2: Modern Cognitive Ability Tests.................................................................................. 14 Chapter 3: On Fluid Intelligence, Working Memory, and Pseudowords ..................................... 22 Chapter 4: Pseudoword Properties and the Implication for Item Properties ................................ 38 Chapter 5: Bilingualism ................................................................................................................ 49 Chapter 6: Pilot Study One ........................................................................................................... 72 Chapter 7: Pseudoword and Word Group Development .............................................................. 76 Chapter 8: Item Development ....................................................................................................... 85 Chapter 9: Pilot Study Two........................................................................................................... 91 Chapter 10: Methods ..................................................................................................................... 94 Chapter 11: Results ..................................................................................................................... 105 Chapter 12: Discussion ............................................................................................................... 113 Appendix A: Verbal Fluid Intelligence Items with Pseudowords .............................................. 125 Appendix B: Linguistic Background Measure ........................................................................... 129 Appendix C: Socioeconomic Status Measure ............................................................................. 130 Appendix D: Demographic Measure .......................................................................................... 131 References ................................................................................................................................... 145 viii List of Tables Table 1. Pilot Study One Condition Properties ........................................................................... 132 Table 2. Pilot Study One Pseudoword Properties ....................................................................... 133 Table 3. Short and Wordlike Pseudoword Properties ................................................................. 135 Table 4. Short and Un-wordlike Pseudoword Properties ........................................................... 136 Table 5. Long and Wordlike Pseudoword Properties ................................................................. 137 Table 6. Long and Un-wordlike Pseudoword Properties ............................................................ 138 Table 7. Short and Wordlike Word Properties ............................................................................ 139 Table 8. Long and Un-wordlike Word Properties ...................................................................... 140 Table 9. Pseudoword and Word Property Means by Stimuli Set ............................................... 141 Table 10. Pilot Study Two Item Stem and Item Difficulties ...................................................... 142 Table 11. Mean Total Score, Vocabulary Score, and Socioeconomic Status (SES) by Stimuli Set .................................................................................................................. 143 Table 12. Mean Item and Total Scores Overall and by Linguistic Sub-group