Comparative Review and Assessment of Inductive and Deductive Psychometrics
Total Page：16
File Type：pdf, Size：1020Kb
Markers of Adulthood Subscale Development 1 Markers of Adulthood Subscale Development: Comparative Review and Assessment of Inductive and Deductive Psychometrics Nathan E. Fosse1 Harvard University Jon E. Grahe Alan Reifman March 10, 2015 1 Direct all correspondence to Nathan E. Fosse at [email protected]. Please cite as follows: Fosse, N., Grahe, J. E., & Reifman, A. (2015, March 31). Markers of Adulthood Subscale Development: Comparative Review and Assessment of Inductive and Deductive Psychometrics. Retrieved from https://osf.io/p8nwq/ Research Report for EAMMI Survey Project, Open Science Framework (https://osf.io/p8nwq/) Markers of Adulthood Subscale Development 2 Executive Summary In this report we outline two psychometric approaches to develop subscales from the 40 Markers of Adulthood items in the Emerging Adulthood at Multiple Institutions (EAMMI) survey. First we describe the inductive approach, drawing from empirical analyses of 20 of the items. Second we describe the deductive approach toward scale construction. In addition to describing the processes of scale construction, we also describe the attributes and psychometric properties of the scales derived. Finally, we assess the inter-construct validity of inductive vis-a-vis deductive approaches, finding high correlations between the two sets of measures. In addition, we draw from use of the 40 MoA items in the extant literature to provide a comparative analysis of the EAMMI items. We find that, despite divergent 'styles' of psychometric reasoning, both approaches are consistent with the strengths and he challenges of constructing scales in the literature. We conclude with a review of the lessons learned. The appendix includes details of the PCA analyses run by the first author, describing the findings in greater detail. Keywords: Markers of Adulthood, emerging adulthood, deductive reasoning, psychometrics, principal components analysis. Research Report for EAMMI Survey Project, Open Science Framework (https://osf.io/p8nwq/) Markers of Adulthood Subscale Development 3 Markers of Adulthood Subscales in the EAMMI Survey In this methodological supplement we describe two psychometric approaches we had undertaken in constructs to construct scales from derived from 40 Likert-type Markers of Adulthood (MoA) items in the survey. We undertook two approaches toward the data, comparing findings, assessing measured by internal reliability (i.e. by the consistency or homogeneity of items comprising subscales) and theoretical coherence (the degree to which subscales make theoretical sense given current understanding of emerging adulthood). We adopted a sample-based, bottom-down, deductive approach as well as a literature-based, top-up inductive approach toward scale measurement. We describe the attributes and the internal validity of each subscale derived, and we compare both approaches empirically and conceptually. We find that both approaches to capture the most important aspects in the MoA items and see future benefits in collaborative psychometrics. While each item assesses the responses of participants to 20 adulthood markers, and may be treated as single indicators, we recognize the potential advantages of advancing emerging adulthood research through dedicated investigation of theoretically-relevant and empirically- valid scales. We undertook different approaches toward generating MoA scales in part out of necessity (that is, the need to produce parsimonious and communicable findings) but also because there is not widespread consensus in constructing scales from the MoA items. The present analyses and description of approaches describe features of the EAMMI data, providing important supplemental analyses of the sample structure. In addition, the present analyses provide some templates and guidelines for future research using similar items. Research Report for EAMMI Survey Project, Open Science Framework (https://osf.io/p8nwq/) Markers of Adulthood Subscale Development 4 The analysis is present in sections that follow. First we provide a synopsis of the EAMMI items. In the next two sections, we describe the data and the two approaches we took in constructing MoA subscales, based in the psychometric literature: inductive approaches, which begin with sample-based data analyses to inform theoretically-meaningful scales, and; deductive approaches, which begin with prior literature and conceptual distinctions and only secondarily to the sample characteristics. The former approach can be considered bottom-up, as it focuses on the data and analytics, while the latter is top-down drawing from the literature to generate informative priors to generate scales. In practice, however, both approaches take into consideration both prior knowledge and the findings at hand.2 We describe the process of scale construction, along with the internal reliability and descriptive statistics of the measures. Comparing both approaches based on comparisons of internal subscale reliability and of conceptual or theoretical coherence, we conclude similarity in the constructs derived. Fourth, we compare each of the subscales we generated with each other empirically, and with respect to prior analyses selected by N. Fosse for their conceptual advances. Based on both empirical and theoretical comparisons, we conclude that the constructs derived match in broad conceptual ways, but we also see promise in advancing he emerging adulthood literature in collaborations that address the following points: (1) outlining in detail the approaches toward scale construction benefit from collaboration, particularly across social scientists with different “styles of thought” (Hacking, 1994); (2) theory and methodological orientations reveal how scale construction can produce different forms of knowledge, particularly without consensus-making among collaborators through sharing syntax code, and analytic decisions; (3) theory-building is 2 A third approach common in psychometrics, the criterion approach, assumes a priori the important criteria for condition or participant classification, and develops scales based on their prediction of this external standard. In contrast, both approaches we describe are internal, as they are based on the properties of the data itself. Research Report for EAMMI Survey Project, Open Science Framework (https://osf.io/p8nwq/) Markers of Adulthood Subscale Development 5 intrinsic to the practice of psychometrics. In the emerging adulthood literature, there is much scientific knowledge to be gained from further investigation of emerging adulthood(s), particularly as this life stage expands and transforms across global society. As a psychological and cultural life stage (Arnett, 2011), it will continue to be crucial to understand the object of inquiry as we also seek to understand its variation across social milieu. Synopsis of EAMMI Markers of Adulthood Items The EAMMI survey contains 40 questionnaire items assessing how important each of 20 Markers of Adulthood (MoA) are important in achieving adulthood and whether the respondents achieved these adulthood identifiers. All items and assessing each of the 20 MoAs were derived from Arnett’s analyses (1997, 2001). This analysis concerns 40 items covering two conceptual domains. First, 20 measure participants’ perceived importance (1 = very important, 4 = not important) of obtaining each MoA. Second, 20 items measured achievement of each MoA (dichotomous, yes or no; or ordinal, from 1 = very true, 3 = not true). Also included in the EAMMI survey is single item measuring whether a respondent felt he or she had reached adulthood (yes, no, somewhat). This factor variable was recoded so that higher scores represented fuller attainment of adulthood. As a global measure of attainment unrelated to any particular MoA, this item was not included in subscale constructions. Following the conceptual distinction adopted in psychometrics (Burisch, 1984; Smith, Fischer, & Fister, 2003), we characterize the subscales we generated from the items the result of two different scale construction processes: inductive and deductive approaches. Inductive approaches rely on information from the data itself, i.e., the correlations or covariances between Research Report for EAMMI Survey Project, Open Science Framework (https://osf.io/p8nwq/) Markers of Adulthood Subscale Development 6 items, to reduce data empirically-meaningful clusters. Theoretical coherence is assessed during the process of analysis, based on both the interpretability of findings but also on assessments in light of prior literature. In contrast, deductive approaches or orientations generate constructs based on exiting literature and theory first, and test internal validity based on those a priori theories. We proceeded as follows. First, J. Grahe computed a principal components analysis (PCA) on the 20 importance items, presenting these subscales to potential authors prior to the invitation to participate in the Special Issue. To incorporate both importance and attainment items, N. Fosse adopted a ‘top-down’ deductive approach, presenting to other authors memoranda and soliciting feedback from corresponding authors. N. Fosse revised subscales based on further review of prior literature, for conceptual clarity, and for communicability. Other authors were invited to create their own subscales to share or to revise these constructs. We describe these two approaches in the following sections, respectively. Inductive Approach (Sample-based, Empirical) The primary advantage of inductive approaches, or