A Practical Guide to Instrument Development and Score Validation in the Social Sciences: the MEASURE Approach

Practical Assessment, Research, and Evaluation Volume 26 Article 1 2021 A Practical Guide to Instrument Development and Score Validation in the Social Sciences: The MEASURE Approach Michael T. Kalkbrenner New Mexico State University - Main Campus Follow this and additional works at: https://scholarworks.umass.edu/pare Recommended Citation Kalkbrenner, Michael T. (2021) "A Practical Guide to Instrument Development and Score Validation in the Social Sciences: The MEASURE Approach," Practical Assessment, Research, and Evaluation: Vol. 26 , Article 1. DOI: https://doi.org/10.7275/svg4-e671 Available at: https://scholarworks.umass.edu/pare/vol26/iss1/1 This Article is brought to you for free and open access by ScholarWorks@UMass Amherst. It has been accepted for inclusion in Practical Assessment, Research, and Evaluation by an authorized editor of ScholarWorks@UMass Amherst. For more information, please contact [email protected]. A Practical Guide to Instrument Development and Score Validation in the Social Sciences: The MEASURE Approach Cover Page Footnote Special thank you to my colleague and friend, Ryan Flinn, for his consultation and assistance with copyediting. This article is available in Practical Assessment, Research, and Evaluation: https://scholarworks.umass.edu/pare/ vol26/iss1/1 Kalkbrenner: The MEASURE Approach A peer-reviewed electronic journal. Copyright is retained by the first or sole author, who grants right of first publication to Practical Assessment, Research & Evaluation. Permission is granted to distribute this article for nonprofit, educational purposes if it is copied in its entirety and the journal is credited. PARE has the right to authorize third party reproduction of this article in print, electronic and database forms. Volume 26 Number 1, January 2021 ISSN 1531-7714 A Practical Guide to Instrument Development and Score Validation in the Social Sciences: The MEASURE Approach Michael T. Kalkbrenner, New Mexico State University The research and practice of social scientists who work in a myriad of different specialty areas involve developing and validating scores on instruments as well as evaluating the psychometric properties of existing instrumentation for use with research participants. In this article, the author introduces The MEASURE Approach to instrument development, an acronym of seven empirically supported steps for instrument development, and initial score validation that he developed based on the recommendations of leading psychometric researchers and based on his own extensive background in instrument development. Implications for how The MEASURE Approach has utility for enhancing the assessment literacy of social scientists who work in a variety of different specialty areas are discussed. Introduction are already financially burdened by the cost of required textbooks. Similarly, applied social scientists (e.g., Assessment literacy is a pertinent issue in social counselors and teachers) who are not affiliated with a sciences research, as researchers tend to assess latent university that provides access to electronic data bases variables (e.g., personality, morale, and other attitudinal might also have limited access to these resources. variables) that are abstract in nature, which are generally appraised by inventories (Gregory, 2016). To The literature is lacking a single refereed journal this end, social science researchers and practitioners article (a one-stop-shop) that includes a practical are responsible for understanding the basic outline of the instrument development and validation foundations, operations, and applications of testing, of scores process based on a number of synthesized including instrument development (Standards for recommendations of prominent expert, contemporary Educational and Psychological Testing, 2014). psychometric researchers. Such an article has potential Instrumentation with strong psychometric support, to provide social scientists with a single and accessible however, tends to be underutilized by social scientists resource for developing their own measures as well as when conducting program evaluation and other types for evaluating the rigor of existing measures for use of research (Tate et al., 2014). The extant literature with research participants. The primary aim of the includes a series of peer-reviewed journal articles (e.g., present author was to introduce The MEASURE Benson, 1998; Kane, 1992; Mvududu & Sink, 2013), as Approach to instrument development. MEASURE well as textbooks and book chapters (e.g., Bandalos & (Figure 1) is an acronym comprised of the first letter of Finney, 2019; DeVellis, 2016; Dimitrov, 2012; Fowler, the following seven empirically supported steps for 2014; Gregory, 2016; Kane & Bridgeman, 2017), which developing and validating scores on measures: (a) collectively outline the instrument development Make the purpose and rationale clear, (b) Establish process as well as guidelines for testing the validity of empirical framework, (c) Articulate theoretical inferences from the scores. However purchasing these blueprint, (d) Synthesize content and scale resources is infeasible for many graduate students, who development, (e) Use expert reviewers, (f) Recruit participants, and (g) Evaluate validity and reliability. Published by ScholarWorks@UMass Amherst, 2021 1 Practical Assessment, Research, and Evaluation, Vol. 26 [2021], Art. 1 Practical Assessment, Research & Evaluation, Vol 26 No 1 Page 2 Kalkbrenner, The MEASURE Approach The MEASURE Approach was developed based on An instrument development study might also be the guidelines of leading psychometricians, primarily necessary if a researcher determines that an existing Benson (1998), DeVellis (2016), Dimitrov (2012), and instrument is inappropriate for use with their target Mvududu and Sink (2013), as well as the author’s population (e.g., cross-cultural fairness issues). In some extensive background and experience with instrument instances, developing an original measure with a development and score validation. Finally, an exemplar diverse population can be more appropriate than description of an instrument development study confirming scores on an established measure (step 7) conducted by Kalkbrenner and Gormley (2020) is that was developed and normed with a different presented to provide an example of applying each step population. Suppose for example, a researcher is in The MEASURE Approach. seeking a screening tool for appraising mental health Figure 1. The MEASURE Approach to Instrument distress among Spanish speaking clients. There might be utility in creating a new screening tool (based on the Development culture) rather than trying to validate scores on an Make the purpose and rationale clear existing measure with Spanish speaking clients, as the Establish empirical framework nature and breadth of the construct of measurement (content validity, step 2) can vary substantially between Articulate theoretical blueprint different cultures. Thus, even if an existing measure of Synthesize content and scale development mental health distress is found to be statistically sound with Spanish speaking clients, it might fail to capture Use expert reviewers unique elements of mental health distress in the culture Recruit participants (see Kane, 2010, for an overview of fairness-related considerations in testing and assessment). After Evaluate validity and reliability making the purpose clear, a researcher should provide a rationale to justify why creating a new instrument is necessary. Step 1: Make the Purpose and When providing a rationale for developing a new Rationale Clear instrument, researchers should (a) present a summary Researchers should first define the purpose of of their review of the extant measurement literature, conducting an instrument development study by telling (b) cite any similar instruments that already exist, and the reader what they are seeking to measure and why (c) articulate the construct(s) that existing measures fail measuring the proposed construct is important to capture in order to highlight a gap in the existing (DeVellis, 2016; Dimitrov, 2012). As part of this step, measurement literature (DeVellis, 2016; Dimitrov, researchers should review the existing literature on the 2012). Finally, test developers should discuss how their proposed construct of measurement to determine if proposed instrument has potential to fill the they can use/adapt an existing measure or if an aforementioned gap in the measurement literature and instrument development study is necessary (Mvududu articulate how filling this gap has significant potential & Sink, 2013). If a measure exists in the literature, to advance future research and practice (see Fu and researchers should carefully evaluate the rigor of the Zhang, 2019, as well as Kalkbrenner and Gormley, instrument development study by comparing the 2020, for examples of providing a rationale for procedures that the test developers employed to instrument development based on these steps). established empirical standards (e.g., The MEASURE Approach). An instrument development study is necessary if the literature is lacking a measure to appraise the researcher’s desired construct of measurement. An instrument development study might also be necessary if a researcher determines that an existing instrument is potentially psychometrically flawed (e.g., lacking reliability or validity evidence, step 7). https://scholarworks.umass.edu/pare/vol26/iss1/1 DOI: https://doi.org/10.7275/svg4-e671 2 Kalkbrenner:

Load more