1 Registered Measurement Report Proposal
Total Page:16
File Type:pdf, Size:1020Kb
1 Registered Measurement Report Proposal: A Large-Scale Test of the Replicability and Generalizability of Survey Measures in Close Relationship and Sexuality Science Stéphanie Gauvin Queen’s University Kathleen E. Merwin Dalhousie University Chelsea D. Kilimnik University of Texas at Austin Jessica Maxwell University of Auckland John Kitchener Sakaluk University of Victoria Author note: This research is being supported by a SSHRC Insight Development Grant awarded to Dr. Sakaluk, a SSHRC Postdoctoral Fellowship awarded to Dr. Maxwell, an Ontario Women’s Health Scholar Award funded by the Ontario Ministry for Health and Long Term Care awarded to Stéphanie Gauvin, a CIHR Doctoral Fellowship awarded to Chelsea Kilimnik, a Joseph-Armand Bombardier Canada Graduate Scholarship funded by SSHRC awarded to Kathleen Merwin, and a Student Research Development Award from the International Academy of Sex Research, Co-Awarded to Stéphanie Gauvin and Kathleen Merwin. 2 Abstract When measurement models are not replicable and/or generalizable, clinical assessments become of questionable utility, and unreplicable findings from studies using those measures will follow. Inspired by recent examinations of measurement in neighboring fields of psychology, we propose a Registered Report, in order to evaluate the replicability and generalizability of 20 well- known and emerging measures assessing elements of romantic relationships and sexuality. After collecting a large sample of that is both sexually and relationally diverse, we will evaluate the taxometric structure, measurement model replicability, reliability, and generalizability of each measure across a multitude of theorized sources of noninvariance. Our results are likely to be of high value to clinical researchers and practitioners alike, as we identify which measures can produce credible assessments, while simultaneously revealing measures with limited replicability and/or generalizability, as well as relational and sexual concepts for which groups may have radically different mental constructions. Key words: invariance; generalizability; measurement; relationships; replicability; sexuality 3 Registered Measurement Report Proposal: A Large-Scale Test of the Replicability and Generalizability of Survey Measures in Close Relationship and Sexuality Science Romantic and sexual relationships are critically important forms of social relationships in adolescence and adulthood, as they confer benefits onto one’s physical health (Holt-Lunstad et al., 2010), emotional well-being (Mikulincer & Shaver, 2007), and sense of happiness and meaning in life (Diener & Seligman, 2002; Muise, Schimmack, & Impett, 2015). It is therefore unsurprising that romantic and sexual relationships have become the topics of widespread psychological scholarship (see Impett & Muise, 2018; Maxwell & McNulty, 2019; Muise, Maxwell, & Impett, 2018), as well as targets of therapeutic intervention when they go awry (see Binik & Hall, 2014; Mitchell et al., 2013). A considerable amount of attention has been paid, in particular, to developing psychological assessments that measure experiences in romantic and sexual relationships (e.g., Milhausen, Sakaluk, Fisher, Davis, & Yarber, 2019), which has enabled close relationships and sexuality scholars to expand their topical breadth. Yet within social (e.g., Earp & Trafimow, 2015) and more recently clinical psychology (Tackett, Brandes, King, & Markon, 2018; Tackett et al., 2017), concerns about problematic methodological trends and the replicability of research findings are being borne out (e.g., Sakaluk, Williams, Kilshaw, & Rhyner, 2019), and the concerns from these “parent disciplines” have also trickled down into the interdisciplinary fields of close relationship research (e.g., Campbell, Loving, & LeBel, 2014; Joel, Eastwick, & Finkel, 2018) and sexual science (Sakaluk, 2016; Sakaluk & Graham, 2018). Rarely mentioned and vastly underappreciated is the importance of sound psychological measurement practices as a central contributor to replicable psychological findings (Flake & Fried, 2019; Sakaluk, 2019; Sakaluk & Fisher, 2019; Sakaluk, Kilshaw, & Fisher, 2019). As 4 Meehl and colleagues argued (1990a, 1990b), measurement is a crucial auxiliary theory for substantive theory testing in psychological research; if the auxiliary theory of measurement is in question, then doubt is cast upon the meaningfulness of whatever conclusions researchers might draw about the variables under study. A first effort to evaluate the soundness of fifteen popular measures in social and personality psychology (Hussey & Hughes, 2018), however, invites cause for concern, as only 60% of the measures demonstrated good validity, and only 33% demonstrated a replicable factor structure. With the present Registered Report, we propose conducting a similar measurement audit to Hussey and Hughes (2018), but specifically for assessments in close relationships and sexual science, using a larger selection of measures and a broader set of criteria for evaluating candidate measures. The Replication Crisis in Clinical, Relationship, and Sexual Science Situating Measurement in the Replication Crisis Psychologists are increasingly calling for greater attention to issues of measurement (e.g., Flake & Fried, 2019, Sakaluk, 2019, Sakaluk, Kilshaw, & Fisher, 2019). Although sometimes it goes unspoken where measurement features in the process of theory-testing, and why it is so critical to the meaningfulness of theory-testing, Meehl’s derivation of the observed conditional (1990a, 1990b) helps to articulate the place and importance of measurement more explicitly. Namely, widespread failures to replicate might come about not because of the usual methodological suspects (e.g., lack of transparency in original research, p-hacking, HARKing; Nelson et al., 2018), but rather, because the effect to be replicated is based upon a lacklustre theory of and/or lackadaisical approach to measurement (Flake & Fried, 2019), thereby undermining the logical chain necessary for substantive theory-testing. 5 Although psychologists might not explicitly conceptualize the process of factor analyzing a measure’s items as generating or testing a theory of measurement, established factor structures are indeed statements of measurement theory (Borsboom, 2005) that can be corroborated or falsified, and these measurement models can be replicated or fail to replicate. Traditionally, social scientists have adopted an attitude of “Measurement, Schmeasurement” (Flake & Fried, 2019); measurement is not an issue taken seriously, as measurement models are typically investigated once or twice and never again (Sakaluk, 2019). A recent measurement-focused replicability investigation, however, suggests this collective mindset may have been deleterious for the field of social and personality psychology. Using a colossal sample size (N = 151,698), Hussey and Hughes (2018) found that only 33% of the common measures they investigated possessed factor structures that replicated well, while only 66% had measurement models that were reasonably consistent across gender and age (i.e., factorial invariance), and only 27% possessed both a replicable measurement structure and good evidence of invariance. As the field of social psychology, in particular, continues to grapple with the mechanisms undermining the replicability of its effects, the spectre of measurement issues raised by Hussey and Hughes’s (2018) investigation presents a chilling contender. Measurement and Replicability in Clinical, Relationship, and Sexual Science Replicability conversations in clinical psychology and sexual science are in their relative infancy (e.g., Sakaluk, 2016; e.g, Tackett et al., 2017). The field of close relationship science, meanwhile, has contributed early and frequently to the discussion (e.g., Campbell et al., 2014). Still, despite differences in the visibility of replicability discussion, there is every reason to presume that measurement may be an equal (if not even more substantial) contributor or detractor to the replicability of research findings in these areas. 6 Clinical and counseling psychologists, for example, move through accredited training programs with packed curricula that leave little in the way of encouragement and/or opportunity for trainees to invest in elective statistical courses, such as in psychological measurement (Ord, Ripley, Hook, & Erspamer, 2016). Close relationship and sexual scientists meanwhile, are beset by the training burden of an interdisciplinary science, in which they must develop competencies in multiple disciplines’ concepts, theories, and methods (Sakaluk, 2019; Wiederman & Whitley Jr., 2001). And even should it be the case that clinical, close relationship, and sexual science are no worse for wear in terms of training in psychological measurement, it is likely that the use of psychological measures takes on a greater applied importance in these areas versus generalist psychological research. That is, clinicians working with clients presenting with relationship and/or sexual problems will depend heavily on the integrity of the assessment tools to identify client needs and track client progress—practices that will have a very real impact on the lives of those seeking clinical services. Together, the importance of the auxiliary theory of measurement (Meehl, 1990a), concerning results from Hussey and Hughes’s (2018) measurement replicability investigation in a neighboring field, and a variety of discipline-specific considerations, render a large-scale