Measurement in Health Psychology
Total Page:16
File Type:pdf, Size:1020Kb
Peters et al. measurement in health psychology Measurement in health psychology: combining theory, qualitative, and quantitative methods to do it right. 6th Methods in Health Psychology Symposium Gjalt-Jorn Ygram A recent debate in Health and quantitative traditions to provide a broad Peters Psychology Review overview of the state of the art, limitations of demonstrated the current practices, and options for improvement. Open University Alexandra Dima importance of careful Moreover, the symposium aimed to give its attention to measurement attendants practical suggestions to apply these University of Amsterdam Anne Marie Plass and operationalisation of insights, as well as facilitate access to their health psychology corresponding tools. Measure Mind Rik Crutzen constructs (Beauchamp, The symposium started with the presentation 2016; Brewer, 2016; de from Gjalt-Jorn Peters of a novel perspective on the Maastricht University Chris Gibbons Vries, 2016; Schwarzer & nature and inter-relations of psychological variables McAuley, 2016; Williams and implications for their measurement. This University of Cambridge Frank Doyle & Rhodes, 2016a, 2016b). perspective facilitates a flexible and theoretically This need is met by rapid promiscuous approach to operationalization and Royal College of Surgeons in developments in the measurement, affording researchers more flexibility Ireland theory and measurement in the development and assessment of measurement of health psychology constructs as evidenced by instruments. This was followed by the presentation recent publications and conference contributions of Anne Marie Plass introducing tools to explore (e.g. Dima et al., 2014). However, these enhanced and improve operationalization in questionnaire methods have been slow to disseminate into development or adaptation using Cognitive research practice. One reason may be that the new Interviewing. Several problems with common perspectives afforded by these developments and assumptions about validity were pointed out and the related tools were not part of the curricula of solutions provided for addressing these. Rik most researchers currently active in health Crutzen provided an overview of the current psychology. This lack of familiarity may manifest practices regarding assessment of the quality of itself as an obstacle that appears difficult to measurement instruments. Although these overcome, thereby obstructing wide-spread use of practices are strongly rooted in classical testing these methods in research. theory, important assumptions of the statistical The goal of the sixth Methods in Health models used were routinely violated. An accessible, Psychology symposium, held at the annual EHPS freely-available procedure for improvement was conference in Aberdeen in 2016, was to address introduced and explained. Alexandra Dima this by increasing attendees’ familiarity with demonstrated stepwise procedures that leverage several new developments in this field. The psychometric techniques to improve the symposium brought together five contributions, understanding and operationalization of combining theory and methods from qualitative psychological constructs. Chris Gibbons introduced Peters et al. measurement in health psychology computer adaptive testing using Concerto, an open aggregation level. Any operationalisation thus source system based on the flexible R and mySQL represents a number of implicit or explicit choices platforms, and discussed its benefits for health regarding these dimensions. psychology research. At the end of the symposium, These three tenets of pragmatic nihilism have Frank Doyle summarized the five previous two implications. First, they provide a foundation contributions and proposed several directions that enables combining theories in a more flexible regarding how these insights can be implemented manner than made possible by integrative theories. in practice to improve the standard of measurement Second, this perspective emphasizes the in health psychology. importance of operationalisations, underlining the The presentations and additional materials are importance of investing in the careful development available on the Open Science Framework through of measurement instruments, and thorough and links on the Health Psychology Methods page on extensive reporting of the specifics and the EHPS website at http://ehps.net/content/health- performance on those measurement instruments as psychology-methods. These materials are available well as disclosure of the instruments themselves. under the Creative Commons Attribution license, Awareness of the dimensions of the tesseract, of unless indicated otherwise. Below, each which each operationalization represents a slice, contribution is briefly summarized from the can aid researchers in scrutinizing the exact items perspective of this symposium. (elements) of both newly developed operationalisations and operationalisations that have been in use for decades. For example, when Pragmatic Nihilism using questionnaires, it is important to pay close attention to the questions used. A very easy, fast, Gjalt-Jorn Ygram Peters and affordable method of identifying potential Health psychology aims to explain and change a problems related to item content and interpretation wide variety of behaviours, and to this end has was provided by Plass in the following talk developed a plethora of theories. Several attempts have been undertaken to build integrative theories, and some even strive for a Theory of Everything Valid or not valid that’s the (also see Peters & Kok, 2016). We argue against question: the limited validity of these efforts; instead, adopting a stance that may ‘proven valid’ measurement be called ‘pragmatic nihilism’ is more fruitful. The first tenet of pragmatic nihilism is that instruments psychological variables, defined in our health Anne Marie Plass psychology theories, are usefully considered as The results of the 2015 landmark study of Nosek metaphors rather than referring to entities that and colleagues suggested that the vast majority of exist in the mind. As a consequence, the second recent psychology studies cannot be replicated, and tenet emphasizes theories’ definitions and it thus became clear that evidence for the most guidelines for the operationalisation of those published findings is not as strong as claimed variables. The third tenet of pragmatic nihilism is (Open Science Collaboration, 2015). It was argued that each operationalisation represents a cross- that replication bias might be due to the different section of a variety of dimensions, such as research methods used, publication bias, or the so- behavioural specificity and duration of the called ‘statistical jackpot’, which indicates that a behaviour, and most importantly, psychological Peters et al. measurement in health psychology study result may be sheer luck, or the result of when a measurement instrument is validated is in endlessly trying various analytic methods until fact largely unsatisfactory. Apart from this, the something pans out. Yet, the quality of the third and utmost critical condition for validity: measurement instruments, used in many social verifying the interpretation of the items for a given scientific studies, was never questionned in target population, is even a largely unknown step, relation to this. Whereas, almost every individual and extremely rare. However, if the elements of a that ever completed a questionnaire has measurement instrument are interpreted differently experienced the unclear nature of this task, giving by a sample than what was intended when the answers to questions that were difficult to instrument was developed, none of the previously understand. A large body of evidence demonstrates gathered data and indicators of validity and that items researchers thought to be perfectly clear reliability still apply. Thus, none of the three are often vague and hard to understand (Markhous, necessary conditions for construct validity are Siksma, & Plass, 2014; Van Kessel, Hendriks, van being met. Yet, we draw our conclusions based on der Hoek, & Plass, 2015; Willis & Artino, 2013). We these data. hardly know how our respondents interpret and One way to establish (better) content validity, understand our questions. and at an earlier stage, is through applying Researchers often make use of existing cognitive testing (Holch et al., 2016; Markhous et measurement instruments that have proven valid al., 2014; Willis, 2005; Willis & Artino, 2013). through the statistical testing of its psychometric Cognitive interviewing involves the study of how qualities. While this seems an excellent approach at survey questions are interpreted, how information first glance, there are serious risks that are being is recalled, and how respondents make decisions to overlooked, in particular regarding the validity provide a particular response. Cognitive assumed. Validity is the extent to which a interviewing is conducted using two key procedures measurement-instrument (scale, or questionnaire) that are combined: 1. Think Aloud, requesting the measures what it claims to measure. There are survey respondents to actively verbalize their three conditions to achieve adequate conceptual thoughts as they attempt to answer the survey coverage of the relevant construct. First, every questions (Willis, 2005; Willis & Artino, 2013), element of a measurement instrument must revealing how they interpret and understand the measure a part of the construct as defined by the questions and answer options, and