1

Validation of a Novel Climate Change Denial Measure Using Item Response Theory

Mr. George Lorama*,, Dr. Mathew Linga, Mr. Andrew Heada, Dr. Edward J.R. Clarkeb

a Deakin University, Geelong, Australia, Misinformation Lab, School of Psychology b Federation University, Mount Helen, Australia, School of Health and Life Sciences

* Corresponding Author, Misinformation Lab, Deakin University, Locked Bag 20001,

Geelong VIC 3220, Australia.

Email address: [email protected]

Declarations of Interest: All authors declare that no conflict of interest exists.

Acknowledgements: This work was supported by Deakin University (4th Year Student

Research Funding; 4th Year Publication Award). 2

Abstract

Climate change denial persists despite overwhelming scientific consensus on the issue.

However, the rates of denial reported in the literature are inconsistent, potentially as a function of ad hoc measurement of denial. This further impacts on interpretability and integration of research. This study aims to create a standardised measure of climate change denial using Item Response Theory (IRT). The measure was created by pooling items from existing denial measures, and was administered to a U.S. sample recruited using Amazon

MTurk (N = 206). Participants responded to the prototype measure as well as being measured on a number of constructs that have been shown to correlate with climate change denial

(authoritarianism, social dominance orientation, mistrust in scientists, and conspiracist beliefs). Item characteristics were calculated using a 2-parameter IRT model. After screening out poorly discriminating and redundant items, the scale contained eight items.

Discrimination indices were high, ranging from 2.254 to 30.839, but item difficulties ranged from 0.437 to 1.167, capturing a relatively narrow band of climate change denial. Internal consistency was high, ω = .94. Moderate to strong correlations were found between the denial measure and the convergent measures. This measure is a novel and efficient approach to the measurement of climate change denial and includes highly discriminating items that could be used as screening tools. The limited range of item difficulties suggests that different forms of climate change denial may be closer together than previously thought. Future research directions include validating the measure in larger samples, and examining the predictive utility of the measure.

Keywords: Climate Change Denial, Measurement, Item Response Theory 3

Validation of a Novel Climate Change Denial Measure Using Item Response Theory

1. Introduction

1.1. Background

Despite overwhelming scientific consensus that climate change is occurring, and is due to human activity (Doran & Zimmerman, 2009; IPCC, 2018), there is still a proportion of the population who deny some or all of the elements of climate change (Capstick & Pidgeon,

2014; Leviston & Walker, 2012; Reser, Bradley, Glendon, Ellul, & Callaghan, 2012). This phenomenon is called ‘climate change denial’ or ‘climate scepticism’ and has been observed in many different contexts (e.g., Capstick & Pidgeon, 2014; Leviston & Walker, 2012; Reser et al., 2012). Public acceptance of the scientific evidence of climate change is important in the transition to a low carbon economy (Poortinga, Spence, Whitmarsh, Capstick, & Pidgeon,

2011), and therefore, climate change denial may have deleterious effects on climate change mitigation efforts.

1.2. Heterogenous rates of climate change denial

Varying rates of climate change denial have been measured in the literature. Leviston and Walker (2012) found that 17.2% of participants did not believe that climate change was occurring, with similar rates found by Capstick and Pidgeon (2014). This is in contrast with

Hornsey, Fielding, McStay, Reser, and Bradley (2016), who found that 4.2% of participants denied climate change, and Reser et al. (2012), who found denial rates of 6.5%. Other studies have found rates somewhere in between, such as Whitmarsh (2011), who found that 12% of participants agreed with the statement “Climate change is not a real problem”.

These studies were undertaken in comparable populations (i.e., Australia & U.K.), making the discrepancies in denial rates quite surprising. This may be a function of true population differences, measurement differences, or a combination of both. It is important to note that these studies used different scales to measure climate change denial. Scales differed 4 in terms of length, question framing, and response options. For example, Leviston & Walker

(2012) simply asked participants whether they believed climate change was happening, with a dichotomous yes/no response option, then asking what they thought was causing climate change, with four response options (e.g., “I think that climate change is happening, but it’s just a natural fluctuation in Earth’s temperatures”). This is in contrast with Reser et al.

(2012), who used four questions and a combination of different response options, such as yes/ no/don’t know, and Likert scales; and Whitmarsh (2011) who asked participants to rate 12 statements on a 5-point Likert scale. Other measures include questions that appear to be double-barrelled, for example, “Even if we do experience some consequences from climate change, we will be able to cope with them” (Capstick & Pidgeon, 2014). The way this particular question is framed asks the reader to imagine that climate change is real, which may cause inconsistent responding for those who deny climate change. Questions such as these may not be measuring the construct of climate change denial reliably.

1.3. Impact of question framing on responses

The measures mentioned in section 1.2 are a small sample of all existing climate change denial measures, with most studies either creating their own measure, or amending an existing one. This has led to a proliferation of climate change denial measures, and the absence of a standard measure may partly explain the differences in observed denial rates between studies. Greenhill, Leviston, Leonard, and Walker (2014) found that differences in question framing and response options affected belief responses. When asked about the causes of climate change, and given options including “both natural and anthropogenic”, the majority of people chose this option. However, when not given this option, participants were split down the middle, choosing either natural or anthropogenic. Differences such as this make it difficult to effectively compare denial rates between studies. Additionally, Leviston,

Leitch, Greenhill, Leonard, and Walker (2011) found that when given more nuanced options 5 for causation beliefs (e.g., “partly human and partly natural”, “mainly natural” etc.), participants were less likely to endorse ‘natural fluctuation’ than ‘human induced’ answers, compared with when given dichotomous response options (natural or human induced). These findings indicate that when measuring climate change denial, subtle variations in items and response options can have significant impacts on observed denial rates.

1.4. Importance of reliable measurement

It is important to have accurate and consistent measurement of public beliefs.

Governments, industries, and organisations observe public sentiment to assist decision- making on important issues such as climate change (Reser et al., 2012). Another important reason for consistent measurement is in gauging the efficacy of interventions that aim to reduce climate change denial. Research has shown mixed evidence for intervention effectiveness (e.g., Benegal & Scruggs, 2018; Hart & Nisbet, 2011; McCright, Charters,

Dentzman, & Dietz, 2016). However, as all of these studies measured denial differently, the variation in denial rates may be partly due to disparity of sensitivity between measures.

Related to this, the absence of a standard scale or measure makes it difficult to integrate research findings and compare interventions. Therefore, it is important to have a consistent, reliable measure of climate change denial in order to improve the ability to integrate and compare research findings. However, the test theory that a measure is built on can impact the adaptability, reliability, and efficiency of the measure.

1.5. Differences in test frameworks

Most psychological measures are founded on one of two dominant test theories:

Classical Test Theory (CTT) or Item Response Theory (IRT). CTT is the underpinning of existing denial measures, and is based on the assumption that an individual’s observed score is a combination of their true score plus some amount of error (de Champlain, 2010). The advantages of CTT are that it has relatively weak assumptions that are easily met, and is 6 designed in such a way that enables simple calculation of summary scores (de Champlain,

2010). However, CTT is a ‘test-dependent’ theory - that is, only the total score of the test can be interpreted, rather than responses to individual items (Bortolotti, Tezza, de Andrade,

Bornia, & de Sousa Júnior, 2013). Generally, responses to a number of items are summed, and the total score is compared to a pre-determined cut-off. The reliance on the total score means that long tests are often required in order to attain an acceptable level of reliability, and additionally, missing response data can be difficult to process. Furthermore, when calculating summary scores using CTT, each individual item is given equal weighting. For example, an item in a climate change denial measure might only be answered affirmatively by individuals who score in the top 5% of climate change denial, but the item would still be given the same weight as an item endorsed by 95% of the sample. This differential difficulty is useful information, but it is information that cannot be used within the CTT framework, impacting on the efficiency of CTT-based tests.

IRT encompasses a family of measurement models that attempt to explain the relationship between a person’s level on an underlying trait (e.g., climate change denial), and their response to a particular test item (de Champlain, 2010). IRT describes this relationship by assigning each test item a difficulty value, indicating the level of the underlying trait at which an individual is 50% likely to ‘endorse’ the question – that is, respond to the question in the affirmative (Fan, 1998). Because each test item provides information about that individual’s level of a given construct or trait, IRT-based tests can be short and efficient, and they allow for easier construction of parallel measures (Bortolotti et al., 2013). Furthermore,

IRT provides additional information regarding item discrimination, which is a measure of how well a test item can distinguish between different levels of the trait or construct being measured. Because both of these item parameters (difficulty and discrimination) are independent of the person characteristics of the sample, IRT-based measures can be 7 administered in different samples and the results meaningfully compared (Fan, 1998).

Additionally, the item parameters generated by IRT allow for adaptive testing (Fan, 1998).

For example, the first question of a test may place an individual within a broad range on the denial spectrum, with each subsequent question efficiently narrowing down their position. In the context of climate change denial, the use of IRT allows for different types of denial to be collapsed into a single continuum. This will allow for more effective comparison of denial rates between studies.

One of the main limitations of IRT is the complexity involved in computing item parameters (Chalmers, 2012). However, advances in computational power and increasing availability of appropriate software packages largely overcome this issue (Chalmers, 2012).

A further limitation is that IRT requires larger sample sizes than CTT to initially calibrate item parameters (Hambleton & Jones, 2005). However, the advent of online rapid recruitment platforms such as Amazon MTurk has provides an efficient and cost-effective method of obtaining large samples. (Buhrmester, Kwang, & Gosling, 2011). Therefore, despite the limitations outlined, the advantages of IRT make it preferable to CTT in the development of a new measure of climate change denial.

1.6. Aims and hypotheses

This study proposed to develop a new measure of climate change denial using IRT.

Using IRT will enable the development of a short and efficient measure, which could potentially be used in contexts such as phone surveys. Using IRT will also improve the ability to compare and integrate research findings. To create the measure, questions were drawn from existing measures. The measure was then administered to a sample in order to obtain difficulty and discrimination indices, and was validated against known correlates of climate change denial: authoritarianism (Häkkinen & Akrami, 2014), social dominance orientation

(Häkkinen & Akrami, 2014), mistrust in scientists (Hmielowski, Feldman, Myers, 8

Leiserowitz, & Maibach, 2014), and conspiracist beliefs (Lewandowsky, Oberauer, &

Gignac, 2013).

Given that the study is exploratory in nature, no specific hypotheses are given.

However, as the items were being drawn from existing climate change denial measures, it is expected that the new measure will have adequate construct validity, as evidenced by convergent validity with known correlates of climate change denial, and additionally, that it will have adequate internal consistency.

2. Methods

2.1. Participants

Two hundred and six participants were recruited using Amazon MTurk, an online rapid recruitment platform. The sample size was determined based on available budget for the study. All participants were U.S. residents, due to Amazon MTurk’s geographical restrictions. Participants were paid USD $2.80 upon completion of the survey. Characteristics of the sample are listed in Table 1, within the results section.

2.2. Materials

Participants completed an online survey that contained a number of scales, including the prototype climate change denial measure. The first part of the survey contained demographic questions, such as age, gender, location (on a scale ranging from major city to rural area), and education level.

Climate change denial was measured using the prototype measure under assessment.

The measure contained 18 items drawn from a number of existing measures (Capstick &

Pidgeon, 2014; Häkkinen & Akrami, 2014; Leiserowitz, Maibach, Roser-Renouf, & Smith,

2011; Leviston & Walker, 2012; McCright & Dunlap, 2011; Poortinga et al., 2011; Reser et al., 2012; Whitmarsh, 2011). Items were selected to ensure a diverse range of questions, covering a range of points on the climate change denial spectrum, e.g., “Climate change is a 9 scam” (Capstick & Pidgeon, 2014) versus “Climate change is completely caused by human activity” (Poortinga et al., 2011). Participants were required to respond to each item with either a yes or no response. The measure can be viewed in supplementary materials.

Mistrust in scientists was measured using the Muenster Epistemic Trustworthiness

Inventory (Hendriks, Kienhues, & Bromme, 2015). Mistrust in scientists has been shown to positively correlate with climate change denial (Hmielowski et al., 2014). Participants were asked “Think about the types of scientists who conduct research in the areas we have previously asked you about. How would you describe them on the following pairs of terms?”.

The measure contained 14 word-pair items (e.g., “competent-incompetent”) which participants rated on a 7-point bipolar scale. For example, a person who viewed scientists as extremely incompetent would rate the “competent-incompetent” item as a seven. Possible scores ranged from 14-98, with higher total scores indicating higher levels of mistrust in scientists. Internal consistency of the scale was found to be very high, ω = .971.

Authoritarianism was measured using the three-factor Aggression-Submission-

Conventionalism scale (Dunwoody & Funke, 2016). Authoritarianism is a personality trait characterised by submission to authority, aggression towards groups targeted by authorities, and conformity to social norms, and has been shown to positively correlate with climate change denial (Altemeyer, 1998; Häkkinen & Akrami, 2014). The scale contained 18 statements, with participants responding on a 7-point Likert scale. Items included statements such as “Traditions interfere with progress” (reverse coded) and “People in positions of authority generally tell the truth”. Possible scores ranged from 18-126, with higher scores indicating greater levels of authoritarianism. Internal consistency was found to be high, ω

= .83.

1Reliability was calculated using McDonalds Omega, which has been shown to be a more robust estimate of internal consistency than Cronbach’s alpha. For more information see Dunn, Baguley, and Brunsden (2014). Interpretation is similar to that of Cronbach’s alpha. 10

Social dominance orientation (SDO) was measured using the shortened SDO7 scale

(Ho et al., 2015). SDO is a personality trait characterised by a general orientation towards hierarchical intergroup relations, and a belief that ‘inferior’ groups should be dominated by

‘superior’ groups (Pratto, Sidanius, Stallworth, & Malle, 1994). Subsequent research has extended the construct to include dominance over the natural environment (Milfont, Richter,

Sibley, Wilson, & Fischer, 2013). SDO has been found to positively correlate with climate change denial (Häkkinen & Akrami, 2014; Milfont et al., 2013). The SDO7 scale contained eight statements, (e.g., “Some groups of people are simply inferior to other groups”), and participants were required to respond to each statement using a 7-point Likert scale. Possible scores ranged from 8-64, with higher scores indicating higher levels of SDO. Internal consistency was found to be very high, ω = .93.

Conspiracist beliefs were measured using the Generic Conspiracist Beliefs Scale

(Brotherton, French, & Pickering, 2013). Broadly, conspiracist beliefs can be defined as the unnecessary adoption of conspiracy theories when other explanations are more plausible

(Brotherton et al., 2013). Past research has found positive associations between conspiracist beliefs and climate change denial (Lewandowsky, Gignac, & Oberauer, 2015). The measure contained 15 statements (e.g., “Groups of scientists manipulate, fabricate, or suppress evidence in order to deceive the public”), and participants were required to respond to each item using one of the following response options: “Definitely not true”; “Probably not true”;

“Not sure/cannot decide”; “Probably true”; “Definitely true”. Possible scores ranged from 15-

75, with higher scores indicating stronger conspiracist beliefs. Internal consistency of the scale was found to be very high, ω = .97.

As part of a larger study on the measurement of contested science, three other measures were included in the survey: a prototype measure of vaccine hesitancy, the

Perceived Sensitivity to Medication scale (Horne et al., 2013), and a one-item measure of 11 political orientation. All measures included in the survey can be viewed in the supplementary materials.

2.3. Procedure

Prior to data collection, ethics approval was obtained from the Deakin University

Human Research Ethics Committee. The sample was obtained by publishing the study on the

Amazon MTurk platform. Members of MTurk were then able to view the Plain Language

Statement and choose to participate if they wished. Participants then clicked on a link to the study, which opened in a new window. Participants then completed the survey, which was hosted on the Qualtrics survey platform. All participants completed both the climate change denial and vaccine hesitancy measures, with the order of questionnaire presentation randomised per participant. Within these measures, item order was also randomised. The study took approximately 10 minutes to complete. Upon completion, participants were provided with a unique code which they could use to claim their reimbursement.

2.4. Statistical Analyses

Statistical analyses were conducted in the R environment (R Core Team, 2018), using the RStudio Integrated Development Environment. After cleaning the data, the ‘ltm’ package

(Rizopoulos, 2018) was used to conduct a 2-parameter IRT analysis on the climate change denial items. Items with poor discrimination were excluded first, before removal of redundant items (having very similar difficulty values). Participant scores on the abridged climate change denial measure were then derived using an Empirical Bayes estimator. These scores were used to model relationships between scores on the abridged climate change denial measure and convergent measures (mistrust in science, authoritarianism, social dominance orientation, conspiracist beliefs). Internal consistency was then examined using McDonald’s

Omega. 12

The script used to run the analyses can be found in the supplementary materials.

Additionally, the study was pre-registered on the Open Science Framework, and the de- identified data obtained from the study will be published at the following link: https://osf.io/3tbxu/

3. Results

3.1. Descriptive statistics

Key descriptive statistics are presented in Table 1.

Table 1 Descriptive Statistics for Relevant Variables Count (%) N 206 Gender Male 114 (55.3) Female 92 (44.7) Education Less than high school 0 (0) High school graduate 47 (22.8) Technical school diploma 2 (1.0) Associate degree 27 (13.1) Bachelor’s degree 103 (50.0) Professional degree 26 (12.6) Doctorate 1 (0.5) Mean (SD) Age 34.89 (10.72) Locationa 5.10 (2.82) Convergent measures Mistrust of scientists (14 – 98) 32.97 (15.57) Authoritarianism (18 – 126) 61.61 (14.81) Social dominance orientation (8 – 64) 21.30 (11.88) Conspiracy beliefs (15 – 75) 38.22 (16.40) Note. a 1 = within a major city, 10 = in a rural area. 3.2. IRT analysis

Prior to the IRT analysis, assumption testing was carried out on the data. The main assumption of IRT is unidimensionality of the measure – that is, the measure should only assess one construct. This assumption was checked using exploratory factor analysis and was 13

found to be met. Participant scores on the climate change denial measure were then analysed

using a 2-parameter IRT model in order to obtain difficulty and discrimination indices for

each item. These values are shown in Table 2. Higher values of difficulty indicate that the

item is measuring higher levels of climate change denial, and higher discrimination values

indicate that the item is better able to distinguish between different levels of denial. This

information, in the form of item characteristic curves (ICCs), is represented graphically in

Figure 1. Steeper slopes represent higher discrimination indices, while higher values on the x-

axis represent higher levels of difficulty.

Table 2 Difficulty and Discrimination Indices of the Initial Scale Item Question Difficulty Discriminatio n ccd0 I am uncertain that climate change is really happening 0.404 1.407 1 ccd0 It is too early to say whether climate change is really a problem 0.145 4.012 2 ccd0 Climate change is just a natural fluctuation in earth’s temperatures 0.022 3.406 3 ccd0 I do not believe climate change is a real problem 0.357 3.082 4 ccd0 I am certain that climate change is really happening* 0.887 2.867 5 ccd0 Claims that human activities are changing the climate are exaggerated 0.158 3.627 6 ccd0 The seriousness of climate change is exaggerated 0.000 6.980 7 ccd0 Floods and heat-waves are not increasing, there is just more reporting 0.307 3.290 8 of it in the media these days ccd0 The seriousness of climate change is exaggerated in the media. 0.010 19.671 9 ccd1 Over the past 12 months, have you punished companies that are -5.933 0.117 0 opposing steps to reduce global warming by NOT buying their products?* ccd1 Do you think global warming should be a high priority for the 0.681 2.118 1 government?* ccd1 Scientists have in the past changed their results to make climate 0.168 2.707 2 change appear worse than it is ccd1 Climate change is a scam 0.386 3.996 14

3 ccd1 The risks associated with climate change are understated* 0.291 0.595 4 ccd1 Climate change is entirely caused by human-activity* -0.203 0.673 5 ccd1 Climate change is just a result of natural variation in the climate. 0.031 3.625 6 ccd1 The so-called "climate threat" is exaggerated. 0.129 4.737 7 ccd1 The climate is not really changing 0.771 2.175 8 Note. * Items were reverse coded.

Difficulties of the initial scale ranged from -5.933 to 0.887. Discrimination ranged

from 0.117 to 19.671.

Figure 1. ICC plot of initial scale.

3.3. Scale refinement

After initial analysis, a more concise scale was derived by excluding items that were

insufficiently sensitive (i.e., low discrimination values) or did not provide additional 15 information (i.e., items that had similar difficulty values). In instances where redundancy was found, the item with higher discrimination was kept. After determining item inclusion and exclusion, the data were again analysed using a 2-parameter IRT model. Resulting difficulty and discrimination indices are displayed in Table 3. This information is also represented graphically in Figure 2.

Table 3 Difficulty and Discrimination Indices of the Abridged Scale Item Question Difficulty Discrimination ccd05 I am certain that climate change is really happening* 1.167 3.037 ccd06 Claims that human activities are changing the climate are 0.536 4.477 exaggerated ccd08 Floods and heat-waves are not increasing, there is just more 0.641 4.446 reporting of it in the media these days ccd09 The seriousness of climate change is exaggerated in the media 0.477 6.487 ccd11 Do you think global warming should be a high priority for the 0.996 2.254 government?* ccd13 Climate change is a scam 0.681 30.839 ccd16 Climate change is just a result of natural variation in the 0.438 3.890 climate. ccd18 The climate is not really changing 0.954 3.469 Note. * Items were reverse coded. 16

Difficulties of the abridged scale ranged from .438 to 1.167. Discrimination of all items was high, ranging from 2.254 to 30.839. 17

Figure 2. ICC plot of abridged scale. 3.4. Correlations with convergent measures

Participant scores on the abridged scale were derived using an Empirical Bayes estimator to allow for correlational testing with convergent measures. Assumptions for correlational analyses were tested by visual inspection of scatterplots and box-and-whisker plots, and no violations were detected. Correlations between standardised scores on the abridged climate change denial measure and the convergent measures are shown in Table 4.

Table 4 Correlations Between Abridged Climate Change Denial Measure and Convergent Measures Measure Pearson Correlation (CIs) p-value Mistrust in science .54 (.43, .63) < .001 Authoritarianism .36 (.24, .48) < .001 Social dominance orientation .55 (.45, .64) < .001 Conspiracist beliefs .34 (.22, .46) < .001

The abridged climate change denial measure had moderate to strong correlations with each of mistrust in science, authoritarianism, social dominance orientation, and conspiracist beliefs.

3.5. Reliability

Internal consistency of the abridged climate change denial measure was calculated using McDonalds Omega, and was found to be very high, ω = .94.

4. Discussion

The current study aimed to create a new measure of climate change denial using Item

Response Theory; to administer the measure to a sample in order to obtain difficulty and discrimination indices; and to validate it against known correlates of climate change denial. A new measure of climate change denial was created by drawing items from previous measures, item characteristics were successfully obtained and were used to refine the scale, and correlations between the refined measure and known correlates of climate change denial were found to be of the expected direction and magnitude. Additionally, the refined measure was 18 found to have very high internal consistency. The measure created appears to be a functional and efficient tool for measuring climate change denial.

4.1. Limited variability of denial

Interestingly, after screening out inefficient and redundant items, the measure appeared to capture a relatively narrow band of climate change denial, as indicated by the limited range of difficulty indices. The difficulty value indicates the level of construct the item is measuring, with higher difficulties capturing higher levels of denial. There are a number of possible reasons for this finding. Climate change denial may simply be a narrower construct than previously thought. Researchers such as Rahmstorf (2004) suggest a ‘stepped’ progression of denial, with ‘trend’ denial (denying that a warming trend exists) at the pinnacle, and ‘impact’ denial (accepting climate change but downplaying the negative impacts) at the milder end of the spectrum. It could be that these different steps of denial are not as clearly differentiated as previously thought, and consequently, item difficulties may be clustered together simply due to the similarity of different forms of denial. Another potential implication is that distinguishing between different forms of denial is unnecessary in the context of measurement, and viewing denial as a unidimensional spectrum would make for easier comparison of denial rates between studies and contexts.

Another potential reason for the limited range of observed difficulties relates to the items themselves, in that the items included in the scale may not capture the complete range of climate change beliefs. As items were taken from a variety of widely used measures, this potential issue would not be confined to this particular measure. Moreover, the measure included several strongly worded items on either end of the spectrum (e.g., “Climate change is a scam” and “I am certain that climate change is really happening”), and items such as these should theoretically capture individuals who sit at either end of the denial spectrum.

4.2. Implications of discrimination values 19

Interesting findings were also revealed from examination of discrimination indices, which indicate how well an item can distinguish between different levels of denial. The initial scale contained a number of items with extremely poor discrimination. These items were screened out during scale refinement, and all items contained within the abridged measure had high discrimination, meaning that items were efficient in assessing levels of climate change denial. However, implications arise from the types of items that had low discrimination. A consistent pattern was observed in both the initial and abridged scales, where reverse coded items tended to discriminate more poorly than non-reverse coded items.

It has been proposed that reversed items may measure a different underlying trait (Magazine,

Williams, & Williams, 2016; Weems & Onwuegbuzie, 2001). While the current measure was found to be unidimensional, the finding that reversed items had poorer discrimination may be useful information for future research, and in the development of related measures.

In contrast, items which were simple, direct, and strongly worded, such as “Climate change is a scam”, provided extremely high discrimination in the abridged measure. This is both a strength of the measure, and important information for future research. Items with high discrimination are useful in brief measures, because they can reliably differentiate respondents on different levels of a construct or trait, resulting in highly efficient and reliable measures. Therefore, highly discriminating items could be used in the development of brief climate change denial measures, and possibly in the brief measurement of other constructs.

The findings also open up the possibility of a one-item screening measure for climate change denial, useful when researchers are not interested in where a person lies on the continuum of climate change denial, but whether they broadly believe in climate change. Situations such as this could arise where belief or disbelief in climate change is a prerequisite for a study, and a one-item screener is used to include or exclude potential participants. Similarly, in the context of computerised adaptive testing, where the order and presence of questions depends 20 on responses to preceding questions, items with such high discrimination would be an efficient first item to assess broad climate change beliefs, with subsequent questions then narrowing down a respondent’s level of denial.

4.3. Limitations and future directions

However, it is also necessary to outline potential limitations of the measure. In using

IRT, the difficulty and discrimination indices of the items are based on the particular sample used to calculate item parameters (de Champlain, 2010). This initial item parameter calculation can have effects on the subsequent measurement of constructs. For example, if the measure were validated on a sample with very high levels of denial, then those items would be given low difficulty values - i.e., they are ‘easy’ for participants in the sample to endorse.

If the measure were then administered to a sample with lower levels of denial, the measure would likely underestimate the level of denial in that second sample - because the participants are only able to endorse a few of the items. Therefore, having the measure validated on a limited sample impacts on generalisability. A solution to this issue would be to estimate the item parameters using a larger, more diverse sample.

Additionally, this study is unable to provide evidence regarding the predictive validity of the new measure. A review of relevant literature indicates that there is still some debate on whether climate change denial in general can predict pro-environmental behaviours (e.g., Van

Rensburg, 2015). Research conducted by Stoll-Kleemann, O’Riordan, and Jaeger (2001) found that while most people in their sample believed climate change was occurring, they built numerous psychological barriers in order to avoid participating in individual or political action to mitigate the impacts of climate change. This indicates that there may be a disconnect between beliefs and behaviour. In the current study, the initial measure did include one quasi-behavioural item, (“Over the past 12 months, have you punished companies that are opposing steps to reduce global warming by NOT buying their 21 products?”), but the item was screened out during scale refinement due to its poor discrimination value, possibly due to the double-barrelled nature of the item or due to its reverse coding. Future research should include some form of behavioural measure to determine the predictive utility of the proposed measure. Examples of this include asking participants to donate money, or have researchers donate money on their behalf, to pro- or anti-climate change organisations, or alternately, asking respondents to sign up to climate change-related petitions or organisations (Bliuc et al., 2015; Rabinovich, Morton, Postmes, &

Verplanken, 2012).

5. Conclusion

Despite the outlined limitations, this study provides a novel, internally consistent, efficient, and easily implementable measure of climate change denial, which has been validated against known correlates of climate change denial. Additionally, the study shows that use of IRT in the measurement of climate change denial is feasible. The results raise questions about the distribution of climate change beliefs, provide implications regarding the discriminant utility of items, and open up potential for one-item screening measures of climate change denial and other constructs. The study also provides areas for future research around the predictive utility of such a measure in terms of pro-environmental behaviours. 22

References

Altemeyer, B. (1998). The other “authoritarian personality”. Advances in Experimental

Social Psychology, 30, 47-92. doi:10.1016/s0065-2601(08)60382-2

Benegal, S. D., & Scruggs, L. A. (2018). Correcting misinformation about climate change:

the impact of partisanship in an experimental setting. Climatic Change.

doi:10.1007/s10584-018-2192-4

Bliuc, A., McGarty, C., Thomas, E. F., Lala, G., Berndsen, M., & Misajon, R. (2015). Public

division about climate change rooted in conflicting socio-political identities. Nature

Climate Change, 5(3), 226-229. doi:10.1038/nclimate2507

Bortolotti, S. L. V., Tezza, R., de Andrade, D. F., Bornia, A. C., & de Sousa Júnior, A. F.

(2013). Relevance and advantages of using the item response theory. Quality &

Quantity, 47(4), 2341-2360. doi:10.1007/s11135-012-9684-5

Brotherton, R., French, C. C., & Pickering, A. D. (2013). Measuring belief in conspiracy

theories: The generic conspiracist beliefs scale. Frontiers in Psychology, 4.

doi:10.3389/fpsyg.2013.00279

Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon's Mechanical Turk.

Perspectives on Psychological Science, 6(1), 3-5. doi:10.1177/1745691610393980

Capstick, S. B., & Pidgeon, N. F. (2014). What is climate change scepticism? Examination of

the concept using a mixed methods study of the UK public. Global Environmental

Change, 24, 389-401. doi:10.1016/j.gloenvcha.2013.08.012

Chalmers, R. P. (2012). mirt: A multidimensional item response theory package for the R

environment. Journal of Statistical Software, 48(6). doi:10.18637/jss.v048.i06 de Champlain, A. F. (2010). A primer on classical test theory and item response theory for

assessments in medical education. Medical education, 44(1), 109-117.

doi:10.1111/j.1365-2923.2009.03425.x 23

Doran, P. T., & Zimmerman, M. K. (2009). Examining the scientific consensus on climate

change. Eos, Transactions American Geophysical Union, 90(3).

doi:10.1029/2009eo030002

Dunn, T. J., Baguley, T., & Brunsden, V. (2014). From alpha to omega: A practical solution

to the pervasive problem of internal consistency estimation. British Journal of

Psychology, 105(3), 399-412. doi:10.1111/bjop.12046

Dunwoody, P. T., & Funke, F. (2016). The Aggression-Submission-Conventionalism Scale:

Testing a new three factor measure of authoritarianism. Journal of Social and

Political Psychology, 4(2), 571-600. doi:10.5964/jspp.v4i2.168

Fan, X. (1998). Item response theory and classical test theory: An empirical comparison of

their item/person statistics. Educational and psychological measurement, 58(3), 357-

381. doi:10.1177/0013164498058003001

Greenhill, M., Leviston, Z., Leonard, R., & Walker, I. (2014). Assessing climate change

beliefs: Response effects of question wording and response alternatives. Public

Understanding of Science, 23(8), 947-965. doi:10.1177/0963662513480117

Häkkinen, K., & Akrami, N. (2014). Ideology and climate change denial. Personality and

Individual Differences, 70, 62-65. doi:10.1016/j.paid.2014.06.030

Hambleton, R. K., & Jones, R. W. (2005). An NCME instructional module on comparison of

classical test theory and item response theory and their applications to test

development. Educational Measurement: Issues and Practice, 12(3), 38-47.

doi:10.1111/j.1745-3992.1993.tb00543.x

Hart, P. S., & Nisbet, E. C. (2011). Boomerang effects in science communication.

Communication Research, 39(6), 701-723. doi:10.1177/0093650211416646 24

Hendriks, F., Kienhues, D., & Bromme, R. (2015). Measuring laypeople's trust in experts in a

digital age: The Muenster Epistemic Trustworthiness Inventory (METI). PLoS One,

10(10). doi:10.1371/journal.pone.0139309

Hmielowski, J. D., Feldman, L., Myers, T. A., Leiserowitz, A., & Maibach, E. (2014). An

attack on science? Media use, trust in scientists, and perceptions of global warming.

Public Understanding of Science, 23(7), 866-883. doi:10.1177/0963662513480091

Ho, A. K., Sidanius, J., Kteily, N., Sheehy-Skeffington, J., Pratto, F., Henkel, K. E., . . .

Stewart, A. L. (2015). The nature of social dominance orientation: Theorizing and

measuring preferences for intergroup inequality using the new SDO(7) scale. Journal

of Personality and Social Psychology, 109(6), 1003-1028. doi:10.1037/pspi0000033

Horne, R., Faasse, K., Cooper, V., Diefenbach, M. A., Leventhal, H., Leventhal, E., & Petrie,

K. J. (2013). The perceived sensitivity to medicines (PSM) scale: An evaluation of

validity and reliability. British Journal of Health Psychology, 18(1), 18-30.

doi:10.1111/j.2044-8287.2012.02071.x

Hornsey, M. J., Fielding, K. S., McStay, R., Reser, J. P., & Bradley, G. L. (2016). Are people

high in skepticism about anthropogenic climate change necessarily resistant to

influence? Some cause for optimism. Environment and Behavior, 48(7), 905-928.

doi:10.1177/0013916515574085

IPCC. (2018). Global warming of 1.5°C. Retrieved from

https://www.ipcc.ch/site/assets/uploads/sites/2/2018/07/SR15_SPM_High_Res.pdf:

Leiserowitz, A., Maibach, E., Roser-Renouf, C., & Smith, N. (2011). Climate change in the

American mind - Americans’ global warming beliefs and attitudes in May 2011.

Retrieved from http://environment.yale.edu/climate/files/ClimateBeliefsMay2011.pdf 25

Leviston, Z., Leitch, A., Greenhill, M., Leonard, R., & Walker, I. (2011). Australians’ views

of climate change. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?

doi=10.1.1.358.327&rep=rep1&type=pdf

Leviston, Z., & Walker, I. (2012). Beliefs and denials about climate change: An Australian

perspective. Ecopsychology, 4(4), 277-285. doi:10.1089/eco.2012.0051

Lewandowsky, S., Gignac, G. E., & Oberauer, K. (2015). The robust relationship between

conspiracism and denial of (climate) science. Psychological Science, 26(5), 667-670.

doi:10.1177/0956797614568432

Lewandowsky, S., Oberauer, K., & Gignac, G. E. (2013). NASA faked the moon landing--

therefore, (climate) science is a hoax: An anatomy of the motivated rejection of

science. Psychological Science, 24(5), 622-633. doi:10.1177/0956797612457686

Magazine, S. L., Williams, L. J., & Williams, M. L. (2016). A confirmatory factor analysis

examination of reverse coding effects in Meyer and Allen's affective and continuance

commitment scales. Educational and psychological measurement, 56(2), 241-250.

doi:10.1177/0013164496056002005

McCright, A., Charters, M., Dentzman, K., & Dietz, T. (2016). Examining the effectiveness

of climate change frames in the face of a climate change denial counter-frame. Topics

in Cognitive Science, 8(1), 76-97. doi:10.1111/tops.12171

McCright, A. M., & Dunlap, R. E. (2011). Cool dudes: The denial of climate change among

conservative white males in the United States. Global Environmental Change, 21(4),

1163-1172. doi:10.1016/j.gloenvcha.2011.06.003

Milfont, T. L., Richter, I., Sibley, C. G., Wilson, M. S., & Fischer, R. (2013). Environmental

consequences of the desire to dominate and be superior. Personality and Social

Psychology Bulletin, 39(9), 1127-1138. doi:10.1177/0146167213490805 26

Poortinga, W., Spence, A., Whitmarsh, L., Capstick, S., & Pidgeon, N. F. (2011). Uncertain

climate: An investigation into public scepticism about anthropogenic climate change.

Global Environmental Change, 21(3), 1015-1024.

doi:10.1016/j.gloenvcha.2011.03.001

Pratto, F., Sidanius, J., Stallworth, L. M., & Malle, B. F. (1994). Social dominance

orientation: A personality variable predicting social and political attitudes. Journal of

Personality and Social Psychology, 67(4), 741-763. doi:10.1037/0022-3514.67.4.741

R Core Team. (2018). R: A language and environment for statistical computing. Vienna,

Austria: R Foundation for Statistical Computing. Retrieved from http://www.R-

project.org/

Rabinovich, A., Morton, T. A., Postmes, T., & Verplanken, B. (2012). Collective self and

individual choice: The effects of inter-group comparative context on environmental

values and behaviour. British Journal of Social Psychology, 51(4), 551-569.

doi:10.1111/j.2044-8309.2011.02022.x

Rahmstorf, S. (2004). The climate sceptics. Retrieved from

http://www.pik-potsdam.de/stefan/Publications/Other/rahmstorf_climate_sceptics_20

04.pdf.

Reser, J. P., Bradley, G. L., Glendon, I., Ellul, M. C., & Callaghan, R. (2012). Public risk

perceptions, understandings and responses to climate change in Australia and Great

Britain. Gold Coast: National Climate Change Adaptation Research Facility.

Rizopoulos, D. (2018). Package ‘ltm’. Retrieved from

https://cran.r-project.org/web/packages/ltm/ltm.pdf

Stoll-Kleemann, S., O’Riordan, T., & Jaeger, C. C. (2001). The psychology of denial

concerning climate mitigation measures: Evidence from Swiss focus groups. Global

Environmental Change, 11(2), 107-117. doi:10.1016/s0959-3780(00)00061-3 27

Van Rensburg, W. (2015). Climate change scepticism: A conceptual re-evaluation. SAGE

Open, 5(2). doi:10.1177/2158244015579723

Weems, G. H., & Onwuegbuzie, A. J. (2001). The impact of midpoint responses and reverse

coding on survey data. Measurement & Evaluation in Counseling & Development,

34(3), 166-177.

Whitmarsh, L. (2011). Scepticism and uncertainty about climate change: Dimensions,

determinants and change over time. Global Environmental Change, 21(2), 690-700.

doi:10.1016/j.gloenvcha.2011.01.016