Agency and self-other asymmetries in perceived bias and shortcomings: Replications of the Bias Blind Spot and extensions linking to free will beliefs
*Subramanya Prasad Chandrashekar Institute of International Business and Governance, Lee Shau Kee School of Business and Administration, Open University of Hong Kong [email protected]
*Siu Kit Yeung, *Ka Chai Yau Department of Psychology, University of Hong Kong, Hong Kong SAR [email protected]; [email protected]
**Chung Yee Cheung, **Tanay Kulbhushan Agarwal, **Cho Yan Joan Wong, **Tanishka Pillai, **Thea Natasha Thirlwell, **Wing Nam Leung , **Yan Tung Li , **Colman Tse Department of Psychology, University of Hong Kong, Hong Kong SAR [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Bo Ley Cheng, Hill Yan Cedar Chan Department of Psychology, University of Hong Kong, Hong Kong SAR [email protected]; [email protected]
^Gilad Feldman Department of Psychology, University of Hong Kong, Hong Kong SAR [email protected]
In press at Judgment and Decision Making Accepted for publication on June, 2021
*Contributed equally, joint first author **Contributed equally, joint fourth author ^Corresponding author Word count: 5730 [excluding tables, table notes, figures, figure captions, and references] Author bios Subramanya Prasad Chandrashekar is a research assistant professor with the Lee Shau Kee School of Business and Administration at the Open University of Hong Kong. His research focuses on social status, lay-beliefs, and morality. Yeung Siu Kit, Yau Ka Chai, Mun Ting Livia Chan , Cheuk Man Jeffrey Ho, Chung Yee Cheung, Cheuk Him Lui , Tanay Kulbhushan Agarwal, Cho Yan Joan Wong, Tanishka Pillai, Thea Natasha Thirlwell, Wing Nam Leung , Yan Tung Li , Cheuk Man Jeffrey Ho, and Colman Tse were students at the University of Hong Kong during academic year 2018. Bo Ley Cheng and Hill Yan Cedar Chan were teaching assistants at the University of Hong Kong psychology department during academic year 2018. Gilad Feldman is an assistant professor with the University of Hong Kong psychology department. His research focuses on judgment and decision-making. Corresponding author
Gilad Feldman, Department of Psychology, University of Hong Kong, Hong Kong SAR; [email protected] Declaration of Conflict of Interest: The author(s) declare no potential conflicts of interests with respect to the authorship and/or publication of this article. Financial disclosure/funding: This research was supported by the European Association for Social Psychology seedcorn grant and the University of Hong Kong Teaching Development Grant. Authorship declaration Gilad led the first two reported replication efforts in his courses. Gilad supervised each step in the project, conducted the pre-registrations, and collected data. Prasad integrated the two replication efforts with validation and extensions of the analyses (Studies 1 and 2). Prasad designed and wrote the pre-registration for follow-up third replication (Study 3). Gilad pre- registered and conducted data collection. Prasad analyzed the data. Prasad and Gilad jointly wrote the final manuscript. Yeung Siu Kit and Yau Ka Chai conducted the US replication (Study 2). Yeung Siu Kit designed and initiated the free-will extensions to the US replication. Mun Ting Livia Chan, Cheuk Man Jeffrey Ho, Chung Yee Cheung, Cheuk Him Lui , Tanay Kulbhushan Agarwal, Cho Yan Joan Wong , Tanishka Pillai, Thea Natasha Thirlwell, Wing Nam Leung, Yan Tung Li, and Colman Tse conducted the Hong Kong replications (Studies 1a and 1b). Bo Ley Cheng and Hill Yan Cedar Chan guided and assisted the replication efforts by the three teams. Contributor Roles Taxonomy
In the table below, employ CRediT (Contributor Roles Taxonomy) to identify the contribution and roles played by the contributors in the current replication effort. Please refer to the url (https://www.casrai.org/credit.html ) on details and definitions of each of the roles listed below.
Yeung Siu Kit and Yau Ka Chai, Mun Ting Livia Chan, Cheuk Man Jeffrey Ho, Chung Yee Cheung, Cheuk Him Lui , Tanay Kulbhushan Agarwal, Cho Yan Bo Ley Joan Wong , Tanishka Pillai, Cheng, Subramanya Thea Natasha Thirlwell, Wing Hill Yan Prasad Gilad Nam Leung, Yan Tung Li, and Cedar Role Chandrashekar Feldman Colman Tse Chan Conceptualization X Pre-registrations X X Data curation X Formal analysis X X Funding acquisition X Investigation X Methodology X Pre-registration peer review / verification X X Data analysis peer review / verification X X Project administration X X Resources X Software X Supervision X X Validation X X Visualization X Writing-original draft X Writing-review and editing X X
Acknowledgements:
Subramanya Prasad Chandrashekar would like to thank Institute of International Business and Governance (IIBG), established with the substantial support of a grant from the Research Grants Council of the Hong Kong Special Administrative Region, China (UGC/IDS 16/17), for its support. Bias Blind Spot: Replications and extensions 1
Abstract
Bias Blind Spot (BBS) is the phenomenon that people tend to perceive themselves as less susceptible to biases than others. In three pre-registered experiments (overall N = 969), we replicated two experiments of the first demonstration of the phenomenon by Pronin et al. (2002).
We found support of the BBS hypotheses, with effects in line with findings in the original study:
Participants rated themselves as less susceptible to biases than others (d = -1.00 [-1.33, -0.67]).
Deviating from the original, we found an unexpected effect that participants rated themselves as having fewer shortcomings (d = -0.34 [-0.46, -0.23]), though there was support for the target's main premise that BBS was stronger for biases than for shortcomings (d = -0.43 [-0.56, -0.29]).
Extending the replications, we found that beliefs in own free will were positively associated with
BBS (r ~ 0.17–0.22) and that beliefs in both own and general free will were positively associated with self-other asymmetry related to personal shortcomings (r ~ 0.16–0.24). Materials, datasets, and code are available on https://osf.io/3df5s/.
Keywords: bias blind spot; bias; free-will beliefs; replication; self-other asymmetries
Bias Blind Spot: Replications and extensions 2
Agency and self-other asymmetries in perceived bias and shortcomings:
Replications of the Bias Blind Spot and extensions linking to free will beliefs
There is a wealth of evidence that human judgment is often affected by motivational and cognitive factors (Kunda, 1990; Nisbett & Ross, 1980). For example, people often rate themselves as being better than average (Dunning et al., 1989), and attribute failure to uncontrollable external factors while claiming credit for success (Miller & Ross, 1975). In light of this, researchers have further sought to understand whether people are aware of their susceptibility to such biases, and demonstrated that self-assessments of one's own bias are often biased as well.
Bias blind spot (BBS) is the phenomenon that people perceive themselves to be less susceptible to biases than others. Research on BBS builds on previous work on actor-observer attributions documenting people's asymmetrical perceptions of themselves compared to others
(Jones & Nisbett, 1972; Pronin et al., 2004), and it would seem that these misalignments extend to differing perceptions of one's own biases compared to those of others.
BBS holds important practical implications. Doctors view themselves as being immune to the impact of gifts from pharmaceutical industry representatives on their own judgments, while perceiving their peers' judgments to be strongly affected (Dana & Loewenstein, 2003). The illusion of objectivity associated with BBS has also been linked with discrimination in hiring
(Uhlmann & Cohen, 2007).
If people are to improve their decision-making and overcome biases, they must first be able to recognize and understand their own biases (Hansen et al., 2014). Yet, BBS suggests that
Bias Blind Spot: Replications and extensions 3 recognizing one's own biases is a tricky challenge, because people's understanding of their own biases is influenced by the very same biases they aim to better understand, thereby resulting in an ironic "blind spot".
We begin by introducing the literature on the BBS and the chosen article for replication, discussing the motivations for the current replication study. We then introduce free-will beliefs as a suggested extension and hypothesize about the relationship between free will beliefs and the bias blind spot effect.
Bias Blind Spot: Theoretical mechanisms
Bias blind spot has been theorized to be a result of the three possible sources: introspection illusion, naïve realism, and self-enhancement motives (Pronin & Kugler, 2007).
The introspection illusion (Nisbett & Wilson, 1977) is people's tendency to place more confidence in introspective information that they can access directly (Ross & Ward, 1996).
People use introspection to access their own intentions, emotions, thoughts, and salient attitudes when seeking to understand and explain their own behaviors. Yet, when aiming to understand and explain others' behaviors, others' thoughts and feelings are not directly accessible and inferences are considered unreliable. People must resort to focusing merely on evaluating others based on their behaviors, which leads to differences in how people perceive their own behavior and decision-making process compared to how they perceive others' (Nisbett & Ross, 1980).
Many such biases in self-other perceptions have been documented over the years. For example, the widely discussed classic actor-observer bias (Jones & Nisbett, 1972) is the perception that others' behaviors reflect stable personality dispositions rather than contextual factors (Jones,
1990).
Bias Blind Spot: Replications and extensions 4
Naïve realism (Ross & Ward, 1995) describes people's tendency to believe that their own perceptions of the world are more objective and accurate than others'. Because one's own thoughts, attitudes, and beliefs come to mind readily and easily, they also appear to be devoid of subjective distortions. As a result, people can fairly easily detect subjectivity in others, yet fail to detect subjectivity in themselves (Pronin, 2007; Pronin et al., 2004; Ross & Ward, 1995; Wilson
& Dunn, 2004). Biases are often considered a deviation from rational objectivity, and it therefore follows that people would perceive themselves as being less biased than others.
Finally, people tend to construe reality in a way that would make for more positive self- evaluations. Biases are commonly perceived as sub-optimal, often referred to as being
"irrational", and so to maintain a positive self-image of being rational and objective, people would need to view themselves as being less biased (Pronin & Kugler, 2007). Ironically, they also resist revising their views about being biased, and persist in their biased perceptions even after having been made aware of their biases (Ehrlinger et al., 2005; Pronin, 2007).
Following the recent developments in psychological science and the growing recognition of the importance of reproducibility and replicability (e.g., Brandt et al., 2014; Open Science
Collaboration, 2015; Van‘t Veer & Giner-Sorolla, 2016; Zwaan et al., 2018), we aimed to revisit the classic work on the BBS with independent replications. Very close direct pre-registered replications are needed to assess previous claims and obtain an accurate estimate of the effect size. We therefore embarked on a series of pre-registered close replications of Pronin et al.'s
(2002) work, with added extensions to further our understanding of the phenomenon. Pronin et al. (2002) provided the first demonstration of the effect by examining people's ratings of susceptibility to common biases in themselves and in others, and showed that people tend to perceive themselves as being less biased, even when being made aware of their potential bias.
Bias Blind Spot: Replications and extensions 5
We chose to conduct very close direct replications and extensions of Pronin et al. (2002) to revisit original findings, reproduce their materials, address and improve original methods and analysis, and extend to examine new directions. Replications contribute to theory specification and form the basis of further theory development (Glöckner & Betsch, 2011).
Choice of target article: Pronin et al. (2002)
We chose Pronin et al.'s (2002) studies for replication for their importance to the self-other asymmetry literature. Specifically, Pronin et al. (2002) found empirical evidence for the view that the sources of BBS are not restricted to self-serving motives. The findings emphasize the primary role of the introspection illusion and naïve realism on BBS effects. The article has had a significant impact on research in social psychology and judgment and decision-making. At the time of writing (March 2021), Google Scholar indicated 1129 citations of the article, laying ground to broader theories about human sense-making and objectivity (Pronin, 2007; Pronin et al., 2004). More recent theoretical assertions from Scopelliti et al. (2015) claim the BBS effect is a meta-bias that can be measured at the individual level. To the best of our knowledge, there are no published direct replications of Pronin et al. (2002).
Given the significance of Pronin et al.'s (2002) work, it is important to ascertain the replicability of the findings using high powered samples. Across the four studies reported in the original paper, the sample size ranged from 24 to 91, with an average sample size of 55 per study.
Pronin et al. (2002) specifically argue that people's introspection fails to detect unconscious processes involved in biases, whereas people's introspection regarding own personal shortcomings — such as the tendency to procrastinate, the fear of public speaking, and the planning fallacy — are more visible and more accessible. Based on this argument, the authors predicted that the self-versus-others asymmetry is larger for biases than for personal
Bias Blind Spot: Replications and extensions 6 shortcomings. Study 1 Survey 2 of Pronin et al. (2002) found empirical support for the view that people rate themselves, compared with others, to be less susceptible to biases, whereas there was no support for self-other asymmetry in ratings of personal shortcomings. Authors interpreted the findings as the demonstration of the stronger role of introspective illusions on BBS. The argument was that introspection may work to reduce self-other asymmetry in personal shortcomings, because shortcomings are more readily accessible and easier to contemplate than cognitive biases. However, it is possible that the original studies reported the lack of significant self-other differences in ratings on susceptibility to shortcomings because of the underpowered samples. Similarly, the same issues with power may have been the reason why they were able to detect the hypothesized effect in only four of the eight self-serving bias items tested. One of the goals of the current replication was to address this shortcoming by testing the replicability of the
BBS effects reported in Pronin et al. (2002) with high-powered samples.
In the current replication, we also contribute toward recent calls for the empirical testing of theories in non-Western populations. Scholars note that the psychology literature is lopsided, with 80 % of study participants in psychology coming from Western, educated, industrialized, rich and democratic (WEIRD) societies (Henrich et al., 2010). Here, we also tested the replicability of the BBS effect among Hong Kong participants.
Extension: Free will beliefs and differences in self-other attributions
We extended our replication and literature on the BBS to examine associations with individual differences. So far, there is little work examining individual differences as predictors of BBS, although one example is by West et al. (2012) who reported failing to find support for associations between cognitive ability and sophistication and the BBS. We were specifically
Bias Blind Spot: Replications and extensions 7 interested in possible associations with free will beliefs as laybeliefs regarding self-regulation and own choice and control (Baumeister & Monroe, 2014).
Belief in free will is the generalized belief that human behavior is free from internal and external constraints across situations (Feldman, 2017; Monroe & Malle, 2014). Studies on the folk understanding of free will found that people normally associate free will with having choice, and understand free will as the absence of internal and external constraints (Baumeister, 2008;
Feldman et al., 2014; Monroe et al., 2014; Monroe & Malle, 2010; Vonasch et al., 2018). Free will beliefs were found to be associated with a range of behavioral and psychological factors, such as academic and job performance (Feldman et al., 2016; Stillman et al., 2010), job satisfaction (Feldman et al., 2018), cooperation (Protzko et al., 2016), and well-being and meaning in life (Crescioni et al., 2016).
How would the concept of free-will relate to biases? First, by definition, people have limited to no control over their own biases, and these seem to mostly occur to regardless of one's own choice, affecting individuals without their awareness. Therefore, stronger beliefs in one's capacity to choose one's path are likely to be associated with perceiving oneself as less biased.
Second, free will beliefs were previously linked with counterfactual thinking (Alquist et al., 2015; Fillon et al., 2021), reflecting stronger introspection, which possibly contributes to learning and avoiding similar mistakes in the future (Feldman et al., 2016). If that is the case, stronger self-reflection associated with stronger free will beliefs may mean the accentuated introspection illusion underlies the bias blind spot.
Third, in a previous study by Pronin and colleague about free will attributions, they demonstrated that "people believe they have more free-will than others" (Pronin & Kugler, 2010).
Bias Blind Spot: Replications and extensions 8
They explained their findings as a new take on the actor-observer bias, arguing that people perceive others’ behavior as more constrained (stable and unchanging personalities, hindering free will), whereas they tend to view their own behavior as intentional and reflective (responsive to situations, adaptive, and changing).
Fourth, free will beliefs were found to be associated with an overestimation of the influence of internal (dispositional and fixed) factors as compared to external (varying and adaptable) factors when evaluating others' behavior (Genschow et al., 2017).
Therefore, we explored the possibility that belief in free will would be positively related to the BBS.
We note that we did not pre-register a priori predictions regarding associations between free will beliefs and the bias blind spot regarding susceptibility to biases. Our predictions focused on self-other differences in personal shortcomings. Given the link between free will and differences in attributions when comparing oneself and others, stronger belief in one's choice and control over one's own actions is likely to be associated with the perception that one's own shortcomings are freely chosen, and less of a flaw. In comparison, stronger free will beliefs are also likely to be associated with the view that others' shortcomings are a result of their disposition, and therefore are more enduring flaws. Combined, the two are suggestive of a positive relationship between free will beliefs and self-other differences regarding personal shortcomings.
Chosen studies for replication: Studies 1 and 2
Our target article for replication, Pronin et al. (2002), consisted of three experiments, and of those we chose to focus on Studies 1 and 2.
Bias Blind Spot: Replications and extensions 9
In Study 1, participants separately rated their own and the average American's susceptibility to eight biases and three personal shortcomings. Pronin et al. (2002) proposed and found support for three hypotheses: 1) participants rate themselves as less susceptible to biases than others, 2) there are no self-other differences detected for ratings of personal shortcomings
(null hypothesis), and, therefore, 3) self-other differences are larger for biases than for shortcomings.
In Study 2, participants compared themselves to others on six personality dimensions, three positive and three negative. Findings were in support of the better than average effect, with participants rating themselves higher on the positive personality dimensions and lower on negative ones, in comparison to others. In support of the bias blind spot, a large majority of participants who exhibited the better than average bias insisted that their ratings were accurate or too modest, even after reading about the better than average effect and their potential bias.
Method
Pre-registrations and open data/code
In each of the replication studies, we first pre-registered the experiment on the Open
Science Framework (OSF), and data collection was launched within one week of registration.
Pre-registrations, disclosures, power analyses, and all materials are available in the
Supplementary Materials. These, together with datasets and R/RMarkdown code, were made available on the OSF at https://osf.io/3df5s/. All measures, manipulations, and exclusions for this investigation are reported, and data collection was completed before analyses. Pre-registrations are all available on the OSF: Study 1a - https://osf.io/f4rb6/; Study 1b - https://osf.io/tez7m/;
Study 2 - https://osf.io/4m35x/; Study 3 - https://osf.io/kqatx/.
Bias Blind Spot: Replications and extensions 10
Data collections
The present investigation included three data collections. We summarized an overview of the three studies in Table 1.
Bias Blind Spot: Replications and extensions 11
Table 1
Summary of studies
# N Sample Effect tested Measures Undergraduate 45 Bias blind spot Pronin et al. (2002) students from HK 1 Undergraduate 43 Better than average effect Pronin et al. (2002) students from HK Bias blind spot Pronin et al. (2002) Better than average effect Pronin et al. (2002) MTurk participants 2 310 Free Will Inventory (FWI) and from the USA Extension: Belief in free Free Will and Determinism will (FAD) Bias blind spot Pronin et al. (2002) Better than average effect Pronin et al. (2002) MTurk participants Free Will Inventory (FWI), Free 3 621 from the USA Extension: Belief in free Will and Determinism (FAD), and will Free will and Determinism plus scale (FAD+)
Studies 1a and 1b were meant as a pre-test of the effects in an undergraduate class covering topics of open science and social psychology, and were designed to replicate the bias blind spot (BBS) effect (Study 1 Survey 2) and the better than average effect (BAE) (Study 2), reported in the target study.
Study 2 was a separate replication of the BBS effect, designed by a different replication team, and with a larger well-powered sample of different demographics. The data collection for
Study 2 was conducted separately within two weeks of the data collection for Studies 1a/b.
Bias Blind Spot: Replications and extensions 12
In light of findings from the first two studies, a year later we followed up with a third data collection. Study 3 aimed to replicate and extend the findings, especially regarding the free will beliefs extensions proposed and tested in Study 2, using an even larger sample that would allow for detecting smaller effects. Also, in Study 3 we tested for replicability of both BBS and BAE reported in the target article.
We briefly introduce the studies below, and detailed procedures for each of the three studies are reported in the Supplementary Materials section.
Study 1a and Study 1b
Undergraduate students taking a social psychology class at the University of Hong Kong designed the replication studies, and were invited to take part in their classmates' studies as participants. A total of 75 students enrolled in the class were invited to participate, and of those
49 undergraduate students chose to take part in the online survey. Of those, we excluded four students who designed Study 1a and six other students who designed Study 1b, resulting in a sample of 45 for Study 1a (Mage = 20.20, SD = 0.99; 31 females) and 43 for Study 1b (Mage =
20.23, SD = 1.04; 32 females).
Participants in Study 1a were presented with descriptions of eight biases and three personal shortcomings in randomized order: self-serving attributions for success or failures, dissonance reduction after free choice, positive halo effect, biased assimilation of new information, reactive devaluation of proposal from one’s negotiation counterparts, perceptions of hostile media bias toward one’s group or cause, fundamental attribution error, self-interest, procrastination, fear of public speaking, and planning fallacy. Full descriptions are detailed in the
Supplementary Materials. For each of the descriptions, participants rated their own susceptibility
Bias Blind Spot: Replications and extensions 13
(biases: 훼 = 0.58; personal shortcomings: 훼 = 0.71) and perceived susceptibility of the average student at the university (biases: 훼 = 0.73; personal shortcomings: 훼 = 0.58). Participants responded to the following: "Indicate to what extent you believe that [you/others] show this effect or tendency?" on a nine-point scale (1 = not at all; 5 = somewhat; 9 = strongly).
Participants in Study 1b were presented with three positive and three negative personality traits in randomized order. The positive personality dimensions assessed were dependability, objectivity, and consideration (훼 = 0.30). The negative personality dimensions assessed were snobbery, deceptiveness, and selfishness (훼 = 0.60). The ratings were made on a 9-point scale (1
= much lower than the average student; 5 = same as the average student; 9 = much higher than the average student).
After rating their personalities, participants were briefed about the better than average effect and asked to reassess their personalities (1 - Objective measures would rate me lower on positive characteristics and higher on negative characteristics than I rated myself; 2 - Objective measures would rate me neither more positively nor more negatively than I rated myself; 3 -
Objective measures would rate me higher on positive characteristics and lower on negative characteristics than I rated myself).
Study 2
A total of 310 American Amazon Mechanical Turk (MTurk) participants completed the study using TurkPrime.com (Litman et al., 2017). Of those, seven participants were excluded
Bias Blind Spot: Replications and extensions 14
1 based on pre-registered exclusion criteria , leaving 303 participants included in the analysis (Mage
= 38.45, SD = 11.58; 166 females).
Participants first rated their free will beliefs on two scales (randomized order) and then rated their own and others’ susceptibility to biases and personal shortcomings. Display of conditions was counterbalanced in a 2 (reference: self vs. others) by 2 (attributes: biases and personal shortcomings) within-subject design (see the Supplementary Materials for more details and full measures). The materials used for testing BAE and BBS effects in Study 2 were the same as those of Study 1a.
For the extension, free will beliefs were measured before the replication using two free will belief subscales: Free Will Inventory (FWI) (Nadelhoffer et al., 2014) (5 items; 1 = Strongly disagree, 7 = Strongly agree; 훼 = 0.91) and Free Will and Determinism (FWD) personal agency subscale (Rakos et al., 2008) (4 items; 1 = Not true at all, 5 = Almost always true; 훼 = 0.92). FWI was chosen as one of the most recently developed measurements of generalized free will beliefs, improving on many of the issues identified in earlier scales. FWD was chosen as it focuses specifically on free will beliefs in the self, which we thought might differ from the more generalized FWI scale. Full measures are provided in the Supplementary Materials.
1 We excluded 7 participants based on pre-registered criteria: self-reported not being serious about filling in the survey (2), not from the USA (6), and failed both the criteria (1).
Bias Blind Spot: Replications and extensions 15
Study 3
A total of 629 American MTurk participants completed the study using TurkPrime.com.
Of those, eight participants were excluded based on pre-registered exclusion criteria2, leaving 621 participants included in the analysis (Mage = 39.15, SDage = 11.88; 346 females). We prevented participants who took part in Study 2 from participating in Study 3.
The study included three parts: 1) free will beliefs, 2) replication of BBS effects (similar to Study 2), 3) replication of BAE (similar to Study 1b). Participants rated their own and other average Americans on susceptibility to biases (self: 훼 = 0.79; others: 훼 = 0.79) and personal shortcomings (self: 훼 = 0.49; others: 훼 = 0.55) (1 = not at all; 9 = strongly), and positive (훼 =
0.45) and negative personalities (훼 = 0.74) (1 = much lower than the average American; 5 = same as the average American; 9 = much higher than the average American).
In addition to the free will beliefs measures we collected in Study 2, we added an additional measure of generalized free will beliefs, using an older scale by Paulhus and Carey
(2011). This was meant to help reexamine our interpretation of the extension findings in Study 2 about differences between free will beliefs in self, and generalized free will beliefs.
Free will beliefs scales included were: Free Will Inventory (FWI; 5 items; Nadelhoffer et al., 2014) (1 = Strongly disagree, 7 = Strongly agree; 훼 = 0.89), Free Will and Determinism personal will sub-scales (FWD; 8 items; Rakos et al., 2008) (0 = Not true at all, 4 = Almost always true; 훼 = 0.74), and Free Will and Determinism plus scale (FAD+; 7 items; Paulhus &
2 We excluded 8 participants based on pre-registered criteria: self-reported not being serious about filling in the survey (2), and not from the USA (6).
Bias Blind Spot: Replications and extensions 16
Carey, 2011) (1 = Not at all true, 5 = Always true; 훼 = 0.85; recoded from a scale of 0 to 4 to match the original scale range).
Mini meta-analysis summary and evaluation criteria for replication design and findings
To summarize the replication results across the three samples, we conducted a mini meta- analysis of the effect sizes observed across the three replication studies (Goh et al., 2016; Lakens
& Etz, 2017). The mini meta-analysis is not only helpful in succinctly summarizing the findings across studies, but also helps obtaining a more precise estimate of the effect size (Braver et al.,
2014; Cumming, 2014).
We characterize the current replications as "very close replications" based on the framework for classification of replications using the criteria by LeBel et al. (2018) (see Table
S25 in the Supplementary Materials).
To interpret the replication results we followed the framework by LeBel et al. (2019).
They suggested a replication evaluation using three factors: (a) whether a signal was detected
(i.e., the confidence interval for the replication Effect size (ES) excludes zero), (b) consistency of the replication ES with the original study’s ES, and (c) precision of the replication’s ES estimate
(see Figure S18 in the supplementary material).
Results
We first conducted replication tests mirroring the tests conducted by Pronin et al. (2002).
Descriptive statistics are provided in Table 2. For the proposed extension, we calculated correlations between measures of free will beliefs and BBS measures. Detailed results for each of the three studies are reported in the Supplementary Materials section (see Table S3).
Bias Blind Spot: Replications and extensions 17
Susceptibility to biases and personal shortcomings
We conducted a series of one-tailed paired samples t-tests, and found that participants showed a general tendency to view themselves as less susceptible to biases across the three samples (Study 1a: Md = -0.75, t(44) = -4.54, p <.001, dz = -0.68, 95% CI [-1.01, -0.35]; Study 2:
Md = -1.15; t(302) = -16.16, p < .001; dz = -0.93, 95% CI [-1.06, -0.79]; Study 3: Md = -1.80; t(620) = -32.04, p <.001, dz = -1.29, 95% CI [-1.39, -1.18]). We conclude these findings to be in support of the BBS hypothesis and the original findings.
Contrary to the predictions made in the original study, we found support for differences in ratings of personal shortcomings comparing self and others in Study 2 (Md = -0.52; t(302) = -
5.22, p < .001; dz = -0.30 , 95% CI [-0.42, -0.18]) and Study 3 (Md = -0.73; t(620) = -10.54, p
<.001, dz = -0.42, 95% CI [-0.51, -0.34]), yet with the weaker effect failing to show support for differences in Study 1a (Md = -0.29; t(44) = -1.13, p = .265; dz = -0.17, 95% CI [-0.47, 0.13]).
Finally, we found support for differences between the self-other bias asymmetry and the self-other personal shortcomings asymmetry across all three samples (Study 1a: Md = -0.46, t(44)
= -1.97, p = .055, dz = -0.29, 95% CI [-0.60, 0.01]; Study 2: Md = -0.62; t (302) = -6.39, p < .001, dz = -0.37, 95% CI [-0.48, -0.25]; Study 3: Md = -1.06; t (620) = -13.01, p <.001, dz = -0.52, 95%
CI [-0.61, -0.44]).
Denying personal susceptibility to the better than average effect
We conducted one-sample one-tailed t-tests and found that participants rated themselves as having personality characteristics that are more positive (Study 1b: M = 5.74, t(42) = 5.09, p <
.001, dz = 0.78, 95% CI [0.43, 1.11]; Study 3: M = 6.42; t(620) = 31.74, p < .001; dz = 1.27, 95%
CI [1.17, 1.38]) and less negative (Study 1b: M = 4.16, t(42) = -4.55, p < .001, dz = -0.69, 95% CI
Bias Blind Spot: Replications and extensions 18
[-1.02, -0.36]; Study 3: M = 3.21; t (620) = -30.38, p <.001; dz = -1.22, 95% CI [-1.32, -1.11]) than others'. We conclude support of the better than average effect.
Bias Blind Spot: Replications and extensions 19
Table 2 Descriptive statistics for all studies
Mean S.D. Skewness Kurtosis Study 1a (N = 45) Self: biases 5.60 0.86 0.69 1.00 Self: personal shortcomings 6.20 1.78 -0.30 -1.02 Others: biases 6.35 0.91 -0.04 -0.05 Other: personal shortcomings 6.49 1.23 -0.48 0.04 Study 1b (N = 43) Positive personality dimensions 5.74 0.96 -0.18 -0.16 Negative personality dimensions 4.16 1.22 0.27 -0.11 Denial of bias 1.98 0.64 0.02 -0.38 Study 2 (N = 303) Self: biases 4.64 1.35 -0.22 0.05 Self: personal shortcomings 5.35 1.88 -0.28 -0.40 Others: biases 5.78 1.16 -0.15 0.73 Others: personal shortcomings 5.87 1.35 -0.03 -0.04 Study 3 (N = 621) Self: biases 4.69 1.30 0.02 0.29 Self: personal shortcomings 5.52 1.71 -0.12 -0.47 Others: biases 6.48 1.04 -0.29 0.28 Others: personal shortcomings 6.25 1.16 -0.04 -0.37 Positive personality dimensions 6.43 1.11 -0.11 0.22 Negative personality dimensions 3.22 1.47 0.47 -0.12 Denial of bias 2.07 0.65 -0.07 -0.62
Bias Blind Spot: Replications and extensions 20
We then proceeded to conduct chi-squared tests to examine denial of susceptibility to the better than average effect, comparing to a 50%-50% split. In Study 1b, we found that despite being made aware of their potential bias, only 9 of the 43 participants (21%) acknowledged any potential bias, leaving 79% of participants still claiming to be better than their average peers (χ2
(1, N = 43) = 14.53, p < .001, dz = 1.43, 95% CI [0.69, 2.16]). We found similar results in Study
3, with only 109 of the 621 participants (18%) admitting bias, leaving 82% denying the bias (χ2
(1, N = 621) = 261.53, p < .001; dz = 1.71, 95% CI [1.50, 1.91]).
We conclude that our findings are in support of denial of the better than average effect bias. The effect size of the replications (Study 1b: dz = 1.43; Study 3: dz = 1.27) was around twice the effect size of the original study (dz = 0.70).
Combining the findings of the three replications in a mini-meta analysis, we summarize the following: (1) participants rated themselves as less susceptible to biases than others, (2) participants rated themselves as less prone to personal shortcomings than others, (3) self-other differences for biases were larger than self-other differences for personal shortcomings, (4) participants rated themselves as possessing more positive personality characteristics and less negative personality characteristics than others, and (5) participants denied correcting for the better than average bias even after having been made aware of it.
Extension: Free will beliefs and shortcomings
We examined the link between free will beliefs and perceived personal shortcomings in self and others. We summarized our findings in Table 3.
All free will beliefs scales were negatively correlated with perceived personal shortcomings in self (r ~ -0.22 [-0.32, -0.11] to -0.09 [-0.17, -0.01]). However, for four out of
Bias Blind Spot: Replications and extensions 21 five free will beliefs measures, we found no support for a link between free will beliefs and perceived personal shortcomings in others (r ~ 0.00 [-0.11, 0.12] to 0.05 [-0.03, 0.13]; FWD in
Study 3 r ~ 0.11 [0.03, 0.18]). These findings seem to indicate that free will beliefs correlate more strongly with perceived shortcomings in self than in others. Supporting this notion, we examined perceived shortcomings in self and in others together, and free will beliefs negatively correlated with personal shortcomings self-other asymmetry (r ~ -0.24 [-0.34, -0.13] to -0.16 [-
0.24, -0.09]).
Additional exploratory findings are reported in the Supplementary Materials (See Table
S24). These suggested that stronger perceptions of personal free will were associated with stronger bias blind spot, but associations were much weaker for generalized free will beliefs.
Bias Blind Spot: Replications and extensions 22
Table 3
Summary of the extensions' hypotheses (confirmatory hypotheses)
Predictions Effect size and CIs (r) NHST p Summary Study 2: United States (N = 303) BFW and personal shortcomings: Self FWI -0.22 [-.32, -.11] < .001 Supported FWD -0.17 [-.28, -.06] 0.003 Supported BFW and personal shortcomings: Others FWI 0.00 [-.11, .12] 0.941 Not supported FWD 0.05 [-.06, .16] 0.357 Not supported BFW and personal shortcomings: Self-other differences FWI -0.24 [-.34, -.13] < .001 Supported FWD -0.22 [-.33, -.11] < .001 Supported Study 3: United States (N = 621) BFW and personal shortcomings: Self FWI -0.16 [-.23, -.08] < .001 Supported FAD+ -0.15 [-.22, -.07] < .001 Supported FWD -0.09 [-.17, -.01] 0.022 Supported BFW and personal shortcomings: Others FWI 0.02 [-.05, .10] 0.551 Not supported FAD+ 0.05 [-.03, .13] 0.231 Not supported FWD 0.11 [.03, .18] 0.007 Supported BFW and personal shortcomings: Self-other differences FWI -0.17 [-.25, -.09] < .001 Supported FAD+ -0.18 [-.25, -.10] < .001 Supported FWD -0.16 [-.24, -.09] < .001 Supported Note. r = Pearson correlation coefficient. BFW = belief in free will; FWI = Nadelhoffer et al. (2014); FAD+ = Paulhus and Carey (2011); FWD = Rakos, Laurene, Skala, and Slane, (2008). Hypotheses and related findings regarding BBS in susceptibility to bias are provided in the Supplementary Materials.
Bias Blind Spot: Replications and extensions 23
General summary: Mini meta-analysis of replication findings
We summarized the replications of Pronin et al. (2002) across the three samples using a mini meta-analysis in Table 4, and plotted the results in Figure 1. We ran the mini meta-analysis with a random-effects model method using the meta package in R (Schwarzer, 2007; see Figure
S15-S16 for detailed forest plots in the Supplementary Materials section).
We found support for self-other differences in perceived susceptibility to bias (d = -1.00 [-
1.33, -0.67]), self-other differences in perceived personal shortcomings (d = -0.34 [-0.46, -0.23]), and the effects were stronger for perceived susceptibility to bias compared to perceived personal shortcomings (d = -0.43 [-0.56, -0.29]). We found support for better than average effects, with higher rating for positive personality dimensions (d = 1.05 [0.57, 1.54]), negative personality dimensions (d = -0.98 [-1.49, -0.47]), and denial of better than average effect (d = 1.69 [1.49,
1.88]).
Our evaluation of the replication findings based on the LeBel et al. (2019) criteria is summarized in the right column in Table 4 (separate analyses for each study are provided Table
S4 of the Supplementary Materials). We conclude strong support for the main hypotheses of the target article, with effects comparable and at times larger than the original's (see Figure 1).
Bias Blind Spot: Replications and extensions 24
Table 4
Summary and comparison of mini meta-analysis effects of the replication to the original article
Replications mini- Original article NHST Replications meta-analysis Cohen's d summary summary Cohen's d Replication summary Bias blind spot (BBS) Self-other asymmetry in susceptibility to biases -1.00 [-1.33, -0.67] -0.86 [-1.28, -0.43] Supported Signal - consistent Signal - inconsistent, Self-other asymmetry in personal shortcomings -0.34 [-0.46, -0.23] 0.28 [-0.10, 0.65] Not supported negative effect Differences in self-other asymmetry between -0.43 [-0.56, -0.29] -0.61 [-1.01, -0.21] Supported Signal - inconsistent, smaller susceptibility to biases and personal shortcomings Better than average effects Positive personality dimensions compared to others 1.05 [0.57, 1.54] 1.60 [1.26, 1.93] Supported Signal - inconsistent, larger
Negative personality dimensions compared to others -0.98 [-1.49, -0.47] -1.23[-1.52, -0.94] Supported Signal - consistent Denial of better than average effect 1.69 [1.49, 1.88] 0.76 [0.29, 1.23] Supported Signal - inconsistent, larger Extension hypothesis- correlations with Belief in free Summary of extension Range of effect size NHST results will hypothesis BFW and personal shortcomings: Self -0.09 [-.17, -.01] to -0.22 [-.32, -.11] Supported Supported BFW and personal shortcomings: Other 0.00 [-.11, .12] to 0.11 [.03, .18] Not- supported Not supported BFW and personal shortcomings: Self-other differences -0.16 [-.24, -.09] to -0.24 [-.34, -.13] Supported Supported Note. Effect size is Cohen's d. Replication summary directly based on LeBel et al., (2018) that considered the following three statistical aspects: (1) whether a signal was detected (i.e., whether the 95% confidence interval, or CI, represented here by the error bars, excludes 0); (2) the consistency of the replication effect-size (ES) estimate with that observed in the original study (i.e., whether the replication's CI includes the original ES point estimate); and (3) the precision of the replication's ES estimate (i.e., the width of its CI relative to the CI in the original study). NHST summary is based on the interpretation of a one-tail p-value with alpha set at 0.05. See Table S3-S4 (Figure S15-S16) of the Supplementary Materials for detailed results of three replication studies. BFW = belief in free will.
Bias Blind Spot: Replications and extensions 25
Figure 1
Comparison of findings in Pronin et al. (2002) to the replications' mini meta-analysis summary
Discussion
Replications of the bias blind spot
We conducted three close replication studies of the bias blind spot effect. Our findings were in support of the main predictions of the BBS effect. The findings of our replication samples are consistent in the samples from USA and Hong Kong. The current replication contributes to the theory development of BBS effects for three main reasons.
Firstly, we empirically qualify the theoretical assertions of Pronin et al. (2002). We addressed one of the potential methodological shortcomings, that is, the underpowered samples in the original study. Using a larger sample of participants, we found support for self-other differences for personal shortcomings, whereas the original article hypothesized and argued findings were in support of a null effect. It is very likely that the null findings reported in the target article were mostly due to insufficient power. Specifically, in their Study 1, the authors expected weaker effects for self-other differences for personal shortcomings than for bias, yet failed to provide evidence for rejecting the null findings. We, on the other hand, did find support for self-other asymmetry in personal shortcomings. Pronin and colleagues (2002) interpreted the findings as being in support of the null, a demonstration that participants are as aware of their own personal shortcomings as they are of others'. Our replication findings suggest that there exists a bias also for personal shortcomings.
Pronin et al. (2002) failed to reject the null regarding specific hypotheses, and so their conclusion was that "in these cases, participants saw themselves as being just as flawed as their peers, if not more so" (p. 372), hinting about support for the null. We found support for rejecting the null regarding that specific case. Replicators often focus on findings that were able to reject
Bias Blind Spot: Replications and extensions 27 the null hypothesis, which is understandable given that the vast majority of findings in published articles until 2011 were about rejecting the null. Yet, our findings here suggest that there might be value in attempting replications for null effects, in cases where an effect was found in one domain but not in another related domain, and especially so when the original sample size was severely underpowered and there were reasons to believe there should have been an effect.
Despite the above divergence in findings, we believe that the theoretical grounds of the target paper remain relevant and that the replications are in support of the original theory and findings, as the replication results lend support for the main introspection hypothesis that self- other differences are larger for susceptibility to bias than for personal shortcoming. Our main recommendation for those studying the BBS is to revise the original conclusions for a null effect for personal shortcomings into expecting an effect, though a weaker effect than for biases.
Secondly, we contribute to understanding of the BBS effect in a cross-cultural setting. We find support for BBS effects across samples from the USA and Hong Kong. For instance, Study 1 used a sample of non-naive students from Hong Kong, who were familiar with heuristics and biases in judgment and decision-making. The sample was slightly underpowered, and yet we were able to detect the effects reported in the classic article. Despite knowledge of other biases, and despite possible cultural differences, the findings were highly consistent with the findings in the classic article and the two MTurk samples. We advise caution in over-interpreting these, given the small sample and the singular bias under examination, yet it offers some support for
BBS being robust enough to withstand concerns such as non-naivete and cross-cultural differences. Concerns around cultural relativity of the findings in psychology seems to be less of a concern with BBS effects (Gergen, 1973; Henrich et al., 2010).
Bias Blind Spot: Replications and extensions 28
Thirdly, going beyond the BBS, we believe that our investigation offers additional insights into the BBS effect. In Studies 2 and 3, we examined the link between free will beliefs and the bias blind spot effects (see summary in Table 3). Overall, our findings supported our predictions that belief in free will would be associated with stronger self-other differences in perceived personal shortcomings (r ~ 0.16–0.24). The differences between self and other ratings of personal shortcomings were mostly driven by the negative association between free will beliefs and personal shortcomings in self. We also found support for an association between free will beliefs and the BBS in susceptibility to bias (r ~ 0.17–0.22), but only with for the free will beliefs scale that concerned the self, rather than in general. In Study 2, these findings were discovered in exploratory analyses, and in Study 3 we pre-registered the predictions and found very consistent effects, suggesting that these were not accidental.
The results suggest that free will belief associations are more about viewing oneself to be free of biases and shortcomings, and less about viewing others as more susceptible to biases and shortcomings. This indicates that free will beliefs in reference to the self are related to the nature of introspection accessed, complementing previous work that argued that free will beliefs impact fundamental socio-cognitive processes (Genschow et al., 2017). Our take-aways from the extensions are the following: 1) free will beliefs are associated with self-other differences in perceived personal shortcoming, 2) free will beliefs in reference to the self are associated with
BBS regarding susceptibility to bias, and 3) different free will beliefs scales gauge different aspects of agency. We advise free will beliefs researchers to incorporate both types of scales and make predictions differentiating between personal and generalized free will beliefs. It is possible that these differences would be especially relevant for biases regarding differences in attributions
Bias Blind Spot: Replications and extensions 29 or perceptions comparing self to others. Overall, given the findings, we see much promise in the direction of examining links between free will beliefs and attribution biases.
It is important to note that our evidence for the relation between belief in free will and
BBS effects is correlational only. Therefore, current findings do not allow for causal inferences.
However, future research may aim to reproduce our findings by implementing experimental designs that manipulate free will beliefs. Furthermore, future research could extend current work in various ways. For example, future work can explore the role of belief in free will on the tendency to attribute responsibility or blame to people susceptible to cognitive biases.
Bias Blind Spot: Replications and extensions 30
References
Alquist, J. L., Ainsworth, S. E., Baumeister, R. F., Daly, M., & Stillman, T. F. (2015). The
making of might-have-beens: Effects of free will belief on counterfactual
thinking. Personality and Social Psychology Bulletin, 41(2), 268-283.
Baumeister, R. F. (2008). Free Will in Scientific Psychology. Perspectives on Psychological
Science, 3, 14–19. doi:10.1111/j.1745-6916.2008.00057.x
Baumeister, R. F., & Monroe, A. E. (2014). Recent research on free will: Conceptualizations,
beliefs, and processes. In Advances in Experimental Social Psychology (Vol. 50, pp. 1-
52). Academic Press. DOI: 10.1016/B978-0-12-800284-1.00001-1
Brandt, M. J., I Jzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., ... &
Van't Veer, A. (2014). The replication recipe: What makes for a convincing
replication?. Journal of Experimental Social Psychology, 50, 217-224. DOI:
10.1016/j.jesp.2013.10.005
Braver, S. L., Thoemmes, F. J., & Rosenthal, R. (2014). Continuously cumulating meta-analysis
and replicability. Perspectives on Psychological Science, 9(3), 333-342.
Crescioni, A. W., Baumeister, R. F., Ainsworth, S. E., Ent, M., & Lambert, N. M. (2016).
Subjective correlates and consequences of belief in free will. Philosophical Psychology,
29, 41–63. DOI: 10.1080/09515089.2014.996285
Cumming, G. (2014). The new statistics: Why and how. Psychological science, 25(1), 7-29.
Dana, J., & Loewenstein, G. (2003). A social science perspective on gifts to physicians from
industry. Jama, 290(2), 252-255. DOI: 10.1001/jama.290.2.252
Bias Blind Spot: Replications and extensions 31
Dunning, D., Meyerowitz, J. A., & Holzberg, A. D. (1989). Ambiguity and self-evaluation: The
role of idiosyncratic trait definitions in self-serving assessments of ability. Journal of
personality and social psychology, 57(6), 1082-1090. DOI: 10.1037/0022-3514.57.6.1082
Ehrlinger, J., Gilovich, T., & Ross, L. (2005). Peering into the bias blind spot: People’s
assessments of bias in themselves and others. Personality and Social Psychology Bulletin,
31, 680–692. DOI: 10.1177/0146167204271570
Feldman, G. (2017). Making sense of agency: Belief in free will as a unique and important
construct. Social and Personality Psychology Compass, 11, 1-15. DOI:
10.1111/spc3.12293
Feldman, G., Baumeister, R. F., & Wong, K. F. E. (2014). Free will is about choosing: The link
between choice and the belief in free will. Journal of Experimental Social Psychology, 55,
239-245. DOI: 10.1016/j.jesp.2014.07.012
Feldman, G., Chandrashekar, S. P., & Wong, K. F. E. (2016). The freedom to excel: Belief in
free will predicts better academic performance. Personality and Individual Differences,
90, 377–383. DOI: 10.17605/OSF.IO/J4K9Z
Feldman, G., Farh, J. L., & Wong, K. F. E. (2018). Agency beliefs over time and across cultures:
Free will beliefs predict higher job satisfaction. Personality and Social Psychology
Bulletin, 44, 304–317. DOI: 10.17605/OSF.IO/C6WM5
Fillon, A., Lantian, A., Feldman, G., & N'gbala, A. (2021) Exceptionality Effect in Agency:
Exceptional Choices Attributed Higher Free Will Than Routine. Manuscript under review.
Preprint retrieved March 2021 from https://psyarxiv.com/sep8f/
Bias Blind Spot: Replications and extensions 32
Genschow, O., Rigoni, D., & Brass, M. (2017). Belief in free will affects causal attributions when
judging others’ behavior. Proceedings of the National Academy of Sciences, 114, 10071–
10076. doi:10.1073/pnas.1701916114
Gergen, K. J. (1973). Social psychology as history. Journal of personality and social
psychology, 26(2), 309–320.
Glöckner, A., & Betsch, T. (2011). The empirical content of theories in judgment and decision
making: Shortcomings and remedies. Judgment & Decision Making, 6(8), 711–721.
Goh, J. X., Hall, J. A., & Rosenthal, R. (2016). Mini meta‐analysis of your own studies: Some
arguments on why and a primer on how. Social and Personality Psychology
Compass, 10(10), 535-549.
Hansen, K., Gerbasi, M., Todorov, A., Kruse, E., & Pronin, E. (2014). People claim objectivity
after knowingly using biased strategies. Personality and Social Psychology Bulletin, 40,
691-699. DOI: 10.1177/0146167214523476
Henrich, J., Heine, S. J., & Norenzayan, A. (2010). Most people are not
WEIRD. Nature, 466(7302), 29-29.
Jones, E. E. (1990). Interpersonal perception. New York: Freeman
Jones, E. E., & Nisbett, R. E. (1972). The actor and the observer: Divergent perceptions of the
causes of behavior. In E. E. Jones et al. (Eds.), Attribution: Perceiving the causes of
behavior. Morristown, NJ: General Learning Press.
Kunda, Z. (1990). The case for motivated reasoning. Psychological bulletin, 108(3), 480-498.
Bias Blind Spot: Replications and extensions 33
Lakens, D., & Etz, A. J. (2017). Too true to be bad: When sets of studies with significant and
nonsignificant findings are probably true. Social Psychological and Personality
Science, 8(8), 875-881.
LeBel, E. P., McCarthy, R. J., Earp, B. D., Elson, M., & Vanpaemel, W. (2018). A unified
framework to quantify the credibility of scientific findings. Advances in Methods and
Practices in Psychological Science, 1(3), 389-402.
LeBel, E. P., Vanpaemel, W., Cheung, I., & Campbell, L. (2019). A brief guide to evaluate
replications. Meta-Psychology, 3.
Litman, L., Robinson, J., & Abberbock, T. (2017). TurkPrime. com: A versatile crowdsourcing
data acquisition platform for the behavioral sciences. Behavior research methods, 49(2),
433-442.
Miller, D. T., & Ross, M. (1975). Self-serving biases in the attribution of causality: Fact or
fiction?. Psychological bulletin, 82(2), 213-225. DOI: 10.1037/h0076486
Monroe, A. E., & Malle, B. F. (2010). From uncaused will to conscious choice: The need to
study, not speculate about people’s folk concept of free will. Review of Philosophy and
Psychology, 1(2), 211-224. DOI: 10.1007/s13164-009-0010-7
Monroe, A. E., & Malle, B. F. (2014). Free will without metaphysics. In A. R. Mele (Ed.),
Surrounding free will. New York, NY: Oxford University Press
Monroe, A. E., Dillon, K. D., & Malle, B. F. (2014). Bringing free will down to Earth: People’s
psychological concept of free will and its role in moral judgment. Consciousness and
cognition, 27, 100-108. doi: 10.1016/j.concog.2014.04.011
Bias Blind Spot: Replications and extensions 34
Nadelhoffer, T., Shepard, J., Nahmias, E., Sripada, C., & Ross, L. T. (2014). The free will
inventory: Measuring beliefs about agency and responsibility. Consciousness and
cognition, 25, 27-41.
Nisbett, R. E., & Ross, L. (1980). Human inference: Strategies and shortcomings of social
judgment. Englewood Cliffs, NJ: Prentice Hall.
Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports on mental
processes. Psychological Review, 8: 231–259. DOI: 10.1037/0033-295X.84.3.231
Open Science Collaboration. (2015, August 28). Estimating the reproducibility of psychological
science. Science, 349. DOI:10.1126/science.aac4716
Paulhus, D. L., & Carey, J. M. (2011). The FAD–Plus: Measuring lay beliefs regarding free will
and related constructs. Journal of personality assessment, 93(1), 96-104.
Pronin, E. (2007). Perception and misperception of bias in human judgment. Trends in Cognitive
Sciences, 11, 37–43. doi:10.1016/j.tics.2006.11.001
Pronin, E., & Kugler, M. B. (2007). Valuing thoughts, ignoring behavior: The introspection
illusion as a source of the bias blind spot. Journal of Experimental Social Psychology,
43(4), 565–578. DOI: 10.1016/j.jesp.2006.05.011
Pronin, E., & Kugler, M. B. (2010). People believe they have more free will than
others. Proceedings of the National Academy of Sciences, 107(52), 22469-22474. doi:
10.1073/pnas.1012046108
Bias Blind Spot: Replications and extensions 35
Pronin, E., Gilovich, T., & Ross, L. (2004). Objectivity in the eye of the beholder: Divergent
perceptions of bias in self versus others. Psychological Review, 111(3), 781-799. DOI:
10.1037/0033-295X.111.3.781
Pronin, E., Lin, D. Y., & Ross, L. (2002). The bias blind spot: Perceptions of bias in self-versus
others. Personality and Social Psychology Bulletin, 28, 369–381. DOI:
10.1177/0146167202286008
Protzko, J., Ouimette, B., & Schooler, J. (2016). Believing there is no free will corrupts intuitive
cooperation. Cognition, 151, 6–9. DOI:10.1016/j.cognition.2016.02.014
Rakos, R. F., Laurene, K. R., Skala, S., & Slane, S. (2008). Belief in free will: Measurement and
conceptualization innovations. Behavior and Social Issues, 17(1), 20-40.
Ross, L., & Ward, A. (1995). Psychological Barriers to Dispute Resolution. In M. P. Zanna (Ed.),
Advances in Experimental Social Psychology (Vol. 27, pp. 255–304). Academic Press.
doi:10.1016/S0065-2601(08)60407-4
Ross, L., & Ward, A. (1996). Naive realism in everyday life: Implications for social conflict and
misunderstanding. In T. Brown, E. S. Reed, & E. Turiel (Eds.), Values and knowledge.
The Jean Piaget symposium series (pp. 103–135). Hillsdale, NJ: Erlbaum.
Schwarzer, G. (2007). meta: An R package for meta-analysis. R news, 7(3), 40-45.
Scopelliti, I., Morewedge, C. K., McCormick, E., Min, H. L., Lebrecht, S., & Kassam, K. S.
(2015). Bias Blind Spot: Structure, Measurement, and Consequences. Management
Science, 61, 2468–2486. doi:10.1287/mnsc.2014.2096
Bias Blind Spot: Replications and extensions 36
Stillman, T. F., Baumeister, R. F., Vohs, K. D., Lambert, N. M., Fincham, F. D., & Brewer, L. E.
(2010). Personal philosophy and personnel achievement: Belief in free will predicts better
job performance. Social Psychological and Personality Science, 1, 43–50. DOI:
10.1177/1948550609351600
Uhlmann, E. L., & Cohen, G. L. (2007). “I think it, therefore it’s true”: Effects of self-perceived
objectivity on hiring discrimination. Organizational Behavior and Human Decision
Processes, 104(2), 207-223. DOI: 10.1016/j.obhdp.2007.07.001
Van't Veer, A. E., & Giner-Sorolla, R. (2016). Pre-registration in social psychology—A
discussion and suggested template. Journal of Experimental Social Psychology, 67, 2-12.
DOI: 10.1016/j.jesp.2016.03.004
Vonasch, A. J., Baumeister, R. F., & Mele, A. R. (2018). Ordinary people think free will is a lack
of constraint, not the presence of a soul. Consciousness and cognition, 60, 133-151. DOI:
10.1016/j.concog.2018.03.002
West, R. F., Meserve, R. J., & Stanovich, K. E. (2012). Cognitive sophistication does not
attenuate the bias blind spot. Journal of personality and social psychology, 103(3), 506-
519. DOI: 10.1037/a0028857
Wilson, T. D., & Dunn, E. W. (2004). Self-knowledge: Its limits, value, and potential for
improvement. Annual Review of Psychology, 55, 493-518. DOI:
10.1146/annurev.psych.55.090902.141954
Zwaan, R. A., Etz, A., Lucas, R. E., & Donnellan, M. B. (2018). Making replication
mainstream. Behavioral and Brain Sciences, e120. DOI: 10.1017/S0140525X17001972
Pronin et al. (2002) replication and extension: Supplementary 1
Pronin et al. (2002) replication + extension: Supplementary materials
Contents Power analyses ...... 4 Study 1a ...... 4 Study 1b...... 4 Study 2...... 4 Study 3...... 4 Open Science ...... 5 Data and code ...... 5 Pre-registrations and Qualtrics study designs ...... 5 Procedure and data disclosures...... 5 Data collection ...... 5 Conditions reporting ...... 5 Data exclusions ...... 5 Variables reporting...... 5 Adjustments to original design ...... 5 Measures used in the experiments ...... 6 Perceived biases and personal shortcomings...... 6 Perceived susceptibility to biases and personal shortcomings (of self) ...... 6 Perceived susceptibility to biases and personal shortcomings (of others) ...... 6 Biases ...... 6 Personal Shortcomings ...... 7 Belief in Free Will scales ...... 7 Scale 1: The Free Will Inventory (Nadelhoffer, Shepard, Nahmias, Sripada, & Ross, 2014) ...... 7 Scale 2: Belief in Free Will Scale- personal agency items (Rakos, Laurene, Skala & Slane, 2008) ...... 8 Scale 3: Free Will and Determinism-personal will subscales of FWD scale (Rakos et al., 2008) ...... 8 Scale 4: Free Will and Determinism Plus scale (FAD+; Paulhus & Carey, 2011) ...... 8 Scale of susceptibility to better-than-average effect...... 9 Summary Tables Across Study 1, Study 2, and Study 3 ...... 10 Table S3 ...... 11 Table S4 ...... 13 Study 1a ...... 15 Table S5 ...... 15 Table S6 ...... 16 Figure S1...... 17 Figure S2...... 18 Table S7 ...... 21 Study 1b...... 22 Table S8 ...... 22 Table S9 ...... 23 Study 2...... 24 Table S10 ...... 24 Table S11 ...... 25 Table S12 ...... 26 Table S13 ...... 31 Table S14 ...... 32 Table S15 ...... 33 Study 3...... 34 Exploratory analyses ...... 34 Additional tables and figures ...... 35 Table S16 ...... 35 Table S17 ...... 36 Table S18 ...... 37 Table S19 ...... 42 Table S20 ...... 43 Table S21 ...... 44 Table S22 ...... 45 Table S23 ...... 46 Summary and Mini-meta analysis...... 47 Mini-meta analysis including the original study ...... 47 Mini-meta analysis excluding the original study ...... 49 Summary of findings on ‘bias blind spot’ across 3 studies ...... 51 Summary of extension hypotheses ...... 52 Table S24 ...... 52 Framework for evaluation of the replications ...... 53 References ...... 55
Power analyses Study 1a Pronin et al. (2002) study that compared participants’ rating on their own biases, personal shortcomings, and self-other differences between biases and personal shortcomings with classmates’ biases (within- subjects design). The effect sizes of the original study were directly calculated from the t-value and the number of participants using the formula suggested by Rosenthal (1991). The effects size (Cohen d) for the differences in susceptibility to biases between self and referent other in the original study (Study 1 Survey 2; Pronin et al., 2002) was .86. For two other contrasts (personal shortcomings, and difference in self-other difference between biases and personal shortcomings with classmates’ biases) the effect sizes were (Cohen d) .28 and .62, respectively. Using G*Power alpha = .05, one-tail (as the direction of hypothesis known), d = 0.5 and power .95 we required sample size of 138. The final sample was 46. Study 1b Study 2 of Pronin et al. (2002) tested participants’ susceptibility for better than average effect. The original study contained three findings that suggested support toward participants’ susceptibility for better than average effect. The effect sizes for each of the predictions were: Positive traits in comparison to others (one-sample t-test; Cohen’s d = 1.61); Negative traits in comparison to others (one sample t-test; Cohen’s d = -1.24); Better-than-average effect (Chi-squared test; Cohen’s d= 0.76). Using G*Power alpha = .05, one-tail (the direction of hypothesis known), d = 0.76 and power .95 we required sample size of 21. The final sample was 43. Study 2 The data for the Experiments 1 and 2 were collected around the same time. We took the approach to conduct replication based on a larger sample. Further, as part of the extension of the replication we pre- registered the hypothesis between belief in free will and susceptibility to personal shortcomings. We based the sample size on the Pearson Correlation Coefficient of 0.21, the average correlation coefficient in Social Psychology. Using GPower 3.1.9.2 for.95 power and.05 alpha with an estimated correlation coefficient of .21 we required a sample size of 289. The final sample was 303. Study 3 The data for Study 3 was collected after conducting analyzing the data from Study 1 and Study 2. The aim of Study 3 was to replicate the original finding with a larger sample and also to test the pre-registered extension hypothesis (with sufficient power) that were documented after analyzing data from Study 2.
We based the sample size on the Pearson Correlation Coefficient of 0.21, the average correlation coefficient in Social Psychology. Using GPower 3.1.9.2 for.95 power and.05 alpha with an estimated correlation coefficient of .17 we required a sample size of 439. The final sample was 621.
Open Science Data and code Data and code will be shared using the Open Science Framework. Review link for data and code of all studies: https://osf.io/3df5s/ Pre-registrations and Qualtrics study designs Study 1a - https://osf.io/f4rb6/; Study 1b - https://osf.io/tez7m/; Study 2 - https://osf.io/4m35x/; Study 3 - https://osf.io/kqatx/. Procedure and data disclosures Data collection Data collection was completed before conducting an analysis of the data.
The data collection for Studies 1a and 1b were combined with data collection of 11 other pre-registered replications, in randomized order.
Study 2 data collection was combined with data collection of other pre-registered replications, in randomized order.
Study 3 was pre-registered and ran data collection with other research on free will beliefs, unrelated to the topic of biases. See pre-registration for Study 3 to learn about the other projects. Conditions reporting All collected conditions are reported. Data exclusions Details are reported in each of the two studies in the materials section of this document Variables reporting All variables collected for this study are reported and included in the provided data. Adjustments to original design We attempted a close replication of the original study by Pronin et al. (2002) yet made several needed adjustments. First, we administered all surveys via an online Qualtrics survey. Second, the two studies from Pronin et al. (2002) chosen for replication included Stanford University undergraduate students and were not paid for completion of the study. The current replication effort of three studies included one undergraduate sample from a university in HK, and two paid samples using Amazon Mechanical Turk (MTurk). Thirdly, Study 2 and Study 3 replications combined the original article' Studies 1 and 2 into an integrated design. Forth, we went beyond the replication and added extensions to the original design to examine the link between agency beliefs and self-other bias asymmetries.
Measures used in the experiments Perceived biases and personal shortcomings In this part, you will be asked to evaluate different aspects of yourself and others, and will then be asked to reflect back on your answers.
Remember, there are no right or wrong answers, so answer to the best of your understanding based on what you believe to be true.
Perceived susceptibility to biases and personal shortcomings (of self) Instruction: In the following, please to read descriptions of human tendencies and biases and indicate to what extent do you believe that you show this effect or tendency? (1 = Not at all; 5 = Somewhat; 9 = Strongly) Perceived susceptibility to biases and personal shortcomings (of others) Instruction: In the following, please to read descriptions of human tendencies and biases and indicate to what extent do you believe that others show this effect or tendency? (1 = Not at all; 5 = Somewhat, 9 = Strongly) Biases Self-Interest Bias Psychologists have claimed that some people show a “self-interest” effect in the way they view political candidates. That is, people’s assessments of qualifications, and their judgments about the extent to which particular candidates would pursue policies good for the American people as a whole, are influenced by their feelings about whether the candidates’ policies would serve their own particular economic and social interests. Reactive Devaluation Psychologists have claimed that some people on both sides of a negotiation show a “reactive devaluation” tendency in their view of the opposing side’s offers for compromise. That is, each side views the other’s compromise proposal less positively than they would view that same proposal if it were presented by a neutral third-party, or if it were not offered at all. Assimilation Bias Psychologists have claimed that some people show a “disconfirmation” tendency in the way they evaluate research about potentially dangerous habits. That is, they are more critical and skeptical in evaluating evidence of danger when they personally engage in such behavior than when they do not personally engage in such behavior. Dissonance Reduction Psychologists have claimed that some people show a “rationalizing” tendency in the way they feel about things they buy. That is, after choosing to buy a particular product, and thereby foregoing other alternatives, people are inclined to see the product they bought as higher in quality, more attractive in its features, and a better “deal” than the alternatives. Self-Serving Bias Psychologists have claimed that some people show a “self-serving” tendency in the way they view their academic or job performance. That is, they tend to take credit for success but deny responsibility for failure; they see their successes as the result of personal qualities, like drive or ability, but their failures as the result of external factors, like unreasonable work requirements or inadequate instruction. Fundamental Attribution Error Psychologists have claimed that some people show a tendency to make “overly dispositional inferences” in the way they view victims of assault crimes. That is, they are overly inclined to view the victim’s plight as one he or she brought on by carelessness, foolishness, misbehaviour, or naiveté. Hostile Media Effect Psychologists have claimed that some people show a “hostile media” effect in the way they view television or newspaper coverage of controversial or partisan issues. That is, they tend to perceive neutral media reports as presenting an inaccurately favourable view of the other side, and an inaccurately negative view of their own side. As a result, they see the media as “hostile,” or “biased against” their side. Positive Halo Effect Psychologists have claimed that some people show a “halo” effect in the way they form impressions of attractive people. That is, when it comes to assessing how nice, interesting, social, able, etc. someone is, people tend to judge an attractive person more positively than he or she deserves. Personal Shortcomings Planning Fallacy Psychologists have claimed that some people show an “optimistic planning” tendency in the way they complete work-related projects. That is, they tend to underestimate the amount of time it takes to get a job done and fail to budget adequate time for themselves. Procrastination Psychologists have claimed that some people show a “delay of effort” tendency in the way they approach challenging or strenuous tasks. That is, they tend to put off starting or finishing projects which they believe will be undesirably time-consuming, arduous, or likely to end in failure. Fear of Public Speaking Psychologists have claimed that some people show a “speaking apprehension” effect in the way they perceive situations involving talking in front of others. That is, they tend to be fearful or nervous when giving an oral presentation or addressing a group of people.
Belief in Free Will scales Scale 1: The Free Will Inventory (Nadelhoffer, Shepard, Nahmias, Sripada, & Ross, 2014) (To what extent do you agree with the following statements?) (7-point scale; 1 = Strongly disagree, 7 = Strongly agree)
1. People always have the ability to do otherwise. 2. People always have free will. 3. How people’s lives unfold is completely up to them. 4. People ultimately have complete control over their decisions and their actions. 5. People have free will even when their choices are completely limited by external circumstances.
Scale 2: Belief in Free Will Scale- personal agency items (Rakos, Laurene, Skala & Slane, 2008) (To what extent do you think the following statements are true?) (5-point scale; 1 = Not true at all, 5 = Almost always true)
1. I am in charge of the decisions I make 2. I decide what action to take in a particular situation 3. I am in charge of my actions even when my life's circumstances are difficult 4. I have free will
Scale 3: Free Will and Determinism-personal will subscales of FWD scale (Rakos et al., 2008) (To what extent do you think the following statements are true?) (5-point scale; 1 = Not true at all, 5 = Almost always true)
1. I am in charge of the decisions I make. 2. I actively choose what to do from among the options I have. 3. I am in charge of my actions even when my life’s circumstances are difficult. 4. My decisions are influenced by a higher power. (R) 5. I have free will even when my choices are limited by external circumstances. 6. I decide what action to take in a particular situation. 7. My choices are limited because they fit into a larger plan. (R) 8. I have free will.
Scale 4: Free Will and Determinism Plus scale (FAD+; Paulhus & Carey, 2011) Instructions - For each of the following statement, please indicate how true those statements are to you (0 - "not true at all", 4 - "always true")
1. People have complete control over the decisions they make. 2. People must take full responsibility for any bad choices they make. 3. People can overcome any obstacles if they truly want to. 4. Criminals are totally responsible for the bad things they do. 5. People have complete free will. 6. People are always at fault for their bad behavior. 7. Strength of mind can always overcome the body's desires.
Scale of susceptibility to better-than-average effect Part 1
Instruction: Please rate yourself relative to the average HKU student on a scale between 1 (much less than the average the average student of the university) to 9 (much more than the average student of the university):
Dependability (being able to be counted on or relied upon) Objectivity (being objective, basing thoughts and actions on objective facts and truth) Consideration for others (caring for and about others) Snobbery (being a snob, thinking about oneself, and acting as being better than others) Deceptiveness (inclination or practice of misleading others through lies or trickery) Selfishness (Prioritising oneself over others)
Part 2
Below is a description of the "Better Than Average" effect. Please read it carefully and then answer the questions in the following page.
Studies have shown that people show a “better than average” effect when assessing themselves relative to other members within their group. That is, 70-80% of individuals consistently rate themselves “better than average” on qualities that they perceive as positive, and conversely, evaluate themselves as having “less than average” amounts of characteristics they believe are negative. Having just read the description about the "better than average" effect, it would be useful to know the accuracy of your self-assessments on the previous part (Part 1).
Please indicate how you think you would be rated on the relevant dimensions by the “most accurate, valid, and objective resources available"
The objective measures would rate me lower on positive characteristics and higher on negative characteristics than I rated myself. (1) The objective measures would rate me neither more positively nor more negatively than I rated myself. (2) The objective measures would rate me higher on positive characteristics and lower on negative characteristics than I rated myself. (3)
Summary Tables Across Study 1, Study 2, and Study 3
Bias blind spot replication and extension (supplementary) 11
Table S3 Summary of findings
Mean Hypothesis N Statistic df p Cohen's d with CI difference The original study Study 1-Survey 2 29 t-statistic Self-other asymmetry in susceptibility to biases -4.64 28 p < .001 -0.8 -0.86 [-1.28, -0.43] Self-other asymmetry in personal shortcomings 1.49 28 p = .15 0.63 0.28 [-0.10, 0.65] Differences in self-other asymmetry between -3.31 28 p = .003 -0.61 [-1.01, -0.21] susceptibility to biases and personal shortcomings -1.43 Study 2 79 t-statistic Positive personality dimensions in comparison to 14.19 78 p < .001 6.44 1.60 [1.26, 1.93] others Negative personality dimensions in comparison to -10.94 78 p < .001 3.64 -1.23[-1.52, -0.94] others Chi-Square Denial of Better-than-average effect 9.97 1 p = .002 2.8 0.76 [0.29, 1.23] Study 1a 45 t-statistic Self-other asymmetry in susceptibility to biases -4.54 44 < .001 -0.75 -0.68 [-1.01, -0.35] Self-other asymmetry in personal shortcomings -1.13 44 0.265 -0.29 -0.17 [-0.47, 0.13] Differences in self-other asymmetry between -1.97 44 0.055 -0.46 -0.29 [-0.60, 0.01] susceptibility to biases and personal shortcomings
Bias blind spot replication and extension (supplementary) 12
Study 1b 43 t-statistic
Positive personality dimensions compared to others 5.09 42 < .001 0.74 0.78 [0.43, 1.11]
Negative personality dimensions compared to others -4.55 42 < .001 -0.84 -0.69 [-1.02, -0.36]
Chi-square Denial of better-than-average effect 14.53 1 < .001 -- 1.43 [0.69, 2.16] Study 2 303 t-statistic Self-other asymmetry in susceptibility to biases -16.16 302 < .001 -1.15 -0.93 [-1.06, -0.79] Self-other asymmetry in personal shortcomings -5.22 302 < .001 -0.52 -0.30 [-0.42, -0.18] Differences in self-other asymmetry between -6.39 302 < .001 -0.62 -0.37 [-0.48, -0.25]. susceptibility to biases and personal shortcomings Study 3 621 t-statistic Self-other asymmetry in susceptibility to biases -32.04 620 <.001 -1.8 -1.29 [-1.39, -1.18] Self-other asymmetry in personal shortcomings -10.54 620 <.001 -0.73 -0.42 [-0.51, -0.34] Differences in self-other asymmetry between -13.01 620 <.001 -1.06 -0.52 [-0.61, -0.44] susceptibility to biases and personal shortcomings
Positive personality dimensions compared to others 31.74 620 <.001 1.42 1.27 [1.17, 1.38]
Negative personality dimensions compared to others -30.38 620 <.001 1.79 -1.22 [-1.32, -1.11]
Chi-square
Denial of better-than-average effect 261.53 1 <.001 -- 1.71 [1.50, 1.91]. Note. In the original article, effect sizes were not reported; we computed Cohen’s d and confidence intervals using the non-central t method (Rosenthal, 1991). Bias blind spot replication and extension (supplementary) 13
Table S4 Summary and comparison of findings of the three replication studies to the original article
Effect size (d) - NHST Effect size (d) Replication summary Original article Summary Study 1a: Hong Kong (N = 45) Self-other asymmetry in susceptibility to biases -0.68 [-1.01, -0.35] -0.86 [-1.28, -0.43] Supported Signal - consistent Self-other asymmetry in personal shortcomings -0.17 [-0.47, 0.13] 0.28 [-0.10, 0.65] Not supported No signal - inconsistent, opposite Differences in self-other asymmetry between susceptibility to biases and personal -0.29 [-0.60, 0.01] -0.61 [-1.01, -0.21] Supported Signal - inconsistent, smaller shortcomings Study 1b: Hong Kong (N = 43) Positive personality dimensions compared to 0.78 [0.43, 1.11] 1.60 [1.26, 1.93] Supported Signal - inconsistent, smaller others Negative personality dimensions compared to -0.69 [-1.02, -0.36] -1.23[-1.52, -0.94] Supported Signal - inconsistent, smaller others Denial of better-than-average effect 1.43 [0.69, 2.16] 0.76 [0.29, 1.23] Supported Signal - consistent Study 2: United states (N = 303) Self-other asymmetry in susceptibility to biases -0.93 [-1.06, -0.79] -0.86 [-1.28, -0.43] Supported Signal - consistent Self-other asymmetry in personal shortcomings -0.30 [-0.42, -0.18] 0.28 [-0.10, 0.65] Not supported Signal- inconsistent, opposite Differences in self-other asymmetry between susceptibility to biases and personal -0.37[-0.48, -0.25] -0.61 [-1.01, -0.21] Supported Signal - inconsistent, smaller shortcomings Study 3: United states (N = 621) Self-other asymmetry in susceptibility to biases -1.29 [-1.39, -1.18] -0.86 [-1.28, -0.43] Supported Signal - inconsistent, larger Self-other asymmetry in personal shortcomings -0.42 [-0.51, -0.34] 0.28 [-0.10, 0.65] Not supported Signal- inconsistent, opposite Bias blind spot replication and extension (supplementary) 14
Differences in self-other asymmetry between susceptibility to biases and personal -0.52 [-0.61, -0.44] -0.61 [-1.01, -0.21] Supported Signal - consistent shortcomings Positive personality dimensions compared to 1.27 [1.17, 1.38] 1.60 [1.26, 1.93] Supported Signal - inconsistent, smaller others Negative personality dimensions compared to -1.22 [-1.32, -1.11] -1.23[-1.52, -0.94] Supported Signal - consistent others Denial of better-than-average effect 1.71 [1.50, 1.91] 0.76 [0.29, 1.23] Supported Signal -inconsistent, larger
Note. Effect size is Cohen's d. Replication summary directly based on LeBel et al., (2018) that considered following three statistical aspects: (1) whether a signal was detected (i.e., whether the 95% confidence interval, or CI, represented here by the error bars, excludes 0); (2) the consistency of the replication effect-size (ES) estimate with that observed in the original study (i.e., whether the replication’s CI includes the original ES point estimate); and (3) the precision of the replication’s ES estimate (i.e., the width of its CI relative to the CI in the original study). NHST summary is based on the interpretation of a one-tail p-value with alpha set at 0.05.
Bias blind spot replication and extension (supplementary) 15
Study 1a Table S5 Study 1a’s Descriptive statistics of the measures of participants’ perceptions of their own susceptibility to eight biases
Fundamental Fear of Personal Self- Self- Reactive Assimilation Cognitive Hostile Halo Original Planning Procrasti attribution public shortcomings serving interest devaluation Bias dissonance media effect 8 biases fallacy nation error speaking avg. N 45 45 45 45 45 45 45 45 45 45 45 45 45 Mean 5.60 5.89 5.73 4.20 4.91 5.84 5.89 6.71 5.60 6.47 5.93 6.20 6.20 Standard deviation 1.78 1.32 1.53 1.85 2.02 1.66 1.93 1.52 0.86 1.98 2.29 2.41 1.78 Skewness 0.06 -0.10 -0.09 0.25 -0.17 -0.36 -0.43 -0.79 0.69 -0.60 -0.28 -0.48 -0.30 Std. error skewness 0.35 0.35 0.35 0.35 0.35 0.35 0.35 0.35 0.35 0.35 0.35 0.35 0.35 Kurtosis -0.65 -0.46 -0.01 -0.41 -0.75 -0.24 -0.71 1.23 1.00 -0.76 -1.05 -1.04 -1.02 Std. error kurtosis 0.70 0.70 0.70 0.70 0.70 0.70 0.70 0.70 0.70 0.70 0.70 0.70 0.70
Bias blind spot replication and extension (supplementary) 16
Table S6 Study 1a’s Descriptive statistics of the measures of participants’ perceptions of others’ susceptibility to eight biases
Fundamental Fear of Personal Self- Self- Reactive Assimilation Cognitive Hostile Halo Original Planning Procrastinat attribution public shortcomings serving interest devaluation Bias dissonance media effect 8 biases fallacy ion error speaking avg. N 45 45 45 45 45 45 45 45 45 45 45 45 45 Mean 6.62 6.58 6.47 6.42 6.44 5.47 5.87 6.93 6.35 6.87 7.02 5.58 6.49 Standard deviation 1.39 1.56 1.32 1.44 1.55 1.93 1.79 1.29 0.91 1.47 1.74 1.78 1.23 Skewness -0.51 -0.57 -0.14 -0.32 -0.34 -0.11 -0.26 -0.21 -0.04 -0.75 -1.18 -0.06 -0.48 Std. error skewness 0.35 0.35 0.35 0.35 0.35 0.35 0.35 0.35 0.35 0.35 0.35 0.35 0.35 Kurtosis -0.23 -0.03 -0.39 -0.32 0.11 -0.50 -0.52 -0.65 0.05 0.71 1.88 -0.56 -0.04 Std. error kurtosis 0.70 0.70 0.70 0.70 0.70 0.70 0.70 0.70 0.70 0.70 0.70 0.70 0.70
Bias blind spot replication and extension (supplementary) 17
Figure S1. Study 1a- Participants’ perceptions of their own and the others’ susceptibility to eight biases in judgment and inference. Error bars indicate ±1 SEM.
Bias blind spot replication and extension (supplementary) 18
Figure S2. Study 1a- Boxplot and violin-plot with jittered data points of the measures of participants’ perceptions of their own and the others’ susceptibility to eight biases.
Bias blind spot replication and extension (supplementary) 19
Figure S3. Study 1a-The participants’’ perception of their own and others’ personal shortcomings. Error bars indicate ±1 SEM.
Bias blind spot replication and extension (supplementary) 20
Figure S4. Boxplot and violin-plot with jittered data points of the three measures of participants’ perceptions of their own and the others’ personal shortcomings Bias blind spot replication and extension (supplementary) 21
Table S7 Study 1a-The results of dependent t-test of self-other asymmetry in each of eight biases
Mean t-statistic df p Cohen's d difference
Original 8 biases (Average) -4.54 44 < .001 -0.75 -0.68 Self-serving -3.06 44 0.002 -1.02 -0.46 Reactive devaluation -2.95 44 0.003 -0.73 -0.44 Assimilation Bias -1.71 44 0.047 -0.53 -0.26 Cognitive dissonance 0.99 44 0.837 0.27 0.15 Fundamental attribution error -4.17 44 < .001 -1.27 -0.62 Hostile media -2.76 44 0.004 -0.96 -0.41 Halo effect -3.88 44 < .001 -1.09 -0.58 Self-interest -3.27 44 0.001 -0.69 -0.49 Procrastination -1.45 44 0.154 -0.56 -0.22 Fear of public speaking 1.14 44 0.260 0.36 0.17 Planning fallacy -1.59 44 0.119 -0.67 -0.24 Personal shortcomings (Average) -1.13 44 0.265 -0.29 -0.17
Bias blind spot replication and extension (supplementary) 22
Study 1b Table S8 Study 1b-Descriptive statistics of the measures of susceptibility to better than average effect
Consideration Mean of Mean of Denial of Dependability Objectivity Snobbery Deceptiveness Selfishness for others positive traits negative traits bias N 43 43 43 43 43 43 43 43 43 Mean 5.37 5.42 6.44 5.74 4.19 3.95 4.33 4.16 1.98 Std. error mean 0.25 0.21 0.22 0.15 0.23 0.28 0.24 0.19 0.10 Standard deviation 1.65 1.35 1.45 0.96 1.50 1.84 1.55 1.22 0.64 Skewness -0.20 -0.34 -1.03 -0.18 0.20 0.43 0.66 0.27 0.02 Std. error skewness 0.36 0.36 0.36 0.36 0.36 0.36 0.36 0.36 0.36 Kurtosis -0.39 -0.36 1.50 -0.16 -0.71 -0.39 0.03 -0.11 -0.38 Std. error kurtosis 0.71 0.71 0.71 0.71 0.71 0.71 0.71 0.71 0.71
Bias blind spot replication and extension (supplementary) 23
Table S9 Study 1b-The results of one sample t-test of better than average effect
Difference from mid- t-statistic df p Cohen's d point of the scale (i.e., 5) Dependability 1.48 42 0.073 0.37 0.23 Objectivity 2.03 42 0.024 0.42 0.31
Consideration for others 6.51 42 <.001 1.44 0.99 Mean of positive traits 5.09 42 <.001 0.74 0.78 Snobbery -3.56 42 <.001 -0.81 -0.54 Deceptiveness -3.73 42 <.001 -1.05 -0.57 Selfishness -2.85 42 0.003 -0.67 -0.43 Mean of negative traits -4.55 42 <.001 -0.85 -0.69
Note. Hₐ population mean > 5 for positive traits; Hₐ population mean < 5 for negative traits
Bias blind spot replication and extension (supplementary) 24
Study 2 Table S10 Study 2- Descriptive statistics of the measures of participants’ perceptions of their own susceptibility to eight biases
Fundamental Fear of Personal Self- Self- Reactive Assimilation Cognitive Hostile Halo Original 8 Planning attribution Procrastination public shortcomings serving interest devaluation Bias dissonance media effect biases fallacy error speaking avg. N 303 303 303 303 303 303 303 303 303 303 303 303 303 Mean 4.29 5.42 4.88 3.57 4.09 4.76 4.77 5.31 4.64 5.27 5.85 4.91 5.35 Std. error mean 0.13 0.12 0.11 0.11 0.12 0.13 0.12 0.12 0.08 0.15 0.15 0.13 0.11 Standard deviation 2.18 2.01 1.84 1.94 2.06 2.22 2.11 2.07 1.35 2.52 2.66 2.24 1.88 Skewness 0.16 -0.23 -0.13 0.44 0.29 -0.06 -0.12 -0.33 -0.22 -0.16 -0.44 0.00 -0.28 Std. error skewness 0.14 0.14 0.14 0.14 0.14 0.14 0.14 0.14 0.14 0.14 0.14 0.14 0.14 Kurtosis -0.79 -0.47 -0.23 -0.44 -0.58 -0.84 -0.60 -0.38 0.05 -1.10 -1.02 -0.88 -0.40 Std. error kurtosis 0.28 0.28 0.28 0.28 0.28 0.28 0.28 0.28 0.28 0.28 0.28 0.28 0.28
Bias blind spot replication and extension (supplementary) 25
Table S11 Study 2- Descriptive statistics of the measures of participants’ perceptions of others’ susceptibility to eight biases
Fundamental Fear of Personal Self- Self- Reactive Assimilation Cognitive Hostile Halo Original Planning attribution Procrastination public shortcomings serving interest devaluation Bias dissonance media effect 8 biases fallacy error speaking avg. N 303 303 303 303 303 303 303 303 303 303 303 303 303 Mean 5.70 6.36 5.63 4.90 5.70 6.17 5.65 6.17 5.78 5.85 6.08 5.69 5.87 Std. error mean 0.10 0.10 0.08 0.10 0.10 0.10 0.09 0.09 0.07 0.11 0.10 0.10 0.08 Standard deviation 1.77 1.70 1.47 1.78 1.66 1.70 1.64 1.62 1.16 1.91 1.71 1.66 1.35 Skewness -0.18 -0.25 -0.04 0.00 -0.29 -0.20 -0.02 -0.26 -0.15 -0.20 -0.28 -0.11 -0.03 Std. error skewness 0.14 0.14 0.14 0.14 0.14 0.14 0.14 0.14 0.14 0.14 0.14 0.14 0.14 Kurtosis 0.00 -0.45 0.36 -0.14 0.27 -0.36 0.00 -0.08 0.73 -0.49 0.05 -0.07 -0.04 Std. error kurtosis 0.28 0.28 0.28 0.28 0.28 0.28 0.28 0.28 0.28 0.28 0.28 0.28 0.28
Bias blind spot replication and extension (supplementary) 26
Table S12 Study 2- Means, standard deviations, and correlations with confidence intervals
Variable M SD 1 2 3 4 5 6 7 1 - Free Will Inventory 5.16 1.38 (0.91) 2 - Belief in Free Will personal will 4.31 0.74 .70** (0.92) [.64, .76] 3 - Susceptibility to biases: Self 4.64 1.35 -0.01 -0.11 (0.81) [-.13, .10] [-.22, .00] 4 - Susceptibility to biases: Others 5.78 1.16 0.02 0.05 .52** (0.84) [-.09, .13] [-.06, .16] [.44, .60] 5 - Personal shortcomings: Self 5.35 1.88 -.22** -.17** .52** .36** (0.63) [-.32, -.11] [-.28, -.06] [.43, .60] [.25, .45] 6 - Personal shortcomings: Others 5.87 1.35 0.00 0.05 .40** .66** .45** (0.65) [-.11, .12] [-.06, .16] [.30, .49] [.59, .72] [.36, .54] 7 - Susceptibility to biases: Self-other asymmetry -1.15 1.24 -0.03 -.17** .60** -.37** .23** -.18** [-.15, .08] [-.28, -.06] [.52, .67] [-.46, -.27] [.13, .34] [-.29, -.07] 8 - Personal shortcomings: Self-other asymmetry -0.52 1.75 -.24** -.22** .25** -.13* .73** -.29** .39** [-.34, -.13] [-.33, -.11] [.14, .35] [-.24, -.01] [.67, .77] [-.39, -.18] [.29, .49] Note. M = mean; SD = standard deviation; N = 303; * p < .05 and ** p < .01. The values in the diagonal indicate the reliability measures (Cronbach alpha); Free Will Inventory = Free Will Inventory (Nadelhoffer et al., 2014); Belief in Free Will personal will = Free-will and determinism personal will sub-scale (Rakos, Laurene, Skala, & Slane, 2008).
Bias blind spot replication and extension (supplementary) 27
Figure S5. Study 2-The participants’ perception of their own and others’ susceptibility to the eight biases measures. Error bars indicate ±1 SEM.
Bias blind spot replication and extension (supplementary) 28
Figure S6. Study 2- Boxplot and violin-plot with jittered data points of the measures of participants’ perceptions of their own and the others’ susceptibility to eight biases.
Bias blind spot replication and extension (supplementary) 29
Figure S7. Study 2-The participants’ perception of their own and of others’ personal shortcomings. Error bars indicate ±1 SEM.
Bias blind spot replication and extension (supplementary) 30
Figure S8. Study 2- Boxplot and violin-plot with jittered data points of the three measures of participants’ perceptions of their own and the others’ personal shortcomings Bias blind spot replication and extension (supplementary) 31 Table S13 Study 2-Results of dependent t-test of self-other asymmetry on the susceptibility to biases
Mean t-statistic df p Cohen's d difference Original 8 biases (Average) -16.16 302 < .001 -1.15 -0.93 Self-serving -11.04 302 < .001 -1.41 -0.63 Self-interest -8.11 302 < .001 -0.94 -0.47 Reactive devaluation -6.79 302 < .001 -0.75 -0.39 Fundamental attribution error -10.82 302 < .001 -1.33 -0.62 Hostile media -12.64 302 < .001 -1.61 -0.73 Halo effect -11.21 302 < .001 -1.41 -0.64 Assimilation Bias -7.42 302 < .001 -0.88 -0.43 Cognitive dissonance -7.25 302 < .001 -0.86 -0.42 Personal shortcomings (Average) -5.22 302 < .001 -0.53 -0.3 Procrastination -3.84 302 < .001 -0.57 -0.22 Fear of public speaking -1.4 302 0.164 -0.22 -0.08 Planning fallacy -5.94 302 < .001 -0.78 -0.34
Bias blind spot replication and extension (supplementary) 32 Table S14 Study 2-Correlations and confidence intervals of free will beliefs measures and measures of participants’ perception of their own susceptibility to biases and of their own perceived personal shortcomings
Variable M SD 1 2 3 4 5 6 7 8 9 10 11 12 1. Free Will Inventory 5.16 1.38 2. Personal will 4.31 0.74 .70** [.64, .76] 3. Others-serving-Self 4.29 2.18 -.12* -.21** [-.23, -.01] [-.32, -.10] 4. Others-interest bias—Self 5.42 2.01 0.08 0.07 .39** [-.04, .19] [-.05, .18] [.29, .48] 5. Reactive devaluation—Self 4.88 1.84 -0.02 -0.08 .40** .34** [-.13, .10] [-.19, .03] [.31, .49] [.24, .44] 6. Fundamental attribution error-Self 3.57 1.94 0.06 -.14* .37** .15** .27** [-.05, .17] [-.25, -.03] [.27, .46] [.04, .26] [.16, .37] 7. Perceptions of hostile media--Self 4.09 2.06 0.04 -0.08 .38** .27** .43** .39** [-.08, .15] [-.19, .03] [.28, .47] [.16, .37] [.33, .52] [.29, .48] 8. Halo effect-Self 4.76 2.22 0.00 -0.02 .36** .27** .44** .33** .37** [-.11, .11] [-.13, .10] [.26, .45] [.17, .37] [.34, .52] [.22, .43] [.26, .46] 9. Assimilation Bias-Self 4.77 2.11 -.11* -.11* .41** .35** .39** .22** .33** .40** [-.22, -.00] [-.22, -.00] [.31, .50] [.25, .45] [.30, .49] [.11, .33] [.23, .43] [.30, .49] 11. Cognitive dissonance-Self 5.31 2.07 0.02 0.00 .32** .33** .45** .27** .35** .40** .38** [-.10, .13] [-.11, .11] [.21, .41] [.23, .43] [.35, .53] [.16, .37] [.25, .45] [.31, .49] [.28, .47] 11. Procrastination-Self 5.27 2.52 -.22** -.13* .26** .19** .28** .16** .15** .39** .37** .31** [-.33, -.11] [-.24, -.02] [.15, .36] [.08, .30] [.18, .38] [.05, .27] [.04, .26] [.29, .48] [.27, .46] [.20, .41] 12. Fear of public speaking-Self 5.85 2.66 -.15** -0.11 .15** .17** .26** .20** .15* .18** .24** .28** .36** [-.26, -.04] [-.22, .01] [.04, .26] [.06, .27] [.15, .36] [.09, .30] [.04, .26] [.07, .29] [.13, .34] [.17, .38] [.26, .46] 13. Planning fallacy-Self 4.91 2.24 -.12* -.15** .38** .14* .36** .34** .27** .33** .36** .38** .48** .26** [-.23, -.01] [-.26, -.04] [.28, .47] [.03, .25] [.26, .46] [.24, .44] [.16, .37] [.23, .43] [.26, .46] [.27, .47] [.38, .56] [.15, .36] Note: *p <.05; **p <.01; ***p <.001; Values in square brackets indicate the 95% confidence interval for each correlation. Free Will Inventory = Free Will Inventory (Nadelhoffer et al., 2014); Belief in Free Will- Personal Will Subscale = Free-will and determinism personal will sub-scale (Rakos, Laurene, Skala, & Slane, 2008).
Bias blind spot replication and extension (supplementary) 33 Table S15 Study 2-Correlations and confidence intervals of free will belief measures and measures of participants’ perception of others’ susceptibility to biases and of others’ personal shortcomings variables
Variable M SD 1 2 3 4 5 6 7 8 9 10 11 12 1. Free Will Inventory 5.16 1.38 2. Personal will 4.31 0.74 .70** [.64, .76] 3. Others-serving – Others 5.70 1.77 -0.04 -0.03 [-.15, .07] [-.14, .08] 4. Others-interest bias-Others 6.36 1.70 0.10 .16** .58** [-.02, .21] [.04, .26] [.50, .65] 5. Reactive devaluation-Others 5.63 1.47 0.02 0.00 .52** .41** [-.10, .13] [-.11, .11] [.44, .60] [.31, .50] 6. Fundamental attribution error -Others 4.90 1.78 0.00 -0.07 .44** .25** .36** [-.11, .11] [-.18, .04] [.34, .52] [.14, .35] [.26, .45] 7. Perceptions of hostile media -Others 5.70 1.66 -0.03 0.05 .43** .43** .46** .35** [-.14, .08] [-.07, .16] [.34, .52] [.33, .52] [.37, .55] [.25, .44] 8. Halo effect –Others 6.17 1.70 0.03 0.09 .39** .46** .39** .20** .45** [-.08, .15] [-.02, .20] [.28, .48] [.37, .55] [.29, .48] [.09, .30] [.36, .54] 9. Assimilation Bias –Others 5.65 1.64 -0.02 0.04 .49** .38** .41** .35** .47** .47** [-.13, .10] [-.07, .15] [.40, .57] [.28, .47] [.31, .50] [.25, .45] [.38, .56] [.38, .56] 11. Cognitive dissonance -Others 6.17 1.62 0.05 0.06 .41** .45** .31** .18** .48** .53** .46** [-.06, .16] [-.05, .18] [.31, .50] [.36, .54] [.21, .41] [.07, .29] [.38, .56] [.44, .60] [.37, .54] 11. Procrastination –Others 5.85 1.91 0.00 0.03 .39** .39** .39** .28** .45** .41** .31** .37** [-.12, .11] [-.08, .14] [.29, .48] [.29, .48] [.29, .48] [.17, .38] [.35, .53] [.31, .50] [.20, .41] [.26, .46] 12. Fear of public speaking -Others 6.08 1.71 -0.02 0.05 .41** .40** .27** .23** .43** .43** .42** .46** .41** [-.13, .10] [-.06, .16] [.31, .50] [.30, .49] [.16, .37] [.12, .34] [.34, .52] [.34, .52] [.33, .51] [.36, .54] [.31, .50] 13. Planning fallacy –Others 5.69 1.66 0.03 0.05 .34** .29** .28** .24** .30** .37** .28** .31** .45** .27** [-.08, .14] [-.07, .16] [.24, .44] [.18, .39] [.17, .38] [.13, .34] [.19, .39] [.27, .46] [.17, .38] [.21, .41] [.36, .54] [.16, .37] Note: *p <.05; **p <.01; ***p <.001; Values in square brackets indicate the 95% confidence interval for each correlation. Free Will Inventory = Free Will Inventory (Nadelhoffer et al., 2014); Belief in Free Will- Personal Will Subscale = Free-will and determinism personal will sub-scale(Rakos, Laurene, Skala, & Slane, 2008).
Bias blind spot replication and extension (supplementary) 34 Study 3 Exploratory analyses We conducted a series of exploratory analyses probing the association between free will beliefs and susceptibility to biases and found that personal will was negatively correlated with susceptibility to bias in self (r = -0.14, p < .001, 95% CI [-0.22, -0.06]) and positively correlated with others’ susceptibility to bias (r = 0.12, p = .003, 95% CI [0.04, 0.20]). Put together, personal will was negatively correlated with self-other asymmetry for susceptibility to bias (r = -0.22, p < .001, 95% CI [-0.29, -0.14]). We found no support for a correlation with the two other measures of free will beliefs (correlations ranged between 0.01 CI [- 0.07, 0.09] and -0.03 CI [-0.11, 0.05]). In an exploratory analysis we found a negative relationship between free-will beliefs and ratings on negative personality dimensions (Free Will Inventory: r = -0.09, p = .033, 95% CI [-0.16, -0.01]; personal agency: r = -0.16, p < .001, 95% CI [-0.23, -0.08]; personal will: r = -0.12, p = .003, 95% CI [-0.20, -0.04]). Positive personality dimensions were generally positively correlated with personal will (r = 0.15, p < .001, 95% CI [0.07, 0.23]; Free Will Inventory: r = 0.04, p = .341, 95% CI [-0.04, 0.12]; Personal agency: r = 0.05, p = .222, 95% CI [-0.03, 0.13]). Denial of bias correlated with Free Will Inventory scale (r = 0.11, p = .007, 95% CI [0.03, 0.19]) and personal agency (r = 0.14, p < .001, 95% CI [0.06, 0.22]), with no support for personal will (r = 0.03, p = .531, 95% CI [-0.05, 0.10]).
Bias blind spot replication and extension (supplementary) 35 Additional tables and figures Table S16 Study 3- Descriptive statistics of the measures of participants’ perceptions of their own susceptibility to biases, and of their own personal shortcomings
Fundamental Original Fear of Personal Self- Self- Reactive Assimilation Cognitive Hostile Halo Planning attribution 8 biases Procrastination public shortcomings serving interest devaluation Bias dissonance media effect fallacy error (avg.) speaking avg. N 621 621 621 621 621 621 621 621 621 621 621 621 621 Mean 4.11 5.47 4.85 3.74 4.04 4.90 5.11 5.28 4.69 5.30 6.33 4.94 5.52 Std. error mean 0.08 0.08 0.07 0.09 0.09 0.08 0.08 0.08 0.05 0.10 0.10 0.09 0.07 Standard deviation 2.09 1.91 1.81 2.12 2.14 2.11 2.01 2.08 1.30 2.49 2.51 2.29 1.71 Skewness 0.29 -0.24 -0.12 0.51 0.32 -0.17 -0.12 -0.24 0.02 -0.22 -0.60 0.00 -0.12 Std. error skewness 0.10 0.10 0.10 0.10 0.10 0.10 0.10 0.10 0.10 0.10 0.10 0.10 0.10 Kurtosis -0.60 -0.12 -0.28 -0.61 -0.68 -0.72 -0.40 -0.55 0.29 -1.04 -0.84 -0.91 -0.47 Std. error kurtosis 0.20 0.20 0.20 0.20 0.20 0.20 0.20 0.20 0.20 0.20 0.20 0.20 0.20
Bias blind spot replication and extension (supplementary) 36 Table S17 Study 3-Descriptive statistics of the measures of participants’ perception of others’ susceptibility to biases and of others’ personal shortcomings
Fundamental Original Fear of Personal Self- Self- Reactive Assimilation Cognitive Hostile Halo Planning attribution 8 biases Procrastination public shortcomings serving interest devaluation Bias dissonance media effect fallacy error (avg) speaking avg. N 621 621 621 621 621 621 621 621 621 621 621 621 621 Mean 6.74 7.14 6.26 5.61 6.34 7.03 6.2 6.54 6.48 6.27 6.42 6.08 6.25 Std. error mean 0.06 0.06 0.06 0.07 0.07 0.06 0.07 0.06 0.04 0.06 0.07 0.06 0.05 Standard deviation 1.58 1.52 1.51 1.76 1.83 1.60 1.64 1.52 1.04 1.59 1.62 1.58 1.16 Skewness -0.55 -0.56 -0.09 -0.12 -0.43 -0.86 -0.14 -0.26 -0.29 -0.33 -0.46 -0.06 -0.04 Std. error skewness 0.10 0.10 0.10 0.10 0.10 0.10 0.10 0.10 0.10 0.10 0.10 0.10 0.10 Kurtosis -0.08 -0.31 -0.24 -0.36 -0.24 0.70 -0.34 -0.28 0.28 -0.10 0.05 -0.32 -0.37 Std. error kurtosis 0.20 0.20 0.20 0.20 0.20 0.20 0.20 0.20 0.20 0.20 0.20 0.20 0.20
Bias blind spot replication and extension (supplementary) 37 Table S18 Study 3 - Means, standard deviations, and correlations
Variable M SD 1 2 3 4 5 6 7 8 9 10 11 1- FWI 5.17 1.25 (.89)
.78** 2-FAD+ 2.88 0.69 (.85) [.74, .81] .58** .56** (.74) 3-FWD 4.10 0.56 [.53, .63] [.50, .61] 0.01 0.01 -.14** 4-Biases: Self 4.69 1.30 [-.07, .09] [-.07, .08] [-.22, -.06] -0.03 -0.02 .12** .30** 5-Biases: Others 6.48 1.04 [-.11, .05] [-.10, .06] [.04, .20] [.23, .37] -.16** -.15** -.09* .38** .30** 6-Shortcomings: Self 5.52 1.71 [-.23, -.08] [-.22, -.07] [-.17, -.01] [.31, .44] [.23, .37] 0.02 0.05 .11** .37** .55** .32** 7-Shortcomings: Other 6.25 1.16 [-.05, .10] [-.03, .13] [.03, .18] [.30, .44] [.49, .60] [.25, .39] 0.03 0.02 -.22** .71** -.46** .13** -0.06 8-Bias: Self-other asymmetry -1.80 1.40 [-.05, .11] [-.06, .10] [-.29, -.14] [.66, .74] [-.52, -.40] [.05, .21] [-.14, .02] 9-Shortcomings: Self-other -.17** -.18** -.16** .12** -0.07 .77** -.35** .17** -0.73 1.73 asymmetry [-.25, -.09] [-.25, -.10] [-.24, -.09] [.05, .20] [-.15, .01] [.74, .80] [-.42, -.28] [.09, .25] 10-Positive personality 0.04 0.05 .15** -.13** .22** -.13** .14** -.28** -.22** 6.42 1.12 dimensions [-.04, .12] [-.03, .13] [.07, .23] [-.21, -.05] [.14, .29] [-.21, -.05] [.07, .22] [-.35, -.21] [-.30, -.15] 11-Negative personality -.09* -.16** -.12** .32** 0.01 .11** 0.07 .30** 0.07 -.34** 3.21 1.47 dimensions [-.16, -.01] [-.23, -.08] [-.20, -.04] [.25, .39] [-.07, .09] [.03, .19] [-.01, .15] [.22, .37] [-.01, .14] [-.40, -.26] 12- Better-than-average .11** .14** 0.03 0.03 -0.03 -0.04 0.01 0.05 -0.05 .14** -0.06 2.07 0.65 denial [.03, .19] [.06, .22] [-.05, .10] [-.05, .11] [-.11, .05] [-.12, .04] [-.07, .09] [-.03, .13] [-.13, .03] [.06, .22] [-.13, .02] Note. M = Mean; SD = Standard deviation; * p < .05; ** p < .01; The values in the diagonal indicate the reliability measures (Cronbach alpha); Free Will Inventory = Free Will Inventory scale (Nadelhoffer et al., 2014; 1-7 scale); FAD+ = Free will and determinism plus scale (Paulhus & Carey, 2011; 1-5 scale); FWD = Free-will and determinism personal will sub-scale (Rakos, Laurene, Skala, & Slane, 2008; 0-4 scale).
Bias blind spot replication and extension (supplementary) 38
Figure S9. Study 3-The participants’ perception of their own and others’ susceptibility to each of the eight biases. Error bars indicate ±1 SEM.
Bias blind spot replication and extension (supplementary) 39
Figure S10. Study 3- Boxplot and violin-plot with jittered data points of the measures of participants’ perceptions of their own and the others’ susceptibility to eight biases.
Bias blind spot replication and extension (supplementary) 40
Figure S11. Study 3-The participants’ perception of their own and others’ personal shortcomings. Error bars indicate ±1 SEM.
Bias blind spot replication and extension (supplementary) 41
Figure S12. Study 3- Boxplot and violin-plot with jittered data points of the three measures of participants’ perceptions of their own and the others’ personal shortcomings Bias blind spot replication and extension (supplementary) 42
Table S19 Study 3- The results of dependent t-test of self-other asymmetry on each of eight biases, and perceived personal shortcomings
Mean t-statistic df p Cohen's d difference Original 8 biases (Average) -32.00 620 <.001 -1.80 -1.29 Self-serving -27.60 620 <.001 -2.63 -1.11 Self-interest -20.40 620 <.001 -1.68 -0.82 Reactive devaluation -16.90 620 <.001 -1.41 -0.68 Fundamental attribution error -18.50 620 <.001 -1.87 -0.74 Hostile media -22.60 620 <.001 -2.29 -0.91 Halo effect -21.10 620 <.001 -2.13 -0.85 Assimilation Bias -12.50 620 <.001 -1.09 -0.50 Cognitive dissonance -14.40 620 <.001 -1.26 -0.58 Personal shortcomings (Average) -10.54 620 <.001 -0.73 -0.42 Procrastination -9.62 620 <.001 -0.97 -0.39 Fear of public speaking -0.82 620 0.412 -0.09 -0.03 Planning fallacy -12.40 620 <.001 -1.14 -0.50
Bias blind spot replication and extension (supplementary) 43
Table S20 Study 3-Descriptive statistics of the measures of susceptibility to better than average effect
Mean of Mean of Consideration Denial of Dependability Objectivity positive Snobbery Deceptiveness Selfishness negative for others bias traits traits N 621 621 621 621 621 621 621 621 621 Mean 6.55 6.32 6.40 6.43 2.88 2.97 3.79 3.22 2.07 Std. error mean 0.06 0.07 0.06 0.04 0.07 0.07 0.07 0.06 0.03 Standard deviation 1.61 1.66 1.59 1.11 1.77 1.81 1.84 1.47 0.65 Skewness -0.41 -0.67 -0.32 -0.11 0.87 0.86 0.37 0.47 -0.07 Std. error skewness 0.10 0.10 0.10 0.10 0.10 0.10 0.10 0.10 0.10 Kurtosis -0.13 0.81 -0.03 0.22 0.10 0.12 -0.10 -0.12 -0.62 Std. error kurtosis 0.20 0.20 0.20 0.20 0.20 0.20 0.20 0.20 0.20
Bias blind spot replication and extension (supplementary) 44
Table S21 Study 3-The results of one sample t-test of better than average effect
Difference from T- statistic df p Cohen's d mid-point Dependability 24.00 620 <.001 1.55 0.96 Objectivity 19.70 620 <.001 1.32 0.79 Consideration for others 22.00 620 <.001 1.40 0.88 Mean of positive traits 31.70 620 <.001 1.42 1.27 Snobbery -29.90 620 < .001 -2.12 -1.20 Deceptiveness -27.90 620 < .001 -2.03 -1.12 Selfishness -16.40 620 < .001 -1.21 -0.66 Mean of negative traits -30.40 620 < .001 -1.79 -1.22
Note. Hₐ population mean > 5 for positive traits; Hₐ population mean < 5 for negative traits
Bias blind spot replication and extension (supplementary) 45
Table S22 Study 3-Correlations with confidence intervals of the measures of susceptibility to biases and perceived personal shortcomings variables (of the self) and belief in free-will scales Variable M SD 1 2 3 4 5 6 7 8 9 10 11 12 13 1. Free Will Inventory 5.17 1.25 2. Personal agency 2.88 0.69 .78** [.74, .81] 3. Personal will 4.10 0.56 .58** .56** [.53, .63] [.50, .61] 4. Self-serving – Self 4.11 2.09 -0.05 -0.07 -.16** [-.13, .03] [-.15, .01] [-.23, -.08] 5. Self-interest bias-Self 5.47 1.91 0.07 .11** 0.05 .36** [-.01, .15] [.03, .18] [-.03, .13] [.29, .42] 6. Reactive devaluation-Self 4.85 1.81 -0.02 -0.04 -.10* .43** .31** [-.10, .06] [-.12, .04] [-.17, -.02] [.36, .49] [.24, .38] 7. Fundamental attribution error –Self 3.74 2.12 0.02 0.04 -.17** .36** .25** .34** [-.06, .09] [-.04, .11] [-.24, -.09] [.29, .43] [.18, .32] [.26, .40] 8. Perceptions of hostile media –Self 4.04 2.14 .09* 0.06 -.12** .32** .25** .41** .46** [.01, .17] [-.02, .14] [-.20, -.05] [.25, .39] [.17, .32] [.35, .48] [.39, .52] 9. Halo effect –Self 4.90 2.11 -0.01 -0.02 -.09* .40** .30** .36** .33** .31** [-.09, .07] [-.10, .06] [-.17, -.01] [.33, .46] [.23, .37] [.29, .43] [.26, .40] [.24, .38] 10. Assimilation Bias –Self 5.11 2.01 -0.07 -0.07 -.08* .24** .27** .31** .31** .25** .34** [-.15, .01] [-.15, .01] [-.16, -.01] [.17, .32] [.20, .35] [.24, .38] [.23, .38] [.17, .32] [.27, .41] 11. Cognitive dissonance –Self 5.28 2.08 0.01 0.02 -0.03 .37** .28** .35** .25** .22** .38** .30** [-.07, .08] [-.06, .10] [-.11, .05] [.30, .43] [.20, .35] [.28, .42] [.17, .32] [.15, .30] [.31, .45] [.22, .37] 12. Procrastination –Self 5.30 2.49 -.21** -.22** -.12** .26** .16** .21** .21** .11** .19** .24** .15** [-.28, -.13] [-.30, -.15] [-.20, -.05] [.18, .33] [.09, .24] [.14, .29] [.13, .28] [.03, .18] [.11, .27] [.17, .31] [.07, .22] 13. Fear of public speaking –Self 6.33 2.51 -.11** -0.06 -0.01 0.03 0.01 .08* 0.06 .10* .09* .11** 0.03 .21** [-.18, -.03] [-.14, .02] [-.09, .07] [-.05, .11] [-.07, .09] [.00, .16] [-.02, .14] [.02, .17] [.02, .17] [.03, .18] [-.05, .11] [.13, .28] 14. Planning fallacy –Self 4.94 2.29 -0.01 -0.02 -0.06 .32** .21** .32** .29** .26** .27** .24** .23** .42** .11** [-.09, .07] [-.10, .06] [-.14, .02] [.24, .38] [.13, .28] [.25, .39] [.21, .36] [.18, .33] [.20, .34] [.17, .31] [.16, .31] [.35, .48] [.03, .19] Note: *p <.05; **p <.01; ***p <.001. Values in square brackets indicate the 95% confidence interval for each correlation. Free Will Inventory = Free Will Inventory scale (Nadelhoffer et al., 2014); Personal agency = Free will and determinism plus scale (Paulhus & Carey, 2011); Personal will = Free-will and determinism personal will sub-scale(Rakos, Laurene, Skala, & Slane, 2008). Bias blind spot replication and extension (supplementary) 46
Table S23 Study 3-Correlations with confidence intervals of the measurers of susceptibility to biases and perceived personal shortcomings variables (of others) and belief in free-will scales
Variable M SD 1 2 3 4 5 6 7 8 9 10 11 12 13 1. Free Will Inventory 5.17 1.25 2. Personal agency 2.88 0.69 .78** [.74, .81] 3. Personal will 4.10 0.56 .58** .56** [.53, .63] [.50, .61] 4. Others-serving – Others 6.74 1.58 -0.01 0.01 .13** [-.09, .07] [-.07, .08] [.05, .21] 5. Others-interest bias-Others 7.14 1.52 -0.01 0.03 .17** .41** [-.09, .07] [-.04, .11] [.09, .24] [.34, .47] 6. Reactive devaluation-Others 6.26 1.51 0 0.03 .12** .36** .39** [-.08, .08] [-.05, .11] [.04, .19] [.29, .43] [.33, .46] 7. Fundamental attribution error –Others 5.61 1.76 -.13** -.14** -.10* .37** .33** .30** [-.21, -.06] [-.22, -.06] [-.17, -.02] [.30, .43] [.26, .40] [.23, .37] 8. Perceptions of hostile media –Others 6.34 1.83 -0.03 -0.04 0.03 .36** .36** .42** .37** [-.11, .05] [-.12, .04] [-.05, .11] [.29, .43] [.29, .42] [.35, .48] [.30, .43] 9. Halo effect –Others 7.03 1.60 0.01 0.02 .17** .38** .37** .39** .20** .27** [-.06, .09] [-.06, .10] [.10, .25] [.31, .44] [.30, .44] [.32, .45] [.13, .28] [.20, .34] 10. Assimilation Bias –Others 6.20 1.64 -0.02 -0.02 0.01 .33** .33** .31** .25** .28** .31** [-.09, .06] [-.10, .06] [-.07, .09] [.26, .40] [.26, .40] [.24, .38] [.18, .33] [.20, .35] [.23, .38] 11. Cognitive dissonance –Others 6.54 1.52 0.05 0.02 .11** .37** .34** .28** .20** .22** .34** .32** [-.03, .13] [-.06, .10] [.03, .19] [.30, .44] [.27, .41] [.21, .35] [.13, .28] [.14, .29] [.27, .41] [.24, .39] 12. Procrastination –Others 6.27 1.59 -0.05 0.02 0.07 .41** .31** .32** .32** .23** .27** .33** .33** [-.13, .03] [-.06, .10] [-.01, .15] [.34, .47] [.24, .38] [.25, .39] [.25, .39] [.16, .31] [.19, .34] [.26, .40] [.26, .40] 13. Fear of public speaking –Others 6.42 1.62 .09* .10* .11** .22** .16** .15** .14** .08* .14** .24** .23** .30** [.01, .17] [.02, .18] [.03, .18] [.15, .30] [.08, .23] [.07, .22] [.06, .21] [.00, .16] [.06, .21] [.17, .32] [.16, .31] [.23, .37] 14. Planning fallacy –Others 6.08 1.58 0.01 -0.02 0.05 .34** .27** .33** .26** .24** .28** .31** .29** .39** .19** [-.07, .09] [-.10, .06] [-.03, .13] [.27, .41] [.19, .34] [.25, .40] [.19, .34] [.16, .31] [.21, .35] [.24, .38] [.22, .36] [.32, .46] [.11, .26] Note: *p <.05; **p <.01; ***p <.001. Values in square brackets indicate the 95% confidence interval for each correlation. Free Will Inventory = Free Will Inventory scale (Nadelhoffer et al., 2014); Personal agency = Free will and determinism plus scale (Paulhus & Carey, 2011); Personal will = Free-will and determinism personal will sub-scale(Rakos, Laurene, Skala, & Slane, 08).
Bias blind spot replication and extension (supplementary) 47
Summary and Mini-meta analysis
Mini-meta analysis including the original study
Figure S13
Forest plots of the mini meta-analyses. CI = confidence interval. ES = effect size (Cohen's d).
Bias blind spot replication and extension (supplementary) 48
Figure S14
Forest plots of the mini meta-analyses testing the Better-than-average effect. CI = confidence interval. ES = effect size (Cohen’s d).
Bias blind spot replication and extension (supplementary) 49
Mini-meta analysis excluding the original study
Figure S15
Forest plots of the mini meta-analyses. CI = confidence interval. ES = effect size (Cohen's d).
Bias blind spot replication and extension (supplementary) 50
Figure S16
Forest plots of the mini meta-analyses. CI = confidence interval. ES = effect size (Cohen's d).
Bias blind spot replication and extension (supplementary) 51
Summary of findings on ‘bias blind spot’ across 3 studies
Figure S17.
Violin plots for mean ratings of the outcome variables. The density of the violin plots represents the density of the data at each value, with wider sections indicating higher density.
Bias blind spot replication and extension (supplementary) 52
Summary of extension hypotheses
Table S24 Summary of the additional results on exploratory extension hypotheses (exploratory).
Predictions Effect size and CIs (r) NHST p Study 2: United States (N = 303) BFW and perceived bias: Self-others differences FWI -0.03 [-.15, .08] 0.565 FWD -0.17 [-.28, -.06] 0.003 Study 3: United States (N = 621) BFW and perceived bias: Self-others differences FWI 0.03 [-.05, .11] 0.869 FAD+ 0.02 [-.06, .10] 0.886 FWD -0.22 [-.29, -.14] < .001 BFW and negative personality dimensions FWI -0.09 [-.16, -.01] 0.033 FAD+ -0.16 [-.23, -.08] < .001 FWD -0.12 [-.20, -.04] 0.003 BFW and positive personality dimensions FWI 0.04 [-.04, .12] 0.341 FAD+ 0.05 [-.03, .13] 0.222 FWD 0.15 [.07, .23] < .001 BFW and denial of better-than-average effect FWI 0.11 [.03, .19] < .007 FAD+ 0.14 [.06, .22] < .001 FWD 0.03 [-.05, .10] 0.531 Note. r = Pearson correlation coefficient. BFW = free will beliefs; Asymmetry = self-other asymmetry. General free will = Nadelhoffer et al. (2014); Personal agency = Paulhus & Carey (2011); Personal will = Rakos, Laurene, Skala, & Slane, (2008).
Bias blind spot replication and extension (supplementary) 53
Framework for evaluation of the replications Table S25. Criteria for evaluation of replications by LeBel et al. (2018). A classification of relative methodological similarity of a replication study to an original study. “Same” (“different”) indicates the design facet in question is the same (different) compared to an original study. IV = independent variable. DV = dependent variable. “Everything controllable” indicates design facets over which a researcher has control. Procedural details involve minor experimental particulars (e.g., task instruction wording, font, font size, etc.).
Target similarity Highly similar Highly dissimilar
Category Direct replication Conceptual replication
Design facet Exact replication Very close Close replication Far replication Very far replication replication
Effect/ Hypothesis Same/similar Same/similar Same/similar Same/similar Same/similar IV operationalization Same/similar Same/similar Same/similar Different Different DV operationalization Same/similar Same/similar Same/similar Different Different IV stimuli Same/similar Same/similar Different Different DV stimuli Same/similar Same/similar Different Procedural details Same/similar Different Physical setting Same/similar Different Contextual variables Different
Bias blind spot replication and extension (supplementary) 54
Figure S18. Criteria for evaluation of replications by LeBel et al. (2019). A taxonomy for comparing replication effects to target article original findings.
Bias blind spot replication and extension (supplementary) 55
References Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., & Van't Veer, A. (2014). The replication recipe: What makes for a convincing replication?. Journal of Experimental Social Psychology, 50, 217-224. DOI: 10.1016/j.jesp.2013.10.005
Kennedy, M. (2010, April). V—Naive Realism and Experiential Evidence. In Proceedings of the Aristotelian Society (Hardback) (Vol. 110, No. 1pt1, pp. 77-109). Oxford, UK: Blackwell Publishing Ltd.
Litman, L., Robinson, J., & Abberbock, T. (2017). TurkPrime. com: A versatile crowdsourcing data acquisition platform for the behavioral sciences. Behavior research methods, 49, 433-442. DOI 10.3758/s13428-016-0727-z
Malle, B. F. (2006). The actor-observer asymmetry in attribution: A (surprising) meta-analysis. Psychological Bulletin, 132, 895–919. DOI: 10.1037/0033-2909.132.6.895
Nadelhoffer, T., Shepard, J., Nahmias, E., Sripada, C., & Ross, L. T. (2014). The free will inventory: Measuring beliefs about agency and responsibility. Consciousness and Cognition, 25, 27-41. DOI: 10.1016/j.concog.2014.01.006
Park, J., Choi, I., & Cho, G. (2006). The actor-observer bias in beliefs of interpersonal insights. Journal of Cross-Cultural Psychology, 37, 630-642. DOI: 10.1177/0022022106290482
Paulhus, D. L., & Carey, J. M. (2011). The FAD–Plus: Measuring lay beliefs regarding free will and related constructs. Journal of personality assessment, 93, 96-104. DOI: 10.1080/00223891.2010.528483
Pronin, E., Lin, D. Y., & Ross, L. (2002). The bias blind spot: Perceptions of bias in self versus others. Personality and Social Psychology Bulletin, 28, 369-381. DOI: 10.1177/0146167202286008
Rakos, R. F., Laurene, K. R., Skala, S., & Slane, S. (2008). Pervasiveness and correlates of belief in free will. Behavior and Social Issues, 17, 20-39. DOI: 10.5210/bsi.v17i1.1929
Rosenthal,R.(1991). Meta-analytic procedures for social research. Newbury Park, CA: SAGE Publications, Incorporated. DOI: 10.4135/9781412984997