And Institution-Related Status Bias in the Peer Review of Abstracts

FEATURE ARTICLE

META-RESEARCH Weak evidence of country- and institution-related status bias in the peer review of abstracts

Abstract Research suggests that scientists based at prestigious institutions receive more credit for their work than scientists based at less prestigious institutions, as do scientists working in certain countries. We examined the extent to which country- and institution-related status signals drive such differences in scientific recognition. In a preregistered survey experiment, we asked 4,147 scientists from six disciplines (astronomy, cardiology, materials science, political science, psychology and public health) to rate abstracts that varied on two factors: (i) author country (high status vs lower status in science); (ii) author institution (high status vs lower status university). We found only weak evidence of country- or institution-related status bias, and mixed regression models with discipline as random- effect parameter indicated that any plausible bias not detected by our study must be small in size.

MATHIASWULLUMNIELSEN*,CHRISTINEFRIISBAKER,EMERBRADY,MICHAEL BANGPETERSENANDJENSPETERANDERSEN

Introduction performance capacity of scientists working at The growth in scientific publishing (Larsen and more or less prestigious institutions (Jost et al., von Ins, 2010) makes it increasingly difficult for 2009; Greenwald and Banaji, 1995; Ridge- researchers to keep up-to-date with the newest way, 2001). According to status characteristics trends in their fields (Medoff, 2006; Col- theory, a scientist’s nationality or institutional lins, 1998). Already in the 1960s, sociologists of affiliation can be considered “status signals” science suggested that researchers, in the midst that, when salient, implicitly influence peer-eval- of this information overload, would search for uations (Berger et al., 1977; Correll and Ber- cues as to what literature to read (Merton, 1968; nard, 2015). Moreover, system justification *For correspondence: Crane, 1965). One important cue was the theory asserts that members of a social hierar- [email protected] author’s location. Academic affiliations would chy, such as the scientific community, regardless Competing interests: The cast a “halo-effect” (Crane, 1967) on scholarly of their position in this hierarchy, will feel moti- authors declare that no work that would amplify the recognition of vated to see existing social arrangements as fair competing interests exist. researchers based at prestigious institutions at and justifiable to preserve a sense of predictabil- Funding: See page 12 the expense of authors from institutions and ity and certainty around their own position nations of lower status. This halo effect ties (Jost et al., 2004; Jost et al., 2002; Magee and Reviewing editor: Peter closely to the idea of the “Matthew effect” in Galinsky, 2008; Son Hing et al., 2011). As a Rodgers, eLife, United Kingdom science, i.e. “the accumulation of differential result, both higher and lower status groups may Copyright Nielsen et al. This advantages for certain segments of the popula- internalize implicit behaviours and attitudes that article is distributed under the tion [of researchers] that are not necessarily favour higher-status groups (Jost, 2013). terms of the Creative Commons bound up with demonstrated differences in Several observational studies have examined Attribution License, which permits unrestricted use and capacity” (Merton, 1988). the influence of country- and institution-related redistribution provided that the From a social psychological perspective, halo effects in peer-reviewing (Crane, 1967; original author and source are country- and institution-related halo effects may Link, 1998; Murray et al., 2018). Most of them credited. arise from stereotypic beliefs about the relative indicate slight advantages in favor of researchers

Nielsen et al. eLife 2021;10:e64561. DOI: https://doi.org/10.7554/eLife.64561 1 of 13 Feature Article Meta-Research Weak evidence of country- and institution-related status bias in the peer review of abstracts

from high status countries (such as the US or UK) between 2000 and 2004. The study found that and universities (such as Harvard University or when abstracts underwent single blind com- Oxford University). However, a key limitation of pared to double blind reviewing the relative this literature concerns the unobserved hetero- increase in acceptance rates was higher for US geneity attributable to differences in quality. authored abstracts compared to non-US auth- Non-experimental designs do not allow us to ored abstracts, and for abstracts from highly determine the advantages enjoyed by scientists prestigious US institutions compared to at high-status locations independent of their abstracts from non-prestigious US institutions. capabilities as scientists, or the content and A recent survey experiment also found that character of their work. professors at schools of public health in the US Here we examine the impact of academic (N: 899) rated one abstract higher on likelihood affiliations on scientific judgment, independent of referral to a peer, when the authors’ affiliation of manuscript quality. Specifically, we consider was changed from a low-income to a high- how information about the geographical loca- income country (Harris et al., 2015). However, tion and institutional affiliation of authors influ- each participant was asked to rate four abstracts ence how scientific abstracts are evaluated by and the results for the remaining three abstracts their peers. In a preregistered survey experi- were inconclusive. Likewise, the study found no ment, we asked 4147 scientists from six disci- evidence of country-related bias in the ratings of plines (astronomy, cardiology, materials science, the strength of the evidence presented in the political science, psychology and public health) abstracts. In another randomized, blinded cross- to rate abstracts that vary on two factors: (i) over study (N: 347), the same authors found that author country (high status vs. lower status sci- changing the source of an abstract from a low- entific nation); (ii) institutional affiliation (high income to a high-income country slightly status vs. lower status university; see Table 1). improved English clinicians’ ratings of relevance All other content of the discipline-specific and recommendation to a peer (Harris et al., abstracts was held constant. A few pioneering studies have already 2017). Finally, a controlled field experiment attempted to discern the influence of national recently examined the “within subject” effect of and institutional location on scholarly peer- peer-review model (single blind vs. double blind) assessments, independent of manuscript quality. on the acceptance rates of full-length submis- One study (Blank, 1991) used a randomized sions for a prestigious computer-science confer- field experiment to examine the effects of dou- ence (Tomkins et al., 2017). The study allocated ble blind vs. single blind peer reviewing on 974 double blind and 983 single blind reviewers acceptance rates in the American Economic to 500 papers. Two single blind and two double Review. The study found no evidence that a blind reviewers assessed each paper. The study switch from single blind to double blind peer found that single blind reviewers were more reviewing influenced the relative ratings of likely than double blind reviewers to accept papers from high-ranked and lower-ranked uni- papers from top-ranked universities compared versities. Another study (Ross et al., 2006) to papers from lower-ranked universities. examined the effect of single blind vs. double Our study contributes to this pioneering work blind peer reviewing on the assessment of by targeting a broader range of disciplines in 13,000 abstracts submitted to the American the social sciences, health sciences and natural Heart Association’s annual Scientific Sessions sciences. This allows us to examine possible

Table 1. Sample distributions for the three-way factorial design across the six disciplines. Number of observations, N, by manipulation (rows) and disciplines (columns). Astronomy Cardiology Pol. science Psychology Public health Manipulation/Discipline (N = 502) (N = 609) Mat. science (N = 546) (N = 1008) (N = 624) (N = 732) Higher status (US) N = 209 N = 191 N = 196 N = 351 N = 216 N = 241 Lower status (US) N = 192 N = 213 N = 187 N = 319 N = 205 N = 237 Lower status (non-US) N = 191 N = 205 N = 163 N = 338 N = 213 N = 254

Nielsen et al. eLife 2021;10:e64561. DOI: https://doi.org/10.7554/eLife.64561 2 of 13 Feature Article Meta-Research Weak evidence of country- and institution-related status bias in the peer review of abstracts

High status (US) Lower status (US) Lower status (Non-US)

Astronomy Cardiology Materials Science Political Science Psychology Public Health

Figure 1. Boxplots displaying the distributions of abstract scores across the three manipulations and six disciplines. Each panel reports results for a specific discipline. The red box plots specify results for respondents exposed to an abstract authored at a high-status institution in the US. The blue box plots specify results for respondents exposed to an abstract authored at a lower-status institution in the US. The green box plots specify results for respondents exposed to an abstract authored at a lower-status institution outside the US. Whiskers show the 1.5 interquartile range. The red, blue and green dots represent outliers, and the grey dots display data points. The boxplots do not indicate any notable variations in abstract scores across manipulations. The online version of this article includes the following figure supplement(s) for figure 1: Figure supplement 1. Rating abstracts on “Originality of the presented research”. Figure supplement 2. Rating abstracts on “Credibility of the results”. Figure supplement 3. Rating abstracts on “Significance for future research”. Figure supplement 4. Rating abstracts on “Clarity of the abstract”. Figure supplement 5. The distribution of abstract scores for the full sample.

between-discipline variation in the prevalence of and moderation effects of the presumed associ- country- or institution-related rater bias. ation between status signals and evaluative out- Our six discipline-specific experiments show comes at an aggregate, cross-disciplinary level. weak and inconsistent evidence of country- We used three measures to gauge the assess- or institution-related status bias in abstract rat- ments of abstracts by peer-evaluators. Abstract ings, and mixed regression models indicate that score is our main outcome variable. It consists of any plausible effect must be small in size. four items recorded on a five-point scale (1=“very poor”, 5 = “excellent”) that ask the peer-evaluators to assess (i) the originality of the Results presented research, (ii) the credibility of the In accordance with our preregistered plan (see results, (iii) the significance for future research, https://osf.io/4gjwa), the analysis took place in and (iv) the clarity and comprehensiveness of the two steps. First, we used ANOVAs and logit abstract. We computed a composite measure models to conduct discipline-specific analyses of that specifies each participant’s total-item score how country- and institution-related status sig- for these four items (Cronbach’s a = 0.778). nals influence scholarly judgments made by Open full-text is a dichotomous outcome mea- peer-evaluators. Second, we employed mixed sure that asks whether the peer-evaluator would regression models with disciplines as random choose to open the full text and continue read- effect parameter to estimate the direct effect ing, if s/he came across the abstract online.

Nielsen et al. eLife 2021;10:e64561. DOI: https://doi.org/10.7554/eLife.64561 3 of 13 Feature Article Meta-Research Weak evidence of country- and institution-related status bias in the peer review of abstracts

A B A B

Figure 2. Plots of odds ratios and estimated proportions derived from the discipline-specific logit models with "Open full text" as outcome. Panel A displays the odds ratios for respondents exposed to manipulation 1 (high-status university, US) or manipulation 3 (lower-status university, non-US). Manipulation 2 (lower-status university, US) is the reference group. Panel B plots the adjusted means for manipulation 1, manipulation 2 and manipulation 3. Error bars represent 95% CIs. As shown in Panel A, peer-evaluators in public health were slightly less likely to show interest in opening the full text when the author affiliation was changed from a lower-status university in the US to a lower-status university elsewhere. The results for the remaining eleven comparisons are inclusive. For model specifications, see Supplementary file 2, Tables S13–S18.

Include in conference is a dichotomous outcome Cohen’s D=±0.30 for the discipline-specific measure that asks whether the peer-evaluator equivalence tests. would choose to include the abstract for an oral The boxplots in Figure 1 display the abstracts presentation, if s/he was reviewing it as a com- ratings for respondents assigned to each manip- mittee member of a selective international scien- ulation across the six disciplines. The discipline- tific conference (the full questionnaire is specific ANOVAs with the manipulations as the included in the Supplementary file 1). between-subject factor did not indicate any As specified in the registered plan, we used country- or institution-related status bias in G*Power to calculate the required sample size abstract ratings (astronomy: F = 0.71, p=0.491, per discipline (N = 429) for detecting a Cohen’s N = 592; cardiology: F = 1.50, p=0.225, f = 0.15 (corresponding to a Cohen’s d = 0.30) N = 609; materials science: F = 0.73, p=0.482, or larger with a = 0.05 and a power of 0.80 in N = 546; political science: F = 0.53, p=0.587, N = 1008; psychology: F = 0.19, p=0.827, the cross-group comparisons with abstract score N = 624; public health: F = 0.34, p=0.715, as outcome. Table 1 presents the final sample N = 732). The h2 coefficients were 0.002 in distributions for the three-way factorial design astronomy, 0.005 in cardiology, 0.003 in materi- across the six disciplines. als science, and 0.001 in political science, psy- To test for statistical equivalence in the disci- chology and public health. pline-specific comparisons of our main outcome A TOST procedure with an a level of 0.05 (abstract score) across manipulations, we per- indicated that the observed effect sizes were formed the two-one-sided tests procedure within the equivalence bound of d = À0.3 and [TOST] for differences between independent d = 0.3 for 11 of the 12 between-subject com- means (Lakens, 2017a). We used the pre-regis- parisons at the disciplinary level (see tered ‘minimum detectable effect size’ of Supplementary file 2, Tables S1–S12). In raw

Nielsen et al. eLife 2021;10:e64561. DOI: https://doi.org/10.7554/eLife.64561 4 of 13 Feature Article Meta-Research Weak evidence of country- and institution-related status bias in the peer review of abstracts

A A BB

Figure 3. Plots of odds ratios and estimated proportions derived from the discipline-specific logit models with "Include in conference" as outcome. Panel A displays the odds ratios for respondents exposed to manipulation 1 (high-status university, US) or manipulation 3 (lower-status university, non- US). Manipulation 2 (lower-status university, US) is the reference group. Panel B plots the adjusted means for manipulation 1, manipulation 2 and manipulation 3. Error bars represent 95% CIs. As shown in Panel A, peer-evaluators in political science were slightly less likely to show interest in opening the full text when the author affiliation was changed from a lower-status university in the US to a high-status university in the US. The results for the remaining eleven comparisons are inclusive. For model specifications, see Supplementary file 2, Tables S19–S24.

scores, this equivalence bound corresponds to a from a lower-status university in the US to a span from À0.8 to 0.8 abstract rating points on lower-status university elsewhere (Odds ratio:. a scale from 4 to 20. In cardiology, the TOST 634, CI: 0.435–0.925). The odds ratios for the test failed to reject the null-hypothesis of non- remaining 11 comparisons across the six disci- equivalence in the evaluative ratings of subjects plines all had 95% confidence intervals spanning exposed to abstracts from higher-status US uni- one. Moreover, in five of the 12 between-subject versities and lower-status US universities. comparisons, the direction of the observed A closer inspection suggests that these find- effects was inconsistent with the a-priori expec- ings are robust across the four individual items tation that abstracts from higher-status US uni- that make up our composite abstract-score. versities would be favoured over abstracts from None of these items show notable variations in lower-status US universities, and that abstracts abstract ratings across manipulations (Figure 1— from lower-status US universities would be fav- figure supplement 1–4). oured over abstracts from lower-status universi- Figures 2 and 3 report the outcomes of the ties elsewhere. discipline-specific logit models with Open full- As displayed in Figure 3, peer-evaluators in text and Include in conference as outcomes. The political science were between 7.0% and 59.4% uncertainty of the estimates is notably larger in less likely to consider an abstract relevant for a these models than for the one-way ANOVAs selective, international conference program, reported above, indicating wider variability in when the abstract’s author was affiliated with a the peer-evaluators’ dichotomous assessments. higher-status US university compared to a lower- As displayed in Figure 2, peer-evaluators in status US university (Odds ratio:. 613, CI:. 406- Public Health were between 7.5% and 56.5% .930). This result goes against a-priori expecta- less likely to show interest in opening the full- tions concerning the influence of status signals text, when the author affiliation was changed on evaluative outcomes. The remaining five

Nielsen et al. eLife 2021;10:e64561. DOI: https://doi.org/10.7554/eLife.64561 5 of 13 Feature Article Meta-Research Weak evidence of country- and institution-related status bias in the peer review of abstracts

models range from 0.86 to 1.05 and have 99% confidence intervals spanning the line of no dif- A ference. The 99% confidence intervals (panel a) indicate that any plausible effect would fall within the bounds of odds ratio = 0.68 and 1.35, which corresponds to a standardized confidence bound of Cohen’s d = À0.21 to 0.17. The wide confidence bounds for the estimated proportion for Include in conference (panel b) reflect the large variations in average assessments of B abstracts across disciplines. Robustness checks based on mixed linear and logit models were carried out to examine the effects of the experimental manipulations on the three outcome measures, while restricting the samples to (i) participants that responded cor- rectly to a manipulation-check item, and (ii) participants that saw their own research as being ‘extremely close’, ‘very close’ or ‘somewhat close’ to the subject addressed in the abstract. Figure 4. Plots of fixed coefficients and adjusted means derived from the mixed-linear All of these models yielded qualitatively similar regression model with "Abstract score" as outcome. Panel A plots the fixed coefficients for results, with small residual effects and confi- manipulation 1 (high-status university, US) and manipulation 3 (lower-status university, non- dence intervals spanning 0 in the linear regres- US). Manipulation 2 (lower-status university, US) is the reference group. Panel B plots the sions and one in the logistic regressions (see adjusted means for manipulation 1, manipulation 2 and manipulation 3. Error bars represent 99% CIs. The figure shows that status cues in the form of institutional affiliation or national Supplementary file 2, Tables S28–S33). More- affiliation have no tangible effects on the respondents’ assessments of abstracts. For model over, a pre-registered interaction analysis was specifications, see Supplementary file 2, Table S25. conducted to examine whether the influence of country- and institution-related status signals was moderated by any of the following charac- disciplines had 95% confidence intervals span- teristics of the peer evaluators: (i) their descrip- ning one, and in six of the 12 comparisons, the tive beliefs in the objectivity and fairness of direction of the observed effects was inconsis- peer-evaluation; (ii) their structural location in tent with a-priori expectations. As indicated in the science system (in terms of institutional affili- panel b, we observe notable variation in the par- ation and scientific rank); (iii) their research ticipants’ average responses to the Include in accomplishments; (iv) their self-perceived scien- conference item across disciplines. tific status. All of these two-way interactions had Figure 4 plots the fixed coefficients (panel a) 99% CI intervals spanning 0 in the linear regres- and adjusted means (panel b) from the mixed sions and one in the logistic regressions, indicat- linear regression with Abstract rating as out- ing no discernible two-way interactions (see come. In accordance with the pre-registered Supplementary file 2, Tables S34–S39). analysis plan, we report the mixed regression models with 99% confidence intervals. The fixed coefficients for the two between-group compari- Discussion sons tested in this model are close to zero, and Contrary to the idea of halo effects, our study the 99% confidence intervals suggest that any shows weak and inconsistent evidence of coun- plausible difference would fall within the bounds try- or institution-related status bias in abstract of À0.26 to 0.28 points on the 16-point abstract- ratings. In the discipline-specific analyses, we rating scale. These confidence bounds can be observed statistically significant differences in converted into standardized effects of Cohen’s two of 36 pairwise comparisons (i.e. 5.6%) and, d = À0.10 to 0.11. of these, one was inconsistent with our a-priori Figure 5 displays odds ratios and 99% confi- expectation of a bias in favour of high-status dence intervals for the mixed logit regressions sources. Moreover, the estimated confidence with Open full-text (panel a, upper display) and bounds derived from the multilevel regressions Include in conference (panel a, lower display) as were either very small or small according to outcomes. The odds ratios for the experimental Cohen’s classification of effect sizes manipulations used as predictors in these (Cohen, 2013). These results align with the

Nielsen et al. eLife 2021;10:e64561. DOI: https://doi.org/10.7554/eLife.64561 6 of 13 Feature Article Meta-Research Weak evidence of country- and institution-related status bias in the peer review of abstracts

A A B

Figure 5. Plots of odds ratios and adjusted means derived from the mixed logit regressions with "Open full text" (upper part) and "Include in conference" (lower part) as outcomes. Panel A displays the odds ratios for respondents exposed to manipulation 1 (high-status university, US) or manipulation 3 (lower-status university, non-US). Manipulation 2 (lower-status university, US) is the reference group. Panel B plots the adjusted means for manipulation 1, manipulation 2 and manipulation 3. Error bars represent 99% CIs. As shown in Panel A, the results for both regression models are inconclusive, and the effect sizes are small. For model specifications, see Supplementary file 2, Tables S26–27.

outcomes of three out of five existing experi- they may still omit to cite them, and instead ments, which also show small and inconsistent frame their contributions in the context of more effects of country- and institution-related status high-status sources. In the future, researchers bias in peer-evaluation (Blank, 1991; might add to this work by examining if these Harris et al., 2015; Harris et al., 2017). How- more subtle processes contribute to shape the ever, it should be noted that even small effects reception and uptake of scientific knowledge. may produce large population-level variations if Third, our conclusion of little or no bias in they accumulate over the course of scientific abstract review does not necessarily imply that careers. Moreover, one could argue that small biases are absent in other types of peer assess- effects would be the expected outcome for a ment, such as peer reviewing of journal and lightweight manipulation like ours in an online grant submissions, where evaluations usually fol- survey context, where the participants are asked low formalized criteria and focus on full-text to make decisions without real-world implica- manuscripts and proposals. Likewise, our study tions. Indeed, it is possible that the effects of does not capture how country- and institution- country- and institution level status signals would related status signals may influence the decisions be larger in real-world evaluation processes, of journal editors. Indeed, journal editors play an where reviewers have more at stake. important role in establishing scientific merits Certain scope conditions also delimit the sit- and reputations and future experimental work uations to which our conclusions may apply. should examine how halo effects may shape edi- First, our findings leave open the possibility that torial decisions. Fourth, although we cover a peer-evaluators discard research from less repu- broad span of research areas, it is possible that table institutions and science nations without our experiment would have produced different even reading their abstracts. In other words, our results for other disciplines. Fifth, it should be study solely tests whether scientists, after being noted that our two dichotomous outcome items asked to carefully read an assigned abstract, on (open full text and include in conference) refer average will rate contributions from prestigious to two rather different evaluative situations that locations more favourably. may be difficult to compare. For instance, a Second, even when peer-evaluators identify researcher may wish to read a paper (based on and read papers from “lower-status” sources, its abstract) while finding it inappropriate for a

Nielsen et al. eLife 2021;10:e64561. DOI: https://doi.org/10.7554/eLife.64561 7 of 13 Feature Article Meta-Research Weak evidence of country- and institution-related status bias in the peer review of abstracts

conference presentation and vice versa. More- disciplinary examination of how country- over, the competitive aspect of selecting an and institution-related status signals influence abstract for a conference makes this evaluative the reception of scientific research. In a con- situation quite different from the decision to trolled experiment that spanned six disciplines, open a full text. we found no systematic evidence of status bias A key limitation is that our experiment was against abstracts from lower status universities conducted in non-probability samples with low and countries. Future research should add to response rates, which raises questions about this work by examining if other processes selection effects. One could speculate that the related to the social, material and intellectual scientists that are most susceptible to status bias organization of science contribute to producing would be less likely to participate in a study con- and reproducing country- and institution-related ducted by researchers from two public universi- status hierarchies. ties in Denmark. Moreover, we decided to include internationally outstanding universities in our high-status category (California Institute of Methods Technology, Columbia University, Harvard Uni- Our study complies with all ethical regulations. versity, Yale University, Johns Hopkins Univer- Aarhus University’s Institutional Review Board sity, Massachusetts Institute of Technology, approved the study (case no. 2019-616-000014). Princeton University, Stanford University, and We obtained informed consent from all partici- University of California, Berkeley). By so doing, pants. The sampling and analysis plan was pre- we aimed to ensure a relatively strong status sig- registered at the Open Science Framework on nal in the abstract’s byline. There is a risk that September 18, 2019. We have followed all of this focus on outstanding institutions may have the steps presented in the registered plan, with led some survey participants to discern the pur- two minor deviations. First, we did not preregis- pose of the experiment and censor their ter the equivalence tests reported for the disci- responses to appear unbiased. However, all par- pline-specific analyses presented in Figure 1. ticipants were blinded to the study objectives Second, in the results section, we report the out- (the invitation email is included in comes of the mixed regression models with Supplementary file 1), and to reduce the abstract score as outcome based on linear mod- salience of the manipulation, we asked each par- els instead of the tobit models included in the ticipant to review only one abstract. Moreover, registered plan. Tables S40–S44 in in the minds of the reviewers, many other factors Supplementary file 2 report the outcomes of than the scholarly affiliations might have been the mixed-tobit models, which are nearly identi- manipulated in the abstracts, including the gen- cal to the results from the linear models der of the authors (Knobloch-Westerwick et al., reported in Tables S25, S28, S31, S36, S37 in 2013; Forscher et al., 2019), the style of writ- Supplementary file 2. ing, the source of the empirical data presented in the abstract, the clarity of the statistical Participants reporting, the reporting of positive vs. negative The target population consisted of research- and mixed results (Mahoney, 1977), the report- active academics with at least three articles pub- ing of intuitive vs. counter-intuitive findings lished between 2015 and 2018 in Clarivate’s (Mahoney, 1977; Hergovich et al., 2010), and Web of Science (WoS). To allow for the retrieval the reporting of journal information and visibility of contact information (specifically email metrics (e.g. citation indicators or Altmetric addresses), we limited our focus to correspond- scores) (Teplitskiy et al., 2020). A final limitation ing authors with a majority of publications falling concerns the varying assessments of abstract into one of the following six disciplinary catego- quality across disciplines. This cross-disciplinary ries: astronomy, cardiology, materials science, variation was particularly salient for the Include political science, psychology and public health. in conference item (Figure 3, panel b), which These disciplines were chosen to represent the may indicate differences in the relative difficulty top-level domains; natural science, technical sci- of getting an oral presentation at conferences in ence, health science, and social science. We did the six disciplines. In the mixed-regression mod- not include the arts and humanities as the major- els based on data from all six disciplines, we ity of fields in this domain have very different tra- attempt to account for this variation by including ditions of publishing and interpret scholarly discipline as random-effect parameter. In sum- quality in less comparable terms. While other mary, this paper presents a large-scale, cross- fields could have been chosen as representative

Nielsen et al. eLife 2021;10:e64561. DOI: https://doi.org/10.7554/eLife.64561 8 of 13 Feature Article Meta-Research Weak evidence of country- and institution-related status bias in the peer review of abstracts

of those domains, practical aspects of access to because they did not see themselves as belong- field experts and coverage in Web of Science ing to one of the targeted disciplines. Others were deciding for the final delineation. We used may not have responded because they were on the WoS subject categories and the Centre for sabbatical, parental leave or sick leave. More- Science and Technology Studies’ (CWTS) meso- over, approximately 16 percent (15,247) of the level cluster classification system to identify eligi- targeted email addresses were inactive or ble authors within each of these disciplines. For bounced for other reasons. A crude estimate of all fields, except for materials science, the WoS the response rate would thus be 5,413/(95,317– subject categories provided a useful field delin- 15,247)=0.07, or seven percent. The gender eation. In materials science, the WoS subject cat- composition of the six respondent samples egories were too broad. Hence, we used the largely resembles that of the targeted WoS pop- article-based meso-level classification of CWTS ulations (Supplementary file 2, Table S46). to identify those papers most closely related to However, the average publication age (i.e. years the topic of our abstract. In our sampling strat- since first publication in WoS) is slightly higher in egy, we made sure that no participants were asked to review abstracts written by (fictive) the respondent samples compared to the tar- authors affiliated with their own research geted WoS populations, which may be due to institutions. the study’s restricted focus on corresponding We used G*Power to calculate the required authors (the age distributions are largely similar sample size for detecting a Cohen’s f = 0.15 across the recruitment and respondent samples). (corresponding to a Cohen’s d = 0.30) or larger The distribution of respondents (per discipline) with a = 0.05 and a power of 0.80 in the disci- across countries in WoS, the recruitment sample, pline-specific analyses with abstract rating as and the respondent sample is reported in outcome. With these assumptions, a comparison Supplementary file 2, Table S47. of three groups would require at least 429 respondents per discipline. Response rates for e- Pretesting and pilot testing mail-based academic surveys are known to be Prior to launching, the survey was pretested with low (Myers et al., 2020). Based on the out- eight researchers in sociology, information sci- comes of a pilot study targeting neuroscientists, ence, political science, psychology, physics, bio- we expected a response rate around 5% and dis- medicine, clinical medicine, and public health. In tributed the survey to approximately 72,000 the pretesting, we used verbal probing techni- researchers globally (i.e. approximately 12,000 ques and “think alouds” to identify questions researchers per field) (for specifications, see that the participants found vague and unclear. Supplementary file 2, Table S45). All data were Moreover, we elicited how the participants collected in October and November 2019. Due arrived at answers to the questions and whether to low response rates in materials science and the questions were easy or hard to answer, and cardiology, we expanded the recruitment sam- why. ples by an additional ~12,000 scientists in each In addition, we pilot-tested the survey in a of these disciplines. In total, our recruitment sample of 6,000 Neuroscientists to (i) estimate sample consisted 95,317 scientists. Each scientist was invited to participate in the survey by email, the expected response rate per discipline, (ii) and we used the web-based Qualtrics software check the feasibility of a priming instrument that for data collection. We sent out five reminders we chose not to include in the final survey, (iii) and closed the survey two weeks after the final detect potential errors in the online version of reminder. Eight percent (N = 7,401) of the the questionnaire before launching, and (iv) ver- invited participants opened the email survey ify the internal consistency of two of the com- link, and about six percent (N = 5,413) composite measures used in the survey (i.e. abstract pleted the questionnaire (for specifications on score and meritocracy beliefs). discipline-specific completion rates, see Supplementary file 2, Table S45). For ethical Procedures reasons, our analysis solely relies on data from In each of the six online-surveys (one per disci- participants that reached the final page of the pline), we randomly assigned participants to one survey, where we debriefed about the study’s of the three manipulations (Table 1). All partici- experimental manipulations. The actual response pants were blinded to the study objectives. rate is difficult to estimate. Some scientists may have refrained from participating in the study

Nielsen et al. eLife 2021;10:e64561. DOI: https://doi.org/10.7554/eLife.64561 9 of 13 Feature Article Meta-Research Weak evidence of country- and institution-related status bias in the peer review of abstracts Manipulations university, per discipline, are listed in We manipulated information about the Supplementary file 2. Table S48, S49. abstract’s institutional source (high status vs. The abstracts were created or adapted for lower status US research institution) and country this study and are not published in their current source (lower status US research institution vs. form. The abstracts used in astronomy, materials lower-status research institution in a select science, political science and psychology were group of European, Asian, Middle Eastern, Afri- provided by relevant researchers in the respec- tive disciplines and have been slightly edited for can and South American countries). The follow- the purposes of this study. The abstracts used in ing criteria guided our selection of universities cardiology and public health represent rewritten for the manipulation of institutional affiliation: versions of published abstracts with numerous Candidates for the “high status” category all alterations to mask any resemblance with the ranked high in the US National Research Coun- published work (the six abstracts are available in cil’s field-specific rankings of doctorate pro- Supplementary file 1). Author names were grams, and consistently ranked high (Top 20) in selected by searching university websites for five subfield-specific university rankings (US each country and identifying researchers in disci- News’ graduate school ranking, Shanghai rank- plines unrelated to this study. ing, Times Higher Education, Leiden Ranking and QS World University Ranking). Measures The universities assigned to the “lower sta- Variable specifications are reported in tus” category were selected based on the PP- Supplementary file 2, Table S50. The outcome top 10% citation indicator used in the Leiden variables used in this analysis are specified University Ranking. For non-US universities, we above. We used dichotomous variables to esti- limited our focus to less esteemed science mate the effect of the manipulations on the out- nations in Southern Europe, Eastern Europe, comes in all regression models. We used the Latin America, Asia, Africa and the Middle East. following measures to compute the moderation Institutions assigned to the “lower-status” cate- variables included in the two-way interaction gory were matched (approximately) across coun- analyses. Our measure of the respondents’ tries with respect to PP-top 10% rank in the descriptive beliefs in the objectivity and fairness field-specific Leiden University ranking. We of peer-evaluation in their own research field decided to restrict our focus to universities in (i.e. meritocratic beliefs) was adapted from ref the Leiden ranking to ensure that all fictive affili- (Anderson et al., 2010). A sample item from ations listed under the abstracts represented this measure reads: “In my research field, scien- research active institutions in the biomedical and tists evaluate research primarily on its merit, i.e. health sciences, the physical sciences and engi- according to accepted standards of the field”. neering, or the social sciences. By matching We adapted the sample item from ref lower-status universities on their PP-top 10% (Anderson et al., 2010). The two other items were developed for this study. Ratings were rank, we ensured that the lower-status universi- based on a five-point scale ranging from (1) ties selected for each discipline had a compara- ‘Strongly agree’ to (5) ‘Strongly disagree’. Based ble level of visibility in the scientific literature. on these items, we computed a composite mea- Since a given university’s rank on the PP-top sure that specifies each participant’s total-item 10% indicator may not necessarily align with its score for these three items (i.e. meritocratic general reputation, we also ensured that none of beliefs) (Cronbach’s a = 0.765). We used two the lower-status universities were within the top- pieces of information to measure the partici- 100 in the general Shanghai, Times Higher Edu- pants’ structural location in the science system cation and QS World University rankings. (i.e. structural location): (i) information about sci- Given this approach, the specific institutions entific rank collected through the survey, and (ii) that fall into the “high status” and “lower sta- information about scientific institution obtained tus” categories vary by discipline. We carried from Web of Science. Our measure of structural out manual checks to ensure that all of the location is dichotomous. Associate professors, selected universities had active research environ- full professors, chairs and deans at top ranked ments within the relevant research disciplines. international research institutions are scored as The universities assigned to each category (high- 1; all other participants are scored as 0. Here, status [US], lower-status [US], lower-status [non- we define top-ranked research institutions as US]), and the average abstract scores per institutions that have consistently ranked among

Nielsen et al. eLife 2021;10:e64561. DOI: https://doi.org/10.7554/eLife.64561 10 of 13 Feature Article Meta-Research Weak evidence of country- and institution-related status bias in the peer review of abstracts

the top 100 universities with the highest propor- and abstract rating. The tobit models were spec- tion of top 10% most cited papers within the ified with a left-censoring at four and a right- past 10 years, according to the Leiden Ranking. censoring at 20. Figure 1—figure supplement 5 We used article metadata from WOS to con- displays the data distribution for the outcome struct an author-specific performance profile for measure abstract rating. The data distribution each respondent (i.e. research accomplish- for this measure was assumed to be normal. ments). Specifically, we assigned researchers We estimated multilevel logistic regressions that were among the top-10% most cited in their with disciplines as random effect parameter to fields (based on cumulative citation impact) to examine the relationship between the manipula- the “high status” group. All other participants tions and the outcome variables Open full-text were assigned to the “lower status” group. Our and Include in conference. Consistent with our measure of self-perceived status was adapted pre-registered analysis plan, we used 95% confi- from the MacArthur Scale of Subjective Social dence intervals to make inferences based on the Status (Adler and Stewart, 2007). We asked the discipline specific ANOVAs and logistic regres- respondents to locate themselves on a ladder sions. To minimize Type I errors arising from with ten rungs representing the status hierarchy multiple testing, we reported the results of all in their research area. Respondents that posi- multilevel regression models with 99% confi- tioned themselves at the top of the ladder were dence intervals. scored as 9, and respondents positioning them- We created two specific samples for the selves at the bottom were scored as 0. moderation analyses. The first of these samples only include respondents that had been exposed Manipulation and robustness checks to abstracts from a high status US university or a As a manipulation check presented at the end of lower status US university. The second sample the survey, we asked the participants to answer was restricted to respondents that had been a question about the author’s institutional affilia- exposed to abstracts from a lower status US uni- tion/country affiliation in the abstract that they versity or a lower status university outside the just read. The question varied depending on US. The Variance Inflation Factors for the predic- manipulation and discipline. For robustness tors included in the moderation analyses (i.e. the checks, we included an item to measure the per- manipulation variables, Meritocracy beliefs, ceived distance between the participant’s own Structural Location, Research Accomplishments research area and the topic addressed in the and Self-perceived status) were all below two. abstract. Responses were based on a five-point We conducted the statistical analyses in scale ranging from (1) ‘Extremely close’ to (5) STATA 16 and R version 4.0.0. For the multilevel ‘Not close at all’. In the statistical analysis, these linear, tobit and logit regressions, we used the response options were recoded into dichoto- “mixed”, “metobit” and “melogit” routines in mous categories (‘Not close at all’, ‘Not too STATA. Examinations of between-group equiva- close’=0, ‘Somewhat close’, ‘Very close’, lence were performed with the R package ‘TOS- ‘Extremely close’=1). TER’ (Lakens, 2017b). Standardized effects were calculated using the R package ‘esc’ Data exclusion criteria (Lu¨decke, 2018). In accordance with our registered plan, respond- Acknowledgements ents that demonstrated response bias (10 items The Centre for Science and Technology Studies of the same responses, e.g. all ones or fives) (CWTS) at Leiden University generously pro- were removed from the analysis. Moreover, we vided bibliometric indices and article metadata. removed all respondents that completed the We thank Emil Bargmann Madsen, Jesper survey in less than 2.5 min. Wiborg Schneider and Friedolin Merhout for Statistical analysis very useful comments on the manuscript. We used one-way ANOVAs and logit models to Mathias Wullum Nielsen is in the Department of perform the discipline-specific, between-group Sociology, University of Copenhagen, Copenhagen, comparisons. We estimated mixed linear regres- Denmark sions and tobit models (reported in [email protected] Supplementary file 2) with disciplines as ran- https://orcid.org/0000-0001-8759-7150 dom effect parameter to measure the relation- Christine Friis Baker is in the Danish Centre for ship between the experimental manipulations Studies in Research and Research Policy, Department

Nielsen et al. eLife 2021;10:e64561. DOI: https://doi.org/10.7554/eLife.64561 11 of 13 Feature Article Meta-Research Weak evidence of country- and institution-related status bias in the peer review of abstracts

of Political Science, Aarhus University, Aarhus, Author response https://doi.org/10.7554/eLife.64561. Denmark sa2 https://orcid.org/0000-0002-3370-021X Emer Brady is in the Danish Centre for Studies in Research and Research Policy, Department of Political Additional files Science, Aarhus University, Aarhus, Denmark Supplementary files https://orcid.org/0000-0002-6065-8096 . Supplementary file 1. Qualtrics survey (astronomy); Michael Bang Petersen is in the Department of the six abstracts used in the survey experiment; email Political Science, Aarhus University, Aarhus, Denmark invitation. https://orcid.org/0000-0002-6782-5635 . Supplementary file 2. Tables S1-S50. Jens Peter Andersen is in the Danish Centre for . Transparent reporting form Studies in Research and Research Policy, Department of Political Science, Aarhus University, Aarhus, Data availability Denmark All data and code needed to evaluate the conclusions https://orcid.org/0000-0003-2444-6210 are available here: https://osf.io/x4rj8/.

Author contributions: Mathias Wullum Nielsen, Con- The following dataset was generated: ceptualization, Resources, Data curation, Formal analysis, Supervision, Funding acquisition, Validation, Database and Author(s) Year Dataset Identifier Investigation, Visualization, Methodology, Writing - URL original draft, Project administration, Writing - review and editing; Christine Friis Baker, Emer Brady, Formal Nielsen MW 2020 https://osf.io/ Open Science x4rj8/ Framework, analysis, Investigation, Visualization, Methodology, x4rj8 Writing - review and editing; Michael Bang Petersen, Formal analysis, Validation, Investigation, Methodol- ogy, Writing - review and editing; Jens Peter Ander- References sen, Conceptualization, Data curation, Formal analysis, Adler N, Stewart J. 2007. The MacArthur scale of Funding acquisition, Validation, Investigation, Visuali- subjective social status. https://macses.ucsf.edu/ zation, Methodology, Project administration, Writing - research/psychosocial/subjective.phpMarch 18, 2021]. review and editing Anderson MS, Ronning EA, Devries R, Martinson BC. 2010. Extending the Mertonian norms: scientists’ Competing interests: The authors declare that no subscription to norms of research. The Journal of competing interests exist. Higher Education 81:366–393. DOI: https://doi.org/10. 1080/00221546.2010.11779057, PMID: 21132074 Received 03 November 2020 Berger J, Fisek H, Norman R, Zelditch M. 1977. Status Accepted 08 March 2021 Characteristics and Social Interaction. New York: Published 18 March 2021 Elsevier. Blank RM. 1991. The effects of double-blind versus Ethics: Human subjects: Aarhus University’s Institu- single-blind reviewing: experimental evidence from the tional Review Board approved the study. We obtained american economic review. The American Economic informed consent from all participants (case no. 2019- Review 81:1041–1067. Cohen J. 2013. Statistical Power Analysis for the 616-000014). Behavioral Sciences. Cambridge, MA: Academic Press. DOI: https://doi.org/10.4324/9780203771587 Funding Collins R. 1998. The Sociology of Philosophies. Grant reference Cambridge, MA: Harvard University Press. Funder number Author Correll S, Bernard S. 2015. Biased estimators? Carlsbergfondet CF19-0566 Mathias Wullum Comparing status and statistical theories of gender Nielsen discrimination. In: Thye S. R, Lawler E. J (Eds). Advances in Group Processes. 23 Bingley: Emerald Aarhus Universi- AUFF-F-2018-7-5 Christine Friis Group Publishing Limited. p. 89–116. DOI: https://doi. tets Forsknings- Baker fond Emer Brady org/10.1016/S0882-6145(06)23004-2 Jens Peter Crane D. 1965. Scientists at major and minor Andersen universities: a study of productivity and recognition. American Sociological Review 30:699–714. The funders had no role in study design, data collection DOI: https://doi.org/10.2307/2091138, PMID: 5824935 and interpretation, or the decision to submit the work Crane D. 1967. The gatekeepers of science: some for publication. factors affecting the selection of articles for scientific journals. The American Sociologist 2:195–201. Forscher PS, Cox WTL, Brauer M, Devine PG. 2019. Little race or gender bias in an experiment of initial Decision letter and Author response review of NIH R01 grant proposals. Nature Human Decision letter https://doi.org/10.7554/eLife.64561.sa1 Behaviour 3:257–264. DOI: https://doi.org/10.1038/ s41562-018-0517-y, PMID: 30953009

Nielsen et al. eLife 2021;10:e64561. DOI: https://doi.org/10.7554/eLife.64561 12 of 13 Feature Article Meta-Research Weak evidence of country- and institution-related status bias in the peer review of abstracts

Greenwald AG, Banaji MR. 1995. Implicit social Link AM. 1998. US and non-US submissions: an cognition: attitudes, self-esteem, and stereotypes. analysis of reviewer bias. JAMA 280:246–247. Psychological Review 102:4–27. DOI: https://doi.org/ DOI: https://doi.org/10.1001/jama.280.3.246, PMID: 10.1037/0033-295X.102.1.4, PMID: 7878162 9676670 Harris M, Macinko J, Jimenez G, Mahfoud M, Lu¨ decke D. 2018. Esc: Effect Size Computation for Anderson C. 2015. Does a research article’s country of Meta-Analysis. R package version 0, 4(1)42. https:// origin affect perception of its quality and relevance? A rdrr.io/cran/esc/ national trial of US public health researchers. BMJ Magee JC, Galinsky AD. 2008. Social hierarchy: The Open 5:e008993. DOI: https://doi.org/10.1136/ self-reinforcing nature of power and status. Academy bmjopen-2015-008993, PMID: 26719313 of Management Annals 2:351–398. DOI: https://doi. Harris M, Marti J, Watt H, Bhatti Y, Macinko J, Darzi org/10.5465/19416520802211628 AW. 2017. Explicit bias toward high-income-country Mahoney MJ. 1977. Publication prejudices: an research: a randomized, blinded, crossover experiment experimental study of confirmatory bias in the peer of English clinicians. Health Affairs 36:1997–2004. review system. Cognitive Therapy and Research 1:161– DOI: https://doi.org/10.1377/hlthaff.2017.0773 175. DOI: https://doi.org/10.1007/BF01173636 Hergovich A, Schott R, Burger C. 2010. Biased Medoff MH. 2006. Evidence of a Harvard and Chicago evaluation of abstracts depending on topic and Matthew Effect. Journal of Economic Methodology 13: conclusion: further evidence of a confirmation bias 485–506. DOI: https://doi.org/10.1080/ within scientific psychology. Current Psychology 29: 13501780601049079 188–209. DOI: https://doi.org/10.1007/s12144-010- Merton RK. 1968. The Matthew effect in science. 9087-5 Science 159:56–63. DOI: https://doi.org/10.1126/ Jost JT, Pelham BW, Carvallo MR. 2002. Non- science.159.3810.56 conscious forms of system justification: implicit and Merton RK. 1988. The Matthew effect in science, II: behavioral preferences for higher status groups. Cumulative advantage and the symbolism of Journal of Experimental Social Psychology 38:586–602. intellectual property. Isis 79:606–623. DOI: https://doi. DOI: https://doi.org/10.1016/S0022-1031(02)00505-X org/10.1086/354848 Jost JT, Banaji MR, Nosek BA. 2004. A decade of Murray D, Siler K, Larivie´re V, Chan WM, Collings AM, system justification theory: accumulated evidence of Raymond J, Sugimoto CR. 2018. Gender and conscious and unconscious bolstering of the status international diversity improves equity in peer review. quo. Political Psychology 25:881–919. DOI: https://doi. bioRxiv. DOI: https://doi.org/10.1101/400515 org/10.1111/j.1467-9221.2004.00402.x Myers KR, Tham WY, Yin Y, Cohodes N, Thursby JG, Jost JT, Rudman LA, Blair IV, Carney DR, Dasgupta N, Thursby MC, Schiffer P, Walsh JT, Lakhani KR, Wang Glaser J, Hardin CD. 2009. The existence of implicit D. 2020. Unequal effects of the COVID-19 pandemic bias is beyond reasonable doubt: a refutation of on scientists. Nature Human Behaviour 4:880–883. ideological and methodological objections and DOI: https://doi.org/10.1038/s41562-020-0921-y, executive summary of ten studies that no manager PMID: 32669671 should ignore. Research in Organizational Behavior 29: Ridgeway CL. 2001. Blackwell Handbook of Social 39–69. DOI: https://doi.org/10.1016/j.riob.2009.10.001 Psychology: Group Processes. Blackwell. DOI: https:// Jost JT. 2013. Cognitive social psychology. In: doi.org/10.1002/9780470998458 Moskowitz G. B (Ed). Social Cognition. Psychology Ross JS, Gross CP, Desai MM, Hong Y, Grant AO, Press. p. 92–105. Daniels SR, Hachinski VC, Gibbons RJ, Gardner TJ, Knobloch-Westerwick S, Glynn CJ, Huge M. 2013. Krumholz HM. 2006. Effect of blinded peer review on The Matilda effect in science communication: an abstract acceptance. JAMA 295:1675–1680. experiment on gender bias in publication quality DOI: https://doi.org/10.1001/jama.295.14.1675, perceptions and collaboration interest. Science PMID: 16609089 Communication 35:603–625. DOI: https://doi.org/10. Son Hing LS, Bobocel DR, Zanna MP, Garcia DM, Gee 1177/1075547012472684 SS, Orazietti K. 2011. The merit of meritocracy. Journal Lakens D. 2017a. Equivalence tests: a practical primer of Personality and Social Psychology 101:433–450. for t tests, correlations, and meta-analyses. Social DOI: https://doi.org/10.1037/a0024618, PMID: 217870 Psychological and Personality Science 8:355–362. 93 DOI: https://doi.org/10.1177/1948550617697177, Teplitskiy M, Duede E, Menietti M, Lakhani KR. 2020. PMID: 28736600 Citations systematically misrepresent the quality and Lakens D. 2017b. TOSTER: Two One-Sided Tests impact of research articles: survey and experimental (TOST) Equivalence Testing. R package version 0. 2- evidence from thousands of citers. arXiv. https://arxiv. 648. https://rdrr.io/cran/TOSTER/ org/abs/2002.10033. Larsen PO, von Ins M. 2010. The rate of growth in Tomkins A, Zhang M, Heavlin WD. 2017. Reviewer scientific publication and the decline in coverage bias in single- versus double-blind peer review. PNAS provided by Science Citation Index. Scientometrics 84: 114:12708–12713. DOI: https://doi.org/10.1073/pnas. 575–603. DOI: https://doi.org/10.1007/s11192-010- 1707323114, PMID: 29138317 0202-z, PMID: 20700371

Nielsen et al. eLife 2021;10:e64561. DOI: https://doi.org/10.7554/eLife.64561 13 of 13