1

The Advantage for Studies of Human Electrophysiology:

Impact on and

Peter E. Clayson*1, Scott A. Baldwin2, and Michael J. Larson2,3

1Department of Psychology, University of South Florida, Tampa, FL

2Department of Psychology, Brigham Young University, Provo, UT

3Neuroscience Center, Brigham Young University, Provo, UT

*Corresponding author at: Department of Psychology, University of South Florida, 4202 East

Fowler Avenue, Tampa, FL, US, 33620-7200. Email: [email protected]

2

Disclosure

Michael J. Larson, PhD, is the Editor-in-Chief of the International Journal of Psychophysiology.

Editing of the was handled by a separate editor and Dr. Larson was blinded from viewing the reviews or comments as well as the identities of the reviewers.

3

Abstract

Barriers to accessing scientific findings contribute to knowledge inequalities based on financial resources and decrease the transparency and rigor of scientific research. Recent initiatives aim to improve access to research as well as methodological rigor via transparency and openness. We sought to determine the impact of such initiatives on open access in the sub-area of human electrophysiology and the impact of open access on the attention articles received in the scholarly literature and other outlets. Data for 35,144 articles across 967 journals from the last 20 years were examined. Approximately 35% of articles were open access, and the rate of publication of open-access articles increased over time. Open access articles showed 9 to 21% more PubMed and CrossRef citations and 39% more mentions than closed access articles. Green open access articles (i.e., author archived) did not differ from non-green open access articles (i.e., publisher archived) with respect to citations and were related to higher

Altmetric mentions. These findings demonstrate that open-access publishing is increasing in popularity in the sub-area of human electrophysiology and that open-access articles enjoy the

“open access advantage” in citations similar to the larger . The benefit of the open access advantage may motivate researchers to make their publications open access and pursue publication outlets that support it. In consideration of the direct connection between citations and journal , journal editors may improve the accessibility and impact of published articles by encouraging authors to self-archive manuscripts on servers.

Keywords: open access, advantage, human electrophysiology (EEG), event-related potentials (ERPs), 4

Open and rigorous scientific practices are increasingly applied throughout the scientific community. These practices include methodological transparency, openly available data, pre- registration of studies, replication of prominent effects, and open access publications. The shift to open and transparent science is part of a concerted effort to improve the rigor, validity, replicability, and availability of research (Nosek & Bar-Anan, 2012; Nosek & Lakens, 2014).

Newfound availability of data and methodology repositories such as the

Framework (OSF), Zenodo, or Databrary (see Gilmore, Kennedy, & Adolph, 2018), and preprint servers such as the ArXiv portfolio provide mechanisms for scientists to make their work accessible. The number of scientists posting their data and code and making their manuscripts available via preprints or other open access formats is increasing (e.g., Hardwicke et al., 2018;

Laakso et al., 2011). Specific scientific areas are raising their focus on access to open and rigorous scientific practices. For example, two flagship journals of international societies in psychophysiology, the International Journal of Psychophysiology and Psychophysiology, offer the registered reports format (along with over 250 total journals and growing; see https://www.cos.io/initiatives/registered-reports), and journals within and psychology, such as the highly-cited journals Communications, NeuroImage, and

Advances in Methods and Practices in Psychological Sciences, have shifted to only open-access formats.

The Budapest Open Access Initiative was among the first to call for increases in self- archiving of manuscripts and availability of open access journals in 2001 as part of an effort to reduce barriers to accessing scholarly works (https://www.budapestopenaccessinitiative.org/).

Similarly, funding agencies, such as the National Institutes of Health (NIH) in the United States or Wellcome Trust in the United Kingdom, require open-access posting of funded studies—often 5 after an embargo period, such as on PubMed Commons for those that receive NIH funding.

Mandating open access is associated with increased rates of manuscript availability as well as increased article impact (Gargouri et al., 2010; Vincent-Lamarre, Boivin, Gargouri, Larivière, &

Harnad, 2015). Approximately 28% of the scholarly literature is currently considered open access (Piwowar et al., 2018)—although these numbers may be an underestimate as multiple publishers increased their open access inventory during the COVID-19 pandemic (Grove, 2020).

Specific sub-areas of science, such as clinical neuroscience, report open access rates as high as

49% (Hanson, Almeida, Traylor, Rajagopalan, & Johnson, 2020). In addition, most publishers now have provisions within their copyright policies for authors to post preprints and/or final non- copy edited and formatted drafts (sometimes referred to as post-prints).

Despite the increase in availability and use of open access formats, there are costs associated with the production and publication of research. Digital space for hosting repositories, web interfaces for manuscript submission and review, typesetting and copy editing, and printing and distribution charges are some of the costs that make open-access publication cost-ineffective for publishers. In the absence of a subscription to cover these costs in open access journals

(or for open access articles), publishers often use digital-only formats or include article processing charges for open access articles that range from reasonable to exorbitant (over $5,000

United States dollars for some journals). Article processing charges may disproportionately affect researchers from countries with lower currency exchange rates (Dove, Chan, Thoma,

Roland, & Bruijns, 2019). Article processing charges shift the burden from institutions or library systems to individual authors and can, problematically, restrict open access publishing to those with grant funding or other financial means. Due to these article processing charges, grant- funded research that is vetted by funding agency reviewers prior to study initiation may be more 6 likely to be fully open access—potentially influencing the quality and perception of open access research.

In of these article processing charges and motivations to increase accessibility to published research, most publishers employ “tiers” of open access with different corresponding fee structures. In gold open access, typically the costliest form of open access, the manuscript is made available by the publisher and is freely available on journal or publisher websites and through search engines. Green open access, in contrast, refers to free or low-cost self-archiving of manuscripts wherein authors upload preprints or post-prints of the work or the article is made available through personal homepages or institutional repositories (see Laakso et al., 2011).

Many journals implement a hybrid model wherein they publish a mix of gold open access and closed access articles. An additional level of open access is diamond or platinum where the papers are archived by the publisher, but the payment for the open access article status is made by the sale of advertising, grant funding, institutions, or governments. For the purposes of the current paper, we separate author-archived green open access from other “non-green” forms of open access that are typically publisher-archived and done at a cost (including gold, diamond, or platinum open access levels).

One particular benefit of open access publishing is increased exposure and citation counts—sometimes referred to as the “open access citation advantage”. Citation counts are often used as a proxy for the scientific impact an article has on the academic literature. Compared to closed-access articles, manuscripts published in an open access format showed high citation indices across scientific disciplines; open-access articles receive approximately 18% more citations on average than closed access articles (Piwowar et al., 2018). The trend of increased citation counts for open access articles is also present in specific sub-areas of science such as 7 (Eysenbach, 2006), general medicine (AlRyalat, Saleh, et al., 2019), oncology

(AlRyalat, Nassar, et al., 2019), cardiovascular disease (Patel et al., 2019), poverty-related disease (Breugelmans et al., 2018), and psychiatry (Hafeez, Jalal, & Khosa, 2019). Moderators of this citation increase at the article level may include impact factor of the journal, specific area of study, and time since publication, with both higher journal impact factor and longer time since publication associated with higher article citation counts independent of open access status (see

Piwowar et al., 2018).

Social media exposure may also increase article citation rates (Shuai, Pepe, & Bollen,

2012). For example, mentions of articles on are associated with increased citations of articles in biomedicine (Thelwall, Haustein, Larivière, & Sugimoto, 2013), psychiatry (Quintana

& Doan, 2016), and ecology (Peoples, Midway, Sackett, Lynch, & Cooney, 2016). Articles randomly assigned to be posted to the platform Twitter had higher citation rates than articles that were randomized not to be posted (Luc et al., 2020). Open access status of manuscripts is also associated with increased social media exposure, although the direction of this relationship (whether open access articles are more likely to be displayed on social media or social media is more likely to reference open access articles) is unclear (e.g., Wang, Liu, Mao, &

Fang, 2015). In addition, scientific impact and exposure go beyond simple citation counts.

Article-level metrics (sometimes referred to as Altmetric scores) that include internet-based mentions, discussion on blogs, exposure on traditional media outlets, bookmarks on reference managers, or mentions on social media provide a higher-level view of article exposure and potential impact than citation counts alone.

As part of the current special issue on open and transparent scientific practices in human electrophysiology, we tested the presence and degree of the open access advantage over the last 8 twenty years in citations in human electrophysiology research, including studies using electroencephalography (EEG) and event-related brain potentials (ERPs). We included articles from as many journals and sources as possible to increase the range and generalizability of the findings. We limited our scope to human electrophysiology due to recent studies showing a range of sample sizes, selective reporting, and adherence to methodological and publication guidelines that should provide a range of study quality and impact (Carbine, Lindsey, Rodeback,

& Larson, 2019; Clayson, Carbine, Baldwin, & Larson, 2019; Clayson, Carbine, & Larson,

2020; Larson & Carbine, 2017). The first aim of the current study was to characterize the prevalence of open access articles in human electrophysiology research. Next, we determined the impact of open access publishing on the attention received by articles via citations and Altmetric mentions. We hypothesized that open access articles would be associated with higher citation counts than closed access articles when accounting for time since publication and impact factor.

Lastly, we compared the impact of manuscripts published as green open access (author archived), non-green open access (typically publisher archived at cost), or closed access. We hypothesized that non-green open access would have higher impact than green open access or closed access due to likely grant funding and author means. Potential moderators included journal impact factor and time since publication as these are known to be associated with citation counts (Piwowar et al., 2018).

Method

All raw data, code for downloading articles and metadata, and code for performing statistical analyses are posted on Open Science Framework (OSF; https://osf.io/yzfdp/).

Article Selection 9

Articles were extracted using a similar approach to other work from our lab (Clayson et al., 2019). Articles on human electrophysiology published between 2000 and 2019 were extracted from the PubMed using the package (R Development Core Team, 2019) rentrez (Winter, 2017). The rentrez package is an interface to the National Center for

Biotechnology Information’s (NCBI’s) E-utilities application programming interface (API). The

NCBI E-utilities API provides access to records in PubMed and other databases. The searches were performed on August 21, 2020 and August 22, 2020.

The search terms used were (ERP[Title/] OR (event-related[Title/Abstract] AND potential[Title/Abstract]) OR (evoked[Title/Abstract] AND potentials[Title/Abstract]) OR

EEG[Title/Abstract] OR electroencephalography[Title/Abstract] OR electroencephalogram[Title/Abstract]) AND English[Language] AND Journal Article[ptyp]

AND Humans[Mesh]). Exclusion criteria included case reports, retracted articles, meta-analyses, literature reviews, conference proceedings/abstracts, editorials, and guidelines papers. These materials were excluded because of the present focus on primary studies using EEG research in human electrophysiology, rather than publications that might discuss EEG more broadly or are a synthesis of existing data. The search terms used for exclusion were (Case Reports[pt] OR

Retracted Publication[pt] OR Meta-Analysis[pt] OR Review[pt] OR Clinical Conference[pt] OR

Duplicate Publication[pt] OR Editorial[pt] OR Guideline[pt] OR Meeting Abstracts[pt] OR

Guideline[Title] OR Practice Guidelines as Topic[MAJR] OR conference proceedings[Journal]).

The information extracted for each article included the title, journal, digital object identifier

(DOI), and number of PubMed citations.

Articles that were downloaded from PubMed that did not contain valid DOIs were not included in any analyses because the DOIs were needed to obtain additional metadata for 10 subsequent analyses. The identified DOIs were used to determine whether the article was open access using the R package roadoi (Jahn, 2019), which interfaces with Unpaywall.org.

Unpaywall is a free service that locates open access scientific papers from article repositories and open access journals, including those listed in CrossRef and the Directory of Open Access

Journals (doaj.org).

The number of CrossRef citations was determined using the R package rcrossref

(Chamberlain, Zhu, Jahn, Boettiger, & Ram, 2020). CrossRef (crossref.org) was designed for the scientific community to improve scholarly communication by maintaining open access resources for tagging scientific outputs with DOIs, sharing and hosting metadata, and providing persistent links to citations that are independent of journals.

Lastly, the number of mentions in non-academic journals gathered from Altmetrics was determined using the R package rAltmetric (Ram, 2017). Altmetrics (altmetric.com) complement citations within scholarly journals by reflecting the attention and influence of articles in non- scholarly outlets. Examples include mentions in the news, on social media, in blog posts, in public policy documents, in patents and software, and on (wikipedia.org).

All of these data were extracted between August 22, 2020 and August 25, 2020. The

2019 five-year impact factor was identified for each journal using the provided by Analytics (jcr.clarivate.com).

Statistical Analysis

There are a few important characteristics of these data to consider. First, articles are nested within journals. Second, the response variable (i.e., number of citations) is discrete count data with a minimum of zero and a positive skew. To accommodate these features of the data, multilevel (for the nesting) negative binomial (for the count) models were used. We used the 11 countfit (Long & Freese, 2014) procedure in Stata (StataCorp, 2019) to compare the fit of negative binomial to zero-inflated negative models. For all outcomes, the negative binomial model outperformed a zero-inflated model, so a zero-inflated model was not used in further statistical analyses. Each multilevel model was then estimated in the R package glmmTMB

(Brooks et al., 2017) using maximum likelihood estimation and the quadratic parameterization of the negative binomial distribution (Hardin & Hilbe, 2007).

Multilevel models were used to predict PubMed citations, CrossRef citations, and

Altmetric mentions. CrossRef.org does not index content itself, but members register the metadata on CrossRef.org directly. Altmetric mentions were used to index attention articles received outside of peer-reviewed journals. These mentions represent article “citations” via outlets such as news media, internet blogs, Twitter, and . The individual predictors for each model included five-year journal impact factor, publication age, open access type, the interaction between impact factor and open access type, the interaction between five-year impact factor and publication age, and the interaction between publication age and open access type. A quadratic relationship was also fit for publication age as a main effect and in each interaction due to the potential impact of open access initiatives adopted in recent years. The five-year impact factor was z-score transformed in order to center the variable to improve model fitting.

Publication age reflected the number of years since publication starting at 0 for 2019. Hence, each one-year increase corresponds to an additional year since publication, so that an article published in 2015 would correspond to a publication age of 4 years. Closed access articles served as the reference level for the open access type predictor. Models were compared using the likelihood ratio test, the Akaike information criterion (AIC; Akaike, 1992) and the Bayesian information criterion (BIC; Schwarz, 1978). Smaller BIC and AIC values indicate better model 12 fit. An intraclass correlation coefficient (ICC) for negative binomial models was calculated for intercept-only models to characterize between- and within-journal variability (Aly, Zhao, Li, &

Jiang, 2014).

Results

35,114 journal articles were extracted from 967 journals across the years 2000 to 2019.

Table 1 presents summary information for the number of citations, Altmetric mentions, and the five-year journal impact factors. These metrics all showed a heavy positive skew. The modal number of citations was 0 and 2 for PubMed citations and CrossRef citations, respectively. The modal number of Atlmetric mentions was 0. With regard to access, 12,158 (35%) articles were open access. Figure 1 shows the frequency of open vs. closed access articles from 2000 to 2019.

There was a steady increase in the number of publications that are open access across time, and the gap between numbers of closed and open access per year appears to be shrinking. The decrease in open access articles in 2019 might reflect currently embargoed manuscripts, which are initially closed but become open access a year after publication. Figure 2 shows the total number of citations and mentions across years as a function of open access type. The Altmetric mentions showed a large spike for one open access article in 2003 due to a single article having

5,885 mentions1. The same article had 551 citations on PubMed and 1,587 citations on CrossRef.

There are some general differences in PubMed and CrossRef citations. PubMed is maintained by the NCBI, and the information deposited into PubMed is curated to ensure accuracy. PubMed primarily indexes journal articles from specific scientific disciplines, rather than other scholarly outputs (e.g., databases or ). CrossRef is a not-for-profit organization

1 The pattern of statistically significant effects was nearly identical when each analysis was run with this outlier excluded. These models are posted on OSF. 13 that is not discipline specific and includes information from formats other than journal articles.

However, CrossRef does not undergo the same level of curation as the PubMed database.

Nonetheless, the relationship between PubMed citations and CrossRef citations for the present data was very strong, r(35,112) = .92, p < .01.

However, PubMed and CrossRef citations were only weakly related to Altmetric mentions, r(35,112) = .14, p < .01; r(35,112) = .15, p < .01, respectively. Unlike PubMed and

CrossRef citations, Altmetrics reflect the attention and reach of articles outside scholarly journals. Therefore, Altmetric mentions index a mix of engagement from public audiences (e.g., news or social media) and scholarly audiences (e.g., post-publication ).

Open vs. Closed Access

We first examined the ICC of intercept-only models predicting PubMed citations,

CrossRef citations, and Altmetric mentions from articles nested within journals (see Aly et al.,

2014). Then, in Model 1 we included a) five-year impact factor, b) the linear effect of publication age, and c) the quadratic effect of publication age. In Model 2 we added the interactions between a) impact factor and the linear publication age effect and b) between impact factor and quadratic publication age effect. In Model 3, the predictors relevant to open access were added and these included a) whether an article was open access, b) the interaction between five-year impact factor and open access, c) the interaction between the linear effect of publication age and open access, and d) the interaction between the quadratic effect of publication age and open access.

The key comparison of interest was the test of model fit between Models 2 and 3, because this comparison indicates whether open access type impacted citations or mentions.

PubMed Citations. The intraclass correlation of the intercept-only model was .43, which indicates that 43% of the variance in PubMed citations is accounted for by between-journal 14 variability and 57% is accounted for by within-journal variability. Model 3, the model with all predictors, showed the best fit based on likelihood ratio tests (휒2s > 48.7, ps < .001), AICs, and

BICs (see Table 2), and Model 3 is interpreted below.

To assist interpretation, the predicted PubMed citations as a function of journal impact factor, publication age, and open access are shown in Figure 3. Predicted citations peaked around

2005, and then declined to 2019, consistent with expectations given time since publication. Open access articles were consistently associated with more citations than closed access articles across time and the three shown five-year impact factor levels (-1 SD, average, +1 SD).

Most of the predictors in Model 3 were statistically significant, and the findings are summarized in Table 2. A one standard deviation difference in five-year impact factor was associated with 43% more citations (rate ratio = exp(0.36) = 1.43). Each additional year since publication was linearly related to 42% more citations (rate ratio = 1.42), and the slope decreased by 1% for each additional year (rate ratio = 0.99; see Figure 3). Open access articles corresponded to 21% more citations (rate ratio = 1.21) than closed access articles. In sum, articles that were older, that were published in higher impact factor journals, and that were open access were associated with more citations.

With regard to interaction effects, for every one standard deviation difference in five-year impact factor, each additional year since publication was linearly related to 1.5% fewer citations

(rate ratio = 0.99). The interaction between impact factor and the quadratic effect of age was not significant. However, open access articles were associated with a 4% linear difference (rate ratio

= 1.04) in citations over closed access articles for every one additional year since publication.

However, the slope for open access articles fell by 0.2% (rate ratio = 0.998). The interaction between impact factor and open access was not significant. 15

CrossRef Citations. The intercept-only model yielded an ICC of .43, which indicates that 43% of the variance in CrossRef citation counts was accounted for by between-journal variability. Model 3, the model with all predictors, showed the best fit based on likelihood ratio tests (휒2s > 76.0, ps < .001), AICs, and BICs (see Table 3), and is interpreted below.

The model predicting CrossRef citations showed a similar pattern of effects to the model predicting PubMed citations for journal impact factor, time since publication, and open versus closed access article type (see Table 3 and Figure 4). Each one standard deviation difference in five-year impact factor corresponded to a 44% difference (rate ratio = 1.44) in the number of citations, and each additional year since publication was associated with a 38% difference (rate ratio = 1.38) in citations. However, the slope of this relationship changed by -1% (rate ratio =

0.99) for each additional year. Articles that were open access were associated with a 9% difference (rate ratio = 1.09) in citations compared to closed access articles.

For each standard deviation increase in five-year impact factor, a one-year difference in the age of an article was linearly related to -1.5% difference (rate ratio = 0.99) in CrossRef citations. The interaction between impact factor and the quadratic effect of publication age was not statistically significant. Open access articles were associated with a 3% linear difference (rate ratio = 1.03) in citations over closed access articles for each additional year since publication, but the slope for open access articles fell by 0.1% (rate ratio = 0.999). Lastly, the interaction between impact factor and open access was not significant.

Altmetric Mentions. The ICC for the intercept-only model was .48, indicating that 48% of the total variance was accounted for by between-journal variability. The model with predictors for open access (Model 3) showed the best fist based on a likelihood ratio test, (휒2s > 76.0, ps <

.001), AICs, and BICs (see Table 4), and is interpreted below. 16

The predicted Altmetric mentions as a function of journal impact factor, publication age, and open access are shown in Figure 5. A one standard deviation difference in five-year impact factor related to an 86% difference (rate ratio = 1.86) in the number of Altmetric mentions. Each additional year since publication was related to a 27% change (rate ratio = 0.72) in mentions, indicating that those articles published more recently had more mentions than older articles. This finding is likely related to the increase use of social media in the last decade. For example,

Facebook was founded in 2004 and Twitter was founded in 2006. This linear relationship between publication age and mentions is also qualified by a significant quadratic effect of age, such that each additional year since publication corresponded to a 1% difference (rate ratio =

1.01) in mentions. Articles that were open access corresponded to a 39% difference (rate ratio =

1.39) in mentions over closed access articles.

With regard to interaction effects, for every one standard deviation difference in five-year impact factor, a one year difference in the age of an article was linearly related to 5% difference

(rate ratio = 0.95) in mentions, but this slope changed by 0.2% (rate ratio = 1.002) for each additional year. Open access articles were associated with a 4% linear difference (rate ratio =

1.04) in citations over closed access articles for each additional year since publication. The remaining interactions were not significant.

Citations and Author-Archived Manuscripts

It is possible that grant-funded research was more likely to be published as open access due to article processing charges imposed by publishers. As noted above, grant-funded studies undergo rigorous review and might also be more likely to be published in higher impact factor journals and receive more recognition. We were interested in whether author-archived manuscripts, such as those posted as preprints or on university repositories, might not receive a 17 benefit in citations from being open access. Hence, we performed secondary analyses, where the

2-level Open Access factor (open vs. closed) was split into three categories: closed access, green open access, and non-green open access. Green open access articles comprise author-archived manuscripts, also referred to as self-archived manuscripts. When an article had green open access and non-green open access versions available, it was coded as non-green open access. The reference level for the non-green open access and green open access variables was closed access.

To compare non-green open access to green open access, models were refit using green open access as the reference level, and the relevant contrasts are described in the text below (Aiken &

West, 1991; West, Aiken, & Krull, 1996). For the sake of simplicity, only models with covariates and interactions were considered, and the summary below focuses on solely analyses related to open access type.

PubMed Citations. Each predictor of the model, except for the main effect of green open access, was significant (see Table 5). Non-green open access and green open access articles corresponded to a 21% and 20% difference in citations over closed access articles, respectively.

The follow-up simple effects contrast indicated a nonsignificant difference between non-green open access and green open access articles, z = 0.09, p = .93.

For non-green open access articles only, there was 8% linear change for each additional year since publication, and this sloped fell by .5% for each additional year. For green open access articles, there was a 2% change in citations for each additional year, and the quadratic effect was not statistically significant. The differences between green and non-green open access in linear and nonlinear changes were both significant (|zs| > 3.9, ps < .01). The remaining interactions with open access type were not statistically significant. 18

CrossRef Citations. Green open access articles were related to a 12% difference in citations over closed access articles, and non-green open access failed to show a statistically significant benefit in citation counts over closed access articles (see Table 5). However, the difference between green and non-green open access articles was not significant, z = -1.38, p =

.17.

For each additional year since publication, there was an 8% linear change in citations for non-green access articles compared to closed access articles, and the slope changed by -.4% for each additional year. These changes were also significantly different from green open access articles (|zs| > 6.2, ps < .01), which showed no statistically significant interactions with publication age. Although green open access and non-green open access articles (vs. closed access) showed no statistically significant interaction with five-year impact factor, the interaction between non-green vs. green open access with five-year impact factor was, z = 2.35, p = .02. For each one standard deviation difference in impact factor, non-green open access articles showed a

5% difference in citations over green open access articles.

Altmetric Mentions. Non-green open access and green open access articles were associated with 18% and 71% change in the number of mentions, respectively (see Table 6). This difference between non-green open access and green open access was significant, z = -4.36, p <

.001.

For every additional year since publication, there was a 25% linear difference and -1% change in slope for mentions of non-green open access articles. For green open access, there was a 7% linear difference and 0.4% change in slope for mentions for each additional year since publication. The difference in linear and nonlinear changes for non-green vs. green open access across publication age were significant, (|zs| > 8.3, ps < .01). 19

Discussion

The present analyses of 35,144 articles from 967 journals indicated that about 35% of articles in the last twenty years in the scientific sub-area of human electrophysiology are open access, and the publishing of open access articles appears to be increasing over time (see Figure

1). Open access articles showed a 9 to 21% benefit in citation counts and a 39% benefit in

Altmetric mentions over closed access articles—consistent with the “open access advantage” in citations. Furthermore, green open access articles (i.e., author archived) showed a similar benefit in citation counts as non-green open access articles and were related to higher Altmetric mentions than closed access articles. Findings of higher citation counts and Altmetric mentions for open access articles were unique from the impact of the time since publication of the article and the five-year impact factor of the journal where the article was published. The Altmetric findings also demonstrate that open access articles received more attention in and outside the academic literature, which is consistent with the impact of open access on the scholarly literature at large (Gargouri et al., 2010; Vincent-Lamarre et al., 2015).

Rates of open-access publishing in human electrophysiology at approximately 35% of manuscripts are numerically above the scientific literature as a whole (approximately 28%;

Piwowar et al., 2018), but below the high rates of open-access articles in similar sub-areas such as clinical neuroscience at approximately 49% (Hanson et al., 2020). Yet, the frequency of open- access publishing in human electrophysiology research is promising as the rates of open- and closed-access publishing were nearly indistinguishable in 2018 and 2019. Both researchers and the public at large benefit from open access research. Citation rates of human electrophysiology articles of between 9% and 21% above closed-access articles compare favorably with the larger scientific literature that suggests an approximately 18% “open access advantage” in citation 20 counts (Piwowar et al., 2018). Our results, thus, correspond with the growing movement for open and transparent publishing and scientific practices.

One notable result was that, in human electrophysiology research, green open access

(author archived articles) was similarly beneficial for increased citation and Altmetric counts relative to non-green open access (primarily gold open access articles). Gold open access articles, while still beneficial relative to closed-access articles, are often associated with article processing charges that can decrease or, at minimum, limit authors without grant support or means from publishing in open access formats. Posting preprints or post-prints of papers to

ArXiv-type hosting repositories, lab websites, or other areas typically falls within the scope of copyright for most journals and can be done at no-to-limited cost. Thus, one implication of the current research is that non-green open access is not necessary to achieve the benefits of open publishing practices in human electrophysiology.

Access to results that advance scientific understanding and progress is improved when published work is available without formidable cost and barriers. Yet, many published manuscripts sit behind paywalls with restricted access only available to those with means or library subscriptions. Such restrictions are often associated with a “double-dipping” wherein publishers receive the product of publicly-funded research at low-to-no cost, reviewers for little or no fee, and high rates of return for access due to subscription fees and bundling practices wherein publishers combine access to lower- and higher-demand journals to increase subscription costs. Scientific societies, universities, and governments are boycotting manuscripts from some publishers, refusing to negotiate with publishers on subscription fees, or discontinuing subscriptions altogether without open access availability of articles in general or articles published by researchers within their system or country (e.g., Ellis, 2018; Else, 2019). 21

Using green open access as a means to provide transparency and accessibility is an alternate way to increase article availability without article processing charges that may increase barriers to open access publishing and have unintended publisher benefits.

The current findings that show an open access advantage for citations and Altmetric mentions in human electrophysiology research, along with the majority of studies that have examined the role of open access publishing practices on citations or other metrics, are observational and based on historical data over the last 20 years. Few studies have experimentally tested the open access advantage. In one of the only experimental studies to date, articles in the British Medical Journal were randomized to open access versus closed access status (Davis, Lewenstein, Simon, Booth, & Connolly, 2008). In the first year after publication, open access articles were downloaded more frequently, but not necessarily cited more often, than closed-access articles. Articles in specific sub-areas (e.g., radiology; Malkawi, Al-Ryalat, Al

Hadidi, Serrieh, & AlRyalat, 2019) as well as journal-level analyses comparing open-access journals (where all articles are open access) versus closed-access journals have not consistently shown the same open access advantage or higher journal-level impact factors (e.g., Atayero,

Popoola, Egeonu, & Oludayo, 2018; Chua et al., 2017; O’Kelly, Fernandez, & Koyle, 2019)— although this discrepancy may be tied to reputations of purely on-line and open access journals that tend to have higher acceptance rates and may be viewed with more skepticism than more traditional academic journals (see Kitayama, 2020, for commentary). Thus, there is the possibility of moderating factors such as research design, sub-area of science, and purely open access journals that can be more rigorously tested as areas for future research. We note, however, that the open access advantage in human electrophysiology research was specific to open access beyond the role of journal impact factor and time since publication, which is remarkable as 22 journal impact factor and time since publication are often thought of as two of the largest contributors to higher citation counts (e.g., Ale Ebrahim et al., 2013).

This study is not without limitations. First, the metrics in the present study included a snapshot of all citations and Altmetric mentions collected at one point in time (August 22-25,

2020). Therefore, we were unable to determine the longitudinal trends of publishing an article as open access over time or whether articles that become open access after an embargo period benefit from the open access status. In a similar vein, we were unable to determine the direct impact of social media exposure on article mentions due to only having a single snapshot of overall Altmetric mentions. Second, it is likely that Altmetric mentions of open access articles are inflated due to the automatic mentions on Twitter via bots that tweet the posting of manuscripts on the ArXiv servers. These bots likely contribute to the large discrepancy in mentions between non-green open access and green open access articles, which include the manuscripts posted on ArXiv servers. However, non-green open access articles still showed a benefit in mentions over green open access articles, suggesting that there is still a benefit of open vs. closed access articles in Altmetric mentions. Third, the present analyses were only conducted on articles with valid DOIs downloaded from PubMed, because DOIs were used to collect information about open access status. Fourth, our analyses only focus on the quantity of citations. It is possible that quality of citations do not differ between open and closed access references or quality favors closed access citations. Future research could try to measure the quality of citations, such as examining whether an article is disputed or supported in a citation using the Scite platform (scite.ai), and examine open or closed access papers have more quality citations. 23

In conclusion, our findings are consistent with the spirit of the new Common Rule that advocates for the value of data shoring and open-source resources as among the best ways to advance knowledge (Federal Policy for the Use of Human Subjects, 2017). Articles on human electrophysiology published as open access were generally associated with greater impact over closed access publications, as indexed by citations and Altmetric mentions. Notably, a benefit of open access publishing was observed for self-archived manuscripts (i.e., green open access), suggesting that authors can circumvent article processing charges in pursuit of reducing barriers to published research. Although initiatives around transparency are becoming more common, the benefit of open access on article attention might motivate researchers to make their publications open access and pursue publication outlets that support it. In consideration of the direct connection between citations and journal impact factor, editors of journals might be able to improve the accessibility and impact of published articles by encouraging authors to self-archive manuscripts on preprint servers.

24

Table 1

Summary Information for PubMed Citations, CrossRef Citations, Altmetric Mentions, and Five-

Year Journal Impact Factors

M (SD) Median Mode 25% – 75% Range

PubMed Citations 10.20 (19.76) 5 0 2 – 11 0 – 799

CrossRef Citations 29.54 (50.27) 15 2 6 – 34 0 – 1,587

Altmetric Mentions 3.97 (41.93) 0 0 0 – 2 0 – 5,885

Five-Year Impact Factor 3.82 (2.24) 3.23 3.66 2.68 – 4.07 0.35 – 59.35

Note: 25% – 75% describes the second and third quartiles of the data.

25

Table 2

Estimates from Multilevel Models Predicting PubMed Citation Counts

Model 1 Model 2 Model 3

Parameter Estimate SE Z Estimate SE Z Estimate SE Z

Intercept 0.04 0.03 1.40 0.05 0.03 1.50* -.06 0.03 -1.64

5-yr Impact Factor 0.28 0.02 17.36* 0.35 0.02 15.80* 0.36 0.02 14.46*

Publication Age 0.36 < .01 89.47* 0.36 < .01 89.23* 0.35 0.01 68.36*

Publication Age2 -.01 < .01 -62.60* -.01 < .01 -62.34* -.01 < .01 -48.67*

Open Access 0.19 0.03 5.78*

IF x Age -.01 < .01 -1.78 -.01 < .01 -2.17*

IF x Age2 <.001 < .01 -0.19 < .001 < .01 0.12

IF x OA -.02 0.02 -1.37

Age x OA 0.04 0.01 4.54*

Age2 x OA -.001 < .01 -3.19*

Fit indices

Deviance 214,363.7 214,315.0 213,687.9

AIC 214,375.7 214,331.0 213,711.9

BIC 214,426.5 214,398.8 213,813.5

Note: IF = five-year journal impact factor; Age = Publication Age; OA = Open Access; AIC =

Akaike information criterion; BIC = Bayesian information criterion. *p < .05 26

Table 3

Estimates from Multilevel Models Predicting CrossRef Citation Counts

Model 1 Model 2 Model 3

Parameter Estimate SE Z Estimate SE Z Estimate SE Z

Intercept 1.18 0.03 47.16* 1.19 0.02 48.26* 1.14 0.03 41.38*

5-yr Impact Factor 0.26 0.01 19.71* 0.36 0.02 19.24* 0.36 0.02 17.26*

Publication Age 0.33 < .01 97.80* 0.33 < .01 97.75* 0.32 < .01 75.75*

Publication Age2 -.01 < .01 -59.47* -.01 < .01 -59.38* -0.01 < .01 -47.03*

Open Access 0.09 0.03 3.32*

IF x Age -.01 < .01 -3.73* -0.02 < .01 -4.09*

IF x Age2 0.003 < .01 1.43 < .001 < .01 1.68

IF x OA -0.01 0.01 -0.83

Age x OA 0.03 0.01 3.92*

Age2 x OA 0.001 < .01 -2.35*

Fit indices

Deviance 284,274.2 284,198.2 283,863.0

AIC 284,286.2 284,214.2 283,887.0

BIC 284,337.0 284,281.9 283,988.6

Note: IF = five-year journal impact factor; Age = Publication Age; OA = Open Access; AIC =

Akaike information criterion; BIC = Bayesian information criterion. *p < .05 27

Table 4

Estimates from Multilevel Models Predicting Altmetric Mentions

Model 1 Model 2 Model 3

Parameter Estimate SE Z Estimate SE Z Estimate SE Z

Intercept 1.95 0.06 35.26* 1.95 0.05 36.03* 1.75 0.06 29.77*

5-yr Impact Factor 0.38 0.03 12.22* 0.65 0.04 15.44* 0.62 0.05 13.00*

Publication Age -0.32 0.01 -39.88* -0.32 0.01 -39.46* -0.32 0.01 -31.99*

Publication Age2 0.01 < .01 20.50* 0.01 < .01 20.24* 0.01 < .01 17.64*

Open Access 0.33 0.06 5.73*

IF x Age -0.06 0.01 -7.44* -0.06 0.01 -6.92*

IF x Age2 0.002 < .01 5.43* 0.002 < .01 4.96*

IF x OA 0.003 0.03 0.10

Age x OA 0.04 0.02 2.10*

Age2 x OA -.001 < .01 -1.58

Fit indices

Deviance 121,808.5 121,716.9 121,449.7

AIC 121,820.5 121,732.9 121,473.7

BIC 121,871.3 121,800.7 122,575.3

Note: IF = five-year journal impact factor; Age = Publication Age; OA = Open Access; AIC =

Akaike information criterion; BIC = Bayesian information criterion. *p < .05 28

Table 5

Estimates from Models Predicting PubMed Citation Counts and CrossRef Citation Counts from

Open Access Type

Estimate SE RR  Z

PubMed Citations

Intercept -0.07 0.03 - - -2.04*

Impact Factor 0.34 0.03 1.41 41% 13.40*

Publication Age 0.35 0.01 1.42 42% 68.43*

Publication Age2 -0.01 < .01 0.99 -1% -48.66*

Non-Green Open 0.19 0.04 1.21 21% 4.50*

Green Open 0.19 0.04 1.20 20% 4.61*

IF x Age -0.01 0.004 0.99 -1% -2.21*

IF x Age2 < .001 < .01 1.00 0.01% 0.71

Non-Green Open x Age 0.08 0.01 1.08 8% 6.59*

Non-Green Open x Age2 -.004 < .01 1.00 -.5% -6.57*

Green Open x Age 0.02 0.01 1.02 2% 2.30*

Green Open x Age2 < .001 < .01 1.00 -.03% -0.50

Non-Green Open x IF 0.02 0.02 1.02 2% 0.78

Green Open x IF -0.03 0.02 0.98 -2% -1.48

CrossRef Citations 29

Intercept 1.12 0.03 - - 40.40*

Impact Factor 0.35 0.02 1.42 42% 16.31*

Publication Age 0.32 < .01 1.38 38% 75.80*

Publication Age2 -0.01 < .01 0.99 -1% -46.99*

Non-Green Open 0.06 0.03 1.06 6% 1.72

Green Open 0.12 0.03 1.12 12% 3.53*

IF x Age -0.02 <.01 0.98 -2% -4.34*

IF x Age2 < .001 < .01 1.00 0.04% 2.45*

Non-Green Open x Age 0.08 0.01 1.08 8% 7.86*

Non-Green Open x Age2 -.004 < .01 1.00 -.4% -7.35*

Green Open x Age 0.004 0.01 1.00 0.4% 0.47

Green Open x Age2 0.001 < .01 1.00 0.07% 1.40

Non-Green Open x IF 0.03 0.02 1.03 3% 1.48

Green Open x IF -0.02 0.01 0.98 -2% -1.37

Note:  = indicates the percent change in the number of citations/mentions; RR = rate ratio; AIC

= Akaike information criterion; BIC = Bayesian information

30

Table 6

Estimates from Models Predicting Altmetric Mentions from Open Access Type

Estimate SE RR  Z

Intercept 1.69 0.06 - - 28.43*

Impact Factor 0.69 0.05 1.99 99% 14.30*

Publication Age -0.32 0.01 0.72 -28% -32.34*

Publication Age2 0.01 < .01 1.01 1% 17.89*

Non-Green Open 0.16 0.07 1.18 18% 2.32*

Green Open 0.54 0.07 1.71 71% 7.54*

IF x Age -0.07 0.01 0.93 -7% -8.36*

IF x Age2 0.003 < .01 1.00 0.3% 6.02*

Non-Green Open x Age 0.23 0.02 1.25 25% 9.78*

Non-Green Open x Age2 -0.01 < .01 0.99 -1% -6.75*

Green Open x Age -0.08 0.02 0.93 -7% -3.69*

Green Open x Age2 0.004 < .01 1.00 0.4% 3.64*

Non-Green Open x IF -0.02 0.05 0.98 -2% -0.55

Green Open x IF -0.03 0.03 0.97 -3% -0.90

Note:  = indicates the percent change in the number of citations/mentions; RR = rate ratio; AIC

= Akaike information criterion; BIC = Bayesian information

31

Figure Captions

Figure 1. The open access status of each article across years. Closed access articles are shown with a solid line, and open access articles are shown with a dashed line.

Figure 2. Total PubMed citations, CrossRef citations, and Altmetric mentions across years as a function of open access.

Figure 3. Predicted PubMed citation counts based on the multilevel model with interaction effects. Separate lines are shown for the average, +1 standard deviation (SD), and -1 standard deviation (SD) of five-year impact factors. Shaded areas represent 95% confidence intervals.

Figure 4. Predicted CrossRef citation counts based on the multilevel model with interaction effects. Separate lines are shown for the average, +1 standard deviation (SD), and -1 standard deviation (SD) of five-year impact factors. Shaded areas represent 95% confidence intervals.

Figure 5. Predicted Altmetric mentions based on the multilevel model with interaction effects.

Separate lines are shown for the average, +1 standard deviation (SD), and -1 standard deviation

(SD) of five-year impact factors. Shaded areas represent 95% confidence intervals.

32

Figure 1

33

Figure 2

34

Figure 3

35

Figure 4

36

Figure 5

37

References

Aiken, L., & West, S. (1991). Multiple regression: Testing and interpreting interactions: Sage Publications, Inc. Akaike, H. (1992). Information theory and an extension of the maximum likelihood principle. In S. Kotz & N. L. Johnson (Eds.), Breakthroughs in statistics: Foundations and basic theory (pp. 610-624). New York, NY: Springer New York. Ale Ebrahim, N., Salehi, H., Embi, M. A., Habibi, F., Gholizadeh, H., Motahar, S. M., & Ordi, A. (2013). Effective strategies for increasing citation frequency. International Education Studies, 6(11), 93-99. AlRyalat, S. A., Nassar, A. A., Tamimi, F., Al-Fraihat, E., Assaf, L., Ghareeb, R., . . . Al-Essa, M. (2019). The impact of the open-access status on journal indices: oncology journals. Journal of Gastrointestinal Oncology, 10(4), 777-782. doi:10.21037/jgo.2019.02.13 AlRyalat, S. A., Saleh, M., Alaqraa, M., Alfukaha, A., Alkayed, Y., Abaza, M., . . . Alshamiry, M. (2019). The impact of the open-access status on journal indices: a review of medical journals. F1000Research, 8, 266. doi:10.12688/f1000research.17979.1 Aly, S. S., Zhao, J., Li, B., & Jiang, J. (2014). Reliability of environmental sampling culture results using the negative binomial intraclass correlation coefficient. Springerplus, 3(40). doi:10.1186/2193-1801-3-40 Atayero, A. A., Popoola, S. I., Egeonu, J., & Oludayo, O. (2018). Citation analytics: Data exploration and comparative analyses of CiteScores of Open Access and Subscription- Based publications indexed in (2014–2016). Data in Brief, 19, 198-213. doi:10.1016/j.dib.2018.05.005 Breugelmans, J. G., Roberge, G., Tippett, C., Durning, M., Struck, D. B., & Makanga, M. M. (2018). Scientific impact increases when researchers publish in open access and international collaboration: A bibliometric analysis on poverty-related disease papers. PloS one, 13(9), e0203156. doi:10.1371/journal.pone.0203156 Brooks, M. E., Kristensen, K., van Benthem, K. J., Magnusson, A., Berg, C. W., Nielsen, A., . . . Bolker, B. M. (2017). glmmTMB balances speed and flexibility among packages for zero-inflated generalized linear mixed modeling. R Journal, 9(2), 378-400. doi:10.32614/rj-2017-066 Carbine, K. A., Lindsey, H. M., Rodeback, R. E., & Larson, M. J. (2019). Quantifying evidential value and selective reporting in recent and 10-year past psychophysiological literature: A pre-registered P-curve analysis. International Journal of Psychophysiology, 142, 33-49. doi:10.1016/j.ijpsycho.2019.06.004 Chamberlain, S. R., Zhu, H., Jahn, N., Boettiger, C., & Ram, K. (2020). rcrossref: Client for various ‘CrossReg’ APIs (Version 1.0.0). Retrieved from https://CRAN.R- project.org/package=rcrossref Chua, S. K., Qureshi, A. M., Krishnan, V., Pai, D. R., Kamal, L. B., Gunasegaran, S., . . . Sood, S. (2017). The impact factor of an open access journal does not contribute to an article’s citations. F1000Research, 6, 208. doi:10.12688/f1000research.10892.1 Clayson, P. E., Carbine, K. A., Baldwin, S. A., & Larson, M. J. (2019). Methodological reporting behavior, sample sizes, and statistical power in studies of event‐related potentials: Barriers to reproducibility and replicability. Psychophysiology, 111(6), 5-17. doi:https://doi.org/10.1111/psyp.13437 38

Clayson, P. E., Carbine, K. A., & Larson, M. J. (2020). A registered report of error-related negativity and reward positivity as biomarkers of depression: P-curving the evidence. International Journal of Psychophysiology, 150, 50-72. doi:https://doi.org/10.1016/j.ijpsycho.2020.01.005 Davis, P. M., Lewenstein, B. V., Simon, D. H., Booth, J. G., & Connolly, M. J. L. (2008). Open access publishing, article downloads, and citations: randomised controlled trial. BMJ (Clinical research ed), 337(jul31 1), a568-a568. doi:10.1136/bmj.a568 Dove, C., Chan, T. M., Thoma, B., Roland, D., & Bruijns, S. R. (2019). A cross-sectional description of open access publication costs, policies and impact in emergency medicine and critical care journals. African Journal of Emergency Medicine, 9(3), 150-155. doi:10.1016/j.afjem.2019.01.015 Ellis, L. (2018). In talks with , UCLA reaches for a novel bargaining chip: Its faculty. The Chronicle of Higher Education. Retrieved from https://www.chronicle.com/article/in-talks-with-elsevier-ucla-reaches-for-a-novel- bargaining-chip-its-faculty/ Else, H. (2019). Thousands of scientists run up against Elsevier’s paywall. Nature. Retrieved from https://www.nature.com/articles/d41586-019-00492-4 Eysenbach, G. (2006). Citation Advantage of Open Access Articles. PLoS biology, 4(5), e157. doi:10.1371/journal.pbio.0040157 Gargouri, Y., Hajjem, C., Larivière, V., Gingras, Y., Carr, L., Brody, T., & Harnad, S. (2010). Self-Selected or Mandated, Open Access Increases for Higher Quality Research. PloS one, 5(10), e13636. doi:10.1371/journal.pone.0013636 Gilmore, R. O., Kennedy, J. L., & Adolph, K. E. (2018). Practical Solutions for Sharing Data and Materials From Psychological Research. Advances in Methods and Practices in Psychological Science, 1(1), 121-130. doi:10.1177/2515245917746500 Grove, J. (2020). Open-access publishing and the coronavirus. Inside Higher Ed. Retrieved from https://www.insidehighered.com/news/2020/05/15/coronavirus-may-be-encouraging- publishers-pursue-open-access Hafeez, D. M., Jalal, S., & Khosa, F. (2019). Bibliometric analysis of manuscript characteristics that influence citations: A comparison of six major psychiatry journals. Journal of psychiatric research, 108, 90-94. doi:10.1016/j.jpsychires.2018.07.010 Hanson, K. A., Almeida, N., Traylor, J. I., Rajagopalan, D., & Johnson, J. (2020). Profile of Data Sharing in the Clinical Neurosciences. Cureus. doi:10.7759/cureus.9927 Hardin, J. W., & Hilbe, J. M. (2007). Generalized linear models and extensions: Stata press. Hardwicke, T. E., Mathur, M. B., MacDonald, K., Nilsonne, G., Banks, G. C., Kidwell, M. C., . . . Frank, M. C. (2018). Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory policy at the journal Cognition. Royal Society Open Science, 5(8), 180448. doi:10.1098/rsos.180448 Jahn, N. (2019). Introduction to roadoi. Retrieved from https://cran.r- project.org/web/packages/roadoi/vignettes/intro.html Kitayama, S. (2020). The open-access model of journal publishing. The Associatin for Psychological Science. Retrieved from https://www.psychologicalscience.org/observer/open-access-journal-publishing Laakso, M., Welling, P., Bukvova, H., Nyman, L., Björk, B.-C., & Hedlund, T. (2011). The Development of Open Access Journal Publishing from 1993 to 2009. PloS one, 6(6), e20961. doi:10.1371/journal.pone.0020961 39

Larson, M. J., & Carbine, K. A. (2017). Sample size calculations in human electrophysiology (EEG and ERP) studies: A and recommendations for increased rigor. International Journal of Psychophysiology, 111, 33-41. doi:10.1016/j.ijpsycho.2016.06.015 Long, J. S., & Freese, J. (2014). Regression models for categorial dependent variables using Stata [3rd Edition]. Luc, J. G. Y., Archer, M. A., Arora, R. C., Bender, E. M., Blitz, A., Cooke, D. T., . . . Antonoff, M. B. (2020). Does Tweeting Improve Citations? One-Year Results from the TSSMN Prospective Randomized Trial. The Annals of Thoracic Surgery. doi:10.1016/j.athoracsur.2020.04.065 Malkawi, L., Al-Ryalat, N., Al Hadidi, E., Serrieh, D., & AlRyalat, S. A. (2019). The Impact of Open Access Status on Journal Indexes of Radiology Journals. American Journal of Roentgenology, 213(4), 736-739. doi:10.2214/AJR.19.21215 Nosek, B. A., & Bar-Anan, Y. (2012). Scientific Utopia: I. Opening Scientific Communication. Psychological , 23(3), 217-243. doi:10.1080/1047840X.2012.692215 Nosek, B. A., & Lakens, D. (2014). Registered reports: A method to increase the credibility of published results. Social Psychology, 45(3), 137-141. doi:10.1027/1864-9335/a000192 O’Kelly, F., Fernandez, N., & Koyle, M. A. (2019). Predatory publishing or a lack of peer review transparency?—a contemporary analysis of indexed open and non-open access articles in paediatric urology. Journal of Pediatric Urology, 15(2), 159.e151-159.e157. doi:10.1016/j.jpurol.2018.08.019 Patel, R. B., Vaduganathan, M., Mosarla, R. C., Venkateswaran, R. V., Bhatt, D. L., & Bonow, R. O. (2019). Open Access Publishing and Subsequent Citations Among Articles in Major Cardiovascular Journals. The American Journal of Medicine, 132(9), 1103-1105. doi:10.1016/j.amjmed.2019.02.009 Peoples, B. K., Midway, S. R., Sackett, D., Lynch, A., & Cooney, P. B. (2016). Twitter predicts citation rates of ecological research. PloS one, 11(11), e0166570-0166511. doi:10.1371/journal.pone.0166570 Piwowar, H., Priem, J., Larivière, V., Alperin, J. P., Matthias, L., Norlander, B., . . . Haustein, S. (2018). The state of OA: A large-scale analysis of the prevalence and impact of Open Access articles. PeerJ, 6(4), e4375. doi:10.7717/.4375 Quintana, D. S., & Doan, N. T. (2016). Twitter article mentions and citations: An exploratory analysis of publications in the American journal of psychiatry. The American Journal of Psychiatry, 173(2), 194. doi:10.1176/appi.ajp.2015.15101341 R Development Core Team. (2019). R: A language and environment for statistical computing. Vienna, Austria. Retrieved from http://www.R-project.org/ Ram, K. (2017). rAltmetric: Retrieves Altmetrics data for any publsihed paper from Almetric.com (Version 0.7.0). Retrieved from https://CRAN.R- project.org/package=rAltmetric Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6(2), 461- 464. Shuai, X., Pepe, A., & Bollen, J. (2012). How the scientific community reacts to newly submitted preprints: Article downloads, twitter mentions, and citations. PloS one, 7(11), e47523-47528. doi:10.1371/journal.pone.0047523 StataCorp. (2019). Stata Statistical Software: Release 16. College Station, TX: StataCorp, LLC. 40

Thelwall, M., Haustein, S., Larivière, V., & Sugimoto, C. R. (2013). Do Altmetrics work? Twitter and ten other social web services. PloS one, 8(5), e64841-64847. doi:10.1371/journal.pone.0064841 Vincent-Lamarre, P., Boivin, J., Gargouri, Y., Larivière, V., & Harnad, S. (2015). Estimating open access mandate effectiveness: The MELIBEA score. Journal of the Association for Information Science and Technology, 67(11), 2815-2828. doi:10.1002/asi.23601 Wang, X., Liu, C., Mao, W., & Fang, Z. (2015). The open access advantage considering citation, article usage and social media attention. , 103(2), 555-564. doi:10.1007/s11192-015-1547-0 West, S. G., Aiken, L. S., & Krull, J. L. (1996). Experimental personality designs: Analyzing categorical by continuous variable interactions. Journal of Personality, 64(1), 1-48. doi:doi:10.1111/j.1467-6494.1996.tb00813.x Winter, D. J. (2017). rentrez: An R package for the NCBI eUtils API. PeerJ Preprints. doi:10.7287/peerj.preprints.3179v2