1 OPEN SCIENCE An Open Science Workflow for More Credible, Rigorous Research Chapter for The Portable Mentor (ed. Mitchell J. Prinstein) Katherine S. Corker Author Note Katherine S. Corker, Grand Valley State University, Department of Psychology, [email protected] https://orcid.org/0000-0002-7971-1678 Thank you to Julia Bottesini and Sarah Schiavone for their thoughtful feedback. All errors and omissions are my own. Preprint version as of March 19, 2021 2 OPEN SCIENCE Abstract Part of what distinguishes science from other ways of knowing is that scientists show their work. Yet when probed, it turns out that much of the process of research is hidden away: in personal files, in undocumented conversations, in point-and-click menus, and so on. In recent years, a movement towards more open science has arisen in psychology. Open science practices capture a broad swath of activities designed to take parts of the research process that were previously known only to a research team and make them more broadly accessible (e.g., open data, open analysis code, pre-registration, open research materials). Such practices increase the value of research by increasing transparency, which may in turn facilitate higher research quality. Plus, open science practices are now required at many journals. This chapter will introduce open science practices and provide plentiful resources for researchers seeking to integrate these practices into their workflow. Keywords: Open science, reproducibility, pre-registration, metascience, research rigor, transparency Author Biography Katie Corker is an associate professor of psychology at Grand Valley State University. She earned her PhD in personality and social psychology at Michigan State University in 2012, just as the replication crisis described in the chapter was unfolding. She is a past president and current executive officer for the Society for the Improvement of Psychological Science (SIPS), which she helped to found in order to transform idle discussions about whether psychology needed improving into tangible actions to improve the field. She came to psychology as an undergraduate student who was delighted to learn that you could use science to study humans. It is therefore fitting that as a professor Katie’s work revolves around the study of scientists and their practices. Outside of working hours, Katie loves to travel, and she is happiest on the trail, at the beach, or with her friends and family at the local pub quiz. 3 OPEN SCIENCE An Open Science Workflow for More Credible, Rigorous Research Recent years have heralded a relatively tumultuous time in the history of psychological science. The past decade saw the publication of a landmark paper that attempted to replicate 100 studies and estimated that just 39% of studies published in top psychology journals were replicable (Open Science Collaboration, 2015). There was also a surplus of studies failing to replicate high profile effects that had long been taken as fact (e.g., Hagger et al., 2014; Harris et al., 2013; Wagenmakers et al., 2016). Taken together, suddenly, the foundations of much psychological research seemed very shaky. As with similar evidence in other scientific fields (e.g., biomedicine, criminology), these findings have led to a collective soul searching dubbed the “replication crisis” or the “credibility revolution” (Nelson et al., 2018; Vazire, 2018). Clearly, something about the way scientists had gone about their work in the past wasn’t effective at uncovering replicable findings, and changes were badly needed. An impressive collection of meta-scientific studies (i.e., studies about scientists and scientific practices) have revealed major shortcomings in standard research and statistical methods (e.g., Button et al., 2013; John et al., 2012; Nuijten et al., 2016; Simmons et al., 2011). These studies point to a clear way to improve not only replicability but also the accuracy of scientific conclusions: open science. Open science refers to a radically transparent approach to the research process. “Open” refers to sharing – making accessible – parts of the research process that have traditionally been known only to an individual researcher or research team. In a standard research article, authors summarize their research methods and their findings, leaving out many details along the way. Among other things, open science includes sharing research materials (protocols) in full, making data and analysis code publicly available, and pre-registering (i.e., making plans public) study designs, hypotheses, and analysis plans. Psychology has previously gone through periods of unrest similar to the 2010s, with methodologists and statisticians making persuasive pleas for more transparency and rigor in research (e.g., Bakan, 1966; Meehl, 1978; Cohen, 1994; Kerr, 1998). Yet, it is only now with improvements in technology and research infrastructure, together with concerted efforts in journals and scientific societies by reformers, that changes have begun to stick (Spellman, 2015). Training in open science practices is now a required part of becoming a research psychologist. The goal of this chapter is to briefly review the shortcomings in scientific practice that open science practices address and then to give a more detailed account of open science itself. We’ll consider what it means to work openly and offer pragmatic advice for getting started. Why Open Science? When introducing new researchers to the idea of open science, the need for such practices seems obvious and self-evident. Doesn’t being a scientist logically imply an obligation to transparently show one’s work and subject it to rigorous scrutiny? Yet, abundant evidence reveals that researchers have not historically lived up to this ideal and that the failure to do transparent, rigorous work has hindered scientific progress. Old Habits Die Hard 4 OPEN SCIENCE Several factors in the past combined to create conditions that encouraged researchers to avoid open science practices. First, incentives in academic contexts have not historically rewarded such behaviors and, in some cases, may have actually punished them (Smaldino & McElreath, 2016). To get ahead in an academic career, publications are the coin of the realm, and jobs, promotions, and accolades can sometimes be awarded based on number of publications, rather than publication quality. Second, human biases conspire to fool us into thinking we have discovered something when we actually have not (Bishop, 2020). For instance, confirmation bias allows us to selectively interpret results in ways that support our pre-existing beliefs or theories, which may be flawed. Self-serving biases might cause defensive reactions when critics point out errors in our methods or conclusions. Adopting open science practices can expose researchers to cognitive discomfort (e.g., pre-existing beliefs are challenged; higher levels of transparency mean that critics are given ammunition), which we might naturally seek to avoid. Finally, psychology uses an apprenticeship model of researcher training, which means that the practices of new researchers might only be as good as the practices of the more senior academics training them. When questionable research practices are taught as normative by research mentors, higher quality open science practices might be dismissed as methodological pedantry. Given the abundant evidence of flaws in psychology’s collective body of knowledge, we now know how important it is to overcome the hurdles described here and transition to a higher standard of practice. Incentives are changing, and open science practices are becoming the norm at many journals (Nosek et al., 2015). A new generation of researchers is being trained to employ more rigorous practices. And although the cognitive biases just discussed might be some of the toughest problems to overcome, greater levels of transparency in the publishing process help fortify the ability of the peer review process to serve as a check on researcher biases. Benefits of Open Science Practices A number of benefits of open science practices are worth emphasizing. First, increases in transparency make it possible for errors to be detected and for science to self-correct. The self- correcting nature of science is often heralded as a key feature that distinguishes scientific approaches from other ways of knowing. Yet, self-correction is difficult, if not impossible, when details of research are routinely withheld (Vazire & Holcombe, 2020). Second, openly sharing research materials (protocols), analysis code, and data provides new opportunities to extend upon research and adds value above and beyond what a single study would add. For example, future researchers can more easily replicate a study’s methods if they have access to a full protocol and materials; secondary data analysts and meta-analysts can perform novel analyses on raw data if they are shared. Third, collaborative work becomes easier when teams employ the careful documentation that is well honed for followers of open science practices. Even massive collaborations across time and location become possible when research materials and data are shared following similar standards (Moshontz et al., 2018). Finally, the benefits of open science practices accrue not only to the field at large, but also to individual researchers. Working openly provides a tangible record of your contributions as a researcher, which may be useful when it comes to applying for funding, awards, or jobs. Markowitz (2015) describes five “selfish” reasons to work reproducibly, chiefly: (a) to avoid 5 OPEN SCIENCE “disaster” (i.e., major errors), (b) because it’s easier, (c) to smooth the peer review process, (d) to allow others to build on your work, and (e) to build your reputation. Likewise, McKiernan et al. (2016) review the ample evidence that articles that feature open science practices tend to be more cited, more discussed in the media, attract more funding and job offers, and are associated with having a larger network of collaborators. Allen and Mehler (2019) review benefits (along with challenges) specifically for early career researchers. All of this is not to say that there are not costs or downsides to some of the practices discussed here.