NSF 19-537 Mid-Scale Research Infrastructure - Project Summary Center for Open Science (COS)
Total Page：16
File Type：pdf, Size：1020Kb
NSF 19-537 Mid-Scale Research Infrastructure - Project Summary Center for Open Science (COS) Overview The Center for Open Science (COS) proposes to undertake a Mid-scale RI-1 Implementation Project entitled, Expanding open infrastructure for robust, open, reproducible research, primarily in association with the SBE Directorate. Reproducibility is the ability to obtain independent evidence supporting scientific findings. Recent investigations suggest that reproducibility of published findings is lower than expected or desired, thus interfering with research progress. COS’s Open Science Framework (OSF) enables rigorous, reproducible science by providing collaboration, registration, and data management support across the entire research lifecycle. With support from NSF 19-537, OSF will become essential mainstream infrastructure supporting researchers’, institutions’, and federal agencies’ priorities to maximize return on investment in research. Adoption of open science behaviors via OSF will enable rigor and reproducibility in harnessing the data revolution, increase accessibility and inclusivity of research for all, stimulate discovery, and accelerate progress in science. Seeded with private philanthropy in 2012, the open-source OSF’s user base and key performance indicators have shown rapid, non-linear growth since inception. OSF is now a popular, robust, and important infrastructure for registering studies and analysis plans; managing and archiving research data, code, and materials; and accelerating open or controlled sharing of research outcomes and all supporting research contents. This mid-scale NSF research infrastructure grant will build on OSF’s success by meeting four major objectives: 1. Expand OSF functionality and workflows to meet the needs of a broad number of research disciplines and communities. 2. Integrate OSF with other established scholarly infrastructure to make search, discovery, transfer, and preservation of scholarly content and metadata more efficient. 3. Manage OSF design for efficient and sustainable long-term service. 4. Provide training to improve effectiveness of registration and results reporting and ensure inclusivity of beneficiaries to these innovations Intellectual Merits These innovations will address the major challenges to reproducibility and will accelerate research progress, maximize the diagnosticity of statistical inferences for confirmatory claims, improve transparency of the research process, mitigate publication bias by making all research studies discoverable regardless of publication status, facilitate the science of science with a searchable database of research plans and outcomes, and increase the accessibility of the research process and infrastructure to individuals, groups, and communities that are historically underserved or underrepresented. This will foster discoverability, reusability, and scientific advances on a shorter timescale. It will do so sustainably with a shared, scalable, customizable, open, community-based infrastructure. Broader Impacts Advancing this infrastructure will have a broad impact on scholarly research, particularly as a complement to changing policies, incentives, and norms toward transparency, rigor, and reproducibility. As journals, funders, and institutions adopt new policies, OSF provides an easy, integrated, and comprehensive solution that enables researchers to focus on doing their best science rather than treating open behaviors as bureaucratic burdens. As a nonprofit committed to open-source solutions, COS can ensure that no organization or researcher is locked-in to our products or locked-out by profit-driven pricing. In five years, we will transition this public goods infrastructure into essential infrastructure for the social-behavioral science research community. We will achieve significant adoption in education and biological sciences, and advance adoption in engineering, geosciences, and other physical sciences. Project Description Center for Open Science NSF - 19-537 Intellectual Merit The Center for Open Science (COS) proposes to undertake a Mid-scale RI-1 Implementation Project entitled, Expanding open infrastructure for robust, open, reproducible research. This proposed project will not have significant environmental or cultural impacts. Scientific Justification Reproducibility, the ability to obtain independent evidence of a scientific claim, is a central tenet of science because it places the burden for “truth” on the quality and repeatability of the evidence, rather than the authority or prestige of its originator (Merton, 1942, 1973; Open Science Collaboration, 2015). Recent investigations across research disciplines and particularly in the behavioral and social sciences (Begley & Ellis, 2012; Camerer et al., 2016, 2018; Cova et al., 2019; Ebersole et al., 2016; Klein et al., 2014, 2018; Open Science Collaboration, 2015; Prinz et al., 2011) suggest that reproducibility of published findings is lower than expected or desired. Many factors contribute to irreproducibility, creating friction that slows the pace of discovery and innovation. These include: ● Norms and incentives prioritize novelty and innovation over credibility and reproducibility: Incentives are strong for producing novel and innovative research results and comparatively weak for assessing and demonstrating reproducibility (Nosek, Spies, & Motyl, 2012). There is broad endorsement of Merton’s norms of transparency, universalism, disinterestedness, and organized skepticism (Merton, 1942, 1973) but similarly broad perception that the research culture reinforces secrecy, self-interestedness, and dogmatism (Anderson et al., 2007). ● Publication bias: A challenge for evaluating the credibility of published literature is that unpublished literature is largely unknown and undiscoverable. Hence, it is difficult to estimate the extent to which the published evidence is a biased representation of reality (Rosenthal, 1979). Reviews suggest that the published literature is pervasively underpowered (Button et al., 2013; Cohen, 1962) and also filled with almost exclusively positive results (Fanelli, 2012; Sterling, 1959). These two observations can’t both be true unless ignoring negative results is common (Greenwald, 1975). ● Questionable research practices: Researchers may wittingly or unwittingly employ questionable research practices, such as p-hacking, that increase the likelihood of falsely obtaining positive results (John et al., 2012; Simmons et al., 2011). A key challenge is presenting the results obtained from exploratory analyses as if they were the outcomes of confirmatory tests in which statistical inferences are diagnostic (Nosek et al., 2018; Wagenmakers et al., 2012). ● Limited access to data, code, and materials for evaluation and reuse: Lack of sharing original data, code, and materials is an impediment to research progress for assessing reproducibility of the original findings, for replicating or extending the original research in new directions, and for novel research applications with existing data (McNutt et al., 2016; Miguel et al., 2014; Nosek et al., 2015). While these challenges are significant and pervasive, they are solvable. Concrete behavior changes will dramatically accelerate reproducibility and, ultimately, the pace of discovery. These include: ● Registration of studies: Registration involves reporting plans for conducting a study in a public registry. This makes studies discoverable whether or not they are ultimately published. At scale, registration of studies and reporting outcomes solves the problem of publication bias, particularly combating the greater likelihood that positive results are published than negative results (Sterling, 1959; Greenwald, 1975; Rosenthal, 1979). ● Preregistration of analysis plans: Preregistration makes clear the distinction between confirmatory analyses that test hypotheses and exploratory analyses that generate hypotheses (Nosek et al., 2018). Preregistration clarifies which analysis decisions were made a priori and which were made 1 post hoc, reducing the likelihood of overconfident interpretations of exploratory results, and confirming the diagnosticity of the hypothesis tests (Nosek et al., 2018; Wagenmakers et al., 2012). ● Accessibility of data, materials, and code: Availability of original research materials enables others to conduct replications of the original findings and to reuse or extend the methodology for novel purposes, and makes it possible to test whether the reported findings are robust to variation in the analysis pipeline (Silberzahn et al., 2018). Also, it enables reuse of data for novel analyses and aggregation with similar datasets. ● Training: These actions are not bureaucratic practices, they are research skills involving planning, methodology, curation, and research ethics. To be effective, researchers need training and resources to facilitate skill acquisition and improvement over time. Registration of studies, preregistration of analysis plans, and accessibility of data, materials, and code address distinct and interdependent aspects of fostering rigor and reproducibility. Broad adoption of these open science behaviors will maximize return on research investment. Preliminary Activities Accomplished: COS’s Role in Open Science COS seeks to change the research culture to accelerate the pace of discovery. As a nonprofit, COS pursues its mission to increase openness, integrity, and reproducibility of research. This mission will be accomplished when open and FAIR (findable, accessible,