NSF 19-537 Mid-Scale Research Infrastructure - Project Summary Center for Open Science (COS)

Overview The Center for Open Science (COS) proposes to undertake a Mid-scale RI-1 Implementation Project entitled, Expanding open infrastructure for robust, open, reproducible research, primarily in association with the SBE Directorate. Reproducibility is the ability to obtain independent evidence supporting scientific findings. Recent investigations suggest that reproducibility of published findings is lower than expected or desired, thus interfering with research progress. COS’s Open Science Framework (OSF) enables rigorous, reproducible science by providing collaboration, registration, and data management support across the entire research lifecycle. With support from NSF 19-537, OSF will become essential mainstream infrastructure supporting researchers’, institutions’, and federal agencies’ priorities to maximize return on investment in research. Adoption of open science behaviors via OSF will enable rigor and reproducibility in harnessing the data revolution, increase accessibility and inclusivity of research for all, stimulate discovery, and accelerate progress in science. Seeded with private philanthropy in 2012, the open-source OSF’s user base and key performance indicators have shown rapid, non-linear growth since inception. OSF is now a popular, robust, and important infrastructure for registering studies and analysis plans; managing and archiving research data, code, and materials; and accelerating open or controlled sharing of research outcomes and all supporting research contents. This mid-scale NSF research infrastructure grant will build on OSF’s success by meeting four major objectives:

1. Expand OSF functionality and workflows to meet the needs of a broad number of research disciplines and communities. 2. Integrate OSF with other established scholarly infrastructure to make search, discovery, transfer, and preservation of scholarly content and metadata more efficient. 3. Manage OSF design for efficient and sustainable long-term service. 4. Provide training to improve effectiveness of registration and results reporting and ensure inclusivity of beneficiaries to these innovations

Intellectual Merits These innovations will address the major challenges to reproducibility and will accelerate research progress, maximize the diagnosticity of statistical inferences for confirmatory claims, improve transparency of the research process, mitigate publication bias by making all research studies discoverable regardless of publication status, facilitate the science of science with a searchable database of research plans and outcomes, and increase the accessibility of the research process and infrastructure to individuals, groups, and communities that are historically underserved or underrepresented. This will foster discoverability, reusability, and scientific advances on a shorter timescale. It will do so sustainably with a shared, scalable, customizable, open, community-based infrastructure.

Broader Impacts Advancing this infrastructure will have a broad impact on scholarly research, particularly as a complement to changing policies, incentives, and norms toward transparency, rigor, and reproducibility. As journals, funders, and institutions adopt new policies, OSF provides an easy, integrated, and comprehensive solution that enables researchers to focus on doing their best science rather than treating open behaviors as bureaucratic burdens. As a nonprofit committed to open-source solutions, COS can ensure that no organization or researcher is locked-in to our products or locked-out by profit-driven pricing. In five years, we will transition this public goods infrastructure into essential infrastructure for the social-behavioral science research community. We will achieve significant adoption in education and biological sciences, and advance adoption in engineering, geosciences, and other physical sciences. Project Description Center for Open Science NSF - 19-537 Intellectual Merit The Center for Open Science (COS) proposes to undertake a Mid-scale RI-1 Implementation Project entitled, Expanding open infrastructure for robust, open, reproducible research. This proposed project will not have significant environmental or cultural impacts. Scientific Justification Reproducibility, the ability to obtain independent evidence of a scientific claim, is a central tenet of science because it places the burden for “truth” on the quality and repeatability of the evidence, rather than the authority or prestige of its originator (Merton, 1942, 1973; Open Science Collaboration, 2015). Recent investigations across research disciplines and particularly in the behavioral and social sciences (Begley & Ellis, 2012; Camerer et al., 2016, 2018; Cova et al., 2019; Ebersole et al., 2016; Klein et al., 2014, 2018; Open Science Collaboration, 2015; Prinz et al., 2011) suggest that reproducibility of published findings is lower than expected or desired. Many factors contribute to irreproducibility, creating friction that slows the pace of discovery and innovation. These include: ● Norms and incentives prioritize novelty and innovation over credibility and reproducibility: Incentives are strong for producing novel and innovative research results and comparatively weak for assessing and demonstrating reproducibility (Nosek, Spies, & Motyl, 2012). There is broad endorsement of Merton’s norms of transparency, universalism, disinterestedness, and organized skepticism (Merton, 1942, 1973) but similarly broad perception that the research culture reinforces secrecy, self-interestedness, and dogmatism (Anderson et al., 2007). ● Publication bias: A challenge for evaluating the credibility of published literature is that unpublished literature is largely unknown and undiscoverable. Hence, it is difficult to estimate the extent to which the published evidence is a biased representation of reality (Rosenthal, 1979). Reviews suggest that the published literature is pervasively underpowered (Button et al., 2013; Cohen, 1962) and also filled with almost exclusively positive results (Fanelli, 2012; Sterling, 1959). These two observations can’t both be true unless ignoring negative results is common (Greenwald, 1975). ● Questionable research practices: Researchers may wittingly or unwittingly employ questionable research practices, such as p-hacking, that increase the likelihood of falsely obtaining positive results (John et al., 2012; Simmons et al., 2011). A key challenge is presenting the results obtained from exploratory analyses as if they were the outcomes of confirmatory tests in which statistical inferences are diagnostic (Nosek et al., 2018; Wagenmakers et al., 2012). ● Limited access to data, code, and materials for evaluation and reuse: Lack of sharing original data, code, and materials is an impediment to research progress for assessing reproducibility of the original findings, for replicating or extending the original research in new directions, and for novel research applications with existing data (McNutt et al., 2016; Miguel et al., 2014; Nosek et al., 2015). While these challenges are significant and pervasive, they are solvable. Concrete behavior changes will dramatically accelerate reproducibility and, ultimately, the pace of discovery. These include: ● Registration of studies: Registration involves reporting plans for conducting a study in a public registry. This makes studies discoverable whether or not they are ultimately published. At scale, registration of studies and reporting outcomes solves the problem of publication bias, particularly combating the greater likelihood that positive results are published than negative results (Sterling, 1959; Greenwald, 1975; Rosenthal, 1979). ● Preregistration of analysis plans: Preregistration makes clear the distinction between confirmatory analyses that test hypotheses and exploratory analyses that generate hypotheses (Nosek et al., 2018). Preregistration clarifies which analysis decisions were made a priori and which were made

1 post hoc, reducing the likelihood of overconfident interpretations of exploratory results, and confirming the diagnosticity of the hypothesis tests (Nosek et al., 2018; Wagenmakers et al., 2012). ● Accessibility of data, materials, and code: Availability of original research materials enables others to conduct replications of the original findings and to reuse or extend the methodology for novel purposes, and makes it possible to test whether the reported findings are robust to variation in the analysis pipeline (Silberzahn et al., 2018). Also, it enables reuse of data for novel analyses and aggregation with similar datasets. ● Training: These actions are not bureaucratic practices, they are research skills involving planning, methodology, curation, and research ethics. To be effective, researchers need training and resources to facilitate skill acquisition and improvement over time. Registration of studies, preregistration of analysis plans, and accessibility of data, materials, and code address distinct and interdependent aspects of fostering rigor and reproducibility. Broad adoption of these open science behaviors will maximize return on research investment. Preliminary Activities Accomplished: COS’s Role in Open Science COS seeks to change the research culture to accelerate the pace of discovery. As a nonprofit, COS pursues its mission to increase openness, integrity, and reproducibility of research. This mission will be accomplished when open and FAIR (findable, accessible, interoperable, reusable) is the default for research output (papers, reports, presentations), content (data, code, materials), and process (registration of studies, preregistration of analysis plans, peer review). COS provides infrastructure for openness and reproducibility, and catalyzes communities to alter norms, incentives, and policies to embrace open science. COS’s strategy for culture and behavior change requires five levels of intervention represented by the pyramid (right). These levels are progressive, reflecting the fact that successful implementation of higher levels depends on successful implementation of lower levels. Infrastructure is the base of the pyramid making behavior change possible. Normative, incentive, and policy interventions require effective infrastructure for enacting open science behavior. OSF Infrastructure COS developed and maintains the Open Science Framework (OSF), a suite of cloud-based applications, which enables rigorous, reproducible science by providing collaboration, registration, and data management support across the entire research lifecycle. OSF is the only free and open solution that allows researchers to manage their projects from study onset and design, to preregistration, through study execution, and, ultimately, to posting research outcomes and archiving all associated research outputs (data, materials, code). OSF supports research across disciplines and generates value by ensuring that the process, content, and outcomes of research are accessible for discovery and reuse. OSF is comprised of four main web-based products that are free to individual researchers: 1) OSF.io, a project management platform for managing research projects, collaborating with others, storing data and materials, connecting to services like Dropbox or Google Drive, and easily controlling public and private access to the research; 2) OSF Papers, a publishing platform for sharing preprints, postprints, and other papers; Over 27,000 papers are published to date, with or without formal peer review, at a rapidly increasing rate (>55/day as of May 2019); 3) OSF Registries captures researchers’ study design and analysis plan prior to observing research outcomes. Upcoming iterations of OSF Registries will enable submission, moderation, and branding by groups or disciplines; and 4) OSF Institutions fosters collaboration, ensures open, persistent access to research, and promotes visibility of research at the institutional level. OSF empowers research communities to cultivate their own change toward openness.

2 OSF hosts over 150,000 registered users, with an average of 200 new researchers joining daily. The figure below illustrates the non-linear growth rate since launch in 2012. Based on the current growth rate, we project to have nearly 200,000 users by the end of 2019. This rapid user growth has made OSF one of the largest repositories and one of the largest registries in existence. OSF now hosts more than 4.5 million files comparing favorably to other general-use repositories such as Zenodo (1.07 million; launched 2013) and Harvard Dataverse (352K; launched 1996). Also, OSF has become the largest registry for basic and preclinical research, with over 27,000 registrations. With the current growth rate, we project to have more than 35,000 registrations by the end of 2019 (see figure right). OSF is not a graveyard of archived and unused material. There are more than 7x as many consumers of content stored on OSF as registered users. More than 1.1 million visitors came to the OSF in 2018, and these consumers viewed OSF content more than 13.7 million times and downloaded OSF content more than 5.5 million times. Remarkably, download rates for OSF content in 2019 are nearly double (187%) that of 2018 with 4.02M downloads as of May 1, 2019. OSF started in social and behavioral sciences, specifically psychology, and adoption rates over time reflect those origins. In a recent sampling of users from different time periods, we found that 90% of new users in 2013 were from the social-behavioral sciences, 57% in 2016, and by 2018, 44% of new users were from the social-behavioral sciences with the rest highly distributed across engineering and social, life, and natural sciences. To assess market penetration in our originating discipline, we conducted a crowdsourced investigation of all active, tenure-track faculty in psychology departments at 69 universities totaling 1,987 faculty. Overall, 35% of faculty had an OSF account, with adoption highest among assistant

professors (44.6%), then associate professors (39.1%), and then full professors (31.3%). The figure at left shows adoption rates over time. If the present trend continues, half of psychology faculty will be OSF users by sometime between early 2021 and late 2022. During the grant period, we expect to conduct similar investigations across ever-broadening disciplinary domains to support strategic planning and resource investment. Based on related evidence, our current speculation is that market penetration in other social- behavioral sciences,

3 education, and near- neighbor life sciences is approximately 15% (±5%), and about half that in natural sciences, engineering, and other areas of life sciences. As such, we have early adopter footholds in all areas of research. We will continue our expansion strategy of viral adoption influence by devoting attention to growth in near-neighbor disciplines to the areas in which we have had most impact. Beyond deepening the outreach across core social-behavioral sciences, our business plan calls for education and preclinical research as the next areas of focus. In 2016, COS launched OSF Preprints, an interface on OSF for sharing and discovering preprints. There are two key features to this service. First, OSF Preprints is supported by SHARE that aggregates preprints across services (e.g., arXiv, bioRxiv) so they are discoverable in a single interface. SHARE is an open database of 65 million research events from 175 sources such as publishers, institutional repositories, federal systems, and registries. SHARE was created by COS in collaboration with the Association of Research Libraries and with support from Association of American Universities and Association of Public Land-Grant Universities. Second, OSF Preprints is brandable so groups can launch their own preprint service for their defined research community. There are 24 preprint services operating on OSF such as EarthArXiv (earth sciences), EcoEvoRxiv (Ecology and Evolutionary Biology), engrXiv (Engineering), INA-rXiv (Indonesian research), AfricArXiv (African research), PsyArXiv (Psychology), SocArXiv (Social sciences), and Thesis Commons. These branded services enable groups to design and operate independent services on shared infrastructure, allowing them to customize their moderation workflow and guidelines, licensing options and discipline taxonomy for submissions. OSF Preprints provides a technical and operational model that we will extend to create OSF Registries. The growth trajectories of OSF adoption, study registration, and archiving and sharing of study materials and data are promising, but there are challenges to address to complete adoption by the mainstream, and to ensure that OSF functionality achieves the goals of increasing research credibility and accelerating discovery. OSF’s registration process needs to be improved and expanded to meet a broader variety of research use cases. OSF needs an efficient and effective result reporting workflow to maximize reporting and discovery of research outcomes. And, OSF needs to become more integrated with other research services to leverage their value and enable research efficiency and infrastructure sustainability. Addressing these needs with an NSF mid-scale infrastructure award will dramatically accelerate progress on improving rigor, credibility, and reproducibility of research. Implementation Plan For some groups, OSF is an essential tool. OSF has grown into a robust infrastructure for registering studies and analysis plans; managing and archiving research data, code, and materials; and helping open and controlled sharing of research outcomes and all supporting research content. With this grant, COS will expand OSF’s functionality and connections to support more diverse research needs and shift behaviors toward robust reproducible research practices. Work will focus on four areas: 1. Expand OSF functionality and workflows to meet the needs of a broad number of research disciplines and communities. COS will build upon OSF’s robust infrastructure, user interface, and user experience to support more discipline-specific workflows, with a particular focus on registration and result reporting. This work will deepen our impact within social-behavioral sciences, encompass our other priority disciplines--preclinical and education research--and move progressively to other fields by first leveraging collaborative networks between near-neighbor research communities. 2. Integrate OSF with other established scholarly infrastructure to make search, discovery, interoperability of content and metadata, and preservation of scholarly content more efficient to better meet the needs of institutional users (e.g., universities, research funders, and federal agencies). 3. Manage OSF design for efficient and sustainable long-term service. Design infrastructure improvements to manage the cost trajectory of OSF for efficient and sustainable scaling as it becomes an essential component of the scientific infrastructure ecosystem.

4 4. Training. Provide direct training in software development to build open solutions, and conduct training for diverse audiences in open research methodology and practices. Reproducibility challenges cannot be solved with solutions that merely archive the final outputs of published research because that does not address publication bias or questionable research practices. OSF is unique among open infrastructure solutions because it supports the entire research lifecycle from conception of ideas to archiving the final outputs of research. This makes it possible to eliminate publication bias and make transparent questionable research practices. As such, successful implementation will: a) rapidly expand adoption of reproducible research practices including study registration; b) increase access to data, materials, and code to allow independent replications of the original findings and extend the value of the research; c) ease discovery of outputs and prevent research outcomes from being locked behind paywalls; and, d) enable continued, sustainable free access for researchers across disciplines, career stages, and diverse demographic backgrounds. Project Activities COS uses the Agile Framework for Product and Engineering teams. Each project activity will begin with community and stakeholder engagement to assess goals, needs, challenges, and gain understanding of workflows. This phase is to empathize with the users to better understand the features and improvements we need to create. Individual interviews, surveys, and focus groups by the Product team will inform project requirements and user stories. UI/UX design is done through collaboration among UX designer, Product, and Engineering to address user, stakeholder, and technical areas. After defining scope, there is an analysis phase to review the current system, architecture, and functionality to define the work plan, set acceptance criteria for testing, and split the work into manageable deliverables to complete the feature or improvement. During implementation there is constant communication among team members to raise gaps and blockers, validate with prototypes, demos and testing, and ideate on challenges. Next, QA testing ensures consistency with the system using an environment that mimics real user data, load, responsiveness, and edge cases. When appropriate, new functionality is released for testing among selected stakeholders before release. COS will employ in-person focus groups, recruiting participants from local higher education institutions including the University of Virginia and Piedmont Virginia Community College, and online user testing. Participants will include a diverse range of experiences, disciplines, academic ranks, institutions (R1 universities, two-year colleges), and demographics. Participants will be paid for their time and sessions will take place on weekdays and weekends to ensure accessibility of participation. Each project activity will include an evaluation plan, determined in the early stages of empathizing and defined through assessment of the needs and goals stakeholders expect as deliverables of the work. 1. Expand OSF functionality and workflows COS will take two approaches to broadening open science behaviors: 1) Improve user experience: OSF adoption rates are non-linear, but we have evidence they are constrained by user interface limitations that can present barriers to mainstream adopters. In 2018, we conducted extensive user testing of OSF interfaces and workflows, identifying priority areas for improvement. We have initiated some of those improvements with a user-focused design and testing process, and expect initial improvements to be released during 2019. This NSF grant will extend and accelerate those improvements across additional OSF use cases. We will begin with a focus on meeting the diversity of research methodologies within the social, behavioral, and education sciences beyond experimental studies including longitudinal research, observational studies, qualitative studies, and research with existing data. Then, we will extend to conducting user research with researchers in other disciplines that are less well-represented in the OSF user community such as preclinical research, engineering, and the natural sciences. Learning the nuances of these communities’ research workflows will help maximize the applicability of the services to a broad range of use cases with appropriate language, symbols, syntax, user flow, and relevant tools for potential integrations.

5 2) Develop registration and reporting workflows in collaboration with specific disciplines: There are commonalities in registration and reporting across disciplines that make it possible for shared infrastructure to be effective across domains. Simultaneously, there are idiosyncratic disciplinary needs, particularly in metadata that can maximize the relevance of the tools and workflows for specific research applications. We will extend our impact in social-behavioral sciences and neighboring disciplines by creating workflows for specific research methodologies and domains, while maintaining common, shared metadata where possible for meta-analysis research across research areas.

1.A. Efficient registration submission workflows 1.A.1 Community engagement on design and implementation of registration formats. OSF presently enables registration of studies with eight different formats from an open-ended format for highly customized registration needs to a highly structured, comprehensive “OSF Preregistration” that reduces researchers’ degrees of freedom for conducting confirmatory tests (Veldkamp et al., 2018). Adoption of these registration formats has approximately doubled yearly since launch in 2012 with a cumulative total of more than 27,000 registrations (as of May 2019). To move completely into the mainstream, these registration services can be improved to ensure ease of use and workflow integration. Implementation of open science best practices has been fastest for experimental research and for data that is easy to anonymize (see Community Priority). In such research, preregistration is highly feasible because the scope of analysis strategies is usually tractable. Also, materials and data are easily shared because they can be anonymized and the data are often easy to obtain making them “cheap” to share. There are, however, many other research formats that do not fit so neatly into for preregistration and open practices. Observational (correlational) studies often involve many variables that can become the basis for multiple investigations. Longitudinal studies are often highly effortful to conduct and planning how the data will be analyzed or reported at the outset is difficult. Existing datasets are the basis for many investigations but they have already been observed. Each of these present challenges for the “pure” notions of preregistration and data sharing because of practical and ethical realities. There are also horizon opportunities to explore: registration in computational research and some natural and life sciences for which preregistration is, so far, a curiosity not a common practice. For this grant, we will focus on the opportunities and challenges for improving openness and reproducibility in more challenging research circumstances, and then implementing technical solutions to support effective behavior execution. COS’s Policy and Product teams will work closely together to review the existing preregistration formats to evaluate their overall effectiveness, and their applicability to emerging areas for registration -- observational studies, longitudinal studies, studies with existing data, and qualitative research. We will revise existing formats or generate customized formats as needed. We will manage a common metadata schema for better aggregation, discovery, and analysis across types. To make OSF content FAIR--findable, accessible, interoperable, and reusable -- we will apply metadata properties from the open, universal DataCite schema to describe files. This metadata will be exposed using schema.org tags to enable discovery of OSF data on Google Dataset Search and the metadata will be available via the OSF API. We will implement the DataCite schema on OSF projects and registrations in the workflow for minting DOIs. As disciplinary communities develop metadata ontologies for their outputs, they can be incorporated in the DataCite schema making files, projects, and registrations richly described and FAIR. Findability of data with rich metadata descriptions through discovery in SHARE, Google/Google Dataset Search, and institutional repositories will lead to efficient reuse. In addition to the user interviews and focus groups employed for each of these feature developments, we will convene in-person working sessions to bring together researchers, funders, and other stakeholders along with our Policy and Product teams to have in depth sessions on workflows, policy adoption, and implementation. We will host a convening meeting for each registration type (observational, qualitative, existing data, longitudinal) to examine the policy adoption and implementation challenges, forecast product and feature solutions that generate a roadmap of actionable solutions. We will continue to engage

6 these groups throughout development to refine content and workflows, and attend domain specific conferences to engage the community about the challenges, opportunities, and implementation. 1.A.2 Branded, community-run registration services. OSF Preprints supports disciplinary, geographic and society research communities with its common infrastructure to launch and operate branded paper services. OSF Registries will do the same for communities focused on registration to enhance rigor and reproducibility. These hosted, branded registries will include customized registration formats that fit the terminology and research design needs that are specific to that community, administration tools for managing and reviewing submissions to the service, and customized filters to facilitate discovery of registrations by consumers. Currently there are eight registration templates on OSF, all developed through collaboration with stakeholders to add rigor and reproducible practices to the research process. With partners defining how to register qualitative, longitudinal, secondary data analysis, systematic review and other study methodology types, we can format rigorous templates for making the details of plans, study designs, and hypotheses more rigorous and reproducible. These templates will be built on a common, standardized metadata schema which can be easily discovered and searched through a discovery interface and programmatically via API. We will add known registries with available metadata to SHARE (all OSF registries, clinicaltrials.gov, and Research Registry are already indexed by SHARE). Via a single interface, researchers will be able to find registrations from multiple sources to maximize discoverability of OSF Registries hosted, branded registries and external registries. Searching across OSF branded registries in one common, aggregated search interface on OSF Registirres will allow researchers and consumers to easily find, reuse and build on research across registries, templates and reported outcomes. Brandable registries will foster innovation in registration formats, adoption of registration in new research communities, and cultivate a culture of information sharing across registries for development and adoption of best practices. 1.B. Customized, efficient result reporting workflows Registering studies reduces the potential effects of publication bias by making all studies on a topic discoverable. Registering analysis plans reduces ambiguity between confirmatory and exploratory aspects of research and improves diagnosticity of statistical inferences (Nosek et al., 2018). However, if the results of the research are never reported, it is difficult to realize the value of either aim. Efficient reporting formats can facilitate deposit and discovery of results for aggregation and reuse. We will create standardardized reporting workflows and enable customization of metadata for specific discipline or study design use cases. This will dramatically improve outcome reporting and reuse of the database of registrations. For example, an effective reporting workflow will remind the researcher of their pre-committed analysis plans and provide a seamless mechanism to report outcomes connected with each of those registered elements. Reporting workflows must also accommodate changes between what was planned and what actually occurred because of reality of data collection, discovery or errors in plans, or violations of statistical assumptions. The goal of registration is not to restrict such deviations. Rather, the purpose is to make the deviations transparent. Good technical implementation should facilitate the methodological purpose and principles. As such, our design process will be a close collaboration between our Policy team focused on the substance and rigor of the forms and the Product team focused on the design and workflows that aid researchers reasoning and reporting about outcomes and deviations. The effectiveness of the result reporting workflows will also have significant implications for the consumers of the registered studies. For example, readers should be able to easily understand how the findings correspond with the registered designs and how unplanned deviations might alter their understanding or confidence. Also, we will provide well-structured data according to FAIR principles to facilitate aggregation of evidence across multiple reports as an aid to cumulative science. Funding and institutional bodies are developing stronger reporting requirements for research. This is creating demand for efficient services integrated with researcher workflows so that well-intended policies are treated as good methodological practice rather than as bureaucratic burdens. Our Policy team works closely with research funders, journals, and publishers to understand the workflow challenges, social pressures, and incentives structure that inhibit adoption of open practices. They will work closely with the

7 Product team to design solutions to these barriers to adoption. To support this work, we have recruited representatives of research funders that wish to improve the rigor and effective reporting of research that they fund. This group will be an advisory committee for feedback on design and implementation of reporting workflows to meet funder and institutional needs. The advisory board consists of Stuart Buck, VP of Research for Arnold Ventures; Dawid Potgieter, Senior Program Officer for Templeton World Charity Foundation; Maryrose Franko, Executive Director of Health Research Alliance; Nicholas Gibson, Senior Program Officer for John Templeton Foundation; and Georgina Humphreys, Clinical Data Sharing Officer for Wellcome Trust. Together, this group awards hundreds of millions of dollars in research funds each year. Their commitment to this work has the potential to shift scientific research towards more open practices and generate more value in research funding. We will follow user product testing principles deploying user interviews and research focus groups, as described above, with stakeholders of different profiles such as: researcher reporting research outcomes, researcher discovering research outcomes, and institutional administrator receiving research outcomes. 1.B.1 Secondary submission workflow for reporting results from registered studies. We will create a result reporting workflow for registered studies. This efficient reporting workflow will provide standardized outcome reporting formats that connect the results of the registered studies with the registration. There are not yet standard workflows for result reporting on OSF. The first step will be to review reporting workflows that exist on other services (e.g., clinicaltrials.gov) and conduct intensive user-testing with potential depositors and consumers. Then, we will establish one to three core result reporting formats that are applicable as widely as possible. After public release of these formats, we will conduct reviews of submitted content and user research to identify gaps for iterative improvement. The product testing approach will follow that described in 1.A.1 for registration workflows. This will lead to improvements of the initial formats and implementation of specialized reporting formats -- such as distinct metadata standards for experimental, observational, longitudinal, qualitative, and existing data research. 1.B.2 Search upgrade with filtering and discovery tools. We will need to update our search infrastructure to formally integrate results reporting with our registration services, and add customized registration formats fitting the terminology and research design needs for communities. The discovery interface and search index will both need to be enhanced to incorporate more searchable fields for exploring contents. Most notably, they will support searching by answers to specific registration form questions, and by relationships between a given OSF project and time stamped, immutable registrations of hypotheses, experimental design, study protocols, codebooks, analysis scripts, and reported outcomes. These features will allow aggregating and isolating specific metadata for research and institutional use cases. For example, researchers conducting meta-analyses will want to find all studies on a particular topic. Funders and institutions will want to track compliance with registration and reporting requirements. Because registration metadata will be standard across all study types, this will improve how consumers can compare registration with reported outcomes within and across domain areas. To achieve this, we will standardize our registration forms, including form data on existing registrations. This will let us index the answers to questions in a consistent way, making them more searchable. We will also introduce the concept of relationships between registrations and reported outcomes, and create and improve user interfaces for search and discovery that meet these use cases. For user research, we will engage our advisory board of research funders to assess their needs as consumers of result reports, and researchers with the personas of searching and discovering registrations for meta-analysis or idiosyncratic investigations. Following initial release, we will conduct the same iterative review and improvement process for metadata and discovery tools. 1.B.3 Automated reminders. We will work with stakeholders to define and implement automated reminders for outcome reporting on OSF. This might include a mechanism at the point of registration for researchers to define a schedule of reminders for uploading their data, sharing their preprint, and connecting their outcomes to their institutional repository. The automated system would send the researcher an email at the predefined time with a call to action and a link back to the registration. We would determine the appropriate lead time and frequency for best user engagement. We will implement a simple initial version for researchers to define their expected completion dates and receive reminders. In

8 parallel, we will conduct a stakeholder review for requirements for a variety of use cases for seeking to improve adherence to registration and reporting incentives or mandates. Reminders will also be useful for identifying studies that were aborted for any reason after registration but before the study was completed. The system will enable researchers to report on the circumstances for aborting the project and meet reporting standards. Finally, an institutional and funder version of automated reminders will facilitate compliance review. After release of the initial automated reminders, we will evaluate usability, timeliness, and effectiveness in encouraging reporting of outcomes. As new registration templates, workflows, and research methodologies are supported, we will iterate to incorporate automated reminders for them. 1.B.4 Connecting registrations, results, papers, presentations, and data. The credibility, reproducibility, and reusability of registered studies can be greatly enhanced by making the content supporting the findings -- e.g., data, materials, protocols, code, presentations, and reports -- easily accessible. OSF facilitates connecting registrations with data, materials, code, and papers either as stored content on OSF or linked storage at other sources. This can be improved with result reporting workflows. During reporting, we will provide options to deposit or provide links to data, materials, code, and papers. We will also implement calls to action on the registration viewing pages for authors to add related content any time they visit the registration before or after the work is completed. Finally, we will enhance the automated reminders for reporting results to include encouragement for depositing data, materials, and code that can be archived with the registration privately, publicly, or with controlled access in the event that the study involved sensitive data that can be archived and shared only with appropriate permissions. 1.C. Evaluation of registration and reporting process and content OSF records copious data about behaviors during registration. Analysis informs product design by identifying fields that are commonly misunderstood or not used effectively. Data can inform design of the workflow of efficient and effective registration. Continuous evaluation of challenges ensures design is supporting good practice and helping researchers focus on planning rigorous and impactful research. A data scientist will conduct continuous evaluation of OSF behavior against the objectives for each service. This analysis will directly inform product development and sustainability strategies. Analyses will include fundamental metrics of the health of the registration and reporting process such as: ● How many registered studies exist on OSF Registries by discipline? What is rate of new submissions, submissions by discipline? How are the rates of registration changing over time? ● With the implementation of automated reminders, is there increased reporting of outcomes? ● With the implementation of easy connections of data, code, supplementation materials, presentations, papers, and reports on registered studies, what proportion of registrations include these connections? ● Are some registration fields more likely to have missing data than others? ● How much time passes between starting a draft registration and submitting it? ● What proportion of draft registrations are ultimately submitted? ● What proportion of registered studies have reported outcomes? ● How many searches are conducted on OSF Registries for registrations and/or reported outcomes? How many on OSF Registries hosted, branded registries? ● What are the view metrics on registrations and reported outcomes from OSF Registries searches? ● What are the download metrics of discovered datasets from OSF Registries searches? ● What percentage of registrations and reported outcomes are submitted by institionally affiliated (SSO through OSFI) users?

Fundamental metrics like these are benchmarks for tracking improvement. For example, it is desirable to have results reported for all registered studies that have been completed. And, if some registered studies are not completed, it is desirable to have that identified--ideally with explanation. There are other questions for which it is not obvious what answer is desirable, but understanding the behavior will be instrumental in ultimately revealing the quality, opportunities, and challenges for registration. For example: ● What proportion of registered studies report changes between initial plans and final outcomes based on researchers self-report? ● Are changes more likely for some parts of the methodology than others?

9 ● Are some types of research less likely to be completed than others based on researcher’s self-report? ● What proportion of registered studies are known to be ultimately published? ● How much time passes between initial registration and reporting? Was the report submitted only after an automated reminder from the system was sent?

These will spur hypotheses for evaluating how registration meets its promise for accelerating progress in science and if there are unintended consequences that are counterproductive. For example, does registration make researchers more conservative in the types of questions that they investigate? If so, it could have implications for managing trade-offs between confirmatory rigor and aggressive innovation. OSF’s API makes a substantial amount of data available to the research community. We will encourage researchers to use OSF data to investigate these questions. Finally, the outcomes of the data science work will inform product development--good technological design can help improve research rigor. 2. Integrate OSF with established scholarly infrastructure In 2018, COS engaged a consultant to develop a three-year business plan. The consultant interviewed 75 stakeholders, including individual researchers, research funders, and decision-makers at universities, societies, and journals. The resulting plan defined the value of the OSF and honed our priorities. One such priority is building better connections with existing scholarly infrastructure. Our institutional service, OSF Institutions, is currently offered as a free service for universities and funders. OSF Institutions provides a discovery page of public OSF content by researchers at the institution, and login authentication using the researchers’ institutional credentials to lower the barrier to on-boarding. Interviewees provided helpful feedback for improvements to OSF Institutions including usage tracking, reporting, and better integration with other research infrastructure. With this grant, COS will improve OSF to better meet the needs of universities and funders. This will accelerate adoption by aligning the infrastructure with institutional policies and incentives, encouraging best practice across domains and career stages, and enhancing the impact of research funding by making the outputs more discoverable and reusable. As of May 2019, 60 institutions have signed up for OSF Institutions, including 22% of the U.S.’s R1 universities (Carnegie Classification). Based on the business plan work, COS is rolling out a fee-for-service for OSF Institutions as part of the long-term sustainability strategy for the infrastructure. Nearly 80% of institution representatives interviewed said they would be willing to pay an annual fee to support OSF as a necessary tool for their researchers. During this grant period, we will roll out and scale up this earned-revenue strategy. The value proposition will be enhanced by integrating OSF with more tools in the scholarly communication ecosystem. Integration with existing services, such as data repositories will help mitigate hosting and storage costs that could inhibit scaling OSF by leveraging institutional investments in data storage and preservation. Advancing integration of OSF with the infrastructure ecosystem will increase the value of OSF for institutions and agencies by meeting monitoring and compliance use cases, and provide a mechanism for scaling adoption while simultaneously mitigating maintenance costs for sustainability. Valuable stakeholder feedback from interviews with deans of libraries, VPs of research, and data management specialists illustrate buy-in: ● “[Having an institutional OSF page] was important for us to signal our support for open science.” (Washington University) ● “I think COS is doing a great job. No complaints. I really like the tool and the organization.” (University of Arizona) ● “We’ve tried to market OSF and are very supportive of it. We definitely want to promote its use, it’s a great tool for project management.” (Oklahoma State University) ● “We are big fans of OSF. I knew about how great it was as a research management tool.” (Duke University) ● “The climate for data sharing and data openness has increased dramatically over the past five years….When Nature and Science are writing about data sharing, they write about OSF.” (VCU) ● “OSF Institutions gives trust to the faculty that the institution thinks this is a good product.” (James Madison University) ● “I was inspired to offer classes in OSF myself. This platform is in a class of its own.” (NYU)

10 ● “We secured a grant to really invest in and promote OSF. Over the summer we are running a number of hands-on workshops.” (Carnegie-Mellon University)

2.A. Connecting registration and result reporting with other services The maturation of registration and reporting requirements for clinical trials at NIH and emergence of similar requirements by others anticipates rapid growth in monitoring and compliance by institutions, funders, and federal agencies. We will enhance OSF to facilitate automated transferring of registration and outcome reporting information to meet institutional, funder, or federal reporting mandates. Already, all metadata for OSF registrations are automatically submitted to SHARE, a database of metadata for millions of research events including publications, grants, registrations, preprints, and datasets. Metadata for preprints on OSF are integrated with Crossref and Google Scholar; projects, registrations, and, soon, datasets are integrated with DataCite and Google, all of which is available via the OSF API. We will add outcome reporting metadata to SHARE and other aggregation services. We will draw on advice and feedback from our funder advisory group and from active collaborative contacts with federal agencies. For example, we have been actively engaged in collaborative discussion with NIH regarding its requirements for registering and reporting clinical trials. The collaborative potential is connecting OSF with NIH to assist researchers meeting those mandates with the efficient OSF registration process and then facilitated deposit into federal repositories (e.g., clinicaltrials.gov). Likewise, COS is involved in two programs with DARPA (SCORE and NGS2) that place a heavy emphasis on incorporating registration into their research programs. There is interest in extending these improvements to research rigor across more programs. We will seek DARPA feedback for enhancing OSF’s connections and reporting mechanisms for their priorities. We will also seek feedback from all NSF directorates that have similar interests. Finally, institutions desire discovery and preservation of outputs such as study designs, analysis plans, hypotheses, registration, and outcome reports. The registrations system will support timely reporting of registration and outcome reporting from OSF to institutional repositories for their researchers. 2.B. Integration with Data Repositories OSF’s API connects services together to make it easier for collaborative teams to prepare and cite their research materials or literature reviews (Mendeley and Zotero), to share and organize files across the services that they use (e.g., Dropbox, GitHub, Google Drive) and then transition those files, if they wish, to an archival repository at the end of the research lifecycle (e.g., figshare, Dataverse). These integrations make research more efficient; it becomes easier to collaborate across multiple services, easier to discover project content in a single OSF interface, and easier to reproduce because of discoverability and recording in the history of the project. A key use case for the OSF API is integration of storage services and data repositories. Researchers have many storage options for data and materials: some are generalist repositories, others are specialized repositories with specific metadata and formatting requirements. OSF renders hundreds of file types directly in the browser, making it effective for discovery. By integrating repositories with OSF, all data for a single project can be discovered in the same interface no matter where it is stored. For example, OSF projects integrate data stored in Dataverse, code stored in GitHub, raw materials stored in Dropbox, and other content in OSF storage to help the data producers manage their work and collaboration, and for the data consumers to discover all of that content. The storage solutions integrated with OSF include OSF storage, Dropbox, GitHub, Amazon S3, Box, Google Drive, figshare, Dataverse, Bitbucket, GitLab, OneDrive, and ownCloud. We will expand the storage integrations to institutional and domain-specific repositories. This will capture the interest articulated in interviews with institutional stakeholders such as “In a dream world I see OSF like a hub, it’s the thing that connects other things together.” (Virginia Tech). We will: (1) improve discoverability of institutional and domain repositories for research producers through their availability in the OSF project interface; (2) improve the deposition and discoverability of data, materials, and code for a single project stored or archived in multiple locations through interoperable integrations on OSF projects; (3) facilitate the transfer of data from real-time active storage during the project lifecycle (e.g., Dropbox) to long-term

11 archival storage solutions (e.g., UVA’s Dataverse: LibraData); and (4) mitigate escalating storage costs associated with scaling (described further in section 3 below). By facilitating a direct connection in OSF projects to institutional or domain repositories, researchers can easily make research outputs, such as preregistrations, analysis plans, datasets, and reported outcomes, discoverable and meet the substantial interest among data repositories to increase their discoverability and use for appropriate data. If generalist repositories like OSF can facilitate discovery of appropriate domain repositories, it will increase the likelihood that data finds its most appropriate home, maximizing FAIRness and reuse. We will: (1) assess institutional repository landscape where custom implementations are often built from a common backend framework to determine which are most commonly used; (2) determine how best to interoperate with institutional repositories from the perspective of the researcher actively managing the research lifecycle and of the repository managers who are curating and monitoring institutional research outputs; (3) identify institutional partner repositories to pilot integrations and provide outreach, support, and training for repositories to connect their services; (4) starting in year two, host a yearly meeting for stakeholders managing and expanding integration of preservation services; these meetings will assess the functions, standards, and shared metadata that are essential for scholarly work, examine what legacy features or workflows are redundant or whose maintenance costs outweigh their utility, explore notable gaps in connections, assess how infrastructure providers better connect to serve users, support providers integration efforts, and collaborate on setting next priorities; (5) provide small grants to preservation services to support their initial work connecting to the OSF API; and (6) build and launch a discovery interface for OSF users to find and transfer data into the most appropriate repositories. Integration with institutional repositories expands the value proposition for institutions supporting OSF. We will undertake a thorough discovery phase with the Product and Engineering teams to determine the most efficient way to build a scalable solution for institutional repository integrations. Ideally, a substantial number of institutions will share a common back-end with similar configurations. If so, then we will build a service for the most common repository solutions to manage the interaction between OSF and the institutional repositories. This technical solution would be comparatively easy to maintain for integration of many institutional repositories. It is also possible that there will be many flavors of institutional repositories interfering with the single service solution. If that is the pervasive circumstance, then we would write a shim service and specification for communicating with OSF. Then, institutions would write their own shim to work with that specification. This solution enables handling of customized solutions, and distributes the workload across institution repository managers. The small grants described in the prior paragraph would be important for making progress on this solution. 2.C. Administrative Tools for Institutional Users The expansion of OSF Institutions to include discoverability of registrations and outcome reporting for monitoring and compliance, and integration of institutional repositories, will be enhanced with administrative tools organized via an administrator dashboard to manage their institutional accounts. Providing institutional administrators with easy, timely, and comprehensive views of their researchers’ OSF activities will enhance the value proposition of the OSF Institutions service and enable administrators to track implementation of open practices at their institutions. 2.D. Evaluation of Integrations COS will improve OSF to better meet the needs of universities and research funders, particularly to improve reported outcomes and the ability to track implementation of open and reproducible practices at institutions. Effectiveness will be assessed through interviews with researchers conducting clinical trials affiliated with a connected institutional repository, and reporting outcomes to funders. Similarly, we will conduct interviews with institutional researchers, data librarians, and administrators to evaluate the usability of repository integrations, ease of adoption within workflows for research output dissemination, and interoperability of systems. We will evaluate the following: ● Increased discoverability of scholarly outputs through new and expanded connections with government grant agencies, funders, compliance reviewers, and institutions.

12 ● Count of reported outcomes disseminated from what number of registered studies; from which discipline areas, registration template type, and affiliated institutions. ● Insights on adoption across a campus through admin panel metrics, seeing registration submission totals, reported outcome totals, with views and downloads on those pieces of content. ● Use of repository connections for archival and preservation of research outputs. How many users enable the connections, on how many projects, how much content is pushed to repositories. 3. Manage design for efficient and sustainable long-term service Funds from an NSF infrastructure grant would support software improvements to distribute data hosting. By improving the API to integrate with institutional repositories, we can (1) reduce the cost trajectory for maintaining OSF as it scales by distributing storage and archival costs across the many existing and effective solutions; and (2) minimize duplication of storage costs by leveraging institutional dollars spent on repositories for storing the same files, simply aggregate the content metadata, and provide a discovery interface on OSF for increased interoperability, accessibility, reusability, and discovery. This will provide a sustainable cost structure for free, open-source infrastructure supporting the scholarly community while simultaneously meeting the demand of research institutions, federal agencies, and individual users. COS will provide small grants to institutions seeking to connect their repository to the OSF API. We will award integration grants each year, prioritizing awards to less-resourced institutions, two-year colleges, historically black colleges and universities (HBCUs), and Hispanic-Serving Institutions (HSIs). We will enable tracking of data hosted on OSF and external integrations to assess scaling, assessing cost per user and then at quarterly intervals to track trends over time. With repository partners, we will review usage metrics on the connected repositories to track trends in discovery of scholarly outputs. 4. Training 4.A. Training for effective registration and result reporting Effective study registration and result reporting are not simple documentation tasks, they are acquired methodological skills that support research rigor and reproducibility. COS has compiled education and training material in the form of a training program, academic papers, an online support knowledge base, video recordings, templates, FAQs, a regular open webinar series, and a support email for technical questions about using OSF. Some of these are scalable solutions for meeting the influx of adopters. We will continue to enhance these materials for registration and reporting advances during the project period. Training support for registration, data sharing, and other open research practices will take three forms: 1) direct in-person training with a cohort of schools serving a diverse population of undergraduate and graduate researchers; 2) free quarterly online webinars that will be publicly accessible and promoted to all research institutions; and 3) documentation and written help guides added to our existing online knowledge base that individual researchers can use for self-teaching. We will provide direct outreach and training to institutions that serve a high percentage of students from underserved or underrepresented groups in STEM. Direct training is not scalable, but providing this complementary service to our online resources will ensure expansion of the value of the infrastructure services to communities that are often left out or left behind. In the first year, we will employ our established training materials, updated with the new features and services, and deliver a mix of in-person and virtual trainings to people at six institutions (all have agreed to participate). Each year, we will recruit new institutions that serve diverse student populations for a new round of training services. We will track trainings delivered, participants, and diversity of participants by gender and race/ethnicity. Training Program: Participating Institutions, Year 1 Cohort Name1 Location1 Carnegie Student % of students %of classification1 enrollment identifying as students graduate and other than White receiving undergrad1 or Caucasian2 Pell grants2 University of California Riverside, CA R1-Doctoral 23,279 84% 56% Riverside University Public

13 Pacific Lutheran Tacoma, WA M2-Master’s 3,122 32% 30% University University Private New York City College New York, NY Baccalaureate 17,279 83% 54% of Technology (CUNY) Associate’s Colleges Public Piedmont Virginia Charlottesville, Associate's Colleges 5,608 34% 25% Community College VA Two-year Public University of Florida Gainesville, FL R1-Doctoral 52,669 43% 28% University Public Virginia Richmond, VA R1-Doctoral 30,675 48% 28% Commonwealth University Public University 1Source - The Carnegie Classification of Institutions of Higher Education 2Source - College Scorecard- Department of Education 4.B. Internship program for student developers and designers contributing to infrastructure COS has an established, high-performing software development team that employs agile development processes. Students interested in software development careers can benefit substantially by participating and contributing to the implementation of new OSF infrastructure. Historically, COS interns have contributed substantially to the organization’s mission. Since 2013, COS’s internship program has included 278 students. Interns received hourly pay ($15-$25/hour) or course credit, when applicable. The majority of interns were in undergraduate programs, but some were pursuing graduate studies. COS conducts both local (University of Virginia, Piedmont Virginia Community College, Virginia Commonwealth University, Virginia Tech) and national recruitment for this program by attending career fairs, posting on job boards, and hosting booths or meet-ups at industry conferences that promote STEM careers to underrepresented communities, such as the Grace Hopper Conference for women in tech. This program has been highly effective for fostering students’ capacity in software development and for promoting diversity. For example, 10% of students who participated in an internship at COS returned for a full-time position after they graduated, and many more went on to full-time positions at other technology organizations. Also, approximately 40% of our interns identified as female, and 47% identified as a race or ethnicity other than White/Caucasian. For this project, we will support 15 intern positions per year with an emphasis on recruiting high-potential students from diverse backgrounds, offering 60 internships throughout the duration of the project. Internships will be offered in the summer as an intensive 10-week session with six to seven participants committing 25-30 hours per week. Fall and Spring sessions will run for 12 weeks and a cohort of four interns will commit 10-15 hours per week. They will receive hourly pay or course credit. Positions will be available in engineering and product and will actively participate in the defining, developing, and testing of features defined in this project. 4.C. Evaluation of Education efforts Every remote or in-person training delivered by COS staff includes an evaluation survey about the knowledge gained, the likelihood of implementing these practices, and suggestions for improvement. These surveys will be complemented by adoption metrics and interviews to assess workshop effectiveness. These will iteratively inform training implementation to improve and reduce barriers to adoption. Also, we will review analytics on our knowledge base and tech support cases to assess if the materials are resolving challenges, and informing decisions about next content generation and webinars. We will track intern program metrics including number of applicants, participants, hours worked, as well as qualitative intro and exit interviews. Interns will complete a presentation to the staff at the end of their internship. Research Community Benefits

14 Researchers are engaged in an active reform movement to increase openness and reproducibility. Surveys and behavioral data demonstrate a rapid shift in adoption of open science practices such as data sharing, materials sharing, and preregistration (e.g., Kidwell et al., 2016; Nosek & Lindsay, 2018; Paluck, 2018). With continued non-linear growth on OSF and elsewhere, openness will quickly become the default, normative practice. Maintaining effective infrastructure is central to achieving this. Following the Holdren memo (Holdren, 2013), all federal agencies with research budgets of $100M or more took steps toward defining policies to make more research openly accessible. There is substantial interest in Congress for improving transparency and reproducibility of research. Committees in both the Senate and House have held hearings and drafts of legislation to improve reproducibility are circulating. NIH and NSF have pursued revisions to researcher biosketches that decrease the emphasis on publication as the sole scholarly contribution, and are providing funding mechanisms for replication and training to promote reproducibility. NIH initiated updates to its clinical trials policy, proposing to extend the mandate for registration to most all experimental research with humans. Organizations like the Council on Government Relations (COGR) and the Federation of Associations for Behavioral and Brain Sciences (FABBS) expressed support of OSF to NIH as a registry service to meet the proposed requirements. National Institute of Standards and Technology (NIST) is actively evaluating OSF for potential adoption as infrastructure supporting intramural research. Open Science by Design: Realizing a Vision for the 21st Century Research (2018), a consensus study report of the National Academies of Sciences, Engineering, and Medicine, concluded: A new generation of information technology tools and services holds the potential of further revolutionizing scientific practice. The ability to automate the process of searching and analyzing linked articles and data can reveal patterns that would escape human perception, making the process of generating and testing hypotheses faster and more efficient. These tools and services will have maximum impact when used within an open science ecosystem that spans institutional, national, and disciplinary boundaries. Open Research Funders Group (ORFG) is a partnership of funding organizations committed to the open sharing of research outputs. In March 2018, ORFG issued a survey (ORFG, 2018) to better understand funder perspectives with respect to supporting open infrastructure. When funders where asked “How much of a priority do you believe open science infrastructure projects should be to the ORGF?,” 50% of the survey respondents ranked infrastructure projects as “one of the highest priorities” with 50% of respondents indicating infrastructure would be an “average priority.” When asked “How familiar are you with open infrastructure systems like the Open Science Framework (OSF)?,” 86.6% of respondents reported they were “Very Familiar” or “Somewhat Familiar” with the OSF. Operations and Utilization Plan

15 Management Plan. COS is organizationally structured to support its three key areas of focus: Research, Policy, and Technology. Each area overlaps and the senior management team is a highly communicative and collaborative team, meeting twice weekly to steward projects, update on progress, and navigate blockers. The technical team, most relevant to this project, is led by David Litherland, who brings over 20 years of experience at leading engineering teams, and Nici Pfeiffer, whose background in engineering and data analysis informs her experience as Director of Product. The senior engineering team bring a depth of technical experience in full stack code development, systems and security design, UX/UI, and quality assurance and testing. The Organizational Chart above illustrates the leadership team and key staff for the project. An asterisk indicates a role that would not be funded through this project. Governance. COS has a highly effective Board of Directors with a diverse representation of leaders across stakeholder communities in science. Brief profiles of the current Board are in the table below. The Board maintains fiduciary responsibility for effective governance of the organization.

Maryrose Franko, Executive Director, Health Research Alliance; Prior: 20 years of program Chair management at Howard Hughes Medical Institute; PhD Molecular Genetics.

Alison Mudditt, Vice CEO, Public Library of Science; Prior: Director, UC Press; Senior leadership Chair at SAGE Publications, Taylor & Francis, and Blackwell Publishers.

Marcia McNutt President, National Academy of Sciences; Prior: Editor-in-Chief Science family of journals; Director of United States Geological Survey; CEO of Monterey Bay Aquarium Research Institute; Professor at MIT in Geophysics

Alan Leshner Emeritus CEO of American Association for the Advancement of Science and Executive Publisher of Science; Prior: Director of NIDA; Deputy Director and Acting Director of NIMH; Professor at Bucknell in Psychology

Alan Kraut Executive Director of Psychological Clinical Science Accreditation System; Prior: Executive Director of Association for Psychological Science

Beth Simone Noveck Professor at NYU Tandon School of Engineering and Visiting Professor Yale Law School; Co-founder/Director of The Governance Lab. Prior: U. S. Deputy Chief Technology Officer, Director White House Open Government Initiative

Jon Hill Managing Director at Investure, an investment firm working primarily with nonprofits to invest their endowments.

Rebecca Saxe Professor of Cognitive Science at MIT; PhD from MIT

Tom Katsouleas, ex Executive Vice President and Provost, University of Virginia; Tom will become officio (voting) the next president of the University of Connecticut and he will be replaced on the Board with the incoming Provost of the University of Virginia.

Brian Nosek, ex officio Executive Director, COS; Professor of Psychology, University of Virginia

Access and use of infrastructure by target research communities. OSF is a free, open-source infrastructure for researchers that is available for use by anyone at any time. That is, OSF is a public good. With more than 1.1 million unique visitors consuming OSF content in 2018, the infrastructure has broad reach across research communities. We expect this to grow to more than 3.2 million unique visitors yearly by the end of the project period, an increase of almost 300%. Also, there are 150,000 registered users now, and we expect to have nearly 200,000 users at the end of 2019. By the end of the five-year project period, we expect this to grow to almost 750,000 registered users, an increase of almost 500%. Moreover, we will conduct specific outreach and training to researchers at institutions with a high

16 percentage of historically underrepresented groups to bridge any visible or invisible barriers to entry. Our key metrics will document success in meeting our goals for increasing access and use of the infrastructure. Planned Metrics. A suite of key performance indicators (KPIs) will document base usage of the OSF and its growth over time. These KPIs indicate fundamental events that indicate actions relating to improving openness, rigor, and reproducibility. The table shows each metric and provides a conceptual definition. Across metrics, higher numbers are “good” in the sense that these metrics are behavioral indicators of researchers using OSF to store, register, share, discover, and use research content.

Metric Definition

Yearly Net Increase in Registered Users Net increase in the number of confirmed OSF users in a calendar year. Yearly Public Projects Net increase in number of public projects on OSF in a calendar year. Yearly Files Net increase in files archived on OSF in a calendar year. Yearly Public Files Net increase in openly shared files on OSF in a calendar year. Yearly Registrations Total number of registrations made on the OSF in a calendar year. Yearly Unique Visitors Total number of consumers of content on OSF in a calendar year. Total number of times a consumer visited the OSF directly (does not count direct interaction with content, e.g., download OSF-content via Yearly Views Google Scholar) in a calendar year. Total number of times content stored on OSF was downloaded by a Yearly Downloads consumer in a calendar year. Net increase of preprints shared on one of the (presently) 24 OSF paper Yearly Preprints Posted sharing services Yearly Result Reports Number of result report forms completed in a calendar year.

In addition to these KPIs, we will track metrics that are directly relevant for some of the planned activities in this proposal including creation of community-operated registries, connections with institutions and other services, and progress on sustainability indicators. There are a variety of other activities described in the proposed work, but description of progress on those activities is better described as milestones for technical and operational development rather than as metrics for ongoing monitoring.

Metric Definition

Activity 1: Number of Total number of branded registries operating on OSF Registries Community-Operated Registries service. Activity 2: Institutional Repositories Total number of institutional repositories connected with OSF connected with OSF for improved discovery Activity 3: OSF Institution Accounts Total number of institutions paying a fee for the OSF (paid) Institutions service to support long-term sustainability Activity 4: Yearly training sessions delivered Yearly total number of training sessions delivered Activity 4: Yearly number of student Yearly total number of student interns at COS that contribute to interns contributing to technical work the building of the OSF services

17 Evaluation of success and impact of NSF investment. Based on past use and assessment of potential growth trajectories, we have identified growth expectations for the KPIs that provide quantitative indicators of success and impact of this investment. These indicators have the advantage of being direct and comparatively easy-to-track.

Metric 2020 2021 2022 2023 2024

Yearly Net Increase in Registered Users 70,000 88,000 108,000 130,000 152,000 Yearly Public Projects 25,000 30,000 36,000 43,000 51,000 Yearly Files 2.1M 2.2M 2.3M 2.4M 2.5M Yearly Public Files 550K 570K 600K 640K 690K Yearly Registrations 12,000 15,000 19,000 24,000 30,000 Yearly Unique Visitors 1.8M 2.0M 2.3M 2.7M 3.2M Yearly Views 14M 15M 16M 17M 18M Yearly Downloads 8M 9M 10M 11M 12M Yearly Preprints Posted 18,000 20,000 23,000 27,000 32,000 Yearly Result Reports 0 10 100 500 1,000 The value of the NSF investment will also become evident by the extent to which institutions adopt and invest in OSF as an infrastructure that supports their researchers and institutional priorities. We identify some growth targets for the activity-defined metrics indicating service to the research community, a growing interdependence between OSF and other scholarly infrastructures, and a growing support base for the long-term sustainability of OSF.

Metric 2020 2021 2022 2023 2024

Activity 1: Number of Community-Operated 1 3 6 10 15 Registries Activity 2: Institutional Repositories 3 8 15 25 30 connected with OSF 11 27 48 72 100 Activity 3: Paid OSF Institution Accounts Activity 4: Yearly training sessions delivered 12 12 12 12 12 (webinar + in-person) Activity 4: Yearly number of student interns 15 15 15 15 15 contributing to technical work These indicators directly account for the usage and progress of the project aims. However, the ultimate success and impact of this project is in its capacity to accelerate discovery by increasing the openness, integrity, and reproducibility of research. Assessment of that is very important, very hard, and beyond the scope of this infrastructure grant. Effective assessment of accelerating discovery and improving reproducibility will require a sustained metascience effort evaluating the quality and reliability of the research literature, and the role of registration, transparency, and sharing in fostering change over time. We are, nevertheless, committed to pursuing such evidence as it is central to COS’s mission. If the services provided by OSF are not advancing these broader objectives, we need to know to alter course and pursue more effective strategies for accelerating science. We do have other funded projects that are highly complementary for these present infrastructure goals such as the DARPA SCORE program. With

18 SCORE, we aim to create automated indicators of credibility of findings and, as a side benefit of the project design, create a database of evidence for change in open science behaviors from 2009 to 2018 in the social-behavioral sciences, and the correlation of those behaviors with change in reproducibility. We will continue to seek funding opportunities for conducting such projects, and we will maximize availability of COS/OSF data for others to pursue independent investigations. Anticipated sources of operations and maintenance funding. COS operates with a $7 million annual budget. In COS’s business plan, we outlined an operating budget for 2019-2021. The pro forma budget outlined: (1) diversifying revenue streams. COS was founded with private philanthropy. Revenue supporting COS has diversified, adding new philanthropic sources, securing government contracts, and building individual giving support. The activities defined in the business plan further diversify the revenue base, targeting a mix by 2021 of 51% research grants, 39% philanthropic support, and 10% earned revenue; and (2) maintaining level expenses. The current staffing model can support the proposed improvements without introducing substantial ongoing costs. COS has secured philanthropic funds or government contracts that account for 50-60% of operating costs into 2022, and a healthy pipeline of renewable grant sources for future funding needs. Additionally, COS has a growing base of annual earned revenue from products and services. These funds will support the operating and maintenance costs of the OSF, among other COS priorities, while new development takes place. Following the conclusion of a formal business planning process in 2018, we are implementing earned-revenue strategies to provide a sustainable, diversified framework, starting with OSF Institutions. Our market analysis revealed that institutions have the willingness and capacity to support OSF as a public goods infrastructure. Assuming an average contribution of $10,000 per institution (a price a dean recently described as “trivial” for what the service does), our projected growth in paid OSF Institutions memberships will account for $1 million of revenue by the end of the five-year project period. Presently, COS devotes approximately $3 million to its product and engineering divisions and about 35% of those costs are attributable to operations and maintenance. As such, we expect that a substantial majority of the ongoing maintenance costs will be offset by earned revenue by year five. Further, the activities in this grant support our ability to (1) drive earned revenue by enhancing the fee-for-service OSF Institutions; (2) meet federal agency use cases, securing government contracts from agencies like DARPA and NIH for projects using OSF; and (3) mitigate costs of scaling by leveraging existing distributed storage solutions to control compounding storage costs. With these solutions in place, we expect to triple OSF usage within the existing expense structure. Future cost growth might provide for increasing web traffic, more international storage locations, or need for increased OSF user support, but costs per user would continue to decrease. COS does not anticipate requiring baselined operating support from NSF following the work defined in this proposal. Broader Impacts Researchers are engaged in an active reform movement to increase openness and reproducibility. Surveys and behavioral data demonstrate a rapid shift in adoption of open science practices such as data sharing, materials sharing, and preregistration (e.g., Kidwell et al., 2016; Nosek & Lindsay, 2018; Paluck, 2018). With continued non-linear growth of open science behaviors and others that improve rigor and reproducibility, these behaviors will quickly become the default, normative practice. This will have substantial broader impact beyond the specifics of this infrastructure proposal. Extendibility to other disciplines. This proposal emphasizes the infrastructure’s service to the social-behavioral sciences, but the OSF is already observing accelerated growth in other research disciplines. In a recent sample, 55% of new users were from fields outside of the social-behavioral sciences. Two additional areas of COS’s focus are education and preclinical research. We also expect that adoption will continue apace across engineering and life and natural sciences. As such, this work could have scalable impact that reaches well beyond its target domains. Public Goods Content and Infrastructure. Most tools and services supporting scholarly communication are proprietary and the basis of business models that limit accessibility of content to the public. The infrastructure and services generated through this proposal will all be public goods -- freely accessible

19 and openly licensed. This represents COS’s fundamental commitment to the principle that academic scientific research is a public good. To maximize the public’s return on investment for supporting science, the science itself -- papers, data, materials, registrations, and the infrastructure hosting it -- should be public goods to the maximum extent possible. Inclusivity. Open, accessible content improves inclusivity to the research process. Producers and consumers of OSF content do not need to be affiliated with a wealthy institution; they only need an internet connection. As such, all stakeholders can produce or consume research in proportion to their interests, skills, and available time to do so. Accessible public infrastructure and content helps level the playing field in contrast to the siloing and exclusivity produced by most research funding going to a very small number of universities and the privileged few having connections to such institutions. Credibility of Science. In modern society, science is one of many sources of knowledge claims competing for public mind share. Now, virtually anyone can make knowledge claims and social media can facilitate attention and dissemination of those claims. Science’s ability to shape public understanding via empirical evidence rests on it unambiguously living up to its core values -- transparency, disinterestedness, universalism, and self-skepticism (Merton, 1942). Science will not maintain its public credibility (and funding support) with a mantra of “trust me”; it must instead respond to “show me.” Accelerated Discovery. The primary benefit of improving transparency, rigor, and reproducibility is accelerating accumulation of knowledge. Our broadest impact is more efficient use of research dollars, more reliable and reusable research output, and faster translation from insight to application. Results from Prior NSF Support NSF Award #1540440, $399,984, 8/15/15-9/30/19 Title: Institutional Re-Engineering of Ethical Discourse in STEM (iREDS) ● PI: Kevin Esterling, University of California, Riverside; co-PI Brian Nosek, Center for Open Science (with other co-PIs) ● Summary of Results: The project is a randomized control trial (RCT) study of the efficacy of a new, project-based ethics training curriculum developed at the University of California, Riverside (UCR). ● For Intellectual merit, the project is a collaboration with the Graduate Division at UCR. The ethics training curriculum is fully integrated with the Open Science Framework (OSF), a free, open, online, cloud-based platform developed by the Center for Open Science (COS) to facilitate within-team communication. The study implemented the design for the RCT that randomly assigns PIs/labs in the UCR College of Natural and Agricultural Sciences (CNAS), Bourns College of Engineering (BCOE), and School of Medicine (SOM) to either a treatment arm, in which the lab receives training on the OSF and the ethics of data management and authorship, or to a control arm. Outcomes are measured by a specially developed survey on ethics practices and attitudes that is centered on developing scales for assessing the quality of ethics discourse within labs. The project enrolled a total of 34 labs with 113 study participants. The data collection is complete and the team is currently analyzing the data for reporting. Broader impacts include an evaluation of best institutional practices to enhance ethical and reproducible science and creation of a culture of ethical STEM which will serve to increase the credibility and trust of science among the public and policy makers. ● List of Publications from Award: To date, no publications have been produced under this award. ● Evidence of Research Products: Rectangular data set of 113 pre and post responses to the survey in the RCT, as well as qualitative ethnographic data collected from eight labs. The data are housed on the OSF and are currently being analyzed.

20 References Cited Center for Open Science (COS) NSF - 19-537

Anderson MS, Martinson BC, & De Vries R (2007) Normative Dissonance in Science: Results from a National Survey of U.S. Scientists. Journal of Empirical Research on Human Research Ethics 2(4):3–14. ​ ​

Begley, C.G., & Ellis, L.M. (2012). Drug development: Raise standards for preclinical cancer research. Nature, 483, 531–533. ​

Button, K.S., Ioannidis, J.P.A., Mokrysz, C., Nosek, B.A., Flint, J., Robinson, E.S.J., & Munafò, M.R. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews ​ Neuroscience, 14, 365-376. doi: 10.1038/nrn3475 ​

Camerer, C. F., Dreber, A., Forsell, E., Ho, T-H., Huber, J., Johannesson, M., ... & Wu, H. (2016). Evaluating replicability of laboratory experiments in economics. Science, 351, 1433-1436. Doi: ​ ​ 10.1126/science.aaf0918

Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., … Wu, H. (2018). Evaluating Replicability of Social Science Experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour. Doi: 10.1038/s41562-018-0399-z ​ ​

Cohen, J. (1962). The statistical power of abnormal-social psychological research: A review. Journal of ​ Abnormal and Social Psychology, 65, 145-153. ​

Cova, F., Strickland, B., Abatista, A. G. F., Allard, A., Andow, J., Attie, M., … Zhou, X. (2018, April 9). Estimating the Reproducibility of Experimental Philosophy. https://doi.org/10.31234/osf.io/sxdah

Ebersole, C. R., Atherton, O. E., Belanger, A. L., Skulborstad, H. M., Allen, J. M., Banks, J. B., ... Nosek, B. A. (2016). Many Labs 3: Evaluating participant pool quality across the academic semester via replication. Journal of Experimental Social Psychology, 67, 68-82. ​ ​

Fanelli, D. (2012). Negative results are disappearing from most disciplines and countries. Scientometrics, ​ ​ 90, 891-904. https://doi.org/10.1007/s11192-011-0494-7 ​

Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, ​ ​ 82, 1–20.

Holdren, J. (2013) Increasing Access to the Results of Federally Funded Scientific Research. Retrieved from:https://obamawhitehouse.archives.gov/sites/default/files/microsites/ostp/ostp_public_access_memo_ 2013.pdf

John, L., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth-telling. Psychological Science, 23, 524-532. ​ ​ ​ ​ doi:10.1177/0956797611430953

Kidwell, M. C., Lazarevic, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L-S., Kennett, C., ... & Nosek, B. A. (2016). Badges to acknowledge open practices: A simple, low-cost, effective method for increasing transparency. PLOS Biology, 14, e1002456. Doi: 10.1371/journal.pbio.1002456 ​ ​

Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Jr., Bahník, S., Bernstein, M. J., … Nosek, B. A. (2014). Investigating variation in replicability: A "many labs" replication project. Social Psychology, 45(3), ​ ​ 142-152. doi: 10.1027/1864-9335/a000178

1

Klein, R.A., Vianello, M., Hasselman, F.,.. Nosek, B.A. (2018). Many Labs 2: Investigating variation in replicability across sample and setting. Advances in Methods and Practices in Psychological Science. ​ ​

McNutt, M., Lehnert, K., Hanson, B., Nosek, B. A., Ellison, A. M., & King, J. L. (2016). Beyond “data/samples available upon request” in the field sciences. Science, 351, 1024-1026. ​ ​ doi:10.1126/science.aad7048

Merton, R. K. (1942). Science and technology in a democratic order. Journal of Legal and Political ​ Sociology, 1, 115–126. ​

Merton, R. K. (1973). The Sociology of Science, Theoretical and Empirical Investigations. The University of Chicago Press, Chicago.

Miguel, E., Camerer, C., Casey, K., Cohen, J., Esterling, K. M., .... Van der Laan, M. (2014). Promoting transparency in social science research. Science, 343, 30-31. doi: 10.1126/science.1245317 ​ ​

National Academies of Sciences, Engineering, and Medicine. (2018). Open Science by Design: Realizing a Vision for 21st Century Research. The National Academies Press. https://doi.org/10.17226/25116.

Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615-631. ​ ​

Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., … Yarkoni, T. (2015). Promoting an open research culture. Science, 348, 1422-1425. doi: 10.1126/science.aab2374 ​ ​

Nosek, B. A., Ebersole, C. R., DeHaven, A., Mellor, D. (2018). The Preregistration Revolution. Proceedings for the National Academy of Sciences. doi: 10.1073/pnas.1708274114

Nosek, B.A., & Lindsay, D.S. (2018). Registration becoming the norm in psychological science. APS ​ Observer. Retrieved from: ​ https://www.psychologicalscience.org/observer/preregistration-becoming-the-norm-in-psychological-scien ce

Open Research Funders Group. (2018). Funder Perspectives on Open Infrastructure. Retrieved from https://static1.squarespace.com/static/5817749f8419c25c3b5b318d/t/5aef5dc2aa4a9960d83d9bb4/15256 36554938/ORFG+Infratructure+Survey+Results.pdf

Open Science Collaboration. (2015). Estimating the Reproducibility of Psychological Science. Science, ​ ​ 349(6251), aac4716. doi: 10.1126/science.aac4716

Paluck, E. L. (2018). Open science practices are on the rise across four social science disciplines. BITSS Annual Meeting. Retrieved from: https://www.bitss.org/events/2018am/

Prinz, F., Schlange, T., & Asadullah, K. (2011). Believe it or not: how much can we rely on published data on potential drug targets? Nature Reviews. Drug Discovery, 10, 712. ​ ​

Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86, ​ ​ 638-641. doi: 10.1037/0033-2909.86.3.638

Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., ... & Nosek, B. A. (2018). Many analysts, one dataset: Making transparent how variations in analytical choices affect results. Advances in ​ Methods and Practices in Psychological Science. doi: 10.1177/2515245917747646 ​

2 Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, ​ ​ 1359-1366.

Sterling, T. D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa. Journal of the American Statistical Association, 54, 30-34. ​ ​

Veldkamp, C. L. S., Bakker, M., van Assen, M. A. L. M., Crompvoets, E. A. V., Ong, H. H., Soderberg, C. K., Mellor, D., Nosek, B. A., & Wicherts, J. M. (2018) Ensuring the quality and specificity of preregistrations. Retrieved from: https://psyarxiv.com/cdgyh/

Wagenmakers, E-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632-638. ​ ​ doi:10.1177/1745691612463078

3