Research Transparency and Reproducibility Training (RT2) Participant Manual Los Angeles, CA September 5-7, 2018 Contents About RT2 ........................................................................................................................................................................................................... 2 Pre-Training Actions ................................................................................................................................................................................ 3 1. Sign up for an OSF account ............................................................................................................................................... 3 2. Review study pre-registration ......................................................................................................................................... 3 3. Install Git and create Github.com account ........................................................................................................... 3 4. Install software for Dynamic Documents – for R or Stata ....................................................................... 4 5. Install Anaconda - for Jupyter Notebooks ............................................................................................................. 4 6. Install LaTeX .................................................................................................................................................................................... 5 7. Install a decent text editor .................................................................................................................................................. 5 Suggested Reading List ........................................................................................................................................................................ 6 Foundational literature ..................................................................................................................................................................... 6 P-curve ............................................................................................................................................................................................................. 6 Researcher Degrees of Freedom ................................................................................................................................................ 7 Power and Priors ...................................................................................................................................................................................... 7 Reproducibility.......................................................................................................................................................................................... 7 Pre-Registration and Pre-Analysis Plans ............................................................................................................................ 7 Replication ................................................................................................................................................................................................... 8 Data De-Identification ........................................................................................................................................................................ 8 Meta-analysis ............................................................................................................................................................................................ 8 Transparency Reporting and Disclosure ............................................................................................................................ 8 Other Useful BITSS Resources ....................................................................................................................................................... 9 Glossary .............................................................................................................................................................................................................10 Meet the RT2 Faculty! ........................................................................................................................................................................... 13 Meet BITSS! ..................................................................................................................................................................................................... 16 Agenda ................................................................................................................................................................................................................ 17 Appendix A: OSF Pre-Registration ............................................................................................................................................ 19 1 About RT2 Welcome to the BITSS community! We are pleased to host you for our three-day Research Transparency and Reproducibility Training (RT2), in Los Angeles, CA. RT2 provides participants with an overview of tools and best practices for transparent and reproducible social science research. We focus on topics such as: ● Pre-registration. The registration of study designs in public repositories prior to data collection allows for better tracking of the universe of studies in a given domain, including studies with null results that are rarely published. This begins to tackle the “file-drawer problem“ whereby only statistically significant findings are reported; ● Pre-analysis plans. The design and use of a pre-analysis plan (PAP)—a step-by-step plan, written before data are analyzed, describing hypotheses and strategies for analyzing data—can help protect against data mining and restrict researcher “degrees of freedom“ in confirmatory research; ● Meta-analysis. Innovations in the design of meta-analyses—dealing with issues of bias, study sample size, and model selection—can improve the quality of inferences made from the analyses of pooled studies; ● Data de-identification. To facilitate review and replicability, researchers can publicly post the data and code needed to replicate findings of published studies. However, this requires an understanding of how to balance maximizing data’s usability with protection of human subjects and data confidentiality by using methods for data de- identification; ● Tools for transparent workflows. There is a plethora of software and online tools to facilitate transparent and reproducible workflows, such as the Open Science Framework (OSF), Git, R packages, dynamic documents, and Jupyter notebooks. BITSS aims for RT2 to have long-term, sustainable impacts on scientific norms and practices as learners and faculty like you continue to incorporate innovative tools and methods into curricula and coursework at your own institutions. To ensure you get the most out of this three-day event, we have prepared this Participant Manual with a list of definitions, a pre-event Reading List, instructions for preparing for the hands-on sessions, including required software downloads, and lists of your fellow participants and RT2 faculty. If you are interested in joining our network of trainers and consultants, please visit our website to learn more about the BITSS Catalyst Program. 2 Pre-Training Actions BITSS requires all participants currently working on their own research to take the following actions before September 5, 2018. Participants who manage, fund, or publish research may also take these actions, but can also be paired with other participants during hands-on sessions to observe the process instead. Please reference the Software section of our Resources page on the BITSS website for more information on this software. 1. Sign up for an OSF account The Open Science Framework (OSF) allows you to store your research files and link together all your research across several platforms, such as Dropbox, Harvard’s Dataverse, and GitHub. It version-controls any files you upload, and you can register a project to create a frozen time-stamped version with a persistent URL. By writing a pre-analysis plan, you could prove to the world that your significant results are not just a successful fishing expedition. Sign up for a free account here. 2. Review study pre-registration In the Improved Specification hands-on session, participants will be able to walk through how to develop a pre-analysis plan and register a study on the OSF. Think of one of your recent studies, or a study that you would like to conduct, and see Appendix 1 below and this Protocol about how to prepare your pre-registration. 3. Install Git and create Github.com account The date-and-initial version of keeping track of changes to your files is not very helpful when working on complex projects or projects involving multiple co-authors. If you want your work to be reproducible, use version control. It has a learning curve even for xkcd-type people, but it’s worth it! (Read Gentzkow and Shapiro, chapter 3 on why.) Software Carpentry and GitHub have great tutorials. To get started, download the GitHub Desktop GUI app. Note that this is only available for Windows and Mac users. Linux users can use the command line or pick one of the other GUIs listed here. If you are comfortable using the command line, we also recommend Windows users install Git Bash. Next, create an account with GitHub.com. GitHub