Data Replication with Code Ocean – a How-To Guide for PA Authors

Data Replication with Code Ocean – A How-To Guide for PA Authors

Simon Heuberger

October 2, 2019

Contact • Code Ocean support: Online browser chat or [email protected] • PA replication team: [email protected]

Workflow for Authors • Create a capsule for your material on the Code Ocean website. If you don’t have a free account yet, you need to sign up for one • Add the member of the PA replication team as a collaborator (use the PA contact email provided above) • Add your material. If you have any questions or problems during the setup, contact the PA replication team with questions. If needed, refer to the Code Ocean help documentation and/or contact the Code Ocean support team (use the email provided above) as well • When you think your material is ready to work in the capsule, email the PA replication team. PA then checks: – The Readme – That all figures and tables are saved to the /results folder and named correctly – Whether all libraries and dependencies are installed via the environment screen, not during runtime • After a successful check, replicate your material by clicking on “Reproducible Run”. This will run your material in the capsule remotely, i.e. you can close your browser and files will continue to be run. If there are any issues/errors during the run and you need help fixing them, email the PA replication team/Code Ocean support • After a successful reproducible run, fill out the publication metadata section of the capsule, and submit your material for publication by clicking on “Submit for publication”. Successful submission requires one complete run of the material without errors • PA gets a notification that the material has been submitted, compares the produced output to the manuscript, and checks the metadata • If aspects are missing or there are discrepancies between the code output and the manuscript, make the required changes to the material and resubmit. This process is repeated until the material is ready for publication • Upon successful comparison/check, Code Ocean publishes your material as a reproducible capsule and publishes a copy to Dataverse for long-term preservation

1 Languages offered by Code Ocean • Code Ocean offers the three data analysis software programs that are most often used in political science: R, Stata, and Python • Code Ocean also offers selected other pieces of more advanced programming software, such as Jupyter (Programming languages on Code Ocean). However, PA’s focus lies on the three standard programs above. If your material uses a language other than R, Stata, or Python, please contact PA before setting up a capsule

How to set up a Code Ocean capsule and install packages in R, Stata, and Python • Create a new blank capsule on your dashboard (What is a capsule?; What is the dashboard?) • Select your base environment (What is a base environment?; Selecting a base environ- mment) – For R, Code Ocean offers versions 3.4.4, 3.5.3, and 3.6 – For Stata, Code Ocean offers Stata 15 with R and Stata 15 with Matlab – For Python, Code Ocean offers versions 3.7.3, 3.7.0, 3.6.3, and TensorFlow 2.7 • Install needed packages on the next screen. It is not possible to install packages inside a script – they need to be installed here. Click on + Add, type in the package name and version (the latter is optional), and click on the tick (Adding packages on Code Ocean) – For R, use CRAN and GitHub – For Stata, use ssc (Using Stata on Code Ocean) – For Python, use conda and pip

How to use two data analysis software programs together • R and Stata – CodeOcean has a pre-provided environment (see the subsection on Stata base environments above) • R and Python – With R as the basis ∗ Add python3-pip (Python 3) or python-pip (for Python 2) via apt-get (What is apt-get?) – With Python as the basis ∗ Add R as a conda package, then add the package you want (e.g. r-ggplot2) ∗ The conda list of R packages is quite limited: Some packages are in a diﬀerent conda channel (e.g. randomizr which is in channel conda-forge, not r), others are not in conda at all (e.g. blockTools) – General rule: If a material is mainly in R, use R as the base. Likewise for Python

Folders in a capsule • The key folders on Code Ocean are /code, /data, and /results • /code – Contains all your script ﬁles • /data – Contains all your data ﬁles

2 – To ensure complete reproducibility, /data is reset to its original form after each run. This means everything your code saves in /data would be lost at the end of a run. To avoid this, save all code output in /results instead (see subsection /results below) • /results – Everything that your code saves needs to be saved in /results (Saving files on Code Ocean). This includes figures, tables, and any created .csv, .Rdata etc. files that other script files read in after creation. For instance, if a .csv file is created in first.R and then needs to be read in by second.R, save and load the .csv to/from /results. Everything saved in /results is available after each run run • run is a shell run script that is central to the working of Code Ocean. In order to run any script file in any programming language, it needs to be listed in run (What is a run script?) • run is not present when you create a new blank capsule. To create it, select an uploaded script file and select “Set as File to Run” with right-point-and-click. run will appear in the Files tab and list the selected script • It is currently not possible to select several script files and select “Set as File to Run”. If your material includes more than one script file to run (which is very likely), create a master script file that reads in the others (then only this master file needs to be listed in run). Alternatively, select “Set as File to Run” for the first file and manually enter the other files into run, e.g. (for all three languages): Rscript "first.R" Rscript "second.R" python -u "first.py" python -u "second.py" stata "first.do" stata "second.do" • If you want a full log of any run script files (i.e. including all executed commands) enter script files in the following way into run: Rscript -e "source(‘first.R’, echo = T)" python -m trace --trace "first.py" For Stata, set up a log file within the .do file

Your material in the capsule • Creating subfolders – Hover the cursor over any folder. A downward arrow will appear on the right. Click it and select “New Folder” to create a new subfolder in this folder • Uploading files/folders – Hover the cursor over any folder. A downward arrow will appear on the right. Click it and select “Upload File(s)” to upload files or folders to this folder. If uploading folders doesn’t work, switch to Chrome as your browser (some browsers like Safari don’t work with this feature) (Uploading files/folders to Code Ocean) – Upload your script files to /code and your data files to /data

3 • Readme – Provide a Readme that briefly describes your material and the related manuscript – The Readme can be uploaded to /code in any format • Saving code output – Your code needs to replicate and create saved output for all code-created figures and tables in the main text as well as in the supplementary material – Everything that your code saves needs to be saved in /results (see section “Fold- ers in a capsule” above) – If you want to save any figures, tables, or data files in subfolders within /results, these subfolders have to be set up in run. For instance, to set up the subfolder /sims in /results, enter the following into run: mkdir -p /results/sims – Figures need to be saved in .pdf format. Tables can be saved in the format you deem most appropriate (e.g. .csv, .txt, .tex etc.) – All figures and tables need to be saved according to their respective numbers in the manuscript, e.g. figure1.pdf, table2.tex etc. • Reading in files – Files that are present when you set up the capsule need to be read in from /data – Files that are created by a script and needed in subsequent scripts need to be saved to and read in from /results (see section “Folders in a capsule” above) • File paths – Always use relative paths, e.g. ../data and ../results, when reading in and saving files (File paths on Code Ocean) – If you have code in a subfolder, don’t forget to add .. as necessary, e.g. load(‘../../data/simulations.csv’). This code would go up two parent di- rectories and then look for ./data/simulations.csv – The working directory is /code. If you source any script files, you thus don’t need relative paths, e.g. source("calculations.R") will work • Running files – After a successful PA check of your setup (see section “Workflow for Authors” above), click on “Commit Changes”, then on “Reproducible Run” – This will run all script files listed in run remotely – You can close your browser once reproduction has started. Files will continue to be run (Running files on Code Ocean) – If there is an error in your code, execution will be halted. Execution will also be halted if your code requires interactive user input • Computing power – Your material will be run on at least a 16-core machine with 120 GB of memory. If your material requires more computing power please contact PA (Computing power on Code Ocean) • Looking at resulting plots – Click on any saved plot and it will open in a new tab. If you see an empty tab, switch to Chrome as your browser (some browsers like Safari don’t work with this feature) • Submitting your material for publication

4 – After a successful reproducible run, click on “Submit for publication” – Successful submission requires one complete run of your material without errors (Publishing on Code Ocean) – If you can’t see the “Submit for publication” button, make sure to commit all changes (Where is the ‘Submit for Publication’ button?) – After submitting your material for publication, add the metadata (see section “Publication metadata” below). PA will then check your material. Upon successful check, Code Ocean makes your material publicly available (What’s involved with a published capsule?)

Publication metadata (must be completed for publication) • Description • Basic info – Capsule name – Research ﬁeld (“Social Sciences”) – Tags (optional) • License info – Software license (“MIT” is recommended) – Data license (“No Rights Reserved” is recommended) • Associated Publication – Select whether the publication associated with this capsule is: ∗ Published ∗ Yet to be published ∗ There is no associated article – If you have a DOI, select “Published”. Otherwise choose “Yet to be published” • Authors and aﬃliations