Reproducible Research with R, LATEX, sweave, and knitr Reproducible Research with R, LATEX, Background sweave, and knitr Scientific Methods Quality Pre- Specification Frank E Harrell Jr Summary Terri Scott Software Department of Biostatistics Sweave Approach Vanderbilt University School of Medicine and Vanderbilt Institute for Clinical & Translational Research Enhancing Sweave Output useR! 2012 Nashville 12 June 2012 Enhanced sweave Report knitr References Outline Reproducible Research with R, LATEX, 1 Background sweave, and knitr 2 Scientific Methods Quality Background 3 Pre-Specification Scientific Methods Quality 4 Summary Pre- Specification 5 Software Summary Software 6 Sweave Approach Sweave Approach 7 Enhancing Sweave Output Enhancing Sweave 8 Example Enhanced Report Handout Output Enhanced sweave Report 9 knitr knitr References Non-reproducible Research Reproducible Research with R, LATEX, sweave, and knitr Misunderstanding statistics Background \Investigator" moving the target Scientific Methods Quality Lack of a blinded analytic plan Pre- Tweaking instrumentation / removing \outliers" Specification Summary Pre-statistician \normalization" of data and background Software subtraction Sweave Poorly studied high-dimensional feature selection Approach Enhancing Sweave Output Enhanced sweave Report knitr canstockphoto.com References Non-reproducible Research, continued Reproducible Research with R, LATEX, sweave, and knitr Programming errors Background Lack of documentation Scientific Methods Failing to script multiple-step procedures Quality Pre- using spreadsheets and other interactive approaches for Specification data manipulation Summary Copying and pasting results into manuscripts Software Sweave Insufficient detail in scientific articles Approach No audit trail Enhancing Sweave Output Enhanced sweave Report knitr References General Importance of Sound Methodology Reproducible Research with Hackam and Redelmeier [2006]: Translation of research R, LATEX, sweave, and evidence from animals to humans knitr Screened articles having preventive or therapeutic Background intervention in in vivo animal model, > 500 citations Scientific Methods 76 \positive" studies identified Quality Median 14 years for potential translation Pre- Specification 37 judged to have good methodological quality (flat over Summary time) Software 28 of 76 replicated in human randomized trials; 34 remain Sweave Approach untested Enhancing Sweave " 10% methodology score " odds of replication × 1.28 Output (0.95 CL 0.97{1.69) Enhanced sweave Report Dose-response demonstrations: " odds × 3.3 (1.1{10.1) knitr Note: The article misinterpreted P-values References Reproducible Research with R, LATEX, sweave, and knitr Background Scientific Methods Quality Pre- Specification Summary Software Sweave Approach Enhancing Sweave Output Enhanced sweave Report knitr References Reproducible Research with R, LATEX, sweave, and knitr Background Scientific Methods Quality Pre- Specification Summary Software Sweave Approach Enhancing Sweave Output Enhanced sweave Report knitr New Yorker Dec 13, 2010 References The Truth Wears Off Reproducible Research with R, LATEX, sweave, and knitr Background Prescribe drugs while they still work Scientific Methods Rhine and ESP: \the student's extra-sensory perception Quality ability has gone through a marked decline" Pre- Specification Regression to the mean Summary Floating definitions of X or Y : association between Software physical symmetry and mating behavior; acupuncture Sweave Approach Enhancing Sweave Output Enhanced sweave Report knitr References The Truth Wears Off, continued Reproducible Research with R, LATEX, sweave, and knitr Selective reporting and publication bias Journals seek confirming rather than conflicting data Background Damage caused by hypothesis tests and cutoffs Scientific Methods 1 Quality Ioannidis: 3 of articles in Nature never get cited, let alone Pre- replicated Specification Biologic and lab variability Summary Software Weak coupling ratio exhibited by decaying neutrons fell by Sweave 10 SDs from 1969{2001 Approach Enhancing Sweave Output \The decline effect is actually a decline of illusion" Enhanced sweave Report knitr References Reproducible Research with R, LATEX, sweave, and knitr Background Scientific Methods Quality Pre- Specification Summary Software Sweave Approach Enhancing Sweave Output Enhanced sweave Report knitr References Biomarker Discoveries Reproducible Research with R, LATEX, sweave, and knitr Background Scientific Izvestia Pravda Methods Quality (News) (Truth) Pre- Specification Summary Big Effects Validated Effects Software Sweave Approach Enhancing Sweave Output Enhanced sweave Report knitr References Strong Inference Reproducible Research with R, LATEX, sweave, and knitr Background Scientific Methods Quality Pre- Specification Summary Software Sweave Approach Enhancing Sweave Output Enhanced sweave Report knitr References Strong (Inductive) Inference, continued Reproducible Research with R, LATEX, sweave, and knitr Devise alternative hypotheses Devise an experiment with alternative possible outcomes Background each of which will exclude a hypothesis Scientific Methods Carry out the experiment Quality Pre- Repeat Specification Summary Regular, explicit use of alternative hypotheses & sharp Software exclusions ! rapid & powerful progress Sweave \Our conclusions . might be invalid if . (i) . (ii) Approach . (iii) . We shall describe experiments which eliminate Enhancing Sweave these alternatives." Output Enhanced sweave Report knitr Platt [1964] References Science Reproducible Research with R, LATEX, A theory which cannot be mortally endangered sweave, and knitr cannot be alive. W. A. H. Rushton Background Religion is a culture of faith; science is a culture of Scientific Methods doubt. Quality Science is the belief in the ignorance of experts. Pre- Specification Richard Feynman Summary Fiction is about the suspension of disbelief; science is Software about the suspension of belief. Sweave Approach James Porter Enhancing Sweave A true scientist is bored by knowledge; it is the Output assault on ignorance that motivates him. Enhanced sweave Report Matt Ridley knitr References System Malfunctions Reproducible Research with Journal publicity, R, LATEX, imact factor Desire to sweave, and Inst. knitr fame, wealth, help patients prestige Background Investigator fame, Scientific Promotion prestige, tenure Methods committees Quality clueless Pre- Specification Quality Research Faculty Scientific Advance Summary freedom Software Journal referees, editors clueless Sweave Approach Statisticians Enhancing funded Sweave Output Passive research after NOGA integrity Enhanced Statisticians sweave Report Inadequate seen as knitr mentoring speedbumps References System Cost of Investigating Research Malpractice Reproducible Research with R, LATEX, sweave, and knitr Background Scientific Methods Quality Pre- Specification Summary Software Sweave Approach Enhancing Sweave Output Enhanced sweave Report knitr References Pre-Specified Analytic Plans Reproducible Research with R, LATEX, sweave, and knitr Long the norm in multi-center RCTs Needs to be so in all fields of research using data to draw Background Scientific inferences (Rubin [2007]) Methods Quality Front-load planning with investigator Pre- too many temptations later once see results (e.g., Specification P = 0:0501) Summary SAP is signed, dated, filed Software Sweave Pre-specification of reasons for exceptions, with exceptions Approach documented (when, why, what) Enhancing Sweave Becoming a policy in VU Biostatistics Output Enhanced sweave Report knitr References What Do Methodologists Offer? Reproducible Research with R, LATEX, sweave, and Biostatisticians and clinical epidemiologists play important roles knitr in Background assessing the needed information content for a given Scientific problem complexity Methods Quality minimizing bias Pre- Specification maximizing reproducibility Summary Software For more information see: Sweave Approach ctspedia.org Enhancing reproducibleresearch.net Sweave Output groups.google.com/group/reproducible-research Enhanced sweave Report knitr References Some Random Thoughts Reproducible Research with R, LATEX, sweave, and Kelvin's curse: The unthinking and inappropriate knitr worship of quantifiable information in medicine Background Feinstein [1977] Scientific Methods . monetization of intellectual property appears to be Quality a powerful force favoring methodological limitations Pre- Specification and an excessive reductionism and fragmentation of Summary biologic knowledge Software Porta et al. [2007] Sweave Approach There is nothing wrong with cancer research that a Enhancing Sweave little less money wouldn't cure. Output Enhanced Nathan Mantel, NCI sweave Report knitr References Goals of Reproducible Analysis/Reporting Reproducible Research with Be able to reproduce your own results R, LATEX, sweave, and knitr Allow others to reproduce your results Time turns each one of us into another person, Background and by making effort to communicate with Scientific Methods strangers, we help ourselves to communicate Quality with our future selves. (Schwab and Claerbout) Pre- Specification Reproduce an entire report, manuscript, dissertation, book Summary with a single system command when changes occur in: Software operating system, stat software, graphics engines, source Sweave Approach data, derived variables, analysis, interpretation Enhancing Sweave Save time Output Provide the ultimate documentation of work done for a Enhanced sweave Report paper knitr http://biostat.mc.vanderbilt.edu/StatReport