Report on the First IEEE Workshop on the Future of Research Curation andNational Research Reproducibility Science Foundation 5-6 November 2016 Award #1641014 Report on the First IEEE Workshop on the Future of Research Curation and Research Reproducibility 5-6 November 2016 WASHINGTON, DC Report for NSF Award #1641014 Page 1 Report on the First IEEE Workshop on the Future of Research Curation and Research Reproducibility 5-6 November 2016 This page intentionally left blank Report for NSF Award #1641014 Page 2 Report on the First IEEE Workshop on the Future of Research Curation and Research Reproducibility 5-6 November 2016 Report on the First IEEE Workshop on The Future of Research Curation and Research Reproducibility Marriott Marquis, Washington, DC, USA 5-6 November 2016 National Science Foundation Award #1641014 Steering Committee Chair: John Baillieul, Boston University Larry Hall, University of South Florida José M.F. Moura, Carnegie Mellon Sheila Hemami, Draper Labs Gianluca Setti, University of Ferrara Michael Forster, IEEE Gerry Grenier, IEEE Fran Zappulla, IEEE John Keaton, IEEE Douglas McCormick and Kenneth Moore, rapporteurs Report for NSF Award #1641014 Page 3 Report on the First IEEE Workshop on the Future of Research Curation and Research Reproducibility 5-6 November 2016 Contents Attendees ...................................................................................................................................................... 6 Preface .......................................................................................................................................................... 7 Executive Summary ....................................................................................................................................... 8 Introduction ................................................................................................................................................ 11 Introductory Presentations ......................................................................................................................... 20 Public Access, Data, and the Research Enterprise ......................................................................... 20 Overview of the Reproducibility Landscape .................................................................................. 22 Plenary Panel: New Forms of Content ........................................................................................................ 26 Reproducible Research .................................................................................................................. 26 Why Is Curation Important? .......................................................................................................... 28 Data-Intensive Research Architectures ......................................................................................... 29 Content, Context, and Curation ..................................................................................................... 32 Group Reports on New Forms of Content and Radically New Approaches to Content Creation .............. 35 Critical elements; curated updates; finding and indexing (Group A) ............................................ 35 Dealing with new content; tools for collaboration (Group B) ....................................................... 36 Essential products, practical actions (Group C) ............................................................................. 37 Curatorial challenges; community advocacy (Group D) ................................................................ 38 Plenary Panel: Peer Review and Quality Assurance ................................................................................... 41 Data, Software, and Reproducibility in Publication ....................................................................... 41 Software Quality Engineering: Paving the Path toward Research Reproducibility ....................... 46 Quality Assurance for Nontraditional Research Products: Is There Any? ..................................... 48 Group Reports on Peer Review and Quality Assurance .............................................................................. 51 New forms of peer validation? Are persistent links possible? (Group A) ...................................... 51 Software challenges; curation and quality engineering (Group B) ................................................ 52 Organizing and paying for quality assurance; models of peer review (Group C) .......................... 54 Different environments vs. a common platform; addressing proprietary code (Group D) ........... 56 Plenary Panel: The Economics of Reproducibility ....................................................................................... 58 Digital First: Publisher’s Value Proposition in the Age of Reproducible Science and Open Data .. 58 Breaking the “Iron Triangle”: Perspectives on Sustainability of Institutional Repositories .......... 61 The Economics of Reproducibility .................................................................................................. 63 Report for NSF Award #1641014 Page 4 Report on the First IEEE Workshop on the Future of Research Curation and Research Reproducibility 5-6 November 2016 The Road to Reproducibility: Research Data Management Initiatives and Researcher Behavior . 65 Group Reports on Economics of Reproducibility ........................................................................................ 67 Stakeholder roles; at what scale do we address challenges? (Group A) ....................................... 67 Policy efforts underway; how should we think about funding and resources? (Group B) ............ 68 Can we publish papers that solely cover research reproduction? Legal issues (Group C) ............ 70 How will reproducibility be funded; what are the biggest challenges for publishers? (Group D) 72 Takeaway Messages .................................................................................................................................... 75 Regarding Research Reproducibility and Open Science ................................................................ 75 Rapidly Evolving Concepts of Research Curation .......................................................................... 75 Sustainability: Creation, Peer Review, Curation ............................................................................ 78 Immediate Next Steps for Research Curation and Peer Review.................................................... 81 NSF and the Evolution of Concepts of Research and Curation ...................................................... 83 Closing Remarks and Next Steps ................................................................................................................. 84 Appendix ..................................................................................................................................................... 87 Agenda ........................................................................................................................................... 87 Background Reading ...................................................................................................................... 90 Endnotes ..................................................................................................................................................... 92 Report for NSF Award #1641014 Page 5 Report on the First IEEE Workshop on the Future of Research Curation and Research Reproducibility 5-6 November 2016 Attendees SIMON ADAR, Code Ocean JACK AMMERMAN, Boston University JOHN BAILLIEUL, Boston University BRUCE CHILDERS, University of Pittsburgh SAMIR EL-GAZALY, University of Arkansas MICHAEL FORSTER, IEEE AMY FRIEDLANDER, National Science Foundation GERRY GRENIER, IEEE SUNIL GUPTA, IEEE LARRY HALL, University of South Florida KAREN HAWKINS, IEEE SHEILA HEMAMI, Draper Labs LINA KARAM, Arizona State University JAMES KELLER, University of Missouri JOHN KEATON, IEEE LISA KEMPLER, MathWorks PATRICIA KNEZEK, National Science Foundation JELENA KOVAČEVIĆ, Carnegie Mellon University MIRIAM LESSER, Northeastern University CLIFFORD LYNCH, Coalition for Networked Information DOUGLAS MCCORMICK, Runestone Associates SHEILA MORRISSEY, Portico ELEONORA PRESANI, Elsevier BERNARD ROUS, ACM GIANLUCA SETTI, University of Ferrara GAURAV SHARMA, University of Rochester EEFKE SMIT, STM International DAVID SMITH, IET MATT SPITZER, Community of Science VICTORIA STODDEN, University of Illinois, Urbana-Champaign MIKE TAYLOR, Digital Science TODD TOLER, John Wiley & Sons JENNIFER TURGEON, Sandia National Labs DAN VALEN, Figshare SHAOWEN WANG, University of Illinois, Urbana-Champaign ERIC WHYNE, Data Machines Corp. FRAN ZAPPULLA, IEEE Report for NSF Award #1641014 Page 6 Report on the First IEEE Workshop on the Future of Research Curation and Research Reproducibility 5-6 November 2016 Preface The introduction to the report that follows identifies 1993 as the birth date of the World Wide Web. This was the beginning of the end of print-based publishing that had been the bedrock of archival communication for more than a thousand years (if we agree that printing first appeared with the publication of the Diamond Sutra in 868). Throughout most of the history of print-based publishing, small groups of people had the financial resources to print and distribute what was published. Those doing the writing in early years were also few in numbers — as were those who could