Towards Open Science in Acoustics: Foundations and Best Practices
Total Page:16
File Type:pdf, Size:1020Kb
DAGA 2017 Kiel Towards Open Science in Acoustics: Foundations and Best Practices Sascha Spors1, Matthias Geier1 and Hagen Wierstorf2 1Institute of Communications Engineering, University of Rostock, Germany, Email: [email protected] 2Filmuniversit¨atBabelsberg KONRAD WOLF The Scientific Method myself Before introducing the open science approach in detail it my future self is worthwhile to review the foundations of the scientific my colleagues method. It may be referred to as `A method of procedure that has characterized natural science since the 17th cen- other researchers tury, consisting in systematic observation, measurement, all people in the world and experiment, and the formulation, testing, and modi- fication of hypotheses.' [1]. Science builds upon a set of science itself well accepted methods which are commonly believed to While research conducted for myself does not require any ensure above formulated principles. Besides these, the transparency in terms of underlying data, research per- reproducibility of results is one of the cornerstones of the formed for the sake of science itself should be as open as scientific method. It may be defined as `Reproducibil- possible to support scientific advancement. ity is the ability of an entire analysis of an experiment or study to be duplicated, either by the same researcher or by someone else working independently, whereas reproducing Open Science an experiment is called replicating it.' [2]. The irrepro- Open science follows the general demand for socializa- ducibility of a wide range of scientific results has drawn tion of knowledge. More specifically, it aims at making significant attention in the last decade [3, 4, 5, 6, 7, 8, 9]. the research process transparent on all levels. The advent In order to track down the problem of irreproducibility, of scientific journals can be seen as a first step towards the application areas of the scientific method have been openness. Open science has been discussed on a broad classified into three branches by [10, 11]: level in the last decade. However the exact meaning of the term is still not fully settled. Many measures are 1. deductive discussed to achieve transparency in the research pro- 2. empirical cess [12, 13, 14, 15, 16]. We review the most common elements of open science in the following, followed by 3. computational application examples in the context of a listening exper- iment. The deductive branch covers results derived e.g. by for- mal logic and mathematics, the empirical branch e.g. sta- Elements of Open Science tistical analysis of controlled experiments. The first two The various measures used to gain open and transparent are traditional branches, while the last one is considered science may be classified as follows: as a potentially novel branch. It covers results derived by large-scale simulations and data-driven computational • Open Source science. The measures that have to be taken to ensure Availability of source code of the implementations reproducibilitly in the traditional branches are quite well used in the research process, e.g. for numerical sim- known. This does not hold for the third, computational, ulations, statistical analyses and graphical user in- branch. Here the main intellectual contributions may be terfaces. encoded in software only. • Open (Science) Data Besides problems in the research methods themselves, re- Availability of the raw/processed data underlying sults are often not reproducible since necessary supple- the research, whereas data does not only refer to mentary material as protocols, data and implementations electronic resources but also to protocols, material are not available. In many cases only the published re- samples, etc. For instance impulse responses, ane- sults are accessible to other researchers. Open science choic stimuli and anthropomorphic data. focuses on the ease of access to scientific data underlying • Open Access research and therefore supports the ease of reproducibil- Free access to published research output, e.g. arti- ity. cles. An alternative view on the problem of reproducibility • Open Methodology is given by asking who should benefit from my re- Detailed documentation of the methodology under- search? Potential answers to this question ordered by lying the research. For instance hypothesis and ex- the degree of transparency are: perimental procedure. 218 DAGA 2017 Kiel • Open Notebook Science 6. Manuscript Availability of the primary record of research results, The design and the results of the perceptual study e.g. laboratory notebooks or detailed mathematical are compiled into a manuscript. It is composed derivations. from text, references and visualizations of the an- alyzed results. The manuscript may be published as • Open Educational Resources Open Access publication. Freely accessible resources for teaching, learning and research. For instance textbooks, slides and exam- 7. Pre-Publication Peer Review ples. The publication process typically involves some kind of quality assessment. The manuscript is evalu- • Open Peer Review [17] ated by independent reviewers who provide ratings Various measures to increase the transparency of the and suggestions for improvement. If accepted the peer review process, e.g. availability of review com- manuscript is then revised. The peer review process ments and crowdsourced review. can be made transparent using Open Peer Review. Example { Listening Experiment 8. Publication The workflow of a typical perceptual experiment serves After successful incorporation of comments from the as an example for the usage of the different elements of pre-publication peer review, the manuscript is ready open science. A perceptual study may be decomposed for publication. It may be accompanied by supple- into the following stages: mentary material in order to improve the ease of re- producibility. In the case of conference papers a pre- 1. Idea sentation may be given which contains other or addi- The results of a preceding experiment or a discus- tional material. The elements involved for publica- sion among colleagues is often the advent for new tion are Open Access, Open Source and Open Data. research projects. These ideas reveal for instance the motivation behind the research. Publishing the 9. Aftermath ideas in the spirit of Open Notebook Science makes After publication, the presented research may be them accessible to the public. replicated by others. Post-publication review will consider the correctness and relevance of the re- 2. Design of Experiment search. The feedback may require to publish errata The experiment is designed on basis of the initial or revise the underlying data and code. Finally open idea. This involves for instance the formulation of a questions may bring up ideas for the next research hypothesis, the definition of an experimental pro- project. cedure, stimuli and subjects. Open Methodology makes the design available. Copyright and Licenses 3. Computation Licenses are an important aspect of open science. Sci- The generation of stimuli may involve mathemati- entific code and data that is made available to the pub- cal deviations for numerical simulations, the imple- lic should be accompanied by a clear license statement. mentation of signal processing, as well as data like Otherwise, due to copyright laws, re-distribution and re- impulse responses and anechoic recordings. For the use is most likely forbidden. The exact legal implications experiment itself often a control logic and graph- are rather complex, hard to oversee and depend on the ical user interface is implemented. These can be country whose copyright laws apply. made available following Open Notebook Science, Open Data and Open Source. In order to explicitly allow re-distribution and re-use { under well-defined conditions { there are a variety of li- 4. Experiment censing frameworks which can be used for open science. The experiment is conducted in the next step. Here For text, images, artwork and similar works, the family the responses are collected from the test subjects of creative commons (CC) licenses1 is a good choice [18], potentially together with other data like head orien- especially the rather permissive variant CC BY and the tation. This constitutes raw data which is processed more restrictive variant CC BY SA. When publishing later in the analysis stage. The data can be pub- source code, one of the many available open source li- lished as Open Data as long as no privacy issues are censes should be used. Examples are the permissive BSD raised. or MIT licenses and the more restrictive GNU General 5. Analysis Public License (GPL), just to name a few. The statistical analysis of the raw data is the ba- To promote wide-spread legal re-use of research results, sis of the generalization of the individual results. a permissive license should be preferred. There are even This may involve the anonymization and outlier re- tools which allow authors to waive all their rights, as moval as first steps. The analysis procedure and far as legally possible, most notably CC0. In order to the processed data can be made transparent using aid in the selection of an appropriate licensing strategy, the elements Open Methodology, Open Source and Open Data. 1https://creativecommons.org 219 DAGA 2017 Kiel recommendations for open science have been given for Personal Experience instance in [19]. Besides using open toolboxes and datasets, our own ex- perience with open science began with the open source Management of Research Data release of the SoundScape Renderer