JYVÄSKYLÄ LICENTIATE THESES IN COMPUTING 18 Antti-Juhani Kaijanaho The Extent of Empirical Evidence that Could Inform Evidence-Based Design of Programming Languages A Systematic Mapping Study JYVÄSKYLÄ LICENTIATE THESES IN COMPUTING 18 Antti-Juhani Kaijanaho The Extent of Empirical Evidence that Could Inform Evidence-Based Design of Programming Languages A Systematic Mapping Study UNIVERSITY OF JYVÄSKYLÄ JYVÄSKYLÄ 2014 The Extent of Empirical Evidence that Could Inform Evidence-Based Design of Programming Languages A Systematic Mapping Study JYVÄSKYLÄ LICENTIATE THESES IN COMPUTING 18 Antti-Juhani Kaijanaho The Extent of Empirical Evidence that Could Inform Evidence-Based Design of Programming Languages A Systematic Mapping Study UNIVERSITY OF JYVÄSKYLÄ JYVÄSKYLÄ 2014 Editor Timo Männikkö Department of Mathematical Information Technology, University of Jyväskylä URN:ISBN:978-951-39-5791-9 ISBN 978-951-39-5791-9 (PDF) ISBN 978-951-39-5790-2 (nid.) ISSN 1795-9713 Copyright © 2014, by University of Jyväskylä Jyväskylä University Printing House, Jyväskylä 2014 GLENDOWER. I can call spirits from the vasty deep. HOTSPUR. Why, so can I, or so can any man; But will they come when you do call for them? — William Shakespeare’s Henry IV Part 1 (III.1) ABSTRACT Kaijanaho, Antti-Juhani The extent of empirical evidence that could inform evidence-based design of pro- gramming languages. A systematic mapping study. Jyväskylä: University of Jyväskylä, 2014, 243 p. (Jyväskylä Licentiate Theses in Computing, ISSN 1795-9713; 18) ISBN 978-951-39-5790-2 (nid.) ISBN 978-951-39-5791-9 (PDF) Finnish summary Background: Programming language design is not usually informed by empirical studies. In other fields similar problems have inspired an evidence-based paradigm of practice. Central to it are secondary studies summarizing and consolidat- ing the research literature. Aims: This systematic mapping study looks for em- pirical research that could inform evidence-based design of programming lan- guages. Method: Manual and keyword-based searches were performed, as was a single round of snowballing. There were 2056 potentially relevant publications, of which 180 were selected for inclusion, because they reported empirical ev- idence on the efficacy of potential design decisions and were published on or before 2012. A thematic synthesis was created. Results: Included studies span four decades, but activity has been sparse until the last five years or so. The form of conditional statements and loops, as well as the choice between static and dynamic typing have all been studied empirically for efficacy in at least five studies each. Error proneness, programming comprehension, and human effort are the most common forms of efficacy studied. Experimenting with programmer participants is the most popular method. Conclusions: There clearly are language design decisions for which empirical evidence regarding efficacy exists; they may be of some use to language designers, and several of them may be ripe for sys- tematic reviewing. There is concern that the lack of interest generated by studies in this topic area until the recent surge of activity may indicate serious issues in their research approach. Keywords: programming languages, programming language design, evidence- based paradigm, efficacy, research methods, systematic mapping study, thematic synthesis Author’s address Antti-Juhani Kaijanaho Department of Mathematical Information Technology University of Jyväskylä, Finland PO Box 35 (Agora), FI-40014 University of Jyväskylä [email protected] Supervisors Professor Tommi Kärkkäinen Department of Mathematical Information Technology University of Jyväskylä, Finland Doctor Vesa Lappalainen Department of Mathematical Information Technology University of Jyväskylä, Finland Doctor Ville Tirronen Department of Mathematical Information Technology University of Jyväskylä, Finland Examiners Doctor Stefan Hanenberg University of Duisburg-Essen, Germany Professor Stein Krogdahl University of Oslo, Norway ACKNOWLEDGEMENTS I would like to thank my supervisors, Professor Tommi Kärkkäinen, Doctor Vesa Lappalainen, and Doctor Ville Tirronen, for encouragement, discussions and feed- back over the years. Doctor Stefan Hanenberg of the University of Duisburg-Essen and Profes- sor Stein Krogdahl of the University of Oslo examined this thesis; their comments reflect fairly, in my opinion, both the strengths and the weaknesses of this thesis, and I expect to take their comments into account when I extend the research re- ported here. My thanks to both. It should be noted that this thesis is published with only minor editorial changes unrelated to the examiners’ comments. The Faculty of Information Technology and the Department of Mathemati- cal Information Technology have generously allowed me to follow my own me- andering path in my postgraduate studies alongside my teaching duties. It has taken about a decade to get to this point. The current Dean, Professor Pekka Neit- taanmäki, and the current Head of Department, Professor Tuomo Rossi, have par- ticularly offered encouragement, challenges and guidance during this process. Professor Emeritus Markku Sakkinen, Doctor Hannakaisa Isomäki, Doc- tor Ville Isomöttönen, Doctor Sami Kollanus, and Mr. Antti-Jussi Lakanen have helped me with discussions, encouragement and critique. Thanks are due to all my other colleagues, as well. In April 2011, I attended (without a paper to present) the 15th International Conference on Evaluation and Assessment in Software Engineering (EASE) at Durham University, England. I enjoyed the presentations and informal discus- sions among the participants very much; they also gave me good ideas and in- spiration. The three anonymous reviewers of the EASE 2012 conference gave me use- ful feedback on an early manuscript detailing the progress of this study. The interlibrary loan service of the university library responded efficiently to every one of my interlibrary loan requests; I had to make a lot of them during this mapping study. Without them, this thesis could not be what it is now. My mother, Ms. Maija Tuomaala, with background in philosophy and in education research, gave me useful feedback on many drafts of this thesis and other related manuscripts. I hereby also thank all my family and friends for encouragement, support and understanding. NOTE ON ENGLISH USAGE This thesis, as is customary, has a single author. I find it awkward to use the first person plural (“we”) about work I have done alone, even though it is somewhat conventional. The more usual method of deliberately obscuring agency by using short passive constructions would be, in many cases, inappropriate (while lin- guistically quite legitimate, see Pullum 2014), as in a systematic secondary study clear indication of who did what is an important part of the audit trail. Thus, like Kitchenham (2010) in her mapping study, I use the singular first person in mine. In situations where I need to refer to a person whose sex is unknown or immaterial, I will generally use the singular “they” (see e. g. Baranowski 2002; Paterson 2011). Of the many less than ideal options available, it is, in my opinion, the best. LIST OF FIGURES FIGURE 1 A high-level representation of the mapping process. This dia- gram omits many details. ...................................................... 55 FIGURE 2 Flow diagram of the study selection process............................ 65 FIGURE 3 Bubble plot of included publications by publication forum and publication year.................................................................... 75 FIGURE 4 Bubble plot of included core publications by publication fo- rum and publication year ...................................................... 78 FIGURE 5 Bubble plot of core sub-studies, excluding experiments, cate- gorized by the facets of efficacy used and the primary research methods used. ...................................................................... 84 FIGURE 6 The number of included publications per year......................... 86 FIGURE 7 The number of included primary studies per publication year .. 87 FIGURE 8 The number of included core studies per publication year ........ 87 FIGURE 9 The number of randomized controlled experiments in the core per publication year .............................................................. 87 FIGURE 10 The number of core studies of conditionals per year................. 88 FIGURE 11 The number of core studies of loops per year........................... 89 FIGURE 12 The slr-tools select form.................................................. 165 LIST OF TABLES TABLE 1 Summary of selection process ................................................ 57 TABLE 2 Summary of manual search.................................................... 58 TABLE 3 Summary of automatic search ................................................ 60 TABLE 4 The overall and exclusive contribution and overlap of the var- ious search modalities ........................................................... 62 TABLE 5 The quasi-gold standard and quasi-sensitivity for manual searches 63 TABLE 6 The quasi-sensitivity and specificity of automatic searches ....... 64 TABLE 7 Pairwise Cohen kappas in the second selection validation ex- ercise. .................................................................................. 68 TABLE 8 Publication forums containing at least two in publications........ 74 TABLE 9 Included studies ................................................................... 76 TABLE 10 Design