JYVÄSKYLÄ STUDIES IN COMPUTING 222 Antti-Juhani Kaijanaho Evidence-Based Programming Language Design A Philosophical and Methodological Exploration JYVÄSKYLÄ STUDIES IN COMPUTING 222 Antti-Juhani Kaijanaho Evidence-Based Programming Language Design A Philosophical and Methodological Exploration Esitetään Jyväskylän yliopiston informaatioteknologian tiedekunnan suostumuksella julkisesti tarkastettavaksi yliopiston vanhassa juhlasalissa S212 joulukuun 4. päivänä 2015 kello 12. Academic dissertation to be publicly discussed, by permission of the Faculty of Information Technology of the University of Jyväskylä, in building Seminarium, auditorium S212, on December 4, 2015 at 12 o’clock noon. UNIVERSITY OF JYVÄSKYLÄ JYVÄSKYLÄ 2015 Evidence-Based Programming Language Design A Philosophical and Methodological Exploration JYVÄSKYLÄ STUDIES IN COMPUTING 222 Antti-Juhani Kaijanaho Evidence-Based Programming Language Design A Philosophical and Methodological Exploration UNIVERSITY OF JYVÄSKYLÄ JYVÄSKYLÄ 2015 Editors Timo Männikkö Department of Mathematical Information Technology, University of Jyväskylä Pekka Olsbo, Ville Korkiakangas Publishing Unit, University Library of Jyväskylä URN:ISBN:978-951-39-6388-0 ISBN 978-951-39-6388-0 (PDF) ISBN 978-951-39-6387-3 (nid.) ISSN 1456-5390 Copyright © 2015, by Antti-Juhani Kaijanaho and University of Jyväskylä Jyväskylä University Printing House, Jyväskylä 2015 GLENDOWER. I can call spirits from the vasty deep. HOTSPUR. Why, so can I, or so can any man; But will they come when you do call for them? — William Shakespeare’s Henry IV Part 1 (III.1) The most elementary and valuable statement in science, the beginning of wis- dom, is “I do not know”. — Data in the Star Trek The Next Generation episode “Where Silence Has Lease” Human beings, he among you is wisest who knows like Socrates that he is actu- ally worthless with respect to wisdom. — attributed to the god in Delphi by Socrates (Plato 2014, at 23b). ABSTRACT Kaijanaho, Antti-Juhani Evidence-Based Programming Language Design. A Philosophical and Method- ological Exploration. Jyväskylä: University of Jyväskylä, 2015, 256 p. (Jyväskylä Studies in Computing ISSN 1456-5390; 222) ISBN 978-951-39-6387-3 (nid.) ISBN 978-951-39-6388-0 (PDF) Finnish summary Diss. Background: Programming language design is not usually informed by empirical studies. In other fields similar problems have inspired an evidence-based paradigm of practice. Such a paradigm is practically inevitable in language design, as well. Aims: The content of evidence-based programming design (EB-PLD) is explored, as is the concept of evidence in general. Additionally, the extent of evidence potentially useful for EB-PLD is mapped, and the appropriateness of Cohen’s kappa for evaluating coder agreement in a secondary study is evaluated. Method: Philosophical analysis and explication are used to clarify the unclear. A system- atic mapping study was conducted to map out the existing body of evidence. Results: Evidence is a report of observations that affects the strength of an argu- ment. There is some but not much evidence. EB-PLD is a five-step process for resolving uncertainty about design problems. Cohen’s kappa is inappropriate for coder agreement evaluation in systematic secondary studies. Conclusions: Coder agreement evaluation should use Scott’s pi, Fleiss’ kappa, or Krippendorff’s al- pha. EB-PLD is worthy of further research, although its usefulness was out of scope here. Keywords: programming languages, programming language design, evidence- based paradigm, philosophical analysis, evidence, systematic map- ping study, coder agreement analysis Author’s address Antti-Juhani Kaijanaho Department of Mathematical Information Technology University of Jyväskylä, Finland PO Box 35 (Agora), FI-40014 University of Jyväskylä [email protected] Supervisors Professor Tommi Kärkkäinen Department of Mathematical Information Technology University of Jyväskylä, Finland Doctor Vesa Lappalainen Department of Mathematical Information Technology University of Jyväskylä, Finland Doctor Ville Tirronen Department of Mathematical Information Technology University of Jyväskylä, Finland Reviewers Professor Matthias Felleisen College of Computer and Information Science Northeastern University, USA Professor Andreas Stefik Department of Computer Science University of Nevada, Las Vegas, USA Opponent Professor Lutz Prechelt Institute of Computer Science Freie Universität Berlin, Germany ACKNOWLEDGEMENTS I would like to thank my supervisors, Professor Tommi Kärkkäinen, Doctor Vesa Lappalainen, and Doctor Ville Tirronen, for encouragement, discussions and feed- back over the years. I am honored to have had Professor Matthias Felleisen of the Northeastern University and Professor Andreas Stefik of the University of Nevada, Las Vegas, as the reviewers of this dissertation and to have Professor Lutz Prechelt of the Freie Universität Berlin as the opponent in the defense. The valuable comments of the reviewers, particularly the extensive comments by Professor Stefik, have resulted in minor changes to the manuscript prior to publication. This dissertation extends my Licentiate Thesis (published as Kaijanaho 2014), which was examined by Doctor Stefan Hanenberg of the University of Duisburg– Essen and Professor Stein Krogdahl of the University of Oslo. My thanks to them for their valuable comments. The Faculty of Information Technology and the Department of Mathemati- cal Information Technology have generously allowed me to follow my own me- andering path in my postgraduate studies alongside my teaching duties. It has taken more than a decade to get to this point. The current Dean, Professor Pekka Neittaanmäki, and the current Head of Department, Professor Tuomo Rossi, have particularly offered encouragement, challenges and guidance during this process. A significant part of this dissertation was written while I was visiting the Chair of Data Management Systems and Knowledge Representation at the Uni- versity of Duisburg–Essen in Essen, Germany, from the beginning of February to the end of April 2015. My thanks to Professor Rainer Unland and Doctor Stefan Hanenberg for making this visit possible at their end; and to the two local doc- toral students—Mr. Hans-Theodor Hünewinkel and Ms. Zohreh Akbari—who shared their office with me during the visit. A thank-you is also due to Professor Tuomo Rossi and the Department of Mathematical Information Technology, who suggested and funded the trip, respectively. While in Essen, I had many deep conversations with Dr. Hanenberg on em- pirical studies and the philosophy of science, among other topics. We hold very different views on the philosophy of science and in particular on the manner in which computing research should be conducted, although we agree on the need for empirical human-factors research; this made the discussions very interesting to me and, I believe, to Dr. Hanenberg as well. My presentation of the ideas in this dissertation were influenced by those discussions. I participated in the Dagstuhl Seminar 15222 “Human-Centric Development of Software Tools” at the end of May 2015 as this dissertation was in the final stages of preparation. I wish to thank the organizers and other participants of the seminar for enlightening discussions which (among other things) illuminated my thinking regarding some of the last difficult issues in this dissertation and thus helped me make this dissertation better. I wish to especially thank the partici- pants of the breakout session on the philosophy of science. Professor Emeritus Markku Sakkinen, Doctor Hannakaisa Isomäki, Doc- tor Ville Isomöttönen, Doctor Sami Kollanus, and Mr. Antti-Jussi Lakanen have helped me with discussions, encouragement and critique. Thanks are due to all my other colleagues, as well. In my Licentiate Thesis acknowledgements, I also thanked the participants of the 15th International Conference on Evaluation and Assessment in Software Engineering (EASE) at Durham University, England in 2011; three anonymous re- viewers of EASE 2012; and the interlibrary loan service of the university library. These thanks apply also to the relevant parts of this dissertation, but for the de- tails, please see the Licentiate Thesis acknowledgements. My mother, Ms. Maija Tuomaala, with background in philosophy and in education research, gave me useful feedback on many drafts of this dissertation and other related manuscripts. I hereby also thank all my family and friends for encouragement, support and understanding. LIST OF FORMAL NOTATION USED IN CHAPTER 4 Notation used in the metalanguage only N the set of all nonnegative integers f0, 1, 2, . g R the set of real numbers [0, 1] f r 2 R j 0 ≤ r ≤ 1 g ]−1, 1] f r 2 R j −1 < r ≤ 1 g P(S) powerset of S L an arbitrary finitary quantifier-free first-order predicate language L1 an extension of L to allow finitary universal and existential quan- tifiers Lw an extension of L to allow countably infinite conjunctions and disjunctions 1 Lw an extension of L to allow finitary universal and existential quan- tifiers and countably infinite conjunctions and disjunctions p, q arbitrary formulas or sentences of the language under discussion K an arbitrary set of sentences of the language under discussion, usually used as (a part of) a premise of an argument H the set of all variable-free terms of the language under discussion S the set of all sentences of the language under discussion p[t/x] the formula obtained by substituting the term t for all free occur- rences of the variable x within p, taking