Working Group Report ITiCSE-WGR ’19, July 15–17, 2019, Aberdeen, Scotland Uk Compiler Error Messages Considered Unhelpful: The Landscape of Text-Based Programming Error Message Research Brett A. Becker∗ Paul Denny∗ Raymond Pettit∗ University College Dublin University of Auckland University of Virginia Dublin, Ireland Auckland, New Zealand Charlottesville, Virginia, USA [email protected] [email protected] [email protected] Durell Bouchard Dennis J. Bouvier Brian Harrington Roanoke College Southern Illinois University Edwardsville University of Toronto Scarborough Roanoke, Virgina, USA Edwardsville, Illinois, USA Scarborough, Ontario, Canada [email protected] [email protected] [email protected] Amir Kamil Amey Karkare Chris McDonald University of Michigan Indian Institute of Technology Kanpur University of Western Australia Ann Arbor, Michigan, USA Kanpur, India Perth, Australia [email protected] [email protected] [email protected] Peter-Michael Osera Janice L. Pearce James Prather Grinnell College Berea College Abilene Christian University Grinnell, Iowa, USA Berea, Kentucky, USA Abilene, Texas, USA [email protected] [email protected] [email protected] ABSTRACT of evidence supporting each one (historical, anecdotal, and empiri- Diagnostic messages generated by compilers and interpreters such cal). This work can serve as a starting point for those who wish to as syntax error messages have been researched for over half of a conduct research on compiler error messages, runtime errors, and century. Unfortunately, these messages which include error, warn- warnings. We also make the bibtex file of our 300+ reference corpus ing, and run-time messages, present substantial difficulty and could publicly available. Collectively this report and the bibliography will be more effective, particularly for novices. Recent years have seen be useful to those who wish to design better messages or those that an increased number of papers in the area including studies on aim to measure their effectiveness, more effectively. the effectiveness of these messages, improving or enhancing them, and their usefulness as a part of programming process data that CCS CONCEPTS can be used to predict student performance, track student progress, • General and reference → Surveys and overviews; • Human- and tailor learning plans. Despite this increased interest, the long centered computing → Human computer interaction (HCI); history of literature is quite scattered and has not been brought • Social and professional topics → Computer science edu- together in any digestible form. cation; Computing education; CS1; • Applied computing → In order to help the computing education community (and re- Education; • Software and its engineering → General program- lated communities) to further advance work on programming error ming languages; Syntax; Compilers; Interpreters; Semantics; messages, we present a comprehensive, historical and state-of-the- art report on research in the area. In addition, we synthesise and present the existing evidence for these messages including the dif- KEYWORDS ficulties they present and their effectiveness. We finally presenta compiler error messages; considered harmful; CS1; CS-1; design set of guidelines, curated from the literature, classified on the type guidelines; diagnostic error messages; error messages; human com- puter interaction; HCI; introduction to programming; novice pro- ∗co-leader grammers; programming errors; programming error messages; re- view; run-time errors; survey; syntax errors; warnings Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation ACM Reference Format: on the first page. Copyrights for components of this work owned by others than the Brett A. Becker, Paul Denny, Raymond Pettit, Durell Bouchard, Dennis J. author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or Bouvier, Brian Harrington, Amir Kamil, Amey Karkare, Chris McDonald, republish, to post on servers or to redistribute to lists, requires prior specific permission Peter-Michael Osera, Janice L. Pearce, and James Prather. 2019. Compiler and/or a fee. Request permissions from [email protected]. Error Messages Considered Unhelpful: The Landscape of Text-Based Pro- ITiCSE-WGR ’19, July 15–17, 2019, Aberdeen, Scotland UK gramming Error Message Research. In 2019 ITiCSE Working Group Reports © 2019 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-6895-7/19/07...$15.00 (ITiCSE-WGR ’19), July 15–17, 2019, Aberdeen, Scotland UK. ACM, New York, https://doi.org/10.1145/3344429.3372508 NY, USA, 34 pages. https://doi.org/10.1145/3344429.3372508 177 Working Group Report ITiCSE-WGR ’19, July 15–17, 2019, Aberdeen, Scotland Uk 1 INTRODUCTION & MOTIVATION - not novice programming students, who can find it Computer programming is an essential skill that all computing difficult to interpret and understand. students must master [141] and is increasingly important in many Despite this importance, in the last 50+ years, little positive has diverse disciplines [167] as well as at pre-university levels [166]. been said about compiler error messages. In this time they have Research from Educational Psychology indicates that teaching and been described as inadequate and not understandable (Moulton learning are subject-specific [135] and that learning programming & Miller, 1967) [146], useless (Wexelblat, 1976) [208], not opti- consists of challenges that are different to reading and writing mal (Litecky, 1976) [120], inadequate [again, 16 years after [146]] natural languages [17, 68] or, for example, physics [34]. It is also (Brown, 1983) [36], frustrating (Flowers et al., 2005) [72], cryptic frequently stated that programming is difficult to learn [127]. As and confusing (Jadud, 2006) [98], notoriously obscure and terse early as 1977 it was explicitly stated that programming could be (Ben-Ari, 2007) [24], undecipherable (Traver, 2010) [202], intimidat- made easier [189]. The perception that programming is intrinsically ing (Hartz, 2012) [84], still very obviously less helpful than they hard however, can vary [125]. Regardless, the general consensus could be (McCall & Kölling, 2015) [139], inscrutable (Ko, 2017)1, for the past several decades from academia to the media, from frustrating [again, 13 years after [72]] and a barrier to progress universities to primary schools, seems to be “Learning to program (Becker et al., 2018) [21]. is hard” [102, p3]. Knuth pointed out ‘mysterious’ error messages as responsible One of the many challenges novice programmers face from the for breaking down the ability to see through layers of abstraction, start are notoriously cryptic compiler error messages, and there as Ramshaw, a former graduate student of Knuth’s recounted:2 is published evidence on these difficulties since at least as early Don [Knuth] claims that one of the skills that you as 1965 [177]. Such messages report details on errors relating to need to be a computer scientist is the ability to work programming language syntax and act as the primary source of with multiple levels of abstraction simultaneously. information to help novices rectify their mistakes. Most program- When you’re working at one level, you try and ignore ming educators are all too familiar with the reality of messages that the details of what’s happening at the lower levels. are difficult to understand and which can be a source of confusion But when you’re debugging a computer program and and discouragement for students. Supporting evidence for this can you get some mysterious error message, it could be be found in the literature spanning many decades. Students have a failure in any of the levels below you, so you can’t reported that such errors are frustrating, and have described them afford to be too compartmentalised. as “barriers to progress” [15]. Further, it has been reported that students have difficulty locating and repairing syntax errors using 1.1 Motivation only the typically terse error messages provided by the average With the users who deal with these messages at the focal point of compiler [53, 182]. These are traits not exclusive to programming this work, our chief motivations originate with how programming error messages – general system error messages also cause similar error messages affect students from two points of view: issues [187]. The effects these messages have on students is mea- (1) Internal student factors. For example: Students find pro- surable. A high number of student errors, and in particular a high gramming error messages to be frustrating [21, 51], intimidat- frequency of repeated errors – when a student makes the same ing [84], as well as confusing and harmful to confidence [100]. error consecutively – have been shown to be indicators of students Rightfully so, students do not trust them [97] but importantly who are struggling with learning to program [16, 98]. This area they do read them [12, 163]. Additionally, their behaviour is of research has seen increased interest in the last 10 years with altered by them [101] – but most programming instructors authors such as Barik [9], Becker et al. [19], Brown & Altadmri [31], have seen that with their own eyes. Denny et al. [50], Karkare (Ahmed et al.)[3], Kohn [106], McCall & (2) External student