Scholarly Horizons: University of Minnesota, Morris Undergraduate Journal Volume 2 Issue 2 Article 5 September 2015 Usability of Error Messages for Introductory Students Paul A. Schliep University of Minnesota, Morris Follow this and additional works at: https://digitalcommons.morris.umn.edu/horizons Part of the Software Engineering Commons Recommended Citation Schliep, Paul A. (2015) "Usability of Error Messages for Introductory Students," Scholarly Horizons: University of Minnesota, Morris Undergraduate Journal: Vol. 2 : Iss. 2 , Article 5. Available at: https://digitalcommons.morris.umn.edu/horizons/vol2/iss2/5 This Article is brought to you for free and open access by the Journals at University of Minnesota Morris Digital Well. It has been accepted for inclusion in Scholarly Horizons: University of Minnesota, Morris Undergraduate Journal by an authorized editor of University of Minnesota Morris Digital Well. For more information, please contact [email protected]. Schliep: Usability of Error Messages for Introductory Students Usability of Error Messages for Introductory Students Paul A. Schliep Division of Science and Mathematics University of Minnesota, Morris Morris, Minnesota, USA 56267 [email protected] ABSTRACT that students struggle with various elements of error mes- Error messages are an important tool programmers use to sages such as terminology and source highlighting [1, 7]. help find and fix mistakes or issues in their code. When Several tools and heuristics are being developed to help ad- an error message is unhelpful, it can be difficult to find the dress issues in error message usability. The goals of these issue and may impose additional challenges in learning the methodologies are to help introductory programmers locate language and concepts. Error messages are especially crit- the issue in the program and guide the programmers to a ical for introductory programmers in understanding prob- solution. The purpose of this paper is to discuss analyses lems with their code. Unfortunately, not all error messages of error message design and its usability for introductory in programming are beneficial for novice programmers. This students in the scope of a class (meaning students' interac- paper discusses the general usability of error messages for in- tions with programming in a lab setting and at home), and troductory programmers, analyses of error messages in com- how these developed methodologies help improve the user pilers and DrRacket, and two methodologies intended to im- experience with error messages. prove error handling. This paper is divided into four sections. In Section 2 we provide background on usability studies, dynamic and static typing, compiler and runtime errors, and an overview of lan- Keywords guages and tools used. In Section 3 we focus on analyses Novice programmers, usability, error messages, usability stud- of the usability of error messages in development environ- ies, compiler errors, syntax errors ments and compiler messages for introductory students and how those analyses were performed. In Section 4 we ex- amine Marceau et al.'s recommendations for error message 1. INTRODUCTION design [5] and then we explore Denny et al.'s syntax error One of the most important foundations of computer pro- enhancement tool [1]. gramming is the communication between the system and the user, specifically in the error messages produced by the 2. BACKGROUND system. These error messages are especially important for In order to discuss the analyses of error messages, we introductory-level computer science students to help them need to understand several concepts related to error types resolve issues in their program because the error messages and usability. These concepts include compiler errors, syn- are the primary source for understanding what is wrong. tax errors, runtime errors, usability studies, and Human- According to Marceau et al., \[students] lack the experience Computer Interaction. We also introduce dynamically and to decipher complicated or poorly-constructed feedback" [4]. statically typed languages and their differences. We then The first rule of good message design is to be sure that the discuss the programming languages and tools used in the error does not add confusion [2]. Difficulties in understand- analysis of the error messages. We will focus on the Racket, ing error messages can often lead the programmers to frus- C++, and Java programming languages and the integrated tration because the error message is either too complicated development environment (IDE) DrRacket. to understand or led them down the wrong path [5], which can sometimes introduce new errors [1]. 2.1 Human-computer interaction and meth- Various studies have been conducted on modern program- ming languages' error messages to evaluate their effective- ods of usability analysis ness in helping novice programmers. The results have shown The study of Human-Computer Interaction, or HCI, is focused on how computer technology is used, specifically on the interfaces between the user and the programs on the computer. As Traver notes, \HCI is a discipline that aims to provide user interfaces that make working with a computer a more productive, effective, and enjoyable task" [7]. Much This work is licensed under the Creative Commons Attribution- of the research presented in this paper is viewed from an Noncommercial-Share Alike 3.0 United States License. To view a copy HCI perspective and emphasizes usability. of this license, visit http://creativecommons.org/licenses/by-nc-sa/3.0/us/ or In order to analyze these messages from an HCI per- send a letter to Creative Commons, 171 Second Street, Suite 300, San Fran- cisco, California, 94105, USA. spective and attain qualitative and quantitative information UMM CSci Senior Seminar Conference, December 2013 Morris, MN. about their usability, a case study may be performed. A Published by University of Minnesota Morris Digital Well, 2015 1 Scholarly Horizons: University of Minnesota, Morris Undergraduate Journal, Vol. 2, Iss. 2 [2015], Art. 5 case study is a research method that closely studies a group 2.3 Statically and dynamically typed of participants (in this paper, the participants are introduc- While we do not mention these concepts in any study, it tory students in the scope of the class) and collects data is important to note how some languages differ in error han- about participants interactions by observations and inter- dling. In statically typed languages, type checking is done views. Many of the studies analyzed in this paper are using at compile-time and objects are bound to types. This means a case study design. Using the data obtained from these that when programming in statically typed languages, a pro- studies a t-test may be performed, which is a statistical ex- grammer will need to pay attention to how a variable's type amination of two data sets and finds if they are significantly is cast. However, statically typed languages provide benefits different from each other. The significance in a t-test is de- such as earlier detection of programming mistakes. termined by the calculated probability, or the p-value. The A dynamically typed language means a type is interpreted p-value is the estimated probability that there is no signifi- at runtime rather than compile time and variables in the lan- cant difference in the data sets. In this paper, a p-value < guage are not bound to types. Since the runtime system of 0.05 will be considered significant. a dynamically typed language deduces type and type con- 2.2 Compiler and runtime errors versions, a programmer does not have to worry about type declaration while writing code. When writing code, a programmer may receive various er- As dynamically and statically typed languages differ in ror messages during different phases of the program, runtime how types are handled, something can be legal in dynami- and compile time. A compiler is a program that converts cally typed and illegal in statically typed language. Consider source code written in a programming language to a lower- the following example: level language understood by a processor or interpreter so that instructions can be executed. A compilation error is re- personName = "Francis" turned when the compiler fails to compile a piece of program personName = 7 source code. A program will not run if there is a compila- tion error because the compiler will not be able to create This sequence of statements is illegal in a statically typed executable code to run if there are errors the compiler finds. language since we are binding a string to personName, then We also discuss several examples of syntax errors in this an integer to personName. This statement would then throw paper. A syntax error is a type of compilation error that an error during compile time. This is a legal statement in a occurs when the code does not conform to the syntactical dynamically typed language, however, since variables do not order expected by the parser. The parser is a program (usu- have types in dynamic languages. Note that this statement ally part of a compiler) that receives input as source code would not work in a purely function language, or a functional and builds it into a structural representation of the input language that does not allow side effects, as you can not while checking for correct syntax of the input. As Kum- change the value of a variable. merfield and Kay note, \The usability of compiler errors is important because syntax error correction is the first step 2.4 Overview of programming languages and in the debugging process. It is not possible to continue pro- tools analyzed gram development until the code compiles. This means it is In section 3, we discuss a study performed by Marceau et a crucial part of the error correction process." [3] al. that analyzes the error messages in the DrRacket inte- Below is an example of a compiler error in Java. Here, grated development environment [4].