Syntax and Sensibility: Using language models to detect and correct syntax errors Eddie Antonio Santos, Joshua Charles Campbell, Dhvani Patel, Abram Hindle, and José Nelson Amaral Department of Computing Science University of Alberta Edmonton, Alberta, Canada {easantos,joshua2,dhvani,hindle1,jamaral}@ualberta.ca Abstract—Syntax errors are made by novice and experienced Listing 1: Syntactically invalid Java code. An open brace ({) is missing programmers alike; however, novice programmers lack the years at the end of line 3. of experience that help them quickly resolve these frustrating errors. Standard LR parsers are of little help, typically resolving 1 public class A{ public static void main(String args[]) { syntax errors and their precise location poorly. We propose 2 3 if (args.length < 2) a methodology that locates where syntax errors occur, and 4 System.out.println("Not enough args!"); suggests possible changes to the token stream that can fix the 5 System.exit(1); error identified. This methodology finds syntax errors by using 6 } language models trained on correct source code to find tokens 7 System.out.println("Hello, world!"); that seem out of place. Fixes are synthesized by consulting the 8 } language models to determine what tokens are more likely at the 9 } estimated error location. We compare n-gram and LSTM (long short-term memory) language models for this task, each trained input to a Java 8 compiler such as OpenJDK’s javac [4], on a large corpus of Java code collected from GitHub. Unlike and it reports that there is an error with the source file, but it prior work, our methodology does not rely that the problem overwhelms the user with misleading error messages to identify source code comes from the same domain as the training data. the location of the mistake made by the programmer. We evaluated against a repository of real student mistakes. Our A.java:7: error: <identifier> expected tools are able to find a syntactically-valid fix within its top- System.out.println("Hello, world!"); 2 suggestions, often producing the exact fix that the student ^ used to resolve the error. The results show that this tool and A.java:7: error: illegal start of type methodology can locate and suggest corrections for syntax errors. System.out.println("Hello, world!"); Our methodology is of practical use to all programmers, but will ^ be especially useful to novices frustrated with incomprehensible A.java:9: error: class, interface, or enum expected syntax errors. } I. INTRODUCTION ^ 3 errors Computer program source code is often expressed in plain Imagine a novice programmer writing this simple program text files. Plain text is a simple, flexible medium that has been for the first time, being greeted with three error messages, preferred by programmers and their tools for decades. Yet, all of which include strange jargon such as error: illegal plain text can be a major hurdle for novices learning how to start of type. The compiler identified the problem as being code [1, 2, 3]. Not only do novices have to learn the semantics at least on line 7 when the mistake is four lines up, on line 3. of a programming language, but they also have to learn how However, an experienced programmer could look at the source to place arcane symbols in the right order for the computer to code, ponder, and exclaim: “Ah! There is a missing open brace understand their intent. The positioning of symbols in just the ({) at the end of line 3!” Such mistakes involving unbalanced right way is called syntax, and sometimes humans—especially braces are the most frequent errors among new programmers [5], novices—get it wrong [1]. yet the compiler offers little help in resolving them. The tools meant for interpreting the human-written source In this paper, we present Sensibility, which finds and fixes code, parsers, are often made such that they excel in understand- single token syntax errors. We address the following problem: ing well-structured input; however, if they are given an input Given a source code file with a syntax error, how can one with so much as one mistake, they can fail catastrophically. accurately pinpoint its location and produce a single token What’s worse, the parser may come up with a misleading suggestion that will fix it? conclusion as to where the actual error is. Consider the Java source code in Listing 1. A single token—an open brace ({) We compare the use of n-gram models and long short-term at the end of line 3—in the input is different from that of the memory neural network (LSTM) models for modelling source correct file that the programmer intended to write. Give this code for the purposes of correcting syntax errors. All models were trained on a large corpus of hand-written Java source error messages the compiler generates and how long it takes for code collected from GitHub. Whereas prior works offer the programmer to fix them. This research shows that novices solutions that are limited to fixing syntax errors within the and advanced programmers alike struggle with syntax errors same domain as the training data—such as only fixing errors and their accompanying error messages. Dy and Rodrigo [22] within the same source repository [6], or within the same developed a system that detects compiler error messages that do introductory programming assignment [7, 8]—Sensibility not indicate the actual fault, which they name “non-literal error imposes no such restriction. As such, we evaluated against messages”, and assists students in solving them. Nienaltowski et a corpus of real syntax errors collected from the Blackbox al. [23] found that more detailed compiler error messages do repository of novice programmers’ activity [9]. not necessarily help students avoid being confused by error In this paper, we focus on correcting syntax errors at the messages. They also found that students presented with error token level. We do not consider semantic errors that can messages only including the file, line, and a short message did occur given a valid parse tree such as type mismatches, and— better at identifying the actual error in the code more often in our abstract models (Section III-B)—misspelled variable for some types of errors. Marceau et al. [24] developed a names. These errors are already handled by other tools given rubric for grading the error messages produced by compilers. a source file with valid syntax. For example, the Clang C++ Becker’s dissertation [25] attempted to enhance compiler error compiler [10] can detect misspelled variable names and—since messages in order to improve student performance. Barik et it has a valid parse tree—Clang can suggest which variable al. [26] studied how developers visualize compilation errors. in scope may be intended. As such, we focus our attention in They motivate their research with an example of a compiler this paper to errors that occur before a compiler can reach this misreporting the location of a fault. Pritchard [27] shows that stage—namely, errors that prevent a valid parse of a source the most common type of errors novices make in Python are code file. In particular, we consider syntax errors that are non- syntax errors. Brown et al. [5] investigated the opinions of trivial to detect using simple handwritten rules, as used in some educators regarding what they believe are the most common parsers [11, 12]. Java programming mistakes made by novices, and contrasted Our contributions include: it with mistakes mined from the Blackbox dataset. They found • Showing that language models can successfully locate and that educators share a weak consensus of what errors are fix syntax errors in human-written code without parsing. most frequent, and furthermore show that across 14;235;239 • Comparing three different language models including two compilation events, educators’ opinion demonstrates a low level n-gram models and one deep neural network. of agreement against the actual data mined. The most common • Evaluating how all three models perform on a corpus of real- error witnessed was unbalanced parenthesis and brackets, which world syntax errors with known fixes provided by students. was ranked the eleventh most common on average by educators. The other most common errors were semantic and type errors. II. PRIOR WORK Earlier research attempted to tackle errors at the parsing Prior work relevant to this research addresses repositories stage. In 1972, Aho [28] introduced an algorithm to attempt to of source code and repositories of mistakes, past attempts at repair parse failures by minimizing the number of errors that locating and fixing syntax errors in and out of the parser, were generated. In 1976, Thompson [29] provided a theoretical methods for preventing syntax errors, deep learning, and basis for error-correcting probabilistic compiler parsers. He applying deep learning to syntax error locating and fixing. also criticized the lack of probabilistic error correction in then- a) Data-sources and repositories: are needed to train modern compilers. However, this trend continues to this day. models and evaluate the effectiveness of techniques on syntax Parr et al. [30] discusses the strategy used in ANTLR, a errors. Brown et al. [9, 13] created the Blackbox reposi- popular LL(*) parser-generator. The parser attempts to repair tory of programmers’ activity collected from the BlueJ Java the code when it encounters an error using the context around Integrated Development Environment (IDE) [14], which is the error so it can continue parsing. This strategy allows used internationally in introductory computer science classes. ANTLR parsers to detect multiple problems instead of stopping Upon installation, BlueJ asks the user whether they consent on the first error. Jeffery [11] created Merr, an extension of the to anonymized data collection of editor events.