EXAMENSARBETE INOM TEKNIK, GRUNDNIVÅ, 7,5 HP STOCKHOLM, SVERIGE 2019 Comparing syntax highlightings and their effects on code comprehension ERIK HÄREGÅRD ALEXANDER KRUGER KTH SKOLAN FÖR ELEKTROTEKNIK OCH DATAVETENSKAP Comparing syntax highlightings and their effects on code comprehension ERIK HÄREGÅRD ALEXANDER KRUGER Bachelor in Computer Science Date: June 7, 2019 Supervisor: Jeanette Hellgren Kotaleski Examiner: Örjan Ekeberg KTH Swedish title: Jämförelse av syntax highlightings och dess effekt på kodförståelse iii Abstract Syntax highlight is a system designed to assist a writer or programmer by dis- playing different parts of a text in a specific color based on its function. In this study, we conducted a practical experiment comparing the effectiveness of two common syntax highlightings by primarily measuring the speed of which par- ticipants could understand a given piece of code. The highlights chosen for the comparison were Atom UI Standard and Styri. Previous studies in this area, most notably by Advait Sarkar from the University of Cambridge, have shown a generally positive effect of syntax highlight but have not compared how different types of syntax highlightings affected the reader. Our study was performed with eight participants from a Technical High school with a minor background in programming who answered six questions where some code was highlighted and some not; the order of the items was found to affect re- sults on some questions. The results do not show any significant advantage to using syntax highlight or a difference between the effectiveness of the two syn- tax highlights. Some participants saw a slight constant advantage from using syntax highlight, but no general or notable conclusion could be made about these participants. iv Sammanfattning Syntax highlight är ett system menat att hjälpa författare eller programmerare att skriva text eller kod genom att ändra textfärgen baserat på dess effekt i en mening eller instruktion. I denna studie genomfördes ett praktiskt experiment som jämförde effekten av två olika syntax highlightings genom att huvudsak- ligen mäta hur lång tid det tog för deltagarna att förstå ett givet kodstycke. Ti- digare studier, av framförallt Advait Sarkar från Cambridge Universitet, inom detta område har visat en generell positiv effekt av syntax highlight men har inte jämfört hur olika syntax highlightings påverkade läsaren. De två syntax highlights som valdes var Atom UI Standard och Styri. Vår studie utfördes med åtta studenter från ett tekniskt gymnasium som hade viss erfarenhet inom pro- grammering. Deltagarna svarade på sex frågor där de blev visade kodstycken. Några av dessa var med syntax highlight och resten hade ingen syntax high- light. Frågornas ordning var konsekvent mellan deltagarna. Vid senare under- sökning visade det sig att frågornas ordning hade påverkat resultatet av vissa frågor. Resultatet visar ingen generell signifikant positiv effekt av att använda syntax highlight eller någon skillnad mellan de två utvalda highlightingsen. Vissa deltagare såg en liten konstant fördel av att använda syntax highlight men inte tillräckligt för en signifikant slutsats. v Acknowledgements We want to acknowledge Jeanette Hellgren Kotaleski for providing quick an- swers when needed and her support throughout the project. We would also like to thank Lovisa Johansson for assisting in finding suitable questions. This study would also not have been possible without Maria Franzén assisting us in finding appropriate participants and providing excellent feedback on our questions. Contents 1 Introduction 1 1.1 Limiting the Scope . .2 1.2 Research Question . .3 1.3 Hypothesis . .3 2 Background 4 2.1 State of the Art . .6 2.2 Difference of the Syntax Highlightings . .6 3 Method 8 3.1 Design . .8 3.2 Selection . .9 3.3 Ethical considerations . 10 4 Results 11 5 Conclusion and Discussion 13 5.1 Conclusion . 13 5.2 Discussion . 13 5.3 Problems with the Study . 14 5.4 Future Work . 16 Bibliography 17 vi Chapter 1 Introduction The core function of Syntax Highlight is to change the text color for the user in the hopes of making the text more accessible and faster to read [1]. It is impor- tant to note that a syntax highlight does not alter the behavior of the code. In the following example 1.2, we have a function called “System.out.println” that is given some text in the form of a string containing the text “HelloWorld”. A function is an instruction that takes something like a string and returns something else. A string is how a computer stores text. Because the “Sys- tem.out.println” is a function the syntax highlight system gives it the color blue; similarly because “HelloWorld” is a string, it is given the color yellow. Different syntax highlightings can give different colors to different categories of a language. Syntax highlight makes it easier to see what purpose each part of the sentence or instruction has; an example can be to visualize what part is a function and what part is a variable such as a string. Figure 1.1: HelloWorld without syntax highlight Figure 1.2: HelloWorld with Styri highlight 1 2 CHAPTER 1. INTRODUCTION This customizability of syntax highlight has led to many programs and larger companies to develop a new style of syntax highlight; examples of this are AtomUI and Googles Material Design [2]. This study hopes to aid in the development of syntax highlightings that are more effective and increase the productivity of programmers. Given the goal, a fair question is why the focus on syntax highlighting? As shown in the famous quote below, programmers spend a lot of time reading code. This means that the speed of which program- mers can read code dramatically affects their overall efficiency. The quote comes from the book Clean Code: A Handbook of Agile Software Crafts- manship” which is the bible of excellent code; the author Robert C Martin is regarded in similar high standards: “Indeed, the ratio of time spent reading versus writing is well over 10 to 1. We are constantly reading old code as part of the effort to write new code.” [3] 1.1 Limiting the Scope There are many different syntax highlightings and we could not reasonably test them all. Therefore we have settled on comparing Atom standard highlight and Styri’s syntax highlight compared to having no syntax highlight. We attempted to select two well-known syntax highlightings in the hopes that their popularity is somehow related to their usefulness [4]. The reason Atom standard highlight and Styri’s syntax highlight were cho- sen was that they are both popular highlights in the Atom editor, one of the most popular code editors in the world. The two highlights are also signifi- cantly different from each-other, which can be seen in the figures later, and as such there might be a difference in their effectiveness. Other studies in the field (see Section 2.1) have used Eye-Tracking as an- other metric to measure the eye movement of participants to assist with the analysis. In this study we opted to not use Eye-Tracking, primarily due to the unreliability of the technology without using head-mounted setups. We in- stead chose to focus on the time and correctness of the answers as our metrics. CHAPTER 1. INTRODUCTION 3 1.2 Research Question Which of Atom standard highlight and Styri’s syntax highlight results in the fastest code comprehension compared to having no syntax highlight? 1.3 Hypothesis In a previous study by Advait Sarkar from the University of Cambridge [5], it was concluded that syntax highlight had a positive effect on the participant’s comprehension of code. In this paper, we aim to study if the system of color being used by a specific syntax highlight can affect the effectiveness of said highlight. We hypothesize that the type of highlight will not play a significant role in the effectiveness and we expect both highlights to perform better than having none at all. The basis of this hypothesis is that both highlights used are popular and as such are likely to perform well; while there are differences between them, it is unlikely that they are major enough to see a radical change in performance. Since most modern code editors feature some form of syntax highlight we find it reasonable that they should have some effect. Chapter 2 Background A language like English was created to facilitate communication between hu- mans and in the same way, programming languages were made to enable com- munication between humans and computers [6]. Where English has sentences made by stringing words together, programming languages are made the same way but words are often called “instructions” or “statements”. By combin- ing words or instructions, we form sentences that adhere to a programmings languages specific grammatical rules. The syntactic rules can be for example that a semi-colon “;” must be used to delimit a statement [7] which represents the end of an instruction or sentence. Unlike spoken languages like English, programming languages must be incredibly precise with their grammar and syntax; there is no room for am- biguity[8]. In this context, ambiguity is defined as being able to create two different interpretations of the same statement. Where it is possible to com- prehend a broken sentence in a human language, this cannot be done by a code compiler since it has to follow specific rules on how to interpret an instruction. Ambiguity is an undesirable trait to have when writing code since it leaves room for the program to misinterpret the instructions, and for that reason is avoided in most programming languages[6]. As such the programmer must write syntactically correct code for the compiler to understand the intentions of the programmer and create a working program. One of the first known editors featuring a system in principle similar to syn- tax highlight was Wilfred Hansen’s code editor Emily from 1969 [1].