“I know it when I see it” – Perceptions of Code ality ITiCSE’17 Working Group Report∗ Jürgen Börstler Harald Störrle Daniel Toll Blekinge Institute of Technology QAware GmbH Linnæus University Karlskrona, Sweden Munich, Germany Kalmar, Sweden [email protected] [email protected] [email protected] Jelle van Assema Rodrigo Duran Sara Hooshangi University of Amsterdam Aalto University George Washington University Amsterdam, The Netherlands Helsinki, Finland Washington, DC, USA [email protected] rodrigo.duran@aalto. [email protected] Johan Jeuring Hieke Keuning Carsten Kleiner Utrecht University Windesheim University of Applied University of Applied Sciences & Arts Utrecht, The Netherlands Sciences Hannover [email protected] Zwolle, The Netherlands Hannover, Germany [email protected] [email protected] Bonnie MacKellar St John’s University Queens, NY, USA [email protected] ABSTRACT The groups showed signicant dierences in the sources they use Context. Code quality is a key issue in software development. The for learning about code quality with education ranked lowest in all ability to develop high quality software is therefore a key learning groups. goal of computing programs. However, there are no universally Conclusions. Code quality should be discussed more thoroughly accepted measures to assess the quality of code and current stan- in educational programs. dards are considered weak. Furthermore, there are many facets to code quality. Dening and explaining the concept of code quality CCS CONCEPTS is therefore a challenge faced by many educators. •General and reference ! Evaluation; •Social and professional Objectives. In this working group, we investigated code quality as topics ! Quality assurance; Computer science education; perceived by students, educators, and professional developers, in Software engineering education; particular, the dierences in their views of code quality and which quality aspects they consider as more or less important. Further- KEYWORDS more, we investigated their sources for information about code Code quality, programming. quality and its assessment. Methods. We interviewed 34 students, educators and professional ACM Reference format: developers regarding their perceptions of code quality. For the Jürgen Börstler, Harald Störrle, Daniel Toll, Jelle van Assema, Rodrigo Duran, Sara Hooshangi, Johan Jeuring, Hieke Keuning, Carsten Kleiner, interviews they brought along code from their own experience to and Bonnie MacKellar. 2017. “I know it when I see it” – Perceptions of Code discuss and exemplify code quality. Quality. In Proceedings of ITiCSE’17, Bologna, Italy, July 3–5, 2017, 15 pages. Results. There was no common denition of code quality among DOI: http://dx.doi.org/10.1145/3059009.3081328 or within these groups. Quality was mostly described in terms of in- dicators that could measure an aspect of code quality. Among these indicators, readability was named most frequently by all groups. 1 INTRODUCTION ∗Working group co-leaders: Jürgen Börstler, Harald Störrle, and Daniel Toll The ability to develop high quality software is a key learning goal of computing programs [17, 23]. However, there are no univer- Permission to make digital or hard copies of part or all of this work for personal or sal measures to assess the quality of code and current standards classroom use is granted without fee provided that copies are not made or distributed for prot or commercial advantage and that copies bear this notice and the full citation are considered weak [6]. Dening the concept of “good” code is on the rst page. Copyrights for third-party components of this work must be honored. therefore a challenge faced by many educators. For all other uses, contact the owner/author(s). Programming textbooks typically emphasize the importance of ITiCSE’17, Bologna, Italy © 2017 Copyright held by the owner/author(s). 978-1-4503-4704-4/17/07...$15.00 quality, but rarely go beyond correctness and commenting in their DOI: http://dx.doi.org/10.1145/3059009.3081328 treatment of quality. The value of thorough commenting beyond ITiCSE’17, July 3–5, 2017, Bologna, Italy J. Börstler et al. Develop materials Recruiting Interviews Transcribing Data handling Analysis • Interview guide interviewees • Part 1: Demographic interviews • Compile “survey • Transcription guide information and data” into one • Participant infor- experience document mation sheet • Part 2: Open question • Data cleaning • Interview script • Part 3: Follow-up questions Figure 1: Overview over the study process. “self-documenting code” is debatable though [13], and there is little items were at a high level. Their study did not show any signi- empirical evidence on the eects of comments on code quality [33]. cant eects of pedagogical code reviews on code quality, which is An early and strong focus on thorough commenting (and tools like quite interesting, since research on inspections in general shows a JavaDoc) might actually take time from more important quality positive impact on quality [3, 21]. issues. Quality aspects like understandability and maintainability Research shows that low level code features may aect code get little attention, although they are of signicant importance for quality. Butler et al., for example, showed that awed identier students’ professional careers. names are associated with low code quality [10]. A recent study Anything a professional software developer will do, be it devel- of 210 open-source Java projects regarding the eect of coding opment, testing or maintenance, will involve reading and under- conventions showed that size, comments and indentation aected standing code that someone else has developed. Eventually, their readability most [22]. Furthermore, recent research in program com- code will also need to be read and understood by others. It is there- prehension shows that misleading names are more problematic than fore important to better prepare students to write code that will be meaningless names [2], but that specic one-letter variables still easy to understand by others. can convey meaning [5]. It has also been shown that low-level struc- Some educators use static analysis tools such as Checkstyle1 or tural dierences aect program understanding, for example that FindBugs2 to support the assessment of programming assignments. for-loops are signicantly harder to understand than if-statements For example, Keuning et al. [18] recently showed that many student [1] and that “maintaining undisciplined annotations is more time programs in the (huge) Blackbox database contain quality issues as consuming and error prone” than maintaining disciplined ones [24]. reported by PMD3, and that students hardly ever x an issue, even There is also a large body of work on software quality measure- when they make use of analysis tools. These tools check a large ment [11], but there is little conclusive evidence on the relationship range of potential quality issues, but it is not clear which of these between the measurements and common software quality attributes tools are suitable or even appropriate in an educational context. [16]. Recent research actually questions the usefulness of many In this working group, we looked into the ways that students, common software measures, since they no longer have any predic- educators, and developers perceive code quality, in order to inves- tive power when controlled for program size [12]. tigate which quality aspects are seen as more or less important, Research also shows that educators do not have a suciently and which sources of information regarding code quality issues are accurate understanding of the programming mistakes that students used by these groups. actually make. A recent large-scale study showed that “educators’ estimates do not agree with one another or the student data” regard- 2 RELATED WORK ing which student mistakes were most common [9]. One conclusion In an ITiCSE’09 working group, we investigated the quality of from these results could be that many educators might base their object-oriented example programs from common introductory pro- teaching materials on invalid invalid assumptions about the issues gramming textbooks [7, 8]. Our results showed that the overall that students have. quality is not as high as one would expect, in particular regarding In our work, we therefore want to elicit rst-hand data from the exemplication of object-oriented properties. students, educators and developers to better understand their per- Stegeman et al. [30, 31] analyzed normative statements about ceptions of code quality. code quality from three popular texts on software development and compiled them into a set of 20 quality aspects. Furthermore, they 3 METHOD interviewed three teachers based on a programming assignment, Our overall goal was to investigate the dierences in perceptions using this set of quality aspects. The quality aspects and the results of code quality held by students, educators, and developers. from the interviews where then used to generate rubrics for the assessment of programming assignments. 3.1 Overall Process Inspired by code inspections [14, 21], Hundhausen et al. pro- posed pedagogical code reviews in which student teams review Since this study involved 10 researchers from dierent countries each others’ code using a checklist of “best practices” [15]. Their and institutions, we developed guides and instructions to ensure focus was on students’ attitudes toward quality and the checklist that the same procedures were followed at each