Learning Syntactic Program Transformations from Examples
Reudismam Rolim∗, Gustavo Soares∗†, Loris D’Antoni‡, Oleksandr Polozov§, Sumit Gulwani¶, Rohit Gheyi∗, Ryo Suzukik, Bjorn¨ Hartmann† ∗UFCG, Brazil, †UC Berkeley, USA, ‡University of Wisconsin-Madison, USA §University of Washington, USA, ¶Microsoft, USA, kUniversity of Colorado Boulder, USA [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Abstract—Integrated Development Environments (IDEs), such update, a developer needs to locate all references to the old as Visual Studio, automate common transformations, such as Re- API and consistently replace them with the new API [1], name and Extract Method refactorings. However, extending these [2]. As another example, in programming courses student catalogs of transformations is complex and time-consuming. A submissions that exhibit the same fault often need similar similar phenomenon appears in intelligent tutoring systems where fixes. For large classes such as massive open online courses instructors have to write cumbersome code transformations that (MOOCs), manually providing feedback to different students describe “common faults” to fix similar student submissions to programming assignments. is an unfeasible burden on the teaching staff.
In this paper, we present REFAZER, a technique for auto- Since applying repetitive edits manually is tedious and matically generating program transformations. REFAZER builds error-prone, developers often strive to automate them. The on the observation that code edits performed by developers space of tools for automation of repetitive code edits contains can be used as input-output examples for learning program IDEs, static analyzers, and various domain-specific engines. transformations. Example edits may share the same structure IDEs, such as Visual Studio [3] or Eclipse [4], include features but involve different variables and subexpressions, which must be that automate some code transformations, such as adding boil- generalized in a transformation at the right level of abstraction. erplate code (e.g., equality comparisons or constructors) and To learn transformations, REFAZER leverages state-of-the-art code refactoring (e.g., Rename, Extract Method). Static ana- programming-by-example methodology using the following key components: (a) a novel domain-specific language (DSL) for de- lyzers, such as ReSharper [5], Coverity [6], ErrorProne [7] and scribing program transformations, (b) domain-specific deductive Clang-tidy [8] automate removal of suspicious code patterns, algorithms for efficiently synthesizing transformations in the DSL, potential bugs, and verbose code fragments. In an education and (c) functions for ranking the synthesized transformations. context, AutoGrader [9] uses a set of program transformations provided by an instructor to fix common faults in introductory We instantiate and evaluate REFAZER in two domains. First, programming assignments. given examples of code edits used by students to fix incorrect programming assignment submissions, we learn program trans- All aforementioned tool families rely on predefined cata- formations that can fix other students’ submissions with similar logs of recognized transformation classes, which are hard to faults. In our evaluation conducted on 4 programming tasks extend. These limitations inspire a natural question: performed by 720 students, our technique helped to fix incorrect submissions for 87% of the students. In the second domain, we arXiv:1608.09000v1 [cs.SE] 31 Aug 2016 use repetitive code edits applied by developers to the same project Is it possible to learn program transformations automatically? to synthesize a program transformation that applies these edits to other locations in the code. In our evaluation conducted on 59 Our key observation is that code edits gathered from reposito- scenarios of repetitive edits taken from 3 large C# open-source ries and version control history constitute input-output exam- projects, REFAZER learns the intended program transformation ples for learning program transformations. in 83% of the cases and using only 2.8 examples on average. The main challenge of example-based learning lies in Keywords—Program transformation, program synthesis, tutor- abstracting concrete code edits into classes of transformations ing systems, refactoring. representing these edits. For instance, Figure 1 shows similar edits performed by different students to fix the same fault in I.INTRODUCTION their submissions for a programming assignment. Although the edits share some structure, they involve different expressions As software evolves, developers edit program source code and variables. Therefore, a transformation should partially to add features, fix bugs, or refactor it. Many such edits have abstract these edits as in Figure 1(d). already been performed in the past by the same developers in a different codebase location, or by other developers in However, examples are highly ambiguous, and many differ- a different program/codebase. For instance, to apply an API ent transformations may satisfy them. For instance, replacing 1 def product(n, term): efficiently synthesize them. Finally, we define functions to rank 2 total, k = 1, 1 the synthesized transformations based on their DSL structure. 3 while k<=n: 4 - total= total *k Evaluation We evaluated REFAZER in two domains: learning 5 + total= total *term(k) transformations to fix submissions to introductory program- 6 k = k+1 7 return total ming assignments and learning transformations to apply repet- itive edits to large code bases. (a) An edit applied by a student to fix the program. Our first experiment is motivated by the recent advances in massive open online courses (MOOCs), where automatically 1 def product(n, term): grading student submission and providing personalized feed- 2 if (n==1): back is challenging due to the large number of students. In 3 return 1 this experiment, we mine existing submissions to programing 4 - return product(n-1, term)*n 5 + return product(n-1, term)*term(n) assignments to collect examples of edits applied by students to fix their code. We then use these examples to synthesize program transformations and we try using the learned trans- (b) An edit applied by another student fixing the same fault. formations to fix any new students’ submissions that exhibit similar types of faults. We say that a submission is “fixed” if it passes the set of tests provided by course instructors. In our evaluation conducted on 4 programming tasks performed by 720 students, REFAZER synthesizes transformations that fix incorrect submissions for 87% of the students. (c) Similar tree edits applied in (a) and (b), respectively. Each edit inserts a concrete subtree to the right hand side of the ∗ Our second experiment is motivated by the fact that certain operator. The two edits share the same structure but involve repetitive tasks occurring during software evolution, such as different variables and expressions. complex forms of code refactoring, are beyond the capabilities of current IDEs and have to be performed manually [15], [16]. In this experiment, we use repetitive code edits applied by developers to the same project to synthesize a program transformation that can be applied to other locations in the (d) A rewrite rule that captures the two edits in (a) and (b). code. We performed a study on three popular open-source C# Fig. 1: An example of a common fault made by different projects (Roslyn [17], Entity Framework [18], and NuGet [19]) students, two similar edits that can fix different programs, and to identify and characterize repetitive code transformations. In a program transformation that captures both edits. our evaluation conducted on 59 scenarios of repetitive edits, REFAZER learns the intended program transformation in 83% of the cases using 2.8 examples on average. The learned trans- formations are applied to as many as 60 program locations.
- foreach(varm in modifiers){if(m.CSharpKind() == modifier) return true;}; + foreach(varm in modifiers){if(m.IsKind(modifier)) return true;}; Fig. 2: Repetitive edits applied to the Roslyn source code to perform a refactoring. when a test fails. To provide more detailed feedback, (e.g., fault 26 locations in the source code. However, the transformation location or its description), teachers typically compile a rubric generated by REFAZER applied this edit to 689 more locations. of common types of faults, and detect them with simple checks. After we presented the results to the Roslyn developers, they With a large variety of possible faults, manually implementing confirmed that the locations discovered by REFAZER should these checks can be laborious and error-prone. have been covered in the original commit. However, many faults are common and exhibit them- selves in numerous unrelated student submissions. Consider III.TECHNIQUE the Python code in Figure 1(a). It describes two submission In this section, we describe our technique for synthe- attempts to solve a programming assignment in the course sizing program transformations from input-output examples. “The Structure and Interpretation of Computer Programs” REFAZER builds on PROSE [14], a framework for program (CS61A) at UC Berkeley1, an introductory programming class synthesis from examples and under-specifications. with more than 1,000 enrolled students. In this assignment, the student is asked to write a porgram that computes the In PROSE, an application designer defines a domain- product of the first n terms, where term is a function. The specific language (DSL) for the desired tasks. The synthesis original code, which includes line 4 instead of line 5, is an problem is given by a spec ϕ, which contains a set of program incorrect submission for this assignment, and the subsequent inputs and constraints on the desired program’s outputs on student submission fixes it by replacing line 4 with line 5. these inputs (e.g., examples of these outputs). PROSE synthe- Notably, the fault illustrated in Figure 1 was a common fault sizes a set of programs in the DSL that is consistent with ϕ, affecting more than 100 students in the Spring semester of using a combination of deduction, search, and ranking: 2016 and Figure 1(b) shows a recursive algorithm proposed • Deduction is a top-down walk over the DSL grammar, by a different student, which contained the same fault. which iteratively backpropagates the spec ϕ on the de- To alleviate the burden of compiling manual feedback, sired program to necessary specs on the subexpressions we propose to automatically learn the rubric checks from of this program. In other words, it reduces the synthesis the student submissions. Existing tools for such automatic problem to smaller synthesis subproblems using a divide- learning [20], [1] cannot generate a transformation that is and-conquer dynamic programming algorithm over the general enough to represent both the edits shown in Figure 1(c) desired program’s structure. due to their limited forms of abstraction. In REFAZER, this • Search is an enumerative algorithm, which iteratively transformation is described as a rewrite rule shown in Fig- constructs candidate subexpressions in the grammar and ure 1(d). This rewrite rule pattern matches any subtree of the verifies them for compliance with the spec ϕ [21]. program’s AST whose root is a operation with a variable * • Ranking is a process of picking the most robust program as the second operand, and inserts a term application on top from the synthesized set of programs that are consistent of that variable. Notice that the rewrite rule abstracts both a with ϕ. Because examples are highly ambiguous, such a variable name and the first operand of the operator. * set may contain up to 1020 programs [14], and quickly eliminating undesirable candidates is paramount for a B. Repetitive codebase edits user-friendly experience. We now illustrate how REFAZER automates repetitive codebase editing. The following example is found in Roslyn, REFAZER consists of three main components, which are the Microsoft’s library for compilation and code analysis for illustrated in Figure 3: C# and VB.NET [17]. Consider the edits shown in Fig- • A DSL for describing program transformations. It con- ure 2, where, for every instance of a comparison with an tains operators that allow partially abstracting edits pro- CSharpKind object returned by the method , the developer vided as examples. The DSL is expressive enough for == replaces the operator with an invocation of the new method representing common transformations but restrict enough IsKind , and passes the right-hand side expression as the to allow efficient synthesis. method’s argument. Such refactoring is beyond the abilities of existing IDEs due to its context sensitivity. In contrast, • Witness functions. In PROSE, a witness function ωF is REFAZER generalizes the two example edits in Figure 2 to the a backpropagation procedure, which, given a spec ϕ on intended program transformation, which can then be applied a desired program on kind F (e), deduces a necessary to all other matching AST subtrees in the code. (or even sufficient) spec ϕe = ωF (ϕ) on its subex- pression e. 3 Witness functions enable efficient top-down When we analyzed the commit 8c146442 in the Roslyn synthesis algorithms of PROSE. repository, we observed that the developer applied this edit to 3 Another view on witness functions ωF is that they simply implement 1http://cs61a.org/ inverse semantics of F , or a generalization of inverse semantics w.r.t. some 2https://github.com/dotnet/roslyn/commit/8c14644 constraints on the output of F instead of just its value. Fig. 3: The workflow of REFAZER. It receives an example-based specification of edits as input, and returns a transformation. transformation ::= Transformation(rule1,..., rulen) of this subsection, we focus on the semantics of rewrite rules, rule ::= Map(λx → operation, locations) which produce the suggested edits. locations ::= Filter(λx → Match(x, match), AllNodes()) A rewrite rule consists of two parts: a location expres- match Context(pattern, path) ::= sion and an operation. A location expression is a Filter pattern ::= token | Pattern(token, pattern1,..., patternn) operator on a set of sub-nodes of a given AST. Its predicate token ::= Concrete(kind, value) | Abstract(kind) λ x → Match(x, Context(pattern, path)) matches each path ::= Absolute(s) | Relative(token, k) sub-node x with a pattern expression. Insert( , , ) operation ::= x ast k Context | Delete(x, ref ) Patterns A pattern expression (pattern, path) | Update(x, ast) checks the context of the node x against a given pattern. | InsertBefore(x,ast) Here pattern is a combination of Concrete tokens (which ast ::= const | ref match a concrete AST) and Abstract tokens (which match only the AST kind). In addition, a path expression specifies const ::= ConstNode(kind, value, ast ,..., ast ) 1 n the expected position of x in the context that is described by Reference( , , ) ref ::= x match k pattern, using an notation similar to XPath [22]. This allows
Fig. 4: A core DSL LT for describing AST transformations. for a rich variety of possible pattern-matching expressions, kind ranges over possible AST kinds of the underlying pro- constraining the ancestors or the descendants of the desired gramming language, and value ranges over all possible ASTs. locations in the input AST. s and k range over strings and integers, respectively. Example 1. Figure 5 shows a transformation that describes our running example from Figure 2. This transformation contains one rewrite rule. Its location expression is • Ranking functions. Since example-based specifications Filter(λx → Context(π, Absolute(""))) are incomplete, the synthesized abstract transformation may not perform the desired transformation on other input programs. We specify ranking functions that rank where a transformation based on its robustness (i.e., likelihood π = Pattern( == , Pattern( . , te, tm), te) of it being correct in general). te = Abstract(
1) Find all nodes in x s.t. their surrounding context matches pattern, and they are located at path within that context. 2) Out of all such nodes, extract the kth one. (a) A synthesized AST transformation. Example 2. For our running example from Figure 2, the desired rewrite rule applies the following operation to all nodes while (receiver.CSharpKind() == selected by the location expression from Example 1: SyntaxKind.ParenthesizedExpression) { ... } Update(x, ConstNode( . , `1,
B. Synthesis algorithm We now describe our algorithm for synthesizing AST trans- formations from input-output examples. Formally, it solves the following problem: given an example-based spec ϕ, find a transformation T ∈ LT that is consistent with all examples (Pi,Po) ∈ ϕ. We denote this problem as T ϕ. Recall that the core methodology of PBE in PROSE is deductive synthesis, or backpropagation. In it, a problem of kind F (T1,T2) ϕ is reduced to several subproblems of kinds T1 ϕ1 and T2 ϕ2, which are then solved recursively. Here ϕ1 and ϕ2 are fresh specs, which constitute necessary (or even sufficient) constraints on the subexpressions T1 and T2 in order for the entire expression F (T1,T2) to satisfy ϕ. In other words, the examples on an operator F are backpropagated to examples on the parameters of F .
As discussed previously, the backpropagation algorithm (c) A list of edits produced after instantiating (a) to (b). relies on a number of modular operator-specific annotations called witness functions. Even though PROSE includes many generic operators with universal witness functions out of Fig. 5: An example of a synthesized transformation and its the box (e.g. list-processing Filter), most operators in application to a C# program, which results in a list of edits. LT are domain-specific, and therefore require non-trivial domain-specific insight to enable backpropagation. The key The main challenges that lie in backpropagation for part of this process is the witness function for the top-level Transformation are: Transformation operator. 1) Given an input-output example (P ,P ), which often Transformation rule rule i o The operator ( 1,..., n) takes represents the entire codebase/namespace/class, find ex- as input a list of rewrite rules and produces a transformation amples of individual edits in the AST of P . that, on a given AST, applies these rewrite rules in all appli- i cable locations, producing a list of edits. The backpropagation 2) Partition the edits into clusters, deducing which of them problem for it is stated in reverse: given examples ϕ of edits were obtained by applying the same rewrite rule. performed on a given AST, find necessary constraints on the 3) For each cluster, build a set of operation examples for the rewrite rules rule1,..., rulen in the desired transformation. corresponding rewrite rule. Algorithm 1 Backpropagation procedure for the DSL operator • Among patterns with non-empty context, favor the shorter Transformation(rule1,..., rulen). ones. Even though context helps prevent underfitting (i.e., Require: Example-based spec ϕ false positive matches), over-specializing to large contexts 1: result ← dictionary for storing examples for each input may lead to overfitting (i.e., false negative matches). 2: for all (Pi,Po) in ϕ do 3: examples ← empty list of refined examples for edits IV. EVALUATION 4: operations ← TREEEDITDISTANCE(Pi, Po) 5: components ← CONNECTEDCOMPONENTS(operations) In this section, we present two empirical studies to evaluate 6: connectedOpsByEdits ← DBSCAN(components) REFAZER, our technique for learning program transformations. 7: for all connectedOps ∈ connectedOpsByEdits do First, we present an empirical study on learning transforma- 8: ruleExamples → connectedOps.MAP( tions for fixing student submissions to introductory Python ops → ops create a single concrete operation based on ) programming assignments (Section IV-A). Then, we present an 9: examples.Add(ruleExamples) 10: end for evaluation of REFAZER on learning transformations to apply 11: result[Pi].Add(examples) repetitive edits to open-source C# projects (Section IV-B). The 12: end for experiments were performed on a PC Core i7 and 16GB of 13: return result RAM, running Windows 10 x64 with .NET Framework 4.6.
A. Fixing introductory programming assignments Finding individual edits We resolve challenge 1 by calculat- In this study, we use REFAZER to learn transformations ing tree edit distance between P and P . Note that the state-of- i o that describe how students modify an incorrect piece of code the-art Zhang-Shasha tree edit distance algorithm [24] manipu- to obtain a correct one. We then measure how often the learned lates single nodes, whereas our operations (and, consequently, transformations can be used to fix the incorrect code submitted examples of their behavior) manipulate whole subtrees. Thus, by other students. Transformations that can be applied across to construct proper examples for operations in L , we group T students are valuable because they can be used to automati- tree edits computed by the distance algorithm into connected cally generate hints to students on how to fix bugs in their components. A connected component of node edits represents code; alternatively, they can also help TAs with writing better a single edit operation over a subtree. manual feedback. We focus our evaluation on the transfer of Partitioning into rewrite rules To identify subtree edits that transformations, and leave the evaluation of hint generation to were performed by the same rewrite rule, we use the DB- future work. SCAN [25] clustering algorithm to partition edits by similarity. Our goal is to investigate both the overall effectiveness of Here we conjecture that components with similar edit distances our technique, and to what extent learned transformations in an constitute examples of the same rewrite rule. education scenario are problem-specific, or general in nature. Algorithm 1 describes the steps performed by the witness If most transformations are general purpose, instructors might function for Transformation. Lines 2-6 perform the steps be able to provide them manually, once. However, if most described above: computing tree edit distance and clustering transformations are problem-specific, automated techniques the connected components of edits. Then, in lines 7-11, for such as REFAZER will be especially valuable. Concretely, we each similar component, we extract the topmost operation to address the following research questions: create an example for the corresponding rewrite rule. This RQ1 How often can transformations learned from student example contains the subtree where the operation was applied code edits be used to fix incorrect code of other students in the input AST and the resulting subtree in the output AST. who are solving the same programming assignment? RQ2 How often can transformations learned from student C. Ranking code edits be used to fix incorrect code of other students who are solving a different programming assignment? The last component of REFAZER is a ranking function for transformations synthesized by the backpropagation algorithm. Benchmark We collected data from the introductory pro- Since LT typically contains many thousands of ambiguous gramming course CS61A at UC Berkeley. More than 1,000 programs that are all consistent with a given example-based students enroll in this course every semester, which has led spec, we must disambiguate among them. Our ranking function the instructors to adopt solutions common to MOOCs such as selects a transformation that is more likely to be robust on video lessons and autograders. For each homework problem, unseen ASTs – that is, avoid false positive and false negative the teachers provide a black-box test suite and the students matches. It is based on the following principles: use these tests to check the correctness of their programs. The system logs a submission whenever the student runs • Favor Reference over ConstNode: a transformation the provided test suite for a homework assignment. This log that resues a node from the input AST is more likely to thus provides a history of all submissions. Our benchmark satisfy the intent than a transformation that constructs a comprises 4 assigned problems (see Table I). For each problem, constant AST. students had to implement a single function in Python. We • Favor patterns with non-root paths, that is patterns that filtered the log data to focus on students who had at least consider surrounding context of a location. A transforma- one incorrect submission, which is required to learn a trans- tion that selects its locations based on surrounding context formation from incorrect to correct state. We analyzed 21,781 is less likely to generate false positives. incorrect submissions, from up to 720 students. TABLE I: Our benchmarks and incorrect student submissions. TABLE II: Summary of results for RQ1. “Submissions” = mean (SD) of incorrect submissions per student; “students” = Assignment Students Incorrect submissions % of students that got their solution fixed by REFAZER; Product product of the first n terms 549 6,410 “fixed” = mean (SD) of submissions required to find the fix. Accumulate fold-left of the first n terms 668 6,410 Repeated function composition, depth n 720 9,924 Batch Incremental P3 Assignment Submissions G G(n) = i=1 i · G(n − i) 379 2,229 students fixed students fixed Product 5.3 (8.2) 501 (91%) 3.57 (6.1) 247 (45%) 4.1 (6.7) Accumulate 8.9 (10.5) 608 (91%) 5.4 (7.9) 253 (38%) 7.5 (9.8) Experimental setup For each problem, each student in the data Repeated 12.7 (15.3) 580 (81%) 8 (10.3) 340 (47%) 9.6 (11.5) set submitted one or more incorrect submissions and, eventu- G 5.5 (9.4) 319 (84%) 1.4 (1.7) 174 (46%) 4.1 (7) ally, a correct submission. We used the last incorrect submis- Total 8.7 (12) 2,008 (87%) 5.2 (8.1) 1,014 (44%) 6.8 (9.7) sion and the correct submission as input-output examples to synthesize a program transformation and used the synthesized absence of data from previous semesters but using existing data transformation to attempt fixing other student submissions. By can double its effectiveness. Table II summarizes the results selecting a pair of incorrect and correct submissions, we learn for the Batch and the Incremental scenarios. a transformation that changes the state of the program from incorrect to correct, fixing existing faults in the code. Students Although we only used students’ last incorrect submis- may have applied additional edits, such as refactorings, though. sions together with their corresponding correct submissions The transformation thus may contain unnecessary rules to as examples for synthesizing transformations, we could find a fix the code. By learning from the last incorrect submission, transformation to fix student solutions 3.5 submissions before we increase the likelihood of learning a transformation that the last incorrect submission, on average. This result suggests is focused on fixing the existing faults. We leave for future that REFAZER can be used to provide feedback to help work the evaluation of learning larger transformations from students before they know how to arrive at a correct solution earlier incorrect submissions to correct submissions, and how themselves. Additionally, providing feedback about mistakes these transformations can help fixing larger conceptual faults can be more important for students who struggle with the in the code. We used the teacher-provided test suites to check assignments. Figure 6 shows the 50 students who had the most whether a program was fixed. submission for the two hardest assignments in our benchmark. Each column shows chronological submissions for one student, For our first research question, we considered two different with the earliest submissions at the top, and the eventual scenarios: Batch and Incremental. In the Batch scenario, for correct submission at the bottom. Red indicates an incorrect each programming assignment, we synthesize transformations submission; blue shows the first time REFAZER was able to for all but one students in the data set and use them to fix automatically fix the student’s code (we only show the earliest the incorrect submissions of the single remaining student, in time and do not re-test subsequent incorrect submissions). As a leave-one-out cross-validation – i.e., we attempt fixing the we can see in the charts, students took dozens (up to 148) submission of a student using only transformations learned submissions. In many cases, our technique already provided a using submissions of other students. This scenario simulates fix after the student attempted half of the submissions. the situation in which instructors have data from one or more previous semesters. In the Incremental scenario, we sort our The transformations learned by REFAZER contain edits data set by submission time and try to fix a submission using with different granularity, ranging from edits to single nodes only transformations learned from earlier timestamps. This in the AST, e.g., updating a constant, to edits that add multiple scenario simulates the situation in which instructors do not statements, such as adding a base case, a return statement, or have previous data. Here the effectiveness of the technique even replacing an iterative solution by a recursive one. We also increases over time. For our second research question, we use noticed transformations containing multiple rules that represent all transformations learned in Batch from one assignment to multiple mistakes in the code. attempt fixing the submissions for a different assignment. Learned transformations are not useful among different In general, each synthesized rule in the transformation programming assignments Using the transformations learned may be applicable to many locations in the code. In our from other assignments, we were able to fix solutions for only experiments, we try to apply each synthesized transformation 7–9% of the students, which suggests that most transforma- to at most 500 combinations of locations. If the transformation tions are problem-specific and not common among different can be applied to further locations, we simply abort and assignments (see Table III). The results suggest that different proceed to the next program transformation. assignments exhibit different fault patterns; therefore, problem- specific training corpora are needed. This finding also suggest Learned transformations are useful within the same pro- that other automatic grading tools that use a fixed or user- gramming assignments In the Batch scenario, REFAZER provided fault rubric (e.g., AutoGrader [9]) are not likely to generated fixes for 87% of the students. While, on average, work on arbitrary types of assignments. students took 8.7 submissions to finish the assignment, the transformations learned using REFAZER fixed the student Qualitative Feedback from Teaching Assistant To validate submissions after an average of 5.2 submissions. In the In- the quality of the learned transformations, we built a user cremental scenario, REFAZER generated fixes for 44% of the interface that allows one to explore, for each transformation, students and required, on average, 6.8 submissions to find a fix. the incorrect submissions that can be fixed with it. We asked The results suggest that the technique can be useful even in the a Teaching Assistant (TA) of the CS61a course to analyze the student B. Applying repetitive edits to open-source C# projects 1 5 9 13 17 21 25 29 33 37 41 45 49 0 first submission In this study, we use REFAZER to learn transformations submission time
-20 that describe simple edits that have to be applied to many locations of a C# code base. We then measure how often the
-40 learned transformation is the intended one and whether it is correctly applied to all the required code locations. Concretely,
# submissions -60 we address the following question: final, correct submission -80 RQ3 Can REFAZER synthesize transformations with repetitive edits to large open-source projects? Submissions avoided Student #16 had 64 submissions until they solved the problem. Submissions remaining Our technique found a transformation that would have fixed their code after 24 submissions, or 42 submissions earlier. Benchmark We manually inspected 404 commits from three large open source projects: Roslyn, Entity Framework, and NuGet. The projects’ size range from 150,000 to 1,500,000 (a) Assignment “Accumulate” lines of code. We consider an edit to be repetitive if it is applied
student to more than two locations in the codebase. We identified 59
1 5 9 13 17 21 25 29 33 37 41 45 49 distinct scenarios of repetitive edits: 27 in Roslyn, 15 in Entity 0 first submission Framework, and 17 in NuGet. The number of edited locations submission time in each scenario ranges from 3 to 60, with a median of 5. -40 Each project contains at least one scenario with more than 19 edited locations. In 15 (25%) out of the 59 scenarios, there -80 are edited locations in more than one file, which are harder to handle correctly for developers. Finally, in 40 (68%) out of # submissions -120 final, correct the 59 scenarios, the edits are complex and context-dependent, submission meaning that a simple search/replace strategy is not enough to -160 correctly apply the edits to all the necessary locations. (b) Assignment “Repeated” Experimental setup We use edits described in the diff in- formation of each scenario as examples of repetitive edits. To Fig. 6: Analysis of the first time REFAZER can fix a student find the number of examples needed for REFAZER to perform submission for the 50 students with the highest number of all the repetitive edits in a commit, we start with a single edit, attempts in two benchmark problems. Blue are submissions then run REFAZER. If only a subset of locations are found, we that might be avoided by showing appropriate feedback from iteratively add more examples from the diff. In this process, a fix generated by REFAZER. we prioritize variety, choosing examples that cover different variations of the transformation. If the edits performed by REFAZER and the edits in the diff do not match, we manually TABLE III: Summary of results of re-using transformations on inspect them to check whether the developer missed a location different assignments. or the locations were incorrectly edited. Original Assignment Target Assignment Helped students Results Table IV summarizes our results. In our evaluation, Product G 28 (7%) REFAZER synthesized transformations for 58 out of 59 sce- Accumulate G 33 (9%) narios. In 38 (64%) scenarios, the synthesized transformations applied the same edits applied by developers, whereas, in 21 scenarios, the transformations applied more edits than fixes found using REFAZER. The TA confirmed that fixes were developers did. We manually inspected these scenarios, and generally appropriate, but also reported some issues. First, a conclude that 11 transformations were correct (i.e., developers single syntactic transformation may represent multiple distinct missed some edits). We reported them to the developers of the mistakes. For instance, a transformation that changes a literal respective projects. So far, they confirmed 1 of these scenarios. to 1 was related to a bug in the stopping condition of a while loop in one student’s code; and also to a bug in the initial value In 10 scenarios (17%) the additional edits were incorrect, of a multiplication which would always result in 0 in another and revealed two limitations of the current DSL. First, some student’s code. In this case, the TA found it hard to provide a edits require further analysis to identify the location to apply meaningful description of the fault beyond “replace 0 to 1”. If them. For instance, in scenario 4, developers edited a local fixes are used to generate feedback, TAs will need additional variable declaration. However, they did not perform the edit tools to merge or split clusters of student submissions. when this variable was reassigned to another object after its declaration. Extending our DSL to support this kind of Finally, our technique relies on test cases for evaluating the operation would require some form of data flow analysis. correctness of fixed programs. Although some fixed programs The second limitation is related to our tree pattern matching. passed the test cases, when test cases were incomplete, some Some examples produced templates that were too general. For faults remained. While reliance on test cases is a fundamental example, if two nodes have different numbers of children, we limitation, when fixes are reviewed in an interactive setting as can currently only match them with respect to their type, which with our TA, our technique can be used to discover the need may be too general. To support this kind of pattern, we plan to for more test cases for particular assignments. include additional predicates in our DSL such as Contains, TABLE IV: Summary of the evaluation on repetitive edits. which does not consider the entire list of children, but checks Scope = scope of the transformation; Ex = number of ex- if any of the children matches a specific pattern. amples; Dev. = number of locations modified by developers; Our technique, on average, required 2.8 examples for syn- REFAZER = number of locations modified by REFAZER. thesizing all transformations in a diff. The number of required Outcomes: = it performed the same edits as the developers; examples may vary based on the examples selected by the # = it performed more edits than the developers (manually developer. Additionally, changes in the ranking functions for validated as correct); = it performed incorrect edits; “—” = giving preference to more general patterns over more restrict it did not synthesize a transformation. ones can also influence the number of examples. We leave a further investigation of example ordering and of our ranking Id Project Scope Ex. Dev. REFAZER Outcome system to future work. 1 EF Single file 2 13 13 2 EF Multiple files 2 4 200 Threats to validity With respect to construct validity, our 3 EF Single file 3 10 10 initial baseline is the diff information between commits. Some 4 EF Multiple files 2 15 19 repetitive edits may have been performed across multiple 5 EF Single file 3 4 4 commits, or developers may not have changed all possible code 6 EF Single file 2 3 3 7 EF Single file 2 3 3 fragments. Therefore, there may be more similar edits in each 8 EF Single file 2 18 18 scenario that were not considered. To reduce this threat, we 9 EF Single file 2 8 35 # manually inspect the additional edits applied by REFAZER. 10 EF Single file 2 4 10 # Concerning internal validity, the selection of example may 11 EF Multiple files 4 8 8 influence the total number of examples needed to perform 12 EF Single file 2 3 3 13 EF Single file 2 12 12 the transformation. As we mentioned, we prioritized examples 14 EF Multiple files 5 5 5 that cover different patterns the transformation should consider. 15 EF Single file 2 3 3 Finally, the sample of repetitive changes may not be represen- 16 NuGet Single file 2 4 4 tative for other kinds of software systems. 17 NuGet Multiple files 2 4 21 18 NuGet Single file 3 3 3 19 NuGet Multiple files 2 31 44 # V. RELATED WORK 20 NuGet Single file 2 3 3 21 NuGet Multiple files 5 8 14 # Example-based program transformations Meng et al. [1], 22 NuGet Single file 3 14 27 [20], [26] propose Lase, a technique for performing repet- 23 NuGet Single file 4 4 4 itive edits using examples. Developers give two or more 24 NuGet Multiple files 2 5 6 25 NuGet Single file 2 3 3 edited methods as examples, and Lase creates a context- 26 NuGet Single file 3 5 5 aware abstract transformation. It uses clone detection and 27 NuGet Single file 2 3 3 dependence analysis techniques to identify methods where the 28 NuGet Single file 3 4 4 transformation should be applied and its context. Lase only 29 NuGet Single file 2 9 44 abstracts names of types, variables, and methods and can only 30 NuGet Single file 2 4 4 31 NuGet Multiple files 3 4 10 # find edits that mismatch with respect to these names. For 32 NuGet Multiple files 3 12 77 instance, Lase cannot abstract the edits shown in Figures 1 33 Roslyn Single file 3 3 21 # and 2 since there are mismatches on expressions. Additionally, 34 Roslyn Multiple files 4 7 7 the edits in Lase have statement-level granularity, therefore 35 Roslyn Multiple files 3 17 18 # 36 Roslyn Single file 2 6 6 limiting the type of expressible patterns. Finally, Lase cannot 37 Roslyn Single file 2 9 9 apply transformations that perform similar edits in the same 38 Roslyn Multiple files 4 26 715 method as shown in Figure 2. We plan to investigate the 39 Roslyn Single file 3 4 4 use of dependence-analysis to improve the quality of the 40 Roslyn Single file 2 4 4 transformations synthesized by REFAZER. 41 Roslyn Single file 6 14 14 42 Roslyn Single file 5 60 — — Other approaches allow expressing program transforma- 43 Roslyn Single file 2 8 8 44 Roslyn Multiple files 3 15 15 tions in a semi-automated way by using examples in com- 45 Roslyn Single file 2 7 7 binations with transformation templates [27], [28]. Unlike 46 Roslyn Single file 4 13 14 these techniques our approach is fully automated. Feser et 47 Roslyn Single file 2 12 12 al. [29] propose a technique for synthesizing data-structure 48 Roslyn Single file 2 4 4 transformations from examples in functional programming 49 Roslyn Single file 2 5 5 50 Roslyn Single file 2 3 3 languages. Nguyen et al. [30] present LibSync, a technique 51 Roslyn Single file 3 11 11 that migrates APIs based on clients that already migrated. 52 Roslyn Single file 2 5 5 Tansey and Tilevich [31] present an example-based technique 53 Roslyn Single file 2 3 5 # to migrate APIs that are based on annotations. HelpMeOut [32] 54 Roslyn Single file 3 5 5 55 Roslyn Single file 3 3 3 learn from examples transformations to fix compilation and 56 Roslyn Single file 3 6 6 run-time errors. Unlike these techniques, REFAZER is not 57 Roslyn Multiple files 2 15 49 # tailored to a specific domain. 58 Roslyn Single file 2 4 7 # 59 Roslyn Single file 3 4 9 Code completion techniques recommend code transforma- tion to developers while they are editing the source code. Raychev et al. [33] use data collected from large code repos- itories to learn likely code completions. Similarly, Foster et al. [34] use a large dataset of common code completions and efficiently search the space of all programs to find one that recommend them to the user based on the code context. Ge behaves correctly. The most prominent search techniques are et al. [35] propose a similar technique for auto-completing enumerative or data-driven. GenProg uses genetic program- a refactoring manually started by the developer. While these ming to repeatedly alter the incorrect program in the hope to techniques are limited by the refactorings present in IDEs and make it correct [44]. Data-driven approaches leverage large in the datasets, REFAZER can automate transformations that online code repositories to synthesize likely changes to the have never been seen before. input program [45]. Prophet [46] is a patch generation system that learns a probabilistic application-independent model of Inductive programming Inductive programming (IP), also correct code from a set of successful human patches. Qlose known as Programming-by-Example, has been an active re- provides ways to rank possible repairs based on a cost met- search area in the AI and HCI communities for over a ric [47]. Unlike these techniques, which use a global model decade [36]. IP techniques have recently been developed for of possible code transformations, REFAZER learns program- various domains including interactive synthesis of parsers [37], specific transformations using examples of code modification imperative data structure manipulations [38], and network poli- — i.e., from both the original and the modified program. cies [39]. Recently, it has been successfully used in industry by FlashFill and FlashExtract [11], [12], [40]. FlashFill is a feature in Microsoft Excel 2013 that uses IP methods to VI.CONCLUSIONS automatically synthesize string transformation macros from We presented REFAZER, a technique for synthesizing syn- input-output examples. FlashExtract is a tool for data extrac- tactic program transformations from examples. Given a set of tion from semi-structured text files, deployed in Microsoft examples consisting of program edits, REFAZER synthesizes a PowerShell for Windows 10 and as the Custom Field and program transformation that is consistent with the examples. Custom Log features in Operations Management Suite (a Our synthesizer builds on the state-of-the-art program synthe- Microsoft log analytics tool). The DSL of REFAZER is inspired sis engine PROSE. To enable it, we develop (i) a novel DSL by the ones of FlashExtract and FlashFill. While FlashFill uses for representing program transformations, (ii) domain-specific the ConstString operator to create new strings and the constraints for the DSL operators, which reduce the space of SubString operator to get substrings from the input string, search for transformations, and (iii) ranking functions for trans- we use NewNode and Reference operators to compose formation robustness, based on the structure of the synthesized the new subtree using new nodes or nodes from the existing transformations. We evaluated REFAZER on two applications: AST. On the other hand, our DSL contains specific operators synthesizing program transformations that describe how stu- for performing tree edits and tree pattern matching. FlashFill dents “fix” their programming assignments and synthesizing and FlashExtract gave rise to PROSE, a novel framework program transformations that apply repetitive edits to large of effective methods for IP [14]. While PROSE has been code bases. Our technique learned program transformations primarily used in the data wrangling domain, our technique that automatically fixed the program submissions of 87% of shows its applicability to a novel unrelated domain – learning the students participating in a large UC Berkeley class and it program transformations. learned the transformations necessary to apply the correct code Synthesis for education Singh et al. [9] propose AutoGrader, edits for 83% of the repetitive tasks we extracted from three a program synthesis technique for fixing incorrect student large code repositories. submissions. Given a set of transformations that represent fixes As future work, we plan to increase the expressiveness for student mistakes (error-model) and an incorrect submission, of our tree pattern expressions to avoid selecting incorrect AutoGrader uses symbolic execution to try all combinations locations due to over-generalization. We aim at investigating of transformations to fix the student submission. While Au- the use of control-flow and data-flow analyses for identifying toGrader requires an error model, REFAZER automatically the context of the transformation, and the inclusion of negative generates it from examples of fixes. In the future, we plan to examples and operators to specify undesired transformations. use the symbolic search of AutoGrader to efficiently explore In the context of HCI research, we want to design new user all transformations learned by REFAZER. interaction models to enable instructors and developers to Rivers and Koedinger [41] propose a data-driven technique apply and debug the results of synthesized transformations. for hint generation. The main idea is to generate concrete In the education domain, we aim at developing new tools for edits from the incorrect solution to the closest correct one. providing personalized feedback for students based on the fixes While they focus on comparing the entire AST, which can have learned by REFAZER. many differences, our technique generalizes transformations In addition to being a useful tool, REFAZER makes two that fix specific mistakes in student submissions. Kaleeswaran novel achievements in PBE. First, it is the first application et al. [42] propose a semi-supervised technique for feedback of backpropagation-based PBE methodology to a domain generation. The technique clusters the solutions based on unrelated to data wrangling or string manipulation. Second, the strategies to solve it. Then instructors manually label in in its domain it takes a step towards development of fully each cluster one correct submission. They formally validate unsupervised PBE, as it automates extraction of input-output the incorrect solutions against the correct one. Although our examples from the datasets (that is, students’ submissions technique is completely automatic, we plan to investigate the or developers’ modifications). We hope that with our future use of formal verification to validate the transformations. work on incorporating flow analyses into witness functions, Program repair Automated program repair is the task of REFAZER will become the first major application of inductive automatically changing incorrect programs to make them meet programming that leverages research developments from the a desired specification [43]. One of the main challenges is to entire field of software engineering. REFERENCES [26] N. Meng, L. Hua, M. Kim, and K. S. McKinley, “Does automated refactoring obviate systematic editing?” in Proceedings of the 37th [1] N. Meng, M. Kim, and K. S. McKinley, “LASE: Locating and applying International Conference on Software Engineering - Volume 1, ser. ICSE systematic edits by learning from examples,” in Proceedings of the ’15. Piscataway, USA: IEEE Press, 2015, pp. 392–402. 35th International Conference on Software Engineering, ser. ICSE ’13. [27] R. Robbes and M. Lanza, “Example-based program transformation,” in Piscataway, USA: IEEE Press, 2013, pp. 502–511. Model Driven Engineering Languages and Systems, ser. Lecture Notes [2] L. Wasserman, “Scalable, example-based refactorings with refaster,” in in Computer Science. Springer Berlin Heidelberg, 2008, vol. 5301, Proceedings of the 2013 ACM Workshop on Workshop on Refactoring pp. 174–188. Tools, ser. WRT ’13. New York, USA: ACM, 2013, pp. 25–28. [28] M. Boshernitsan, S. L. Graham, and M. A. Hearst, “Aligning devel- [3] Microsoft, “Visual Studio,” 2016, at https://www.visualstudio.com. opment tools with the way programmers think about code changes,” [4] The Eclipse Foundation, “Eclipse,” 2016, at https://eclipse.org/. in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ser. CHI ’07. New York, USA: ACM, 2007, [5] JetBrains, “ReSharper,” 2016, at https://www.jetbrains.com/resharper/. pp. 567–576. [6] Synopsys, Inc., “Coverity,” 2016, at http://www.coverity.com/. [29] J. K. Feser, S. Chaudhuri, and I. Dillig, “Synthesizing data structure [7] Google, “Error-prone,” 2016, at http://errorprone.info/. transformations from input-output examples,” Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and [8] Google, “Clang-tidy,” 2016, at http://clang.llvm.org/extra/clang-tidy/. Implementation, vol. 50, no. 6, pp. 229–239, 2015. [9] R. Singh, S. Gulwani, and A. Solar-Lezama, “Automated feedback [30] H. A. Nguyen, T. T. Nguyen, G. Wilson, Jr., A. T. Nguyen, M. Kim, and generation for introductory programming assignments,” Proceedings of the 34th ACM SIGPLAN Conference on Programming Language Design T. N. Nguyen, “A graph-based approach to API usage adaptation,” in Proceedings of the ACM International Conference on Object Oriented and Implementation, vol. 48, no. 6, pp. 15–26, 2013. Programming Systems Languages and Applications, ser. OOPSLA ’10. [10] S. Gulwani, J. Hernndez-Orallo, E. Kitzelmann, S. H. Muggleton, New York, USA: ACM, 2010, pp. 302–321. U. Schmid, and B. Zorn, “Inductive programming meets the real world,” [31] W. Tansey and E. Tilevich, “Annotation refactoring: inferring upgrade Communications of the ACM, vol. 58, no. 11, pp. 90–99, 2015. transformations for legacy applications,” in Proceedings of the 23rd [11] S. Gulwani, “Automating string processing in spreadsheets using input- Annual ACM SIGPLAN Conference on Object Oriented Programming output examples,” in Proceedings of the 38th Annual ACM SIGPLAN- Systems Languages and Applications, ser. OOPSLA’08. Atlanta, USA: SIGACT Symposium on Principles of Programming Languages, ser. ACM, 2008, pp. 295–312. POPL ’11. New York, USA: ACM, 2011, pp. 317–330. [32] B. Hartmann, D. MacDougall, J. Brandt, and S. R. Klemmer, “What [12] V. Le and S. Gulwani, “FlashExtract: A framework for data extraction would other programmers do: Suggesting solutions to error messages,” by examples,” in Proceedings of the 35th ACM SIGPLAN Conference in Proceedings of the SIGCHI Conference on Human Factors in on Programming Language Design and Implementation, ser. PLDI ’14. Computing Systems, ser. CHI ’10. New York, USA: ACM, 2010, New York, NY, USA: ACM, 2014, pp. 542–553. pp. 1019–1028. [13] D. Edge, S. Gulwani, N. Milic-Frayling, M. Raza, R. Adhitya Saputra, [33] V. Raychev, M. Schafer,¨ M. Sridharan, and M. Vechev, “Refactoring C. Wang, and K. Yatani, “Mixed-initiative approaches to global editing with synthesis,” in Proceedings of the 28th ACM SIGPLAN Interna- in slideware,” in Proceedings of the 33rd Annual ACM Conference on tional Conference on Object Oriented Programming Systems Languages Human Factors in Computing Systems. ACM, 2015, pp. 3503–3512. and Applications, ser. OOPSLA ’13. New York, USA: ACM, 2013, [14] O. Polozov and S. Gulwani, “FlashMeta: A framework for inductive pp. 339–354. program synthesis,” in Proceedings of the ACM international conference [34] S. R. Foster, W. G. Griswold, and S. Lerner, “WitchDoctor: IDE support on Object oriented programming systems languages and applications, for real-time auto-completion of refactorings,” in Proceedings of the ser. OOPSLA ’15. New York, NY, USA: ACM, 2015, pp. 542–553. 34th International Conference on Software Engineering, ser. ICSE ’12. [15] A. Hindle, E. T. Barr, Z. Su, M. Gabel, and P. Devanbu, “On the New Jersey, USA: IEEE Press, 2012, pp. 222–232. naturalness of software,” in Proceedings of the 34th International [35] X. Ge, Q. L. DuBose, and E. Murphy-Hill, “Reconciling manual Conference on Software Engineering, ser. ICSE ’12. Piscataway, NJ, and automatic refactoring,” in Proceedings of the 34th International USA: IEEE Press, 2012, pp. 837–847. Conference on Software Engineering, ser. ICSE ’12. Piscataway, USA: [16] G. Kniesel and H. Koch, “Static composition of refactorings,” Science IEEE Press, 2012, pp. 211–221. of Computer Programming, vol. 52, no. 1-3, pp. 9–51, 2004. [36] H. Lieberman, Your wish is my command: Programming by example. [17] Microsoft, “Project Roslyn,” 2011, at https://github.com/dotnet/roslyn. Morgan Kaufmann, 2001. [18] Microsoft, “Entity Framework 6,” at http://www.asp.net/ [37] A. Leung, J. Sarracino, and S. Lerner, “Interactive parser synthesis entity-framework. by example,” in Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation, Portland, OR, [19] Microsoft, “NuGet 2,” at https://github.com/nuget/nuget2. USA, June 15-17, 2015, 2015, pp. 565–574. [20] N. Meng, M. Kim, and K. S. McKinley, “Systematic editing: Generating [38] R. Singh and A. Solar-Lezama, “Synthesizing data structure manipula- program transformations from an example,” in Proceedings of the 32nd tions from storyboards,” in Proceedings of the 19th ACM SIGSOFT ACM SIGPLAN Conference on Programming Language Design and symposium and the 13th European conference on Foundations of Implementation, ser. PLDI ’11. New York, USA: ACM, 2011, pp. software engineering, 2011, pp. 289–299. 329–342. [39] Y. Yuan, R. Alur, and B. T. Loo, “NetEgg: Programming network [21] R. Alur, R. Bodik, G. Juniwal, M. M. Martin, M. Raghothaman, S. A. policies by examples,” in Proceedings of the 13th ACM Workshop on Seshia, R. Singh, A. Solar-Lezama, E. Torlak, and A. Udupa, “Syntax- Hot Topics in Networks, 2014, pp. 20:1–20:7. guided synthesis,” in FMCAD. IEEE, 2013, pp. 1–17. [40] M. Mayer, G. Soares, M. Grechkin, V. Le, M. Marron, O. Polozov, [22] World Wide Web Consortium, “XPath,” 1999, at https://www.w3.org/ R. Singh, B. Zorn, and S. Gulwani, “User interaction models for TR/xpath/. disambiguation in programming by example,” in Proceedings of the [23] M. Pawlik and N. Augsten, “RTED: A robust algorithm for the tree 28th Annual ACM Symposium on User Interface Software & edit distance,” Proceedings of the VLDB Endowment, vol. 5, no. 4, pp. Technology, ser. UIST ’15. New York, USA: ACM, 2015, pp. 291–301. 334–345, 2011. [41] K. Rivers and K. R. Koedinger, “Data-driven hint generation in vast [24] K. Zhang and D. Shasha, “Simple fast algorithms for the editing solution spaces: a self-improving python programming tutor,” Interna- distance between trees and related problems,” SIAM Journal on Com- tional Journal of Artificial Intelligence in Education, pp. 1–28, 2015. puting, vol. 18, no. 6, pp. 1245–1262, 1989. [42] A. K. Shalini Kaleeswaran, Anirudh Santhiar and S. Gulwani, “Semi- [25] M. Ester, H.-P. Kriegel, J. Sander, and X. Xu, “A density-based supervised verified feedback generation,” in Proceedings of the 16th algorithm for discovering clusters in large spatial databases with noise.” ACM SIGSOFT International Symposium on Foundations of Software AAAI Press, 1996, pp. 226–231. Engineering, ser. FSE ’16. New York, NY, USA: ACM, 2016. [43] C. Goues, S. Forrest, and W. Weimer, “Current challenges in automatic [45] V. Raychev, M. Vechev, and E. Yahav, “Code completion with statistical software repair,” Software Quality Journal, vol. 21, no. 3, pp. 421–443, language models,” in Proceedings of the 35th ACM SIGPLAN Confer- 2013. ence on Programming Language Design and Implementation, ser. PLDI [44] C. Le Goues, M. Dewey-Vogt, S. Forrest, and W. Weimer, “A systematic ’14. New York, USA: ACM, 2014, pp. 419–428. study of automated program repair: Fixing 55 out of 105 bugs for $8 [46] F. Long and M. Rinard, “Automatic patch generation by learning each,” in Proceedings of the 34th International Conference on Software correct code,” Proceedings of the 43rd Annual ACM SIGPLAN-SIGACT Engineering, ser. ICSE ’12. Piscataway, USA: IEEE Press, 2012, pp. Symposium on Principles of Programming Languages, vol. 51, no. 1, 3–13. pp. 298–312, 2016. [47] L. D’Antoni, R. Samanta, and R. Singh, Qlose: Program Repair with Quantitative Objectives. Springer International Publishing, 2016, pp. 383–401.