Deuce: Direct Manipulation Source Program Editor

Bachelor’s Thesis Advisor: Ravi Chugh Winter 2017

Grace Lu The University of Chicago [email protected]

1 Introduction Abstract Sketch-n-Sketch (Resources #1, References [1] Commonly used text editors today often and [2]) is a direct manipulation programming require users to manipulate code in a lin- system written in Elm that automatically gener- ear fashion. These editors, such as Sub- ates code when interacting with the system. This lime and vim, provide limited features allows a user to generate a program with mini- for users to easily and quickly restructure mal text editing. Figure 1 depicts the interface of code. Users therefore manually edit code this system. The left side is the text editor or by adding and deleting text or copying code box; the right side is the output canvas. Us- and pasting lines to make changes. The ers are able to draw shapes including lines, rec- goal of Deuce as a direct manipulation tangles, ellipses, and polygons in the output can- text editor is to provide tools to help pro- vas. After drawing the shape, Sketch-n-Sketch grammers improve the readability and automatically generates the code that renders maintainability of code with minimal ef- these shapes, as seen in the code box. Sketch-n- fort, especially as code bases grow in Sketch supports programs written in a simple size. Such tools rely on being able to in- lisp-like programming language called Little teract with the program beyond simply (#3). Users are able to write Little programs in rewriting or reorganizing text. Deuce in- the code box that will generate shapes when the troduces new methods of interacting with program is run, or interact with the output can- the text through keyboard and mouse, in- vas to have code be automatically written. The cluding selection of text and drag-and- interface provides additional features in the out- drop of code elements. These direct ma- put canvas such as editing the size and color of nipulation text editing features are made shapes without needing to manipulate variables possible by the editor keeping track of through text editing in the code box. Users are information beyond just the text that able to invoke these features by clicking on se- makes up the program. This requires an lection zones and using sliders on the shapes. editor that is structure-aware. Deuce is Higher-level features include being able to set not a standalone text editor but rather a variables of shapes equal to each other (Make layer on top of text editors to make cer- Equal), such as making the x coordinate of the tain classes of editing and refactoring rectangle equal to the x coordinate of the line, or tasks easier to perform and less prone to creating a relation between multiple shapes, such errors. This layer gives users the option as one shape being half the width of another (Re- to switch back and forth between tradi- late). tional text editing and interactive pro- gram manipulation. Furthermore, this ed- itor is unique in that it focuses on provid- ing features for refactoring functional code, which lacks the rigid formatting of other programming paradigms. of menu buttons and configuration options, the DNDRefactoring editor provides the user with the ability to select drag sources and drop targets to easily interact with the text program. This drag-and-drop feature allows users to perform program transformations with a single move- ment as opposed to clicking through menu op- tions and selecting configurations. Therefore, users are able to directly interact with variables, expressions, statements, methods, and other ele- Figure 1. Sketch-n-Sketch interface. The left side ments in a program [4]. DNDRefactoring sup- depicts the code box in which users can view and ports two classes of refactorings—within the edit the text of Little programs. The right side Java editor or between the Package Explorer and depicts the output canvas in which users can Outline View. Editor refactorings with drag-and- draw and manipulate shapes, after which the drop include promoting local variable to field, system automatically generates or changes the extracting temp variable, introducing parameter, code that renders the shapes on the left. extracting method, and moving methods or members; refactorings between and within the The Sketch-n-Sketch text editor seen on the left Package Explorer and Outline View include side of the interface uses the Ace code editor moving methods, moving types to files or pack- (#4) written in JavaScript. Ace provides many ages, converting types, and extracting classes. built-in features such as line numbers and scroll- To test the performance of the DNDRefactoring ing. This editor provides the foundation for the tool, the researchers conducted user studies. In Deuce interactive layer. these user studies, the researchers timed how long it took participants to perform refactorings Deuce is a layer built on the text editor portion with and without the DNDRefactoring tool. Re- of Sketch-n-Sketch as an extension of Ace. Simi- sults shows the participants were able to perform lar to the direct manipulation program generator, almost all common refactorings faster in the Deuce aims to minimize time spent text editing DNDRefactoring editor than in the regular to create a more fluid experience when interact- Eclipse interface. In fact, refactorings were per- ing with the program text. This layer allows us- formed on average 3 times faster, showing drag- ers to interact with the text through mouse fea- and-drop to be a more efficient tool for code re- tures including selection and dragging in addi- structuring [4]. tion to traditional text editing. Therefore, Deuce 2.2 Barista is not considered a replacement for text editors but rather an extension. With these new interac- Similarly, Ko and Myers of Carnegie Mellon tion elements, users will be able to more easily University created a programming environment edit and refactor complex code. called Barista to overcome limitations of plain text editors. Barista provides data structures, al- 2 Related Work gorithms, and interactive techniques to augment traditional text editors [3]. These new tools and 2.1 DNDRefactoring interactions will improve usability and utility of Researchers at the University of Illinois at Urba- code editors to increase programmer productivi- na-Champaign created a tool called Drag-and- ty. This framework allows users to move back Drop Refactoring (DNDRefactoring) in the and forth between structured and unstructured Eclipse IDE (integrated development environ- versions of the code. Barista uses a model-view- ment) to support the restructuring of code controller architecture in which the model is an through mouse movements [4]. Their work ap- abstract syntax tree composed of structures and plies specifically to the Java programming lan- tokens, the view is a tree of interactive views, guage and focuses on common code restructur- and the controller is the event handlers. An ab- ing tasks seen in object-oriented programming. stract syntax tree representation of the code is While the Eclipse editor itself does provide re- created internally to be able to maintain the factoring tools, users often experience difficulty structure of a program and also provides a visual finding, invoking, and configuring the features to user interface to enable interactive editing fea- work as intended. To overcome the proliferation tures. With the abstract syntax tree, users of Barista are able to match delimiters and quickly 4.1 Item Identification see errors in code such as improper scoping of The first step in creating a direct manipulation variables since these are easily identifiable when text editor is indexing elements within the pro- the program structure is determined. For gram. This means attaching identifiers to expres- frontend features, the framework allows for rich sions, patterns, and target positions in the pro- annotation metadata to be displayed and pro- gram. This information is needed to be able to vides focused views on specific blocks of code. select elements and also move them around In addition, the environment supports a drag- within a program. Deuce therefore is a structure- type feature to help with movement of code aware editor. Plain text editors such as Ace do blocks. This feature is implemented with the not keep track of the structure of the program. keyboard, with users holding down a modifier key near statements or delimiters and using the Deuce keeps track of expressions, patterns, and arrow keys to select the target location. target positions when parsing through code. IDs 3 Deuce for these items are kept track of in the model. Furthermore, a dependence graph of variables in Deuce (#2) aims to accomplish many of the the code is created. This allows Deuce to keep same goals as DNDRefactoring and Barista in track of dependencies between parts of the pro- creating a more interactive program editor that gram to determine which transformations are provides refactoring features beyond normal valid and which are out of scope. This is similar plain text editing. These additional features that to the abstract syntax tree structure Barista cre- augment traditional text manipulation aim to ates. make refactoring easier and more reliable. 4.2 Item Selection: Initial Design Similar to Barista, the Deuce project is split into To enable the user to interact with the program two parts: the frontend user interface and the beyond text manipulation, a selection feature backend program transformations. The program was introduced into the Deuce editor. The first transformations rely on the user interface to be iteration involved the user being able to select an able to trigger backend changes to the code. Sec- item such as an expression, pattern, or target po- tion 4 describes the creation of the Deuce user sition based on the mouse position of the click, interface and the features it provides; Section 5 and then the item would be highlighted using details the program transformations that are Ace editor highlights as seen in Figure 2. Initial- made possible by the frontend. ly, the Ace mouse position was used to identify click positions. Later on, a function was written The focus of the completed work has been the to calculate the exact row and column position of user interface discussed in Section 4. This inter- a mouse click using pixel position for higher ac- active interface is necessary for the program curacy. transformations discussed in Section 5 to be im- plemented.

4 Deuce: User Interface Since Deuce is an added layer built on top of Ace, the state of Deuce is kept track of in the backend Elm code in a model. This model main- tains information about the code box and its con- tents. With this information, the code box can be displayed as intended.

Code items that Deuce keeps track of include Figure 2. Ace highlights. The code box currently expressions, patterns, and target positions. Tar- shows highlights of all items—expressions (or- get positions are spaces before and after expres- ange), patterns (yellow), and target positions sions and patterns. (green). Items that are selected appear in darker shades of the highlight color.

For the selection of larger expressions such as “def” and “let” expressions, which are composed of smaller expressions, the left parentheses were first used for selection. In the next iteration, the keywords themselves (i.e. “def” and “let”) were used for selection to provide the user with a Figure 3. Bounding box for an expression. larger area in which to select the expression. For a list of patterns, the left open bracket allowed 4.4 Item Selection: Improved Design for the selection of the list. The use of SVGs expanded beyond just “def’ and “let” expressions to include all expressions, Target positions are spaces within expressions patterns, and target positions. The use of SVGs and patterns. In addition to allowing for selection replaces the reliance on mouse position to de- of expressions and patterns, target positions are termine selected elements. With SVGs being also selectable by clicking anywhere within the used for item selection, the use of mouse posi- row and column position of the space. These are tion for selection can be replaced by event han- needed when moving pieces of code around the dlers tied to the SVGs. Therefore, all expres- program. sions, patterns, and target positions are indicated with SVGs instead of Ace highlights. For ex- Item selections occur when a user clicks down pressions and patterns, the SVGs are polygons; on the selection area of an item; deselection sim- for target positions, the SVGs are circles. These ilarly occurs when the user clicks on an already circles no longer occupy the entire row and col- selected item. umn space position but rather are centered in the 4.3 Bounding Box middle of the space.

As discussed in Section 4.2 about Item Selec- In the Deuce layer, all polygons and circles that tion, larger expressions such as “def’ and “let” indicate expressions, patterns, and target posi- were only selectable by the keywords them- tions are calculated but hidden until hovered or selves. This greatly limits the selectable area for selected. All of these items are calculated recur- large, nested expressions. Therefore, instead of sively so that larger expressions are below only allowing for selection of “def” and “let” for smaller expressions, with patterns on top of ex- larger expressions and using Ace highlights to pressions and target positions on the very top, on indicate selected items, SVGs (Scalable Vector top of patterns. This allows the user to be able to Graphics) are drawn over items. SVG is a vec- “click through” to larger expressions. See the tor-based image format. This gives greater con- Figures below to better understand this feature. trol over the look of hovered and selected items in Deuce and also allows for event handlers to be attached to these items. Instead of using the mouse position to determine which elements are being hovered over or selected, SVGs allows Deuce to simply attach functions that are execut- ed when the SVG is hovered or clicked. Figure 4. Overlapping expressions. As shown, this function definition includes smaller sub- For the SVGs to be properly drawn over expres- expressions. Highlighted in orange is the larger sions that span multiple lines, the start and end expression. Highlighted in yellow is a nested columns are calculated for each line of the ex- smaller expression. Before the use of SVGs for pression by using the unparsed text representa- selection, the orange expression was only se- tion of lines. Using this information, the points lectable by clicking on the word “def” and the of a polygon to be drawn over the expression can yellow expression was only selectable by click- be calculated. This results in a SVG polygon that ing on the word “let”. Now, the orange expres- bounds the entire expression as seen in Figure 3. sion can be selected by the left parenthesis, word “def”, and space around the target position seen after the word “def” (described in Figures 5-1 and 5-2). In addition, the orange expression can be selected by the right parenthesis, too. Similar- ly, the yellow expression can be selected by its the start and end positions of the mouse are parentheses, the word “let”, and other spaces stored in the model so that the items associated that are not occupied by other SVG elements. with these positions can be identified.

Drag-and-drop is a feature that allows for pro- gram transformations to be performed in a sim- ple and smooth motion. This removes the need for multiple selections or clicking through op- tions. Figure 5-1. Mouse hovering target position cir- cle. 4.6 Deuce Mode Features Given Deuce is an added layer on top of the Ace text editor, features were added to be able to eas- ily switch between Deuce and Ace mode. When the shift key is pressed, the user is in Deuce mode and is able to see selectable items when

Figure 5-2. Mouse hovering the row-column hovering over the item; the user is also able to space position of target position seen in Figure select an item. When the shift key is not pressed, 5-1, but outside of the area of the SVG circle. the code box functions as a normal Ace editor, When the mouse hovers the SVG circle of the though the selected items will still appear in the target position, the user can select the target program (i.e. an expression selected with the position. When the mouse moves out of the target shift key pressed will still appear selected when position but stays in the row-column position of the shift key is not pressed). Deselection of items the space of the target position, the larger ex- can only occur in Deuce mode. In both Deuce pression shown in yellow can be selected if and Ace modes, though, the user can press the clicked. escape key to remove all selected items. This will reset the Deuce state to not contain any se- Before the use of SVGs over expressions, pat- lected elements. terns, and target positions, hovering and selec- 5 Deuce: Program Transformations tion of items was handled by determining which item the mouse position corresponded to in With the Deuce user interface in place, program terms of row-column position. With SVGs, event transformations can take advantage of these in- handlers can be attached to indicate which items teraction features to accomplish general tasks. are currently being hovered over (onMouseOver) Deuce currently supports drag-and-drop defini- or clicked on (onMouseClick). tion reordering described in Section 5.1. The other program transformations discussed in this Since the SVGs are drawn in the Deuce layer, section are currently being implemented. The which overlays the Ace layer, this affects basic program transformations discussed in this sec- Ace features such as scrolling. The position of tion are based on general programming tasks polygons and circles drawn in the Deuce layer often done in functional languages. therefore need to be manually recalculated when a scroll occurs to correctly position the SVGs While the user interface is the main focus of the over selected elements in the code box. work described in this paper, the following pro- gram transformations will be supported by the 4.5 Mouse Drag-and-Drop developed interface features. The figures in this With identification and selection of items in section show how the implemented user inter- place, drag-and-drop of elements is possible. face will allow a user to perform specific refac- Selection allows the user to choose what to be torings. moved where, and identification allows the pro- gram to know which items are selected. A drag 5.1 Definition Reordering in the Deuce layer involves a start position initi- Currently, Deuce supports movement of varia- ated with the click of the mouse and holding this bles through drag-and-drop. Specifically, a user click while moving to an end position where the can click down on a pattern and drag it to a tar- click is released. Throughout this drag motion, get position before an expression or to a target position before or after a pattern. This allows the user to reorder a definition and also have the as- sociated value be moved through a single drag- and-drop motion as opposed to moving the pat- tern name and also the value of the pattern in separate edits of the code.

Figure 8. After dragging the second “x1” onto

Figure 6-1. Selected pattern (orange) and target the first “x1”, the two “x1” variables have been (black circle) position for drag-and-drop. combined into one definition at the top of the program that is used in both places.

Users may also be interested in the reverse ac- tion—splitting definitions when values are no longer shared. For example, if the two places “x1” is used do not share the same value any- Figure 6-2. Program transformation after click- more, the user could drag the “x1” to the places ing on “color” seen in Figure 6-1 and keeping it would be uniquely defined as shown in Figure the mouse down while moving to the target posi- 9. This allows users to easily and quickly rede- tion indicated by the black dot in Figure 6-1 be- fine variable values. fore releasing the mouse. This moved “color” to the target position and also moved the value of color (117) to the associated location. 5.2 Combining and Splitting Definitions This program transformation allows users to cre- ate or remove variables. Combining definitions allows a user to clean up code by creating a re- usable definition. For example, Figure 7 shows two “x1” variables defined in separate places. If the user wants to create a single “x1” variable to be used in both places with the same value, one of the “x1” variables could be dragged on top of Figure 9. Variable to be split. If the value of the other to create the output seen in Figure 8. “x1” is no longer shared, the user should drag The value of global “x1” is then the value of the “x1” to each place it will be uniquely defined. “x1” that was not dragged. Combining defini- The result of this transformation is shown in tions can therefore create more readable and re- Figure 10. usable code.

Figure 10. Splitting variables. The split “x1”

Figure 7. Two variables to be combined. values have the same value but now have a local scope. Therefore, the user can change the value of one without affecting the other. 5.3 Introducing Variables Users may want to create new variables within a program to increase readability of code. For ex- ample, as seen in Figure 11, one variable is an expression defined in terms of another variable. The code may be made cleaner by extracting this variable out and then using the new variable name in its place as seen in Figure 12.

Figure 11. Instance to introduce a new variable.

Figure 13. Repetitive code snippets.

Figure 12. Introducing a new variable. 5.4 Introducing Lambdas When similar pieces of code are frequently re- used within a program, a lambda function can be introduced to reduce the program size. As shown in Figure 13, the definitions for “line1”, “line2”, and “line3” use the same code with different var- iable values. Therefore, a function that parame- terizes the changing values can be created to Figure 14. Introducing a lambda function to re- simplify the code (Figure 14). move repetitive code snippets. 5.5 Adding or Removing Function Argu- To perform this program transformation, the user ments drags the similar definitions on top of one anoth- er to combine them. In the example shown in Building off of the previous program transfor- Figure 13, the user can drag the “line2” defini- mation (Introducing Lambdas), there may be tion on top of the “line1” definition, which certain parameters to the created function or any would create the “line1” function seen in Figure other defined function that should be added and 14. Then the user would drag the “line3” defini- others that should be removed. For example, in tion in Figure 13 on top of the newly created Figures 14 and 15, the “width” variable is set to “line1” function to create the program seen in be 5 for all calls to the “line1” function. The user Figure 14. may instead want “width” to be a parameter to the “line1” function. To invoke this transfor- mation, the user would drag the 5 inside of the “line1” function definition into the parameter list for the function as depicted in Figure 15. This would then change the function to have a “width” parameter and also add an argument value of 5 to each place the “line1” function is called, as seen in Figure 16.

5.6 Reordering Function Arguments To reorder function arguments, users should drag the argument to be moved onto a target po- sition. In the following example in Figure 18, the user wants to move “x2” before “y1” but after “x1”. This results in Figure 19, where the “x2” parameter is after the “x1” parameter, and the corresponding “x2” values in the calls to the Figure 15. Adding a function argument to a func- “line1” function have also moved to the corre- tion. sponding places.

Figure 16. Adding a function argument also edits Figure 18. Reordering a function argument. the function at call-sites.

Similarly, function arguments can also be re- moved. To invoke this transformation, the user should drag the argument to be removed directly on top of the parameter, as seen in Figure 17. The result will then be Figure 14. Note the “width” parameters are also removed at the call- sites. Figure 19. Reordered function argument also While all “width” values in Figure 17 have a changes argument ordering at call-sites. value of 5, the value that is dragged on top of the 5.7 Button Transformations parameter variable will be the value that is set in the function. For example, if the second call to In addition to drag-and-drop program transfor- “line1” in Figure 17 had a “width” value of 6 mations, Deuce can support transformations in- while the other “width” values remained the duced by selection of items and button clicks. same with values of 5, and this 6 value was For example, if the user wants to make two vari- dragged on top of “width”, the “width” value in ables equal, one option is to use the drag-and- Figure 14 would be 6. drop method described in Section 5.2. Another option would be to select the two variables to be set equal and click a “Make Equal” button that becomes available when the selected items satis- fy the requirements of using the button. This se- lection option to perform program transfor- mations can be used in addition or in place of drag-and-drop options. The risk of relying on selection and buttons is the proliferation of men- us in the user interface. The addition of menus, though, could provide the user with more options Figure 17. Removing a function argument. and configurations to produce the exact refactor- ing outcome intended. Nevertheless, drag-and- drop allows the user to easily perform program transformations with a single motion. 6 Future Work ture-aware, keeping track of elements of the pro- gram and their relationships. With an interactive The user interface features implemented for the user interface and useful program transfor- Deuce layer lay the foundation for a wide variety mations, Deuce can improve the programming of refactorings. These refactorings include, but experience by making common tasks simpler are not limited to, the program transformations and quicker to perform. discussed in Section 5 (Deuce: Program Trans- formations). 8 Acknowledgements

On the frontend, the user interface can continual- I would like to thank Professor Ravi Chugh, my ly be iterated on. As discussed in Section 4 thesis advisor, for giving me this opportunity to (Deuce: User Interface), multiple design itera- work on the Sketch-n-Sketch project and for tions were implemented and improved on. Addi- providing continuous guidance throughout the tional features that can be added to the user in- entire endeavor. I would also like to thank Pro- terface include elements to better guide a user fessor Shan Lu for her advice and mentorship. I through Deuce. This may include arrows to indi- am also grateful for the Liew Family College cate where an item may be moved after it has Research Fellows Research Grant for supporting been selected, a before and after comparison of this project. the code once a transformation has been per- formed, and a short pause before the actual code Resources change is made so that it is not so sudden. These would allow a new user to better understand the #1 Sketch-n-Sketch project: features Deuce provides and also give all users https://ravichugh.github.io/sketch-n- more ideas about how to refactor the program. sketch/index.html

On the backend, much of the future work is de- #2 Deuce code (last commit by author): scribed in Section 5. These transformations rely https://github.com/ravichugh/sketch-n- on both the user interface and structure-aware sketch/commit/c794383cfe3e60c8490c739a4e6533af components of Deuce. With more program trans- 7f2fe511 formations in place, users are able to see the full #3 Little language syntax guide: potential of the direct manipulation source pro- gram editor to make editing and refactoring easi- https://github.com/ravichugh/sketch-n- er. sketch/blob/master/README.md #4 Ace Editor: With more features on both the frontend and https://ace.c9.io/#nav=about backend in place, user studies can be conducted to test the performance of Deuce. Similar to the References user studies done by the DNDRefactoring team, users will be asked to perform various program [1] R. Chugh, B. Hempel, M. Spradlin, and J. Albers, transformations in Deuce and in the regular Ace “Programmatic and Direct Manipulation, Together editor. The tests will measure the time needed in at Last,” in Proceedings of the ACM SIGPLAN both cases to complete a task and also the users’ Conference on Programming Language Design opinions. These studies will be useful in improv- and Implementation (PLDI), 2016. ing Deuce’s user interface and also program [2] B. Hempel and R. Chugh, “Semi-Automated SVG transformation features. This feedback will drive Programming via Direct Manipulation,” in Pro- future work. ceedings of the ACM Symposium on User Inter- face Software and Technology (UIST), 2016. 7 Conclusion [3] A.J. Ko and B.A. Myers, “Barista: An Implemen- tation Framework for Enabling New Tools, Inter- Deuce aims to make editing and refactoring of action Techniques and Views in Code Editors”, in programs easier and less prone to errors. This is Proc. ACM Conference on Human Factors in done by creating a layer on top of a traditional Computing Systems, 2006. text editor that provides features that make such tasks easier to invoke and execute. To be able to [4] Y.Y. Lee, N. Chen, R.E. Johnson, “Drag-and- provide these features, the editor must be struc- Drop Refactoring: Intuitive and Efficient Program Transformation,” in ICSE, 2013.