Sketch-n-Sketch: Output-Directed Programming for SVG

Brian Hempel Justin Lubin Ravi Chugh Department of Computer Science, University of Chicago, USA {brianhempel,justinlubin,rchugh} @ uchicago.edu

1/8/2019 Sketch-n-Sketch { a e Sketch-n-Sketch File Code Tools View Options Output Tools Context: Program

Undo Redo Clean Up Run b 1 equiTriPt [x3, y3] [x2, y2] = 2 [ (x2 + x3 + sqrt 3! * (y2 - y3))/ 2!, (y2 + y3 - sqrt 3! * (x2 - x3)) / 2!] c 3 4 oneThirdPt [x3, y3] [x, y] = d 5 [ x / 1.5!+ x3 / 3!, y / 1.5! + y3 / 3!] 6 7 point = [39, 314] 8 9 point2 = [490, 301] 10 11 makeKochPts depth point point2 = 12 let oneThirdPt2 = oneThirdPt point point2 in f 13 let oneThirdPt3 = oneThirdPt point2 point in 14 let equiTriPt2 = equiTriPt oneThirdPt3 oneThirdPt2 in 15 if depth < 2 then 16 [point, oneThirdPt3, equiTriPt2, oneThirdPt2] 17 else 18 let makeKochPts2 = makeKochPts (depth - 1) point oneThirdPt3 in 19 let makeKochPts3 = makeKochPts (depth - 1) oneThirdPt3 equiTriPt2 in 20 let makeKochPts4 = makeKochPts (depth - 1) equiTriPt2 oneThirdPt2 in 21 let makeKochPts5 = makeKochPts (depth - 1) oneThirdPt2 point2 in ⦀ 22 concat [makeKochPts2, makeKochPts3, makeKochPts4, makeKochPts5] 23 24 depth = 3{1-5} 25 26 topPts = makeKochPts depth point point2 27 28 botCorner = equiTriPt point2 point 29 30 rightPts = makeKochPts depth point2 botCorner 31 32 leftPts = makeKochPts depth botCorner point 33 34 snowflakePts = concat [topPts, rightPts, leftPts] 35 36 polygon1 = 37 let pts = snowflakePts in 38 let [color, strokeColor, strokeWidth] = [124, 360, 2] in 39 polygon color strokeColor strokeWidth pts 40 41 svg (concat [ 42 [polygon1] 43 ])

Figure 1. A recursive program built through output-directed program transformations, without any text-based programming.

ABSTRACT tem for creating programs that generate vector graphics. For creative tasks, programmers face a choice: Use a GUI and To enable output-directed interaction at more stages of pro- sacrifice flexibility, or write code and sacrifice ergonomics? gram construction, we expose intermediate execution products for manipulation and we present a mechanism for contex- To obtain both flexibility and ease of use, a number of sys- tual drawing. Looking forward to output-directed program- tems have explored a workflow that we call output-directed ming beyond vector graphics, we also offer generic refactor- programming. In this paradigm, direct manipulation of the ings through the GUI, and our techniques employ a domain- program’s graphical output corresponds to writing code in a agnostic provenance tracing scheme. general-purpose programming language, and edits not possible with the mouse can still be enacted through ordinary text edits To demonstrate the improved expressiveness, we implement arXiv:1907.10699v4 [cs.HC] 10 Aug 2019 to the program. Such capabilities provide hope for integrating a dozen new parametric designs in SKETCH-N-SKETCH with- graphical user interfaces into what are currently text-centric out text-based edits. Among these is the first demonstration of programming environments. building a recursive function in an output-directed program- ming setting. To further advance this vision, we present a variety of new output-directed techniques that extend the expressive power of Author Keywords SKETCH-N-SKETCH, an output-directed programming sys- Output-Directed Programming; SVG; Sketch-n-Sketch

CCS Concepts • Software and its engineering → Application specific devel-

This is the authors’ preprint version. It is posted here for your personal use. opment environments; Language features; • Human-centered The definitive version was published in the following venue. computing → Graphical user interfaces; UIST’19, October 20–23, 2019, New Orleans, LA, USA © 2019 Copyright held by the owner/author(s). Publication rights licensed to ACM. ISBN 978-1-4503-6816-2/19/10. . . $15.00 DOI: https://doi.org/10.1145/3332165.3347925 INTRODUCTION A number of systems have made initial steps towards realizing Direct manipulation [40] graphical user interface (GUI) appli- this paradigm. After first drafting a program using ordinary cations are ubiquitous. Every day, direct manipulation GUIs text edits, several ODP systems allow direct manipulation of are used by millions for office tasks such as presentation and the output to change constant literals in the program [46,9, document preparation as well as for specialized creative tasks 24, 39, 29]. To also relieve the initial text editing burden, for such as engineering and graphic design. Direct manipulation’s programs with graphical output a few systems provide ODP intuitive point-click-modify experience enables a large number mechanisms for program construction [31, 38, 19], akin to of people to leverage computers to create novel artifacts. drawing tools in graphics editors. Even so, experts may sometimes forgo a GUI application and All prior ODP systems presented their workflow as a mix of create their artifact using a programming language. In contrast text edits and direct manipulation. Although the selling point to the intuitive experience offered by direct manipulation GUIs, of ODP is that text edits can compensate for any missing direct text-based coding does not offer immediate and direct artifact manipulation features, the standing question is how thoroughly construction, but programmers use text-based languages to direct manipulation can subsume text-based editing. Therefore, gain flexibility not afforded by any GUI application. in this work we wonder: What kinds of programs can be constructed entirely through output manipulations? Both paradigms—direct manipulation and text-based programming—have proven useful. Naturally, there have New ODP Techniques in SKETCH-N-SKETCH been efforts to combine their distinct strengths. We extend our prior work on SKETCH-N-SKETCH [9, 19], For example, some professional creative applications do of- a programming system for creating vector graphics, with new fer a scripting API (e.g. Maya [2]), but, beyond perhaps an ODP techniques that enable the system to construct 16 ex- without initial macro recording, script construction is a text editing ample programs text edits on the code—even though process with no direct manipulation support. Furthermore, in ordinary text editing remains possible at any time during the the programming world, non-text paradigms such as blocks- construction of each example. Specifically, we: [4] or wires-based visual programming [43] allow direct ma- 1. Expose intermediate execution values for manipulation, nipulation of program elements. Even so, most professional instead of just the final output. programming remains a strictly plain text activity. 2. Offer expression focusing to enable contextual drawing. Instead, could direct manipulation augment text-based pro- 3. Expose generic code refactoring tools through output- gramming, providing user interfaces more like standard cre- directed interactions. ative application GUIs, such as graphics editors? Rather than directly manipulating program elements construed as blocks 4. Use generic run-time tracing to track value provenance, and wires, could one construct a textual program by directly to associate output values with source code locations. manipulating a program’s output? Although the SKETCH-N-SKETCH system is specialized for Output-Directed Programming (ODP) programs that output vector graphics, the four principles above We refer to this paradigm—in which a programmer constructs are relevant for future output-directed programming systems a plain text program by mouse-based operations on the pro- of all kinds. gram’s output—as output-directed programming (ODP). Each Paper Outline and Supplementary Materials operation triggers a transformation on the text-based program, To introduce the output-directed programming workflow in akin to automatic refactoring tools. Successive operations SKETCH-N-SKETCH, in the next section we walk through the build the program step by step. Because the program is plain creation of a simple example program. After this overview, we text throughout the process, desired changes that are not pos- present the core ODP mechanisms and our four innovations in sible through the provided direct manipulation interactions more detail, followed by an examination of the construction of can be enacted by ordinary text editing. The system treats the the 16 example programs. We conclude by discussing current plain text as the artifact’s primary representation so that text limitations of SKETCH-N-SKETCH, related work, and future editing does not disable future direct manipulation actions. directions for ODP. Our implementation, examples, videos, This ODP paradigm for altering a program by manipulating and appendices are available as Supplementary Materials and its output offers two tantalizing possibilities: on the Web (http://ravichugh.github.io/sketch-n-sketch/).

• Creative applications could represent the user’s artifact OVERVIEW not as an opaque internal data structure but as a visible, SKETCH-N-SKETCH is a bimodal programming environment, editable, textual program. Direct manipulation of the arti- as depicted in Figure 1: the left pane is an ordinary source fact can be freely mixed with computational generation. code text editor; the right pane renders the scalable vector • For general-purpose programming, ODP could facilitate graphics (SVG) [47] design generated by the code and also general program construction via output-directed ma- offers a graphical interface for performing transformations on nipulations. For experts, these ODP interactions might the SVG output (i.e., with mouse-based manipulations). The accelerate common program construction tasks, and for programmer may perform keyboard text edits on the code at novices, ODP interactions might provide an approachable any time during program construction, but this ability will not pathway into programming. be used in this paper. 7/5/2019 Sketch-n-Sketch Sketch-n-Sketch File Examples Code Tools Output Tools View Options

7/5/2019 Sketch-n-Sketch Built-In Tools Sketch-n-Sketch File ExamplesCurrCode Tent le: ools UntitledOutput Tools * View Options Context: Program

Current le: Untitled * Undo Redo Clean Up Context: Program Built-In Tools Run Undo Redo Clean Up 1 Run 1 2 y = 127 2 square1 = square 140 [158, 127] 100 3 3 4 x = 158 4 y = 251 User-Dened Tools 5 To introduce the basic ODP workflow in User-Dened Tools 6 w = 156 5 y = 127 7 SKETCH-N-SKETCH, we will walk through Standard Library Tools 6 x = 158 Standard Library Tools 8 color = 362 x = 158 9 the7 construction of a reusable design for the 10 strokeWidth = 5 SKETCH8 topLeft-N-SKETCH = [x, ylogo] (Figure 2). The 11 topLeft=[x,y] 12 logoFunc x y w color strokeWidthconstruction9 = will proceed in four consecutive 13 let topLeft = [x, y] in 10 w = 100 14 let square1 = square 140phases: topLeft w in (i) drawing the shapes, (ii) relating Figure 2. w = 156 11 15 let y2 = y + w in graphical features to align shapes and equate Lambda logo. 16 let xYPair = [ x+ w, y2] in12 square2 = square 140 topLeft w square1= square 140 topLeftw 17 let line1 = line color strokeWidthcolors, (iii) topLeftabstracting xYPair in the design into a reusable function, 18 let line2 = line color strokeWidth13 [x, y2] [ (2! * x + w)/ 2!, (2! * y 19 let polygon1 = and,14 finally,y2 = y (iv) + wrefactoring the design to clean up the gen- y2=y+w 20 let pts = [[x, y], [ x+ w15, y], xYPair, [x, y2]] in 21 let [color, strokeColorerated, strokeWidth code.] = Throughout,[529, 360, 5] in the code shown is as produced by 22 polygon color strokeColorSKETCH16 strokeWidthline1-N-S = KETCHpts line in 0. Newlines5 topLeft inserted[ ⦀x+ w, fory2] formatting pur- line1= line05 topLeft[x+w, y2] 23 [square1, line1, line2, polygon117 ] 24 poses in this paper are “escaped” by a “\” backslash. The final 25 logo = logoFunc x y w colorcode strokeWidth18 forline2 this example= line 0 is5 shown[x, y2 in] [Figure (2! * 11x .+ w)/ 2!, (2! * y + wline2) / 2=! ]line 05[x, y2]\ 26 19 [ (2! *x+w)/ 2!, (2! *y+w) / 2!] 27 svg (concat [ 28 logo 20 svg (concat [ ... 29 ]) Drawing21 [ Shapessquare1], The22 initial [square2 program], template is nearly blank, defining only an Figure 4. Code after drawing one square and snap-drawing two lines (right). Freeze annotations, written⦀ !, tell SKETCH-N-SKETCH not to empty23 list [line1 of SVG], shapes: 24 [line2] change those constants when shapes are moved on the canvas [9]. svg25( concat]) [ ]) from the top-left to the bottom-right corner and a second line Starting from this bare program, the programmer would first from the bottom-left corner to the center point (these two steps like to get some shapes into the program’s output. A tradi- are shown in the right side of Figure 4). SKETCH-N-SKETCH tional programming environment would require looking up interprets these snaps as constraints which should always hold, the drawing library’s documentation and perhaps copy-pasting i.e., the lines should still coincide with the corners and center some example code into the program. even if the square is moved or resized. SKETCH-N-SKETCH encodes these constraints in the program via shared variables: file:///Users/brian/Documents/open_source/sketch-n-sketch/build/out/index.htmlThe SKETCH-N-SKETCH output pane, however, imitates a three new variables are1/1 introduced for the square’s top-left x traditional drawing application. The programmer clicks on and y coordinates and its width w, and the spatial properties the “square” tool from the toolbox (Figure 1a) and drags the of both the square and the two lines are defined in terms of mouse on the canvas. When she releases the mouse, a new simple math on those variables (Figure 4). square1 definition is inserted in the program, based on a call to the square standard library function with arguments for position and size derived from the mouse drag operation. A square1 variable usage is also inserted into the shape list at Relating Properties the end of the program, and thereby a red square appears in The programmer would like the color and width of the two the program output on the canvas (not shown). lines always to be identical. SKETCH-N-SKETCH offers tools 7/5/2019 Sketch-n-Sketchfor relating features after they are drawn. To relate the col- square1= square 0 [158, 127] 156 7/5/2019 Sketch-n-Sketch Sketch-n-Sketch File Examples Code Tools Output Tools View OptionsSketch-n-Sketch Fileors,Examples the programmerCode Tools Output first Tools selectsView Options the two lines (indicated by

file:///Users/brian/Documents/open_source/sketch-n-sketch/build/out/index.htmlsvg( concat [ yellow highlights in Figure 5) to expose sliders for each Built-In Tools 1/1 Current le: Untitled * Context: Program Current le: Untitled * Context: Program Built-In Tools [square1] line’s color and stroke width. As mentioned earlier, she Undo Redo Clean Up Run Undo]) Redo Clean Up 1 can manipulate the sliders to change constants in the pro- 1 Run 2 y = 127 2 square1 = square 140 [15 3 gram, but here she instead clicks the whole line2 color slider 4 x = 158 line1 3 Shapes on the canvas may be to select the property User-De ned Tools 4 svg (concat [ square1 5 directly moved and resized as 6 topLeft = [x, y]itself (a selected prop- User-De ned Tools Standard Library Tools 5 [square1] 7 Standard Library Tools 6 in]) a traditional graphics editor, 8 w = 156 erty is indicated in line2 9 causing SKETCH-N-SKETCH to 10 square1 = squaregreen,140 topLeft asw shown in automatically change appropri- 11 Figure 5) and then se- 12 y2 = y + w ate numeric constants in the pro- 13 lects the other line’s 14 line1 = line 0 5 topLeft [ x+ w, gram [9]. Additionally, when a 15 color slider as well. Figure 5. Selecting a feature. shape is selected, sliders appear 16 line2 = line 0 5 [x, y2] [ (2! * 17 Whenever an item is selected, SKETCH-N-SKETCH displays a to allow similar modification of Figure 3. Color slider.18 svg (concat [ 19 [square1], floating menu of output tools. The programmer would like the non-shape attributes such as, e.g., fill color (Figure 3) or20 stroke[line1], 21 [line2] two selected colors to always be the same, so she moves her width. In our example, the programmer drags the color22 slider]) mouse to the⦀ MAKE EQUAL tool, revealing a list of possible to change the square’s fill color from red to a shade of green; the appropriate number in the program is updated (the color program transformations in a submenu—in this case there is only one, but if the lines had been different colors two results number 0 above becomes⦀ 140, shown in Figure 4). wouldOutput Tools be offered to let the programmer decide which line’s To create the lines for the lambda symbol, the programmer colorAdd to Output should take priority (as simulated in Figure 6). Moving selects the “line” tool and then “snap-draws” [17] two lines herReorder mouse in List over the single “Equalize by removing line2 color” Dupe ( D) between point features of the green square: she draws one line resultMerge in the submenu shows a preview of the new output on Output Tools Group Add to Output Abstract Reorder in List Repeat... Dupe ( D) Group Abstract Repeat...

file:///Users/brian/Documents/open_source/sketch-n-sketch/build/out/index.html 1/1 otherwise-overlapping widgets. Hovering over a list widget 7/5/2019 highlights the corresponding code expression in the program Sketch-n-Sketch Sketch-n-Sk(notetch shown),File allowingExamples theCode T programmerools Output T to distinguishools View theOptions var-

ious lists. She selects the list widget corresponding to the Built-In Tools Current le: Untitled[square1, * line1, line2]Context: literalProgr (asam shown in Figure 7). Undo Because Redo Clean Up this list literalRun is assigned to a variable, the list widget 1 is labeled with the variable name, squareLineLine. 2 y = 127 3 A label on the canvas may be clicked logo 4 x = 158 to enact a RENAME refactoring on User-Dened Tools 5 6 topLeftthe = [ variablex, y] (Figure 8). After renam- Standard Library Tools 7 ing the squareLineLine variable to 8 w = 156 Figure 6. The output tools panel appears when an item on the canvas is logo, she then, with the list still se- 9 lected, invokes the ABSTRACT tool selected. Each tool may offer multiple results; hovering the mouse10 oversquare1 = square 140 topLeft w each result previews the change on the code and on the canvas. 11 to construct a function that returns 12 y2 = y the+ w selected item, namely the list 13 [square1, line1, line2]. The 14 color = 0 the canvas and presents a diff of the change in the code box ABSTRACT tool performs an “Extract Figure 8. Renaming. (not shown)—in this case the appearance will not change15 but 16 strokeWidthMethod” = 5 refactoring. Definitions (with free variables) used an appropriately-named color variable will be introduced17 and only in the construction of [square1, line1, line2] are used for both lines. The programmer is happy with this change18 line1 =heuristically line color strokeWidth pulled into t the new function, producing a func- and clicks to apply it. She repeats this workflow to equalize19 20 line2 =tion line named color logoFuncstrokeWidthparameterized [ over the remaining free the stroke width of the two lines as well. 21 variables y, x, w, color, and strokeWidth: 22 squareLineLine = [square1, line1⦀ Abstracting the Design 23 ... The programmer would like to make her logo design reusable—24 svg (concatlogoFunc [ yxw color strokeWidth= 7/5/2019 Sketch-n-Sketch 25 squareLineLinelet topLeft=[x,y] in in other words, she would like to create a function that, given26 ]) let square1= square 140 topLeftw in Sketch-n-Sketch File Examples Code Toolssize andOutput T colorools arguments,View generatesOptions a logo appropriately. let y2=y+w in let line1= Output Tline colorools strokeWidth topLeft[x+w, y2] in She gathers the three shapes into a single expression by select- let line2= line color strokeWidth[x, y2B]\uilt-In Tools Current le: Untitled * Context: Program > logoFunc Focus squareLineLine ing the shapes and invoking the GROUP tool in the floating [ (2! *x+w)/ 2!, (2! *y+w) / 2!] in Add to Output Undo Redo Clean Up Run menu. Analogous to the grouping functionality of traditional [square1, line1, line2] graphics editors, GROUP in SKETCH-N-SKETCH gathers the Reorder in List 1 logo= logoFuncyxw color strokeWidth shapes into a single list definition, which is used in place of Dupe ( D) 2 y = 127 logoFunc ... Group 3 the individualy shapesx in thew final shapecolor list: strokeWidth 7/5/2019 Sketch-n-Sketch Abstract 4 x = 158 ... The programmer has created a Repeat... User-Dened Tools 5 Sketch-n-SketchsquareLineLineFile Examples=[square1Code T,ools line1Output T, line2ools] View Options reusable function, which she 6 w = 156 Current le: Untitled *svg( concat [ Context: Program could manually call multiple Built-In Tools 7 squareLineLine times in the program. Conve- FigureS 9.tand Customard Librar drawingy Tools tool 8 color = 0 Undo Redo ]) Clean Up Run niently, however, type inference via type inference. 9 1 in SKETCH-N-SKETCH notices 10 strokeWidth = 5 2 y = 127 3 Now that the shapes are grouped that this function accepts an (x,y)-coordinate and a width, so 11 squareLineLine 4 x = 158 into a list, SKETCH-N-SKETCH this function appears in the tool box on the right-hand side 12 -- *** Focused Definition *** User-Dened Tools 5 offers that list itself as an ob- as a custom drawing tool (Figure 9).1 User-defined drawing 13 logoFunc y x w color strokeWidth Standard Library Tools 6 topLeft = [jectx, y] for selection on the can- file:///Users/brian/Documents/open_source/sketch-n-sketch/build/out/index.htmltools behave the same way as the standard library drawing 1/1 14 let topLeft = [x, y] in 7 tools such as the square and line tools already demonstrated: 15 let square1 = square 140 topLe8 w = 156 vas. Lists are represented on the 9 drawing on the canvas with the tool will insert a new call to 16 let y2 = y + w in canvas with list widgets—boxes 10 square1 = squarewith dotted140 topLeft borders w (intended the custom function into the program. 17 let line1 = line color strokeW11 18 let line2 = line color strokeW12 y2 = y + w to evoke the elements of a list). 19 [square1, line1, line2] 13 When the programmer hovers Refactoring 20 14 color = 0 her mouse over a shape, list wid- Although the programmer has produced a reusable design, 15 gets are shown for each of the 21 logo = -- *** Example Call *** Figure 7. List widgets. she may change her mind about particular details in the 16 strokeWidthlist = 5 expressions in the program 22 logoFunc y x w color strokeWid17 ⦀ program. For example, the programmer may like to clean 23 18 line1 = linein color which strokeWidth the shape appeared.t In this example there are four up the code and also may want to add a border to the de- 24 svg (concat [ 19 such lists: the initial list literal [square1, line1, line2] 20 line2 = line color strokeWidth [ sign. SKETCH-N-SKETCH provides features to help imple- 25 logo plus three expressions at the end of the program that eval- 21 ment code changes such as these two. uate to lists (the squareLineLine variable usage, the list 26 ]) 22 squareLineLine = [square1, line1⦀ 23 literal [squareLineLine], and the flattened list value pro- 1Type inference mechanisms to distinguish between x and y coor- 24 svg (concatduced [ by the call concat [squareLineLine]). Notice in dinates, distances, and other numeric values are described in the 25 squareLineLineFigure 7 how SKETCH-N-SKETCH chooses a layout to avoid appendix included in the Supplementary Materials. 26 ])

Output Tools Focus squareLineLine Add to Output Reorder in List Dupe ( D) Group Abstract Repeat...

file:///Users/brian/Documents/open_source/sketch-n-sketch/build/out/index.html 1/1 file:///Users/brian/Documents/open_source/sketch-n-sketch/build/out/index.html 1/1 Similar to list widgets, call widgets—drawn with solid 1 borders—are placed around items that are produced by func- 2 y = 127 tion calls in the program. The programmer clicks the call 3 widget for the first call to logoFunc, which (a) exposes the 4 x = 158 7/5/2019 Sketch-n-Sketch 5 Sketch-n-Sketch File function’sExamples argumentsCode Tools Output T aboveools the calledView Options widget, and (b) focuses 6 w = 156 any changes to just that function. The rest of the program 7 Current le: Untitled * output (there isContext: nonePr currently)ogram > logoFunc disappears from the canvas for 8 color = 362 Built-In Tools Undo Redo Clean Upthe durationRun of the focused editing session. 9 1 10 strokeWidth=5 2 y = 127 When a function is fo- logoFunc 11 3 y x w color strokeWidth 12 logoFuncxyw color strokeWidth= 4 x = 158 cused, its arguments 13 let topLeft=[x,y] inUser-Dened Tools 5 may be renamed, re- 14 let square1= square 140 topLeftw in 6 w = 156 moved, or reordered; let y2=y+w in 7 15 Standard Library Tools 8 color = 0 an argument may be 16 let xYPair=[x+w, y2] in 9 added by selecting a 17 let line1= line color strokeWidth topLeft xYPair in 10 strokeWidth = 5 18 let line2= line color strokeWidth[x, y2]\ 11 shape feature within 19 [ (2! *x+w)/ 2!, (2! *y+w) / 2!] in 12 -- *** Focused Definitionthe function *** and choos- 20 let polygon1= 13 logoFunc y x w color strokeWidth 14 let topLeft = [x,ing y] inADD ARGUMENT 21 let pts = [[x,y], [x+w,y], xYPair,[x, y2]] in 15 let square1 = squarefrom 140 the topLe output tools Figure 10. Focused editing. 22 let [color, strokeColor, strokeWidth] = \ 16 let y2 = y + w in [529, 360, 5] in menu. In this case, the programmer clicks the REORDER 23 17 let line1 = line color strokeW 24 polygon color strokeColor strokeWidth pts in 18 let line2 = line AcolorRGUMENT strokeW arrows to place the x argument before y. 25 [square1, line1, line2, polygon1] 19 [square1, line1, line2] 20 26 Now the programmer would like to add a border. After chang- 27 logo= logoFuncxyw color strokeWidth 21 logo = -- *** Exampleing Call the *** two existing lines to black via color sliders, she then 22 logoFunc y x w color strokeWid⦀ 28 23 snap-draws a polygon to the four corners of the square with 29 svg( concat [ 24 svg (concat [ the “Polygon” tool from the toolbox. Because she is focused 30 logo 25 logo 31 ]) 26 ]) on this function, the new polygon, called polygon1, is added to the return expression of the logoFunc function instead of Figure 11. The final code for the lambda logo example (Figure 2), to the program’s final shape list (see lines 20-25 of Figure 11). produced entirely by output-directed manipulations on the canvas. With the polygon selected, she uses the polygon’s sliders to set its fill color to transparent and to adjust its stroke width.

The lambda logo program is now completed to the program- tion regions for width and height), colors (via color sliders), mer’s satisfaction; the final code is shown in Figure 11. The or strokes (via stroke width sliders). When the programmer programmer was able to create the program entirely through makes a selection and then invokes a transformation, the se- output-directed manipulations on the canvas by using tools for lected sub-values are “arguments” to the transformation and drawing, relating, abstracting, and refactoring. are interpreted in a transformation-specific way. DESIGN AND IMPLEMENTATION Looking forward to applying ODP to larger programs, manipu- SKETCH-N-SKETCH is a browser application written in the lation of the final output alone will be insufficient. The output functional language Elm [14] extended with custom providers by itself does not represent the process by which the output for mutation and exceptions. was produced. We believe some of the intermediate process should be exposed on the canvas for manipulation. file:///Users/brian/Documents/open_source/sketch-n-sketch/build/out/index.htmlFigure 12 depicts the workflow for output-directed program- 1/1 ming in SKETCH-N-SKETCH. The programmer-facing lan- Therefore, in addition to the sub-values of the final output de- guage for users is a standard functional language with Elm- scribed above, the programmer may also select two additional like syntax. A text-based program written in this language is kinds of terms associated with the program’s execution. First, evaluated, and its SVG output value is rendered on the output we also expose widgets on the canvas that correspond to inter- canvas. The output is overlaid with widgets for selecting and mediate values produced during execution. Second, we allow manipulating sub-values (discussed below). The programmer the programmer to select a sub-expression in the program to may draw a shape, or drag an item to change a number in denote the context under focus. These additional kinds of the program [9], or make a selection and choose a program selections—intermediate values and focused sub-expressions— transformation from the floating tools menu (Figure 6). The serve as further arguments to program transformations and resulting modified program is placed in the code box and is enable the programmer to modify portions of the computation re-evaluated producing new output on the canvas. that do not have a direct representation in the final output.

Most of SKETCH-N-SKETCH’s tools (summarized in Fig- To highlight the key ideas behind our implementation, we ure 13) operate on selections on the canvas. Different kinds of discuss these two novel selection types below. We also discuss sub-values of the final output may be selected: whole shapes the degree to which several of our transforms are ordinary may be selected, or sub-features of shapes may be selected— automatic refactorings (i.e. not specific to SVG), as well as namely, positional properties (via points rendered on the cor- the generic provenance tracing that the transformations rely ners and edge midpoints of shapes), size properties (via selec- upon to associate canvas selections with program locations. [x,y] as point = [208, 256]

A B halfW = 102 e ⇒ … ⇒ v } xOffset=x+ halfW xOffset2=x- halfW

Program halfH = 145 D C Transformations yOffset=y- halfH

Figure 12. Output-directed programming workflow. (A) Program e eval- yOffset2=y+ halfH uates to value v of type SVG. (B) Graphical widgets for manipulating the output value, intermediate values, and the program. (C) Programmer se- Figure 14. Point and offset widgets are displayed based on program lects from toolbox (cf. Figure 1a) and draws on canvas, or manipulates execution to allow manipulation of intermediate computations. Depicted existing shapes and selects a tool from the floating tools menu (Figure 6). is an early step in the creation of Figure 15xii. (D) One or more candidate programs are produced; the programmer previews the changes and then chooses one (Figure 6). also insert a new point for the base of the offset. To ease the Drawing #Ex Abstracting #Ex creation of symmetry, offset amounts may snap to each other # DRAW SHAPE 16 # GROUP 8 # DRAW CUSTOM SHAPE 4 # ABSTRACT 9 while drawing. The symmetry is enforced by introducing a # DUPE 2 # MERGE 0 shared variable for offset distance. For example, the program # DRAW POINT 3 # REPEAT OVER FUNCTION CALL 2 in Figure 14 is rendered on the canvas as a point widget from # DRAW OFFSET 9 # REPEAT OVER EXISTING LIST 3 which four offsets emanate. The variables halfW and halfH # DELETE 4 # REPEAT BY INDEXED MERGE 2 are shared offset amounts produced by snap-drawing (and # FOCUSED DRAWING 1 # FILL HOLE 2 # SNAP DRAWING (UI) 15 # LIST WIDGETS (UI) 9 subsequently renamed by the programmer). The ends of the offsets can serve as snap targets for future drawing (e.g. the Relating #Ex Refactoring #Ex rhombuses in Figure 15xii). Note that Figure 14 was created # MAKE EQUAL 12 # FOCUS EXPRESSION 3 by the using the “Point and Offset” tool, but the rendered wid- # RELATE 2 # RENAMEIN OUTPUT 15 gets would be the same if the program were instead written by # DISTANCE FEATURES (UI) 6 # ADD/REMOVE/REORDER ARG. 6 # REORDER LIST 4 text edits—only the code matters. SKETCH-N-SKETCH emits # ADDTO OUTPUT 1 widgets from the evaluator when appropriate computation # SELECT TERMINATION COND. 1 patterns are encountered. Figure 13. Program transformations and user interface features (UI) in SKETCH-N-SKETCH. Those marked with # are new to our system; A third type of intermediate widget—a list widget—is emitted the remainder are improvements upon [19]. The #Ex column indicates whenever an execution step evaluates to a list of graphical the number of examples in Figure 15 in which the feature was used. items. A list widget is drawn as a dashed box encompassing the items (Figure 7). Any list of shapes, list of points, or list of lists in the program can thus be selected. List widgets obviate Intermediate Value Widgets the need for a graphics-specific construct to denote groups of To allow manipulation of computation steps before the final shapes—a group is just a list of shapes, and the list widget output, SKETCH-N-SKETCH displays widgets for three kinds facilitates selection of, and thus operations on, the group. of intermediate values produced during execution: point val- ues, offset values, and list values. Expression Focusing Points (number-number pairs, e.g. [10, 20]) produced at To enable manipulation of subsections of the program, a pro- intermediate execution steps are exposed for manipulation as gram expression may be selected to control the syntactic scope point widgets on the canvas. Between selected points, we also of changes made to the code. Most notably, focusing a shape draw distance features for selecting Cartesian distances. group (i.e. list of shapes) or a function causes drawing opera- tions to insert shapes into that group or function rather than the In graphics code, it is common to define offsets from some final output. Accordingly, a recursive design can be created x y base or values. Therefore, during execution, when a nu- by drawing a function inside itself. meric amount is added to or subtracted from an x or y coordi- nate, an offset widget is drawn on the canvas as a horizontal or When a function call produces graphical elements, a call wid- vertical arrow from the initial point, where the length of the get is displayed around the returned elements as a box with a arrow is the numeric amount of the offset. The arrow itself gray solid border. Clicking the gray border focuses the func- may be selected or dragged to change the offset amount. tion call (Figure 10). The remainder of the program’s output is hidden and drawing operations are interpreted inside the To complement the new point widgets and offset widgets, a focused function. Clicking the call widget again or pressing “Point and Offset” tool in the toolbox (Figure 1b) allows the the “escape” key returns focus back to the entire program. programmer either to click on the canvas to add a point defini- tion to the program (e.g. [x, y] as point = [10, 20]) Literal (non-function) definitions may also be focused. If a or to drag on the canvas to add a new offset definition (e.g. canvas selection refers to the right hand side of an assignment, xOffset = x + 5). Offsets not drawn from an existing point a FOCUS EXPRESSION tool is offered to focus that assignment. Generic Refactoring limitations by discussing the tasks from the Watch What I Do: Looking forward to applying ODP to domains beyond vec- Programming by Demonstration [34] benchmark suite that the tor graphics, a number of SKETCH-N-SKETCH’s tools apply current system cannot complete without text edits. standard automatic refactorings. Authoring Most notably, labels are drawn next to most widgets on the The lambda logo walkthrough in the overview framed the canvas to aid comprehension—any label may be clicked to SKETCH-N-SKETCH workflow as a four stage draw-relate- RENAME the associated variable and its uses. abstract-refactor authoring process. In practice, design con- SKETCH-N-SKETCH also includes tools for refactoring func- struction cannot always be cleanly delineated into precise tions. The ABSTRACT tool performs a generic “Extract stages in a fixed order. For example, while constructing the de- Method” refactoring, building a new function parameterized signs of Figure 15, we often specified the relationships before over the expression’s free variables. Also, when a function call drawing the shapes: we laid out the desired parameterization is focused by clicking its call widget, SKETCH-N-SKETCH of- of the design using points and offsets (as in, e.g., Figure 14) fers actions to add, remove, or reorder arguments. Finally, the and afterwards attached shapes using snap-drawing. MERGE tool performs clone elimination. The #Ex column of Figure 13 lists how many of the Figure 15 examples utilized each tool. Indeed, besides shape drawing Provenance Tracing and variable renaming, the most widely used functionality was While the widgets for intermediate values expose relevant SKETCH-N-SKETCH’s snap-drawing ability—not surprising steps of the computation on the canvas, occasionally the trans- given that the goal was to create parametric designs. formations require knowledge about parts of the computation that are not directly represented by widgets. For example, It is notable that to encode spatial constraints we more often the ADD ARGUMENT tool searches for every expression that preferred to snap-draw rather than use the MAKE EQUAL tool. affected the selected value and separately offers each such MAKE EQUAL is more flexible, but not only does it require expression as a possible new argument to the function. extra clicks compared to snap-drawing, MAKE EQUAL can offer a large number of different but hard-to-distinguish ways To help the tools answer these kinds of questions, the to enforce a constraint (Figure 6 is a tame example). To help, SKETCH-N-SKETCH evaluator performs tracing on every exe- SKETCH-N-SKETCH employs a ranking heuristic that seems cution step: the resultant value is tagged with the expression to work well in practice: MAKE EQUAL prefers changes that being evaluated as well as pointers to the prior (tagged) val- rewrite terms near each other and later in the program. The ues used in the immediate computation; transitively following least used tool for specifying relationships was the RELATE these prior tagged values reveals the dependencies of the com- tool. RELATE guesses a mathematic relationship between se- putation. To answer containment queries, we additionally add lected items—we only used it for constraints involving thirds. pointers from list elements to the list(s) containing them. Our tracing discards certain control flow information, most notably Offsets plus snaps and MAKE EQUAL was sufficient, but not pattern match de-structuring operations; future versions of always convenient, for creating reasonable parameterizations SKETCH-N-SKETCH may adopt the Transparent ML [1] trac- of the designs. Laying out offsets beforehand requires fore- ing scheme to preserve this information. Neither our tracing thought. A future SKETCH-N-SKETCH might benefit from nor Transparent ML is specific to SVG—we suspect generic tools to break constraints or invert dependencies after the fact. tracing will be useful for ODP in multiple domains. As indicated in Figure 13, we did not use the MERGE tool and SKETCH-N-SKETCH becomes sluggish on larger examples. three tools were used only once, all on the most challenging Although tracing theoretically adds memory and time over- example, the Koch snowflake. The MERGE tool merges multi- head to evaluation, the culprit is not tracing itself but is usually ple copies of a shape into a function—we prefer ABSTRACT comparison between large traces or widget processing. instead because it requires only a single example. The three tools only used for the Koch fractal all played a role in the CASE STUDY OF ODP EXAMPLES workflow to specify recursion, which we now recount. To explore the expressiveness of ODP in SKETCH-N-SKETCH, we implemented 16 parametric designs summarized in Recursive Koch Snowflake Figure 15. Drawn from various sources [9, 19, 34,5,8], We highlight the key steps in creating the recursive von Koch the designs exercise different features: 6 designs are param- snowflake design [45] (Figure 1, Figure 15i). This task re- eterized functions that appear as drawing tools at the end quires manipulating intermediates—program has no output of construction, 6 involve repetition by position, 1 involves throughout most of its development—and focused editing is repetition by other features (radius and color, Figure 15xiii), needed to build the recursive function. Here we emphasize the and 1 uses recursion (Figure 15i). All 16 programs, spanning steps to specify the recursion, although the video in the Sup- 427 lines of code total, were built entirely via output-directed plementary Materials walks through the entire construction. manipulations, without any text editing in the code box. Each side of the Koch snowflake is based on a recursive motif Below, we discuss common patterns in how we used the tools shown in Figure 1f. The motif requires two helper functions: 1 to implement the examples. Next, we recount the key steps a function that, given two points, computes a point 3 of the in building the recursive Koch curve and explain the process way between them (oneThirdPt, Figure 1d), and a function to construct designs involving repetition. Finally, we describe that, given two points, computes a third point that completes an Sketch-n-Sketch File Code Tools View Options Output Tools

Sketch-n-Sketch File Code Tools View Options Output Tools Current le: Untitled * Context: Program Current le: Untitled * Context: Program Sketch-n-Sketch File Code Tools View Options Output Tools Sketch-n-Sketch File Code Tools View Options Output Tools Undo Redo Clean Up Run Current le: Untitled * Context: Program Undo Redo Clean Up Run Sketch-n-Sketch File Code Tools View Options Output Tools Current le: Untitled * Context: Program Undo Redo Clean Up Run 1 1 Sketch-n-Sketch File Code Tools View Options Output Tools 1 equiTriPt [x3, y3] [x2, y2] = Undo Redo Clean Up Run Current le: Untitled * 2 [left, top] as topLeft Context:= [84, Pr 147ogram] Sketch-n-Sketch File Code Tools View Options Output Tools 2 [ (x2 + x3 + sqrt 3! * (y2 - y3))/ 2!, (y2 + y3 - sqrt 3! * (x2 - x3)) / 2!] [ , y ] p [ g ( q (p 2 w = 126 3 3 9 Current le:Sketch-n-Sk Untitledetch File * Code Tools View Options Output Tools Context: Program Undo Redo Clean Up Run Current le: Untitled * Context: Program 4 oneThirdPt [x, y] [x3, y3] = 4 height = 345 Current le: Untitled * Context: Program 5 [ x / 1.5!+ x3 / 3!, y / 1.5! + y3 / 3!] 10 hangingTray topPoint hangDistance = 3 Undo Redo Clean Up UndoRun Undo Redo RedoClean Up Clean Up Run Run 6 1 5 11 let [x, y] as point = topPoint in 85 fstOffset = x13 + baseW 21 squareByCenter2Func3 center2 = 86 22 squareByCenter fill2 center2 (squareW / 2!) 7 point = [39, 314] 2 point = [82, 136] 4 color = 36623 6 stoneWidth = 85 12 let yOffset = y + hangDistance in 87 [_, y13] = onLine2 1 8 24 squareByCenter2Func4 center2 = 3 88 9 point2 = [490, 301] 7 13 let [x1, y1] as point1 = [x, yOffset] in 2 [25x, squareByCentery] as point fill center2 = (squareW[66, / 2148!) ] 4 h = 239 89 sndOffset = y13 - cutW 5 26 10 14 let trayHalfW = 85 in 90 3 27 boxyXFunc ([x, y] as point) squareW n = 8 width = 331 28 let xOffset = x + squareW in 11 makeKochPts depth point point2 = 5 91 [x14, y14] as point14 = [fstOffset, y13] 15 let left = x1 - trayHalfW in 6 strokeWidth4 h429 = let =141xOffset2 8 = x - squareW in 12 let oneThirdPt2 = oneThirdPt point2 point6 inw = 444 9 92 30 let [x1, y1] as point1 = [xOffset, y] in 13 let oneThirdPt3 = oneThirdPt point point2 in 16 let right = x1 + trayHalfW in 93 y14Offset = y14 - cutW 5 31 let y1Offset = y1 - squareW in 7 10 archFunc ([left, top] as topLeft) width height stoneWidth = 94 7 32 let ySep =0! - squareW in 14 let equiTriPt2 = equiTriPt oneThirdPt3 oneThirdPt2 in 17 let tray = ellipse 46 point1 trayHalfW 21 in 6 w33 = let274 upRightPts = nPointsSepBy n [x1, y1Offset] squareW ( ySep) 15 if depth < 2 then 8 floorRect = rect 36 point w h 11 let lintel = rect 124 topLeft width stoneWidth in 95 boxBack = 34 let [x2, y2] as point2 = [xOffset2, y] in 18 let color = 434 in 96 let pts = [point10, [x10, y10Offset], [x12, y12Offset], point35 let y2Offset = y2 - squareW in 16 [point, oneThirdPt3, equiTriPt2, oneThirdPt2] 7 9 97 let [color, strokeColor, strokeWidth8 line1Func] = [color, 360, 2] in 36 ([ let xyOffset3, y= y2] + squareWas in point) = 17 else 12 let pillarTop = top + stoneWidth in 19 let strokeWidth = 5 in 8 fill37 let =yOffset4 362 = y1 + squareW in 10 tableRect = rect 188 point (w / 3!) h 98 polygon color strokeColor strokeWidth pts 38 let upLeftPts = nPointsSepBy n [x2, y2Offset] ySep ySep in 18 let makeKochPts2 = makeKochPts (depth - 1) point oneThirdPt3 in 13 let pillarHeight = height - stoneWidth in 20 let wire2 = line color strokeWidth point [right, y1] in 99 11 9 let xOffset9 39 let downRightPts = x= nPointsSepBy + w n [x1in, yOffset4] squareW squar 19 let makeKochPts3 = makeKochPts (depth - 1) oneThirdPt3 equiTriPt2 in 21 let wire1 = line color strokeWidth point [left, y1] in 100 boxRight = 40 let downLeftPts = nPointsSepBy n [x2, yOffset3] ySep squareW i 20 let makeKochPts4 = makeKochPts (depth 12- 1) svgequiTriPt2 (concat oneThirdPt2 [ in 14 let leftPillar = rect 16 [left, pillarTop] stoneWidth pillarH 101 Skletetch-n-Sk pts = [point12etch, [x12, y12OffsetFile ], [Codex14, y14Offset Tools10], pointh41 View= let73 squareByCenter1Options = squareByCenter 408 point (squareW / 2!) ⦀ Output Tools 42 let repeatedSquareByCenter2Func = ⦀ Sketch-n-Sk22 [etchtray,File wire2Code, wire1 Tools ] View Options Output Tools 102 let [color, strokeColor, 10strokeWidth ] = line[color, 360 , color2] in strokeWidth point [xOffset, y] 21 let makeKochPts5 = makeKochPts (depth 13- 1) oneThirdPt2 [floorRect point2], in 15 let rightPillar = rect 220 [ width - stoneWidth+ left, pillar 11 43 map squareByCenter2Func upRightPts in 22 concat [makeKochPts2, makeKochPts3, makeKochPts4, makeKochPts5] 103 polygon color strokeColor strokeWidth pts 44 let repeatedSquareByCenter2Func21 = 23 45 map squareByCenter2Func2 downRightPts in 23 14 [tableRect] 16 [lintel, leftPillar, rightPillar] 104 11 12 batteryFunc ([x, y] as point) h4 w fill h = Current le:24 UntitledbaseCenter * = [centerX, 477] Context: Program Curr105 entboxBot le: = Untitled * 46 let repeatedSquareByCenter2Func3 = Context: Program 24 depth = 3{1-5} 15 ]) 13 47 let ⦀ map bodysquareByCenter2Func3 = rect upLeftPts fill in point w h4 in 17 106 let pts = [point10, point12, point14, onLine2] in 25 48 let repeatedSquareByCenter2Func4 = Undo 25 Redo Clean Up Run 12 left = 10449 map squareByCenter2Func4 downLeftPts in 18 arch = archFunc topLeft width height stoneWidth 107 let [color, strokeColor, strokeWidth] = [color, 360,14 2] in let head = rect fill [ x+ w, (h4 - h + 2! * y) / 2!] 40 h in 26 topPts = makeKochPts depth point point2 2 26 color = 214 108 Undo polygon color Redo strokeColorClean strokeWidth Up pts 50 let squareByCenterSingleton = [squareByCenter1] in Run 51 concat [squareByCenterSingleton, repeatedSquareByCenter2Func4 27 109 15 [body, head] 19 3 rhombusFunc27 [x, y] halfW halfH = 13 52 28 botCorner = equiTriPt point2 point 110 boxLeft = 16 53 boxyX = boxyXFunc point squareW n 4 let xOffset = x + halfW in 1 29 20 svg (concat [ 28 pillar = line color 20 point2 baseCenter 111 let pts = [point10, [x10, y10Offset], [x13, sndOffset], onLin54 Sketch-n-Sketch File Code Tools View Options Output Tools 5 let xOffset2 = x - halfW in ⦀ 112 let [color, strokeColor, 14strokeWidthtop] = [color =, 360119,17 2] in battery55 boxyXFunc1 = boxyXFunc batteryFunc [341, 621] squareW point 3{0-10} h4 w fill h 30 rightPts = makeKochPts depth point2 botCorner 21 arch 29 ⦀ 2 pt2 = [351, 271] 56 Sketch-n-Sketch31 File Code Tools View Options Output Tools Sketch-n-Sketch File Code Tools View Options 6 let yOffset =Output y - halfHTools in 113 polygon color strokeColor strokeWidth pts 18 57 boxyXFunc2 = boxyXFunc [513, 216] squareW 1{0-10} 22 ]) ⦀ 30 base = ellipse color baseCenter 147 20 1143 58 7 let yOffset2 = y + halfH in 115 boxFront = 15 19 svg59 svg ((concatconcat [ [ Current le: Untitled * Context: Program 31 116 let pts = [[x13, sndOffset], [x14, y14Offset], point14, onLin60 boxyX, Current le: Untitled * 8 let pts = [[xContext:, yOffset Progr],am [xOffset, y], [x, yOffset2], [xOffset 4 pt1 = [122, 382] 20 61 battery boxyXFunc1, Current le: Untitled * Context: Program 9 let32 [strokeWidthcolor, strokeColor = 15 , strokeWidth] = [118, 360, 2] in 117 let [color, strokeColor, 16strokeWidthrungs] = [color, 360=, 2] in 62 boxyXFunc2 Undo Redo Clean Up ⦀ Run 1185 polygon color strokeColor strokeWidth pts 21 ])63 ]) (i) Koch Undo Snowflake Redo Clean Up ★ (ii) Precision FloorRun Plan ☆ (iii) Mondrian Arch λ (iv) Balance Scale ☆ (v) Box Volume ★ (vi) Xs λ★ (vii) Battery λ (viii) Ladder ★ 10 33 polygon color strokeColor strokeWidth pts 119 1 [x1, y1]= [444, 358] Undo Redo Clean Up Run 11 1206 svgarrowFunc (concat [ pt1 17pt2 = map line1Func (nVerticalPointsSepBy 4{0-10} [left, top] 50) 1 34 leftArm = line color strokeWidth point2 point 2 12 rhombusFunc2 ([x, y] as point) = Sketch-n-Sketch File Code Tools View Options 1217 [topDownTemplate let onLine2], Output = T oolsonLine pt1 pt2 0.7754620659147587 in 1 [left, top] as topLeft = [84, 117] 2 point31 =LOC [307, 334] 9 LOC,35 3 MO 15 LOC, 4 MO 122 [boxBack34 ],LOC, 2118 MO 81 LOC 3 x= 56 44 LOC 13 LOC, 5 MO 18 LOC 13 let halfW = 40 in Sketch-n-Sketch File Code Tools View Options Output Tools 36 rightArm = line color strokeWidth point2 point3 1238 [boxRight let ],onPerpendicularLine2 = onPerpendicularLine onLine24 pt2 1! i 2 3 (vs. 2514 LOC, let 27halfH MO) = 83 in Curr(vs.ent le: 23 Untitled LOC, * 9 MO) 124 [boxBot(vs.], 53 LOC,Context: 40 Pr ogrMO)am (vs. 21 LOC, 2 MO) 37 125 [boxLeft], 19 bot = 346 5 halfGauge = 58 3 w = 320 4 r = 166 15 rhombusFunc point halfW halfH Current le: Untitled9 * let onPerpendicularLine3 = onPerpendicularLineContext: onLine2 Program pt2 -1! 38 hangDistance = 232 Undo Redo Clean Up 126 [boxFrontRun ] 6 ⦀ 4 5 16 10127 ]) let line1 = 20line 0 5 pt1 pt2 in 6 attachmentPts = nPointsOnCircle 7{0-10} 0.0628000000000000117 branchHalfW39{-3.14 = 48 1 point = [236, 241] Undo Redo Clean Up Run 7 yOffset = y1 - halfGauge 5 ∂ = 23 Sketch-n-Sketch File Code Tools View Options Output Tools 11 let line2 = line 0 5 onPerpendicularLine2 pt2 in 7 18 40 hangingTray1 = hangingTray point3 hangDistance2 21 leftLine = line color strokeWidth8 [left, top] [left, bot] Current le: Untitled * Context: Program 1 6 8 color = 434 19 y1Offset41 = branchY - branchHalfW 3 circles = 12 let line3 = line 0 5 onPerpendicularLine3 pt2 in 9 yOffset2 = y1 + halfGauge Undo Redo Clean Up Run 2 [x, y] as point = [118, 193] 7 lambdaFunc ([left, top] as topLeft) w ∂ leftColor botColor bigCo 9 20 42 hangingTray2 = hangingTray point hangDistance4 map (\i -> 13 [line1, line222, line3] 10 1 21 y1Offset2 = branchY + branchHalfW 5 circle (if mod i 2! == 0! then 03 else 466) point (22 + i * 11 railOverExtension = 40 8 let bot = top + w in 2 boxes10 = spokeFunc point2 = 43 22 6 (reverse (zeroTo 5{0-15})) 4 woodHalfL14 = 98 23 rightLine = line color strokeWidth [ left+ w, top] [ left+ w, bot 9 let [x1, bot] as botLeft = [left, bot] in 3 map11 (\i -> line color 5 point point2 44 svg (concat [ ⦀ 12 4 rect 200 [ 50 + i * 76, 110] 55 195) 23 x1Offset = branchLeft + 405 7 5 15 arrow = arrowFunc pt1 pt2 13 firstTieX = x + railOverExtension 10 let right2 = x1 + w in 5 12(zeroTo 7{0-15}) 45 [pillar], 8 svg (concat [ 6 24 6 taperStartX = x + woodHalfL 24 14 13 spokes = 46 [base], 16 11 let right3 = left + w in 7 svg (concat [ 25 branch = 9 circles 7 15 y1Offset = y1 - halfGauge 8 boxes14 map spokeFunc attachmentPts 26 let47 pts [ leftArm= [[branchLeft], , y1Offset], [x1Offset10, ])branchY], [branc 17 arrowFunc1 = arrowFunc25 svg [295(concat, 378] [432[ , 428] 12 let yOffset2 = top + ∂ in 9 ]) 8 [x1, y1] as point1 = [taperStartX, y] 16 15 27 let48 [ color [rightArm, strokeColor], , strokeWidth] = [29, 360, 2] in 13 let xOffset2 = left + ∂ in 9 18 17 y1Offset2 = y1 + halfGauge 16 carFunc center2 = 28 polygon color strokeColor strokeWidth pts 26 rungs, 49 hangingTray1, 10 pencilHalfW19 =arrowFunc2 45 = arrowFunc [324, 535] [245, 413] 18 14 let leftOffset = x1 + ∂ in 17 squareByCenter 48 center2 25 29 50 hangingTray2 11 27 [leftLine], 19 endTiesX = ⦀x1 - railOverExtension 15 let botOffset = bot - ∂ in 18 30 deadspace = 72 20 ⦀ 51 ]) 12 top = y1 - pencilHalfW 20 16 let rightOffset = right2 - ∂ in 19 cars = 31 21 svg (concat [ ⦀ 13 28 [rightLine] 21 pointsBetweenSepBy2 = pointsBetweenSepBy [firstTieX, y1] [endTie 17 let [right, bot] as botRight = [right2, bot] in 20 map carFunc attachmentPts 32 leafAttachmentStartX = branchLeft + deadspace 33 14 bot = y122 + pencilHalfW arrow, 22 21 ⦀ 29 ]) 18 let yOffset = bot - ∂ in 34 leafAttachmentEndX = x1Offset - deadspace 15 23 arrowFunc1, 23 tieOverExtension = 32 22 capFunc point2 = 19 let midpoint2 = midpoint topLeft botRight in 35 16 tipX = x1 + 183 24 ⦀ 23 circle 364 point2 9 24 arrowFunc2 20 let [x, _] = midpoint2 in 36 leafAttachmentPts = pointsBetweenSepBy [leafAttachmentStartX, br 17 25 rectByCenter1Func point2 = 24 ⦀ 37 18 body = rectByCenter25 ]) 44 point woodHalfL pencilHalfW 26 rectByCenter 24 point2 17.5 (halfGauge + tieOverExtension) 21 let fstOffset = x - ∂ in 25 caps = (ix) Logo (via Three Tris) λ (x) N38 Boxesleaves = (xi) Ferris Wheel (xii) Tree19 Branch (xiii) Target (xiv) 27Pencil Tip λ (xv) Arrows λ (xvi) Rails 22 let [_, y] = midpoint2 in 26 map capFunc attachmentPts ★ 39 map rhombusFunc2★ leafAttachmentPts ★ ★ ★ ⦀ ★ ★ ★ 20 ratio = 0.651569678605651 28 repeatedRectByCenter1Func = 23 let sndOffset = y + ∂ in 27 40 21 29 map rectByCenter1Func pointsBetweenSepBy2 41 svg (concat [ 24 let leftTri = 28 ring1 = 40ring LOCcolor 7 point r 7 LOC 25 LOC 31 LOC 8 LOC 30 28 LOC 20 LOC 23 LOC 42 [branch], 22 leadStartBotPt = onLine [x1, bot] [tipX, y1] ratio 25 let pts = [[left, yOffset2], [fstOffset, y], [x1, botOffset] 29 31 color = 446 43 leaves 23 30 hub = circle 362 point 44 32 26 let [color, strokeColor, strokeWidth] = [leftColor, 360, 2] 44 ]) 24 leadStartTopPt = onLine [x1, top] [tipX, y1] ratio 31 27 polygon color strokeColor strokeWidth pts in 25 33 strokeWidth = 17 Figure32 svg (concat 15. [ Example programs created solely via output-directed manipulations. LOC = Source lines of code. = Final design is a function that 26 shavedWood = 34 λ 28 let botTri = 33 [hub], 27 let pts = [[x1, bot], [x1, top], leadStartTopPt, leadStartBotP 35 line1 = line color strokeWidth [x, yOffset] [x1, y1Offset] 29 let pts = [[leftOffset, bot], [x, sndOffset], [rightOffset, 34 cars, 28 let [color, strokeColor, strokeWidth] = [464, 360, 0] in 36 30 let [color, strokeColor, strokeWidth] = [botColor, 360, 2] i appears35 spokes, as a drawing tool. COMPARISON TO [19]: If an example can similarly be created in [19], code metrics for [19] appear in parentheses. 29 polygon color strokeColor strokeWidth pts 37 line2 = line color strokeWidth [x, yOffset2] [x1, y1Offset2] 31 polygon color strokeColor strokeWidth pts in 36 [ring1], 38 30 37 caps 39 svg (concat [ 32 let bigTri = Math operations (MO) indicate code simplifications due to our31 improvements.lead = # (resp. $) = Task cannot (resp. can with undesired parameterization) 38 ]) 40 repeatedRectByCenter1Func, 33 let pts = [[xOffset2, top], [right3, top], [right, yOffset]] 32 let pts = [leadStartBotPt, leadStartTopPt, [tipX, y1]] in 41 34 let [color, strokeColor, strokeWidth] = [bigColor, 360, 2] i 33 let [color, strokeColor, strokeWidth] = [409, 360, 0] in 35 polygon color strokeColor strokeWidth pts in be created in [19]. SOURCE OF EXAMPLES: WWID: PBD34 [34]: polygon Taskscolor strokeColor marked strokeWidth pts with underline (dashed = only partially completed). Lillicon [5]: 36 [leftTri, botTri, bigTri] 35 37 (vii) Battery. QuickDraw [8]: (viii) Ladder.SKETCH-N-SKETCH36[19]:svg (concat(ix) [ Logo (via Three Tris).SKETCH-N-SKETCH [9]: (x) N Boxes, (xi) Ferris Wheel. 37 [body], 38 lambda = lambdaFunc topLeft w ∂ 26 234 144 38 [shavedWood], 39 39 [lead] 40 svg (concat [ 40 ]) 41 equilateral triangle (equiTriPt, Figure 1e). The two helpers After drawing the are created without text edits via the RELATE and MAKE function between EQUAL tools, respectively. Each helper takes in two points the remaining three and is thus exposed as a drawing tool (Figure 1c). pairs of points the design looks like a The Koch motif is created by fractal (Figure 17). oneThirdPt snap-drawing However, only one forward and backwards to point is output from yield the 1 and 2 points; 3 3 the function—the equiTriPt is used to create white points are a point equidistant from the Figure 17. Recursive case. 1 2 only intermediates. 3 and 3 points. The bottom Moreover, most of those intermediates are inside calls to the of Figure 16 shows this base case. Although focused on the function’s recursive case, initial motif after abstraction. to modify the output of the base case, the programmer can Note that when a function is click the call widget of a call to the base case (the inner light focused, as in Figure 16, input gray border in Figure 17) to focus that call instead (Figure 18). points are rendered in orange Figure 16. Initial Koch motif. and output points in blue. The code is not recursive yet: The ADDTO OUTPUT and REORDER makeKochPts point point2= IN LIST tools are used to place the let oneThirdPt2= oneThirdPt point point2 in selected points into the output of the let oneThirdPt3= oneThirdPt point2 point in base case (resulting in the list on line equiTriPt oneThirdPt3 oneThirdPt2 16 of Figure 1). Once complete, the recursive case may be refocused, and, Drawing the function inside itself causes SKETCH-N-SKETCH using the same two tools, its output is to insert a recursive skeleton and the recursive call: specified to be the concatenation of the Figure 18. Base case. makeKochPts point point2= lists returned from calls the the base case (line 22 of Figure 1). let oneThirdPt2= oneThirdPt point point2 in let oneThirdPt3= oneThirdPt point2 point in What remains is to choose a termination condition. The fo- let equiTriPt2= equiTriPt oneThirdPt3 oneThirdPt2 in cused call widget for makeKochPts displays the conditional if ??terminationCondition then not <| ??terminationCondition for the recursive case. equiTriPt2 Clicking the conditional offers termination conditions. Cur- else let makeKochPts2= makeKochPts point oneThirdPt3 in rently, SKETCH-N-SKETCH implements only one: fixed depth, equiTriPt2 in which a depth argument is decremented on the recursive calls. These changes are visible in the final code (Figure 1f). To avoid infinite recursion, the temporary guard expression With all the points for the Koch snowflake now on the canvas, ??terminationCondition false evaluates to the first time a polygon is attached to them by selecting the “Polygon” tool true the function is encountered, and if the function has and clicking the list widget to select the points. With widgets appeared earlier in the call stack, affecting termination at a hidden, the final snowflake is shown in Figure 1. fixed depth. The termination condition may be selected later. rhombusFunc2([x,y] as point) = Parametric Computer-Aided Design (CAD) let halfW = 40 in Feature-based parametric CAD systems record user actions as let halfH = 83 in a series of steps that together act as a program encoding the rhombusFunc point halfW halfH creation of the design. If an element’s property is changed, repeatedRhombusFunc2= dependent actions in the sequence are re-run. Among CAD map rhombusFunc2 leafAttachmentPts systems, EBP [33] is notable for offering a programming by demonstration workflow to create loops and conditionals. Figure 19. Code fragment from Figure 15xii produced by the REPEAT OVER EXISTING LIST tool. The tool creates a function (rhombusFunc2) Drawing With Constraints that produces a shape given a single point, and maps that function over Several visual design systems integrate constraint specifica- a list of points defined elsewhere in the program (leafAttachmentPts). tion (e.g. [48, 22]). Notably, Apparatus [37], Recursive Draw- ing [36], and Geometer’s SketchPad [21] support recursion. Repetition Like CAD, but unlike SKETCH-N-SKETCH, these systems Of the Figure 15 examples, 7 utilize SKETCH-N-SKETCH’s do not represent the design in a general-purpose, text-based tools for creating repetitive designs. A selected shape may be programming language. repeated either over an existing list of points in the program or Constraint-Oriented Programming (COP) over a new call to any function that returns a list of points (the Other constraint-oriented systems, following in the footsteps default toolbox in Figure 1a contains several such functions). of SketchPad [42], explicitly view building a constrained sys- Figure 19 illustrates code produced by these repetition tools. tem as a programming task [6, 20, 15]. While offering varying For creating designs that vary over a non-spatial attribute, degrees of visibility into the code, these systems are distin- SKETCH-N-SKETCH alternatively offers a programming by guished by running constraint solvers alongside the program. demonstration workflow: after manually laying out the first We instead rely on standard execution semantics. few shapes of a repetitive design, the REPEAT BY INDEXED Programming by Demonstration (PBD) MERGED tool syntactically merges the shape expressions into Several PBD approaches [10] use shape drawing as a domain a single function that is called with an argument i that takes for exploring non-textual programming techniques. These sys- on the values 0, 1, 2, etc. Using a simple form of sketch- tems usually rely on a visual rather than representation of the based synthesis [41], a special tool called FILL HOLE resolves program (e.g. [23, 25]) or show actions step-by-step [28]. We syntactic differences between the original shape expressions also use shape drawing as our application domain, but we do by synthesizing expressions that refer to the index variable i. not hide the program text. Although not as visual as peer PBD Figure 15x and xiii were constructed with this workflow. systems, Tinker [26] is notable for supporting recursion by demonstration; indeed, any Lisp expression may be created, al- Limitations: Remaining Tasks beit via manipulations performed on a symbolic representation Of the 15 tasks in the Watch What I Do: Programming by not far removed from the underlying Lisp. More recently, PBD Demonstration [34] benchmark suite that may be interpreted techniques have been developed for data visualization [44], as parametric drawings, our system is able to fully complete 4. mobile applications for collaboration [12], web scraping [3, What is needed to complete all the tasks without text edits? 7], API discovery [49], and hand-drawing recognition [13]. Two of these remaining WWID: PBD tasks can be partially Output-Directed Programming completed in this work (Figure 15v, vi). To fully complete Several recent systems augment a regular text-based program- them, “Box Volume” would require an interaction to com- ming experience with abilities to directly manipulate output pute and display the numeric volume of the folded box, and to enact code changes. The transformations available may be “Xs” would require more precise control over what definitions “small” (such as changing constants [9, 24, 29], strings [46, are abstracted. (Not all uses of a squareWidth parameter 39, 24, 29], or list literals [29]), but several systems enable are pulled into the abstraction, causing the design to render “larger” program changes via output manipulation. incorrectly when drawing an X with different sized squares.) Transmorphic [38] re-implements the Morphic UI frame- The remaining nine tasks are diverse; no single feature would work [27] but with stateless views. Transmorphic retains help with more than two or three. A prominent missing feature Morphic’s ability to manipulate shapes by affecting changes is arbitrary text boxes, with other elements placed relative to to a view’s (text-based) code rather than changing live object the text size. Beyond this, several examples require various list state. Adding and removing shapes as well as changing a operations. SKETCH-N-SKETCH would also need to reason shape’s primitive properties are both supported. about intersections of lines with shape edges, to offer ways to specify overlapping and containment constraints, and to solve APX [31, 32], like SKETCH-N-SKETCH, is a two-pane (code different kinds of such constraints simultaneously. Finally, box and output canvas) environment for creating shape- one example would require creating if-then-else branches drawing programs (APX also supports dynamic visual simula- outside of a recursive or hole-filling setting. tions). On the canvas, manipulating shape attributes changes appropriate numbers in the program; a few larger changes (e.g. RELATED WORK grouping and inserting shapes) are also supported. However, Many related approaches provide direct manipulation tools to most of APX’s interactions focus on directly manipulating (a) transform programs and/or (b) build parametric designs. terms in the code box rather than on the canvas. Prior SKETCH-N-SKETCH ODP for Novices Our prior work on SKETCH-N-SKETCH [19] introduced a Our work so far has assumed the SKETCH-N-SKETCH user is draw-relate-abstract workflow, including the ability to select comfortable working in code to understand program operation. sub-values in the output and invoke certain program transfor- ODP interactions might also help those with little program- mations. Figure 13 indicates which tools also appeared in ming experience—such as domain experts or students—to this prior work. In the present work, besides exposing inter- quickly produce rudimentary programs. Design considera- mediates for manipulation, offering expression focusing, and tions for novices should be investigated. adding repetition and generic refactoring tools, we also sought to improve the prior draw-relate-abstract tools. Widget Visibility Because program execution may involve a large number of The prior SKETCH-N-SKETCH [19] relied on syntactic restric- intermediate evaluation steps, even simple programs might tions: shapes followed a strict “left-top-right-bot” bounding clutter the canvas with widgets, making it unusable. Therefore, box parameterization, special functions were needed to com- our current implementation hides most widgets by default and pose shapes, and the program needed to be a series of top-level uses heuristics to determine when to show them—generally, definitions followed by a list literal “main” expression refer- upon the mouse hovering over some associated shape. Addi- ring to the top-level definitions. If, through text-editing, the tionally, widgets from intermediate expressions in standard programmer veered outside of this syntactic subset of the lan- library code—outside the visible program—are generally not guage, many output-directed tools became unavailable. shown. In the future, these visibility choices could be more In this work, we relaxed these syntactic restrictions and moved systematically controlled, e.g., by source code annotations or the operation of SKETCH-N-SKETCH’s tools closer to ordi- user interface options. nary programming concepts. The drawing tools now operate Customizing Widgets based on types rather than a syntactic bounding box construct. Mechanisms for customization may help address the open No longer are special function calls needed to compose shapes: question of how to render graphical representations of pro- groups are now ordinary lists. Our generic provenance tracing gram evaluation and of domain objects that are not inherently enables the tools to transform shapes not defined at the top visual. One approach would be to design an API for “toSvg” level. We also now offer multiple results for transformations. functions that specify how to render graphical representations We additionally improved code generation. Four of the ex- of intermediate data structures. Analogous to “toString” amples in Figure 15 (ii, iii, iv, vii) can also be constructed functions that render text, a toSvg API might similarly render without text edits in the prior SKETCH-N-SKETCH [19], al- composite data structures via recursively calling toSvg for beit with undesirable parameterizations for two of the four each of the structure’s elements. (ii, iv). Figure 15 lists the source lines of code (LOC) and Synthesizing Program Transformations math operations (MO) for the programs in both the present and Each program transformation offered by SKETCH-N-SKETCH prior versions of SKETCH-N-SKETCH. Reductions in LOC has been hand-coded one-by-one. This effort is time- are incidental—our drawing tools now insert single-line func- consuming, error-prone, and limits the composability of trans- tion calls instead of wordier multi-line definitions. Reductions formations. Program synthesis techniques have been suc- in math operations, however, do indicate better code: the prior cessfully incorporated into refactoring tasks for class-based SKETCH-N-SKETCH sometimes inserted large math expres- languages [35]; perhaps such techniques can be extended to sions too complex for human comprehension. Besides discard- streamline the specification of output-directed transforms that ing the bounding boxes that required properties such as shape operate simultaneously on both abstract and concrete syntax. width to be computed (e.g. as right - left), we also more aggressively reuse variables and simplify math in the MAKE User Interactions for Deciphering Human Intent EQUAL tool, aided by connecting SKETCH-N-SKETCH to an There are often multiple valid program changes associated external solver (REDUCE [18]). with a specific user action. To resolve ambiguity, our sys- During our upgrades, some functionality was disabled but tem displays code and output differences and asks the user would be useful to restore. Most notably, unneeded by our to choose. Richer interactions are needed to explain the dif- examples, the current SKETCH-N-SKETCH lacks a path tool. ferences (e.g., change impact analysis [16, 11]) and to bet- Also, tools to re-parameterize shapes en masse could be useful, ter resolve the user’s intent (e.g., by asking additional ques- if, for example, a bounding box parameterization is desired. tions [30]). In order for the live and immediate experience afforded by CONCLUSION AND FUTURE WORK output-directed programming to scale into usable systems for We improved expressiveness in SKETCH-N-SKETCH, show- different programming tasks and user scenarios, developing ing how to construct a variety of non-trivial programs entirely and refining such user interaction techniques remain crucial through direct manipulation. Our long-term goal, however, challenges to tackle in future work. is not to create a practical drawing tool but to use this par- ticular application domain as a laboratory for advancing the ACKNOWLEDGEMENTS expressiveness of output-directed programming for a variety Our thanks to Philip Guo and the anonymous reviewers for of future programming settings. There are several avenues to helpful feedback. This work was supported by U.S. National explore in this direction. Science Foundation Grant No. 1651794. REFERENCES Krahn. 2014. Babelsberg/JS - A Browser-Based [1] Umut A. Acar, Amal Ahmed, James Cheney, and Roly Implementation of An Object Constraint Language. In Perera. 2013. A Core Calculus for Provenance. Journal European Conference on Object-Oriented Programming of Computer Security (2013). (ECOOP). [2] Autodesk Inc. Maya. [16] Malcom Gethers, Bogdan Dit, Huzefa Kagdi, and Denys http://www.autodesk.com/products/maya/overview. Poshyvanyk. 2012. Integrated Impact Analysis for Managing Software Changes. In International [3] Shaon Barman, Sarah Chasins, Rastislav Bodik, and Conference on Software Engineering (ICSE). Sumit Gulwani. 2016. Ringer: Web Automation by Demonstration. In Object-Oriented Programming, [17] Michael Gleicher and Andrew Witkin. 1994. Drawing Systems, Languages, and Applications (OOPSLA). with Constraints. The Visual Computer: International Journal of Computer Graphics (1994). [4] Andrew Begel. 1996. LogoBlocks: A Graphical Programming Language for Interacting with the World. [18] Anthony C. Hearn. 1968. REDUCE: A User-Oriented Advanced Undergraduate Project, MIT Media Lab. Interactive System for Algebraic Simplification. In Interactive Systems for Experimental Applied [5] Gilbert Louis Bernstein and Wilmot Li. 2015. Lillicon: Mathematics. Academic Press. Using Transient Widgets to Create Scale Variations of Icons. Transactions on Graphics (TOG) (2015). [19] Brian Hempel and Ravi Chugh. 2016. Semi-Automated SVG Programming via Direct Manipulation. In [6] Alan Borning. 1981. The Programming Language Symposium on User Interface Software and Technology Aspects of ThingLab. Transactions on Programming (UIST). Languages and Systems (TOPLAS) (October 1981). [20] Allan Heydon and Greg Nelson. 1994. The Juno-2 [7] Sarah E. Chasins, Maria Mueller, and Rastislav Bodik. Constraint-Based Drawing Editor. In Technical Report 2018. Rousillon: Scraping Distributed Hierarchical Web 131a, Digital Systems Research, Digital Equipment Data. In Symposium on User Interface Software and Corporation. Technology (UIST). [21] R. Nicholas Jackiw and William F. Finzer. 1993. The [8] Salman Cheema, Sumit Gulwani, and Joseph LaViola. Geometer’s Sketchpad: Programming by Geometry. In 2012. QuickDraw: Improving Drawing Experience for Watch What I Do: Programming by Demonstration. MIT Geometric Diagrams. In Conference on Human Factors Press. in Computing Systems (CHI). [22] Jennifer Jacobs, Sumit Gogia, Radomír Mech, and [9] Ravi Chugh, Brian Hempel, Mitchell Spradlin, and Joel R. Brandt. 2017. Supporting Expressive Procedural Jacob Albers. 2016. Programmatic and Direct Art Creation Through Direct Manipulation. In Manipulation, Together at Last. In Conference on Proceedings of the 2017 CHI Conference on Human Programming Language Design and Implementation Factors in Computing Systems, Denver, CO, USA, May (PLDI). 06-11, 2017. [10] Allen Cypher, Daniel C. Halbert, David Kurlander, [23] David Kurlander. 1993. Graphical Editing by Example. Henry Lieberman, David Maulsby, Brad A. Myers, and Ph.D. Dissertation. Columbia University. Alan Turransky (Eds.). 1993. Watch What I Do: Programming by Demonstration. MIT Press. [24] Kevin Kwok and Guillermo Webster. 2016. Carbide Alpha. https://alpha.trycarbide.com/. [11] Bogdan Dit, Michael Wagner, Shasha Wen, Weilin Wang, Mario Linares Vásquez, Denys Poshyvanyk, and [25] Henry Lieberman. 1993a. Mondrian: A Teachable Huzefa H. Kagdi. 2014. ImpactMiner: A Tool for Graphical Editor. In Watch What I Do: Programming by Change Impact Analysis. In International Conference on Demonstration. MIT Press. Software Engineering, Companion Proceedings [26] Henry Lieberman. 1993b. Tinker: A Programming By (ICSE-C). Demonstration System for Beginning Programmers. In [12] Jonathan Edwards, Jodie Chen, and Alessandro Warth. Watch What I Do: Programming by Demonstration. MIT 2016. Live End-User Programming. In LIVE Workshop. Press. [13] Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, [27] John H. Maloney and Randall B. Smith. 1995. and Josh Tenenbaum. 2018. Learning To Infer Graphics Directness and Liveness In the Morphic User Interface Programs From Hand-Drawn Images. In Conference on Construction Environment. In Symposium on User Neural Information Processing Systems (NIPS). Interface Software and Technology (UIST). [14] Evan Czaplicki. 2012-2019. Elm. http://elm-lang.org. [28] David L. Maulsby, Ian H. Witten, and Kenneth A. Kittlitz. 1989. Metamouse: Specifying Graphical [15] Tim Felgentreff, Alan Borning, Robert Hirschfeld, Jens Procedures by Example. In Conference on Computer Lincke, Yoshiki Ohshima, Bert Freudenberg, and Robert Graphics and Interactive Techniques (SIGGRAPH). [29] Mikaël Mayer, Viktor Kuncak,ˇ and Ravi Chugh. 2018. [39] Christopher Schuster and Cormac Flanagan. 2016. Live Bidirectional Evaluation with Direct Manipulation. Programming by Example: Using Direct Manipulation Proceedings of the ACM on Programming Languages for Live Program Synthesis. In LIVE Workshop. (PACMPL), Issue OOPSLA (2018). [40] Ben Shneiderman. 1983. Direct Manipulation: A Step [30] Mikaël Mayer, Gustavo Soares, Maxim Grechkin, Vu Beyond Programming Languages. Computer (August Le, Mark Marron, Oleksandr Polozov, Rishabh Singh, 1983). Benjamin Zorn, and Sumit Gulwani. 2015. User Interaction Models for Disambiguation in Programming [41] Armando Solar-Lezama. 2008. Program Synthesis by by Example. In Symposium on User Interface Software Sketching. Ph.D. Dissertation. UC Berkeley. and Technology (UIST). [42] Ivan Sutherland. 1963. Sketchpad, A Man-Machine [31] McDirmid, Sean. 2015. A Live Programming Graphical Communication System. Ph.D. Dissertation. Experience. In Future Programming Workshop, Strange MIT. Loop. https://onedrive.live.com/download?cid= [43] William Robert Sutherland. 1966. The On-line 51C4267D41507773&resid=51C4267D41507773%2111492& Graphical Specification of Computer Procedures. Ph.D. authkey=AMwcxdryTyPiuW8 Dissertation. Massachusetts Institute of Technology. https://www.youtube.com/watch?v=YLrdhFEAiqo. [44] Victor, Bret. 2013. Drawing Dynamic Visualizations. [32] McDirmid, Sean. 2016. The Future of Programming will http: be Live. In Curry On. //worrydream.com/#!/DrawingDynamicVisualizationsTalk. https://www.youtube.com/watch?v=bnqkglrSqrg. [45] Helge von Koch. 1904. Sur une courbe continue sans [33] Guy Pierra, Jean-Claude Potier, and Patrick Girard. tangente, obtenue par une construction géométrique 1996. The EBP System: Example Based Programming élémentaire. Arkiv för Matematik, Astronomi och Fysik 1 System for Parametric Design. In Modelling and (1904), 681–704. Graphics in Science and Technology. Springer Berlin Heidelberg. [46] Xiaoyin Wang, Lu Zhang, Tao Xie, Yingfei Xiong, and Hong Mei. 2012. Automating Presentation Changes in [34] Richard Potter and David Maulsby (Eds.). 1993. A Test Dynamic Web Applications via Collaborative Hybrid Suite for Programming by Demonstration. In Watch Analysis. In International Symposium on the What I Do: Programming by Demonstration. MIT Press. Foundations of Software Engineering (FSE). [35] Veselin Raychev, Max Schäfer, Manu Sridharan, and Martin Vechev. 2013. Refactoring with Synthesis. In [47] World Wide Web Consortium (W3C). Scalable Vector Object-Oriented Programming, Systems, Languages, Graphics (SVG) 1.1 (Second Edition). and Applications (OOPSLA). http://www.w3.org/TR/SVG11/. [36] Schachman, Toby. 2012. Recursive Drawing. Master’s [48] Haijun Xia, Bruno Araújo, Tovi Grossman, and Daniel J. thesis. New York University Interactive Wigdor. 2016. Object-Oriented Drawing. In Conference Telecommunications Program. on Human Factors in Computing Systems (CHI). http://recursivedrawing.com/. [49] Kuat Yessenov, Ivan Kuraj, and Armando Solar-Lezama. [37] Schachman, Toby. 2015. Apparatus. http://aprt.us/. 2017. DemoMatch: API Discovery from Demonstrations. In Conference on Programming [38] Robin Schreiber, Robert Krahn, Daniel H. H. Ingalls, Language Design and Implementation (PLDI). and Robert Hirschfeld. 2016. Transmorphic: Mapping Direct Manipulation to Source Code Transformations. APPENDIX or introducing a variable). This internal workflow has been We provide a few additional details about our design and an effective way to express and resolve equality constraints implementation; discuss examples from prior work [19]; and within template code. describe two examples from Figure 15 in more detail. Specifying the Focused Expression Type Inference A focused function and call are denoted by special comments, Because SKETCH-N-SKETCH designs are ordinary programs, automatically inserted into the program. Comments, unlike custom functions that produce shapes can be incorporated internal expression identifiers, are preserved across arbitrary as new drawing tools in the user interface. Any function that text edits to the program. takes two points as input—or one point and a horizontal and/or vertical distance—is exposed as a custom drawing tool. Completing the Examples from the Prior Sketch-n-Sketch To determine which numbers represent horizontal/vertical dis- Our prior work [19] presented three examples that require text tances and to provide reasonable default values for e.g. colors edits both in the prior system and ours. Below, we discuss the and stroke widths, type inference tags each type with a set of additional features that would be required to complete these zero or more inferred roles. Functions in the standard library tasks entirely via output-directed manipulations. (the bottom portion of Figure 1a) have role annotations in Two of the remaining examples in [19] utilize paths (“Garden their type declarations. By using these functions, role infor- Logo” and “Coffee Mug”). Unneeded by our examples, we mation enters the program and propagates via type inference, disabled the path tool pending internal updates to allow path usually obviating the need for manual annotation. A similar drawing to work with snaps. Besides a path tool, a REWRITE scheme called brands (implemented via named mixin traits) is AS OFFSETS tool would be useful to rewrite the coordinates used in [31], although we additionally apply certain built-in of selected points to be computed relative to an anchor point usage rules—e.g. that a number added to an X value must be a (akin to the domain-specific rewriting performed by the prior HorizontalDistance—to infer roles in more cases. GROUP tool). “Garden Logo” would also benefit from special tooling for mirrored drawing across a vertical symmetry line. Value Holes Tagging values with provenance enables a helpful trick that The third remaining example (“Snip Polygon”) relied on spe- we use to implement several of the new transformations (most cial tooling to draw a polygon with its points positioned rela- notably the snap-drawing): to indicate that a particular pro- tive to a bounding box. That tooling was discarded along with gram location should somehow evaluate to a particular value, the pervasive bounding box parameterization and would have we transiently insert that value as a leaf in the abstract syntax to be restored (perhaps as a REWRITE RELATIVE TO BOUNDS tree. Before displaying the program, each such value hole is tool). Additionally, to fully develop the design without text ed- filled by inspecting the provenance of the value and choosing its, SKETCH-N-SKETCH would need an output-directed mech- an expression that evaluates to the value (usually by using anism for specifying min/max calculations. Example (xii): Tree Branch Example (xiii): Target Tools highlighted: DRAW OFFSET,REPEAT OVER LIST Tools highlighted: REPEAT BY INDEXED MERGE,FILL HOLE

To demonstrate offset widgets and repetition tools, we describe Repeating over a point list allows copies of shapes to vary part of the tree branch task shown in Figure 15ii. The overar- in their spacial positions, but not over any other attributes ching strategy is to build a rhombus abstracted over its center, such as size or color. To support other repetition scenarios and then repeat it over a list of points to form the leaves. We where the varying attributes could be calculated from an index omit mentioning uses of RENAME. (i.e. 0, 1, 2,...), we offer a workflow which we now briefly demonstrate by the construction of a target (Figure 15iii). The rhombus is constructed around a central point using offsets drawn with the “Point or Offset” tool. Offset amounts may To start, we draw three concentric circles snapped to the same snap to each other while drawing, which inserts a variable for center point and change the color of the middle circle. After the shared offset amount (halfW and halfH below). selecting the three circles, we invoke REPEAT BY INDEXED [x,y] as point = [208, 256] MERGE, from which we select the second of two results (which differs from the first only in that it adds the reverse halfW = 102 call seen below, so that i = 0 for the smallest circle).

xOffset=x+ halfW ... circles= xOffset2=x- halfW map (\i -> circle ??(1 => 0, 2 => 466, 3 => 0) \ halfH = 145 point\ ??(1 => 114, 2 => 68, 3 => 25)) yOffset=y- halfH (reverse (zeroTo 3{0-15})) ... yOffset2=y+ halfH ... The program maps an anonymous function that takes an index (\i -> ...) over the list [2,1,0]; each index is thus trans- We then draw a polygon snapped to the offset endpoints and formed into one of the circles. The anonymous function is a ABSTRACT the resulting rhombus into a function parameter- syntactic merger of the original three circle definitions. Their ized over [x, y], halfW, and halfH. We will attach instances differences—radius and color—have been turned into what of this function over the branch. we call programming-by-example (PBE) holes, represented by ??(...) The branch construction . The first PBE hole above can be read as “the first 0 is shown at right. On top time this expression is executed it should return , the second 466 0 of a brown polygon, two time it is executed it should return , and the third time ”. “deadspace” offsets are SKETCH-N-SKETCH drawn inwards from the employs sketch-based ends of the branch to form synthesis [41] to the start and end locations resolve these holes. of the attachment points As shown at right, for the leaves. Between all suggested fillings these locations we draw the involve i, the only pointsBetweenSepBy standard library function that returns variable that differs in points separated from their neighbors by a fixed distance. the execution environments. For the first hole, we choose With this particular function, making our branch longer will the mod i 2 == 0! conditional to obtain alternating colors. add more leaves rather than spacing them out. For the second we choose the only option, a base + i ∗ width Finally, to repeat our leaf rhombus over the points, we first expression, to calculate the radii of our circles. Similar to the Koch snowflake, the inserted code zeroTo 3{0-15} contains select the one copy of the rhombus and invoke the REPEAT a slider annotation allowing us to manipulate the number of OVER LIST tool to repeat over the attachment points we just drew. This tool creates a new function abstracted over just a circles. Figure 15xiii shows five circles for the final design. single point (rhombusFunc2 below) and maps that function over our leafAttachmentPts, completing our leafy branch. ... rhombusFunc2([x,y] as point) = let halfW = 40 in let halfH = 83 in rhombusFunc point halfW halfH ... repeatedRhombusFunc2= map rhombusFunc2 leafAttachmentPts ...