4 Drawing on SketchPad: Refl ections on Computer Science and HCI

Joseph A. Konstan University of Minnesota, Minneapolis, Minnesota, U.S.A.

I. Sutherland, 1963: “Sketchpad: A Man–Machine Graphical Communication System”

Sutherland’s SketchPad system, paper, and dissertation provide, for me, the answer to the oft-asked question: “why should HCI belong in a computer science program?” I fi rst came across the paper “SketchPad: A Man–Machine Graphical Communica- tion System” from the 1963 AFIPS conference nearly thirty years after it had been published (Sutherland 1963). I was nearing completion of my own dissertation in which I was exploring a variety of techniques for constructing user interface toolkits. At the time, the paper seemed little more than a handy reference in which I could trace the lineage of constraint programming in user interfaces—from Sketchpad, through Borning’s ThingLab (Borning 1981), to my own work, with various hops and detours along the way. I guess I was young and in a hurry. It was only a few years later when I started to teach this material that I realized how much of today’s computing traces its roots to Sutherland’s work.

What was this tremendous paper about? A drawing program. Not just any drawing program, but one that took full advantage of computing, a million-pixel display (albeit with a slow pixel-by-pixel refresh), a light pen, and various buttons and dials to empower users to draw and repeat patterns, to integrate constraints with draw- ings so as to better understand mechanical systems, and to draw circuit diagrams as input to simulators. Indeed, by placing constraints on drawings, the user could readily link shapes together or constrain them. Ignoring the advent of color and images, one can readily argue that deployed drawing programs didn’t surpass SketchPad until the early generations of computer-aided design (CAD) software packages. Indeed, although several low-distribution CAD systems became available 24 Joseph A. Konstan over the next two decades, in many ways the widespread availability and use of sophisticated CAD tools can be traced back to the founding of AutoDesk and the release of AutoCAD in 1982. For me, however, the importance of SketchPad lies in its implicit argument that HCI and advances in the science of computing are so closely intertwined. I’ll provide just a few examples.

Pointing with a light pen A substantial part of Sutherland’s work related to the use of a light pen as an input device. What’s impressive about this work is not simply the engineering needed to make it work, but Sutherland’s awareness of the human need to point at things rather than generically at the display. His pseudo pen location—a semantic mapping of the pen’s location into drawing space—is the predecessor of modern pointing and dragging operations that snap to object attach- ment points and grids (see, e.g., Bier and Stone 1986). The taming of the light pen is a multifaceted triumph. First, it is no mean feat simply to fi nd the pen in the fi rst place (a task that requires that the user point to an illuminated pixel on the screen). Sutherland’s explanation of “inking up” and the provision of a default illuminated space helped here. But more important, precise pointing was exceedingly diffi cult, as a result of both natural hand movements and the occlusion of the target by the pen and the user’s hand. Having a pseudo pen location addresses these issues by identifying points where the user is most likely to be interested in pointing. Thus, the user needn’t point exactly at a line to select it (merely close enough) and can connect ends of segments together by simply moving one close enough to the other.

Rendering of lines, circles, and text As a new faculty member, I was assigned to teach computer graphics. Of course, this meant that I had to learn computer graph- ics. One of the fi rst areas to excite me was two-dimensional rendering—particularly, incremental algorithms for rendering lines and curves. Imagine my surprise when seeing that Sutherland had anticipated much of this work (if without some of the later elegance of effi cient integer algorithms). Even more impressive is the care he evidently put into making sure that the circles would look right. He did this in the straightforward way (by identifying points around an eighth of a circle, and then replicating those points around the circle), but I haven’t found earlier work laying out the algorithm. Similarly, he already had font tables for display of text and numbers, which greatly increased the capabilities of the display for engineering drawings, particularly since it allowed labels to be rendered at different angles. Drawing on SketchPad 25

Constraints and their display Constraints were the reason I found SketchPad, but in the end I was more impressed with the fact that Sutherland had already antici- pated the need to display constraints over the drawings to which they were applied. His constraint language is certainly not for the novice, but then again, neither was his system. He supports a full range of position, orientation, and shape constraints that make it possible to, for example, create a regular hexagon by constraining the six lines of the hexagon to be the same size, and constraining the vertices to lie on a circle. The famous example of a modeled bridge with stresses on it gives additional insight into the power of the constraint systems, though its power becomes most apparent in the appendix to his dissertation where he lists all the constraint types. And the visualization of those constraints, though not necessarily a scalable solu- tion, made it possible to at least begin to understand the network of forces that were acting on your drawing behind the scenes. As a constraint programmer myself, I found that even applications with only a few dozen constraints could quickly overfl ow my head and force me to grapple with the complexity. How I would have liked to have a visualization tool such as Sutherland’s.

The data structures, algorithms, and programming structures As I see it, Sketch- Pad anticipated object-oriented (OO) programming (in ways that would later be extended through other user interface–centered research), and contains within it a collection of interesting data structures and algorithms. The ring buffers and inher- itance structures developed to make SketchPad programmable have much of the functionality of early class-based OO inheritance systems. Fundamentally, Sketch- Pad anticipated over fi fty years ago many of the programming ideas that are still with us today. As I pull together these examples and think of the paper as a whole, the paper presents the fi rst answer to the question posed above. It loudly asserts that HCI belongs as part of computer science because the needs of innovative interfaces drive forward the science of computing. Indeed, computer science has always been advanced through attempts to solve hard problems. Much of what we’ve learned about reliable computing and software engineering has emerged from trying to solve aerospace problems. Today we are advancing our theory of formal languages by solving problems in gene and protein bioinformatics. But the broader challenge of making a computer that can interact with humans in a manner supportive of human creativity—this is a challenge that has pushed forward nearly every aspect of our fi eld, from graphics (for interfaces) to operating systems (for responsive multipro- gramming) to speech recognition to networking and beyond. Sutherland’s quest for 26 Joseph A. Konstan a “man–machine graphical communication system” advanced interactive computing immeasurably, and our continuing quest to make computers serve as effective tools and partners with humans continues to drive the fi eld forward. To me, this is the key argument why computer science as a discipline must embrace applications- oriented computing in general, and HCI in particular. What lesson can we draw from Sutherland’s SketchPad work about the challenges that remain for HCI and computer science? I fi nd three lessons here: There is still a great amount of work to do on computing systems designed to serve as tools for expert users. HCI researchers, especially in the fi rst decades of the fi eld, focused substantial effort on novices, on walk-up-and-use interfaces, and on “knowledge workers” whose knowledge was channeled into fairly generic tools. Less work has been done on the potential for computing systems that may involve custom hardware and extensive investments in training. There are, of course, notable exceptions, including some of the work done on systems for CAD, air traffi c control, intelligence analysts, and others. But too often our textbooks and our widely pro- moted techniques fail to reach beyond to generic knowledge worker. Just to mention one simple example from SketchPad, I’d like to see more dials and buttons! Basic computing hardware and software has advanced to the point where few of us need to invent much to create new interfaces; but how often are we handicapping ourselves by accepting whatever today’s desktop computer has to offer? There is a wealth of exciting research that breaks this mold, adding everything from touch sensors to new displays to location awareness into our devices and applications. Some of this research is grounded in application ideas, but much of it is “for its own sake” technology. As a fi eld, we need to bring together the creativity behind these new interaction ideas with concern for users and applications. Sutherland created SketchPad as a way to think about computers for users—it is hard to believe that the details of interaction would have come out right it he’d thought of himself as a researcher exploring pixel displays and light pens for their own sake. It is critical to keep a presence of HCI within computer science—critical to both HCI and computer science. It is wonderful that there are many venues where HCI is advanced. Just in the academy, it has homes in cognitive science, design, informa- tion science, and various social science and engineering disciplines. There are good reasons for all of these connections, and they can help HCI move forward. However, it would be a big mistake if HCI migrated out of computer science departments entirely. From the perspective of HCI, it would cut us off from the venue where the future capabilities of computing are being developed. If we’re not there to help set the agenda, the next generations of networking, software tools, handheld comput- Drawing on SketchPad 27 ers, and development techniques will not be to our liking or particularly supportive of user experience. At the same time, computer science departments need us. They need someone “on the inside” who understands and promotes the importance of the human–computer connection. Failing that, it is too easy to imagine the fi eld being driven by the wrong problems and focusing too much on computation and not enough on communication. These ideas bring me back to Sutherland. I use the SketchPad paper at the start of my graduate seminar on HCI (which attracts a wide range of students), and for many years it has helped to bring home the importance of HCI in shaping comput- ing. The last time I taught this course, however, this paper elicited puzzled reactions. “Isn’t this how computers just are?” asked one of the students. Sutherland’s ideas have been in the core of computing for so long that we now have a generation of computer scientists who take them as a given. That’s a great success for our fi eld, but also a reason to make sure that we’re regularly nurturing the next SketchPads out there.