SATIN: A Toolkit for Informal Ink-based Applications Jason I. Hong and James A. Landay Group for User Interface Research, Computer Science Division University of California, Berkeley Berkeley, CA 94720-1776 USA +1 510 643 7354 {jasonh, landay}@cs.berkeley.edu ABSTRACT based interaction techniques have been developed, such as Software support for making effective pen-based gesturing1 and pie menus [5], these techniques have not yet applications is currently rudimentary. To facilitate the been widely adopted because they are difficult and time- creation of such applications, we have developed SATIN, a consuming to implement. Java-based toolkit designed to support the creation of With respect to input and output for pens, we are at a stage applications that leverage the informal nature of pens. This similar to that of windowing toolkits in the early 1980s. support includes a scenegraph for manipulating and Many example applications and many novel techniques rendering objects; support for zooming and rotating objects, exist, but there are no cohesive frameworks to support the switching between multiple views of an object, integration creation of effective pen-based applications. As a first step of pen input with interpreters, libraries for manipulating ink towards such a framework, we have developed SATIN2,a strokes, widgets optimized for pens, and compatibility with toolkit for supporting the creation of informal ink-based Java’s Swing toolkit. SATIN includes a generalized applications [15]. From a high-level perspective, there were architecture for handling pen input, consisting of three research goals for SATIN: recognizers, interpreters, and multi-interpreters. In this • paper, we describe the functionality and architecture of Design a generalized software architecture for SATIN, using two applications built with SATIN as informal pen-based applications, focusing on how to examples. handle sketching and gesturing in a reusable manner • Develop an extensible toolkit that simplifies the Keywords creation of such informal pen-based apps toolkits, pen, ink, informal, sketching, gesture, recognition, • Distribute this toolkit for general use by researchers interpreter, recognizer, SATIN As a first step, we surveyed existing pen-based applications INTRODUCTION (both commercial and research) in order to determine what Sketching and writing are natural activities in many shared functionality would be most useful. Afterwards, we settings. Using pen and paper, a person can quickly write implemented the first iteration of the toolkit in Java, and down ideas, as well as draw rough pictures and diagrams, built our first significant application with it, DENIM [26] deferring details until later. The informal nature of pens (see Fig. 1). From the lessons learned, we developed the allows people to focus on their task without having to worry second iteration of SATIN, and built another application, about precision. SketchySPICE. However, although more and more computing devices are In this paper, we first outline functionality common in coming equipped with pens, there are few useful pen-based existing pen-based applications, and take a look at current applications out there that take advantage of the fact that 1 software support for pen-based interfaces. We continue by pens are good for sketching . Most applications use pens describing the high-level and then detailed design of the only for selecting, tapping, and dragging. These SATIN toolkit. Specifically, we focus on a generalized applications simply treat the pen as another pointing device, architecture for handling pen input, consisting of three ignoring its unique affordances. components: recognizers, interpreters, and multi- Furthermore, the few compelling applications that do exist interpreters. We describe how pen input is handled in terms are built from scratch, despite the fact that many of them of the two applications, DENIM and SketchySPICE. We share the same kinds of functionality. This is because of the conclude with an evaluation of the toolkit, as well as our rudimentary software support for creating pen-based plans for future work and a discussion of lessons learned. applications. Despite the fact that many new and useful pen- 1 By sketching, we mean the process of drawing roughly and LEAVE BLANK THE LAST 2.5cm OF THE LEFT COLUMN quickly. We use the term ink for the strokes that appear. By gesturing, we mean a pen-drawn stroke that issues a command ON THE FIRST PAGE 2 FOR US TO PUT IN The SATIN project page and software download is at: THE COPYRIGHT NOTICE! http://guir.berkeley.edu/projects/satin Commercial Software Support for Pens PalmOS [8] offers some very simple pen input processing. The default behavior is to process strokes and taps in the silk screen area as key events, with all other strokes passed on to the application for processing. PalmOS also provides some APIs for getting individual stroke points, enabling and disabling the Graffiti shorthand recognizer, and for getting the last known location of the pen. Microsoft Windows for Pen Computing [29] provides minimal support for pens. Text entry areas were replaced either by handwriting edit controls (hedit)orbyboxed edit controls (bedit), in which individual characters can be written. Simple gesture recognition was also supported. These extensions give the developer very little support for building informal ink-based applications. Figure 1 – A screenshot of DENIM, a sketch-based web site design tool created on top of SATIN In Windows CE [30], pen input is treated as a subset of PEN APPLICATION SPACE mouse input. Applications can receive messages when the Recently, there have been many applications developed that pen is moved, goes down, comes up, and is double-tapped. use sketching and gesturing. We performed a survey of Windows CE also provides simple handwriting recognition. these applications, looking specifically for examples of NewtonOS [2] uses sheets of paper as its input metaphor. informal ink-based interaction, ones that step away from Users can write on these sheets without having to explicitly rigid structure and precise computation, instead supporting save. Furthermore, users can specify several ink modes in ambiguity, creativity, and communication [15]. Many pen- which strokes are processed as text, as shapes, or left based research systems have headed in the direction of unprocessed as raw ink. Recognition errors can be informal interfaces in recent years, either by not processing corrected by choosing from an n-best list. Gestures are also the ink [11, 41, 43] or by processing the ink internally while integrated into the system. Drawing a zig-zag shape over a displaying the unprocessed ink [14, 24, 32, 40]. word or shape, known as scrubbing, deletes that object. The applications we examined include design tools [9, 12, Holding down the pen for a second activates select mode. 14, 20-22, 24, 43, 47]; whiteboard applications [1, 32, 33, After select is enabled, the user can drag the pen and either 37]; annotation tools [41, 44-46]; note-taking applications highlight or circle the objects to select. Lastly, NewtonOS [10, 11, 42]; and applications demonstrating new provides an extensive widget set for pens, designed to interaction techniques [19, 28, 40]. These applications minimize the amount of end-user writing necessary. share much functionality with each other, including: Perhaps the most sophisticated commercial support for pens • Pen input as ink was in GO Corporation’s PenPoint [6]. PenPoint is an operating system built from the ground up to support pens. • Pen input as gestures Besides providing many of the services described above, • Pen input for selecting and moving such as gestures and pen widgets, PenPoint also has such • Interpreters that act on ink input features as live embedding of documents within documents, • Manipulation of other kinds of objects besides ink and extensive integration of gesture recognition and • Grouping of objects handwriting recognition. • Layering of objects There are two main differences between SATIN and the • Time indexing of ink input systems described above. First, all of the systems listed • Transformation of ink to other cleaned-up objects above are designed to build formal user interfaces, and are • Immediate and deferred processing of ink thus focused on handwriting recognition and form entry Later, in the process of developing DENIM, our first tasks. In contrast, SATIN is targeted towards the application, we discovered we needed techniques for development of informal ink-based applications. The managing information, and turned to using zooming and second difference is extensibility. Aside from handwriting semantic zooming, as demonstrated in Pad++ [3] and Jazz recognition, the systems listed above provide minimal [4]. We decided that this functionality was useful enough to support for manipulating and processing ink. In contrast, developers that it should be included in the toolkit. one of our primary goals with SATIN was to give EXISTING PEN FRAMEWORKS developers flexibility in how ink is processed and to make it In this section, we outline existing frameworks for simple to do so. For example, new gestures cannot be added developing pen-based applications, and describe where in the systems described above. SATIN builds on their ideas. Research Software Support for Pens SATIN is built in Java, using JDK1.33. SATIN uses Java2D Simple ink and gesture support is provided in Artkit [16]. for rendering, and makes extensive use of the Java core Artkit uses the notion of sensitive regions, invisible classes as well as the Swing windowing toolkit [31]. rectangles that can be placed on top of screen objects. The Fig. 2 shows how a pen-based application would be built sensitive region intercepts stroke input, and processes the using SATIN, Swing, and Java. Roughly speaking, SATIN input in a recognition object, which possibly forwards a can be partitioned into twelve interrelated concepts (See higher-level event to the screen object underneath. Table 1). Each of these concepts is briefly summarized in Mankoff et al., extended the subArctic toolkit [17] to the next section.