CHI 2007 Proceedings · Novel Navigation April 28-May 3, 2007 · San Jose, CA, USA Command Strokes with and without Preview: Using Pen Gestures on Keyboard for Command Selection Per Ola Kristensson Shumin Zhai Dept. Computer and Information Science IBM Almaden Research Center Linköpings universitet, Linköping, Sweden 650 Harry Road, San Jose, California, USA [email protected] [email protected] ABSTRACT processor and 30GB of storage are already available on the This paper presents a new command selection method that market today. The increasingly limiting bottleneck to provides an alternative to pull-down menus in pen-based information flow in ultra-mobile computing, however, is mobile interfaces. Its primary advantage is the ability for the lack of efficient user interfaces on these devices. In users to directly select commands from a very large set comparison to desktop or laptop computers, today’s without the need to traverse menu hierarchies. The handsets and tablet PCs are far more cumbersome to use. proposed method maps the character strings representing the commands onto continuous pen-traces on a stylus When a desktop keyboard and a mouse are not an option, an keyboard. The user enters a command by stroking part of its electronic pen provides a mobile and flexible alternative character string. We call this method “command strokes.” means of input. The current work focuses on pen-based We present the results of three experiments assessing the command selection. The basic and de-facto standard usefulness of the technique. The first experiment shows that method of issuing commands on a pen-based computer is command strokes are 1.6 times faster than the de-facto the same as on a desktop PC: hierarchical pull-down menus. standard pull-down menus and that users find command The limitations of pull-down menus on desktop computers strokes more fun to use. The second and third experiments have long been recognized [9]. Pull-down menus are much investigate the effect of displaying a visual preview of the more problematic on a pen-based touch-screen computer currently recognized command while the user is still for a number of reasons. First, the pen (stylus) and its articulating the command stroke. These experiments show holding hand often obscure the very items on the pull-down that visual preview does not slow users down and leads to menu the user needs to find and select. Second, pen motion significantly lower error rates and shorter gestures when on a screen has to be one-to-one in scale. This is in contrast users enter new unpracticed commands. to other pointing devices—such as the mouse—whose control-to-display gain is rate-accelerated, so that one does Author Keywords not have to move over a large distance to reach a far-away commands, pen gesture, shorthand, keyboard shortcuts menu on a large display. Pull-down menus on mobile devices also pose a space management problem, forcing ACM Classification Keywords application developers to either remove functionality or H5.2. Information interfaces and presentation: User introduce an even more complex pull-down menu Interfaces – input devices and strategies, interaction styles, navigation system than on their desktop counterparts. theory and methods; I5.1. Pattern recognition: What comes to pull-down menus’ rescue on desktop and Implementation – interactive systems. laptop computers are keyboard shortcuts for frequent commands such as Copy and Paste. Without a keyboard, INTRODUCTION these shortcuts are often what the user misses the most on a Advances in mobile computing hardware are rapidly pen-based device. Despite the flexibility of the pen, increasing the processing power, wireless bandwidth, and command selection on mobile devices remains inefficient storage capacity in handsets, electronic tablets, and other and unsatisfactory. ultra-mobile computers. Palmtop computers with a 1GHz We present a new and practical command entry technique for pen computers called command stroke (CS). CSs are Permission to make digital or hard copies of all or part of this work for pen-gesture traces defined on a graphical keyboard personal or classroom use is granted without fee provided that copies are according to the letters in the commands such as c-o-p-y not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, and p-a-s-t-e. CSs offer users a complementing method of or republish, to post on servers or to redistribute to lists, requires prior directly selecting any command without needing to browse specific permission and/or a fee. a menu hierarchy. CHI 2007, April 28–May 3, 2007, San Jose, California, USA. Copyright 2007 ACM 978-1-59593-593-9/07/0004...$5.00. The development of CSs followed an iterative process. An early incarnation of the concept was compared with 1137 CHI 2007 Proceedings · Novel Navigation April 28-May 3, 2007 · San Jose, CA, USA traditional pull-down menus. Encouraged by the positive regular QWERTY desktop keyboard). This causes two results and informed by the feedback gained from the problems. First, very short pattern sequences are confusable experiment we developed a second iteration of the in the recognition process. Second, the pen-gestures of technique we call command strokes with preview (CSP). some different commands can be quite similar from a user’s The structure of this paper reflects the iterative point of view. The pen-gestures Ctrl-C for Copy and Ctrl-X development process: We first present the initial for Cut are in fact very close to each other (cf. Figure 1). incarnation of command strokes and its evaluation. Thereafter we present the second iteration of the method Q W E T and two experiments that investigate how the visual preview functionality impacts end-users. Last, we compare A S D F our work to previous research and point to future work. SHIFT Z X C V COMMAND STROKES Ctrl Alt The concept of CS grew out of SHARK shorthand [5, 14], a word recognition system also known as ShapeWriter. A Figure 1. The line traces of Ctrl-C and Ctrl-Shift-E as user-drawn CS on a graphical keyboard is recognized by a short command strokes on a stylus QWERTY keyboard pattern recognizer that compares its geometrical shape, not Long Command Strokes the keys crossed on the keyboard, with all CS templates to A long CS is defined by a stylus keyboard trace from a return the user intended command. Since this pattern modifier key through a few or all letters in the name of a recognition approach takes advantage of the constrained command. Figure 2 shows an example of a long CS for the number of commands (as opposed to the infinite number of action Copy whose template starts on Ctrl and intersects the possible shapes or geometric patterns on a keyboard), it has letter keys C-O-P-Y in sequence. Since long CSs are richer an inherent error tolerance. We use the same recognition in shape features, they should be more error tolerant than method that we previously reported for word recognition in short CSs: as long as the user’s gesture is geometrically [5], which gives an in-depth explanation of the recognition similar to the long command pattern, the command can be process. recognized and executed (Figure 2). Short Command Strokes In the first design iteration CSs were divided into two classes: short command strokes and long command strokes. Q W E R T Y U I O P A short CS is a pen-gesture trace from one or more A S D F G H J K L modifier keys, such as Ctrl, Alt, Fn, Shift or Command to a letter key corresponding to the physical keyboard hotkey Z X C V B N M conventions, for example, Ctrl-C for Copy. In a desktop Ctrl Alt environment frequently used commands such as Copy and Paste have mnemonic key identifiers indicating the hotkey Figure 2. The solid red line shows the long CS template for shortcut. Copy and the dashed blue line shows a user’s actual pen- gesture of the command Taking advantage of the flexibility of the pen, short CSs transform the physical keyboard hotkeys to a fluid form. A EXPERIMENT 1 short CS is typically defined as a straight stroke. The user’s There are many issues with CSs that can be studied. In the input can be matched against a collection of pen-gesture first experiment we focused on a few basic questions: Can templates: a user needs only to draw an approximate pen- CSs offer any performance advantage over pull-down gesture to invoke a command, as long as it is closer to the menus, the de-facto standard technique for entering intended pen-gesture template than other templates in the commands on tablet PCs and smart phones? What are the database. A novice user starts out by tracing the key quantitative tradeoffs between short and long CSs? Which sequences. Over time, the pen-gesture becomes ingrained in technique do users prefer? the user’s memory and can be quickly executed without looking much at the keys. The technique can be expanded Procedure and Design to any arbitrary sequence of keys. For instance, Figure 1 We recruited 12 volunteers for participation in the shows the hotkey trace Ctrl-Shift-E (for Track Changes in experiment. Their ages ranged between 20 and 50. The MS Word). experiment was conducted on a 1GHz Tablet PC with a 12" Short CSs, being directly mapped from the physical TFT display with 1024 × 768 pixel resolution. The test keyboard hotkeys, have limitations due to their heritage.