CHI 2001 • 31 MARCH – 5 APRIL Papers Accuracy Measures for Evaluating Computer Pointing Devices I. Scott MacKenzie Tatu Kauppinen Miika Silfverberg Dept. of Computer Science Nokia Research Center Nokia Research Center York University P.O. Box 407 P.O. Box 407 Toronto, Ontario FIN-00045 Nokia Group FIN-00045 Nokia Group Canada M3J 1P3 Finland Finland [email protected] [email protected] [email protected] ABSTRACT The pointing device most common in desktop systems is the In view of the difficulties in evaluating computer pointing mouse, although others are also available, such as trackballs, devices across different tasks within dynamic and complex joysticks, and touchpads. Mouse research dates to the 1960s systems, new performance measures are needed. This paper with the earliest publication from English, Engelbart, and proposes seven new accuracy measures to elicit (sometimes Berman [6]. The publication in 1978 by Card and colleagues subtle) differences among devices in precision pointing tasks. at Xerox PARC [4] was the first comparative study. They The measures are target re-entry, task axis crossing, established for the first time the benefits of a mouse over a movement direction change, orthogonal direction change, joystick. Many studies have surfaced since, consistently movement variability, movement error, and movement offset. showing the merits of the mouse over alternative devices Unlike movement time, error rate, and throughput, which are (e.g., [7, 9, 13]). based on a single measurement per trial, the new measures This paper focuses on the evaluation of computer pointing capture aspects of movement behaviour during a trial. The devices in precision cursor positioning tasks. The primary theoretical basis and computational techniques for the contribution is in defining new quantitative measures for measures are described, with examples given. An evaluation accuracy that can assist in the evaluations. with four pointing devices was conducted to validate the measures. A causal relationship to pointing device efficiency PERFORMANCE EVALUATION (viz. throughput) was found, as was an ability to discriminate The evaluation of a pointing device is tricky at best, since it among devices in situations where differences did not involves human subjects. There are differences between otherwise appear. Implications for pointing device research classes of devices (e.g., mouse vs. trackball) as well as are discussed. differences within classes of devices (e.g., finger controlled trackball vs. thumb-controlled trackball). Generally, between- Keywords class differences are more dramatic, and hence more easily Computer pointing devices, performance evaluation, detected through empirical evaluations. performance measurement, cursor positioning tasks The most common evaluation measures are speed and INTRODUCTION accuracy. Speed is usually reported in its reciprocal form, The popularization of the graphical user interface (GUI) movement time (MT). Accuracy is usually reported as an began in 1984 with the Apple Macintosh. Since then, GUIs error rate – the percentage of selections with the pointer have evolved and matured. A key feature of a GUI is a outside the target. These measures are typically analysed over pointing device and “point-and-click” interaction. Today, a variety of task or device conditions. pointing devices are routinely used by millions of computer An ISO standard now exists to assist in evaluating pointing users. devices. The full standard is ISO 9241, “Ergonomic design for office work with visual display terminals (VDTs).” Part 9 is “Requirements for non-keyboard input devices” [8]. Permission to make digital or hard copies of all or part of this work for ISO 9241-9 proposes just one performance measurement: personal or classroom use is granted without fee provided that copies are throughput. Throughput, in bits per second, is a composite not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or measure derived from both the speed and accuracy in republish, to post on servers or to redistribute to lists, requires prior responses. Specifically, specific permission and/or a fee. SIGCHI’01, March 31-April 4, 2001, Seattle, WA, USA. IDe Copyright 2001 ACM 1-58113-327-8/01/0003…$5.00. Throughput = ( 1 ) MT anyone. anywhere. 9 Papers CHI 2001 • 31 MARCH – 5 APRIL where borne out in more thorough analyses, for example, in considering movement path. = D + Consider the trackball’s means to effect pointer motion. To IDe log2 1 ( 2 ) move the pointer a long distance, users may “throw” the ball We with a quick flick of the index finger, whereas more precise The term IDe is the effective index of difficulty, in “bits.” It is pointer movement is effected by “walking” the fingers across calculated from D, the distance to the target, and We , the the top of the ball. These behaviours, which are not possible effective width of the target. The use of the “effective” width with other pointing devices, may affect the pointer’s path. (We) is important. We is the width of the distribution of Such effects may not surface if analyses are limited to selection coordinates computed over a sequence of trials, movement time or error rates. calculated as Dragging tasks are particularly challenging for trackballs. This has been attributed to an interaction between the muscle = × We 4.133 SDx ( 3 ) groups to effect pointer motion (index finger) vs. those to press a button (thumb) [11]. In the study cited, however, only where SDx is the standard deviation in the selection coordinates measured along the axis of approach to the target. movement times and error rates were measured. Since these are gross measures (one per trial), their power in explaining This implies that We reflects the spatial variability (viz. accuracy) in the sequence of trials. And so, throughput behaviour within a trial is limited. Here we see a clear need captures both the speed and accuracy of user performance. for more detailed measures that capture characteristics of the See [5, 10] for detailed discussions. pointer’s path. Several measures are possible to quantify the smoothness (or NEW ACCURACY MEASURES lack thereof) in pointer movement, however analyses on the Besides discrete errors or spatial variability in selection path of movement are rare in published studies. (For coordinates, there are other possibilities for accuracy and exceptions, see [1, 12].) One reason is that the computation is each provides information on aspects of the interaction. In a labour-intensive. The pointer path must be captured as a “perfect” target selection task, the user moves the pointer by series of sample points and stored in a data file for subsequent manipulating the pointing device; the pointer proceeds analysis. Clearly, both substantial data and substantial follow- directly to the centre of the target and a device button is up analyses are required. pressed to select the target (see Figure 1). An example of a task where the path of the pointer is important is shown in Figure 2. When selecting items in a hierarchical pull-down menu, the pointer’s path is important. If the path deviates too far from the ideal, a loss of focus occurs and the wrong menu item is temporarily active. Such behaviour is undesirable and may impact user performance. Figure 1. A “perfect” target-selection task In practice, this behaviour is rare. Many variations exist and all occur by degree, depending on the device, the task, and other factors. In this section, we identify some of these behaviours and formulate quantitative measures to capture them. We are not suggesting that it is wrong to report error rates. Rather, our goal is to augment this with more expressive measures of accuracy — measures that can assist in characterizing possible control problems that arise with pointing devices. Movement Variability Figure 2. The importance of pointer path Devices like mice, trackballs, joysticks, and touchpads have a variety of strengths and weaknesses, and these are well Several measures are now proposed to assist in identifying documented in past studies [4, 5, 7, 9, 11]. However, analyses problems (or strengths) for pointing devices in controlling a tend to focus on gross measures such as movement time and pointer’s movement path. Figure 3 shows several path error rates. These measures adequately establish “that there is variations. Note that the pointer start- and end-point are the a difference", but their power in eliciting “why there is a same in each example. Clearly, accuracy analyses based only difference” is limited. Establishing “why” is more likely on end-point variation cannot capture these movement variations. 10 Volume No. 3, Issue No. 1 CHI 2001 CHI 2001 • 31 MARCH – 5 APRIL Papers We begin by proposing several simple measures that require Orthogonal Direction Change (ODC). In Figure 3d, two only that certain salient events are logged, tallied, and direction changes occur along the axis orthogonal to the task reported as a mean or ratio. axis. Each change is logged as one orthogonal direction Target Re-entry (TRE). If the pointer enters the target change (ODC). If this measure is substantial (measured over region, leaves, then re-enters the target region, then target re- repeated trials), it may signal a control problem in the entry (TRE) occurs. If this behaviour is recorded twice in a pointing device. sequence of ten trials, TRE is reported as 0.2 per trial. A task The four measures above characterize the pointer path by with one target re-entry is shown in Figure 3a. logging discrete events. Three continuous measures are now proposed: movement variability, movement error, and movement offset. Movement Variability (MV). Movement variability (MV) is (a) a continuous measure computed from the x-y coordinates of the pointer during a movement task. It represents the extent to which the sample points lie in a straight line along an axis parallel to the task axis.