CHAPTER 5 Haptic Interaction Design for Everyday Interfaces

By Karon E. MacLean

This chapter sets about to provide the background and orientation needed to set a novice designer on his or her way to bringing haptics successfully into an interactive product. To define appropriate roles for haptic interaction, it is necessary to integrate a basic awareness of human capabilities on one hand and current device technology on the other. Here, I explore this integration by first summarizing the most salient constraints imposed by both humans and hardware. I then proceed to relate perceptual, motor, and attentional capabil- ities to a selection of emerging application contexts chosen to be relevant to contemporary design trends and opportunities. These include abstract communication and notification, augmentation of graphical user interfaces, expressive control, affective communication, and mobile and handheld computing.

Our touch (haptic) sense is such an integral part of our everyday experience that few of us really notice it. Notice it now, as you go about your business. Within and beneath our skin lie layers of ingenious and diverse tactile receptors comprising our tactile sens- ing subsystem. These receptors enable us to parse textures, assess temperature and mate- rial, guide dexterous manipulations, find a page’s edge to turn it, and deduce a friend’s mood from a touch of his hand. Intermingled with our muscle fibers and within our joints are load cells and position transducers making up our proprioceptive sense, which tell our nervous systems of a limb’s position and motion and the resistance it encounters. Without these and their close integration with our body’s motor control, it would be ex- ceedingly difficult to break an egg neatly into a bowl, play a piano, walk without tripping, stroke a pet, write, draw, or even type. Touch is our earliest sense to develop (Montagu, 1986). It has evolved to work in a tight partnership with vision and hearing in many ways we are only beginning to understand, as we study processes (such as hand-eye coordination) and how we process conflicting or competing information from different senses. In stark contrast to the importance of touch in our everyday experience, the use of touch is marginalized in contemporary computer interfaces, overlooked in the rush to accom- modate graphical capability in desktop-based systems. The primary advances have been in feel-focused improvements in nonactuated pointing tools for both function and aes- thetics. Scroll wheels have been designed for the user to click with just the right resistance and frequency; and most cell phones now come with vibrators that indicate incoming calls. Meanwhile, the use of haptic feedback in the consumer sphere is largely limited to gaming, and tactile feedback to simple cell phone alerts.

DOI 10.1518/155723408X342826. Copyright 2008 Human Factors and Ergonomics Society. All rights reserved. 149 150 Reviews of Human Factors and Ergonomics, Volume 4

So much more is possible, or will be soon. On the human side, the horizon is expand- ing as we improve our understanding of haptic perception; on the technology side, hap- tic display innovation and pricing are just starting to benefit from the volume effects of inclusion in large consumer bases. The goal of this chapter, which is targeted at the novice haptic application designer with a background in human-computer interaction (HCI) and usability engineering, is to describe this new horizon and to provide some guidance about how to use an intimate, yet unfamiliar, modality with maximal effectiveness. I assume a context of ubiquitous computing rather than virtual reality (Weiser, 1991); although the former has received far less attention by haptics researchers, it has the potential to directly influence many more users. In this chapter I describe the constraints within which designers must work—some de- fined by fixed human abilities, others by hardware limits that will certainly change. The potential functionality of haptic feedback is then outlined through discussions of a num- ber of basic interactions and related application contexts in which haptic interfaces are well positioned to play an important role. Taken together, these perspectives are intended to help designers identify and successfully exploit new uses of haptic feedback.

A ROLE FOR HAPTICS IN THE CHANGING LANDSCAPE OF COMPUTATION

Origins and Definitions Active haptic devices are interfaces to computers or networks that exchange power (e.g., forces, vibrations, heat) through contact with some part of the user’s body, following a programmed interactive algorithm. For example, a force feedback device can physically render a computed virtual environment model within limits such as its workspace and actuator torque. Cell phone vibrators and force feedback game joysticks are also active hap- tic interfaces; whereas the vibrator is only a display, the joystick is both an input and an output device, and its control is considerably more complex. The concept has been explicitly in place in the robotics world since around 1990, with first credit usually given to Minsky’s textural explorations (Minsky, Ouh-Young, Steele, & Behensky, 1990) and Brooks’s scientific “hapticizations” (Brooks, Ouh-Young, Batter, & Kilpatrick, 1990). However, these early examples were foreshadowed by a vast body of work in teleoperation and rehabilitation research extending back to World War II. Although the need for custom hardware once limited haptics research to a small community of robotics engineers, the release of the first (expensive) commercial desktop haptic dis- play—Sensable Technologies’ Phantom in 1996—opened the world of haptic interfaces to professionals in other fields (Massie & Salisbury, 1994; Sensable Technologies, 1996). It is now possible to purchase commodity haptic devices (e.g., mice, gaming joysticks and steering wheels, vibrating cell phones). You can also create your own: A variety of tactors (vibrotactile displays) are available from discount electronics sources and drivable from your computer’s audio card. Simple, reasonably high performance force feedback displays can be assembled out of commonly used components with only a little mechatronic prowess (Hayward & MacLean, Haptic Interaction Design for Everyday Interfaces 151

2007), and open-source and proprietary software libraries ease their programming. Al- though a vast divide remains between the display fidelity of current haptic versus graphic technology, the entry level for utilizing haptic feedback has become much lower. But as always, coming up with a “killer app” is more of a challenge, and getting it right even more so.

Current Trends: The Expanding Reach of Computation Whereas haptic display advances move along at one pace, the role and reach of compu- tation and networking in the world outside the research lab are evolving and expanding at a blinding rate. It is worth singling out aspects of these changes with acute relevance to the new possibilities enabled by active haptic feedback.

Networking. Everyone is connected, in ever more places and for a larger part of their time. Being in constant contact with others has become a way of life for many since the Internet and cell phone use became dominant features of society in the 1990s. Technically, these high-speed networks have moved virtual touching—whether of other people or of virtual representations of electronic constructs, by wire and in real time—out of science fiction and into the present.

Ubiquity of computing devices. Computation is everywhere, not just on the desk. It is used on the go, in nontraditional contexts, and without the benefit of a big graphic dis- play. As everyone knows who has used a cell phone for more than a phone call or a PDA for more than entering an address, or braved an automobile’s onboard communications console to change the radio station, there is a need and an opportunity in this sparse visual real estate for enhanced information display.

Multitasking. As a direct consequence of the preceding factors, people increasingly do more than one thing at once—because they can and because they now feel they must. Frequently this means their eyes are busy with one task while their ears, hands, and/or voice are taking care of something unrelated. For this reason—as much as the absence of a large, high-resolution screen—having additional information conduits besides vision seems like it might be a useful thing, if our caffeinated brains can handle it.

Virtualization of personal presence. In many ways, personal presence has become optional: Employees work from home, students attend class remotely, parents and chil- dren live on nonintersecting schedules, everyone shops online. The convenience is unde- niable, but the net effect when the bricks-and-mortar version of the institution is nearly empty has a profound impact on group dynamics and personal engagement. Social sci- entists are busy identifying the key elements of personal engagement that are lost in this development, from nonverbal conversational cues in individual interactions (e.g., Knapp & Hall, 2005; Sproull & Kiesler, 1991) to impacts on larger collectives (for a discussion, see Menzies, 2005). The need for remote collaboration and communication tools that fos- ter inclusiveness, nonverbal support, and a sense of social presence for both professional and personal contexts has been well recognized (Grudin, 1988). 152 Reviews of Human Factors and Ergonomics, Volume 4

Information management. The network creates a veritable firehose of information pouring across us. One can think of these as a volume challenge (there is simply too much information, and it needs to be filtered and triaged in some way before we see it) and as an attention challenge (contributing to fragmentation). Fragmentation. An important artifact of constant connection and task juggling is con- tinual interruption, even when a person’s primary task requires continuity. Jobs not pro- tected from the barrage must be accomplished in time slices, whether the slicing is due to an incoming e-mail or to the need to cross the street while getting GPS directions. Despite the utility of this situation for those seeking to procrastinate, interface schemes that can help to manage, filter, batch, or otherwise mitigate the intrusiveness of connection-derived interruptions could have value. Setting aside the fascinating controversy of whether these changes are leading to an overall improvement in productivity and quality of life in the developed world (Menzies, 2005)—particularly for those who have come of age in the Internet era and seem adapted to it—many would agree that to some extent, our gadgets currently manage us, and we would prefer to have it the other way around. Can haptic technology help?

Partner Disciplines Clearly, haptic interfaces and augmentations are just one of many emerging tools that can be brought to bear on problems related to increased connectivity. Furthermore, re- search on machine recognition of user availability for communication, emotion model- ing, computer-supported collaboration, the role of attention in perception and cognition, as well as advances in haptic hardware are just a few of the related and highly active threads essential to progress in this area. But beyond the fact that we need a versatile toolkit to solve our interaction problems, it is important to acknowledge the intensely interdisciplinary nature of the challenge of creating useful and aesthetic haptic feedback. It cannot be overemphasized that (a) haptic interfaces are generally used in multisensory and multitasking environments and, as such, must be designed with a holistic view of context, and (b) an interdisciplinary outlook and partnering with experts in these other fields is bound to improve design outcomes. In the next two sections, I provide a multidisciplinary overview of the key issues in human haptic perception and current experimental haptic hardware, respectively, that will influence design. The first is a fixed quantity about which we still have much to learn; the second is a moving target. Relevant primary sources and more focused reviews will be cited where applicable and available.

HUMAN CONSTRAINTS

What Designers Need to Know About Human Haptics Human haptic perception has not yet been studied to the same extent as have vision and audition, but it received some initial attention with the surge in human factors research Haptic Interaction Design for Everyday Interfaces 153 in the World War II era, and interest has grown steadily since the early 1990s. It is unique among the senses in being physically and functionally integrated with motor control, and it very often works in close synchrony with other senses. Examples include hand-eye coor- dination and the way we interrogate an object’s material properties through an integra- tion of feel and sound (Klatzky, Pai, & Krotkov, 2000). Classic and contemporary texts that broadly address the touch sense from a psychophys- ical standpoint include Gescheider (1985); Goldstein (1999); Heller and Schiff (1991); Kandel, Schwartz, and Jessell (2000), Katz (1925/1989); Klatzky and Lederman (2003); and Rock (1984).

The Sensors Like vision and audition, the haptic sense comprises a hierarchy of subsystems. Touch includes two such modules: tactile (sensations arising from stimulus to the skin—heat, pressure, vibration, slip, pain) and proprioceptive (which provides our knowledge of body positions, forces, and motions via end organs located in muscles, tendons, and joints). You use proprioception to assess the weight of a brick in your hand, and if you close your eyes while someone else moves your arms around, proprioception will tell you where they are. Haptic interfaces are generally directed at either the tactile or the proprioceptive systems because of configuration constraints.

Tactile sensors. Our skin is our primary tactile sensory organ. There are three main functional categories of tactile receptors, all located in different layers of the skin, and there is a fascinating variety in how they work mechanically and neurologically, as laid out in several perception texts (e.g., Goldstein, 1999; Rock, 1984). Thermoreceptors recognize changes in skin temperature; mechanoreceptors (of which there are half a dozen) sense pres- sure, vibration, and slip; and nocioreceptors sense pain. To encompass sensations ranging from pin-pricks to broad, steady pressure, they vary in their mechanical principle of action, skin depth, and response speed. It is important to understand a few generally applicable details about tactile sensor resolution, or acuity. Spatially, resolution is determined by the size of the receptor’s field, but if there is a high density of receptors, the resulting overlap and “crosstalk” reduce effec- tive resolution. When a point stimulus is applied in overlapping fields, the perceptual resolution becomes the size of the union of the two fields. Figure 5.1 illustrates how density varies with body location, showing the higher con- centration of receptors in areas of glabrous (nonhairy) skin. The sensors in glabrous skin can detect finer detail, as measured by smaller spatial thresholds than for nonglabrous skin. With regard to temporal resolution, a given type of receptor has a successiveness thresh- old, which describes how closely spaced a series of stimuli can be for a recipient to iden- tify them as separate. This determines, for example, the lowest and highest frequency of vibrations we can distinguish. Fast adapting sensors (FA) capture transients and slowly adapting (SA) sensors are for static stimuli. We can sense much higher frequencies through certain tactile sensors than via proprioception; appropriately so, because skin can be vibrated much more quickly than a limb or even a fingertip. Tactile sensitivity, as opposed to acuity, refers to the stimulus’ detection threshold. 154 Reviews of Human Factors and Ergonomics, Volume 4

Figure 5.1. Distribution of tactile sensors in skin over entire body. High sensor density corre- lated to body regions involved in exploration and manipulation. Reproduced with permission from Kandel et al. (2000).

Again, this is a function of specific receptor type, and the overall percept of signal strength varies depending on the stimulus site and mechanical characteristics of the stimulus. Some important mechanoreceptors (the most commonly implicated in today’s tac- tile displays) are illustrated in Figure 5.2 and include Merkel receptors (SA), which respond to pressure at about 0–10 Hz; Meissner corpuscles (RA), which respond to “taps” within 3–50 Hz; Ruffini cylinders (SA), which respond to stretching of skin or move- ment of joints at 0–10 Hz, and Pacinian corpuscles (RA), which respond to rapid vibra- tion within 100–500 Hz. The “sweet spot” for vibrotactile sensitivity is considered to be 250 Hz (Shimoga, 1992). A characterization of the mechanical impedance of fingertip skin lends further insight into these limits (Hajian & Howe, 1997). Detailed recent reviews of research in tactile sensing for the purposes of tactile communication can be found in Jones and Sarter (2008) and Pasquero (2006).

Proprioceptive sensors. Proprioception (which is closely related to kinesthesia) is how we get information from most force feedback displays. Its receptors are generally embed- ded in our muscle fibers and joints, although sometimes skin stretching also gives cues. There are two primary types of muscle mechanoreceptors. Our force sensors (Golgi tendon organs) measure force via localized tension and are located serially between muscles and tendons. Position and motion sensors (muscle spindles) are located in parallel among mus- cle fibers and are excited by changes in muscle length (e.g., active and passive stretching). Haptic Interaction Design for Everyday Interfaces 155

Figure 5.2. Cross section of glabrous skin (areas without hair follicles, e.g., the fingertips). Reproduced with permission from Goldstein (1999).

Each of these sensor types plays a special role in motor control; this is further discussed in Goldstein (1999) and Rock (1984). Many psychophysical studies have sought to determine human capabilities such as the resolution with which we can track forces—that is, maintain a desired force level with or without some form of feedback on the force we are providing, or, similarly, our ability to follow a specified position trajectory in space. Goals of these studies include determining both the involvement of different kinds of sensory receptors and the performance achiev- able. Studies might be performed, for example, by testing performance with either the entire hand anesthetized, just the skin, or not at all. Comparing performance under these different conditions reveals which sensors (e.g., tactile vs. proprioceptive) are contribut- ing to performance. Thus, we have learned that humans are reported to track small forces quite accurately; for example, in ideal conditions (visual feedback available) with errors ranging from 2%–3% when gripping (Mai, Avarello, & Bolsinger, 1985) to 15% when pushing against a normal surface (Srinivasan & Chen, 1993). Performance degrades with- out visual feedback or access to texture (Lederman & Klatzky, 2004); that is, both tactile and proprioceptive sensing are apparently involved. The just noticeable difference (JND), expressed as a percentage, is a common measure of both tactile and proprioceptive sensory resolution. A JND of n percent implies an expo- nential resolution curve. At low torque levels, we can sense values relatively close together, but as the absolute torque or force level increases, absolute sensory resolution decreases accordingly. This raises the question of whether we discriminate torque or force directly or if we instead sense compliance or something else. Wu, Basdogan, and Srinivasan (1999) found evidence to support the idea that we are attending to the work done in compressing a spring. That is, as predicted by Gibson (1962), our “object invariant” is the relationship between force and position rather than either one individually. Here, resultant torque discrimination was measured at a JND of 13% and compliance (displacement/force) at 22%. Thus, the actual sensation probably depends on the work performed in depressing the spring. Our motor control bandwidth is how fast we can move our own limbs or digits and 156 Reviews of Human Factors and Ergonomics, Volume 4 is much lower than the rate of motion we can perceive. Proprioceptive sensing occurs around 20–30 Hz, compared with 10–10,000 Hz for tactile sensing. Control, however, sat- urates around 5–10 Hz, a limit determined by mechanical resonance—whether it’s mov- ing your eyes, fingers, or legs. Hasser and Cutkosky (2002) provided an example of how this has been modeled. More detail on frequency ranges can be found in Shimoga (1992).

The “Bidirectional Sense”: Sensorimotor Integration and Motor Control Sensorimotor control guides our physical motion in coordination with our touch sense (Lederman & Klatzky, 1997). There is a different balance of position and force control when we are exploring an environment (e.g., lightly touching a surface) versus manipu- lating an object. In the latter case we might be going through a preprogrammed sequence of movements and relying only subconsciously, if at all, on the touch sense. Planning future motion also engages sensorimotor functions (Gillespie, 1999). In this case it is relevant to define active touch as user-driven spatial exploration over time (Gibson, 1962), a different concept than the active haptic feedback defined earlier. A haptic designer needs to be aware of the way a user is going to grasp a handle or probe; consider the possibilities of grip suggested by the different handles shown in Fig- ure 5.3. The amount of force you can generate depends on the way you hold it—the grip employed brings into play different muscle groups, which in turn differ in their force generation and sensory resolution capabilities. Power grasps are designed for strength and stability, involving the palm as well as the fingers. Maximum forces can be in the range

Figure 5.3. The form of the handle invites a particular kind of grasp: the hand’s configuration determines the strength versus precision that can be applied. Haptic Interaction Design for Everyday Interfaces 157 of 5 to hundreds of Newtons. Precision, or pinch, grasps are less strong, generating up to 25% of the force, and are characterized by apposition of the thumb and distal joints of the fingers. In either case, fatigue can be an issue sooner or later: When you get tired, you gen- erate less force and are less able to control it (MacKenzie & Iberall, 1994; Napier, 1956). Lederman and Klatzky’s work with exploratory procedures emphasizes the “percep- tual functions of the hand”—that is, “how action is employed in the service of perceptual goals” (Lederman & Klatzky, 1996, p. 431). These motions, shown in Figure 5.4, are intu- itive and ingrained; subjects have used them freely and consistently when asked to extract specific object properties from a sample.

Crossmodal Integration and Interference Haptic design is nearly always multimodal design; the touch sense is generally used in con- junction with other sensory modalities, whether their roles are to reinforce the same task or to handle different tasks performed at the same time. Touch-derived input plays a unique role in this context, and theories continue to develop on how sensory information is integrated and how conflicting information is resolved. The emerging short answer is that the task matters. Multisensory processing occurs in the superior colliculus (SC) in the midbrain. Much

Figure 5.4. Exploratory procedures. The nature of desired physical information guides strategic touching (“action for perception”). Reproduced with permission from Lederman and Klatzky (1996). 158 Reviews of Human Factors and Ergonomics, Volume 4 of what we know about how it works comes from studies of orienting and attentional focus, often using animal models. The SC contains specialized neurons that receive input from multiple sensory modalities, and consistent information arriving on these conduits can be synthesized in a number of ways. Enhancement happens when consistent stimuli in two or more modalities reinforce one another, strengthening the overall percept. Neu- rons sum and transform input from different senses, an effect that is most dramatic when the unimodal stimuli are weak. However, there are both facilitation and inhibition effects of intensity: a moderate acces- sory stimulus usually facilitates a primary stimulus, but an intense accessory stimulus usu- ally inhibits the primary one (Stein & Meredith, 1993). When one source is more accurate, more precise, and/or faster than the other, complementary information can be merged (Newell, Bülthoff, & Ernst, 2003). Linear integration is thought to be at work; for exam- ple, Lederman and Klatzky (2004) proposed this mechanism for texture perception. Habituation occurs when the initial effects of the accessory differ from later effects; for example, one becomes accustomed to a loud sound. Finally, temporally or spatially adja- cent stimuli can mask one another. But if they are too far apart or too discordant qual- itatively, stimuli will dissociate; they fail to “fuse,” undermining one another and leading to confusion (Stein, 1998). Older theories of so-called sensory dominance followed a winner-take-all structure whereby, in the case of a conflict, one sense “captures” the others (Warren, Welch, & McCarthy, 1981). Vision was generally considered dominant to other modalities, given the best scientific data of the time (e.g., Rock & Victor,1964). More recent observations of hap- tic dominance suggest a more complex view (including Heller, Calcaterra, Green, & Brown, 1999; Hershberger & Misceo, 1996; Robles-De-La-Torre & Hayward, 2001). Current models encode an integration theory that weights conflicting data, with some influence still provided by less dominant modalities (Bertelson & de Gelder, 2004; Yuille & Bülthoff, 1996). Context of the activity is thus critical to predicted outcome (what is sensed): In this theory, our brains place a higher value on the information, which either comes from a more reliable source (often based on prior knowledge) or gives a more relevant view given the task at hand (Ernst & Bülthoff, 2004; Welch & Warren, 1986). The most sophisticated models make their predictions by combining signal noise minimization with signal reli- ability schemes; through the manipulation of reliability, responses can be made to shift from visual to haptic dominance (Ernst & Banks, 2002). A series of multisensory studies involving manipulated multimodal stimuli provide concrete examples of how these ideas might be advantageously exploited in application design. In experiments based on a Phantom haptic device to render forces consistently or discordantly to graphically displayed visual cues, the following emerged: (a) Vision dominates proprioceptively sensed hand position when in conflict (Srinivasan, Beau- regard, & Brock, 1996); (b) “stiffness” of auditory cues dominates haptics in assessments of stiffness, but this is a weaker dominance than seen for vision (DiFranco, Beauregard, & Srinivasan, 1997); and (c) the influence of both vision (perspective cues) and haptics (stiffness cues) was shown to be reduced in the face of distorting information from the other modality, with the two systems apparently attempting to fuse in an optimal man- ner (Wu et al., 1999), thus foreshadowing current models (Ernst & Banks, 2002; Guest & Spence, 2004). Haptic Interaction Design for Everyday Interfaces 159

Attention and Multitasking

As interfaces are increasingly required to make extremely efficient use of limited percep- tual resources, attentional processing and its allocation across sensory modalities has become an active research topic. If, for example, one plans to offload the visual sense by deliver- ing information haptically, one should know whether this transfer of work will actually reduce the total demand on users’ processing resources or make the situation even worse. One influential operational response to this question, Wickens’s multiple resource theory (Wickens, 1980, 2002), makes predictions about the human operator’s ability to per- form in high-workload, multitask environments by viewing processing as constrained by several pools of limited “resources.” In this framework, one factor presumed to define resource pools is sensory modality. Interference among tasks results if common process- ing resources are required for the successful completion of two or more tasks. Thus, for example, there might be more interference between two visual tasks than between a visual and a haptic task, all else being equal. Since the 1980s, this approach and variants have provided a popular tool with which to analyze factors such as user stress, workload, strate- gizing, and task failure. However, it leaves some questions unanswered (e.g., Hancock & Szalma, 2003). Meanwhile, other studies demonstrate that there is still dissent as to the degree of inde- pendence in the attentional resources used by different modalities (e.g., Driver & Spence, 1998; Duncan, Martens, & Ward, 1997). Inclusion of haptics in these studies (usually in the form of tactile stimuli used to aid in spatial orienting) is relatively recent (e.g., Gray & Tan, 2002; Macaluso, Frith, & Driver, 2002 ; Spence, Nicholls, Gillespie, & Driver, 1998; Young, Tan, & Gray, 2003), but these studies show patterns similar to that of vision and audition. However, even if perceptual resources are modular, this doesn’t guarantee peo- ple’s ability to use them independently. For example, try to write an e-mail while talking on the telephone. This task conflict is not perceptual; there are many other potential points of interference, such as the linguistic shared resource in this example. Where does this leave us? Empirical studies in applied settings demonstrate real costs to attentional competition (e.g., in demonstrating the dangers of mobile phone use while driving, as in Patten, Kircher, Ostlund, & Nilsson, 2004); task fragmentation seems to be involved and is another source of productivity loss (Oulasvirta, Tamminen, Roto, & Kuorelahti, 2005). However, current attentional models are not yet reliable and compre- hensive enough to be the sole basis of specifying attentional factors in design. Thus, until the models improve, testing interfaces under representative workload conditions contin- ues to be important, although this requires subtlety as well.

Affect The emotional significance of interpersonal touch for humans is undisputed. It consti- tutes a critical part of our early development, from infancy through adolescence (Montagu, 1986). It is increasingly seen to play an important role in mental health and the treatment of conditions ranging from cancer to critical illness in infants (Field, 2003; Gumtau, 2005; McCorkle & Hollenbach, 1990). 160 Reviews of Human Factors and Ergonomics, Volume 4

So-called nontouching cultures, such as North America’s, in comparison with Latin countries at the other extreme, exhibit signs of relationship problems emerging in young children (Field, 1999) and show different patterns of peer behavior in adolescents (Field, 2003). In other aspects of social behavior, casual touching has been shown to have a con- siderable impact on the positive attitude of the touched toward the toucher, in contexts as disparate as consumer behavior—for example, restaurant tipping (Crusco & Wetzel, 1984; Hornik, 1992)—professional relationships (Fisher, Rytting, & Heslin, 1976), and the readiness of a student to engage in classroom participation (Guéguen, 2004). Between couples, touch is a conduit of many social messages, including hostility, sexual interest, nurturance and dependence, affiliation, and the level of intimacy (Collier, 1985). It is part of a social contract,“for even the best words lack the honesty of touch” (Montagu, 1986). Aesthetically, we see the same kind of fundamental role for touch, as is eloquently artic- ulated in Ackerman (1990). Montero (2006) argued that fundamental aesthetic judgments are available through proprioception, taking the physical experience of activities such as dance as a starting point (see also Foster, 1986). Others believe it influences visual art. Discussing the ideas of Bernhard Berenson (renowned Renaissance art critic of the early 1900s, e.g., Berenson, 1930), Perricone argued,

It is the quality of life-likeness, and of “material significance” that stimulates our tac- tile imagination. Artworks that embody such qualities survive as masterpieces, whereas ones that do not become merely intellectual or historical curiosities. Without volume, bulk, inner substance, and texture, in short without the stuff of touch, the poetry, character, and plot of an art have no anchor in the world, no place for our imagination to take hold.... Touch is the “bass line” of art experience. (Perricone, 2007, p. 91)

This is reinforced by the oft-noted high incidence of haptic metaphor in aesthetic descrip- tion: balanced, powerful, rhythmical, tense, dynamic, integrated, delicate, moving, touching. But aside from art, what about our emotional response to “everyday” haptics? Norman (2005) described three levels of emotional responses: visceral, behavioral, and reflective. Viscerally, we have unconscious reactions that vary little across people, mood, culture, gender, age, and so on. In the middle, our behavioral affect responses are part of our everyday actions and problem solving and reflect, for example, “the pleasure of using a good tool effectively” (p. 23). Reflective responses involve conscious thought, study, and interpretation; they vary from person to person and between task contexts for the same individual and are most closely aligned with aesthetics. Things that are attractive at any of these levels work better, Norman claimed, because they make you feel good. This in turn makes you think more creatively, whereas negative affect tends to narrow your focus (Norman, 2003). With maturing product bases, aesthet- ics increasingly define and differentiate successful interfaces from failures (Jordan, 2000), and the field of affective design has emerged in the last decade with the goal of eliciting specified emotional experiences from users (Picard, 1997). Touch does contribute to the holistic environment that cumulatively drives a person’s affect state up or down. In addition to the evidence cited earlier regarding the importance of everyday interpersonal touch, user studies show that hedonic qualities (pleasurability) Haptic Interaction Design for Everyday Interfaces 161 of touched materials can be evaluated psychophysically (Essick, James, & McGlone, 1999; Knowles & Sheridan, 1966; Swindells, MacLean, Booth, & Meitner, 2007); the more recent of those studies rely on measures of valence and arousal, which have been identified as primary axes of affective response (Russell & Weiss, 1989). Affective haptic design can take us in one of two primary directions: toward a focus on what feels good in touched interfaces—for example, either active or passive manual controls—or toward computer-mediated interpersonal touch (for a recent review, see Haans & IJsselsteijn, 2006). Specific examples of both directions are discussed in the last section.

HAPTIC SYSTEMS: THE OTHER BOTTLENECK

In this section I describe haptic hardware and rendering techniques at a general level, focus- ing on techniques that are more relevant to interaction design than to robotics. The goal of this discussion is to call attention to the hardware and software obstacles that stand between us and the vision of total multimodal interface transparency.

Kinds of Haptic Hardware There are two primary types of haptic devices. Force feedback devices are designed to act on proprioception by providing forces that react to our movements in space (Figure 5.5). Tactile displays target the skin and tend to be more localized. Some reviews give an in-depth view of contemporary tactile and force feedback systems and how they work (Chouvardas, Miliou, & Hatalis, 2005; Hayward et al., 2004; Hayward & MacLean, 2007; Pasquero, 2006; Srinivasan & Basdogan, 1997). With force feedback, the essential idea is that the user holds an actuated (powered) link and moves it about. These movements are measured, and in response, forces are computed (rendered) according to a virtual representation of the displayed physical environment. These devices have degrees of freedom (axes along which the grasped or touched end- effector can move) ranging from 1 (e.g., a knob) to many. Common values in nonresearch

Figure 5.5. A force feedback knob. A computer algorithm takes sensed knob position as input into a virtual model and produces an output torque, which is delivered through the knob’s motor. Photo used with permission of Colin Swindells. 162 Reviews of Human Factors and Ergonomics, Volume 4 desktop setups are 2 (planar devices) and 3 (allowing a match to a graphical three- dimensional space). The latter has been popularized by Sensable Device’s Phantom (Massie & Salisbury, 1994), the first commercially available force feedback display. An astonishing variety of approaches have been taken to the problem of tactile dis- play. Pasquero (2006, especially Figure 5.3) contains a discussion (see also Chouvardas et al., 2005; Hayward & MacLean, 2007; Jones & Sarter, 2008). Because of their efficiency, small size, and low power requirements, the most commonly deployed technologies (e.g., in mobile devices and touch screens) use piezoelectric, voice coil, solenoid, or eccentric rotating motor actuation. These approaches tend to vary along dimensions of strength and expressive capabil- ity (number of distinct sensations achievable; see MacLean & Enriquez, 2003); generally there is a trade-off between these two characteristics. For example, an eccentric rotating motor produces a salient but indistinct “buzzy” feeling. A piezo or voice coil device, on the other hand, can produce a sharp attack and have more controllable dimensions, but the sensations tend to be subtle when delivered by compact devices. A special case is that of a macro-micro configuration, in which a tactile display is mounted on the tip of a force feedback device and provides low-force, high-frequency feedback to complement the larger static forces of the base display (e.g., Kontarinis & Howe, 1995).

Haptic Hardware Today Since the beginning of the current century, several companies have been selling high- performance 3+-dimensional bench- and desktop displays targeting various research and professional niche markets (e.g., animation, visualization, and surgical simulation or tele- operation). In subsequent years, some less costly desktop devices with lower but still respectable performance (3 axes actuated and 6 axes sensed), priced at the higher end of gaming technology, have spurred a new jump in application activity (Novint Technolo- gies, 2006; Sensable Technologies Inc., 2003). Immersion Corporation began testing the consumer market even before the aforemen- tioned efforts, having licensed technology for a range of force and tactile devices to com- panies such as Logitech. Immersion’s I-Force technology is used in gaming devices (joysticks and steering wheels, etc.) as well as in both a 2-D force feedback (FEELit) and the vibro- tactile (I-Feel) mouse, which were introduced in 2000 and 2001, respectively (Immersion Corporation, 2000). Unlike the gaming devices, neither mouse did well in the marketplace, with impor- tant usability lessons. Informal customer feedback suggested that the force feedback mouse suffered from its small workspace, whereas the vibrotactile device was annoying and failed to provide the benefits of firm programmable resistance. However, other, more targeted, uses have been found for this kind of feedback (Chan, MacLean, & McGrenere, 2008). Beyond the desktop market, Immersion provided the design of the longer-lived but con- troversial BMW iDrive (BMW World, 1999), a haptic automotive multifunction interface and, more recently, its TouchSense technology for embedded touch screens, which pro- vides vibrotactile feedback in response to interaction with graphical widgets. More spe- cialized devices are mentioned in specific application contexts later in this chapter. For all these technologies, an open question is how to use them well for both currently Haptic Interaction Design for Everyday Interfaces 163 posed and as-yet-unidentified application concepts. As our experience grows, this ques- tion and its answers will inform and spur the design of future haptic technology.

Software and Control Architectures The computed application needs to connect to the display hardware (sensing and receiv- ing actuator commands) in a timely manner. Stable rendering of force feedback generally requires closed-loop speeds between 500 and 1,000 KHz (the stiffer the system being ren- dered, the higher the rate required). At present, there are many choices of communica- tions conduits, depending on the desired speed, distance, network bandwidth, and communication load. Factors influencing these outcomes include whether the device uses a dedicated or a public network, parallel port, USB 2.0 protocol, local audio channel, or slower connection to an embedded local controller (Hayward & MacLean, 2007; MacLean & Snibbe, 1999). Computationally intensive models (e.g., for 3-D surgical simulations) might require separate CPUs dedicated to the graphical model (slow but intensive updates) and the hap- tic rendering engine (faster but less intensive updates). At the other end of the contin- uum are games, which generally use a peripheral microcontroller with a small set of local macros, triggered periodically with relatively slow communications from the application host computer.

Modeling and Rendering Rendering refers to the translation between the computed interaction model (which might be, for example, a static or dynamic collection of virtual objects to be encountered with an actuated probe) and the command sent to the actuated device. For force-controlled (as opposed to position-controlled) force feedback devices, this results in a force applied to the user’s body, which changes based on both the user’s measured position and the state of the virtual model. A primary challenge here is in obtaining both high fidelity and con- trol stability in rendering force feedback (for some tutorials and reviews, see Basdogan & Srinivasan, 2002; Hayward & MacLean, 2007; Otaduy & Lin, 2006; Salisbury, Barbagli, & Conti, 2004). A major impetus to the development of these techniques comes from surgical applications. Open-source libraries intended to facilitate application development and sharing of techniques have also emerged (e.g., Conti, Morris, Barbagli, & Sewell, 2003; Haptic Library, 2005), as have proprietary and mixed tools (e.g., Immersion Corporation, 2001, 2004; Sensable Technologies Inc., 2004). For tactile displays, rendering refers to how an individual or spatial array of actuators is made to move in response to a variety of interactive variables. Representative examples range from a single vibrotactile display (Chan, MacLean, & McGrenere, 2005; Immersion Corporation, 2004), to tactor arrays (Tan, Gray et al., 2003; van Erp & van Veen, 2003), to planar shape displays (Levesque et al., 2007). The low-level principles tend to be rela- tively simple, given that individual elements are generally “feed-forward” and stable, but the psychophysical percepts can be far more complex (e.g., Luk, Pasquero, Little, MacLean, Levesque, & Hayward, 2006; Tan, Reed et al., 2003). Responses to these stimuli are subject 164 Reviews of Human Factors and Ergonomics, Volume 4 to both temporal and spatial masking (Craig, 1985; Enriquez & MacLean, 2008a; Gallace et al., 2006; MacLean & Enriquez, 2003) and numerous illusory qualities (Hayward, 2008). Generating the model of the environment to be rendered can be a serious business, for tactile and force rendering alike. Models can be created ad hoc, and often 3-D graphical models are modified for haptic display through the application of various kinds of hap- tic texture mapping (surfaces on the models that can be felt with a 3-D haptic display). But sometimes we wish to render something the way it feels in the real world, with high fidelity. One approach to this is the haptic camera concept, whereby an actuated probe sam- ples the real environment, processes the data by a number of possible techniques, and then redisplays it on an appropriate device (MacLean, 1996; Miller & Colgate, 1998; Swindells & MacLean, 2007). James and Pai (2001) took this a step further with tightly coupled audio-haptic displays of the result. On the premise that the instant of contact is the key to material distinctive- ness (Klatzky et al., 2000), Kuchenbecker et al. (2006) described a means of capturing and rendering the dynamics of impulsive contact with different materials, allowing these mate- rials to be rendered with distinctive-feeling “fingerprints.” Limitations What stands between current haptic simulations and passing the haptic Turing test—that is, attaining the ideal state at which the user is unable to distinguish the feel of real ver- sus rendered objects? State-of-the-art display hardware and current rendering algorithms reach this criterion only in isolated instances, when the device’s actual physical handle and configuration cor- respond perfectly to the environment being simulated, which in turn is computationally easy to handle (for example, a Phantom stylus being used to simulate stroking a stylus- shaped tool on a soft, textured surface). Some examples of frequent problems, and con- sequent areas of needed improvement, include the following: • Subtle instability: “Jitter” or “activeness” (usually in a force feedback device, but possible in some types of tactile displays) caused by inadequate hardware dynamic range and/or refresh rate (Choi & Tan, 2002; Colgate & Brown, 1994; Gillespie & Cutkosky, 1996). This tends to be exacerbated for stiff models (e.g., a soft-tissue surgical simulation is rarely active, but a bone simulation might be). • Forces feel “spongy.” This could be because the motors are too weak (e.g., to meet size, weight, responsiveness, or power compromises), the transmission is too compliant, or up- dates are too slow. Conversely, a softened controller is one compromise solution to simula- tion activeness. • The display’s handle, configuration, and/or form factor is inappropriate. For example, some tactile displays must be explored “in place” rather than by moving one’s hand over a large surface. Other displays may use a single handle (e.g., a stylus) to physically grasp for tool inter- action rather than the wide range of tool shapes to which people are accustomed, with their varied grasp and affordances. Still other handles may suffer from limited workspace or have the wrong geometry or degrees of freedom. • The device is too large, heavy, or power-hungry to function properly in the intended envi- ronment. • The model just feels wrong, possibly because the virtual model is unrealistic (usually over- simplified) or the rendering technique inadequate. Haptic Interaction Design for Everyday Interfaces 165

WHEN TO USE HAPTIC FEEDBACK

In this section, I explore how haptic feedback can be deployed to solve several low-level, generically described interactive functions. Each is illustrated with applications whose properties make them good (if challenging) platforms for deploying haptic feedback. The application areas often overlap across interactive functions, so some appear more than once. Neither the function list nor the examples are exhaustive, not least because this is where the field is evolving fastest in response to new needs and new hardware. We can consider haptic value in terms of functionality, emotion, and aesthetics. We search for ways in which it can improve task performance or expand capabilities, allow us to communicate through technological conduits, or find interactions more pleasurable and satisfying. A good starting point is an examination of touch interactions in the nontech- nological world. Our sensors and neurological and social wiring are likely to be well evolved or conditioned to handle the things we do naturally, comfortably, and with easy precision in this domain.

Precise Force Versus Position Control An important clue is revealed when comparing ungrounded gestures (conversational em- phasis, demonstration of emotion, or indication of a relatively discrete command, e.g., stop, come, look over there) with those that entail resistance (almost any kind of tool use, from chopping vegetables to precisely controlling a violin string’s vibration). For humans, precision requires resistance. We are not very good at absolute position control (try to reach out and touch a specific point in space with your hand without looking or groping for landmarks), but we are quite skilled at discerning and producing small vari- ations in force resistance. So to accomplish fine position control, we need something solid to push against. In the tactile domain, by running your finger over a flat surface, you can detect slight variations in texture—rough spots or razor-thin scratches—but not small variations in height over several inches or feet. As noted earlier, the bulk of our tactile sensors, partic- ularly the most sensitive ones, are triggered as the skin moves over a spatial texture at a particular rate. The implication for design is that some kind of grounded resistance is desirable for a precise task. This could be provided by a programmed force feedback system or, alterna- tively, by a passive ground (e.g., tabletop) while nongrounded feedback (such as impulses or vibrations) supplies the programmed feedback. In the latter case, the user’s input will be isometric (without measurable motion), so position sensing cannot be used to meas- ure user intent; pressure might be more suitable. When precision is not needed and broad, expansive gestures are appropriate, nongrounded systems (such as a limb-mounted tac- tile display) might do the job.

Guidance Both force and tactile feedback can be used to provide direct spatial guidance, either by leading with forces or by orienting attention in a particular direction. Spatial orientation 166 Reviews of Human Factors and Ergonomics, Volume 4 usually takes the form of applying a signal to a body location, which then draws visual attention in the same direction (Bertelson & de Gelder, 2004; Driver & Spence, 1994, 2004; Spence & Driver, 2004; Spence et al., 1998). Guidance implies a more continuous engage- ment that is best delivered through grounded force feedback, which could vary in preci- sion and subtlety (e.g., steering a car or aircraft, drawing a calligraphic character, or learning a surgical procedure). Force feedback guidance applications tend to vary across the spectrum of control shar- ing with the intelligent system. A training application would generally entail strong guid- ance, at least to begin with, whereas a skilled driver might want to make use of some system information while ultimately maintaining control (see the Shared Control section). Control can thus be shared with the system, another user, or both.

Training. In instructional applications, the user is expected to follow the lead of the teacher. The teacher could be the system (i.e., a computer program) or another human. The latter case is an instance of shared control/remote collaboration, which, because of intricacies of the control energetics involved, is discussed in more detail later. Haptics has been shown to have value in the training of sensorimotor tasks. Adams, Klowden, and Hannaford (2001) found haptics to be beneficial when it is included in a virtual reality training segment prior to the performance of a Lego™ manual assembly task. Likewise, Morris et al. (2007) showed an advantage of combined haptic-visual train- ing modes over either haptic or visual mode alone for tasks that have a force component. This is consistent with the earlier observation of Feygin, Keehner, and Tendick (2002) that “while visual training was better for teaching the trajectory shape, temporal aspects of the task were more effectively learned from haptic guidance” (p. 40). There are many ways of implementing the construction of training forces, including some that inspire the modification of the displays themselves. Kikuuwe and Yoshikawa (2001) discussed a scheme to haptically display both the activating pressure and position of the trainer to the trainee; because the trainee can be constrained only by pressure or position at a given time, the other quantity must be conveyed in an abstracted form (e.g., position displayed visually while pressure is felt as pressure). Using calligraphic writing as an example, Avizzano, Solis, and Bergamasco (2002) iden- tified human actions using human Markov models, then restricted the user along a pre- defined trajectory (see mention of virtual fixtures later in this chapter). Gillespie, O’Modhrain, Tang, and Zaretsky (1998) and Teo, Burdet, and Limk (2002) proposed means of making the teaching role more sophisticated by measuring the stu- dent’s resistance and backing off as the need for guidance decreased, thus providing a simultaneous assessment capability. This allows the quantitative assessment of progress. The approach has also been exploited to good effect in physical therapy—for example, in rehabilitation of stroke patients (Krebs, Hogan, Aisen, & Volpe, 1998). A special case of haptic training is the use of simulation for training surgical skills, espe- cially for the difficult and sensorially deprived endoscopic and laparoscopic procedures. Together with haptic augmentation of surgical teleoperation, this is probably the haptic application receiving the greatest research attention at present, through a combination of apparent good match, real need (Delp, Loan, Basdogan, & Rosen, 1997; Vozenilek, Huff, Reznek, & Gordon, 2004), and significant research funding opportunities. Because there Haptic Interaction Design for Everyday Interfaces 167 is an extensive literature on this specific application, the reader is referred to comprehen- sive reviews by Camarillo, Krummel, and Salisbury (2004); Howe and Matsuoka (1999); Satava (1999); and Wagner, Stylopoulos, Jackson, and Howe (2007).

Shared control. When control is balanced cooperatively between user and machine, the system might have knowledge of the sensed and networked environment and relevant databases but not of the user’s goals, although it might be able to infer them. If the sys- tem did know the user’s goals, of course, it might be harder to argue for the value of pro- viding guidance for the user to reach them, rather than simply “jumping” there. One way of handling this delicate balance is by sharing the forces that are felt and provided, by both system and user, in the process of exerting system control. This concept seems especially natural in navigational or steering contexts, with their single loci of control, which tradi- tionally has been specified in a physical manner by the user (e.g., using a steering wheel). This kind of force sharing lies on a continuum of abstraction that has, at one end, bilat- eral force-reflecting telerobots, perhaps the most straightforward version. These systems consist of a remote robot located in the work environment, connected by a network to a local robot of compatible kinematic design that a user moves, often wearing it as an exo- skeleton. In one configuration, the user “feels” forces sensed by a remote robot and redis- played on a local robot, while the local user’s motions are sensed by the local robot and translated directly into actuation forces exerted by the remote robot. This scheme allows the local user to be sensitive to the impedance of the remote environment with conse- quently improved dexterity and error management. A vast amount of research and industrial implementation on this topic has occurred since the early work of Jacobsen, Smith, Backman, and Iversen (1991), and an in-depth treatment is beyond the scope of this chapter. An example of how a telerobot begins to act as a shared-force system is illustrated in Taylor et al. (1999): The basic force reflec- tion in a microsurgical application is augmented with some system intelligence to remove hand tremor and otherwise make the robot’s precision and sensitivity available to the human guide. Preceding the articulation of the shared-control concept is that of the virtual fixture, first proposed by Rosenberg (1994), which extends to a virtual environment the utility of a physical ruler or other constraint used to help guide a task by keeping it within spec- ified boundaries. Programmed forces can provide the constraint. One means of using shared forces for mixed-initiative guidance is to relax the hard constraint of this fixture idea. For example, one could model a bead being drawn on a string rather than on an in- flexible rod, or by creating soft constraints that the user could feel, be gently repelled by, or “punch through” if desired. However, many of these approaches can be tricky to implement. The user’s interactions with the rendered system can lead to oscillations because of the user’s instinctive reaction to certain kinds of guidance (Forsyth & MacLean, 2006). Usable solutions depend on the task, but ideally they will build on a database that will be derived from both user models (of reflexive and cognitive responses to disturbances, including control actions perceived as being intrusive) and empirical user studies in both abstract and reasonably realistic contexts. 168 Reviews of Human Factors and Ergonomics, Volume 4

A number of practitioners have implemented schemes by which force control is shared between user and system in some way. In an example of nonlinear input force scaling, Payandeh (2001) allowed a surgeon to preset a tool force limit (in minimally invasive sur- gery, it is important to minimize interaction forces to avoid tissue damage), and then the system enforced that limit via a tunable virtual spring, regardless of the actual forces exerted by the surgeon. The user’s mindset and awareness of where the control balance lies is a variable to be managed. Griffin, Provancher, and Cutkosky (2003) looked at some of the potentially neg- ative side-effects of sharing control in an organization in which the automatic controller handles lower-level manipulation and the human operator is concerned more with higher- level, so-called supervisory actions. Users responded to system intervention with gener- ally improved performance and satisfaction, and manipulation errors were reduced when users were notified that an intervention had occurred. In a different study, Enriquez and MacLean (2004) used programmable throttle pedal resistance that increased as a driver approached a (simulated) leading vehicle from behind, as if there was a virtual spring between the two cars. Here, reliability of the “warning” signal was found to be critical; the driver’s predisposition to utilize the aid was influenced by the past reliability of the signal, with differing responses to past misses versus false alarms (see also Tipper & Kingstone, 2005). In addition, the subjects’ aggressiveness in tailgating in this simulation was found to be influenced by subtle differences in experi- ment instructions, illustrating the difficulty of accurate testing and modeling in time- and safety-critical environments. Others have offered variants on shared control based on virtual fixtures. Using a force feedback steering wheel, Griffiths and Gillespie (2004) showed that fixtures-based assis- tance improves lanekeeping in a driving task and reduces visual demand, both by around 30%. Forsyth and MacLean (2006) also used the fixture of a system-known path to be fol- lowed in a steering task. They addressed problems of instability in high-bandwidth follow- ing by constituting the control signal from a look-ahead prediction algorithm. Kragic, Marayong, Li, Okamura, and Hager (2005) used a segmented task model of a microsurgical procedure to recognize a user’s task state in real time and then to provide context-appropriate assistance. Assistance could include, for example, guidance toward a virtual fixture consisting of a position deduced to be desirable based on the task state. If the user had a reason to go elsewhere, he or she could overcome the guidance. In O’Malley, Gupta, Gen, and Li (2006), a force-sharing driving system actively demon- strated desired motions during virtual environment interactions.

Remote collaboration. Remote collaboration that is reliant on force feedback signals can be considered a special case of shared control, wherein the sharing is done with at least one other human user (with the automatic controller potentially still assuming an impor- tant role). A well-known early instance of collaborative teleoperation that did not involve force feedback allowed (any) user to join in controlling a robot over the Internet. Goldberg et al. (2000); Basdogan, Ho, Srinivasan, and Slater (2000); and Sallnas et al. (2000) explored the utility of haptic feedback in both performance in the collaborative task and the sense of presence and “togetherness” that it engendered, both finding a positive impact of hap- tics. Smith and MacLean (2007) further found an effect of mediating virtual physical Haptic Interaction Design for Everyday Interfaces 169 metaphors on promoting this sense of interpersonal connection in a more explicitly social context (discussed later). In a more complex setup involving surgical training, Nudehi, Mukherjee, and Ghodoussi (2005) set up a force reflection between mentor and trainee in which the balance between the two shifted as learning progressed, each feeling the size of the difference between them. This work raised the notion of control authority between the two collaborators. Finally, as described in the next section, remote collaboration can be aided by haptics through tactile communication as well as sharing of forces.

Abstract Communication and Information Display: Haptic Icons The idea of using haptics to display abstract information has roots in communication aids for the blind, with the Optacon (a pioneering pin-array device that allowed scanning of printed text and conversion to a tactile display), which is notable for its widespread and long-lived use (Linvill & Bliss, 1966). A recent review of this application space can be found in Tan and Pentland (2001), backed up by many reviews of relevant tactile psy- chophysics (Gallace, Tan, & Spence, 2007; Jones & Sarter, 2008; Pasquero, 2006). In the tactile domain, abstract information transmission has centered on haptic icons or their equivalent: brief informative haptic signals (usually vibratory) to which informa- tion has been attached. A comprehensive recent overview of the state of the art in trans- parent tactile communication can be found in MacLean (2008).

Representational or abstract? The jury is out on whether the best approach to haptic signaling is to use metaphorically derived symbols or more arbitrarily assigned associa- tions, in a comparison similar to the contrast between auditory icons (Gaver, 1993) and earcons (Blattner, Sumikawa, & Greenberg, 1989) in the auditory modality. The likely pros and cons are fairly obvious: Metaphorically derived, symbolic notations seem likely to be easier to learn and remember. From a philosophical and social science perspective, Gumtau (2006) observed that “the linguistic model might not be as helpful as a semiotic model in what is mainly a communication of non-verbal cues. It seems that although people voice their desire to establish logical codes, they naturally draw on aesthetic and social codes that help them design meaning successfully” (p. 254). But there are also serious design challenges to making large and usable sets of sym- bolic icons, particularly when the rendering palette is as limited as current tactile display hardware. (Imagine how well symbolic graphics would work using a few grayscale pixels to cover all possibilities.) These challenges include controlling signal salience (a semiotic association might not respect appropriate relative intensity) and perceptual spacing. Both problems are handled relatively easily when the need for semiotic connection is dropped, such as using a process of perceptual optimization on a proposed signal set (e.g., MacLean & Enriquez, 2003, and see the following text). One approach to increasing the controllability of the representational approach is to ascertain a set of basic primitives using careful testing, with the goal of using them across multiple contexts—for example, to represent ordinal data in a variety of situations (Tang, McLachlan, Lowe, Chalapati, & MacLean, 2005). Another approach is for designers to 170 Reviews of Human Factors and Ergonomics, Volume 4 carefully create their own codes, drawing on an existing knowledge base accessed by users (Brown & Kaaresoja, 2006; Chan et al., 2005). Alternatively, users appear to be able to cre- ate their own codes when given the means, on the basis of either emotive associations (Brave & Dahley, 1997; Chang, O’Modhrain, Jacob, Gunther, & Ishii, 2002; Fogg, Cutler, Arnold, & Eisbach, 1998) or informative ones (Enriquez & MacLean, 2008b). The latter case may be a cue for how to join the two approaches. In this study, the exper- imenters (Enriquez and MacLean) believed they had created purely arbitrary links, but they discovered that users had instead created their own semantic mnemonics. These per- sonally derived cues seemed as logical to users as the associations that the experimenters had intended to be semantic. This was borne out in evaluation by an equivalent ability to learn and retain the two kinds of associations; that is, perhaps we can make anything behave as a semiotic link. Psychophysical evidence for acuity. A discussion of the learnability of haptically en- coded signals must begin with a consideration of people’s ability to use the haptic chan- nel for this kind of information transmission, in terms of perceptual acuity and neural mapping. Direction may be taken from detailed psychophysical studies of texture percep- tion, both active and passive, as discussed earlier. Examples of relevant psychophysical studies include those on texture orientation (Hughes, 2006), humans’ encoding of texture spatially rather than temporally (Connor & Johnson, 1992), active versus passive exploration (Klatzky et al., 2003), subjective scaling and exploration strategies for friction (Smith & Scott, 1996), and the qualitatively differ- ent mechanisms and capabilities involved in the perception of rough and fine textures (Bensmaïa & Hollins, 2000; Hollins & Risner, 2000). Recent work has demonstrated that temporal as well as spatial cues are involved in roughness perception (e.g., Cascio & Sathian, 2001; Connor & Johnson, 1992; Gamzu & Ahissar, 2001). Likewise, studies have been conducted both on methods for rendering artificial tex- tures and on the perception thereof. The majority of these studies have focused on textures sensed or displayed through a probe, such as that provided by a Phantom 3D haptic inter- face, a perceptual mode that differs in significant ways from the use of bare-fingered or whole-hand grasps (Klatzky & Lederman, 1999). Whereas a stylus (another kind of probe) is often used to interact with a potentially activated handheld device, the transducer would invariably be a vibrator of some sort rather than a DC motor, with different rendering capabilities and methods. Thresholds have been found for real and virtual textures (Tan, Gray, Young, & Traylor, 2006), as well as the ability of humans to scale roughness through a vibrotactile mouse (Klatzky & Lederman, 2006). From a hardware and control standpoint, Campion and Hayward (2005) identified some system control limits, and Choi and Tan (2004) identi- fied the sources of perceptual instability (a sense of activeness in simulated texture) in force feedback renderings of texture. Finally, Winfield, Glassmire, Colgate, and Peshkin (2007) demonstrated a highly effec- tive use of vibration to modulate perceived friction, and Levesque et al. (2007), a novel Braille-inspired display that acts on a principle of skin stretch. Human ability to learn haptically represented abstractions. Regardless of the ap- proach used to construct a stimulus-meaning link, in deploying the haptic channel for Haptic Interaction Design for Everyday Interfaces 171 the transmission of this kind of abstracted information, we are asking individuals to use their touch sense in a manner they do not encounter in the natural world. Thus, another important question is that of brain plasticity for those users asked to pick up this skill after childhood, and a relevant place to look for an answer is in studies of human ability to learn Braille after childhood. A first step in learning Braille is to develop tactual acuity. Barraga and Erin (1992) listed the following steps in this process, moving from simple to complex:

a. Awareness and attention b. Structure and shape c. Part-to-whole relationships d. Graphic representations e. Braille symbols

The first of these is perhaps the most applicable to synthetic tactile symbols, which are more likely to be temporally rather than spatially encoded. Parents of children who need to learn Braille are encouraged to immerse their children in rich and guided haptic expe- riences that include experiencing different tactile qualities, shape sorting, and concepts of parts and wholes, as well as to think in terms of two-dimensional graphical represen- tations (Blake, 2003). In later stages, Braille is introduced by placing Braille labels on famil- iar objects. Symbols with a single meaning are introduced first, and more complex symbols are learned later (Barraga & Erin, 1992). To what extent can this skill be achieved after childhood? In comparing capacity for attaining tactile hyperacuity between blind and sighted adults, Grant, Thiagarajah, and Sathian (2000) noted that the performance of blind individuals, though initially better in some respects, can be matched by sighted individuals who receive training. Hamilton and Pascual-Leone (1998) provided neural evidence for brain plasticity; see also Gallace, Tan, and Spence (2007) for a review including further functional magnetic resonance imaging (fMRI) support for this conclusion. The presence of significant individual differences in tactile acuity and ability to learn abstract associations has been recognized, including both occasional hyperacuity (Craig, 1977) and, anecdotally, the so-called haptically challenged. It is not known whether this range arises through perceptual function or cognitive abilities. Differences in how indi- viduals organize their perceptual space have also been noted, with the most salient dimen- sions being related in similar ways across individuals; in both cases, less salient dimensions contributed differently to individuals’ organization of the stimuli (Hollins, Bensmaïa, Karlof, & Young, 2000). Both types of variation (in ability and organization) have impli- cations for the widespread introduction of haptic information displays. An important area of future work is the need to better attribute the causes of both poor and exemplary hap- tic function and to ascertain whether training and awareness can improve the former.

Creating learnable icons. Research and current practice in creating and teaching hap- tic icons to users—those that are both metaphorically inspired and completely arbitrary— suggest two primary issues that need to be addressed. The first is to ensure that the stimuli are perceptually discernable, and the second is to ensure that peoples’ preferences and abilities for organizing them are well understood. One tool for achieving this is to use 172 Reviews of Human Factors and Ergonomics, Volume 4 multidimensional scaling (MDS) to perceptually optimize stimulus sets. In this approach, users provide dissimilarity data about a stimulus set (e.g., by rating or sorting exemplars), which are then used to create a perceptual map (MacLean & Enriquez, 2003; Pasquero, Luk, Little, & MacLean, 2006; Ternes & MacLean, 2008). A simple example of such a map is shown in Figure 5.6, which displays descriptions of stimuli and user-derived perceptual organization (i.e., dimensions and spacing). The MDS map can be (a) iteratively revised until the desired perceptual organization is achieved (Chan et al., 2005; MacLean & Enriquez, 2003) and (b) used to choose a subset of stim- uli for actual use in an application, again according to their desired perceptual spacing. This method can be used both for independent creation of stimuli intended for arbitrary mapping to meanings and for adjustment of a prototype set of representational icons whose meanings are chosen a priori (Chan et al., 2005). Perceptual optimization does not directly solve the problem of linking stimulus to meaning; in the case of abstract connections, this is a matter of learning. In discussing short-term tactual memory, Millar (1999) argued that “convergence and overlap of infor- mation from different sources is crucial to parsimonious organization for memory and

(a) (b)

Figure 5.6. A simple example of a perceptual MDS map of 30–36 stimuli. Iterations 1 (a) and 2 (b) of a stimulus design cycle; the first map was used to improve the stimulus spacing in the second. The changes in stimuli made to achieve better use of the design space can be seen by comparing the two legends. Reproduced with permission from MacLean and Enriquez (2003). Haptic Interaction Design for Everyday Interfaces 173 recall” (p. 747). This suggests that even to learn a stimulus that might later be invoked purely through the haptic channel, a multisensory reinforcement learning process could be advantageous. In terms of empirical results, users have already demonstrated good ability to learn asso- ciations that are either metaphorically matched by the designer (e.g., Brown & Kaaresoja, 2006; Chan et al., 2005; Tang et al., 2005), deliberately arbitrary (Enriquez, MacLean, & Chita, 2006), or chosen by the user. In the studies mentioned here, learning has taken the form of repeated exposure/testing cycles of stimulus-meaning pairs until a given perform- ance criterion is reached. However, some researchers have also taken a further step of icon testing and optimization under realistic environmental “stress testing,” adjusting the stim- uli for relative distinctiveness and salience as needed. For example, in some circumstances, a controlled degradation in salience or detectabil- ity is desirable when workload increases; some important icons are still being noticed, but less critical ones “wash out” when more urgent tasks are in play (Chan et al., 2008). The study comparing the deliberately arbitrary and user-chosen approaches to icon development shows no difference in retention after a two-week interval (Enriquez & MacLean, 2008b), which raises the interesting question of whether multimodal reinforce- ment, as advocated by Millar (1999), might be more important even than semiotic con- tent for the stimulus itself.

Current status of dimensionality and learnability research. Several studies on the per- ceived dimensionality of real surfaces sets the upper limit for what one is likely to achieve by synthetic means. These studies have generally used MDS, collecting user data through ratings of individual material samples with respect to several requested dimensional scales. This use of MDS then demonstrates the number of dimensions required to express the variability in those responses. In the first of these studies, Hollins, Faldowski, Rao, and Young (1993) and Hollins et al. (2000) found three dimensions for 17 stimuli— rough/smooth, hard/soft, sticky/slippery or, perhaps, springiness. The last dimension showed more variation among individuals and was possibly less heavily weighted. More recently, Ballesteros, Reales, de Leon, and Garcia (2005) examined 20 textures and discovered just two dimensions for natural tex- ture: rough/smooth and slippery/adherent. Noting that two to three dimensions are as many as are likely to be found by an MDS algorithm for a set of 17 stimuli, Bergmann, Tiest, and Kappers (2006) compared the most samples (124) and found 4 dimensions, with the most clearly interpretable attributed to roughness and compressibility (these, however, might be composite percepts). This last result has the most intuitive credibility; whereas two or three dimensions might explain the variability in texture alone, this seems inadequate for explaining the full range of ex- pressiveness in the larger world of real tactile stimuli. However, today’s devices are far from capturing this entirety, so two to three simultaneously displayed and perceived dimensions might be a practical estimate for now. The foregoing results are for natural textures. What has been produced synthetically? Stimulus dimensions employed to date in rendering abstract haptic signals include vibra- tory frequency, amplitude, waveform/roughness, rhythm, and spatial location (Brown, Brewster, & Purchase, 2006; MacLean & Enriquez, 2003; Michelitsch, Ruf, van Veen, & 174 Reviews of Human Factors and Ergonomics, Volume 4 van Erp, 2002; van Erp & Spapé, 2003) to achieve usable set sizes on the order of 15–36 stimuli; at most 3 of these dimensions are ever used at once. A systematic exploration of rhythm space has increased this number to 84 in a perceptually optimized set, by com- pounding 21 rhythms with 2 frequencies and 2 amplitudes (Ternes & MacLean, 2008). Application of haptic icons. A number of researchers have explored the use of haptic abstract representations in simulated application contexts, through the display of either fully tactile or mixed-force and tactile representations for which participants were required to learn associations. Nesbitt (2002) used force feedback displayed on a Phantom—includ- ing texture, hardness, and object inertia in a virtual model—to convey stock market data. Using metaphorical relationships, Chan et al. (2005, 2008) added mouse-based tactile feedback (using an intentionally low-end commodity device, to test feasibility) to a remote collaboration task. These signals were used to mediate turntaking via the haptic channel in order to free voice and visual channels. The premise was that this would improve col- laborative quality, as defined by measures such as equitability of time in control and con- trol turnover rates, through a background awareness of others’ wish to participate. The 7 mediating icons were easily learned in 3 min for the set; subjects maintained 97% accu- racy of identification even under substantial multimodal workload. The icons were utilized in a graded (i.e., appropriate) way in a fully simulated multitasking context, and results of the observational study suggested that collaboration dynamics were positively affected. Brown and Kaaresoja (2006) tested users’ ability to form iconic associations to cell phone callers, using 3 dimensions (roughness, intensity, and rhythm) to create 9 icons dis- played on the same eccentric-type motor as in the Chan et al. (2005) study. Meaningful associations relating to cell phone messages were learned with a 72% recognition rate but were not tested in a realistic context. Two recently introduced devices are based on a tactile skin stretch array that has poten- tial as a comfortable and expressive display. In this concept, parallel piezo actuators suc- cessively compress or stretch the skin and create dynamic sensations such as a “caterpillar” wave and many variants, including simple or spatially complex vibrations. Both devices have been tested for utility in abstract haptic representation. In a handheld device (THMB: 8 actuators aligned in a single row), the novel skin-stretch sensation was first perceptually characterized for its dimensionality, and initial studies were performed for its use in mobile handheld interactions such as scrolling and notifi- cation (Luk et al., 2006). In a desktop device called Stress2 (a 2-D array of individually controllable actuators targeted at blind Braille users), tactile graphics are used to test asso- ciations to pictorial concepts (Wang, Levesque, Pasquero, & Hayward, 2006). This mix of fully and semi-situated tests together give a strong indication that people are able to learn and utilize complex haptic signals, even in demanding environments. The most important next steps are to more fully understand the interplay with other percep- tual, attentional, and cognitive demands in the challenging environments where they will be most useful.

Notifications and Background Awareness Passive touch cues (which the observer receives rather than seeks; Gibson, 1962) can be used to notify of state changes or events and to maintain nonintrusive ambient levels of Haptic Interaction Design for Everyday Interfaces 175 background awareness. Such cues might be delivered through a tactile display or overlaid on a force feedback signal being used for another, possibly related, function. Typically, this kind of functionality targets multitasking environments, when the user’s primary atten- tion, as well as visual resources and possibly the hands, are engaged in another task. Several issues pertain to this situation. In terms of physical configuration, it is usually necessary for such displays to be in continual contact with the stimulus site, so that signals will not be missed. Because it may be desirable to keep the hands free for more dextrous roles, the glabrous skin of the fingertips is often unavailable as the delivery site, which for the most part leaves the less sensitive hairy skin (Gallace et al., 2007). In indi- vidual projects, vests and belts have been used (Jones, Nakamura, & Lockyer, 2004; van Erp, van Veen, Jansen, & Dobbins, 2005), the back (Tan, Gray, et al., 2003; Yanagida, Kakita, Lindeman, Kume, & Tetsutani, 2004), and the tongue (Bach-y-Rita, Kaczmarek, Tyler, & Garcia-Lara, 1998). Applications and contexts in which hands can be used for background displays include steering wheels (Enriquez, Afonin, Yager, & MacLean, 2001) trackpoint (Campbell et al., 1999), mouse (Chan et al., 2005, 2008), and, increasingly, mobile devices (Kaaresoja, Brown, & Linjama, 2006; Luk et al., 2006; Poupyrev & Maruyama, 2003, and see section below). More fundamentally, Gibson (1962) argued that passive touch, wherein the person focuses on the “events at the sensory surface,” is not a natural state: “In a tactual situation, the observer will explore with his fingers unless prevented,” and the experience is of the observer’s environment (pp. 489–490). Active touch is predominant in naturalistic en- vironments in which people are seeking information (Sarter, 2006). However, because delivery sites are generally not on the hands and information is typically intended to be nonattentive, the premise that passively received information display will be less effective will be tricky to test, but it is an observation to keep firmly in mind. Design for multitasking environments. In terms of the design of the signals themselves, the multitasking nature of the environment is all-important. To manage intrusiveness and allow the user to optimize his or her allocation of attention, signals must be designed with variable salience. That is, important events or urgent events or changes should register as “louder” than less important ones (Chan et al., 2005). In addition to issues of sensory adaptation to the signals, the user’s interruptibility is not a constant. In the car, for example, interruptibility differs substantially depending on whether the driver is sitting at the roadside versus engaged in a turn. In the office, some tasks require protection from routine interference, and yet certain events might always be important enough to warrant an interruption. Researchers in the field of sensor-based computing are working on the problem of automatically and continuously monitoring user interruptibility by detecting various aspects of the context, such as location (Moran & Dourish, 2001; Pantic, Pentland, Nijholt, & Huang, 2006; Schmidt, Beigl, & Gellersen, 1999; Scholl, Hasvold, Henriksen, & Ellingsen, 2007) and in modeling and detecting user mental or emotional state (Chen & Vertegaal, 2004; Fogarty et al., 2005; Horvitz, Kadie, Paek, & Hovel, 2003).

Augmentation of Graphical User Interfaces The augmentation of GUIs, whether using a vibrotactile or force feedback mouse on a 176 Reviews of Human Factors and Ergonomics, Volume 4 desktop or on mobile displays, should be addressed as an obvious candidate for the use of haptic feedback. However, to date it has not proved extremely productive. Part of the problem is that GUIs were designed to be graphical, and it is difficult simply to layer another modality on a visually based framework and expect it to work well. Most efforts here have been to give physical edges to things, including graphical widg- ets—such as clickable icons and taskbars—or more specialized objects—for example, word processor textual units or edges of areas segmented by a photo editor (Miller & Zelez- nik, 1998). A classic and inspiring problem is the annoying way one “falls off” a pull-down menu after riding it down through several levels. In a study based on a steering model (Accot & Zhai, 1997) for this kind of interaction, Dennerlein, Martin, and Hasser (2000) found performance improvements when haptic feedback reduced the incidence of losing the trajectory by providing boundaries using a force feedback mouse. However, this kind of solution begs the question of why, if the sys- tem already has enough knowledge of the user’s goals to provide this level of constraint, it does not just take him or her directly to the desired destination. Obviously, some assistance can be provided by haptic feedback in this context. Akamatsu, MacKenzie, and Hasbrouq (1995) showed benefits of tactile feedback in point- ing tasks, suggesting it improved the targeting parameters. Haptic widgets and other con- straints can still provide overrideable force guidance (support rather than obstruction), as well as other kinds of information. Dynamic trajectory prediction is another possible approach. However, performance results for both predicted trajectories and complex widg- ets have been inconclusive; for example, see the studies by Oakley, Adams, Brewster, and Gray (2002) using a Phantom with desktop GUIs. Handheld mobile devices, more extensively considered later and by Lewis et al. (2008) in this volume, provide another possibility for GUI augmentation, with vibrotactile feedback applied to stylus or hands during graphical manipulations such as text selec- tion, scrolling, and button clicks (Kaaresoja et al., 2006; Leung, MacLean, Bertelsen, & Saubhasik, 2007; Poupyrev, Okabe, & Maruyama, 2004). Because of its degraded visual information and restricted manual parameters, this augmentation might provide a more definite performance benefit. Another approach, of course, is to devise completely new interaction techniques based on haptic or tactile feedback (e.g., Smyth & Kirkpatrick, 2006) and as inspired in a nontactile format by Harrison, Fishkin, Gujar, Mochon, and Want (1998). When these techniques are shown to be effective, there still remains the prac- tical obstacle of making haptic feedback necessary for an interaction before everyone has access to it.

Expressive Control A digression on expressive capacity. Expressive is used here to refer to the quality or power of expressing an attitude, emotion, or other communicative information. The term expressive interface is often used as shorthand for a tool that supports artistic or inter- personal communication (e.g., a music controller or a facial gesture); however, I use the term expressive capacity can be used to more broadly describe the richness of a commu- nication channel for any purpose, including its dimensionality or continuity, the degree of control it affords the user, and the ease and naturalness with which desired acts can Haptic Interaction Design for Everyday Interfaces 177 be completed (MacLean & Enriquez, 2003). More precisely, expressive capacity includes the following: a. Density: number of “bits” of information that can be transmitted b. Controllability: accuracy of conveyance (comprising the act of communication, transmission, and successful interpretation by the recipient) c. Directness: versus encoded nature of the required actions (similar to the difference between direct-manipulation and command-line interfaces in HCI) d. Responsiveness: the immediate confirmatory and/or gratifying feedback provided to the user e. Emotiveness: the amount (range, subtlety, etc.) of emotion that can be infused into the channel. By this measure, a computer keyboard is a highly expressive interface on the first two counts, but it fails miserably on the third and fourth. The fifth is tricky: The product of typing (the printed word) can be highly emotive in every way, both visually (ask a type- setter) and in linguistic meaning. But the act of typing is not particularly emotive, unless you happen to be using a keyboard that senses how hard you’re striking and adds to the message appropriately for a flame, a love note, or a particularly elegant bit of computer code. One such device has used water and steam for this purpose (Mann, 2005).

Expressive for whom? This raises the interesting question of whether an input device should be classified as expressive based on its output if using it doesn’t feel expressive. This may seem semantic, but the difference is important. To be expressive in communication with others when the input device doesn’t feel that way requires far more training and ability to abstract the expressed concept than otherwise. We can do it with typing because we are experts at language, but many people find it much easier to say or gesture expres- sive things than to write them. Conversely, traditional (and difficult-to-use) acoustic instru- ments such as a violin or piano or the voice tend to exhibit flaws in expressive output in the hands of the untrained. However, the budding musician might really enjoy the bang- ing or belting out because of the immediate feedback of sounds and responsive feel.

Haptic interfaces and expressive capacity. Physicality seems a completely natural— indeed, essential—property for control tasks requiring both emotiveness and precision. As noted earlier, on the other hand, an ungrounded gestural interface might be best for purely emotive control because of its lack of constraints. A keyboard is hard to beat when you wish to indirectly but exactly specify the greatest possible range of actions—maximum controllability. Taking a cue from the most-studied expressive interface application of music con- trollers, many argue that the resistance and feedback of either passive or active (computer- controlled) forces or vibrations are essential to controllability (Askenfelt & Jansson, 1992; Gillespie, 1992; Rovan & Hayward, 2000; Verillo, 1992). For example, Bongers (1997) reported that “tactually displaying attributes of the sound (e.g., pitch, volume enve- lope, timbre) enables the performer to improve muscular control... and led to the realiza- tion that the link between sound source and its ‘feel’ was missing” (p. 1). Along with others (e.g., Gillespie, 1999; O’Modhrain, 2001), Luciani, Florens, and Castagne (2005) further linked this effect to a consistency or closing of the control loop, using the term ergotic to refer to cases in which there is a mechanical interaction between 178 Reviews of Human Factors and Ergonomics, Volume 4 the subject and the sound source. They reported: “force feedback devices and physically based models are indeed required for implementing virtual ergotic action-sound systems, able to engrave the energetic consistency of the physical action in the sound” (p. 592). Although it is usually positive to be able to both hear and explore the physics (whether real or virtual) of the system one is interacting with, computer-controlled devices that enable this also bring constraints: tethering and a loss of workspace, weight, motors and electrical power, and a lack of generality in the control actions and handles that can be used. For example, the need for extremely tight synchronization between action and sound can push the limits of non-real-time operating systems (e.g., Beamish, MacLean, & Fels, 2004). Thus, until haptic technology catches up with our needs, creativity and compro- mise are needed to carry out this mandate. Several recent resources give guidance in meet- ing this challenge, from the standpoint of both the fundamental interactions themselves and of their mechatronic implementation (Bongers, 2000, 2006; Cook, 2001; O’Sullivan & Igoe, 2004). To date, applications of haptics to music control include both enhancing the experi- ence of the listener of a musical performance (Gunther & O’Modhrain, 2003) and, more commonly, aiding the performer (Beamish et al., 2004; Gillespie & O’Modhrain, 1995; Gunther & O’Modhrain, 2003; Nichols, 2002; Rovan & Hayward, 2000; Verplank, Gurevich, & Mathews, 2002). Most of these examples are characterized by the strong individuation of instrument to application—that is, type of music to be created and gestures employed. These are not general-purpose devices. Music interfaces are certainly not the only place where expressiveness and controlla- bility are called for (MacLean, 2000). A simulation of the feel of a bristled paintbrush is demonstrated in Baxter et al. (2001). Some initial explorations in animation control demonstrate the need for new and sophisticated controllers and interaction techniques to address severely underactuated systems. Donald and Henle (2000) used a Phantom haptic display to manipulate high-degree-of-freedom trajectories collected from motion capture data, whereas Zhao and Panne (2005) confronted the need for direct control of 13 degrees of freedom during real-time, physically based animation synthesis using a non- actuated gamepad.

Use of tangible, analog metaphor and abstraction in expressive control. Along with digitization of once-tangible tasks and ubiquitous computing necessitating controllers throughout one’s world (not just the desktop), there comes the frequent need to manage information or control systems through very simple input devices. When they are hapti- cized, this generally comes down to knobs and sliders or possibly planar displays. When information tasks have roots in predigital interactions, useful metaphors can be exploited to aid control. Snibbe et al. (2001) introduced—and MacLean, Shaver, and Pai (2002) extended—the idea of a mediating virtual physical metaphor for interacting with media. The haptic representation in this case is not of the media itself but of a metaphor- ical tool that will aid in the task at hand (a virtual clutched connection with a high-inertia roll of “film” that runs on the computer screens as the roll spins). This layer of applied tangibility arises from earlier mechanisms for handling celluloid film and allows a more fluid handling of the information than cursor clicks of stop/start buttons. The film roll example also illustrates how, when traversing digital media, we jump Haptic Interaction Design for Everyday Interfaces 179 between discrete and continuous forms of the material, its content, and aggregations (as articulated in Ramos & Balakrishnan, 2003, in a nonhaptic implementation). A video stream is a succession of frames—discrete when played slowly, but merged into a fluid per- ceptual experience when speeded up. Spinning the virtual wheel or roll described earlier allows one to move seamlessly between these states as the “tick tick” of individual frames speeds into a texture as the frame rate fuses visually. A set of voice mail messages are discrete objects, but when played, they are continu- ous streams. A cable TV channel is also a stream. A collection of channels consists of indi- vidual items, but if they are represented in the right way, a viewer can skim over them as if they were a texture; that is, he or she can feel for the variation that indicates a prop- erty of interest. Shahrokni et al. (2006) discussed the use of haptic modes as another means of navigating these differences, an approach also used in Beamish et al. (2004) to accom- modate different purposes for moving through a clip of music. Michelitsch et al. (2002) proposed a start for a classification of dimensions for this sort of interaction (continuous/ discrete, scale range, order of control—e.g., velocity vs. position—and functions specific to the application). They also specified a set of haptic primitives and implementation sug- gestions. Other thought-provoking taxonomies are provided in A. Bongers (2006) and Klemmer, Li, Lin, and Landay (2004).

Communication of Affect Whether the need is to establish a more “human” connection with a remote collaborator, to support computer-mediated communication between users or between user and com- puter, or simply to make manual controls that feel good (or appropriately bad), we need to better understand what feels good or bad and what underlies emotional communica- tion through touch. What feels good? Let us start by considering affective reactions engendered by haptic stimuli, independent of their social context. For application building, it is good to know which sensations are generally regarded as nice or nasty when trying to predict responses to messages carrying emotional meanings. Knowles and Sheridan (1966) were possibly the first to assess subjective responses to the feel of manual controls—in their case, for the purpose of making more usable aircraft cockpits. Their work was notable in its early ac- ceptance of the importance of feel in usability, despite possible independence from meas- urable impact on immediate performance. Using current affective assessment techniques (biometric and self-report), Swindells, MacLean, Booth, and Meitner (2006) looked at responses to textures and assessed the impact of these responses on performance. In these studies, a complex interdependency was found between preference and performance, raising the possibility that preference can override performance when the latter is not all-critical. A process for affective design for manual controllers (whether active haptic or old-fashioned mechanical) is described in Swindells (2007). Haptics and emotional communication. “Feather, scent, and shaker” are three concepts for supporting simple intimacy—visual, olfactory, and haptic (Strong & Gaver, 1996). In offering these concepts, the authors noted, 180 Reviews of Human Factors and Ergonomics, Volume 4

Most current collaborative systems demand explicit communication. They rely on symbolic messages—usually language—which means that communicative acts must be overtly articulated by the sender, and that their reception is a relatively focused and attention-demanding endeavor for the recipient. The use of symbols also implies that the process is one of transferring information, whether about facts or opinions or beliefs. Finally, the broad purpose of current systems is to support goal-oriented behavior such as planning, design, or problem-solving, in which communication serves some external aim. (p. 30) Haans and IJsselsteijn (2006) provided an excellent review on mediated social touch: “the ability of one actor to touch another actor over a distance by means of tactile or kines- thetic feedback technology” (p. 153). The earliest examples include a pair of prototypes in which users could exert forces on one another’s hands over a physical (e.g., bicycle- brake-cable) tether:“In touch” (Brave & Dahley, 1997) and “Hand Jive” (Fogg et al., 1998). The latter points out the need for physical indirection in the connection, to trigger a more interesting interplay and avoid a tendency toward thumb-wrestling. A “haptic doorknob” offered a more ephemeral and asynchronous force connection, by representing haptically as well as aurally the sensed mood of a room earlier in the day, messages left by passers-by, and so on (MacLean & Roderick, 1999). ComTouch is a rep- resentation of a remote hand-squeeze, suggesting bilateralism with a squeeze at one end resulting in a vibration at the other (Chang et al., 2002). Haptic Instant Messaging sug- gests a means of lightweight emotional contact using a vibrotactile layer for chat sessions on mobile devices (Rovers & van Essen, 2004), and “Hug over a Distance” (Mueller et al., 2005) prototypes a vibrotactile vest for couples. “Tap Tap” uses a vibrotactile display in a scarf for distributed touch therapy (Bonnanni, Lieberman, Vaucelle, & Zucherman, 2006). These individual case studies are provocative and have led to some insights, but their mixed success suggests that more systematic investigation is needed to understand how, exactly, emotion is communicated through touch. Stripping the interaction down to a single-degree-of-freedom forced knob connection, with visual and verbal contact removed, Smith and MacLean (2007) tested the role of two factors (personal relationship and the virtual metaphor used in the haptically rendered interaction) on users’ abilities to convey and receive emotion. Results showed that four basic emotions could be communicated with good success and that more intimate mediating metaphors worked the best. The fact that couples liked, but strangers usually disliked, the interaction suggested that emotional connection (and its taboos) were experienced by users. Bailenson, Yee, Brave, Merget, and Koslow (2007) also explored virtual interpersonal touch in a different format, comparing prerecorded emotion in a handshake with a real (nonmediated) version; they found identification performance better than chance, but not as good as unmediated. In ongoing work, Yohanan, Chan, Hopkins, Sun, and MacLean (2005; Yohanan & MacLean, 2008) have explored the basis of haptic emotional commu- nication through the use of a simulated animal model. This research sets aside the load- ing of interpersonal touch by instead measuring how people convey and read feelings and mood to and from a mechatronically sensed lap pet. Mobile and Handheld Computing Accessing data and executing simple tasks while on the go is the information application Haptic Interaction Design for Everyday Interfaces 181 of the 21st century. Many see mobile computing, with its built-in attributes of divided attention, eyes and sometimes hands busy—all with hyperportable aids—as a natural place for enhanced haptic interfaces. This is evinced by a small explosion of both tech- nological and interaction development in this domain, mostly since about the year 2000. There is an enormous engineering constraint at play in this development: Actuators must be tiny and power-efficient yet able to create perceptually significant and diverse stimuli. Although hardware issues fall beyond the scope of the current chapter, most of the papers cited in this section contain information about this topic as well as a variety of approaches to hardware design (Poupyrev et al., 2002, provide a particularly useful overview of the landscape). In the mobile context, the user needs both to issue directives and to receive informa- tion in a form that is usable in his or her current context. In many—and probably the best—envisioned scenarios, these two roles are tightly integrated. Feedback helps the user navigate through an information space as well as deliver the sought-after result in a closed- loop negotiation that does not involve the eyes. In the following paragraphs, I discuss a few of these efforts on the basis of style of interaction.

Large gestures. When one defines large to imply at least wrist-scale (as opposed to finger) motions, these gestures have an obvious role. O’Modhrain (2004) argued com- pellingly for the considered role of body motion and proprioceptive cues in mobile ap- plications, particularly when the result of an interaction is evoked remotely. Taken to the extreme, there is no single “device” locus but, instead, a body-centric information map that is accessed with gestures dubbed body mnemonics (Angesleva, Oakley, Hughes, & O’Modhrain, 2003). More limited variants use gesture in the frame of a handheld device, most of which sense tilting with accelerometers (beginning with Harrison et al., 1998) and display con- firmatory vibrotactile feedback for tasks such as navigating through one-dimensional lists or menus (Linjama & Kaaresoja, 2004; Oakley & O’Modhrain, 2005; Poupyrev et al., 2002). Yao and Hayward (2006) used tilt and a carefully crafted physical virtual model of rolling to control an illusion of varying lengths of a tubular container, whereas “Shoogle” uses a shaking metaphor to promote interactive exploration to display infor- mation (Williamson, Murray-Smith, & Hughes, 2007). Both examples leverage our expec- tations of external physical systems as opposed to O’Modhrain’s more body-centric view. Such illusions, which exploit the user’s expectations about the external world, are another way in which information can be conveyed. Those gestural input approaches that have been subjected to performance evaluations show some promise of performance im- provement, as well as compelling and natural interaction modes. However, care must be taken when they are coupled with a screen, in which case, tilting can interfere with its view.

Finger-scale gestures. Smaller, finger-scaled gestures typify interaction with another class of displays: either stylus/finger movements and coupled vibrotactile feedback, or force or tactile feedback in response to a thumb’s scrolling. A number of vibrotactile touch screens have supported the former in explicitly handheld contexts, with a focus on developing interaction paradigms to exploit the technology (Kaaresoja et al., 2006, 182 Reviews of Human Factors and Ergonomics, Volume 4 and Poupyrev et al., 2004, both employ piezo technology). Immersion’s Touchsense tech- nology, based on a solenoid actuator, has been marketed in a larger range of applications and comes with a toolkit of effects that can be combined in different ways (Immersion Corporation, 2006). These technologies differ significantly in terms of strength and richness of signal. Piezo actuators, such as voice coils, do not deliver the same “punch” but are capable of richer, more varied waveforms, including a sharp attack. This difference in technologies deter- mines where each can be used effectively. A haptic icon will not be distinguishable by a user when it is displayed on a piezo device sitting in her pocket; that same user, however, would be able to feel a solenoid vibrating there. Piezos and voice coils may therefore be better for detailed information display and navigation aids through the finger or stylus, using haptic signaling (e.g., Brown & Kaaresoja, 2006; MacLean & Enriquez, 2003; Ternes & MacLean, 2008, and see the earlier section) or WIMP (window, icon, menu, pointing device) navigation aids (Leung et al., 2007; Poupyrev, 2004). Other larger-scale vibratory actuators will be more suitable for pronounced notifications and alerts reflecting a more limited vocabulary. The thumb is a special case: It is more naturally used for pointing than for navigat- ing. The touch screen approach is not as obviously relevant because of the hand’s geom- etry, although future device designers might find a way to exploit this; for example, a variable-friction display incorporated into a side panel, such as described in Winfield et al. (2007). Other approaches have engaged the thumb’s motion explicitly, with interface components that move in the presence of feedback. The earliest example was envisioned in Snibbe et al. (2001), and prototyped in MacLean et al. (2002), as a small force feedback wheel carefully configured in a handheld form factor to enable navigating and control- ling of streaming media through the implementation of metaphorical interactions. Taking an entirely different approach, Luk et al. (2006) and Pasquero et al. (2007) miniaturized a piezo skinstretch tactile display and mounted it on a passive slider, allow- ing for thumb movement in browsing and navigation. A more recent prototype has springs at the ends of the slider to support velocity control modes, which have been found to be the most effective means of navigating large spaces in the context of the thumb’s limited range of motion (and device real estate constraints). In summary, mobile devices represent a context in which haptic interfaces could play an important role. However, it is clear that to do so, haptics must be included in these devices’ dominant interactive metaphor from the start. In part because of the difficult operating environment and the lack of visual feedback, the haptic role must be holistic and consistent throughout the potentially many and diverse tasks that the device supports. The interaction design challenges are as significant as the technical constraints, and their solutions must lead the technical solutions.

SUMMARY

In this chapter I have provided the background and orientation that could set a novice designer on his or her way toward bringing haptics successfully into an interactive prod- uct. A basic but integrated awareness of human capabilities on one hand and current Haptic Interaction Design for Everyday Interfaces 183 device technology on the other should help to define appropriate roles for haptic interac- tion. I have explored a number of these roles explicitly, relating knowledge about percep- tual and motor capabilities to the needs and constraints of key application contexts. How to execute good haptic interaction design in a systematic, generalizable way is a young and evolving topic not addressed here; however, the refinement of such methods is clearly the research community’s next task. The convergence of an information-based economy with improving haptic technology makes this research focus an exciting one with the potential to significantly improve our technology-saturated lives.

REFERENCES

Accot, J., & Zhai, S. (1997). Beyond Fitt’s law: Models for trajectory-based HCI tasks. In Proceedings of CHI ’97 (pp. 295–302). New York: Association for Computing Machinery. Ackerman, D. (1990). A natural history of senses. New York: Random House. Adams, R. J., Klowden, D., & Hannaford, B. (2001). Virtual training for a manual assembly task. haptics-e, the Electronic Journal of Haptics Research, 2(2), 1–7. Akamatsu, M., MacKenzie, I. S., & Hasbrouq, T. (1995). A comparison of tactile, auditory, and visual feedback in a pointing task using a mouse-type device. Ergonomics, 38, 816–827. Angesleva, J., Oakley, I., Hughes, S., & O’Modhrain, S. (2003). Body mnemonics: Portable device interaction design concept. Poster presented at UIST’03 Vancouver, Canada. Askenfelt, A., & Jansson, E. V. (1992). On vibration sensation and finger touch in stringed instrument playing. Music Perception, 9(3), 311–350. Avizzano, C. A., Solis, J., & Bergamasco, M. (2002, March). Teaching to write Japanese characters using a haptic interface. In Proceedings of the 10th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (pp. 255–262). Los Alamitos, CA: IEEE Computer Society. Bach-y-Rita, P., Kaczmarek, K. A., Tyler, M. E., & Garcia-Lara, M. (1998). Form perception with a 49-point elec- trotactile stimulus array on the tongue: A technical note. Journal of Rehabilitation Research and Development, 35(4), 427–430. Bailenson, J. N., Yee, N., Brave, S., Merget, D., & Koslow, D. (2007). Virtual interpersonal touch: Expressing and recognizing emotions through haptic devices. Human-Computer Interaction, 22(3), 325–353. Ballesteros, S., Reales, J. M., de Leon, L. P., & Garcia, B. (2005). The perception of ecological textures by touch: Does the perceptual space change under bimodal visual and haptic exploration? In Proceedings of Eurohaptics Conference, 2005 and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2005. World Haptics 2005 (pp. 635–638). Los Alamitos, CA: IEEE Computer Society. Barraga, N., & Erin, J. (1992). Visual handicaps and learning. Austin: ProEd. Basdogan, C., Ho, C. H., Srinivasan, M., & Slater, M. (2000). An experimental study on the role of touch in shared virtual environments. In ACM Transactions on Computer-Human Interaction (TOCHI) (Special issue on human-computer interaction and collaborative virtual environments), 7(4), 443–460. Basdogan, C., & Srinivasan, M. A. (2002). Haptic rendering in virtual environments. In K. Stanney (Ed.), Hand- book of virtual environments (pp. 117–134). Mahwah, NJ: Erlbaum. Baxter, B., Baxter, W., Lin, M., Manocha, D., & Wendt, J. (2001). DAB: Interactive haptic painting with 3D vir- tual brushes. In ACM SIGGRAPH 2001 video review on Animation theater program. New York: ACM Press. Beamish, T., MacLean, K., & Fels, S. (2004). Manipulating music: Multimodal interaction for DJs. In Proceedings of ACM Conference on Human Factors in Computing Systems (CHI ‘04), CHI Letters (vol. 6, no. 1, pp. 327–334). New York: ACM Press. Bensmaïa, S. J., & Hollins, M. (2000). Complex tactile waveform discrimination. Journal of the Acoustical Society of America, 108(3), 1236–1245. Berenson, B. (1930). The Italian painters of the Renaissance. Oxford: Oxford University Press. Bergmann Tiest, W. M., & Kappers, A. M. L. (2006). Analysis of haptic perception of materials by multidimen- sional scaling and physical measurements of roughness and compressibility. Acta Psychologica, 121(1), 1–20. 184 Reviews of Human Factors and Ergonomics, Volume 4

Bertelson, P., & de Gelder, B. (2004). The psychology of crossmodal attention. In C. Spence & J. Driver (Eds.), Crossmodal space and crossmodal attention (pp. 141–177). Oxford: Oxford University Press. Blake, S. J. (2003). Teaching reading in Braille. Retrieved October 27, 2008, from http://blindness.growingstrong. org/ed/aa010401a.htm Blattner, M. M., Sumikawa, D. A., & Greenberg, R. M. (1989). Earcons and icons: Their structure and common design principles. Human-Computer Interaction, 4(1), 11–44. BMW World. (1999). The BMW iDrive console. Retrieved October 27, 2008, from http://www.bmwworld.com/ technology/idrive.htm Bongers, B. (1997). Tactile display in electronic musical instruments. In Proceedings of IEEE Colloquium on Developments in Tactile Displays (Digest No. 1997/012, pp. 7/1–7/3). Los Alamitos, CA: IEEE Press. Bongers, B. (2000). Physical interfaces in the electronic arts. Interaction, theory and interfacing techniques for real-time performance. In M. Wanderley & M. Battier (Eds.), Trends in gestural control of music (pp. 41–70). Paris: IRCAM-Centre Pompidou. Bongers, B. (2006). Interactivation: Towards an e-cology of people, our technological environment, and the arts (No. 2006-12). Amsterdam: Proefschrift Amsterdam, VU. Bonnanni, L., Lieberman, J., Vaucelle, C., & Zucherman, O. (2006). TapTap: A haptic wearable for asynchronous distributed touch therapy. In Proceedings of the Conference on Human Factors in Computing Systems (CHI ’06; pp. 580–585). New York: ACM Press. Brave, S., & Dahley, A. (1997). inTouch: A medium for haptic interpersonal communication. In Proceedings of the Conference on Human Factors in Computing Systems (CHI’97, pp. 363–364). New York: ACM Press. Brooks, F., Ouh-Young, M., Batter, J. J., & Kilpatrick, P. J. (1990). Project GROPE—Haptic displays for scientific visualization. Computer Graphics, 24(4), 177–185. Brown, L. M., Brewster, S. A., & Purchase, H. C. (2006). Multidimensional tactons for non-visual information presentation in mobile devices. In Proceedings of 8th Conference on Human-Computer Interaction with Mobile Devices and Services (pp. 231–238). New York: ACM Press. Brown, L. M., & Kaaresoja, T. (2006). Feel who’s talking: Using tactons for mobile phone alerts. In Proceedings of CHI ’06 Extended Abstracts on Human Factors in Computing Systems (pp. 604–609). New York: ACM Press. Camarillo, D. B., Krummel, T. M., & Salisbury, J. K. (2004). Robotic technology in surgery: Past, present, and future. American Journal of Surgery, 188(4A), 2S–15S. Campbell, C. S., Zhai, S., May, K. W., & Maglo, P. (1999). What you feel must be what you see: Adding tactile feedback to the trackpoint. In Proceedings of Interact’99 (pp. 383–390). Amsterdam: IOS Press. Campion, G., & Hayward,V. (2005). Fundamental limits in the rendering of virtual haptic textures. In Proceedings of 1st Worldhaptics Conference (WHC ’05; pp. 263–270). Los Alamitos, CA: IEEE Press. Cascio, C. J., & Sathian, K. (2001). Temporal cues contribute to tactile perception of roughness. Journal of Neuro- science, 21(14), 5289–5296. Chan, A., MacLean, K. E., & McGrenere, J. (2005). Learning and identifying haptic icons under workload. In Proceedings of 1st Worldhaptics Conference (WHC ’05; pp. 432–439). Los Alamitos, CA: IEEE Press. Chan, A., MacLean, K. E., & McGrenere, J. (2008). Designing haptic icons to support collaborative turn-taking. International Journal of Human-Computer Studies, 66, 333–355. Chang, A., O’Modhrain, S., Jacob, R., Gunther, E., & Ishii, H. (2002). ComTouch: Design of a vibrotactile com- munication device. In Proceedings of Conference on Designing Interactive Systems (DIS’02; pp. 312–320). New York: ACM Press. Chen, D., & Vertegaal, R. (2004). Using mental load for managing interruptions in physiologically attentive user interfaces. In Proceedings of CHI ’04 Extended Abstracts on Human Factors in Computing Systems (pp. 1513–1516). New York: ACM Press. Choi, S., & Tan, H. Z. (2002). An analysis of perceptual instability during haptic texture rendering.In Proceed- ings of 10th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (pp. 129–136). Los Alamitos, CA: IEEE Computer Society. Choi, S., & Tan, H. Z. (2004). Toward realistic rendering of haptic textures. IEEE Computer Graphics and Appli- cations (Special issue—Haptic rendering beyond visual computing), 24(2), 40–47. Chouvardas, V. G., Miliou, A. N., & Hatalis, M. K. (2005). Tactile displays: A short overview and recent devel- opments. In Proceedings of the 5th International Conference on Technology and Automation (pp. 246–251). Thessaloniki, Greece: Alexander Technical Educational Institute (ATEI) of Thessaloniki. Haptic Interaction Design for Everyday Interfaces 185

Colgate, J. E., & Brown, J. M. (1994). Factors affecting the Z-width of a haptic display. In Proceedings of IEEE International Conference on Robotics & Automation (pp. 3205–3210). Los Alamitos, CA: IEEE Computer Society. Collier, G. (1985). Emotional expression: Mahwah, NJ: Erlbaum. Connor, C. E., & Johnson, K. O. (1992). Neural coding of tactile texture: Comparison of spatial and temporal mechanisms for roughness perception. Journal of Neuroscience, 12(9), 3414–3426. Conti, F., Morris, D., Barbagli, F., & Sewell, C. (2003). CHAI 3D: The open source haptics project. Retrieved October 27, 2008, from http://www.chai3d.org/index.html Cook, P. (2001). Principles for designing computer music controllers. In Proceedings of Conference on New Inter- faces for Musical Expression (pp. 1–4). New York: ACM Press. Craig, J. C. (1977). Vibrotactile pattern perception: Extraordinary observers. Science, 196(4288), 450–452. Craig, J. C. (1985). Tactile pattern perception and its perturbations. Journal of the Acoustical Society of America, 77(1), 238–246. Crusco, A. H., & Wetzel, C. G. (1984). The Midas touch: The effects of interpersonal touch on restaurant tip- ping. Personality & Social Psychology Bulletin, 10(4), 512–517. Delp, S., Loan, P., Basdogan, C., & Rosen, J. M. (1997). Surgical simulation: An emerging technology for emer- gency medical training. Presence: Teleoperators and Virtual Environments, 6(2), 147–159. Dennerlein, J., Martin, D., & Hasser, C. (2000). Force feedback improves performance for steering and combined steering-targeting tasks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’00; pp. 423–429). New York: ACM Press. DiFranco, D. E., Beauregard, G. L., & Srinivasan, M. A. (1997). The effect of auditory cues on the haptic percep- tion of stiffness in virtual environments. In Proceedings of the 6th Annual Symposium on Haptic Interfaces for Virtual Environments and Teleoperator Systems, ASME/IMECE (pp. 17–22). Atlanta: ASME Press. Donald, B. R., & Henle, F. (2000, April). Using haptic vector fields for animation motion control. In Proceedings of IEEE International Conference on Robotics and Automation (ICRA ’00; pp. 3435–3442). Los Alamitos, CA: IEEE Robotics and Automation Society. Driver, J., & Spence, C. J. (1994). Spatial synergies between auditory and visual attention. In C. Umiltà & M. Moscovitch (Eds.), Attention and performance XV: Conscious and nonconscious information processing (pp. 311–331). Cambridge: MIT Press. Driver, J., & Spence, C. (1998). Crossmodal attention. Current Opinion in Neurobiology, 8, 245–253. Driver, J., & Spence, C. (2004). Crossmodal spatial attention: Evidence from human performance. In C. Spence & J. Driver (Eds.), Crossmodal space and crossmodal attention (pp. 141–177). Oxford: Oxford University Press. Duncan, J., Martens, S., & Ward, R. (1997). Restricted attentional capacity within but not between sensory modalities. Nature, 387, 808–810. Enriquez, M., Afonin, O., Yager, B., & MacLean, K. (2001). A pneumatic tactile notification system for the driv- ing environment. In Proceedings of Workshop on Perceptive User Interfaces (PUI ‘01; pp. 1–7). New York: ACM Press. Enriquez, M., & MacLean, K. (2004, March). Impact of haptic warning signal reliability in a time- and-safety- critical task. In Proceedings of the 12th Annual Symposium on Haptic Interfaces for Virtual Environments and Teleoperator Systems, IEEE-VR2004 (pp. 407–415). Los Alamitos, CA: IEEE Computer Society. Enriquez, M., & MacLean, K. E. (2008a). Backward and common-onset masking of vibrotactile stimuli. Brain Research Bulletin (special issue on robotics and neuroscience), 75(6), 761–769. Enriquez, M., & MacLean, K. E. (2008b). The role of choice in longitudinal recall of meaningful tactile signals. In Proceedings of IEEE Haptic Symp (HAPTICS ‘08; pp. 49–56). Los Alamitos, CA: IEEE Computer Society. Enriquez, M., MacLean, K. E., & Chita, C. (2006). Haptic phonemes: Basic building blocks of haptic communi- cation. In Proceedings of 8th International Conference on Multimodal Interfaces (ICMI ’06; pp. 302–309). New York: ACM Press. Ernst, M. O., & Banks, M. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashion. Nature, 415, 429–433. Ernst, M. O., & Bülthoff, H. H. (2004). Merging the senses into a robust percept. Trends in Cognitive Sciences, 8(4), 162–169. Essick, G. K., James, A., & McGlone, F. P. (1999). Psychophysical assessment of the affective components of non- painful touch. NeuroReport, 10(10), 2083–2087. 186 Reviews of Human Factors and Ergonomics, Volume 4

Feygin, D., Keehner, M., & Tendick, F. (2002). Haptic guidance: Experimental evaluation of a haptic training method for a perceptual-motor skill. In Proceedings of 10th Symposium on Haptic Interfaces for Virtual Envi- ronment and Teleoperator Systems (pp. 40–47). Los Alamitos, CA: IEEE Computer Society. Field, T. (1999). Preschoolers in America are touched less and are more aggressive than preschoolers in France. Early Childhood Development and Care, 151, 11–17. Field, T. (2003). Touch. Cambridge: MIT Press. Fisher, J. D., Rytting, M., & Heslin, R. (1976). Hands touching hands: Affective and evaluative effects of an inter- personal touch. Sociometry, 39(4), 416–421. Fogarty, J., Hudson, S. E., Atkeson, C. G., Avrahami, D., Forlizzi, J., Kiesler, S., Lee, J. C., & Yang, J. (2005). Pre- dicting human interruptibility with sensors. ACM Transactions on Computer-Human Interaction, 12(1), 119–146. Fogg, B., Cutler, L. D., Arnold, P., & Eisbach, C. (1998). HandJive: A device for interpersonal haptic entertain- ment. In Proceedings of Conference on Human Factors in Computing Systems (pp. 57–64). New York: ACM Press. Forsyth, B., & MacLean, K. E. (2006). Predictive haptic guidance: Intelligent user assistance for the control of dynamic tasks. IEEE Transactions on Visualization and Computer Graphics, 12(1), 103–113. Foster, S. L. (1986). Reading dancing: Bodies and subjects in contemporary American dance. Berkeley: University of California Press. Gallace, A., Tan, H. Z., & Spence, C. (2006). Failure to detect tactile change: A tactile equivalent to the change blindness phenomenon. Psychonomic Bulletin and Review, 13(2), 300–303. Gallace, A., Tan, H. Z., & Spence, C. (2007). The body surface as a communication system: The state of the art after 50 years. Presence: Teleoperators & Virtual Environments, 16(6), 655–676. Gamzu, E., & Ahissar, E. (2001). Importance of temporal cues for tactile spatial-frequency discrimination. Journal of Neuroscience, 21(18), 7416–7427. Gaver, W. (1993). Synthesizing auditory icons. In Proceedings of Conference on Human Factors in Computing Systems (CHI’93; pp. 228–235). New York: ACM Press. Gescheider, G. (1985). Psychophysics: Method, theory and application (2nd ed.). Mahwah, NJ: Erlbaum. Gibson, J. J. (1962). Observations on active touch. Psychological Review, 69(6), 477–490. Gillespie, B. (1992). The Touchback keyboard. In Proceedings of the International Computer Music Conference (pp. 447–448). San Jose, CA: International Computer Music Association. Gillespie, B. (1999). Haptics in manipulation. In P. R. Cook (Ed.), Music, cognition and computerized sound: An introduction to psychoacoustics (pp. 229–245). Cambridge: MIT Press. Gillespie, B., & Cutkosky, M. (1996). Stable user-specific haptic rendering of the virtual wall. In Proceedings of ASME International Mechanical Engineering Conference and Exposition, Symposium on Haptic Interfaces (pp. 397–406). New York: ASME Press. Gillespie, B., & O’Modhrain, S. (1995). The Moose: A haptic user interface for blind persons with application to the digital sound studio (No. STAN-M-95). Stanford, CA: Stanford University Department of Music. Gillespie, B., O’Modhrain, S., Tang, C. P., & Zaretsky, D. (1998). The virtual teacher. In Proceedings of the ASME Dynamic Systems and Control Division (pp. 171–178). Atlanta: ASME Press. Goldberg, K., Chen, B., Solomon, R., Bui, S., Farzin, B., Heitler, J., Poon, D., & Smith, G. (2000). Collaborative teleoperation via the Internet. In Proceedings of IEEE International Conference on Robotics and Automation (ICRA; pp. 2019–2024). Los Alamitos, CA: IEEE Robotics and Automation Society. Goldstein, E. B. (1999). Sensation and perception (5th ed.). Belmont, CA: Wadsworth. Grant, A., Thiagarajah, M., & Sathian, K. (2000). Tactile perception in blind Braille readers: A psychophysical study of acuity and hyperacuity using gratings and dot patterns. Perceptual Psychophysics, 62(2), 301–312. Gray, R., & Tan, H. Z. (2002). Dynamic and predictive links between touch and vision. Experimental Brain Research, 145, 50–55. Griffin, W. B., Provancher, W. R., & Cutkosky, M. R. (2003). Feedback strategies for shared control in dexterous telemanipulation. In Proceedings of Intelligent Robots and Systems, 2003 (IROS 2003; vol. 2793, pp. 2791–2796). Los Alamitos, CA: IEEE Press. Griffiths, P., & Gillespie, R. B. (2004). Shared control between human and machine: Haptic display of automa- tion during manual control of vehicle heading. In Proceedings of the 12th International Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (HAPTICS’04; pp. 358–366). Los Alamitos, CA: IEEE Computer Society. Haptic Interaction Design for Everyday Interfaces 187

Grudin, J. (1988). Why CSCW applications fail: Problems in the design and evaluation of organization of orga- nizational interfaces. In Proceedings of the 1988 ACM Conference on Computer-Supported Cooperative Work (pp. 85–93). New York: ACM Press. Guéguen, N. (2004). Nonverbal encouragement of participation in a course: The effect of touching. Social Psy- chology of Education, 7(1), 89–98. Guest, S., & Spence, C. (2004). Tactile dominance in speeded discrimination of textures. Experimental Brain Research, 150(2), 201–207. Gumtau, S. (2005, March). Tactile semiotics: The meanings of touch explored with low-tech prototypes. In Pro- ceedings of First Worldhaptics Conference (WHC ’05; pp. 651–652). Los Alamitos, CA: IEEE Computer Society. Gumtau, S. (2006). Freshly squeezed touch into bits: Towards the development of a haptic design palette. Virtual Reality, 9(4), 250–259. Gunther, E., & O’Modhrain, S. (2003). Cutaneous grooves: Composing for the sense of touch. Journal of New Music Research, 32, 369–381. Haans, A., & IJsselsteijn, W. (2006). Mediated social touch: A review of current research and future directions. Virtual Reality, 9(2), 149–159. Hajian, A. Z., & Howe, R. D. (1997). Identification of the mechanical impedance at the human fingertip. ASME Journal of Biomechanical Engineering, 119(1), 109–114. Hamilton, R. H., & Pascual-Leone, A. (1998). Cortical plasticity associated with Braille learning. Trends in Cog- nitive Sciences, 2(5), 168–174. Hancock, P. A., & Szalma, J. L. (2003). The future of neuroergonomics. Theoretical Issues in Ergonomics Science, 4(1), 238–249. Hasser, C. J., & Cutkosky, M. R. (2002). System identification of the human hand grasping a haptic knob. In Proceedings of the 10th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (pp. 171–180). Los Alamitos, CA: IEEE. Haptik Library. (2005). Haptik Library, the open source library for haptics. Retrieved October 27, 2008, from http:// www.haptiklibrary.org/ Harrison, B. L., Fishkin, K. P., Gujar, A., Mochon, C., & Want, R. (1998). Squeeze me, hold me, tilt me! An explo- ration of manipulative user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Com- puting Systems (pp. 17–24). New York: ACM Press. Hayward, V. (2008). A brief taxonomy of tactile illusions and demonstrations that can be done in a hardware store. Brain Research Bulletin (special issue on robotics and neuroscience), 75(6), 742–752. Hayward, V., Astley, O. R., Cruz-Hernandez, M., Grant, D., & Robles-De-La-Torre, G. (2004). Haptic interfaces and devices. Sensor Review, 24(1), 16–29. Hayward, V., & MacLean, K. E. (2007). Do it yourself haptics—Part I. IEEE Robotics and Automation Society Magazine, 14(4), 88–104. Heller, M. A., Calcaterra, J., A., Green, S. L., & Brown, L. (1999). Intersensory conflict between vision and touch: The response modality dominates when precise, attention-riveting judgments are required. Perception & Psychophysics, 61, 1384–1398. Heller, M. A., & Schiff, W. (Eds.). (1991). The psychology of touch. Mahwah, NJ: Erlbaum. Hershberger, W. A., & Misceo, G. F. (1996). Touch dominates haptic estimates of discordant visual-haptic size. Perception & Psychophysics, 58, 1124–1132. Hollins, M., Bensmaïa, S., Karlof, K., & Young, F. (2000). Individual differences in perceptual space for tactile textures: Evidence from multidimensional scaling. Perception & Psychophysics, 62(8), 1534–1544. Hollins, M., Faldowski, R., Rao, S., & Young, F. (1993). Perceptual dimensions of tactile surface texture: A mul- tidimensional scaling analysis. Perception & Psychophysics, 54, 697–705. Hollins, M., & Risner, S. R. (2000). Evidence for the duplex theory of tactile texture perception. Perception & Psychophysics, 62(4), 695–705. Hornik, J. (1992). Tactile stimulation and consumer response. The Journal of Consumer Research, 19(3), 449–458. Horvitz, E., Kadie, C., Paek, T., & Hovel, D. (2003). Models of attention in computing and communication: From principles to applications. Communications of the ACM, 46(3), 52–59. Howe, R. D., & Matsuoka, Y. (1999). Robotics for surgery. Annual Review of Biomedical Engineering, 1, 211–240. Hughes, B. (2006). Haptic exploration and the perception of texture orientation. haptics-e, the Electronic Journal of Haptics Research, 4(2). 188 Reviews of Human Factors and Ergonomics, Volume 4

Immersion Corporation. (2000). The Immersion I-Feel and Feel-It Mouse, I-Force game controllers. Retrieved October 27, 2008, from http://www.immersion.com/ Immersion Corporation. (2001). Immersion Studio for gaming devices. Retrieved October 27, 2008, from http:// www.immersion.com/ Immersion Corporation. (2004). VibeTonz tactile feedback system. Retrieved October 27, 2008, from http://www. immersion.com/mobility Immersion Corporation. (2006). TouchSense® technology for touch screens. Retrieved October 27, 2008, from http:// www.immersion.com Jacobsen, S. C., Smith, F. M., Backman, D. K., & Iversen, E. K. (1991, February). High performance, high dex- terity, force reflective teloperator II. In Proceedings of ANS Topical Meeting on Robotics and Remote Systems, Albuquerque, NM. James, D., & Pai, D. (2001). A unified treatment of elastostatic contact simulation for real time haptics. haptics-e, the Electronic Journal of Haptics Research, 2(1). Jones, L. A., Nakamura, M., & Lockyer, B. (2004). Development of a tactile vest. In Proceedings of Haptic Inter- faces for Virtual Environment and Teleoperator Systems (HAPTICS ’04; pp. 82–89). Los Alamitos, CA: IEEE Computer Society. Jones, L., & Sarter, N. (2008). Tactile displays: Guidance for their design and application. Human Factors, 50, 90–111. Jordan, P. W. (2000). Designing pleasurable products: An introduction to the new human factors. London: Taylor & Francis. Kaaresoja, T., Brown, L. M., & Linjama, J. (2006). Snap-crackle-pop: Tactile feedback for mobile touch screens. In Proceedings of Eurohaptics (poster; pp. 565–566). Paris: Eurohaptics Society. Kandel, E. R., Schwartz, J. H., & Jessell, T. M. (Eds.). (2000). Principles of neural science (4th ed.). New York: McGraw-Hill. Katz, D. (1925/1989). The world of touch. Mahwah, NJ: Erlbaum. Kikuuwe, R., & Yoshikawa, T. (2001). Haptic display device with fingertip presser for motion/force teaching to human. In Proceedings of IEEE International Conference on Robotics and Automation (pp. 868–873). Los Alamitos, CA: IEEE Robotics and Automation Society. Klatzky, R. L., & Lederman, S. J. (1999). Tactile roughness perception with a rigid link interposed between skin and surface. Perception & Psychophysics, 61(4), 591–607. Klatzky, R. L., & Lederman, S. J. (2003). Touch. In A. F. Healy & R. W. Proctor (Eds.), Experimental psychology (Vol. 4, pp. 147–176). New York: Wiley. Klatzky, R. L., & Lederman, S. J. (2006). The perceived roughness of resistive virtual textures: I. Rendering by a force feedback mouse. ACM Transactions on Applied Perception, 3(1), 1–14. Klatzky, R., Lederman, S., Hamilton, C., Grindley, M., & Swendsen, R. (2003). Feeling textures through a probe: Effects of probe and surface geometry and exploratory factors. Perception & Psychophysics, 65(4), 613–631. Klatzky, R. L., Pai, D. K., & Krotkov, E. P. (2000). Perception of material from contact sounds. Presence: Tele- operators and Virtual Environments, 9(4), 399–410. Klemmer, S. R., Li, J., Lin, J., & Landay, J. A. (2004). Papier-Mâché: Toolkit support for tangible input. In Pro- ceedings of ACM Conference on Human Factors in Computing Systems (CHI ’04; pp. 399–406). New York: ACM Press. Knapp, M., & Hall, J. (2005). Nonverbal communication in human interaction (6th ed.). Florence, KY: Wadsworth Publishing. Knowles, W. B., & Sheridan, T. B. (1966). The “feel” of rotary controls: Friction and inertia. Human Factors, 8, 209–215. Kontarinis, D. A., & Howe, R. D. (1995). Tactile display of vibratory information in teleoperation and virtual environments. Presence: Teleoperators and Virtual Environments, 4, 387–402. Kragic, D., Marayong, P., Li, M., Okamura, A. M., & Hager, G. D. (2005). Human-machine collaborative systems for microsurgical applications. International Journal of Robotics Research, 24(9), 731–742. Krebs, H. I., Hogan, N., Aisen, M. L., & Volpe, B. T. (1998). Robot-aided neuro-rehabilitation. IEEE—Transactions on Rehabilitation Engineering, 6(1), 75–87. Kuchenbecker, K. J., Fiene, J., & Niemeyer, G. (2006). Improving contact realism through event-based haptic feed- back. IEEE Transactions on Visualization and Computer Graphics, 12(2), 219–230. Haptic Interaction Design for Everyday Interfaces 189

Lederman, S. J., & Klatzky, R. L. (1996). Action for perception: Manual exploratory movements for haptically processing objects and their features. In A. Wing, P. Haggard, & R. Flanagan (Eds.), Hand and brain: Neuro- physiology and psychology of hand (pp. 431–446). San Diego: Academic Press. Lederman, S. J., & Klatzky, R. L. (1997). Haptic aspects of motor control. In F. Boller & J. Grafman (Eds.), Hand- book of neuropsychology (vol. 11, pp. 131–148). New York: Elsevier. Lederman, S. J., & Klatzky, R. L. (2004). Multisensory texture perception. In B. E. Stein, C. Spence, & G. Calvert (Eds.), The handbook of multisensory processes (pp. 107–122). Cambridge: MIT Press. Leung, R., MacLean, K. E., Bertelsen, M. B., & Saubhasik, M. (2007). Evaluation of haptically augmented touch- screen GUI elements under cognitive load. In Proceedings of 9th International Conference on Multimodal Interfaces (ICMI ’07; pp. 374–381). New York: ACM Press. Levesque, V., Pasquero, J., & Hayward, V. (2007). Braille display by lateral skin deformation with the STReSS^2 tactile transducer. In Proceedings of World Haptics ’07 (Second Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems; pp. 115–120). Los Alamitos, CA: IEEE Computer Society. Linjama, J., & Kaaresoja, T. (2004). Novel, minimalist haptic gesture interaction for mobile devices. In Proceedings of the Third Nordic Conference on Human-Computer Interaction (pp. 457–458). New York: ACM Press. Linvill, J. G., & Bliss, J. C. (1966). A direct translation reading aid for the blind. In Proceedings of Institute of Electrical and Electronics Engineers (pp. 40–51). Los Alamitos, CA: IEEE Press. Luciani, A., Florens, J., & Castagne, N. (2005). From action to sound: A challenging perspective for haptics. In Proceedings of First Worldhaptics Conference (WHC ‘05; pp. 592–595). Los Alamitos, CA: IEEE Computer Society. Luk, J., Pasquero, J., Little, S., MacLean, K., Levesque, V., & Hayward, V. (2006). A role for haptics in mobile inter- action: Initial design using a handheld tactile display prototype. In Proceedings of ACM Conference. on Human Factors in Computing Systems (CHI ’06; pp. 171–180). New York: ACM Press. Macaluso, E., Frith, C., & Driver, J. (2002 ). Supramodal effects of covert spatial orienting triggered by visual or tactile events. Journal of Cognitive Neuroscience, 14, 389. MacKenzie, C., & Iberall, T. (1994). The grasping hand (vol. 104). Amsterdam: North-Holland. MacLean, K. E. (1996). The haptic camera: A technique for characterizing and playing back haptic environments. In Proceedings of the 5th Annual Symposium on Haptic Interfaces for Virtual Environments and Teleoperator Systems, ASME/IMECE (pp. 459–468). Atlant: ASME Press. MacLean, K. E. (2000). Designing with haptic feedback. In Proceedings of IEEE Robotics and Automation (ICRA ’00; pp. 783–788). Los Alamitos, CA: IEEE Robotics and Automation Society. MacLean, K. E. (2008). Foundations of transparency in tactile information design. IEEE Transactions on Haptics, 1(2), 84–95. MacLean, K. E., & Enriquez, M. (2003). Perceptual design of haptic icons. In Proceedings of EuroHaptics (pp. 351–363). Paris: Eurohaptics Society. MacLean, K. E., & Roderick, J. B. (1999). Smart tangible displays in the everyday world: A haptic door knob. In Proceedings of the IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM ’99; pp. 203–208). Atlant:a ASME Press. MacLean, K. E., Shaver, M. J., & Pai, D. K. (2002). Handheld haptics: A USB media controller with force sens- ing. In Proceedings of IEEE Haptic Symposium (HAPTICS ’02; pp. 311–318). Los Alamitos, CA: IEEE Computer Society. MacLean, K. E., & Snibbe, S. S. (1999). An architecture for haptic control of media. In Proceedings of the 8th Annual Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, ASME/IMECE (pp. 219–228). New York: ASME Press. Mai, N., Avarello, M., & Bolsinger, P. (1985). Maintenance of low isometric forces during prehensile grasping. Neuropsychologia, 23(6), 805–812. Mann, S. (2005). “flUId streams”: Fountains that are keyboards with nozzle spray as keys that give rich tactile feedback and are more expressive and more fun than plastic keys. In Proceedings of the 13th Annual ACM International Conference on Multimedia (MULTIMEDIA ’05; pp. 181–190). New York: ACM Press. Massie, T. H., & Salisbury, J. K. (1994). The PHANToM haptic interface: A device for probing virtual objects. In Proceedings of the Third Annual Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, ASME/IMECE (pp. 295–301). New York: ASME Press. 190 Reviews of Human Factors and Ergonomics, Volume 4

McCorkle, R., & Hollenbach, M. (1990). Touch and the acutely ill. In K. Barnard & B. T. Brazelton (Eds.), Touch—The foundation of experience. Madison, CT: International Universities Press. Menzies, H. (2005). No time: Stress and the crisis of modern life. Vancouver, Canada: Douglas & McIntyre. Michelitsch, G., Ruf, A., van Veen, H. A., & van Erp, J. B. (2002). Multi-finger haptic interaction within the MIAMM project. In Proceedings of EuroHaptics (pp. 193–205). Paris: Eurohaptics Society. Millar, S. (1999). Memory in touch. Psicothema, 11(4), 747–767. Miller, B. E., & Colgate, J. E. (1998). Using a wavelet network to characterize real environments for haptic dis- play. In Proceedings of the 7th Annual Symposium on Haptic Interfaces for Virtual Environment and Tele- operator Systems, ASME/IMECE (pp. 257–264). New York: ASME Press. Miller, T., & Zeleznik, R. (1998). An insidious haptic invasion: Adding force feedback to the X desktop. In Pro- ceedings of the 11th Annual ACM Symposium on User Interface Software and Technology (UIST ’98; pp. 59–64). New York: ACM Press. Minsky, M., Ouh-Young, M., Steele, O., Brooks, F. P., Jr., & Behensky, M. (1990). Feeling and seeing: Issues in force display. ACM Computer Graphics, 24(2), 235–242. Montagu, A. (1986). Touching: The human significance of the skin (3rd ed.). New York: Harper & Row. Montero, B. (2006). Proprioception as an aesthetic sense. Journal of Aesthetics & Art Criticism, 64(2), 231–242. Moran, T., & Dourish, P. (2001). Introduction to this special issue on context-aware computing. Human-Computer Interaction, 16(2), 87–95. Morris, D., Tan, H., Barbagli, F., Chang, T., & Salisbury, K. (2007, March). Haptic feedback enhances force skill learning. In Proceedings of 2nd Worldhaptics Conference (WHC ’07; pp. 21–26). Los Alamitos, CA: IEEE Com- puter Society. Mueller, F., Vetere, F., Gibbs, M., Kjedskov, J., Pedell, S., & Howard, S. (2005). Hug over a distance. In Proceedings of CHI’05 Extended Abstracts (pp. 1673–1676). New York: ACM Press. Napier, J. R. (1956). The prehensile movements of the human hand. Journal of Bone and Joint Surgery, 38(B), 902–913. Nesbitt, K. (2002). Experimenting with haptic attributes for display of abstract data. In Proceedings of Eurohap- tics 2002 (pp. 150–155). Paris: Eurohaptics Society. Newell, F. N., Bülthoff, H. H., & Ernst, M. O. (2003). Cross-modal perception of actively explored objects. In Proceedings of Eurohaptics Conference (pp. 291–299). Paris: Eurohaptics Society. Nichols, C. (2002). The vBow: Development of a virtual violin bow controller haptic human-computer inter- face. In Proceedings of the Conference on New Interfaces for Musical Expression (NIME ’02; pp. 1–4). Limerick, Ireland: University of Limerick. Norman, D. A. (2002). Emotion & design: Attractive things work better. Interactions, 9(4), 36–42. Norman, D. A. (2005). Emotional design: Why we love (or hate) everyday things. New York: Basic Books. Novint Technologies, Inc. (2006). Novint Falcon: 3D game controller. Retrieved October 27, 2008, from http:// home.novint.com/products/novint_falcon.php Nudehi, S. S., Mukherjee, R., & Ghodoussi, M. (2005). A shared-control approach to haptic interface design for minimally invasive telesurgical training. IEEE Transactions on Control Systems Technology, 13(4), 588–592. Oakley, I., Adams, A., Brewster, S. A., & Gray, P. D. (2002). Guidelines for the design of haptic widgets. In Proceedings of 16th British HCI Group Annual Conference (pp. 195–211). London: Springer-Verlag. Oakley, I., & O’Modhrain, S. (2005). Tilt to scroll: Evaluating a motion based vibrotactile mobile interface. In Proceedings of the First Worldhaptics Conference (WHC ’05; pp. 40–49). Los Alamitos, CA: IEEE Computer Society. O’Malley, M. K., Gupta, A., Gen, M., & Li, Y. (2006). Shared control in haptic systems for performance enhance- ment and training. Journal of Dynamic Systems, Measurement, and Control, 128(1), 75–85. O’Modhrain, S. (2004). Touch and go—Designing haptic feedback for a hand-held mobile device. BT Technol- ogy Journal, 22(4), 139–145. O’Modhrain, M. S. (2001). Playing by feel: Incorporating haptic feedback into computer-based musical instruments. Unpublished dissertation, Stanford University, Stanford, CA. O’Sullivan, D., & Igoe, T. (2004). Physical computing. Boston: Thomson Course Technology. Otaduy, M. A., & Lin, M. C. (2006). High fidelity haptic rendering. San Rafael, CA: Morgan & Claypool Pub- lishers. Oulasvirta, A., Tamminen, S., Roto, V., & Kuorelahti, J. (2005). Interaction in 4-second bursts: The fragmented Haptic Interaction Design for Everyday Interfaces 191

nature of attentional resources in mobile HCI. In Proceedings of ACM Conference on Human Factors in Com- puting Systems (CHI ’05; pp. 919–928). New York: ACM Press. Pantic, M., Pentland, A., Nijholt, A., & Huang, T. (2006). Human computing and machine understanding of human behavior: A survey. In Proceedings of the 8th International Conference on Multimodal Interfaces (ICMI ’06; pp. 239–248). New York: ACM Press. Pasquero, J. (2006). Survey on communication through touch (No. TR-CIM 06.04). Montreal, Canada: Center for Intelligent Machines, McGill University. Pasquero, J., Luk, J., Levesque, V., Wang, Q., Hayward, V., & MacLean, K. E. (2007). Haptically enabled handheld information display with distributed tactile transducer. IEEE Transactions on Multimedia, 9(4), 746–753. Pasquero, J., Luk, J., Little, S., & MacLean, K. (2006). Perceptual analysis of haptic icons: An investigation into the validity of cluster sorted MDS. In Proceedings of 14th IEEE Haptic Symposium (HAPTICS ’06; pp. 437–444). Los Alamitos, CA: IEEE Computer Society. Patten, C., Kircher, A., Ostlund, J., & Nilsson, L. (2004). Using mobile telephones: Cognitive workload and atten- tion resource allocation. Accident Analysis & Prevention, 36(3), 341–350. Payandeh, S. (2001). Application of shared control strategy in the design of a robotic device. In Proceedings of American Control Conference (pp. 4532–4536). Evanston, IL: American Automatic Control Council. Perricone, C. (2007 ). The place of touch in the arts. Journal of Aesthetic Education, 41, 90–104. Picard, R. W. (1997). Affective computing. Cambridge: MIT Press. Poupyrev, I., & Maruyama, S. (2003). Tactile interfaces for small touch screens. In Proceedings of 16th Annual ACM Symposium on User Interface Software and Technology (UIST’03; pp. 217–220). New York: ACM Press. Poupyrev, I., Maruyama, S., & Rekimoto, J. (2002). Ambient touch: Designing tactile interfaces for handheld devices. In Proceedings of 15th Annual ACM Symposium on User Interface Software and Technology (UIST’02; pp. 51–60). NewYork: ACM Press. Poupyrev, I., Okabe, M., & Maruyama, S. (2004). Haptic feedback for pen computing: Directions and strategies. In ACM Extended Abstracts of the ACM Conference on Human Factors in Computing Systems—CHI 2004 (pp. 1309–1312). New York: ACM Press. Ramos, G., & Balakrishnan, R. (2003). Fluid interaction techniques for the control and annotation of digital video. In Proceedings of the 16th Annual ACM Symposium on User Interface Software and Technology (UIST ’03; pp. 105–111). New York: ACM Press. Robles-De-La-Torre, G., & Hayward, V. (2001). Force can overcome object geometry in the perception of shape through active touch. Nature, 412, 445–448. Rock, I. (1984). Perception. New York: Scientific American Library. Rock, I., & Victor, J. (1964). Vision and touch: An experimentally created conflict between the two senses. Science, 143(3606), 594–596. Rosenberg, L. (1994). Virtual fixtures: Perceptual overlays enhance operator performance in telepresence tasks. Unpublished Ph.D. dissertation, Stanford University, Stanford, CA. Rovan, J., & Hayward, V. (2000). Typology of tactile sounds and their synthesis in gesture-driven computer music performance. In M. Wanderley & M. Battier (Eds.), Trends in gestural control of music (pp. 297–320). Paris: IRCAM. Rovers, A. F., & van Essen, H. A. (2004). HIM: A framework for haptic instant messaging. In Proceedings of ACM Conference on Human Factors in Computing Systems (CHI ’04; pp. 1313–1316). New York: ACM Press. Russell, J. A., & Weiss, A. (1989). Affect grid: A single-item scale of pleasure and arousal. Journal of Personality and Social Psychology, 57(3), 493–502. Salisbury, K., Barbagli, F., & Conti, F. (2004). Haptic rendering: Introductory concepts. IEEE Computer Graphics and Applications Magazine (special issue on haptic rendering), 24(2), 24–32. Sallnas, E., Rassmus-Grohn, K., & Sjostrom, C. (2000). Supporting presence in collaborative environments by haptic force feedback. Transactions on Computer-Human Interaction (TOCHI), 7(4), 461–476. Sarter, N. B. (2006). Multimodal information presentation: Design guidance and research challenges. Interna- tional Journal of Industrial Ergonomics, 36(5), 439–445. Satava, R. M. (1999). Emerging technologies for surgery in the 21st century. Archives of Surgery, 134(11), 1197–1202. Schmidt, A., Beigl, M., & Gellersen, H.-W. (1999). There is more to context than location. Computers & Graphics Journal, 23(6), 893–902. 192 Reviews of Human Factors and Ergonomics, Volume 4

Scholl, J., Hasvold, P., Henriksen, E., & Ellingsen, G. (2007). Managing communication availability and interrup- tions: A study of mobile communication in an oncology department. In Proceedings of Pervasive Computing (Vol. 4480, pp. 234–250). Toronto: Springer. Sensable Technologies Inc. (1996). PHANTOM desktop 3D touch system. Cambridge, MA. Sensable Technologies Inc. (2003). PHANTOM® Omni™ haptic device. Retrieved October 27, 2008, from http:// www.sensable.com/haptic-phantom-omni.htm Sensable Technologies Inc. (2004).Retrieved October 27, 2008, from http://www.sensable.com/products- openhaptics-toolkit.htm Shahrokni, A., Jenaro, J., Gustafsson, T., Vinnberg, A., Sandsjö, J., & Fjeld, M. (2006). One-dimensional force feed- back slider: Going from an analogue to a digital platform. In Proceedings of the 4th Nordic Conference on Human-Computer Interaction (NordiCHI ’06; pp. 453–456). New York: ACM Press. Shimoga, K. B. (1992). Finger force and touch feedback issues in dexterous telemanipulation. In Proceedings of Fourth Annual Conference on Intelligent Robotic Systems for Space Exploration (pp. 159–178). Los Alamitos, CA: IEEE Computer Society. Smith, A. M., & Scott, S. H. (1996). Subjective scaling of smooth surface friction. Journal of Neurophysiology, 75(5), 1957–1962. Smith, J., & MacLean, K. E. (2007). Communicating emotion through a haptic link: Design space and method- ology. International Journal of Human Computer Studies (special issue on affective evaluation—innovative approaches), 65(4), 376–387. Smyth, T. N., & Kirkpatrick, A. E. (2006). A new approach to haptic augmentation of the GUI. In Proceedings of the 8th International Conference on Multimodal Interfaces (pp. 372–379). New York: ACM Press. Snibbe, S. S., MacLean, K. E., Shaw, R., Roderick, J. B., Verplank, W., & Scheeff, M. (2001). Haptic metaphors for digital media. In Proceedings of ACM Symposium on User Interface Software & Technology (UIST ’01; pp. 199–208). New York: ACM Press. Spence, C., & Driver, J. (Eds.). (2004). Crossmodal space and crossmodal attention. Oxford: Oxford University Press. Spence, C., Nicholls, M. E. R., Gillespie, N., & Driver, J. (1998). Cross-modal links in exogenous covert spatial orienting between touch, audition, and vision. Perception & Psychophysics, 60(4), 544–557. Sproull, L., & Kiesler, S. (1991). Connections: New ways of working in the networked organization. Cambridge: MIT Press. Srinivasan, M. A., & Basdogan, C. (1997). Haptics in virtual environments: Taxonomy, research status, and chal- lenges. Computers and Graphics, 21(4), 393–404. Srinivasan, M. A., Beauregard, G. L., & Brock, D. (1996). The impact of visual information on the haptic per- ception of stiffness in virtual environments. In Proceedings of the 5th Annual Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, ASME/IMECE (pp. 555–559). New York: ASME Press. Srinivasan, M. A., & Chen, J. (1993). Human performance in controlling normal forces of contact with rigid objects. In Proceedings of ASME: Advances in Robotics, Mechatronics, and Haptic Interfaces (pp. 119–125). New York ASME Press. Stein, B. E. (1998). Neural mechanisms for synthesizing sensory information and producing adaptive behaviors. Experimental Brain Research, 123(1/2), 124–135. Stein, B. E., & Meredith, M. A. (1993). The merging of the senses. Cambridge: MIT Press. Strong, R., & Gaver, B. (1996). Feather, scent, and shaker: Supporting simple intimacy. In Proceedings of ACM Conference on Computer Supported Cooperative Work (CSCW ’96; pp. 29–30). New York: ACM Press. Swindells, C. (2007). Incorporating affect into the design of 1-D rotary physical controls. Unpublished Ph.D. dis- sertation, University of British Columbia, Vancouver. Swindells, C., & MacLean, K. E. (2007, March). Capturing the dynamics of mechanical knobs. In Proceedings of 2nd Worldhaptics Conference (WHC ’07; pp. 194–199). Los Alamitos, CA: IEEE Press. Swindells, C., MacLean, K. E., Booth, K. S., & Meitner, M. (2006). A case-study of affect measurement tools for physical user interface design. In Proceedings of Graphics Interface, Canadian Human-Computer Communica- tions Society (GI 2006). Wellesley, MA: A. K. Peters, Ltd. Swindells, C., MacLean, K. E., Booth, K. S., & Meitner, M. (2007). Exploring affective design for physical con- trols. In Proceedings of ACM Conference on Human Factors in Computing Systems (CHI ’07; pp. 933–942). New York: ACM Press. Tan, H. Z., Adelstein, B. D., Traylor, R., Kocsis, M., & Hirleman, E. D. (2006). Discrimination of real and virtual Haptic Interaction Design for Everyday Interfaces 193

high-definition textured surfaces. In Proceedings of Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (HAPTICS’06; p. 1). Los Alamitos, CA: IEEE Computer Society. Tan, H. Z., Gray, R., Young, J. J., & Traylor, R. (2003). A haptic back display for attentional and directional cue- ing. haptics-e: The Electronic Journal of Haptics Research, 3(1). Tan, H. Z., & Pentland, A. (2001). Tactual displays for sensory substitution and wearable computers. In W. Barfield & T. Caudell (Eds.), Fundamentals of wearable computers and augmented reality (pp. 579–598). Mahwah, NJ: Erlbaum. Tan, H. Z., Reed, C. M., Delhorne, L. A., Durlach, N. I., & Wan, N. (2003). Temporal masking of multidimen- sional tactual stimuli. Journal of the Acoustical Society of America, 114(6), 3295–3308. Tang, A., McLachlan, P., Lowe, K., Chalapati, R. S., & MacLean, K. E. (2005). Perceiving ordinal data hap- tically under workload. In Proceedings of 7th International Conference on Multimodal Interfaces (ICMI ’05; pp. 244–251). City and publisher to come at proof. Taylor, R., Jensen, P., Whitcomb, L., Barnes, A., Kumar, R., Stoianovici, D., Gupta, P., Wang, Z., Dejuan, E., & Kavoussi, L. (1999). A steady-hand robotic system for microsurgical augmentation. International Journal of Robotics Research, 18(12), 1201–1210. Teo, C. L., Burdet, E., & Lim, H. P. (2002). A robotic teacher of Chinese handwriting. In Proceedings of 10th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (pp. 335–341). Los Alamitos, CA: IEEE Computer Society. Ternes, D., & MacLean, K. E. (2008). Designing large sets of haptic icons with rhythm. In Proceedings of Euro- haptics (pp. 199–208). Paris: Eurohaptics Society. Tipper, C., & Kingstone, A. (2005). Is inhibition of return a reflexive effect? Cognition, 97(3), B55–B62. van Erp, J. B. F., & Spapé, M. M. A. (2003). Distilling the underlying dimensions of tactile melodies. In Proceed- ings of Eurohaptics (pp. 111–120). Paris: Eurohaptics Society. van Erp, J. B., van Veen, H. A., Jansen, C., & Dobbins, T. (2005). Waypoint navigation with a vibrotactile waist belt. ACM Transactions on Applied Perception, 2(2), 106–117. van Erp, J. B., & Spapé, M. M. A. (2003). Distilling the underlying dimensions of tactile melodies. In Proceedings of Eurohaptics (pp. 111–120). Paris: Eurohaptics Society. van Erp, J. B., & van Veen, H. A. (2003). A multi-purpose tactile vest for astronauts in the International Space Station. In Proceedings of Eurohaptics (pp. 405–408). Paris: Eurohaptics Society. Verillo, R. T. (1992). Vibration sensation in humans. Music Perception, 9(3), 281–302. Verplank, B., Gurevich, M., & Mathews, M. (2002). The PLANK: Designing a simple haptic controller. In Proceed- ings of the Conference on New Interfaces for Musical Expression (NIME ’02; pp. 1–4). Limerick, Ireland: University of Limerick Vozenilek, J., Huff, J. S., Reznek, M., & Gordon, J. A. (2004). See one, do one, teach one: Advanced technology in medical education. Academic Emergency Medicine, 11(11), 1149–1154. Wagner, C. R., Stylopoulos, N., Jackson, P. G., & Howe, R. D. (2007). The benefit of force feedback in surgery: Examination of blunt dissection. Presence: Teleoperators and Virtual Environments, 16(3), 252–262. Wang, Q., Levesque, V., Pasquero, J., & Hayward, V. (2006). A haptic memory game using the STRESS2 tactile display. CHI ‘06 Extended Abstracts on Human Factors in Computing Systems (pp. 271–274). New York: ACM Press. Warren, D. H., Welch, R. B., & McCarthy, T. J. (1981). The role of visual-auditory compellingness in the ventril- oquism effect: Implications for transitivity among the spatial senses. Perception and Psychophysics, 9, 557–564. Weiser, M. (1991). The computer for the 21st century. Scientific American, 265(3). 94–110. Welch, R. B., & Warren, D. H. (1986). Intersensory interactions. In K. R. Boff, L. Kaufman, & J. P. Thomas (Eds.), Sensory processes and perception (Vol. 1, chapter 25). New York: Wiley. Wickens, C. D. (1980). The structure of attentional resources. In R. S. Nickerson (Ed.), Attention and perform- ance (vol. 8, pp. 239–257). San Diego: Academic Press. Wickens, T. D. (2002). Elementary signal detection theory. Oxford: Oxford University Press. Williamson, J., Murray-Smith, R., & Hughes, S. (2007). Devices as interactive physical containers: The Shoogle system. In CHI ’07 Extended Abstracts on Human Factors in Computing Systems (pp. 2013–2018). New York: ACM Press. Winfield, L., Glassmire, J., Colgate, J. E., & Peshkin, M. (2007). T-PaD: Tactile pattern display through variable friction reduction. In Proceedings of 2nd World Haptics Conference (WHC ’07; pp. 421–426). Los Alamitos, CA: IEEE Computer Society. 194 Reviews of Human Factors and Ergonomics, Volume 4

Wu, W.-C., Basdogan, C., & Srinivasan, M. A. (1999). Visual, haptic and bimodal perception of size and stiffness in virtual environments. In Proceedings of the 8th Annual Symposium on Haptic Interfaces for Virtual Envi- ronment and Teleoperator Systems, ASME/IMECE (pp. 19–26). New York: ASME Press. Yanagida, Y., Kakita, M., Lindeman, R. W., Kume, Y., & Tetsutani, N. (2004). Vibrotactile letter reading using a low-resolution tactor array. In Proceedings of Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (HAPTICS ’04; pp. 400–406). Los Alamitos, CA: IEEE Computer Society. Yao, H.-Y., & Hayward, V. (2006). An experiment on length perception with a virtual rolling stone. In Proceed- ings of Eurohaptics (pp. 325–330). Paris: Eurohaptics Society. Yohanan, S., Chan, M., Hopkins, J., Sun, H., & MacLean, K. E. (2005). Hapticat: Exploration of affective touch. In Proceedings of 7th International Conference on Multimodal Interfaces (ICMI ’05; pp. 244–251). New York: ACM Press. Yohanan, S., & MacLean, K. E. (2008, April). The Haptic Creature project: Social human-robot interaction through affective touch. In Proceedings of the 2nd AISB Symposium on the Role of Virtual Creatures in a Com- puterised Society (pp. 7–11). Brighton, UK: Society for the Study of Artificial Intelligence and Simulation of Behaviour. Young, J. J., Tan, H. Z., & Gray, R. (2003). Validity of haptic cues and its effect on priming visual spatial atten- tion. In Proceedings of 11th International Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (pp. 166–170). Los Alamitos, CA: IEEE Computer Society. Yuille, A. L., & Bülthoff, H. H. (1996). Bayesian decision theory and psychophysics. In D. C. Knill & W. Richards (Eds.), Perception as Bayesian inference (pp. 123–161). Cambridge: Cambridge University Press. Zhao, P., & van de Panne, M. (2005). User interfaces for interactive control of physics-based 3D characters. In Proceedings of I3D: ACM SIGGRAPH 2005 Symposium on Interactive 3D Graphics and Games (pp. 87–94). New York: ACM Press.