sign in sign up
<<
Home , Mobile computing, Thad Starner, Wearable computer

THE CHALLENGES OF WEARABLE COMPUTING: PART 2

WEARABLE COMPUTING PURSUES AN INTERFACE IDEAL OF A CONTINUOUSLY

WORN, INTELLIGENT ASSISTANT THAT AUGMENTS MEMORY, INTELLECT,

CREATIVITY, COMMUNICATION, AND PHYSICAL SENSES AND ABILITIES.

MANY CHALLENGES AWAIT WEARABLE DESIGNERS. PART 2 BEGINS WITH THE

CHALLENGES OF NETWORK RESOURCES AND PRIVACY CONCERNS. THIS

SURVEY DESCRIBES THE POSSIBILITIES OFFERED BY WEARABLE AND,

IN DOING SO, DEMONSTRATES ATTRIBUTES UNIQUE TO THIS CLASS OF

COMPUTING.

Challenges throughput. Another serious issue is open The most immediately striking challenge standards to enable interoperability between in designing wearable is creating different services. For example, only one long- appropriate interfaces. However, the issues of range radio should be necessary to provide power use, heat dissipation, networking, and telephony, text messaging, Global Positioning privacy provide a necessary framework in (GPS) correction signals, and so on. which to discuss interface. Part 1 of this arti- For wearable computers, networking cle covers the first two of these issues; Part 2 involves communication off body to the fixed begins with the networking discussion. network, on body among devices, and near body with objects near the user. Each of these Networking three network types requires different design Georgia Institute of As with any , the decisions. Designers must also consider pos- amount of power and the type of services sible interference between the networks. Technology available can constrain networking. Wearable computers could conserve resources through Off-body communications. Wireless commu- improved coordination with the user inter- nication from mobile devices to fixed infra- face. For example, the speed at which a given structure is the most thoroughly researched of information packet is transferred can be bal- these issues. On the consumer side, analog cel- anced against latency, energy costs, and finan- lular phones and digital amateur-radio cial costs. Often, bits per second per watt is a repeaters provided the first glimpse of future more meaningful measure of a particular wire- problems; these systems would often drop less networking technology than maximum connections as the user moved. Communica-

54 0272-1732/01/$10.00  2001 IEEE tions researchers developed systems based on several standards—cellular digital packet data Cellular phones (CDPD), Global System for Mobile Com- Although not wearable computers, the original analog cellular phones provide a study in munications (GSM), time-division multiple- design constraints related to networking and interfaces of devices. Initially, one of the most access (TDMA), and code-division important features of these phones was a that resembled, as closely as possi- multiple-access (CDMA)—to help this prob- ble, a push button telephone. This feature provided a key improvement over the half-duplex, lem. Today, next-generation communications push-to-talk walkie-talkies of the time. systems (2. and ) will further improve As the technology matures, a variety of fundamentally different interfaces are becoming connection reliability and aggregate through- popular, such as instant messaging. Alternative networking paradigms will also appear because put. The “Cellular phones” sidebar discusses of the expense of deploying cellular towers, upgrading infrastructure continuously, and shrink- networking and interface issues applicable to ing available bandwidth. These new paradigms will take advantage of asynchronous com- wearable computers and their design. munication and other modes of human-to-human interaction, such as those used in Internet No matter what technologies finally dom- store-and-forward networks. Even today, cellular phones are becoming more like wearable com- inate, some challenges will remain. First, none puters. Higher-end microprocessors, multitasking operating systems, and bitmapped displays of the current networking systems will be now exist in cellular phones. Cellular phone manufacturers embed sensors in these devices ubiquitous. Users will always face situations to determine if the user is in a meeting or walking on a noisy street. The cellular phone uses in which mobile devices will not be in range this information to determine if the user is interruptible and the mode to use as an incoming- of a network cell. Although additional cell call alert. Indeed, researchers are developing some systems in which the cellular phone com- deployment and satellite use is slowly address- municates a worker’s current task to a remote expert who can provide advice. ing this problem, it will remain unprofitable to provide coverage for some areas. However, an interesting concept is to employ automo- bandwidth. When the user is working offline, biles as repeaters for the wearable user’s wire- the system employs this cache and updates any less data traffic.1 Although a wearable changes when network connectivity becomes has a relatively small battery and available.2 But what happens when a conflict antenna, cars can carry much larger equip- occurs? For example, suppose a businessman ment. In addition, drivers rarely stray further updates his calendar while disconnected, and, than a few miles from their automobiles. Fur- during the same time, his assistant also sched- thermore, even when driven into a remote ules an appointment. In practice, such conflicts location, a car is often within communication are rare, but this problem raises the issue of range of other cars. For example, imagine a where to locate a file’s correct, or master, copy. series of cars along rural highways acting as Most wireless mobile devices by nature repeaters to route wireless data to a local, fixed adhere to a thin-client approach to comput- networking center, such as a US post office. ing. In other words, the device provides just This wireless service does not need to be real enough processing power, user interface, and time to provide value; store-and-forward net- data storage to access services that are based working has been used successfully with the on a fixed-infrastructure located else- Internet for decades. where. Mobile devices without wireless access, For such a wireless participatory network- such as the original PalmPilot, provide asyn- ing scheme to succeed, developers must chronous services where docking the device address standardization, security, quality of ser- with the fixed network updates the file’s mas- vice, and synchronization. This style of ad hoc ter copy. However, the rapid increase in networking of mobile devices complicates tra- mobile mass-storage capacity, increasing at a ditional issues of resource discovery and rout- rate that surpasses that of Moore’s law, makes ing. Furthermore, mobile nodes traveling at a strong case for the mobile device maintain- variable speeds create difficulties for many ing the master copy. Today’s ruggedized, pock- wireless systems. Fortunately, this area of et-size hard drives can store 48 Gbytes. Soon, research has become very active in recent years. mobile users will maintain a terabyte on body, Another way to alleviate coverage issues is to making local storage space a nonissue. In fact, employ aggressive caching. By observing the users might prefer keeping all their data with wearer’s network use, the them, making the file data physically secure, can speculate about what the user will access accessible at any time, and always the author- next and cache material using spare network itative copy.

JULY–AUGUST 2001 55 WEARABLE COMPUTING

Caching, revision control, and intelligent will also need standards for resource discov- agents can emulate remote access to the user’s ery and node arbitration to enable communi- on-body personal directories when wireless cation. Solving this problem for wearables connectivity is unavailable. Although this strat- could prove easier than for ubiquitous com- egy could prove inconvenient for the occasional puting because a wearer will only occasional- third-party user, it gives the wearer the most ly add or remove a device from his body convenient access. For applications like calen- network. dars, the wearer knows her copy is definitive. Although on-body communication requires Retaining information locally could also significantly less energy than off-body net- help conserve battery power. In terms of works, energy use becomes critical because power, wireless network access requests are each device must have its own, relatively small generally more expensive than local access battery. This is a current challenge for the requests. Thus, if the user is responsible for the and IEEE 802.15 communities. majority of the data accesses, maintaining a Some experimental systems, such as BBN’s local copy will conserve power. Modeling user BodyLAN, require as little as 4-nJ/bit to trans- activities can also conserve power. Many wire- mit, while maintaining moderate bandwidth.4 less networks scale transmission power to the With such energy conservation, low-band- minimum necessary to maintain a connection. width, on-body sensors and interface devices Thus, transmitting a 10-Mbyte e-mail mes- could last for a year on one charge. As a side sage when the wearer is near a receiver could benefit, such low transmission power makes require significantly less power than when only US Federal Communications Commission a weaker signal is available. If the wearable approval significantly easier. computer understands a particular message’s Privacy and contention between different urgency level and can predict the wearer’s wearers’ body networks becomes an issue with future location, it could delay transmission on-body wireless networks. Zimmerman’s Per- until the wearer is closer to a receiver. In fact, sonal Area Network addresses this problem by with multiple wireless services to choose from exploiting near-field effects. In a sense, the and pricing schemes that depend on time of wearer’s body contains the transmissions, and transmission, the evaluation criteria of when devices must touch the user to receive a con- to transmit could extend to include financial nection.5 Post and Orth take a different tack cost as well as efficiency. Expected latency by researching clothing that has embedded could also help determine when and which electrical components.6 By careful use of tra- real-time or interactive services to use. ditional fabrics and connectors and the design of new conductive threads, these researchers Interoperability. Unfortunately, the hardware demonstrated functional, washable clothing needed to access more than one wireless ser- where the dressing process makes electrical vice burdens the user with extra equipment. connections. For example, snaps attached Software radios could improve this situation.3 with conductive thread can serve as connec- Many wireless-modem components can be tions among electronics in the shirt, belt, and emulated using digital signal processing. pants. Such experimentation has also led to Downloading the appropriate software to the new interfaces, including keyboards embroi- radio can change its communication standards dered onto the user’s jacket. and protocols (of course, wireless devices will still require certain analog components such as Communicating with near-body objects. Com- antennas). If this vision proves practical, wear- munication with near-body objects provides able computers will not only use different yet another set of challenges to wearable com- wireless networking services based on geo- puter design. Many consumer electronics graphical location, cost, and power, but could manufacturers are now proposing network- also replace certain common, portable con- ing standards based on mature radio and sumer electronics, such as GPS receivers, infrared transceivers. Most of these standards radios, and televisions. assume that the device has access to a signifi- As with mobile ad hoc networking, such cant energy supply. However, Hull, Neaves, wireless services between devices on the body and Bedford-Roberts propose embedding an

56 IEEE MICRO RFID tag reader in the user’s shirt cuff and passive RFID tags in devices with which the user wishes to communicate or track.7 When- ever the user reaches for a tagged device, ener- gy is transmitted wirelessly from the user’s shirt cuff to the device, which collects the power and responds with a few bytes of information. This information can be the device’s unique ID or, in the future, the cur- rent state of the device’s low-power sensors. The wearable computer can then use its high- Figure 1. The Locust infrared transponder er-power wireless connection to transmit these system mounted in an overhead fluorescent results to the network at large. Thus, a tagged light fixture. The solar cell on the right pro- device’s location and state is uploaded to the vides power. network each time it is moved by a user with a tag reader. This scheme realizes some of the advantages of ubiquitous sensors and net- nection. Although researchers have not yet working without the inconvenience and cost implemented a secure system, a user could of ubiquitous batteries. target information to one or several users by The Locust positioning system, shown in including a cryptographic key in an upload Figure 1, provides another example of an to a given Locust. In this way, the system cre- implementation of a power-restrictive net- ates a simple form of low-power, location- work scheme. While several technologies based networking. provide communication and location infor- mation outdoors, the Locust’s primary pur- Privacy pose is to provide location information inside a building. Each Locust consists of a micro- Those who design systems which handle controller, infrared transmitter, and infrared personal information therefore have a spe- receiver. To avoid the maintenance overhead cial duty: They must not design systems of batteries, each Locust is mounted near an which unnecessarily require, induce, per- overhead light and generates power with a suade, or coerce individuals into giving up small solar panel. By mounting these devices personal privacy in order to avail them- close to light sources, the infrared transmit- selves of the benefit of the system being ter/receiver pair receives a view of the work- designed.8 space in a room. The infrared transmitter repeats a unique ID every second with an off- These words, written by Leonard Foner, set determined by its ID to avoid repeated seem especially applicable to wearable com- collisions with other Locust. By listening for puters, which could become storehouses of these IDs and having a corresponding map users’ most intimate information. Indeed, of the area, wearable computers can deter- designers of early sys- mine their location. The wearable can then tems often cite privacy as one of the key user repeat this information to networks in the concerns in adopting their technology.9 User environment, depending on the wearer’s pri- privacy concerns are not equivalent to secu- vacy preferences. A user can upload small rity concerns. Security involves the protec- amounts of data to a particular Locust, and tion of information from unauthorized users; the Locust relays this data along with its privacy is the individual’s right to control the unique ID. Thus, a user can annotate a given collection and use of personal information.8 area with specific information for discovery When considering security and privacy, sys- by another, later user. Note that the infor- tem designers must consider what threats the mation can stand alone or act as a pointer to system might face, such as those posed by more data, such as sound files, animations, crackers, employees, employers, the courts, or programs stored on a traditional network and so on. For an example of an early wear- and accessed by a user’s real-time wireless con- able computer and how its design addressed

JULY–AUGUST 2001 57 WEARABLE COMPUTING

identify themselves to other employees and Thorp/Shannon wearable predictor security personnel, and to unlock various Probably the first electronic wearable computer mentioned in the literature is also one of doors. Each badge continually announces its the most carefully designed with regard to privacy and unobtrusiveness. Edward Thorp and presence to the environment through radio designed, built, and field-tested an analog wearable computer that yield- transmissions. Receivers in the environment ed an expected gain of 44 percent in the game of roulette by predicting the most favored report the badge’s location to a central system. octant for the ball to land. When combined with other sensors, this sys- Roulette is a game of chance, popular in many casinos. A small ball spins on a shallow tem can determine when an individual has sloped track mounted above a spinning wheel that has numbered pockets to receive the ball. entered a restricted space without a badge or Players bet on the pockets, trying to pick the one in which the ball will land. As the ball and appropriate authorization. Although this sys- wheel spin, gamblers continue placing bets until it appears that the ball will fall from the track tem could be a highly effective security sys- in the next two or three revolutions. tem, it raises several privacy concerns. The game was long considered unpredictable and became a challenge to the two Mass- A common user perception about such secu- achusetts Institute of Technology professors who were interested in and statis- rity systems is that employers could spy on tics. Thorp and Shannon determined that by timing the speed of the ball and the wheel and them or monitor activities such as time spent in calculating the differential equations that determined their movement, they could determine the restroom or the length of coffee breaks. In the ball’s path with fair accuracy. By 1961, Thorp and Shannon had created an analog wear- fact, employers could install additional, hidden able computer to aid a gambler’s bets. receivers to monitor employee actions covert- Because of the potential serious consequences of being discovered with an electronic ly. Even if a concerned badge wearer removes device at a gaming table in the early 1960s, unobtrusiveness and privacy were primary con- the badge for a given situation, the aggregate cerns. Significantly, not only was it necessary to keep private the information generated by information collected over several days or the wearable, but Thorpe and Shannon needed to hide the computer’s existence from onlook- months can still reveal behavior patterns. ers. Thus, power, networking, and interface considerations were subservient to minimizing Assuming that the user does not mind such the probability of detection. intrusions or is sufficiently recompensed for The system was divided into separate parts used by an observer and a bettor. The observ- them, the security measures a business uses to er wore the computer and timing system, while the bettor wore an earphone so he could protect its employee’s privacy is still an issue. As receive instructions regarding how to bet. To camouflage the system, the speaker on the bet- an extreme example, consider an employee tor’s system was placed in one ear canal, and the wires to it were colored to match skin and who has a former spouse stalking her. Does the hair. The observer timed the ball and rotor using microswitches in his shoes. Thus, both the business sufficiently protect data collected input and output devices were unobtrusive. The computer was placed in a cigarette-pack- from the security system so that the stalker can- age-size box; it remained on the observer. To minimize the risk of detection, an inductive not determine the employee’s work hours? wireless scheme connected the observer’s computer to the bettor’s receiver. Only surveil- Technically, it is possible to make active lance devices that were very close to the system could detect its communications. In design- badges secure. A badge system can use encryp- ing primarily for privacy, Thorp and Shannon purposely crippled their system; compared to a tion technology that only gives a master oper- general-purpose wearable computer; it was a single purpose machine with an awkward ator access to a given badge’s descrambled interface, limited power system, and rudimentary connectivity. The system was used suc- signature. However, this master operator cessfully in a Las Vegas casino in the mid-1960s, but the machine was fragile, preventing might be bribed or manipulated into unwit- large-scale winnings. Worried about getting caught, they delayed revealing their system tingly revealing critical information. until 1969, when they described it in a journal.1 Legislation could provide another privacy threat to employees. For example, the US Freedom of Information Act and similar state Reference laws might require releasing information that 1. E. Thorp and Anonymous, “Optimal Gambling Systems for Favorable Games,” a government employee mistakenly thought Rev. of the Int’l. Statistics Inst., vol. 37, no. 3, 1969. was private. In addition, court subpoenas might demand information recovered and stored by ubiquitous computing systems. the need for privacy, see the “Thorp/Shan- Already, US authorities have tapped automatic non wearable roulette predictor” sidebar. automobile toll-pass systems to help prove Depending on the perceived threat, securi- court cases. For the active-badge security sys- ty and privacy concerns can conflict. For tem, a simple solution is to erase or overwrite example, consider an active-badge system that records when they are no longer needed. Even a company deploys as a security measure. so, employees must still trust employers to be Employees must wear badges at all times to diligent in this regard.

58 IEEE MICRO Even if technology and policy address these physical range and network connectivity to concerns, active-badge systems are vulnerable that provided through the user’s wearable to yet another form of attack. Simply moni- computer, the wearable computer becomes a toring the amount of traffic from various natural control point for all user-related infor- badge-receiving stations provides data on a mation. In some cases, an electric field gen- person’s path through a building on how erated by the wearable computer could many people are in a given area. This traffic wirelessly power the sensors, following the analysis can be aggregated or combined with model of passive RFID tags. In this manner, other sources of information to reveal poten- sensors and objects “wake-up” as the user tially damaging data. passes through their environment. Without a An alternative to active badges is to design user with an appropriate power system near- systems in which the user solely controls the by, the sensors are unavailable. This method resultant information. In other words, the helps limit abuse of such systems by a remote user’s wearable computer would concentrate, third party. When such a scheme is not pos- process, and filter any data collected or dis- sible, the user could give explicit permission tributed about the user. In this way, the user for the sensing to occur by turning on the controls the degree of functionality and can sensor’s power. The sensor should turn off balance it against the amount of information automatically when it can no longer sense the revealed. user or the user’s network connection. The GPS and Locust (see Part 1) position Although not preventing potential abuses, systems demonstrate this design philosophy. such schemes help make abuse more incon- Both systems place transmission beacons in venient. the environment: low earth orbit for GPS and One major challenge to preserving privacy the local environment for Locust. By carrying in wearable computing is the dissemination a compatible receiver operating independently of information and ideas on potential abuses from the infrastructure, users can benefit from and protection schemes. Wearable comput- these systems without revealing their where- ing system designers should have a neutral abouts. However, certain functionality, such as forum to discuss techniques and introduce letting others inquire about a user’s location, standards. Or perhaps the community should would not be available without the user devise a privacy protection scheme designed retransmitting his position to the environ- with ratings displayed on various systems and mental infrastructure. components. For example, a wearable com- Although such a retransmission scheme puter component could protect privacy by might seem gratuitous, it provides certain ben- way of the following barriers: efits. The first is that the user has finer con- trol over what information is revealed. Even • Physical. In this approach, some mecha- if the user does not reveal any information, nism maintains a physical barrier the wearable computer knows his location— between data and potential abusers. The an important piece of contextual information. barriers could range from a system in Another benefit is that the worn component which users always carry crucial data on simply listens to beacon signals instead of con- their bodies to a methodology that tinuously broadcasting one itself, which gives secures data in a safe when not in use. security badges longer battery life. Also, users Other physical safeguards could include might more willingly accept technology in this shielding the wearable computer to min- form. By giving employees explicit control imize unintentional wireless emissions. over personal information, an employer shows • Technological. These approaches use secu- respect and confidence in employees’ use of rity methods such as encryption and bio- the technology. metric identifiers—fingerprints, iris In general, wearable computing could pro- scans, and so on—as barriers. vide users with a sense of control with respect • Legislative. Laws could specify conditions to privacy. This issue will gain in importance under which privacy is considered vio- as sensing systems become common through- lated. The law could tailor associated out homes and offices. By limiting a sensor’s penalties to particular technologies,

JULY–AUGUST 2001 59 WEARABLE COMPUTING

ensuring relevance and avoiding misin- Clothing, design, and fashion. Wearable com- terpretation in court. puting represents an unusual intersection of • Social. Wearable systems could use exist- , engineering, design, and fashion. The ing social conventions to build barriers. same basic computer components found in a For example, systems could store sensi- mainframe also comprise a wearable, but tive data in physical articles—such as design decisions for wearables must account data repositories resembling diaries or for the restrictions of portability and usabili- wallets—that a particular culture would ty over the need for speed and throughput. normally consider personal. In addition, there is a surprising social aspect • Obscuring. The wearable computer could to wearable computing. The design, tailoring, hide sensitive information in directories and expense of a wearable computer can reflect with large quantities of nonsensitive the user’s taste and importance, as does a information. Thus, a casual investigator banker’s choice of business suits. Natick Army could not look at all the files to determine Research Lab researchers have indicated to the which are the most revealing. author that in the armed forces, wearers’ per- ception of design and expense affects their Although some of these protections are very acceptance and opinion of the most basic sup- weak, different combinations allow a user to plies, such as mountaineering boots. specify various levels of inconvenience for a To make a computer wearable for an extend- would-be interloper. In some cases, the user ed period of time, designers must carefully might want the barriers to be breached given determine size, body placement,11 clothing enough persistence. For example, if the user has type,6 and wired interconnections to other a serious accident, the user might want col- components. This situation could reverse the leagues to discover his medical history, where- normal design process for computers; with as in normal circumstances he would consider wearables, developers might first decide on a this information private. Perhaps a medical form factor and then determine functionality information stored in the user’s and associated electronics. Such a philosophy wallet would provide an appropriate level of resulted in the PalmPilot when other stylus- protection against casual prying. Such a card based handheld devices were failing.12 Unlike would let bystanders help the user in an emer- much of and electrical engi- gency. However, if the main concern is exploita- neering, where ungainly prototypes might not tion by the user’s medical insurance company, affect an experiment’s quality, simple variations more levels of security would be necessary. in the form factor of a research-grade wearable With care, wearable designers can specify bar- computer could prove significant. rier combinations to adapt to the changing political, technical, and social climates in dif- interfaces: Making simple things sim- ferent markets and geographic areas. ple and complex things possible. To improve portability, for wearable computers Interface design tend to be small. For example, the Twiddler In the following discussion, the term interface one-handed keyboard can fit in a pocket, and is used as a generalization to refer to the numer- the flat panel display in a pair of MicroOpti- ous fields that address human and computer cal eyeglasses is about the size of a grain of rice. interaction. This includes, but isn’t limited to, Unfortunately, unlike computer processors, human-computer interfaces, psychophysics, there is a limit to how small these devices can human factors, ergonomics, industrial design, become. The resolution of the human eye lim- and fashion. Wearable computing interfaces its a usable display’s size and resolution. Simi- are a topic for an entire textbook; this article larly, the size of the user’s fingers limits the cannot begin to summarize the work in the placement and number of keys on a keyboard. field to date or even the areas yet to be explored Wearable peripheral designers must consider (for an overview, see the Proceedings of the IEEE trade-offs between usability, portability, and International Symposium on Wearable Comput- unobtrusiveness for every device they make. ers).10 Instead, this discussion is meant to stim- As previously discussed, one vision of wear- ulate curiosity in the field. able computers involves a body-centered wire-

60 IEEE MICRO less network through which the user can inte- grate peripherals simply by pocketing them. A day in the life Such a system eliminates redundancy in A fictitious police detective’s typical day demonstrates how wearable computers may portable consumer electronics; it could also benefit from specialization of peripherals: When documenting a crime scene, the detective enable interfaces that would otherwise prove removes a small camera and microphone from his car’s glove compartment and fastens them too costly to implement because of the repeat- to his lapel. The camera stores video on the detective’s pocket wearable. With a small key- ed design cost for each independent device. board attached to his belt, the detective privately adds text annotations to the video stream. More importantly, such a system structure At the office, the detective stashes the camera and keyboard in his coat pocket, and approach- encourages the design of specialized periph- es a large screen and interface area festooned with a variety of knobs normally associated erals for particular tasks. The “A day in the with video editing. The wearable interfaces with this equipment, and the detective spends life” sidebar presents a scenario that illustrates the next few hours splicing together segments of his field observations to make his report. this point. Given an insight by his editing, the detective decides to question a suspect. He holsters his Some wearable computer users might pre- gun, which can only be activated to fire when it is kept within two meters of his wearable. fer general-purpose equipment that allows The detective borrows a pair of sunglasses with a small display, camera, and microphone maximum flexibility and constant availabili- embedded in them. He also picks out a longer-range network transceiver so that the others ty. Unfortunately, such general-purpose sys- at the station can monitor his progress. As he questions the suspect, the detective compares tems provide a bit of a conundrum socially. the suspect’s answers to his report and images from the crime scene displayed on the sun- Onlookers do not know if the wearer is using glasses. As a result, he discovers an inconsistency in the suspect’s story and decides to arrest the system as a camcorder, a cellular phone, a him. Although not as convenient as the lapel camera to manipulate, the sunglasses’ camera text editor, or a game system. Thus, an performs well, inconspicuously documenting and reporting the suspect’s resistance to arrest. onlooker does not know if she can conve- Back at the police station, another police officer, who monitors all detectives’ progress via niently interrupt the user. Of even greater con- their long-range transmitters in the field, alerts nearby units to the situation. Back-up offi- cern to the onlooker is whether or not the cers respond to the location, and the suspect is arrested safely. wearer records their conversation covertly. The peripherals in this scenario were designed for specific tasks and form factors. Using While virtually undetectable video and audio each device was simple—any software or hardware reconfiguration on the wearable was recording equipment has been available for performed transparently when the user approached the peripheral. All the peripherals linked some time, as evidenced by investigative news into the detective’s personal machine, allowing maintenance of crucial information on the programs, the similarity of head-up displays detective’s body. The detective makes an active choice in what functionality he wishes to carry to camcorder eyepieces causes confusion with versus the inconvenience of added weight and bulk the peripherals might cause. onlookers. In the past, the form factor of portable devices helped constrain their per- ceived uses. For example, a microphone hang- by scientists who approached this problem in ing from a box at a reporter’s side designates some form, for example, Bush’s Memex, the box as an audio recording device. How- Wiener’s Cybernetics, Licklider’s Man-Machine ever, with wearable computers, the form fac- Symbiosis, and Englebart’s implementation of tor and uses are not yet commonly known. NLS/AUGMENT, to name a few. Providing some external cue about the tasks In the pursuit of such an interface, wear- being performed by the wearer from minute able computing provides a set of advantages to minute could prove important as wearable not available before. Wearable computers are computers become more widespread. physically close to the user, highly portable, quickly accessed, and designed to consume a Intellectual tools fraction of the user’s full attention. PDAs, the One of the early applications of computers commercial devices most similar to a wear- was to calculate ballistic trajectories, a task for able, provide a contrast to these traits. Users which the human mind is not well suited. In often store PDAs in a pocket or carrying case. artificial intelligence, research efforts try to cre- So, although PDAs are physically close, it can ate machines that perform tasks the human take users a significant amount of time to mind and body do perform well. An interest- access the interface. In addition, the use of a ing challenge that serves as a compromise is to stylus-based PDA requires both hands, and create systems that augment a user’s natural abil- the interface requires most of the user’s visu- ities through computational components. This al attention. Thus, it is difficult to use PDAs idea is not new; the history of computing is while, for example, walking down the street, filled with systems and philosophies described repairing an automobile, or even having a

JULY–AUGUST 2001 61 WEARABLE COMPUTING

conversation. However, a wearable equipped user and a set of goals, context is defined as with a display built into eyeglasses and a one- those environmental features not created handed or speech-driven interface lets the user explicitly for input to the system.14 Lamming concentrate on a primary task while the wear- showed how context could be effectively used able provides information support. For exam- with Xerox’s PARCTabs.15 His Forget-me-not ple, in repairing a car, a mechanic only has to system sensed particular office activities, such make a small eye movement to refer to the as personal location, encounters with others, car’s technical manual in a head-up display. activities, telephone calls, file User attention is the scarcest resource for exchanges, and printing. It demonstrated the wearable computing. In particular, hand-eye use of complex queries to augment memory. coordination is at a premium as the user lives For example, suppose the user remembers that in the physical world while accessing virtual he discussed a business plan with a colleague information. Interfaces should provide the a week ago and was interrupted by a telephone most support for the smallest investment of call. However, he does not remember who attention diverted from the user’s primary called or why it was important enough to task. This idea is at odds with the window-, interrupt the conversation. Forget-me-not lets icon-, menu-, and pointer-driven interfaces the user search these events from the past and, generally found on desktops; desktop appli- from the context of the interruption, provide cations often assume that the application itself the user with the name of the person who is the primary focus of the user’s attention. called and a record of the user’s actions after- Hence, wearable computer systems might not ward, which might illuminate the importance use traditional GUIs for many tasks. of the call. More sophisticated on-body perception sys- Note taking and immediacy of interface. A tems can capture a more complete sense of wearable equipped with a head-up display and context. Through sensors placed near where one-handed keyboard allows rapid note taking the user’s natural senses are located, the wear- in most situation. In some cases, the wearable able receives a first-person view of the wear- provides a less obtrusive and more efficient er’s interactions with others and the world. method of recording information than is pos- Recently, researchers have sible with traditional means, such as pen and begun experimenting with wearable comput- paper. For example, in the author’s case, typ- ers. Clarkson uses on ing on a one-handed keyboard takes half as wearable-based audio and low-resolution much time as writing by hand. video to identify interesting events during the Wearable computer users who use their user’s everyday life.16 These characterizations machines during conversation tend to opti- can act as additional features when looking mize their systems such that they can begin for particular pieces of information. Although taking notes within two seconds of realizing not wearable, Moore has demonstrated a the need. Given this level of speed and access, vision system that identifies objects, such as a everyday users generally take a huge variety of book or keyboard, by tracking how the user notes, ranging from how to fix a given piece of manipulates these objects.17 Schiele and Starn- equipment to what they need to do in the next er have developed wearable gesture and object 10 minutes. Researchers have noted infor- recognizers.18 Although these are research sys- mally that in some domains, access to the tems, they demonstrate how a wearable com- interface in under two seconds results in sig- puter’s first-person perspective could capture nificantly more use than systems that require the day-to-day experiences of wearers without a longer delay.13 This “two second rule” pro- burdening them with having to specify explic- vides an initial heuristic for defining the itly what to capture. acceptable delay in accessing the interface for wearable systems. Just-in-time information. Capturing informa- tion does not have much meaning unless it is Perception and context. A wearable can also indexed and retrievable in a timely fashion. A retrieve the context in which notes were taken; major question in the wearable computing such context is useful for indexing. Given a community is how to present information to

62 IEEE MICRO Figure 2. Remembrance Agent. Every 10 seconds, the current text in the top buffer forms an automatic query that returns the one-line summaries of potentially interesting documents in the bottom buffer. a wearer. Researchers at Carnegie Mellon Uni- “software that proactively retrieves and pre- versity have reported several successes using sents information based on a person’s local wearable computers for inspection, mainte- context in an accessible yet nonintrusive man- nance, and repair, especially in industrial and ner.”14 Creating these mobile, nonintrusive military domains.19 However, in her doctoral computer interfaces is a distinct challenge and thesis at the Georgia Institute of Technology, research focus. Ockerman reports that her wearable task guid- A particular just-in-time ance system inhibits experienced pilots’ per- agent of interest is the Remembrance Agent.14 formance in inspecting their aircraft.20 The Although information retrieval systems exist for apparent discrepancy is more than likely due to many applications, almost all of these systems the users’ relative experience levels with the concentrate on written text and query-based task and the authority they assign to the wear- information retrieval on demand. For example, able computer. When given a checklist, Ock- they can answer questions such as “When is that erman’s experienced users tend to over rely on conference’s paper deadline?” or “Who’s an it. To help prevent this problem, the wearable expert on this particular ?” However, interface should encourage the user to exercise they do not help the user remember to ask a judgment and physically touch the equipment. question or what question to ask. The Remem- Adapting the user interface based on active brance Agent addresses these problems. Its asso- sensing of the inspected object might alleviate ciative forms of recall might remind a user that this problem. Providing overviews of the task’s an important conference exists, or that there are purpose and each step’s context in the check- references to a particular algorithm the user list has been shown to improve performance. might have missed. In short, with experienced users, wearable The Remembrance Agent, shown in computers should provide formal structure yet Figure 2, performs this task by continuously encourage independent thought and adapta- displaying relevant information to an indi- tion of the interface to the situation. vidual user in his current context. In its cur- While Ockerman’s task guidance systems rent form, the Remembrance Agent software expect explicit user interaction, such as when uses text that a user is reading or writing as a the wearer notes an observation or signals that query into its relevance engine. At the bottom a subtask is complete, another style of inter- of the user’s screen, one-line summaries of face suggests pieces of information based on potentially useful documents (e-mails, papers, current context. Rhodes describes these as books, notes, and so on) appear as a list in just-in-time information retrieval agents or order of potential relevance. If a document is

JULY–AUGUST 2001 63 WEARABLE COMPUTING

indeed interesting, this summary alone may ters between wearers with selfish, and possibly be a sufficient reminder for the user of the conflicting, goals. For example, if two package document’s existence. Alternatively, if the delivery drivers encounter each other at a drop- summary intrigues the user, he can view a full off point, their wearables can compare delivery annotation with a simple sequence of key- schedules and determine if, by exchanging strokes. packages, the drivers can minimize their routes. Fickas et al. simulated such negotia- tions in large-scale wearable communities and overlays explored the role of deception as well as meth- ods for building such communities.23 information on the physical Tailoring augmented-reality systems world. Many systems presented in this discussion involve information interfaces that are portable and personal, and demand as little user attention as necessary. In some forms of Although early wearable prototypes of the augmented reality, however, the information Remembrance Agent attempted to couple has immediate bearing on a physical object audio and visual perception to information and its properties. In such cases, coupling the retrieval, the resulting systems were not prac- virtual interface to the hand-eye coordination tical for everyday use.18 However, recent and visual attention needed to interact with advances in desktop-based systems that rec- the physical object is appropriate and even ognize gesture, faces, and speech indicate that desirable. Examples of this are augmented these types of mobile systems could become realities that provide sensory enhancement or feasible in the near future. that help compensate for physical handicaps. As stated earlier, augmented reality overlays Facilitating collaboration information on the physical world. For exam- Many occupations are inherently mobile, ple, a doctor could wear a head-up display to with spontaneous meetings often happening see the results of an X-ray overlaid on an actu- away from any desktop. In such cases, wear- al patient. In such situations, the information able computers might aid communication and provided by the head-up display, in this case, collaboration. Kraut et al. found that remote the skeleton, is not incongruous with the assistance significantly improves task perfor- patient’s body presented to the doctor’s unaid- mance in wearable applications.21 In this study, ed eyes. In fact, the head-up display needs to a technician wore a head-mounted camera and accurately track the patient with low latency display combination to enable a remote expert to maintain the doctor’s illusion of seeing to see the technician’s workplace and to dis- through the patient. This ’s accu- play appropriate manual pages. Kortuem iden- racy and stability is especially important when tified several collaborative primitive functions performing surgery. During surgery, updat- that wearable computers can exhibit, including ing the visualization in a timely manner is also remote awareness, presence, presentation, important. Thus, coupling the visualization pointing, and manipulation. Kortuem exper- to the surgeon’s actions is beneficial. imented with body-stabilized spatial informa- On the other end of the spectrum, some aug- tion displays to support 3D collaboration. mented realities might need only the loosest Although the state of the current hardware coupling. For example, augmented realities often interferes with exploring these princi- might trigger audio events based on which room ples, an intuitive collaborative interface that the wearer visits or even go so far as to commu- does not overwhelm the wearer’s attention nicate to the wearer primarily through ambient seems feasible.22 interfaces. Between these two extremes lies a An intriguing collaboration idea exploits continuum of interfaces for further research. communities of wearable agents acting on As mentioned earlier, one particularly behalf of their owners.23 These agents negoti- intriguing idea in augmented reality is to ate for cooperation during physical encoun- extend the Web to physical reality. Figure 3

64 IEEE MICRO (a) (b) (c)

Figure 3. Augmenting reality with hypertext links. When a computer vision system first locates a tag, it renders an arrow on top of the live video in the wearer’s head-up display; the arrow indicates a hyperlink (a). If the user shows interest by staring at the object, the system displays appropriate text labels (b). If the user approaches the object, the system shows movie sequences or 3D graphics (c). demonstrates an early implementation of such ger them creates a form of just-in-time infor- a system using computer vision and head-up mation retrieval agents. Augmented reality displays.18 The user wears a combination could also assist synchronous collaboration head-up display and camera. Computer vision between Web users by enabling shared visu- continuously search the video alization of file systems, design tools, and images for unique tags that indicate an object information searches. has information associated with it. Comput- er graphics then overlay the user’s visual field Hardware and software engineering to communicate this information. In the sys- In creating wearable systems, many trade- tem shown in Figure 3, when the system first offs occur with respect to the challenges dis- locates a tag, an arrow is rendered to indicate cussed in this article. Balancing these a hyperlink. If the user shows interest, the sys- characteristics during design, runtime, and tem displays the appropriate text labels. Fig- maintenance is the domain of software and ure 3b shows the overlying text rotating in hardware engineering. Academic and indus- conjunction with the tag, demonstrating that trial groups are beginning to organize collab- the system can recover rotation with respect orations and workshops to explore what to the user’s head. If the user approaches the software and hardware engineering mean in object, the system displays 3D graphics or the wearable computing domain, but much movie sequences, as shown in Figure 3c. work remains. In the past decade, several researchers have Most general-purpose wearable computers begun addressing how to improve the sensing described in the literature do not address the and human-computer interface problems of complicated design choices implicit in such augmented-reality systems. The problems systems. Devices like those discussed in the associated with a “real ” are pacemakers (see Part 1) and cellular phones an area of active research, but a true field test sidebars of this article, use special-purpose cannot occur until the requisite equipment wearable computers and related infrastructure becomes more accessible and a sufficiently hardware. Each design prioritizes one attribute large population of users is supportable. significantly more than others. Although In many senses, this augmented-reality progress in hardware will enable more func- extension of the Web combines many prop- tionality in a wearable computer, the balance erties of the intellectual tools described earli- between privacy, power, networking, and er. A wearer can attach annotations to physical interface will be continually revisited with objects, thus performing a form of context- each new development in the field. dependent note taking. Letting others access these notes achieves a form of asynchronous earable computing pursues an interface collaboration. Indeed, coupling notes to a Wideal in which the computer persists physical context or to a set of actions that trig- and provides constant access to information

JULY–AUGUST 2001 65 WEARABLE COMPUTING

3. V. Bose, “The Impact of Software Radio on Further resources Wireless Networking,” and Comm. Rev., vol. 3, no. 1, Jan. 1999, •http://www.cc.gatech.edu/ccg pp. 30-37. •http://www.media.mit.edu/wearables 4. P. Carvey, “Technology for the Wireless •http://www.charmed.com Interconnection of Wearable Personal Elec- •For a complete bibliography for this article, visit http://computer.org/micro. tronic Accessories,” VLSI Signal Processing IX, IEEE Press, Piscataway, N.J., 1996, pp. services, senses and models context, augments 13-22. and mediates the user’s interactions with the 5. T. Zimmerman, Personal Area Networks environment, and interacts seamlessly with the (PAN): Near-Field Intra-Body Communica- user. Much work remains. Perception on the tion, master’s thesis, Media Laboratory, body is a relatively new endeavor since appro- Massachusetts Inst. Technology, Cam- priate sensors are just now becoming available. bridge, Mass., 1995. While much study has centered on low-atten- 6. E. Post et al., “E-broidery: Design and Fabri- tion interfaces for automobiles and aircraft, lit- cation of Textile-Based Computing,” IBM tle has been done for users of personal head-up Systems J., vol. 39, no. 3, 2000, pp. 840-850. displays. Most ambitiously, wearable comput- 7. R. Hull, P. Neaves, and J. Bedford-Roberts, ing will enable and encourage development of “Towards Situated Computing,” Proc. Intl. intelligent agents that model the user’s minute- Symp. Wearable Computers, IEEE CS Press, by-minute behavior in an effort to predict Los Alamitos, Calif., 1997, pp. 146-153. future needs and goals. In the words of Lick- 8. L. Foner, Political Artifacts and Personal Pri- lider, if even partly successful, “the resulting vacy: The Yenta Multi-Agent Distributed partnership will think as no human brain has Matchmaking System, doctoral dissertation, ever thought.” MICRO Media Laboratory, Massachusetts Inst. Technology, 1999. Acknowledgments 9. R. Want et al. “An Overview of the PARCTab Thanks to the Spring 2001 Mobile and Ubiquitous Computing Experiment, IEEE Ubiquitous Computing class at the Georgia Personal Communications, vol. 2, no. 6, Dec. Institute of Technology, which field-tested 1995, pp. 28-33. many of this article’s examples and its more 10. Proc. Int’l Symp. Wearable Computers esoteric ideas. Thanks are also due to the (ISWC), IEEE CS Press, Los Alamitos, Calif., Georgia Institute of Technology Contextual 1997-2001. Computing Group, the Massachusetts Insti- 11. F. Gemperle et al., “Design for Wearability,” tute of Technology Wearable Computing Pro- Int’l. Symp. Wearable Computers, IEEE CS ject, and the members of wearables list. In Press, Los Alamitos, Calif., 1998, pp. 116-122. particular, Brad Rhodes, Lenny Foner, Rob 12. P. Dillon, “The Next Small Thing,” Fast Com- Melby, Kent Lyons, Dan Ashbrook, John Nis- pany, vol. 15, June 1998, pp. 93-113. sen, and Chip Maguire contributed their time 13. B. Shneiderman, Designing the User Inter- to help guide structure and provide details and face, 3rd ed., Addison-Wesley, Reading, references. Mass.,1997, pp. 358-362. This article is dedicated to the memory of 14. B. Rhodes, Just-In-Time Information Claude Shannon, hacker par excellence. Retrieval, doctoral dissertation, Media Lab- oratory, Massachusetts Inst. Technology, References Cambridge, Mass., 2000. 1. T. Kanter, G. Maguire, and T. Smith, 15. M. Lamming and M. Flynn, Forget-Me-Not: “Rethinking Wireless Internet with Smart Intimate Computing in Support of Human Media,” Proc. Nordic Radio Symp. (NRS 01), Memory, tech. report RXRC TR 94-103, Nordic Radio Soc., Sweden, 2001. Rank Xerox Research Center, Cambridge, 2. M. Satyanarayanan, “Fundamental Chal- UK, 1993. lenges in Mobile Computing,” Proc. Symp. 16. B. Clarkson and A. Pentland, “Recognizing Principles of , ACM User’s Context from Wearable Sensors: Press, New York, 1996. Baseline System,” tech.report TR-519,

66 IEEE MICRO Media Laboratory, Massachusetts Inst. Technology, Cambridge, Mass., 2000. 17. D. Moore, Vision-Based Recognition of Actions Using Context, doctoral dissertation, Dept. Electrical Engineering, , Next- Atlanta, 2000. 18. T. Starner and et al., “Augmented Reality Through Wearable Computing,” Presence, generation vol. 6, no. 4, Winter, 1997, pp. 386-398. 19. A. Smailagic and D. Siewiorek, “The CMU Mobile Computers: A New Generation of Computer Systems,” Proc. IEEE Int’l. Com- courses puter Conf. (COMPCON), IEEE CS Press, Los Alamitos, Calif., 1994, pp. 467-473. 20. J. Ockerman, Preventing Operator Over- for the Reliance on Task Guidance Systems, doctor- al dissertation, Dept. Industrial and Systems next Engineering, Georgia Tech, Atlanta, 2000. 21. R.E. Kraut, M.D. Miller, and J. Siegel, “Col- generation laboration in Performance of Physical Tasks: Effects on Outcomes and Communication.” Proc. ACM Conference on Computer Sup- of computer ported Cooperative Work, ACM Press, New York, 1996, pp. 57-66. professionals 22. M. Billinghurst et al., “A Wearable Spatial Conferencing Space,” Proc. Intl. Symp. Wearable Computers, IEEE CS Press, Los Alamitos, Calif., 1998, pp. 76-83. Influence what our 23. S. Fickas et al., “When Cyborgs Meet: Build- ing Communities of Cooperating Wearable Agents,” Proc. Intl. Symp. Wearable Com- students learn. Review the puters, IEEE CS Press, Los Alamitos, Calif., 1999, 124-132. latest draft of Computing Curricula 2001. Thad Starner is an assistant professor at the Georgia Institute of Technology. His research http://computer.org/education/curricula2001 interests include intelligent agents, wearable computing, computer vision, and . Starner has a PhD from the Massachusetts Institute of Technology Media Prepared by the Laboratory and has been wearing general-pur- IEEE Computer Society/ACM joint task force pose computers as part of his daily life for eight years. He is a member of the IEEE Com- on Computing Curricula 2001 puter Society, the ACM, and the AAAS.

Direct questions and comments about this article to Thad Starner, Georgia Institute of Technology, College of Computing, 801 Atlantic Dr., Atlanta, GA 30332-0280; [email protected].

JULY–AUGUST 2001 67