Digitally Enhanced Reality

Designing Wearable Personal Assistants for Surgeons: An Egocentric Approach

A wearable personal assistant prototype based on facilitates touchless interaction with x-ray images, allows clinicians to browse electronic patient records (EPRs) while on the move, and supports synchronized ad hoc remote collaboration.

ncreasingly powerful wearable com- cal setup in the way evolution designed it to puters suggest that a tight integration interpret and act in the world. Our focus here between human and computer is achiev- is on human perception, cognition, and action. able. However, to reach a deep integra- How we think we interpret the world around tion that offers timely cognitive support, us in everyday life is not how we (our brains) Iwe need to better understand how humans actually do it. In the last few decades, research perceive, think, and act in the world. An ideal in cognitive science, perception psychology, and wearable intelligent assistant “augments mem- neuropsychology has resulted in some remark- ory, intellect, creativity, communication, and able findings, some of which are still debated: physical senses and abilities.”1 We prefer the term “wearable • About 95 percent of measurable brain activity 2 Shahram Jalaliniya and personal assistant” (WPA) to is unconscious. emphasize the tight integration • The 5 percent of human conscious cognitive Thomas Pederson between a single mind, body, processing (attention) is volatile and easily in- IT University of Copenhagen and computer. terrupted by internal unconscious processes Human body-and-mind- or external stimuli. centric design approaches can • Human attention does not multitask.3 complement existing technology-driven efforts • By the time external stimuli grasps our at- (that is, efforts based on available state-of-the- tention (if it does), it has already undergone art hardware) in addressing many of the chal- significant filtering and transformation by un- lenges of human-computer systems because, conscious processes.4 ultimately, the power of these systems depends • Human routine actions are often initiated and on the level of integration.1 Although we can controlled by unconscious cognitive processes adapt and modify the artificial cognitive archi- triggered by direct external stimuli, leaving tecture (the “computer” system), we cannot conscious processes out of the loop. change the body and brain of human agents. As system designers,­ we can only ensure that As developers of interactive systems, we our WPAs talk to our relatively static biologi- should care about these findings, because the

22 PERVASIVE computing Published by the IEEE CS n 1536-1268/15/$31.00 © 2015 IEEE systems we aim to design are closer World space to the user’s body and mind than classical personal computing devices Perception space have ever been. Inspired by these Action space findings, we propose an information Recognizable set flow model that considers the percep- tion, cognition, and action of human Examinable set agents while (unlike in more classical Selected set HCI models) dealing with conscious Manipulated set and unconscious cognition separately. This lets us take a more holistic view of the role of WPAs, making it evi- dent that the explicit interaction be- tween WPAs and users occurs in an information-rich context, in which Figure 1. The situative space model (SSM).9 We developed the SSM to capture what our brains process much more than a specific human agent can perceive and not perceive, reach and not reach, at any we traditionally model as system de- given moment in time. In particular, the perception space is the space around the signers. It also lets us start speculat- agent that can be perceived at each moment. The action space is the space around ing about functionalities that WPAs the agent that is currently accessible to the agent’s physical actions. could offer that interface directly with the unconscious part of our cognitive processes, something that is undoubt- edly still hard to implement in prac- not just a single targeted artifact or Action and Perception Instead tice, even if successful attempts have system. of Input and Output indeed been made.5 • The proximity principle assumes In the egocentric interaction paradigm, Although this article’s main focus is that proximity plays a fundamen- the modeled individual must be viewed our egocentric approach to the design tal role in determining what can be as an agent that can move about in a of WPAs inspired by modern cognitive done, what events signify, and what mixed-reality environment (an environ- science, we also discuss our experiences agents are up to. ment consisting of both directly acces- deploying the framework in the hospi- • Changeability of the environment sible everyday “real” entities and vir- tal domain and our initial WPA proto- and of the agents’ relationship with tual/digital objects accessed through type for orthopedic surgeons based on the environment takes into account mediating digital devices), not as a user the Google Glass platform. agents’ more or less constant body performing a dialogue with a com- movements, including the head, puter. Adopting the physical-virtual Egocentric Interaction hands, and sensing organs. equity principle, we suggest substitut- Both system designers and users in- • The physical-virtual equity principle ing the concepts of (device) input and creasingly face a new HCI paradigm pays equal attention to interaction output with (human agent) action and that redefines the relationship be- with both virtual (digital) objects perception. tween the human, computer system, (classical HCI) and physical objects and world: an egocentric interaction (classical ergonomics). The New “Desktop” paradigm.6 This paradigm extends To facilitate the design of egocentric and modifies the classical user-cen- The term “egocentric” signals that it is interaction systems such as the WPAs tered approach in HCI7 on several the body and mind of a specific individ- we focus on here, we developed a situ- points: ual that (literally) acts as the center of ative space model (SSM) to capture reference, so all modeling is anchored what a specific human agent can per- • Situatedness acknowledges the pri- to this individual’s body and mind in ceive and not perceive, reach and not macy of the agents’ current bodily this interaction paradigm. The term is reach, at any given moment in time situation at each point in time in analogously used in psychology and (see Figure 1). This model is for the guiding and constraining agents’ to denote the conceptual emerging egocentric interaction para- behavior. and spatial frames of reference that hu- digm what the virtual desktop is for • Attention to the complete local en- mans by necessity rely on when think- the PC/WIMP (window, icon, menu, vironment emphasizes the need to ing and acting in the world and when pointing device) interaction paradigm: consider the entire environment, collaborating with others.8 more or less ­everything of interest to a

July–september 2015 PERVASIVE computing 23 Digitally Enhanced Reality

(such as vision) toward phenomena of External Human body and mind world interest (pathway 8-10-2-4). Perception Cognition 12 13 Cognition Introvert Human cognition is divided into un- (in-body) conscious and conscious processing phenomena 5 Unconscious 6 Conscious (arrows 12 and 13 in Figure 2, respec- cognitive cognitive Wearable processing 7 processing tively). The human agent receives in- personal 3 Body 4 put from sensors capturing in-body assistant sensors 9 8 phenomena (such as proprioceptive 14 2 information about limb positions and 1 for maintaining homeostasis) and from Action World Body sensors capturing information from the phenomena actuators (muscles) external world. No external world or 10 in-body phenomena is subject to con- 11 15 scious cognitive processing before it has been unconsciously processed (path- ways 2-4-6 and 5-6, respectively). Conscious processing is slower Figure 2. A body-and-mind-centric model of how wearable personal assistants (WPAs) than unconscious processing. For in- fit into the flow of perception, cognition, and action of human agents. Classical HCI stance, muscular reactions to immedi- models are typically concerned only with pathway 15-3 (the bold red arrows) and how ate threats are initiated unconsciously it relates to conscious cognitive processing. Yet by and large, our perception of the (pathway 4-9) long before conscious world (pathway 2-4) and our perception of our body state (arrow 5) is beyond processes are engaged. We protect our our conscious control. faces with our hands instinctively from approaching projectiles such as hockey pucks even when we are consciously aware of the fully protective shields of specifichuman ­ agent is assumed to, and Perception-Cognition-Action transparent material in front of us. supposed to, happen here. We describe Loop only the perception and action spaces Figure 2 shows a simplified model of Action here and point the reader elsewhere for information flows occurring as the Human action is initiated and con- more details.9 result of a human agent acting in the trolled by a mix of conscious and un- The perception space is the space world. The purpose is not to provide conscious cognitive processes. An ex- around the agent that can be perceived a completely accurate account but ample of an activity mostly driven by at each moment. Like all spaces and a good-enough model for designing an unconscious perception-cognition- sets of the SSM, it is agent-centered, ­future interactive systems. action loop is walking along a well- varying continuously with the agent’s known road with no exposure to ob- movements. Different senses have dif- Perception stacles (pathways 2-4-9-10 and 11). An ferently shaped perception spaces, By and large, our perception of the example of an activity that uses a com- with different operating requirements, world (pathway 2-4 in Figure 2) and bination of conscious and unconscious range, and spatial and directional res- our perception of our body state (ar- cognition is attempting to thread a nee- olution with regard to the perceived row 5) is beyond our conscious control. dle, which demands focused visual and sources of the sense data. Compare However, conscious cognitive processes tactile conscious attention (pathway ­vision and hearing, for example. influence unconscious processes (ar- 2-4-6-8-10) in parallel with uncon- The action space is the space around row 7), as in the case when we deliber- scious detailed control of hand, finger, the agent that is currently accessible to ately address our attention to a certain and arm muscles (pathway 5-9-11). the agent’s physical actions. Objects speaker in a crowd and automatically within this space can be directly acted (thanks to unconscious processing), to The Wearable Personal upon. The outer range limit is less de- some degree, single out the voice we Assistant in the Loop pendent on object type than that of the want to hear. We can also consciously By including unconscious cognitive perception space and is basically deter- and indirectly affect unconscious pro- processing and all perceivable world mined by the agent’s physical reach. cessing by orienting our body sensors phenomena (including everyday

24 PERVASIVE computing www.computer.org/pervasive ­objects such as coffee cups and foot- • Implicit input is action performed by Situation identification is one such balls), the model in Figure 2 provides a a human agent without the conscious mechanism. By implicitly monitoring more complete perspective of the con- intention of communicating with the body state through pathway 10-1 (for text in which a WPA operates than clas- WPA. For example, a human agent example, body posture, galvanic skin sical HCI models, which are typically acts in the real world (moves about, response, and heartbeat), and correlat- concerned only with pathway 15-3 (red manipulates objects, and interacts ing it with the state of the nearby world arrows in the figure) and how it relates with other human agents), which is (arrow 1), the WPA has a reasonable to conscious cognitive processing. It partially sensed by the WPA (path- platform for determining the human becomes evident that any information ways 8-10-1 and 9-10-1). agent’s current situation. generated by a WPA (arrow 3) is just • Explicit output occurs when a WPA Another mechanism is subliminal one source of information among many addresses the conscious mind. The cueing. Certain phenomena measured others that hit the unconscious and con- WPA creates a change in the agent’s best close to the body (for example, scious parts of our brains, which to- perception space (Figure 1) that the eye movements, facial expressions, and gether try to make sense of it all. Heu- human agent cannot avoid con- electromyography [EMG]) can pro- ristics for serving that information in sciously perceiving (pathway 3-4-6), vide important­ insights into ongoing a timely manner, arriving at successful thereby inviting the human agent to conscious and unconscious cognitive “attention management,” is probably act (arrows 10, 11, and 15). processing. They can therefore help de- best based on knowing what else is hit- • Implicit output occurs when a WPA termine the intensity level and type of ting the senses in parallel. The context- addresses the unconscious mind. The stimuli that could be used for subliminal aware systems community has investi- WPA generates a phenomena in the cueing,13 and for the WPA to sublimi- gated this for years but often using a agent’s perception space (Figure 1) nally direct the human agent’s gaze in a system- or device-centered approach. that reaches the unconscious part certain direction (pathway 3-4-9). We believe that a human-centric ap- of cognition (pathway 3-4) but not Finally, work has also been done in proach toward determining what mat- the conscious part (pathway 6)—for mediated reality. If the WPA is suffi- ters in a given situation (for example, example, through ambient displays ciently integrated into the visual per- using the SSM in Figure 1) will reveal such as the “dangling string”11). ception flow, beyond Google Glass interesting complementary information. see-through and partially covering mon- Note that conceptually, the hard- By placing actuators and sensors on ocular HMDs and toward ’s ware user interface of a WPA (such or very close to the body and keeping EyeTap vision,14 the WPA could act both as the head-mounted display [HMD], them there for large parts of the day, as a filter and highlighter. In this way, microphone, loudspeaker, and Google we would argue that the WPA can po- it could directly alter the perception of Glass touchpad) receives input from tentially sense and affect several of the the surrounding world (arrow 2) so as the human agent and provides output information flows shown in Figure 2 to facilitate tasks. Naturally, security and that the human agent can sense in the with more precision than more tradi- ethics become important topics if devel- shape of world phenomena (arrows 10 tional interactive systems (such as PCs opment leads us in this direction. and 2). In Figure 2, these information and smartphones). This leads to the in- flows are re-represented as separate triguing idea of future WPAs being able A WPA for Orthopedic information flows (arrows 15 and 3) to to facilitate the transition from “felt Surgeons facilitate the following discussion. sense” tacit knowledge, generated as Healthcare is a highly collaborative the human agent experiences the world, work domain, and even single-user Implicit Input and Output to knowledge that the agent can con- hospital systems, such as electronic pa- Although seldom clearly defined in the sciously reflect on and articulate,12 aug- tient record (EPR) systems, are used for HCI literature, the distinction between menting human cognition at the core. coordination and collaboration. More- explicit and implicit input and output Space limitations, and our wish to dis- over, because of the spatial distribution is useful for discussing some important cuss possibilities that are more directly of departments, wards, meeting rooms, properties of WPAs10: applicable in the near future, make us and offices, clinicians move around a end this section by mentioning some lot. Their mobile yet collaborative work • Explicit input is action intentionally more concrete potential mechanisms style forces them to use pagers, mobile and consciously directed toward the for using implicit input and output phones, and other devices to communi- WPA. For example, a human agent in the context of WPAs (all currently cate, which can interrupt and interfere navigates the GUI presented on the explored by the pervasive computing with ongoing activities.15 Google Glass HMD by swiping the community but not necessarily in the However, previous attempts to develop touch area (pathway 8-15). context of designing WPAs). WPAs for clinicians have faced several

July–september 2015 PERVASIVE computing 25 Digitally Enhanced Reality

surgery, an assistant or nurse operates the mouse and keyboard for surgeons. We observed a complex orthopedic surgery (scoliosis surgery) in which the surgical team used the navigation system to increase accuracy of the op- eration. In surgeries using the naviga- tion system, a CT scan of the patient is displayed on a large screen, while the navigation system tracks the positions of surgical tools in relation to the pa- tient’s coordinate systems. Thus, the surgeon needs to look at the screen and at the same time use the surgical tools to operate (see Figure 3b). In such situ- ations, the surgeon needs to frequently switch visual focus between the surgi- cal site and the screen. During some complex surgeries, the surgeon might (a) (b) call an experienced colleague to help. In such cases, the experienced surgeon Figure 3. Design framework for a WPA prototype for surgeon assistance: (a) the provides guidance either over the phone navigation system and several large screens to monitor medical images in the or in person. operating room, and (b) surgeons look at the screens while operating on the patient. In trauma cases, the acute depart- ment calls the on-call orthopedic sur- geon, and the surgeon must go to the technical and human-related chal- other surgeons. During these meetings, acute department immediately. The lenges.16 Emerging unobtrusive eye- surgeons discuss general topics such as acute department sends a short text wear computers such as Google Glass administrative issues and special pa- message to the surgeon giving a brief raise hope for solving some of the tech- tient cases. After regular meetings, all history of the patient. The surgeon nical issues. Furthermore, we believe surgeons meet in radiology conference reads the message before arriving at that our egocentric design approach rooms to review patients’ latest x-rays, the acute department. In trauma cases, demystifies some of the human-related MRIs, and CT scans; the medical im- a group of clinicians from different de- complexities by defining the concept of ages are presented on several large partments work together to save the pa- a WPA based on human needs for dif- screens. During the radiology meetings, tient from life-threatening injuries. ferent kinds of assistance. We are cur- surgeons discuss important cases and Visiting patients in the ward is also rently developing a WPA for orthopedic make notes in special notebooks. Af- part of the daily routine. Surgeons surgeons to support them throughout terward, surgeons booked for different make their ward rounds together with their workday based on our egocentric types of orthopedic surgeries go to the a nurse, moving from patient to pa- design approach. operating rooms (or theaters). Operat- tient. The surgeon makes a diagnosis To understand the healthcare work ing rooms are prepared based on the and prescribes treatment facilitated domain, we conducted an ethnographic type of surgery planned because each by the nurse, who has general knowl- study in Rigshospital in Copenhagen. type requires particular medical infra- edge of the patients and can provide an As part of the ethnographic study, we structures. For example, some complex overview of their current conditions. shadowed an on-call orthopedic sur- orthopedic surgeries require a naviga- Before ward rounds, the surgeon needs geon during a workday. Our initial ob- tion system to monitor a 3D model of to review the patient records and re- servations showed that surgeons have the surgical site and the position of the cent medical images of the patients on among the highest mobility in the hos- operation instruments. Several screens a computer in his or her office. After pital. Moreover, we observed several in the operating room allow surgeons to ward rounds, the surgeon reports re- orthopedic surgeons in different types monitor patient records, x-rays, MRIs, sults from the rounds using a Dicta- of orthopedic surgeries. and CT scans (Figure 3a). phone. Later, administrative personnel A typical surgeon’s workday starts Because computers and their periph- transcribe the recorded voice for the with a daily or weekly meeting with erals are difficult to sterilize during a EPR system.

26 PERVASIVE computing www.computer.org/pervasive Assistance type Because of clinicians’ high mobil- ity, they usually bump into each other Perception Action Cognitive in corridors, wards, and other parts assistance assistance assistance of the hospital. In these ad hoc col- Mobility Briefing on the Data entry on the laborative situations, they might talk move move about a particular patient or medical Mobile access to patient records task. They might then move to one physician’s office to look at a patient’s medical information. In some cases, they go to the ward to visit a partic- ork Task reminder ular patient and discuss appropriate Interruption treatments. Context-aware interruption WPAs in Hospitals: Design Framework Multitasking Display Predicting info Based on the egocentric design ap- information in eye requirements proach and the ethnographic study, we Characteristics of hospital w Support multi- developed a WPA design framework for modal interaction supporting clinicians—in particular, surgeons (see Figure 4). The main char- Collaboration Synchronous and Asynchronous and local collaboration local collaboration acteristics of hospital work are the core elements of the framework, represented by the five rows in Figure 4, while the Sterility Touchless restrictions interaction columns correspond to the three kinds of assistance defined by deploying our egocentric design approach. The boxes Figure 4. A design framework for WPAs in hospital settings. The rows present the within present our initial 12 ideas for main characteristics of hospital work, while the columns correspond to the three our healthcare WPA, which we briefly kinds of assistance defined by deploying our egocentric design approach. The boxes discuss here. within present our initial 12 ideas for our healthcare WPA, while those in bold red were implemented in an initial Google Glass prototype based on feedback from Perception Assistance surgeons. Perception assistance includes both augmenting the world by adding needed information to the perception space and simplifying the world by filtering wirelessly to the EPR system facilitates Display information in eye. During out potentially distracting or irrelevant information and knowledge sharing surgery, the surgical team needs to phenomena from the perception space. and can potentially be further simpli- monitor important information on fied by automatic retrieval based on a display. For example, during some Briefing on the move. To utilize time on ­location (history) of the clinicians (for orthopedic surgeries, it is necessary the move, the WPA provides informa- example, data for the patient near- to take periodic x-rays of the patient tion relevant to the situation that the est the clinician’s current location), or (fluoroscopy). Fluoroscopic surgeries surgeon will soon be entering. For bookmarked x-ray images for a sur- force the surgeon to frequently switch ­example, surgeons called to the acute geon attending a radiology conference. focus between the surgical site and department get important health sta- the screen. The WPA can display this tus information about the emergency Telepresence. Although digital resources information directly to the surgeon’s patient from the WPA. can be easily shared through mobile eyes, letting the surgeon maintain fo- devices, sharing physical resources, cus on the patient, reducing surgery Mobile access to patient records. The people, and tacit knowledge is still time and avoiding complications from main reason for mobility in hospitals a challenge. WPAs could help share x-ray exposure. is the need to be in different physical tacit knowledge among clinicians by places, to get in contact with a particu- offering ad hoc telepresence sessions Support for . lar person, or to access knowledge and between a remote specialist and local The WPA facilitates parallel activities shared resources. A WPA connected clinicians. by providing appropriate input and

July–september 2015 PERVASIVE computing 27 Digitally Enhanced Reality

­output modalities according to ac- through asynchronous collaboration. cause misunderstandings and delays. A tion and perception restrictions of the For example, clinicians update the WPA could act as an interface between situation in which clinicians perform time schedules of personnel on white- stationary computers and the surgeon medical tasks, as captured by the SSM boards, enter descriptions of medical through touchless modalities such as (Figure 1). For instance, when the user tasks performed on patients during a speech, body gestures, and gaze. is performing a visual task, the WPA work shift into computer systems, and could switch automatically or manu- so on. Ubiquitous WPAs would allow A WPA Prototype for Selected Scenarios We interviewed three orthopedic sur- We selected three scenarios for further geons to evaluate the utility of the 12 ideas in practice. First, we explained implementation: mobile access to patient the ideas to the surgeons and asked them to rank the usefulness of each records, telepresence, and touchless interaction idea using a 5-point Likert scale. Next, we discussed the ideas in more detail and display information in the eye. with open questions about the situa- tion in which the proposed idea could ally to aural modality for displaying for binding virtual objects to physical play a role, how frequently these situa- helpful information. objects, locations, or situations. tions occur, what information is needed in each situation, and what specific Synchronous and local collaboration. Action Assistance ­restrictions and requirements (such as In synchronous and local collabora- The main focus of action assistance is sterility) should be observed. Based on tive scenarios, WPAs can increase to make the surrounding environment the results of the interview study, we se- awareness among team members. For easier to manipulate by providing in- lected three main scenarios for further example, during a surgery, the atten- formation relevant to the task at hand implementation: mobile access to pa- tion point of a surgeon on the surgical through appropriate modalities. tient records, telepresence, and touch- site, tracked by a gaze tracker, could be less interaction and display informa- shared with the surgical team. Predicting information requirements. tion in the eye. The WPA would be able to recognize Because of Google Glass’s unobtru- Cognitive Assistance clinicians’ activities and allow them to siveness, its support of various input Many ideas for perception assistance access relevant information quickly, channels, such as voice commands, head ultimately target cognitive assistance. without sacrificing their connection to motion, and touchpad. Consequently, Hence, there is some overlap in the as- the patient or the procedure at hand. we developed the WPA prototype on the sistance mechanisms targeting percep- Google Glass platform. The first proto- tion and cognition. What we list here Data entry on the move. In a clinical set- type supports three selected scenarios. are support mechanisms that aim to af- ting, everything needs to be properly re- fect cognitive processes more directly. corded, for legal reasons and to ensure Mobile Access to Patient Records continuity of care. For instance, after According to the interviewed surgeons, Context-aware interruption. As a contin- visiting a patient in the ward, clinicians mobile access to patient records through uously running device that stays with should record diagnosis results and deci- the WPA provides valuable support in clinicians at all times, a WPA could de- sions. The WPA could support data en- situations such as making ward rounds, termine current context and determine try on the move by automatically record- working in the operating room, per- whether or not its wearer is interrupt- ing the visited patient’s ID and the date, forming ad hoc collaborations. How- ible in a given situation. time, and author (WPA user). ever, each situation requires the WPA to provide different interaction modalities. Task reminder. To mitigate the risk of a Touchless interaction. As noted earlier, For example, the WPA should support clinician forgetting a task that was in- in most operating rooms, several com- touchless interaction in the operating terrupted, the WPA could remind the puters and large displays monitor dif- room because of the sterility restrictions, user about the (state of) the interrupted ferent medical information before and whereas during ward rounds, surgeons tasks after interruption. during surgery, and because of sterility can use a Google Glass touchpad to pro- restrictions, input devices are handled vide input to the device. In fact, support Asynchronous and local collaboration. by an assistant or a nurse instructed for multimodal interaction is a crucial Many medical tasks are coordinated by the surgeon. This can sometimes characteristic for the WPA to provide

28 PERVASIVE computing www.computer.org/pervasive mobile access to the patient records. But different strategies could be applied to support multimodalities; the interaction modalities can be switched automati- cally or manually in different situations. To compare the effect of automatic and manual switching between mo- dalities on user performance, we con- ducted a lab experiment in which we (a) (c) (d) asked participants to complete a physi- cal task (a hotwire game, as illustrated Gesture recognition system in Figure 5a) while simultaneously answering simple math questions ei- ANT interface ther displayed on the HMD or played 18 InVesalius through the headset. Participants Inertial sensor Hand (ETHOS) answered the visual questions through gesture detector head gestures, and the aural questions Medical using audio modality (speech). We image Foot viewer measured participants’ performance gesture based on the speed and error rate in detector the hotwire task. The results of the ex- Floor sensor (Sensfloor) periment show that performance was (b) higher when participants answered the questions using the audio modal- ity than when they used visual and Figure 5. Early experiments informing the design of the WPA prototype: (a) a 18 gesture-based modalities, probably be- participant completing the Hotwire task ; (b) a participant touchlessly interacting 19 cause the auditory modality interferes with medical images on the screens, (c) a head-mounted display for teleguidance 17 less with the motorically and visually applications ; and (d) a wearable laser pointer. demanding hotwire task. Further- more, the participants’ performance was higher when the modalities were switched automatically than when they faster and more accurate when the QR or touch gestures on the touchpad (tap were switched manually, but manual code is available. for zooming in, long press for zoom- switching was preferred because of the ing out, and swipe up for rotation). To higher controllability. Switch between textual data and medical navigate through an enlarged ­image Based on findings from this experi- images. Patient records are distributed in real-time, the Google Glass head ment, we implemented redundant in- in several pages (cards) on the Google tracker sensor is used, allowing the put channels for the WPA and let users Glass. Because of the display size in user to quickly scan over the com- choose appropriate modalities manu- Google Glass, the textual part of the plete image by moving his or her head ally in different situations. The main patient records is shortened. However, ­(Figure 6). steps for mobile access of the patient medical images (x-rays, CT scans, and records on the Google Glass applica- so on) are the most important part for Touchless Interaction tion are as follows. orthopedic surgeons. Users can switch We studied the utility of touchless in- between cards using voice commands teraction techniques in the operating Find the records. To find and retrieve a (such as “Ok Glass,” “next,” or “pre- room in a lab experiment (Figure 5b).19 patient’s health records, the system pro- vious”) or performing swipe gestures Based on the experiment’s promising re- vides two main channels: voice and QR (swipe left and swipe right) on the sults, we designed a touchless interaction code. Users can filter the patient list by Google Glass touchpad. module for the WPA that enables sur- saying the patient’s name or social secu- geons to provide touchless input to other rity number. They can also find the pa- Interact with medical images. Users computers in the operating room using tient’s records by reading the patient’s can zoom in or out, rotate, and navi- voice commands and head gestures. QR code using the front-view camera gate through a zoomed-in view of the The surgeon can interact with two of Google Glass. The latter method is ­medical images using voice commands systems: a 3D medical imaging ­system

July–september 2015 PERVASIVE computing 29 Digitally Enhanced Reality

Mobile access to the patient records Touchless to share a still image taken by the lo- Telepresence interaction cal surgeon with a remote expert. The remote expert can use a mobile appli- cation to see the shared image and pro- vide guidance through vocal communi- cation and also by adding sketches on the still image. The graphical content provided by the remote expert is super- imposed on the local surgeon’s HMD in real time. Our comparative study on a remote pointing scenario using a laser pointer and HMD technologies (Figures 5c and 5d) revealed the challenge of stabilizing content provided by the remote ­expert on the local side when there is a live video stream between local and ­remote sides.17 Thus, we used still image and vocal communication in the telepres- ence module. In addition, because of Figure 6. Screenshots of the main cards (pages) of three applications. Users can switch the technical limitations of Google between cards using voice commands (such as “Ok Glass,” “next,” or “previous”) or Glass, the experienced quality of the performing swipe gestures on the Google Glass touchpad. To navigate through an live video stream is lower than still im- enlarged image in real-time, the Google Glass head tracker sensor is used, allowing the ages, which makes still images the bet- user to quickly scan over the complete image by moving his or her head. ter choice. The feedback from two sur- geons who tried the telepresence system also showed that in orthopedic surger- ies, sharing still images is more useful (InVesalius) or a 2D image viewer to x-ray on the screen, in which the screen than live streams. However, sharing review x-rays and other 2D images. is divided into five areas: up, down, live video can of course be useful in The surgeon can switch between 2D left, right, and middle. By moving the other situations.20 images, zoom in or out, and navigate pointer from the middle to the four through an enlarged image on the sta- other areas on the Google Glass screen, tionery screen via voice commands and the enlarged image moves in the same t the time of this writing, head gestures through the WPA. To direction on the big screen. we just finished an empiri- interact with the 3D imaging system, In addition to supporting touchless cal evaluation of the system the user should first choose the desired interaction, this module allows ­users at the ITX hospital simula- view (Axial, Sagittal, or Coronal). The to take a snapshot from the statio- tionA facilities in Copenhagen involv- surgeon can adjust the view of differ- nery screens and display it in surgeon’s ing two orthopedic surgeons who tried ent slices of the 3D model continuously Google Glass HMD (see Figure 6). the WPA in two scenarios: surgery and by vertical movements of the head. ward rounds. Preliminary results con- To send commands using head move- Telepresence firm the WPA’s utility and practicality ments, we divided the Google Glass The three surgeons we interviewed in realistic settings. However, the sur- screen into three areas distributed ver- noted that, during complex surgeries, geons and patients reflected on some tically. The user’s vertical head move- the surgeon might need help from an social challenges in the ward rounds ment is mapped to the position of a expert colleague. In such situations, scenario. Our future plan is to im- pointer on the screen. When the pointer the surgeon asks the expert colleague prove interaction aspects of the WPA to enters the top area, the depth view of to personally help in the operating ­address the social challenges and other the 3D model increases, whereas cross- room or to provide guidance through a minor issues. ing the lower border with the pointer phone call. To enhance the effectiveness From a theoretical perspective, we decreases the depth. The same method of collaborations over the phone, the hope to help extend current context- is used to navigate through an enlarged WPA’s telepresence module is designed­ awareness research to better explore­

30 PERVASIVE computing www.computer.org/pervasive the Authors more powerful future personal assis- Shahram Jalaliniya is a PhD fellow at the IT University of Copenhagen, where tance systems. Our proposed egocentric­ he is a member of the Pervasive Interaction Technology (PIT) Lab. His research design approach, which includes interests include wearable computing, HCI, pervasive computing, and mul- timodal interaction. Jalaliniya has a master’s degrees in information systems ­unconscious cognitive processing as part from Lund University and in software and technology from the IT University of of the system design, is well-grounded Copenhagen. He is a member of IEEE. Contact him at [email protected]. in modern cognitive science but poses huge challenges for us as engineers. We believe that the pervasive computing Thomas Pederson is an associate professor at the IT University of Copenha- community will take on this challenge gen, where he is a member of the Pervasive Interaction Technology (PIT) Lab with increasing success as body-worn and the Interaction Design Group. His research interests include HCI, context- aware systems, and pervasive/. Pederson has a PhD in technology and the modeling of human computing science from Umeå University, Sweden. He is a member of the cognition continue to improve. ACM. Contact him at [email protected].

Acknowledgments

We thank Ulrik Kaehler Olesen of the Department of Traumatology and Orthopedic Surgery at the Rigshospitalet, Copenhagen, as well as our fel- low Pervasive Interaction Technology (PIT) Lab members for their valuable input at various stages Human-Computer Interaction, L. Erlbaum Cooperative Work (CSCW), vol. 14, no. 2, of the design of the WPA prototype. This work Associates, 1986. 2005, pp. 131–160. was supported by the EU Marie Curie Network iCareNet under grant number 264738. 8. I. Poupyrev et al., “Egocentric Object 16. K. Adamer et al., “Developing a Wear- Manipulation in Virtual Environments: able Assistant for Hospital Ward Empirical Evaluation of Interaction Tech- Rounds: An Experience Report,” The niques,” Computer Graphics Forum, Internet of Things, Springer, 2008, References vol. 17, no. 3, 1998, pp. 41–52. pp. 289–307.

1. T. Starner, “The Challenges of Wearable 9. T. Pederson, L.-E. Janlert, and D. Surie, “A 17. S. Jalaliniya, T. Pederson, and S. Computing: Part 1,” IEEE Micro, vol. 21, Situative Space Model for Mobile Mixed- Houben, “Wearable Laser Pointer Ver- no. 4, 2001, pp. 44–52. Reality Computing,” IEEE Pervasive Com- sus Head-Mounted Display for Tele- puting, vol. 10, no. 4, 2011, pp. 73–83. Guidance Applications?” Adjunct Proc. 2. G. Lakoff and M. Johnson, Philosophy ACM Int’l Symp. Wearable Comput- in the Flesh: The Embodied Mind and 10. A. Schmidt, “Implicit Human Computer ers (ISWC), 2014, pp. 193–200; doi: Its Challenge to Western Thought, Basic Interaction through Context,” Personal 10.1145/2641248.2641354. Books, 1999. Technologies, vol. 4, nos. 2–3, 2000, pp. 191–199. 18. S. Jalaliniya, “Towards Dynamically 3. C. Rosen, “The Myth of Multitasking,” Switching Multimodal User Interfaces,” The New Atlantis, vol. 20, Spring, 2008, MS thesis, IT Univ. of Copenhagen, 2014. pp. 105–110. 11. M. Weiser and J.S. Brown, Designing Calm Technology, PowerGrid J., vol. 1, 19. S. Jalaliniya et al., “Touch-less Interac- no. 1, 1996, pp. 75–85. 4. S.J. Brooks et al., “Exposure to Subliminal tion with Medical Images Using Hand Arousing Stimuli Induces Robust Activa- & Foot Gestures,” Adjunct Proc. ACM tion in the Amygdala, Hippocampus, Ante- 12. E. Gendlin, Experiencing and the Creation Conf. Pervasive and Ubiquitous Comput- rior Cingulate, Insular Cortex and Primary of Meaning: A Philosophical and Psycho- ing (Ubicomp), 2013, pp. 1265–1274; doi: Visual Cortex: A Systematic Meta-Analy- logical Approach to the Subjective, North- 10.1145/2494091.2497332. sis of FMRI Studies,” Neuroimage, vol. 59, western Univ. Press, 1962. no. 3, 2012, pp. 2962–2973. 20. B.A. Ponce et al., “Telementoring: Use 13. M. Vidal, D.H. Nguyen, and K. Lyons, of in Orthopedic 5. R.W. DeVaul, A.S. Pentland, and V.R. “Looking At or Through? Using Eye Education, J. Bone & Joint Surgery, vol. Corey, “The Memory Glasses: Subliminal Tracking to Infer Attention Location for 96, no. 10, 2014, p. e84; doi:10.2106/ vs. Overt Memory Support with Imper- Wearable Transparent Displays,” Proc. JBJS.M.00928. fect Information,” Proc. IEEE Symp. ACM Int’l Symp. Wearable Computers Wearable Computers (ISWC), 2003, (ISWC), 2014, pp. 87–90. pp. 146–146.

6. T. Pederson, L.-E. Janlert, and D. Surie, 14. S. Mann, “Continuous Lifelong Capture of “Towards a Model for Egocentric Inter- Personal Experience with EyeTap,” Proc. action with Physical and Virtual Objects,” 1st ACM Workshop Continuous Archival Proc. 6th Nordic Conf. Human-Com- and Retrieval of Personal Experiences, puter Interaction: Extending Boundaries (CARPE), 2004, pp. 1–21. ­(NordiCHI), 2010, pp. 755–758. 15. J.E. Bardram and C. Bossen, “Mobility Selected CS articles and columns 7. D.A. Norman and S.W. Draper, User Cen- Work: The Spatial Dimension of Collabo- are also available for free at tered System Design, New Perspectives on ration at a Hospital,” Computer Supported http://ComputingNow.computer.org.

July–september 2015 PERVASIVE computing 31