Literature study: Wearable Control Room

/LWHUDWXUHVWXG\FRQFHUQLQJZHDUDEOHFRPSXWHUVIRUXVHLQSURFHVVSODQWV This report is a literature study concerning different aspects of wearable . The report is meant to be used mainly as background information for defining relevant projects within the program “wearable control room” as well as an introduction of this research topics for Ph.D. students and other interested readers.

,QWURGXFWLRQ This report presents the state of the research concerning wearable computers with focus on use within process plants. In addition to wearable computers, the report focuses on human factors as well as the organisational and social aspects of substituting the existing centralised control room with distributed wearable computers. Research related to wearable computers has existed for several years, but most of the research has concentrated on other aspects and goals than developing a wearable control room for process plants.

A wearable can generally be defined as (Bass 1997):

• it may be used while the wearer is in motion; • it may be used while one or both hands are free or occupied with other tasks; • it exists within the corporeal envelope of the user, i.e., it should be not merely attached to the body but becomes an integral part of the person's clothing; • it must allow the user to maintain control; • it must exhibit constancy, in the sense that it should be constantly available. The following chapters outline the state of work related to different topics, applications and finally, universities and research institutions working with wearable computers. Chapter 2 pre- sents a short review of equipment available on the market. Chapter 3 continues with a review of different communication technologies used in wearable computing. This chapter is an exten- sion of chapter 3. In chapter 4, the integration of components for information storage, support, training, etc. is presented. The next four chapters review different aspects of man- machine interaction. Chapter 5 focuses on the problems of visualisation; what should be visua- lised and how should it be visualised. Chapter 6 gives an introduction to cognitive science related to wearable computers. In chapter 7, literature concerning the human-human interac- tion, or the social aspects of wearable control rooms, is reviewed. The organisation structure may also be influenced by the introduction of distributed wearable control rooms and chapter 8 presents relevant references.

The reviews of these topics are not necessarily related to wearable computers. However, expe-

- 1 - Literature study: Wearable Control Room riences from, for example, mobile workplaces within consultancies may be transferred to pro- cess plants. After the chapters concerning particular topics, chapter 9 reviews different applications of interest. Finally, in chapter 10, relevant universities and research institutes working on subjects of interest for this work are presented.

(TXLSPHQW A wearable computer system is equivalent to a typical . However, it consists of a head mounted display or eyeglasses, input devices, communication link to send or receive up-to-date information from remote servers, a wearable computer, a battery and optional user and environmental sensors. The physical characteristics such as weight and size are significant as the operator should be able to work without being disturbed by the wearable computer system (e.g., , Birmingham University). This chapter reviews the trends within trackers and displays. The following chapter focuses on input devices.

Whenever information is presented to human beings, it is important to present right informa- tion at right time. Especially, this is essential for where computer-generated information is superimposed on transparent eye-glasses. Different types of trackers determine the positions of the user’s head and/or hands. Today, four types of trackers are used: (1) mag- netic trackers, (2) “WUHJKHWV´tracker, (3) optical trackers and (4) GPS. Magnetic trackers deter- mines a set of 3D coordinates based on magnetism, but these trackers are easily influenced by metal. The second category imitates the human vestibular sense system consisting of fine hair reacting on movements. These trackers are often inaccurate. The optical trackers are very pre- cise, but these are expensive. Finally, GPS is an accepted system for geographical positioning, but within small distances GPS is still too imprecise.

The head mounted display or eyeglasses present visual information to the user. Today, different techniques are available and accepted for information presentation depending on the user’s tasks. Head mounted displays, or helmets, are mainly used to present 3D worlds that are detached from the reality and hence, its name “” (University of North Carolina). When the user moves his head the virtual landscape follows these movements. In virtual real- ity, the user can interact with the visualisation in different ways. In some , the virtual objects are overlaid with synthetical information (Carnegie Mellon University).

A head mounted display can also consist of a single screen placed just above one of the user’s eyes. Hence, information that the user needs to perform a particular work task can be displayed on the screen (Chalmers University of Technology, Birmingham University). The user can see

- 2 - Literature study: Wearable Control Room the reality as well as the information screen. Transparent eyeglasses has a different function compared to the two types of head mounted displays presented. Information is superimposed on the reality (e.g., & Lab, Colorado School of Mines). This technique is called augmented reality. Today, it seems that augmented reality may perform some tasks that previous needed to be modelled in virtual reality. Examples are systems over- loaded by artificial information such as non-existing buildings and tag names in complex plants (Darmstadt University of Technology, MIT). Wearable computer systems using overlaid information require accurate trackers to determine the position of the user’s head and the syn- thetical information. Other problems are the resolution of the displays and graphic accelera- tors. However, a new type of visualisation called retinal scanners is just introduces (University of Washington). The scans a low power beam of light which "paints" an image directly onto a user’s retina rather than a screen. Three laser sources (red, green, and blue) are combined to provide a full RGB color scale. These displays will see applications, not only in high end military and medical systems, but in aids for people with low vision and ulti- mately in everyday use with computers, telecommunications and television.

1DYLJDWLRQWHFKQRORJLHV(speech, touch pad, eye- and head tracking, etc.) Within a wearable computer system, different communication links exist between the user and the wearable computer (input device) as well as between the wearable computer and the remote . This section focuses on communication from the user to the wearable computer system.

Most of the wearable computer systems use speech recognition technology as the input device. Examples are the speech-driven wearable computer (manufactured by Speech Systems Inc.), wearable computers within the emergency service (e.g., Industrial Ergonomics Group at Bir- mingham University and West Yorkshire Ambulance Service) and the current prototype of the Factory Automation Support Technology (e.g., Research Institute). Using voice activating input devices releases both hands of the user to perform other tasks.

Other kinds of input devices are hand-held keyboards (alternatively put around an arm), mice and hand controls (sort of joystick). Head tracking is also used in some applications (surgery at RiT: the surgeon controls the direction of a camera within the patient by head movements (Austad and Pedersen 1996)).

- 3 - Literature study: Wearable Control Room

6\VWHPLQWHJUDWLRQ(support system, , CSCW, etc.) Electronic Performance Support System (EPSS): incorporates task specific information, train- ing and assistance at the worksite when most needed. Such systems include hypermedia, expert systems, computer assisted instruction and intelligent agents. The FAST framework is an extension of the EPSS in that FAST is a wearable EPSS. (Birmingham University)

9LVXDOLVDWLRQ(user interface; what should be presented and how should it be visualised) What should be visualised?

How should it be visualised? (Feiner et al. 1993)

The typical graphical user interface does not work with voice input. The older, command- based or function key-based user interface is much more effective. (Experiences from “A wearable computer for quality assurance inspectors in a food processing plant”).

&RJQLWLYHVFLHQFHZLWKLQZHDUDEOHFRPSXWHUV Cognitive models are used to measure the human reliability, to predict and to understand actions taken by humans, to improve human-machine systems, etc. Best gives a general intro- duction to cognitive psychology (Best 1992). The following topics are of special interest for wearable control rooms:

• work tasks performed by the operators • mental models • augmented reality

The research on artificial intelligence and expert systems have combined computers and cogni- tion (Sheridan 1986). However, Newell and Simon’s model of human problem solving (Newell and Simon 1972) and Rasmussen’s decision ladder (Rasmussen 1983, Rasmussen 1986) are two widely used cognitive models of a reasoner, that is a human being, an expert system, etc. Especially, the decision ladder model is developed for environments within the process indus- try. Anderson’s model reflects cognitive structures (Anderson 1995). Other theoretical models that take perception into consideration are suggested (Gibson 1979; Gibson and Walk 1960; Kelso 1995). On the other hand, Hollnagel has suggested to treat human reliability as a concept rather than a number (Hollnagel 1991) and hence, refers to the use of fuzzy systems within cognitive psychology.

- 4 - Literature study: Wearable Control Room

6RFLDODVSHFWV Reports from Cornell University (“Social connectivity in the mobile workplace”): The goal of the research was to understand how employees were affected by the social changes in their work environment; how employees coped with reduced opportunities for face-to-face contact; and how functions such as informal organisational learning were carried out in the mobile work environment.

2UJDQLVDWLRQDOFKDQJHV The purpose of this project is to replace the central control room as known today with distrib- uted wearable control rooms. Hence, this will impact the organisational structure and environ- ment. Information will be available to employees at different skill levels and the traditional organisation hierarchy may change. The introduction of distributed control rooms may extend the demands for collaborative work. A report from Cornell University presents experiences of new workplace strategies. The report addresses the influence of the work environment on project coordination; work patterns; communication within and outside the team and within and outside one’s discipline; learning; social interaction and client reactions (“The ecology of collaborative work”).

To some extent, experiences from alternative, or mobile, workplaces can be transferred to wearable control rooms.

$SSOLFDWLRQV Today, wearable computers are in use in different environments and for different applications. Some systems exist as prototypes whereas other wearable computing systems are integrated in real environments. Examples of applications where the wearable computer system is in focus are PDLQWHQDQFH (process plants, the US Marine, airlines), VLWXDWLRQDODZDUHQHVV (war fighters, emergency services, process plants), VXSSRUW V\VWHPV (emergency services, process plants), WHOHRSHUDWLRQ (emergency services, off-shore), ORJLVWLF (industrial applications) and OHDUQLQJ (). Within maintenance, the operator makes use of voice-input data, check regulations, transfer information and makes timely decisions in the field. Using the voice and hands is a natural way to conduct inventory. With a wearable computer, the operator’s hands are free to count items or climb ladders, while the wearable computer sends data directly to the reducing errors. Situational awareness includes access to updated and relevant data concerning weather, maps or building plans to aid communications and help coordinate relief efforts. Head-mounted

- 5 - Literature study: Wearable Control Room video cameras can also document and communicate conditions for adjustment. Support sys- tems are used in a wide range of fields; equal to all fields is that the wearable computer system has access to and can transfer information while leaving the user’s hands free to physically activities. Finally, learning how to repair, inspect or perform a certain task no longer needs to be limited to the classroom. The wearable computer can assist in training and certification by allowing the user to bring an interactive, electronic instructor to the work site.

Especially within architectural and medical environments, different systems of 3D visualisa- tion such as virtual reality and augmented reality have been developed. Virtual reality can give the user a visual feeling of “walking” through a 3D-world illustrating, for example, a building or the inside of a machine. The following sections present 3D visualising applications.

7KH:DONWKUX3URMHFWThis project is done at University of North Carolina, Department of . The overall goal of the Walkthrough Project is to create interactive com- puter graphics systems that enable a user to experience an architectural model by simulating a walk through of the model (Brooks 1992). The main goals of the Walkthrough project are to: a) drive existing dynamic graphics engines to the utmost, b) push forward the development of methods for tracking position and orientation, c) have users evaluate our systems frequently so that we can identify aspects of a system which most impair the illusion of real presence, and d) learn more about the behavior of people in simulations. The long-term goal is to develop a per- sonal, portable visualisation system that will allow users to walk through and interact with models of meaningful complexity while receiving realistic visual, proprioceptive, and auditory feedback at interactive rates (>25 updates per second).

=NH\DQHZPHWKRGIRUPHUJLQJYLUWXDOUHDOLW\At Carnegie Mellon University, a group works on overlaying real images with synthetical images in real time. The z-key method will extend the possibilities of synthesizing virtual reality and make applications such as tele-oper- ation, tele-presence, training systems with simulation and games playing in virtual reality, pos- sible and/or more effective.

$XJPHQWHGUHDOLW\Besides walk-through systems and virtual reality, augmented reality over- lays the reality with synthetical images. Hence, the user can see the real world and in addition, computer-generated graphics. Often, the user is able to interact with the system by a helmet, eye-glasses and/or virtual reality gloves. The technology can be used to support maintenance, repair, training, assembly and logistics. Two examples are presented from Colorado School of Mines and Boeing, respectively. At Colorado School of Mines, they work with a scenario of

- 6 - Literature study: Wearable Control Room (PC) maintenance. This prototype can automatically determine the position and orientation of the PC with respect to the maintenance person's head. It continually displays graphical overlays showing the user the location of parts within the PC and guidelines for actions to perform. Boeing uses augmented reality for daily maintenance and repair of air- planes. The operator wears transparent eye-glasses on which useful information is superim- posed related to the positions of the eyes. Examples of information are hidden wires in the fuselage, locations of fuses and experiences concerning previous problems. Boeing has worked within this field for several years and is probably the company that is in front of this research topic.

&DUQHJLH0HOORQ8QLYHUVLW\3URMHFW'HVFULSWLRQNavigator 2 is a multimedia wearable computer for a Boeing aircraft inspection application. It is used by US Air Force personnel. An inspector uses the system to examine the skin of a KC 135 aircraft for cracks and corrosion during introduction to depot-level maintenance. The location and type of each defect found is recorded on the Navigator 2 (using a graphical representation of the aircraft as a location indi- cator). This type of inspection requires crawling over all of the aircraft's skin, typically stand- ing on a "cherry picker" but also attached via safety harness and standing on top of the aircraft. The primary input is a joystick providing general 2-dimensional input that is useful for posi- tioning on the geographic-based input where location is important. The joystick is used in con- junction with speech to mark discrepancies. Field evaluations for aircraft inspection at McClellan Air Force Base indicate not only approximately a 20 percent savings in inspection time but also a dramatically reduced inspection data entry time from hours to minutes.

*HRUJLD7HFK(366DQG)$67

0HGL:HDU 2UHJRQ8QLYHUVLW\ 2YHUYLHZMEDIWEAR is an application being currently developed as a part of the Wearable Computers research project. Such computers are meant to close the gap between the mobile and constantly changing human workspace and the tradi- tional way of using computers. Some of the advantages that wearable computers introduce are:

Information is accessible at hand wherever the user needs it.

As the user interacts with the environment, the changes made are timely and accurately updated.

Availability of precise information beyond the scope of the user's knowledge. Sensors enable wearable computers to actively acquire and process information about the environment. The keyword in this context is perception broadening. Simple and natural way of interfacing the

- 7 - Literature study: Wearable Control Room computer by speech and vision. For MEDIWEAR, three main application areas were identi- fied:

The first area is to support nurses in home care. The main goal of home care is to be able to send patients home at an earlier stage, thereby reducing overall cost and sup- porting the healing process by letting patients be in the environment they are more comfortable with. Second, a mobile computer which is taken home by a patient in order to monitor vital functions that could otherwise not be monitored outside a hospital. We call this mobile com- puter a patient wearable. The issue of patient acceptance of human-computer connections has yet to be addressed.

Third, a mobile computer which informs paramedics outside the hospital. The paramedic wear- able combines different functions that help the paramedic to accomplish typical tasks more rapidly.

6FLHQWLILFLQVWLWXWLRQV •At %LUPLQJKDP 8QLYHUVLW\, a group called Industrial Ergonomics Group works with human factors of wearable computers. The research is focused on “wearability”, the conse- quences of the physical equipment on operation and feedback to be presented to the wearer, and the dialogue between human and computer. The first item refers to the physical charac- teristics of products and users such as weight, size and flexibility. The second item con- cerns input/output technologies as well as information search and retrieval from limited displays. The third item considers human factors in dialogue design and interaction. Other centres of wearable computers are MIT and BT (British Telecom). •At &KDOPHUV8QLYHUVLW\RI7HFKQRORJ\, a group at Department of Production Engineering works on a project called Super Operator. The operator in a manufacturing plant uses a wearable computer (Xybernaut) to perform maintenance and in operating the plant. The project uses standard equipment and develops its own software based on . •At MIT (0DVVDFKXVHWWV,QVWLWXWHRI7HFKQRORJ\), the research focuses on development of wearable computers regarding the technological challenges as well as the use for ordinary human beings. The goal of wearable computers is to become a seamless extension of the body and mind. has published papers concerning smart clothing, that is, incor- porating a wearable computer into an ordinary pair of eyeglasses and ordinary clothing such as jogging shoes or a jacket (Mann 1997). The scientific purpose is idealistic com-

- 8 - Literature study: Wearable Control Room

pared to, for example, the Industrial Ergonomics Group at Birmingham University. • The Wearable Computing Research Group at 8QLYHUVLW\RI2UHJRQ mainly focuses on the technical and user interaction issues of wearable computing. The group has been collabo- rating with the US army and the marine corps to develop usable wearable computers. MediWear is a project for use within home care, patient treatment and paramedical ser- vices. •At *HRUJLD 7HFK, they develop applications to support plant personnel. Their research takes two directions: (1) system and information integration (EPSS) and (2) wearable com- puters for training and decision support (FAST). • The Wearable Computer Systems research group at &DUQHJLH0HOORQ8QLYHUVLW\ develops prototypes of wearable computers to improve the demands to the equipment and communi- cation. Especially, maintenance and plant operation are examples of applications that could be successively supported by wearable computers. The research group discusses factors within the organisation such as the consequences of down-sizing and productivity improvement goals, etc. The main goal is the need of functionality, that is, the user should be able to dedicate all his attention to the task at hand with no distraction provided by the system itself. Furthermore, the research group states that interface design must take into consideration the user tasks by developing mental models of the user to match the interface element with the user tasks. • There are actually two interesting research groups at &ROXPELD8QLYHUVLW\; the Computer Graphics & User Interface Lab and the Mobile Computing Lab. The first environment con- centrates on different visualisation techniques one of which is user interfaces for aug- mented reality, that is wearable computers. The group works on two augmented reality systems for use in structural engineering and architectural applications. One of the main point is that the overlaid material is tied to the physical world and follows the movements of the user’s head and body. The general challenges of the Mobile Computing Lab are the development of small, portable computers and wireless communication. In detail, the Mobile Computing Lab works on the following problems: (1) operation in spite of discon- nection and/or intermittent connection, (2) efficient use of limited bandwidth (wireless links current operate only at kilobit-to-megabit speeds), (3) dynamic service location and (4) dynamic load balancing (to improve latency and throughput). • The International Workplace Studies Program (IWSP) at &RUQHOO 8QLYHUVLW\ conducts research on the ecology of new ways of working. The intent of the research is to help orga-

- 9 - Literature study: Wearable Control Room

nizations gain a competitive advantage by better understanding how innovative workplace strategies contribute to the effectiveness of: (1) individuals, (2) teams and (3) the organiza- tion as a whole.

5HIHUHQFHV Anderson, J. R. 1995. Cognitive Psychology and its Implications. New York: W. H. Freeman and Company. Austad, A. and Pedersen, E. M. 1996. Use of a Robotic Manipulator and Stereoscopic Vision Laparoscopic Surgery. Diploma at Department of Engineering Cybernetics, NTNU. Bass, L. 1997. Conveners report of CHI’97 Workshop on Wearable Computers, Personal Com- munication to attendees. (http://www.bham.ac.uk/ManMechEng/ieg/w1.html) Best, J. B. 1992. Cognitive Psychology. West Publishing Company. Brooks Jr., F. P. 1992. Walkthrough Project: Final technical report to National Science Founda- tion Computer and Information Science Engineering. UNC-CH Computer Science Technical Report #TR92-026. Feiner, S., MacIntyre, B. Haupt, M. and Solomon, E. 1993. Windows on the world: 2D win- dows for 3D augmented reality. Proceedings of ACM Symposium on User Interface Software and Technology, Atlanta GA, 145-155. Gibson, J. J. 1979. The Ecological Approach to Visual Perception. Boston: Houghton Mifflin. Gibson, J. J. and Walk, R. D. 1960. The visual cliff. Scientific American. 202, 64-71. Hoff, W. A., Lyon, T. and Nguyen, K. 1996. -based registration techniques for augmented reality. In: Proceedings of Intelligent Robots and Computer Vision XV. Bos- ton, Massachusetts, Nov. 19-21. 2904, 538-548. Hollnagel, E. 1991. What is a man that can be expressed by a number? Proceedings of the International Conference on Probabilistic Safety Assessment and Management, 501- 506. Kelso, J. A. S. 1995. Dynamic Patterns. MA: MIT Press. Kortuem, G. 1998. Some issues in the design of user-interfaces for collaborative wearable computers. Position paper for the VRAIS’98 Workshop on Interfaces for Wearable Computers. Kortuem, G. 1996. Software technologies for wearable computers. Position paper. Mann, S. 1997. Smart Clothing: The Wearable Computer and WearCam. Personal Technolo- gies. (1). Mann, S. 1997. Wearable computing: A first step toward personal imaging. Computer (2).

- 10 - Literature study: Wearable Control Room

Najjar, L. J., Thompson, J. C. and Ockerman, J. J. A wearable computer for quality assurance inspectors in a food processing plant. Digest of papers: The First International Sympo- sium on Wearable Computers, 163-164. Newell, A. and Simon, H. A. 1972. Human Problem Solving. Englewood Cliffs, NJ: Prentice- Hall. Rasmussen, J. 1983. Skills, rules, and knowledge; signals, signs, and symbols, and other dis-

tinctions in human performance models. ,((( 7UDQVDFWLRQV RQ 6\VWHPV 0DQ DQG &\EHUQHWLFV. (3), 257-266. Rasmussen, J. 1986. Information Processing and Human-Machine Interaction: An Approach to Cognitive Engineering. North-Holland. Sheridan, T. B. 1986. Forty-five years of man-machine systems: History and trends. Proceed- ings of 2nd IFAC Conference on Analysis, Design and Evaluation of Man-Machine Systems 1985. Smith, B., Bass, L. and Siegel, J. 1995. On site maintenance using a wearable computer sys- tem. CHI’95 Interactive Posters. Siegel, J., Kraut, R. E., John, B. E. and Carley, K. M. 1995. An empirical study of collaborative wearable computer systems. CHI’95 Proceedings. Thompson, C. Ockerman, J. J., Najjar, L. J. and Rogers, E. Factory automation support tech- nology (FAST): A new paradigm of continuous learning using a wearable. Digest of papers: The First International Symposium on Wearable Computers, 31-38. Webster, A., Feiner, S. MacIntyre, B. Massie, W. and Krueger, T. Augmented reality in archi- tectural construction, inspection and renovation.

85/DGGUHVVHVIRUXQLYHUVLWLHVDQGUHVHDUFKLQVWLWXWLRQV Birmingham University, Industrial Ergonomics Group (http://www.bham.ac.uk/ManMechEng/ ieg/ieg2.htm) Carnegie Mellon University, Wearable Computer Systems (http://www.cs.cmu.edu/afs/ cs.cmu.edu/project/vuman/www/home.html) Chalmers University of Technology, Department of Production Engineering (http:// www.pe.chalmers.se/) Columbia University, Computer Graphics & User Interface Lab (http://www.cs.columbia.edu/ graphics/) Columbia University, Mobile Computing Lab (http://www.mcl.cs.columbia.edu/) Cornell University, The International Workplace Studies Program (IWSP) (http://

- 11 - Literature study: Wearable Control Room

iwsp.human.cornell.edu/) Darmstadt University of Technology, Department & Virtual Reality, Fraunhofer Project Group for Augmented Reality at ZGDV (http://www.igd.fhg.de/www/igd-a4/ar/ ) The French National Institute for Research in Computer Science and Control (INRIA), Syntim Research Group (http://www-syntim.inria.fr/syntim/syntim-eng.html) Massachusetts Institute of Technology, Wearable Computing home page (http:// lcs.www.media.mit.edu/projects/wearables/) University of North Carolina (http://www.cs.unc.edu/~walk/) University of Oregon, Computer & Information Science, Wearable Computing Research Group (http://www.cs.uoregon.edu/research/wearables/) University of Washington, Human Interface Technology Lab (http://www.hitl.washington.edu/ )

- 12 -