MIXED REALITY ENTERTAINMENT WITH WEARABLE COMPUTERS FONG SIEW WAN (Master in Engineering, National University of Singapore) A THESIS SUBMITTED FOR THE DEGREE OF MASTER OF ENGINEERING DEPARTMENT OF ELECTRICAL AND COMPUTER ENGINEERING NATIONAL UNIVERSITY OF SINGAPORE 2003 Acknowledgement I would like to express my gratitude to all those who gave me the possibility to complete this thesis. First and foremost, I want to thank the Defence Science and Technology Authority (DSTA) of Singapore for the financial support in this research project. Especially to the officers in-charged, Ms Lilian Ng and Mr Choo Hui Wei, I thank you for all your suggestions and encouragement throughout the project. I am deeply indebted to my supervisor Dr. Adrian Cheok whose help, stimu- lating suggestions and encouragement helped me in all the time of research for and writing of this thesis. My former colleagues from the Digital Systems and Applications lab, Power lab, and Human Interface Technology lab supported me in numerous aspects of my work. I want to thank them for all their help, support, interest and valuable hints. Especially I am obliged to Dr Chen XiangDong, Ms Liu Wei, Dr Simon Prince, Dr Farzam Farbiz, and Mr Goh Kok Hwee. I also want to thank Mr Lee Meng Huang for all his assistance during the initial phase of the project and demonstration. Especially, I would like to give my special thanks to my family whose patient love enabled me to complete this work. i Contents Abstract v List of Figures vii 1 Introduction 1 1.1 Objectives . 3 1.2 Scope . 3 1.3 Research Contributions . 5 1.4 Organization . 8 2 Background: Wearable Computer & Augmented Reality 10 2.1 Augmented Reality . 11 2.2 Historical Context and Fundamental Issue . 12 2.3 Hardware: Commercial and Research Systems . 14 2.3.1 Input Devices . 17 2.3.2 Output Devices . 23 2.3.3 Sensors and Tracking Devices . 27 2.4 Software Applications . 31 2.4.1 Soldier Battlefield Applications . 31 2.4.2 Medical Applications . 33 ii 2.4.3 Manufacturing Applications . 36 2.4.4 Navigation and Tracking Applications . 37 2.4.5 Entertainment Applications . 38 3 Hardware Development of Wearable Computer 42 3.1 Design and Construction of Wearable Computer . 43 3.1.1 Prior Experiences . 44 3.1.2 DSTAR Wearable Computer . 49 3.2 Hardware Design Details . 49 3.2.1 Fabrication of Wearable Computer System . 50 3.2.2 Technical Design Details . 56 3.2.3 Power Consumption of System . 63 3.2.4 Discussion on Problems Encountered . 64 3.2.5 Limitations of Wearable Computer Systems . 66 4 Wearable Computer Applications: Game City and Interactive Theater 69 4.1 Game City: A Ubiquitous Large Area Multi-Interface Augmented Reality Game Space for Wearable Computers . 71 4.1.1 Introduction . 71 4.1.2 Game City Interface . 74 4.2 Interactive Theater . 87 4.2.1 Introduction . 87 4.2.2 Background Theory . 89 4.2.3 Interactive Theater System . 95 5 Application: Human Pacman 103 iii 5.1 Introduction . 104 5.2 Background . 106 5.3 Previous Works . 109 5.4 System Design . 110 5.4.1 Software Details . 111 5.5 Gameplay . 116 5.5.1 Main Concepts: Team Collaboration, Ultimate Game Objec- tives and the Nature of Pac-World . 116 5.5.2 Pacman, Ghost, and Helper . 120 5.5.3 Actual Gameplay . 124 6 Software Design and HCI Issues in Human Pacman 132 6.1 Mobile Service and Ubicomp Issues . 133 6.1.1 Mobile Computing . 134 6.1.2 Ubicomp . 136 6.2 Human Computer Interaction Design in Human Pacman . 141 6.3 Challenges of Wearable Computing Applications . 142 7 Summary 145 Bibliography 151 iv Abstract Computing technology is rapidly advancing for the past few decades; providing us with ever greater computing power, storage capacity, and portability. With the recent proliferation of portable computing devices such as laptop, palmtop, and tablet PC, I am envisioning a future of mobile computing becoming the mainstream technology thereby reducing the desktop to a historical relic. Mobile computing is realized with the employment of a wearable computer. In this thesis, I will describe the design and development of the wearable computer named ‘DSTAR’. This powerful wearable computer is complete with a head mounted display (HMD) with camera attached, a main system of small form factor (PC 104 form factor), and a novel input device (Twiddler2). In addition to that, I have also added a Wireless LAN card, an inertial sensory system (InertiaCube2), a Bluetooth device, and a Global Positioning System (GPS) receiver (or a Dead Reackoning Module, DRM, in the later system) to enable the wearable computer to support the implementation of augmented reality and networking software applications. Three wearable computing applications are developed: ‘Game City’, ‘Interactive Theater’, and ‘Human Pacman’. These systems support multi-players in a wide outdoor area with total mobility in an attempt to renew traditional physicalness in gameplay in computer entertainment. Tracking and navigation modules are incorporated by overlaying the video stream captured by the head-mounted camera with 2D text or 3D virtual objects. At the same time, wireless LAN is set up to support communication between players so as to explore the various aspect of social gaming. Tangible interaction between physical object and its virtual counterpart is incorporated into the gameplay to provide the player with a new experience of v direct physical interaction with his computer. Also, they can experience seamless transitions between the virtual and the real world in the systems. In the last part of the thesis, various mobile computing problems encountered in the areas of wireless communication, mobility, and portability are described in details. I will also take a look of the theme of ubiquitous computing embedded in our applications. I studied several issues such as tangible interface and context awareness in outdoor environment in the domain of ‘Human Pacman’. Lastly, I present the reader with human computer interaction (HCI) design concerns of the system. Design decisions are justified in adherence to the wisdom from HCI studies. vi List of Figures 2.1 Steve Mann’s WearComps.(Image used with permission, courtesy of Prof. Steve Mann) . 15 2.2 MIT Media Lab Wearable Computer, MIThril.(Image used with per- mission, courtesy of Prof. Alex(Sandy) Pentland, MIT Media Lab) . 17 2.3 Finger Trackball. 21 2.4 Conceptual Diagram of an Optical Seethrough HMD. 24 2.5 Conceptual Diagram of a Video Seethrough HMD. 25 2.6 Referential Commonly Employed in Virtual Reality and Augmented Reality. 29 2.7 Battlefield Augmented Reality System. (Photograph used courtesy of Naval Research Lab) . 33 2.8 A View Through a SeeThrough HMD Shows a 3D Model of Demol- ished Building at Its Original Location. (Image used with permis- sion, courtesy of Prof. Steven Feiner, Computer Graphics and User Interface Lab, Columbia University.) . 38 2.9 AR Game: AquaGauntlet. (Image used with permission, courtesy of MR Systems Laboratory , Canon Inc.) . 39 vii 3.1 MatchboxPC Single Board Computer.(Image used with permission, courtesy of Vaughan Pratt (chairman & CTO, Tiqit computers, Inc.).) 45 3.2 MicroOptical Head Mounted Display. (Image used with permission, courtesy of MicroOptical, Inc.) . 45 3.3 MatchboxPC Wearable Computer System Configuration. 46 3.4 Espresso Wearable Computer System Configuration. 49 3.5 DSTAR Wearable Computer Components. 51 3.6 DSTAR Wearable Computer System Configuration. 51 3.7 DSTAR Wearable Computer as Worn by the Author. 52 3.8 DSTAR Wearable Computer HMD. 53 3.9 Hardware Components Inter-connections. 53 3.10 Battery Pad on Wearable Computer. 54 3.11 Wires are Concealed Inside the Jacket. 55 3.12 Power Regulator at the Back Pocket. 55 3.13 The Motherboard with Its External Connections. 56 3.14 GPS Sensor. 57 3.15 Twiddler in the Front Pocket. 57 3.16 Twiddler2: Keyboard and Mouse. (Photograph used courtesy of Handykey Corporation.) . 58 3.17 Sony NP-F960 Lithium Ion battery. 59 3.18 Design Schematics for Power Supply. 60 3.19 InertiaCube2 Inertial Measurement Unit. (Photograph used with permission, courtesy of InterSense, Inc.) . 62 3.20 Dead Reckoning Module. (Photograph used with permission, cour- tesy of Robert W. Levi (President, Point Research Corporation).) . 63 viii 4.1 Game-City Concept. 75 4.2 Wearable Computer Sensor System for Outdoor Augmented Reality. 76 4.3 Wearable Computer Outdoor Augmented Reality Interface. 77 4.4 Tangible Interaction: Opening a Real Box to Find Virtual Treasures Inside. 79 4.5 TouchSpace Communication System Part (1). 81 4.6 TouchSpace Communication System Part (2). 82 4.7 Communication between Wearable Computer and TouchSpace System. 83 4.8 Looking for the Castle through a “Magic 3D Window”. 85 4.9 Collaboratively Fighting the Witch. 85 4.10 Navigation in VR mode. 86 4.11 Live Human Actor Content Rendered from the Appropriate View- point in Real Time. 95 4.12 Live 3D Viewpoint System. 96 4.13 The Pose of the Head Mounted Camera is Estimated (Bottom Left), and the Equivalent View of the Subject is Generated (Bottom Right) from the Incoming Video Streams (Top). This is then Rendered into the Image (Bottom Left) and Displayed in the Hmd. 97 4.14 Interactive Theater Concept Diagram. 98 4.15 Wearable Computer Outdoor Interface. 99 4.16 Virtual Static Actors in Outdoor Locations. 99 4.17 Hardware and Software Outline and Pseudo-code of Interactive The- atre Algorithm. 100 4.18 Capture of Live Actor. 101 4.19 Real Time 3D Live Display of Actor in Interactive Theatre. 102 ix 5.1 A Player at ParaParaParadise Arcade Game. 109 5.2 Top Level System Design of Human Pacman. 111 5.3 Flowchart of software on server.