TOUCH SPACE: AN EMBODIED COMPUTING MIXED REALITY GAME SPACE

Wang Weihua, Xubo Yang, Adrian David Cheok, Mark Billinghurst, and Hirokazu Kato National University o/Singapore, Human Interface Technologies Lab University o/Washington, and Hiroshima City University

Abstract: This paper presents a novel computer entertainment system which recaptures human touch and physical interaction with the real-world environment as essential elements of the game play, whilst also maintaining the exciting fantasy features of traditional computer entertainment. Our system called "Touch-Space" is an embodied (ubiquitous, tangible, and social) computing based mixed reality (MR.) game space which regains the social aspects of traditional game plays. In this novel game space, the real-world environment is an essential and intrinsic game element and the human's physical context influences the game play. It allows tangible interactions between players and virtual objects, and collaborations between players in different levels of reality. Thus, the system re-invigorates computer entertainment systems with social human-to-human and human-to-physical touch interactions.

Key words: embodied computing; social computing; mixed reality; game space

1. Introduction

Humans, as social creatures fmd physical interaction, touch, and human• to-human presence essential for the enjoyment of life [Bowlby 83]. Computer entertainment also can provide humans with enjoyment, by allowing virtual fantasy and imaginative play activity to be carried out. However, present computer entertainment focuses the user's attention

The original version of this chapter was revised: The copyright line was incorrect. This has been corrected. The Erratum to this chapter is available at DOI: 10.1007/978-0-387-35660-0_65 R. Nakatsu et al. (eds.), Entertainment Computing © IFIP International Federation for Information Processing 2003 464 W. Weihua, X Yang, A. Cheok, M. Billinghurst, H. Kato. and natural interactions such as gestures, body language and movement, gaze, and physical awareness are lost. Seeing these inadequacies of present computer games, it is interesting to fmd an approach to retain the exciting elements brought by computer games, as well as simultaneously regaining the natural physical world interactions embedded in pre-computer games. Our research work proposes a novel approach to create such a novel computer game space by employing the theory of embodied computing together with mixed reality. Our game space is termed Touch-Space. In Touch-Space, games are situated and carried out in the physical world, and they recognize the physical co-location of players and objects in the world as essential elements of the game mechanics. Players must walk around within a large room-size area and pick up real objects to physically interact with the game space, in the similar way as they are playing a traditional non-computer game. What enhances the physical space is that the real object and real environment may be augmented with virtual objects or virtual figures. Thus, the benefits and excitement of computer entertainment is also incorporated into the physical space. The system supports multiple simultaneous participants playing together, while maintaining the social, non-mediated interaction between players. Through a seamless traversable interface, the players can transit to and from fully immersive virtual environment. Thus, players will experience a novel full spectrum game experience ranging from physical reality, augmented reality, to virtual reality, in a seamless way, which is featured with tangible interfaces and social interaction.

2. Related Works

In this section we will discuss previous research projects that are related to our work. AR2 Hockey [Ohshima 98] allows two users to hit a virtual puck on a real table where they can see each other through a HMD. AquaGuantlet [Tamura 01] is a multi-player game where players fight with strange invaders coming from the virtual world through some egg-shape objects into the physical game space. The Magic Book [Billinghurst 01] uses a book metaphor to demonstrate the seamless transitions between augmented reality and virtual reality. The "Pirates!" game [Bjork 01] implements a multi-player mobile game on PDAs with proximity sensing technology to incorporate player's contextual information as important elements of the game mechanics. It also emphasizes the social non-mediated interaction between players by taking the co-location between players as one of intrinsic elements of the game. Touch Space: An Embodied Computing Mixed Reality Game Space 465

Our Touch-Space draws ideas from the Magic Book for the seamless transitions in MR space, and draws ideas from "Pirates!" to integrate the contextual and social information into the game space. However, it differs from the existing game interfaces distinctly in the following manner: AR2 Hockey and AquaGuantlet are only AR games which have no transition between AR and VR, and the players are still focused on digital objects rather than exploring the physical environment together with other cooperators. However, Touch-Space provides a full spectrum of the game experience in physical reality, AR, and VR spaces through a seamless transition interface. Furthermore, players explore the physical game space together, and use the physical world information to interact with the game space. In Touch-Space human-to-physical world and human-to-human interaction is central and essential. "Pirates!" uses the contextual information in a physical space to interact with a 2D game on a small PDA screen. Thus, the players have to mentally map between the small 2D game context and the 3D physical space. However, Touch-Space directly deploys the physical space information in a 3D mixed reality space with tangible interfaces, and the game context is directly associated with the physical game space without an indirect mental mapping process being required. In essence the virtual objects are tangible and become naturally embedded in the real environment. The Magic Book uses a book metaphor, so that the collaborations are carried out in a small-scale, close-up configuration. In this system, body movement and other large-scale contextual information is not of concern. However, Touch-Space explores a MR space with a room-scale size and movement about this space is essential. Thus a primary concern is the contextual information in a large-scale space, and how to merge the human• to-physical interaction, human-to-human interaction and human-to-virtual interaction in a large-scale configuration. For example, the players walk around in the game space, using their locations and co-locations to interact with the system and to keep close social contact with other players. They also physically interact with real and virtual objects in the space.

3. Technology and Background Theory

In order to maintain the exciting fantasy features of traditional computer entertainment in the physical game space, digital objects must seamlessly appear in the physical world. Touch-Space achieves this by using augmented reality techniques. Azuma [Azuma 97] has defined three characteristics that are integral to an augmented reality interface. First, it combines the real and the virtual. Second, it is interactive in real time. Third, it is registered in 466 W. Weihua, X Yang, A. Cheak, M. Billinghurst, H. Kata. three dimensions. Mixed reality covers the full Reality-Virtuality spectrum proposed in [Milgram 94]. By using the mixed reality techniques, Touch• Space supports a full spectrum game experience, ranging from physical reality, to augmented reality, and to virtual reality In order to have human touch and physical interaction with the environment as an essential feature of the game play, the theory of embodied computing is applied in the system. In this way, it moves the computer interface away from the traditional keyboard and mouse and into the environment, supporting a more interactive behaviour. Combining mixed reality with embodied computing allows us to create a rich physical environment, where digital 3D objects are embedded and are manipulated directly in a collaborative manner with natural interactions in the physical world. Embodied computing [Dourish 0 I] is a next generation computing paradigm which involves the elements of ubiquitous computing [Weiser 91], tangible computing [Ishii 97], and social computing [Suchman 87]. It places computation and interaction throughout and with the environment, as well as incorporating the sociological organization of interactive behavior. Ubiquitous computing provides advantages for creating a novel game space as it provides technology that is designed to be embedded in the natural human environment which responds to people's needs and actions in a contextual manner [Abowd 00]. Tangible computing directly links the digital world and the physical world and allows the computational world to engage and employ our physical and tactile skills which we are intimately familiar with. It also provides a seamless method of allowing natural physical interaction and collaboration between people. Social computing allows advantages for the novel game space by incorporating the integration of people's interaction with technology that does not follow formal theoretical abstracts or procedures but are improvised naturally in real-time. In Touch-Space, tangible interaction is combined with mixed reality to allow tangible mixed reality interaction. Thus the digital world can be directly manipulated using natural hand interaction, and as the tangible objects are distributed over a wide space, full body movements and movement together with the other players is essential to the game interaction. As will be detailed below, the players must walk around within a large room-size area and pick up real objects to physically interact with the 3D digital objects that are directly and seamlessly tied to the physical objects. Furthermore digital 3D landscapes are seamlessly integrated into the movement ofa person's hand with a wand. It can be seen that ubiquitous computing deals with computing in the environment and with activities that take place in the context of the environment. Tangible interaction deals with using the physical world and Touch Space: An Embodied Computing Mixed Reality Game Space 467 physical object manipulation to interact with the digital world. Both share the view that interaction with computers should exploit our natural familiarity with the physical environment and physical objects. Both tie the computer interaction with physical activities in such a manner that the computer is embedded in the activity. In this way, the environment and physical objects become the computer interface. Touch space uses the social computing concepts to allow a natural physical and social interaction between players. Furthermore natural interactions such as gestures, body language and movement, gaze, and physical awareness are always visible in the space. The system also allows multiple simultaneous participants playing together in a human-to-human social manner, with non-mediated interaction between players. Furthermore, as will be detailed below the system even supports a high level of social interaction between different spaces. Through a seamless traversable interface, the players can transit to and from fully immersive virtual environment. Even when one person is in virtual reality, and the other is in physical reality, both can interact and view each other's movements.

4. Touch Space Interface

In this section, we will introduce the Touch-Space interface through details of the game carried out in it. The game has three stages, in which two players should collaboratively finish some tasks, and then rescue a princess in a castle controlled by a witch. In this game, a princess is captured by a witch and is locked in the witch's castle that is located in a mysterious land. To play the game firstly the two players need to find two map pieces of the mysterious land and other necessary treasures. Secondly, they should fly above the land and look for the castle, after then, they need to fight and defeat the witch. Finally, they enter the castle to find the princess, and thus they complete the game mission. Thus, the game consists of three main game stages which are detailed below: Stage 1: Physical Land Exploration Stage This is an augmented reality experience stage, which allows the users to experience tangible interaction and physical world contextual awareness. In this stage, two players need to collect enough treasures and avoid dangers in the room-size game area. Treasures and dangers may be virtual objects augmented on real objects. Physical movement and interaction is essential, as the players must physically walk around in the room and interact with the objects. Note that this entails that human's physical context directly influences the game play, and the player's bodily movement is an essential 468 W. Weihua, X Yang, A. Cheok, M Billinghurst, H Kato. and intrinsic feature of the game play. When the players obtain all necessary treasures, they finish stage I and thus start the next stage. Stage 2: Virtual Land Exploration Stage This is also an augmented reality experience stage, which stresses the seamless interaction with virtual objects and virtual figures with body movement, and the seamless transitions between AR world and the VR world. In this stage, a virtual land is embedded seamlessly to the physical ground in a ubiquitous manner. The two players need to find the castle in it. They will hold a small virtual 3D window functioning as a see-through lens through which they can see part of a virtual land which appears on the ground. Since they can only see the part of the virtual land corresponding to their current physical body location, they need to walk their body around in the room-size area to be able to find the castle, thus increasing body movement and interaction. Once the players find the castle, they will see the witch flying out of the castle and the small magic window which they hold, and the witch will "jump" into the physical environment where they are located. The witch will fly above the players and throw fireballs to the players. The players need to physically move their body to avoid being hit by the fireballs and at the same time use their virtual gunfire to shoot at the witch. When they defeat the witch, by flying into the 3D window, they will experience a seamless transition from the AR world into the VR world and start the next stage. Figure 1 shows the scrolling 3D landscape when the player is looking for the castle. A small virtual plane is attached to the wand as an avatar for the user, as if the user is exploring the game space by manipulating the plane.

Figure 1. Looking for the castle through a Figure 2. Collaboratively "magic 3D window" fighting the witch As Figure 2 shows, two players can collaboratively battle with a virtual witch hovering around them. When the player presses one button on her wand, she sees that the small plane zooms in toward her until a cockpit occupies her full view. Then she can fly toward the castle as if she is sitting in the plane, which is shown in Figure 3. Thus the player experiences a Touch Space: An Embodied Computing Mixed Reality Game Space 469 transition from AR to VR in a very realistic and seamless manner. It is important to note is that there is a constant social interaction and communication ability between the human players no matter what mode of reality they are in, even if they are in different realities.

Figure 3. Seamless transition from AR to VR Stage 3: Virtual Castle Exploration Stage This is a VR experience stage, featured with fully immersive VR navigation. In this stage, the players will see they are in the virtual castle and they need to navigate in it trying to find the princess. In this case the players can see each other as avatars in the VR environment. A very interesting feature of this stage is that if one player has entered the VR environment, whilst the other is still in the AR environment (stage 2), the VR player will see the players hand movements represented by an airplane in the sky tied to the AR players hand movements. Furthermore, the AR mode player will see the VR mode player as a small avatar moving around in her 3D window. Thus there is a constant communication of body movements even between different types of realities. With support of above technologies and interfaces, the players can actually experience a full spectrum of embodied interaction with the new mixed reality game space. This demonstrates a natural and seamless interaction between the physical and virtual world, and re-invigorates computer entertainment systems with human-to-human and human-to• physical social interactions.

5. Conclusions and Future Work

Our goal is to explore future MR and wearable computing technologies involving new styles of applications. We envision a new type of game experience that has two main features: integrated Ubiquitous context• awareness and sociality into the computer interaction context, which entails ubiquitous, tangible, and social computing (and thus directly applies the theory of embodied interaction); and a seamless merging of physical world, augmented world and virtual world exploration experience. Furthermore, by providing a seamless transition between physical world and virtual world, 470 W Weihua, X Yang, A. Cheak, M. Billinghurst, H. Kata. the players can enjoy both tangible physical game experience and fantasy virtual game experience. Our visions on the new game experience and the new type of CSCP (Computer Supported Cooperative Play) will bring great challenges and many open issues for the computer entertainment research. Furthermore, we aim that in the future, with the development of the wearable computing technologies, this embodied interaction MR space experience will be able to be carried out in a much larger-scale space, and embedded into our everyday life physical environment. The future of the development will be a large MR collaboration space with wearable, networked sites. For example we could have football field sized games (for example we playa combination "ball" and "shoot-up" game over a football field sized space), or city wide adventure mixed reality entertainment (for example a spy adventure that takes place over a city).

References

[Abowd 00] G. D. Abowd, and E. D. Mynatt, Charting past, present and future research in ubiquitous computing. ACM Transactions on Computer-Human Interaction, 7(1), 29-58, 2000. [Azuma 97] R. T. Azuma, "A survey of augmented reality," Presence, 6(4), pp 355-385, 1997. [Billinghurst 01] M. Billinghurst, H.Kato, I.Poupyrev, The MagicBook: Moving Seamlessly between Reality and Virtuality In IEEE Computer Graphics and Applications, May/June, 2001, pp. 2-4. [Bjork 01] S. Bjork, J. Falk, R. Hansson, P. Ljungstrand, Pirates! - Using the Physical World as a Game Board. Interact 2001, IFIP TCl3 Conference on Human-Computer Interaction, July 9-13, Tokyo, Japan. [Bowlby 83] J. Bowlby, Attachment and Loss, Volume I: Attachment. New York: Basic Books, 1983 [Dourish 01] P. Dourish, Where the Action Is: The Foundations of Embodied Interaction, MIT Press, 2001. [Ishii 97] H. Ishii and B. Ullmer, "Tangible bits: towards seamless interfaces between people, bits and atoms", Proc ACM Con! Human Factors in Computing Systems, CH/'97, Atlanta, Georgia USA, 1997. [Milgram 94] P. Milgram, F. Kishino, A Taxonomy of Mixed Reality Visual Displays. IECE Trans. on Information and Systems, vol. E77-D, no. 12, pp.1321-1329, 1994. [Ohshima 98] T. Ohshima, K. Satoh, H. Yamamoto, H. Tamura: "AR2 Hockey system: A collaborative mixed reality system," Trans. VRSJ, vol.3, no.2, pp.55-60, 1998. [Suchman 87] L. Suchman, Plans and situated actions: The problem of human-machine communication, Cambridge: Cambridge University Press, 1987. [Tamura 01] H. Tamura, H. Yamamoto and A. Katayama:" Mixed Reality: Future Dreams Seen at the Border between Real and Virtual Worlds," Computer Graphics and Applications, vol.21, no.6, pp.64-70. 2001. [Weiser 91] M. Weiser, "The computer for the twenty-first century", Scientific American, 265(3), pp. 94-104, 1991.