Mixed Reality in Virtual World Teleconferencing
Total Page:16
File Type:pdf, Size:1020Kb
Mixed Reality in Virtual World Teleconferencing Tuomas Kantonen (1), Charles Woodward (1), Neil Katz (2) (1) VTT Technical Research Centre of Finland, (2) IBM Corporation ABSTRACT In this paper we present a Mixed Reality (MR) teleconferencing application based on Second Life (SL) and the OpenSim virtual world. Augmented Reality (AR) techniques are used for displaying virtual avatars of remote meeting participants in real physical spaces, while Augmented Virtuality (AV), in form of video based gesture detection, enables capturing of human expressions to control avatars and to manipulate virtual objects in virtual worlds. The use of Second Life for creating a shared augmented space to represent different physical locations allows us to incorporate the application into existing infrastructure. The application is implemented using open source Second Life viewer, ARToolKit and OpenCV libraries. KEYWORDS: mixed reality, virtual worlds, Second Life, teleconferencing, immersive virtual environments, collaborative Figure 1. Illustration of Mixed Reality teleconference: augmented reality. Second Life avatar among real people, wearing ultra light weight data glasses, sharing a virtual object on the table, inside virtual room, displayed in CAVE. INDEX TERMS: H.4.3 [Information System Applications]: Communications: Applications – computer conferencing,The structure of the paper is as follows. Section 2 describes the teleconferencing, and video conferencing; H.5.1 [Information background and motivation for our work. Section 3 explains Systems]: Multimedia Information Systems – artificial,previous work related to the subject. Section 4 gives an overview augmented, and virtual realities. of the system we are developing. Section 5 goes into some explanation of Second Life technical detail. Section 6 gives a 1 INTRODUCTION description of our prototype implementation. Section 7 provides a The need for effective teleconferencing systems is increasing, discussion of results, as well as items for future work. mainly due to economical and environmental reasons Conclusionsas are in the section 8. transporting people for face•to•face meetings consumes lot of time, money and energy. Massively multi•user virtual 3D worlds 2 BACKGROUND have lately gained popularity as teleconferencing environments. There are several existing teleconference systems, ranging from This interest is not only academic as one of the largest virtual old but still often used audio teleconferencing and video conferences was held by IBM in late 2008 with over 200teleconferencing to web•based conferencing applications. 2D participants. The conference, hosted in a private installment of groupware and even massively multi•user 3D virtual worlds have Second Life virtual world, was a great success saving alsoan been used for teleconferencing. estimated $320,000 compared to the expense of having the Each of these existing systems has its pros and cons. conference held in the physical world [1]. Conference calls are quick and easy to set up without other In this paper, we present a system for mixed reality hardware than a mobile phone, yet it is limited to audio only and teleconferencing where a mirror world of a conference room is requires a separate channel e.g. for document sharing. created in Second Life and the virtual world is displayed in the Videoconferencing adds a new modality as pictures of participants real•life conference room using augmented reality techniques. The are transferred but it requires more hardware and bandwidth, real people’s gestures are reflected back to Second Life. Thebeing quite expensive in the high•end. Web•conferencing is participants are also able to interact with shared virtual objects on lightweight and readily supports document and application the conference table. A synthetic illustration of such a setting is sharing but it lacks natural interaction between users. shown in figure 1. We see several advantages of using a 3D virtual environment, such as Second Life or OpenSim among many other platforms, as [email protected] alternative means for real•time teleconferencing and [email protected] collaboration. First, the users are able to see all meeting [email protected] participants and get a sense of presence not possible in a traditional conference call. Second, the integrated voice capability of 3D virtual worlds provides spatial and stereo audio. Third, the 3D environment itself provides a visually appealing shared meeting environment that is just not possible with other means of teleconferencing. However, the lack of natural gestures constitutes IEEE Virtual Reality 2010 a major drawback for real interaction between the participants. 20 - 24 March, Waltham, Massachusetts, USA 978-1-4244-6236-0/10/$26.00 ©2010 IEEE 179 3 RELATED WORK virtual worlds. We call the systemAugmented Collaboration in In our work, virtual reality and augmented reality is combined in Mixed Environments (ACME). similar manner as in the original work by Piekarski et al. [2]. In the ACME system, some participants of the meeting occupy Their work was quite limited in the amount of augmenteda space in Second Life while others are located around a table in virtuality as only position and orientation of users werereal world. The physical meeting table is replicated in Second Life transferred into the virtual environment. Our work focuses on to support virtual object interactions as well as avatar occlusions. interaction between augmented reality and a virtual environment. The people in real world see the avatars augmented around a real Therefore our work is closely related to immersive telepresence world table, displayed by video see through glasses, immersive environments such as [3, 4]. Several different immersive 3D video stereoscopic walls or within a video teleconference screen. conferencing systems are described in [5]. Participants in Second Life see the real world people as avatars Local collaboration in augmented reality has been studied for around the meeting table, augmented with hand and body example in [6, 7]. Collaboration is achieved by presenting co• gestures. Both the avatars and real people can interact with virtual located users the same virtual scene from their respectiveobjects shared between them, on the virtual and physical viewpoints and providing the users simple collaboration tools conference tables respectively. such as virtual pointers. Remote AR collaboration has mostly The main components of the system are: co•located users been limited to augmenting live video such as in [8] or laterwearing video•see•throught HMD, a laptop for each user running augmenting a 3D model reconstructed from multiple videothe modified SL client, a ceiling mounted camera above each user cameras as in [9]. Remote sharing of the augmented virtualfor hand tracking and remote users using the normal SL client. objects and applications has been studied for example in [10]. The system is designed for restricted conference room Our work uses Second Life and the open source implementation environments where meeting participants are seated around a well of Second Life server called OpenSim, which are multi•userlit, uniformly colored table. As an alternative to HMDs, a CAVE virtual worlds, as the virtual environment for presenting shared style stereo display environment or a plain old video screens can virtual objects. Using Second Life in AR has been previously be used. studied by Lang et al. [11] as well as Stadon [12] although their Figure 2 shows how the ACME system is experienced in a work does not include augmented virtuality. meeting between two participants, one attending the meeting in In the simplest case, augmented virtuality can be achieved by Second Life and the other one in real life. It should be noted that displaying real video inside a virtual environment as in [13]. This the system is designed for multiple simultaneous remote and co• approach has been also used for virtual videoconferencing in [14] located users. A video of the ACME system is available at [20]. and augmenting avatar heads in [15]. Another form of augmented 6 IMPLEMENTATION virtuality is avatar puppeteering where human body gestures are recognized and used to control the avatar, either only the avatars face as in [16] or the whole avatar body as in [17]. However, only 6.1 General little previous work has been presented on augmenting Second The ACME system is implemented by modifying the open source Life avatars with real life gestures. The main exception is the VR• Second Life viewer [21]. The viewer is kept backward compatible Wear system [18] for controlling avatar’s facial expressions. with original Second Life so that, even though more advanced features might require server side changes, all major ACME 4 SECOND LIFE VIRTUAL WORLD features are also available when the user is logged in to the Second Life is a free, massively multi•user on•line game•like 3D original Second Life world. virtual world for social interaction. It is based on community The SL client was run on Dell Precision M6400 laptops (Intel created content and it even has a thriving economy. The virtual Mobile Core 2 Duo 2.66GHz, 4GB DDR3 533MHz). Logitech world users, called residents, are represented by customizable QuickCam Pro for Notebooks USB cameras (640x480 RGB, 30 avatars and can take part in different activities provided by other FPS) were used for video•see•through functionality, while residents. Unibrain Fire•I firewire camera (640x480 YUV, 7.5 FPS) was For interaction, Second Life features spatial voice