日 本 ロボ ッ ト学 会 誌Vol.24 No,5,pp.573-575.2006 573

解 説 Humans in Real and Virtual Space

•\ Studies of Interaction and Collaboration Mediated by Informa- tion Technology•\

Ruzena, Bajcsy*1, Klara Nahrstedt*2 and Lisa Wymore*3 *1EECS Department , University of California, Berkeley *2CS Department. University of Illinois, Urbana-Champaign *3 Theater . Dance and Performance Studies Depaituient, University of California. Berkeley

1. Introduction In this paper we will present Information and Technology as a tool for interaction and col- laboration. Then we will explore how this technology is influencing art and use Dance as an example. Finally we will show how Artists/dancers challenge technology. This work is supported by NSF Grant SCI-0549-231 and by CITRIS at UC Berkeley.

First, we will explain the technology which drives Fig. 1 Tele-immersion setup: The system consists of cam- this collaboration. It is called Tele-Immersion. Telc- era clusters and display system immersion is an emerging technology that enables coop- erative interaction between geographically distributed sites. This is achieved through realistic reconstruction of the scenes and people in them in virtual space in real time. The system consists of three major components.

2. Data Acquisition Component

Multiple cameras that capture a 360 degree view of the scene and the activity that happens in the scene. These images have to be processed in such a way that the surrounding views are transformed in the cloud of dense three dimensional points in real time that verita- Fig. 2 Overview of Tele-immersion bly represent the people acting in their environment. It is expected that each site has such data, acquisition polarized glasses and the three dimensional objects capabilities. can be manipulated with a wireless Three dimensional Fig. 1 shows the laboratory setup at the University of mouse. California, Berkeley and the overview of Tele-immersion The challenge here is to create the real time display; is in Fig. 2. hence there is not much time to perform surface fitting and other enhancement algorithms. Furthermore it is in 3. Interactive Display this component where also integration of different data We have stereo display which can be viewed through sets takes place including the integration of the three dimensional information from different sites.

原 稿 受 付 2006年4月2111 キ ー ワ ー ド:Tele-immersion,Virtual Rea.lity, Rea1-time Com- 4. Data Transmission puter Vision, b4ultimedia-network-protocols, Information Technology and Dance *1,3Berkeley In order to connect geographically distributed sites , CA 94720, USA *2Urbana , IL 61801-2302, USA we need to consider the throughput of large data sets

日本 ロ ボ ッ ト学 会誌24巻5号 11 2006年7月 574 Ruzena Bajcsy Klara Nahrstedt Lisa Wymore This technology also enables us to explore the ef- fect of Computer games which always have been about tele-presenceand teleportation. Bruno Latour [14]sug- gested that machines and humans form a collective and are continuously acting together in an associative chain of relationships that is only interrupted as people move to different levels of complexity in the process. The interactive capabilities of this technology allow us in- crease and interchange dynamically the role between the Fig. 3 Group interactions in Tele-immersion meeting in viewer/audience and the performer/artists. Perhaps the virtual space this is the most exciting part of this human/machine symbiosis. via networks, including several switches, since we are Needless to say. that bringing this technology to audi- using the common networks for this purpose. The band- ences exposes the technological inadequacies and hence width, delays are all issues that need to be addressed stimulates improvements and perhaps radical rethink- via quality of services protocols. The tradeoffs between ing of the technology. This is the best feedback we can the computation needed for data compression and de- get. compression versus sending the uncompressed data is References our challenge. There are several other groups [1]- [10] who have [1] J. Lanier:•gVirtually there,•hScientific American, pp.66-75, 2001. build data acquisition capabilities such as ours or sim- [2] J. Mulligan, V. Isler and K. Daniilidis:•gPerformance evaluation ilar. Also some other groups made the network con- of stereo for tele-presence.•hProceedings of the Eighth IEEE In- nection between supercomputing center and Japan as ternational Conference on Computer Vision (ICCV'01), vol.2, p p.5.58 -565. 2001. an example, but with a dedicated network connec- [3] J. Mulligan. V. Isler and K. Daniilidis•gTrinocular stereo: a tion [11][12]. As far as we know we are the first group real-tine algorithm and its evaluation,•hInternational Journal of Computer Vision, vol.47, no.1-3, pp.51 61, 2002. to, utilize all off the shelf components and the regular [4] J. Leigh.. A. Johnson, M. Brown, D. Sandin and T. DeFanti: "Visualization in teleimmersive environments Internet2 network. ,•hComputer, vol.32, no.12, pp.66-73, 1999. We have performed preliminary experiments with our [5] A. Hilton, D. Beresford, T. Gentils, R. Smith. W. Sun and collaborator a dancer making a performance locally J. Illingworth:•gWhole-body modelling of people from multi- view images to populate virtual worlds,•hVisual Computer: and also distributive. Fig. 3 demonstrates this meeting International Journal of , vol.16, no.7, in Virtual space. pp.411-436, 2000. From these experiments following questions have [6] R. Raskar, G. Welch, NI. Cutts, A. Lake, L. Stesin and H. Fuchs: "The office of the future: A unified approach to image -based risen: modeling and spatially immersive displays,•hProceedings of How does the dancer feel seeing herself in the Virtual ACM SIGGRAPH, pp.179-188, 1998. [7] P. Kauff and O. Schreer:•gAn immersive 3d video-conferencing world, and at the same time in the physical world? system using shared virtual team user environments,•hACM Is Technology in this case a powerful actor, or pow- Conf. on Collaborative Virtual Environments, 2002. [8] H. Baker. D. Tanguay, I. Sobel, D. Celb, M. Gross, erless actor or collaborator? W. Culhertson and T. Malzbender:•gThe coliseum immersive What is real/physical and virtual`? teleconferencing system.•hInternational Workshop on Immer- There seem to be some ecology between these two sive Telepresence, 2002. [9] G. Cheung, T. Kanade, J. Bouguet and M. Holler:•gA real states though ecology does not necessarily mean har- time system for robust 3d voxel reconstruction of human mo- mony or an ideal balance of forces and energies that tions.•hIEEE Conf. Computer Vision and Pattern Recognition, pp.714-720, 2000. sustain the connections between humans and machines. [10] R. Sara and R. Bajcsy:•gOn occluding contour artifacts What is the distance between the viewer and the iln- in stereo vision,•hIEEE Conf. Computer Vision and Pattern ages? According to Burnet [13]this will be diminished Recognition, pp.852-857, 1997. [11] Y. Ohno. J. Nliura and Y. Shirai:•gTracking Players and Es- when the images become physical and the bodies be- timation of the 3D Position of a Ball in Soccer Games,•hPro- ceedings of International Conference on Pattern Recognition come holographic. We have the opportunity to test this (ICPR'00), vol.1, 2000. in our laboratories. [12] A. Mankin, L. Gharai, R. Riley, NM.P. Maher and J. Flidr:•gThe

J RSJ Vol.24 No.5 12 July, 2006 Humans in Real and Virtual Space 575

Design of a Digital Amphitheater,•hProceedings of the 10th In- ages in the age of new media and digital revolution. MIT Press, ternational Workshop on Network and Sup- 2005, port for Digital Audio and Video (NOSSDAV 00). 2000. [14] B. Latour: Reassembling the social fabric:An Introduction to [13] R. Burnett: How Images Think: The transformation of im- Actor-Network-Theory. Oxford University Press, 2005.

Ruzena Bajcsy Klara Nahrstedt Dr. Ruzena Bajcsy is a professor in the Klara Nahrstedt is a full professor at Electrical Engineering and Computer Sci- the University of Illinois at Urbana- ence Department at the University of Cali- Champaign, Computer Science Depart- fornia, Berkeley. She was Director of CIT- ment. Her research interests are directed RIS at the University of California, Berke- toward multimedia distributed systems, ley between 2001 and 2005. She was As- quality of service (QoS) management in sistant Director of the Computer Information Science and wired and mobile ad hoc networks, QoS routing, QoS-aware Engineering Directorate (CISE) at the National Science resource management in distributed multimedia systems. Foundation between 1998 and 2001. She was a professor QoS-aware middleware systems, Quality of Protection in of computer science and engineering and Director of Gen- multimedia systems, and tele-immersive applications. She eral Robotics and Active Sensory Perception Laboratory, is the recipient of the Early NSF Career Award, the Ju- which she founded in 1978, at the University of Pennsylva- nior Xerox Award, and the IEEE Communication Society nia. Dr. Bajcsy is a pioneering researcher in machine per- Leonard Abraham Award for Research Achievements. She ception, robotics and . She has done is the editor-in-chief of the ACM/Springer Multimedia Sys- seminal research in the areas of human-centered computer tems Journal, and the Ralph and Catherine Fisher Profes- control, cognitive science, robotics, computerized radiolog- sor. Klara Nahrstedt received her BA in mathematics from ical/medical image processing and artificial vision. She is Humboldt University, Berlin, in 1984, and M.Sc. degree in highly regarded, not only for her significant research con- numerical analysis from the same university in 1985. She tributions, but also for her leadership in the creation of a was a research scientist in the Institute for Informatik in world-class robotics laboratory, recognized world wide as a Berlin until 1990. In 1995 she received her Ph.D. from the premiere research center. She is a member of the National University of Pennsylvania in the Department of Computer Academy of Engineering, as well as the Institute of Medicine. and Information Science. She is the member of ACM and She is especially known for her wide-ranging, broad outlook IEEE. in the field and her cross-disciplinary talent and leadership in successfully bridging such diverse areas as robotics and Lisa Wymore artificial intelligence, engineering and cognitive science. Dr. Lisa Wymore began her graduate study Bajcsy received her master's and Ph.D. degrees in electrical at the University of Illinois, Urbana- engineering from Slovak Technical University in 1957 and Champaign, where she was awarded a Cre- 1967, respectively. She received a Ph.D. in computer sci- ative and Performing Arts Fellowship, an ence in 1972 from , and since that time Outstanding Achievement Award, and a has been teaching and doing research at Penn's Department hloe Family Award for her creativity. After of Computer and Information Science. Prior to her work at the University of Pennsylvania, she taught during the graduating with an M.F.A. in Dance in 1998, she moved to Chicago and began her career as dancer, choreographer, and 1950s and 1960s as an instructor and assistant professor in teacher. She has danced with numerous dance companies as the Department of Mathematics and Department of Com- well as independent choreographers. For her choreography, puter Science at Slovak Technical University in . Wymore has been twice awarded Illinois Arts Council Fel- In 2001 she received an honorary doctorate from University lowships, and has been awarded several Community Arts of Ljubljana in . In 2001 she became a recipient of Assistant Program Grants from the Chicago Department the ACM A. Newell award. of Cultural Affairs She is working on a project with Viet- namese performance artist in the next two years. Wymore is the Co-Artistic Director of Smith/Wymore Disappearing Acts; a dance-theater-performance group based in San Fran- cisco. Wymore is now an Assistant Professor at the Univer- sity of California Berkeley in the Department of Theater, Dance, and Performance Studies where she is collaborating with Ruzena Bajcsy on a project utilizing Tele-Immersion technology. The project is tentatively called the Resonance Project. Wymore is also in the process of finishing her La- ban/Bartenieff Movement Analysis Certification this coming year.

日本 ロ ボ ッ ト学 会 誌24巻5号 13 2006年7月