Optical tracking for QAVE, a CAVE-like Virtual Reality System Domagoj Baričević, Hrvoje Dujmić, Matko Šarić, Ivan ðapić University of Split, FESB Split, Croatia [email protected], [email protected], [email protected], [email protected] the displays are properly aligned, and if the perspective of the Abstract: In this paper we present the optical tracking system for the QAVE (Quake + CAVE) system that we developed based image shown is properly adjusted as the user moves around, on the computer game Quake III Arena. The QAVE system is the immersion will be complete. an immersive CAVE-like virtual reality system based on the The name CAVE is a registered trademark and can open source id Tech 3 game engine (formerly known as the technically only be used for systems that are licensed; this is Quake 3 engine) and the open source computer vision library why the term “CAVE like system” is often used when OpenCV. The QAVE features an optical tracking system that utilizes two cameras and two color markers (one for the head, discussing these types of systems in general. one for the hand). The use of open source software and optical An interesting example of a CAVE like system is CaveUT, tracking makes the system very cost-effective. The QAVE developed by Jeffrey Jacobson and his collaborators [2], [3]. system demonstrates that computer games are a viable platform CaveUT is based on Epic Games' computer game Unreal for scientific research, especially in the development of virtual Tournament and uses Unreal Tournament’s built in support reality applications. for game modifications to create a CAVE like system. This paper presents a game based CAVE like system 1. INTRODUCTION based on id Software’s “id Tech 3” engine (formerly known as the “Quake 3” engine), one of the most widely used In recent years games have been increasingly used for engines to date. The first game powered by the engine was scientific purposes. Games are, by nature, flexible so with Quake III Arena, a direct competitor to Unreal Tournament. some small changes they can be easily put to other uses. At In August of 2005 its source code was released under the the core of a game is what is known as the “game engine”, GPL license and so became available to everyone and the software technology that powers a game. Anything that is enabled many applications including our system, the specific to a particular game such as textures, models, “QAVE”. sounds, music, video, levels, scripts etc. is not part of the Our system, QAVE (Quake + CAVE) is the first CAVE engine. This means that you can have two very different like system powered by the Quake III engine. There is games that are built using the same engine. One of the most another system that combines CAVEs and Quake, Paul obvious uses for game engines is in the field of virtual reality Rajlich’s CaveQuake [4]. However our system (QAVE) and since games are by their nature virtual environments. CaveQuake are diametrically opposed: CaveQuake enables a Virtual Reality (VR) is a field that has long interested regular CAVE system to play the Quake game; on the other scientists, engineers, artists and the general public. Virtual hand QAVE uses the Quake game to create a CAVE like reality has many applications; it is used in engineering to help system. visualize data, simulate complex mechanisms, and so on. It is also used in entertainment, medicine (imaging, surgery), training (pilots, astronauts), psychological therapy (phobias, 2. GAMES IN DEVELOPMENT OF VR SYSTEMS post traumatic stress), etc. The main goal of virtual reality is to immerse a human into Games are of interest in research because they can provide a virtual world. Some of the most immersive systems inexpensive, fast, well rendered, interactive virtual worlds developed to date are the CAVE like systems, named after with realistic physics models. They often have free the first such system developed by the Electronic development tools that can be used to create game content Visualization Laboratory at the University of Illinois at (player models, game maps, etc.) and also to modify the Chicago and demonstrated at SIGGRAPH 92 [1]. The CAVE game. Given that the gaming industry is a multi billion dollar (CAVE Automatic Virtual Environment) is basically a small business, games will only get even more advanced, therefore room where the walls, the floor and the ceiling are computer even better platforms for research [5], [6]. displays. Each of these displays shows the view of the virtual The games most often used in research today are either environment that can be seen from the position of the user’s part of the Quake series or Unreal series of games. These avatar. If the user’s position changes the displays have to games belong to the First Person Shooter (FPS) genre. A FPS change their view so it corresponds to the new perspective. If is a type of action game with a 3D world viewed from the limited. On the other hand Quake III Arena is completely player's perspective. Although the first FPS games only had open source so we have absolute freedom to modify the code the first person view (hence the name), modern FPS games and adjust it to our needs. However this advantage is also its have no such restriction. drawback. CaveUT is built as a regular modification of UT Modern FPS games are based on the client server and therefore it can be (and has been) easily upgraded to new architecture. Although it’s actually one program, the game is versions. On the other hand, the QAVE system has changed logically split into two parts. The server runs the simulation, parts of the game that were not initially modifiable, because while the client provides a user interface. One of the main of that it can't be easily upgraded to new versions. reasons for this separation is support for multiple players. The multiplayer mode of FPS games is the key to creating a CAVE like system. The Quake and Unreal games (like 3. QAVE OVERVIEW most multiplayer FPS games) offer a spectator mode in which players in a multiplayer game can just observe the QAVE is a CAVE like system that uses a modified version action without participating in it. The spectator mode can be of id Software's Quake III Arena computer game as its used in free floating mode where the players can fly the graphics and physics engine. The system allows for real time camera through the world, or in follow mode in which the active perspective correction using an optical tracking spectator follows one of the players in the game and sees the system, allows for up to 30 individual screens, and has world from his/her perspective. This mode is what is used support for datagloves. Head position is used for perspective when creating the multi screen display of a CAVE like correction. Hand gestures in tandem with the relative position system. of the hand in relation to the head are used to control movement inside the virtual environment. In a QAVE (or CaveUT) system there is one computer assigned to each screen of the display, there is also a computer assigned to the user. Each computer runs an instance of the game, one computer (usually the user's) starts a multiplayer game and acts as the server and the other computers connect as clients. The game instance controlled by the user represents the user's avatar; the clients all switch to spectator mode and select the avatar as the player to follow. So far all this can be accomplished with no changes to the game and at this point all the screens would be showing the same view. However by making some modification to the game the views can be rotated so that, for example, in a three wall CAVE, one view is looking forward, one is rotated 90 degrees left, an the last one is rotated 90 degrees to the right. To be fully immersive a CAVE like system must maintain the proper perspective for all of the different parts of the display. Simply rotating the view will only provide proper perspective if the user is in the very middle of the display. To allow for correct perspective from other positions CAVE like systems have to provide perspective correction. The position of the player has to be known or determined and the view frustum (the portion of the world that is rendered) has to be adjusted accordingly. This basically turns a static screen into an active window into the virtual world. In CaveUT adjusting the frustum is done by using a library (VRGL) that acts as an intermediary between the game and the OpenGL library, in QAVE the game itself has been modified to allow perspective correction. QAVE and CaveUT share many of the same traits, most notably the low cost; however our system (QAVE) has one major advantage over CaveUT. Unreal Tournament is only partially open source so the ability to change the game is Figure 1 - Construction used during development Figure 1 shows the construction used during QAVE’s of the QAVE user. QDC then calculates the perspective development. The physical and logical structure of the QAVE correction and passes the results to the QAVE Arena (QA) system is shown in Figure 2. doing the rendering for that screen, the QA then adjusts the The QAVE system is comprised of three programs: QAVE rendering of the image for its screen. The corrections are Control Center, QAVE Display Controller and QAVE Arena. field of view angles for four directions (up, down, left, and right) and are sent to the screen’s QA. As the name implies, the QAVE Control Center (QCC) controls the functioning of the QAVE, more precisely it QAVE Arena (QA), a modified version of Quake III accepts and interprets gestures from the user's dataglove and Arena, is the hart of the QAVE system. It runs the simulation, also performs motion tracking. QCC is the only part of the renders the virtual environment and even allows connections QAVE system that is directly controlled by the operator and between multiple QAVEs. There are three roles that may be the user. The user interacts with QCC (and therefore the played by QA, running in a number of instances on different QAVE system) by means of a dataglove and the tracking computers. The first role of QAVE Arena is the user’s avatar system. Currently, QCC supports 5DT datagloves but it can and deals with the actual actions within the virtual easily be reprogrammed to use other datagloves or other environment; the second role is the QAVE display, these types of input devices. The QCC’s tracking system uses the QAs are in charge of the rendering; the third role is the role OpenCV computer vision library [7], of the server that actually creates the virtual environment. The QAVE Display Controller (QDC) acts as an Usually the user’s QA will also act as the server and the intermediary between QCC and QA and is the program that display QAs will be the clients but this is not necessary. Any performs the calculations necessary for perspective instance of QA can be a server, even those that are not part of correction. It was created to minimize the changes done to the the QAVE system itself, this means that somebody inside a code of the game itself. Every screen has an instance of QDC QAVE system can connect to any QA anywhere in the world. paired with that screen’s QA. Both programs run on the same Furthermore this allows several different QAVE systems to computer. Since each instance of QDC is assigned its own be connected to each other so that different people from screen it has to be configured with the screen’s position, across the world can be in the same virtual environment. QDC receives messages from the QCC via a network QA is controlled by messages received from the QAVE connection. One of these messages contains the head position Control Center and the QAVE Display Controller.
QAVE Arena (screen)
QAVE Display Controller
QAVE user
QAVE Arena (server)
QAVE operator
LAN QAVE Control Center QAVE Arena (user)
LAN/Internet Figure 2 - QAVE logical and physical diagram Quake III Arena has what are known as console variables. The QAVE's tracking system uses the open source The console is used by players to change the game's settings OpenCV library originally developed by Intel [7]. The library and status while it is running; this is done by issuing implements several computer vision algorithms including commands and setting variables. QAVE Arena has defined many tracking algorithms; the one used by the QAVE is the several console variables in addition to those it inherited from CAMSHIFT (Continuously Adaptive Mean Shift) algorithm Quake III Arena. Some of these variables are used to [8]. In the QAVE system the object to track is determined by configure QA with the orientation of the screen; others are its histogram. The histogram is specified by the QAVE used to adjust the perspective correction, to rotate the whole operator selecting the object (head and hand in our case) on display when necessary and so on. The configuration the image captured by the camera. variables are set by the operator manually but the variables After the head and hand have been found on the camera for perspective correction, display rotation, head position and image it is necessary to calculate their real world positions. relative hand position are set by QA automatically when it QCC is configured with the cameras’ position, orientation gets a message from QDC specifying the new values. and field of view. The system knows the location of the head and hand in the image coordinates. To determine the real world (3D) coordinates a fairly simple geometric calculation 4. QAVE MESSAGES is performed. Object positions are determined in the following way. The The QAVE system is made up of multiple components CAMSHIFT algorithm lets us know where the object appears which need to work together, this is accomplished by passing to be from the perspective of the camera. Since we also know messages. The different QAVE Arena instances communicate the position and orientation of the camera it is fairly simple to with each other to maintain the virtual world like any FPS determine a line that starts at the camera and passes through and use internal systems inherited from Quake III Arena; they the object. Now we have two lines in space, one for each also accept messages sent by QCC and QDC that are unique camera, and these lines cross where the object is, so to get the to the QAVE system. The QAVE messages are either position of the object we need to find the intersection of the windows messages that are used to send information from lines. Normally, the lines we get will not actually be QCC or QDC to the QA they control (and therefore running intersecting; instead, they will pass very near each other. This on the same computer), or they are network broadcast is not a problem since the method we use to find the messages (UDP datagrams) that are sent from QCC to QDC. intersection actually finds the shortest line segment between Although QCC and an instance of QDC can be running on the lines. The segment is defined by its end points; we the same computer they still communicate via UDP. The flow consider the object to be positioned at the average of these of messages is one way, QCC only sends messages, QDC two points, the length of the segment can be an indicator of sends and receives but only in one direction: from QCC and how precise the position is, the shorter the line, the better. to QA. Although the tracking system is fast and quite precise it is QCC sends QA messages regarding the user’s interaction not always robust. Like a lot of other optical tracking with the environment, specifically it sends movement systems, QAVE’s optical tracking system uses color markers. commands. However, most other applications have stable backgrounds There are four types of messages sent by QCC to the and lighting conditions, with the QAVE this is not the case. QDCs: The screens of the QAVE show bright dynamic images that 1. messages that specify the position of the user’s head can not always be predicted, if a screen is in a camera’s field 2. messages that specify the position of the user’s hand of view and the picture contains the same colors that are used 3. messages that specify view rotation by the markers, the tracking system can easily get confused 4. messages that shut down the QDCs and lose track of the object it was tracking. Luckily, in QDC sends QA messages specifying the perspective systems that do not have fully surrounding displays this can correction, head and hand position (in virtual world units), be resolved by positioning the camera so that it doesn’t face a and view rotation. screen. Another problem with the screens is that they are often the main source of lighting, this lighting is usually colored which can then change the perceived color of objects but that normally doesn’t cause major problems. Low light 5. TRACKING SYSTEM and changes in the intensity of the light are a bigger problem. QAVE uses an optical tracking system based on two Tracking the user’s hand has special problems of its own. cameras and two color markers (one for the head and one for Unlike the head which is positioned above the body of the hand). Details about the previous version of the tracking user and is never hidden by it, the hand is much lower and the system that used a single camera can be found in [9]. user’s body often hides it from at least one camera, the only way to alleviate this problem is to position the cameras in such a way that it is not possible or likely for the hand to be problematic lighting and background conditions that are an occluded by the body. inescapable part of CAVE like systems. QAVE offers a complex interactive virtual environment, and although the system is not perfect in this early stage of 6. USING THE QAVE development, it does provide a viable platform for the development of immersive virtual reality applications. The The QAVE has been designed with a simple but very QAVE system is an open and scalable system, and with efficient user interface; there are only two basic actions the additional development it can be improved and provide a user needs to perform to interact with the system: pointing high level VR experience. and dragging. When moving around inside the QAVE the There are aspects that should be improved upon. One perspective will be automatically corrected but the user will aspect is better tracking. Although upgrades have been made not move inside the virtual environment (the point of view to the tracking system there is still room for improvement, a will, but not the avatar). If the user wants to move somewhere possible way to accomplish this is to move from the visible s/he just has to point in the desired direction and gesture range of the light spectrum to the near infra red. Another whether s/he wants to go forward or reverse (by using the aspect is full stereoscopic support which would require some little finger), fast or slow (using the thumb). Figure 3 shows more game modification but also a way of tracking the user’s the gesture used to move in reverse. With displays that don’t head orientation. This can be accomplished by using an have a full 360 degree view it is sometimes necessary to inertial orientation tracker or by further improving the optical rotate the view, to do this the user simply “grabs” the view, tracking system. drags it around in the direction it needs to be rotated, and then lets it go. REFERENCES [1] C. Cruz Neira, D. Sandin, T. DeFanti, R. Kenyon, J. Hart: “The CAVE®: Audio Visual Experience Automatic Virtual Environment”, Communications of the ACM, vol 35, no 6, pp. 65 72, 1992. [2] J. Jacobson, M. Lewis: “Game Engine Virtual Reality with CaveUT”, IEEE Computer, vol. 38, no. 4, pp. 79 82, 2005. [3] J. Jacobson, M. Le Renard, J. Lugrin, M. Cavazza: “The CaveUT system: immersive entertainment based on a game engine”, Proceedings of the 2005 ACM SIGCHI international Conference on Advances in Computer Entertainment Figure 3 - Glove gesture for reverse movement Technology (Valencia, Spain, June 15 17, 2005). ACE '05. ACM Press, New York, NY, 184 187. [4] CAVE QUAKE II, 7. CONCLUSION AND FUTURE WORK brighton.ncsa.uiuc.edu/~prajlich/caveQuake/ [5] M. Lewis, J. Jacobson: “Game Engines in Scientific In this paper we presented the optical tracking system for Research”, Communications of the ACM, vol. 45, no 1, pp. the QAVE, a CAVE like system based on the id Tech 3 27 31, 2002. computer game engine. We have shown how a game can be [6] Michael Zyda: “From Visual Simulation to Virtual adapted to support the creation of an immersive multi screen Reality to Games”, IEEE Computer, vol. 38, no 9, pp. 25 32, display for a CAVE like system. The id Tech 3 engine has 2005. also been altered to support active perspective correction and [7] Open Source Computer Vision Library, enable control with an external program. QAVE is comprised www.intel.com/technology/computing/opencv/ of three programs running on several computers and in [8] Gary R. Bradski: “Real Time Face and Object Tracking multiple instances, the central program coordinates the as a Component of a Perceptual User Interface”, 4th IEEE functioning of the system by using a simple unidirectional Workshop on Applications of Computer Vision (WACV'98), message passing protocol. The system utilizes head tracking 1998. to facilitate perspective correction, and head and hand [9] D. Baričević, H. Dujmić, M. Šarić: “QAVE – a Game tracking combined with dataglove gesture recognition to based Immersive Virtual Reality System”, Proceedings of the create an easy to use user interface. The optical tracking 18th International Conference on Information and Intelligent system is fast and precise but not always robust due to the Systems, pp. 357, Varaždin, Croatia, 12 14 September 2007.