Integrating Real and Virtual Objects in Virtual Environments

Mary Whitton1, Benjamin Lok2, Brent Insko3, Fred Brooks1

1University of North Carolina at Chapel Hill 2University of Florida 33Dlabs, Inc. Chapel Hill, NC Gainesville, FL Austin, TX {whitton|brooks}@cs.unc.edu [email protected] [email protected]

Abstract

Integration of real objects into a virtual environment (VE), creating a mixed environment (ME), helps overcome one of VEs greatest weaknesses: nothing to really touch—no tools to hold, no walls to run into. Avatars, particularly self-avatars, are a special class of such real objects. No VE system today supports immediate and infrastructure-free integration of real objects into the virtual scene. Successful integration of real objects into VEs places demands on the VE system in several areas; it must acquire both shape and appearance (textures) models of the real objects, track real objects, merge the representation of the real object into the virtual scene, and simulate appropriate behaviors for situations such as object collisions. We draw on work in our own labs to frame a discussion of different approaches to integrating real and virtual objects, including strengths, weaknesses, and open problems. Keywords: Mixed environments, 3D interaction, cognition in VR, passive haptics, visual hulls, integrating real and virtual, mixed reality, virtual environments, VE systems

1 The Need

This paper examines the challenges of implementing an effective system to integrate virtual objects and representations of real objects in immersive virtual environments (VEs). Our purpose is to present the range of problems that must be solved before we can routinely have systems that will robustly, and without complex, costly, and motion restricting infrastructure, allow us to interact with both real and virtual objects in a virtual environment. Importantly, the general problem includes the special case of integrating avatars into the VE. You have only to consider how disconcerting it is not to see your own body in a VE to understand the need.

We need to see our bodies so that we don’t violate the expectations of seeing ourselves that we have developed in the real world and so we can confirm where are in the VE. We need to see our feet for confident locomotion, including collision avoidance; we need to see our hands so that we can use them to do things. In a VE, you can’t perform fine motor tasks that require visual feedback of your finger positions if the hand model has only three states—full open, midway closed, and fully closed. Having fully functional hands is a startlingly simple goal, but a goal that is startlingly difficult to achieve.

Figure 1. First person view of dial being activated (left). User performing action at passive haptic barrier (middle) and with passive haptic barrier, but not exterior wall, removed (right). Dotted lines added to clarify position of virtual barrier. User posture reflects his lack of confidence in location of virtual dial.

1.1 Fundamental issues in integrating real and virtual The major challenges of integrating representations of dynamic real objects in virtual environments are:

Acquisition: Make representations of real objects look right in the VE. The VE system must know what the real object looks like—both its shape and its appearance. This is a problem in modeling. Modeling is particularly complicated when the real objects are articulated or deformable, as are human bodies, since the shape of the ensemble of segments depends both on modeling the constraints on the motion of individual segments and on accurate tracking of the segments.

Tracking: Know what’s where. The VE system has to know where the real objects are in the real world and it has to know which object is which. This is a tracking problem and a problem of differentiating among objects once the positions are known.

Merging real and virtual: Put the representation of the real object in the right place, at the right time. To avoid cue conflict, real and virtual instances of the same object must be spatially registered. This is a problem of calibration and registration of coordinate systems, and of having sufficient information to resolve occlusion relationships between the purely virtual objects and those with physical correlates. While potentially less noticeable and distracting in immersive VE than in augmented reality, the mis-registration of real and virtual objects due to overall system latency cannot be ignored.

Simulation: Make objects behave properly. This concerns the simulation that is controlling object behaviors in the application. While collision detection and response is the most common simulation model running in VEs, it is only one of many that may control behaviors and interactions of real and purely virtual objects. For our purposes, we consider collision response to include generating substitute sensory cues for haptics that are missing in interactions between real and purely virtual objects or pairs of purely virtual objects.

1.2 Scope

This paper is about mixed environment (ME) systems where real objects are used to augment an immersive virtual environment and users wear head-mounted displays. Objects that are forever out-of-reach of all participants aren’t a concern here, as it is sufficient for them to be purely virtual, and, in this paper, we consider the use of haptics only in its traditional sensory role, and not as a means to display supplemental data. The next sections describe three lab- based systems that provide concrete examples on which to base a discussion of the issues.

2 Integrating Real Objects that have Geometric Models The simplest, oldest, and still popular way of integrating a real object into a virtual scene is to generate a geometric model of it and simply add it, with all its properties, to the scene description. For static objects, this step is all that is required; for objects that can move, a tracker or trackers must be attached to the object and the object’s coordinate system reconciled with that of the rest of the scene.

2.1 The Techniques

Static real objects: passive haptics. Incorporating real representations of static objects was motivated by the desire to provide the user with haptic feedback he “touches” objects in the VE. The idea of passive haptics is to create quickly assembled, relatively inexpensive approximate physical models of real objects such as walls and fixtures and to locate them in the VE system space so that they are registered with the virtual model. The virtual model, viewed by the user in a head-mounted display, supplies the visual details and the physical model provides haptic feedback when the user touches or collides with it.

We have built passive haptics from plywood, fiberboard, and from polystyrene construction blocks. We strive for, but don’t always achieve, ¼” construction and ¼” placement accuracy. Figures 1 and 2 show the passive haptics in use.

Dynamic real objects. Models of real objects become part of the scene description. For proper positions of the virtual representation of the moving objects, their changing locations must be reported to the system, typically from a magnetic, acoustic, or optical tracker attached to the object. The pose of the model, the position and orientation of each of its component parts, is established on a frame-by-frame basis from tracker readings. When a limited number of trackers are available, the real objects are limited in number and complexity.

2.2 Effectiveness

Passive Haptics. Insko showed that the addition of a 1½” plywood ledge corresponding to the virtual ledge in UNC’s Pit VE, Figure 2, caused a statistically significant increase in the rise in heart rate that occurs when users enter the room and find themselves on a ledge high over the room below [Insko01, Meehan02]. He also showed that participants who trained for a blindfolded maze navigation task with a passive haptic representation of the maze navigated the real maze faster and with fewer errors than those trained in a purely virtual environment. Users trained without passive haptics got visual and auditory cues to collisions between their hands and the maze structure. Note that there were audio cues to collisions which were outside the field-of-view of the user.

Over 300 individuals have experienced the Pit either with or without the ReddiForm™ block walls that can guide the user, but not provide real support. It is our observation that feeling the walls, in addition to seeing them, provides users with a powerful confirmation of where they are in the space. With the walls, most users walk along the ledge with modest confidence; without the walls, users often don’t move at all or take only baby steps.

Dynamic Real Objects. Tracked real objects have been used as components in user interfaces, and for applications such as treatment of phobias. An early use of dynamic real objects was to represent the avatar of the user’s (tracked) hand; later the technique was used to track real objects that provide tactile augmentation when user touches the corresponding virtual object [Hoffman96]. Lindeman’s work evaluated the impact of having real objects to support GUIs and widgets within the VE [Lindeman01]. A VE system developed and used at the University of Washington was effective in reducing fear of spiders through controlled exposure to virtual spiders in the virtual environment. Midway through treatment the therapists added a tracked, furry toy spider to the system to provide further increase realism. At the end of therapy, the client’s percentile on a fear-of-spiders scale had decreased from 99th to 71st and, demonstrating transfer of training from VE to the real world, she was able to engage in outdoor activities such as camping [Carlin97].

2.3 Plusses and Minuses of Real Objects with Geometric Models

Acquisition: Look right—shape and appearance. The quality of the shape and appearance of the rendered instance of the real objects depends on the amount of effort and time put into creating the model and appearance data such as material properties, textures generated as artwork, or textures captured with cameras. Modeling de novo is difficult and slow, and does not permit users to take new items into a VE on short notice.

Easy customization of avatar shape and appearance is an issue and goal. Geometric model-based systems require either that the model be generated anew for each person or that a generic model be calibrated to the user’s hand size and shape. In either case, custom textures need to be captured to provide realistic appearance. Avatar-Me is a company in Great Britain that makes systems for quickly generating custom avatars of individuals from four photographs that provide scaling values and custom textures for realistic appearance of clothes and face [Hilton 00].

Tracking: What’s where? Object location is as good as the VE system’s tracking component. As tracker’ reports are individually identifiable, and assuming one tracker per rigid object or per component of an articulated object, there is no difficulty establishing which real object is which and where it is. Object motion and velocity are easily calculated from tracker reports and can be then used as parameters in simulations. The problem is that tracking an object with as many degrees of freedom as a hand requires a large number of trackers, and the components of the tracking devices that are mounted on the hand quickly begin to interfere with natural hand use. Trackers dependent upon line-of-sight between sensor and active or passive beacons are particularly problematic because of the redundancy of devices required to ensure that data sufficient to produce accurate tracking results can be captured.

Figure 2. The UNC Pit virtual environment (left) and a user standing on the passive haptic ledge (right).

In addition to commercial marker-based motion capture systems, there are some lower infrastructure, special- purpose marker-less tracking systems that have ability to directly control articulated bodies. Examples include ShapeWrap II from Measurand, Inc., VRLogic’s 5DT Dataglove, and the Rutgers RM II Hand Master which can not only control hand pose, but also provide haptic feedback. To date, the Dataglove least interferes with natural hand motion.

With few exceptions, either a sensor, marker, or a beacon must be mounted on the real object to track it. Sensor system design always involves trade-offs in ability to separate signal from noise and differentiate one signal from another, in sensitivity to small signal changes, in data update rate, and in such product considerations as size, power consumption, wireless or not, and safety. Additional trade-offs involve sensor, marker, and emitter placement, calibration, and redundancy-robustness.

Merge real and virtual: Be in the right place at the right time. The accuracy of the registration of the real and virtual instances of an object depends on the care spent positioning passive haptics, calibrating trackers, and reconciling coordinate systems among models, real space, and trackers. These tasks are simply hard work and, in the case of trackers, are limited by the characteristics of the sensing devices and beacons themselves, e.g., stability, linearity, and sensitivity. Because geometric models of the real objects can be made part of the scene description, normal rendering with a depth buffer ensures proper occlusions. There is little incremental computational overhead in this type of system, and so system latency increases are negligibly small.

Simulation: Behave properly. Most advanced collision detection and response systems today assume geometric object definitions, so there is no model conversion overhead for this technique. The technique suffers, however, from the universal VE problem of virtual objects not being able to cause physical responses in real objects. Options for mitigating the effects of missing haptic feedback are considered in section 5.

Other issues. Passive haptics are not feasible when the virtual space exceeds the limits of the tracked space of the VE system.

3 Image-based objects and avatars

The ability to extemporaneously add real objects to a VE, with no explicit modeling or a priori tracking infrastructure is the primary benefit of image-based techniques. An approximation of the object’s shape, such as a visual hull, is sometimes sufficient.

3.1 The Technique

A shape-from-silhouette concept, the visual hull for a set of objects is the tightest volume around the objects that can be obtained by examining only the object silhouettes as seen by a set of n cameras [Laurentini 1994]. Visual hull approaches capture a working volume using outside-looking-in cameras, and construct geometric approximations of objects in the scene from the set of images [Matusik00, Matusik01]. Because the space captured by the cameras has been carefully mapped into the coordinate system of the virtual world, the virtual representations of real objects are at known positions in the VE.

The camera images are used for appearance data as well as for shape. An auxiliary camera worn on the user’s HMD can provide nearly view-correct appearance data, which can be mapped onto the objects generated from the images. Shape and appearance are updated as fast as the object reconstruction software can run.

Figure 3. Visual hull of user's hands is used in the cloth simulation model and enables user to part the curtains.

3.2 Effectiveness

Lok hypothesized that a real-time mixed environment system would make user interaction with objects more natural, and that spatial cognitive tasks—tasks that require problem solving while manipulating objects and maintaining mental model of spatial relationships among them—could be performed better if real objects were incorporated into the VE.

Block arrangement study. We performed a study similar to, and based on, the block design portion of the Wechsler Adult Intelligence Scale (WAIS) [Lok 03a]. Participants manipulated four or nine identical wooden blocks to make the top faces of the blocks match a target pattern. We compared participants’ task performances, measured as time to completion, in a purely virtual environment and in two mixed environment systems, using their performancesin real space as a baseline.

For the cognitive manual task we investigated, interacting with real objects provided a substantial performance improvement over interacting with only virtual objects. Although participant task performance in all the VE conditions was substantially worse than in real space, task performance in the mixed environments was significantly better (2-times faster) than for purely virtual environment, and much closer to performance in real space. Handling real objects makes task behavior in the VE more like behavior in the real-world. Even in our simple task, we saw evidence that manipulating virtual objects sometimes caused participants to develop VE-specific approaches. This would be undesirable in a training application.

Interacting with virtual curtains. Figure 3 shows a user parting a set of virtual curtains with his hands. The visual hull representation of his hands is being used as an element in the physically-based mechanical simulation controlling the cloth curtain behavior. Although the system had a low update rate and was visually imprecise, the small group that experienced it almost universally said, “This is how it ought to be; I can just use my hands naturally to cause things to happen” [Lok 03b].

3.3 Plusses and Minuses of Image-Based Objects and Avatars

Acquisition: Look right—shape and appearance. Visual hull systems require not only the infrastructure for capturing and real-time processing of data from multiple cameras, but also require that the user and real objects be placed in a featureless surround so that object silhouettes are easily extracted by image subtraction. Real-time performance requires trade-offs in model update rate, modeling resolution, and reconstruction quality. Lok’s reconstruction system used 4 NTSC resolution cameras and generated a 320 x 240 pixel reconstruction. He estimated latency at 0.3 seconds and with 1 cm reconstruction error. The number of cameras, their placement, and the resolution of their sensors constrain model shape and appearance accuracy.

Improving visual hull models is a goal. In determining shape, purely image-based systems have the problem of inability to resolve self-occlusions, the inability to properly capture concavities, and, without additional data, inability to differentiate multiple objects in a scene, e.g. fingers, particularly when they are self-occluding. Data that can be added to improve scene segmentation into identifiable objects includes markers on the objects or color coding the objects, e.g. wearing a glove with each finger a different color. Both of these solutions add infrastructure and, in a system that uses the real-time images to texture the objects, reduces the realism of the object’s appearance. Advanced visual hull techniques can build very good hulls, but they run slowly and off-line, making them unsuitable as yet for use in VE systems [Matusik02].

Tracking: What’s where? Image-based approaches expand the types of real objects that can be used in a VE to include those not easily tracked with magnetic, optical, or acoustic trackers, e.g. highly articulated and deformable objects. Rather than reply on trackers attached to objects, these approaches identify, on a frame by frame basis, which units of space in the working volume are occupied by the real objects. Essentially, many points on the surface of the real object are tracked. There are two primary drawbacks: there is no inherent way to associate the recovered 3D points with the real object they correspond to, and there is no guaranteed correspondence between the visual hull computed in one frame with that computed in the next. This means that if a visual hull contains two objects, they cannot be differentiated for purposes of associating data such as surface properties or mass, nor can they be individually tracked over time, as may be required for physical simulations.

Merge real and virtual: Be in the right place at the right time. Models constructed from visual hulls may be polygon models, voxel grids, or depth meshes. Models in these formats can be combined with virtual objects using straightforward polygon rendering, depth buffers, or volume visualization techniques that produce correct occlusion relationships between the hull object and a purely virtual object.

A common technique for mitigating the effects of system latency is, at the last minute, to apply a transform based on a prediction of where the object is, or will be when the frame rendering is complete. Since a position and velocity history is not inherently available for individual objects represented by a visual hull system, prediction is not possible except in special cases with this technique.

Simulation: Behave properly. Lok’s system includes a collision response model consisting of detection, recovery, and physically accurate response for collisions between real and virtual objects. Like other systems, it cannot make real objects respond to collisions with virtual ones. Since objects cannot be individually differentiated within a hull, we cannot determine the exact path of objects as they come to be in collision with a purely virtual object. This means that the computed response to the collision can only be approximate. This reduces the usefulness of visual hull techniques when the application requires precision simulations.

4 Just-in-Time Models and Dynamic Physical Correlates

The third example of a mixed environment system creates models just before they are needed. The visual fidelity of the real object is higher than in the real-time system. The problem driving the development of this system is early testing of fit and assembly for NASA payloads. A typical problem involves real objects for which CAD models exist, tools for which CAD models don’t exist, and objects for which only CAD models exist.

Figure 4. Assigning marker position to scanned model of pliers (left), using pliers with part (middle), HMD view of using real objects (right).

4.1 What it is

The Just-in-Time Model system generates a model of a real object in a matter of minutes by scanning it with a Cyberware 3D laser scanner. A post-scanning process creates a polygonal model and generates textures for it based on camera images. To track the real object, we register and affix simple colored markers to the it. We track it with outside-looking-in web cameras using the STRAPS tracking library [Jackson04]. The primary user wears a head- mounted display (HMD) to view the 3D models. He can pick up tools and parts and see the corresponding models move. Other team members can participate by viewing the ME using a Magic Lens (tracked Tablet PC) and via a LAN or the Internet. Figure 4 shows a model of pliers captured with the scanner and being used in a virtual scene.

Notably, the system tracks two kinds of real objects: real parts that are modeled just-in-time, for instance the pliers in Figure 4, and real objects that approximate as-yet-not-manufactured parts, for instance the white cardboard box in Figure 5 (left) that represents the fuel tank. We call the approximate objects physical correlates.

4.2 Effectiveness

As an initial evaluation of system usability, we conducted an informal study using actual NASA payload models, parts and tools [Wang05]. In an abstracted layout task, the user had to position a fuel tank and then insert a circuit board in a slot inside the fuselage of a prototype airplane. However, the fuel tank blocked the slot, preventing the user from inserting the board. The user had to recognize the design flaw and adjust the assembly sequence accordingly.

Three graduate students who were unfamiliar with the system performed the task. All users detected the assembly error and subsequently removed the fuel tank, inserted the board, and remounted the fuel tank, Figure 5. Users maneuvered a real circuit board that was modeled using our scanning pipeline. They inserted the circuit board into a physical slot (passive haptics) that was represented virtually by CAD models of the slot and the surrounding payload subcomponent. The virtual fuel tank exists only as a CAD model and the user handled a physical correlate of the tank, a small white box with attached color markers.

4.3 Plusses and Minuses of Just-in-Time Models

Acquisition: Look right—shape and appearance. High-resolution color, texture, and precise geometric shape of the virtual object are obtained using a commercial 3D laser scanner. However, scanners are relatively expensive, and the operation of aligning and combining color imagery with the 3D point-cloud representation of the object surface is not a fully automatic operation.

Figure 5. Working on the task and the virtual scene before and after assembly.

Tracking: What’s where? Wang et al. found commercial magnetic and acoustic trackers unable to track the number of objects they expect to have in their scenes. They developed the outside-looking-in web-cam-based tracking system that senses and locates color markers attached to the objects being tracked. In their initial implementation, positional error can be as large as 4.5 cm. This technique is limited by the ability of the cameras to differentiate colors, by the angle of the markers relative to the cameras, and by lighting within the scene. The technique isn’t robust in the event of occlusions in the sensor-to-marker line-of-sight.

In addition to calibration, system latency is one of the largest contributors to mis-registration of real and virtual objects in VEs. There is about 120 ms of latency added by the optical tracker in the system reported by Wang. This is enough to affect the user’s experience negatively. Meehan and colleagues showed that user sense of presence was higher in a system with 50 ms than one with 90 ms of end-to-end latency [Meehan02]; [Mania 04] has shown that variations in system latency of as little as 17 ms can be noticed, independent of baseline latency.

Merge real and virtual: Be in the right place at the right time. Since this system operates exclusively with polygonal representations of the real objects, we can easily apply traditional lighting, rendering, and shading algorithms.

Simulation: Behave properly. Although the most advanced collision packages today can operate on a variety of data structures, polygonal models are most often assumed. This system is based on polygonal representations and so can be used with a wide variety of simulation software. With the assistance of the hardware of today’s programmable graphics cards, collisions can be detected and appropriate responses can be computed even for collisions between geometric models of articulated and deformable bodies and dynamically changing geometric models.

5 Simulation: Mitigating the effects of Missing Haptics

Providing effective haptic cues for VE users who are moving about in their tracked space is a challenging open problem. Grounded, active haptic devices such as a Phantom® or a floor-mounted exoskeleton generally are unsuitable for users who are moving. Many approaches are being explored; none yet dominates. Here we list some recent work in this area.

Wearable and portable active haptics. Wireless, wearable mechanical tactors are now available and are being tested to see if they provide a level of haptic feedback that will support task performance. These systems can be, and often are, used to display data that is not haptic. Work by Lindeman and his colleagues is representative [Lindeman04]. At UNC, we frequently have users carry a Logitech ® Cordless Rumblepad ™ that vibrates when they collide with a virtual wall. We’ve yet to investigate whether receiving the collision cue in the hands instead of on the body part that is in collision has an effect on performance or training.

Haptic Hand. In a recent proof-of-concept activity at UNC [Kohli05] we investigated a way to provide haptic feedback for using menus that does not involve the user carrying a prop. We lock a menu to the palm of the user’s non-dominant hand and he manipulate widgets using the other hand. Users (n=7) reported the system was easier to use when they had both fingertip-to-palm contact and visual confirmation of contact.

Alternative and augmenting sensory cues. The idea here is to compensate for missing haptic cues by stimulating one or more of the other senses—most often by providing visual cues or auditory ones. Overall user satisfaction with a somewhat soft haptic feedback device (a mid-50s vintage Argonne Remote Manipulator with a 15Hz cut-off frequency in the control electronics) was increased by adding a large electrical relay that pulled in with a loud clunk when a collision occurred [Brooks 1990]. Lok and Insko both had objects in collision change color; Insko also gave an audio cue.

The visual sense is powerful. When visiting in the UNC lab several years ago and looking at molecular models on a very high quality stereo projection system, Mel Slater reached out and touched a portion of the model where the link between two atoms was a somewhat sharp point. Slater uttered a very audible, “Ouch!” as he had a illusion of haptics generated only by the visual.

Minimizing cue conflict. Visual interpenetration of virtual solid objects lessens the strength of the VE illusion. Burns [Burns05] investigated the extent of user’s tolerance of mismatch of visual and proprioceptive cues as part of a series of studies to determine the least objectionable visual response when such interpenetration occurs. Results to date show that we much more quickly notice interpenetration than positional mismatch of virtual and real arms and hands, another instance of vision dominating proprioceptive cues.

6 How good is good enough?

The quality of our best VE systems limits the usability and performance evaluation testing we can do. If the best today is only just good enough to integrate real and virtual objects in a lab-based application, we can’t confidently generalize results to generate design guidelines for applications that will be used in the field. We must have a very high performance system before we can meaningfully back off on performance on a particular element of the system to test at what level the degradation begins negatively effecting usability or performance.

Developing a robust, high performance, minimal infrastructure VE system that will integrate real and virtual objects in VE won’t be a simple matter. The open issues are non-trivial and, in many cases, open research problems in sub- disciplines of computer science—tracking, modeling, rendering, image-based techniques, and simulation. Since our VE problems are not necessarily those researchers’ driving problems, their results may require considerable additional work before they can be used in a VE system. The breadth of sub-disciplines used to build virtual environment systems demonstrates the magnitude of the system engineering and integration task that VE system developers face. It also reveals the scale of the opportunities for evaluative studies to insure that we build effective virtual environments.

Acknowledgements. The preparation of this document was supported by the Office of Naval Research, the NIH National Institute for Biomedical Imaging and Bioengineering, and the University of Florida. The authors gratefully acknowledge the input of Ming Lin and Leonard McMillan on the topics of collision models and visual hulls, respectively. Ron Adres of Reddiform supplied the Styrofoam blocks used in passive haptics construction. The authors would like to acknowledge the work of Samir Naik in the development of the real-time object reconstruction system; Xiyong Wang, John Quarels, Aaron Kotranza, and Danette Allen in the development of the mixed environment system; Eric Burns and Luv Kohli for their work on managing avatar hand behavior and the haptic hand, respectively.

References

[Brooks90] Brooks, Jr., F. P., M. Ouh-Young, J. J. Batter, and P. J. Kilpatrick, 1990: "Project GROPE: Haptic Displays for Scientific Visualization," Computer Graphics: Proc. SIGGRAPH '90, August, 177-185, Dallas, TX. [Burns05] Burns, Eric, Razzaque, S., Whitton, M., McCallus, M., Panter, A., Brooks, F., “The Hand is Slower than the Eye: A quantitative exploration of visual dominance over proprioception,” Proceedings of IEEE Virtual Reality 2005, (Bonn, Germany March 2005), IEEE Computer Society. To appear. [Carlin97] Carlin, A.S., Hoffman, H. and Weghorst, S. (1997). “Virtual Reality and Tactile Augmentation in the Treatment of Spider Phobia: A Case Study. Behavior Research and Therapy, 35(2), 153-158. [Hilton00] A. Hilton, D. Beresford, T. Gentils, R. Smith, W. Sun, and J. Illingworth. Whole-Body Modelling of People from Multiview Images to Populate Virtual Worlds, The Visual Computer, 16 (7), pp. 411-436 (2000). ISSN 0178-2789. [Insko01] Insko, B. (2001). Passive Haptics Significantly Enhances Virtual Environments. Ph.D. Dissertation, Department of Computer Science, University of North Carolina at Chapel Hill. Available as UNC Computer Science Technical Report #01-017. [Jackson04] Jackson, J., Lok, B., Kim, J., Xiao, D., Hodges, L., and Shin, M. “ Straps: A Simple Method for Placing Dynamic Avatars in an Immersive Virtual Environment”, Technical Report 04-02, University of North Carolina at Charlotte. [Kohli05] Kohli, L., Whitton, M., "The Haptic Hand: Providing User Interface Feedback with the Non-Dominant Hand in Virtual Environments," Proceedings of Graphics Interface 2005 (May 2005). To appear. [Laurentini94] A. Laurentini. The Visual Hull Concept for Silhouette-Based Image Understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol 16, No. 2, 150-162, February 1994. [Lindeman99] Lindeman, R., J. Sibert, and J. Hahn (1999). Hand-Held Windows: Towards Effective 2D Interaction in Immersive Virtual Environments. Proceedings of IEEE Virtual Reality '99, IEEE Press, 205-212. [Lindeman01} Lindeman, R., John L. Sibert, James N. Templeman, "The Effect of 3D Widget Representation and Simulated Surface Constraints on Interaction in Virtual Environments," Proc. of IEEE Virtual Reality 2001, pp. 141-148. [Lindeman04] Lindeman, R.W., Page, R., Yanagida, Y., Sibert, J.L., "Towards Full-Body Haptic Feedback: The Design and Deployment of a Spatialized Vibrotactile Feedback System,,” Proc. of ACM Virtual Reality Software and Technology (VRST) 2004, Nov. 10-12, 2004, Hong Kong, China, pp. 146-149 [Lok03a] Lok, Benjamin, Samir Naik, Mary Whitton, and Frederick Brooks. “Effects of Interaction Modality and Avatar Fidelity on Task Performance and Sense of Presence in Virtual Environments,” Journal on Presence: Teleoperators and Virtual Environments, 12, 6: 615-628. (2003). [Lok03b] Lok, Benjamin, Samir Naik, Mary Whitton, and Frederick Brooks (2003). “Incorporating Dynamic Real Objects into Virtual Environments,” Proceedings of ACM 2003 Symposium on Interactive 3D Graphics, Monterey, CA (April 28-30), ACM SIGGRAPH: 31-41. Images in Communications of the ACM, July 2003, 46.7: 48. [Mania04] Mania, K., Adelstein, B., Ellis, S., and Hill, M. “Perceptual sensitivity to head tracking latency in virtual environments with varying degrees of scene complexity” in Proceedings of the ACM 1st Symposium on Applied Perception in Graphics and Visualization, 2004, 39-47. [Matusik00] W. Matusik, C. Buehler, R. Raskar, S. Gortler and L. McMillan. Image-Based Visual Hulls. In SIGGRAPH 00 Proceedings, Annual Conference Series, pages 369-374. [Matusik01] W. Matusik, C. Buehler, and L. McMillan, "Polyhedral Visual Hulls for Real-Time Rendering", Eurographics Workshop on Rendering 2001. [Matusik02] Matusik, Wojciech, Hanspeter Pfister, Addy Ngan, Paul Beardsley, and Leonard McMillan, "Image- Based 3D Photography Using Opacity Hulls," Proceedings of ACM SIGGRAPH 2002, (in Computer Graphics Proceedings, Annual Conference Series, July 2002), pp. 427-437. [Meehan02] Meehan, M., B. Insko, M. Whitton and F. P. Brooks Jr. (2002). "Physiological Measures of Presence in Stressful Virtual Environments." ACM Transactions on Graphics 21(3): 645-652. (Proceedings of ACM SIGGRAPH 2002, San Antonio, Texas). [Wang05] Wang, Xiyong, Aaron Kotranza, John Quarels, Benjamin Lok, Danette Allen (2005). “Rapidly Incorporating Real Objects for Evaluation of Engineering Designs in a Mixed Reality Environment,” To Appear in New Directions in 3D User Interfaces Workshop, IEEE Virtual Reality 2005, Bonn, Germany.