Usage and Evaluation of the Spatial AR Hologram for Exhibits

Keywords: augmented reality, , exhibition technology, user evaluation

Minju Kim, Graduate School of Culture Technology, KAIST, South Korea

Kwangyun Wohn, Graduate School of Culture Technology, KAIST, South Korea

Abstract

Pervasive uses of digital technologies to enhance audiences’ experiences and better understanding have been widespread throughout museums. Computer graphics and animation technologies offer effective ways to create exhibits by representing the scenes that are difficult and inefficient to display in the real world. Virtual reality takes one step further by immersing the audiences in the virtual space, and thereby maximizes their experiences with the exhibit simulations through the real-time visitor interaction. Holographic display has been used for a long time in exhibition field in a way that rendering of virtual image through a semi-transparent screen is spatially aligned with the physical exhibits. Compared with VR which all takes place in the virtual space, the holographic display enhances the audiences’ perception of reality by displaying the additional information on the physical object with the flavor of augmented reality (AR).

As an attempt to the further extend the potential of the holographic display, we propose a new exhibition system that combines the virtual reality technology with the conventional holographic display, and create a prototype to verify its effectiveness. The distinctive features are as follows: 1) Spatial Visualization: Even though the conventional holographic display is regarded as 3D, it displays images on a single layer offering a limited depth effect. Whereas we introduce another layer of display screen, thereby forming a 2-layer display and apply the stereoscopic imaging to the front display. 2) Spatial Exploration: Camera faces the audience and tracks their positions and gestures to enable interaction between the system and the audience.

In this paper, we present the design of our system, focusing the user study on how effective is the system in creating the 3D spatial perception. Our initial investigation suggests that the newly conceived holographic display offers the users not only more accurate 3D spatial perception, but also better spatial awareness and realism. Furthermore, it helps the users understand exhibition contents easier and better.

1. Introduction

Extensive usage of digital technologies has been widespread throughout museums. Digital technologies such as computer graphics, virtual reality, augmented reality and holographic display not only help create exhibits but also enhance audiences’ experiences (Kajinami, 2010). Of those, the holographic display has been used for a long time in exhibition field in a way that rendering virtual image through a semi-transparent screen is spatially aligned with the physical exhibits (Debenham et al., 2011; Jim, 1999). Even though the conventional holographic display is regarded as 3D in the sense that it displays the image in the air, it displays images on a single layer which lacks the depth effect. As such, there are limits on creating exhibits and make audience immersed.

In this paper, we propose Spatial AR Hologram, a new exhibition system that combines virtual reality technology with the conventional holographic display. Frist, we make the space continuous 3D using a 2-layer display and applying the stereoscopic imaging to the front layer. It allows the system to express exhibits that are not subject to a space limit. Second, we enable audiences to experience the exhibits consistent by applying 3D interaction. Depth camera faces the audience and tracks their positions to enable interaction between the system and the audience.

We describe the design of our system, focusing the user evaluation on how effective is the system in creating the 3D spatial perception. Our investigation suggests that the newly created holographic display provides a three dimensional spatial perception with a good sense of 3D immersion for users. Finally we prototype AR exhibits and it is distinctly different with conventional holographic exhibits. Furthermore, it not only offers better spatial awareness and realism, but also helps the audiences understand exhibition contents easier and better.

2. Related Work

2.1 Exhibition Technology: CG/VR/ AR/

Pervasive use of digital technologies to enhance audiences’ experiences and better understanding has been widespread throughout museums (Bimber et al, 2003; Basballe et al, 2010). Computer graphics and animation technologies offer effective ways to create exhibits by re-presenting the scenes that are difficult and inefficient to display in the real world. Virtual reality takes one step further by immersing the audiences in the virtual space, and thereby maximizes their experiences with the exhibit simulations through the real-time visitor interaction.

Holographic display has been used for a long time in exhibition field in a way that rendering virtual image through a semi-transparent screen is spatially aligned with the physical exhibits. Recent research (Hannigan, 2001; Bimber et al. 2005; Hilliges et al. 2012; Gingrich et al. 2013) has shown that compared with VR which all takes place in the virtual space, holographic display enhances the audiences’ perception of reality by displaying additional information on the physical object with the flavor of augmented reality. Even though conventional holographic display is 3D, it does not have the capacity to provide three-dimensional AR hologram experience. That is, the images projected on the holographic screen cannot evoke physical depth cues. As a result, there are limits and restrictions on exhibits display. When displaying AR exhibits, virtual image through a semi-transparent screen is not spatially aligned with the physical exhibits. Thus, it causes spatial inconsistency and difficulty in understanding exhibits.

2.2 3D Visualization

3D visualization techniques have been widely used for simulating data, exploring and manipulating information using the three-dimensional graphics in information visualization field, especially scientific visualization. Smallman et al. (2001) claimed that the purpose of 3D visualization is to make people understand data correctly and efficiently by exploiting the 3D spatial perception. Recent research has applied the distinctive features of 3D visualization such as web-contents visualization (Jun et al., 1993), virtual environments simulations (Rhyne, 2002; George et al., 2004), and many others. They reported that the 3D increased the reality of the contents and improved the visual experience of the audience.

The use of stereoscopic enables expressing 3D graphic realistically by creating or enhancing the illusion of depth in an image. Binocular perception makes representing 3D space possible, so it has been utilized in the 3D operation interaction field. According to surveys, Mulder et al. (2004) and Fischer et al. (2007) proposed virtual surgery training and virtual workbench that overlay virtual images on the physical objects through a transparent screen. Applying stereoscopic technology allows rendered virtual images to be spatially aligned with the physical object in the same space. However Prema et al. (2006) found that the display area is limited to the frontal part and there is the limitation using at exhibition that handles complex and various information. To overcome this limitation, we have developed a multi-layer based spatial AR hologram that made the best use of three- dimensional space around physical exhibits.

3. Overview of Spatial AR Hologram System

In this paper, we present Spatial AR Hologram that shows additional information on exhibits effectively in exhibition field. The newly conceived system provides the audiences not only accurate 3D spatial perception but also with better spatial awareness. Furthermore, it helps the audience understand content easier and better in exhibition field that deals with complex and various information such as text, image, video, 3D object and so on. To achieve this, we need to solve the spatial visualization and the spatial exploration problems. In this paper, we define convergence of the spatial visualization and the spatial exploration as ‘Spatial Experience’. The distinctive features are as follows.

In order to express spatial visualization, we installed multi-layer display and applied 3D stereoscopic technologies. Stereoscopic technologies enable complex information to be merged around the physical object naturally. Multi-layer displays offer spatial impression by increasing the number of displays and arranging them separate in the physical space. In the case of additional exhibits’ information, it includes indirect background information and foreground information which explain exhibits directly. In general, the audience moves around the exhibition hall and looks this way and that. Therefore, for the spatial exploration, it is important to offer additional information with visual consistency depending on user’s physical point of view. The camera faces the audience and tracks their positions to visualize exhibit contents by guaranteeing their exhibition experience in real-time. As a result, it enables active interaction between the system and the audience.

4. System Implementation

The proposed spatial AR holographic display uses two displays (one is the semi-transparent film and the other is the monitor) placed in parallel and separated 1 meter physically by depth. The space in between is used to place the physical object which serves as the exhibit artifact. The semi-transparent film reflects the image which is emanated from the projector underneath, forming a virtual image appearing on the mid-air. The other display, placed in the back, serves as the backdrop scene at this moment, but will produce the stereoscopic 3D scene later, as described in the following section. With this configuration, the system can enhance the audience’s spatial experience by allowing them to Figure 1. Spatial AR Holographic display system look at the physical objects while displaying digital exhibits on the front and back displays at the same time. Our physical configuration for Spatial AR Hologram is illustrated in (Figure 1).

We use Musion Eyeliner 42 inch transparent display for the front display (Musion EyeLiner™, 2014). The physical object and the back layer display are viewed by the audience through the semi-transparent front display. This front display reflects light toward the audience from a projector mounted under the display. The projector displays a 1024×768 pixels image at 4000 ANSI LUMEN. A 50’’ Samsung TV monitor is used as the back layer. We installed the controllable lights inside the system that switch the scene between virtual reality and augmented reality naturally and improve the visibility of the real exhibits. The light system is controlled by using ARDUINO. One depth camera which is installed in front of the system faces the audience and tracks the position in real time. To block the light and enhance usability, we covered the entire system with a black cloth except for the frontal side which serves as the interaction space with the audience.

4.1 Spatial Visualization: multi-layer display with stereoscopic imaging.

To visualize spatial images, we applied a stereoscopic visualization at the front display and a 2D image at the back display respectively. Polarized 3D system, one of the three-dimensional representation methods, is not applicable in this case because of the diffused reflection properties. Accordingly, we applied the anaglyph 3D image which can be implemented easily. We installed two virtual cameras at a position one’s two eyes apart in the horizontal direction in Processing (Casey, 2007). Two cameras view the same image and extract each image as red and cyan. As an experimental approach, although we current use an anaglyph 3D image, active shutter 3D system is preferable for that it is better at three-dimensional representation and high resolution implementation. Meanwhile, according to the characteristic of human visual perception, vergence cannot occur in two different places, so we have made version plane at the front display. Applying stereoscopic images on both displays is an open question and further investigation will be needed (Figure 2).

Figure 2. Spatial visualization Figure 3. Spatial exploration

4.2 Spatial Exploration: Head tracking with depth camera

Depth camera (Microsoft Kinect) in front of the system faces the audience and tracks the head to enable the motion parallax (Figure 4). It tracks the location of the audience’s head and links it with the two cameras which render graphic elements in the virtual three-dimensional space. This allows the audience to view the digital exhibits as their attention is correctly registered on the physical exhibits consistently. Therefore, it ensures the continuity of the spatial experience, increasing the interactivity between the audience and exhibits (Figure 3).

Figure 4. Depth camera view: user head tracking

5. User Evaluation

5.1 Comparison of 3D spatial perception

In this paper, we tested our system with users, focusing the user study on how effective the system is in creating the 3D spatial perception. In order to verify the 3D spatial perception, it was unnecessary to operate back layer display at the same time. Twelve participants with past experiences with 3D user interfaces (age 20-32, 5 male and 7 female) were recruited to participate in the study (Figure 5).

First, users had the time to familiarize themselves with the system. Then, they performed the task, arranging virtual images spatially aligned with the physical object. They performed three different randomized tasks for 3D spatial perception configuration, comparing existing 2D interface and 3D interface that applied stereoscopic visualization, and view-dependent 3D interface that applied both 3D visualization and 3D exploration. We formally evaluated the accuracy, task completion time, and the total number of tries. Users were asked to complete the task as accurately and quickly as possible. The total experiment time for each participant was between 30-40 min.

Figure 5. Experiment environment for user study. Virtual 2D and 3D images (left), the virtual 2D and 3D images with physical object (right)

Results showed clear differences mostly between the 2D and 3D interface and showed no reliable differences between 3D and view-dependent 3D interface. 3D interface that applied stereoscopic visualization offers the users more accurate 3D spatial perception. Especially in the case of view-dependent 3D that applied the user interaction, participants performed tasks reliably faster and error rate was the lower, meaning it is offering consistent depth perception to the users. 2D, on the other hand, showed high at all aspects. Users’ comments include “recognizing the space in 2D interface was difficult since the virtual image always seemed located in front of the physical object”, “I feel as if I am adjusting the image size in the 2D flat screen rather than the depth of the virtual image”. In contrast, in the case of 3D and view-dependent 3D interface, they easily positioned virtual images and the physical object like the one object without confusion. Proposed system applying 3D visualization and 3D exploration make the users reproduce three-dimensional spatial perception more quickly and accurately (Figure 6).

Figure 6. Results of the user evaluation (task error rate/ task finishing time/ and total trial time)

5.2 Spatial awareness, realism, contents understanding

After viewing the exhibit contents based on view-dependent 3D, 3D, and 2D interface through the system, qualitative evaluation was carried out. The questionnaires are listed below (Table 1).

Virtual image seems to float in the space, not on the screen. 3D spatial awareness Virtual image and the physical object is spatially aligned in real space. Easy to form physical space perception around the physical object. Virtual image seems to be around the real physical object. 3D spatial realism AR contents is natural like the real. Easy to be immersed in AR contents. Satisfaction and Easy to understand the exhibits contents. contents understanding No visual discomfort to observe the contents. I want to see the contents using this system in the future.

Table 1. Questionnaires to survey spatial awareness, realism and contents understanding

Questionnaire responses (7-point Likert scale) indicate that the Spatial AR Hologram interface were easy to form depth perception and spatial awareness (6.19). Participants, however, performed more poorly in perceiving three-dimensional spatial realism in the 2D than view-dependent 3D interface (3.36 vs. 6.33). Users’ comments include “it was difficult to be immersed in AR hologram contents represented in 2D interface”, “I had to observe the physical object and virtual image separately to understand it correctly.” Also, satisfaction of the contents through the system is the highest in view-dependent 3D interfaces, and there was no great difference between view-dependent and 3D interface (6.3 vs. 5.63). We verify that the proposed system offers the users not only more accurate 3D spatial perception, but also a better spatial awareness and realism.

6. Exhibition Prototyping

Spatial AR Hologram supports many application and display possibilities. One of the characteristics of Spatial AR Hologram is the ability for users to experience seamless mix of real space and virtual contents not restricted to a defined area. This leads to possibilities of more interesting museum exhibits. In this paper, we shows AR exhibits contents that represents and explains three-story Stone Pagoda of Gameunsa site which is located in Gyeongju-si, South Korea. At first, we installed miniature of three-story stone pagoda between a front display and back display. Then, related exhibit information is displayed on each display, describing direct information on the front and indirect and background information on the back display (Figure 7).

Figure 7. Spatial AR hologram Exhibition Prototyping

Exhibition prototype is implemented using Processing, and stereo 3D image is projected on the front display and 2D image on the back display at the same time. To make it simultaneously, after making the image file at Processing respectively and link each of that to the OSC file. Two virtual cameras are positioned apart by various distance in the horizontal direction in Processing. Controlling the distance of them makes the virtual images shown in various depth plane. Virtual three-story stone pagoda is produced in 3D Maya and light system is implemented by using ARDUINO.

The result of prototyping shows significant differences from conventional holographic display system such as l-layer based display and interface creating 2D images. It offers the users not only more accurate 3D spatial perception, but also efficient exhibit information visualization. Audiences’ comments include that the newly conceived system helps them understand the exhibition contents easier and better. Furthermore, audience can have a vivid experience as if they were watching a three-story stone pagoda in reality at nearby museum without having to go to the place.

7. Conclusion and Future works

In this paper, we have presented a new exhibition system, Spatial AR Hologram that incorporates the virtual reality technology with the conventional holographic display. To visualize spatial 3D, we use a 2-layer display and apply the stereoscopic imaging to the front display. It allows the system to deliver the spatial information naturally around the physical objects in real space. Also, it makes it possible to create a continuous space, as well as visualize complex information efficiently.

Our results highlight that the newly conceived holographic display applying 3D visualization and 3D exploration provides the users with not only more accurate 3D spatial perception, but also a better spatial awareness. Furthermore, it helps the users understand exhibition contents easier and better.

In addition to the 3D spatial perception problem, there are two other interesting and challenging issues. The first one is fully exploiting two layers of displays plus the space in-between so that whatever information to be presented is displayed automatically or semi-automatically in such a way that the audience experiences a coherent and continuous 3D space. Another is that of interactivity. It is not only the audience but also the presenter (or the docent) who can take the advantage of the interactive nature of the display system. We expect our new AR holographic display to be useful not only in exhibition but also in other diverse fields such as interactive presentations, stage performances, information visualization and many others.

8. Acknowledgement

This research was partially supported by NRF and the BK21 Plus Framework.

References

1. Basballe, D. A. and K. Halskov. (2010). “Projections on museum exhibits: engaging visitors in the museum setting”. Proceedings of the 22nd Conference of the Computer-Human Interaction Special Interest Group of Australia on Computer-Human Interaction. 80-87. 2. Bimber, O., L. M. Encarna, and D. Schmalstieg. (2003). “The virtual showcase as a new platform for augmented reality digital storytelling”. Proceedings of the workshop on Virtual environments 2003. 87-95. 3. Casey R. Ben F. (2007). "Processing: A Programming Handbook for Visual Designers and Artists". The MIT Press. 712 pages. 4. Debenham, P., G. Thomas and J. Trout. (2011). “Evolutionary augmented reality at the Natural History Museum”. Mixed and Augmented Reality (ISMAR), 2011 10th IEEE International Symposium on. 249-250. 5. Fischer G.S., Deguet A., Csoma C., Taylor R.H., Fayad L., Carrino J.A., Zinreich S.J., and Fichtinger G. (2007). "MRI Image Overlay: Application to Arthrography Needle Insertion". Comput Aided Surg. 12(1), 2-14. 6. George L., and Costas V. (2004). “Virtual museums for all: Employing game technology for edutainment”. Virtual Reality. 8(2), 96-106. 7. Gingrich, O., A. Renaud and E. Emets. (2013). “KIMA: a holographic telepresence environment based on cymatic principles”. ACM SIGGRAPH 2013 Art Gallery. 332-343. 8. Hannigan, B. (2001). “Augmented Reality as a New Media Experience”. Proceedings of the IEEE and ACM International Symposium on Augmented Reality (ISAR'01), IEEE Computer Society. 197-. 9. Hilliges, O., D. Kim, S. Izadi, M. Weiss and A. Wilson. (2012). “HoloDesk: direct 3d interactions with a situated see-through display”. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2421-2430. 10. Jim S. (1999). "The Science Behind the Ghost: A Brief History of Pepper's Ghost", Hahn, Benjamin Blom, New York, 189 pages. 11. Jun R. and Mark G. (1993). “The Information Cube: Using Transparency in 3D information visualization”. Proceedings of the Third Annual Workshop on Information Technologies & systems. 125-132. 12. Kajinami, T., O. Hayashi, T. Narumi, T. Tanikawa and M. Hirose. (2010). "Digital Display Case: Museum exhibition system to convey background information about exhibits". Virtual Systems and Multimedia (VSMM), 2010 16th International Conference on. 230-233. 13. Mulder, J. D. and B. R. Boscker. (2004). “A modular system for collaborative desktop VR/AR with a shared workspace”. Virtual Reality. 27-31. 14. Musion EyeLiner™. (2014). http://www.musion.com/ 15. Prema V, Roberts G, and Wuensche BC. (2006). “3D visualisation techniques for multi-layer display technology”. IVCNZ '06. 251-256. 16. Rhyne, T.-M. (2002). "Computer games and scientific visualization". Commun. ACM. 45(7), 40-44. 17. Smallman, H. S., M. S. John, H. M. Oonk and M. B. Cowen. (2001). "Information Availability in 2D and 3D Displays". IEEE Comput. Graph. Appl. 21(5), 51-57.