2019 IEEE Games, Entertainment, Media Conference (GEM) A Large Curved Display System in Virtual Reality for Immersive Data Interaction Lizhou Cao Chao Peng Jeffery T. Hansberger Computer Science Department Computer Science Department Army Research Laboratory University of Alabama in Huntsville University of Alabama in Huntsville Huntsville, Alabama, United States Huntsville, Alabama, United States Huntsville, Alabama, United States [email protected] [email protected] [email protected] Abstract—This work presents the design and implementation and can support users to have a natural way of interaction with of a large curved display system in a virtual reality (VR) data. environment that supports visualization of 2D datasets (e.g., In this paper, we propose design criteria for a large curved images, buttons and text). By using this system, users are allowed to interact with data in front of a wide field of view and gain display system in VR. We describe the implementation of the a high level of perceived immersion. We exhibit two use cases system and two use cases of the system, including (1) a virtual of this system, including (1) a virtual image wall as the display image wall as the display component of a 3D user interface, component of a 3D user interface, and (2) an inventory interface and (2) an inventory interface for a VR-based educational for a VR-based educational game. The use cases demonstrate game. capability and flexibility of curved displays in supporting varied purposes of data interaction within virtual environments. The rest of the paper is organized as follows. Section II Index Terms—virtual reality, curved display, visualization, presents the related work. Section III presents our design cri- human-computer interaction teria. Section IV describes the implementation of our system. Section V describes two use cases. Section VI concludes the I. INTRODUCTION work and proposes future work. Virtual reality (VR) with a head mounted display (HMD) II. RELATED WORK is a computed simulated environment and has been adopted widely to visualize 3D objects. It allows users to control the This section reviews previous work related to the develop- camera view with natural head and hand movements, and gives ment of 2D interfaces in VR, large display systems, and curved the users an immersive experience of interaction. However, display systems. S. M, Slobounov et al. [9] performed an EEG there are not sufficient studies in the literature that discusses study about the modulation of cortical activity in 2D versus the VR’s capability of presenting 2D datasets such as image 3D. Based on their experimental results, they concluded that collections, buttons, and text, which are important media for a fully immersive 3D environment requires allocation of more the storage and dissemination of information. brain and sensory resources for cognitive/motor control than In real-world situations, data can be presented within large a 2D environment, and they argued that 3D VR tasks could physical display systems, from which users can gain an enhance the performance of many applications. B. Ens et al. immersive experience [8]. The physical setting of such a large [1] presented a framework to tie variations of 2D information display system usually requires users to move their bodies in space together. Their framework was used to organize multiple front of the system in order to get closer to particular data surfaces into a curved formation, but individual surfaces were items that they want to interact with. When wearing an HMD, still flat. a user’s view of environment is blocked. For safety reasons, Ed Lantaz [5] reviewed the large display development in users usually prefer stationary interactions while using a VR digital cinemas. He argued that the large display is able to application. For example, a user may choose to stand at the provide the immersive view and will be the future of displays. center point of the virtual space to view a scatter plot. We K. Reda et al. [6] reviewed the approaches that used hybrid- investigate the possibility to create a curved display in VR. reality environments for visualizing large and heterogeneous This approach provides a wide field of view and supports a data. They argued that a hybrid-reality environment blurs high level of perceived immersion that is similar to the user’s the boundary between traditional virtual environments and experience when using a real-world curved display. In our tiled display walls. A. Febretti et al. [3] presented a soft- daily life, we use curved TVs [7], curved display gaming ware framework of hybrid reality environments for large-scale machines [2], and curved display walls [10]. Also, IMAX visualization. They showed applications supported by their movie theaters are often featured with curved screens. We framework. believe that curved display systems can be mimicked in VR Jon Shaw [8] gave a review of the history of dome theaters. He argued that because dome screens can provide larger field- This work was supported by the DOD grant W911NF-16-2-0016. of-view images and increase the subjective feeling in expanse, 978-1-7281-2404-9/19/$31.00 ©2019 IEEE naturalness, depth, and powerfulness, they are more closely to our natural neural responses, and makes makes the experience feel more “real”. S. Grogorick et al. [4] presented the design and implementation of a real-time stereo dome projection system. The work showed that large curved display systems could give users immersive experiences in real world. From these previous work, a conclusion is that a real-world curved display system can support a high level of perceived immersion. We believe this kind of system can be a good way for users to interact with data in VR with an HMD. III. DESIGN CRITERIA Our system needs to satisfy user demands in immersion, Fig. 1. The articulation of the canvas and curved screen within our system interactivity, extendibility, and easy-to-use. We discuss our architecture. The orthographic camera captures the content on the canvas, design criteria for those demands in the following subsections. which is then rendered onto the render texture. After that, the render texture is mapped onto the curved screen. A. Immersion The system should use the advantages of VR technologies to allow users gain a high level of perceived immersion, which IV. IMPLEMENTATION will improve user engagement. Similar to the large display We have implemented our system using the Unity3D game systems in the real world, we believe that a simulation of a engine and programmed functional features using C# scripting large screen in VR could satisfy this criterion. We designed language. In this section, we explain the architecture of our our system with a wide field of view. Since we want users system and the detail of the implementation. to interact with data by standing at a center point, the field of the view needs to be adjustable. The perception should be A. Architecture non-distorted from the far edge of the display. The architecture of our system includes three parts: a canvas B. Interactivity used to organize the data, a curved screen to visually represent The system should allow a range of user interactions that the data, and a virtual handheld object (VHO) used as a range from selection to the manipulation of data items. The navigation aid or an instructional tablet. This architecture is system should provide users with operations that support data designed with the inspiration from the setting of curved TVs interaction, such as pointing to the screen, clicking on and in everyday life. In our system, the canvas is same as a moving data items, zooming in and out the view of the screen, broadcasting studio that provides contents. The curved screen and panning around on the screen. can be considered as a curved TV. A VHO can be used as a C. Extendibility remote control. The canvas is the base space of the curved display system. Extendibility refers to the capability of adding display pan- It is a flat rectangular window that all data elements will be els to form a larger system for more complicated operations. inside. An orthographic camera is placed in front of the canvas Multiple display panels can be added and organized for the with the view size bound to the window size of the canvas. appropriate layout. Individual panels may be adjustable in size, During the run-time of the system, the game engine renders the position, and orientation. Also, the panels should be able to camera’s view onto a texture, which is called render texture. communicate to each other so data items can be shared among In order to maintain a high level of visual quality for the them. rendered canvas, we set the resolution of the render texture D. Easy-to-use based on the size of the curved display (the curved screen). The system needs to be user-friendly and modularized so The render texture will be assigned to the polygonal mesh of that it can be easily reconfigured for different applications and the curved display using the surface’s UV texture coordinates, data types. In our design, the system can be broken down which define the region on the surface the texture maps to. to a back-end module and a front-end module. The back- Fig. 1 illustrates how the canvas is used to create the curved end module handles the data structure and the organization screen through a render texture. of the dataset in the same way as traditional 2D interfaces. The curved screen is created from a polygonal plane, first The front-end module is used to give visual representations by bending the plane along the width direction and then by desired by users, as well as handling input commands for data bending it along the height direction.