Creating High Fidelity 360° Degree Virtual Reality with High Dynamic Range Spherical Panorama Images
Total Page:16
File Type:pdf, Size:1020Kb
Zi Siang See, Lizbeth Goodman, Craig Hight, Mohd Sharizal Sunar, Arindam Dey, Yen Kaow Ng, Mark Billinghurst (2019) Creating high fidelity 360° virtual reality with high dynamic range spherical panorama images. Virtual Creativity, Volume 9, Numbers 1-2, 1 December 2019, pp. 73-109(37), Intellect. (author’s copy) Creating High Fidelity 360° Degree Virtual Reality with High Dynamic Range Spherical Panorama Images Abstract This research explores the development of a novel method and apparatus for creating spherical panoramas enhanced with high dynamic range (HDR) for high fidelity Virtual Reality 360 degree (VR360) user experiences. A VR360 interactive panorama presentation using spherical panoramas can provide virtual interactivity and wider viewing coverage; with three degrees of freedom, users can look around in multiple directions within the VR360 experiences, gaining the sense of being in control of their own engagement. This degree of freedom is facilitated by the use of mobile displays or head-mount-devices. However, in terms of image reproduction, the exposure range can be a major difficulty in reproducing a high contrast real-world scene. Imaging variables caused by difficulties and obstacles can occur during the production process of spherical panorama facilitated with HDR. This may result in inaccurate image reproduction for location-based subjects, which will in turn result in a poor VR360 user experience. In this paper we describe a HDR spherical panorama reproduction approach (workflow and best practice) which can shorten the production processes, and reduce imaging variables, and technical obstacles and issues to a minimum. This leads to improved photographic image reproduction with fewer visual abnormalities for VR360 experiences, which can be adaptable into a wide range of interactive design applications. We describe the process in detail and also report on a user study that shows the proposed approach creates images which viewers prefer, on the whole, to those created using more complicated HDR methods, or to those created without the use of HDR at all. 1. Introduction Mobile Virtual Reality (VR) experiences have recently become popular, often based on immersive panorama images and 360 degree videos. Computational photography techniques involving high dynamic range (HDR) and spherical panoramas can be used to produce images for this type of photo-realistic VR. However this can be challenging because of the visual abnormalities produced by the combination of spherical panoramas and HDR. In this research, a technique is presented for fast HDR spherical panorama acquisition and post- processing that produces results indistinguishable from more complex techniques. This could enable much less complicated production of VR-ready content compared with traditional methods, thus opening out the field to users of all levels of ability, serving to democratise the medium whilst encouraging active engagement by a wide variety of users. Typically, VR360 experiences can be viewed on a personal mobile display and/or a VR head-mount-display (HMD). Both of these displays allow users to look around a spherical panorama using the interaction of gaze-control, hands-free navigation, or other user interface methods. This research describes an image-based solution using HDR spherical panorama images for high fidelity Virtual Reality 360 degree (VR360) interactive panorama projections. In this context, ‘high fidelity’ means that the proposed solution produces a highly detailed and accurate HDR spherical panorama image which resembles the scenic environment, with the aim of increasing fidelity, or high definition visual similarity and clarity, closest to the original scene depicted, and which can be used for VR360. However, the problem of inaccurately reproduced HDR spherical panorama can result in a biased representation for interactive panorama experiences, and difficulty for image-based 360 navigation and visualization. Hence there is a need to explore a feasible solution for panorama imaging reproduction that can be optimized for VR360 experiences. Such a 1 Zi Siang See, Lizbeth Goodman, Craig Hight, Mohd Sharizal Sunar, Arindam Dey, Yen Kaow Ng, Mark Billinghurst (2019) Creating high fidelity 360° virtual reality with high dynamic range spherical panorama images. Virtual Creativity, Volume 9, Numbers 1-2, 1 December 2019, pp. 73-109(37), Intellect. (author’s copy) solution will naturally be of benefit to creative practitioners from the gallery library archive museum (GLAM) sector, geographic information system (GIS) practices, virtual tourism, and a variety of practices within the overall domains of the creative industries as well as still unexplored potential within spatial documentation and scanning workflows in other sectors. In this paper we first summarize related background work, and then describe our method for creating HDR spherical panorama images. Then we present results of a user study comparing user perceptions of images generated with our technique to other existing methods. Finally we conclude the paper and describe directions for future work. 2. Background This research is based on earlier work in HDR imaging and methods for creating spherical panorama images for VR360 user experiences. VR360 can be used for various creative presentations, scientific documentation of subjects, or other applications. As noted above, however, panorama image reproduction has a few challenges which can result in the inaccurate reproduction of 360 imagery, including well-known factors of parallax error, nadir angle difficulty, and limited dynamic range, and these will be discussed in this section. 2.1 High Dynamic Range (HDR) Imaging The work by Debevec and Malik (1997) for High Dynamic Range (HDR) imaging is one of the early contributions that created the current trend of multiple exposures HDR computational photography. This is a technique that aims to increase the dynamic range of a photographic image by using combined multiple exposures, resulting in improved luminance reproduction in shadow and highlighted areas. The HDR photography process usually involves multiple exposures of images in order to combine additional dynamic range information acquired from the extra exposures for providing a more accurately representation of the wide range of real-world lighting conditions (Banterle et al. 2018, Reinhard et al. 2010, Ma et al. 2018, Nightingale 2012, Mantiuk et al. 2015, Yue et al. 2018). For example, Fairchild describes creation of HDR images from up to 18 differently exposed images for extremely high contrast scenes (Fairchild 2007). Most recently, a number of methods for expanding greater dynamic range from low dynamic range (LDR) images, also known as Expansion Operators, have been studied (Banterle et al., 2018) for reproducing HDR. It can be challenging to create HDR images due to needing multiple source exposures, which can cause different types of errors. For example, a ghosting effect may occur when moving objects such as people or cars are captured across multiple images (Karađuzović- Hadžiabdić et al. 2017, Steinmuller and Gulbins 2010, Reinhard et al.2010). Misalignment issues may exist when the source images are acquired without using a stable photographic equipment setup (Rafi et al 2007, Reinhard et al 2010), such as on a moving vehicle or pedestrian. Unpredictable changes of lighting or weather conditions can happen during long photo shoots producing different lighting conditions for some of the images. Figure 1 illustrates a common visual abnormality in an HDR image, a ghosting effect due to moving objects during the image acquisition. Multiple exposures for fusing globally/locally-adapted HDR Figure 1. Multiple exposures for HDR imaging with visual abnormalities, such as ghosting and blurring. 2 Zi Siang See, Lizbeth Goodman, Craig Hight, Mohd Sharizal Sunar, Arindam Dey, Yen Kaow Ng, Mark Billinghurst (2019) Creating high fidelity 360° virtual reality with high dynamic range spherical panorama images. Virtual Creativity, Volume 9, Numbers 1-2, 1 December 2019, pp. 73-109(37), Intellect. (author’s copy) 2.2 Spherical Panorama Spherical panorama computational photography techniques can produce wide angle panoramic images that are usually unable to be acquired from a conventional single angle photo. The typical technique involves photographic acquisition of images from multiple angles (Felinto et al. 2012, Ueberheide 2018, Guo et al. 2018, Gawthrop 2007, Kent 2017, DiVerdi et al. 2009, Koeva et al. 2017, Gledhill et al. 2003, Awang Rambli et al. 2009, Fisher et al. 2015, Fangi 2018), which are then combined into a panorama covering the extended viewing angle. The production process usually requires careful planning with very little room for imaging errors in order to archive a complete and usable multiple angle acquisition (Felinto et al. 2012, Jung et al. 2017). For the spherical panorama images presented in this paper, they are usually viewed in a equirectangular projection (a full spherical panorama can commonly be projected into a ratio to cover 360° x 180°). The use of a multi-shot or multi-row setup (Chen 1995, Benosman and Kang 2001) that captures multiple angle images can introduce errors. For example, parallax error can result in inaccurate photographic reproduction (Kent 2017, Fangi 2018). This occurs when two or more images that are to be stitched together were acquired from slightly dissimilar viewpoints which cannot be visually merged correctly in post-processing, resulting in a stitching error. Insufficient