Creating High Fidelity 360° Degree Virtual Reality with High Dynamic Range Spherical Panorama Images

Abstract This research explores the development of a novel method and apparatus for creating spherical panoramas enhanced with high dynamic range (HDR) for high fidelity Virtual Reality 360 degree (VR360) user experiences. A VR360 interactive panorama presentation using spherical panoramas can provide virtual interactivity and wider viewing coverage; with three degrees of freedom, users can look around in multiple directions within the VR360 experiences, gaining the sense of being in control of their own engagement. This degree of freedom is facilitated by the use of mobile displays or head-mount-devices. However, in terms of image reproduction, the exposure range can be a major difficulty in reproducing a high contrast real-world scene. Imaging variables caused by difficulties and obstacles can occur during the production process of spherical panorama facilitated with HDR. This may result in inaccurate image reproduction for location-based subjects, which will in turn result in a poor VR360 user experience. In this paper we describe a HDR spherical panorama reproduction approach (workflow and best practice) which can shorten the production processes, and reduce imaging variables, and technical obstacles and issues to a minimum. This leads to improved photographic image reproduction with fewer visual abnormalities for VR360 experiences, which can be adaptable into a wide range of interactive design applications. We describe the process in detail and also report on a user study that shows the proposed approach creates images which viewers prefer, on the whole, to those created using more complicated HDR methods, or to those created without the use of HDR at all.

1. Introduction Mobile Virtual Reality (VR) experiences have recently become popular, often based on immersive panorama images and 360 degree videos. Computational photography techniques involving high dynamic range (HDR) and spherical panoramas can be used to produce images for this type of photo-realistic VR. However this can be challenging because of the visual abnormalities produced by the combination of spherical panoramas and HDR. In this research, a technique is presented for fast HDR spherical panorama acquisition and post- processing that produces results indistinguishable from more complex techniques. This could enable much less complicated production of VR-ready content compared with traditional methods, thus opening out the field to users of all levels of ability, serving to democratise the medium whilst encouraging active engagement by a wide variety of users. Typically, VR360 experiences can be viewed on a personal mobile display and/or a VR head-mount-display (HMD). Both of these displays allow users to look around a spherical panorama using the interaction of gaze-control, hands-free navigation, or other user interface methods. This research describes an image-based solution using HDR spherical panorama images for high fidelity Virtual Reality 360 degree (VR360) interactive panorama projections. In this context, ‘high fidelity’ means that the proposed solution produces a highly detailed and accurate HDR spherical panorama image which resembles the scenic environment, with the aim of increasing fidelity, or high definition visual similarity and clarity, closest to the original scene depicted, and which can be used for VR360. However, the problem of inaccurately reproduced HDR spherical panorama can result in a biased representation for interactive panorama experiences, and difficulty for image-based 360 navigation and visualization. Hence there is a need to explore a feasible solution for panorama imaging reproduction that can be optimized for VR360 experiences. Such a 1

solution will naturally be of benefit to creative practitioners from the gallery library archive museum (GLAM) sector, geographic information system (GIS) practices, virtual tourism, and a variety of practices within the overall domains of the creative industries as well as still unexplored potential within spatial documentation and scanning workflows in other sectors. In this paper we first summarize related background work, and then describe our method for creating HDR spherical panorama images. Then we present results of a user study comparing user perceptions of images generated with our technique to other existing methods. Finally we conclude the paper and describe directions for future work.

2. Background This research is based on earlier work in HDR imaging and methods for creating spherical panorama images for VR360 user experiences. VR360 can be used for various creative presentations, scientific documentation of subjects, or other applications. As noted above, however, panorama image reproduction has a few challenges which can result in the inaccurate reproduction of 360 imagery, including well-known factors of parallax error, nadir angle difficulty, and limited dynamic range, and these will be discussed in this section.

2.1 High Dynamic Range (HDR) Imaging The work by Debevec and Malik (1997) for High Dynamic Range (HDR) imaging is one of the early contributions that created the current trend of multiple exposures HDR computational photography. This is a technique that aims to increase the dynamic range of a photographic image by using combined multiple exposures, resulting in improved luminance reproduction in shadow and highlighted areas. The HDR photography process usually involves multiple exposures of images in order to combine additional dynamic range information acquired from the extra exposures for providing a more accurately representation of the wide range of real-world lighting conditions (Banterle et al. 2018, Reinhard et al. 2010, Ma et al. 2018, Nightingale 2012, Mantiuk et al. 2015, Yue et al. 2018). For example, Fairchild describes creation of HDR images from up to 18 differently exposed images for extremely high contrast scenes (Fairchild 2007). Most recently, a number of methods for expanding greater dynamic range from low dynamic range (LDR) images, also known as Expansion Operators, have been studied (Banterle et al., 2018) for reproducing HDR. It can be challenging to create HDR images due to needing multiple source exposures, which can cause different types of errors. For example, a ghosting effect may occur when moving objects such as people or cars are captured across multiple images (Karađuzović- Hadžiabdić et al. 2017, Steinmuller and Gulbins 2010, Reinhard et al.2010). Misalignment issues may exist when the source images are acquired without using a stable photographic equipment setup (Rafi et al 2007, Reinhard et al 2010), such as on a moving vehicle or pedestrian. Unpredictable changes of lighting or weather conditions can happen during long photo shoots producing different lighting conditions for some of the images. Figure 1 illustrates a common visual abnormality in an HDR image, a ghosting effect due to moving objects during the image acquisition.

Multiple exposures for fusing globally/locally-adapted HDR

Figure 1. Multiple exposures for HDR imaging with visual abnormalities, such as ghosting and blurring. 2

2.2 Spherical Panorama Spherical panorama computational photography techniques can produce wide angle panoramic images that are usually unable to be acquired from a conventional single angle photo. The typical technique involves photographic acquisition of images from multiple angles (Felinto et al. 2012, Ueberheide 2018, Guo et al. 2018, Gawthrop 2007, Kent 2017, DiVerdi et al. 2009, Koeva et al. 2017, Gledhill et al. 2003, Awang Rambli et al. 2009, Fisher et al. 2015, Fangi 2018), which are then combined into a panorama covering the extended viewing angle. The production process usually requires careful planning with very little room for imaging errors in order to archive a complete and usable multiple angle acquisition (Felinto et al. 2012, Jung et al. 2017). For the spherical panorama images presented in this paper, they are usually viewed in a equirectangular projection (a full spherical panorama can commonly be projected into a ratio to cover 360° x 180°). The use of a multi-shot or multi-row setup (Chen 1995, Benosman and Kang 2001) that captures multiple angle images can introduce errors. For example, parallax error can result in inaccurate photographic reproduction (Kent 2017, Fangi 2018). This occurs when two or more images that are to be stitched together were acquired from slightly dissimilar viewpoints which cannot be visually merged correctly in post-processing, resulting in a stitching error. Insufficient dynamic range in the captured images is another obstacle in panorama imagery (Brown and Lowe 2006, Reinhard et al. 2010). The lighting conditions at the capture location may involve various types of light sources, and sometimes the luminance in exposure value is too great in a high contrast scene (Debevec and Malik 1997). Finally, the consistency of the white balance and exposures is vital in the multiple angle images using a multi-row setup and fast changing weather conditions can result in white balance inconsistency in different multiple angle images (Diverdi et al. 2009). Figure 2 demonstrates common difficulties and obstacles for spherical panorama creation using the multiple angle images method (in the form of equirectangular projections). A smaller amount of multiple angle images could potentially avoid image stitching errors and human handling variables during acquisition (Guo et al. 2018), however the appropriate amount of images depends on the limitations of the configuration available, and the desired panoramic image resolution. A shorter acquisition time allows for less occurrence of imaging variables and easier pre-visualization of the forecast spherical panorama reproduction outcome. For instance, moving objects such as a vehicle or person is ideally captured on a single angle instead of reappearing in multiple angles.

Camera or hand-shake

Limited dynamic range Ghosting /Parallax error

Figure 2. Example of hand-held panorama capture exaggerating and exhibiting visual abnormalities.

2.3 Spherical Panorama and HDR Earlier research has shown how HDR and spherical panorama can be combined together to provide an immersive VR experience. Various studies (Felinto et al. 2012, Brown and Lowe 2006; Steinmuller and Gulbins, 2010; Okura et al., 2014; Silk., 2011; Gawthrop, 2007; Popovic et al., 2014) have shown an interest in working with both the spherical panorama and high dynamic range (HDR) techniques and use cases. Figure 3 shows a typical configuration for generating an HDR spherical panorama. In this example, there are 9 pictures taken from each of 8 orientations at different exposure levels for a total of 72 source images. To achieve this with conventional photography, 9 exposures are acquired for each angle, ranging from darker to brighter images. This must be repeated for at least 8 angles (1 zenith, 1 nadir, 6 horizontal angles). Once the acquisition process is completed, a considerable amount of time is required to merge the multiple image exposures for each angle. Once completed, the HDR image for each angle can then be stitched into a panorama using commercial tools such as PTGui (PTGui, 2018). Some panorama software will work with various error correction features for multiple exposures but these have limited control for the use of digital negatives. However, major imaging errors tend to occur in multiple angle HDR images. In fact, it can at times prove extremely difficult to manage the stitching process successfully. As a result, familiar reproduction difficulties can include ghosting and misaligned bracketed images in multiple angles which cause poor quality in the panorama image outcome and ultimately in VR user experiences.

Spherical panorama multi-row HDR that requires multiple exposures configuration covering 360 degrees. • 9 exposures required for each angle • 8 angles required (image sequence) (example numbers of sequence)

-4, -3, -2, -1, 0, +1, +2, +3, +4 (EV)

Total images required = 72 images (simulated scenario) • 8 angles required (image sequence) • 9 exposures required for each angle

Figure 3. Photography production process for the combination of HDR and spherical panorama images.

As can be seen from the related work suggested in this section, there are well known methods for creating HDR and spherical images, although there are sources of error in various cases, and these have relied on time consuming conventional methods for stitching the source HDR images together into the panorama. For example, as multiple exposures are 4

required for each angle, such an approach will result in a lengthier capturing period and imaging variables can easily be introduced (Felinto et al. 2012; Fangi 2018), such as constantly changing real-world lighting and ghosting errors caused by moving objects. In the next section we describe a method we have developed for more efficient HDR spherical panorama creation.

3. Method and Apparatus As a practice-based research (Gray and Malins, 2004; Candy, 2006), we describe our method for efficient HDR spherical panorama creation. We first describe how we capture the source images, then the computation we used to create a series of HDR images from a single digital negative (RAW) image, and finally our approach for combining the HDR images into a spherical panorama.

3.1 Preparation Our research observed the results of extending dynamic range from digital negatives. Such findings are being used to find out if RAW extended exposure values (EV) can be practical for reproducing HDR from a single acquisition. This idea was mainly inspired by the workflow described by Debevec and Malic (1997) where such solutions were achieved by scanning the analogue film multiple times to obtain a greater dynamic range. The study attempted to observe the capability of a digital negative for how close the potential extendable dynamic range from single acquisition can match the actual native exposure values of multiple exposures acquisition. Digital negative which is also known as RAW format on digital camera equipment was found to have a possible extension of dynamic range (Adobe, 2019; Hill, 2010; Gakken, 2008) in the aspect of exposure quantification, potentially identical to how the exposure can be adjusted on a physical camera but crucially this is performed during the digital imaging post-processing stage with the help of RAW processors. A recent study by Hasinoff et al. (2016) used a group of burst RAW mobile-based images and recombined them to produce a high quality denoised image, with improved dynamic range, sharpness and clarity. In this study, the method and apparatus for producing high dynamic range (HDR) image from single source of acquisition has the speculated feature that image results can be accomplished completely free from ghosting effect and the misalignments issues common in high resolution panoramic capturing workflow (multi-row). Two cameras and their digital negatives have been studied, by mimicking native multiple exposures HDR. In the early stage of this study, Nikon D3x was used to verify the extendable dynamic range reproduced by a single source of acquisition in RAW, and the study observed the extended exposure range values from RAW at -4EV, -3EV, -2EV, -1EV, +1EV, +2EV, +3EV, +4EV to compare with the original value of these native exposures acquired from the camera for multiple exposures. The test was limited to the RAW produced by D3x at base ISO of 100. Nikon manufactured D850 with larger sensor became available in the later part of this study and was being used for testing for dynamic ranges, compared to the earlier attempts with D3x. Digital negatives produced by these two cameras behaved differently. The D850 (with base ISO64) is speculated to have greater usable dynamic range over the D3x. In the ideal situation, this would achieve a result comparable to the method of using 7-9 multiple exposures for HDR in a conventional photographic concept. Quantifying the extendable dynamic range was essential for this study. Sekonic J510 and Litemaster Pro L-478DR light meters were used for exposure reading and pre- visualisation. Figure 4 (a) illustrates exposures being inspected from D3x and (b) from D850,

both using nikkor 50mm f1.8mm, having the exposure parameters of -4EV, -3EV, -2EV, - 1EV, 0EV, +1EV, +2EV, +3EV, +4EV with RAW processor used and presented in the study, the approximation of exposure range was observed where the Neutral5 patch of the Gretagmacbeth colour checker chart were used to measure the darkest 0 to brightness 255 pixel value from the 18 photographed exposure sequences ranging from extreme darkness and brightness samples. Figure 5 (a) illustrates an example of native exposure (0EV) of D3x being extended with the range of -4EV and +4EV using RAW processor, and figure 5 (b) for D850 with similar settings for pixel value reading for observation. Image processing and preparations were all handled in Adobe 1998 color space for consistency.

(a) (b)

Figure 4. (a). Neutral patch reading from D3X, (b) from D850.

(a)

-4EV 0EV +4EV

(b)

-4EV 0EV +4EV

Figure 5. (a) native exposure (0EV) of RAW being extended with the range of -4EV and +4EV using RAW processor for D3x, and (b) for D850

Table 1 shows the pixel value reading of an 18 exposure sequence (S1-18) for D3x using CaptureOne Pro (C1Pro), with S1 being darkest and S18 the brightest, and exposures were adjusted manually in post-processing from S1 to S18 on a one stop increment basis, showing how RAW extended EVs perform on D3x with base ISO100 using C1Pro. The set of figure 6 (a), (b), (c) and (d) demonstrates how RAW extended EVs compared to the native EVs. When -EVs were extended from RAW, the intention was to perform highlight recovery (brighter details in image area). As can be seen, recovery did not perform too well in extending -3EV and -4EV as shown in figure 6 (c) and (d). However, figure 6 (a) and (b) have shown that the RAW extended EVs of -1EV and -2EV performed relatively close to how native EV was captured in the digital photograph workflow. The set of figure 7 (a), (b),

(c) and (d) shows shadow recovery worked quite well for RAW extended +1EV to about +3EV. However, real-world performance of exposure value recovery using RAW extended dynamic range approach must be tested to verify how do they work in actual photographic scenarios. Digital noise of images generated from RAW is another important aspect to look at.

Table 1. Pixel value reading with RAW processor (CaptureOne Pro) at Neutral5 Patch, for D3x with ISO100.

Extended Extended Extended Extended Native Extended Extended Extended Extended Sequence -4EV -3EV -2EV -1EV 0EV +1EV +2EV +3EV +4EV S01 0 0 0 0 0 0 1 2 3 S02 0 0 0 0 1 1 2 4 8 S03 0 0 0 0 1 3 6 10 20 S04 0 0 1 2 3 8 12 21 35 S05 0 1 2 4 6 15 26 43 69 S06 1 2 5 8 15 25 45 70 118 S07 2 4 8 16 26 47 73 120 179 S08 4 7 15 26 43 70 116 174 222 S09 8 15 27 45 73 117 173 222 247 S10 16 26 44 72 116 174 222 247 254 S11 27 45 70 116 175 221 248 254 255 S12 45 71 117 173 222 247 254 255 255 S13 70 115 172 220 249 254 255 255 255 S14 123 179 224 250 255 255 255 255 255 S15 152 207 241 255 255 255 255 255 255 S16 154 207 242 255 255 255 255 255 255 S17 155 208 242 255 255 255 255 255 255 S18 155 208 242 255 255 255 255 255 255

(a) (b) (c) (d)

Figure 6. Native and Raw extended -EVs (D3x, C1 Pro 7) for (a) -1EV, (b) -2EV, (c) -3EV, and (d) -4EV

(a) (b) (c) (d)

Figure 7. Native and Raw extended +EVs (D3x, C1 Pro 7) for (a) +1EV, (b) +2EV, (c) +3EV, and (d) +4EV

The same test was performed using the camera configuration of Nikon D850, with base ISO64, and table 2 shows pixel value reading for 18 exposure sequence (S1-18). Figures 8 (a), (b), (c) and (d) have shown how RAW extended EVs performed for -1EV to -4EV.

Similar to the previous camera used for testing, the D850 shows a fall in performing dynamic range recovery in -3EV and -4EV. It may be speculated that extending a recovering of image highlight works quite safely up to -3EV with such an equipment setup. In the situation of - 3EV, some pixels value image details will not be recovered completely. Figures 9 (a), (b), (c) and (d) illustrate usable stable performance in terms of shadow recovery for +1EV to +4EV. Image sequences in RAW extended +3EV and +4EV were observed to have very minimum digital noise which make a dynamic range recovery much more usable and predictable in real-world use cases in acquiring images for VR360 recreation.

Table 2. Pixel value reading with RAW processor (CaptureOne Pro) at Neutral5 Patch, for D850, with ISO64.

Extended Extended Extended Extended Native Extended Extended Extended Extended Sequence -4EV -3EV -2EV -1EV 0EV +1EV +2EV +3EV +4EV S01 0 0 0 0 0 1 1 3 4 S02 0 0 0 0 1 2 2 6 12 S03 0 0 0 1 2 3 6 11 21 S04 0 0 1 2 3 6 11 20 36 S05 0 1 1 3 6 12 20 35 60 S06 1 2 3 6 12 20 35 59 99 S07 1 3 6 12 21 36 60 100 149 S08 3 7 12 21 36 58 99 149 199 S09 6 12 21 36 59 99 150 198 233 S10 12 21 36 59 97 149 198 232 249 S11 20 36 59 98 149 198 232 249 252 S12 36 59 98 149 198 233 249 253 254 S13 59 99 150 198 232 250 253 254 255 S14 100 149 199 232 252 254 254 254 255 S15 125 175 217 245 255 255 255 255 255 S16 139 188 228 249 255 255 255 255 255 S17 139 188 228 249 255 255 255 255 255 S18 140 189 228 250 255 255 255 255 255

(a) (b) (c) (d)

Figure 8. Native and Raw extended -EVs (D850, C1 Pro 11) for (a) -1EV, (b) -2EV, (c) -3EV, and (d) -4EV

(a) (b) (c) (d)

Figure 9. Native and Raw extended +EVs (D850, C1 Pro 11) for (a) +1EV, (b) +2EV, (c) +3EV, and (d) +4EV

3.2 An Algorithm for HDR from Single Source of RAW In this section, the development of an improvised algorithm for producing an HDR image from a single source of RAW data is presented. The algorithm process can be operated by mapping or masking the individually RAW extended +2EV and -2EV layers (or +3EV and -3EV) using the monotonic RGB channel from 0EV as a base guide or reference. Figure 10 illustrates the HDR spherical panorama reproduction process we refined for preparing HDR image, having a RAW extended dynamic range extracted from a single source of RAW capture that has been processed from RAW processor used in the study. This gives an increased dynamic range result of approximately 12.5EV (14.5EV experimented in section 5) as indicated in the observation previously shown in figure 6, 7, 8, and 9.

Proposed configuration, multiple angle images acquisition

RAW processing, EV parameter, extending dynamic range

Extended EV (range -1ev to -4ev) 0EV (neutral /native) Extended EV (range +1ev to +4ev)

Masking on the expanded Allow manual selection of extended EV (example +2 EV) layer on top range for darker /brighter scene of the neutral (0EV) layer

monotonic mask from monotonic mask from RGB RGB channel channel (inverted) Highlight recovery Shadow recovery

Apply Image: 50% opacity on top of each layer.

Allow manual opacity of HDRI on top of LDRI

Blended result consistent for multiple angles, HDRI expanded from single source of RAW.

Figure 10. Algorithm, process of preparing the HDR extended from single RAW. 9

The algorithm for reproducing an HDR image from a single RAW source can reduce the imaging errors caused from multiple exposure images. Our algorithm was inspired by the work of Debevec and Malik (1997) who recovered extended dynamic range by combining multiple scans from a single source of analogue negative film. We describe an HDR method that uses a single acquisition that extends the dynamic range from the digital negative RAW format. The HDR image reproduced from single acquisition avoids the obstacles and issues that occur in a set of HDR images reproduced from multiple exposures. This provides an outcome with zero ghosting error, zero misalignment issues, and the least acquisition time. Figure 11 is the HDR computation (proposed solution) for working with source image from single acquisition of RAW format. This solution describes a HDR method that uses a single acquisition that extends the dynamic range in the digital negative RAW format which could be used for multiple angles for spherical panorama creation. Due to the fact that each HDR image is reproduced from a single source of capture using digital negative, this provides an intended outcome where problems associated with multiple exposures HDR were avoided. These familiar problems include ghosting error, misalignment issue and a lengthy acquisition time. Therefore, no multiple exposures correction features are needed (for ghosting and misalignment) for such an approach as the extended exposures are generated from same source of RAW. As we continued to test with a number of different RAW processers, most of these processors demonstrated fairly similar outcomes, but may have dissimilar strength in extracting extended EVs in extreme shadow and highlight areas.

HDR Im age Computation Input: Source image in digital negative RAW format (e.g. NEF) Output: An HDR image reproduced from a single acquired RAW image having its dynamic range extended from RAW

1. Three images, I, I+, I- are processed from the single source of RAW.

2. I+ and I- are respectively altered by +2EV and -2EV where dynamic range are extended. That new m old is, the luminance lx of each pixel x is recalculated as (lx) = 2 (lx) , where m is the change in EV.

3. A new image, I’+, is constructed out of I+, by preserving only the highlight regions of I+.

Similarly, a new image I’- is constructed out of I-, by preserving only the non-highlight regions of I-.

4. I’+ is then overlaid on top of I’- with 50% opacity (can be in any sequence if I’- is overlaid on I’+). The results of the two are then combined to form the final image I’.

Figure 11. HDR image computation using single RAW acquisition.

3.3 Creating the Spherical Panorama This section describes a method for combining spherical panorama and HDR images to create an HDR spherical panorama image for VR360. The early configuration method and apparatus using D3x with full-frame fish-eye Nikkor 16mm f2.8 lens in this study is capable of producing an image with width of 16000 pixels and height of 8000 pixels, equivalent to about 130 megapixels of large spherical image rendering. Using D850 with a similar fish-eye 16mm f2.8 lens, the solution can produce an outcome with the resolution of 22000 by 11000 pixels, giving it a large 242 megapixels reproduction of HDR spherical panorama. During the user study (using D3x), most of the images were reduced to 8000 x 4000 pixels at 300dpi for data collection and analysis. The creative component shown in section 5 of this study (using D850), however, has been preserved in the highest resolution at 242 megapixels (22000 x 11000 pixels) for inspection.

The prototype is an image capturing apparatus, based on the modified Manfrotto 303SPH with several additional customizations, including rotation components (horizontal rotation knob 2) added to the main aluminium arm as shown in figure 12 (Manfrotto, 2017). This configuration permits manual rotation for acquiring the nadir angle by having the extended arm pointing down to the nadir without much visible obstacles. Multiple angle images can be correctly obtained, which enables a high degree of genuine visual reproduction acquired from the original source.

Vertical rotation knob modification • Option of using default configuration by Manfrotto 303SPH • Modified extension ring or similar may be considered for flexible reach/view of nadir Additional space for capturing nadir

Horizontal rotation knob 1 modification • Additional levelling adjustment. • Additional counter weight is recommended.

Added horizontal rotation knob 2 • Must be added for capturing nadir

Figure 12. “See Through 360” Prototype Configuration.

Figure 13 demonstrates the prototype being deployed during multiple angle images acquisition from steps 1, 2 and 3. In an ideal situation, this set of proposed method and apparatus is capable of having a significantly faster capturing process, with an acquisition process of less than 30 seconds in laboratory condition (with assumed controlled lighting). Post-processing now has the advantage of working with genuinely acquired imagery resources. During post-processing, a choice is given to the creation process to either prepare each HDR angle with RAW extended EV individually or using stitched panorama images of native (0EV) and RAW extended EVs (-2EV, +2EV) for forming a final outcome of HDR spherical panorama, as shown in figure 13. This is achieved using the workflow and post- processing computation illustrated in figures 10 and 11. For the HDR workflow being used in

this study, we also mean globally-adapted HDR or tone mapping. This means the HDR image constructed by multiple exposures or additional dynamic range are compressed into a low dynamic range (LDR) display, and the output can then be made available for a wide range of image file formats (jpeg, tiff) and display scenarios (monitors, mobile screens and head- mount-displays). Using our method of HDR spherical panorama creation for VR360, we can reduce the manual or digital post-processing correction required if the equipment setup is carefully configured with minimal parallax error and human mistakes; with this a nadir angle can be reproduced with genuine content from the real-world scene, or having reduced or minimal visual errors. This is achievable by having the nadir perspective reproduced from additional images (step 3) in the multi-row configuration illustrated in figure 13.

Spherical panorama prototype HDR Spherical Panorama configuration covering 360 degrees • HDR that requires single source acquisition multiple angles, including nadir. (Extended dynamic range -2V and +2EV can • 11 angles required (image sequence) be reproduced from RAW processing)

(step 1)

Panorama image with RAW extended +EV

Panorama image with original 0EV (step 2)

Panorama image with RAW extended -EV

HDR Spherical Panorama (step 3)

Total images required = 11 images (successful scenario) • 11 angles required (image sequence) • 1 acquisition (RAW) required for each angle for HDR for panorama stitching

Speculated benefits for VR360 reproduction: • Potentially reduce reproduction processes • Potentially reduce or avoid difficulties and obstacles

Figure 13. High fidelity solution for VR360 using HDR spherical panorama

4. Experimental Validation A key question with the proposed method and apparatus (our method) is how good are the HDR spherical panorama images created using our method compared to those made with a more traditional approach (as shown in figure 3)? In order to evaluate this, a user experience study was conducted. Our target application was VR viewing of 360 degree panorama images. Virtual Reality 360 degree (VR360) interactive panorama image presentations can be made available for various platforms, including desktop computers, mobile and wearable devices. There are several mobile-based VR viewers that can be used with a smartphone. In this study, we hypothesized that a high fidelity reproduction of location-based imagery content could create a realistic user viewing experience for VR360 interactive panorama presentations. There is a lot of interest in exploring panoramas reproduced with a high amount of visual information which can make onlookers believe that what they are looking at is real (Benosman and Kang 2001). It is also ideal that the luminance reproduced in the images could accurately represent the photographed real-world locations (Reinhard et al. 2010, Banterle 2018). A user experience study was conducted validating the performance and quality of the panorama produced by our approach. The VR360 experiences were tested with a variety of display devices, including a computer monitor, mobile display and head-mount-device display. Our study focused more on the perceived image quality (in the context of HDR spherical panorama) instead of comparing the usability of various devices. We did, however, include three types of displays (computer screen, mobile screen, and HMD) for comparing our HDR spherical panorama tests.

4.1 Participant Demographics In this experiment we recruited a total of 84 participants distributed randomly in three groups. Table 3 provides their demographic information and previous experience with interactive panorama and VR displays on a scale of 1 to 7, where 1 is no experience and 7 is very experienced.

Table 3. Participant demographic.

Display Age Exp. with Interactive Exp. with VR Displays M (No of. Participants) M (SD) Panoramas M (SD) (SD)

Desktop/Laptop (29) 21.8 (3.4) 2.96 (1.5) 2.48 (1.62)

Female (18) 20.6 (1.3) 2.88 (1.36) 2.56 (1.46)

Male (11) 23.7 (4.9) 3.1 (1.7) 2.36 (1.91)

Mobile device (27) 22.6 (4.8) 3.3 (1.46) 2.07 (1.49)

Female (14) 20.4 (0.7) 3.21 (1.31) 2.07 (1.59)

Male (13) 25 (6.1) 3.38 (1.66) 2.08 (1.44)

Oculus Gear VR (28) 22.5 (4.9) 2.89 (1.53) 1.11 (0.42)

Female (13) 21.9 (3.7) 2.54 (1.61) 1.23 (0.6)

Male (15) 23.1 (5.9) 3.21 (1.42) 1 (0)

Overall (84) 22.3 (4.4) 3.05 (1.48) 1.89 (1.41)

4.2 User Experience Study To evaluate the perceived quality of images a user experience study was performed. Three VR360 types of sampling methods were investigated Type A (with LDR), Type B (our approach with early configuration using D3x), and Type C (with multiple exposures HDR spherical panorama). To further investigate the perceived quality on different kinds of displays we decided to use a desktop/laptop display (15 inches display with 1920x1080 pixels resolution), mobile device (Samsung manufactured SM-N9208ZDUXME/Note 5 with 2560x1440 pixels resolution), and Oculus Gear VR HMD (Samsung manufactured SM- R322NZWAXAR) with the same mobile device. Three groups of participants were recruited, one for each display type; 29 participants for the desktop group, 27 for mobile displays, and 28 for Oculus Gear VR. There were also six different sets of images which we processed with the three sampling methods, which resulted in 18 total images. Figure 14 shows six sets of sample images differentiating Type A, B and C. Participants were shown three versions of each image in a randomized order and repeated the process for all six sets of images. Each image was shown to the participants for less than a minute, so the total time involved for viewing 18 images was less than 18 minutes. After seeing the first set, participants were asked to rank them in order of their perceived image quality as shown in table 4. These were answered by rating three types of the images in each set, by the rank order of Most Realistic, Average Realistic and Least Realistic. This ranked data was recorded and transposed for analysis on a scale of 1 to 3, where Most Realistic = 1 (best), Average Realistic = 2, and Least Realistic = 3 (worst). For the user experience study, the numbers of “1 to 3” were not visible to the users; these were only used for back-end statistical records and analysis purposes. Data was checked to ensure that users did not provide any duplicate entries for any of these answers. Table 5 illustrates the differences in participant responses to Type A, B and C images. Our hypothesis was: Type B (our method) would result in the lowest rank across all different display types and images. As the ranking data was nonparametric in nature, we used Related- Samples Friedman’s Two-Way ANOVA to analyse it separately for each display type. The two repeated measure factors were sampling method and images. For the post-hoc analysis we used the Dunn-Bonferroni test from the statistical package SPSS. The analysis will be reported in four sections. First, the overall ranking is presented for each image, followed by the reported ranking of the three display types. The three types of images for the six sets are sorted in a counter-balanced order during the user study.

Table 4. Six sets of images as each contains Type A, B and C being ranked.

Images Description

Set 1 - 6 Can you please rank the following images, from 1 to 3 in order on how photo-realistic is the image quality? Each set contains: Type A Most realistic Type B Average realistic Type C Least realistic Note: Six sets of images were arranged in counter-balanced order comparing Type A, B, C.

Type A Type B Type C

Set 1

Set 2

Set 3

Set 4

Set 5

Set 6

Figure 14. Example of type A, B (our method) and C of image comparisons.

Table 5. Differences of Type A, B and C in six sets of the images being tested.

Comparison Description

Type A Generic LDR-based panorama method (approx. 8.5EV): 8 images were acquired (1 zenith, 6 horizontal, 1 nadir: 16mm full-frame fish eye lens were used on Nikon manufactured D3x, each image was in 24.5 megapixels). Typically, these required 25-35 seconds to complete the acquisition.

Type B Our method (with approx. RAW extended 12.5EV): 11 images were acquired (1 zenith, 6 horizontal, 4 nadir: using a customized camera-rig as shown in figure 13, 16mm full-frame fish eye lens were used on Nikon manufactured D3x, each image was in 24.5 megapixels). Typically, these required 30-45 seconds to complete the acquisition.

Type C Generic multiple exposures HDR panorama (approx. 16.5EV): 72 images were acquired, each angle requiring 9 images for generated multiple exposures HDR images (1x9 zenith, 6x9 horizontal, 1x9 nadir: 16mm full-frame fish eye lens were used on Nikon manufactured D3x, each image was in 24.5 megapixels). Typically, these might require 600-1200 seconds to complete the acquisition due to unknown environment variables. Subject to the nature of the multiple exposures HDR approach, the amount of time needed can be significantly higher as compared to a non-HDR approach.

All Displays Together Initially we considered all 84 participants’ data together, and did not separate out depending on display type used to view the content. Participants ranked the images viewed from the three image creation methods on a the choices of Most Realistic, Average Realistic, Least Realistic (with the back-end statistical scale of 1 to 3, where 1 = best and 3 = worst). Overall, we noticed significant differences between the different types of image creation methods. Referring to figure 14, for images 1 to 4, Type A was found to be ranked significantly higher (i.e worst, least preferred) than Type B and Type C, corresponding to a worse ranking score. In Image 5, Type B was ranked significantly lower (better, preferred) than Type A and Type C. In Image 6, Type B was ranked significantly lower than Type A and Type C; Type A was ranked significantly lower than Type C. In all images, however, Type B was ranked lowest, which showed a clear preference for Type B. The average ranking for each type and statistical analysis is shown in Table 6. In all cases A was ranked worse than B and C, and in most cases B and C are ranked the same. In two sets of images B is ranked the lowest (better, preferred). Both sets of images B ranked the best were produced in an indoor environment, which had fixed sources of lightings and the real-life dynamic range may not have as much contrast as an outdoor environment with constantly changing sunlight.

Table 6. Average Ranks: All Displays together. In post hoc analysis, < indicates ranked significantly lower than and > indicates ranked significantly higher than.

Type A Type B Type C χ² (2)= , p Post hoc analysis

Image 1 2.44 1.67 1.89 26.6, p <.001 A > B & C

Image 2 2.7 1.57 1.73 63.17, p < .0001 A > B & C

Image 3 2.67 1.62 1.71 56.38, p < .0001 A > B & C

Image 4 2.51 1.67 1.82 34.02, p < .0001 A > B & C

Image 5 2.14 1.35 2.51 59.74, p < .0001 B < A & C

Image 6 2.12 1.23 2.65 87.5, p < .0001 B < A & C, A < C

Computer Desktop or Laptop Screen This section reports on images displayed on desktop or laptop screens to 29 participants. The average ranking for each type and statistical analysis is shown in Table 7. We noticed that for Image 2, Type A was ranked significantly higher than Type B and Type C. For Image 3, Type A was ranked significantly higher than Type B. In images 4 to 6, Type C was ranked significantly higher than Type B. In Image, 6, Type C was also ranked significantly higher than Type A. Overall, a general preference for Type B is evident in all images.

Table 7. Average Ranks: Computer or Laptop Displays. In post hoc analysis, only conditions showing significant differences are shown and < indicates ranked significantly lower than and > indicates ranked significantly higher than.

Type A Type B Type C χ² (2)= , p Post hoc analysis

Image 1 2 1.69 2.31 5.57, p = .061 No Significant Differences

Image 2 2.59 1.62 1.79 15.38, p < .0001 A > B & C

Image 3 2.41 1.72 1.86 7.72, p = .021 A > B

Image 4 2.03 1.66 2.31 6.28, p = .043 C > B

Image 5 2.14 1.55 2.31 9.17, p = .01 C > B

Image 6 1.86 1.41 2.72 25.72, p < .0001 C > A & B

Mobile Device Screen In this section, we only consider images displayed on a mobile device screen. A total of 27 participants viewed the images on a mobile screen. The average ranking for each type and statistical analysis is shown in Table 8. For all images, Type B was ranked the lowest (i.e better, preferred). In Image 1, Type B was ranked significantly lower than Type A. In Image 17

2, Type A was ranked significantly higher that Type B and Type C. In images 3 and 4, Type C was ranked significantly higher than Type A and Type B. In images 5 and 6, Type B was ranked significantly lower (better, preferred) than Type A and Type C.

Table 8. Average Ranks: Mobile Device Displays. In post hoc analysis, only conditions showing significant differences are shown and < indicates ranked significantly lower than and > indicates ranked significantly higher than.

Type A Type B Type C χ² (2)= , p Post hoc analysis

Image 1 2.44 1.63 1.93 9.18, p = .01 B < A

Image 2 2.59 1.59 1.81 14.89, p = .001 A > B & C

Image 3 2.67 1.56 1.78 18.67, p < .0001 C > A & B

Image 4 2.56 1.7 1.74 12.52, p = .002 C > A & B

Image 5 2.04 1.33 2.63 22.74, p < .0001 B < A & C

Image 6 2.15 1.26 2.59 24.89, p < .0001 B < A & C

Oculus Gear VR Screen Twenty-eight participants viewed the images through a Oculus Gear VR display. In this section we discuss about this ranking data. The average ranking for each type and statistical analysis is shown in Table 9. In images 1 to 4, we noticed that Type A was ranked significantly higher than Type B and Type C. In images 5 and 6, Type B was ranked significantly lower (better, preferred) than Type A and Type C. Again, in all images Type B was the clear preferred option for participants.

Table 9. Average Ranks: Oculus Gear VR Displays. In post hoc analysis, only conditions showing significant differences are shown and < indicates ranked significantly lower than and > indicates ranked significantly higher than.

Type A Type B Type C χ² (2)= , p Post hoc analysis

Image 1 2.89 1.68 1.43 34.36, p < .0001 A > B & C

Image 2 2.93 1.5 1.57 36.29, p < .0001 A > B & C

Image 3 2.93 1.57 1.5 36.29, p < .0001 A > B & C

Image 4 2.96 1.64 1.39 39.93, p < .0001 A > B & C

Image 5 2.25 1.14 2.61 32.64, p < .0001 B < A & C

Image 6 2.36 1 2.64 43.14, p < .0001 B < A & C

Figure 15 shows the mean rank based on the three types of displays tested, the whiskers represent +- 95% confidence intervals. We observed users’ preferences towards Type B images (our method) especially for the image set 5 and 6, which were panorama

images acquired in an indoor environment (See figure 14). Image set 5 was acquired in an automobile showroom with fixed lighting conditions at midnight. Image set 6 was acquired in a casual office environment with some amount of sunlight from the windows. Both of these sets of images required a slower shutter speed for image acquisition. For instance, in the case of using the Type B method, the image in set 5 were acquired at 0.67 seconds for each angle. From a technical perspective, images taken with a slower shutter speed are more likely to have ghosting effects caused by multiple exposures fusion and moving objects. This degrades the viewing experiences for 360 degrees panoramas in the case of Type C. To illustrate, a user who preferred Type B image commented that he felt “image Type C has a strange moving effect for the subject sitting on sofa, and bottom (viewing angle) has error.” We think this meant ghosting visual errors caused by the multiple exposures HDR image (as demonstrated in figure 16).

Figure 15. Mean rank based on three types of displays (the whiskers represent +- 95% confidence intervals).

It was also observed that a few users who tested with mobile devices commented that the visibility of the smaller mobile screen might make it too difficult to evaluate the images as fewer distinguishable details could be seen. Some HMD users appeared to have difficulty in differentiating and evaluating Types B and C for images 1 to 4 (for those samples acquired from outdoor conditions), and were confused as to which provided the better image quality. Two aspects that lead to this situation were observed. First, Type C images have greater coverage (16.5EV) of shadow and highlight of outdoor environment, and Type B covers only about 12.5 EV using current configurations (our early configuration using D3x). Secondly, Type C images were prepared using a Photomatix HDR processor which demonstrated excess saturation in image rendering which may have increased user preferences towards using this type of HDR images for outdoor environments. In addition to the survey, we asked participants who preferred Type B images for their comments on the image quality. Users who preferred Type B said “.. I prefer image Type B, the environment looks more natural. ”, “the floor which is shown in image Type B is cleaner than in the other image types”, “I won't be able to see the tripod at Type B, color is good too.”, “the clarity of image Type B is extremely good.”, “brightness is important with photo details”, “Image Type B has more details and colour is nice.”, and “sample Type B has balance of tone”. Overall users were very impressed with the image quality and details shown in images produced by (our) method B. According to some users who preferred Type C images, they seemed to prefer the panorama photos with exaggerated colors as compared to photo-realistic colors. Users expressed that the exaggerated colors may provide a more beautified look and feel, and may be tolerated with some visual errors. For example they said “.. both image Type B and Type C are good, Type B is cleaner but the Type C has beautiful color... ”, “image Type C has good color but looks incorrect when viewing down”, and “image Type C has very rich tone, but I prefer the clean bottom when viewing the image Type B”. Users felt “.. image Type C is very colourful, saturated, and the lighting seems to give good feeling (it is more artificial, like a post-production made movie).”, “image Type C is probably too colorful (however less photo- realistic), but it feels good.” and it “… has a lot of details in building, but the bottom has error and some content is blurry.”. On the other hand, some users commented that it was difficult to choose between Type B and Type C experiences. For example, one user said “Image Type B and Type C are both quite good in brightness.” and “some images are confusing to make a decision”. For future studies, we would be interested in investigating further to find out if the viewing experience provided by the Type B method could be improved with an increased dynamic range closer to Type C’s 16.5 EV.

4.3 Perception In additional to the main user study in section 4.2, we requested 84 participants to provide some additional input on the perceived importance of dynamic range (shadow and highlight), and the level of photo-realism and resolution. We wanted to understand if the user preferences were different in dissimilar types of displays for 360 viewing experiences (computer screen, mobile screen and HMD). Participants were also asked a set of subjective Likert scale questions, shown in Table 10. These were answered on a scale of 1 to 7, where 1 = strongly disagree (not very important) and 7 = strongly agree (very important). We ran a 20

one-way ANOVA (Display being the independent variable) followed by Tukey HSD post- hoc test to analyze the data in recorded in Likert-scale based questions, with the results shown in table 11.

Table 10. Likert scale questions of user’s perception.

Descriptions

Question 1. For interactive panorama, is visual information available in the shadow and highlighted areas important?

Question 2. When viewing an interactive panorama, is photo-realism important to you?

Question 3. For interactive panorama, is image resolution important to you?

Table 11. Mean (SD) values of rating for three displays in three different questions.

Display Desktop (n=29) Mobile (n=27) VR (n=28)

Question 1 4.79 (1.00) 5.07 (1.27) 5.61 (0.74)

Question 2 5.38 (1.24) 5.93 (1.21) 6.25 (0.97)

Question 3 5.21 (1.26) 6.19 (1.15) 6.82 (0.39)

We noticed a significant effect on displays on participants’ rating in Question 1: F(2, 83) = 4.67, p=.012. The post-hoc test revealed that VR display was rated significantly higher than the Desktop display (p=.01). This indicates that participants felt that for interactive panoramas, visual information available in the shadow and highlighted areas was more important in VR displays than in Desktop displays. We did not find any other significant differences.

On the other hand, we noticed a significant effect of display type for the participants’ rating for Question 2: F(2, 83) = 4.23, p=.018. The post-hoc test revealed that the VR display was rated significantly higher than the Desktop display (p=.014). This indicates that participants felt photorealism was more important in VR displays than in Desktop displays. We did not find any other significant differences.

There was a significant effect of display type for the participants’ rating for Question 3: F(2, 83) = 18.44, p < .001. The post-hoc test revealed that VR display was rated significantly higher than Desktop displays (p < .001) and almost significantly higher than Mobile displays (p=.057). This indicates that participants felt that image resolution was more important in VR displays than in the two other display types.

We observed that resolution in VR displays is crucial for providing a usable viewing experience. The level of spatial awareness is higher as HMD users experience an undistracted 360 visual content. This appears to lead to a greater level of immersion where the VR display users viewed spherical panorama intuitively with 3 Degrees of Freedom (Yaw, Pitch, and Roll), and the quality of photographic image reproduction was much more noticeable as 21

Zi Siang See, Lizbeth Goodman, Craig Hight, Mohd Sharizal Sunar, Arindam Dey, Yen Kaow Ng, Mark Billinghurst (2019) Creating high fidelity 360° virtual reality with high dynamic range spherical panorama images. Virtual Creativity, Volume 9, Numbers 1-2, 1 December 2019, pp. 73-109(37), Intellect. (author’s copy) compared to a computer screen or mobile display. As illustrated in the analysis, obtaining and presenting adequate dynamic range in 360 content can be as essential to having sufficient resolution for photo-realistic spherical panorama reproductions. This is especially noticeable for the case of HMD VR display users. The user experience study presented in this section was qualitative, empirical and subjective, and the users experienced a similar set of images processed using different approaches. However, in addition to the user experience study, Section 3.1 illustrated quantitative data that supported the idea that RAW extended EVs are relatively close to the native EV, and this implies that the usable EVs extended from a single captured RAW could be used for generating HDR spherical panorama for high fidelity VR360. The key intention for this work is not to compare various HDR algorithms but to focus instead on comparing the feasibility of using a single captured RAW generated HDR image with conventional multiple exposures HDR images. There are a number of multiple exposures HDR methods and algorithms (as shown in the previous work) and some HDR processers or algorithms work better than others with various de-ghosting or alignment features. However, the concern in this work is on exploring how may we avoid these image abnormalities by rethinking our capturing workflow. In the future, it is crucial to explore ways to increase the capability of obtaining dynamic range in our Type B approach using a single source of digital negative closer to the exposure values of Type C. Doing so provides image producers or photographers greater control of the EVs to be preserved based on different kinds of indoor and outdoor scenarios. Figure 16 and 17 shows a comparison of VR360 images using our method (Type B) compared to a generic conventional multiple exposures panorama (Type C).

(a) (b)

Figure 16. comparing (a) Type B our method, and (b) Type C multiple exposures HDR in VR360.

(a) (b)

Figure 17. comparing (a) Type B our method, and (b) Type C multiple exposures HDR in VR360. 22

5. Field Work Demonstration

This section features several VR360 produced using Nikon D850 (which was available in the later stage of this study) with the proposed method and apparatus, as a demonstration of practise-based research and application. All VR360 images were produced using RAW extended HDRs. Figure 18 shows a VR360 of “Susan Gilmore Beach”. For this VR360, a subjective choice of dynamic range expansion was made using RAW extended - 2EV and +2EV for giving this HDR spherical panorama result. Image quality of 242 megapixels has provided a high level of detail for this VR360, free from major imagery errors and abnormalities. Figure 19 shows this VR360 being magnified (zoomed-in) for inspection with a high level of preserved tonality, and such VR360 outcomes are compatible with commonly available web browser displays and enabled with HMD feature for an adequate VR experience.

Figure 18. HDR spherical panorama of “Susan Gilmore Beach”.

Figure 19. Magnified details of near-error-free VR360.

“The Memorial Walk” (also known as the Anzac Memorial Walk) provides a significant view toward the shoreline of Newcastle, Australia, as shown in figure 20. This image is used for a VR360 that shows a successful attempt at capturing 360 content in a low- light and windy sunset scenario. The image details have been well-preserved. HDR using RAW dynamic range has been used (-2EV and +2EV) in reproducing this adequate VR360 with a high level of authentic imagery information. Figure 21 shows VR360 content being enabled with stereo view on a mobile device, and this can be adapted on an easily available HMD, accessible by users.

Figure 20. HDR spherical panorama of “Memorial Walk, Newcastle”.

Figure 21. VR360 content being enabled with stereo view on a mobile device.

Figure 22 demonstrates similar content being viewed at micro level on a digital display. Figure 23 shows a high level of authenticity in viewing the nadir angle of this VR360. This example involved zero image manipulation using image correction tools or software. The use of the method and apparatus proposed in this study for the VR360 illustrated and achieved high fidelity results for image reproductions which resemble the original conditions of the on-site locations. Having an authentically captured nadir angle to be included in a high resolution (242 megapixels) VR360 clearly provides a great amount of imagery detail enhanced with HDR for a reliable visual experience.

Figure 22. magnified details of near-error-free VR360.

Figure 23. Adequate reproduction of nadir in VR360.

The “Customs House” is an important heritage landmark of Newcastle, Australia (figure 24). It was a public building designed by British colonial architect James Barnet. This VR360 demonstrated the use of the proposed method and apparatus with "first attempt success", reproduced with HDR using a RAW extended -1EV and +3EV (with +1EV used as a reference image) to cope with the extreme lighting conditions (natural and human-made light sources). As it can be seen, the nadir angle was genuinely produced, without a need for post-processing image manipulation. Figure 25 shows preserved details at a very high magnification.

Figure 24. High fidelity VR360 of “Customs House”.

Figure 25. Magnified view of the VR360 of “Customs House”. 26

As shown in the example of “Customs House”, RAW extended -1EV and +3EV were used instead of RAW extended -2EV and +2EV. Figure 26 demonstrates the various flexible use cases of HDR reproduced with RAW extended dynamic range. This illustrates several possible ways to use the proposed solution for a wide range of high contrast VR360 scenarios.

RAW Extendable EVs

-4EV -3EV -2EV -1EV 0EV +1EV +2EV +3EV +4EV

-2EV 0EV +2EV

-3EV 0EV +3EV

-1EV +1EV +3EV

-3EV -1EV +1EV

Figure 26. Various flexible use cases of HDR reproduced with RAW extended dynamic range

Figure 27 shows a range of practise-based results of VR360 reproduced using the proposed method and apparatus, using various RAW extended dynamic range settings for different lighting scenarios. We assume a wide range of users, creators and producers of VR360 will benefit from this type of imaging solution and approach as it allows 360 content authoring using HDR spherical panorama to be achieved with minimum difficulty and visual abnormalities. As is being explored in a proliferating set of case studies, virtual reality has a wide range of possible applications and implementations. Knowledge about and an understanding of the empirical method and apparatus outlined in this study, could especially benefit VR content providers, AR/MR/XR content creators, imaging system manufacturers, photographers, software developers and mobile applications developers. The solution could be applied to a variety of scenarios in tourism industries, cultural heritage, urban morphology, architectural visualisation, museums, art galleries, manufacturing industries, commercial showrooms and corporate information technology projects. We assume a quick and robust 360 capture workflow might also be attractive to law enforcement seeking accurate documentation of evidence paired with consistently gained spatial information. Quick and accurate 360 photographic capture could prove an asset in disaster management scenarios, data capture around environmental change and so on. Allied with more automated workflows, such spatialised image capture could foster new applications tasking with finding value from the density of visual data contained in such ground-level sensing techniques. Another long- 27

(a) Newcastle Beach (b) Secret Path of the Cliff

-2EV, 0EV, +2EV -3EV, 0EV, +3EV

(e) Verandah (f) Hangout Space

-2EV, 0EV, +2EV -2EV, 0EV, +3EV

(g) The Obelisk (h) King Edward Park

-2EV, 0EV, +2EV -2EV, 0EV, +2EV Figure 27. Various flexible use cases of VR360 produced with RAW extended dynamic range. 28

6. Discussion “Reducing the problem by simplifying the process” is exactly the idea we wish to introduce in this project. The user study conducted in this study (section 4) revealed that some users were uncertain if “more is good”. HDR spherical panorama can now be accomplished with usable nadir using the proposed solution as demonstrated in various examples of VR360 applications shown in this study.

Using our method, we have found that extending the high dynamic range with digital negative (RAW) is feasible. In this case it was possible to extend the exposures captured by the Nikon D3x by gaining additional RAW extended EVs of -2EV and +2EV. With this RAW extended dynamic range, the additional -2EV and +2EV can be used to reproduce a HDR image from a single acquisition as tested om the user study (section 4.2). RAW extended dynamic range up to -3EV and +3EV using D850 in later part of the study (section 5) has also found working with great flexibility in extendable dynamic range can be useful in many real-world situations for reproducing VR360 content. It is arguable as to what is the ideal range for reproducing an adequate HDR (Reinhard et al 2010). Additional values extended from the RAW image which exceeds the range of -3EV and +3EV can be explored in future studies. For creating HDR spherical panoramas, some issues can be fixed by having manual or digital post-processing correction. As a photographic approach, it is vital to plan the pre- visualization imagery outcomes in order to minimize certain predicted difficulties from the beginning (Adams 1983). Carefully planning for the acquisition process should aim to reduce human mistakes and minimize the chances of getting parallax error. This can result in optimum HDR spherical panorama results without wasting too much effort or resources on post-processing corrections. The suggested method of having the nadir perspective reproduced with additional images shown in our production framework is one way to produce a clean result. Table 8 provides a list of some of the benefits of our approach compared to more traditional methods. As can be seen, our method should lead to key benefits in obtaining HDR spherical panorama using the least amount of time with reduced image errors which require rectification.

Table 8. Improvements from our approach combining HDR and spherical panoramas

Benefit and Optimization HDR Spherical Panorama

Single acquisition approach for Obstacles which have been avoided: RAW extended dynamic range has • Inconsistent HDR adaptation when avoided the following problems: stitching panorama with multiple • Ghosting error when using angles Reduced image errors multiple exposures • Nadir error which require rectification • Inconsistent lighting conditions for multiple exposures Obstacles to be minimized: • Misalignment when using • Parallax error multiple exposures • Ghosting error in multiple angles

Single acquisition using RAW The solution and prototype instead of multiple exposures. The configuration allows genuine nadir proposed HDR workflow allows us angle to be captured, this brings the Reduced or redesigned to decide the degree of RAW capability to capture authentic visuals processing steps extended dynamic range to be used for high fidelity VR360 without seeing for the image outcomes during post- the “photographer or equipment”. processing.

Only single acquisition for each Multiple exposures are not required for Reduced capturing time angle is required when using RAW each angle, so the entire acquisition extended dynamic range approach. process can be much faster.

As a reflection of the learning outcomes during the process of creative component creation, it is worthwhile to discuss some of the key understandings and considerations. The notion a "lean process" has been emphasised and highlighted throughout this study, as it suggests an approach focused on avoiding errors from the beginning instead of fixing them later, with extra effort and wasted resources. Producing high fidelity VR360 using HDR spherical panorama with dynamic range extended from a single source of digital negative (RAW) can be achieved with some of the following suggestions for best practice: i) As a first step, predict the intended image outcome supported by individual decisions for choosing feasible photographic scenarios, ideal timing and locations to avoid accidently capturing undesirable fast-moving objects. ii) It is recommended to use the GretagMacbeth Color Chart or X-Rite ColorChecker for maintaining a constant white balance for multiple angle images. A pre-shot can be captured, using the equipment used for multiple angles, to obtain a reliable white balance value to be applied for all angles. iii) Plan ahead to perform a complete capture for the entire set of multiple angle images. The total time needed will be shorter for outdoor daylight conditions as compared to night and indoor conditions. It has been found that optimum results can be achieved in the priority sequence of capturing the horizontal angles, then zenith and finally the nadir angles. A shorter acquisition time suggests the possibility to keep imaging variables and errors to a minimum, from both the aspects of HDR and spherical panorama reproduction. 30

iv) Avoid working with image sources that require heavy rectification. HDR extended from a single source is a way to allow robust image stitching using panorama imaging processors. Working with a clean source of HDR images for spherical panorama allows a vast amount of imaging detail and clarity of imagery features to be preserved. v) If the option to choose the extent of the exposure value (EV) to be preserved is given, we recommend capturing the maximum range EV with the right equipment and reviewing it later to finalise which dynamic range should be used for an adequate VR360 output. vi) Avoid photographic situations with unmanageable shadows cast by the placement of equipment, tripod or even the photographer. As shown in the examples of Image set 3 (from figure 14), the nadir angles were hampered by minor shadows which degraded the outcome of the HDR spherical panorama. In some situations, this can be managed by using manual correction in the post-processing stage with little intervention. Another situation to avoid, if possible, is capturing the nadir angle with a reflective flooring surface. Such a scenario complicates the accuracy of stitching as multiple angles may not have a good consistency in nadir angles. Figure 28 shows panoramic content can be made available for VR360 viewing experiences on various displays or platforms - such as using a HMD powered by a mobile phone or viewing the similar content using a mobile device. Some mobile user interfaces may even include a feature for toggling the display to a stereo-view (as with VR360s presented in Section 5 “Field Work Demonstration”), which would be compatible with a number of personal peripherals such as Google Cardboard or other mobile-based HMDs. These viewing experiences are becoming handy and widely available, from social media to corporate websites where the demand of working with spherical panorama is likely to increase in various practical use cases, as well as practices which are not geared toward public display, as outlined briefly above.

Figure 28. HMD and mobile device viewing similar spherical panorama (VR360 experience)

Before VR360 had become ubiquitous (though still relatively expensive for the average person), the level of fidelity which VR360 now provides was not commonly possible using standard photographic equipment, and as a result many people did not have sufficient access to the power of image creation. Now, using VR 360, everyone with access to the equipment can, at least for one moment, immerse themselves into a different virtual world. This power, and the joy and learning that it can bring, should in our view be available to all. 31

However, a number of off-the-shelf items of equipment and common solutions do not produce and provide VR360 images that are entirely free from imaging errors. VR360 is becoming increasingly available for use in a variety of sectors and for end-users in quite different practical domains. Just within the creative indsutries, for example, experts and newcomers alike are showing interest in using it for transmedia immersive storytelling; for telling community stories; for citizen journalism; for virtual tourism; and for connected health. For example, patients with limited physical mobility can now enjoy VR360 and various virtual reality experiences as a mode of ‘virtual travel’ - this brings a new level of inclusive experience in terms of presence, active engagement and immersion. To achieve these inclusive aims, however, a greater level of detail in the preservation and presentation of the images is required in order to achieve a high enough fidelity in reproduction of VR360 for an adequate user experience. This research aims to fill this gap in existing knowledge and know-how, to democratize the field of VR360 and improve the quality of the experience for all.

7. Conclusion An experimental method and apparatus have been explored for producing high fidelity VR360, one which creates HDR enhanced panorama image quality with few visual abnormalities. Being able to capture a dynamic range resembling the quality of multiple exposures immediately reduces some of the difficulties with other HDR and spherical panorama methods, including parallax errors, insufficient luminance, ghosting effects in multiple exposures HDR, inconsistent HDR lighting distribution, and nadir angle image errors. A user experience study was conducted to evaluate the quality of the images captured for VR360 and the perceived usability with different display types. It was found that increased luminance in the adapted panorama image was vital in terms of user preference. The VR360 view which aims to reproduce real-world environments requires an image quality which has minimal or zero visual abnormalities so that the user achieves a higher photo- realistic virtual experience. Due to a higher level of spatial awareness for HMD users, the photographic aspects of 360 content such as dynamic range and resolution are clearly noticeable in virtual experiences, compared to desktop/laptop displays and mobile devices. The proposed solution presented in this study is favourable as implied in the user study, especially in some indoor conditions. A number of users expressed a difficulty in choosing between our method and the conventional multiple exposures HDR spherical panorama in some outdoor scenarios. A key interest of this study was to provide an image-based solution for creating high dynamic range enhanced spherical panorama images for high fidelity VR360 interactive panoramic projection. The solution developed in this study has made several improvements in the technical process of creating HDR imaging and spherical panorama. The HDR algorithm for creating sufficient high dynamic range from a single acquisition of digital negative in RAW has shown that working with a multiple exposures workflow is optional. The proposed method provides a VR360 reproduction outcome that is free from major imaging errors such as ghosting effects and misalignment issues. This saves acquisition time, digital storage, and human effort which leads to a reduced cost factor and recording of genuine photographic visual information in HDR. For spherical panorama, the improvised multi-row configuration may allow sufficient nadir image information to be acquired. And the nadir angle can be adequately reproduced using the authentic image information captured 32

from the real-world scene instead of relying heavily on post-processing compensation or corrective manipulation. The optimisation in this study highlighted the importance of reducing redundant human activities during the production process from acquisition to post- processing. Future research may extend the context of this study towards using HDR content in Augmented Reality (AR) and other mixed reality (MR) applications. In such scenarios, an HDR spherical panorama can be used for AR markers and trackers on mobile devices and wearable computers. In the future, this research may be expanded to consider various additional suitable areas of applications using HDR spherical panorama, such as hybrid approaches of photogrammetry 3D scanning in the context of digital heritage, assistive technologies and inclusive design.

8. Acknowledgement The researchers would like to thank Prof Harold Thwaites, Assistant Prof Xia Sheng Lee and researchers who provided their support and valuable feedback to this study. This is a continuous effort to produce, collect, study and make available a database of image content for research development. This research was supported by the Australian Government Research Training Program (RTP) Scholarship and Funding.

9. Reference Adams, A., 1983. Examples: The Making of 40 Photographs, Little, Brown & Co., Boston. Adobe. 2019. Digital Negative (DNG). https://helpx.adobe.com/photoshop/digital- negative.html [accessed 25 March 2019] Awang Rambli, D. R., Sulaiman, S., Nayan, M.Y., and Asoruddin, A.R., 2009. A Step-wise Zoom Technique for Exploring Image-based Virtual Reality Applications. World Academy of Science, Engineering and Technology, Vol 50, 197-200. Banterle, F., Artusi, A., Debattista, K., and Alan Chalmers. (2018) Advanced High Dynamic Range Imaging. CRC Press, Taylors & Francis Group. Benosman, R. and Kang, S.B., 2001. Panoramic Vision: Sensors, Theory, and Applications. Springer. Brown, M. and Lowe, D.G., 2006. Automatic Panoramic Image Stitching using Invariant Features. International Journal of Computer Vision, 74(1), 59–73, 2007. Capture One (2017) https://www.phaseone.com/en/Products/Software/Capture-One-Pro (accessed 19 December 2017) Candy, L. (2006). Practice-Based Research: A Guide. CCS Report: 2006-V1.0 November. University of Technology Sydney. Chen, E., 1995. QuickTime VR - an image-based approach to virtual environment navigation. In SIGGRAPH ’95.

Debevec, P.E. and Malik, J., 1997. Recovering High Dynamic Range Radiance Maps from Photographs, SIGGRAPH, 369-378. DiVerdi, S., Wither, J., and Höllerer, T., 2009. All Around the Map: Online Spherical Panorama Construction. Computers & Graphics, 33(1), 73-84. Fairchild, M.D., 2007. "The HDR photographic survey," IS&T/SID 15th Color Imaging Conference, Albuquerque, 233-238. Fangi, G., Pierdicca, R., Sturari, M., and Malinverni, E. S. (2018) Improving Spherical Photogrammetry Using 360° Omni-cameras: Use Cases and New Applications. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2. Felinto, D., Zang, A.R., Velho, L., 2012. Production Framework for Full Panoramic Scenes with Photorealistic Augmented Reality. XXXVIII Latin American Conference of Informatics (CLEI). Fisher, E.C., Akkaynak, D., Harris, J., Herries, A.I.R., Jacobs, Z., Karkanas, P., Marean, C.W., McGrath, J. (2015) Technical Considerations and Methodology for Creating High- resolution, Color-Corrected, and Georectified Photomosaics of Stratigraphic Sections at Archaeological Sites. Journal of Archaeological Science. Gakken Co., 2008. Mastering in RAW. Canon Singapore. Gawthrop, P., 2007. Creating Spherical Panoramas with the Canon 5D and 15mm Fisheye Lens. http://www.lightspacewater.net/Tutorials/PhotoPano2/paper/ [accessed 16 September 2012] Gray, C., and Malins, J. (2004). Visualizing research: a guide to the research process in art and design. Ashgate Publishing Limited. Guo, Y., Zhao, R., Wu, S., and Wang, C (2018) Image capture pattern optimization for panoramic photography. Multimedia Tools and Applications, Volume 77, Issue 17, pp. 22299–22318. Gledhill, D., Tian, G.Y., Taylor, D., and Clarke, D., 2003. Panoramic imaging—a review. Computers & Graphics, 27(3), 435-445. Hasinoff, S. W., Sharlet, D., Geiss, R., Adams, A., Barron, J. T., Kainz, F., Chen, J., Levoy, M. (2016) Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM Transactions on Graphics, Volume 35, Issue 6. Hill, R., 2010. Photographer's Guide to RAW in Photoshop. Future Publishing, UK. Jung, J., Lee, J., Kim, B., and Lee, S. (2017) Upright adjustment of 360 spherical panoramas. Virtual Reality 2017, IEEE. Karađuzović-Hadžiabdić, K., Telalović, J. H., & Mantiuk, R. K. (2017). Assessment of multi- exposure HDR image deghosting methods. Computers & Graphics, 63, 1-17. Kent, B. R., (2017) Spherical Panoramas for Astrophysical Data Visualization. The Astronomical Society of the Pacific.

Koeva, M., Luleva, M., and Maldjanski, P. (2017) Integrating Spherical Panoramas and Maps for Visualization of Cultural Heritage Objects Using Virtual Reality Technology. Sensors 2017. Ma, K., Duanmu, Z., Yeganeh, H., and Wang, Z. (2018) Multi-Exposure Image Fusion by Optimizing A Structural Similarity Index. IEEE Transactions on Computational Imaging, Volume 4, no. 1, pp. 60-72. Mantiuk , R. K., Myszkowski, K., Seidel, H. (2015) High Dynamic Range Imaging. Wiley Encyclopedia of Electrical and Electronics Engineering. Manfrotto (2017) https://www.manfrotto.co.uk/multi-row-panoramic-head (accessed 29 March 2017) Nightingale, D., 2012. Practical HDR, Second Edition: A complete guide to creating High Dynamic Range images with your Digital SLR. Focal Press. Okura, F., and Kanbara, M (2014) Aerial full spherical HDR imaging and display. Virtual Reality, Volume 18, Issue 4, pp 255-269, Springer. Popovic, V., Seyid, K., Pignat, E., Çogal, Ö., Leblebici, Y. (2014) Multi-camera platform for panoramic real-time HDR video construction and rendering. Journal of Real-Time Image Processing, December 2016, Volume 12, Issue 4, pp 697-708. PTGui (2018) https://www.ptgui.com/features.html (accessed 30 November 2018) Rafi, A., Tinauli, M., and Izani, M., 2007. High Dynamic Range Images: Evolution, Applications and Suggested Processes. Information Visualization, 2007. IV '07. 11th International Conference 2007. Rehm, L., 2009. D3x In Depth Review, 2009, http://www.dpreview.com/reviews/nikond3x [accessed 01 December 2010] Reinhard, E., Heidrich, W., Debevec, P., Pattanaik, S., Ward, G., and Myszkowski, K., 2010. High Dynamic Range Imaging (2nd Edition), Acquisition, Display, and Image-based lighting, ELSEVIER. Silk. S., (2011) High Dynamic Range Panoramic Imaging with Scene Motion. M.A.Sc Thesis, University of Ottawa Steinmuller, E. and Gulbins, J. (2010) The Digital Photography Workflow Handbook - From Import to Output. Steinmuller Photo, Germany. Ueberheide, M., Muehlhausen, M., and Magnor, M. (2018) Low Cost Setup for High Resolution Multiview Panorama Recording and Registration. 2018 26th European Signal Processing Conference (EUSIPCO). Yue, G., Hou, C., Gu, K., Mao, S., and Zhang, W. (2018) Biologically Inspired Blind Quality Assessment of Tone-Mapped Images. IEEE Transactions on Industrial Electronics, Volume 65, no. 3, pp. 2525-2536.