OZO Post Production Workflow

OZO+ P O S T PRODUCTION WORKFLOW

Updated 29th August 2017 References Creator v2.1

Camera Overview ...... 4 Post Workflow Overview ...... 4 Importing Assets ...... 5 Importing Raw Captures ...... 6 Frames and Timecode ...... 8 Viewing Metadata and Mastering Parameters ...... 9 Raw Capture Metadata Tab ...... 9 Raw Capture Mastering Parameters ...... 9 DPX Metadata/Mastering Tab ...... 12 OpenEXR Metadata/Mastering Tab ...... 12 Stitched Metadata/Mastering Tab ...... 13 EXB Metadata/Mastering Tab ...... 13 WAV Metadata/Mastering Tab ...... 13 Overview of Media Exports ...... 13 OZO Creator Fast-Turnaround Workflow Example Outline ...... 15 Export Equirectangular DPX/EXR Files ...... 17 Export Editorial Previews (360 Editorial) ...... 18 Batch Queuing the Export of Multiple Assets ...... 20 The Job Queue ...... 20 Export Audio Files ...... 22 Exporting Processed Audio ...... 22 Exporting Ambisonic Audio ...... 23 Exporting OZO Audio ...... 23 Export Per-sensor Files from Editorial Selects ...... 23 Marking In/Out Points in Captures ...... 24 Frame Number Offset ...... 24 Exported Metadata Information ...... 25 Editorial Decision-making Using OZO Creator ...... 25 Selecting “Hero” Takes ...... 25 Basic Editing using Compositions ...... 25 Exporting Compositions ...... 27 Frame Rate Conversion using OZO Creator ...... 27 Color Match and Denoise Imagery ...... 29 Using the Tones Controls ...... 29 Using the Color Wheel parameters ...... 31 Using Denoise Parameters to Improve Imagery ...... 32 Using the Clarity parameters ...... 34 Using the Sharpening Parameters ...... 35 Comparing the Results of Different Filter Settings on Stitched Footage ...... 36 Stitching Overview ...... 36 Stitching imagery using OZO Creator ...... 37 Importing Fisheye Imagery for Stitching...... 37 Stitching Directly from Raw Captures ...... 37 Setting Seam Positions ...... 37 Manually Setting Backseam Convergence ...... 39 Exporting Stitched Files ...... 41 3

Automated Backseam Convergence ...... 43 Best Practices and Additional Notes for Stitching using OZO Creator ...... 43 Stitching Files Using 3rd-Party Services and Tools ...... 44 Stitching DPX Files using Deluxe ...... 44 Stitching DPX or EXR Files using The Foundry’s Cara VR tools for Nuke ...... 45 Stitching High Bitrate MP4s using Jaunt Cloud ...... 45 Stitching DPX Files with other 3rd-Party Software ...... 45 VFX ...... 46 Metadata Parameter Definitions ...... 46 Using the OpenEXR Depth Map Export ...... 47 Using OZOEXB Captures to Create an HDR plate ...... 47 Final Color ...... 48 Titles and Credits ...... 49 Exporting Encoded files ...... 49 Encoding MP4VR Files with Depth Maps ...... 53 Viewing MP4 Files using OZO Preview ...... 53 Sound Mixing ...... 54 Reviewing the Mix using an HMD ...... 55 Scripting with OZO Creator ...... 57 Scripted Command Line Usage Examples ...... 57

Camera Overview The OZO+ camera is a 3D, 360-degree virtual reality camera that consists of eight 2048 x 2048 (2K x 2K) sensors, with approximate sensitivity of ISO 400. Each sensor is paired with a wide-angle 195-degree FOV lens, and captures images that overlap adjacent sensors to provide natural stereoscopic views. Cam1 (Side L), Cam2 (Front L), Cam3 (Front R), and Cam4 (Side R) represent the four sensor views around the “equator” of the camera. Cam 5 (Top L), Cam 6 (Top R), Cam 7 (Bottom L) and Cam 8 (Bottom R) complete the 8 camera sensor viewpoints.

By design, the camera captures stereoscopic (3D) imagery through approximately 260 degrees (+/- 130 degrees from center) oriented to the camera front and sides, with the balance of the 360-degree field of view captured with monoscopic (2D) imagery. Please note however, that when the images are processed by OZO Creator, the final stitched output only has 230 degrees of stereoscopic imagery.

There are additionally eight microphones spaced around the camera body for recording spatial audio, which can be exported as a 4.0, 5.0, 7.0, or 8.0 channel 48kHz 24-bit PCM WAV file, first-order AmbiX Ambisonic spatial audio, and as Nokia’s OZO Audio Interchange format.

The camera records both picture and sound either to its own internal 500GB Digital Cartridge, or to an external Blackmagic HyperDeck Studio Pro 2 or Blackmagic HyperDeck Shuttle recorder via a 1.5Gbps HD-SDI cable.

The camera compresses the eight 2K x 2K pixel images captured by the sensors into a single compressed data stream that emulates a 10-bit 1920 x 1080 full HD resolution video frame. Sound captured by the microphones is recorded as 8 discrete PCM channels, synced with the video. To post produce the sound and images from these raw camera recordings, first export to appropriate media and industry file formats using OZO Creator software.

OZO Creator and the associated system requirements can be downloaded from Nokia’s website: https://ozo.nokia.com/vr/ozo-creator-virtual-reality-software/

Post Workflow Overview Post production with OZO+ is conceptually similar to a standard post production workflow. Just like with a typical production, editing, color correction, VFX, titles, sound mixing, transcoding to deliverable formats, etc. are required

The main difference is that instead of starting with a single stream of images from the capture, or two streams of stereoscopic footage (on a 3D project) a single “raw” capture is exported, and becomes either four or eight individual streams of images (depending on camera settings). Then a stitching process must be applied to create either a Left/Right eye stereoscopic output, or optionally a monoscopic output.

This document will help clarify where the stitching process best fits into the process, and step through a typical professional post workflow with OZO+ footage, concentrating on the usage of OZO Creator. 5

A high-level overview of production with the OZO camera

Importing Assets OZO Creator allows for the import of several kinds of raw and processed assets.

 Captures are either .ozoraw files recorded onboard the camera, or .mov files recorded using a Blackmagic Hyperdeck Studio Pro 2 or Blackmagic Shuttle

 DPX are the per-sensor DPX files exported from OZO Creator. These can additionally be de-noised or color-matched using 3rd party tools, prior to importing back into OZO Creator

 EXR are the per-sensor OpenEXR files exported from OZO Creator. These can additionally be de- noised or color-matched using 3rd party tools, prior to importing back into OZO Creator

 Stitched files are Left and/or Right eye DPX or EXR files that have been stitched into spherical/latlong/equirectangular format panoramas from the raw captures or per-sensor DPX or EXR files.

Please note that to be imported into Creator, these stitched files must conform to the filename structure “*[left/right]*[%07d].[dpx/exr]“. So frame 1 of the stitched stereoscopic sequence might be “myProject_left_0000001.dpx” and “myProject_right_0000001.dpx”, for example. The left and right eye files may reside in separate folders.

When importing stitched files, OpenEXR format Depth Maps generated by OZO Creator may also be imported. These depth maps will be “linked” to the stitched files that are imported concurrently. These can be used together with the stitched files to export MP4VR files with both stitched and depth map video tracks.

 EXB files are .ozoexb files recorded onboard the OZO+ camera. These 31-frame-long files are generated when using the Exposure Bracketing feature in OZO Remote, and are meant for the purpose of creating of high-dynamic-range (HDR) plates.

 Audio files are multi-channel interleaved files either exported directly from OZO Creator, or multi- channel mixes exported from sound mixing programs such as Pro Tools or Steinberg Nuendo. Please note that these WAV files must conform to a particular channel order specification for proper spatial audio reproduction (discussed in the Sound Mixing portion of this document) and must be 48KHz 24- bit PCM files. The supported number of channels for imported audio include 4.0, 5.0, 7.0 and 8.0. Creator also supports the export/import of first-order AmbiX Ambisonic audio with a W,Y,Z,X channel order.

When more than one type of asset is imported in a project, a list of abbreviations is displayed at the top of the Stage that correspond to those assets. Just click on an abbreviation to access the list of corresponding assets currently imported into the project.

The Stage with imported Captures (CAP), DPX files (DPX), EXR files (EXR), Stitched files (STI), and WAV files (WAV).

Importing Raw Captures The first step after capturing imagery is to follow best practices and make backup copies of the captures.

After backups have been made, start Nokia’s OZO Creator software, and import captures into the program by either dragging and dropping them onto the Stage pane of Creator, or by clicking the “+” button in the Stage pane or navigating to Stage in the main menu, and then selecting Import Capture. Then select the capture(s) that need(s) to be imported.

Close up of the “Stage” pane located at the upper-left of the Creator UI

At this point, the full raw captures can be played back using the transport controls. An individual capture is selected in the Stage when it is highlighted and UI elements in the interface are blue, and the “[N] Asset(s) Selected” message is shown next to the Export button.

Stage focus Creator UI

If the UI elements in the interface are green, then the Composition is active, and the transport controls play back any content that is loaded on the timeline. A green “Composition Selected” message will also be displayed next to the Export button.

Composition focus Creator UI

To edit and play back edited footage when in the Composition focus mode, drag-and-drop material from the Stage to the Composition timeline (alternatively, control + click (Mac) or right click (Windows) on an asset and select Add to Composition).

Frames and Timecode OZO Creator can display either timecode or frames in the UI. The UI can be switched between the two using the following methods.

 Click on the Duration heading above the timecode/frame duration readout to switch between displaying frames and timecode.  Ctrl + click (Mac) or right-click (Windows) on either the Current position or Duration readout to bring up a contextual menu that allows selection between Timecode and Frames.  Navigate to OZO Creator>Preferences and click the appropriate radio button to select either Timecode or Frames.

The Current Position, Duration, Mark In, and Mark Out fields allow for direct number entry as well. This allows for typing or copy-and-pasting values into the fields.

 With Frames selected, click to place a cursor at the mouse pointer location, or double-click to select the entire field.  With Timecode selected, click to place a cursor at the mouse pointer location, double-click to select the two-digit field at the mouse location, or triple-click to select the entire field. 9

Viewing Metadata and Mastering Parameters

Raw Capture Metadata Tab Double-clicking an asset or single-clicking on the “tag” icon of a raw capture in the Stage will open up the Mastering/Metadata tab.

If Metadata is selected, information about the selected capture can be viewed:

 Camera ID  Date of capture  Time of capture (UTC)  Duration of capture  File size  Exposure setting  Illuminant setting

The metadata tab of a raw capture.

Raw Capture Mastering Parameters In the Mastering tab, a number of controls are available to adjust the results of exported imagery and audio. Please note that adjustments to these parameters affects nearly all exported imagery and audio.

When an asset’s “tag” icon is highlighted, it indicates that some of the Mastering parameters have been adjusted from their default values. To reset all the Mastering adjustments on an asset, the “Reset Mastering” menu option can be selected from the asset’s context menu. Additionally, individual Mastering groups (Colors, Tones, Denoise, Sharpening, etc.) can be toggled on and off to compare before/after changes to default values. Please note that these toggles also affect exported media. Meaning that if a group is turned off, the group’s parameters will have no effect on the output.

From a workflow perspective, it is highly recommended to set the Audio, Rotation, and confirm the Illuminant selection in the Colors mastering section on all newly-imported per-captures as the very first step in the process. 10

The Audio section allows for adjustment of a capture’s audio gain.

 The audio volume can be changed by -/+20dB by adjusting the Gain. The waveform in the Creator UI will animate in conjunction with adjustments to this control, and peaks in the waveform will turn red to indicate audio clipping if the Gain is raised too much. Pressing the Auto (“A”) button will analyze the audio of the capture and automatically maximize the Gain value without clipping.

The Rotation section contains only a single parameter for rotating footage.

 The Rotate Asset 180° selection will rotate the imagery and audio of a capture by 180 degrees to accommodate an underslung (upside-down) mounting of the OZO+ camera.

The Colors section contains options for adjusting the color conversion applied to the raw captures:

 If needed, the Illuminant setting of the capture can be overridden by de-selecting the As Captured checkbox and then adjusting the Illuminant value using the slider. While different Illuminant settings visually change the color temperature, it is not a standard color temperature control. The Illuminant setting will provide for the most accurate color reproduction when correctly matched to the specific spectrum of lighting used in the scene. If fine color adjustments are still needed when the correct Illuminant is selected, those adjustments can be applied using the Temperature, Tint and Tones controls, discussed later in this documentation.

 Import CSV file is used to import a file containing spectral data that is captured by taking a measurement using a high-end color meter (e.g. Sekonic C-7000). Once imported into Creator, this data will be constructed into a “custom” Illuminant which will be applied in place of the standard Illuminant settings. This can potentially provide more accurate color reproduction when multiple lights with different spectra (e.g. fluorescent, LED, sunlight) are simultaneously used to light a scene. Example CSV files with this information in the proper format can be found in: o For Windows systems: C:\Program Files\Nokia OZO\OZO Creator\Resources\Illuminants. o For Mac systems: . Using Finder, navigate to: ~ /Applications/OZOCreator.app/Contents/Resources/Illuminants. Please note that to access the Contents folder, either right click or Ctrl + click on the OZO Creator app and select Show Package Contents. . Copy the.csv files within the folder. . Create an Illuminants directory on the Desktop and copy the files to this new directory. . Navigate to this folder in Creator and select the .csv file.

 Temperature sets the white balance to a custom color temperature. Decrease Temperature to correct a video taken with a lower color temperature of light; this setting makes the image colors bluer to compensate for the lower color temperature (yellowish) of the ambient light. Conversely, increase Temperature to correct a video taken with a higher color temperature of light; the image colors become warmer (yellowish) to compensate for the higher color temperature (bluish) of the ambient light.

 Tint sets the white balance to compensate for a green or magenta tint. Decrease Tint to add green to the image; increase Tint to add magenta.

 The Saturation control allows for the saturation of the image to be adjusted from -100 (fully desaturated) to +50 (increasingly saturated). This control can also be used to help counteract the desaturating effect of the Color denoise filter. This parameter defaults to a value of 0 (no adjustment).

In a typical workflow, the following Tones, Denoise, and Sharpening parameters can be adjusted later, after the editing process has narrowed the field down to only the captures needed for the final piece.

The Color Wheels section contains controls to adjust the saturation and luminance of the Shadows, Midtones and Highlights. These controls are discussed in more detail in the Using the Color Wheel controls section of this documentation.

Each section contains the following controls:  A Color Wheel  Saturation  Luminance  Luminance Range

The Tones section contains controls for adjusting the levels of each of the red, green, and blue channels of the imagery on both a Global and Per-sensor basis. These controls are discussed in more detail in the Using the Tones controls section of this documentation.

Controls include:  White level  Gamma  Black level  Highlight recovery

The Denoise section contains controls for three filters that can improve image quality by removing different kinds of noise from the imagery. These filters are discussed in more detail in the Using Denoise Parameters to improve imagery section of this documentation.

Available filters include:  Temporal  Spatial  Color

The Clarity filter allows the user to to increase mid-frequency contrast within a specified luminance range of the image such that images become clearer, potentially reducing any unwanted “haze.”

Available filters include:  Amount  Radius  Luminance change limit  Range

The Sharpening section contains controls for a sharpening filter that can be applied to imagery. These controls are discussed in more detail in the Using the Sharpening controls section of this documentation.

Controls include:  Amount  Radius  Masking

DPX Metadata/Mastering Tab The Metadata tab for Fisheye DPX and Stitched DPX assets provides metadata about the camera, exposure and Illuminant values, etc. of the imported footage.

In the Mastering tab, a Rotate Asset 180° control is provided to accommodate an underslung (upside-down) mounting of the camera.

OpenEXR Metadata/Mastering Tab The metadata tab for Fisheye OpenEXR and Stitched OpenEXR assets provides all the information that the DPX Metadata tab provides.

In the Mastering tab, a Rotate Asset 180° control is provided to accommodate an underslung (upside-down) mounting of the camera.

Additionally, two controls in the OpenEXR section are provided for the conversion of these files. Please note that adjustments to these values will affect the appearance of exported imagery.

 The Scene referenced check-box enables a conversion from scene referenced gamma (gamma 1.0) to monitor gamma (approx. 2.2 gamma). Unchecking this box makes the EXR files become quite dense and contrasty. It is recommended that this is left checked.

 The Max value slider adjusts the clip point of the input floating point values when reading the EXR files. By default, this is set to 1.0. This setting may be useful to adjust if color correction using a floating point value other than 1.0 as “white” is applied to EXR files prior to re-importing them. However, under typical conditions, leave this set to 1.0. 13

Stitched Metadata/Mastering Tab The Metadata tab for Stitched DPX and EXR assets provides more limited metadata than other assets, primarily including the date and time of the stitch process, the duration, and the file size of the imported footage.

In the Mastering tab, a Rotate Asset 180° control is provided to accommodate an underslung (upside-down) mounting of the camera.

EXB Metadata/Mastering Tab The Metadata tab for EXB captures provides the same metadata as with raw captures.

In the Mastering tab, the same set of controls are provided as with raw captures, with the exception of Audio, as EXB files are recorded without sound.

WAV Metadata/Mastering Tab The Metadata tab for WAV files contains information about the imported audio file, such as duration, number of channels, etc.

In the Mastering tab, a Gain control is provided for adjusting the gain of the selected WAV file.

Overview of Media Exports A number of different media formats can be exported from OZO Creator. Simply select an asset or multiple assets in the Stage and press the Export button to open up the Export Asset window.

The Format menu will be populated with different options, depending on the format of the selected Asset in the Stage.

 Fisheye DPX files are 10-bit rec709 color space files. These are 16.8 MB per frame, or about 234GB/minute.

 Fisheye OpenEXR files are 16-bit floating point linear gamma (scene referenced) files. These will match the DPX files in color and contrast when a rec709 color space is applied to them. There are two advantages of OpenEXR over DPX. The first is that EXR is a 16 bit floating point format versus DPX’s 10 bit integer format. And the second is that EXR files have lossless compression, whereas DPX files have no compression and thus take up more drive space. However, there is no significant dynamic range, color resolution, or image quality advantage over DPX. EXR files do take longer to export due to the processing time associated with compressing them. The size varies due to the content, but they tend to be around 9 – 14 MB per frame, or 126 – 197GB/minute.

 Fisheye CinemaDNG files are raw sensor data files. These will need to be opened and converted in a program such as Adobe Premiere Pro or DaVinci Resolve to de-mosaic/de-Bayer the raw sensor data and apply a color space conversion. So DNG files add some complication to the workflow as compared to DPX or OpenEXR. However, the main advantages of CinemaDNG over those two formats are that these files tend to be smaller, and they provide considerably more control over dynamic range and color to the user. The size is approximately 8.4MB/frame, or 118 GB/minute.

 Fisheye High Bitrate MP4s are YUV 4:2:0 rec709 color space h.264-encoded MP4 files. This is the format of choice if stitching with the Jaunt cloud. As these are compressed files, the bitrate and size can vary considerably, depending on the detail captured by the sensor. Typical values range from 70Mbps – 180Mbps per MP4 file, or approximately 3.6 – 10.3GB/minute.

 Per-Sensor Equirectangular DPX are spherical/latlong/equirectangular versions of each fisheye image (per-sensor) in DPX Format. Files are 10-bit rec709 color space files.

 Per-Sensor Equirectangular EXR are spherical/latlong/equirectangular versions of each fisheye image (per-sensor) in EXR Format. Files are 16-bit floating point linear gamma.

 Stitched DPX will stitch a 360-degree panorama and output a latlong/spherical/equirectangular DPX image per frame. These have the same color space/gamma/bit-depth as the Fisheye DPX.

Stitched OpenEXR will stitch a 360-degree panorama and output a latlong/spherical/equirectangular EXR image per frame. These have the same color space/gamma/bit-depth as the Fisheye OpenEXR.

 Stitched MP4 creates an h.264-encoded file for play back using many applications or platforms, such as YouTube, Facebook, or Samsung Gear VR. A common example would be to encode a MP4 at UHD resolution with top-bottom packing for use on a Samsung Gear VR with the Oculus Video app. Note that this selection has different audio options than MP4VR.

 Stitched MP4VR creates an h.264-encoded file that is primarily intended for play back using the applications that incorporate the Nokia Player SDK as it adds support for 2 channel spatial OZO audio and encoded depth map video tracks. This format can be played back with any applications that play back standard MP4 files. However please note that the audio will not be properly spatialized if played back in an application that does not utilize the Nokia Player SDK.

 Depth Maps OpenEXR will export Left and Right eye OpenEXR format “stitched” latlong/equirectangular files that contain scene depth information in the depth.Z layer. These images utilize a scale of 1 cm to 1.0 floating point value and correlate to stitched imagery that is exported from OZO Creator using the same settings.

 360 Editorial MOV is an h.264 encoded MOV file created by taking the side left (cam1) and side right (cam4) images, applying a fisheye-to-spherical/latlong transformation on them, and then simply placing them side-by-side in a single frame. It also contains a SMPTE timecode track and a stereo downmix of the audio. The data footprint typically lands in the 15 to 40 MB per minute range. 15

 Processed Audio WAV converts the native 360° OZO audio format to a “ring” of loudspeaker channels. These are exported as interleaved 48KHz 24-bit PCM WAV files with either 4.0, 5.0, 7.0, or 8.0 channels. Note that the “elevation” component of the captured audio is lost with this output.

 Ambisonic Audio WAV converts the native OZO audio format to the broadly-compatible first-order AmbiX ambisonic format with a W,Y,Z,X channel order. This provides for a full 360° of spatial audio, but with lower spatial precision as compared to the native OZO audio format.

 OZO Audio Interchange outputs the native OZO audio format as an 8.0 channel WAV file with additional metadata. This can be used to import lossless OZO audio into Steinberg Nuendo and achieve full 360° audio playback with maximum spatial precision.

OZO Creator Fast-Turnaround Workflow Example Outline There are a number of different possible workflows for the post-production of OZO+ footage, each with different advantages and disadvantages, and utilizing a wide array of 3rd party tools. It is unfortunately not possible to cover all permutations in this document.

However, the following step-by-step outline that concentrates primarily on the use of OZO Creator for a “fast- turnaround” use case may help in understanding the optimal workflow for a given project. This is just one possible suggestion for a workflow. So deviate as needed to accommodate different production decisions, schedules, availability of different 3rd party tools, etc.

1. Ingest raw captures into OZO Creator 2. Under Audio in the Mastering tab, press the Auto button to set the maximum Gain level on the selected captures 3. If the camera was underslung on any shots, enable Rotate Asset 180° under Rotation for those assets 4. Identify the desired takes, and delete the unwanted takes from the Stage 5. Select all remaining takes, export 360 Editorial files 6. Import 360 Editorial files into the 3rd party non-linear editor (NLE) of choice and edit the footage 7. Re-open the project in OZO Creator. Use those editorial decisions to manually trim each of the desired takes in OZO Creator to the needed length. a. Only one range can be selected at a time on a given asset in the Stage. If multiple selects from a single take are needed, the same take will need to be imported into OZO Creator multiple times to allow for a given range to be individually selected on each imported version of the asset. 8. Set the Mastering parameters for each of the trimmed takes, periodically saving the project in OZO Creator a. First, adjust the Illuminant to the setting that best represents the actual lighting in the scene (or import an associated CSV file). b. Adjust the Temperature and Tint to arrive at the desired overall color.Adjust Highlight recovery and Shadows recovery. 16

c. Adjust Saturation. d. Use the Color Wheels to adjust the color, saturation, and luminance in the Highlights, Midtones, and/or Shadows if required. e. Ensure Spherical View is selected, drag view to the right or left to expose the Back seam, and then match the Per-sensor color if needed. f. Ideally using a head-mounted display (or “HMD,” e.g. Oculus Rift), set the Global tones to achieve the desired look. g. Iterate between steps b. through g. as needed until the desired color is achieved. h. Move the playhead several frames into the timeline, adjust the Temporal filter if needed. i. Adjust the Spatial filter if needed. j. Adjust the Color filter if needed. k. Ideally using an HMD, set the Clarity parameters to taste. l. Ideally using an HMD, set the Sharpening parameters to taste. 9. Adjust the seam locations taking into account the width and position of the seams. Note: If "3D Panorama (monoscopic poles)" is selected in the export options. the position and widths of the horizontal seams are used to control the falloff from stereo to mono at the zenith and nadir. 10. Select all raw captures, press Export a. Select Stitched MP4 b. Choose the appropriate options to generate the desired format (often based on the specs for the intended distribution platform) c. Queue exports and start Job Queue 11. Import the stitched footage into the NLE and conform to the offline edit 12. Render from the NLE to generate mezzanine/transcode masters or direct-deliverable formats.

Export Equirectangular DPX/EXR Files The Per-sensor Equirectangular Export options will create per-sensor latlong/spherical/equirectangular/layers from the fisheye DPX/EXR files. Since these files are output prior to the stitching process, these files can be useful when manually fixing any artifacts created during automated stitching.

Equirectangular Side Left/Cam 1 image Equirectangular Front Left/Cam 2 image

Equirectangular Front Right/Cam 3 image Equirectangular Side Right/Cam4 image

Equirectangular Top Left/Cam 5 image Equirectangular Top Right/Cam 6 image

Equirectangular Bottom Left/Cam 7 image Equirectangular Bottom Right/Cam 8 image 18

Export Editorial Previews (360 Editorial) If an external Non Linear Editing (NLE) system is being used for editing, such as Final Cut Pro, Avid Media Composer, or Adobe Premiere Pro, the first step is to export 360 Editorial files from OZO Creator.

The 360 Editorial is an h.264 encoded MOV file created by taking the side left (cam1) and side right (cam4) images, applying a fisheye-to-spherical/latlong transformation on them, and then simply placing them side-by- side in a single frame.

Due to the physical separation of these two sensors on the camera, a visible seam in the very center of the frame will be present. This is normal, as this image is not truly “stitched.”

As these are compressed files, the data size can vary considerably, depending on the content. The data footprint typically lands in the 15 to 65 MB per minute range.

An example of a 360 Editorial with burned-in 4 channel loudspeaker angles and timecode

A SMPTE timecode track is encoded into the 360 Editorial. The timecode can optionally be burned into the picture at the upper-center of the image by enabling the Show timestamp option.

Additionally, an overlay for the intended audio mix (4.0, 5.0, 7.0, 8.0) can be optionally burned in using the Show speaker layout option. This way, the sound mixer has a visual reference for producing a 360-degree positional audio mix from watching 2D playback.

The 360 Editorial is exported with a 2-track stereo mixdown of the recorded audio to use as a sync reference. 19

To export a 360 Editorial from a full capture, either control + click (Mac) or right click (Windows) on a capture in the stage or navigate to Create in the main menu, and then choose Export from the menu. Or simply click the Export button in the UI. This will open the Export Asset window.

Exporting a 360 Editorial in the Export Asset window

By default, the In and Out point selection is set to the length of the full asset. But if these selections were adjusted, the Range value can be changed to All to easily export the full length of the asset. The Single frame (Playhead) selection will export only the single frame located at the current playback cursor/playhead location.

It is highly recommended that the full length of the assets is exported to 360 Editorials first to do the offline edit. Then use those offline edit decisions to determine the takes and selects that are to be used for the online edit. If the 360 Editorials are trimmed before exporting, it will be difficult to accurately reference the edits back to the original, un-trimmed raw captures.

There are a few options for naming the exported asset under the File Naming menu.

 Use asset name will use the exact name of the raw capture + “editorial.mov” (e.g. “Capture0001_editorial.mov”)

 Use camera ID and time of capture (UTC) will use the name of the camera that captured the footage, along with the Coordinated Universal Time (UTC) date and time that the capture was initiated (e.g. “20160221_151007_ PC01001071”). In most circumstances, this will match the output of the Use asset name option if the asset was recorded on the media module in the camera.

 Custom allows the user to specify the exact name that they want for the file. If batch exporting from multiple captures, the software will auto-increment a number onto the end of the filename prefix to avoid overwriting files. (e.g. “360Editorial-1.mov,” “360Editorial-2.mov,” etc.)

Then choose 360 Editorial under the Format drop-down, Browse to a location to choose the Destination folder to which files will be exported. Select the Resolution, Show speaker layout and Show timestamp options. Finally, click Queue. The export task is now added to the Job Queue.

Batch Queuing the Export of Multiple Assets Batch queuing the export of multiple jobs to the Job Queue can be a time saver for any part of the workflow where a number of assets all need to be exported using the same settings.

For example, the raw capture assets in a project typically all need to be exported to 360 Editorial files using exactly the same export settings. If this is the case, simply shift + click or ctrl + click (Windows) or cmd + click (Mac) to select the needed assets in the Stage and then click the Export button. Choose the desired settings in the Export Asset window and click Queue. Export jobs for all of the selected assets will be added to the Job Queue.

However, please note that stitched exports must be individually selected and exported, otherwise the locations of seams will not be accurately rendered into the exported file.

The Job Queue The Job Queue provides a way to queue up a list of tasks, and then execute those tasks in one single rendering pass. This is optimally used to set up a queue of exports during the day, and then execute the processing overnight, which can be valuable from a workflow efficiency perspective.

Any exports that are set up will only be queued. So these jobs will not begin to process until the Job Queue is started. To initiate the execution of the queue, click on the Job Queue button at the lower left of the application.

The Job Queue button located in the lower left of the UI

Pressing this button will bring up the Job Queue window. Clicking Start will start the queue, clicking Clear All will remove all the jobs, and clicking the “X” button to the right of individual jobs removes those jobs from the queue. 21

If the queue extends beyond the bottom of the window, scrolling up and down the queue can be done by clicking and dragging anywhere within the center of the window.

Prior to starting the queue, the Close button will simply close the window. After the queue has been started, the Close button will turn into a Cancel button and, if pressed, will provide a prompt to cancel the entire queue. Note that all queued jobs will be lost if cancelling after starting the execution of the queue.

The Import stitched image assets automatically once exporting finishes option will be highlighted if Stitched DPX or Stitched OpenEXR jobs have been queued. Clicking the check-box will trigger OZO Creator to import the stitched imagery into the existing project as soon as the Job Queue has been completed. This can be a time saver if stitching a number of assets, or stitching single frames using different settings from the same asset, in order to quickly compare and contrast the results in OZO Creator.

Three different jobs queued in the Job Queue window

The Job Queue is saved when the project is saved. So it is good practice to save often when generating a large queue of jobs.

Export Audio Files

Exporting Processed Audio Multi-channel audio can be exported concurrently when exporting the 360 Editorials. These files can be synced to the 360 Editorial MOV files on a timeline in an NLE such as Premiere Pro or Final Cut Pro, synced in a DAW such as Pro Tools, or muxed to the MOV files using a program such as ffmpeg.

Select Processed Audio in the Format window of the Export Asset window, and then choose the Range, the File Naming option, the Destination, and the Selected Speaker Layout. Although there is a selection of outputs, the typical speaker layout selections are 5.0 or 7.0.

Exporting a 5 channel 48KHz 24 bit PCM WAV File from a raw capture in the Stage

As long as the audio is exported with the same In and Out point selection Range as the 360 Editorial, the exported audio will be in perfect sync with the 360 Editorial file.

The file size for audio will depend on the number of output channels selected:

 4 channels - 23 MB/minute

 5 channels - 28.8 MB/minute

 7 channels - 40.3 MB/minute

 8 channels - 46.1 MB/minute 23

Despite the coincidence that the camera has eight microphones, and that Creator provides an option to export an eight channel audio file, please note that there is actually no direct 1:1 mapping of microphone signals to exported audio tracks with this selection.

OZO Creator applies a conversion from mic signals to standard WAV audio when exporting Processed Audio. And as part of this process there is a loss of elevation (i.e. the height, or “Z” component) in the exported audio. So exporting Processed Audio is essentially a conversion from a "sphere" of captured mic signals to a "ring" of loudspeaker format audio.

Exporting Ambisonic Audio Ambisonic audio can also be exported from raw captures using OZO Creator.

The ambisonic audio format is exported as a first-order AmbiX WAV file with channel order W,Y,Z,X. This format provides for less spatial precision than a typical 5.0 channel Processed Audio export. But the ambisonic format has the advantage of providing a full 360° of spatial audio, including the elevation component that the Processed Audio exports lack.

Exporting OZO Audio The OZO Audio Interchange format is only supported by the Steinberg Nuendo DAW software. But if mixing using Nuendo, this export is the best option as it will provide for a full 360° of spatial audio with full spatial precision.

Export Per-sensor Files from Editorial Selects Once the offline editorial process using the 360 Editorial proxy files is completed, the next step is to trim and export the full resolution imagery. If a 3rd party stitching tool will be used, and/or 3rd party tools will be used to color-match, denoise, or apply any other processes prior to stitching, then export per-sensor files to begin the online conform and finishing process.

An example of an exported DPX frame. The angled horizon is normal. 24

Referencing the burned-in timecode in the 360 Editorial or the information in an Edit Decision List (EDL) generated by a Non Linear Editor (NLE), manually trim the asset of the corresponding raw capture to match.

Note that OZO Creator does not support the input of EDL files for automated batch trimming and exporting. To trim the capture, the timecodes listed in the EDL/XML/AAF file have to be manually entered.

Marking In/Out Points in Captures There are a few methods for exporting a section of a capture. First, start by selecting an asset in the Stage.

 Set the selection in and out points by using the mouse to position the playback cursor and click Mark In, and then reposition the cursor and click Mark Out.

 Click on the timecode/frames value for each of the Mark In and Mark Out and type in or paste the desired values.

 Adjust the triangular selection handles that hang just below the timeline to the desired in and out points.

Once the asset has been trimmed on the timeline, and all Mastering controls are appropriately set, open the Export Asset window and select either Fisheye DPX, Fisheye OpenEXR, Fisheye Cinema DNG, or Fisheye High Bitrate MP4s from the Format drop-down menu.

These export selections will output an image sequence or compressed file for each of the eight sensors on the camera.

Once the output image format is chosen, select a Destination, and then choose the File Naming for the folder into which the exported files will be written. Each format exports to a slightly different folder structure, but the frames ultimately end up written to their respective camera’s directories, e.g.: /cam1/20161017_105500_PC01001093_cam1_0000000.dpx /cam2/20161017_105500_PC01001093_cam2_0000000.dpx etc.

Frame Number Offset For all DPX, DNG, and EXR (i.e. image sequence) exports, a Frame number offset option is provided. This provides the ability to set the start frame of the output to any integer value from 0 through 999,999.

This option is not available for audio or MOV/MP4/MP4VR (i.e. encoded) exports.

Exported Metadata Information Additionally, OZO Creator will make another folder, labeled with the term “metadata” in some fashion, which is located inside the named export folder. There will be two files inside of this folder.

The first file is a “metadata” file that Creator and other Nokia software will use to properly de-warp the fisheye lens distortion of the exported image files. This is a binary file and can only be read by Nokia software.

The second file is a human-readable “.txt“ file that contains a number of parameters that can be used in compositing software, such as Nuke, to de-warp the individual fisheyes. These metadata parameters are explained in the VFX section of this documentation.

A third, optional file is a text file labeled [name of export]_motionmetadata.txt which contains the gyroscopic metadata from the camera recorded during the capture. This metadata may be helpful in stabilizing the footage in a post process. This file is only generated when exporting either Fisheye DPX, EXR, or DNG files in conjunction with selecting the option Include motion metadata with export. This file provides comma- separated values for the frame number, roll, pitch, and heading (yaw), as well as a timestamp.

Editorial Decision-making Using OZO Creator OZO Creator provides basic review and editing functionality for a simple edit of footage from the camera (i.e. no fades/dissolves or import of external media such as titles), or for the selection of “hero” takes prior to exporting footage.

Selecting “Hero” Takes Once a number of captures are imported into the Stage pane of OZO Creator, simply click on one capture after another and play them back using the transport controls. This allows a quick selection of the captures that contain the take(s) that are to be used in the edit, and an elimination of unwanted captures.

Basic Editing using Compositions Please note that assets cannot be mixed-and-matched on the timeline (e.g. a raw capture cannot be put on the same Composition timeline as fisheye DPX files, or stitched DPX files).

Use the steps below to assemble a rough edit:

 First, use the Mark In and Mark Out functionality to isolate (“trim”) the desired portion of all needed captures in the Stage.

 Drag the first selected asset from the Stage to the timeline. This will put the UI into Composition focus mode (green UI elements).

 Continue to drag “trimmed” captures from the Stage to the timeline, and drag them around on the timeline to assemble them. 26

 If an asset requires further trimming, use the mouse to position the playback cursor on the timeline to the start point of the selection, and click Trim In. Then set the timeline cursor to the end point of the selection, and click Trim Out. This can also be done by clicking and dragging on the head or tail of the green “block” that represents the asset. Click and drag on the middle of the green asset “block” to reposition the clip on the timeline.

 If multiple ranges from a single capture have to be used, pull the same capture back onto the timeline after the first trim action, and trim the new “pull” of the capture to the additional selection. It is not possible to make multiple selections from a single instance of a capture on the timeline.

The original “source” timecodes of these selections will not be readily seen after trimming, as the timecode display in OZO Creator only indicates the “timeline” or “output” timecode and not the “source” timecode of each segment on the timeline. However, any exported files do retain the “source” timecode in the metadata of the file header, regardless of their position on the timeline.

The full length of the edited Composition can be exported by pressing the Export button and choosing All. To export just a selection, set the In and Out points, press the Export button, and choose Selection (In-Out).

To set the In and Out points, simply click and drag on the two triangular “flags” just below the timeline. These will “snap-to” the edges of individual cuts on the timeline. This can also be done by positioning the playback cursor and pressing the Mark In and Mark Out buttons (or typing in a timecode/frame value above the respective button) to respectively set the In and Out points.

If more precision is needed when editing, click and drag the Zoom control to the right to zoom into the timeline. The green scroll bar below the timeline can then be used to scroll the view. To see the entirety of a long timeline, click the Auto zoom (“A”) button to the right of the Zoom slider, or drag the Zoom control to the left to zoom out.

If assembling a Composition from raw captures, the audio of the raw captures will be trimmed and assembled along with the video and exported separately if needed. But please note that imported WAV files cannot be edited on the timeline. If Processed Audio or Ambisonic WAV import is needed, it is recommended to utilize a single interleaved multi-channel WAV file, edited in a 3rd party program, which syncs to the edited Composition timeline.

Once the timeline is edited, it can be exported to many supported export formats, depending on the format of the assets used in the Composition.

Press the Export button (ensure the UI is displaying the green Composition focus mode), and make the appropriate selections for the playback platform of choice. This is discussed in more detail in the Exporting Encoded Files section later in this documentation. 27

Exporting Compositions Please note that the naming conventions are a bit different for the Export Composition window as compared to the Export Asset window, as the timeline of a Composition can be made up of multiple captures.

 Use project name will use the name of the current OZO Creator project.

 Use project name and current time will combine the name of the OZO Creator project and the time of the initiated export to create the name of the exported file.

 Custom allows the user to specify the exact name that they want for the file.

The Export Composition window set up to export Stitched DPX files

Frame Rate Conversion using OZO Creator The Frame Rate Conversion feature in OZO Creator applies an optical-flow-based motion interpolation process to generate additional frames in between those frames natively captured by the camera. This allows for the possibility of producing smoother motion when up-converting to 60fps and outputting a 60fps encoded file.

Alternatively, a slow-motion effect can be created when up-converting to a higher frame rate (e.g. 120fps) but outputting a file encoded with a lower frame rate (e.g. 30fps). So if the edit calls for a slow motion version of a shot, this feature can be used to provide a smooth effect, rather than using simple (and juddery) frame- doubling to slow down the motion.

Frame rate conversions can be applied to raw captures, fisheyes and stitched image sequence sources. 28

There are two parameters available to control this effect. The selection of the Format will determine if one or both parameters will appear in the Export Asset window:

 Interpolated Frame Rate appears when any RGB or YUV image export is selected in the Format menu (i.e. DNG files cannot be frame rate converted). This setting determines the amount of frames generated by the interpolation process. When raw captures are chosen as the source for interpolation, options include 30, 60, 90, 120, 180, and 240. When image sequences such as Stitched DPX or Stitched OpenEXR are chosen as the source for interpolation, options include None, 2x, 3x, 4x, 6x, and 8x.

 Playback Frame Rate only appears when exporting to a stitched and encoded file format (e.g. MP4, MOV). This parameter defines the playback frame rate of the encoded file. Options include 30 and 60 frames per second. The final effect on the footage is determined by the combination of this parameter and the Interpolated Frame Rate parameter.

Below are examples of the effect when combining different Interpolated Frame Rate and Playback Frame Rate parameter values to raw captures and exporting encoded files (e.g. MP4s).

Effect: None, native frame rate and duration: Interpolated Frame Rate: 30 Playback Frame Rate: 30

Effect: 2x speed, “fast-motion”: Interpolated Frame Rate: 30 Playback Frame Rate: 60

Effect: Smoothed motion, no change to duration: Interpolated Frame Rate: 60 Playback Frame Rate: 60

Effect: Half-speed, “slow-motion”: Interpolated Frame Rate: 60 Playback Frame Rate: 30

If a maximum-quality frame rate conversion is the focus, it may be advantageous to first apply the frame rate conversion and export image sequences (DPX, EXR), manually address any possible artifacts introduced by the frame rate conversion process, and then reimport the “touched up” image sequences into OZO Creator before finally encoding to MP4.

It should be noted that using extremes in the interpolated frame rate and the playback frame rate could lead to undesired effects. The user is encouraged to become familiar with these features and compare output effects using different combinations of the interpolated and playback frame rates to verify if the desired output is achieved. 29

Color Match and Denoise Imagery Once the per-sensor files of the select takes are exported, color match the sensor outputs to one another to ensure the most seamless stitched presentation.

It has so far proven best to select the sensor output with the best color and contrast as the “hero” camera, and do a “best light” or “pre-grade” color correction on that output using mostly basic color correction tools. Then proceed to match the other sensor outputs to the color corrected “hero” camera.

The idea is not to do a “creative” color correction at this point because there are a number of time-consuming steps that follow, including stitching. And any significant change to the color downstream may require backing up and re-doing several steps. So it is recommended that a very basic color correction be applied at this point to give the most flexibility for setting the final “look” after stitching and VFX have been completed.

It is also recommended to apply a denoising process on the individual DPX files, as long as the denoising process does not significantly soften the image. This will help reduce the visibility of noise and compression artifacts in the footage, and likely improve the results of any optical-flow-based stitching process. The Temporal denoise filter provided in OZO Creator will do a good job removing random noise. If a 3rd party motion-compensated temporal denoising pass is needed, many of our customers and partners have suggested that they get good de-noising results using the Neat Video noise-reducing plugin.

Another reason that it is best to apply de-noising prior to stitching is because the stitching process will convert to spherical (also called “latlong” or “equirectangular”) imagery. This results in the visible stretching and warping of the image towards the “poles” of the sphere (i.e. the top and bottom of the imagery), which has the effect of stretching out and warping the noise as well. So it will likely be more difficult to remove the noise after stitching, rather than beforehand.

Using the Tones Controls The tone of exported footage can also be adjusted within OZO Creator using the Tones parameters. This adjustment is applied after the color conversion using the selected Illuminant parameters in the Colors mastering section. So first ensure that all Illuminant parameters are set appropriately, before using the RGB color controls.

It is also recommended to set the Saturation value, located in the Colors mastering section, prior to adjusting the per-sensor controls.

Please note that it will take a moment after releasing the mouse button for OZO Creator to process the adjustments made to these parameters. So there will be a short delay between setting a value, and the image updating with the results of that change.

The parameters available to adjust tones include:

 White level adjusts the level of the white point in the image. Note that this parameter will not reveal any additional highlight detail. Use the Highlight Recovery parameter for that function.

 Gamma adjusts the level of the midtones in the image.

 Black level adjusts the level of the black point of the image.

 The Highlight recovery slider controls how much highlight detail is recovered in the brightest (clipped) portions of the image. A setting of 0 results in no recovery, and a setting of 140 applies maximum recovery. Values between 140 – 150 additionally darken the highlights of the image to allow for extra range if post compositing/painting is desired in the highlight areas. Please be aware that any highlight detail added into the image will not not have full color accuracy. This value can be adjusted if undesired desaturation effects are seen in the highlights. If this is not a significant concern, then a setting of 140 is recommended to generate maximum dynamic range imagery.

 The Shadows recovery slider controls how much shadow detail is recovered in the darkest (crushed) portions of the image. A setting of 0 results in no recovery, and a setting of 100 applies maximum recovery.

Shadow/Highlight Recovery Options

These parameters can be adjusted on both a Global (all-sensor) basis and on a Per-sensor basis, with the exception of Highlight/Shadow recovery, which is always applied globally to all sensors. Simply click the appropriately-labeled button in the RGB Levels section to select the mode.

These parameters can also be adjusted on each of the red (R), green (G), and blue (B) channels independently, as well as on an “overall” basis (again, with the exception of Highlight/Shadow recovery). Simply select either RGB to apply an equal correction to all three channels simultaneously, or click one of the R, G, or B buttons to select the individual channel to be adjusted.

The curve in the displayed graph will animate along with these parameters to help indicate how the adjustments are affecting the output imagery. Please note that adjustments to these parameters may result in additional clipping of highlights or crushing of black detail.

Adjusting the tone on a per-sensor basis can be especially helpful to correct for less-than-optimal lighting conditions (e.g. the sun located near the horizon and glaring directly into one sensor, resulting in “warm” tones and/or “fogged” black levels).

Per-sensor tone adjustments may best be achieved in Spherical view (View>Spherical View), as this view will most visibly show tone and color differences between sensors. Additionally, click-and-drag the panorama slightly to the left or right in Spherical view so that the back seam is visible for matching the color between the Side L and Side R sensors.

Global tone adjustment, however, is best achieved when the adjustments are reviewed using an HMD. This will provide a viewing environment closer to the final consumption display, as many HMDs have significantly different dynamic range capability and color space than traditional monitors. Thus, while it can be slightly awkward to switch between adjusting the UI on a monitor and reviewing the footage in an HMD, making global color decisions using an HMD will provide the most accuracy in achieving the desired “look.”

Using the Color Wheel parameters The color wheel parameters in the Mastering tab can be used to independently adjust the color, brightness, and saturation in the Shadows, Midtones and Highlights.

The Color Wheel controls 32

The Color Wheels controls are as follows:

 Color Wheel – The control point in the center of the Color Wheel is used to select the hue and the intensity of that hue for each of the Shadows, Midtones, and Highlights. o Positioning the control point at different angles will change the hue to that indicated on the Color Wheel. o Placing the control point further away from the center of the wheel increases the intensity of the hue. o The angular position of the control point is displayed above the color wheel. This value ranges from 0° to 360°.

 Luminance – allows adjustment to the brightness of the defined luminance range.

 Saturation – allows adjustment to the color saturation of the defined luminance range.

 Luminance Range – this control is used to set the luminance ranges that define the Shadows, Midtones, and Highlights.

The user can return to the default settings by clicking the return button .

Using Denoise Parameters to Improve Imagery When exporting imagery from a raw capture, it is encouraged to use OZO Creator’s image processing filters to improve the quality of the imagery.

The controls for these parameters are located in the Mastering tab of each individual raw capture under the Denoise heading, and the values are saved on a per-asset basis. This makes it possible to tailor these settings for all of the different raw captures in a project. These per-asset settings will be saved when the project is saved.

It is important to note that only one filter, the Temporal denoise filter, is applied to the raw sensor data. As the DNG files are un-color-converted raw sensor data, the Temporal filter is the only filter whose effects can be seen in all image exports, including exported DNG files. This is unlike any other available filters, as the remaining filters are applied after the sensor data has been de-mosaiced/de-Bayered and color converted to RGB space. So their effects can only be seen in the DPX, EXR, MP4, and MOV files.

The Temporal parameters control a filter that removes random noise using information from multiple adjacent frames:

 The Strength parameter adjusts the aggressiveness of the filter. This parameter primarily determines if noise is removed. Higher values are more aggressive, and lower values are less aggressive. The higher the value, the more frame-to-frame variation in pixel values the algorithm accepts when removing noise. Please note if the value is set too high, too much variation may be allowed, and artifacts may appear in some content. Also note that if the value is too low, the algorithm may interpret the frame- to-frame random noise as movement, and the filter becomes less effective. This parameter defaults to a value of 40.

 The Framecount parameter adjusts the number of frames used to reduce the noise. This parameter primarily determines how much noise is removed. Higher values equate to potentially lower noise levels. So, under typical circumstances, it is recommended to set this value as high as possible. This parameter defaults to a value of 10.

Please note that the Temporal denoise algorithm uses previous frames to do the filtering. Therefore, it is not as effective in the first 10 frames of the clip. Move the playhead at least 10 frames into the timeline before adjusting the filter to see the full effect.

The Spatial parameter controls a spatial noise filter which helps to further reduce the intensity of random noise. It is strongly recommended to first set the Temporal filter before adjusting the value for this filter, as the temporal filtering is applied to the imagery first:

 The Strength parameter adjusts the aggressiveness of this spatial filter. Higher values will more aggressively reduce noise, but this comes at the expense of potentially softening some image detail. This parameter defaults to a value of 20.

The Color parameters control a filter that reduces color/chroma artifacts. These saturated “false-color” chroma artifacts can often be seen on the edges of high-contrast, high-frequency detail:

 Strength adjusts the aggressiveness of the filter. Higher values will more aggressively remove these artifacts, but may also increasingly desaturate real image content. This parameter defaults to a value of 40.

 Radius controls the size of the filter. Larger values will more thoroughly remove “blotchy” lower- frequency (larger-sized) chroma artifacts. These are often seen in darker areas of the image. Please note that larger values will also desaturate more and larger regions of real image content. This parameter defaults to a value of 1.

To more closely inspect the results of these filters when adjusting the values, it is recommended to zoom in on the fisheye imagery using the Image Zoom control. This is achieved either by scrolling with the mouse wheel when the cursor is placed over the image, or by clicking on the zoom percentage value to open an Image Zoom pop-up window, and then adjusting the slider. The maximum zoom value is 300%, which approximates the size 34

of the imagery when stitched and viewed in an HMD. To close the pop-up window, click on the zoom percentage value again.

The Image Zoom pop-up window

When zoomed in, the image can also be repositioned by clicking and dragging on it.

It is also important to note that, in addition to improving image quality, the proper use of these image processing filters can also improve the results of the optical flow used for the Fine Stitch (highest quality) stitching and the Depth Map outputs in OZO Creator. So it is especially recommended to spend the time optimizing these parameters on each of the raw capture assets in a project, prior to stitching the assets with OZO Creator.

More in-depth information on the Denoise parameters can also be found in the Tips for Improving Image Quality on OZO Footage document available in the Learn page on ozo.nokia.com.

Using the Clarity parameters The Clarity filter allows the user to increase mid-frequency contrast within a specified luminance range of the image such that images become clearer, potentially reducing any unwanted “haze.”

This filter is applied to imagery at the fisheye stage when exporting fisheye DPX, EXR, or MP4 frames.

 The Amount parameter adjusts the overall amount of clarity applied to the footage. Higher values add more contrast to areas in the selected Range. Lower values decrease the contrast to areas in the selected Range making the image appear softer.

 The Radius parameter adjusts the kernel size of the Clarity filter. The scale ranges from 50 to 300, with a default set to 175. The smaller Radius values primarily affect higher frequencies in the image and so the effect more closely approximates to a sharpening filter. Larger Radius values additionally affect lower frequencies, and thus can change the apparent contrast of the image.

 The Luminance change limit defines the maximum amount of change possible in the affected portions of the image. In practical terms, this parameter is used to prevent “crushing” of black areas and “clipping” of white areas, and crucially helps to reduce visible “overshoots” created by this filter which appear around the edges of high-contrast detail.

 The Range parameter defines the luminance range in the image to which the filter will be applied. The scale ranges from 0 to 100. Shadows fall in the 0 – 25 range, Midtones fall in the 25 – 75 range, Highlights fall into the 75 – 100 range. The default value is 25 – 75 and thus only affect the Midtones. Leaving the range set to the Midtones also helps to prevent clipping/crushing, and focuses the effect of the filter where it has the most impact on the image.

It is strongly recommended to review the results in an HMD to best gauge how these parameters will affect the experience on the final consumption display. For example, the image displayed in native fisheye geometry on a 2D computer monitor is much different in appearance than when viewing it mapped to a sphere in 3D using an HMD.

Using the Sharpening Parameters The Sharpening filter is used to provide detail enhancement on OZO+ footage, optimally resulting in a “crisper” output image with improved apparent resolution.

This filter is applied to imagery at the fisheye stage when exporting fisheye DPX, EXR, or MP4 frames. Conversely, the sharpening is applied to imagery after stitching when exporting stitched footage.

 The Amount parameter adjusts the overall amount of sharpening applied to the footage. Higher values apply more sharpness, and lower values apply less.

 The Radius parameter adjusts the kernel size of the sharpening filter. Larger values will more intensely sharpen larger details in the image, but can also result in more visible overshoots. Lower values sharpen only the finest detail in the image, but the generated overshoots tend to be less visible than with larger Radius values. The optimum Radius selection will depend very heavily on the kind of detail being sharpened.

 The Masking parameter is used to control the effect of an edge mask used to help reduce visible overshoots and the increased visibility of noise when sharpening details in the image. Lower values will reduce the effect of the edge mask and allow for more high-contrast edges and low-contrast noise to be sharpened, which may result in visible overshoots on edges of sharpened content and visible noise patterns. Higher values will increase the effect of the edge mask and reduce the visibility of overshoots and sharpened noise. But values significantly higher than the default may reduce the visible effects of the sharpening filter.

It may be best to first turn up the Amount parameter to 100% and the Radius parameter to 3.0 to very clearly see the effects of the filter. Then set the Masking parameter to ensure the sharpening on the most crucial detail in the footage is not being masked out. Once the Masking parameter is set appropriately, adjust the Amount and Radius parameters to taste.

It is strongly recommended to review the sharpened result in an HMD to best gauge how these parameters will affect the experience on the final consumption display. For example, the sharpness of an image displayed in 36

native fisheye geometry on a 2D computer monitor is much different in appearance than when viewed mapped to a sphere in 3D using an HMD.

Comparing the Results of Different Filter Settings on Stitched Footage One way to quickly compare and contrast multiple different sets of parameters on stitched footage is to select the Single frame (Playhead) option in the Range menu of the Export Asset window. This selection works best when also enabling the Import stitched image assets automatically once exporting finishes option available in the Job Queue window. Please note that this option is only present when selecting Stitched DPX or Stitched OpenEXR exports.

 Adjust the desired Mastering parameters  Export Stitched DPX or EXR using the Single frame (Playhead) option  For easy reference, name the file using a Custom name that reflects the chosen settings  Queue the job  Repeat until all desired variations of Denoise parameters have been queued for rendering.  After opening the Job Queue, but before executing the queue, ensure that the Import stitched image assets automatically once exporting finishes option is selected

Using this procedure, a number of these jobs can be queued, and then automatically imported into the list of stitched assets in OZO Creator upon completion of the queue. The resulting single frame versions with different mastering parameters can then be compared within the Creator UI, or optimally using an HMD.

Stitching Overview Stitching is the process by which a single panoramic image is created by combining imagery from multiple sensor positions. The desired result is to create a visually seamless transition from one sensor’s perspective to another, so that the appearance is that of one contiguous image. Stitching OZO+ footage can be done with OZO Creator, or with a number of 3rd party applications.

After any optional color matching and de-noising has been applied to the per-sensor files, the files proceed to the stitching process in order to create left and right eye spherical/latlong/equirectangular images for use in viewing stereoscopic 3D imagery using a head mounted display (HMD), such as a Samsung Gear VR, Oculus Rift, or HTC Vive.

Expect that any stitching process will create some visible artifacts in the interpolated (“seam”) areas that will require manual intervention to fix. These artifacts tend to be the most visible on objects that are closest to the camera, as this is the point at which the parallax difference between sensor angles is the greatest. Large- parallax imagery provides the greatest challenge for any stitching process.

For further information on optimizing footage for the stitching process, please refer to the Shooting for Stitching guide in the Learn page on ozo.nokia.com. 37

Stitching imagery using OZO Creator OZO Creator uses computational stitching to create 360° panoramic frames from raw captures and per-sensor fisheye images.

Importing Fisheye Imagery for Stitching To stitch panorama frames from the per-sensor fisheye images, the per-sensor images must first be imported into OZO Creator. Either navigate to Stage in the main menu, or click the “+” in the Stage pane. Then select either Import DPX or Import EXR (depending on the file format of the per-sensor fisheyes).

 The DPX folder (or EXR folder) field should be set to point into a folder that contains per-sensor DPX or OpenEXR frames.

 The Metadata file field should be set to point into a folder that contains the metadata for the specific DPX or EXR frames that are being used as the source (the metadata path is auto-populated when selecting DPX / EXR files and metadata that conform to the default path and filename output from Creator).

 Then click Import to add these assets to the Stage.

To select these imported DPX or EXR frames for stitching, click on the asset in the Stage.

Note that if there is more than one kind of asset imported into the project, select the appropriate file format at the top of the Stage pane first (i.e. “DPX” or “EXR”) to access the proper list of assets, and then select the specific asset to stitch.

Stitching Directly from Raw Captures Stitching can also be done directly from the raw captures. Simply ensure a raw capture is selected in the Stage, optimize the Mastering parameters, and then proceed with the stitching process.

Setting Seam Positions Once the correct asset is selected, enable Spherical view by clicking the “square” icon in the image viewing pane, pressing “V,” or by navigating to View>Spherical View from the main menu. This view will provide a latlong/spherical/equirectangular panorama view using a fast stitch process.

Once in Spherical view, enable the Seam Mode by pressing the Seams icon in the image viewing pane or via the Seams parameter in the Mastering tab. This will highlight the seam areas with a magenta color. A spot check can be done on the footage (the speed of the drives will likely not allow for full speed playback, especially if stitching from DPX or EXR files) and note if there is close proximity detail or movement within seam areas.

Vertical stitching seam area locations and sizes are to some extent adjustable. As the content in seam areas is interpolated, its quality may not be as good as the image quality in non-interpolated areas. Thus it may be 38

useful to move or resize seams to place them in areas where less important content is located. The seams must be contained in an area of the 360-degree panorama where two adjacent cameras’ field of view overlap, and thus cannot be positioned outside of this overlap area.

Horizontal seams are also present, and highlighted in green. These seams are not used for stitching. They are, however, used to control the falloff from stereo to mono at the zenith and nadir poles of the camera. Like the vertical seams, the position and width of the horizontal seams can be adjusted to some degree. The changes will be applied if 3D Panorama (monoscopic poles) is selected in the export options.

To widen or narrow a seam, click and drag on the edges of the seam. To move the seam, click and drag in the center of the seam. Please be aware that there are limitations on the extent the seams can be repositioned.

The seams can be reset to the original values by clicking the return button in the Seams parameters.

To get out of the Seams mode, simply press the Escape button on the computer’s keyboard.

The default placement of the third seam may lead to stitching artifacts on this particular footage. Please note that the visualization here is not necessarily an accurate rendition of any artifacts that may be generated.

Note how the second seam has been narrowed so that the character’s hand is no longer within the seam area. The third seam has been moved to the left and narrowed so that the character is no longer in the seam area. This will likely result in a stitch with fewer visible artifacts.

Adjustments made to the seam positions will be saved on a per-asset basis when the project is saved. So any adjustments made to the locations of the seams here will affect how each of the shots are stitched.

Manually Setting Backseam Convergence It is recommended to adjust the Backseam Convergence parameter in the Mastering tab whilst viewing the image in Spherical view to find the best setting for the visual convergence of detail in the captured scene. Increasing this value will reveal closer proximity details (thus reducing the magnitude of the vertical “slice” through them). But it will potentially cause distant objects to be viewed “doubled.” In other words, the same distant detail may be seen to the left and right of the seam. So setting the Backseam Convergence parameter will likely require a compromise between near and far detail reproduction.

Click and drag the mouse cursor in a non-seam area of the image, and pan the image left or right to better expose the back seam. The image will “wrap around” and show how the left and right edge of the latlong/spherical/equirectangular image will connect.

With the image panned to the right, and Backseam Convergence set to 1.090, notice how the buildings within the blue circled area don’t align perfectly and appear doubled.

With the Backseam Convergence now set to 1.027, the buildings line up much more accurately.

This setting is saved on a per-asset basis, and will be applied to the back seam if Manual is selected in the Backseam convg. menu of the Export Asset window when exporting stitched footage. 41

Exporting Stitched Files Once the correct asset is selected, and the optimal seam positions are determined, click the Export button at the lower left of the UI (Stage focus, blue UI elements). Then select the necessary stitched format.

The Export Asset window

The Range, File naming, File name prefix, Destination, and Frame number offset fields should already be familiar.

Format selects the output file format:

 Stitched DPX will stitch a 360-degree panorama and output a latlong/spherical/equirectangular DPX image per frame.

 Stitched OpenEXR will stitch a 360-degree panorama and output a latlong/spherical/equirectangular EXR image per frame. These are compressed files, so expect that EXR will take longer to process than the DPX files, as computation is required to apply the compression.

Stitching Quality has two options:

 Fine Stitch (highest quality): Frames are stitched with computational stitching at high quality. This option should be used to get the highest possible output quality. But this comes with a cost of rendering speed. The processing time is highly variable and will depend heavily on the content in the 42

seam areas and the speed of the input and output drives. In general, a faster stitching rate can be achieved with: o higher speed drives used for input and output o content with less complexity of motion, especially in the seam areas o the selection of Stitched DPX instead of Stitched OpenEXR for the export format

 Fast stitch (normal quality): Frames are stitched with a computationally simpler process as compared to the Fine Stitch quality. But the result is that the stitching runs at a much faster rate. Similar to the Fine Stitch (highest quality) option, a faster stitching rate can be achieved with: o higher speed drives used for input and output o the selection of Stitched DPX instead of Stitched OpenEXR for the export format

Horizon stabilization provides the option to automatically stabilize the footage based on the OZO’s orientation sensors.

Frame type specifies the panorama type:

 3D panorama (monoscopic poles) outputs separate left and right eye panoramas with a blend to mono at the zenith and nadir of the spherical panorama. This is the recommended stereoscopic setting for most content.  3D panorama outputs separate left and right eye panoramas with stereoscopic poles. Note that this option typically provides a less comfortable viewing experience than the 3D panorama (monoscopic poles) option.  2D panorama (Left eye) outputs frames for the left eye only.  2D panorama (Right eye) outputs frames for the right eye only.

Exporting 2D panoramas is a bit faster than 3D panoramas because the other eye output is omitted.

Resolution provides several options for specifying the output resolution of the panoramic images. Please note that 4K is the highest “native-resolution” output:

 8K (8192x4096)*  8K UHD (7680x4320)*  6K (6144x3072)*  4K (4096x2048)  4K UHD (3840x2160)  3K (3072x1536)  HD (1920x1080)

*denotes a selection that is upscaled using an advanced algorithm during the stitching process 43

Automated Backseam Convergence The Export Asset window also includes an option to calculate and apply an automated Backseam convergence when exporting Stitched DPX/EXR/MP4/MP4VR files.

The Backseam convg. parameter includes the following options:

 The Manual option uses the user-specified Backseam convergence value set in the UI.

 The Automatic (fixed value) option uses the first frame of the sequence to calculate the optimal convergence values for a number of points along the vertical back seam. This can better account for the different convergence distances typically needed to properly converge all the detail at differing depths along the entirety of the seam. This option typically creates a more “invisible” back seam than the Manual option does and thus is the recommended option for most static camera shots.

 The Automatic (15 frame re-calculation) option works similarly to the Automatic (fixed value) option, except that it re-calculates and applies the optimal convergence every 15 frames. The re-calculation is not smoothly “animated” between keyframes. Thus, if there is significant change in the content in the back seam area, the detail in the seam may visibly “shift” on the frames when the re-calculation is applied.

Best Practices and Additional Notes for Stitching using OZO Creator Stitching seam areas are interpolated from two or more fisheye images, and at best the stitching seam areas are not noticeable. But in certain situations various kinds of interpolation artifacts such as ghosting, blurring, twitching, or garbage pixels may show up.

To ensure the best possible panorama stitching quality, here are some best practices and notes that may be helpful:

 During filming it is best to keep most of the action in non-interpolated areas, if possible. The area right in front of the camera has a quite large FOV of seamless area. Thus it provides the best opportunity to stage action close to the camera and still keep subjects out of interpolated seams.

 The closer the filmed content is to the camera, the more disturbing the interpolation artifacts are. Slow moving content at close distances in interpolated areas is perhaps the most "dangerous" combination.

 Foreground content in which color is similar to the background may more readily result in interpolation artifacts than content with high contrast.

 Thin items such as electric wires, ropes, poles, etc. in the interpolated areas close to the camera are very likely to cause ghosting artifacts.

 Recommended minimum distance for content in interpolated areas is about 1.5 - 2.0 meters, or 5 - 6 feet. 44

 The 3D effect is gradually faded out to 2D at the sides of the camera, around -90 to -120 and 90 to 120 degrees from the camera’s "zero" angle (directly forward).

 The location of the seams, other than the back seam, can be manipulated in OZO Creator. Proper adjustment of the Backseam convergence slider and/or choosing the most appropriate selection of the Backseam convg. parameter in the Export Asset window will help provide the best visual convergence of detail to the rear of the camera.

 Bright light sources, especially above or below the camera, may cause lens flares or light streaks which have different shapes and/or locations in different sensors due to the fisheye lens optics. This may potentially confuse the interpolation algorithm.

 In general, image quality & the stereo effect is not optimal directly above and below the camera. Proper use of the 3D panorama (monoscopic poles) selection and adjustment of the stereoscopic-to- monoscopic horizontal seams can help to account for this.

 The five first and five last frames of the stitching process may typically contain more unwanted artifacts than subsequent frames. Thus it is suggested to have a few extra frames of handle at the start and end of the stitching range.

 Panorama stitching takes a lot of time. Depending on the computer’s capabilities, hard drive read/write speed, and the chosen settings, stitching one frame at high quality and high resolution settings may take a number of seconds.

Other best practices and suggestions can be found in the Shooting for Stitching document available in the Learn page on ozo.nokia.com.

Stitching Files Using 3rd-Party Services and Tools

Stitching DPX Files using Deluxe The same proprietary technology developed by Nokia for the automated stereo stitching of footage has been licensed for use by Deluxe, a full-service, internationally present post-production company. Their services can be used for stitching OZO+ footage to receive stereoscopic output from color-matched and de-noised per- sensor files.

Deluxe can additionally handle end-to-end workflow for the post production of footage captured using the OZO+, including editing, audio mixing, color, de-noising, stitching, and the encoding and packaging of deliverables. 45

Stitching DPX or EXR Files using The Foundry’s Cara VR tools for Nuke Nokia has worked with The Foundry to provide native support for stitching OZO+ footage using the Cara VR toolset within Nuke. This allows for image workflows that fully remain inside Nuke nearly from start to finish.

OZO+ specific procedures for stereoscopically stitching footage using Cara VR include:

 In the c_CameraSolver node, be sure to select Nokia OZO in the Preset menu.

 Click Setup Rig and then point the browser to the “.txt” file containing the metadata that was exported concurrently with the DPX/EXR/DNG files from OZO Creator. This will properly set up all of the intrinsic and extrinsic parameters of the particular camera used to shoot the footage within the c_CameraSolver node.

 If required, use Match and Refine in C_CameraSolver to fine tune the camera parameters.

 In the c_Stitcher node, ensure that Enable Stereo Stitch is selected. Under the Stereo tab, the Eye Separation parameter can be adjusted to control the apparent depth in the scene, and the Falloff Type can be adjusted to control depth merging at the poles.

 Under the Stereo tab of the c_Stitcher node, it may be desired to set the Eye Separation parameter to a value of 0.086, as that is closer to the true inter-axial distance of each of the lenses on the camera.

 3D elements can be integrated in a stereoscopic stitch using the c_RayRenderer node. Set the Projection Mode to Spherical and select Stereo Scan Enable on the Camera tab. The Eye Separation and Falloff Type can then be set to match the stitch.

Stitching High Bitrate MP4s using Jaunt Cloud Jaunt’s cloud stitching platform also supports stereoscopic stitching using footage captured using the camera. This platform accepts the Fisheye High Bitrate MP4 export (not DPX/EXR/DNG files) as the source to stitch, and ambisonic audio as the compatible spatial audio format.

The Fisheye High Bitrate MP4s are h.264 compressed files. So their size will vary depending on content. But expect a data footprint in the neighborhood of 2.5GB – 5.5GB per minute of exported High Bitrate MP4 footage.

For further information on stitching using the Jaunt platform, visit https://www.jauntvr.com/camera/ozo/

Stitching DPX Files with other 3rd-Party Software There are also multiple software stitching solutions available for stitching OZO+ footage monoscopically, including VideoStitch and Kolor Autopano. These are compatible with content captured using the camera. Please note that these tools cannot create stereoscopically stitched OZO+ footage. 46

VFX After stitching, the images will likely need to go through visual effects and paint to manually clean up and repair any artifacts from the stitching process.

The artists will likely require the per-sensor color corrected/de-noised/etc. images to use in repairing artifacts or reconstructing imagery. Providing the artists with the “human readable” metadata information created during the export of the DPX/EXR/DNG files, as well as the definitions below, should give them the information they need to take a fisheye image, and create a spherical/latlong/equirectangular image than can be used to paint/composite into the stitched image. OZO Creator exports the metadata for DPX/EXR/as a text file with the filename ending in _metadata.txt. This will be found in the metadata folder associated with the export.

Metadata Parameter Definitions Each lens will have the following parameters:

 The Center parameter refers to the X and Y coordinates of the center of the lens, measured with respect to the center of the sensor. Since the sensor is 2048X2048, the two values should both be close to 1024 (1024 being half of 2048, and thus the value of a perfectly centered lens)

 The Focal parameter describes the image circle radius in pixels at 1.0 radians (about 57 degrees) FOV. Typically Nuke or other compositing programs will use "Degrees" for this value. So it is likely that a conversion has to be applied to get to an accepted value.

 The Distortion parameters (which are named k1 - k4) are always bundled with Focal value. They describe a polynomial function that approximates lens mapping curve. Four distortion parameters should in theory give a bit more accuracy than three parameters.

o The distortion parameters use the Open CV 3.0 fish eye model described here: http://docs.opencv.org/3.0.0/db/d58/group__calib3d__fisheye.html

o Specifically regarding the distortion function, the deviation from f.theta is described by a polynomial, theta_D = theta ( 1 + k_1 . theta^2 + k_2 . theta^4 + k_3.theta^6 + k_4.theta^8).

 The Position parameter is used to describe the location in 3D space of each of the eight sensors using the mid-center-right sensor and optical axis as the origin frame of reference.

Alternatively, OZO Creator, Nuke’s Cara VR toolset, and Deluxe can optionally output per-sensor spherical/latlong/equirectangular images as part of the stitching process (e.g. the Convert input images into spherical images option in OZO Creator), bypassing the need for VFX artists to create these images themselves. This is likely the best option from a workflow efficiency perspective.

Use the post-stitched footage to composite special effects into the imagery. CGI needs to be composited into stitched spherical imagery.

Using the OpenEXR Depth Map Export To assist with applying depth-based effects and compositing of CG elements, OZO Creator provides an option to export depth maps by selecting the Depth Maps OpenEXR export option.

This export option uses multiple vantage points (camera viewpoints) to provide the calculated depth in the scene for the stereo portion of the image. The depth maps are exported as Left and Right eye OpenEXR format latlong/equirectangular files, with the depth information contained in the depth.Z layer. These OpenEXR files containing the depth information are exported separately from the stitched Left and Right eye panoramas. Please note that this export will take at minimum several seconds per frame, the speed of which will vary depending primarily on the speed of the computer’s GPUs.

This depth map correlates to Fine Stitch (highest quality) stitched Left and Right eye files that are output with exactly the same resolution. The Resolution options provided for the Depth Maps OpenEXR export option correlate to those for the various stitched export formats.

Please note, however, that the back of the camera (i.e. the far left and far right portions of the latlong depth map images) will not have all of the depth provided. As the back of the camera has only monoscopic viewpoints, depth cannot be calculated in these areas.

In this depth map, 1 centimeter of depth is equal to a floating point value of 1.0. So the Gain for the viewing window of a compositing application will need to be reduced quite considerably to see something other than a pure white image (a gain of approximately 0.0005 should be sufficient). Please note that the depth in the depth map is truncated beyond a distance of about 22.4 meters, as depth values beyond this distance currently cannot be accurately calculated.

Using OZOEXB Captures to Create an HDR plate OZO Remote allows for the easy capture of bracketed exposures in a scene using the Exposure Bracketing functionality. The format of a recording taken with Exposure Bracketing is an OZOEXB file, which is 31 frames in length, with each frame captured at a different exposure setting.

When exported to standard image formats, this capture enables the creation of a single high-dynamic-range (HDR) plate using 3rd party tools. This HDR plate can then be composited into a shot which is captured at the same camera location to provide for imagery with more apparent dynamic range than the camera can capture natively.

Note that Temporal denoise filtering is disabled on these captures. Since the difference from frame to frame is so great due to the difference in exposure, this filter would have little practical effect anyway.

It is suggested that Highlight Recovery also be left at a value of 0, since the dynamic range of the HDR plate will be generated from the bracketed exposures. Additionally, adding in significant amounts of Highlight Recovery may introduce undesired desaturated highlight details into the generated HDR plate.

There are a number of workflow possibilities to extend the dynamic range of a shot using these EXB captures.

If OZO Creator will be used to stitch the shot into which HDR elements will be composited, it may be best to export the EXB file as Fisheye DPX or EXR frames. Then use a 3rd-party tool to generate a single HDR fisheye for each viewpoint, as long as the HDR result can be made consistent between viewpoints. Finally, import and stitch those HDR fisheyes into Creator to generate the equirectangular HDR plate.

Alternatively, it is also possible to export the EXB file as Fisheye DPX or EXR frames, re-import those fisheye frames, and then export them as Stitched DPX or EXR frames. Then, use those Left and/or Right eye stitched frames as the source to generate the HDR plate(s). Be aware that the Fine Stitch (highest quality) stitching may not provide optimal, temporally-consistent results on some content in the extreme exposure ranges that contain significant white clipping or black crushing. So with this workflow, manual cleanup may be needed on either the stitched exposure brackets prior to HDR generation, or on the final HDR 360° panorama itself.

If a 3rd party stitching solution will be used, then it is recommended to start by exporting Fisheye DPX or EXR frames from the EXB file and then run those frames through the 3rd party stitching and HDR-generating tool pipeline. This way, the alignment of the images will be optimal as the HDR plate will be stitched using the same tool used to stitch the scene into which it will be composited. Depending on the stitching and HDR-generating tools available, it may be simplest to stitch first, and then generate the HDR plate from the stitched files. Otherwise, generate HDR versions of each of the fisheye viewpoints, and then stitch the result to generate the equirectangular HDR plate. Experiment to see what works best for the combination of tools available.

It is also possible that, rather than combining all of the exposures into a single HDR plate to use as a composite source, each alternate-exposure can be stitched and then used individually as a composite source for different elements of the image. This is another perfectly valid way to extend the dynamic range of the final imagery.

Final Color Final color on the stitched images can typically be started in parallel with the VFX work. This is an industry- standard full color correction pass to set the “look” of the show.

The left and right eyes are both made up of image information from different portions of the four equatorial sensors on the camera. Thus there shouldn’t be a need to do too much color balancing between the eyes, as long as the color matching pass on the per-sensor fisheye imagery was precisely done.

One significant thing to take into consideration is the multitude of HMDs that will be used to view this material. It is recommended to output color bars and profile the intended display (Samsung phones, Oculus Rift, HTC Vive, etc.) as much as possible prior to starting the color correction pass, so that colorists have a better sense of the target they need to reach.

While most phones and other displays profess to use a standard Rec709 or sRGB gamut, a custom viewing look up table (LUT) may be needed to best match the color correction environment to the consumption display. Alternatively, a color space that best represents the color and contrast of the HMD may be needed. Phones in particular (the Samsung Galaxy S6 and S7 being popular models for viewing VR content using the Gear VR system) may have a much bluer white point than is standard for television monitors.

Titles and Credits Titles and credits should, under normal circumstances, be made small enough so that the viewer does not have to look around in the HMD to view them. The FOV in most HMDs is quite narrow (usually between 96 and 110 degrees). So appropriately-sized credits will look quite small on the stitched latlong left and right eyes when viewing on a monitor.

It is good practice to present multiple copies of the credits which are spaced out from one another at 90 or 120 degrees so that the viewer will see one of the copies, regardless of the direction they are looking when the credits begin.

The credits can also be scrolled vertically or horizontally. It is generally not recommended to do this over complete black, as it might induce slight motion sickness in the viewer. In the case of scrolling credits, it might be better to “anchor” the viewer with some kind of static background over which the credits move.

Credits can also be placed in 3D space. But if the credits are over black, then a monoscopic presentation is most often used.

Exporting Encoded files It is possible to export 360 Editorial or MP4 “review” versions of the footage as work progresses, or even export directly to a stitched MP4 output from the raw captures as part of a “fast-turnaround” workflow. Depending on the input, certain options may or may not be selectable.

There are two options in the Format menu of the Export Asset window for exporting stitched and encoded files:

 Stitched MP4 – this selection creates an h.264-encoded file for playback using the platform or application of choice, for example, YouTube, Facebook, or Samsung Gear VR. A common example would be to encode a MP4 at UHD resolution with top-bottom packing for use on a Samsung Gear VR with the Oculus Video app. Note that this selection has different audio options than Stitched MP4VR.

 Stitched MP4VR – this selection creates an h.264-encoded file that is primarily intended to play back using applications that incorporate the Nokia Player SDK. This file can additionally be encoded with depth map video tracks if a stitched image sequence and an associated depth map image sequences are imported into OZO Creator and used as the source for encoding. However, this format is 50

compatible with any applications that play back standard MP4 files. Note that this selection has different audio options than Stitched MP4.

Stitching Quality is available only if exporting from non-stitched imagery (captures, fisheye DPX/EXR files), as OZO Creator will first stitch and then encode/export the footage. These options are the same as with the Stitched DPX/Stitched OpenEXR functionality:

 Fine Stitch (highest quality): Frames are stitched with computational stitching at high quality.

 Fast stitch (normal quality): Frames are stitched with a computationally simpler process as compared to Fine Stitch quality.

Output Quality determines the quality/bitrate of the encoded MP4 file:

 Very High provides the best quality and a fairly high bitrate (typically in the range of 96 – 184 Mbps for UHD, depending on all the export settings). This option takes a bit longer to process than either Normal or High, and will also require approximately twice as much space as files encoded with the High setting.

 High provides good quality and reasonable bitrate (typically in the range of 48 – 92 Mbps for UHD, depending on all the export settings). This option provides probably the best balance of quality, speed, and space, as it produces noticeably better image quality than Normal, but doesn’t take as long to encode or use as much disk space as Very High.

 Normal will take less time to encode and create a smaller file as compared to High (typically in the range of 6 – 21 Mbps for UHD, depending on all the export settings). Compression artifacts will likely be visible in Normal encodes.

Backseam convg. Is available only if exporting from non-stitched imagery (captures, fisheye DPX/EXR files). These options are the same as with the Stitched DPX/Stitched OpenEXR functionality:

 Manual utilizes the manually set Backseam convergence parameter.  Automated (fixed value) calculates optimal back seam convergence using the first frame of the sequence.  Automated (15 frame re-calculation) calculates and applies optimal back seam convergence every 15 frames, starting with the first frame of the sequence.

Profile provides a selection of presets for generating encodes compatible with Nokia SDK-enabled applications on phones and on computers, as well as allowing for user-defined custom encoding options:

 OZO Player Mobile changes the encoding settings to a preset configuration that is readable by mobile applications, especially those that support the OZO Player SDK. This outputs a stereoscopic MP4 file which is output with a top (Left eye) – bottom (Right eye) configuration at UHD resolution.

 OZO Player Desktop changes the encoding settings to a preset configuration that is readable by desktop computer applications using the OZO Player SDK, such as OZO Preview. This outputs a stereoscopic MP4 file that contains two full UHD streams, one for each of the Left and Right eyes.

 Online 360° 3D changes the encoding settings to a preset configuration that is broadly accepted by most online distribution platforms. This selection will also reveal the Inject spatial media metadata checkbox, which if enabled will inject metadata into a copy of the output file. This metadata is needed for online video platforms to properly identify and play back the video in a 360° format stereoscopically, and optionally with ambisonic audio. Please ensure that the target online video platform is compatible with the preset configuration. Any one of these settings can be adjusted to suit the target online video platform. If changes are made, the profile will change to Custom.

 Online 360° 2D changes the encoding settings to a preset configuration that is broadly accepted by most online distribution platforms. This selection will also reveal the Inject spatial media metadata checkbox, which if enabled will inject metadata into a copy of the output file. This metadata is needed for online video platforms to properly identify and play back the video in a 360° format monooscopically, and optionally with ambisonic audio. Please ensure that the target online video platform is compatible with the preset configuration. Any one of these settings can be adjusted to suit the target online video platform. If changes are made, the profile will change to Custom.

 Custom allows for a user-defined configuration to encode for the playback platform of choice.

Frame type works essentially the same as it does in the Stitched DPX/Stitched OpenEXR options. This specifies the panorama type:

 3D panorama (monoscopic poles) encodes a file with both left and right eyes, and if encoding from non-stitched imagery, will stitch with a blend to mono at the zenith and nadir.  3D panorama encodes a file with both left and right eyes, and if encoding from non-stitched imagery, will retain stereoscopic imagery at the zenith and nadir.  2D panorama (Left eye) encodes a file with the left eye only.  2D panorama (Right eye) encodes a file with the right eye only.

Frame packing determines how the left and right eyes are encoded into the file. This option is only available when 3D panorama is selected under Frame Type:

 Separate encodes the left and right eye into two full-resolution individual streams within the file. This is the highest-quality option, as it retains full resolution for each of the individual eyes. However, it is less compatible than Top-bottom. Please ensure that the playback device and playback application can accommodate this encode.

 Top-bottom (sometimes called “over-under”) encodes both the left eye and right eye into a single frame. The eyes are stacked on top of one another in this frame. The left eye is on top, and the right eye is on the bottom. Please note that this option reduces the vertical resolution of each of the eyes in order to pack both into a single frame. 52

 Top-bottom (double height) also encodes the left and right eye with a similar arrangement as the Top- bottom selection. However, please note that this option uses the full vertical resolution of each eye, and so this selection will double the height of the Resolution options. Additionally, many players are unable to decode the large resolution frames this option outputs. So this selection is primarily used for outputting a “master” encoded file that can be used to transcode to multiple playback formats, or for uploading to platforms such as YouTube that will transcode for playback.

Resolution provides several options for specifying the output resolution of the panoramic images:

When Separate or Top-bottom is selected:

 8K (8192x4096)*  8K UHD (7680x4320)*  6K (6144x3072)*  4K (4096x2048)  4K UHD (3840x2160)  3K (3072x1536)  HD (1920x1080)

When Top-bottom (double height) is selected:

 8K (8192x8192)*  8K UHD (7680x8640)*  6K (6144x6144)*  4K (4096x4096)  4K UHD (3840x4320)  3K (3072x3072)  HD (1920x2160)

* denotes a selection that is upscaled using an advanced algorithm during the stitching process

Audio provides multiple options for encoding the audio of the MP4 file (please note that these options are different if Stitched MP4VR is selected):

 4.0 will encode four channels of loudspeaker-configuration audio into the MP4 file  5.0 will encode five channels of loudspeaker-configuration audio into the MP4 file  7.0 will encode seven channels of loudspeaker-configuration audio into the MP4 file  8.0 will encode eight channels of loudspeaker-configuration audio into the MP4 file  Mono will encode a single monophonic track of audio into the MP4 file  Ambisonics will encode first-order, four-channel AmbiX ambisonic audio into the MP4 file.  Disabled will disable the encoding of audio altogether and output an MOS MP4 file.

If MP4VR is selected, there will only be two Audio options when stitching directly from a raw capture:

 OZO Audio 2 channels + meta will encode spatial audio that is compatible with the Nokia Player SDK. Applications that do not use this SDK will not be able to properly play back this audio. This option is only available when stitching to MP4VR directly from a raw capture.

 Disabled will disable the encoding of audio altogether and output an MOS MP4VR file.

Further information on how to generate MP4 files for different platforms can be found on the Learn page on ozo.nokia.com:  YouTube-compatible MP4 files, please refer to Exporting stitched MP4s to YouTube  Samsung GearVR-compatible MP4 files, please refer to Sideloading content to Samsung Gear VR.  Facebook-compatible MP4 files, please refer to Exporting stitched MP4s to Facebook  Vimeo-compatible MP4 files, please refer to Exporting MP4s to Vimeo

Encoding MP4VR Files with Depth Maps MP4VR files can be used in conjunction with Nokia’s Player SDK to provide playback of a single file with stitched video and synced depth information to an application such as Unity. This assists with the real-time depth- based compositing of CG assets and effects.

To generate an MP4VR file that includes depth map information:

 Export Stitched DPX or EXR frames using OZO Creator  Export Depth maps using OZO Creator. Other than the Format selection, be sure to utilize the same settings used to generate the stitched DPX or EXR frames  Import the stitched footage into OZO Creator, browsing to the appropriate locations of the stitched footage and the associated depth maps when prompted  Select the stitched footage under the STI tab of the Stage, and export the footage to the MP4VR format, ensuring that the Include depth map checkbox is selected in the Export Asset window

Viewing MP4 Files using OZO Preview OZO Preview can be freely downloaded from Nokia’s website: https://ozo.nokia.com/vr/get-ozo-virtual-reality-products/

OZO Preview can be used to view MP4 files generated by OZO Creator. Exported footage can be reviewed using either the computer monitor and mouse for 2D playback, or a supported HMD for 3D head-tracked playback, along with a pair of regular headphones for audio reproduction (including binaural audio, if the export was encoded with it).

Preview will be able to identify the stereoscopic/monoscopic format of the file through the file suffix.

 _TB.mp4 designates a top-bottom (over/under) stereoscopic file  _2LR.mp4 designates an MP4 containing two, full resolution separated left and right eye streams  _M.mp4 designates a monoscopic (2D) panorama of either a left eye or a right eye

When viewing the exported footage in OZO Preview, select Show Info Overlay to display an angle overlay, as well as the current time and total run time of the footage.

The Info Overlay displays the current angle, the current time (upper left), and the total run time (upper right)

Additionally, OZO Preview can sync to Pro Tools using Midi time code (MTC) generated by the Pro Tools application. This allows synced playback of audio in Pro Tools while viewing the footage in an HMD environment in order to create the most accurate spatial audio mix for presentation. This is discussed in more detail in the Sound Mixing section of this documentation.

Sound Mixing OZO+ captures spatial audio with eight microphones positioned over the surface of the device. The microphone signals are stored with the raw captures. These microphone signals are later processed with software to enable accurate headphone and loudspeaker playback.

Audio capturing itself does not require anything specific for VR playback. Typical means can be used to capture the audio, such as close miking, coincident microphones, microphone arrays, etc. In addition, ADR dialogue or Foley effects can be added in the mixing phase as is normally done.

Audio mixing for the Nokia OZO+ can be performed in the same way as normally done for films with 5.1 or 7.1 mixes. Essentially, if the mix works with a properly configured loudspeaker playback system, it should work with Nokia’s binaural headphone playback. No special compensation is needed during the mixing process to account for the conversion from multi-channel playback to spatial binaural, as the binaural rendering algorithms will handle that.

However, one large difference between mixing for traditional theatrical or home video displays and mixing for VR is the precision required in panning and defining the perceived distance to the source. 55

VR mixes must be panned extremely precisely, otherwise the auditory direction of the source and the visual direction of the same source in the image will not match, and this will negatively impact the immersion for the viewer, and potentially result in confusion.

Similarly, the perceived auditory distance to the source should match as closely as possible the perceived visual distance to the same source. This can be achieved through the application of varying levels of location- appropriate reverb and adjustments to the volume of the audio source.

That being said, realism may have to be compromised a bit to preserve clarity, and thus the mix typically should not re-create a perfect auditory reproduction of the environment. This is because the environment, if reproduced perfectly, will likely be too “ambient” to provide the viewer with enough clarity to easily discern dialogue, especially from distant sources.

The currently supported loudspeaker layouts are (positive degrees to the right of center):

 4.0 channels: -45, 45, -135, and 135 degrees

 5.0 channels: -30, 30, 0, -110, and 110 degrees

 7.0 channels: -30, 0, 30, -90, 90, -150, and 150 degrees

 8.0 channels: 0, 45, 90, 135, 180, -135, -90, and -45 degrees

Please note that when muxing audio to picture using OZO Creator to encode MP4 files for the Nokia Player SDK, the channel order in the final mixed multichannel 48kHz 24-bit WAV file has to match the aforementioned supported loudspeaker layouts for proper spatial positioning. So, for a 5 channel mix, track 1 would correspond to the Left channel (-30), track 2 to the Right (30), track 3 to the Center (0), track 4 to the Left Rear (-110), and track 5 to the Right Rear (110).

It is recommended to mix using loudspeakers equidistantly located from the listener at the specified angles for the chosen number of channels. Loudspeakers still currently provide the best “master” audio reproduction for mixing. The mix can be reviewed with a binaural rendering plugin and a pair of high-quality headphones. However, it is advised not to mix entirely using headphones, as the mix may not translate well to other sets of headphones.

This mix should be synced to monitor playback of a 360 Editorial with the same loudspeaker angles burned in, which provides a 2D visual representation of the position of the source within the 360-degree sound field. This is probably the best way to “rough-in” the mix.

Reviewing the Mix using an HMD The mix can be reviewed using an Oculus DK2 HMD by using Midi time code (MTC) to sync OZO Preview to the playback of Pro Tools when on a Macintosh. Loudspeaker audio playback using a video-synced HMD is currently the most accurate way to judge if the mix of the audio matches the visual content. 56

Midi control can be used to start OZO Preview playback from digital audio workstations (DAWs) when they are running in different computers in the same network. This enables audio playback from DAW and video playback from OZO Preview to run in sync.

On a Mac, to use this feature with Pro Tools, first make sure "Network midi" is set up in "Audio Midi Setup" in OS X preferences and that in OZO Preview "MTC Sync" command in "View" menu is enabled.

Then do the following in Pro Tools:

 Set up a session ("Setup" > "Session") to start at 00:00:00:00.

 In "Setup" > "Peripherals...", set the MTC Generator to be the network session.

 Go to the next tab "Machine Control" and enable the master.

 Select the network midi session.

 In "Setup" > "Preferences", go to the "Synchronization" tab and activate both "Machine Chases Memory Location" and "Machine Follows Edit Insertion/Scrub".

 In the transport controls activate "GEN MTC".

The project might need some adaptation to account for approximately 600ms of startup buffering latency of the player, after which the content should be in sync.

Another, perhaps less complex way to review the mix in an HMD is simply to have the locked picture edit sideloaded to a Gear VR system. Then manually sync the start of the playback in the HMD and the playback in the DAW of choice.

Simply ensure that you are in the optimal location within a properly arranged loudspeaker array while reviewing playback with an HMD, and you should get a fairly accurate representation of how the mix will work with the sequence after being encoded for binaural playback.

Reviewing footage in any HMD will block the view of the controls, so it is unlikely that immediate adjustments can be made while reviewing the mix in this fashion. You will likely have to make mental notes, take off the HMD to make adjustments, and then review again. Despite the inconvenience, it is highly recommended that the mix is reviewed using this method before finalizing, because it best represents the end user experience.

Scripting with OZO Creator OZO Creator also includes scripting functionality for the distribution of batch exports. The scripting functionality is run by executing OZO Creator from a command line prompt and specifying a JSON script that contains all of the import/export/parameter information necessary.

JSON scripts can easily be generated through the OZO Creator application. The procedure is exactly the same as queueing an export, except instead of pressing the Export button in the Export Asset window to add the process to the Job Queue, press the Save As JSON button instead. OZO Creator will then write out a JSON file to the specified Destination folder.

If on a Mac, run the OZO Creator scripting functionality by using Terminal to navigate to /Applications/OZO/OZO\ Creator.app/Contents/MacOS/ (assuming that the application is installed in its default location) and then type “./OZO\ Creator [path to and name of JSON file]”

If using a Windows PC, run the OZO Creator scripting functionality by using the Command Prompt (cmd.exe). Assuming that the application is installed in its default location, navigate to C:\Program Files\Nokia OZO\OZO Creator. Type “OZO Creator” [path to and name of JSON file], for example “OZO Creator” C:/Users/”User Name”/Folder/File. Please note:  The OZO Creator command has inverted commas, “OZO Creator”  A forward slash is used for the path name  Where the file path has two words, inverted commas are used, “User Name”

Scripted Command Line Usage Examples The following documentation is displayed by typing the command line syntax ./OZO\ Creator -–examples on Mac or “OZO Creator” –-examples on Windows in the install location. However, the easiest way to obtain an example of the script is to simply write out a JSON file from the Export function in OZO Creator.

OZO Creator command line usage examples

The JSON file is the primary interface. The command line arguments can be used instead if there is only one input file. At the moment there can be only one output.

Refer to --help text for descriptions of the properties.

=== Capture -> Fisheye DPX or EXR === JSON file: { "inputs": [ { "file": "/path/to/capture.mov" } ], "outputs": [ { "file": "/path/to/output_exr/exr/cam%d/%%07d.exr", "metadataFile": "/path/to/output_exr/metadata/foo.Metadata", "metadataTextFile": "/path/to/output_exr/metadata/first_frame_metadata.txt" } ] 58

}

The output file type is detected from output file. Command line arguments for similar operation would be: --input.file /path/to/capture.mov \ --output.file /path/to/output_exr/exr/cam%d/%%07d.dpx

=== Capture -> Editorial === { "inputs": [ { "file": "/path/to/capture.mov", "in": 0, "out": 5 } ], "process": { "editorial": true }, "outputs": [ { "file": "/path/to/output_editorial.mov" } ] }

"in" and "out" for an input can be used to select in and out points for a capture.

=== Capture -> Stitched DPX === { "inputs": [ { "file": "/path/to/capture.mov", "in": 0, "out": 3 } ], "process": { "outputWidth": 1920, "outputHeight": 1080, "stitch": { "type": "Separated" } }, "outputs": [ { "file": "/path/to/output_stitched/%s_%%07d.exr" } ] }

=== Fisheye DPX -> mp4 === { "inputs": [ { "file": "/path/to/output_dpx/dpx/cam%d/%%07d.dpx", "metadataFile": "/path/to/output_dpx/metadata/foo.metadata", "frameCount": 5 } ], "process": { "outputWidth": 1920, "outputHeight": 1080, "stitch": { "type": "Separated" } 59

}, "outputs": [ { "file": "/path/to/output_stitched.mp4" } ] }

For image sequence inputs the frameCount has to be specified. This is because reading huge image sequence from a network disk can be slow. 60