Dolby Atmos Immersive Audio From the Cinema to the Home Nick Engel Senior Director Consumer Entertainment Technology AES Melbourne Section Meeting 11 December 2017 History of Cinema • 1980s - Dolby SR noise reduction (Analog) • 1990s – Dolby Digital Surround EX™ (5.1) • 2010 - Dolby Surround 7.1 © 2016-17 DOLBY LABORATORIES, INC. 2 Spatial Audio Description: Channel Based 3 3 © 2016-17 DOLBY LABORATORIES, INC. Spatial Audio Description: Channel Based 4 4 © 2016-17 DOLBY LABORATORIES, INC. Spatial Audio Description: Object Based © 2016-17 DOLBY LABORATORIES, INC. 5 Spatial Audio Description: Object Based • Position © 2016-17 DOLBY LABORATORIES, INC. 6 Spatial Audio Description: Object Based • Position • Size © 2016-17 DOLBY LABORATORIES, INC. 7 Spatial Audio Description: Object Based • Position • Size • Diffusion © 2016-17 DOLBY LABORATORIES, INC. 8 Spatial Audio Description: Object Based Object • audio essence + metadata Metadata • Spatial metadata • Position • Size • Diffusion • Gain • Object distance • Zone • Width • Screen Scaling • Snap © 2016-17 DOLBY LABORATORIES, INC. 9 Spatial Audio Description Benefits of Channel Based • Established work-flow, tools and techniques • Easy to render & monitor • Direct artistic control over key loudspeakers (e.g. Center, LFE) • Efficient storage (with scene simplification) © 2016-17 DOLBY LABORATORIES, INC. 10 Spatial Audio Description Liabilities of Channel Based • Spatial Resolution Limited by Channel Count © 2016-17 DOLBY LABORATORIES, INC. 11 Spatial Audio Description Liabilities of Channel Based • Spatial Resolution Limited by Channel Count © 2016-17 DOLBY LABORATORIES, INC. 12 Spatial Audio Description Liabilities of Channel Based • Spatial Resolution Limited by Channel Count • Not Scalable or Adaptable © 2016-17 DOLBY LABORATORIES, INC. 13 Spatial Audio Description Liabilities of Channel Based • Spatial Resolution Limited by Channel Count Channel Config. • Not Scalable or Adaptable Audio Track DCP Renderer Panner Channel-based Spatial Audio Descipion Speaker Positions Audio Track DCP Renderer Panner Object-based Spatial Audio Descipion © 2016-17 DOLBY LABORATORIES, INC. 14 Spatial Audio Description Benefits of Object Based • Scalable © 2016-17 DOLBY LABORATORIES, INC. 15 Spatial Audio Description Benefits of Object Based • Scalable © 2016-17 DOLBY LABORATORIES, INC. 16 Spatial Audio Description Benefits of Object Based • Scalable © 2016-17 DOLBY LABORATORIES, INC. 17 Spatial Audio Description Benefits of Object Based • Scalable © 2016-17 DOLBY LABORATORIES, INC. 18 Spatial Audio Description Liabilities of Object Based • Less efficient for complex soundscapes • One object authored, distributed and rendered for every cricket or raindrop? • Learning curve for mixers © 2016-17 DOLBY LABORATORIES, INC. 19 Hybrid Channel-based + Object-based Audio Base Audio Bed Sound “Objects” Dolby Atmos Complex Audio Textures + Discrete and Moving Sounds = Unlimited Creative Freedom Audio plus Positional Data Extremely Flexible Translates to any speaker configuration • Audio Bed contain stationary ambient sounds – similar to a channel based mix • Objects consist of audio plus positional information e.g. explosions, animals, vehicles, bullets, dialogue, etc. • Objects move throughout the three dimensional listening space as content is played back © 2016-17 DOLBY LABORATORIES, INC. 20 Loudspeaker Layout 2121 © 2016-17 DOLBY LABORATORIES, INC. Loudspeaker Layout 2222 © 2016-17 DOLBY LABORATORIES, INC. Loudspeaker Layout 2323 © 2016-17 DOLBY LABORATORIES, INC. Loudspeaker Layout 2424 © 2016-17 DOLBY LABORATORIES, INC. Loudspeaker Layout Screen Loudspeaker Surround Loudspeaker 2525 © 2016-17 DOLBY LABORATORIES, INC. Loudspeaker Layout 2626 © 2016-17 DOLBY LABORATORIES, INC. Loudspeaker Layout 2727 © 2016-17 DOLBY LABORATORIES, INC. Loudspeaker Layout 2828 © 2016-17 DOLBY LABORATORIES, INC. Loudspeaker Layout 2929 © 2016-17 DOLBY LABORATORIES, INC. Reference Level 3030 © 2016-17 DOLBY LABORATORIES, INC. Reference Level 3131 © 2016-17 DOLBY LABORATORIES, INC. Mixer Survey All of the Dolby field engineers around the world and over 60 mixers where interviewed for their opinions… 3232 © 2016-17 DOLBY LABORATORIES, INC. Experimental Mixes Incredibles § Skywalker Sound § Michael Semanick § AMS NEVE Console DFC § Screen and Surround Height Avatar § Skywalker Sound § Gary Summers § 3D Audio Panner § Screen and Surround Height, and Surround Point Source (surround direct) 3333 © 2016-17 DOLBY LABORATORIES, INC. Double Blind Subjective Tests Ambience 3.0 2.0 1.40 1.12 0.98 0.75 1.0 0.57 0.39 0.29 0.00 -0.08 0.00 -0.18 0.0 -0.39 -0.70 -0.64 -1.0 -2.0 -3.0 MS#5#1 #5#2SS #5#3 #5#4 #5 © 2016-17 DOLBY LABORATORIES, INC. 34 © 2016-17 DOLBY LABORATORIES, INC. © 2016-17 DOLBY LABORATORIES, INC. 36 WORLD OF ATMOS Content creation Distribution Playback Devices 38 BRINGING IMMERSIVE AUDIO TO THE HOME TVs CINEMA PCs HOME THEATER BLU-RAY™ MOBILE DEVICES SOUND BARS AVRs VIRTUAL REALITY GAMING Bringing Atmos to the Home Creation Delivery Playback Spatial Coding Connected Home Consumer Theatre Devices Devices Audio Coding Built In Speakers © 2014 Dolby Laboratories, Inc. DOLBY ATMOS AUDIO An Atmos scene is made up of audio objects Audio objects consist of a audio stream and associated metadata Up to 159 discrete objects can be represented during content creation In addition to objects, an Atmos scene can be described in terms of channel beds or Intermediate Spatial Format (ISF) © 2016-17 DOLBY LABORATORIES, INC. HOME CINEMATIC CONTENT CREATION Creation Encoding QC Delivery Near-field Mix Pro Tools TrueHD DMD Blu-ray DMP or DD+JOC DD+ JOC bitstream Dolby Atmos in ES, TS or MP4 Master File or MP4 w/ DUP with video .atmos DD+JOC and 3rd Dolby RMU H264 party DMG muxer AVR/ or (Optional) Non- DP580 DMG Atmos ProRes Video 2.0 and 5.1 or LPCM 5.1 or 7.1 and 7. 1 Mix 2.0 Mix in a .mov container PCM Dolby © 2016-17 DOLBY LABORATORIES, INC. EXAMPLE LIVE WORKFLOW Stadium DP591 Object Audio Contribution Contribution Audio Console Encoding Encoder Fiber or Decoder (ED2) Satellite DP580 Object Audio Monitoring Metadata DP590 Monitoring Object Audio (ED2) Authoring Playout Home DP591 Object Audio Transmission Encoding Encoder (DD+ JOC) STB DP580 Object Audio Monitoring (DD+ JOC) © 2016-17 DOLBY LABORATORIES, INC. Bringing Atmos to the Home Creation Delivery Playback Spatial Coding Connected Home Consumer Theatre Devices Devices Audio Coding Built In Speakers © 2016-17 DOLBY LABORATORIES, INC. The Challenge: Delivering 5.1 Creation Delivery Playback Spatial Coding 5.1 Content Atmos Connected Home 5.1 Content Consumer192,000 bps 4,608,000 bps Devices Theatre Compression Ratio: 23.4xDevices Audio Coding Built In Speakers © 2016-17 DOLBY LABORATORIES, INC. The Challenge: Delivering Atmos Creation Delivery Playback Spatial Coding Atmos Content Atmos Connected Home ATMOS Content Consumer384,000 bps 150,000,000 bps Devices Theatre Compression Ratio: 390xDevices Audio Coding Built In Speakers © 2016-17 DOLBY LABORATORIES, INC. DATA REDUCTION PART 1 90-minute Object-based (OBI) Movie – 100 Object-Channel/Track Stats Format Bitrate Peak-bitrate File Size AES3 Pairs Metadata Metadata (90 minute) Required Channel Sync Complexity PCM+Metadata Constant 115.2Mb/s 77.76GB 50 OOB Extremely High Immersive content requires new thinking! © 2016-17 DOLBY LABORATORIES, INC. DATA REDUCTION PART 1 – SPATIAL CODING Screen Screen Spatial coding Bed Object Dynamic Object © 2016-17 DOLBY LABORATORIES, INC. 48 DATA REDUCTION PART 2 – AUDIO CODING • Dolby Digital (AC-3): Traditional transform codec • Dolby Digital Plus (E-AC3): Supports Dolby Atmos (JOC) • Dolby TrueHD: Lossless codec, supports Dolby Atmos • Dolby AC-4: Dolby Atmos, Accessibility, Personalization • MAT: Container format for Audio over HDMI © 2016-17 DOLBY LABORATORIES, INC. 49 Dolby Digital Plus JOC DD+ 5.1 Dolby Digital Plus Audio Frame DD+ 5.1 JOC with metadata Dolby Digital Plus Audio Frame 192kbps 384kbps © 2016-17 DOLBY LABORATORIES, INC. 50 DD+ JOC Encoder: Object Based Immersive (OBI) content DD+ JOC Encoder v1.10 Object Audio Metadata EMDF (OAMD) Metadata JOC Creation/ Parametric Formatting Data Object 1 L L Object 2 R R DD+ 5.1-ch Object 3 C C DD+ frame core EMDF OAR v2.x JOC Encoder (BSID 16) Object 4 LFE LFE encode Object 5 Ls Ls Object 6 Rs Rs Object 7 Object 8 Object 9 Object 10 Object 11 Object 12 Object 13 Object 14 Object 15 LFE CONFIDENTIAL INFORMATION 51 © 2016-17 DOLBY LABORATORIES, INC. DD+ JOC Decoder (DD+ JOC bitstream, 5.1-channel core) Control Metadata / Dolby Digital Plus Decoder Parameters Decoder Config Info Input Decoder core EMDF data output Handling (OAMD payload) JOC Data OAMD Data S EMDF u b s Decode t r e L a F D B Dolby Digital Plus F m D D r a i E o I0 I0 D a e e c Bitstream n 16-channel PCM P l R r c c k n e t a o o C o E m r E (Objects and/or channels – d d JOC Decoder r / s n n e e m a e d d assignment indicated by OAMD) p r DRC / dialnorm data CONFIDENTIAL INFORMATION 52 © 2016-17 DOLBY LABORATORIES, INC. DELIVERY – DOLBY AUDIO CODECS Dolby AC-4 is the new audio coding system that improves today’s stereo and 5.1 experiences and enables efficient delivery of the next generation audio experiences like immersive and personalized 1994 2008 2015 © 2016-17 DOLBY LABORATORIES, INC. 53 AC-4 - State of the art coding tools Stereo coding Parametric spatial coding Video-frame synchronous optimization Temporal shaping of tools audio framing quantization noise AC-4 DECODER MDCT DOMAIN QMF DOMAIN TIME DOMAIN Two Stereo Advanced Advanced Spectral Audio Companding DRC & DE SRC Limiter