High Dynamic Range Master Class

Matthew Goldman Senior Vice President Technology, TV & Media Ericsson & Executive Vice President, Society of Motion Picture & Television Engineers Do we see or do we “make”?

Are these cubes A) All straight B) Some curved or C) All curved?

Let’s find out….

› UHDTV discussions often don’t consider the human visual system (HVS) › The eye is relatively low resolution, so the HVS builds detail over time via saccadic motion and eye tracking and using contrast and color, combining input data with internal reference models based on memory › This is not at all how cameras work

HD 1920x1080

4K UHD 3840x2160

High Dynamic Range 8K UHD 7680x4320

Image Resolution 10-bit Sampling

Wide Color 8b = Visible Banding Gamut

High Frame Rate

1 arc minute*

*limit of Fovea Centralis 0.5 arc minute

1 arc minute HDTV field-of-view ~30°

4K UHDTV field-of-view ~60°

x° D W x° H D = (W/2)/tan(x) Screen size = √(H2+W2)

(2160p)

Source: http://www.rtings.com/info/4k-ultra-hd-uhd-vs-1080p-full-hd-tvs

© Ericsson 2016 | 2016-08-31 | Page 10 High Dynamic Range (HDR) › HDR immersion not limited to strict viewing distance – Benefits large screens (including HD) and tablets and phones › From transmit side, HDR is potentially more economically viable to deploy than 4K UHDTV › Once you have seen HDR, you realize how much better than current TV it is › Cameras can capture HDR now, but we can’t see it at home

Pictures are richer, more lifelike and sharper with HDR. Seeing is believing.

Low contrast image looks ‘softer’ as More dynamic range can reveal more some detail is harder to see detail – especially edges – and looks Which image has highersharper resolution? (although the pixel resolution is the same)

Low contrast image looks ‘softer’ as More dynamic range can reveal more some detail is harder to see detail – especially edges – and looks sharper (although the pixel resolution is the same)

Snellen chart: Pelli-Robson chart: Impact of size/distance Impact of contrast on resolution on resolution

SD/HD/4K TV Today: HDR TV: Low dynamic range means subtle High dynamic range means contrast differences in the original subtle contrast differences in content (and which many cameras the original content can be can capture) are not maintained – captured and transmitted to detail is missing. the consumers, revealing previously hidden detail.

Light Dark Grey Grey

Light Dark Grey? Grey?

Is ‘A’ or ‘B’ lighter?

30 fps 8 bit sample depth 50//60, 100/120 fps 10/12 bit sample depth

They are the same

30 fps 8 bit sample depth 50//60, 100/120 fps 10/12 bit sample depth

8b = Visible Banding

High Dynamic Range 10-bit Sampling

The combination of HDR, WCG and higher sample precision technologies – acts as a single feature! Whether

Wide Color Gamut 3840 x 2160p 1920 x 1080p or

Candela per square meter (cd/m2) or “nit”

Cinema today: 48 cd/m2 – In dark viewing environment

Reference white for TV production: 100 cd/m2 – Rec. ITU-R BT.1886 – Based on 1930s CRT!

Typical LCD TV today (standard dynamic range, SDR): 300-400 cd/m2

HDR TVs, now to future: 1,000 to 4,000 cd/m2

Standard Dynamic Range, Standard Dynamic Range, High Dynamic Range, Lowlight Exposure Highlight Exposure (simulated by tone mapping)

Images source: K. McCoy. Licensed under CC BY-SA 3.0 via Wikimedia Commons

› SDR: Video generally ≤ 1.25x; Cinema generally ≤ 2.7x › HDR: May be up to 100x Source: Report ITU-R BT.2390

Television: 100 cd/m2 peak white (Rec. ITU-R BT.1886) TV today Based on 1930s CRT

Cinema: 48 cd/m2 peak white Digital Cinema in dark viewing environment

Simultaneous dynamic range cd/m2 (nit) Light Level 10-8 10-6 10-4 10-2 100 102 104 106 108

Television: 100 cd/m2 peak white (Rec. ITU-R BT.1886) TV today Based on 1930s CRT

Cinema: 48 cd/m2 peak white Digital Cinema in dark viewing environment

4Q15: 1000-1200 cd/m2 HDR TV Future: Likely much higher

Simultaneous dynamic range cd/m2 (nit) Light Level 10-8 10-6 10-4 10-2 100 102 104 106 108

Outer triangle: UHDTV primaries Rec. ITU-R BT.2020 Inner triangle: HDTV primaries Rec. ITU-R BT.709

BT.2020 + Z

Z Y

BT.709 + Z X Y X

© Ericsson 2016 | 2016-08-31 | Page 28 Visual Quality: sample Bit depth › Today, all direct-to-consumer digital TV uses 8-bit sampling Visible banding › Banding (posterization) with 8b, especially in plain areas – Sky, backgrounds, graphics, logo – Very noticeable with slow changes, such as fades › Significantly improved PQ with 10-bit sample bit depth – No bandwidth cost in the compressed domain – HEVC Main-10 Profile allows 8-bit or 10-bit operation

› HDR and WCG exacerbates 8-bit 10-bit issues with 8-bit sampling © Ericsson 2016 | 2016-08-31 | Page 29 Television = tele + vision From camera to display

› To see what’s happening from a far distance, the scene γ needs to be captured, transmitted to a remote location, then L = V where γ = 2.35 reconstituted for HDTV (ITU-R BT.709) › Cameras convert scene light to an electrical signal, suitable of being transmitted over long distances, using an opto-electronic transfer function (OETF) › Display convert an electrical signal back to scene light using an electro-optical transfer function (EOTF) › For over 60 years, the cathode ray tube (CRT) was the universal display technology used › The response of a CRT to an input signal is not linear and its EOTF is commonly known as gamma

Scene Transmission Medium Scene Capture Display OETF EOTF © Ericsson 2016 | 2016-08-31 | Page 30 Sample Bit Depth

› Step size / Luminance (dL/L) is the measure of visibility › Levels below the Barten’s contrast sensitivity function (dashed curve) are masked from the HVS › Mapping signal levels to display luminance (EOTF) is known as the gamma curve (a straight line in log space) › 8-bit gamma-coded has large, visible steps across the range › 10-bit gamma-coded reduces this dramatically

© Ericsson 2016 | 2016-08-31 | Page 31 Recent / on-going Standards SMPTE › ST 2084:2014 – High Dynamic Range Electro-Optical Transfer Function of Mastering Reference Displays – Defines “display referred” curve with absolute luminance values based on human visual model – Called Perceptual Quantizer (PQ) › ST 2086:2014 – Mastering Display Color Volume Metadata Supporting High Luminance and Wide Color Gamut Images – Specifies mastering display primaries, white point, and min/max luminance › Draft ST 2094-x – Content-Dependent Metadata for Color Volume Transformation of High Luminance and Wide Color Gamut Images – Specifies dynamic metadata used in the color volume transformation of source content mastered with HDR and/or WCG imagery, when such content is rendered for presentation on a display having a smaller color volume › New project: HDR & WCG Signaling on Streaming Interfaces – To define a signaling representation & carriage mechanism for real-time interfaces to identify HDR and WCG content so that it is properly processed in a production facility as well as correctly displayed in professional reference displays using SMPTE interface standards

© Ericsson 2016 | 2016-08-31 | Page 32 Recent / on-going Standards ITU-R WP6C › Rec. ITU-R BT.2100 – Parameter values for high dynamic range television systems for production & international programme exchange › Report ITU-R BT.2390 – High dynamic range television for production and international programme exchange (companion report to BT.2100) › Interim Report from RG-24 HDR-TV – HDR signalling requirements for programme production and international exchange arising from Recommendation ITU-R BT.2100 – Possibly modify Rec. ITU-R BT.1120-8 Digital interfaces for HDTV studio signals, and Rec. ITU-R BT.2077-1 Real-time serial digital interfaces for UHDTV signals – Transfer function signaling could enable auto-selection of appropriate EOTF by a display: › BT.1886 EOTF if the SDR-TV (i.e., BT.709 HD or BT.2020 UHD) flag is received › BT.2100 PQ EOTF if the PQ flag is received › BT.2100 HLG EOTF if the HLG flag is received

© Ericsson 2016 | 2016-08-31 | Page 33 HDR Transfer functions › Production reference for CRT peak white level is 100 nits – Now referred to as standard dynamic range (SDR) – SDR camera OETF comes from a desire to simplify analog TV electronics › Inverse of CRT gamma (EOTF)

› Rec. ITU-R BT.2100 defines 2 HDR transfer functions: – SMPTE ST 2084 Perceptual Quantization (PQ) EOTF 10-bit levels over wider range – Hybrid Log Gamma (HLG) OETF › (originally from BBC/NHK) – Pros & Cons for each › Consequences for the production chain

FD  EOTFE10000 Y

1 m1  max E1 m2  c ,0  Y    1   1 m2   c2  c3E 

Where: E´ denotes a nonlinear color value {R’, G’, B’} or { L’, M’, S’} in PQ space [0,1]

FD is the luminance of a displayed linear component {RD, GD, BD} or YD or ID, in cd/m2 Y denotes the normalized linear color value, in the range [0:1] So that when R’=G’=B’, the displayed pixel is achromatic

m1 = 261016384 =0.1593017578125 m2 = 25234096×128=78.84375 c1 = 34244096=0.8359375=c3-c2+1 c2 = 24134096×32=18.8515625 c3 = 23924096×32=18.6875 © Ericsson 2016 | 2016-08-31 | Page 35 Hybrid Log Gamma (HLG) OETF

 E 2 0  E 1 E  OETFE  alnE b c 1 E

Where: E is the signal for each colour component {Rs, Gs, Bs} proportional to scene linear light and scaled by camera exposure, normalized to the range [0:12] E´ is the resulting non-linear signal {R’, G’, B’} in the range [0:1] a = 0.17883277, b = 0.28466892, c = 0.55991073 Source: Report ITU-R BT.2390

© Ericsson 2016 | 2016-08-31 | Page 36 Both End-to-End HDR Systems Rec. ITU-R BT.2100 – Parameter values for high dynamic range television systems for production & international programme exchange • Common reference opto-optical transfer function (OOTF); compensates for non-linearity between displayed light and the light captured by the camera End-to-end HDR systems, both identical in mastering environment

SMPTE ST 2084 PQ EOTF

HLG OETF

Source: BBC Research & Development © Ericsson 2016 | 2016-08-31 | Page 37 Live Broadcast Chain – SDR

Compression & Scene Acquisition Display Transmission

OETF EOTF ITU-R BT.709 ITU-R BT.1886 (peak white 100 nits)

Compression & Scene Acquisition Display Transmission

? ? ? High Dynamic Range Content High Dynamic Range Display >16 f-stops

Camera OETF & Compress Profess- Graphics Compress Ad Insert Consumer STB EOTF, sensor Map to content ional & (DTH) Decode Graphics Remap Color exchange) Decode Production Overlay & Full Size Space Display & Down- converted

Video Racking

CCU ctrl

Video Vision Racking Video Mixer CCU ctrl Router Video To Studio (Switcher)

Other sources Graphics

› The reference monitor dilemma

› HDR Reference Monitors today do not have a standard

› So we can either: – Define the standard (as with SDR TV today), or – Signal what was used

› The choice affects interoperability of content

HDR to SDR HDR SDR Conversion

These all need to have matched levels ..... or you can never create the SDR version

© Ericsson 2016 | 2016-08-31 | Page 43 Backward compatibility checklist 1. What exactly is meant by backward compatibility? 2. Which technologies are we trying to address? 1. Dynamic range 2. Color gamut / Color space 3. Sample bit depth 4. Spatial resolution 5. Temporal resolution Before we all sign up to a 6. Video coding standard for delivery-to-consumer (DTH) backward compatibility 3. What is “good enough” picture quality? scheme, have we really understood what will happen 1. For legacy, conventional HD service at each stage of the content 2. For new Ultra HD service (1080p or 2160p HDR+) chain? 4. What new costs will backward compatibility create? 1. In production 2. In post-production 3. In distribution 4. In delivery to consumer © Ericsson 2016 | 2016-08-31 | Page 44 HDR+ “backward compatibility”

› Q1: Are we trying to make an HDR signal that also can be viewed on any conventional SDR TV/display? (stream backward compatibility) Or › Q2: Are we trying to make a signal that can be converted by an intermediate step to be shown on any conventional SDR TV/display? (display backward compatibility) Or › Q3: Are we planning to simulcast, as we do today with HD/SD, and are planning to do with UHD/HD? Historical note: Prior to HD being launched, backward compatibility was a big concern – after in-depth review, not so much

We could … › Create a signal that works with existing HD/UHD STBs/DTVs – But then we’re restricted to 8-bit AVC 1080i › Create a signal that decodes with a new STB but looks “okay” on an SDR TV › Create a signal that is optimal for HDR (1080p or 2160p) – And then either: › Use the existing transmission for SDR, or › Convert in new STBs

› Existing HD SDR TV services will remain – Large populations of SDR-only or AVC-only STBs/DTVs › What does this mean?

Cost-effectively producing both HDR and SDR means SDR HD transmission one production chain will remain and so is available HDR must be suitable for for use by SDR-only UHDTVs deriving SDR

SDR HD transmission will remain unless there is an effective way to do backward compatibility with a single service that does not compromise either the new HDR image or the legacy SDR image …

© Ericsson 2016 | 2016-08-31 | Page 47 Backward Compatibility to legacy HD › Dynamic range: ITU-R BT.2100 (PQ or HLG)  BT.709/BT.1886 (Gamma) › Color space: ITU-R BT.2020  BT.709 › Sample bit depth: 10b  8b (for delivery-to-consumers) › Spatial resolution: 3840x2160  1920x1080 or 1280x720 (as applicable) › Temporal resolution: 50-60 fps progressive  25-30 fps interlaced › Video coding standard: HEVC  to AVC or MPEG-2 (for delivery-to-consumers) › With “broadcast quality” images in both HDR and SDR formats – For legacy conventional HD service – For new Ultra HD service (1080p or 2160p HDR+) Simulcast required unless all of these conditions are met*

*Note: the above still excludes modulation, transport, and audio BC! © Ericsson 2016 | 2016-08-31 | Page 48 HDR+ Approaches: “PQ10”-based

› SMPTE ST 2084 PQ HDR + Rec. ITU-R BT.2020 color – One of two HDR+ formats defined in new Rec. ITU-R BT.2100 Image parameter values for high dynamic range television for use in production and international programme exchange › Single layer baseline is non backwards compatible › HDR10 = PQ10 + reference display metadata – Metadata = SMPTE ST 2086 HDR static metadata + MaxCLL + MaxFALL – Specified by Blu-ray® Disc Association, DECE, CTA, UHD Alliance for pre-produced content – Uses HEVC Color Remapping Information SEI message

› Backwards compatibility possible by using proprietary add-on schemes involving dual layers or single layer + “HDR enhancement” metadata › Some examples – Dolby Vision: Dual layer, HDR baseline + “SDR reconstruction” enhancement layer – Dolby Vision Live: Single layer, HDR baseline + optional “Display Adaptation” metadata (SMPTE ST 2094-1 & 2094-10 HDR dynamic metadata) › Also optional ICtCp color space + optional closed-loop Re-shaper – Technicolor “SL-HDR1” (formerly “Prime”): Single layer, SDR baseline + “HDR Reconstruction” metadata (SMPTE ST 2094-1 & 2094-30 HDR dynamic metadata) – Qualcomm: HDR10 + “Dynamic Range Adjustment” metadata

› Hybrid-Log Gamma (HLG) HDR + Rec. ITU-R BT.2020 color – One of two HDR+ formats defined in new Rec. ITU-R BT.2100 Image parameter values for high dynamic range television for use in production and international programme exchange › Single layer with no metadata › Backwards compatible for Live TV (if BT.2020 color space container maintained) › Possibly backward compatible to HD (BT.709) color space with tone mapping algorithm – But some in industry do not believe doing BC this way is necessary for 4K (very limited in-field) – Instead, plan to simulcast conventional HD SDR with 4K HDR+

› Fewer constraints than live production – Post-production can produce for just that specific content – Knows about pictures “in the future” – A much more controlled environment

› Could exist in multiple formats if needed – Although more economical if there is only 1 format for all

© Ericsson 2016 | 2016-08-31 | Page 53 Live TV challenges › Live TV ecosystems have special needs › To get on-air in early adoption (2017-2019?), must keep the solution simple and “forgiving”, to work as best as possible with existing live workflows › Some proposed HDR schemes require metadata or dual layer streams › This information may get dropped until the HDR Live TV ecosystem matures (islands of implementations always occur in technology displacements) › For early Live TV workflows, use HDR schemes that do not require metadata or dual layers so that if lost/missing/not produced, renderer is still able to produce “broadcast quality” HDR images › “Bread & butter” will be conventional HD for a long time –Simulcast likely required in early deployments

RWT Live An impressive swim from wake

JeanJean Wake Wake SplitSplitSplit TimeTime Time 1:22:121:22:12 1:22:12

Several HDR+ schemes had been proposed, based on discussions with Hollywood studios / Blu-ray Disc Association / display manufacturers. None of whom, however, produce Live TV programming!

Jean Wake Split Time 1:22:12

This is the basic picture … Now in real time we are adding lower thirds …

RWT Live An impressive swim from wake

JeanJean Wake Wake SplitSplit Time Time 1:22:12 1:22:12 JeanJeanJean Wake WakeWake SplitSplitSplit Time TimeTime 1:22:12 1:22:121:22:12

Adding transitions and logos, freeze frames, Adding logos and captioning downstream slow motion, repositioning, etc.

Logo replacement Down-conversion

Up Next This content often needs further downstream image manipulation (mixing, wipes, fades, keying, graphics) … “Squeeze & Tease” Format conversion

Use Case Broadcast 1 Network/Source

Use Case Broadcast Broadcast 2 Network/Source Affiliate/Remote

Use Case Subsequent Broadcast Broadcast Broadcaster 3 Network/Source Affiliate/Remote (e.g., international distribution*)

*or MVPD (Cable or Satellite Provider ) in some regions © Ericsson 2016 | 2016-08-31 | Page 59 Dual-layer & “HDR Enhancement” metadata operational problems Dual layer and single layer + “HDR enhancement” metadata are difficult to apply to Live TV workflows

If there are dual-layers or enhancement metadata that need to be reconstituted into a single stream to be processed and re-encoded, then how do I do the following?: – Transition between two sources, for example in a live studio program? – Apply effects (which layer am I applying them to?) – Add graphics (which layer am I applying them to?) – Format or standards convert (which layer am I converting and how?) – Add logos or closed captions – Cut between programs and adverts

› Remember audio loudness issues?

› Advertisers may use the opportunity to grab attention by introducing huge steps in light levels

Uncompressed Compressed (consumer-grade) 4K (2160p) vs. 1080i HD 400% circa 250% “HDR+” (HDR+WCG+10bit) 25-30% circa 0-20% HFR (50-60fps  100-120fps) 200% circa 30%

In some cases, bandwidth also required to simulcast legacy HD bitstreams in addition to new UHD HDR+ bitstreams

› If bandwidth constraints prevent a broadcaster from offering all of the new technologies, then focus on the “best bang for the bit” –1080p50/60 HDR+

› Take advantage of all modern displays’ ability to up-convert 1080p to 4K (2160p) –Of course, HDR+ support required to render HDR+

› UHDTV is all about the UHD-1 Phase 2 2160p100/120* consumer experience CP “B” (HFR) Bit depth: 10-bit Source: DVB Colour Space: BT.2020 CM-UHDTV – But UHD-1 Phase 1 does not Dynamic Range: HDR Backwards compatibility options: Desired include immersive technologies! UHD-1 Phase 2 Audio: Next generation audio CP “A” (HDR+) 2160p50/60* (e.g. Object-based ) Bit depth: 10 bit Colour Space: BT.2020 › UHD-1 Phase 2 defined Dynamic Range: HDR Backwards compatibility options: Desired commercially … technical Audio: Next generation specs expected end 2016 UHD-1 Phase 1 2160p50/60* audio (e.g. Object-based)

Bit depth: 8-bit, 10-bit, Colour Space: Rec. BT.709 and BT.2020 (signalled) Dynamic Range: Standard Audio: Existing DVB Toolbox Hooks for High Frame Rate compatibility 2019/2020 included

2017/2018

2014/2015

* The specification also includes fractional variants of the 60fps family (1/1.001)

© Ericsson 2016 | 2016-08-31 | Page 65 Other Recent/on-going standards activities › CTA – Draft CTA-861-G – A DTV Profile for Uncompressed High Speed Digital Interfaces (core of HDMI). To add HDR dynamic metadata & HLG OETF to HDR support already in CTA-861.3 › Ultra HD Forum – Ultra HD Forum Guidelines (for UHD Phase A) – Supports 1080p and 2160p HDR+ – PQ10 and HLG10 for Live TV workflows High Dynamic Range Display › UHD Alliance Peak Brightness: >1000 nits – UltraHD Premium specs and logo certification program Black Level: <0.05 nits – Broadcasting Sub-Group to consider the definition or or Peak Brightness: >540 nits recommendation of a reference display environment Black Level: <0.0005 nits › ATSC, SCTE, others … – Standardizing the use of HDR+ in direct-to-home/consumer systems

© Ericsson 2016 | 2016-08-31 | Page 66 MPEG HEVC HDR “Fast TracK” › Call for Evidence for HDR/WCG (February 2015) – Input contributions showed that low bitrate applications did not perform as expected, including poor texture rendition, color shifts after compression, color boundary overflow & chroma sub-sampling issues › January 2016: Decision made not to add new HDR tools to HEVC due to lack of significant improvements of candidate proposals over improved encode-side only techniques (no changes to bitstream syntax or reference decoder) › 3 techniques improved the “anchors” (source images used in evaluation) – Luma Adjustment (Ericsson) › Preprocessing before encoding › Removes subsampling artifacts – Chroma QP offset (Ericsson) › Encoder optimization › Removes chrominance artifacts – Luma QP offset (Ericsson & Sharp) › Encoder optimization › Increases detail › Next step: “Best practices” recommendation & verification testing

Original 4:4:4 Conventional 4:2:0 Ericsson 4:2:0 © Ericsson 2016 | 2016-08-31 | Page 68 (no compression) (no compression) Luma adjustment (ERICSSON)

Original 4:4:4 Conventional 4:2:0 Ericsson 4:2:0 (no compression) (no compression) © Ericsson 2016 | 2016-08-31 | Page 69 Chroma qp offset (ERICSSON)