Media #WWDC17

High• Efficiency Image File Format

Session 513

Davide Concion, Apple/ Team

© 2017 Apple Inc. All rights reserved. Redistribution or public display not permitted without written permission from Apple. • De facto standard for image compression—JPEG

• Requirements for a new format

• HEIF—The answer

• HEIF anatomy

• The codec of choice—HEVC • De facto standard for image compression—JPEG

• Requirements for a new format

• HEIF—The answer

• HEIF anatomy

• The codec of choice—HEVC JPEG > De facto Standard for Image Compression

Hundreds of millions of JPEG images are captured, created, uploaded, and shared daily JPEG > De facto Standard for Image Compression

Hundreds of millions of JPEG images are captured, created, uploaded, and shared daily

Limitations • Compression • Auxiliary images • Animations JPEG > 1992

ITU/MPEG standard

JPEG JPEG standard

1992 JPEG > 1992

ITU/MPEG MPEG1 MPEG2 MPEG4 AVC HEVC standard

JPEG JPEG JPEG2000 JPEG XR standard

1999

1992 1996 2000 2003 2009 2013 JPEG > 1992

ITU/MPEG MPEG1 MPEG2 MPEG4 AVC HEVC standard

JPEG JPEG JPEG2000 JPEG XR standard

HEIF

1999

1992 1996 2000 2003 2009 2013 2015 • De facto standard for image compression—JPEG

• Requirements for a new format

• HEIF—The answer

• HEIF anatomy

• The codec of choice—HEVC Requirements for a New Format Requirements for a New Format

State-of-the-art compression Requirements for a New Format

State-of-the-art compression

HW acceleration Requirements for a New Format

State-of-the-art compression

HW acceleration

Deep and wide color Requirements for a New Format

State-of-the-art compression

HW acceleration

Deep and wide color

4:4:4 color sampling Requirements for a New Format

State-of-the-art compression

HW acceleration

Deep and wide color

4:4:4 color sampling

HDR Requirements for a New Format

State-of-the-art compression

HW acceleration

Deep and wide color

4:4:4 color sampling

HDR

Auxiliary images Requirements for a New Format

State-of-the-art compression

HW acceleration

Deep and wide color

4:4:4 color sampling

HDR

Auxiliary images

Animation, burst, and playback intent Requirements for a New Format

State-of-the-art compression Multiple-image

HW acceleration

Deep and wide color

4:4:4 color sampling

HDR

Auxiliary images

Animation, burst, and playback intent Requirements for a New Format

State-of-the-art compression Multiple-image

HW acceleration Multi-resolution

Deep and wide color

4:4:4 color sampling

HDR

Auxiliary images

Animation, burst, and playback intent Requirements for a New Format

State-of-the-art compression Multiple-image

HW acceleration Multi-resolution

Deep and wide color Tiles

4:4:4 color sampling

HDR

Auxiliary images

Animation, burst, and playback intent Requirements for a New Format

State-of-the-art compression Multiple-image

HW acceleration Multi-resolution

Deep and wide color Tiles

4:4:4 color sampling Rich

HDR

Auxiliary images

Animation, burst, and playback intent Requirements for a New Format

State-of-the-art compression Multiple-image

HW acceleration Multi-resolution

Deep and wide color Tiles

4:4:4 color sampling Rich metadata

HDR Other media types

Auxiliary images

Animation, burst, and playback intent Requirements for a New Format

State-of-the-art compression Multiple-image

HW acceleration Multi-resolution

Deep and wide color Tiles

4:4:4 color sampling Rich metadata

HDR Other media types

Auxiliary images Extensible

Animation, burst, and playback intent • De facto standard for image compression: JPEG

• Requirements for a new format

• HEIF—The answer

• HEIF anatomy

• The codec of choice—HEVC HEIF > Standard

High Efficiency Image File Format HEIF > Standard

High Efficiency Image File Format

ISO standard—ISO/IEC 23008-12 (June 2015—Version 1) • http://mpeg.chiariglione.org/standards/mpeg-h/image-file-format HEIF > Standard

High Efficiency Image File Format

ISO standard—ISO/IEC 23008-12 (June 2015—Version 1) • http://mpeg.chiariglione.org/standards/mpeg-h/image-file-format

Version 2 of the spec should be made public soon HEIF > Standard

High Efficiency Image File Format

ISO standard—ISO/IEC 23008-12 (June 2015—Version 1) • http://mpeg.chiariglione.org/standards/mpeg-h/image-file-format

Version 2 of the spec should be made public soon

Reference model from MPEG • http://mp4ra.org HEIF > Standard

High Efficiency Image File Format

ISO standard—ISO/IEC 23008-12 (June 2015—Version 1) • http://mpeg.chiariglione.org/standards/mpeg-h/image-file-format

Version 2 of the spec should be made public soon

Reference model from MPEG • http://mp4ra.org

GPAC/MP4Box has recently added basic support for HEIF • https://gpac.wp.imt.fr HEIF > Container

HEIF specifies a structural format HEIF container (a “container”) ftyp—Declares the brands in the file

meta — Information about items

mdat—Media Data

EXIF thumbnail image data HEIF > Container

HEIF specifies a structural format HEIF container (a “container”) ftyp—Declares the brands in the file Derived from ISO Base Media File Format

(ISOBMFF - ISO/IEC 14496-12), meta — Information about items based on Apple QuickTime • MP4, timed media mdat—Media Data • MPEG-21, digital items thumbnail image data HEIF > Container

HEIF specifies a structural format HEIF container (a “container”) ftyp—Declares the brands in the file Derived from ISO Base Media File Format

(ISOBMFF - ISO/IEC 14496-12), meta — Information about items based on Apple QuickTime • MP4, timed media mdat—Media Data • MPEG-21, digital items EXIF thumbnail Any compression codec can be image data included in a HEIF file HEIF > Container

HEIF specifies a structural format HEIF container (a “container”) ftyp—Declares the brands in the file Derived from ISO Base Media File Format

(ISOBMFF - ISO/IEC 14496-12), meta — Information about items based on Apple QuickTime • MP4, timed media mdat—Media Data • MPEG-21, digital items EXIF thumbnail Any compression codec can be image data included in a HEIF file HEIF > File Extension

Payload Extension—single image Extension—sequence

HEVC .HEIC .HEICS

H.264 .AVCI .AVCS

Any Codec .HEIF .HEIFS HEIF > File Extension

Payload Extension—single image Extension—sequence

HEVC .HEIC .HEICS

H.264 .AVCI .AVCS

Any Codec .HEIF .HEIFS HEIF > File Extension

Payload Extension—single image Extension—sequence

HEVC .HEIC .HEICS

H.264 .AVCI .AVCS

Any Codec .HEIF .HEIFS • De facto standard for image compression—JPEG

• Requirements for a new format

• HEIF—The answer

• HEIF anatomy

• The codec of choice—HEVC HEIF Anatomy

• Items

• Roles of images

Image• properties

Properties• association

Image• sequences

Tiles• HEIF Anatomy > Items HEIF Anatomy > Items

AVC

HEVC

HEVC HEIF Anatomy > Items

Primary Derived Image

AVC

HEVC

HEVC HEIF Anatomy > Items

EXIF

XMP

MPEG-7

Primary Derived Image

AVC

HEVC

HEVC HEIF Anatomy > Items

EXIF

Properties XMP MPEG-7 Initialization

… Primary Size Derived Image

Mirror AVC … HEVC

HEVC HEIF Anatomy > Items

EXIF

Properties XMP cdsc MPEG-7 Initialization

… Primary Size Derived Image

Mirror AVC … dimg HEVC HEVC HEIF Anatomy

• Items

• Roles of images

Image• properties

Properties• association

Image• sequences

Tiles• HEIF Anatomy > Roles of Images

Primary Representative image HEIF container

Master meta - Information about items Full-resolution displayable image

pitm - Primary Item Box Thumbnail Smaller resolution

Auxillary Alpha plane or a depth map iref - Item Reference Box dimg - Derived Image Hidden Not for display thmb - Thumbnail auxl - Auxiliary Image Derived ‘grid’, ‘iden’, ‘iovl’ cdsc - Content describe

Equivalent Alternative images HEIF Anatomy

• Items

Roles• of images

Image• properties

Properties• association

Image• sequences

Tiles• HEIF Anatomy > Image Properties > Descriptive

Decoder configuration Codec initialization HEIF container

meta—Information about items Image spatial extents Width and height

iprp—Item Properties Pixel aspect ratio Pixel aspect ratio

ipco—Item Property Container Colour information ICC profile ispe—Image Spatial Extents

Relative location Position of the reconstructed image colr—Colour Information hvcC—HEVC Decoder Conf. Auxiliary images Type of auxiliary image properties HEIF Anatomy > Image Properties > Transformative

Clean aperture Cropping of image HEIF container

meta—Information about items Image rotation 0, 90,180,270 degrees rotation

iprp—Item Properties Image mirroring Image mirroring

ipco—Item Property Container

clap—Clean Aperture irot—Rotation imir—Mirroring HEIF Anatomy

• Items

Roles• of images

Image• properties

Properties• association

Image• sequences

Tiles• HEIF Anatomy > Image Properties > Association

HEIF container HEIF container meta—Information about items ipma—Item Properties Association iprp—Item Properties item 1—association ipco—Item Property Container property 1 property 2 hvcC—Decoder configuration 1 ispe—spatial extent 1 width: 500 height: 500

ispe—spatial extent 2 item 5—association width: 1000 height: 1000 property 3 hvcC—Decoder configuration 2 item 6—association ispe—spatial extent 3 property 4 width: 320 height: 240 property 5 HEIF Anatomy > Image Properties > Association

HEIF container HEIF container meta—Information about items ipma—Item Properties Association iprp—Item Properties item 1—association ipco—Item Property Container property 1 property 2 hvcC—Decoder configuration 1 1 ispe—spatial extent 1 2 width: 500 height: 500 ispe—spatial extent 2 3 item 5—association width: 1000 height: 1000 property 3 hvcC—Decoder configuration 2 4 item 6—association ispe—spatial extent 3 5 property 4 width: 320 height: 240 property 5 HEIF Anatomy > Image Properties > Association

HEIF container HEIF container meta—Information about items ipma—Item Properties Association iprp—Item Properties item 1—association ipco—Item Property Container property 1 property 2 hvcC—Decoder configuration 1 1 ispe—spatial extent 1 2 width: 500 height: 500 ispe—spatial extent 2 3 item 5—association width: 1000 height: 1000 property 3 hvcC—Decoder configuration 2 4 item 6—association ispe—spatial extent 3 5 property 4 width: 320 height: 240 property 5 HEIF Anatomy > Image Properties > Association

HEIF container HEIF container meta—Information about items ipma—Item Properties Association iprp—Item Properties item 1—association ipco—Item Property Container property 1 property 2 hvcC—Decoder configuration 1 1 ispe—spatial extent 1 2 width: 500 height: 500 ispe—spatial extent 2 3 item 5—association width: 1000 height: 1000 property 3 hvcC—Decoder configuration 2 4 item 6—association ispe—spatial extent 3 5 property 4 width: 320 height: 240 property 5 HEIF Anatomy > Image Properties > Association

HEIF container HEIF container meta—Information about items ipma—Item Properties Association iprp—Item Properties item 1—association ipco—Item Property Container property 1 property 2 hvcC—Decoder configuration 1 1 ispe—spatial extent 1 2 width: 500 height: 500 ispe—spatial extent 2 3 item 5—association width: 1000 height: 1000 property 3 hvcC—Decoder configuration 2 4 item 6—association ispe—spatial extent 3 5 property 4 width: 320 height: 240 property 5 HEIF Anatomy

• Items

Roles• of images

Image• properties

Properties• association

• Image sequences

Tiles• HEIF Anatomy > Sequences

‘moov’ box—‘trak’ box—‘’ handler HEIF container

ftyp—Declares the brands in the file

meta—Information about items

mdat—Media Data

moov—Movie Box

track—video track—sound track—pict HEIF Anatomy > Sequences

‘moov’ box—‘trak’ box—‘pict’ handler HEIF container

Tracks have roles too ftyp—Declares the brands in the file

meta—Information about items

mdat—Media Data

moov—Movie Box

track—video track—sound track—pict HEIF Anatomy > Sequences

‘moov’ box—‘trak’ box—‘pict’ handler HEIF container

Tracks have roles too ftyp—Declares the brands in the file

Controlling the playback meta—Information about items • Edit list mdat—Media Data • Looping moov—Movie Box track—video track—sound track—pict HEIF Anatomy > Sequences

‘moov’ box—‘trak’ box—‘pict’ handler HEIF container

Tracks have roles too ftyp—Declares the brands in the file

Controlling the playback meta—Information about items • Edit list mdat—Media Data • Looping moov—Movie Box track—video Inter prediction track—sound track—pict HEIF Anatomy

• Items

Roles• of images

Image• properties

Properties• association

• Image sequences

Tiles• HEIF Anatomy > Tiles

Tiles • rloc • grid HEIF Anatomy > Tiles

Tiles • rloc • grid

Why tiles? • Parallelism • Memory • Cropping HEIF Anatomy > Tiles

Tiles • rloc • grid

Why tiles? • Parallelism • Memory • Cropping

System tiles vs. HEVC tiles Agenda

• De facto standard for image compression—JPEG

• Requirements for a new format

• HEIF—The answer

• HEIF anatomy

• The codec of choice—HEVC The Codec of Choice—HEVC

ITU/MPEG MPEG1 MPEG2 MPEG4 AVC HEVC standard

JPEG JPEG JPEG2000 JPEG XR standard

1999

1992 1996 2000 2003 2009 2013 The Codec of Choice—HEVC

ITU/MPEG HEVC standard

JPEG JPEG standard 2X

1999

1992 1996 2000 2003 2009 2013 The Codec of Choice—HEVC

ITU/MPEG HEVC standard

JPEG JPEG standard

Hardware Support

1999

1992 1996 2000 2003 2009 2013 The Codec of Choice—HEVC Coding tools that outperform JPEG

• Block size

• Angular prediction

• Entropy coding

• Adaptive quantization

• Deblocking, SAO HEVC > Intra Coding Tools > Block Size

8 8

JPEG HEVC > Intra Coding Tools > Block Size

8 8 16 32 64 8 8 16 32

64

JPEG HEVC HEVC > Intra Coding Tools > Angular Prediction

8

8

JPEG HEVC > Intra Coding Tools > Angular Prediction

8

8

JPEG HEVC HEVC > Intra Coding Tools > Entropy Coding

• Huffman • + • QP

JPEG HEVC > Intra Coding Tools > Entropy Coding

• Huffman • CABAC• + • QP

JPEG HEVC HEVC > Intra Coding Tools > Adaptive Quantization

• HUFFMAN • CABAC

JPEG HEVC > Intra Coding Tools > Adaptive Quantization

• HUFFMAN • CABAC • + • QP

JPEG HEVC HEVC > Intra Coding Tools > Deblocking > SAO

Deblocking • Global smoothing sharp edges

SAO • Local smoothing HEVC > Intra Coding Tools > Deblocking > SAO

Deblocking • Global smoothing sharp edges

SAO • Local smoothing HEIF Configuration with Camera App

HEIF container

ftyp—Declares the brands in the file

meta—Information about items

mdat—Media Data

EXIF thumb compressed data main image compressed data HEIF Configuration with Camera App

.HEIC HEIF container

ftyp—Declares the brands in the file

meta—Information about items

mdat—Media Data

EXIF thumb compressed data main image compressed data HEIF Configuration with Camera App

.HEIC HEIF container

HEVC—main still profile ftyp—Declares the brands in the file

meta—Information about items

mdat—Media Data

EXIF thumb compressed data main image compressed data HEIF Configuration with Camera App

.HEIC HEIF container

HEVC—main still profile ftyp—Declares the brands in the file Images are encoded in 512x512 tiles meta—Information about items

mdat—Media Data

EXIF thumb compressed data main image compressed data HEIF Configuration with Camera App

.HEIC HEIF container

HEVC—main still profile ftyp—Declares the brands in the file Images are encoded in 512x512 tiles meta—Information about items Thumb is 320x240 mdat—Media Data

EXIF thumb compressed data main image compressed data HEIF Configuration with Camera App

.HEIC HEIF container

HEVC—main still profile ftyp—Declares the brands in the file Images are encoded in 512x512 tiles meta—Information about items Thumb is 320x240 mdat—Media Data

EXIF EXIF thumb compressed data main image compressed data HEIF Configuration with Camera App

.HEIC HEIF container

HEVC—main still profile ftyp—Declares the brands in the file Images are encoded in 512x512 tiles meta—Information about items Thumb is 320x240 mdat—Media Data

EXIF EXIF Depth stored as auxiliary image + XMP thumb compressed data main image compressed data HEIF Configuration with Camera App

.HEIC HEIF container

HEVC—main still profile ftyp—Declares the brands in the file Images are encoded in 512x512 tiles meta—Information about items Thumb is 320x240 mdat—Media Data

EXIF EXIF Depth stored as auxiliary image + XMP thumb compressed data main image compressed data Meta first, then thumbs, then main image HEIF Configuration with Camera App

.HEIC HEIF container

HEVC—main still profile ftyp—Declares the brands in the file Images are encoded in 512x512 tiles meta—Information about items Thumb is 320x240 mdat—Media Data

EXIF EXIF Depth stored as auxiliary image + XMP thumb compressed data main image compressed data Meta first, then thumbs, then main image Summary Summary

De facto image file format Summary

De facto image file format

Format requirements Summary

De facto image file format

Format requirements

HEIF Summary

De facto image file format

Format requirements

HEIF

HEIF in depth Summary

De facto image file format

Format requirements

HEIF

HEIF in depth

HEVC More Information https://developer.apple.com/wwdc17/513 Related Sessions

Introducing HEIF and HEVC Executive Ballroom Tuesday 4:10PM

Working with HEIF and HEVC Hall 2 Friday 11:00AM Labs

HEIF and HEVC Lab Technology Lab A Wed 9:00AM–11:00AM

HEIF and HEVC Lab Technology Lab F Fri 12:00PM–1:50PM