Media #WWDC17
High• Efficiency Image File Format
Session 513
Davide Concion, Apple/Image Compression Team
© 2017 Apple Inc. All rights reserved. Redistribution or public display not permitted without written permission from Apple. • De facto standard for image compression—JPEG
• Requirements for a new format
• HEIF—The answer
• HEIF anatomy
• The codec of choice—HEVC • De facto standard for image compression—JPEG
• Requirements for a new format
• HEIF—The answer
• HEIF anatomy
• The codec of choice—HEVC JPEG > De facto Standard for Image Compression
Hundreds of millions of JPEG images are captured, created, uploaded, and shared daily JPEG > De facto Standard for Image Compression
Hundreds of millions of JPEG images are captured, created, uploaded, and shared daily
Limitations • Compression • Auxiliary images • Animations JPEG > 1992
ITU/MPEG standard
JPEG JPEG standard
1992 JPEG > 1992
ITU/MPEG MPEG1 MPEG2 MPEG4 AVC HEVC standard
JPEG JPEG JPEG2000 JPEG XR standard
1999
1992 1996 2000 2003 2009 2013 JPEG > 1992
ITU/MPEG MPEG1 MPEG2 MPEG4 AVC HEVC standard
JPEG JPEG JPEG2000 JPEG XR standard
HEIF
1999
1992 1996 2000 2003 2009 2013 2015 • De facto standard for image compression—JPEG
• Requirements for a new format
• HEIF—The answer
• HEIF anatomy
• The codec of choice—HEVC Requirements for a New Format Requirements for a New Format
State-of-the-art compression Requirements for a New Format
State-of-the-art compression
HW acceleration Requirements for a New Format
State-of-the-art compression
HW acceleration
Deep and wide color Requirements for a New Format
State-of-the-art compression
HW acceleration
Deep and wide color
4:4:4 color sampling Requirements for a New Format
State-of-the-art compression
HW acceleration
Deep and wide color
4:4:4 color sampling
HDR Requirements for a New Format
State-of-the-art compression
HW acceleration
Deep and wide color
4:4:4 color sampling
HDR
Auxiliary images Requirements for a New Format
State-of-the-art compression
HW acceleration
Deep and wide color
4:4:4 color sampling
HDR
Auxiliary images
Animation, burst, and playback intent Requirements for a New Format
State-of-the-art compression Multiple-image
HW acceleration
Deep and wide color
4:4:4 color sampling
HDR
Auxiliary images
Animation, burst, and playback intent Requirements for a New Format
State-of-the-art compression Multiple-image
HW acceleration Multi-resolution
Deep and wide color
4:4:4 color sampling
HDR
Auxiliary images
Animation, burst, and playback intent Requirements for a New Format
State-of-the-art compression Multiple-image
HW acceleration Multi-resolution
Deep and wide color Tiles
4:4:4 color sampling
HDR
Auxiliary images
Animation, burst, and playback intent Requirements for a New Format
State-of-the-art compression Multiple-image
HW acceleration Multi-resolution
Deep and wide color Tiles
4:4:4 color sampling Rich metadata
HDR
Auxiliary images
Animation, burst, and playback intent Requirements for a New Format
State-of-the-art compression Multiple-image
HW acceleration Multi-resolution
Deep and wide color Tiles
4:4:4 color sampling Rich metadata
HDR Other media types
Auxiliary images
Animation, burst, and playback intent Requirements for a New Format
State-of-the-art compression Multiple-image
HW acceleration Multi-resolution
Deep and wide color Tiles
4:4:4 color sampling Rich metadata
HDR Other media types
Auxiliary images Extensible
Animation, burst, and playback intent • De facto standard for image compression: JPEG
• Requirements for a new format
• HEIF—The answer
• HEIF anatomy
• The codec of choice—HEVC HEIF > Standard
High Efficiency Image File Format HEIF > Standard
High Efficiency Image File Format
ISO standard—ISO/IEC 23008-12 (June 2015—Version 1) • http://mpeg.chiariglione.org/standards/mpeg-h/image-file-format HEIF > Standard
High Efficiency Image File Format
ISO standard—ISO/IEC 23008-12 (June 2015—Version 1) • http://mpeg.chiariglione.org/standards/mpeg-h/image-file-format
Version 2 of the spec should be made public soon HEIF > Standard
High Efficiency Image File Format
ISO standard—ISO/IEC 23008-12 (June 2015—Version 1) • http://mpeg.chiariglione.org/standards/mpeg-h/image-file-format
Version 2 of the spec should be made public soon
Reference model from MPEG • http://mp4ra.org HEIF > Standard
High Efficiency Image File Format
ISO standard—ISO/IEC 23008-12 (June 2015—Version 1) • http://mpeg.chiariglione.org/standards/mpeg-h/image-file-format
Version 2 of the spec should be made public soon
Reference model from MPEG • http://mp4ra.org
GPAC/MP4Box has recently added basic support for HEIF • https://gpac.wp.imt.fr HEIF > Container
HEIF specifies a structural format HEIF container (a “container”) ftyp—Declares the brands in the file
meta — Information about items
mdat—Media Data
EXIF thumbnail image data HEIF > Container
HEIF specifies a structural format HEIF container (a “container”) ftyp—Declares the brands in the file Derived from ISO Base Media File Format
(ISOBMFF - ISO/IEC 14496-12), meta — Information about items based on Apple QuickTime • MP4, timed media mdat—Media Data • MPEG-21, digital items EXIF thumbnail image data HEIF > Container
HEIF specifies a structural format HEIF container (a “container”) ftyp—Declares the brands in the file Derived from ISO Base Media File Format
(ISOBMFF - ISO/IEC 14496-12), meta — Information about items based on Apple QuickTime • MP4, timed media mdat—Media Data • MPEG-21, digital items EXIF thumbnail Any compression codec can be image data included in a HEIF file HEIF > Container
HEIF specifies a structural format HEIF container (a “container”) ftyp—Declares the brands in the file Derived from ISO Base Media File Format
(ISOBMFF - ISO/IEC 14496-12), meta — Information about items based on Apple QuickTime • MP4, timed media mdat—Media Data • MPEG-21, digital items EXIF thumbnail Any compression codec can be image data included in a HEIF file HEIF > File Extension
Payload Extension—single image Extension—sequence
HEVC .HEIC .HEICS
H.264 .AVCI .AVCS
Any Codec .HEIF .HEIFS HEIF > File Extension
Payload Extension—single image Extension—sequence
HEVC .HEIC .HEICS
H.264 .AVCI .AVCS
Any Codec .HEIF .HEIFS HEIF > File Extension
Payload Extension—single image Extension—sequence
HEVC .HEIC .HEICS
H.264 .AVCI .AVCS
Any Codec .HEIF .HEIFS • De facto standard for image compression—JPEG
• Requirements for a new format
• HEIF—The answer
• HEIF anatomy
• The codec of choice—HEVC HEIF Anatomy
• Items
• Roles of images
Image• properties
Properties• association
Image• sequences
Tiles• HEIF Anatomy > Items HEIF Anatomy > Items
AVC
HEVC
HEVC HEIF Anatomy > Items
Primary Derived Image
AVC
HEVC
HEVC HEIF Anatomy > Items
EXIF
XMP
MPEG-7
Primary Derived Image
AVC
HEVC
HEVC HEIF Anatomy > Items
EXIF
Properties XMP MPEG-7 Initialization
… Primary Size Derived Image
Mirror AVC … HEVC
HEVC HEIF Anatomy > Items
EXIF
Properties XMP cdsc MPEG-7 Initialization
… Primary Size Derived Image
Mirror AVC … dimg HEVC HEVC HEIF Anatomy
• Items
• Roles of images
Image• properties
Properties• association
Image• sequences
Tiles• HEIF Anatomy > Roles of Images
Primary Representative image HEIF container
Master meta - Information about items Full-resolution displayable image
pitm - Primary Item Box Thumbnail Smaller resolution
Auxillary Alpha plane or a depth map iref - Item Reference Box dimg - Derived Image Hidden Not for display thmb - Thumbnail auxl - Auxiliary Image Derived ‘grid’, ‘iden’, ‘iovl’ cdsc - Content describe
Equivalent Alternative images HEIF Anatomy
• Items
Roles• of images
Image• properties
Properties• association
Image• sequences
Tiles• HEIF Anatomy > Image Properties > Descriptive
Decoder configuration Codec initialization HEIF container
meta—Information about items Image spatial extents Width and height
iprp—Item Properties Pixel aspect ratio Pixel aspect ratio
ipco—Item Property Container Colour information ICC profile ispe—Image Spatial Extents
Relative location Position of the reconstructed image colr—Colour Information hvcC—HEVC Decoder Conf. Auxiliary images Type of auxiliary image properties HEIF Anatomy > Image Properties > Transformative
Clean aperture Cropping of image HEIF container
meta—Information about items Image rotation 0, 90,180,270 degrees rotation
iprp—Item Properties Image mirroring Image mirroring
ipco—Item Property Container
clap—Clean Aperture irot—Rotation imir—Mirroring HEIF Anatomy
• Items
Roles• of images
Image• properties
Properties• association
Image• sequences
Tiles• HEIF Anatomy > Image Properties > Association
HEIF container HEIF container meta—Information about items ipma—Item Properties Association iprp—Item Properties item 1—association ipco—Item Property Container property 1 property 2 hvcC—Decoder configuration 1 ispe—spatial extent 1 width: 500 height: 500
ispe—spatial extent 2 item 5—association width: 1000 height: 1000 property 3 hvcC—Decoder configuration 2 item 6—association ispe—spatial extent 3 property 4 width: 320 height: 240 property 5 HEIF Anatomy > Image Properties > Association
HEIF container HEIF container meta—Information about items ipma—Item Properties Association iprp—Item Properties item 1—association ipco—Item Property Container property 1 property 2 hvcC—Decoder configuration 1 1 ispe—spatial extent 1 2 width: 500 height: 500 ispe—spatial extent 2 3 item 5—association width: 1000 height: 1000 property 3 hvcC—Decoder configuration 2 4 item 6—association ispe—spatial extent 3 5 property 4 width: 320 height: 240 property 5 HEIF Anatomy > Image Properties > Association
HEIF container HEIF container meta—Information about items ipma—Item Properties Association iprp—Item Properties item 1—association ipco—Item Property Container property 1 property 2 hvcC—Decoder configuration 1 1 ispe—spatial extent 1 2 width: 500 height: 500 ispe—spatial extent 2 3 item 5—association width: 1000 height: 1000 property 3 hvcC—Decoder configuration 2 4 item 6—association ispe—spatial extent 3 5 property 4 width: 320 height: 240 property 5 HEIF Anatomy > Image Properties > Association
HEIF container HEIF container meta—Information about items ipma—Item Properties Association iprp—Item Properties item 1—association ipco—Item Property Container property 1 property 2 hvcC—Decoder configuration 1 1 ispe—spatial extent 1 2 width: 500 height: 500 ispe—spatial extent 2 3 item 5—association width: 1000 height: 1000 property 3 hvcC—Decoder configuration 2 4 item 6—association ispe—spatial extent 3 5 property 4 width: 320 height: 240 property 5 HEIF Anatomy > Image Properties > Association
HEIF container HEIF container meta—Information about items ipma—Item Properties Association iprp—Item Properties item 1—association ipco—Item Property Container property 1 property 2 hvcC—Decoder configuration 1 1 ispe—spatial extent 1 2 width: 500 height: 500 ispe—spatial extent 2 3 item 5—association width: 1000 height: 1000 property 3 hvcC—Decoder configuration 2 4 item 6—association ispe—spatial extent 3 5 property 4 width: 320 height: 240 property 5 HEIF Anatomy
• Items
Roles• of images
Image• properties
Properties• association
• Image sequences
Tiles• HEIF Anatomy > Sequences
‘moov’ box—‘trak’ box—‘pict’ handler HEIF container
ftyp—Declares the brands in the file
meta—Information about items
mdat—Media Data
moov—Movie Box
track—video track—sound track—pict HEIF Anatomy > Sequences
‘moov’ box—‘trak’ box—‘pict’ handler HEIF container
Tracks have roles too ftyp—Declares the brands in the file
meta—Information about items
mdat—Media Data
moov—Movie Box
track—video track—sound track—pict HEIF Anatomy > Sequences
‘moov’ box—‘trak’ box—‘pict’ handler HEIF container
Tracks have roles too ftyp—Declares the brands in the file
Controlling the playback meta—Information about items • Edit list mdat—Media Data • Looping moov—Movie Box track—video track—sound track—pict HEIF Anatomy > Sequences
‘moov’ box—‘trak’ box—‘pict’ handler HEIF container
Tracks have roles too ftyp—Declares the brands in the file
Controlling the playback meta—Information about items • Edit list mdat—Media Data • Looping moov—Movie Box track—video Inter prediction track—sound track—pict HEIF Anatomy
• Items
Roles• of images
Image• properties
Properties• association
• Image sequences
Tiles• HEIF Anatomy > Tiles
Tiles • rloc • grid HEIF Anatomy > Tiles
Tiles • rloc • grid
Why tiles? • Parallelism • Memory • Cropping HEIF Anatomy > Tiles
Tiles • rloc • grid
Why tiles? • Parallelism • Memory • Cropping
System tiles vs. HEVC tiles Agenda
• De facto standard for image compression—JPEG
• Requirements for a new format
• HEIF—The answer
• HEIF anatomy
• The codec of choice—HEVC The Codec of Choice—HEVC
ITU/MPEG MPEG1 MPEG2 MPEG4 AVC HEVC standard
JPEG JPEG JPEG2000 JPEG XR standard
1999
1992 1996 2000 2003 2009 2013 The Codec of Choice—HEVC
ITU/MPEG HEVC standard
JPEG JPEG standard 2X
1999
1992 1996 2000 2003 2009 2013 The Codec of Choice—HEVC
ITU/MPEG HEVC standard
JPEG JPEG standard
Hardware Support
1999
1992 1996 2000 2003 2009 2013 The Codec of Choice—HEVC Coding tools that outperform JPEG
• Block size
• Angular prediction
• Entropy coding
• Adaptive quantization
• Deblocking, SAO HEVC > Intra Coding Tools > Block Size
8 8
JPEG HEVC > Intra Coding Tools > Block Size
8 8 16 32 64 8 8 16 32
64
JPEG HEVC HEVC > Intra Coding Tools > Angular Prediction
8
8
JPEG HEVC > Intra Coding Tools > Angular Prediction
8
8
JPEG HEVC HEVC > Intra Coding Tools > Entropy Coding
• Huffman • + • QP
JPEG HEVC > Intra Coding Tools > Entropy Coding
• Huffman • CABAC• + • QP
JPEG HEVC HEVC > Intra Coding Tools > Adaptive Quantization
• HUFFMAN • CABAC
JPEG HEVC > Intra Coding Tools > Adaptive Quantization
• HUFFMAN • CABAC • + • QP
JPEG HEVC HEVC > Intra Coding Tools > Deblocking > SAO
Deblocking • Global smoothing sharp edges
SAO • Local smoothing HEVC > Intra Coding Tools > Deblocking > SAO
Deblocking • Global smoothing sharp edges
SAO • Local smoothing HEIF Configuration with Camera App
HEIF container
ftyp—Declares the brands in the file
meta—Information about items
mdat—Media Data
EXIF thumb compressed data main image compressed data HEIF Configuration with Camera App
.HEIC HEIF container
ftyp—Declares the brands in the file
meta—Information about items
mdat—Media Data
EXIF thumb compressed data main image compressed data HEIF Configuration with Camera App
.HEIC HEIF container
HEVC—main still profile ftyp—Declares the brands in the file
meta—Information about items
mdat—Media Data
EXIF thumb compressed data main image compressed data HEIF Configuration with Camera App
.HEIC HEIF container
HEVC—main still profile ftyp—Declares the brands in the file Images are encoded in 512x512 tiles meta—Information about items
mdat—Media Data
EXIF thumb compressed data main image compressed data HEIF Configuration with Camera App
.HEIC HEIF container
HEVC—main still profile ftyp—Declares the brands in the file Images are encoded in 512x512 tiles meta—Information about items Thumb is 320x240 mdat—Media Data
EXIF thumb compressed data main image compressed data HEIF Configuration with Camera App
.HEIC HEIF container
HEVC—main still profile ftyp—Declares the brands in the file Images are encoded in 512x512 tiles meta—Information about items Thumb is 320x240 mdat—Media Data
EXIF EXIF thumb compressed data main image compressed data HEIF Configuration with Camera App
.HEIC HEIF container
HEVC—main still profile ftyp—Declares the brands in the file Images are encoded in 512x512 tiles meta—Information about items Thumb is 320x240 mdat—Media Data
EXIF EXIF Depth stored as auxiliary image + XMP thumb compressed data main image compressed data HEIF Configuration with Camera App
.HEIC HEIF container
HEVC—main still profile ftyp—Declares the brands in the file Images are encoded in 512x512 tiles meta—Information about items Thumb is 320x240 mdat—Media Data
EXIF EXIF Depth stored as auxiliary image + XMP thumb compressed data main image compressed data Meta first, then thumbs, then main image HEIF Configuration with Camera App
.HEIC HEIF container
HEVC—main still profile ftyp—Declares the brands in the file Images are encoded in 512x512 tiles meta—Information about items Thumb is 320x240 mdat—Media Data
EXIF EXIF Depth stored as auxiliary image + XMP thumb compressed data main image compressed data Meta first, then thumbs, then main image Summary Summary
De facto image file format Summary
De facto image file format
Format requirements Summary
De facto image file format
Format requirements
HEIF Summary
De facto image file format
Format requirements
HEIF
HEIF in depth Summary
De facto image file format
Format requirements
HEIF
HEIF in depth
HEVC More Information https://developer.apple.com/wwdc17/513 Related Sessions
Introducing HEIF and HEVC Executive Ballroom Tuesday 4:10PM
Working with HEIF and HEVC Hall 2 Friday 11:00AM Labs
HEIF and HEVC Lab Technology Lab A Wed 9:00AM–11:00AM
HEIF and HEVC Lab Technology Lab F Fri 12:00PM–1:50PM