On Audio-Visual File Formats Summary • digital audio and digital video • container, codec, raw data • different formats for different purposes Reto Kromer • AV Preservation by reto.ch • audio-visual data transformations Film Preservation and Restoration Hyderabad, India 8–15 December 2019
1 2
Digital Audio • sampling Digital Audio • quantisation
3 4 Sampling • 44.1 kHz • 48 kHz • 96 kHz • 192 kHz digitisation = sampling + quantisation
5 6
Quantisation • 16 bit (216 = 65 536) • 24 bit (224 = 16 777 216) • 32 bit (232 = 4 294 967 296) Digital Video
7 8 Digital Video Resolution • resolution • SD 480i / SD 576i • bit depth • HD 720p / HD 1080i • linear, power, logarithmic • 2K / HD 1080p • colour model • 4K / UHD-1 • chroma subsampling • 8K / UHD-2 • illuminant
9 10
Bit Depth Linear, Power, Logarithmic • 8 bit (28 = 256) «medium grey» • 10 bit (210 = 1 024) • linear: 18% • 12 bit (212 = 4 096) • power: 50% • 16 bit (216 = 65 536) • logarithmic: 50% • 24 bit (224 = 16 777 216)
11 12 Colour Model • XYZ, L*a*b* • RGB / R′G′B′ / CMY / C′M′Y′ • Y′IQ / Y′UV / Y′DBDR • Y′CBCR / Y′COCG • Y′PBPR
13 14
15 16 17 18
RGB24
00000000 11111111 00000000 00000000 00000000 00000000 11111111 00000000 00000000 00000000 00000000 11111111
00000000 11111111 11111111 11111111 11111111 00000000 11111111 11111111 11111111 11111111 00000000 11111111
19 20 Compression Uncompressed • uncompressed + data simpler to process • lossless compression + software runs faster • lossy compression – bigger files • chroma subsampling – slower writing, transmission and reading • born compressed Examples: TIFF, DPX, DNG, OpenEXR
21 22
Lossless Compression Lossy Compression
+ smaller files optimised for image acquisition and/or • postproduction + faster writing, transmission and reading optimised for access – data processing complexer • – software runs slower Examples (mezzanine): ProRes 422, ProRes 4444, DNxHD, DNxHR Examples: JPEG 2000, FFV1 Examples (access): H.264 (AVC), H.265 (HEVC), AV1
23 24 Chroma Subsampling 4:4:4
4:4:4 • 4 • 4:2:2 ● ● ● ● ● ● ● ● • 4:2:0 / 4:1:1 4 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●
25 26
4:2:2 4:2:0
2 ● ○ ● ○ ● ○ ● ○ 2 ● ○ ● ○ ● ○ ● ○ 2 ● ○ ● ○ ● ○ ● ○ 0 ○ ○ ○ ○ ○ ○ ○ ○ ● ○ ● ○ ● ○ ● ○ ● ○ ● ○ ● ○ ● ○ ● ○ ● ○ ● ○ ● ○ ○ ○ ○ ○ ○ ○ ○ ○
27 28 4:1:1 Born Compressed • optimised for both image acquisition and 1 ● ○ ○ ○ ● ○ ○ ○ postproduction 1 ● ○ ○ ○ ● ○ ○ ○ Examples: CineForm RAW, ProRes RAW ● ○ ○ ○ ● ○ ○ ○ ● ○ ○ ○ ● ○ ○ ○
29 30
Bayer
31 32 Illuminant • D50 • D55 • D65 • D75
33 34
35 36 37 38
0111010100101010100010110101011110 0100110101010101010100001011101010 0111010100101010100010110101011110 0001110101010101010100001011101010 0110101010010101010001011010101111 0010101010101010000101110101010000 0111010100101010100010110101011110 File Structure 0101010101010101000010111010100110 1001011101010010101010001011010101 1110010101010101010000101110101010 0111010100101010100010110101011110 0101010101010101001101010100000001 0010100010101010101001010101010101
39 40 File Structure Audio-Visual Container audio-visual container (wrapper) • MP4 • MXF audio MOV Matroska (.mkv) codec video codec • • • AVI audio data video data
• Flash
41 42
Single Images Audio Codec • folder • MXF • WAVE • FLAC • TAR • Matroska (.mkv) • BWF • ZIP
• CinemaDNG • AAC • MP3
43 44 Video Codec (Master) Video Codec (Mezzanine) images streams • ProRes 422, ProRes 4444, ProRes RAW • TIFF • 8 bit raw • DNxHD, DNxHR • DPX • 10 bit raw • CineForm RAW • JPEG 2000 • HuffYUV • OpenEXR • FFV1 • DNG
45 46
Video Codec (Access) • H.264, H.265 (HEVC), AV1 Data is anything but «raw».
47 48 Audio Data Video Data • pcm_s16le • rgb48le • yuv444p16le • pcm_s24le • rgb24 • yuv422p10le • pcm_s32le • rgb72le • uyvy422 • yuv420p • yuv444p24le • bayer_bggr16le • bayer_bggr24le
49 50
What is inside my DPX? • log neg encoding • log RGB encoding or quasi-log encoding gamma encoding or power function • encoding • scene-linear encoding
51 52 Principles
The archive must be able to handle • the file formats it holds.
File Formats • open source • simple to use and well documented • widely used by the community
53 54
Different Purposes archive master format: Elena Rossi-Snook: ➔ for preservation mezzanine format: Archiving without access ➔ for professional use in post-production isn’t preservation, dissemination formats: ➔ for widely spreading and easy access it’s hoarding.
55 56 Archive Master (Today) Mezzanine (Today) film video • folder, TIFF, 2K, RGB, 4:4:4, 16 bit • ProRes 4444, 2K • MXF, DPX, 2K, R′G′B′, 4:4:4, 10 bit • DNxHR, 2K video • ProRes 422 HQ, HD • AVI, «raw», HD, Y′CBCR, 4:2:2, 10 bit • DNxHD 175x, HD • Matroska, FFV1, HD, Y′CBCR, 4:2:2, 10 bit audio audio • BWF, 48 kHz, 24 bit • BWF, 96 kHz, 24 bit • WAVE, 48 kHz, 24 bit • FLAC, 96 kHz, 24 bit
57 58
Dissemination (Today) Archive Master and Mezzanine
MP4 film Matroska, FFV1, 2K, RGB, 4:4:4, 16 bit Video • • H.264, SD, yuv420p, «lossy» video • H.264, HD, yuv420p, «lossy» • Matroska, FFV1, HD, Y′CBCR, 4:2:2, 10 bit Sound audio • AAC, 44.1 kHz, 16 bit • Matroska, FLAC, 96 kHz, 24 bit • AAC, 48 kHz, 16 bit
59 60 Access Reading
WebM (a subset of Matroska) Reto Kromer: Matroska and FFV1: One File Video Format for Film and Video Archiving?, in «Journal of Film Preservation», n. 96 (April • H.265, HD, yuv420p 2017), FIAF, Brussels, Belgium, p. 41–45 • AV1, HD, yuv420p ➔ https://retokromer.ch/publications/ Sound JFP_96.html • FLAC, 48 kHz, 16 bit
61 62
container: codec: • folder • TIFF • TAR • DPX • ZIP • JPEG 2000 Pros & Cons • MXF • FFV1 • Matroska • OpenEXR • CineForm RAW • ProRes RAW
63 64 avantages disavantages
TIFF data easier DPX bigger files OpenEXR to process Transformations
JPEG 2000 data complexer FFV1 smaller files to process
65 66
Audio Exemple
tristimulus transfer function colour diff. encode subsampling filter interpolation filter colour diff. decode transfer function tristimulus MOV demultiplex WAVE [M] [M] [M] [M] decode pcm_s24le
′ ′ filter B B
′ ′ pcm_s16le XYZ RGB XYZ G G RGB encode 4:4:4 ′ ′ 4:2:2 4:4:4 R R R
R R AAC C C C B B B
C multiplex C C ′ ′ ′ MP4 Y Y Y
67 68 Video Exemple Audio-Visual Exemple
MOV MOV demultiplex demultiplex ProRes 422 HQ WAVE decode decode ProRes 422 HQ yuv422p10le pcm_s24le filter yuv422p10le filter yuv420p pcm_s16le H.264 encode AAC encode yuv420p multiplex H.264 multiplex MP4 MP4
69 70
Acknowledgements AV Preservation by reto.ch
Swiss Federal Institute of Technology chemin du Suchet 5 • 1024 Écublens • Massachusetts Institute of Technology Switzerland • Kinemathek Lichtspiel, Bern Web: reto.ch Twitter: @retoch • Charles Poynton Email: [email protected] • Dave Rice & Misty De Meo • Agathe Jarczyk & David Pfluger
71 72