On Audio-Visual File Formats Summary • digital audio and digital video • container, codec, raw data Reto Kromer • AV Preservation by reto.ch • different formats for different purposes Open-Source Tools and Resources audio-visual data transformations for Audio-Visual Archives • Elías Querejeta Zine Eskola Donostia (San Sebastián), Spain 19, 21, 26 and 28 May 2020
1 2
Digital Audio • sampling Digital Audio • quantisation
3 4 Sampling • 44.1 kHz • 48 kHz • 96 kHz • 192 kHz digitisation = sampling + quantisation • 500 kHz
5 6
Quantisation • 16 bit (216 = 65 536) • 24 bit (224 = 16 777 216) • 32 bit (232 = 4 294 967 296) Digital Video
7 8 Digital Video Resolution • resolution • SD 480i / SD 576i • bit depth • HD 720p / HD 1080i • linear, power, logarithmic • 2K / HD 1080p • colour model • 4K / UHD-1 • chroma subsampling • 8K / UHD-2 • illuminant
9 10
Bit Depth Linear, Power, Logarithmic • 8 bit (28 = 256) «medium grey» • 10 bit (210 = 1 024) • linear: 18% • 12 bit (212 = 4 096) • power: 50% • 16 bit (216 = 65 536) • logarithmic: 50% • 24 bit (224 = 16 777 216)
11 12 Colour Model • XYZ, L*a*b* • RGB / R′G′B′ / CMY / C′M′Y′ • Y′IQ / Y′UV / Y′DBDR • Y′CBCR / Y′COCG • Y′PBPR
13 14
15 16 RGB24
00000000 11111111 00000000 00000000 00000000 00000000 11111111 00000000 00000000 00000000 00000000 11111111
00000000 11111111 11111111 11111111 11111111 00000000 11111111 11111111 11111111 11111111 00000000 11111111
17 18
Compression • uncompressed • lossless compression • lossy compression • chroma subsampling • born compressed
19 20 Uncompressed Lossless Compression
+ data simpler to process + smaller files + software runs faster + faster writing, transmission and reading – bigger files – data processing complexer – slower writing, transmission and reading – software runs slower
Examples: TIFF, DPX, DNG, OpenEXR Examples: JPEG 2000, FFV1
21 22
Lossy Compression Chroma Subsampling
optimised for image acquisition and/or 4:4:4 • postproduction • 4:2:2 optimised for access • • • 4:2:0 / 4:1:1 Examples (mezzanine): ProRes 422, ProRes 4444, DNxHD, DNxHR Examples (access): H.264 (AVC), H.265 (HEVC), AV1
23 24 4:4:4 4:2:2
4 ● ● ● ● ● ● ● ● 2 ● ○ ● ○ ● ○ ● ○ 4 ● ● ● ● ● ● ● ● 2 ● ○ ● ○ ● ○ ● ○ ● ● ● ● ● ● ● ● ● ○ ● ○ ● ○ ● ○ ● ● ● ● ● ● ● ● ● ○ ● ○ ● ○ ● ○
25 26
4:2:0 4:1:1
2 ● ○ ● ○ ● ○ ● ○ 1 ● ○ ○ ○ ● ○ ○ ○ 0 ○ ○ ○ ○ ○ ○ ○ ○ 1 ● ○ ○ ○ ● ○ ○ ○ ● ○ ● ○ ● ○ ● ○ ● ○ ○ ○ ● ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ● ○ ○ ○ ● ○ ○ ○
27 28 Born Compressed Bayer
optimised for both image acquisition and • postproduction
Examples: CineForm RAW, ProRes RAW
29 30
31 32 Illuminant • D50 • D55 • D65 • D75
33 34
35 36 File Structure
37 38 0111010100101010100010110101011110 0100110101010101010100001011101010 File Structure 0111010100101010100010110101011110 0001110101010101010100001011101010 audio-visual container (wrapper) 0110101010010101010001011010101111 audio 0010101010101010000101110101010000 codec video codec 0111010100101010100010110101011110 0101010101010101000010111010100110 1001011101010010101010001011010101 audio video data 1110010101010101010000101110101010 data 0111010100101010100010110101011110 0101010101010101001101010100000001 0010100010101010101001010101010101
39 40 Audio-Visual Container Single Images • MP4 • MXF • folder • MXF • MOV • Matroska (.mkv) • TAR • Matroska (.mkv) • AVI • ZIP
• CinemaDNG • Flash
41 42
Audio Codec Video Codec (Master) • WAVE • FLAC images streams • BWF • TIFF • 8 bit raw • DPX • 10 bit raw • JPEG 2000 • HuffYUV • OpenEXR • FFV1 • AAC • DNG • MP3
43 44 Video Codec (Mezzanine) Video Codec (Access) • ProRes 422, ProRes 4444, ProRes RAW • H.264, H.265 (HEVC), AV1 • DNxHD, DNxHR • CineForm RAW
45 46
Audio Data • pcm_s16le Data is anything • pcm_s24le but «raw». • pcm_s32le
47 48 Video Data What is inside my DPX? • rgb48le • yuv444p16le • log neg encoding • rgb24 • yuv422p10le • log RGB encoding or quasi-log encoding rgb72le uyvy422 gamma encoding or power function • • • encoding yuv420p • scene-linear encoding • yuv444p24le • • bayer_bggr16le • bayer_bggr24le
49 50
Principles
The archive must be able to handle • the file formats it holds.
File Formats • open source • simple to use and well documented • widely used by the community
51 52 Different Purposes archive master format: Elena Rossi-Snook: ➔ for preservation mezzanine format: Archiving without access ➔ for professional use in post-production isn’t preservation, dissemination formats: ➔ for widely spreading and easy access it’s hoarding.
53 54
Archive Master (Today) Mezzanine (Today) film video • folder, TIFF, 2K, RGB, 4:4:4, 16 bit • ProRes 4444, 2K • MXF, DPX, 2K, R′G′B′, 4:4:4, 10 bit • DNxHR, 2K video • ProRes 422 HQ, HD • AVI, «raw», HD, Y′CBCR, 4:2:2, 10 bit • DNxHD 175x, HD • Matroska, FFV1, HD, Y′CBCR, 4:2:2, 10 bit audio audio • BWF, 48 kHz, 24 bit • BWF, 96 kHz, 24 bit • WAVE, 48 kHz, 24 bit • FLAC, 96 kHz, 24 bit
55 56 Dissemination (Today) Archive Master and Mezzanine
MP4 film Matroska, FFV1, 2K, «RGB», 4:4:4, 16 bit Video • • H.264, SD, yuv420p, «lossy» video • H.264, HD, yuv420p, «lossy» • Matroska, FFV1, HD, Y′CBCR, 4:2:2, 10 bit Sound audio • AAC, 44.1 kHz, 16 bit • Matroska, FLAC, 96 kHz, 24 bit • AAC, 48 kHz, 16 bit
57 58
Access Reading
WebM (a subset of Matroska) Reto Kromer: Matroska and FFV1: One File Video Format for Film and Video Archiving?, in «Journal of Film Preservation», n. 96 (April • «H.265», HD, yuv420p 2017), FIAF, Brussels, Belgium, p. 41–45 • AV1, HD, yuv420p ➔ https://retokromer.ch/publications/ Sound JFP_96.html • FLAC, 48 kHz, 16 bit
59 60 container: codec: • folder • TIFF • TAR • DPX • ZIP • JPEG 2000 Pros & Cons • MXF • FFV1 • Matroska • OpenEXR • CineForm RAW • ProRes RAW
61 62
avantages disavantages
TIFF data easier DPX to process bigger files OpenEXR Transformations
JPEG 2000 smaller files data complexer FFV1 to process
63 64 colour diff. encode colour diff. decode transfer function tristimulus tristimulus transfer function interpolation filter subsampling filter
[M] [M] [M] [M] ′ ′ B B ′ ′ XYZ RGB XYZ RGB G G 4:4:4 4:2:2 ′ ′ 4:4:4 R R R R R C C C B B B C C C ′ ′ ′ Y Y Y
65 66
Data Transformations Audio Exemple
QuickTime demultiplex demultiplex WAVE decode decode pcm_s24le filter filter pcm_s16le encode encode AAC multiplex multiplex MP4
67 68 Video Exemple Audio-Visual Exemple
QuickTime QuickTime demultiplex demultiplex ProRes 422 HQ WAVE decode decode ProRes 422 HQ yuv422p10le pcm_s24le filter yuv422p10le filter yuv420p pcm_s16le H.264 encode AAC encode yuv420p multiplex H.264 multiplex MP4 MP4
69 70
Acknowledgements AV Preservation by reto.ch
Swiss Federal Institute of Technology zone industrielle Le Trési 3 • 1028 Préverenges • Massachusetts Institute of Technology Switzerland • Kinemathek Lichtspiel, Bern Web: reto.ch Twitter: @retoch • Charles Poynton Email: [email protected] • Dave Rice & Misty De Meo • Agathe Jarczyk & David Pfluger
71 72