On Audio-Visual File Formats Summary • digital audio and digital video • container, codec, raw data Reto Kromer • AV Preservation by reto.ch • different formats for different purposes audio-visual data transformations La preservazione digitale • del film e del video Università di Udine a Gorizia 7–10 maggio 2018

1 2

Digital Audio • sampling Digital Audio • quantisation

3 4 Sampling • 44.1 kHz • 48 kHz • 96 kHz • 192 kHz digitisation = sampling + quantisation

5 6

Quantisation • 16 bit • 24 bit • 32 bit Digital Video

7 8 Digital Video Resolution • resolution • SD 480i / SD 576i • bit depth • HD 720p / HD 1080i • linear, power, logarithmic • 2K / HD 1080p • colour model • 4K / UHD-1 • chroma subsampling • 8K / UHD-2 • illuminant

9 10

Bit Depth Linear, Power, Logarithmic • 8 bit «medium grey» • 10 bit • linear: 18% • 12 bit • «logarithmic»: 50% • 16 bit • 24 bit

11 12 Colour Model • XYZ • RGB / R′G′B′ / CMY / C′M′Y′ • Y′IQ / Y′UV / Y′DBDR • Y′CBCR / Y′COCG • Y′PBPR

13 14

RGB24

00000000 11111111 00000000 00000000 00000000 00000000 11111111 00000000 00000000 00000000 00000000 11111111

00000000 11111111 11111111 11111111 11111111 00000000 11111111 11111111 11111111 11111111 00000000 11111111

15 16 Chroma Subsampling • 4:4:4 • 4:2:2 • 4:2:0 / 4:1:1

17 18

4:4:4 4:2:2

4 ● ● ● ● ● ● ● ● 2 ● ○ ● ○ ● ○ ● ○ 4 ● ● ● ● ● ● ● ● 2 ● ○ ● ○ ● ○ ● ○ ● ● ● ● ● ● ● ● ● ○ ● ○ ● ○ ● ○ ● ● ● ● ● ● ● ● ● ○ ● ○ ● ○ ● ○

19 20 4:2:0 4:1:1

2 ● ○ ● ○ ● ○ ● ○ 1 ● ○ ○ ○ ● ○ ○ ○ 0 ○ ○ ○ ○ ○ ○ ○ ○ 1 ● ○ ○ ○ ● ○ ○ ○ ● ○ ● ○ ● ○ ● ○ ● ○ ○ ○ ● ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ● ○ ○ ○ ● ○ ○ ○

21 22

Illuminant • D50 • D55 • D65 • D75

23 24 25 26

File Structure

27 28 0111010100101010100010110101011110 0100110101010101010100001011101010 File Structure 0111010100101010100010110101011110 0001110101010101010100001011101010 audio-visual container (wrapper) 0110101010010101010001011010101111 audio 0010101010101010000101110101010000 codec video codec 0111010100101010100010110101011110 0101010101010101000010111010100110 1001011101010010101010001011010101 audio video data 1110010101010101010000101110101010 data 0111010100101010100010110101011110 0101010101010101001101010100000001 0010100010101010101001010101010101

29 30

Audio-Visual Container Audio Codec • MP4 • MXF • WAVE • BWF • MOV • Matroska (.mkv) • FLAC • AVI

• AAC • Flash • MP3

31 32 Video Codec • H.264, H.265 (HEVC) Data is anything • ProRes 422, ProRes 4444, ProRes RAW • DNxHD, DNxHR but «raw». • CineForm RAW • FFV1

33 34

Audio Data Video Data • pcm_s16le • rgb48le • yuv444p16le • pcm_s24le • rgb24 • yuv422p10le • pcm_s32le • rgb72le • uyvy422 • yuv420p • yuv444p24le • bayer_bggr16le • bayer_bggr24e

35 36 «Single-Image Codec» «Single-Image Container» • TIFF • FFV1 • folder • MXF • DPX • Motion JPEG • JPEG 2000 • TAR • Matroska (.mkv) • OpenEXR • ZIP • CinemaDNG • DNG

37 38

What is inside my DPX? • log neg encoding • log RGB encoding or quasi-log encoding gamma encoding or power function • encoding • scene-linear encoding

39 40 Principles

The archive must be able to handle • the file formats it holds.

File Formats • open source • simple to use and well documented • widely used by the community

41 42

Different Purposes archive master format: Elena Rossi-Snook: ➔ for preservation mezzanine format: Archiving without access ➔ for professional use in post-production isn’t preservation, dissemination formats: ➔ for widely spreading and easy access it’s hoarding.

43 44 Archive Master (Today) Mezzanine (Today) film video • folder, TIFF, 2K, RGB, 4:4:4, 16 bit • Apple ProRes 4444, 2K • MXF, DPX, 2K, R′G′B′, 4:4:4, 10 bit • Avid DNxHR, 2K video • Apple ProRes 422 HQ, HD • AVI, «raw», HD, Y′CBCR, 4:2:2, 10 bit • Avid DNxHD 175x, HD • Matroska, FFV1, HD, Y′CBCR, 4:2:2, 10 bit audio audio • BWF, 48 kHz, 24 bit • BWF, 96 kHz, 24 bit • WAVE, 48 kHz, 24 bit • FLAC, 96 kHz, 24 bit

45 46

Dissemination (Today) Archive Master and Mezzanine

MP4 film Matroska, FFV1, 2K, «RGB», 4:4:4, 16 bit Video • • H.264, SD, yuv420p, «lossy» video • H.264, HD, yuv420p, «lossy» • Matroska, FFV1, HD, Y′CBCR, 4:2:2, 10 bit Sound audio • AAC, 44.1 kHz, 16 bit • Matroska, FLAC, 96 kHz, 24 bit • AAC, 48 kHz, 16 bit

47 48 Access Reading

WebM (a subset of Matroska) Reto Kromer: Matroska and FFV1: One File Video Format for Film and Video Archiving?, in «Journal of Film Preservation», n. 96 (April • «H.265», HD, yuv420p 2017), FIAF, Brussels, Belgium, p. 41–45 Sound ➔ https://retokromer.ch/publications/ • «FLAC», 48 kHz, 16 bit JFP_96.html

49 50

container: codec: • folder • Cineon, DPX • TAR • TIFF, TI/A • ZIP • JPEG 2000 Pros & Cons • MXF • FFV1 • Matroska • OpenEXR • AXF • CineForm (VC-5) • ProRes RAW

51 52 tristimulus transfer function colour diff. encode subsampling filter interpolation filter colour diff. decode transfer function tristimulus Transformations [M] [M] [M] [M] ′ ′ 4 4 B B ′ ′ XYZ RGB XYZ G RGB G 4:4: ′ 4:2:2 ′ 4:4: R R R R R C C C B B B C C C ′ ′ ′ Y Y Y

53 54

Data Transformations Audio Exemple

MOV demultiplex demultiplex WAVE decode decode pcm_s24le filter filter pcm_s16le encode encode AAC multiplex multiplex MP4

55 56 Video Exemple Audio-Visual Exemple

MOV MOV demultiplex demultiplex ProRes 422 HQ WAVE decode decode ProRes 422 HQ yuv422p10le pcm_s24le filter yuv422p10le filter yuv420p pcm_s16le H.264 encode AAC encode yuv420p multiplex H.264 multiplex MP4 MP4

57 58

Acknowledgements AV Preservation by reto.ch

Swiss Federal Institute of Technology chemin du Suchet 5 • 1024 Écublens • Massachusetts Institute of Technology Switzerland • Kinemathek Lichtspiel, Bern Web: reto.ch Twitter: @retoch • Charles Poynton Email: [email protected] • Dave Rice & Misty De Meo • Agathe Jarczyk & David Pfluger

59 60