On Audio-Visual File Formats Summary • digital audio and digital video • container, codec, raw data different formats for different purposes Joshua Ng • Archives New Zealand • Reto Kromer • AV Preservation by reto.ch • audio-visual data transformations FFmpeg for audio-visual archivists IASA/JTS, Hilversum, The Netherlands 3 October 2019

1 2

Digital Audio • sampling Digital Audio • quantisation

3 4 Sampling • 44.1 kHz • 48 kHz • 96 kHz • 192 kHz digitisation = sampling + quantisation

5 6

Quantisation • 16 bit • 24 bit • 32 bit Digital Video

7 8 Digital Video Resolution • resolution • SD 480i / SD 576i • bit depth • HD 720p / HD 1080i • linear, power, logarithmic • 2K / HD 1080p • colour model • 4K / UHD-1 • chroma subsampling • 8K / UHD-2 • illuminant

9 10

Bit Depth Linear, Power, Logarithmic • 8 bit «medium grey» • 10 bit • linear: 18% • 12 bit • power: 50% • 16 bit • logarithmic: 50% • 24 bit

11 12 Colour Model • XYZ, L*a*b* • RGB / R′G′B′ / CMY / C′M′Y′ • Y′IQ / Y′UV / Y′DBDR • Y′CBCR / Y′COCG • Y′PBPR

13 14

RGB24

00000000 11111111 00000000 00000000 00000000 00000000 11111111 00000000 00000000 00000000 00000000 11111111

00000000 11111111 11111111 11111111 11111111 00000000 11111111 11111111 11111111 11111111 00000000 11111111

15 16 Compression Uncompressed • uncompressed + data simpler to process • + software runs faster • – bigger files • compressed-born – slower writing, transmission and reading

Examples: TIFF, DPX, DNG, OpenEXR

17 18

Lossless Compression Lossy Compression

+ smaller files optimised for image acquisition and/or • postproduction + faster writing, transmission and reading optimised for access – data processing complexer • – software runs slower Examples (mezzanine): ProRes 422, ProRes 4444, DNxHD, DNxHR Examples: JPEG 2000, FFV1 Examples (access): H.264 (AVC), H.265 (HEVC), AV1

19 20 Chroma Subsampling 4:4:4

4:4:4 • 4 • 4:2:2 ● ● ● ● ● ● ● ● • 4:2:0 / 4:1:1 4 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

21 22

4:2:2 4:2:0

2 ● ○ ● ○ ● ○ ● ○ 2 ● ○ ● ○ ● ○ ● ○ 2 ● ○ ● ○ ● ○ ● ○ 0 ○ ○ ○ ○ ○ ○ ○ ○ ● ○ ● ○ ● ○ ● ○ ● ○ ● ○ ● ○ ● ○ ● ○ ● ○ ● ○ ● ○ ○ ○ ○ ○ ○ ○ ○ ○

23 24 4:1:1 Born-Compressed • optimised for both image acquisition and 1 ● ○ ○ ○ ● ○ ○ ○ postproduction 1 ● ○ ○ ○ ● ○ ○ ○ Examples: CineForm RAW, ProRes RAW ● ○ ○ ○ ● ○ ○ ○ ● ○ ○ ○ ● ○ ○ ○

25 26

Illuminant • D50 • D55 • D65 • D75

27 28 29 30

0111010100101010100010110101011110 0100110101010101010100001011101010 0111010100101010100010110101011110 0001110101010101010100001011101010 0110101010010101010001011010101111 0010101010101010000101110101010000 0111010100101010100010110101011110 File Structure 0101010101010101000010111010100110 1001011101010010101010001011010101 1110010101010101010000101110101010 0111010100101010100010110101011110 0101010101010101001101010100000001 0010100010101010101001010101010101

31 32 File Structure Audio-Visual Container audio-visual container (wrapper) • MP4 • MXF audio MOV (.mkv) codec • • • AVI audio data video data

• Flash

33 34

Single Images Audio Codec • folder • MXF • WAVE • FLAC • TAR • Matroska (.mkv) • BWF • ZIP

• CinemaDNG • AAC • MP3

35 36 Video Codec (Master) Video Codec (Mezzanine) images streams • ProRes 422, ProRes 4444, ProRes RAW • TIFF • 8 bit raw • DNxHD, DNxHR • DPX • 10 bit raw • CineForm RAW • JPEG 2000 • • OpenEXR • FFV1 • DNG

37 38

Video Codec (Access) • H.264, H.265 (HEVC), AV1 Data is anything but «raw».

39 40 Audio Data Video Data • pcm_s16le • rgb48le • yuv444p16le • pcm_s24le • rgb24 • yuv422p10le • pcm_s32le • rgb72le • uyvy422 • yuv420p • yuv444p24le • bayer_bggr16le • bayer_bggr24le

41 42

What is inside my DPX? • log neg encoding • log RGB encoding or quasi-log encoding gamma encoding or power function • encoding File Formats • scene-linear encoding

43 44 Principles Different Purposes • The archive must be able to handle the file archive master format: formats it holds. ➔ for preservation mezzanine format: • open source ➔ for professional use in post-production simple to use and well documented dissemination formats: • ➔ for widely spreading and easy access • widely used by the community

45 46

Archive Master (Today)

Elena Rossi-Snook: film • folder, TIFF, 2K, RGB, 4:4:4, 16 bit Archiving without access • MXF, DPX, 2K, R′G′B′, 4:4:4, 10 bit isn’t preservation, video • AVI, «raw», HD, Y′CBCR, 4:2:2, 10 bit it’s hoarding. • Matroska, FFV1, HD, Y′CBCR, 4:2:2, 10 bit audio • BWF, 96 kHz, 24 bit • FLAC, 96 kHz, 24 bit

47 48 Mezzanine (Today) Dissemination (Today) video MP4 • ProRes 4444, 2K Video • DNxHR, 2K • H.264, SD, yuv420p, «lossy» • ProRes 422 HQ, HD H.264, HD, yuv420p, «lossy» DNxHD 175x, HD • • Sound audio • AAC, 44.1 kHz, 16 bit • BWF, 48 kHz, 24 bit AAC, 48 kHz, 16 bit • WAVE, 48 kHz, 24 bit •

49 50

Archive Master and Mezzanine Access film WebM (a subset of Matroska) • Matroska, FFV1, 2K, «RGB», 4:4:4, 16 bit Video video • «H.265», HD, yuv420p • Matroska, FFV1, HD, Y′CBCR, 4:2:2, 10 bit Sound audio «FLAC», 48 kHz, 16 bit • Matroska, FLAC, 96 kHz, 24 bit •

51 52 Reading

Reto Kromer: Matroska and FFV1: One for Film and Video Archiving?, in «Journal of Film Preservation», n. 96 (April 2017), FIAF, Brussels, Belgium, p. 41–45 Pros & Cons ➔ https://retokromer.ch/publications/ JFP_96.html

53 54 container: codec: avantages disavantages • folder • , DPX TAR TIFF TIFF • • DPX data easier • ZIP • JPEG 2000 DNG to process bigger files • MXF • FFV1 OpenEXR • Matroska • OpenEXR CineForm RAW FFV1 smaller files data complexer • JPEG 2000 to process • ProRes RAW

55 56 tristimulus transfer function colour diff. encode subsampling filter interpolation filter colour diff. decode transfer function tristimulus Transformations [M] [M] [M] [M] ′ ′ B B ′ ′ XYZ RGB XYZ G RGB G 4:4:4 ′ 4:2:2 ′ 4:4:4 R R R R R C C C B B B C C C ′ ′ ′ Y Y Y

57 58

Data Transformations Audio Exemple

MOV demultiplex demultiplex WAVE decode decode pcm_s24le filter filter pcm_s16le encode encode AAC multiplex multiplex MP4

59 60 Video Exemple Audio-Visual Exemple

MOV MOV demultiplex demultiplex ProRes 422 HQ WAVE decode decode ProRes 422 HQ yuv422p10le pcm_s24le filter yuv422p10le filter yuv420p pcm_s16le H.264 encode AAC encode yuv420p multiplex H.264 multiplex MP4 MP4

61 62

Acknowledgements AV Preservation by reto.ch

Swiss Federal Institute of Technology chemin du Suchet 5 • 1024 Écublens • Massachusetts Institute of Technology Switzerland • Kinemathek Lichtspiel, Bern Web: reto.ch Twitter: @retoch • Charles Poynton Email: [email protected] • Dave Rice & Misty De Meo • Agathe Jarczyk & David Pfluger

63 64