Image and Video Compression Coding Theory Contents 1 JPEG 1 1.1 The JPEG standard .......................................... 1 1.2 Typical usage ............................................. 1 1.3 JPEG compression ........................................... 2 1.3.1 Lossless editing ........................................ 2 1.4 JPEG files ............................................... 3 1.4.1 JPEG filename extensions ................................... 3 1.4.2 Color profile .......................................... 3 1.5 Syntax and structure .......................................... 3 1.6 JPEG codec example ......................................... 4 1.6.1 Encoding ........................................... 4 1.6.2 Compression ratio and artifacts ................................ 8 1.6.3 Decoding ........................................... 10 1.6.4 Required precision ...................................... 11 1.7 Effects of JPEG compression ..................................... 11 1.7.1 Sample photographs ...................................... 11 1.8 Lossless further compression ..................................... 11 1.9 Derived formats for stereoscopic 3D ................................. 12 1.9.1 JPEG Stereoscopic ...................................... 12 1.9.2 JPEG Multi-Picture Format .................................. 12 1.10 Patent issues .............................................. 12 1.11 Implementations ............................................ 13 1.12 See also ................................................ 13 1.13 References ............................................... 14 1.14 External links ............................................. 15 2 Color space 16 2.1 Examples ............................................... 16 2.2 Conversion .............................................. 17 2.3 RGB density ............................................. 17 2.4 Lists .................................................. 17 2.4.1 Generic ............................................ 17 2.4.2 Commercial ......................................... 18 i ii CONTENTS 2.4.3 Special-purpose ....................................... 18 2.4.4 Obsolete ........................................... 18 2.5 Absolute color space ......................................... 18 2.5.1 Conversion .......................................... 19 2.5.2 Arbitrary spaces ........................................ 19 2.6 See also ................................................ 19 2.7 References ............................................... 19 2.8 External links ............................................. 20 3 Color vision 21 3.1 Wavelength and hue detection .................................... 21 3.2 Physiology of color perception .................................... 21 3.2.1 Theories ........................................... 22 3.2.2 Cone cells in the human eye ................................. 22 3.2.3 Color in the human brain ................................... 23 3.2.4 Subjectivity of color perception ............................... 24 3.2.5 In other animal species .................................... 24 3.3 Evolution ............................................... 25 3.4 Mathematics of color perception ................................... 26 3.5 Chromatic adaptation ......................................... 27 3.6 See also ................................................ 27 3.7 References .............................................. 27 3.8 External links ............................................. 29 4 YUV 30 4.1 History ................................................. 30 4.2 Conversion to/from RGB ....................................... 31 4.2.1 SDTV with BT.601 ...................................... 31 4.2.2 HDTV with BT.709 ...................................... 32 4.2.3 Notes ............................................. 32 4.3 Numerical approximations ....................................... 32 4.3.1 Studio swing for BT.601 ................................... 33 4.3.2 Full swing for BT.601 ..................................... 33 4.4 Luminance/chrominance systems in general .............................. 33 4.5 Relation with Y′CbCr ......................................... 34 4.6 Types of sampling ........................................... 34 4.7 Converting between Y′UV and RGB ................................. 34 4.7.1 Y′UV444 to RGB888 conversion ............................... 35 4.7.2 Y′UV422 to RGB888 conversion ............................... 35 4.7.3 Y′UV411 to RGB888 conversion ............................... 35 4.7.4 Y′UV420p (and Y′V12 or YV12) to RGB888 conversion .................. 36 4.7.5 Y′UV420sp (NV21) to RGB conversion (Android) ...................... 36 CONTENTS iii 4.8 References ............................................... 37 4.9 External links ............................................. 37 5 YCbCr 38 5.1 Rationale ............................................... 38 5.2 YCbCr ................................................. 38 5.2.1 ITU-R BT.601 conversion .................................. 40 5.2.2 ITU-R BT.709 conversion .................................. 40 5.2.3 ITU-R BT.2020 conversion ................................. 41 5.2.4 JPEG conversion ....................................... 41 5.3 CbCr Plane at Y = 0.5 ........................................ 41 5.4 References ............................................... 41 5.5 External links ............................................. 42 6 Chroma subsampling 43 6.1 Rationale ............................................... 43 6.2 How subsampling works ........................................ 43 6.3 Sampling systems and ratios ...................................... 44 6.4 Types of sampling and subsampling .................................. 44 6.4.1 4:4:4 ............................................. 44 6.4.2 4:2:2 ............................................. 44 6.4.3 4:2:1 ............................................. 44 6.4.4 4:1:1 ............................................. 44 6.4.5 4:2:0 ............................................. 45 6.4.6 4:1:0 ............................................. 46 6.4.7 3:1:1 ............................................. 46 6.5 Out-of-gamut colors .......................................... 46 6.6 Terminology .............................................. 46 6.7 History ................................................. 46 6.8 Effectiveness .............................................. 47 6.9 Compatibility issues .......................................... 47 6.10 See also ................................................ 47 6.11 References ............................................... 47 7 Discrete cosine transform 49 7.1 Applications .............................................. 49 7.1.1 JPEG ............................................. 49 7.2 Informal overview ........................................... 50 7.3 Formal definition ........................................... 51 7.3.1 DCT-I ............................................ 51 7.3.2 DCT-II ............................................ 51 7.3.3 DCT-III ........................................... 51 iv CONTENTS 7.3.4 DCT-IV ........................................... 51 7.3.5 DCT V-VIII ......................................... 51 7.4 Inverse transforms .......................................... 52 7.5 Multidimensional DCTs ....................................... 52 7.6 Computation ............................................. 53 7.7 Example of IDCT ........................................... 53 7.8 See also ................................................ 53 7.9 Notes ................................................. 54 7.10 Citations ................................................ 54 7.11 References ............................................... 54 7.12 Further reading ............................................ 55 7.13 External links ............................................. 55 8 H.264/MPEG-4 AVC 56 8.1 Naming ................................................ 56 8.2 History ................................................ 57 8.2.1 Versions ........................................... 57 8.3 Applications .............................................. 58 8.3.1 Derived formats ....................................... 59 8.4 Design ................................................ 59 8.4.1 Features ........................................... 59 8.4.2 Profiles ............................................ 61 8.4.3 Levels ............................................ 63 8.4.4 Decoded picture buffering .................................. 63 8.5 Implementations ........................................... 63 8.5.1 Software encoders ...................................... 64 8.5.2 Hardware ........................................... 64 8.6 Licensing ............................................... 64 8.7 See also ................................................ 65 8.8 References .............................................. 65 8.9 Further reading ............................................ 66 8.10 External links ............................................. 67 9 Group of pictures 68 9.1 Description .............................................. 68 9.2 GOP Structure ............................................ 68 9.3 References ............................................... 69 10 Video compression picture types 70 10.1 Summary ............................................... 70 10.2 Pictures/Frames ............................................ 70 10.3 Slices ................................................