A STUDY OF MPEG-2 AND H.264 VIDEO CODING A Thesis Submitted to the Faculty of Purdue University by Michael Igarta In Partial Fulfillment of the Requirements for the Degree of Master of Science in Electrical and Computer Engineering December 2004 ii If we knew what it was we were doing, it would not be called research, would it? Albert Einstein iii ACKNOWLEDGMENTS I would like to thank my advisory committee for supporting me in this work. I would especially like to thank Professor Edward Delp for serving as my advisor throughout graduate school. After taking his video systems class, I grew a strong interest in the field of digital video. This led to my summer project involving video compression, which was followed by my involvement in a “secret” project involving digital video, which then led to this thesis. Thank you Professor Delp for giving me all these opportunities and a chance to work on something that I truly enjoy. Professor Delp, I can truly say that working in your lab has been a once-in-a-lifetime experience. I will never forget it. I would also like to thank everyone in the VIPER lab for their support. Yuxin, thank you for offering advice on my project as well as helping with my defense. And I cannot thank Eugene enough for everything, from his programming tips, advice on how to use Word, help with editing my thesis, and simply just being there as a friend. Last but not least I would like to thank my family for always supporting me. I would not be where I am today without the love from my family. My parents have always stood behind me no matter what I wanted to do. And especially to Abbie and Sea, thank you very much for always being there for me. You both are my number one inspiration for me to succeed. iv TABLE OF CONTENTS Page LIST OF TABLES ................................. vi LIST OF FIGURES ................................ viii ABSTRACT .................................... xiii 1 INTRODUCTION ............................... 1 1.1 The Digital Video Compression Problem ................ 1 1.2 Digital Image Capture .......................... 3 1.3 Basic Video Color Science ........................ 4 1.3.1 The Human Visual System .................... 4 1.3.2 Development of Color Television ................. 5 1.3.3 Color Space Conversions ..................... 5 1.3.4 Motion Compensated Video Compression Overview ...... 10 1.4 History of Previous Video Coding Standards .............. 17 1.4.1 H.261 ............................... 20 1.4.2 MPEG-1 .............................. 20 1.4.3 MPEG-2 .............................. 24 1.4.4 MPEG-4 Part 2: Visual ..................... 24 1.4.5 H.264 / MPEG-4 Part 10: Advanced Video Coding ...... 27 2 A COMPARISON OF MPEG-2 AND H.264 VIDEO CODING ............................ 28 2.1 Transform Video Coding ......................... 28 2.1.1 Transform Coding in MPEG-2 .................. 28 2.1.2 Undesirable Properties of the DCT ............... 32 2.1.3 Transform Coding in H.264 ................... 32 2.2 Motion Compensated Video Coding ................... 38 v Page 2.2.1 Motion Compensation in MPEG-2 ............... 38 2.2.2 Motion Compensation in H.264 ................. 43 2.2.3 In-Loop Deblocking Filter in H.264 ............... 50 2.3 Transmission of Compressed Video ................... 51 2.3.1 Transmission of MPEG-2 Compressed Video .......... 51 2.3.2 Transmission of H.264 Compressed Video ............ 55 2.3.3 Error Resilient Coding Tools in H.264 .............. 57 2.4 Profiles and Applications ......................... 59 2.4.1 MPEG-2 Profiles ......................... 59 2.4.2 H.264 Profiles ........................... 59 3 EXPERIMENTAL RESULTS ......................... 63 3.1 Testing Method .............................. 63 3.1.1 Software Used ........................... 63 3.1.2 Measuring Quality of Digital Video ............... 65 3.2 Testing Each Feature in H.264 ...................... 67 3.2.1 H.264 Test Configurations .................... 68 3.3 Experiment 1: Test of the Additional Macroblock Sizing ....... 70 3.4 Experiment 2: Test of the H.264 In-Loop Filter ............ 70 3.5 Experiment 3: Test of Context-Adaptive Binary Arithmetic Coding . 75 3.6 Experiment 4: Test Using Increased Number of Reference Pictures .. 75 3.7 Experiment 5: Test of Combination of Features of H.264 ....... 80 3.8 Experiment 6: MPEG-2 versus H.264 Video Compression ....... 80 4 CONCLUSIONS ................................ 92 4.1 Suggestions for Future Work ....................... 92 APPENDICES ................................... 94 Appendix A: Input Sequences ......................... 94 Appendix B: Full Experimental Results ....................104 LIST OF REFERENCES .............................162 vi LIST OF TABLES Table Page 1.1 Typical digital video formats with their uncompressed data rates assum- ing 12-bits/pixel. ............................... 2 1.2 Typical transmission/storage capacities. .................. 3 1.3 ITU-R BT.601 Digital Video Parameters. .................. 7 1.4 Variable length codes for motion vectors in MPEG-2. The zero motion vector is the most occurring, hence, it is assigned the shortest code. ... 18 2.1 Intracoding feature comparsion between MPEG-2 and H.264. ....... 33 2.2 Motion compensation comparsion between MPEG-2 and H.264. ..... 39 2.3 Scope of Video Coding Layer and Network Abstraction Layer ...... 55 2.4 Data partitions in H.264. ........................... 58 2.5 MPEG-2 Profiles. This chart shows supported chrominance sub-sampling, maximum resolution, maximum data rates, supported picture types, and picture rate. .................................. 60 2.6 H.264 defines three profiles designed to suit a wide variety of applications. 61 3.1 Overview of MPEG-2 configuration. ..................... 64 3.2 Overview of H.264 Configurations. ..................... 69 3.3 Each feature in H.264 contributes a small but measurable improvement. Larger numbers indicate better performance. ................ 70 3.4 Summary of PSNR improvement for H.264 video. Results were summa- rized according to resolution. VQEG input sequence results were sepa- rated for comparison purposes. ....................... 83 A.1 QCIF (176 × 144) resolution sequences. ................... 95 A.2 CIF (352 × 240) resolution sequences. .................... 95 A.3 CIF (352 × 288) resolution sequences. .................... 96 A.4 ITU-R 601 (720 × 480) resolution interlaced sequences. .......... 96 A.5 High Definition (1280 × 720) resolution sequences. ............. 96 vii Table Page A.6 VQEG PAL (720 × 576) resolution interlaced sequences. ......... 97 A.7 VQEG NTSC (720 × 480) resolution interlaced sequences. ........ 97 B.1 Full results for QCIF (176 × 144) sequences. ................148 B.2 Full results for QCIF (176 × 144) sequences. ................149 B.3 Full results for CIF (352 × 240) sequences. .................150 B.4 Full results for CIF (352 × 240) sequences. .................150 B.5 Full results for CIF (352 × 288) sequences. .................151 B.6 Full results for CIF (352 × 288) sequences. .................152 B.7 Full results for CIF (352 × 288) sequences. .................153 B.8 Full results for CIF (352 × 288) sequences. .................154 B.9 Full results for (704 × 480) sequences. ....................154 B.10 Full results for VCEG (720 × 480) sequences. ...............155 B.11 Full results for VCEG (720 × 480) sequences. ...............156 B.12 Full results for VCEG (720 × 480) sequences. ...............157 B.13 Full results for VCEG (720 × 480) sequences. ...............158 B.14 Full results for VCEG (720 × 480) sequences. ...............158 B.15 Full results for (720 × 480) sequences. ....................159 B.16 Full results for VCEG (720 × 576) sequences. ...............160 B.17 Full results for VCEG (720 × 576) sequences. ...............161 B.18 Full results for VCEG (720 × 576) sequences. ...............161 viii LIST OF FIGURES Figure Page 1.1 YCrCb sub-sampling formats. The 4:2:0 format is commonly used for digital video. ................................. 8 1.2 Interlaced video alternates the display of top and bottom fields. Only a single field is displayed at any given time but a complete picture is perceived by the viewer. ........................... 9 1.3 The spatial domain is the set of pixels within a single picture. The tem- poral domain involves multiple pictures of the video. ........... 10 1.4 Block diagram outlining steps from video encoding to video decoding. .. 11 1.5 A video sequence consists of a hierarchy of different units, with the sample being the smallest addressable unit. ..................... 12 1.6 A picture consists of slices. Each slice consists of macroblocks. Mac- roblocks are 16 × 16 pixels. ......................... 13 1.7 Two consecutive pictures from the ‘mobile’ sequence. The bottom image shows the difference image between the two images. Very little change is seen between the two consecutive pictures. ................. 15 1.8 Different types of predictive pictures. Arrows point from the reference picture to the prediction picture. ...................... 16 1.9 The scope of video coding standard only defines the decoder and not the encoder. .................................... 19 1.10 Timeline of video coding standards. MPEG-2/H.262 and MPEG-4: Part 10/H.264 were joint projects of MPEG and ISO. .............. 20 2.1 Quantization matrices for DCT coefficients for intracoded data (top) and intercoded data (bottom). .......................... 30 2.2 The normal zig-zag scanning order, shown on the left, is used for frame- coded pictures in MPEG-2.