HEVC OPTIMIZATION IN MOBILE ENVIRONMENTS by Ray Garcia A Dissertation Submitted to the Faculty of The College of Engineering and Computer Science in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy Florida Atlantic University Boca Raton, FL May 2014 Copyright by Ray Garcia 2014 ii ACKNOWLEDGEMENTS As we journey through life, individual achievements are rarely individual but a collection of help, encouragement, and support of a multitude of people involved directly or indirectly through the endeavor. My scholastic effort for the dissertation is no different. First and foremost I want to thank my wife for her patience and support. Without this the manuscript would not have been possible. In addition, the staff at Florida Atlantic University was invaluable in providing guidance and recommendations. I am very thankful to my advisor, Dr. Hari Kalva, my committee members Dr. Borko Furht, Dr. Imad, Mahgoub, Dr. Daniel Raviv and graduate department staff Jean Mangiaracina. I am truly grateful to all. iv ABSTRACT Author: Ray Garcia Title: HEVC Optimization in Mobile Environments Institution: Florida Atlantic University Dissertation Advisor: Dr. Hari Kalva Degree: Doctor of Philosophy Year: 2014 Recently, multimedia applications and their use have grown dramatically in popularity in strong part due to mobile device adoption by the consumer market. Applications, such as video conferencing, have gained popularity. These applications and others have a strong video component that uses the mobile device’s resources. These resources include processing time, network bandwidth, memory use, and battery life. The goal is to reduce the need of these resources by reducing the complexity of the coding process. Mobile devices offer unique characteristics that can be exploited for optimizing video codecs. The combination of small display size, video resolution, and human vision factors, such as acuity, allow encoder optimizations that will not (or minimally) impact subjective quality. The focus of this dissertation is optimizing video services in mobile environments. Industry has begun migrating from H.264 video coding to a more resource intensive but compression efficient High Efficiency Video Coding (HEVC). However, v there has been no proper evaluation and optimization of HEVC for mobile environments. Subjective quality evaluations were performed to assess relative quality between H.264 and HEVC. This will allow for better use of device resources and migration to new codecs where it is most useful. Complexity of HEVC is a significant barrier to adoption on mobile devices and complexity reduction methods are necessary. Optimal use of encoding options is needed to maximize quality and compression while minimizing encoding time. Methods for optimizing coding mode selection for HEVC were developed. Complexity of HEVC encoding can be further reduced by exploiting the mismatch between the resolution of the video, resolution of the mobile display, and the ability of the human eyes to acquire and process video under these conditions. The perceptual optimizations developed in this dissertation use the properties of spatial (visual acuity) and temporal information processing (motion perception) to reduce the complexity of HEVC encoding. A unique feature of the proposed methods is that they reduce encoding complexity and encoding time. The proposed HEVC encoder optimization methods reduced encoding time by 21.7% and bitrate by 13.4% with insignificant impact on subjective quality evaluations. These methods can easily be implemented today within HEVC. vi HEVC OPTIMIZATION IN MOBILE ENVIRONMENTS LIST OF TABLES .............................................................................................................. x LIST OF FIGURES .......................................................................................................... xii 1 INTRODUCTION ...................................................................................................... 1 1.1 Motivation ............................................................................................................ 2 1.2 Contribution ......................................................................................................... 3 1.3 Outline .................................................................................................................. 4 2 PROBLEM DESCRIPTION ....................................................................................... 5 2.1 H.264 vs. HEVC Subjective Evaluation .............................................................. 7 2.2 Decision Optimization .......................................................................................... 7 2.3 Complexity Reduction with HVS Factors ............................................................ 8 3 BACKGROUND ........................................................................................................ 9 3.1 Overview of Video Compression ......................................................................... 9 3.1.1 HEVC overview .......................................................................................... 14 3.2 Overview of HVS ............................................................................................... 19 3.2.1 Retina .......................................................................................................... 19 3.2.2 Smooth Pursuit Eye Movement .................................................................. 23 3.2.3 Transparent Motion Perception ................................................................... 24 vii 4 LITERATURE REVIEW ......................................................................................... 25 4.1 Subjective Quality .............................................................................................. 25 4.1.1 Metrics for Quality Evaluation ................................................................... 25 4.1.2 Subjective Evaluation ................................................................................. 27 4.2 Complexity Reduction ........................................................................................ 35 4.2.1 H.264 ........................................................................................................... 36 4.2.2 HEVC .......................................................................................................... 42 5 HEVC AND H.264 SUBJECTIVE EVALUATION ................................................ 58 5.1 Background ........................................................................................................ 58 5.2 Evaluation Methods ............................................................................................ 62 5.3 Experiments ........................................................................................................ 66 5.4 Results ................................................................................................................ 68 5.5 Discussion .......................................................................................................... 71 5.6 Concluding Remarks .......................................................................................... 78 6 HEVC DECISION OPTIMIZATION ...................................................................... 79 6.1 Background ........................................................................................................ 79 6.2 Method ............................................................................................................... 81 6.3 Prediction Modeling ........................................................................................... 83 6.4 Results ................................................................................................................ 84 6.4.1 Data Reading Primer ................................................................................... 84 viii 6.5 Data Analysis ..................................................................................................... 86 6.5.1 Prediction Model Analysis .......................................................................... 90 6.6 Application ......................................................................................................... 94 7 ADAPTING LOW BIT RATE SKIP MODE IN MOBILE ENVIRONMENT ....... 96 7.1 Background ........................................................................................................ 96 7.2 HEVC Elements ................................................................................................. 99 7.2.1 Quad-Tree ................................................................................................... 99 7.2.2 Skip Mode Method ................................................................................... 100 7.3 Proposed Method .............................................................................................. 102 7.4 Experiments ...................................................................................................... 105 7.5 Results .............................................................................................................. 110 7.6 Concluding Remarks ........................................................................................ 121 8 CONCLUSION ....................................................................................................... 122 9 FUTURE WORK ...................................................................................................