Complexity Scalable MPEG Encoding Stephan Mietens CIP-DATA LIBRARY TECHNISCHE UNIVERSITEIT EINDHOVEN Mietens, Stephan O. Complexity scalable MPEG encoding / by Stephan O. Mietens. - Eindhoven : Technische Universiteit Eindhoven, 2004. Proefschrift. - ISBN 90-386-2040-3 NUR 992 Trefw.: complexiteit van berekening / beeldcodering / codering. Subject headings: computational complexity / video coding / encoding. Schlagwörter: Skalierbarkeit / MPEG / Videokompression / Encoding. ­c Copyright 2004 Stephan Mietens All rights are reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior written permission from the copyright owner. Complexity Scalable MPEG Encoding PROEFSCHRIFT ter verkrijging van de graad van doctor aan de Technische Universiteit Eindhoven, op gezag van de Rector Magnificus, prof.dr. R.A. van Santen, voor een commissie aangewezen door het College voor Promoties in het openbaar te verdedigen op woensdag 18 februari 2004 om 16.00 uur door Stephan Oliver Mietens geboren te Frankfurt am Main, Duitsland Dit proefschrift is goedgekeurd door de promotoren: prof.dr.ir. P.H.N. de With en prof.dr.ir. R.L. Lagendijk Copromotor: Prof. Dr.-Ing. habil. C. Hentschel ISBN 90-386-2040-3 to my family Contents 1 Introduction 1 1.1 Problem statement ....................... 1 1.1.1 Context . ....................... 1 1.1.2 Terms and system requirements ............ 3 1.1.3 Approaches towards MPEG complexity scalability . 5 1.1.4 Structure of the conducted research .......... 6 1.2 Programmable hardware . ................. 8 1.2.1 Emerging flexible architectures ............ 8 1.2.2 Signal flow-graph programming ............ 10 1.2.3 Dynamic multi-window TV . ............ 12 1.3 Scalable system design ..................... 13 1.3.1 Motivation for embedded applications . ...... 13 1.3.2 Scalable systems using scalable video algorithms . 15 1.3.3 Overall system architecture with scalability ...... 16 1.4 MPEG-2 video compression standard . ............ 19 1.4.1 Basics of MPEG video coding . ............ 21 1.4.2 Processing of MPEG video coding ........... 22 1.4.3 MPEG Scalability . ................. 28 1.5 Outline of the thesis ....................... 30 1.6 Background of the chapters . ................. 33 2 Discrete cosine transformation 37 2.1 Introduction ........................... 37 2.2 Mathematical background . ................. 38 2.2.1 Forward Transformation ................ 38 2.2.2 Inverse Transformation ................. 39 2.2.3 Periodic properties of the cosine function . ...... 40 i ii Contents 2.3 Examples of DCT algorithms . ................. 40 2.3.1 Lee-Huang recursive algorithm ............ 41 2.3.2 Cho-Lee 2-D algorithm ................. 41 2.3.3 Arai-Agui-Nakajima 1-D algorithm .......... 42 2.3.4 Alternative DCT algorithms . ............ 43 2.4 Complexity comparison of DCT algorithms .......... 44 2.5 Accuracy-reduced DCT ..................... 47 2.5.1 Merhav-Vasudev multiplication-free algorithm . 47 2.5.2 Pao-Sun adaptive DCT modeling ........... 48 2.5.3 Lengwehasatit-Ortega var.-complexity algorithm . 48 2.6 Discussion ............................ 49 3 Complexity scalable DCT computation 51 3.1 Introduction ........................... 51 3.2 DCT-algorithm analysis ..................... 53 3.2.1 Concept for the new technique . ............ 53 3.2.2 Trade-off criterion for computations and quality . 54 3.2.3 Priority weighting . ................. 54 3.3 Implementation aspects of a fast-DCT algorithm analysis . 55 3.3.1 Database construction ................. 55 3.3.2 Construction of a database for analysis results ..... 57 3.3.3 Algorithmic example . ................. 59 3.4 Enhancements using priority weighting ............ 60 3.5 Experimental results ...................... 61 3.5.1 Computational results of the experiments . ...... 61 3.5.2 Pictorial results of the experiments ........... 63 3.6 Discussions and conclusions . ................. 66 4 Motion estimation 71 4.1 Introduction ........................... 71 4.2 Block-matching criteria ..................... 72 4.3 Fast ME algorithms ....................... 74 4.3.1 Full Search (2DFS) . ................. 74 4.3.2 One Dimensional Full Search (1DFS) . ...... 74 4.3.3 Block-Based Gradient Descent Search (BBGDS) . 76 4.3.4 Diamond Search . ................. 76 4.3.5 Three Step Search (TSS) ................ 77 4.3.6 New Three Step Search (NTSS) ............ 77 4.3.7 Simple Recursive Motion Estimation (simple RME) . 77 4.4 Motion-vector refinement . ................. 80 Contents iii 4.5 Comparison of ME algorithms ................. 81 4.5.1 Picture quality of motion-compensated frames . 81 4.5.2 Computational complexity comparison . ...... 83 4.5.3 Complexity vs. picture quality . ............ 84 4.6 Conclusion ........................... 85 5 Scalable motion estimation 87 5.1 Introduction ........................... 87 5.1.1 A view on the ME process . ............ 87 5.1.2 State-of-the-art ME . ................. 88 5.1.3 Contribution of this chapter to scalable ME ...... 90 5.2 Notations ............................ 91 5.3 SMART, a structural-level technique . ............ 92 5.3.1 Algorithm . ....................... 92 5.3.2 Modifications in the MPEG encoder architecture . 94 5.3.3 Scalability aspects . ................. 96 5.3.4 Experimental verification ................ 99 5.4 Block classification supporting the DCT and ME . ......103 5.4.1 Algorithm . .......................104 5.4.2 Experimental verification ................105 5.5 CARES, a vector-selection level technique ...........106 5.5.1 A concept for content-dependent ME . ......106 5.5.2 Algorithm . .......................108 5.5.3 Experimental verification ................109 5.6 Discussion and conclusions . .................111 6 System experiments and enhancements 113 6.1 Experimental environment . .................113 6.2 Inserting scalable DCT and related optimizations . ......115 6.2.1 Effect of proposed scalable DCT ............115 6.2.2 Selective DCT computation based on block classifica- tion ...........................118 6.2.3 Cyclical DCT for interframe coding ..........120 6.3 Combining SMART and CARES into one scalable ME system 123 6.4 Combined effect of scalable DCT and scalable ME ......128 6.4.1 Open-loop MPEG encoder . ............129 6.4.2 Closed-loop MPEG encoder . ............131 6.4.3 MPEG encoder with variable GOP and bit rate . 132 6.5 Discussion on further optimization of the scalable encoder . 135 6.6 Conclusions ...........................137 iv Contents 7 Conclusions 139 7.1 Conclusions of the thesis chapters . ............139 7.2 Memory-resource usage .....................143 7.3 Future work on MPEG scalability . ............143 7.3.1 System optimization . .................143 7.3.2 Bit-based processing . .................144 7.3.3 Modifying the input signal . ............144 7.3.4 Differentiating scalable DCT for intra- and intercoding 145 7.4 Applicability to other video applications ............145 7.4.1 MPEG-incompatible DCT coefficient coding .....145 7.4.2 Segmentation (MPEG-4) ................146 7.4.3 Multi-temporal ME (H.264) . ............146 References 149 A Derivation of computation order 159 B Computational complexity of ME algorithms 165 B.1 Number of block comparisons .................166 B.2 Computational complexity of a single block comparison . 167 B.3 Vector Refinement .......................167 B.4 Example configurations for motion estimation . ......168 B.5 Statistics of real computational effort . ............169 C Reduced processing resolution 171 C.1 Scalable DCT based on bit-slices ................171 C.1.1 Overview . .......................171 C.1.2 Architecture .......................172 C.1.3 Experimental results . .................175 C.2 Bit-based computation of ME . .................176 C.2.1 Simply reduced bit-resolution of input values .....176 C.2.2 One-Bit Motion Estimation (1BME) ..........176 D Test sequences 181 D.1 “Girl” . ............................181 D.2 “Voit” . ............................182 D.3“Teeny”.............................182 D.4 “Renata” . ............................183 D.5 “Stefan” . ............................183 D.6 “Foreman” ............................184 Contents v D.7 “Table tennis” . .......................184 Summary 187 Samenvatting 189 Zusammenfassung 193 Acknowledgments 197 Biography 199 CHAPTER 1 Introduction his chapter forms an introduction to the research that aims at realizing computational complexity scalability for MPEG Ìencoding. The chapter commences with the general problem statement of scalable MPEG processing: i.e. a scalable range of pic- ture quality with corresponding resource usage of the system. Re- source usage is expressed in terms of computation power, memory and communication bandwidth. Additional sections present an exemplary flexible and programmable hardware architecture showing the con- cept of scalable video processing, the principles of scalable system design, and a brief description of the MPEG coding standard, in or- der to deepen the insights of the problem statement. Furthermore, this chapter summarizes the subsequent individual chapters and presents their publication history. 1.1 Problem statement 1.1.1 Context The current burgeoning market for consumer digital video applications has lead to widely accepted consumer products based on MPEG coding, such as 2 Introduction DVD players, set-top boxes and digital video cameras. This thesis