Video Coding with Superimposed Motion-Compensated Signals Videocodierung mituberlagerten ¨ bewegungskompensierten Signalen Der Technischen Fakult¨at der Friedrich-Alexander-Universit¨at Erlangen-N¨urnberg zur Erlangung des Grades Doktor-Ingenieur vorgelegt von Markus Helmut Flierl Erlangen, 2003 Als Dissertation genehmigt von der Technischen Fakult¨at der Friedrich-Alexander-Universit¨at Erlangen-N¨urnberg Tag der Einreichung: 13.03.2003 Tag der Promotion: 27.06.2003 Dekan: Prof. Dr. rer. Nat. A. Winnacker Berichterstatter: Prof. Dr.-Ing. B. Girod Prof. Dr.-Ing. A. Kaup Acknowledgments This work required a substantial effort and I am very grateful to many people who made this endeavor possible. I would like to thank my supervisor Prof. Bernd Girod for the opportunity to join his group in Erlangen and Stanford as well as to benefit from the inspiring environment. I would also like to thank Prof. Andr´e Kaup for his interest in my work and reviewing my thesis as well as Prof. Heinrich Niemann and Prof. Wolfgang Koch for joining the committee. I am thankful to my fellow students Anne Aaron, Chuo-Ling Chang, Jacob Charareski, Joachim Eggers, Peter Eisert, Niko F¨arber, Sang-Eun Han, Frank Hartung, Mark Kalman, Yi Liang, Marcus Magnor, James Mammen, Prashant Ramanathan, Shantanu Rane, Marion Schabert, Eric Setton, Wolfgang S¨orgel, Eckehard Steinbach, Klaus Stuhlm¨uller, Jonathan Su, Matthias Teschner, Lutz Trautmann, Thomas Wiegand, and Rui Zhang for many stimulating discussions, joint work, and proofreading. In addition, I am thankful for helpful discussions with Dr. Nick Kingsbury. I am also grateful to the people at the Telecommunications Laboratory and Information Systems Laboratory, especially to Ursula Arnold and Kelly Yilmaz for their invaluable adminis- trative support. My special thanks belong to my family and friends for all the support they gave. Erkenne die Musik dessen vollendete T¨one verklingen. F¨uhle die Stille die zum Tr¨aumen verf¨uhrt. Erfahre den unendlichen Raum der hinschwindenden Ansammlung sich vollendender T¨one. — Deinem Odysseus vii Contents 1 Introduction 1 2 Background and Related Work 7 2.1CodingofVideoSignals............................ 7 2.2MotionCompensation............................. 8 2.2.1 BidirectionalMotionCompensation.................. 9 2.2.2 Overlapped Block Motion Compensation . 10 2.2.3 Variable Block Size Motion Compensation . 11 2.2.4 Multiframe Motion Compensation . 12 2.2.5 Superimposed Motion Compensation . 13 2.3MotionEstimation............................... 16 2.3.1 Rate-Constrained Motion Estimation . 16 2.3.2 Rate-Constrained Estimation of Superimposed Motion . 18 2.3.3 QuantizerSelectionattheResidualEncoder............. 20 2.3.4 EfficientMotionEstimation...................... 22 2.4 Theory of Motion-Compensated Prediction . 23 2.4.1 FrameSignalModel.......................... 24 2.4.2 Signal Model for Motion-Compensated Prediction . 25 2.4.3 Signal Model for Multihypothesis Prediction . 26 2.4.4 PerformanceMeasures......................... 28 2.4.5 Conclusions............................... 29 2.5 Three-Dimensional Subband Coding of Video . 29 2.5.1 Motion Compensation and Subband Coding . 30 2.5.2 Motion-CompensatedLiftedWavelets................. 31 viii Contents 3 Motion-Compensated Prediction with Complementary Hypotheses 33 3.1Introduction................................... 33 3.2 Extended Model for Superimposed Motion-Compensated Prediction . 34 3.2.1 Superimposed Prediction and Correlated Displacement Error . 34 3.2.2 ComplementaryHypotheses...................... 36 3.2.3 GradientofthePredictionErrorVariance.............. 41 3.3HypotheseswithAdditiveNoise........................ 42 3.3.1 AveragingFilter............................. 43 3.3.2 WienerFilter.............................. 45 3.4Forward-AdaptiveHypothesisSwitching................... 52 3.4.1 SignalModelforHypothesisSwitching................ 53 3.4.2 MinimizingtheRadialDisplacementError.............. 53 3.4.3 EquivalentPredictor.......................... 56 3.4.4 Motion Compensation with Complementary Hypotheses and For- ward-AdaptiveHypothesisSwitching................. 57 3.5PictureswithVaryingNumberofHypotheses................ 59 3.6Conclusions................................... 61 4 ITU-T Rec. H.263 and Superimposed Prediction 63 4.1Introduction................................... 63 4.2VideoCodingwithSuperimposedMotion................... 64 4.2.1 SyntaxExtensions........................... 64 4.2.2 CoderControl.............................. 64 4.3ExperimentalResults.............................. 66 4.3.1 Multiple Hypotheses for Constant Block Size . 66 4.3.2 Multiple Hypotheses for Variable Block Size . 67 4.3.3 Multiple Hypotheses and Multiple Reference Pictures . 69 4.4Conclusions................................... 76 5 ITU-T Rec. H.264 and Generalized B-Pictures 77 5.1Introduction................................... 77 5.2PredictionModesforB-Pictures........................ 78 5.2.1 Overview................................ 78 5.2.2 DirectMode............................... 79 5.2.3 SuperpositionMode.......................... 80 5.2.4 Rate-Distortion Performance of Individual Modes . 80 5.3SuperimposedPrediction............................ 83 5.3.1 Bidirectionalvs.SuperpositionMode................. 83 Contents ix 5.3.2 Two Combined Forward Prediction Signals . 84 5.3.3 EntropyCoding............................. 89 5.4EncoderIssues................................. 90 5.4.1 RateConstrainedModeDecision................... 90 5.4.2 Rate Constrained Motion Estimation . 91 5.4.3 Rate Constrained Superimposed Motion Estimation . 92 5.4.4 Improving Overall Rate-Distortion Performance . 93 5.5Conclusions................................... 97 6 Motion Compensation for Groups of Pictures 99 6.1Introduction................................... 99 6.2CodingScheme................................. 99 6.2.1 Motion-Compensated Lifted Haar Wavelet . 100 6.2.2 Motion-Compensated Lifted 5/3 Wavelet . 101 6.2.3 ExperimentalResults..........................101 6.3 A Mathematical Model of Motion-Compensated Subband Coding . 108 6.3.1 Motion-Compensated Lifted Haar Wavelet . 108 6.3.2 Motion-Compensated Lifted 5/3 Wavelet . 109 6.3.3 SignalModel..............................109 6.3.4 TransformCodingGain........................113 6.4Conclusions...................................115 7 Summary 117 A Mathematical Results 123 A.1 Singularities of the Displacement Error Covariance Matrix . 123 A.2AClassofMatricesandtheirEigenvalues..................123 A.3 Inverse of the Power Spectral Density Matrix . 124 A.4PowerSpectralDensityofaFrame......................126 Glossary 127 Bibliography 131 x Contents Inhaltsverzeichnis 1 1Einf¨uhrung 1 2 Grundlagen und Stand der Forschung 7 2.1CodierungvonVideosignalen......................... 7 2.2Bewegungskompensation............................ 8 2.2.1 BidirektionaleBewegungskompensation................ 9 2.2.2 Bewegungskompensation mituberlappenden ¨ Bl¨ocken........ 10 2.2.3 Bewegungskompensation mit Bl¨ocken variabler Gr¨oße........ 11 2.2.4 Bewegungskompensation mit mehreren Referenzbildern . 12 2.2.5 UberlagerteBewegungskompensation.................¨ 13 2.3 Bewegungssch¨atzung.............................. 16 2.3.1 Bewegungssch¨atzungmitRatennebenbedingung........... 16 2.3.2 Sch¨atzunguberlagerter ¨ Bewegung mit Ratennebenbedingung . 18 2.3.3 Wahl des Quantisierers beim Pr¨adiktionsfehler-Encoder....... 20 2.3.4 Effiziente Bewegungssch¨atzung..................... 22 2.4 Theorie der bewegungskompensierten Pr¨adiktion............... 23 2.4.1 Bildsignalmodell............................ 24 2.4.2 Signalmodell f¨ur bewegungskompensierte Pr¨adiktion......... 25 2.4.3 Signalmodell f¨ur Multihypothesen-Pr¨adiktion............ 26 2.4.4 Bewertungsmaße............................ 28 2.4.5 Schlussfolgerungen........................... 29 2.5Drei-DimensionaleTeilbandcodierungvonVideo............... 29 2.5.1 Bewegungskompensation und Teilbandcodierung . 30 2.5.2 BewegungskompensierteWavelets................... 31 3 Bewegungskompensierte Pr¨adiktion mit komplement¨aren Hypothesen 33 3.1Einleitung.................................... 33 3.2 Erweitertes Modell f¨ur dieuberlagerte ¨ bewegungskompensierte Pr¨adiktion . 34 3.2.1 Uberlagerte¨ Pr¨adiktion und korrelierte Versatzfehler . 34 3.2.2 Komplement¨areHypothesen...................... 36 3.2.3 Gradient der Pr¨adiktionsfehlervarianz................. 41 3.3HypothesenmitadditivemRauschen..................... 42 3.3.1 Mittelungsfilter............................. 43 3.3.2 WienerFilter.............................. 45 1This translation of the table of contents is a requirement according to the ”Pr¨ufungsordnung f¨ur die Technischen Fakult¨at der Friedrich-Alexander-Universit¨at Erlangen-N¨urnberg”. Contents xi 3.4AdaptiveHypothesenauswahl......................... 52 3.4.1 Signalmodell f¨urdieAuswahlvonHypothesen............ 53 3.4.2 Minimierung des radialen Versatzfehlers . 53 3.4.3 Aquivalenter¨ Pr¨adiktor......................... 56 3.4.4 Bewegungskompensation mit komplement¨aren und adaptiv aus- gew¨ahltenHypothesen......................... 57 3.5BildermitvariierenderAnzahlanHypothesen................ 59 3.6Schlussfolgerungen............................... 61 4 ITU-T Rec. H.263 unduberlagerte ¨ Pr¨adiktion 63 4.1Einleitung.................................... 63 4.2 Videocodierung mituberlagerterBewegung.................. ¨ 64 4.2.1 SyntaxErweiterung........................... 64 4.2.2 Algorithmen f¨urdenEncoder..................... 64 4.3ExperimentelleErgebnisse..........................