Resilient Video Coding for Wireless and Peer-To-Peer Networks
Total Page:16
File Type:pdf, Size:1020Kb
Resilient Video Coding for Wireless and Peer-to-peer Networks Resilient Video Coding for Wireless and Peer-to-peer Networks Proefschrift ter verkrijging van de graad van doctor aan de Technische Universiteit Delft, op gezag van de Rector Magnificus Prof. dr. ir. J. T. Fokkema, voorzitter van het College voor Promoties, in het openbaar te verdedigen op 2 oktober 2007 om 10.00 uur door Jacco Reynoud TAAL elektrotechnisch ingenieur geboren te Rotterdam. Dit proefschrift is goedgekeurd door de promotor: Prof. dr. ir. R. L. Lagendijk Samenstelling promotiecommissie: Rector Magnificus, voorzitter Prof. dr. ir. R. L. Lagendijk, Technische Universiteit Delft, promotor Prof. dr. ir. P. F. A. Van Mieghem Technische Universiteit Delft Prof. dr. ir. H. J. Sips Technische Universiteit Delft Prof. dr. ir. C. H. Slump Universiteit Twente Prof. dr. ir. P. H. N. de With Technische Universiteit Eindhoven Univ.-Prof. Dr.-Ing. Eckehard Steinbach Technische Universität München Dr. A. Reibman AT&T Research A This document is typeset with the LTEX ÑeÑÓiÖ typesetting system. The photo on the cover is taken in April 2006 in the pedestrians tunnel connecting the old and new part of Shanghai. This work was carried out in the ASCI graduate school. ASCI dissertation series number 155 Advanced School for Computing and Imaging ISBN: 978-90-9022238-7 Copyright c 2007 by J. R. Taal All rights reserved. No part of this material may be reproduced or transmitted in any form or by any means, electronic, mechanical, including photocopying, recording or by any information storage and retrieval system, without the prior permission of the copyright owner. Voor Jenneke Preface The work that resulted in this thesis was carried out at Delft University of Technology in three different projects. In each of these project video coding was part of the research, but every time the emphasis was put on a different viewpoint towards video coding was taken. The first project, Ubiquitous Communications (UBICOM), was totally funded by the Delft University of Technology. The UBICOM scenario was a low-power wearable augmented reality system where a virtual world could be projected onto the real world, using semi transparent virtual reality helmet. In the Context Aware User and Terminal (CACTUS) project a modern user, equipped with wearable devices was central. Devices, worn by the user or in- tegrated in rooms, buildings, cars, etc, will be able to communicate with each other in an ad hoc manner. To support his tasks, his personal device commu- nicates and negotiates on his behalf user with other devices such as screens, input devices and for instance vending machines. CACTUS, as sub project of the FREEBAND project, was funded by the Ministry of Economic Affairs in The Netherlands. In the I-SHARE project, we focused on sharing resources and data among devices, people and groups of people. In this scenario, Internet wireless net- works and peer-to-peer (P2P) networks play an important role to interconnect devices and to facilitate sharing. I-SHARE was also funded by the Ministry of Economic Affairs, as a sub project of the FREEBAND project. This thesis concentrates on the aspect of video compression in these projects. Each project had a different viewpoint on video compression. Not only the sce- nario in which compression was used differed, but also the chosen solutions to implement video compression. Although we concentrate on video com- pression, we approach the task of video compression from the environment in which it has to operate, namely a networked device, from which different constraints are imposed on the video coder. J. R. Taal, Delft, March 2007. i Acknowledgements Although doing a Ph.D. track was a decision of my own, I know many people have supported me in pursuing it. First I would like to thank my promotor Inald Lagendijk for his support and patience with me. I always enjoyed the deep discussions we had. Thanks for giving me this opportunity, working with you is a privilege. I thank my roommates, I have had many in the projects: (in order of ap- pearance) Maarten Ditzel and Hylke van Dijk, both for taking care of me as a fresh student on the 16th floor. Hylke for the many interesting and abstract discussions we had about our work. Maarten, thanks for the really nice time, the volleyball games and our friendship. Jan, thanks for the fun we had on the 9th floor; too bad you could not finish your thesis. Ronald, we share a sense of humour. It was really nice to have you in the same room on the 10th floor for these years. Your have an interesting research topic and wish you luck in writ- ing your thesis, which you probably do with ease. Jenneke, the first woman in the same office, it was a privilege. Your enthusiasm has always brought cheer in our room. The people that I perhaps owe most, were not my roommates. Ivaylo, thank you. For the work we done together and the many papers we wrote together. I will never forget our first trip to China. Johan, your enthusiasm worked on me. We had and still have great ideas, some of them are now being materialized. The Information and Communication Theory Group has always been inspir- ing and competitive, but also very lively and fun. I thank all my colleagues in the group. This is the best group of all. I like to thank Henk Sips and Dick Epema for your support and cooperation during the projects. During my journey up and down the building, I have met and worked with many people. I would like to thank all that I have missed here. I like to thank Prof. He of Tsinghua University, Beijing, China, for hosting me and Ivaylo in 2003. It has been one of the most interesting experiences I had. I like to thank Chen Zhibo for the really nice cooperation on our paper. I wish you all good luck in your career. My best friends, Maurice and Michiel. The holidays and weekends we spent iii ACKNOWLEDGEMENTS together are unforgettable and a big part of my life. They helped me through the process. I hope our lives will stay entangled forever. I thank Rob Becqué, who sadly could not finish his thesis, for his inspira- tion. Finally, I thank my family for their love and support. Roy and Cindy, I am very proud of the life you have build up together with your children Jiska and Minke. Mom, I suspect you have always known that I could do this. You always supported me in whatever step I took. You are the best mom in the world. iv Contents Preface i Acknowledgements iii Contents v List of Tables ix List of Figures x List of Abbreviations xiii 1 Introduction 1 1.1 Advances in Video Compression . 2 1.2 ProblemDescription ....................... 4 1.3 OrganizationandScope . .. .. .. .. .. .. 5 1.4 ContributionsoftheThesis . 8 I Network Aware Video Coding 9 2 Video Streaming Model 11 2.1 Introduction ........................... 11 2.2 The Video Streaming Model . 13 2.2.1 Overview......................... 13 2.2.2 Application........................ 15 2.2.3 VideoCoder ....................... 15 2.2.4 Network Adaptation Layer . 17 2.3 Scenario.............................. 18 2.4 Discussion ............................ 19 3 Network Aware Video Coding Strategies 21 3.1 Introduction ........................... 21 v CONTENTS 3.2 Real-time adaptive video coding . 22 3.2.1 Introduction ....................... 22 3.2.2 Video Coder Behavior Model . 23 3.2.3 PredictionGain.. .. .. .. .. .. .. 26 3.2.4 Frame prediction and Frame skipping . 27 3.2.5 MotionCompensation. 29 3.2.6 Joint effect of frame skip and motion compensation . 31 3.2.7 IntraBlockRefresh. 32 3.2.8 Quantization and Variable Length Coding . 33 3.2.9 End-to-end quality metric . 35 3.2.10 Resourceusage. 40 3.2.11 Parameter Estimation . 42 3.3 Scalable, Layered and Progressive Coding . 42 3.3.1 Introduction ....................... 42 3.3.2 Differential Layered Coding . 44 3.3.3 Progressive Coding: Bit-plane Coding . 45 3.4 Multiple Description Coding . 46 3.4.1 Odd-Frame / Even-Frame Streaming . 48 3.4.2 Theory and special cases of MDC . 49 3.4.3 MD Nested Quantization . 53 3.4.4 Multiple Description Correlating Transform . 55 3.4.5 multiple description (MD) Channel Codes . 57 3.4.6 MD-FEC.......................... 57 3.5 Discussion ............................ 59 II Streaming Scenarios 61 4 End-to-end optimization of mobile video streaming using QoS 65 4.1 Introduction ........................... 65 4.2 QoSnegotiation:ARC. .. .. .. .. .. .. .. 67 4.3 Mobile Video Streaming Implementation . 69 4.3.1 VideoEncoder ...................... 70 4.3.2 Protocols......................... 71 4.4 Experimental Evaluation . 71 4.5 Discussion ............................ 74 5 Error Resilient Video Compression using Behavior Models 75 5.1 Introduction ........................... 75 5.2 ProblemFormulation. 77 5.3 Source Encoder Optimization Criterion . 80 5.4 RateDistortionModel . 82 5.4.1 Rate Distortion Behavior Model of Intra-frames . 83 vi 5.4.2 Rate Distortion Behavior Model of Inter-frames . 84 5.4.3 Channel Induced Distortion Behavior Model . 88 5.5 ModelValidation ......................... 93 5.5.1 Optimal Rate allocation . 94 5.5.2 Rate Allocation for a Channel With Residual Errors . 95 5.5.3 Selecting the Optimal GOP Length . 97 5.5.4 Optimal Rate Allocation for Whole Sequences . 99 5.6 Discussion ............................ 101 6 Optimized Video-Streaming over 802.11 by Cross-Layer Signaling 103 6.1 Introduction ........................... 103 6.2 Communication system architecture . 104 6.3 Link adaptation basics . 106 6.4 AutomaticMACratecontrol . 108 6.4.1 Statistics-based automatic rate control . 108 6.4.2 SNR-based automatic rate control . 109 6.4.3 Hybrid automatic rate control . 110 6.5 Cross-layersignaling .