Faculdade de Engenharia da Universidade do Porto Platforms for Handling and Development of Audiovisual Data José Pedro Sousa Horta Project/Dissertation concerning Mestrado Integrado em Engenharia Informática Advisor: Professor Eurico Carrapatoso July 2008 © José Pedro Sousa Horta, 2008 Platforms for Handling and Development of Audiovisual Data José Pedro Sousa Horta Project/Dissertation concerning Mestrado Integrado em Engenharia Informática Approved in public display by the jury: President: Prof. Doutor António Fernando Coelho _________________________________________________ Examiner: Prof. Doutor Nuno Magalhães Ribeiro Vowel: Prof. Doutor Eurico Manuel Carrapatoso July 17th 2008 Abstract Everywhere around us, digital is replacing analogue. Such is especially true in the audiovisual: be it in consumer or professional market, the advantages of computer-based media have quickly persuaded investors. To choose in which technologies, software should be based, proper understanding of the major aspects behind digital media is essential. An overview of the state of the art regarding the compression of digital video, the existing container formats and the multimedia frameworks (that enable easier playback and editing of audiovisual content) will be given, preceded by an introduction of the main topics. The professional video market is particularly focused on this document, due to the context of its elaboration, predominantly MPEG-2, MXF and DirectShow. In order to obtain more accurate results, fruit of the practical contact with the technology, a hands-on programming project was accomplished using DirectShow to playback both local and remote WMV files. Compression algorithms are continuously being improved and new ones invented, the designers of container formats are paying increased attention to metadata and some serious alternatives to DirectShow are emerging. In a continuously mutating environment, light is shed at what has, what is and what will be happening in the upcoming years on the digital video scene. Results show that the choice of a particular video coding format or multimedia framework depends heavily on the purpose of its application and that there is no universal solution. Various use scenarios are given an answer as to which is the best course of action concerning which technologies to apply. The software developed was ultimately given a positive feedback by the client. Keywords: Digital video compression, Digital video containers, Multimedia frameworks; DV, Motion JPEG 2000, MPEG 2 / 4 / 21, ASF, MXF, DirectShow, Helix DNA, QuickTime i ii Resumo Um pouco por todo o lado, o formato digital está a substituir o analógico. Este facto torna-se realmente óbvio no mundo do audiovisual: seja no mercado profissional ou de consumo, as vantagens da media baseada na informática rapidamente convenceram os investidores. De modo a facilitar a escolha de quais tecnologias digitais que devem ser empregues, um conhecimento suficiente dos principais aspectos por detrás do tratamento de media numérica é essencial. Uma vista geral do estado da arte dos formatos de compressão e transporte de vídeo digital e das plataformas multimédia (que permitem o fácil manuseio e edição de conteúdo audiovisual) será efectuada, precedida por uma introdução aos tópicos principais. O mercado de vídeo profissional é particularmente tido em conta neste documento, devido ao contexto da sua elaboração, predominantemente as tecnologias MPEG-2, MXF e DirectShow. De modo a obter resultados mais pertinentes, foi concluído um projecto prático baseado em DirectShow com o intuito de reproduzir ficheiros WMV, locais e remotos. Os algoritmos de compressão estão continuamente a ser melhorados e inventados, os formatos de armazenamento têm cada vez recursos dedicados a metadata e sérias alternativas ao DirectShow começam a emergir. Num ambiente em contínua mutação, é levantado o véu sobre o que aconteceu, o que está a acontecer e o que irá acontecer nos próximos anos, na área do vídeo digital. Os resultados obtidos indicam que a escolha de um formato de ficheiro de vídeo particular ou de uma framework multimédia depende imenso da utilização pretendida, não existindo soluções universais. Vários cenários são considerados e várias soluções recomendadas acerca das melhores opções tecnológicas a adoptar. A aplicação implementada foi avaliada positivamente pelo cliente. iii iv Acknowledgements I would like to thank all my colleagues at MOG Solutions for an outstanding work environment that considerably eased the flow of work. My special thanks to my MOG Solutions responsible, Vitor Teixeira, for letting me find my own pace and for his valuable knowledge, to Luis Barral for his priceless expertise on DirectShow programming and his will to share it, to Orlando Ribeiro for his constant availability to help and to Pedro Campos for keeping me motivated along these four and a half months. Big thanks to my faculty advisor Prof. Eurico Carrapatoso for kindly embracing this project. And a final thank you to my brother Frederico, for providing me with his layman’s insight when reviewing this document. v vi Table of Contents 1. ITRODUCTIO ........................................................................................................................................ 1 1.1 SCOPE ............................................................................................................................................................ 1 1.2 MOTIVATION .................................................................................................................................................. 2 1.3 OBJECTIVES ................................................................................................................................................... 3 1.4 STRUCTURE .................................................................................................................................................... 4 1.5 DIGITAL VIDEO .............................................................................................................................................. 5 2. DIGITAL VIDEO COMPRESSIO .......................................................................................................... 7 2.1 DIGITAL DATA COMPRESSION ......................................................................................................................... 8 2.1.1 Lossless Compression ............................................................................................................................ 8 2.1.1.1 Run-Length Encoding ..................................................................................................................................... 8 2.1.1.2 Entropy Encoding ........................................................................................................................................... 9 Huffman Coding .................................................................................................................................................. 10 Arithmetic Coding ............................................................................................................................................... 11 2.1.2 Lossy Compression .............................................................................................................................. 13 2.1.2.1 Markov Sources ............................................................................................................................................ 13 2.1.3 Transform Coding................................................................................................................................ 14 2.1.3.1 Discrete Cosine Transform ........................................................................................................................... 14 2.1.3.2 Wavelet Transform ........................................................................................................................................ 16 2.2 DIGITAL IMAGE COMPRESSION ..................................................................................................................... 18 2.2.1 Sampling .............................................................................................................................................. 18 2.2.2 Quantization ........................................................................................................................................ 19 2.2.3 Predicting Image Values ...................................................................................................................... 21 2.2.4 Chroma Subsampling .......................................................................................................................... 21 2.3 DIGITAL VIDEO COMPRESSION ..................................................................................................................... 24 2.3.1 Motion Estimation and Compensation ................................................................................................ 24 2.3.1.1 Frame Types .................................................................................................................................................. 25 Intra Frames......................................................................................................................................................... 25 Predicted Frames ................................................................................................................................................