Tessa Bosschem Low-Complexity Techniques for 2D-To-3D Conversion

Low-complexity techniques for 2D-to-3D conversion Tessa Bosschem Promotor: prof. dr. ir. Rik Van de Walle Begeleiders: ir. Sebastiaan Van Leuven, ir. Glenn Van Wallendael Masterproef ingediend tot het behalen van de academische graad van Master in de ingenieurswetenschappen: computerwetenschappen Vakgroep Elektronica en Informatiesystemen Voorzitter: prof. dr. ir. Jan Van Campenhout Faculteit Ingenieurswetenschappen en Architectuur Academiejaar 2010-2011 Low-complexity techniques for 2D-to-3D conversion Tessa Bosschem Promotor: prof. dr. ir. Rik Van de Walle Begeleiders: ir. Sebastiaan Van Leuven, ir. Glenn Van Wallendael Masterproef ingediend tot het behalen van de academische graad van Master in de ingenieurswetenschappen: computerwetenschappen Vakgroep Elektronica en Informatiesystemen Voorzitter: prof. dr. ir. Jan Van Campenhout Faculteit Ingenieurswetenschappen en Architectuur Academiejaar 2010-2011 Acknowledgments During the realization of this thesis I have been accompanied and helped by many people. It is now a great pleasure to have the opportunity to thank them. First of all, I would like to show my gratitude to my promoter Rik Van de Walle and my supervisors Sebastiaan Van Leuven, Glenn Van Wallendael and Jan De Cock. Their encouragement, guidance and enthusiasm enabled me to develop an understanding of the subject. Without their help and good advice this dissertation would not have been possible. I also owe my gratitude to the people of the Vlaamse Radio- en Televisieomroep (VRT), for providing me the necessary material in order for my thesis to succeed. I wish to thank my friends, who supported me during the dicult times and provided emotional support whenever necessary. Last but not least, it is an honor for me to thank my family, my parents and my sister, for helping me in every possible way and for supporting me from the beginning until the end. I want to thank my parents for giving me the opportunity to undertake my studies and believing in me. i Toelating tot bruikleen "De auteur geeft de toelating deze masterproef voor consultatie beschikbaar te stellen en delen van de masterproef te kopiëren voor persoonlijk gebruik. Elk ander gebruik valt onder de beperkingen van het auteursrecht, in het bijzonder met betrekking tot de verplichting de bron uitdrukkelijk te vermelden bij het aanhalen van resultaten uit deze masterproef." "The author gives permission to make this master dissertation available for consultation and to copy parts of this master dissertation for personal use. In the case of any other use, the limitations of the copyright have to be respected, in particular with regard to the obligation to state expressly the source when quoting results from this master dissertation." Tessa Bosschem, juni 2011 ii Low-complexity techniques for 2D-to-3D conversion Tessa Bosschem A thesis submitted in partial fulllment of the requirements for the degree of Master in Engineering Science: Computer Science Year 2010 - 2011 Promoter: prof. dr. ir. Rik Van de Walle Supervisors: ir. Sebastiaan Van Leuven, ir. Glenn Van Wallendael Faculty of Engineering and Architecture Ghent University Summary The upswing of 3D television has resulted in the need for the ecient creation of 3D video material. In addition to methods to retrieve 3D information directly from a scene, there is an increasing need for well-working 2D-to-3D conversion techniques. This dissertation aims at developing a new 2D-to-3D conversion technique, specically designed to meet the needs of a chroma keying application. Therefore, in this work a new algorithm was proposed, exploiting these characteristics and providing a straightforward technique to create a 3D experience from 2D video streams. A comparison was made between distinct 2D-to-3D techniques and visual comfort user assessments were executed to obtain subjective results. Based on these results, conclusions were drawn about the perceived depth, the preferred parallax eect and the technique that produces the most enjoyable 3D eect. The most signicant result was the overall preference of the observers for 3D videos generated with the proposed depth extraction method, evaluated for the use case of a weather forecast. Keywords: 2D-to-3D conversion, visual comfort, stereoscopy, chroma key iii Low-complexity techniques for 2D-to-3D conversion Tessa Bosschem Supervisor(s): ir. Sebastiaan Van Leuven, ir. Glenn Van Wallendael, prof. dr. ir. Rik Van de Walle Abstract—This article describes the development of a new 2D-to-3D con- III. 2D-TO-3D CONVERSION TECHNIQUES version algorithm for the conversion of existing 2D video material. A low- complexity technique is developed and applied to a chroma keying appli- In this section, different 2D-to-3D conversion techniques will cation to illustrate its usefulness. Additional functionalities, such as the be discussed. The first subsection describes state of the art extraction of objects with a more pronounced 3D effect and the possibility techniques while the following subsections propose the two de- to choose between 3D “behind” and “in front of” the display screen, are available. To be able to evaluate the new system, subjective experiments signed algorithms. Each algorithm was designed to retrieve regarding visual discomfort were executed. The performance of distinct depth information from a 2D video stream and generate left and 2D-to-3D conversion methods was investigated, with an additional explo- right view sequences. These left eye and right eye views can be ration of the effect of increasing depth levels, parallax effect and amount of 3D animations. The experiments show that, compared to state of the art im- offered to a stereoscopic player which redirects them to the cor- plementations, the newly designed algorithm is found to generate the most rect eye. The brain does the remaining work by creating a per- pleasing results in case of a chroma keying situation. This demonstrates the ception of depth. These techniques can be applied to the weather applicability of our technique. forecast situation and the results are used for testing purposes. Keywords— 2D-to-3D conversion, visual comfort, stereoscopy, chroma key, background subtraction A. State of the art techniques Investigating state of the art 2D-to-3D conversion methods I. INTRODUCTION enabled the making of a well-founded comparison between the HREE -dimensional television is nowadays often seen as a current techniques and the newly designed algorithms. Two dis- Tmajor milestone in the ultimate visual experience of mul- tinct state of the art methods are considered during the eval- timedia. The concept of stereoscopy1 has been around for ages uation of the algorithms. The first method, the open source but only recently a real breakthrough from conventional 2D tele- MakeMe3D software[4], uses self-developed object recognition vision to 3D television is expected to come soon. The mass pro- and motion analysis to extract depth information. The second duction of 3D video material is essential for 3D television to method employs the input video stream as first view and gener- become a succes and penetrate the living rooms. One possible ates the second view by introducing a frame difference. A frame approach for generating 3D content from existing 2D content are is shown to the left eye, while the previous frame is shown to the 2D-to-3D conversion techniques. These techniques pose several right eye. Both algorithms accept one merged video stream as difficulties such as disocclusion problems and the need for flaw- input for their algorithm. less extraction of depth information. In this work, the goal is to avoid these issues by exploiting the characteristics of chroma B. Video plus depth keying applications for the conversion to three-dimensional ma- After the classification made by the background subtraction terial. method (described in paragraph II), it is possible to immedi- ately generate depth maps corresponding to each frame of the II. BACKGROUND SUBTRACTION METHOD merged video stream. The depth maps are represented by YUV sequences, only containing non-zero values for the luminance In this research, the main focus lies on the conversion of two- component. Left and right view are then generated using a dimensional video used in chroma keying applications. Such Depth Image Based Rendering (DIBR) toolbox[3]. This ap- an application allows to employ two video streams as input for proach will result in disoccluded regions in the generated views, the conversion algorithm: the video with the foreground object which are gaps of missing pixel information. The depth maps recorded in front of the chroma key screen and an additional are interpreted according to the MPEG N8038 informative rec- background video stream. In a first phase, a background subtrac- ommendations and disocclusions are filled by background pixel tion algorithm is applied to separate foreground objects from the extrapolation. chroma key screen. Afterwards, pixels classified as background can be substituted by the corresponding pixels in the background C. Proposed generation of left and right view video stream. This will generate a merged media file, the video as seen on television (in applications as e.g. weather forecasts The classification, as made by the background subtraction or news broadcasting). For this article we evaluate the use case method, remains the same but left and right view are generated of a weather forecast situation. earlier in the process, during the classification of foreground and background pixels. The possibility to assign depth before the 1Stereoscopy refers to the technique used to obtain a 3D effect by presenting video streams are merged together is one of the major advan- two images separately to the left and right eye of the viewer. tages of the proposed method. Fig. 1. Left and right frame generated using the new algorithm, with resulting stereo image. A cut-and-paste technique is applied to obtain the desired depth effect and to assign similar depth levels to the same pixels as in previous method. However, in this case, there is no need to Fig.

Load more