Capturing and Post-Processing of Stereoscopic 3D Content for Improved Quality of Experience
Total Page:16
File Type:pdf, Size:1020Kb
Capturing and Post-Processing of Stereoscopic 3D Content for Improved Quality of Experience by Di Xu B.Sc., Beijing Normal University, 2003 M.A.Sc., University of Victoria, 2007 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF Doctor of Philosophy in THE FACULTY OF GRADUATE STUDIES (Electrical & Computer Engineering) THE UNIVERSITY OF BRITISH COLUMBIA (Vancouver) February 2013 © Di Xu, 2013 Abstract 3D video can offer real-life viewing experience by providing depth impression. 3D technology has not yet been widely adopted due to challenging 3D-related issues, ranging from capturing to post-processing and display. At the capturing side, lack of guidelines may lead to artifacts that cause viewers headaches and nausea. At the display side, not having 3D content customized to a certain aspect ratio, display size, or display technology may result in reduced quality of experience. Combining 3D with high- dynamic-range imaging technology adds exciting features towards real-life experience, whereas conventional low-dynamic-range content often suffers from color saturation distortion when shown on high-dynamic-range displays. This thesis addresses three important issues on capturing and post-processing 3D content to achieve improved quality of experience. First, we provide guidelines for capturing and displaying 3D content. We build a 3D image and video database with the content captured at various distances from the camera lenses and under different lighting conditions. We conduct comprehensive subjective tests on 3D displays of different sizes to determine the influence of these parameters to the quality of 3D images and videos before and after horizontal parallax adjustment. Next, we propose a novel and complete pipeline for automatic content-aware 3D video reframing. We develop a bottom-up 3D visual attention model that identifies the prominent regions in a 3D video frame. We further provide a dynamic bounding box that crops the video and avoids annoying problems, such as jittering and window violation. Experimental results show that our algorithm is both effective and robust. ii Finally, we propose two algorithms for correcting saturation in color images and videos. One algorithm uses a fast Bayesian-based approach that utilizes images’ strong spatial correlation and the correlations between the R, G, and B color channels. The other algorithm takes advantage of the strong correlation between the chroma of the saturated pixels and their surrounding unsaturated pixels. Experimental results show that our methods effectively correct the saturated 2D and 3D images and videos. Our algorithms significantly outperform the existing state-of-the-art method in both objective and subjective qualities, resulting in plausible content that resembles real-world scenes. iii Preface This thesis presents research conducted by Di Xu, in collaboration with Drs. Lino Coria and Colin Doutre, under the guidance of Dr. Panos Nasiopoulos. A list of publications resulting from the work presented in this thesis is provided on the following pages. The work presented in Chapter 2 has been published in [P1-P4] and submitted to [P5]. The content of Chapter 3 is submitted to [P6], a provisional patent application has been filled based on the material in the chapter [P7], part of the chapter is published in [P8], and an application is published in [P9]. Chapter 4 appears in [P10-P12]. The work presented in Chapters 2 and 3 of this thesis was performed by Di Xu and Dr. Lino Coria, including data acquisition, algorithm designing, and manuscript writing. Di Xu was the main contributor for implementing the proposed algorithms, conducting the experiments, and analyzing the results. Dr. Panos Nasiopoulos provided guidance and editorial input into the manuscript writing. The work presented in Chapter 4 was primarily performed by Di Xu, including designing and implementing the proposed algorithms, performing all experiments, analyzing the results, and writing the manuscripts. The work was conducted with suggestions, feedback, and manuscript editing from Dr. Colin Doutre and the guidance and editorial input of Dr. Panos Nasiopoulos. The first and last chapters of this thesis were written by Di Xu, with editing assistance from Dr. Nasiopoulos. iv List of Publications Based on Work Presented in This Thesis Chapter 2 [P1] D. Xu, L. E. Coria, and P. Nasiopoulos, "Guidelines for an Improved Quality of Experience in 3D TV and 3D Mobile Displays," Journal of the Society for Information Display , vol. 20, no. 7, pp. 397–407, July 2012. [P2] D. Xu, L. E. Coria, and P. Nasiopoulos, "Quality of Experience for the Horizontal Pixel Parallax Adjustment of Stereoscopic 3D Videos," in Proc. of IEEE International Conference on Consumer Electronics (ICCE) , Las Vegas, NV, USA, Jan. 13-16, 2012, pp. 398-399. [P3] L. Coria, D. Xu, and P. Nasiopoulos, "Quality of Experience of Stereoscopic Content on Displays of Different Sizes: A Comprehensive Subjective Evaluation," in Proc. of IEEE International Conference on Consumer Electronics (ICCE) , Las Vegas, NV, USA, Jan. 9-12, 2011, pp. 778-779. [P4] D. Xu, L. Coria, and P. Nasiopoulos, "Guidelines for Capturing High Quality Stereoscopic Content Based on a Systematic Subjective Evaluation," in Proc. of IEEE International Conference on Electronics, Circuits, and Systems, (ICECS) , Athens, Greece, Dec. 12-15, 2010, pp. 166-169. [P5] D. Xu, L. Coria, and P. Nasiopoulos, “A quality metric for 3D content: The effect of capturing parameters,” submitted, Feb. 2013. Chapter 3 [P6] D. Xu, L. Coria, and P. Nasiopoulos, "Smart Stereoscopic 3D Video Reframing Based on a 3D Visual Attention Model," submitted, Nov. 2012. [P7] D. Xu, L. Coria, and P. Nasiopoulos, “Automatic Stereoscopic 3D Video Reframing,” US Provisional Patent Application No. 61/714119, filed October 15, 2012. [P8] L. Coria, D. Xu, and P. Nasiopoulos, "Automatic Stereoscopic 3D Video Reframing," in Proc. of 3DTV-CONFERENCE 2012 , ETH Zurich, Switzerland, Oct. 15-17, 2012. [P9] M. T. Pourazad, D. Xu, and P. Nasiopoulos, "Random Forests Based View Generation for Multiview TV," in Proc. of IEEE International Conference on Tools with Artificial Intelligence (ICTAI) , Athens, Greece, Nov. 7-9, 2012. v Chapter 4 [P10] D. Xu, C. Doutre, and P. Nasiopoulos, "Correction of Clipped Pixels in Color Images," IEEE Transactions on Visualization and Computer Graphics , vol. 17, no. 3, pp. 333-344, Mar. 2011. [P11] D. Xu, C. Doutre, and P. Nasiopoulos, "An Improved Bayesian Algorithm for Color Image Desaturation," in Proc. of IEEE International Conference on Image Processing (ICIP) , Hong Kong, Sep. 2010, pp. 1325-1328. [P12] D. Xu, C. Doutre, and P. Nasiopoulos, "Saturated-Pixel Enhancement for Color Images," in Proc. of IEEE International Symposium on Circuits and Systems (ISCAS) , Paris, France, May 30th to Jun. 2nd, 2010, pp. 3377-3380. vi Table of Contents Abstract ......................................................................................................................... ii Preface .......................................................................................................................... iv Table of Contents ........................................................................................................ vii List of Tables ................................................................................................................. x List of Figures ............................................................................................................... xi List of Acronyms ........................................................................................................ xiv Acknowledgments........................................................................................................ xv Dedication .................................................................................................................. xvii 1 Introduction and Overview ..................................................................................... 1 1.1 3D and High Dynamic Range Video Technology Overview ....................... 3 1.1.1 Stereoscopic 3D Video Representation ............................................................. 3 1.1.2 3D Display Technologies.................................................................................. 5 1.1.3 High Dynamic Range Imaging Technology ...................................................... 7 1.1.4 Quality Assessment Methods for Visual Media ................................................. 8 1.2 Capturing and Displaying Stereoscopic 3D Content ................................... 9 1.2.1 Camera Setup ................................................................................................. 10 1.2.2 Horizontal Parallax Adjustment ...................................................................... 13 1.3 Reframing Technology Overview ............................................................. 13 1.3.1 Existing Reframing Solutions ......................................................................... 14 1.3.2 Visual Attention Model and Automatic Reframing Techniques ....................... 15 1.4 Color Distortion Due to Saturation in Low Dynamic Range Content ........ 17 1.4.1 Existing Algorithms for Saturated-Pixel Enhancement .................................... 17 1.5 Thesis Outline .......................................................................................... 22 2 Guidelines for an Improved Quality of Experience in 3D TV and 3D Mobile Displays .......................................................................................................................