Automatic 2D-To-3D Conversion of Single Low Depth-Of-Field Images
Total Page:16
File Type:pdf, Size:1020Kb
AUTOMATIC 2D-TO-3D CONVERSION OF SINGLE LOW DEPTH-OF-FIELD IMAGES Serendra Reddy Supervisor: Associate Professor Fred Nicolls Thesis Presented for the Degree of DOCTOR OF PHILOSOPHY in the Department of Electrical Engineering UNIVERSITY OF CAPE TOWN 2016 Abstract This research presents a novel approach to the automatic rendering of 3D stereoscopic disparity image pairs from single 2D low depth-of-field (LDOF) images. Initially a depth map is produced through the assignment of depth to every delineated object and region in the image. Subsequently the left and right disparity images are produced through depth image- based rendering (DIBR). The objects and regions in the image are initially assigned to one of six proposed groups or labels. Labelling is performed in two stages. The first involves the delineation of the dominant object-of-interest (OOI). The second involves the global object and region grouping of the non-OOI regions. The matting of the OOI is also performed in two stages. Initially the in focus foreground or region-of-interest (ROI) is separated from the out of focus background. This is achieved through the correlation of edge, gradient and higher-order statistics (HOS) saliencies. Refinement of the ROI is performed using k-means segmentation and CIEDE2000 colour-difference matching. Subsequently the OOI is extracted from within the ROI through analysis of the dominant gradients and edge saliencies together with k-means segmentation. Depth is assigned to each of the six labels by correlating Gestalt-based principles with vanishing point estimation, gradient plane approximation and depth from defocus (DfD). To minimise some of the dis-occlusions that are generated through the 3D warping sub-process within the DIBR process the depth map is pre-smoothed using an asymmetric bilateral filter. Hole-filling of the remaining dis-occlusions is performed through nearest-neighbour horizontal interpolation, which incorporates depth as well as direction of warp. To minimising the effects of the lateral striations, specific directional Gaussian and circular averaging smoothing is applied independently to each view, with additional average filtering applied to the border transitions. Each stage of the proposed model is benchmarked against data from several significant publications. Novel contributions are made in the sub-speciality fields of ROI estimation, OOI matting, LDOF image classification, Gestalt-based region categorisation, vanishing point detection, relative depth assignment and hole-filling or inpainting. An important contribution is made towards the overall knowledge base of automatic 2D-to-3D conversion techniques, through the collation of existing information, expansion of existing methods and development of newer concepts. i Acknowledgements I would like to thank my supervisor Fred Nicolls for his valuable guidance and input into the structure and direction of the thesis. I would like to thank my family Anupa, Leya and Zaydin for their motivation, support and patience. I would like to thank especially my parents Thayalan, Dolly, Sarojini and Vasi for their unwavering dedication and encouragement over the years. I would also like to acknowledge the financial assistance of the Durban University of Technology. ii CONTENTS I. INTRODUCTION ...................................................................................................................... 1 A. Preamble ................................................................................................................................... 1 B. Research Challenges ................................................................................................................ 7 C. Summary of the Proposed Approach ..................................................................................... 11 D. Novel Contributions ............................................................................................................... 15 E. Structure of the Thesis ........................................................................................................... 16 II. UNSUPERVISED REGION-OF-INTEREST EXTRACTION BASED ON THE CORRELATION OF GRADIENT AND HIGHER-ORDER STATISTICS SALIENCIES .................................... 18 A. Introduction ............................................................................................................................ 18 B. Related Work ......................................................................................................................... 19 C. Proposed Approach ................................................................................................................ 24 1) Saliency Maps ................................................................................................................. 26 2) ROI Detection and Extraction ......................................................................................... 28 D. Results and Discussion ........................................................................................................... 37 E. Summary ................................................................................................................................ 51 F. Conclusion .............................................................................................................................. 52 III. UNSUPERVISED MATTING OF THE OBJECT-OF-INTEREST IN LOW DEPTH-OF-FIELD IMAGES ................................................................................................................................. 54 A. Introduction ............................................................................................................................ 54 B. Proposed Approach ................................................................................................................ 56 1) OOI Baseline Reference .................................................................................................. 57 2) LDOF Image Classification ............................................................................................ 57 3) OOI Matting .................................................................................................................... 63 C. Results .................................................................................................................................... 70 D. Discussion .............................................................................................................................. 74 E. Conclusion .............................................................................................................................. 75 IV. AUTONOMOUS DEPTH MAP GENERATION OF SINGLE 2D LOW DEPTH-OF-FIELD IMAGES USING DEFOCUS, LINEAR PERSPECTIVE AND GESTALT PRINCIPLES ........ 77 A. Introduction ............................................................................................................................ 77 B. Related Work ......................................................................................................................... 79 C. Proposed Approach ................................................................................................................ 89 1) Relative Depth Descriptor ............................................................................................... 89 2) Relative Depth Assignment ............................................................................................ 94 D. Results and Discussion ......................................................................................................... 102 E. Conclusion ............................................................................................................................ 107 iii V. VANISHING POINT DETECTION OF MAN-MADE ENVIRONMENTS ............................ 109 A. Introduction .......................................................................................................................... 109 B. Previous Approaches ............................................................................................................ 110 C. Proposed Method ................................................................................................................. 117 1) Straight Edge Segment Detection ................................................................................. 118 2) Straight Line Interpretation and Co-Planar Refinements .............................................. 120 3) Candidate Vanishing Points .......................................................................................... 122 4) Optimal Vanishing Point Estimation ................................................................................... 124 D. Results and Discussion ......................................................................................................... 124 1) Benchmarking ............................................................................................................... 124 2) Testing ........................................................................................................................... 129 E. Conclusion ............................................................................................................................ 132 VI. STEREOSCOPIC IMAGE SYNTHESIS OF SINGLE 2D LOW DEPTH-OF-FIELD IMAGES USING DEPTH IMAGE-BASED RENDERING ........................................................................ 134 A. Introduction .........................................................................................................................