Timescape Image Panorama Registration Techniques
Total Page:16
File Type:pdf, Size:1020Kb
Timescape Image Panorama Registration Techniques by Heider K. Muhsin Ali, M.Sc A thesis submitted to the Faculty of Graduate and Postdoctoral Affairs in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Electrical and Computer Engineering Ottawa-Carleton Institute for Electrical and Computer Engineering (OCIECE) Department of Systems and Computer Engineering Carleton University Ottawa, Ontario November 2016 © Heider K. Muhsin Ali, 2016 Abstract Panoramic image viewing in its common form simulates a comprehensive spatial or temporal view. The term panorama indicates the concepts of space (or place) and time. The space sense of panorama has dominated as the default concept since it has been scientifically and commercially applied for decades. However, the temporal concept of panorama, which has been recently investigated by researchers and historians, offers a new context of the definition of panorama. A novel application in the field of panorama display has been realized and presented by this work which aims to participate effectively in cultural heritage preservations around the world and to provide an easy access tool to demonstrate a timelined panorama view of landmarks. Consequently, a new concept of panorama has been accomplished through a timelined display of registered historic and modern images for many famous landmarks around the world. This is achieved by merging geographical and historical information into a single attractive temporal and spatial panoramic view; also known as a timescape. This comprehensive view is achieved from a collection of historic and modern images of such landmarks available on the Internet. Hundreds of thousands of landmark images covering more than one century of time have been collected using websites like Flickr and Google Images. We use Gabor filters and neural networks to select a subset of images that have roughly the same view for each landmark in order to keep the most suitable images that will be used to build and demonstrate the temporal panorama. The processing of the selected set of historic and modern images has to contend with radiometric differences, geometric distortions, and ii severely differing capture systems and environments in order to prepare them for alignment. Finally, the aligned images; warped or registered, were prepared for presentation in a timelined image display. The image registration is done using optimized optical flow of extracted SIFT features after applying a set of similarity measure standards to select the highest precision subset of aligned images and to develop the timescape display. iii Acknowledgements I am so grateful to the GOD (ALLAH) for his uncountable bounties on me including the ability and help to perform and write this thesis. I would like to express my deep and sincere appreciation to my thesis supervisor, Professor Anthony Whitehead, for his precious guidance throughout the period of my graduate study. This thesis would not have been possible without his direction and assistance. His vast knowledge and continuous guidance have helped shape the contributions in this thesis. I express my great appreciation to my wife, Zahra who dedicated her never ending love for me and was my partner towards achievement, and to my adorable daughters and son, who were patient with me while being always busy. I also appreciate the love and support of my brothers and sisters who are always the source inspiration and encouragement for me. iv Table of Contents Abstract .............................................................................................................................. ii Acknowledgements .......................................................................................................... iv Table of Contents .............................................................................................................. v List of Table ...................................................................................................................... xi List of Illustrations ......................................................................................................... xiii List of Appendices ......................................................................................................... xxii Chapter 1: Introduction ................................................................................................... 1 1.1 Cultural Heritage Preservation ....................................................................................... 1 1.2 Panoramas....................................................................................................................... 2 1.3 Historic Photos Challenges………………………………………………………….....3 1.4 Thesis Objectives………………………………………………………………………4 1.5 System Structure………………………………………………………………………6 1.6 Thesis Contributions…………………………………………………………………..8 1.7 Thesis Organization…………………………………………………………………...9 Chapter 2: Background and Related Work ................................................................. 10 2.1 Historical Photos and Documents................................................................................. 11 2.2 Photographic Technology Evolution ............................................................................ 14 2.2.1 Technical Aspects of Photography……………….……………..….………..17 2.2.2 Exposure and Rendering……………………………………….……..……...18 2.3 Image Subset Selection……………………………………………………………….19 2.3.1 Image classification approaches……………………………………...……..19 2.3.2 Image Classification Techniques……………………………………………21 v 2.3.3 Image Selection Literature Review………………………………….……….23 2.4 Automatic Image Registration…………………………………………………………24 2.4.1 Registration Methods and Classification…………………………………..……..25 2.4.1.1 Automation Degree………………………………………………..………25 2.4.1.2 Single and Multi-Modal Methods…………………………………..……..26 2.4.1.3 Linear and Nonlinear Transformation……………………………………..26 2.4.1.4 Spatial and Frequency Domain Methods…………………………...……..26 2.4.1.5 Parametric and Nonparametric Image Registration……………………..…26 2.4.1.5.1 Parametric Geometric Transformations…………………………..….27 2.4.1.5.1.1 Rigid Transformations……………………………...……..27 2.4.1.5.1.2 Affine Transformations………………………….………...28 2.4.1.5.1.3 Phase Correlation…………………………………………..29 2.4.1.5.2 Nonparametric Geometric Transformations…………….…...………29 2.4.2 Similarity Measure Criteria…………………………………………………….…..29 2.4.2.1 Similarity Measures for Linear Relations……………………………..……..30 2.4.2.1.1 Sum of Squared Differences………………………...………………..30 2.4.2.1.2 Histogram Difference Measure…………………………..……………30 2.4.2.1.3 Cross Correlation……………………………………..……………..31 2.4.2.2 Similarity Measures for Nonlinear Relations…………………...……...…….31 2.4.2.2.1 Correlation Ratio…………………………………………….………31 2.4.2.2.2 Structural Similarity Index (SSIM)……………………….…………31 2.4.2.2.3 Root-Mean Square Error (RMSE)………………..………………….32 2.4.2.2.4 Mutual Information……………………………..……………………32 2.5 Feature Extraction and Matching……………………………..……………..………….33 2.5.1 Low-Level Feature Extraction…………………….………………………….…33 2.5.1.1 Edge Detection……………………………….…………..………..……33 vi 2.5.1.2 Phase congruency………………………………………………...……..34 2.5.1.3 Localized feature extraction………………………………….……….35 2.5.1.4 Describing Image Motion……………………………………………..36 2.5.2 Feature Extraction Standard Techniques……………………………………….…..37 2.5.3 High-Level Feature Extraction………………………………………………..……45 2.5.3.1 Deformable Templates…………………………………………..……..46 2.5.3.2 Parts-Based Shape Analysis……………………………………...……..47 2.5.3.3 Active Contours/Snakes……………………………………………..….47 2.5.3.4 Shape Skeletonization……………………………………………..……48 2.6 Optimization Process in Image Registration………………………………….………49 2.6.1 Gauss-Newton (GN) Optimization Algorithm…………………………...……..51 2.6.2 Levenberg-Marquardt (LM) Optimization Algorithm………………………….51 2.6.3 Quasi-Newton (QN) Optimization Algorithm……………………………….....52 2.6.4 Nonlinear Conjugate Gradient (NCG) Optimization Algorithm……………….52 2.6.5 Markov Random Fields (MRF)………………………………………..……….52 2.6.6 Belief Propagation…………………………………………………..………….54 2.7 Panorama Types………………………………………………………………...……56 2.7.1 Spatial Panoramas………………………………………………………...……..56 2.7.1.1 2-D Spatial Panoramas…………………………….………….………..57 2.7.1.2 3-D Spatial Panoramas…………………………………..……..……..…58 2.7.2 Temporal Panoramas……………………………………………………..…….60 2.7.2.1 Time-lapse Videos………………………………………………..……..60 2.7.2.2 Historic Perspective Panorama (HPP)…………………………….…….61 2.7.3 Timelined Panorama………………………………………………………..…..62 2.7.3.1 Rideau Timescapes………………………………………………….….62 vii Chapter 3: Image Subset Selection ................................................................................ 63 3.1 Timeline Based Image Data Set……………………………………………………..….64 3.2 Gabor Functions and Filters for Feature Extraction………………………………..….65 3.3 Neural Networks………………………………………………………………………71 3.3.1 Neural Network Training………………………………………………...……73 3.4 Experimental Results and Analysis……………………………………………………82 Chapter 4: Automatic Feature Extraction ............................................................... 89 4.1 Feature extraction: Optical Flow of Extracted Features………………………………90 4.1.1 Dense SIFT Features and Optical Flow…………………………..…….…...…92 4.1.2 Results and Discussion………………………………………..………….……94 4.2 Feature extraction: Standard Techniques………………………….…………..……...96 4.2.1 Feature Extraction and Matching Techniques……………….……………..….96 4.2.1.1 Matching Procedure…………………………….…………….....…..98 4.2.1.2 Disparity Gradient Filter…………………………..………………..…99 4.2.2 Harris Corner Detector………………………………………..………………….99