University of Calgary PRISM: University of Calgary's Digital Repository Graduate Studies The Vault: Electronic Theses and Dissertations 2013-12-05 Indoor Navigation based on Fiducial Markers of Opportunity Lakhani, Mazhar Ali Lakhani, M. A. (2013). Indoor Navigation based on Fiducial Markers of Opportunity (Unpublished master's thesis). University of Calgary, Calgary, AB. doi:10.11575/PRISM/26507 http://hdl.handle.net/11023/1176 master thesis University of Calgary graduate students retain copyright ownership and moral rights for their thesis. You may use this material in any way that is permitted by the Copyright Act or through licensing that has been assigned to the document. For uses that are not allowable under copyright legislation or licensing, you are required to seek permission. Downloaded from PRISM: https://prism.ucalgary.ca UNIVERSITY OF CALGARY Indoor Navigation based on Fiducial Markers of Opportunity by Mazhar Ali Lakhani A THESIS SUBMITTED TO THE FACULTY OF GRADUATE STUDIES IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE DEPARTMENT OF ELECTRICAL AND COMPUTER ENGINEERING CALGARY, ALBERTA NOVEMBER, 2013 © Mazhar Ali Lakhani 2013 Abstract Indoor navigation has gained significant importance in the last decade due to its applications in the location based services industry. The outdoor navigation technology of GNSS by itself does not offer good solutions due to presence of multipath and low SNR in indoor environments. Complimentary sensors like IMU, magnetometers, cameras, etc., have their own set of problems. A camera however, provides a wealth of information for position estimation. The main aim of this thesis is to analyse camera based navigation systems using stationary figures of opportunity. Unlike traditional navigation systems that utilise coded markers, this thesis performs navigation based on planar stationary figures that are inherently present in the environment, hence the more structure the more information. The novelty of this thesis lies in its usage of Hough transforms to do the same. Initial experimental results demonstrate that an average webcam quality camera provides enough information to extract reasonably good navigation coordinates under specific conditions. ii Acknowledgements To my supervisor, Dr. John Nielsen and my co-supervisor, Dr. Gérard Lachapelle, this thesis could not exist without your guidance and encouragement. Thank you for your gracious support and insight and for providing me with this opportunity to explore my interests in the field of navigation. I would also like to thank all the wonderful and diverse members of Position Location And Navigation (PLAN) Group especially my team members Yuqi Lee and Neha Dawar. I would also like to thank Piyush Rawat for assisting me in writing my thesis. To the administrative staff at the Electrical Engineering Department, thank you for all your hard work behind the scenes and making my research possible. To my parents, thank you for instilling in me a love for engineering and natural sciences. Thank your also for your continued support and encouragement through the many years that led up to the completion of my thesis. iii Table of contents Abstract…………………………………………………………………………………………...ii Acknowledgements……………………………………………………………………….…...…iii Table of Contents…………………………………………………………………………………iv List of Figures…………………………………………………….……………...………......…...vi List of Tables……………………………………………………………………………………..ix List of Abbreviations……………………………………………………………………………...x 1. Introduction………………………………………………………………………………...…1 1.1 The Problem……………………………………………………………………………….1 1.2 Different Approaches to Location Estimation………………….………………………....3 1.3 Vision Sensor…………………………………………………………………...…………3 1.4 Coded Fiducial Marker……………………………………………………………………4 1.5 Fiducial Marker Based Indoor Navigation System……………………………………….6 1.6 Thesis Outline……………………………………………………………………………..9 2. Background………………………………………………………………………………..…10 2.1 Feature Point……………………………………………..………………………………10 2.2 Good Features to Track……………..…………………………………………………...11 2.3 Preprocessing…………………………………………………………………………….15 2.3.1 Box Filter………………………………………………………………………...18 2.3.2 Gaussian Smoothing……………………………………………………………..19 2.3.3 Thresholding……………………………………………………………………..22 2.3.4 Edge Detection………………………………………………………………......24 2.4 Contour Detection……………………………………………………………………….30 2.5 Perspective Transform…………………………………………………………………...33 2.6 Camera Calibration………………………………………………………………………43 2.7 RANSAC………………………………………………………………………………...49 2.8 Fiducial Marker of Opportunity…….……………………………………………………53 3. Proposed System and Concepts……………………………………………………………..58 3.1 Proposed FM Algorithm……………………………………………………………...…58 iv 3.2 The Marker Coding Scheme…………………………………………………………….59 3.3 Obtaining Rotation Angles from Perspective Transform…………………………….….62 3.4 Simultaneous Localisation and Mapping Techniques…………………………………...66 4. Experimental Validation………………………………………………………………….....69 4.1 Back Projection Experiment Trials………………………………………………………70 4.2 Known Trajectory – Turn Table Experiment Trials……………………………………..73 4.3 Transition from FM to FMO – Experiment Trials…………………………...………….76 4.4 Trajectory Estimation for Long Range – Experiment Trials…………………………….81 4.5 Fiducial Marker Of Opportunity………………………………………………………...86 4.5.1 Homography Estimation using HT………………………………………………87 4.5.2 Homography Estimation using HT for Image Sequence………………………...91 4.5.3 Homography Estimation using HT for Real-time Captured Video…………….102 4.6 Angle Estimation from Perspective Transform…………………………………..….109 5. Conclusions…………………………………………………………………………………112 5.1 Contributions…………………………………………………………………………...112 5.2 Future work…………………………………………………………………………….114 Bibliography……………………………………………………………………………………117 A. Optical Flow……………………………………………………………………………122 B. Affine Transformation………………………………………………………………….126 C. Back-Projection…………………………………………………………………………127 D. Code………………………………………………………………………………..…...129 v List of Figures Figure 1.1 Difference in accuracy requirements between indoor and outdoor environments ... ….2 Figure 1.2 Agama V-1325R camera observing a stationary fiducial marker on the wall .............. 5 Figure 1.3 Coded fiducial marker as observed by the camera ........................................................ 5 Figure 1.4 True Geometry of the fiducial marker ........................................................................... 6 Figure 1.5: A fiducial marker of opportunity detected by Hough transform……………………..7 Figure 2.1 Pixel intensity variation of the ROI in a scene ………………………………….…...11 Figure 2.2 Good Features to Track algorithm results for a test marker in two consecutive frames…………………………………………………………………………………………….14 Figure 2.3 Difference between the intensity values for a noisy image vs. a smooth image….….18 Figure 2.4 A box filter kernel of size 4x4……………………………………………………..…19 Figure 2.5 Image undergoing box filtering…………...……………………………………….…19 Figure 2.6 surface view and top view of a gaussian kernel with var(x) = 1 var(y) =1…………..20 Figure 2.7 Image undergoing gaussian smoothing………………………………………….…...21 Figure 2.8 Examples of different types of thresholding schemes……………………………….23 Figure 2.9 Image undergoing Canny edge detection…………………………………………….25 Figure 2.10 Kernel derivation for Sobel operator………………………………………………..28 Figure 2.11 Sobel operator output to an input image applied in horizontal, vertical and dual (horizontal-vertical) axis………………………………………………………………………....30 Figure 2.12 Flowchart for contour map of an image………………………………………….…31 Figure 2.13 Contour map of a test image undergoing various stages described in figure 2.9…...32 Figure 2.14 Image undergoing translation along X-axis by 100 pixels………………………….34 Figure 2.15 Image rotation of the selected region with respect to the image center by 30 degrees clockwise…………………………………………………………………………………………34 Figure 2.16 A camera observing a stationary marker from different viewing angles …..….…...36 Figure 2.17 A quadrilateral ABCD in camera’s ith image frame undergoing PT to form quadrilateral A’B’C’D’ in (i+1)th frame…………………………………………..………….….37 Figure 2.18 Figure depicting the camera and world coordinate system………………………....38 vi Figure 2.19 Pinhole camera model setup………………………………………….…………….39 Figure 2.20 Pinhole camera for a 3D object………………………………………..……………39 Figure 2.21 Types of Radial distortions….……….…………………………………….……….45 Figure 2.22 Camera Calibration for a chessboard pattern…………..……………………….…..47 Figure 2.23 Calibration Re-projection Error………………………..……………………….…...48 Figure 2.24 Mathematical model estimation using RANSAC and Least Squares……..………..53 Figure 2.25 Hough transform of a line…………………..……….……………………………....55 Figure 2.26 Line under consideration L1………………..………….……………………………56 Figure 2.27 Hough Transform of the line L1…………………………….…………………..…..57 Figure 2.28 Line detection using HT…………………………………….…………………..…..57 Figure 3.1 Flowchart of the proposed computer vision algorithm……...………….….….……...59 Figure 3.2 Example of a FM with unique ID…………………………………………………….62 Figure 3.3 Example of a coded fiducial marker with three dimensional data association…….....62 Figure 3.4 Block diagram of the overall SLAM algorithm with camera observables integrate with IMU to estimate the final trajectory……………………………………………………………...67 Figure 4.1 AGAMA V-1325R with a 2MP CMOS image sensor…………….…………………70 Figure 4.2 Diagram showing 6 DOF Experiment setup……………………….……….………..70 Figure 4.3 Camera observable of a test FM…………………………………….………………..71 Figure 4.4 Inverse perspective transformation to calculate the back projection