ABSOLUTE DEPTH USING LOW-COST LIGHT FIELD CAMERAS By Shreedhar Rangappa A Doctoral Thesis submitted in partial fulfilment of the requirements for the award of Doctor of Philosophy of Loughborough University January 2018 Acknowledgments I would firstly like to convey my sincere thanks to my first supervisor, Dr Jon Petzing, for his guidance, support, patience and encouragement throughout various stages of my PhD journey. I always imagined PhD journey as unseen track of a rally, being driver myself and Jon as my co-driver, who constantly guided me to stay on track and reach the goal. I wish to thank my second supervisor, Dr Peter Kinnell, for his wonderful ideas that really made me think of different perspectives of solving problems. Also, I wish to thank Intelligent Automation Centre and the Loughborough University for supporting throughout my PhD journey. It would have been difficult to manage my thoughts, ideas and feelings without someone to listen and respond to it, which was extremely well handled by my friends and loved ones, to whom I convey my thanks. Finally, thanks to my parents (RJ) for their unconditional love and ∞ support. ತಾ벿 ಮ郍ತು 郍ಂದೆಗೆ ಅರ್ಪಿಸತತೆುೇನೆ Dedicated to mom & dad Abstract Digital cameras are increasingly used for measurement tasks within engineering scenarios, often being part of metrology platforms. Existing cameras are well equipped to provide 2D information about the fields of view (FOV) they observe, the objects within the FOV, and the accompanying environments. But for some applications these 2D results are not sufficient, specifically applications that require Z dimensional data (depth data) along with the X and Y dimensional data. New designs of camera systems have previously been developed by integrating multiple cameras to provide 3D data, ranging from 2 camera photogrammetry to multiple camera stereo systems. Many earlier attempts to record 3D data on 2D sensors have been completed, and likewise many research groups around the world are currently working on camera technology but from different perspectives; computer vision, algorithm development, metrology, etc. Plenoptic or Lightfield camera technology was defined as a technique over 100 years ago but has remained dormant as a potential metrology instrument. Lightfield cameras utilize an additional Micro Lens Array (MLA) in front of the imaging sensor, to create multiple viewpoints of the same scene and allow encoding of depth information. A small number of companies have explored the potential of lightfield cameras, but in the majority, these have been aimed at domestic consumer photography, only ever recording scenes as relative scale greyscale images. This research considers the potential for lightfield cameras to be used for world scene metrology applications, specifically to record absolute coordinate data. Specific interest has been paid to a range of low cost lightfield cameras to; understand the functional/behavioural characteristics of the optics, identify potential need for optical and/or algorithm development, define sensitivity, repeatability and accuracy characteristics and limiting thresholds of use, and allow quantified 3D absolute scale coordinate data to be extracted from the images. The novel output of this work is; an analysis of lightfield camera system sensitivity leading to the definition of Active Zones (linear data generation – good data) and In- active Zones (non-linear data generation – poor data), development of bespoke calibration algorithms that remove radial/tangential distortion from the data captured using any MLA based camera, and, a light field camera independent algorithm that allows the delivery of 3D coordinate data in absolute units within a well-defined measurable range from a given camera. Keywords: Lightfield, Plenoptic Camera, Micro Lens Arrays, Coordinate metrology, Sensitivity, Accuracy, 3D data sets. Table of Contents Depth recording problems using 2d photosensor ............................................... 4 New imaging technique: Microlens array based light field ............................... 5 Price tag of industrial light field cameras .......................................................... 6 Low-cost light field cameras .............................................................................. 7 Problem due to black-box effect ........................................................................ 8 Objectives and aims ........................................................................................... 9 Chapter description .......................................................................................... 11 Plenoptic function and light field ..................................................................... 15 2.1.1. Theory ............................................................................................................... 15 2.1.2. Ray-Space Representation ................................................................................ 17 Plenoptic camera systems ................................................................................ 19 2.2.1. Early developments .......................................................................................... 19 2.2.2. Methods of light field acquisition ..................................................................... 21 Microlens array based light field camera ......................................................... 23 Commercially available MLA based light field cameras ................................. 26 Conclusion........................................................................................................ 27 3.1 Introduction ...................................................................................................... 31 3.2 Image acquisition using conventional cameras (2D) ....................................... 32 3.3 Image acquisition in MLA based light field cameras ...................................... 34 3.3.1 Pinhole approximation of LF cameras .............................................................. 35 3.3.2 Light propagation through MLA ...................................................................... 38 3.4 Features of MLA based light field cameras ..................................................... 41 3.4.1 Sub-aperture images ......................................................................................... 42 3.4.2 Post-focusing .................................................................................................... 44 3.4.3 All-in-focus and depth data .............................................................................. 49 3.5 Potential features of MLA based light field cameras for metrology ................ 51 3.6 Conclusion........................................................................................................ 52 4.1 Introduction ...................................................................................................... 57 4.2 Raw image of light field camera ...................................................................... 58 4.3 Geometric and light ray dependent MLA cord ................................................ 64 4.4 Mounting issue of MLA on Photosensor ......................................................... 67 4.5 Algorithm to generate the MLA cord ............................................................... 70 4.6 Conclusion........................................................................................................ 78 5.1. Introduction ...................................................................................................... 83 5.2. Features of the Lytro cameras .......................................................................... 84 5.2.1. Performance of the Lytro cameras .................................................................... 84 5.2.2. Lytro software depth sensitivity to optical distortion ....................................... 87 5.3. Depth Calibration setup .................................................................................... 92 5.4. Specification of the Lytro cameras, software and external factors .................. 92 5.5. Response curve ................................................................................................. 96 5.6 Pixel resolution of the Lytro cameras .............................................................. 98 5.7 Result 3D Measurement ................................................................................. 103 5.8 Lytro response to uniform illumination ......................................................... 109 5.9 Conclusion...................................................................................................... 111 Introduction .................................................................................................... 116 Theory of perspective views .......................................................................... 121 Order of views ................................................................................................ 124 Importance of touch to focus .......................................................................... 127 Base disparity between perspective views ..................................................... 129 Quality of complex lens systems.................................................................... 139 Depth estimation in absolute scale ................................................................. 148 Conclusion.....................................................................................................