Phd Thesis, University of Sydney
Total Page:16
File Type:pdf, Size:1020Kb
LIGHT-FIELD FEATURES FOR ROBOTIC VISION IN THE PRESENCE OF REFRACTIVE OBJECTS Dorian Yu Peng Tsai MSc (Technology) BASc (Enginnering Science with Honours) Submitted in fulfillment of the requirement of the degree of Doctor of Philosophy 2020 School of Electrical Engineering and Computer Science Science and Engineering Faculty Queensland University of Technology Abstract Robotic vision is an integral aspect of robot navigation and human-robot interaction, as well as object recognition, grasping and manipulation. Visual servoing is the use of computer vision for closed-loop control of a robot’s motion and has been shown to increase the accuracy and performance of robotic grasping, manipulation and control tasks. However, many robotic vision algorithms (including those focused on solving the problem of visual servoing) find refractive objects particularly challenging. This is because these types of objects are difficult to perceive. They are transparent and their appearance is essentially a distorted view of the background, which can change significantly with small changes in viewpoint. What is often overlooked is that most robotic vision algorithms implicitly assume that the world is Lambertian—that the appearance of a point on an object does not change significantly with respect to small changes in viewpoint1. Refractive objects violate the Lambertian assumption and this can lead to image matching inconsistencies, pose errors and even failures of modern robotic vision systems. This thesis investigates the use of light-field cameras for robotic vision to enable vision-based motion control in the presence of refractive objects. Light-field cameras are a novel camera technology that use multi-aperture optics to capture a set of dense and uniformly-sampled views of the scene from multiple viewpoints. Light-field cameras capture the light field, which simul- taneously encodes texture, depth and multiple viewpoints. Light-field cameras are a promising alternative to conventional robotic vision sensors, because of their unique ability to capture view-dependent effects, such as occlusion, specular reflection and, in particular, refraction. First, we investigate using input from the light-field camera to directly control robot motion, a process known as image-based visual servoing, in Lambertian scenes. We propose a novel light-field feature for Lambertian scenes and develop the relationships between feature motion and camera motion for the purposes of visual servoing. We also illustrate in both simulation and using a custom mirror-based light-field camera, that our method of light-field image-based visual servoing is more tolerant to small and distant targets and partially-occluded scenes than monocular and stereo-based methods. Second, we propose a method to detect refractive objects using a single light field. Specifi- cally, we define refracted image features as those image features whose appearance have been distorted by a refractive object. We discriminate between refracted image features and the surrounding Lambertian image features. We also show that using our method to ignore the re- fracted image features enables monocular structure from motion in scenes containing refractive objects, where traditional methods fail. We combine and extend our two previous contributions to develop a light-field feature capable of enabling visual servoing towards refractive objects without needing a 3D geometric model of the object. We show in experiments that this feature can be reliably detected and extracted from the light field. The feature appears to be continuous with respect to viewpoint, and is therefore be suitable for visual servoing towards refractive objects. 1This Lambertian assumption is also known as the photo-consistency or brightness constancy assumption. This thesis represents a unique contribution toward our understanding of refractive objects in the light field for robotic vision. Application areas that may benefit from this research include manipulation and grasping of household objects, medical equipment, and in-orbit satellite ser- vicing equipment. It could also benefit quality assurance and manufacturing pick-and-place robots. The advances constitute a critical step to enabling robots to work more safely and reli- ably with everyday refractive objects. Statement of Original Authorship The work contained in this thesis has not been previously submitted to meet requirements for an award at this or any other higher education institution. To the best of my knowledge and belief, the thesis contains no material previously published or written by another person except where due reference is made. QUT Verified Signature Dorian Tsai March 2, 2020 Acknowledgements To my academic advisors, Professor Peter Ian Corke, Dr. Donald Gilbert Dansereau and Asso- ciate Professor Thierry Peynot, I would like to offer my most heartfelt gratitude. They shared with me an amazing knowledge, insight, creativity and enthusiasm. I am grateful for the re- sources and opportunities they provided, as well as their guidance, support and patience. In addition, I wish to convey my appreciation to Douglas Palmer and Thomas Coppin who were my fellow plenopticists for many helpful and stimulating discussions. Thanks to Dr. Steven Martin who helped with much of the technical engineering aspects of building and mounting light-field cameras to various robots over the years. Thanks to Dominic Jack and Ming Xu for being an excellent desk buddy. Thanks to Prof. Tristan Perez, Associate Professor Jason Ford and Dr. Timothy Molloy for helping to get me started on my PhD journey in inverse differential game theory applied to the birds and the bees, until I changed topics to light fields and robotic vision six months later. Thanks to Kate Aldridge, Sarah Allen and all of the other administrative staff in the Australian Centre for Robotic Vision (ACRV) for organising so many conferences and workshops, and keeping things running smoothly. This research was funded in part from the Queensland University of Technology (QUT) Post- graduate Research Award, the QUT Higher Degree Tuition Fee Sponsorship, the QUT Excel- lent Top-Up Scholarship, and the ACRV Top-Up Scholarship, as well as financial support in the form of employment as a course mentor and research assistant. The ACRV scholarship was supported in part by the Australian Research Council Centre of Excellence for Robotic Vision. Lastly, a very special thanks goes to the many faithful friends and family and colleagues who’s backing and constant encouragements sustained me through this academic marathon and grad- uate with a degree. I am especially indebted to Robin Tunley and Miranda Cherie Fittock for their camaraderie and steady moral support. Thank you all very much. Contents Abstract List of Tables vii List of Figures ix List of Acronyms xiii List of Symbols xv 1 Introduction 1 1.1 Motivation . 1 1.1.1 Limitations of Robotic Vision for Refractive Objects . 3 1.1.2 Seeing and Servoing Towards Refractive Objects . 5 1.2 Statement of Research . 8 1.3 Contributions . 9 1.4 Significance . 10 1.5 Structure of the Thesis . 11 2 Background on Light Transport & Capture 15 2.1 Light Transport . 15 i ii CONTENTS 2.1.1 Specular Reflections . 16 2.1.2 Diffuse Reflections . 16 2.1.3 Lambertian Reflections . 17 2.1.4 Non-Lambertian Reflections . 17 2.1.5 Refraction . 19 2.2 Monocular Cameras . 21 2.2.1 Central Projection Model . 21 2.2.2 Thin Lenses and Depth of Field . 23 2.3 Stereo Cameras . 26 2.4 Multiple Cameras . 28 2.5 Light-Field Cameras . 29 2.5.1 Plenoptic Function . 30 2.5.2 4D Light Field Definition . 32 2.5.3 Light Field Parameterisation . 34 2.5.4 Light-Field Camera Architectures . 36 2.6 4D Light-Field Visualization . 42 2.7 4D Light-Field Geometry . 44 2.7.1 Geometric Primitive Definitions . 44 2.7.2 From 2D to 4D . 46 2.7.3 Point-Plane Correspondence . 56 2.7.4 Light-Field Slope . 58 3 Literature Review 61 3.1 Image Features . 61 3.1.1 2D Geometric Image Features . 62 CONTENTS iii 3.1.2 3D Geometric Image Features . 65 3.1.3 4D Geometric Image Features . 66 3.1.4 Direct Methods . 69 3.1.5 Image Feature Correspondence . 70 3.2 Visual Servoing . 72 3.2.1 Position-based Visual Servoing . 73 3.2.2 Image-based Visual Servoing . 75 3.3 Refractive Objects in Robotic Vision . 81 3.3.1 Detection & Recognition . 82 3.3.2 Shape Reconstruction . 85 3.4 Summary . 92 4 Light-Field Image-Based Visual Servoing 95 4.1 Light-Field Cameras for Visual Servoing . 95 4.2 Related Work . 97 4.3 Lambertian Light-Field Feature . 99 4.4 Light-Field Image-Based Visual Servoing . 100 4.4.1 Continuous-domain Image Jacobian . 100 4.4.2 Discrete-domain Image Jacobian . 102 4.5 Implementation & Experimental Setup . 104 4.5.1 Light-Field Features . 104 4.5.2 Mirror-Based Light-Field Camera Adapter . 105 4.5.3 Control Loop . 106 4.6 Experimental Results . 107 4.6.1 Camera Array Simulation . 108 iv CONTENTS 4.6.2 Arm-Mounted MirrorCam Experiments . 110 4.7 Conclusions . 117 5 Distinguishing Refracted Image Features 119 5.1 Related Work . 122 5.2 Lambertian Points in the Light Field . 126 5.3 Distinguishing Refracted Image Features . 128 5.3.1 Extracting Image Feature Curves . 130 5.3.2 Fitting 4D Planarity to Image Feature Curves . 132 5.3.3 Measuring Planar Consistency . 137 5.3.4 Measuring Slope Consistency . 138 5.4 Experimental Results . 140 5.4.1 Experimental Setup . 140 5.4.2 Refracted Image Feature Discrimination with Different LF Cameras . 141 5.4.3 Rejecting Refracted Image Features for Structure from Motion . 148 5.5 Conclusions . 153 6 Light-Field Features for Refractive Objects 157 6.1 Refracted LF Features for Vision-based Control . 158 6.2 Related Work . 159 6.3 Optics of a Lens . 161 6.3.1 Spherical Lens .