Biologically Motivated Modeling and Imitating the Chameleon's Vision System Ofir Avni Technion - Computer Science Department - M.Sc. Thesis MSC-2006-27 - 2006 Technion - Computer Science Department - M.Sc. Thesis MSC-2006-27 - 2006 Biologically Motivated Modeling and Imitating the Chameleon's Vision System Research Thesis Submitted in partial fulfillment of the requirements for the degree of Master of Science in Computer Science Ofir Avni Submitted to the Senate of the Technion|Israel Institute of Technology HESHVAN 5767 HAIFA NOVEMBER 2006 Technion - Computer Science Department - M.Sc. Thesis MSC-2006-27 - 2006 Technion - Computer Science Department - M.Sc. Thesis MSC-2006-27 - 2006 The research was done under the supervision of Assoc. Prof. Ehud Rivlin in the department of Computer Science I wish to thank my advisor, Assoc. Prof. Ehud Rivlin, for his guidance and support through my long research. I would also like to thank Prof. Gadi Katzir from the department of Biology in the university of Haifa, for his guidance in the biological aspects of this research. The support and guidance of Dr. Hector Rotstein and Dr. Francesco Borrelli on issues regarding control theory was also very valuable. I would also like to thank Shay Ohayon, which provided help on pose estimation, and the source code for his tracking system, which served as a basis for mine. The generous financial help of the Technion is gratefully acknowledged. Technion - Computer Science Department - M.Sc. Thesis MSC-2006-27 - 2006 Technion - Computer Science Department - M.Sc. Thesis MSC-2006-27 - 2006 Contents Abstract 1 1 Introduction 3 1.1 Biologically Motivated Approach . 3 1.2 Tracking the Eyes of the Chameleon . 4 1.3 The Chameleon as a Model . 4 1.4 Thesis Outline . 8 2 The Visual System of the Chameleon 11 3 Related Work 15 3.1 3D Pose Estimation . 15 3.2 Environment Visual Scanning . 18 3.3 Active Target Tracking . 20 4 Tracking the Eyes of the Chameleon 23 4.1 Tracking the Chameleon Head . 23 4.2 Tracking the Eyes . 34 4.3 Calibration of Multiple Cameras and Mirrors . 38 4.4 Experiments . 44 Technion - Computer Science Department - M.Sc. Thesis MSC-2006-27 - 2006 5 Scanning the Environment 47 6 Low Level Control - Smooth Pursuit and Saccade 53 6.1 Active Vision Model . 53 6.2 Smooth Pursuit Control . 56 6.3 Saccade Controller . 57 6.4 Switching between Smooth Pursuit and Saccade . 60 7 Simulations and Experiments 61 7.1 System Architecture . 61 7.2 Scanning Method Simulations . 62 7.3 Smooth Pursuit and Saccade Controllers: Experimental Validation . 66 8 Conclusions 69 References 75 Hebrew Abstract ix Technion - Computer Science Department - M.Sc. Thesis MSC-2006-27 - 2006 List of Figures 1.1 The robotic head and a chameleon head . 6 1.2 System architecture . 7 2.1 A map of the ganglion cells in the chameleon eye . 12 2.2 Comparison of Chameleon Eye with a Typical Lizard Eye . 13 2.3 Focus/De-focus Movements of the Chameleon Eyes . 14 4.1 Body, world and camera coordinates systems . 24 4.2 A chameleon with the artificial features attached to it . 31 4.3 The parameters of the appearance of a feature . 32 4.4 Pattern of a feature . 33 4.5 Eye Model . 34 4.6 Finding the direction of the eye by line-sphere intersection . 36 4.7 Difference image reveals the location of the eyelid . 37 4.8 Mirror as a virtual camera . 40 4.9 Experiments setup . 45 4.10 Results of tracking the eyes . 46 6.1 Pan-tilt model and coordinate systems . 54 6.2 System model . 56 6.3 Smooth Pursuit min-max MPC Controller . 58 Technion - Computer Science Department - M.Sc. Thesis MSC-2006-27 - 2006 6.4 Saccade controller for different horizons N . 59 7.1 Simulation setup for the 1D case . 63 7.2 Simulation results for the search method in 1D . 63 7.3 Simulation results for search method using four cameras . 65 7.4 Simulation results for search method in 2D . 65 7.5 Experimental results for tracking sinusoidal signal . 67 7.6 Experimental results for tracking a target with constant acceleration . 67 Technion - Computer Science Department - M.Sc. Thesis MSC-2006-27 - 2006 Abstract This thesis takes the biologically motivated approach with regard to the visual system of the chameleon. The chameleon has a unique visual system, in which the two eyes scan independently the environment. In the first part of this work, a complex and innovative computer vision system, which uses cameras and mirrors, is designed and implemented in order to track the direction of the eyes of chameleons. In this part the problem of pose estimation with multiple cameras and mirrors is formulated and solved. The problem is formulated as a minimization problem of the three-dimensional geometric errors given the location of features found in the image and their three-dimensional model. The direction of the eyes is found by founding the eyelid in the image, and based on a geometric model of the eye. Preliminary results from this part, which includes biological experiments, indicates that chameleons scan the environment using a \negative correlation" strategy. That is, when one eye scans forward, the other, with high probability, scans backwards. In the second part, a robotic system based on the chameleon visual system is con- structed. Based on the observation from the first part, a new method for visual scanning with a set of independent pan-tilt cameras is presented. The method combines information about the environment and a model for the motion of the target to perform optimal scan- ning based on stochastic dynamic programming. For the implementation, a model-based control strategy is developed that performs target tracking. A switching control, combining smooth pursuit and saccades, is proposed. Robust and Minimum-time Model Predictive Control (MPC) theory is used for the design of the control law. Finally, simulative and experimental validation of the approach is presented. The scanning algorithm was simulated in matlab, and the resulting scanning pattern has a remarkable resemblance to the scanning behavior of chameleons. The target tracking method was implemented in a real-time system of two cameras mounted on top of pan-tilt heads. Experimental results of tracking a target are presented. 1 Technion - Computer Science Department - M.Sc. Thesis MSC-2006-27 - 2006 Technion - Computer Science Department - M.Sc. Thesis MSC-2006-27 - 2006 Chapter 1 Introduction 1.1 Biologically Motivated Approach In Biologically Motivated Robotics, biologists and scientists are working together to create robotic systems that mimic biological behaviors. The benefits from this kind of work are mutual. Biologists, for instance, can test models for biological behaviors on a real, usually simplified, system. As models are tested on real systems new ideas and corrections to the models may rise, improving the understanding of biological behaviors. Scientists, on the other hand, may get from the biological behaviors new ideas on how to solve engineering problems. These new ideas have the advantage of a \working solution" as they are tested and improved on a daily basis in the course of evolution. In this dissertation, we follow the biologically motivated approach to investigate the visual system of the chameleon, or more specific, eye movements in the chameleon. The chameleon has interesting eye movements since the laterally positioned eyes are able to move independently, each over a wide range. The eyes perform saccadic movements - i.e. large and rapid reorientation of the eye from one direction to the other - thus enabling the chameleon to quickly scan its surrounding. This arrangement of the visual system raises questions like what is the correct way to scan the environment, and how it should be carried out. In this dissertation we will try to answer those questions. This work explores several fields which compose the biologically motivated research. In the first part, the advanced abilities of computer vision are used to conduct biological research. A computer-vision system is designed and implemented to track the eye move- ments of chameleons. Using this system, experiments, which are biological in nature, are 3 Technion - Computer Science Department - M.Sc. Thesis MSC-2006-27 - 2006 4 CHAPTER 1. INTRODUCTION performed. In the second part, based on inputs from the biological experiments, a model that describes the biological behavior is formed. Last, a robotic system is implemented that mimic the biological behavior. 1.2 Tracking the Eyes of the Chameleon Tracking the eyes of a chameleon can be considered as the estimation of the 3D pose of an articulated body, where the head of the chameleon is considered as a rigid body, and each of the two eyes has additional two degrees of freedom. There are numerous works dealing with estimating pose of articulated body, many of them in the context of tracking the human body. In [Drouin et al., 2003] for example, the authors track and estimate a skeletal model of a human body. The tracked person is dressed with a black outfit, and distinguishable markers are placed on the joints, which enables easier tracking of the joints. In [Nickels and Hutchinson, 2001] the authors present a model-based approach to track articulated bodies, where the kinematic model is known. In our system, the chameleon's head is marked with distinguishable features which help to identify the head's pose. The direction of each eye is found relative to the head. In order to find the direction of the eye, the eyelid is needed to be found. This brings a difficulty since the range of motion of each eye is large, and the eyelid will not always be visible using one camera.