Flying Insects and Robots Dario Floreano · Jean-Christophe Zufferey · Mandyam V

Flying Insects and Robots Dario Floreano · Jean-Christophe Zufferey · Mandyam V. Srinivasan · Charlie Ellington Editors

Flying Insects and Robots

123 Editors Prof. Dario Floreano Dr. Jean-Christophe Zufferey Director, Laboratory of Intelligent Systems Laboratory of Intelligent Systems EPFL-STI-IMT-LIS EPFL-STI-IMT-LIS École Polytechnique Fédérale de Lausanne École Polytechnique Fédérale de Lausanne ELE 138 ELE 115 1015 Lausanne 1015 Lausanne Switzerland Switzerland dario.floreano@epfl.ch jean-christophe.zufferey@epfl.ch

Prof. Mandyam V. Srinivasan Prof. Charlie Ellington Visual and Sensory Neuroscience Animal Flight Group Queensland Brain Institute Dept. of Zoology University of Queensland University of Cambridge QBI Building (79) Downing Street St. Lucia, QLD 4072 Cambridge, CB2 3EJ Australia UK [email protected] [email protected]

ISBN 978-3-540-89392-9 e-ISBN 978-3-540-89393-6 DOI 10.1007/978-3-540-89393-6 Springer Heidelberg Dordrecht London New York

Library of Congress Control Number: 2009926857

ACM Computing Classiﬁcation (1998): I.2.9, I.2.10, I.2.11

c Springer-Verlag Berlin Heidelberg 2009 This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use.

Cover illustration: Flying insect image reproduced with permission of Mondolithic Studios Inc.

Cover design: KünkelLopka, Heidelberg

Printed on acid-free paper

Springer is a part of Springer Science+Business Media (www.springer.com) Preface

Flying insects represent a fascinating example of evolutionary design at the micro- scopic scale. Their diminutive size does not prevent them from perceiving the world, flying, walking, jumping, chasing, escaping, living in societies, and even finding their way home at the end of a long day. Their size and energy constraints demand extremely efficient and specialized solutions, which are often very different from those that we are accustomed to seeing in larger animals. For example, the visual system of flying insects, which features a compound eye comprising thousands of ommatidia Ð “little eyes” Ð represents a dramatic alternative to the design of our own eyes, which we share with all vertebrates and which has driven the design of today’s cameras. Do insect eyes differ from human eyes only superficially with respect to the optical and imaging characteristics, or do the nervous systems of their owners process the information that they receive in different ways? Several aspects of this question are explored in this book. The nervous system of flying insects not only coordinates the perception and motion of the animal at extremely high speed and in very dynamic conditions, but it actively monitors features in the surrounding environment, supports accurate landings in very tiny spots, handles recovery from high turbulence and collisions, directs the exploration of the environment in search of food, shelter or partners, and even enables the animal to remember how to return to its nest. Flying insects move their wings by using the whole thorax to produce fast, resonat- ing, respiration-like contractions that result in the movement of the wing appendages, whose morphology and constituent materials then modify the basic, passive flapping motion through the air. As such, these creatures represent a fascinating source of inspiration for engineers aiming to create increasingly smaller and autonomous robots that can take to the air like a duck to water, and go where no machine has gone before. At the same time, robotic insects can serve as embodied models for testing scientific hypotheses that would be impossible to study in numerical simulations because of the difficulty in creating realistic visual environments, capturing the physics of fluid dynamics in very turbulent and low-speed regimes, reproducing the elastic properties of the active and passive materials that make up an insect body, and accurately modelling the perception-action loops that drive the behavior of the system. Despite much recent progress, both the functioning of flying insects and the design of micro flying robots are not yet fully understood, which makes this trans- disciplinary area of research extremely fascinating and fertile with discoveries. This book brings together for the first time highly selected and carefully edited contribu- tions from a community of biologists and engineers who share the same passion for

v vi Preface understanding the design principles of flying insects and robots. The book is the off- spring of a stimulating meeting with the same title and organizers that was held in the summer of 2007 at Monte Verità in Switzerland. After the meeting, we decided to assemble a carefully edited volume that would serve both as a tutorial introduction to the field and as a reference for future research. In the months that followed, we solicited some of the participants, as well as additional authors whose research was complementary and would fit the book plan, to write chapters for a larger audience. The authors and the editors spent most of 2008 writing, revising, and cross-linking the chapters in order to produce a homogeneous, accessible, and yet up-to-date book. Approximately half of the book is written from a biological perspective and the other half from an engineering perspective, but in all cases the authors have attempted to use plain terminology that is accessible to both sides and they have made several links and suggestions that cut across the traditional divide between biology and engineering. The book starts with a description of today’s advanced methods used to study flying insects. After this, the reader is taken through a description of the perceptual, neuronal, and behavioral properties of flying insects and of their implications for the design of sensors and control strategies necessary to achieve autonomous navigation of miniature flying vehicles. Once this ground has been covered, the reader is gradu- ally introduced to the principles of aerodynamics and control suitable for microsys- tems with flapping wings and to several examples of robots with fixed and flapping wings that are inspired by the principles of flying insects. We encourage readers to photocopy the figures of Chapter 15, cut out the drawings, and assemble a moving model of the thorax, which should provide an intuitive understanding of the typical workout routine of a flying insect. Two chapters venture into the area of robots that live and transit between the ground and the air. Although these chapters are more speculative from a biological perspective, they highlight the fact that flying insects are also terrestrial animals and that robots capable of a transition between terrestrial locomotion and flight can have several advantages. Finally, the book closes with two engineering chapters: one dedicated to energy supply and the possible use of solar cells to power micro aerial vehicles, and another to the technology available today and in the near future for realizing autonomous, flying micro robots. We sincerely hope that you will enjoy and learn from this book as much as we did throughout the entire creation and editing of this project. We would like to express our deep gratitude to the contributors of all the chapters, who enthusiastically presented their knowledge and achievements in such a small space and time, and who displayed amazing patience and dedication in revising their own material and that of their col- leagues. Last, but not least, we would like to thank Ronan Nugent at Springer for welcoming this project, following it through all its production stages while accom- modating many of our requests, and making sure that it is presented in a form that best fits its content both on the Internet and in the printed edition.

Lausanne Dario Floreano Brisbane Jean-Christophe Zufferey Cambridge Mandyam V. Srinivasan 12 June 2009 Charlie Ellington Contents

1 Experimental Approaches Toward a Functional Understanding of Insect Flight Control ...... 1 Steven N. Fry 2 From Visual Guidance in Flying Insects to Autonomous Aerial Vehicles ...... 15 Mandyam V. Srinivasan, Saul Thurrowgood, and Dean Soccol 3 Optic Flow Based Autopilots: Speed Control and Obstacle Avoidance 29 Nicolas Franceschini, Franck Ruffier and Julien Serres 4 Active Vision in Blowflies: Strategies and Mechanisms of Spatial Orientation ...... 51 Martin Egelhaaf, Roland Kern, Jens P. Lindemann, Elke Braun, and Bart Geurten 5 Wide-Field Integration Methods for Visuomotor Control ...... 63 J. Sean Humbert, Joseph K. Conroy, Craig W. Neely, and Geoffrey Barrows 6 Optic Flow to Steer and Avoid Collisions in 3D ...... 73 Jean-Christophe Zufferey, Antoine Beyeler, and Dario Floreano 7 Visual Homing in Insects and Robots ...... 87 Jochen Zeil, Norbert Boeddeker, and Wolfgang Stürzl 8 Motion Detection Chips for Robotic Platforms ...... 101 Rico Moeckel and Shih-Chii Liu 9 Insect-Inspired Odometry by Optic Flow Recorded with Optical Mouse Chips ...... 115 Hansjürgen Dahmen, Alain Millers, and Hanspeter A. Mallot 10 Microoptical Artificial Compound Eyes ...... 127 Andreas Brückner, Jacques Duparré, Frank Wippermann, Peter Dannberg, and Andreas Bräuer 11 Flexible Wings and Fluid–Structure Interactions for Micro-Air Vehicles ...... 143 W. Shyy, Y. Lian, S.K. Chimakurthi, J. Tang, C.E.S. Cesnik, B. Stanford, and P.G. Ifju

vii viii Contents

12 Flow Control Using Flapping Wings for an Efﬁcient Low-Speed Micro-Air Vehicle ...... 159 Kevin D. Jones and Max F. Platzer 13 A Passively Stable Hovering Flapping Micro-Air Vehicle ...... 171 Floris van Breugel, Zhi Ern Teoh, and Hod Lipson 14 The Scalable Design of Flapping Micro-Air Vehicles Inspired by Insect Flight ...... 185 David Lentink, Stefan R. Jongerius, and Nancy L. Bradshaw 15 Springy Shells, Pliant Plates and Minimal Motors: Abstracting the Insect Thorax to Drive a Micro-Air Vehicle ..... 207 Robin J. Wootton 16 Challenges for 100 Milligram Flapping Flight ...... 219 Ronald S. Fearing and Robert J. Wood 17 The Limits of Turning Control in Flying Insects ...... 231 Fritz-Olaf Lehmann 18 A Miniature Vehicle with Extended Aerial and Terrestrial Mobility . 247 Richard J. Bachmann, Ravi Vaidyanathan, Frank J. Boria, James Pluta, Josh Kiihne, Brian K. Taylor, Robert H. Bledsoe, Peter G. Ifju, and Roger D. Quinn 19 Towards a Self-Deploying and Gliding Robot ...... 271 Mirko Kovac,ˇ Jean-Christophe Zufferey, and Dario Floreano 20 Solar-Powered Micro-air Vehicles and Challenges in Downscaling .. 285 André Noth and Roland Siegwart 21 Technology and Fabrication of Ultralight Micro-Aerial Vehicles ... 299 Adam Klaptocz and Jean-Daniel Nicoud Contributors

Richard J. Bachmann BioRobots, LLC, Cleveland, USA Geoffrey Barrows Centeye, Washington, DC, USA, [email protected] Antoine Beyeler Laboratory of Intelligent Systems, EPFL, Lausanne, Switzerland Robert H. Bledsoe United States Marine Corps and the Naval Postgraduate School, USA Norbert Boeddeker Department of Neurobiology and Center of Excellence ‘Cognitive Interaction Technology’, Bielefeld University, Bielefeld, Germany, [email protected] Frank J. Boria Department of Mechanical Engineering at the University of Florida, USA Nancy L. Bradshaw Faculty of Aerospace Engineering, Delft University of Technology, 2629 HS Delft, The Netherlands; Experimental Zoology Group, Wageningen University, 6709 PG Wageningen, The Netherlands Andreas Bräuer Fraunhofer Institute for Applied Optics and Precision Engineering, Albert-Einstein-Str. 7, D-07745, Jena, Germany Elke Braun Department of Neurobiology & Center of Excellence “Cognitive Interaction Technology”, Bielefeld University, D-33501 Bielefeld, Germany Floris van Breugel Cornell Computational Synthesis Lab, Cornell University, Ithaca, New York, USA, [email protected] Andreas Brückner Fraunhofer Institute for Applied Optics and Precision Engineering, Albert-Einstein-Str. 7, D-07745, Jena, Germany, [email protected] C.E.S. Cesnik Department of Aerospace Engineering, University of Michigan, Ann Arbor, Michigan, USA, [email protected] S.K. Chimakurthi Department of Aerospace Engineering, University of Michigan, Ann Arbor, Michigan, USA, [email protected] Joseph K. Conroy Autonomous Vehicle Laboratory, University of Maryland, College Park, MD, USA, [email protected] Hansjürgen Dahmen Cognitive Neurosciences, University of Tübingen, Germany, [email protected]

ix x Contributors

Peter Dannberg Fraunhofer Institute for Applied Optics and Precision Engineering, Albert-Einstein-Str. 7, D-07745, Jena, Germany Jacques Duparré Fraunhofer Institute for Applied Optics and Precision Engineering, Albert-Einstein-Str. 7, D-07745, Jena, Germany Martin Egelhaaf Department of Neurobiology & Center of Excellence “Cognitive Interaction Technology”, Bielefeld University, D-33501 Bielefeld, Germany, [email protected] Ronald S. Fearing Biomimetic Millisystems Lab, University of California, Berkeley, CA, USA, [email protected] Dario Floreano Laboratory of Intelligent Systems, EPFL, Lausanne, Switzerland, Dario.Floreano@epfl.ch Nicolas Franceschini Biorobotics Lab, Institute of Movement Science, CNRS & Univ of the Mediterranean, Marseille, France, [email protected] Steven N. Fry Institute of Neuroinformatics (INI) & Institute of Robotics and Intelligent Systems (IRIS) Ð Swiss Federal School of Technology Zürich, Switzerland Bart Geurten Department of Neurobiology & Center of Excellence “Cognitive Interaction Technology”, Bielefeld University, D-33501 Bielefeld, Germany J. Sean Humbert Autonomous Vehicle Laboratory, University of Maryland, College Park, MD, USA, [email protected] Peter G. Ifju Department of Mechanical and Aerospace Engineering, University of Florida, Gainsville, FL, USA, ifju@ufl.edu Kevin D. Jones Naval Postgraduate School, Monterey, CA, USA, [email protected] Stefan R. Jongerius Faculty of Aerospace Engineering, Delft University of Technology, 2629 HS Delft, The Netherlands Roland Kern Department of Neurobiology & Center of Excellence “Cognitive Interaction Technology”, Bielefeld University, D-33501 Bielefeld, Germany Josh Kiihne United States Marine Corps and the Naval Postgraduate School, USA Adam Klaptocz Lab of Intelligent Systems, EPFL, Lausanne, Switzerland, adam.klaptocz@epfl.ch Mirko Kovac Laboratory of Intelligent Systems, EPFL, Lausanne, Switzerland, Mirko.Kovac@epfl.ch Fritz-Olaf Lehmann Institute of Neurobiology, University of Ulm, Albert-Einstein-Allee 11, 89081 Ulm, [email protected] David Lentink Experimental Zoology Group, Wageningen University, 6709 PG Wageningen, The Netherlands; Faculty of Aerospace Engineering, Delft University of Technology, 2629 HS Delft, The Netherlands, [email protected] Y. Lian Department of Aerospace Engineering, University of Michigan, Ann Arbor, Michigan, USA, [email protected] Contributors xi

Jens P. Lindemann Department of Neurobiology & Center of Excellence “Cognitive Interaction Technology”, Bielefeld University, D-33501 Bielefeld, Germany Hod Lipson Cornell Computational Synthesis Lab, Cornell University, Ithaca, New York, USA, [email protected] Shih-Chii Liu Institute of Neuroinformatics, University of Zürich and ETH Zürich, Zürich, Switzerland, [email protected] Hanspeter A. Mallot Cognitive Neurosciences, University of Tübingen, Germany, [email protected] Alain Millers Cognitive Neurosciences, University of Tübingen, Germany, [email protected] Rico Moeckel Institute of Neuroinformatics, University of Zürich and ETH Zürich, Zürich, Switzerland, [email protected] Craig W. Neely Centeye, Washington, DC, USA, [email protected] Jean-Daniel Nicoud Didel SA, Belmont, Switzerland, [email protected] André Noth Autonomous Systems Lab, ETHZ, Zürich, Switzerland, [email protected], [email protected] Max F. Platzer Naval Postgraduate School, Monterey, CA, USA, [email protected] James Pluta United States Marine Corps and the Naval Postgraduate School, USA Roger D. Quinn Department of Mechanical and Aerospace Engineering at Case Western Reserve University, Cleveland, USA, [email protected] Franck Ruffier Biorobotics Lab, Institute of Movement Science, CNRS & Univ of the Mediterranean, Marseille, France, franck.ruffi[email protected] Julien Serres Biorobotics Lab, Institute of Movement Science, CNRS & Univ of the Mediterranean, Marseille, France, [email protected] W. Shyy Department of Aerospace Engineering, University of Michigan, Ann Arbor, Michigan, USA, [email protected] Roland Siegwart Autonomous Systems Lab, ETHZ, Zürich, Switzerland, [email protected] Dean Soccol ARC Centre of Excellence in Vision Science, Queensland Brain Institute, University of Queensland, St. Lucia, QLD 4072, Australia, [email protected] Mandyam V. Srinivasan ARC Centre of Excellence in Vision Science, Queensland Brain Institute, University of Queensland, St. Lucia, QLD 4072, Australia, [email protected] B. Stanford Department of Mechanical and Aerospace Engineering, University of Florida, Gainsville, FL, USA, bstan@ufl.edu xii Contributors

Wolfgang Stürzl Department of Neurobiology and Center of Excellence ‘Cognitive Interaction Technology’, Bielefeld University, Bielefeld, Germany, [email protected] J. Tang Department of Aerospace Engineering, University of Michigan, Ann Arbor, Michigan, USA, [email protected] Brian K. Taylor Department of Mechanical and Aerospace Engineering at Case Western Reserve University, Cleveland, USA Zhi Ern Teoh Cornell Computational Synthesis Lab, Cornell University, Ithaca, New York, USA, [email protected] Saul Thurrowgood ARC Centre of Excellence in Vision Science, Queensland Brain Institute, University of Queensland, St. Lucia, QLD 4072, Australia, [email protected] Ravi Vaidyanathan Department of Mechanical Engineering at the University of Bristol, Bristol, UK; Department of Systems Engineering at the Naval Postgraduate School, CA, USA, [email protected] Frank Wippermann Fraunhofer Institute for Applied Optics and Precision Engineering, Albert-Einstein-Str. 7, D-07745, Jena, Germany Robert J. Wood Harvard Microrobotics Laboratory, Harvard University, Cambridge, MA, USA Robin Wootton School of Biological Sciences, Exeter University, Exeter EX4 4PS, UK, [email protected] Jochen Zeil ARC Centre of Excellence in Vision Science and Centre for Visual Sciences, Research School of Biological Sciences, The Australian National University, Biology Place, Canberra, ACT 2601, Australia, [email protected] Jean-Christophe Zufferey Laboratory of Intelligent Systems, EPFL, Lausanne, Switzerland, Jean-Christophe.Zufferey@epﬂ.ch Chapter 1 Experimental Approaches Toward a Functional Understanding of Insect Flight Control

Steven N. Fry

Abstract This chapter describes experimental 1.1 Introduction approaches exploring free-flight control in insects at various levels, in view of the biomimetic design Flying insects achieve efficient and robust flight con- principles they may offer for MAVs. Low-level flight trol despite size constraints and hence limited neu- control is addressed with recent studies of the aerody- ral resources [6, 12]. This is achieved from closely namics of free-flight control in the fruit fly. The ability integrated and often highly specialized sensorimotor to measure instantaneous kinematics and aerodynamic control pathways [19], making insects an ideal model forces in free-flying insects provides a basis for the system for the identification of biological flight con- design of flapping airfoil MAVs. Intermediate-level trol mechanisms, which can serve as design principles flight control is addressed by presenting a behavioral for future autonomous micro-air vehicles (MAVs) [18, system identification approach. In this work, the 16, 38, 62]. While the implementation of biomimetic motion processing and speed control pathways of the design principles in MAVs and other technical devices fruit fly were reverse engineered based on transient is inherently appealing, such an approach has its pit- visual flight speed responses, providing a quantitative falls that can easily lead to misconceptions [54]. control model suited for biomimetic implementation. A first problem relates to the immense complex- Finally, high-level flight control is addressed with ity of biological systems, in particular flight con- the analysis of landmark-based goal navigation, for trol mechanisms. The multimodal sensorimotor path- which bees combine and adapt basic visuomotor ways represent a high-dimensional control system, reflexes in a context-dependent way. Adaptive control whose function and underlying physiology are under- strategies are also likely suited for MAVs that need to stood only partially. A second problem relates to perform in complex and unpredictable environments. the often substantially different spatial and tempo- The integrative analysis of flight control mechanisms ral scales of insects and MAVs. A meaningful trans- in free-flying insects promises to move beyond iso- fer of a control mechanism identified in a small lated emulations of biological subsystems toward a insect to its typically much larger robotic counter- generalized and rigorous approach. part is non-trivial and requires detailed knowledge of the system dynamics. For example, it is not obvious how to control a robot based on motion processing principles derived from insects [41, 62], which perform maneuvers much faster and based on completely different locomotion principles than their robotic counterparts. S.N. Fry () This chapter presents recent experimental appro- Institute of Neuroinformatics, University of Zurich and ETH aches aimed at a functional understanding of insect Zurich; Institute of Robotics and Intelligent systems, ETH flight control mechanisms. To this end, flight con- Zurich trol in insects is addressed at various levels, from the e-mail: [email protected]

D. Floreano et al. (eds.), Flying Insects and Robots, 1 DOI 10.1007/978-3-540-89393-6_1, © Springer-Verlag Berlin Heidelberg 2009 2 S.N. Fry biomechanics of flapping flight to flight control strate- equipped with virtual reality technology [27]. The gies and high-level navigational control. The experi- identification of the control dynamics in the form of mental approaches share in common a detailed anal- a controller provides a powerful strategy to transfer ysis of the time-continuous processes underlying the biological control principles into the robotic con- control of free flight under highly controlled and yet text, including MAVs [17, 16], also see Chap. 3. meaningful experimental conditions. ¥ High level: Landmark navigation. Autonomous MAVs should ultimately be able to flexibly solve meaningful tasks, such as navigate through cluttered, unpredictable, and potentially dangerous 1.1.1 Chapter Overview environments, and safely return to their base. Here, too, insects can serve as a model system, as some ¥ Low level: Biomechanics. A formidable challenge species show the impressive ability to acquire the for the design of MAVs is to generate sufficient knowledge of specific locations in their environ- aerodynamic forces to remain aloft, while control- ment (e.g., nest, food site), which they repeatedly ling these forces to stabilize flight and perform visit over the course of many days [56], also see maneuvers. The sensorimotor system of insects has Chaps. 2 and 7. evolved under size constraints that may be quite The third example in this chapter describes an similar to those of MAVs. Consequently, the biolog- experimental approach aimed to explain landmark- ical solutions enabling flight in these small animals based goal navigation in honey bees from a detailed may provide useful design principles for the imple- analysis of individual maneuvers. Goal navigation mentation of MAVs. is explained with basic sensorimotor control mech- The first example in this chapter addresses low- anisms that are combined and modified through the level flight control with a detailed description of learning experience. The ability to achieve robust, free-flight biomechanics in the fruit fly Drosophila adaptive, and flexible flight control as an emergent [29, 30]. 3D high-speed videography and dynamical property of basic sensorimotor control principles force scaling were combined to resolve the move- offers yet more interesting options for the design ments of the wings and the resulting aerodynamic of autonomous MAVs with limited built-in control forces. The time-resolved analysis reveals aerody- circuits. namic and control requirements of insect flight, which are likewise essential to MAV design [60, 17, 16, 58], also see Chaps. 11Ð16. 1.2 Low-Level Flight Control – ¥ Intermediate level: Visuomotor reflexes. To navigate autonomously in a cluttered environment, Biomechanics of Free Flight MAVs need to sense objects and produce appropriate responses, such as to avoid an impending A detailed knowledge of flight biomechanics provides collision. Insects meet this challenge with reflex- the foundation for our understanding of biological ive responses to the optic flow [33], i.e., the per- flight control strategies Ð or their implementation in ceived relative motion of the environment during MAVs. Flight control is ideally studied in free flight, flight. The so-called optomotor reflexes mediate in which the natural flight behavior of an insect can various visual flight responses, including attitude be measured under realistic sensory and dynamic flight control, collision avoidance, landing, as well as con- conditions. To understand how a flapping insect stabi- trol of heading, flight speed, and altitude. Optomo- lizes its flight and performs maneuvers, the underlying tor reflexes provide a powerful model system to mechanisms must be studied at the level of single wing explore visual processing and flight control princi- strokes Ð not an easy task, considering the tiny forces ples, reviewed in [12, 21, 46, 50], also see Chaps. 2, and short timescales involved. 4, and 17. The example described in this section shows how The second example in this chapter describes the application of 3D high-speed videography and a rigorous system analysis of the fruit fly’s visual dynamic force scaling using a robotic fly wing allowed flight speed response using TrackFly, a wind tunnel such a detailed analysis of free-flight biomechanics to 1 Experimental Approaches to Insect Flight Control 3 be performed in the fruit fly Drosophila, a powerful by a black cylindrical cup filled with vinegar, the flies model system for the design of flapping airfoil MAVs approached the center of the chamber, where they often [18, 17, 16, 58]. hovered before landing on the cup or instead performed a fast turning maneuver (saccade) in order to avoid colliding with it. These flight sequences were filmed 1.2.1 Research Background using three orthogonally aligned high-speed cameras, whose lines of sight intersected in the middle of the flight chamber. The aerodynamic basis of insect flight has remained Next, the wing and body kinematics were extracted enigmatic due to the complexities related to the intrin- using a custom-programmed graphical user interface sically unsteady nature of flapping flight [57]. A solid (Fig. 1.1B). The wing kinematics were then played theoretical basis for quantitative analyses of insect through a dynamically scaled robotic wing (Robofly, flight aerodynamics was provided by Ellington’s influ- Fig. 1.1C) to measure the aerodynamic wing forces ential theoretical work based on time-averaged mod- (arrows in Fig. 1.1B) throughout the filmed flight els [22]. At the experimental level, dynamically scaled sequence. Combining the kinematic and force data robotic wings provided the technological breakthrough allowed a direct calculation of instantaneous aerody- allowing aerodynamic effects to be explored empiri- namic forces, torques, and power (see below). cally at the timescale of a single wing stroke [44].

1.2.2.1 Hovering Flight

1.2.2 Experiments Hovering flight offers itself for an analysis of the aerodynamic requirements of flapping flight without the To perform a time-resolved biomechanical analysis complications resulting from body motion. The anal- of free flight, the wing and body movements of fruit ysis of such a hovering sequence, consisting of six flies were recorded using 3D high-speed videography consecutive wing strokes, is shown in Fig. 1.2. The (Fig. 1.1A). For this, hungry flies were released into precisely controlled wing movements are character- a small flight chamber (side length 30 cm). Attracted ized by a high angle of attack and maximal force

A B C Motors

HS Wing

Head Mineral oil C Fly’s path HS HS Abdomen Model wing Aerodynamic LED force

0.8 m

Fig. 1.1 Measurement of kinematics and forces. (A) Setup: positions of the head and abdomen to obtain their 3D positions. Flies were filmed with three orthogonally aligned high-speed Arrows show the aerodynamic forces measured using Robofly. (5000 fps) cameras (HS). Arrays of near-infrared light emitting (C) “Robofly”: Plexiglas wings (25 cm in length) were flapped diodes (LEDs) were used for back-lighting. Flies were attracted in mineral oil at the appropriate frequency to match the Reynolds to a small cylindrical cup (C), in front of which they were number of the fly’s flapping wings in air (and hence the fluid filmed within the small overlapping field of view of the cameras dynamics). The up-scaled fluid dynamic forces were measured (shown as a wire-frame cube). (B) Kinematic extraction: Wing with strain gauges on the wings and the aerodynamic forces act- and body kinematics were measured using a graphical user inter- ing on the fly’s wings (shown in B) were back-calculated. Figure face, which allowed a user to match the wing silhouettes and the modified from [30] 4 S.N. Fry

A C 1.5 Measured Model 1 Stroke angle 90 0.5 0 Net force (N) ation Devi- (deg) –90 0 of Angle attack D 5 Lift

N) 5

–5 0 Thrust Flight forces –5 (10 Nm) –8 Down Up 0 10 20 30 0 Time (ms) Pitch torque (10 –5 B –5 E 1 × 10 N 500 Total ) –1

Down Aerodyn.

Up Power (W kg Intertial –200 Down Up Stroke cycle

Fig. 1.2 Hovering flight. (A) Wing kinematics and flight forces. instantaneous aerodynamic force. The measured force is shown Data from six consecutive wing beats are shown. For a definition together with the force predicted by a quasi-steady model [20]. of the stroke angles refer to [30]. (B)Wingmotionandforces. (D) Instantaneous pitch torque. Pitch torque (black line ± S.D.) Successive wing positions during a hovering stroke cycle are oscillates around a mean of zero during hovering. (E) Instanta- shown with matchstick symbols (dots show the leading edge). neous specific flight power (in W/kg muscle mass). Traces show Instantaneous aerodynamic forces are shown with arrows.Axes total mechanical power (±S.D.), which is composed of the aero- indicate horizontal (± 90◦) and vertical (± 10◦) stroke positions. dynamic and inertial power required for wing motion. Figure The arrow shown with the fly shows the aerodynamic force modified from [30] averaged over the stroke cycle. (C) Quasi-steady analysis of production during the middle of the downstroke and generated throughout the stroke cycle must cancel each the early upstroke (Fig. 1.2A, B). As expected for hov- other out precisely, as shown in Fig. 1.2D. Finally, ering, the drag of the downstroke and upstroke can- the instantaneous power was calculated directly from cels itself out, while the mean lift offsets the fly’s the scalar product of wing velocity and the forces act- weight. The aerodynamic forces generated by the wing ing on the wings (Fig. 1.2E). The power associated movements are largely explained with a quasi-steady with aerodynamic force production peaks around the model that takes into account translational and rota- middle of each half-stroke, when aerodynamic forces tional effects of the wing motion (Fig. 1.2C). The main and wing velocity are maximal. Conversely, the power discrepancy between the modeled and measured forces required to overcome wing inertia reverses its sign is a phase delay, which is likely due to unsteady effects during each half-stroke due to the deceleration of the (e.g., wingÐwake interactions) not considered here. wings toward the end of each half-stroke. The total A further requirement of stable hovering flight is mechanical power, the sum of these two components, a precise balance of the aerodynamic torques over is positive for most part of the wing stroke. Power is the course of a wing stroke. To maintain a constant negative when the wings decelerate while producing body pitch, for example, the substantial torque peaks little aerodynamic force, which occurs briefly toward 1 Experimental Approaches to Insect Flight Control 5 the end of each half-stroke. During this phase, the 1.2.3 Conclusions mechanical power could be stored elastically and partially retrieved during the subsequent half-stroke to The physical constraints may be similar for MAVs and reduce the total power requirements. The potential flies operating at similar size scales, and the flight con- reduction of flight power in fruit flies, however, is quite trol mechanisms evolved in insects can therefore pro- limited (in the order of 10%). vide valuable design principles for MAVs. The application of high-speed videography and aerodynamic force measurements using dynamically scaled robots pro- 1.2.2.2 Maneuvering vides detailed insights into the requirements of insect flight control that can help identify important design This approach was taken further to explore how flies constraints for MAVs. modify their wing kinematics during flight maneuvers. This analysis reveals critical aspects of flight control Flight sequences containing saccadic turning maneu- in Drosophila that need to be considered also for MAV vers were filmed and the aerodynamic wing forces design. Precise and fast wing actuation appears most again measured using Robofly. Figure 1.3A shows critical for flight control. As shown by the example wing tip trajectories during such maneuvers, labeled of pitch torque, the instantaneous torques produced by according to the yaw torque they produced. At the the wings vary considerably and must be precisely bal- onset of a turn, the outside wing tilts backward and its anced over the course of a stroke cycle. As shown by amplitude increases (light gray tip trajectories). Con- the analysis of yaw torque during turning maneuvers, versely, the inside wing tilts forward and its amplitude even subtle changes in wing motion are sufficient to decreases (dark gray trajectories). The resulting dif- induce fast turns within a few wing beats. Precise and ference in yaw torque generated by the two wings is fast sensorimotor control loops are obviously required sufficient to accelerate the fly to over 1000◦/s within for flight control using similar morphologies. about five wing strokes [29]. The changes in stroke The experiments also indicate less critical features plane angle and stroke amplitude over the course of a of flapping flight control, at least at the size scale of saccade are shown in Fig. 1.3B. To maximally acceler- the fruit fly. Wing stiffness and surface structure, for ate at the onset of the saccade, the difference in stroke example, may be relatively unimportant under certain amplitude between the outside and inside wing is only conditions. The simple Plexiglas wing used in Robofly around 5◦, while the stroke plane angle differs by a was sufficient to reproduce the required aerodynamic mere 2◦. Even during extreme flight maneuvers, there- forces without the need to mimic the quite complicated fore, the changes in wing kinematics are quite small. structure of the fly’s wing. The feasibility of a stiff, Stroke plane angle (deg) ABYaw torque 15 Stroke 3 amplitude High 10 2 Low 5 1 0 0 –5 –1 30 deg Stroke –2

Stroke amplitude (deg) –10 plane angle

–20 –10 0 10 20 30 40

Time (ms)

Fig. 1.3 Maneuvering. (A) Changes in wing kinematics asso- respectively. (B) Bilateral changes in wing kinematics during ciated with yaw torque production during saccades. Wing tip a turning maneuver. At the onset of a turn, the outside wing trajectories were measured during free-flight turning maneuvers increases the stroke amplitude and stroke plane angle (backward (saccades). Wing tip trajectories associated with high and low tilt) relative to the inside wing. Figure modified from [29] aerodynamic yaw torques are shown as light and dark gray lines, 6 S.N. Fry light-weight wing structure for flapping lift production identification of visuomotor control pathways in the was recently demonstrated in [59]. As a further sim- fruit fly. The characterization of biological control plification, quasi-steady mechanisms of aerodynamic pathways in the form of a control model allows more force production dominate in the production of aerody- direct and meaningful transfer of biological flight con- namic forces of flapping wings, at least in the fruit fly. trol principles into a robotic context, including MAVs Unsteady effects, such as wingÐwing and wingÐwake [17, 16]. interactions, play a minor role, such that simple analyt- ical tools can be applied at least in first approximation. Finally, elastic storage plays a comparatively small role given the small mass of the wings, and therefore does 1.3.1 Research Background not present a significant design constraint. In conclusion, the measurement of instantaneous Pioneering experiments explored the transfer prop- wing positions in free flight, together with the aero- erties of optomotor turning reflexes in insects. This dynamic forces measured in the robotic wing, is suf- was achieved with a simple preparation, in which ficient to robustly quantify several relevant aspects insects were stimulated using a rotating drum and of flight biomechanics. The example of the fruit fly, their intended turning responses measured using ele- itself a powerful model for MAV design [61, 60], gant techniques [36, 23, 34]. The response tuning of reveals a suitable strategy to get flapping MAVs off the optomotor turning reflexes provides the foundation for ground. The next substantial challenge is to actively a cohesive theory of optic flow processing in insects to stabilize flight, for which exceedingly fast and pre- this day, reviewed in [6, 4, 37, 5], also see Chaps. 4 cise wing control is required. The impressive advances and 5. in flight biomechanics provide a solid foundation for While tethering provides a simple method to deliver biomimetic MAV design that takes into account the stimuli without influence of the behavioral reactions requirement for flight control. (referred to as open loop in the biological literature), the results of such experiments are difficult to interpret functionally [49]. Tethering disrupts various reafferent 1.3 Intermediate-Level Flight Control – feedback circuits, which leads to significant behavioral artifacts [30] and prevents the analysis of flight control Visuomotor Reflexes under realistic dynamical conditions. Visual reflexes have also been extensively explored An intermediate level of flight control involves reflex- in free flight, reviewed in [12, 6, 46], in which case ive responses to sensory input. On the one hand, the visual input is coupled to the insect’s flight behav- they can mediate corrective maneuvers to recover ior (natural closed-loop condition) [45, 49]. This cou- from disturbances and unstable flight conditions to pling hinders a rigorous system identification because increase dynamic system stability. On the other hand, the stimuli are no longer under complete experimen- they can mediate flight maneuvers to suitably respond tal control. Nevertheless, data obtained in this way can in an unpredictable environment. For example, an provide valuable insight into flight control mechanisms MAV equipped with motion sensors can sense an [40, 11, 3], Chap. 2. A simpler behavioral analysis object appearing in front and respond with an avoid- becomes possible from measuring free-flight behavior ance maneuver to prevent a collision. Equipped with under steady-state conditions [15, 47, 2], but cannot the appropriate sensors and flight control strategies, address questions relating to flight control dynamics. autonomous MAVs can navigate more safely and effi- ciently within cluttered and unpredictable environments (Chaps. 3 and 8). The extremely efficient and robust visuomotor 1.3.2 Experiments reflexes of insects can provide design principles for biomimetic flight control strategies in autonomous A functional understanding of biological flight con- MAVs. The second part of this chapter describes trol principles that can be meaningfully transferred behavioral experiments aimed at a rigorous system into MAVs requires careful consideration of the 1 Experimental Approaches to Insect Flight Control 7 multimodal reafferent pathways. Below I describe a respond to the linear velocity (TF/SF) of the patterns, recent experimental approach aimed at a system iden- which serves as a control signal for flight speed [27]. tification of visuomotor pathways that can serve as The visual tuning properties of visual flight speed design principles for MAV control. responses differ from optomotor turning responses, which instead show a response maximum at a particular temporal frequency (TF) of displayed patterns [34]. 1.3.2.1 System Analysis of Visual Flight Speed Next, system identification procedures were applied Control Using Virtual Reality Display to obtain a controller that was able to reproduce the Technology transient open-loop response properties, i.e., reproduce the transient changes in flight speed after onset To perform a system identification of the fruit fly’s of the optic flow stimulus. The controller was then motion-dependent flight speed control pathways, a used to predict the speed responses under more real- wind tunnel was equipped with virtual reality display istic visual closed-loop conditions, and the results con- technology (TrackFly [27], Fig. 1.4). firmed with data obtained from flies tested in closed An automated procedure was implemented to loop. A detailed quantitative account of the procedures induce flies to fly to the center of the wind tunnel and data is published elsewhere ([27], [28], Rohrseitz and then stimulate them with horizontally moving sine and Fry, in prep.). gratings of defined temporal frequency (TF), spatial frequency (SF), and contrast. To hold the linear image velocity (defined as TF/SF) constant in the fly’s eyes, 1.3.3 Conclusions the grating speed was adjusted continuously to compensate for the displacement of the fly (one-parameter The reflexive flight control pathways of insects can open-loop paradigm). The automated high-throughput provide powerful control architectures for biomimetic system allowed a large data set of visual responses MAVs. For a meaningful interpretation of the biologi- to be measured for a broad range of temporal and cal measurements, however, the behavioral context and spatial frequencies. The results show that fruit flies relevance of multimodal feedback must be carefully considered. Free-flight experiments are ideal to explore flight control under realistic flight conditions, but the difficulty of delivering arbitrary stimuli in a controlled M manner is a hindrance for detailed behavioral system LP identification. The described approach works around this limita- tion by allowing a particular parameter (here: pattern Cam Cam speed) to be presented in open loop, without disrupt- Projector ing the remaining stimuli. From the measured transient responses, linear pattern velocity was identified as the relevant control parameter for visual flight speed con- Wind trol. Based on this high-level understanding, the underlying visual computations and neural structures can be Fly further explored. LP Next, the transient responses were used to reverse M engineer the control scheme underlying flight speed control (Fig 1.5). The measurements performed in open and closed loop are quantitatively explained by Fig. 1.4 TrackFly. A wind tunnel was equipped with a virtual a proportional control law, which is simpler still than reality display system (only working section of the wind tunnel a PID controller recently suggested for insects [24], is shown). Visual stimuli were presented to free-flying flies in Chap. 3. open loop, i.e., the pattern offset was adjusted to the fly’s posi- A rigorous system identification approach in tion along the wind tunnel in real time. M: Mirror; LP: Light path; Cam: Video camera. For details see [27] biology provides a functional understanding of the 8 S.N. Fry

Pattern Speed Flight speed Wind tunnel slip speed signal command Control TF, SF Vision

Open/closed loop Locomotor limit

Flight speed Fig. 1.5 Control model. The fly’s flight speed responses are speed, which is constrained by measured locomotor limits. The quantitatively explained by a simple control model. The visual simulated flight speed responses in open and closed loop (note system computes pattern velocity as input to a controller of flight switch symbol) were verified experimentally using TrackFly underlying neuromotor pathways and characterizes 1.4.1 Research Background their dynamics in a concrete control model. The control strategy can then be meaningfully transferred into The mechanisms by which flying insects use land- MAVs even if the underlying neuromotor mechanisms marks to return to a learned place was pioneered by remain only partially known. Tinbergen’s (1932) [52] classic neuroethological studies in the digger wasp. His approach, followed by many later researchers (e.g., for flying honey bees [1, 7]; review [55]), was to induce search flights in experi- 1.4 High-Level Flight Control – mentally modified visual surroundings and conclude Landmark-Guided Goal Navigation from the search location the internal visual representation of the visual environment. High-level flight control strategies are ultimately A similar approach in honey bees performed half a required to enable MAVs to perform meaningful tasks, century later led to the influential snapshot model [8], for which sensory input must be processed in a context- which explained goal-directed flight control to result dependent way. For example, an MAV could rely on from a comparison between the current retinal image the same visual objects encountered along its flight and a template image formerly stored at the goal loca- path during the outbound trip and to return to its home tion. The ways in which insects represent locations as base, requiring a context-dependent processing of the visual memories and use these to return to a learned visual input. place are studied experimentally in increasing detail, Some insect species reveal the amazing ability to see recent reviews in [13, 10]. Not least due to its return to quite distant places that they previously vis- algorithmic formulation, the snapshot and related mod- ited. Honey bees, for example, learn the location of a els found widespread appeal in the robotic community rewarding food source, which they repeatedly visit to and were further explored in numeric [43] and robotic collect food for the hive (also see Chaps. 2 and 7). The [39, 42] implementations, reviewed in [25, 53]. third example of this chapter describes experiments exploring the basic control principles underlying such complex, context-dependent behaviors. 1.4.2 Experiments Landmark-based goal navigation is explained with visuomotor control mechanisms that are modified The experiments giving rise to the snapshot model through learning experience. Complex flight behaviors were first replicated and extended to explore landmark- result as an emergent property of basic flight control based goal navigation in more detail [31]. The results strategies and their interactions with the environment. were suggestive of alternative visuomotor control Similar control strategies could allow MAVs with lim- strategies, which were subsequently explored with ited resources to likewise perform well in complex, detailed analyses of individual approach flights using real-world applications. more advanced video tracking techniques [26]. 1 Experimental Approaches to Insect Flight Control 9

Ai Aii B

Initial training Feeder 5

0 Aiii Aiv Flight durations (s) 1204060 80 100 120 Flights

Fig. 1.6 Learning experiments. (A) Successive approach flights with arrows) closer toward the final feeder position on consec- of a single bee. The experiments were performed in a cylindrical utive foraging trips. Lines show the flight trajectories toward tent (¯ 2.4 m). The bee entered on the left and flew to an incon- the temporary and final feeder positions. (ii) Flights 1Ð20. (iii) spicuous feeder (location indicated with a stippled cross-hair) Flights 51Ð70. (iv) Flights 101Ð120. (B)Durationofsuccessive 0.5 m in front of a black paper square attached to the back wall flights. With increasing experience, flight duration decreased to (shown as a black bar on the right). (i) Initial training. The bee about 2 s after about 50 flights. Figure modified from [32] was trained by displacing a temporary feeder (location shown

Detailed measurements of approach flights of bees right visual field and performed more convoluted flight to a goal location were made, taking care not to dis- paths toward the goal (Fig. 1.7B). In the identical sit- rupt their natural behavior. To explore the relevance uation, a second bee approached the feeder with a dif- of learning, every single approach flight of a single ferent approach pattern, but like the first bee held the bee was measured (Fig. 1.6). By moving a tempo- cylinder in the right visual field (Fig. 1.7C). rary feeder stepwise through a uniform flight tent, This simple rule was even used by bees trained the bee was trained to fly toward a permanent feeder with two differently colored cylinders (marked L and located in front of a single landmark (Fig. 1.6 Ai). R in Fig. 1.7D). The bees simply relied on one of the The approach flights of this inexperienced bee were cylinders (in this case the right cylinder) for the ini- slow and quite convoluted. The first 20 flights of the tial approach, again holding it in the right visual field. same bee with the permanent feeder were faster, but The detailed structure of an approach flight, together still revealed turns and loops reminiscent of search with the measured body axis direction, is shown in flights (Fig. 1.6 Aii). The bee’s approaches became (Fig. 1.7E). progressively faster and smoother as it gained expe- These and other experiments provide a coherent rience during successive foraging trips (Fig. 1.6 Aiii). view on the visuomotor strategies employed by bees After about 100 flights, the bee approached the feeder to locate a goal in different environmental situations. with straight and fast trajectories. Duration of suc- Bees with little experience with a landmark setting cessive flights decreased from about 5Ð10 s to 2Ð3 s (Fig. 1.6 Ai, Aii) or in the absence of a suitable (i.e., (Fig. 1.6B). near-frontal) landmark (Fig. 1.7C) perform search-like Next, individual bees were trained using landmark flights. If a landmark is suitably located behind the goal settings that differed in position, number, and color of (Fig. 1.6), an experienced bee will simply head toward the landmarks. First, a bee was trained with a single it and find its goal. To do so, the bee needs to fixate the black cylinder (•) located just to the right of the feeder landmark in a frontal position, as symbolized with the (Fig. 1.7A, left; feeder location is marked with cross- curved arrows in Fig. 1.8A. If the bee tends to keep hairs). The bee approached the cylinder and performed the landmark in the right visual field (Fig. 1.8B), it occasional left turns, roughly aimed at the feeder posi- will tend to make turns to the left, as required for the tion. During these approaches, the bee held the cylin- final goal approach (Fig. 1.7). Finally, the bee can rely der roughly in the frontal-right visual field (Fig. 1.7A, on one of several landmarks during its goal approach, right). A different bee trained with the cylinder fur- if it associates it with the appropriate retinal position ther to the right side still held the cylinder in the and/or turning direction (Fig. 1.8C). These results are 10 S.N. Fry

A B 1 1 Frequency 0 Frequency 0 180 90 0 90 180 180 90 0 90 180 Azimuth distribution Azimuth distribution (deg) (deg) C 1 D L 1

R L R Frequency Frequency 0 0 180 90 0 90 180 180 90 0 90 180 Azimuth distribution Azimuth distribution (deg) (deg) E

Fig. 1.7 Approach flights and landmark azimuth during ent bee. (D) Approach flights of two individual bees in the pres- approach flights. (A) Left: 40 successive approach flights of an ence of two cylinders of different colors. The bees headed toward individual bee with a cylinder (•) positioned at an angular dis- the right (R) cylinder. (E) Typical example of a bee’s approach tance of 15◦ from the feeder (+). Right: Distribution of landmark flight. The bee’s position (dots) and body axis direction (lines) positions in the bee’s visual field. (B) Cylinder placed 40◦ to were measured at 50 Hz using a pan-tilt tracking system [26]. the right of the feeder. (C)Asin(B), with data from a differ- Data are subsampled for clarity. Figure modified from [32]

Landmark ACB

Feeder 1 2 1

1 1 2 1 1 2 Flight path

Fig. 1.8 Visuomotor guidance model. (A) Frontal landmark. ing the approach. Lines show hypothetical flight paths. Curved Bees fixate the landmark frontally. (B) Lateral landmark. Bees arrows symbolize a learned visuomotor association. For details perform biased turns to the left, keeping the cylinder in the right see [32] visual field. (C) Two landmarks. Bees use one landmark dur- consistent with various experiments performed in both process. Relatively unstructured, search-like flights are flying and walking insects, e.g., [9, 35]. observed in bees with limited experience with the landmark setting. A suitable landmark is used to direct the flight toward the goal, while successful flight motor patterns are reinforced by operant learning to increase 1.4.3 Conclusions the efficiency and reliability of the approach flights. Control strategies based on a flexible and adapt- Landmark-based goal navigation in honey bees is able employment of basic control loops may also be explained as an emergent property of basic visuomotor suited for MAVs with limited storage and processing reflexes, which are modified by a continuous learning capacity to enable successful landmark navigation. A 1 Experimental Approaches to Insect Flight Control 11 possible scenario could consist of exploring unfamil- robotic environment with appropriate consideration of iar terrain and subsequently patrolling along suitable the behavioral context and scaling issues. routes. The flexible use of comparatively basic visuo- The presented research examples motivate closer motor control strategies can more likely meet the high interactions between biological research of flight con- requirements for fast and robust flight control required trol mechanisms and engineering design of MAVs. by MAVs than a single complex and hard-wired algo- Such interactions promise significant benefits to both rithm. Flexible adaptation to varying environmental fields, in that biologists can aim at more rigorous quan- conditions and increasing experience provides a pow- titative analyses of flight control and engineers can erful strategy for flight control in complex and unpre- aim at more meaningful biomimetic implementations. dictable environments. Indeed, this aim will likely have been reached when the common fascination about flight control becomes the defining element of a coherent, interdisziplinary research effort. 1.5 Closing Words Acknowledgments I wish to thank to reviewers for useful com- Insects perform complex flight control tasks despite ments, Chauncey Grätzel for advice on the writing, Jan Bar- tussek, Vasco Medici and Nicola Rohrseitz for useful com- their small size and presumably limited neural capac- ments and discussions. The work described in this chapter was ity. The fact that insects nevertheless excel in their funded by the following institutions: Human Frontiers Science flight performance is explained with a close integration Program (HFSP), Swiss Federal Institute of Technology (ETH) of specialized sensorimotor pathways. Zurich; Swiss National Science Foundation (SNSF), University of Zurich and Volkswagen Foundation. While it is intriguing to take inspiration from flying insects for the design of autonomous MAVs, the high complexity of an insect’s multimodal flight control system renders this task non-trivial and prone to References misconceptions [54, 14, 50]. It may therefore be hardly fruitful Ð and indeed counter-productive Ð to take 1. Anderson, A.: A model for landmark learning in the honey- superficial inspiration from biology and implement the bee. Journal of Comparative Physiology A 114, 335Ð355 purported principles in robots without due care. (1977) Instead, biologists and engineers should take advan- 2. Baird, E., Srinivasan, M.V., Zhang, S., Cowling, A.: Visual tage of the fact that insects can achieve superior flight control of flight speed in honeybees. Journal of Experimen- tal Biology 208(20), 3895Ð3905 (2005) control with possibly quite basic, but highly integrated 3. Boeddeker, N., Kern, R., Egelhaaf, M.: Chasing a dummy control principles. To this end, detailed biological stud- target: Smooth pursuit and velocity control in male ies are required that address flight control mecha- blowflies. Proceedings of the Royal Society of London nisms at various levels, including biomechanics, neural Series B Biological Sciences 270(1513), 393Ð399 (2003) 4. Borst, A., Egelhaaf, M.: Principles of visual motion detec- processing, sensorimotor integration, and high-level tion. Trends in Neurosciences 12(8), 297Ð306 (1989) behavioral strategies. The experimental approaches 5. Borst, A., Haag, J.: Neural networks in the cockpit of the described in this chapter show that advanced con- fly. Journal of Comparative Physiology A 188(6), 419Ð37 cepts and technologies can help provide the functional (2002) 6. Buchner, E.: Behavioral analysis of spatial vision in insects. understanding of biological flight control principles In: M.A. Ali (ed.) Photoreception and Vision in Inverte- required for meaningful biomimetic implementations brates, pp. 561Ð621. Plenum Press, New York (1984) in MAVs. 7. Cartwright, B.A., Collett, T.S.: How honey bees use land- Not only can engineers profit from rigorous bio- marks to guide their return to a food source. Nature 295, 560Ð564 (1982) logical research of flight control, but the concepts 8. Cartwright, B.A., Collett, T.S.: Landmark learning in bees: and tools applied in engineering [17, 16] can like- Experiments and models. Journal of Comparative Physiol- wise be meaningfully applied to explore biological ogy A 151, 521Ð543 (1983) control principles in a rigorous and quantitative way 9. Chittka, L., Kunze, J., Shipman, C., Buchmann, S.L.: The significance of landmarks for path integration in hom- [51, 48, 49, 27]. The control principles thus identified ing honeybee foragers. Naturwissenschaften 82, 341Ð343 in insects can then be transferred more easily into a (1995) 12 S.N. Fry

10. Collett, T.S., Graham, P., Harris, R.A., Hempel de Ibarra, 29. Fry, S.N., Sayaman, R., Dickinson, M.H.: The aerody- N.: Navigational memories in ants and bees: Memory namics of free-flight maneuvers in Drosophila. Science retrieval when selecting and following routes. Advances in 300(5618), 495Ð498 (2003) the Study of Behavior 36, 123Ð172 (2006) 30. Fry, S.N., Sayaman, R., Dickinson, M.H.: The aerodynam- 11. Collett, T.S., Land, M.F.: Visual control of flight behaviour ics of hovering flight in Drosophila. Journal of Experimen- in the hoverfly Syritta pipiens L. Journal of Comparative tal Biology 208(12), 2303Ð2318 (2005) Physiology 99, 1Ð66 (1975) 31. Fry, S.N., Wehner, R.: Honey bees store landmarks in 12. Collett, T.S., Nalbach, H.O., Wagner, H.: Visual stabiliza- an egocentric frame of reference. Journal of Comparative tion in arthropods. Reviews of Oculomotor Research 5, Physiology A 187(12), 1009Ð1016 (2002) 239Ð63 (1993) 32. Fry, S.N., Wehner, R.: Look and turn: Landmark-based goal 13. Collett, T.S., Zeil, J.: Places and landmarks: An arthro- navigation in honey bees. Journal of Experimental Biology pod perspective. In: S. Healy (ed.) Spatial Representation 208(20), 3945Ð3955 (2005) in Animals, pp. 18Ð53. Oxford University Press, Oxford, 33. Gibson, J.J.: The visual perception of objective motion and New York (1998) subjective movement. 1954. Psychological Review 101(2), 14. Datteri, E., Tamburrini, G.: Biorobotic experiments for the 318Ð23 (1994) discovery of biological mechanisms. Philosophy of Science 34. Götz, K.G.: Optomotorische Untersuchung des visuellen 74(3), 409Ð430 (2007) Systems einiger Augenmutanten der Fruchtfliege 15. David, C.T.: Compensation for height in the control of Drosophila. Kybernetik 2, 77Ð92 (1964) groundspeed by Drosophila in a new, ’barber’s pole’ wind 35. Graham, P., Fauria, K., Collett, T.S.: The influence of tunnel. Journal of Comparative Physiology A 147, 485Ð493 beacon-aiming on the routes of wood ants. Journal of (1982) Experimental Biology 206(3), 535Ð541 (2003) 16. Deng, X.Y., Schenato, L., Sastry, S.S.: Flapping flight 36. Hassenstein, B., Reichardt, W.: Systemtheoretische Anal- for biomimetic robotic insects: Part II - Flight control yse der Zeit-, Reihenfolgen- und Vorzeichenauswertung bei design. IEEE Transactions on Robotics 22(4), 789Ð803 der Bewegungsperzeption des Rüsselkäfers Chlorophanus. (2006) Zeitschrift für Naturforschung 11b, 513Ð524 (1956) 17. Deng, X.Y., Schenato, L., Wu, W.C., Sastry, S.S.: Flap- 37. Hausen, K.: Decoding of retinal image flow in insects. ping flight for biomimetic robotic insects: Part I - System Reviews of Oculomotor Research 5, 203Ð35 (1993) modeling. IEEE Transactions on Robotics 22(4), 776Ð788 38. Jeong, K.H., Kim, J., Lee, L.P.: Biologically inspired artifi- (2006) cial compound eyes. Science 312(5773), 557Ð561 (2006) 18. Dickinson, M.H.: Bionics: Biological insight into mechan- 39. Lambrinos, D., Möller, R., Labhart, T., Pfeifer, R., Wehner, ical design. Proceedings of the National Academy of Sci- R.: A mobile robot employing insect strategies for naviga- ences of the United Statesof America 96(25), 14, 208Ð9. tion. Robotics and Autonomous Systems 30, 39Ð64 (2000) (1999) 40. Land, M.F., Collett, T.S.: Chasing behaviour of houseflies 19. Dickinson, M.H.: Insect flight. Current Biology 16(9), (Fannia canicularis). Journal of Comparative Physiology A R309ÐR314 (2006) 89, 331Ð357 (1974) 20. Dickson, W.B., Dickinson, M.H.: The effect of advance 41. Liu, S.C., Usseglio-Viretta, A.: Fly-like visuo-motor ratio on the aerodynamics of revolving wings. Journal of responses on a robot using aVLSI motion chips. Biologi- Experimental Biology 207(24), 4269Ð4281 (2004) cal Cybernetics 85(6), 449Ð457 (2001) 21. Egelhaaf, M., Kern, R.: Vision in flying insects. Current 42. Möller, R.: Insect visual homing strategies in a robot with Opinion in Neurobiology 12(6), 699Ð706 (2002) analog processing. Biological Cybernetics 83, 231Ð243 22. Ellington, C.P.: The aerodynamics of hovering insect flight. (2000) Philosophical Transactions of the Royal Society B: Biolog- 43. Möller, R.: Do insects use templates or parameters for land- ical Sciences 305, 1Ð181 (1984) mark navigation? Journal of Theoretical Biology 210, 33Ð 23. Fermi, G., Reichardt, W.: Optomotorische Reaktionen der 45 (2001) Fliege Musca domestica. Kybernetik 2, 15Ð28 (1963) 44. Sane, S.P., Dickinson, M.H.: The aerodynamic effects of 24. Franceschini, N., Ruffier, F., Serres, J.: A bio-inspired fly- wing rotation and a revised quasi-steady model of flapping ing robot sheds light on insect piloting abilities. Current flight. Journal of Experimental Biology 205(8), 1087Ð1096 Biology 17(4), 329Ð335 (2007) (2002) 25. Franz, M.O., Schöllkopf, B., Mallot, H.A., Bülthoff, H.H.: 45. Schuster, S., Strauss, R., Götz, K.G.: Virtual-reality tech- Where did i take that snapshot? Scene based homing niques resolve the visual cues used by fruit flies to evaluate by image matching. Biological Cybernetics 79, 191Ð202 object distances. Current Biology 12(18), 1591Ð4 (2002) (1998) 46. Srinivasan, M.V., Zhang, S.: Visual motor computations in 26. Fry, S.N., Bichsel, M., Müller, P., Robert, D.: Tracking insects. Annual Review of Neuroscience 27, 679Ð96 (2004) of flying insects using pan-tilt cameras. Journal of Neuro- 47. Srinivasan, M.V., Zhang, S., Lehrer, M., Collett, T.S.: Hon- science Methods 101(1), 59Ð67 (2000) eybee navigation en route to the goal: Visual flight con- 27. Fry, S.N., Rohrseitz, N., Straw, A.D., Dickinson, M.H.: trol and odometry. Journal of Experimental Biology 199(1), TrackFly: Virtual reality for a behavioral system analysis 237Ð44 (1996) in free-flying fruit flies. Journal of Neuroscience Methods 48. Tanaka, K., Kawachi, K.: Response characteristics of visual 171(1), 110Ð117 (2008) altitude control system in Bombus terrestris. Journal of 28. Fry, S.N., Rohrseitz, N., Straw, A.D., Dickinson, M.H.: Experimental Biology 209(22), 4533Ð4545 (2006) Visual control of flight speed in Drosophila melanogaster. 49. Taylor, G.K., Bacic, M., Bomphrey, R.J., Carruthers, A.C., Journal of Experimental Biology 212, 1120Ð1130 (2009) Gillies, J., Walker, S.M., Thomas, A.L.R.: New experimen- 1 Experimental Approaches to Insect Flight Control 13

tal approaches to the biology of flight control systems. Jour- 57. Weis-Fogh, T.: Quick estimates of flight fitness in hovering nal of Experimental Biology 211(2), 258Ð266 (2008) animals, including novel mechanisms for lift production. 50. Taylor, G.K., Krapp, H.G.: Sensory systems and flight sta- Journal of Experimental Biology 59, 169Ð230 (1973) bility: What do insects measure and why? Advances in 58. Wood, R.J.: Design, fabrication, and analysis of a 3DOF, Insect Physiology 34, 231Ð316 (2008) 3 cm flapping-wing MAV. Intelligent Robots and Systems, 51. Taylor, G.K., Zbikowski,ú R.: Nonlinear time-periodic mod- 2007. IROS 2007, pp. 1576Ð1581 (2007) els of the longitudinal flight dynamics of desert locusts 59. Wood, R.J.: The first takeoff of a biologically inspired at- Schistocerca gregaria. Journal of the Royal Society Inter- scale robotic insect. Robotics, IEEE Transactions on 24(2), face 2(3), 197Ð221 (2005) 341Ð347 (2008) 52. Tinbergen, N.: über die Orientierung des Bienenwolfs 60. Wu, W., Shenato, L., Wood, R.J., Fearing, R.S.: Biomimetic (Philantus triangulum Fabr.). Zeitschrift für Vergleichende sensor suite for flight control of a micromechanical fly- Physiologie 16, 305Ð334 (1932) ing insect: Design and experimental results. Proceedings of 53. Vardy, A., Möller, R.: Biologically plausible visual hom- the 2003 IEEE International Conference on Robotics and ing methods based on optical flow techniques. Connection Automation (ICRA 2003), vol. 1, pp. 1146Ð1151. IEEE Science 17(1), 47 Ð 89 (2005) Press, Piscataway, NJ (2003) 54. Webb, B.: Validating biorobotic models. Journal of Neural 61. Wu, W.C., Wood, R.J., Fearing, R.S.: Halteres for the Engineering 3(3), R25ÐR35 (2006) micromechanical flying insect. Proceedings of the IEEE 55. Wehner, R.: Spatial vision in arthropods. Handbook of International Conference on Robotics and Automation, Sensory Physiology, vol. VII/6C, pp. 287Ð616. Springer, ICRA 2002 1, 60Ð65 (2002) Berlin, Heidelberg, New York, Tokyo (1981) 62. Zufferey, J.C., Floreano, D.: Fly-inspired visual steering of 56. Wehner, R.: Arthropods. In: F. Papi (ed.) Animal homing, an ultralight indoor aircraft. IEEE Transactions on Robotics pp. 45Ð144. Chapman & Hall (1992) 22(1), 137Ð146 (2006) Chapter 2 From Visual Guidance in Flying Insects to Autonomous Aerial Vehicles

Mandyam V. Srinivasan, Saul Thurrowgood, and Dean Soccol

Abstract Investigation of the principles of visually cannot infer the distances to objects or surfaces from guided flight in insects is offering novel, computa- the extent to which the directions of gaze must con- tionally elegant solutions to challenges in machine verge to view the object, or by monitoring the refrac- vision and robot navigation. Insects perform remark- tive power that is required to bring the image of the ably well at seeing and perceiving the world and navi- object into focus on the retina. Furthermore, compared gating effectively in it, despite possessing a brain that with human eyes, the eyes of insects are positioned weighs less than a milligram and carries fewer than much closer together and possess inferior spatial acuity 0.01% as many neurons as ours does. Although most [16]. Therefore, the precision with which insects could insects lack stereovision, they use a number of inge- estimate range through binocular stereopsis would be nious strategies for perceiving their world in three much poorer and restricted to relatively small dis- dimensions and navigating successfully in it. Over the tances, even if they possessed the requisite neural appa- past 20 years, research in our laboratory and elsewhere ratus [15]. Not surprisingly, then, insects have evolved is revealing that flying insects rely primarily on cues alternative strategies for dealing with the problems of derived from image motion (“optic flow”) to distin- visually guided flight. Many of these strategies rely guish objects from backgrounds, to negotiate narrow on using image motion, generated by the insect’s own gaps, to regulate flight speed, to compensate for head- motion, to infer the distances to obstacles and to con- winds and crosswinds, to estimate distance flown and trol various manoeuvres (Franceschini, Chap. 3 of this to orchestrate smooth landings. Here we summarize volume) [10, 16, 21, 22]. some of these findings and describe a vision system This pattern of image motion, known as “optic currently being designed to facilitate automated terrain flow”, is used in many ways to guide flight. For exam- following and landing. ple, distances to objects are gauged in terms of the apparent speeds of motion of the objects’ images, rather than by using complex stereo mechanisms [12, 2.1 Introduction 17, 26]. Objects are distinguished from backgrounds by sensing the apparent relative motion at the boundary [18]. Narrow gaps are negotiated safely by balancing Insect eyes differ from vertebrate or human eyes in the apparent speeds of the images in the two eyes [11, a number of ways. Unlike vertebrates, insects have 19]. The speed of flight is regulated by holding con- immobile eyes with fixed-focus optics. Therefore, they stant the average image velocity as seen by both eyes [1, 5, 27]. This ensures that flight speed is automatically lowered in cluttered environments and that thrust M.V. Srinivasan () is appropriately adjusted to compensate for headwinds Queensland Brain Institute and ARC Centre of Excellence and tail winds [2, 27]. Visual cues are also used to com- in Vision Science, University of Queensland, St. Lucia, pensate for crosswinds. Bees landing on a horizontal QLD 4072, Australia e-mail: [email protected] surface hold constant the image velocity of the surface

D. Floreano et al. (eds.), Flying Insects and Robots, 15 DOI 10.1007/978-3-540-89393-6_2, © Springer-Verlag Berlin Heidelberg 2009 16 M.V. Srinivasan et al. as they approach it, thus automatically ensuring that beneath the eye is given by flight speed is close to zero at touchdown [24]. For- aging bees gauge distance flown by integrating optic Vf flow: they possess a visually driven “odometer” that ω = rad/s (2.1) h is robust to variations in wind, body weight, energy expenditure and the properties of the visual environment [4, 6, 7, 23]. From this relationship it is clear that, if the bee’s In this chapter we concentrate on two aspects of horizontal flight speed is proportional to her height visually guided navigation in flying insects, which are above the surface (as shown by the data), then the heavily reliant on optic flow. One relates to landing angular velocity of the image of the surface, as seen by on a horizontal surface. The other concerns a related the eye, must be constant as the bee approaches it. This behaviour, terrain following, which involves maintain- angular velocity is given by the slope of the regression ing a constant height above the ground and following line. The angular velocity of the image varies from one the local fluctuations of terrain altitude. trajectory to another, but is maintained at an approximately constant value in any given landing. An analysis of 26 landing trajectories revealed a mean image angular velocity of ca. 500◦/s [24]. 2.2 Landing on a Horizontal Surface These results reveal two important characteristics. First, bees landing on a horizontal surface tend to We begin by summarizing the results of experiments approach the surface at a relatively shallow descent that were conducted a few years ago in our laboratory angle. Second, landing bees tend to hold the angular to elucidate the visual mechanisms that guide landing velocity of the image of the ground constant as they in honeybees. How does a bee land on a horizontal sur- approach it. face? When an insect makes a grazing landing on a flat What is the significance of holding the angular surface, the dominant pattern of image motion gener- velocity of the image of the ground constant during ated by the surface is a translatory flow in the front-to- landing? One important consequence is that the hor- back direction. What are the processes by which such izontal speed of flight is then automatically reduced landings are orchestrated? as the height decreases. In fact, by holding the image Srinivasan et al. [24] investigated this question by velocity constant, the horizontal speed is regulated to video-filming trajectories, in three dimensions, of bees be proportional to the height above the ground, so that landing on a flat, horizontal surface. Two examples of when the bee finally touches down (at zero height), her landing trajectories, reconstructed from the data, are horizontal speed is zero, thus ensuring a smooth land- shown in Fig. 2.1a,b. A number of such landings were ing. The attractive feature of this simple strategy is that analysed to examine the variation of the instantaneous it does not require explicit measurement or knowledge height above the surface (h), instantaneous horizon- of the speed of flight or the height above the ground. tal (forward) flight speed (Vf), instantaneous descent Thus, stereoscopic methods of measuring the distance speed (Vd) and descent angle (α). These variables are of the surface (which many insects probably do not illustrated in Fig. 2.1c. Analysis of the landing trajecto- possess) are not required. What is required, however, ries revealed that the descent angles were indeed quite is that the insect be constantly in motion, because the shallow. The average value measured in 26 trajectories image motion resulting from the insect’s own motion was ca. 28◦ [24]. is crucial in controlling the landing. Figure 2.2a,b shows the variation of flight speed The above strategy ensures that the bee’s horizon- with height above the surface, analysed for two landing tal speed is zero at touchdown, but does not regulate trajectories. These data reveal one of the most striking the descent speed. How does the descent speed vary and consistent observations of this study: Horizontal during the landing process? Plots of descent speed ver- speed is roughly proportional to height, as indicated by sus height reveal a linear relationship between these the linear regression on the data. When a bee flies at a two variables, as well. Two examples are shown in horizontal speed of Vf (cm/s) at a height of h (cm), the Fig. 2.2c,d. This finding implies that landing bees (i) angular velocity ω of the image of the surface directly adjust their forward (i.e. flight) speed to hold the 2 Flying Insects and Autonomous Aerial Vehicles 17

Fig. 2.1 (a, b) Three-dimensional reconstruction of two typical above surface; Vf (cm/s): horizontal (forward) flight speed; Vd landing trajectories, from video films. Vertical lines depict the (cm/s): vertical (descent) speed; α (deg or rad): descent angle -1 height above surface. (c) Illustration of some of the variables [α=(tan (Vf /Vd)]. Adapted from [24] analysed to investigate the control of landing. h (cm): height image velocity of the ground constant and (ii) cou- require explicit knowledge of instantaneous height or ple the descent speed to the forward speed, so that the flight speed. Second, forward and descent speeds are descent speed decreases with the forward speed and adjusted in such a way as to hold the image veloc- also becomes zero at touchdown. ity constant. This is advantageous because the image These results reveal what appears to be a surpris- velocity can then be maintained at a level at which ingly simple and effective strategy for making graz- the visual system is most sensitive to deviations from ing landings on flat surfaces. A safe, smooth landing the “set” velocity, thereby ensuring that the control of is ensured by following two simple rules: (a) adjusting flight is as precise as possible. An alternative strat- the speed of forward flight to hold constant the angu- egy, for instance, might be to approach the surface lar velocity of the image of the surface as seen by the at constant flight speed, decelerating only towards the eye and (b) making the speed of descent proportional end. Such a constant speed approach, however, would to the forward speed, i.e. flying at a constant descent cause the image velocity to increase rapidly as the sur- angle. This produces landing trajectories in which the face is approached and to reach levels at which the forward speed and the descent speed decrease progres- image velocity measurements may no longer be pre- sively as the surface is approached, both approaching cise enough for adequate flight control. This situation zero at touchdown. would be avoided by the bee’s landing strategy, which What are the advantages, if any, of using this land- holds the image velocity constant. Third, an interesting strategy? We can think of three attractive features. ing by-product of the bee’s landing strategy is that the First, the strategy is very simple because it does not projected time to touchdown is constant throughout the 18 M.V. Srinivasan et al.

Fig. 2.2 (a, b) Variation of horizontal flight speed (Vf)with straight lines are linear regressions through the data, as rep- height (h) above the surface for two different landing trajec- resented by the equations; r denotes the regression coefficient. tories. (c, d) Variation of descent speed (Vd) with height (h) Adapted from [24] above the surface for two different landing trajectories. The landing process (details in [24]). In other words, if, at the ground and to fly parallel to the surface by follow- any time during the landing process, the bees were to ing any fluctuations of height (terrain following). The stop decelerating and continue downward at constant strategy would now be not to fly towards a target on the velocity, the time to contact the ground would be the ground, but instead towards a distant target or the hori- same, regardless of where this occurs in the landing zon. This will ensure a level flight attitude. Flight speed trajectory. From the landing trajectories, one calculates is held constant by maintaining a constant forward a projected time to touchdown of about 0.22 s. Thus, it thrust (which should be effective, at least in still air). appears that landing bees allow themselves a “safety The height above the ground is regulated by adjusting margin” of a fifth of a second to prepare for touch- the altitude so that the magnitude of the optic flow gen- down if they were to abandon the prescribed landing erated by the ground is constant. If the forward flight strategy at any point, for whatever reason, and proceed speed is Vf and the desired height above the ground is towards the ground in the same direction without fur- h, the magnitude of the optic flow that is to be main- ther deceleration. tained is given by Eq. (2.1) above as simply the ratio of Vf to h. If the measured optic flow is greater than this target value, it signifies that the altitude is lower than 2.3 Terrain Following the desired value and a control command is then generated to increase altitude until the target magnitude of optic flow is attained. If the optic flow is lower than the A simple modification of the landing strategy target value, the opposite control command is issued, described above can be used, in principle, to regu- to again restore the optic flow to its desired level. late the height above the ground during cruising flight. This strategy has been tested successfully in simula- Here the aim is to maintain a constant height above tions, in tethered robots in the laboratory and in some 2 Flying Insects and Autonomous Aerial Vehicles 19 freely flying model aircraft (Franceschini, Chap. 3 of A’ this volume; Zufferey et al., Chap. 6 of this volume) η O Mirror [3, 13, 14, 25, 28]. f θ surface r P

Camera 2.4 Practical Problems with the β Camera nodal point Measurement of Optic Flow image

When flying at low altitudes or during landing, the image of the ground can move very rapidly, making it difficult to obtain accurate estimates of optic flow. Ground A Here we describe a specially shaped mirror surface Fig. 2.4 Geometry of camera/mirror configuration for the that, first, scales down the speed of image motion as design of mirror profile. Modified from [20] seen by the camera and, second, removes the perspective distortion (and therefore the distortion in image velocity) that a camera experiences when viewing a stant velocity in the image plane of the camera, inde- pendent of the position of A in the ground plane. That horizontal plane that stretches out to infinity in front η is, we require d = L, where η is the distance of the of the aircraft. Ideally, the moving image that is cap- dt tured by the camera through the mirror should exhibit point A from the centre of the camera’s image plane, a constant, low velocity everywhere, thus simplifying and L is the desired constant velocity; f is the focal the optic flow measurements and increasing their accu- length of the camera. The derivation of a mirror profile racy (see Fig. 2.3). that meets these objectives is given in [20] and we shall not repeat it here. Instead, we use an example profile to illustrate the performance of the mirror. Figure 2.5 shows one example of a profile of the reflective surface and includes the computed ray paths. In this example the camera faces forward, in the direc- Video tion of flight. The nodal point of the camera is at (0.0). Reflective camera surface The image plane of the camera is to the left of this Ground point and is not included in the figure, but its reflection plane about the nodal point (an equivalent representation) is depicted by the vertical line to the right of the nodal point. Parameters used in this design are as follows: Fig. 2.3 Illustration of a system for visually guided terrain fol- = = = = lowing and landing. The optical system is shown on an enlarged V 1000.0 cm/s; h 100.0 cm; r0 10.0 cm; f scale relative to the aircraft in order to illustrate its configuration 3.5 cm; L=2.0 cm/s, where V is the speed of the aircraft, h is the height above the ground, L is the image velocity, f is the focal length of the camera and r0 is the distance from the nodal point of the camera to the 2.5 A Mirror-Based Vision System for tip of the reflective surface. If the camera was looking directly downwards at the ground, the image velocity Terrain Following and Landing of the ground would have been 35 cm/s; with the mirror, the image velocity is reduced to 2.0 cm/s. Thus, We seek a mirror profile that will map equal distances the mirror scales down the image velocity by a factor along the ground in the flight direction to equal dis- of 17.5. placements in the image plane of the camera. The The curvature of the mirror is highest in the region geometry of the cameraÐmirror configuration is shown that images the ground directly beneath the aircraft, in Fig. 2.4. A is the camera image of a point A on the because this is the region of the ground that moves at ground. We want A (the image of A) to move at a con- the highest angular velocity with respect to the camera, 20 M.V. Srinivasan et al.

Fig. 2.5 Example of 5 computed mirror proﬁle. Camera nodal point Adapted from [20] Mirror profile

y –5

–10

Forward –15 5 10 15 20 25 x and which therefore requires the greatest reduction the optic axis. The circle in Fig. 2.6a maps to the hor- of motion. We see from Fig. 2.5 that equally spaced izontal line in Fig. 2.6b. Regions below the line rep- points on the ground along the line of flight map to resent areas in front of the aircraft and regions above equally spaced points in the camera image. This con- it represent areas behind. Thus, the mirror endows the firms the correct operation of the surface. camera with a large field of view that covers regions in Figure 2.6a illustrates the imaging properties of the front of, below and behind the aircraft. mirror, positioned with its axis parallel to and above a A consequence of the geometrical mapping pro- plane carrying a checkerboard pattern. Note that the duced by the mirror is that, for straight and level flight mirror has removed the perspective distortion (fore- parallel to the ground plane, the optic flow vectors will shortening) of the image of the plane. The scale of the have constant magnitude along each radius but will mapping depends upon the radial direction. Compres- be largest along the vertical radius and smallest (zero) sion is lowest in the vertical radial direction and high- along the horizontal radius. est in the horizontal radial direction. This is illustrated in Fig. 2.7, which shows the Figure 2.6b shows a digitally remapped view of the optic flow vectors that are generated by a simulated image in Fig. 2.6a, in which the polar co-ordinates of level flight over an ocean. Figure 2.7a shows the flow each pixel in Fig. 2.6a are plotted as Cartesian co- field in the raw image and Fig. 2.7b the flow field ordinates. Here the vertical axis represents radial dis- in the remapped image. In the remapped image, all tance from the centre of the image of Fig. 2.6a and the flow vectors are oriented vertically. The magnitudes horizontal axis represents the angle of rotation about of the vectors are constant within each column and

Fig. 2.6 (a) Illustration of imaging properties of mirror and (b) remapped version of image in Fig. 2.6a. Adapted from [20] a b