Neuromorphic Control of Dynamic Systems by Prince Singh M.S., Aerospace Engineering, University of Illinois-UC M.Sc., Applied Mathematics, Imperial College London B.S., Aerospace Engineering, Embry-Riddle Aero. Univ. Submitted to the Department of Aeronautics and Astronautics in partial fulfillment of the requirements for the degree of Doctor of Philosophy at the MASSACHUSETTS INSTITUTE OF TECHNOLOGY June 2018 @ Massachusetts Institute of Technology 2018. All rights reserved. Signature redacted A uthor ...................... Department of Aeronautics and Astronautics 0--lay 2 48-204- Certified by.......... Signature redacted ... Professor Emilio Frazzoli Professor of Aeronautics and Astronautics - Thesis Supervisor Certified by.......... Signature redacted.... Professor Hamsa Balakrishnan Associate Professor of Aeronautics and Astronautics / A -n Thesis Advisor Certified by......... Signature redacted ...... Dr. Karl Iagnemma Principal Research Scientist redacted Thesis Advisor Accepted by .. Signature Professor fiamsa Balakrishnan Associate Professor of Aeronautics and Astronautics Chair, Graduate Program Committee MASSACHUSETTS INSTITUTE OF TECHNOLOGY 0 JUN 2 8 2018 LIBRARIES 2 Neuromorphic Control of Dynamic Systems by Prince Singh Submitted to the Department of Aeronautics and Astronautics on May 29, 2018, in partial fulfillment of the requirements for the degree of Doctor of Philosophy Abstract Arguably, the agility of a robotic system is dictated by the speed of its processing pipeline, i.e., the speed of data acquisition plus data processing from a robot's on-board vision sensors. Specifically, one ideally hopes that this pipeline offer fresh measurements at a high temporal resolution with low-latency in a computationally-cheap manner for efficient control. This desirable situation may be hard to come by for majority of the current vision-based robotic applications that rely on the traditional CCD-/CMOS- pipeline, as one would be in search for traditional cameras that offer a high sampling rate (thus, high temporal resolution) whose potentially redundant (thus, not fresh) and synchronous output must be processed with low- latency in a computationally-cheap manner. For instance, processing the synchronous series of conventional camera images, which embed possibly redundant levels of intensities may greatly hinder the fast reaction times required by robots while expending power. This issue warrants the need for faster sensors in order to truly address the underlying perception problem for high-performance systems that need to operate under power constraints. To this end, we capitalize upon the merits of a recently introduced biologically inspired and computationally-cheap alternative to traditional cameras-called Neuromorphic Vision Sensors whose pixels independently and asynchronously (thus, high temporal resolution) fire, in the order of micro-seconds (thus, low-latency and high temporal resolution), a stream of non-redundant (thus, fresh) brightness changes represented as binary numbers ( 1), termed retinal events, based on a trigger condition that is defined on a logarithmic scale. These properties offer a faster process- ing pipeline and hint that the Neuromorphic sensor would be a promising candidate to facilitate high-speed robotic applications. However, existing computer-vision based algorithms designed for processing periodic mea- 3 surements cannot be directly adapted to process retinal events, as these are fired aperiodically, and are ambiguous since they are binary. As an additional challenge, in practice, many retinal events are misfired due to the presence of underlying sensor circuitry noise (not associated to physi- cal brightness changes in the environment) and we term these as spurious events. The merits and operational constraints of this vision sensor man- dates the development of a corresponding control-theoretic setup. Thus, the contributions of this dissertation are twofold: 1) to design a control algorithm that processes de-noised retinal events to facilitate a prescribed control task, and 2) to propose a de-noising procedure that mitigates the effect of spuriosity in retinal events. The first part of this dissertation, investigates the problem of control- ling (i.e., stabilization and regulation) a Continuous-Time Linear Time Invariant (CT-LTI) system using retinal events generated from an ideal- istic model of a Neuromorphic Vision Sensor, which is an instance of a broad family of signal change detection sensors frequently encountered in practice. The contribution is to present a novel control design procedure that stabilizes and regulates a hybrid system, consisting of the CT-LTI system and the discrete-event signal change observation model, to a de- sired set-point. Moreover, the set of thresholds (sufficient conditions) for the given system to fulfill the prescribed control task is provided. The proposed controller is then extended to handle the case of noise in both the system dynamics as well as the observation model; thus, accounts for spurious events in this setting. The second part of this dissertation proposes a de-noising algorithm- Spuriosity Filter (SF)-and is motivated by the practical need to reduce spurious events whilst working with general observation models. The con- struction of SF is based on the fundamental lack of spatial correlation between spurious events and the algorithm trades off pixel resolution to produce a cleaner event stream on larger spatial scales by seeking a form of 'consensus' between neighboring pixels. At the core of our analysis lies a formal equivalence relation, defined as a means to track brightness, between our filter and a lower-resolution neuromorphic sensor with reduced noise levels. As a consequence of the principled analysis, we highlight impor- tant properties that any filter, which processes asynchronous noisy retinal events must respect and have not been accounted for by existing works. The effectiveness of the proposed control-theoretic setup for the illus- trative task of heading regulation is illustrated over a range of systems: from numerical experiments to a laboratory testbed. Thesis Supervisor: Professor Emilio Frazzoli Title: Professor of Aeronautics and Astronautics 4 Acknowledgments Several people have made my time at MIT fruitful and my life manageable. First and foremost, I have been extremely fortunate to have Prof. Emilio Frazzoli as my supervisor. Emilio's insightful vision, his limitless creativity, his everlasting enthusiasm, and importantly his humility has always left me in awe and propelled me to push myself daily to become a better student of life. I am truly grateful for the autonomy Emilio gave and trusted me with from day one of my time here while gently providing me with course corrections. Finally, I am indebted to Emilio's patience with me on many occasions and am extremely thankful for his style of training that ensures the best interests of his students so that they can succeed in their future endeavors. I am honored to have Prof. Hamsa Balakrishnan and Dr. Karl Iag- nemma serve on my thesis committee. Their invaluable feedback has im- mensely improved the quality of this work. I have been delighted to receive their mentorship and constant support during my thesis work. Further, I thank my thesis readers, Drs. Ali Agha (JPL) and Harshad Khadilkar (TCS), for agreeing to serve in this capacity, for sparing their time to provide excellent feedback on my work, and for always being available on short notice to provide me with good advice in order to make my life here manageable. Several researchers have played critical roles in shaping my research direction. My deepest thanks go to Profs. Sze Zheng Yong (ASU), Luca Carlone (MIT), and Andrea Censi (ETH-Z). They have endowed upon me a wealth of information and have facilitated many stimulating discussions leading to the fruition of these works. I am thankful to count Valerio Varricchio as a dear friend whose perpetual creativity has truly inspired 5 me as a researcher. I thank Valerio for always being supportive during my lowest points. It has been my privilege to have been a member of LIDS, a conducive place for learning comprised of amazing faculty and supportive administrative staff, and to have collaborated with its numerous bright students-many to individually name here and much of whom have become life-long friends! Lastly, I would not be where I am today without the blessings of my late mother, the support of my father and my wife. A substantial portion of the material in this dissertation is written with and without editing or rearrangement from the following four papers. @ 2016 IEEE. Reprinted, with permission, from Prince Singh, Sze Zheng Yong, Jean Gregoire, Andrea Censi, and Emilio Frazzoli. Stabiliza- tion of linear continuous-time systems using neuromorphic vision sensors. In Decision and Control (CDC), 2016 IEEE 55th Conference on, Las Ve- gas, NV. @ 2017 IEEE. Reprinted, with permission, from Prince Singh, Sze Zheng Yong, and Emilio Frazzoli. Stabilization of stochastic linear continuous- time systems using noisy neuromorphic vision sensors. In American Con- trol Conference (ACC), 2017, Seattle, WA. @ 2018 IEEE. Reprinted, with permission, from Prince Singh, Sze Zheng Yong, and Emilio Frazzoli. Regulation of Linear Systems using Event-Based Detection Sensors. In Transactions on Automatic Control (TAC), 2018. (To appear) 6 @ 2018. ELSEVIER. Reprinted, with permission, from Prince Singh, Valerio Varricchio, Sze Zheng Yong, Ali-Akbar Agha-Mohammadi,