DOCSLIB.ORG

Exploration of Airsim Using C and Rust in the Context of Safetycritical Systems

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Exploration of Airsim Using C and Rust in the Context of Safetycritical Systems DEGREE PROJECT IN COMPUTER ENGINEERING, FIRST CYCLE, 15 CREDITS STOCKHOLM, SWEDEN 2018 Exploration of AirSim using C and Rust in the Context of SafetyCritical Systems DANIEL AROS BANDA JOEL WACHSLER KTH ROYAL INSTITUTE OF TECHNOLOGY SCHOOL OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE Exploration of AirSim using C and Rust in the context of Safety-Critical Systems JOEL WACHSLER DANIEL AROS BANDA Bachelor in Computer Science Date: June 20, 2018 Supervisor: George Ungureanu Examiner: Ingo Sander Swedish title: Utforskning av AirSim med hjälp av C och Rust inom ramen för Säkerhetskritiska System School of Electrical Engineering and Computer Science iii Abstract AirSim is a new simulator developed as a plugin for the Unreal En- gine, aiming to be a useful tool aiding the development and testing of algorithms for autonomous vehicles. Due to AirSim still being in its infancy there is little to none research available of its possibilities or detailed guidelines and tutorials on how to use its APIs. Rust is a new systems programming language with the purpose of being safe, practical and concurrent which through design choices can solve some of the major drawbacks of the C programming language such as memory leaks, thread management, and segmentation faults. Researching the features of AirSim and its different ways of communi- cating, we determine the possibility of implementing a custom flight controller in Rust and C able to control a drone in the simulator and evaluate the capabilities of Rust compared to C. This is conducted by reading available documentation for AirSim, studying the source code and learning about the communication protocols used by AirSim. This thesis results in an implementation of a custom flight controller in Rust and C that controls a drone in AirSim using a communication protocol named MAVLink which enables fine-grained control of the motors. The conclusion made about the comparison of Rust and C is that both languages were able to implement the safety-critical func- tionality of the flight controller and that Rust provided capabilities which could be useful when developing safety-critical systems. Keywords AirSim, Simulation, C, Rust, Safety-Critical Systems, Flight Controller, MAVLink iv Sammanfattning AirSim är en ny simulator utvecklad som ett plugin för Unreal Engi- ne, med målet att fungera som ett hjälpmedel inom utveckling och testning av algoritmer för autonoma fordon. På grund av att AirSim fortfarande är väldigt ungt finns väldigt lite forskning tillgänglig om dess möjligheter eller detaljerade riktlinjer och beskrivningar för an- vändningen av dess APIer. Rust är ett nytt programmeringsspråk med målet att vara säkert, prak- tiskt och parallellt vilket genom designval kan lösa några av de största problemen med programmeringsspråket C som till exempel minness- läckor, trådhantering och segmenteringsfel. Genom att undersöka funktionerna i AirSim och dess olika sätt att kommunicera, utforskar vi möjligheten av att utveckla en egen flyg- kontroller i Rust och C som kan styra en drönare i simulatorn och utvärdera Rust i förhållande till C. Detta genomförs genom att läsa tillgänglig dokumentation för AirSim, studera källkoden och lära oss de kommunikationsprotokoll som används av AirSim. Denna avhandling resulterar i implementationen av en egen flygkon- troller i Rust och C som styr en drönare i AirSim med kommunika- tionsprotokollet MAVLink, vilket möjliggör en noggrann kontroll av motorerna. Slutsatsen gällande Rust och C är att båda språken fun- gerade väl för implementationen av säkerhetsritiska funktioner i flyg- kontrollern samt att Rust erbjöd förmågor som kan visa sig vara an- vändbara vid utveckling av säkerhetskritiska system. Nyckelord AirSim, Simulering, C, Rust, Säkerhetskritiska System, Flygkontroller, MAVLink Contents 1 Introduction 1 1.1 Background . 1 1.2 Problem . 1 1.3 Purpose . 2 1.4 Goals . 2 1.5 Method . 3 1.6 Delimitations . 4 1.7 Document overview . 4 2 Background 6 2.1 AirSim . 6 2.1.1 AirSim architecture . 7 2.1.2 Flight controllers . 8 2.1.3 Vehicles . 9 2.2 Flight controller . 9 2.2.1 Aircraft principal axes . 10 2.2.2 Sensors . 10 2.2.3 PID controller . 11 2.3 RPC . 12 2.4 MAVLink . 12 2.5 Safety-critical systems . 13 2.6 The C programming language . 13 2.6.1 Memory management . 13 2.6.2 Dangling pointers . 14 2.7 The Rust programming language . 15 2.7.1 Ownership . 15 2.7.2 Borrowing . 16 2.7.3 Memory management . 16 v vi CONTENTS 3 AirSim 18 3.1 AirSim settings . 18 3.2 AirSim API using RPC . 19 3.3 AirSim API using MAVLink messages . 20 3.3.1 MAVLink messages used by AirSim . 21 3.4 The notion of time . 24 4 AirSim findings and controller implementation 25 4.1 System requirements . 25 4.1.1 Operating System and setup . 26 4.2 Communication . 26 4.2.1 RPC . 26 4.2.2 MAVLink . 27 4.2.3 Network communication . 30 4.2.4 Serial communication . 31 4.3 Enabling the MAVLink distance sensor . 31 4.4 Multiple vehicle simulation . 32 4.5 Custom flight controller . 33 4.5.1 Choosing API . 34 4.5.2 Architecture using MAVLink messages . 34 4.5.3 Implemented functionality . 36 4.6 Code examples . 38 5 Evaluation and conclusions 39 5.1 AirSim . 39 5.2 Rust and C in safety-critical systems . 40 5.3 Goals . 41 5.4 Future work . 43 5.4.1 Rust and C comparison . 43 5.4.2 Flight controller . 43 5.4.3 Multiple vehicle simulation . 43 5.4.4 AirSim . 44 Bibliography 45 List of Figures 2.1 Realistic graphics in AirSim provided by Unreal Engine . 7 2.2 Core components of the AirSim architecture . 8 2.3 The car vehicle in the default environment of AirSim . 9 2.4 The drone vehicle in the default environment of AirSim . 9 2.5 Principal axes of an aircraft . 10 2.6 Block diagram of a Proportional Integral Derivative (PID) controller with a feedback loop . 11 2.7 RPC procedure call overview . 12 2.8 MAVLink message transportation overview . 13 3.1 RPC takeoff procedure call example . 20 3.2 MAVLink message sensing and receiving in AirSim . 22 4.1 Spawning multiple vehicles in AirSim . 33 4.2 Flight controller (FC) functionality summary . 33 4.3 How simple flight, AirSim and Remote procedure call (RPC) are connected . 34 4.4 Custom flight controller architecture . 35 4.5 The drone hovering above the takeoff position using the custom FC . 36 vii Listings 2.1 Heap memory allocation in C . 14 2.2 Dangling pointer in C . 15 2.3 Ownership in Rust . 16 2.4 Borrowing in Rust . 16 2.5 Rust compiler preventing a dangling pointer . 17 2.6 Heap memory management in Rust . 17 3.1 AirSim settings file example . 19 3.2 Changing the port of the default Application Program- ming Interface (API) server . 19 3.3 AirSim settings to enable MAVLink communication . 21 3.4 AirSim settings to set the simulation clock to Scalable- Clock and change its speed to a factor 2 . 24 4.1 AirSim API communication in C++ . 27 4.2 MAVLink message parsing in C . 27 4.3 Example of decoding common MAVLink message sent by AirSim in C . 28 4.4 Receiving, parsing and decoding common MAVLink mes- sages sent by AirSim in Rust . 28 4.5 Actuator controls messaging in C . 29 4.6 Actuator control messaging in Rust . 30 4.7 AirSim settings for over the network control . 31 4.8 AirSim settings to enable serial communication . 31 viii List of Tables 3.1 MAVLink messages AirSim is listening for . 21 3.2 MAVLink messages sent by AirSim . 22 3.3 Fields of a HIL_ACTUATOR_CONTROLS message . 23 3.4 Fields of a HIL_SENSOR message . 23 ix Acronyms API Application Programming Interface. 2–5, 7, 8, 18–20, 25, 26, 29– 31, 40, 42, 44 CC Companion computer. 7 FC Flight controller. 1–4, 7–12, 18, 22, 24–27, 30, 37–40, 42–44 GC Garbage Collector. 14, 17 GCS Ground Control Station. 12 GPS Global Positioning System. 10–12, 27 IDE Integrated development environment. 41 IMU Inertial Measurement Unit. 10, 12 JSON JavaScript Object Notation. 18 MAVLink Micro Air Vehicle Link. 8, 12, 18–23, 26–36, 40, 42–44 PID Proportional Integral Derivative. 11, 27, 38, 43 RC Remote control. 7–9 RPC Remote procedure call. 8, 12, 19, 20, 26, 30, 31 segfault Segmentation fault. 14, 17, 39, 41 UART Universal asynchronous receiver-transmitter. 43 UDP User Datagram Protocol. 20, 31, 35, 43 x Chapter 1 Introduction Developing software systems where the consequences of failure may result in death, damage to the environment or serious financial loss, known as safety-critical systems [7], usually have a high cost of verifi- cation and validation [19]. Simulating these systems can reduce these costs by finding design flaws in the early stages of development and engineers being able to model the system without resorting to proto- typing [8]. 1.1 Background AirSim is a new open source simulator developed by Microsoft AI & Research [18]. The goal of this simulator is to be a useful tool in the de- velopment of autonomous vehicles and the gathering of training data for machine intelligence [15]. Connecting the simulator to a safety- critical system, in the form of a Flight controller (FC), developed in the C programming language and the new programming language Rust [9] the suitability for each of these languages in the context of safety-critical systems can be compared and evaluated without the us- age of a real drone. 1.2 Problem Due to the young age of AirSim, there is little to none research based upon the simulator, making it unclear of its potential and capabilities.
Load more

Recommended publications
  • Getting Started with PX4 for Contributors Mark West PX4 Community Volunteer Format
    Getting Started with PX4 For Contributors Mark West PX4 Community Volunteer Format A word about format Limited Time + Big Subject = Compression Some Subjects Skipped = Look in Appendix Demo Videos Clipped = Look in Appendix Who is this for? Anybody Levels: L1: Operate a PX4 vehicle L2: Build a PX4 vehicle L3: Build the source L4: Modify the source L5: Contribute L1: Operate You want to operate a drivable unit See Appendix : Operate Vehicle L2: Build a Vehicle You want to build a drivable unit See Appendix : Build Vehicle L3: Build the Source You want to build the image from source Big step up from L2 Choose Toolchain Container L3: Build the Source : Toolchain All the tools you need to build an image / exe Compilers / Linkers / Tools / Etc Installation: Manual : If needed Convenience Scripts : Preferred L3: Build the Source : Container Container? Like a better VM, with toolchain installed Easy to install, easy to update, no interaction Image = container snapshot L3: Build the Source : Container Container images built in layers px4io/px4-dev-ros-melodic px4io/px4-dev-simulation-bionic px4io/px4-dev-base-bionic ubuntu L3: Build the Source : Container Containers are isolated, all interaction is explicit File System -v ~/src_d/Firmware:/src/firmware/:rw File System Directory Directory Network Port -p 14556:14556/udp Network Port Host Docker run params Container L3: Build the Source : Toolchain or Container? Container if possible Toolchain if no suitable container * But you can make your own containers L3: Build the Source : Toolchain Demo Build image for SITL Simulation 1) Install Toolchain 2) Git PX4 Source 3) Build PX4 4) Takeoff L3: Build the Source : Container Demo Build image for SITL Simulation 1) Install Docker 2) Git PX4 Source 3) Locate Container 4) Run Container 5) Build PX4 6) Build available on host computer L4: Modify the Source Next step up in complexity Why? Fix bugs, add features Loop: Edit, Build, Debug IDE: Visual Studio, Eclipse, QT Creator, etc.
    [Show full text]
  • Reinforcement Learning and Trustworthy Autonomy
    Reinforcement Learning and Trustworthy Autonomy Jieliang Luo, Sam Green, Peter Feghali, George Legrady, and Çetin Kaya Koç Abstract Cyber-Physical Systems (CPS) possess physical and software inter- dependence and are typically designed by teams of mechanical, electrical, and software engineers. The interdisciplinary nature of CPS makes them difficult to design with safety guarantees. When autonomy is incorporated, design complexity and, especially, the difficulty of providing safety assurances are increased. Vision- based reinforcement learning is an increasingly popular family of machine learning algorithms that may be used to provide autonomy for CPS. Understanding how visual stimuli trigger various actions is critical for trustworthy autonomy. In this chapter we introduce reinforcement learning in the context of Microsoft’s AirSim drone simulator. Specifically, we guide the reader through the necessary steps for creating a drone simulation environment suitable for experimenting with vision- based reinforcement learning. We also explore how existing vision-oriented deep learning analysis methods may be applied toward safety verification in vision-based reinforcement learning applications. 1 Introduction Cyber-Physical Systems (CPS) are becoming increasingly powerful and complex. For example, standard passenger vehicles have millions of lines of code and tens of processors [1], and they are the product of years of planning and engineering J. Luo () · S. Green · P. Feghali · G. Legrady University of California Santa Barbara, Santa Barbara, CA, USA e-mail: [email protected]; [email protected]; [email protected]; [email protected] Ç. K. Koç Istinye˙ University, Istanbul,˙ Turkey Nanjing University of Aeronautics and Astronautics, Nanjing, China University of California Santa Barbara, Santa Barbara, CA, USA e-mail: [email protected] © Springer Nature Switzerland AG 2018 191 Ç.
    [Show full text]
  • A Simple Platform for Reinforcement Learning of Simulated Flight Behaviors ⋆
    LM2020, 011, v2 (final): ’A Simple Platform for Reinforcement Learning of Simulated . 1 A Simple Platform for Reinforcement Learning of Simulated Flight Behaviors ⋆ Simon D. Levy Computer Science Department, Washington and Lee University, Lexington VA 24450, USA [email protected] Abstract. We present work-in-progress on a novel, open-source soft- ware platform supporting Deep Reinforcement Learning (DRL) of flight behaviors for Miniature Aerial Vehicles (MAVs). By using a physically re- alistic model of flight dynamics and a simple simulator for high-frequency visual events, our platform avoids some of the shortcomings associated with traditional MAV simulators. Implemented as an OpenAI Gym en- vironment, our simulator makes it easy to investigate the use of DRL for acquiring common behaviors like hovering and predation. We present preliminary experimental results on two such tasks, and discuss our cur- rent research directions. Our code, available as a public github repository, enables replication of our results on ordinary computer hardware. Keywords: Deep reinforcement learning Flight simulation Dynamic · · vision sensing. 1 Motivation Miniature Aerial Vehicles (MAVs, a.k.a. drones) are increasingly popular as a model for the behavior of insects and other flying animals [3]. The cost and risks associated with building and flying MAVs can however make such models inaccessible to many researchers. Even when such platforms are available, the number of experiments that must be run to collect sufficient data for paradigms like Deep Reinforcement Learning (DRL) makes simulation an attractive option. Popular MAV simulators like Microsoft AirSim [10], as well as our own sim- ulator [7], are built on top of video game engines like Unity or UnrealEngine4.
    [Show full text]
  • Chapter 3 Airsim Simulator
    MASTER THESIS Structured Flight Plan Interpreter for Drones in AirSim Francesco Rose SUPERVISED BY Cristina Barrado Muxi Universitat Politècnica de Catalunya Master in Aerospace Science & Technology January 2020 This Page Intentionally Left Blank Structured Flight Plan Interpreter for Drones in AirSim BY Francesco Rose DIPLOMA THESIS FOR DEGREE Master in Aerospace Science and Technology AT Universitat Politècnica de Catalunya SUPERVISED BY: Cristina Barrado Muxi Computer Architecture Department “E come i gru van cantando lor lai, faccendo in aere di sé lunga riga, così vid’io venir, traendo guai, ombre portate da la detta briga” Dante, Inferno, V Canto This Page Intentionally Left Blank ABSTRACT Nowadays, several Flight Plans for drones are planned and managed taking advantages of Extensible Markup Language (XML). In the mean time, to test drones performances as well as their behavior, simulators usefulness has been increasingly growing. Hence, what it takes to make a simulator capable of receiving commands from an XML file is a dynamic interface. The main objectives of this master thesis are basically three. First of all, the handwriting of an XML flight plan (FP) compatible with the simulator environment chosen. Then, the creation of a dynamic interface that can read whatever XML FP and that will transmit commands to the drone. Finally, using the simulator, it will be possible to test both interface and flight plan. Moreover, a dynamic interface aimed at managing two or more drones in parallel has been built and implemented as extra objective of this master thesis. In addition, assuming that two drones will be used to test this interface, it is required the handwriting of two more FPs.
    [Show full text]
  • An Aerial Mixed-Reality Environment for First-Person-View Drone Flying
    applied sciences Article An Aerial Mixed-Reality Environment for First-Person-View Drone Flying Dong-Hyun Kim , Yong-Guk Go and Soo-Mi Choi * Department of Computer Science and Engineering, Sejong University, Seoul 05006, Korea; [email protected] (D.-H.K.); lattechiff[email protected] (Y.-G.G.) * Correspondence: [email protected] Received: 27 June 2020; Accepted: 3 August 2020; Published: 6 August 2020 Abstract: A drone be able to fly without colliding to preserve the surroundings and its own safety. In addition, it must also incorporate numerous features of interest for drone users. In this paper, an aerial mixed-reality environment for first-person-view drone flying is proposed to provide an immersive experience and a safe environment for drone users by creating additional virtual obstacles when flying a drone in an open area. The proposed system is effective in perceiving the depth of obstacles, and enables bidirectional interaction between real and virtual worlds using a drone equipped with a stereo camera based on human binocular vision. In addition, it synchronizes the parameters of the real and virtual cameras to effectively and naturally create virtual objects in a real space. Based on user studies that included both general and expert users, we confirm that the proposed system successfully creates a mixed-reality environment using a flying drone by quickly recognizing real objects and stably combining them with virtual objects. Keywords: aerial mixed-reality; drones; stereo cameras; first-person-view; head-mounted display 1. Introduction As cameras mounted on drones can easily capture photographs of places that are inaccessible to people, drones have been recently applied in various fields, such as aerial photography [1], search and rescue [2], disaster management [3], and entertainment [4].
    [Show full text]
  • Peter Van Der Perk
    Eindhoven University of Technology MASTER A distributed safety mechanism for autonomous vehicle software using hypervisors van der Perk, P.J. Award date: 2019 Link to publication Disclaimer This document contains a student thesis (bachelor's or master's), as authored by a student at Eindhoven University of Technology. Student theses are made available in the TU/e repository upon obtaining the required degree. The grade received is not published on the document as presented in the repository. The required complexity or quality of research of student theses may vary by program, and the required minimum study period may vary in duration. General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain Department of Electrical Engineering Electronic Systems Research Group A Distributed Safety Mechanism for Autonomous Vehicle Software Using Hypervisors graduation project Peter van der Perk Supervisors: prof.dr. Kees Goossens dr. Andrei Terechko version 1.0 Eindhoven, June 2019 Abstract Autonomous vehicles rely on cyber-physical systems to provide comfort and safety to the passen- gers. The objective of safety designs is to avoid unacceptable risk of physical injury to people. Reaching this objective, however, is very challenging because of the growing complexity of both the Electronic Control Units (ECUs) and software architectures required for autonomous operation.
    [Show full text]
  • Autonomous Mapping of Unknown Environments Using A
    DF Autonomous Mapping of Unknown Environments Using a UAV Using Deep Reinforcement Learning to Achieve Collision-Free Navigation and Exploration, Together With SIFT-Based Object Search Master’s thesis in Engineering Mathematics and Computational Science, and Complex Adaptive Systems ERIK PERSSON, FILIP HEIKKILÄ Department of Mathematical Sciences CHALMERS UNIVERSITY OF TECHNOLOGY Gothenburg, Sweden 2020 Master’s thesis 2020 Autonomous Mapping of Unknown Environments Using a UAV Using Deep Reinforcement Learning to Achieve Collision-Free Navigation and Exploration, Together With SIFT-Based Object Search ERIK PERSSON, FILIP HEIKKILÄ DF Department of Mathematical Sciences Division of Applied Mathematics and Statistics Chalmers University of Technology Gothenburg, Sweden 2020 Autonomous Mapping of Unknown Environments Using a UAV Using Deep Reinforcement Learning to Achieve Collision-Free Navigation and Ex- ploration, Together With SIFT-Based Object Search ERIK PERSSON, FILIP HEIKKILÄ © ERIK PERSSON, FILIP HEIKKILÄ, 2020. Supervisor: Cristofer Englund, RISE Viktoria Examiner: Klas Modin, Department of Mathematical Sciences Master’s Thesis 2020 Department of Mathematical Sciences Division of Applied Mathematics and Statistics Chalmers University of Technology SE-412 96 Gothenburg Telephone +46 31 772 1000 Cover: Screenshot from the simulated environment together with illustration of the obstacles detected by the system. Typeset in LATEX, template by David Frisk Gothenburg, Sweden 2020 iv Autonomous Mapping of Unknown Environments Using a UAV Using Deep Reinforcement Learning to Achieve Collision-Free Navigation and Ex- ploration, Together With SIFT-Based Object Search ERIK PERSSON, FILIP HEIKKILÄ Department of Mathematical Sciences Chalmers University of Technology Abstract Automatic object search in a bounded area can be accomplished using camera- carrying autonomous aerial robots.
    [Show full text]
  • Simulating GPS-Denied Autonomous UAV Navigation for Detection of Sur- Face Water Bodies
    This may be the author’s version of a work that was submitted/accepted for publication in the following source: Singh, Arnav Deo& Vanegas Alvarez, Fernando (2020) Simulating GPS-denied autonomous UAV navigation for detection of sur- face water bodies. In Proceedings of 2020 International Conference on Unmanned Aircraft Systems:ICUAS. Institute of Electrical and Electronics Engineers Inc., United States of America, pp. 1792-1800. This file was downloaded from: https://eprints.qut.edu.au/211603/ c IEEE 2020 This work is covered by copyright. Unless the document is being made available under a Creative Commons Licence, you must assume that re-use is limited to personal use and that permission from the copyright owner must be obtained for all other uses. If the docu- ment is available under a Creative Commons License (or other specified license) then refer to the Licence for details of permitted re-use. It is a condition of access that users recog- nise and abide by the legal requirements associated with these rights. If you believe that this work infringes copyright please provide details by email to [email protected] License: Creative Commons: Attribution-Noncommercial 4.0 Notice: Please note that this document may not be the Version of Record (i.e. published version) of the work. Author manuscript versions (as Sub- mitted for peer review or as Accepted for publication after peer review) can be identified by an absence of publisher branding and/or typeset appear- ance. If there is any doubt, please refer to the published source. https://doi.org/10.1109/ICUAS48674.2020.9213927 Simulating GPS-denied Autonomous UAV Navigation for Detection of Surface Water Bodies Arnav Deo Singh Fernando Vanegas Alvarez Queensland University of Technology Queensland University of Technology Brisbane, Australia Brisbane, Australia [email protected] [email protected] Abstract— The aim to colonize extra-terrestrial planets has NASA has been developing a UAV for the exploration of been of great interest in recent years.
    [Show full text]
  • Unrealcv: Virtual Worlds for Computer Vision
    Session: Open Source Software Competition MM’17, October 23-27, 2017, Mountain View, CA, USA UnrealCV: Virtual Worlds for Computer Vision Weichao Qiu1, Fangwei Zhong2, Yi Zhang1, Siyuan Qiao1, Zihao Xiao1, Tae Soo Kim1, Yizhou Wang2, Alan Yuille1 1. Johns Hopkins University, 2. Peking University {qiuwch,zfw1226,edwardz.amg,joe.siyuan.qiao}@gmail.com {zxiao10,tkim60}@jhu.edu,[email protected],[email protected] ABSTRACT realism, while the newest video games such as GTA do not have UnrealCV1 is a project to help computer vision researchers build good APIs to access its internal data and sometimes have license virtual worlds using Unreal Engine 4 (UE4). It extends UE4 with a issues. Second, the modifications of one video game can notbe plugin by providing (1) A set of UnrealCV commands to interact transfered to others and needs to be redone case by case. In or- with the virtual world. (2) Communication between UE4 and an der to use a video game as a virtual world for computer vision external program, such as Caffe. UnrealCV can be used in two researchers, some extensions need to be done: (1) the video game ways. The first one is using a compiled game binary with UnrealCV needs to be programmly accessible through an API, so that an AI embedded. This is as simple as running a game, no knowledge of agent can communicate with it (2) the information in the virtual Unreal Engine is required. The second is installing UnrealCV plugin worlds need to be extracted to achieve certain tasks, such as ground to Unreal Engine 4 (UE4) and use the editor of UE4 to build a new truth generation or giving rewards based on the action of agents.
    [Show full text]
  • Airsim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles
    AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles Shital Shah1, Debadeepta Dey2, Chris Lovett3, Ashish Kapoor4 Abstract Developing and testing algorithms for autonomous vehicles in real world is an expensive and time consuming process. Also, in order to utilize recent advances in machine intelligence and deep learning we need to collect a large amount of annotated training data in a variety of conditions and environments. We present a new simulator built on Unreal Engine that offers physically and visually realistic simulations for both of these goals. Our simulator includes a physics engine that can operate at a high frequency for real-time hardware-in-the-loop (HITL) simulations with support for popular protocols (e.g. MavLink). The simulator is designed from the ground up to be extensible to accommodate new types of vehicles, hardware platforms and software protocols. In addition, the modular design enables various components to be easily usable independently in other projects. We demonstrate the simulator by first implementing a quadrotor as an autonomous vehicle and then experimentally comparing the software components with real-world flights. 1 Introduction Recently, paradigms such as reinforcement learning [12], learning-by-demonstration [2] and transfer learning [25] are proving a natural means to train various robotics systems. One of the key challenges with these techniques is the high sample com- plexity - the amount of training data needed to learn useful behaviors is often pro- hibitively high. This issue is further exacerbated by the fact that autonomous vehi- cles are often unsafe and expensive to operate during the training phase. In order to seamlessly operate in the real world the robot needs to transfer the learning it does in simulation.
    [Show full text]
  • C Copyright 2020 Kyle Lindgren Robust Vision-Aided Self-Localization of Mobile Robots
    c Copyright 2020 Kyle Lindgren Robust Vision-Aided Self-Localization of Mobile Robots Kyle Lindgren A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy University of Washington 2020 Reading Committee: Blake Hannaford, Chair Jenq-Neng Hwang E. Jared Shamwell Program Authorized to Offer Degree: Electrical & Computer Engineering University of Washington Abstract Robust Vision-Aided Self-Localization of Mobile Robots Kyle Lindgren Chair of the Supervisory Committee: Professor Blake Hannaford Electrical & Computer Engineering Machine learning has emerged as a powerful tool for solving many computer vision tasks by ex- tracting and correlating meaningful features from high dimensional inputs in ways that can exceed the best human-derived modeling efforts. However, the area of vision-aided localization remains diverse with many traditional approaches (i.e. filtering- or nonlinear least-squares- based) often outperforming deep approaches and none declaring an end to the problem. Proven successes in both approaches, with model-free methods excelling at complex data association and model-based methods benefiting from known sensor and scene geometric dynamics, elicits the question: can a hybrid approach effectively combine the strengths of model-free and model-based methods? This work presents a new vision-aided localization solution with a monocular visual-inertial architec- ture which combines model-based optimization and model-free robustness techniques to produce scaled depth and egomotion estimates. Additionally, a Mixture of Experts ensemble framework is presented for robust multi-domain self-localization in unseen environments. Advancements in virtual environments and synthetically generated data with realistic imagery and physics engines are leveraged to aid exploration and evaluation of self-localization solutions.
    [Show full text]
  • Generating Data to Train a Deep Neural Network End-To-End Within a Simulated Environment
    Freie Universität Berlin Master thesis at Department of Mathematics and Computerscience Intelligent Systems and Robotic Labs Generating Data to Train a Deep Neural Network End-To-End within a Simulated Environment Josephine Mertens Student ID: 4518583 [email protected] First Examiner: Prof. Dr. Daniel Göhring Second Examiner: Prof. Dr. Raul Rojas Berlin, October 10, 2018 Abstract Autonomous driving cars have not been a rarity for a long time. Major man- ufacturers such as Audi, BMW and Google have been researching successfully in this field for years. But universities such as Princeton or the FU-Berlin are also among the leaders. The main focus is on deep learning algorithms. However, these have the disadvantage that if a situation becomes more complex, enormous amounts of data are needed. In addition, the testing of safety-relevant functions is increasingly difficult. Both problems can be transferred to the virtual world. On the one hand, an infinite amount of data can be generated there and on the other hand, for example, we are independent of weather situations. This paper presents a data generator for autonomous driving that generates ideal and unde- sired driving behavior in a 3D environment without the need of manually gen- erated training data. A test environment based on a round track was built using the Unreal Engine and AirSim. Then, a mathematical model for the calculation of a weighted random angle to drive alternative routes is presented. Finally, the approach was tested with the CNN of NVidia, by training a model and connect it with AirSim. Declaration of Authorship I hereby certify that this work has been written by none other than my person.
    [Show full text]