View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by Archivio istituzionale della ricerca - Politecnico di Milano Model-based Real-time Testing of Drone Autopilots Andrea Patelli∗ and Luca Mottola∗+ ∗Politecnico di Milano (Italy), +SICS Swedish ICT Contact author: [email protected] ABSTRACT erates in a setting where developers struggle to gain run-time Key to the operation of robot drones is the autopilot soft- information useful for verifying the correctness. Besides the ware that realizes the low-level control. The correctness of hurdles at testing autopilots in real executions, resource- autopilot implementations is currently mainly verified based constraints hamper extensive logging and monitoring. on simulations. These may overlook the timing aspects of To complicate matters, many autopilot functionality need control loop executions, which are however fundamental to to execute in real-time, that is, subject to given time con- dependable operation. We report on our experience in ap- straints. A paradigmatic example is that of safety function- plying model-based real-time testing to Ardupilot, a widely ality, which must be operated within strict time bounds to adopted autopilot. We describe our approach at deriving a be effective. Should these bounds be violated, the drone model of Ardupilot's core functionality and at reducing the may harm surrounding people or objects. Testing real-time model to enable practical testing. Our work reveals that software, however, is challenging even in mainstream com- Ardupilot may fail in meeting the time constraints associ- puting, as testified by the bast literature on the subject [12]. ated to critical functionality, such as enabling fail-safe oper- The result of such a state of affairs is that autopilot soft- ation. Through controlled experiments, we demonstrate the ware often undertakes little testing compared to the key real-world occurrence of such erroneous executions. role it plays. Further, most such testing is essentially per- formed in simulations and looks at functional requirements. However, simulations provide little coverage, whereas non- 1. INTRODUCTION functional requirements such as real-time functionality are Aerial vehicles, ground robots, and aquatic rovers enable tackled by over-provisioning the hardware so that the soft- sophisticated applications [5, 16, 24]. Key to their depend- ware runs \fast enough" not to cause problems in everyday able operation is the autopilot software that implements the use. This provides no definitive guarantee on the correct low-level motion control. Autopilots process various sensor functioning of autopilots, which is unacceptable especially inputs, such as accelerations and GPS coordinates, to oper- for safety functionality. ate the electrical motors that set the vehicle's attitude. The Contribution. This paper presents our work and experi- high-level inputs come from a ground-control station (GCS) ence at gaining a deeper insight into the proportions, depth, where mission parameters are configured, or from a human and implications of the issues above. operator who drives the drone using a remote controller. We take Ardupilot [3], a widely employed open-source Problem. The autopilot is responsible for a number of autopilot implementation we describe in Sec.2, and apply safety functionality triggered in response to faults. For ex- state-of-the-art model-based real-time testing techniques for ample, it may force the drone to land should the battery verifying the correctness of crucial safety functionality. As voltage drop below a threshold. Correctly implementing this illustrated in Sec.3, the techniques we adopt are both ex- functionality is fundamental: accidents are often reported haustive, that is, given certain inputs, they can verify if whose causes may be attributed, in a way or the other, to developer-provided correctness properties hold in every pos- malfunctioning autopilots [19]. sible execution, and operate off-line. The latter feature is Implementing autopilot software is, nevertheless, challeng- immensely useful to overcome the aforementioned practical ing. Size, cost, and energy concerns require autopilots to run hurdles in testing autopilots right on the drones. on resource-constrained embedded hardware. This forces de- We build a model of Ardupilot's core functionality in two velopers to employ low-level languages and to aggressively steps. Sec.4 describes how we derive the code's control flow optimize implementations. Moreover, autopilot software op- graph in a fully automated way, which guarantees the model is structurally equivalent to the code. Next, we perform ex- Permission to make digital or hard copies of all or part of this work for personal or tensive measurements of the execution times of Ardupilot's classroom use is granted without fee provided that copies are not made or distributed code over a Pixhawk board in a minimally invasive way, us- for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the ing custom hardware and during actual executions. author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or The model derived this way, however, turns out impracti- republish, to post on servers or to redistribute to lists, requires prior specific permission cal to carry out the actual testing process. As it is common and/or a fee. Request permissions from [email protected]. when applying exhaustive techniques, we stumble upon state DroNet’16, June 26 2016, Singapore, Singapore space explosion problems that hamper the conclusion of the © 2016 Copyright held by the owner/author(s). Publication rights licensed to ACM. ISBN 978-1-4503-4405-0/16/06. $15.00 verification process [13, 25]. To overcome these, Sec.5 il- DOI: http://dx.doi.org/10.1145/2935620.2935630 lustrates model reduction techniques we apply to shave the setup fast loop scheduler is rendered based on a model of flight dynamics embedded within SITL. A GCS can connect to the simulated Ardupilot instead of a real drone, and issue commands to drive way- time point navigation and remotely log the mission execution. Using SITL provides a practical means to quickly test functional properties of Ardupilot, or to check Ardupilot's Figure 1: Ardupilot's control loop. behavior in specific circumstances. Further, tests can be scripted. This allows one to build automatic testing frame- works, for example, to perform regression testing on nightly builds as new features are added. state space, ultimately obtaining a model usable on ordinary personal computers. 2.2 Related Work The outcome of the testing process on the reduced model, In embedded software, the lack of exhaustiveness is widely described in Sec.6, indicates that, for example, specific exe- acknowledged as a drawback of simulation-based testing [8]. cutions are possible where Ardupilot's fail-safe mode might The testing outcome only applies to that specific execution be arbitrarily delayed. This means that, even though a fault with given inputs. Simulations may also fall short in real- happens that requires the fail-safe mode to be enabled, the ism [8], for example, as it is difficult to accurately model latter might not operate immediately. Rather, the drone the execution times of embedded hardware. However, test- continues working (or not working) as in the absence of the ing on top of resource-constrained hardware is likely difficult fault. Through real-world controlled experiments using a without generating \heisenbugs": fictitious software defects custom-built aerial drone, we demonstrate the not-so-rare induced by probing the executions [15]. occurrence of such erroneous executions. Model-based testing complements simulations [11] by op- We conclude the paper by offering in Sec.7 an outlook on erating on an abstract representation of the system, which how the insights we present in this work may represent an allows to test a system without concretely running it. The input to further developments in the field. models are input to a tool together with developer-provided properties, that is, an encoding of what is considered the cor- 2. BACKGROUND rect behavior. The tool returns whether the model satisfies We describe the autopilot we consider and cast our work the properties in every possible execution, and is therefore in the broader context. exhaustive. If a property is violated, a counter-example is produced that shows an execution where a property does 2.1 Ardupilot not hold. Because of these features, model-based testing is Ardupilot is an open-source project [3] at the basis of widely used in safety critical software [8]. many commercial products, including those of 3DRobotics [1] Applying model-based testing to the special case of real- and many others, and boasts a large on-line community. It time software requires dedicated models and tools [12]. For well exemplifies the state of the art in the field of autopi- example, various kinds of timed automata [11, 13] to model lot development, platforms, and testing. Other existing au- the behavior of the system exist. Different model represen- topilots, such as OpenPilot [17] and Cleanflight [10], show tations require different algorithms to check the properties of similar characteristics. interest [13,25]. A vast literature exists on the subject [12], which shows how these techniques can be effectively applied Implementation. Fig.1 shows the execution of Ardupi- to a number of different application domains. lot's control loop. The fast loop only includes critical mo- Among these domains is also the one of avionics [6,21,22]. tion control functionality. The time left from the execu- However, compared to traditional avionic systems, robot tion of fast loop is given to an application-level scheduler drones are peculiar in at least three respects. First, many that distributes it among non-critical tasks, such as log- state-of-the-art systems, including Ardupilot, are the result ging.