Efficient and Agile Underwater Robots

Srihar kth royal institute of technology S ha Bhat Hydrobatics: Efficient and Agile Underwater Robots

Licentiate Thesis in Vehicle and Maritime Engineering Hydrobatics: Efficient and Agile Underwater Robots

SRIHARSHA BHAT

TRITA-SCI-FOU 2020:44

ISBN 978-91-7873-721-5 2020 KTH www.kth.se Stockholm, Sweden 2020 Hydrobatics: Efficient and Agile Underwater Robots

SRIHARSHA BHAT

Academic Dissertation which, with due permission of the KTH Royal Institute of Technology, is submitted for public defence for the Degree of Licentiate of Engineering on Monday the 14th of December 2020, at 2:00 p.m. in U1, Brinellvägen 28A, Stockholm.

Licentiate Thesis in Vehicle and Maritime Engineering KTH Royal Institute of Technology Stockholm, Sweden 2020 © Sriharsha Bhat

Cover photo: Swedish Maritime Robotics Center (SMaRC)

TRITA-SCI-FOU 2020:44 ISBN 978-91-7873-721-5

Printed by: Universitetsservice US-AB, Sweden 2020 i

Abstract

The term hydrobatics refers to the agile maneuvering of underwater vehicles. Hydro- batic capabilities in autonomous underwater vehicles (AUVs) can enable increased maneuverability without a sacrifice in efficiency and speed. This means innovative robot designs and new use case scenarios are possible. Benefits and technical challenges related to hydrobatic AUVs are explored in this thesis. The dissertation contributes to new knowledge in simulation, control and field applications, and provides a structured approach to realize hydrobatic capabilities in real world impact areas.

Three impact areas are considered - environmental monitoring, ocean production and security. A combination of agility in maneuvering and efficiency in performance is crucial for successful AUV applications. To achieve such performance, two technical challenges must be solved. First, these AUVs have fewer control inputs than degrees of freedom, which leads to the challenge of underactuation. The challenge is described in detail and solution strategies that use optimal control and model predictive control (MPC) are highlighted. Second, the flow around an AUV during hydrobatic maneuvers transitions from laminar to turbulent flow at high angles of attack. This renders flight dynamics modelling difficult. A full 0-360 degree envelope flight dynamics model is therefore derived, which combines a multi-fidelity hydrodynamic database with a generalized component-buildup approach. Such a model enables real-time (or near real-time) simulations of hydrobatic maneuvers including loops, helices and tight turns.

Next, a cyber-physical system (CPS) is presented – it safely transforms capabilities derived in simulation to real-world use cases in the impact areas described. The simulator environment is closely integrated with the robotic system, enabling pre- validation of controllers and software before hardware deployment. The small and hydrobatic SAM AUV (developed in-house at KTH as part of the Swedish Maritime Robotics Center) is used as a test platform. The CPS concept is validated by using the SAM AUV for the search and detection of a submerged target in ﬁeld operating conditions.

Current research focuses on further exploring underactuated control and motion planning. This includes development of real-time nonlinear MPC implementations running on AUV hardware, as well as intelligent control through feedback motion planning, system identiﬁcation and reinforcement learning. Such strategies can enable real-time robust and adaptive control of underactuated systems. These ideas will be applied to demonstrate new capabilities in the three impact areas. ii

Keywords: Autonomous Underwater Vehicles, Modeling, Simulation, Control, Field Testing, Cyber-physical Systems. iii

Sammanfattning

Termen hydrobatik avser förmåga att utföra avancerade manövrer med undervattensfarkoster. Syftet är att, med bibehållen fart och räckvidd, utvigda den operationella förmågan i manövrering, vilket möjliggör helt nya användningsområden för maxi- mering av kostnadseﬀektivitet. I denna avhandling undersöks fördelar och tekniska utmaningar relaterade till hydrobatik som tillämpas på undervattensrobotar, vanligen kallade autonoma undervattensfarkoster (AUV). Avhandlingen bidrar till ny kunskap i simulering, reglering samt tillämpning i experiment av dessa robotar genom en strukturerad metod för att realisera hydrobatisk förmåga i realistiska scenarier.

Tre nyttoområden beaktas - miljöövervakning, havsproduktion och säkerhet. In- om dessa nyttoområden har ett antal scenarios identifierats där en kombination av smidighet i manövrerbarhet samt effektivitet i prestanda är avgörande för robotens förmåga att utföra sin uppgift. För att åstadkomma detta måste två viktiga tekniska utmaningar lösas. För det första har dessa AUVer färre styrytor/trustrar än frihetsgra- der, vilket leder till utmaningen med underaktuering. Utmaningen beskrivs i detalj och lösningsstrategier som använder optimal kontroll och modellprediktiv kontroll belyses. För det andra är flödet runt en AUV som genomför hydrobatiska manövrar komplext med övergång från laminär till stark turbulent flöde vid höga anfallsvinklar. Detta gör flygdynamikmodellering svår. En full 0-360 graders flygdynamikmodell härleds därför, vilken kombinerar en multi-tillförlitlighets hydrodynamisk databas med en generaliserad strategi för komponentvis-superpositionering av laster. Det- ta möjliggör prediktering av hydrobatiska manövrar som t.ex. utförande av looping, roll, spiraler och väldigt snäva svängradier i realtids- eller nära realtids-simuleringar.

I nästa steg presenteras ett cyber-fysikaliskt system (CPS) – där funktionalitet som härrör från simuleringar kan överföras till de verkliga användningsområdena på ett eﬀektivt och säkert sätt. Simulatormiljön är nära integrerad i robot-miljön, vilket möjliggör förvalidering av reglerstrategier och mjukvara innan hårdvaruimplemen- tering. En egenutvecklad hydrobatisk AUV (SAM) används som testplattform. CPS- konceptet valideras med hjälp av SAM i ett realistiskt sceanrio genom att utföra ett sökuppdrag av ett nedsänkt föremål under fältförhållanden.

Resultaten av arbetet i denna licentiatavhandling kommer att användas i den fort- satta forskningen som fokuserar på att ytterligare undersöka och utveckla ny me- todik för reglering av underaktuerade AUVer. Detta inkluderar utveckling av real- tidskapabla ickelinjära MPC-implementeringar som körs ombord, samt AI-baserade reglerstrategier genom ruttplaneringsåterkoppling, autonom systemidentiﬁering och förstärkningsinlärning. Sådan utveckling kommer att tillämpas för att visa nya möj- ligheter inom de tre nyttoområdena. iv

Nyckelord: Autonoma Undervattensfarkoster (AUV), Modellering, Simulering, Reglerteknik, Fältprovning, Cyber-fysikaliska System(CPS). Preface

The project on hydrobatics is part of the Swedish Maritime Robotics Center (SMaRC), and is funded by the Swedish Foundation for Strategic Research / Stiftelsen för Strate- gisk Forskning (SSF).

I would like to thank my principal supervisor Assoc. Prof. Ivan Stenius for his invaluable support, guidance and camaraderie throughout the course of the PhD. I hope this collaboration leads us to even more exciting research and exploration. I would like to thank co-supervisor Prof. Dimos Dimarogonas for his guidance on control theory and research directions, and I wish to deepen this collaboration during my upcoming work on topics in underactuated control.

Colleagues working on the SAM project at SMaRC have been extraordinary in their support and tenacity, and it has been a privilege to work with Joseﬁne Sever- holt, Carl Ljung, Anton Svensson, Nils Bore, Ignacio Torroba, Özer Özkahraman, Christopher Sprague and Yiping Xie on the ﬁeld tests in sun, rain, wind, snow and ice! I want to acknowledge the excellent contributions of Tianlei Miao in his master’s thesis which provided valuable input for one of the papers included in this work. It has also been an enriching experience to collaborate with other SMaRC colleagues at KTH including Clemens Deutsch, Elias Strandell Erstorp, Viktor Lidström, Sebas- tian Thune, Filip Söderling, Lazaro Moratelli, Niklas Rolleberg, Prof. Peter Sigray, Prof. Jakob Kuttenkeuler, Assoc. Prof. John Folkesson and Prof. Petter Ögren among others. Thanks also to Joana Fonseca, for our collaboration in teaching.

I would like to thank other partners at SMaRC for providing valuable opportunities for discussion, workshops and experiments. These include Prof. Anna Wåhlin from the University of Gothenburg (for valuable discussions on environmental sensing, guidance on the Hugin AUV, and opportunities for ﬁeld tests), Prof. Nina Kirch- ner from Stockholm University (for ASIAQ, and ﬁeld tests at Askö), Prof. Fred- erik Gröndahl, Jean-Baptiste Thomas (for access and insight into algae farms) and Prof. Magnus Burman (for ASIAQ) from KTH, and the organizations MMT Swe-

v vi

den, Saab, FMV, FOI and WASP (for collaboration, technical discussions, access to test sites, workshops and joint tests).

A word of gratitude also to friends, family and well-wishers spread all over the world (including but not limited to India, Singapore, Sweden, Germany, USA, France, Japan, Lebanon, China ...), with whom I have shared several memorable ex- periences. I do not want to mention names here, since that would inevitably mean I would miss someone out. Most importantly, I want to thank my parents for their incessant support and love, and I would not be at this stage without them.

Whenever I faced challenges through the course of my research, I have been in- spired and motivated by reading works by Isaac Asimov, Haruki Murakami, Milan Kundera, Anna Akhmatova and Percy Bysshe Shelley, listening to enthralling pieces by the Late-Romantic and Modern composers Dmitri Shostakovich, Gustav Mahler, Pyotr Tchaikovsky and Igor Stravinsky; as well as following superlative sporting per- formances by Novak Djokovic, Rafael Nadal, Roger Federer, and the cricket teams of India and the West Indies. I would not have had the fortitude to come this far if it were not for these stalwarts.

My apologies in case I have not mentioned someone here, it is not intentional. Last, I hope the reader learns something new after reading the content in this thesis.

With warm regards,

Sriharsha Bhat, November 2020, Stockholm, Sweden. Appended Papers and Author Contributions

This thesis is a compilation of the following appended papers, which includes the described author contributions.

Paper A

S. Bhat and I. Stenius, "Hydrobatics: A Review of Trends, Challenges and Opportu- nities for Eﬃcient and Agile Underactuated AUVs," 2018 IEEE/OES Autonomous Underwater Vehicle Symposium (AUV), Porto, Portugal, 2018, pp. 1-8.

S. Bhat performed the study, authored the publication and presented the work at the conference. I. Stenius contributed with supervision, detailed discussions on the method, highlighting key contributions, writing and reviewing the text.

Paper B

S. Bhat, I. Stenius and T. Miao, "Real-time ﬂight simulation of hydrobatic AUVs over the full 0-360 degree envelope", Manuscript submitted to the IEEE Journal of Oceanic Engineering, 2020, pp.1-17.

S. Bhat wrote the paper, devised the method together with I.Stenius, generated results and discussed them. I. Stenius had the initial idea, supervised the work, was closely involved in discussions, added ﬁgures and text, and established the method together with S. Bhat. T. Miao performed studies in hydrodynamics that provided results for the paper.

vii viii

Paper C

S. Bhat, I. Stenius, N. Bore, J. Severholt, C. Ljung and I. Torroba Balmori, "Towards a Cyber-Physical System for Hydrobatic AUVs," OCEANS 2019 - Marseille, Mar- seille, France, 2019, pp. 1-7.

S. Bhat wrote and presented the paper, was one of the initiators of the idea together with I. Stenius and N. Bore, developed the control system and compiled the results. I. Stenius supervised the work, initiated the idea, and supported with hardware and software development as well as experiments, writing and review. N. Bore was responsible for a majority of the software development. J. Severholt and C. Ljung were responsible for hardware development and I. Torroba contributed with simulation results and localization modules.

Paper D

S. Bhat, I. Torroba, Ö. Özkahraman, N.Bore, C.Sprague, Y.Xie, I. Stenius, J. Sever- holt, C. Ljung, J.Folkesson and P. Ögren, "A Cyber-Physical System for Hydrobatic AUVs: System Integration and Field Demonstration" 2020 IEEE/OES Autonomous Underwater Vehicle Symposium (AUV), St.Johns, Newfoundland, Canada 2020, pp. 1-8.

S. Bhat took a leading role in writing and presenting the paper, developed the control system and was closely involved with describing the method and generating the results. I. Stenius supervised the work, together with J. Folkesson and P. Ögren. Ö. Özkahraman developed the mission planning and execution software, I. Torroba the localization module, C. Sprague the path planner and Y. Xie the target detection software. N. Bore developed the simulation environment, while J. Severholt and C. Ljung developed the hardware systems. All authors were closely involved with ﬁeld experiments that were presented in the paper. Ö. Özkahraman and I. Torroba were closely involved with writing and editing the text. Contents

1 Introduction 1 1.1 Impact areas ...... 2 1.2 Motivating hydrobatics ...... 5 1.3 Research question and contributions ...... 5 1.4 For the interested reader ...... 7

2 Underactuation 9 2.1 The challenge of underactuation ...... 9 2.2 An underactuated hydrobatic AUV ...... 10 2.3 Addressing the challenge ...... 12 2.4 For the interested reader ...... 15

3 Flight Dynamics 16 3.1 Component buildup workﬂow ...... 17 3.2 Multi-ﬁdelity look-up tables (LUTs) ...... 19 3.3 Results of hydrobatics simulations ...... 20 3.4 For the interested reader ...... 22

4 Cyber-physical Systems 26 4.1 Controller validation with Simulink ...... 27 4.2 Software validation with Stoneﬁsh ...... 28 4.3 Results of ﬁeld tests ...... 29 4.4 For the interested reader ...... 30

5 Conclusions and outlook 32 5.1 Real-time nonlinear MPC ...... 33 5.2 Intelligent control and motion planning ...... 34 5.3 New scenarios ...... 35

Bibliography 37

ix List of Figures

1.1 The big picture: linkages of applications in underwater robotics to UN Sustainable Development Goals (pictograms from [4])...... 2 1.2 Envisioned examples of use case scenarios and impact areas (courtesy: top left Fredrik Gröndahl, top middle: Nina Kirchner, and top right Saab AB.)...... 3 1.3 AUV performance trade-oﬀs: Flight and Hover style vehicles. (AUVs courtesy Saab AB)...... 6 1.4 Linkages between contributions made in the thesis...... 7 1.5 The SAM AUV developed at KTH as part of SMaRC, is a hydrobatic research platform...... 8

2.1 The hardware subsystems on SAM: 1) Battery pack; 2) Longitudinal Center of Gravity (LCG) trim; 3) Variable Buoyancy System (VBS); 4) Transversal center of gravity (TCG) trim; 5) Thrust vectoring and propulsion (picture courtesy Joseﬁne Severholt)...... 11 2.2 An illustration of a proposed underactuated control framework including a model predictive control, motion planning, system identi- ﬁcation and supervisory control...... 14

3.1 Flow separation around a slender body at high angles of attack. Symmetric vortex shedding is seen here, which would transition to asymmetric vortices and turbulent wakes at higher angles (from [40], reprinted in [41], used with permission from Elsevier)...... 17 3.2 A real-time AUV simulator can reflect real world-flight dynamics behavior and subsystems for generating hydrobatic maneuvers and tuning control strategies. Here, the SAM AUV is depicted, both in the simulation environment of the hydrobatics simulator and in reality during field tests. Actuator subsystems on the AUV such as the LCG and VBS are also modeled in simulation [15]...... 18

x LIST OF FIGURES xi

3.3 A schematic illustration of look-up tables capturing the nonlinear hydrodynamic forces for each sub-component over the full envelope.The dynamics of all components are combined to obtain the full AUV dynamics model [15]...... 19 3.4 The simulation framework in Simulink including component models with lookup tables, actuator models, sensors, state estimators, links to external systems, and plots. On the right example lookup tables over the full envelope for the lift and drag coefficients of the hull are presented...... 21 3.5 Trajectory of a simulated tight looping maneuver (unit: m). The maneuver takes the AUV over 0 − 360◦ in pitch [15]...... 22 3.6 Trajectory of a turbo-turn maneuver (unit: m). The AUV goes through 0 − 360◦ in yaw [15]...... 22 3.7 Trajectory of a helix maneuver (unit: m). Such a maneuver can enable inspection of targets in confined spaces [15]...... 23 3.8 Trajectory of a stationary LCG sweep maneuver (unit: m). The maneuver enables static pitch control [15]...... 23 3.9 Comparison of yaw angle and rate for a turbo-turn maneuver between a simulation run(red) and field test(blue). Qualitiatively similar behavior is seen between simulation and reality [15]...... 24 3.10 The behavior of the AUV in field tests during the Turbo-Turn maneuver in (a) is qualitatively similar to the simulated motion in (b). This enables the use of the simulator as a development tool, as maneuvers and control strategies can be pre-tested in simulation [15]...... 25

4.1 The SAM cyber-physical system architecture integrating a user interface for mission planning, control, autonomy and localization software, AUV hardware, and simulation tools. Two simulation environments are integrated – the Stonefish simulator to rehearse missions and integrate software, and the hydrobatics simulator to simulate advanced maneuvering [17]...... 27 4.2 Controllers can be evaluated first in simulation models in Simulink, and later deployed to the ROS environment for system integration and hardware deployment...... 28 4.3 Mission rehearsal with multiple SAM AUVs in the Stonefish simulation environment (left) and in the Neptus mission planning interface (right). SAM-1 is executing a mission while SAM-2 follows SAM-1. A sunken target (car) can be seen in the background [17]...... 29 xii LIST OF FIGURES

4.4 The spiral search mission in Askö in which the car is found. The plotted points are from the localization module. Colour indicates time, blue(start) to red(end). The red marker shows where the car is, known from a prior multibeam echosounder survey. Green lines show the planned path. Green marker shows the estimated location of the car from the on-board localization unit [17]...... 30 4.5 The trajectory of SAM in 3D (blue to red gradient in time). The desired waypoints are shown in green. Note the diﬀerent scales of the horizontal and vertical axes.The AUV could hold depth with an accuracy of 0.5m [17]...... 31

5.1 Future ideas in intelligent control for underactuated robotics. . . . . 35 5.2 Photographs of new scenarios: kelp farms (top, picture courtesy KosterAlg) and calving glacier fronts (bottom, picture courtesy Elias Strandell Erstorp/Sebastian Thune)...... 36 List of Tables

2.1 Overlapping control between subsystems on SAM [16]...... 12

xiii

Chapter 1

Introduction

Our society can perhaps turn to the oceans to ﬁnd answers to grand challenges such as climate change, and the need for sustainable development and security. Oceans cover the majority of the earth’s surface and are key to life – they contribute signiﬁcantly to biodiversity and oxygen generation, and absorb 50% of global CO2 emissions [1]. They have been intricately linked to human society throughout history, but are still largely unexplored and unknown. Exploring the world below water may yet hold the answer to deep questions about the origin of life and our past and future as a species. Technological advances in autonomous underwater vehicles (AUVs) can spark a revolution in exploring and understanding our oceans by rendering access to and observation of areas previously considered inaccessible. The next generation of underwater robots may revolutionize our view of our world with new perspectives and information; just like how satellites and space exploration have impacted earth and planetary observation since the 1970s. Hydrobatic AUVs can be part of this revolution [2, 3].

The term hydrobatics stems from aerobatics, and refers to agile maneuvering of underwater vehicles. Hydrobatic capabilities can enable autonomous underwater vehicles (AUVs) to be efficient in range and speed, as well as agile in maneuvering, thereby opening doors to disruptive designs and impactful new use case scenarios. The upcoming chapters will throw light on the societal benefits of hydrobatic AUVs and describe the challenges in generating elegant control strategies for hydrobatics. Novel ways of efficiently simulating such robots and translating simulations to real- world scenarios will be presented, and ideas for the future will be introduced.

1 2 CHAPTER 1. INTRODUCTION

Figure 1.1: The big picture: linkages of applications in underwater robotics to UN Sustainable Development Goals (pictograms from [4]).

1.1 Impact areas

Current advances in AUVs are driven by the need for long-term underwater presence with reduced human intervention and the capability to operate in unknown environments. From the results of such research, AUVs will allow us to explore unknown environments and gather valuable data to help answer challenging questions on climate and society that affect humanity as a whole. AUVs can prove to be useful tools in turning the oceans into a sustainable resource for food and energy. AUVs can safeguard society through challenging missions to track intruders and identify targets far beyond the capabilities of human pilots and divers. Underwater robots can impact UN Sustainable Development Goals [4] to different extents, as described in Figure 1.1. There is a direct link to G13: Climate Action and G14: Life Below Water since AUVs can be used as sensor platforms to gather data on environmental parameters. The use of AUVs can enable sustainable use of the oceans, and accel- erate other technologies such as renewable energy and biomaterials, linking to G6: Clean Water and Sanitation, G7: Affordable and Clean Energy, G12: Respon- sible Consumption and Production. Furthermore, the development of cutting-edge AUV technologies and applications is closely linked to G9: Industry, Innovation and Infrastructure. In this context therefore, hydrobatics can be an enabler in key impact areas including environmental sensing, ocean production and security (see Figure 1.2). CHAPTER 1. INTRODUCTION 3

Figure 1.2: Envisioned examples of use case scenarios and impact areas (courtesy: top left Fredrik Gröndahl, top middle: Nina Kirchner, and top right Saab AB.).

Considering environmental sensing, AUVs can be a powerful tool for gathering data for health monitoring of the oceans. AUVs could be used to explore regions underneath the polar ice shelves, providing critical environmental measurement and images that would enable us to better understand phenomena such as climate change, microplastic eﬀects or acidiﬁcation. Operations would take place in largely unknown environments with great uncertainty and potentially extreme condi- 4 CHAPTER 1. INTRODUCTION

tions [5]. An AUV may be deployed and recovered from a research vessel or through an under-ice docking mechanism (e.g. [6]). During the mission, the vehicle would navigate autonomously, detect and map interesting features and generate high resolu- tion environmental data throughout the water column. With hydrobatic capabilities, it would be possible to aim sensors in any direction for high spatial resolutions and panoramic images (e.g. [7]). A successful AUV in this scenario would have to be agile and maneuverable to avoid obstacles and disturbances in launch, recovery and data-gathering/inspection; while being eﬃcient and robust for long range transit and navigation in a hostile environment.

The concept of blue growth refers to treating the oceans as a sustainable resource, and ocean production with seaweed/algae farming is one example [8]. Different algal species have the potential to be farmed for food, raw materials, energy and nu- trients. Algae farms will range from a few hectares to several square kilometres in area, and teams of resident AUVs (such as [9]) can be used as valuable tools to in- spect the site, monitor crop health, and support licensing and harvesting operations. These AUVs will perceive and ’see’ the health of the crop by aiming environmental sensors, and fuse sensor information to efficiently navigate through an obstacle rich environment by maneuvering between moorings, cables and plants. The AUVs would cooperate with each other, as well as with human divers to perform missions for inspection, monitoring and harvesting. Being resident at the farm, the AUVs will dock with seafloor stations to recharge, exchange data and receive new instructions [10, 11]. Hydrobatics becomes relevant here since docking, precision maneuvering, obstacle avoidance and sensor aiming capabilities are crucial for successful operations.

Focusing on security, AUVs can offer force multiplier capabilities in target tracking, mine-hunting, anti-submarine operations and reconnaissance. Agile AUVs could cooperate with submarines and surface vessels in detecting and encircling targets, or perform reconnaissance surveys. Mobile docking for launch and recovery (e.g. from submarines or larger AUVs) becomes a critical performance [12, 13]. Furthermore, AUVs will be perceptive and intelligent to detect and identify dangerous objects (e.g. mines) and intruders. Multiple AUVs could cooperate with other agents to hunt and encircle targets and reconnoitre unknown environments [14]. Such scenarios place high demands on autonomy, cooperation and maneuverability; where efficiency and agility are both a requirement. This means hydrobatic capabilities can offer beneficial performance characteristics. CHAPTER 1. INTRODUCTION 5

1.2 Motivating hydrobatics

Considering the impact areas presented, docking, perception in unknown environments and swarms for cooperative autonomy become force multipliers to enhance AUV operational capabilities. It is clear that hydrobatics can play a key role as an enabler of such operations and is closely linked with these technology trends.

Robotic operations underwater have unique requirements in comparison to land, air and space; including a very dynamic environment and limited communication availability meaning autonomy, endurance and perception are crucial. These operational challenges necessitate design prioritization based on mission critical performance. Slender, streamlined and underactuated flight-style AUVs (optimized for efficiency and speed) are usually utilized for long range missions, but these lack the capability to perform advanced precision maneuvers. On the other hand, fully or over-actuated hover style AUVs are used if inspection or manipulation is considered, since agility in six degrees-of-freedom (DOF) is prioritized at the cost of speed and range. There are performance gaps in both flight style and hover style AUVs (as illustrated in Fig. 1.3), and bridging this gap can be an enabler of entirely new capabilities.

Specifically, hydrobatic capabilities mean that the robots can be efficient over long transits, while being agile in critical situations to achieve the objective. This means that a traditional target conflict can be resolved – leading to disruptive designs and use cases!

1.3 Research question and contributions

Given the motivations and impact areas described, a key research question is therefore- How can we push the boundaries of underactuated AUVs to make them more agile in 6DOF and therefore capable of hydrobatic maneuvers while still being eﬃcient in speed and range?

Exploring this research question and turning the idea of hydrobatics into reality is the focus of this work. Crucial technical challenges in underactuation and flight dynamics are identified and ideas are proposed to address them. To translate these ideas to real-world scenarios, a framework is presented to evaluate new control strategies in simulation environments, and to then deploy these strategies to hardware in field operating conditions. The following contributions are thus made:

1. Paper A: A literature study is performed on trends, challenges and opportunities related to hydrobatic AUVs [2]. 6 CHAPTER 1. INTRODUCTION

Figure 1.3: AUV performance trade-oﬀs: Flight and Hover style vehicles. (AUVs courtesy Saab AB).

2. Paper B: An efficient means of simulating hydrobatic maneuvers in real time over the full 0 − 360◦ flight envelope is presented [15]. 3. Paper C: A cyber-physical system concept to transform the control strategies and autonomy packages developed in simulation to real hardware [16] is introduced. 4. Paper D: The cyber-physical system is demonstrated in a real-world inspection scenario [17] The linkage and flow between the different papers is presented in Figure 1.4.

There is also a certain urgency in developing this next generation of AUVs and transforming ideas to demonstrations. Based on current evidence, we are running out of time before the climate and environment are irrevocably altered [18]; and thus the sooner new ideas are realized, the better chance of them being industrialized to have a positive impact on these challenges. This motivates a large portion of this work to aim towards real-world demonstrations. The real-world AUV on which the ideas are tested is the hydrobatic SAM AUV (see Figure 1.5) developed at KTH (as part of the Swedish Maritime Robotics Center (SMaRC)). CHAPTER 1. INTRODUCTION 7

Figure 1.4: Linkages between contributions made in the thesis.

1.4 For the interested reader

The impact areas in environmental sensing, ocean production and security are further described in Paper A. Technology trends in docking, perception and swarms are highlighted in detail with references to recent literature in AUV research in that paper. Further information on use case scenarios is available in [19]. 8 CHAPTER 1. INTRODUCTION

Figure 1.5: The SAM AUV developed at KTH as part of SMaRC, is a hydrobatic research platform. Chapter 2

Underactuation

Underactuation is a very exciting and yet challenging aspect of hydrobatics. Hy- drobatic capabilities require controllability in unstable flow conditions with fewer control inputs than possible degrees of freedom. The complexity of the same neces- sitates several innovative techniques to perform and exploit hydrobatic maneuvers with minimum effort. The focus here is to describe and define the challenge, to demarcate its boundaries, and to suggest solution strategies.

2.1 The challenge of underactuation

Following Newton’s second law of motion, the dynamics of an underwater robot are typically represented by a second order ordinary diﬀerential equation as

x¨ = f(x, x,˙ u, t) . (2.1) In (2.1), x is the state vector describing positions, u is the vector containing all control inputs and t is the time variable [20]. The dynamics of the system in (2.1), can usually be divided into a passive space f1 and a controllable space f2

x¨ = f1(x, x,˙ t) + f2(x, x,˙ t)u . (2.2) It can be seen from (2.2) that the system is now linearly dependent on the control inputs u. By deﬁnition, this system is fully actuated if the rank of f2 is equal to the dimension of the state vector x, i.e.

rank (f2(x, x,˙ t)) = dim(x) . (2.3) A dynamic system is deﬁned as fully actuated from a control perspective if for each degree of freedom there is a unique control input actuating that degree of freedom. All the state variables in x are connected to control inputs in u for such a

9 10 CHAPTER 2. UNDERACTUATION

system. Note that this definition does not take into account saturation effects on the controller inputs. Now, a dynamic system (on the form of (2.2)) is defined as underactuated if the rank of f2 is less than the dimension of the state vector x, i.e.

rank (f2(x, x,˙ t)) < dim(x) . (2.4) In this case a unique control input does not actuate each degree of freedom, and control inputs from u will be able to inﬂuence some accelerations of x, but not all. A form of underactuation may also be given by a system that is fully actuated according to deﬁnition in (2.3), but has severe saturation limits on the possible control inputs.

An AUV is a 6 degree-of-freedom mechanical system - the state vector x has a dimension of 6, while the rank of f2 can range from as low as 2 or 3 for a flight style vehicle and up to 6 (or even more) for a hover style AUV or Remote Operated Ve- hicle (see Figure 1.3). Existing 6DOF hover-style robots are usually fully-actuated (and sometimes even over-actuated); with a number of thrusters enabling decoupled control of the motion in specific degrees of freedom. This makes them control trace- able between inputs and motions, and a degree of linearization of system dynamics is possible. While having fewer actuators would make the system less complex (and thereby more robust and efficient); controlling its behavior to achieve precise motions is not trivial.

Coupling between terms and complex dynamics (f1 in (2.2)) makes it particu- larly challenging to trace motions to controls [21]. The control challenge increases when hydrobatic maneuvers come into the picture, since requirements on position- ing, response time and accuracy become more demanding. It is important to predict the system dynamics to enable advanced maneuvers of underactuated vehicles. The natural dynamics as well as the unstable regimes of the system can then be exploited to obtain the desired motions. Following the natural dynamics of the system (rather than opposing them by brute force in a fully actuated robot) also enables energy eﬃ- cient control, as typically very limited control inputs are required to gently steer the vehicle along a natural dynamic path of the system. Therefore, eﬀective methods for underactuated system control and motion planning are necessary in order to fully exploit the abilities of the available actuators.

2.2 An underactuated hydrobatic AUV

The key challenge that arises due to underactuation is to control agile 6DOF maneuvers with minimum actuators and control inputs. We consider the slender SAM AUV that is being developed at the Swedish Maritime Robotics Center as a case study to further explore the underactuated control problem. SAM, short for Small and Aﬀordable Maritime Robot, is designed to demonstrate hydrobatic capabilities, CHAPTER 2. UNDERACTUATION 11

Figure 2.1: The hardware subsystems on SAM: 1) Battery pack; 2) Longitudinal Center of Gravity (LCG) trim; 3) Variable Buoyancy System (VBS); 4) Transversal center of gravity (TCG) trim; 5) Thrust vectoring and propulsion (picture courtesy Josefine Severholt). and has a unique actuator configuration to enable agile maneuvers [16, 17]. The SAM AUV is around 1.5m long and weighs close to 15kg. SAM’s actuators include counter-rotating propellers (in a duo-prop configuration), a thrust vectoring nozzle, a movable battery-pack for longitudinal changes in center of gravity (c.g.) position (LCG), rotating weights for transversal changes to c.g. position (TCG) and a variable buoyancy subsystem (VBS) that can pump water in and out of a tank (see Figure 2.1).

It is possible to combine the eﬃciency of the duo-prop propulsion system and the slender form factor with heightened maneuverability (thanks to the trim and buoyancy subsystems) by utilizing these actuator subsystems in tandem. SAM can be made to pitch up to ±90◦ using the LCG system, hover in position using the VBS to maintain its depth; or even roll while stationary using the TCG. Furthermore, new control sequences for tight turns and loops can be utilized using the combination between the counter-rotating propellers and thrust vectoring.

Controlling such a system is however not straightforward, since the subsystems do not independently influence specific degrees of freedom. The interactions and influences between the actuator subsystems and the respective degrees of freedom are presented in Table 2.1. Such an actuator configuration means that most degrees of freedom in SAM cannot be independently influenced, and thus require nuance in 12 CHAPTER 2. UNDERACTUATION

Table 2.1: Overlapping control between subsystems on SAM [16].

Subsystem x y z roll pitch yaw LCG × VBS × × TCG × Thrust vectoring × × × × Counter-rotating propellers × × control. For example, the depth z can be inﬂuenced by the VBS, and also by the thrust vectoring and propulsion. When the VBS is used, the pitch angle is also inﬂuenced, since there is a change in the moment balance. Such interactions mean that single- input single-output control strategies can have limitations, and motivate a model based multi-input multi-output (MIMO) control approach. Furthermore, some of the subsystems have saturation limits, which means that additional constraints need to be considered.

2.3 Addressing the challenge

The focus is to derive eﬃcient underactuated control strategies to realize hydrobatic maneuvers, and to maintain stability and controllability in the presence of uncertainties, disturbances and constraints.

A set of linear single-input single-output controllers (e.g. using proportional- integral-derivative (PID) control) can be used to track individual output to controls at the simplest level, but this has obvious limitations due to couplings and nonlinearities unless decoupled states are tracked. Cascading these linear controllers can address some of these nonlinear couplings and this is a common strategy to enable diving or steering control. Such strategies enable us to obtain a functional system in a simple and straightforward manner, but these are not necessarily optimal. The natural dynamics of the system are not considered, constraints cannot be easily addressed and the control objective cannot be customized.

A key point is thus to incorporate the dynamics of the system in the control actions, enabling eﬃcient controls. A model of the system’s dynamics of the form of (2.1) would be useful in deriving such control strategies. This way, relevant couplings could be exploited by the controller. Techniques in optimal control and nonlinear model predictive control (MPC) can be useful to bring in such a nonlinear dynamics model and customized objectives and constraints into the picture. These can help in prioritizing eﬃciency, adaptability and robustness. CHAPTER 2. UNDERACTUATION 13

In the ﬁeld of optimal control, the control problem (shown here in continuous time) is reformulated as an optimization problem, with the goal of minimizing (or maximizing) an objective function subject to a set of constraints as

R tf min φ(xf ) + f0(x, u)dt, ti

s.t x˙ = d(x, u), (2.5) x ∈ X, u ∈ U,

where x represents the states, u represents the controls, φ(xf ) is the cost at the ﬁnal state xf , ti and tf refer to the initial and ﬁnal time, f0 is the objective function at each time instant. The dynamics of the system are represented by a nonlinear ODE through the function d(x, u). The states and controls are constrained to sets X and U. The problem can be solved using a variety of analytical and numerical techniques, the most prominent being Dynamic Programming [22] or Pontryagrin’s Minimum Principle [23]. Direct numerical optimization methods such as Nonlinear Program- ming are also applicable [24].

A common objective term could be to minimize deviations from a setpoint. This objective can be augmented with additional terms to minimize effort, or minimize time (or a combination of these criteria). The constraints include the system dynamics model (linear or nonlinear), initial (and final) states, limits on actuators/controls/states and customized safety limits. On solving the optimization problem, a sequence of control inputs (u∗(t) ) and the respective optimal state trajectories (x∗(t)) are derived, and the optimal control can be applied to the system. However, the applied control is open-loop, and the effect of disturbances on the state are not considered. Disturbances can be accounted for if the optimal control u∗ is recalcu- lated for a pre-defined prediction horizon t + T at each time instant t (i.e. ti = t and tf = t + T in the optimal control problem above). In this case, we arrive at finite horizon model predictive control (MPC).

The stability of the system can be guaranteed if a linearized system model is used at the last prediction step t + T to check for infinite horizon convergence [25]. Additionally, uncertainties can potentially be considered in the form of additional constraints, model parameters or controller augmentations (e.g. by constraining the system to a tube or funnel of reachable states through an additional controller that propogates the effect of uncertainties [26, 27, 28]). The model can be updated based on measured response from the environment [29] – this can lead us to adaptive control, or even dual control where the model is identified and simultaneously controlled. 14 CHAPTER 2. UNDERACTUATION

Figure 2.2: An illustration of a proposed underactuated control framework including a model predictive control, motion planning, system identiﬁcation and supervisory control.

Different control objectives can be considered and updated from a motion planner or supervisory controller, thereby allowing for different behaviors and actions. It is therefore crucial to derive a flexible, light and real-time model predictive controller that is capable of running on target hardware (e.g. [30]), so that a foundation for further augmentations can be established. This is the focus of current research. Once such strategies are attempted in the field or in simulation environments to generate training data-sets, techniques in reinforcement learning could also be used to augment the control policies, enabling model-free adaptive control [31, 32, 33]. These ideas can be brought together in a framework such as that presented in Figure 2.2.

While techniques in nonlinear model predictive control have shown promise with theoretical and simulation results (especially in autonomus driving, aircraft flight control and chemical process control), there are very few implementations on AUVs in field conditions (such as e.g. [30]). Ideas in feedback motion planning have been demonstrated for flying [34] and grasping [35], but not yet underwater, where the measurement uncertainty is significantly higher. Reinforcement learning, as well as other combinations of learning and control are currently gaining interest, and there is therefore motivation to apply such combinations to hydrobatics. Therefore, a pri- ority is to derive elegant strategies for underactuated control using these techniques in challenging environments. Solutions found for the underactuated control problem in hydrobatics can also then be potentially translated to other fields, especially for flight control, manipulation and biomimetics.

We will be able to apply new techniques to perform very elegant maneuvers with the help of underactuated control strategies. To do so, however, better understanding and modeling of the AUV’s ﬂight dynamics is crucial. CHAPTER 2. UNDERACTUATION 15

2.4 For the interested reader

The challenge of underactuation is described in detail in Paper A, and literature from underactuated robotics is presented suggesting means to solve the problem. The SAM AUV is presented in greater detail in Papers C and D, together with implementations of linear and cascaded controllers. Chapter 3

Flight Dynamics

An AUV can enter high angles of attack during agile hydrobatic maneuvers, which can lead to unsteady, transitional and turbulent flow conditions around the body (see Figure 3.1). Such conditions go far beyond the usual well described and understood flow region before stall – this means modeling the flight physics becomes an extremely complex task. Very few papers in the AUV literature attack this challenge [36, 37] and questions remain open. Most publications on AUV simulation models focus on generalized frameworks and predictions at low angles of attack [38, 39], since most current AUV applications do not face such flow conditions except in extreme scenarios (such as heavy cross-flow). Modeling over the full envelope is very difficult to perform in real time since most solution strategies are either not accurate enough, or not applicable in real-time.

Focusing on this challenge however, offers new opportunities. The ability to simulate hydrobatic AUVs in real-time enables us to further explore their capabilities in the specific use case scenarios described previously. Accurate and efficient simulation models can be used to prototype new hydrobatic maneuvers and to derive new underactuated control strategies (based on different actuator configurations) at low risk and cost (see Figure 3.2). Furthermore, having efficient prediction models enables the use of optimal and model predictive controllers for the scenarios considered, and improves the efficacy of the control strategies developed. Inspiration to derive such models is garnered from the aerospace domain, where analytical and semi-empirical (ASE) high angle-of-attack models have been used for simulating aircraft, missiles and spacecraft in critical scenarios. Computational techniques such as computational fluid dynamics (CFD) offer increasingly accurate flow predictions albeit at a high computational cost [42]. If data from field tests are available, system identification techniques may enable us to learn the dynamics from the data [43]. These varied techniques are threaded together to generate a nonlinear simula-

16 CHAPTER 3. FLIGHT DYNAMICS 17

Figure 3.1: Flow separation around a slender body at high angles of attack. Symmet- ric vortex shedding is seen here, which would transition to asymmetric vortices and turbulent wakes at higher angles (from [40], reprinted in [41], used with permission from Elsevier). tion model that is capable of real-time simulations over the full 0 − 360◦ angle of attack envelope [15]. Two main innovations have been made to enable fast and efficient simulations without sacrificing accuracy – a generalized component build-up workflow (to improve efficiency) and a multi-fidelity lookup table (LUT) concept (to improve accuracy).

3.1 Component buildup workﬂow

A key idea is to treat individual components of the AUV (including the hull, wings, fins, actuators etc.) separately with their own dynamic properties (e.g. mass, inertia, damping) and to combine them in the full simulation model. This enables easy as- sembly of simulation models for complex systems, and through the composition of simple sub-components allows for very fast simulations. Such a component build-up approach is straightforward, general and flexible; and enables almost any robot to be simulated if the dynamics are well defined. Such flexibility is desirable if different robots and actuator configurations need to be tested. Using this approach, different models of varying fidelity levels can be combined and superposed. However, a trade-off is that interaction effects between components (such as shadowing of control surfaces) are difficult to capture in such an approach. 18 CHAPTER 3. FLIGHT DYNAMICS

Figure 3.2: A real-time AUV simulator can reflect real world-flight dynamics behavior and subsystems for generating hydrobatic maneuvers and tuning control strategies. Here, the SAM AUV is depicted, both in the simulation environment of the hydrobatics simulator and in reality during field tests. Actuator subsystems on the AUV such as the LCG and VBS are also modeled in simulation [15]. CHAPTER 3. FLIGHT DYNAMICS 19

Figure 3.3: A schematic illustration of look-up tables capturing the nonlinear hydrodynamic forces for each sub-component over the full envelope.The dynamics of all components are combined to obtain the full AUV dynamics model [15].

3.2 Multi-ﬁdelity look-up tables (LUTs)

The component buildup approach offers speed, but its accuracy depends on the individual component models. Therefore, look-up tables (LUTs) are used to represent the hydrodynamic forces and moments due to the different AUV components (see Figure 3.3). These LUTs are populated using the best available data for the components over the full 0 − 360◦ angle of attack envelope and over an operating range of velocities (Reynolds’ numbers). A multi-fidelity paradigm is used to populate the lookup tables. High fidelity data might be available readily at low angles of attack (e.g. for a wing in laminar flow), while the available data might be less accurate at high incidence angles (e.g for a slender body post-stall). The best available datasets at each flight condition are assembled to obtain a full envelope database and capture nonlinear dynamic behavior qualitatively.

To compose such a multi-ﬁdelity database, any appropriate dataset may be used. Here, a variety of input sources are exempliﬁed including: 1. Jorgensen’s analytical formulation for slender bodies at high angles of attack, which was originally intended for rockets and spacecraft [44];

2. DATCOM, a comprehensive compendium of aerodynamic calculation methods for ﬁghter aircraft; which also includes methods for slender bodies at high angles of attack [45];

3. Computational Fluid Dynamics (CFD) calculations [46]; and 20 CHAPTER 3. FLIGHT DYNAMICS

4. Analytical and numerical methods for airfoils and wings including lifting line theory and solvers such as XFOIL [47].

These methods have different fidelity levels and can be combined to ensure that the best available data is used to capture the full flight envelope. Rapid methods such as DATCOM and lifting line theory can be used to populate the database over the entire operating region, while slower but higher fidelity methods such as CFD can be used to provide more accurate snapshots in critical flow regions. Furthermore, if data from experiments from field tests are available, such data can also be imported to the database. It is to be noted that the different methods need to be dimensional- ized appropriately when applied to AUVs.

Nonlinear behavior and post-stall effects can be captured with these LUTs. The LUTs are pre-computed before the simulation is run. During the simulation, the forces are interpolated from the LUTs at each flight condition (Reynolds’ number, angle of attack) of the AUV. System identification techniques may be used to learn new dynamics models to improve the LUTs in areas of low accuracy.

3.3 Results of hydrobatics simulations

The two ideas presented are combined using software from Mathworks 1 (Matlab and Simulink) for real-time flight simulation of hydrobatic maneuvers [15]. The SAM AUV introduced previously (with the actuator configuration presented in Fig- ure 2.1) is used as a case study. The hydrodynamic LUTs are pre-computed from the computer aided design (CAD) geometries of the different components, and the components and actuator models are assembled in Simulink (Figure 3.4).

Simulations are performed by time-stepping the assembled dynamics model using a Runge-Kutta solver, and diﬀerent actuator inputs are tested to obtain hydrobatic maneuvers (unit quaternions are used to represent orientations so that singularities due to Euler angles at 90◦ pitch angle can be avoided). Studied hydrobatic maneuvers include :

1. Tight looping, where the AUV performs a vertical ﬂip by asynchronously cycling the propellers and thrust-vectoring system through maximum values (Figure 3.5);

2. Turbo-turning, where the AUV turns with a very small turning radius, using a similar sequence as above (Figure 3.6);

1https://se.mathworks.com/ CHAPTER 3. FLIGHT DYNAMICS 21

Figure 3.4: The simulation framework in Simulink including component models with lookup tables, actuator models, sensors, state estimators, links to external systems, and plots. On the right example lookup tables over the full envelope for the lift and drag coeﬃcients of the hull are presented.

3. Helical inspection, where the AUV follows a helical trajectory in a conﬁned environment (Figure 3.7); and

4. Stationary LCG sweep, where the AUV pitches up to ±90◦ using only the Longitudinal Center of Gravity subsystem thus enabling inspections without the use of the propellers (Figure 3.8).

Each of these maneuvers can be rapidly simulated, and the aim is to use the control sequences developed in the actual AUV. These real-time ﬂight simulations can show great value in designing intelligent controllers.

The turbo-turn sequence has also been verified in the field, and the simulation model has shown qualitatively similar behavior to real world tests (see Figures 3.9 and 3.10). Such maneuvers can be used in real-world scenarios – for example the turbo-turn can enable maneuvering and inspections in very confined areas, while the LCG sweep can make the AUV extremely agile and enable sensor aiming at specific targets.

To further augment the simulation models used, it can be beneﬁcial to learn from the response of the real system in ﬁeld tests, and thereby update the dynamics parameters and LUTs. 22 CHAPTER 3. FLIGHT DYNAMICS

Figure 3.5: Trajectory of a simulated tight looping maneuver (unit: m). The maneuver takes the AUV over 0 − 360◦ in pitch [15].

Figure 3.6: Trajectory of a turbo-turn maneuver (unit: m). The AUV goes through 0 − 360◦ in yaw [15].

3.4 For the interested reader

A closer look at the equations involved can help the reader appreciate the nuances involved in modeling hydrobatic AUVs. The dynamics of an AUV can be represented CHAPTER 3. FLIGHT DYNAMICS 23

Figure 3.7: Trajectory of a helix maneuver (unit: m). Such a maneuver can enable inspection of targets in conﬁned spaces [15].

Figure 3.8: Trajectory of a stationary LCG sweep maneuver (unit: m). The maneuver enables static pitch control [15]. 24 CHAPTER 3. FLIGHT DYNAMICS

Figure 3.9: Comparison of yaw angle and rate for a turbo-turn maneuver between a simulation run(red) and ﬁeld test(blue). Qualitiatively similar behavior is seen between simulation and reality [15]. in a vectorial form [38] as

MRBν˙ + CRB(ν)ν + MAν˙ + CA(ν)ν+

D(ν)ν + g(η) = τC , (3.1) where η refers to the position vector, ν is the velocity vector, (Mν˙ + CRBν) captures the forces due to rigid body dynamics with the mass and inertia matrix M and the matrix containing Coriolis and centripetal forces CRB. g(η) represents the hydrostatic terms while (MAν˙ + CAν + D(ν)ν) captures the hydrodynamic forces due to added mass (MA,CA) and damping (D(ν)). Populating these matrices, and performing time integration of these equations of motion enables flight simulation. While computing the rigid body and hydrostatic terms is straightforward, there is a great deal of complexity and nuance in representing the hydrodynamic terms. For most AUVs at low angles of attack, constant values are used for the coefficients in these matrices since they do not change sharply over the operating region. However, the nonlinear hydrodynamic behavior needs to be captured in the case of hydrobatic AUVs, and LUTs can be beneficial. Detailed information on the methods, validation and maneuvering results from real-time flight simulation over the full envelope can be found in Paper B . CHAPTER 3. FLIGHT DYNAMICS 25

(a) Right Turbo-Turn, Top View, Experiment

(b) Right Turbo-Turn, Top View, Simulation

Figure 3.10: The behavior of the AUV in ﬁeld tests during the Turbo-Turn maneuver in (a) is qualitatively similar to the simulated motion in (b). This enables the use of the simulator as a development tool, as maneuvers and control strategies can be pre-tested in simulation [15]. Chapter 4

Cyber-physical Systems

Translating novel capabilities and control strategies derived in virtual simulation environments to real-world scenarios is essential. However, such translation is also tremendously diﬃcult, since additional disturbances such as waves and currents, wakes from other vehicles and objects, uncertainties in sensing and communication, actuator delays and saturation, couplings caused due to unmodelled eﬀects and environmental factors such as temperature, salinity or density could render the results of simulations to be inaccurate or even invalid. Therefore, there is a need for a structured approach to reduce the risks and uncertainties associated with this transition. Close intertwining between virtual and physical worlds can be crucial for successful application cases, and this motivates tight integration of simulation environments with the full robotic system. This means that a network of sensors and actuators is tightly coupled with software, intelligence and simulations across platforms, length and time scales. Such intertwined systems are called cyber-physical systems (CPS), and they can provide a streamlined pathway to translate new ideas to reality with reduced risk.

This concept is illustrated in Figure 4.1. The software and hardware subsystems are all part of a common sensing and computing network. The SAM AUV is at the center of the cyber-physical system, and utilizes the Robot Operating System (ROS) 1 as its common network communication interface between software components. The actuators and sensors within SAM utilize a common communication protocol with the Controller Area Network bus (CAN bus) 2. To blur the distinctions between hardware and software with minimal delay and high robustness, a translation link exists between the CAN-bus and ROS 3. The AUV system is coupled with virtual

1https://www.ros.org/ 2https://uavcan.org/ 3https://github.com/smarc-project/uavcan_ros_bridge

26 CHAPTER 4. CYBER-PHYSICAL SYSTEMS 27

Operator SAM SW

Control Signals Plan IMC-ROS CAN-ROS Bridge Bridge Sensor Data Neptus Status Plan

Topics Status Web SAM Hardware Interface Behavior Sensor Data Dead Reckoning Tree Pose Pose Action

Controllers Controllers Motion Planner Set Points Controllers Control Signals Stonefish Simulator

Simulink Simulator

Figure 4.1: The SAM cyber-physical system architecture integrating a user interface for mission planning, control, autonomy and localization software, AUV hardware, and simulation tools. Two simulation environments are integrated – the Stoneﬁsh simulator to rehearse missions and integrate software, and the hydrobatics simulator to simulate advanced maneuvering [17]. validation of subsystems, controllers, maneuvers and missions. Therefore simulations of varying time and length scales are used to model and validate various vehicle and mission scenarios. Simulator environments (such as the hydrobatics simulator presented in Chapter 3) can publish and subscribe to topics within the ROS network with the same interface as the real AUV, enabling rapid validation in the loop. In this way the hardware, onboard software, and external evaluation environments are tightly coupled. This system can also be scaled to include multiple robots and further external tools for mission planning and payload post-processing.

4.1 Controller validation with Simulink

A tight link to virtual validation means that one can fail several times in simulation until a fault-tolerant and robust system is achieved. The cyber-physical link enables the best solutions to be looped between reality and simulation with high confidence. With a focus on hydrobatics, control strategies designed within the real-time flight simulator can be readily translated to the real AUV hardware (see Figure 4.2). High risk maneuvers such as the turbo-turn sequence can be optimized in simulation before field testing. Furthermore controllers for depth, pitch and heading control can be tuned and then deployed to hardware. Similarly, if model predictive controllers, or reinforcement learning policies are developed in simulation, these can also be translated to hardware thanks to the system structure. 28 CHAPTER 4. CYBER-PHYSICAL SYSTEMS

Figure 4.2: Controllers can be evaluated ﬁrst in simulation models in Simulink, and later deployed to the ROS environment for system integration and hardware deployment.

4.2 Software validation with Stoneﬁsh

To further expand on the cyber-physical system; a second simulation environment 4 based on the open-source Stonefish software package [48] provides a virtual world to test the entire software system on the AUV. The environment can be represented including specific bathymetry, submerged objects and water conditions, albeit with less numerical and physical accuracy than the flight simulator (due to processing constraints). The control systems linked to hydrobatics also interact closely with the autonomy, perception and localization modules. The focus of the Stonefish simulator is to enable the evaluation of planning, autonomy and perception packages. In this way, full system runs can be performed for early fault detection. Entire missions (e.g. to search for a submerged target or to survey an area) can be rehearsed, and high risk activities such as the multi-agent operations can be explored (see Figure 4.3).

The two simulator environments complement each other, and ensure that when new autonomy software is deployed to the actual AUV, it is pre-validated. There are challenges with translating controllers and localization systems to reality, since several disturbances and nonlinearities can necessitate ﬁne-tuning. However, a basic foundation has been constructed as a springboard for better ﬁne-tuning on board on the real environment. 4https://github.com/smarc-project/sam_stonefish_sim CHAPTER 4. CYBER-PHYSICAL SYSTEMS 29

Figure 4.3: Mission rehearsal with multiple SAM AUVs in the Stoneﬁsh simulation environment (left) and in the Neptus mission planning interface (right). SAM-1 is executing a mission while SAM-2 follows SAM-1. A sunken target (car) can be seen in the background [17].

4.3 Results of ﬁeld tests

Taking the CPS concept in Figure 4.1 to reality, an inspection scenario has been realized with the SAM AUV in the Baltic Sea at the island of Askö in field operating conditions (details can be found in [17]). SAM was used to autonomously search for and detect a submerged passenger car (a Mini Cooper) using its payload cameras. To realize such a scenario, a search mission was planned as a sequence of waypoints using the Neptus mission planning interface 5. A behavior tree enabled autonomous mission execution, sending targets to specific actions. A motion planner read these targets and generated setpoints for controllers. A set of flight and trim controllers enabled the AUV to reach the target waypoints, acting on feedback from the localization module. A perception module processed the feed from the payload cameras, and fed back information on target detection to the behavior tree. Furthermore, SAM was capable of re-planning in the case of detection, and could react to emergency situations. This entire system was thus validated in a real-world use case (see Figure 4.4 for the results of the search mission overlaid with a satellite image, and Figure 4.5 for the 3D trajectory of SAM computed using dead reckoning during the mission). Additionally, hydrobatic capabilities such as the turbo-turn sequence and static pitch and depth control were also demonstrated in the field to improve mission execution, showing the translation of these capabilities to the real world.

The ﬁeld test demonstrated that a streamlined sequence exists for translating ideas from simulation to reality. The elements of the cyber-physical system enable us to appreciate and exploit the nuances of the use case scenarios. For example, the

5https://www.lsts.pt/toolchain/neptus/ 30 CHAPTER 4. CYBER-PHYSICAL SYSTEMS

Figure 4.4: The spiral search mission in Askö in which the car is found. The plotted points are from the localization module. Colour indicates time, blue(start) to red(end). The red marker shows where the car is, known from a prior multibeam echosounder survey. Green lines show the planned path. Green marker shows the estimated location of the car from the on-board localization unit [17]. use of the simulation tools enabled fine-tuning of the interaction between the higher level planners and the lower level controllers and reduced the chance of conflicting objectives. While traditional flight control strategies using cascaded feedback controllers have been used for maneuvering at this stage, the overall system architecture offers a strong framework for the development of further novel control and motion planning strategies. This offers inspiration for further work in more advanced control strategies for challenging use cases such as commercial algae farm operations or under-ice monitoring. Such cyber-physical systems can link closely to the revolution in Industry 4.0 (with a focus on interconnection between systems, autonomy and information transparency [49, 50]), and can enable rapid development of new capabilities for challenging and novel use cases.

4.4 For the interested reader

The cyber physical system concept is explored in Papers C and D. Detailed descrip- tions of hardware and software subsystems, as well as simulation environments are provided, and results and challenges are further highlighted. Paper C introduces the idea and focuses on the ROS-CAN bus link, while Paper D focuses on system integration for ﬁeld testing. CHAPTER 4. CYBER-PHYSICAL SYSTEMS 31

Figure 4.5: The trajectory of SAM in 3D (blue to red gradient in time). The desired waypoints are shown in green. Note the diﬀerent scales of the horizontal and vertical axes.The AUV could hold depth with an accuracy of 0.5m [17]. Chapter 5

Conclusions and outlook

The reader has been introduced to the concept of hydrobatics in the course of this work. Hydrobatic AUVs have been defined, and impact areas in environmental sensing, ocean production and security have been delineated. A first key challenge in underactuated control has been introduced, and solution strategies have been proposed. A second key challenge in flight dynamics modeling has been addressed, and real-time simulations of hydrobatic maneuvers have been demonstrated. Such simulations have been translated to real-world use cases through a cyber-physical system; which encompasses not just hydrobatic control strategies, but also autonomy packages and hardware. Through this sequence, hydrobatics has been taken from concept to reality.

Revisiting the research question, the boundaries of underactuated AUVs have been demarcated by identifying the technical challenges in underactuated control and flight dynamics modeling (Paper A). A novel framework for handling advanced hydrodynamic modelling is a key contribution of this thesis (Paper B). Further, the hydrodynamic models together with the cyber-physical system concept have been developed and verified in both simulations and experiments (Papers C and D) to enable us to further push the boundaries for new strategies in underactuated control. This can enable us to realize AUVs that are both efficient in speed and range, as well as agile in maneuvering.

The current focus is therefore to further explore the challenge of underactuation, by designing a lightweight model-predictive controller to control hydrobatic maneuvers in ﬁeld experiments. Moving forward, further intelligence to handle uncertainties and adapt to changing conditions will be added to the system. This will enable eﬃcient and adaptive underactuated control strategies using new ideas and techniques in motion planning, behavior trees, reinforcement learning and system

32 CHAPTER 5. CONCLUSIONS AND OUTLOOK 33

identiﬁcation. Further use case scenarios and capabilities of the system will also be prioritized, such as precision maneuvering, docking and swarming. Upcoming publications will therefore focus on both new control strategies and application cases.

5.1 Real-time nonlinear MPC

In order to address the underactuated control problem, an efficient and simple nonlinear model predictive controller that can run in real-time on the AUV hardware is proposed. The prediction model used combines the 6 degree-of-freedom rigid body kinematics (including quaternion rotations to enable 0 − 360◦ maneuvering in all directions) and a vectorial nonlinear flight dynamics model (previously presented in equation (3.1) in Chapter 3). This flight dynamics model can be augmented with the multi-fidelity LUTs to improve accuracy at high angles of attack. The prediction model is given by

η˙ = Jq(η)ν , −1 (5.1) ν˙ = Meff (τC − Ceff(ν) − g(η)) , where η contains the positions and orientations and ν comprises the linear and an- gular velocities. Meff represents the combined mass and inertia matrix including added mass terms, Ceff represents the combined velocity dependent terms including Coriolis forces, centripetal forces and hydrodynamic damping, g(η) represents hydrostatics, and τC contains the control forces and moments from the propulsion and thrust vectoring subsystems. If necessary, trim actuators such as the LCG or VBS subsystems can be modeled by adding the mass and c.g. positions as additional states.

The nonlinear model predictive control problem is formulated for output reference tracking, where the optimal control must minimize the deviation of the output from a reference value. An output function o(η, ν) converts the orientations in the T state vector η ν from quaternion parameters to Euler angles so state devi- T ations can be tracked easily. The output state is given by ηout νout , while T the reference state to be tracked is given by ηref νref . The model predictive controller aims to minimize the deviation from the reference trajectory, while also minimizing the use of the control τC as well as reducing control transients τ˙C . The objective function J is given by 34 CHAPTER 5. CONCLUSIONS AND OUTLOOK

Z t+T T J = [(ηref(t) − ηout(t)) Q(ηref(t) − ηout(t))+ t T T τC (t)R1τC (t) + τ˙C (t)R2τ˙C (t)]dt (5.2) where t is the time, T is the prediction horizon, and Q, R1 and R2 are weighting matrices. In addition to the prediction model and the objective function, additional limits on actuators as well as constraints on states can be speciﬁed. Direct methods in nonlinear programming such as Sequential Quadratic Programming (SQP) can be used to obtain the optimal control [24]. Linearized (Jacobian) matrices of the dynamics model, the objective function and the output function can be used to speed up the solution time for the optimization solver. This setup can be evaluated in simulation (e.g. using the MPC toolbox in Matlab), and ongoing research work focuses on implementing and evaluating this controller in the AUV hardware.

Such a controller can be customized, and it is possible to easily extend it to include additional constraints, objectives, and allow model updates. The solver speed could limit use of optimal and predictive control for online real-time implementations, especially if nonlinear dynamics in 6DOF is considered (as it is here). This is because the large state space and the nonlinear dynamic constraints make the optimization problem difficult to solve. Linearizing the model about specific reference states simplifies the problem, and can enable rapid solutions (e.g. using convex optimization or linear-quadratic regulation). Warm start techniques could also be used for increased speed. Additional custom constraints can be added to make the controller more robust.

5.2 Intelligent control and motion planning

The MPC can be augmented with a feedback motion planner to make the system more intelligent. Robustness can be improved by using feedback motion planning to plan safe trajectories and add funnel-like state constraints to the MPC [34]. Behavior trees can be used to switch between diﬀerent control objectives based on the mission state, thereby acting as supervisory or event driven controllers [51]. For example, an energy-optimal controller can be used for transit and simple maneuvers; while time- optimal control can be prioritized in emergencies or criticial scenarios. Augmenting behavior trees with control barrier functions can enable guaranteed safety limits [52]. Techniques in machine learning can be used to identify dynamics models of the system from sensor feedback [53] to enable adaptive or dual control. The optimization solver can be improved as well, using techniques such as deep reinforcement learning to oﬀer real-time and rapid optimal control solutions to dynamic systems in the CHAPTER 5. CONCLUSIONS AND OUTLOOK 35

Figure 5.1: Future ideas in intelligent control for underactuated robotics. presence of uncertainty[32]. Learnt models of the system can also be used to improve the real-time simulation models, thereby enabling a more accurate simulation testbed. Data from experiment and simulation can be used to derive adaptive control policies using reinforcement learning [54, 55]. These ideas are summarized in Figure 5.1.

5.3 New scenarios

The intelligent controllers thus derived will be used in real-world use case scenarios. These include: 1. Operations within an algae farm environment (see Figure 5.2, top) considering maneuvering in the presence of obstacles, docking with stationary/moving targets and cooperation with other agents for inspection or manipulation. 2. Operations in a calving glacier front (see Figure 5.2, bottom). Missions in such a hostile, dynamic environment encompass tests of robustness, adaptability, as well as agile launch and recovery. In particular, multi-agent tasks, docking and near/under ice operations will be explored. This increases the focus on autonomy in a hostile and challenging environment. Such demonstrations will put the concepts introduced through a stress test, allowing the solutions to be optimized and industrialized.

To conclude, the methods outlined in this thesis form the foundation to create new hydrobatic capabilities for use in new applications and hostile conditions, thereby pushing the frontiers of what we know today! 36 CHAPTER 5. CONCLUSIONS AND OUTLOOK

Figure 5.2: Photographs of new scenarios: kelp farms (top, picture courtesy KosterAlg) and calving glacier fronts (bottom, picture courtesy Elias Strandell Er- storp/Sebastian Thune). Bibliography

[1] SMHI. Ocean absorption of carbon dioxide. Sept. 2019. url: https:// www.smhi.se/en/theme/ocean- absorption- of- carbon- dioxide-1.13092. [2] Sriharsha. Bhat and Ivan. Stenius. “Hydrobatics: A Review of Trends, Chal- lenges and Opportunities for Eﬃcient and Agile Underactuated AUVs”. In: 2018 IEEE/OES Autonomous Underwater Vehicle Workshop (AUV). 2018, pp. 1–8. doi: 10.1109/AUV.2018.8729805. [3] D. A. Duecker et al. “Micro Underwater Vehicle Hydrobatics: A Submerged Furuta Pendulum”. In: 2018 IEEE International Conference on Robotics and Automation (ICRA). May 2018, pp. 7498–7503. doi: 10 . 1109 / ICRA . 2018.8461091. [4] United Nations Department of Economic and Social Aﬀairs. The 17 Goals. url: https://sdgs.un.org/goals. [5] Peter King et al. “Deploying an AUV beneath the Sørsdal Ice Shelf: Recom- mendations from an expert-panel workshop”. In: 2018 IEEE/OES Autonomous Underwater Vehicle Workshop (AUV) (2018), pp. 1–6. [6] Peter W. Kimball et al. “The ARTEMIS under-ice AUV docking system”. English (US). In: Journal of Field Robotics (2017). issn: 1556-4959. doi: 10.1002/rob.21740. [7] Josep Bosch et al. “Creating 360 underwater virtual tours using an omnidi- rectional camera integrated in an AUV”. In: OCEANS 2015 - Genova (2015), pp. 1–7. [8] Jean-Baptiste Thomas. “Insights on the sustainability of a Swedish seaweed industry”. PhD thesis. KTH Royal Institute of Technology, 2018. [9] A. B. Phillips et al. “Agile design of low-cost autonomous underwater vehicles”. In: OCEANS 2017 - Aberdeen. June 2017, pp. 1–7. doi: 10.1109/ OCEANSE.2017.8084772.

37 38 BIBLIOGRAPHY

[10] Y. Sato et al. “Autonomous docking of hovering type AUV to seaﬂoor charg- ing station based on acoustic and visual sensing”. In: 2017 IEEE Underwater Technology (UT). Feb. 2017, pp. 1–6. doi: 10.1109/UT.2017.7890282. [11] N. A. Cruz et al. “A lightweight docking station for a hovering AUV”. In: 2017 IEEE Underwater Technology (UT). Feb. 2017, pp. 1–7. doi: 10.1109/UT. 2017.7890314. [12] R. Berg and H. Wicklander. “Swedish AIP Submarine Development”. In: in Undersea Defence Technology Conference (UDT 2016). Oslo, Norway, 2016. [13] Philip A. Wilson. “Autonomous Homing and Docking Tasks for an Underwa- ter Vehicle”. In: IFAC Proceedings Volumes 42.18 (2009). 8th IFAC Con- ference on Manoeuvring and Control of Marine Craft, pp. 304–309. issn: 1474-6670. doi: https://doi.org/10.3182/20090916-3-BR- 3001.0078. [14] Özer Özkahraman and Petter Ögren. “Underwater Caging and Capture for Autonomous Underwater Vehicles”. In: Global OCEANS. IEEE. 2020. [15] Sriharsha Bhat, Ivan Stenius, and Tianlei Miao. “Real-time ﬂight simulation of hydrobatic AUVs over the full 0-360 degree envelope”. Manuscript submitted to IEEE Journal of Oceanic Engineering. 2020. [16] Sriharsha Bhat et al. “Towards a Cyber-Physical System for Hydrobatic AUVs”. In: OCEANS 2019 - Marseille. 2019, pp. 1–7. doi: 10.1109/OCEANSE. 2019.8867392. [17] Sriharsha Bhat et al. “A Cyber-Physical System for Hydrobatic AUVs: System Integration and Field Demonstration”. In: IEEE OES AUV2020 Symposium. St. John’s, NL, Canada, Sept. 2020. [18] Intergovernmental Panel for Climate Change (IPCC). Special Report: Global Warming of 1.5 degrees Celsius. Tech. rep. Intergovernmental Panel for Cli- mate Change (IPCC), 2018. [19] Ivan Stenius, Peter Sigray, and Gunnar Linn. SMaRC Swedish Maritime Robotics Centre: Mid-Term Evaluation Report 2020. Tech. rep. Stockholm: KTH Royal Institute of Technology, 2020. [20] Russ Tedrake. Underactuated Robotics: Algorithms for Walking, Running, Swimming, Flying, and Manipulation (Course Notes for MIT 6.832). url: http://underactuated.mit.edu/ (visited on 10/11/2018). [21] Sonia Martınez, Jorge Cortés, and Francesco Bullo. Motion Planning and Control Problems for Underactuated Robots. 2002. url: arXiv : math / 0209213. [22] Richard Ernest Bellman. Dynamic Programming. USA: Dover Publications, Inc., 2003. isbn: 0486428095. BIBLIOGRAPHY 39

[23] L. S. Pontryagin. The Mathematical Theory of Optimal Processes. CRC Press, 1987. [24] John T. Betts. Practical Methods for Optimal Control and Estimation Using Nonlinear Programming. Second. Society for Industrial and Applied Mathe- matics, 2010. doi: 10.1137/1.9780898718577. [25] H. Chen and F. Allgöwer. “A Quasi-Inﬁnite Horizon Nonlinear Model Predic- tive Control Scheme with Guaranteed Stability”. In: Automatica 34.10 (1998), pp. 1205–1217. issn: 0005-1098. doi: https://doi.org/10.1016/ S0005-1098(98)00073-9. [26] D. Q. Mayne et al. “Tube-based robust nonlinear model predictive control”. In: International Journal of Robust and Nonlinear Control 21.11 (2011), pp. 1341– 1353. doi: 10.1002/rnc.1758. [27] A. Nikou, C. K. Verginis, and D. V.Dimarogonas. “A Tube-based MPC Scheme for Interaction Control of Underwater Vehicle Manipulator Systems”. In: 2018 IEEE/OES Autonomous Underwater Vehicle Workshop (AUV). 2018, pp. 1–6. [28] D. Limon et al. “Robust tube-based MPC for tracking of constrained linear systems with additive disturbances”. In: Journal of Process Control 20.3 (2010), pp. 248–260. issn: 0959-1524. doi: https://doi.org/10. 1016/j.jprocont.2009.11.007. [29] Tor Aksel N. Heirung, B. Erik Ydstie, and Bjarne Foss. “An MPC Approach to Dual Control”. In: IFAC Proceedings Volumes 46.32 (2013). 10th IFAC In- ternational Symposium on Dynamics and Control of Process Systems, pp. 69– 74. issn: 1474-6670. doi: https://doi.org/10.3182/20131218- 3-IN-2045.00151. [30] Leo V Steenson et al. “Model predictive control of a hybrid autonomous underwater vehicle with experimental veriﬁcation”. In: Proceedings of the In- stitution of Mechanical Engineers, Part M: Journal of Engineering for the Maritime Environment 228.2 (2014), pp. 166–179. [31] R. S. Sutton, A. G. Barto, and R. J. Williams. “Reinforcement learning is direct adaptive optimal control”. In: IEEE Control Systems Magazine 12.2 (1992), pp. 19–22. [32] F. L. Lewis, D. Vrabie, and K. G. Vamvoudakis. “Reinforcement Learning and Feedback Control: Using Natural Decision Methods to Design Optimal Adap- tive Controllers”. In: IEEE Control Systems Magazine 32.6 (2012), pp. 76– 105. [33] Bahare Kiumarsi et al. “Optimal and Autonomous Control Using Reinforce- ment Learning: A Survey”. In: IEEE Transactions on Neural Networks and Learning Systems PP (Dec. 2017), pp. 1–21. doi: 10.1109/TNNLS.2017. 2773458. 40 BIBLIOGRAPHY

[34] Anirudha Majumdar and Russ Tedrake. “Funnel libraries for real-time robust feedback motion planning”. In: The International Journal of Robotics Re- search 36.8 (2017), pp. 947–982. doi: 10.1177/0278364917712421. [35] Russ Tedrake et al. Learning to Fly like a Bird. 2009. url: https : / / groups.csail.mit.edu/robotics-center/public_papers/ Tedrake09.pdf. [36] Jason Evans and Meyer Nahon. “Dynamics modeling and performance evaluation of an autonomous underwater vehicle”. In: Ocean Engineering 31 (Oct. 2004), pp. 1835–1858. [37] K. P. Watson et al. “Prediction of submersible maneuvering performance at high incidence angles”. In: Proceedings of OCEANS ’93. Oct. 1993, II289– II294 vol.2. doi: 10.1109/OCEANS.1993.326108. [38] T. Fossen. Handbook of Marine Craft Hydrodynamics and Motion Control. John Wiley & Sons Ltd., Apr. 2011. [39] A. J. Healey and D. Lienard. “Multivariable sliding mode control for autonomous diving and steering of unmanned underwater vehicles”. In: IEEE Journal of Oceanic Engineering 18.3 (July 1993), pp. 327–339. issn: 0364- 9059. doi: 10.1109/JOE.1993.236372. [40] Van Dyke, ed. An Album of Fluid Motion. 4th. Department of Mechanical Engineering, Stanford University, Stanford, California: The Parabolic Press, 1982. [41] Russell M. Cummings et al. “Computational challenges in high angle of attack ﬂow prediction”. In: Progress in Aerospace Sciences 39.5 (2003), pp. 369– 384. issn: 0376-0421. doi: https://doi.org/10.1016/S0376- 0421(03)00041-1. [42] A. B. Phillips. “Simulations of a Self Propelled Autonomous Underwater Ve- hicle”. PhD thesis. University of Southampton, School of Engineering Sci- ences, 2010. [43] O. Hegrenaes, O. Hallingstad, and B. Jalving. “Comparison of Mathematical Models for the HUGIN 4500 AUV Based on Experimental Data”. In: 2007 Symposium on Underwater Technology and Workshop on Scientiﬁc Use of Submarine Cables and Related Technologies. 2007, pp. 558–567. [44] Leland H Jorgensen. Prediction of static aerodynamic characteristics for space- shuttle-like and other bodies at angles of attack from 0 deg to 180 deg. Tech. rep. 19730006261. USA: National Aeronautics and Space Administration (NASA), 1973. [45] R D Finck. USAF (United States Air Force) Stability and Control DATCOM (Data Compendium). Technical database. McDonnell Aircraft Co, St. Louis, MO, Apr. 1978. BIBLIOGRAPHY 41

[46] Tianlei Miao. “Generation of a full-envelope hydrodynamic database for hydrobatic AUVs: Combining numerical, semi-empirical methods to calculate AUV hydrodynamic coefficients”. MA thesis. Stockholm, Sweden: KTH Royal Institute of Technology, Aug. 2019. [47] Mark Drela. “XFOIL: An Analysis and Design System for Low Reynolds Number Airfoils”. In: Low Reynolds Number Aerodynamics. Ed. by Thomas J. Mueller. Berlin, Heidelberg: Springer Berlin Heidelberg, 1989, pp. 1–12. isbn: 978-3-642-84010-4. [48] Patryk Cieślak. “Stonefish: An Advanced Open-Source Simulation Tool De- signed for Marine Robotics, With a ROS Interface”. In: OCEANS 2019 - Mar- seille. June 2019. doi: 10.1109/OCEANSE.2019.8867434. [49] M. Hermann, T. Pentek, and B. Otto. “Design Principles for Industrie 4.0 Sce- narios”. In: 2016 49th Hawaii International Conference on System Sciences (HICSS). 2016, pp. 3928–3937. doi: 10.1109/HICSS.2016.488. [50] R. Drath and A. Horch. “Industrie 4.0: Hit or Hype? [Industry Forum]”. In: IEEE Industrial Electronics Magazine 8.2 (2014), pp. 56–58. doi: 10.1109/ MIE.2014.2312079. [51] M. Colledanchise and P. Ögren. “How Behavior Trees Modularize Hybrid Control Systems and Generalize Sequential Behavior Compositions, the Sub- sumption Architecture, and Decision Trees”. In: IEEE Transactions on Robotics 33.2 (2017), pp. 372–389. [52] Özer Özkahraman and Petter Ögren. Combining Control Barrier Functions and Behavior Trees for Multi-Agent Underwater Coverage Missions. 2020. arXiv: 2008.09427 [cs.RO]. [53] Lennart Ljung. “Perspectives on System Identification”. In: IFAC Proceedings Volumes 41.2 (2008). 17th IFAC World Congress, pp. 7172–7184. issn: 1474- 6670. doi: https://doi.org/10.3182/20080706-5-KR-1001. 01215. [54] Pieter Abbeel et al. “An Application of Reinforcement Learning to Aerobatic Helicopter Flight”. In: Advances in Neural Information Processing Systems 19. Ed. by B. Schölkopf, J. C. Platt, and T. Hoffman. MIT Press, 2007, pp. 1– 8. [55] Andrei A. Rusu et al. “Sim-to-Real Robot Learning from Pixels with Progres- sive Nets”. In: 1st Annual Conference on Robot Learning, CoRL 2017, Moun- tain View, California, USA, November 13-15, 2017, Proceedings. Vol. 78. Proceedings of Machine Learning Research. PMLR, 2017, pp. 262–270.