Master's Thesis

Autonomous Sailboat Prototype Sensors and Electronics Implementation with Machine Learning for Navigation

A Master's Thesis Submitted to the Faculty of the Escola Tècnica d'Enginyeria de Telecomunicació de Barcelona Universitat Politècnica de Catalunya by Esteban Francisco Chacón Mosquera

In partial fulfilment of the requirements for the degree of MASTER IN TELECOMMUNICATIONS ENGINEERING

Advisor: Rosa M. Fernández Cantí Co Advisor: José A. Lázaro Villa

Barcelona, January 2021

Title of the thesis: Autonomous Sailboat Prototype Sensors and Electronics Implementation with Machine Learning for Navigation

Author: Esteban Francisco Chacón Mosquera

Advisor: Rosa M. Fernández Cantí

Co Advisor: José A. Lázaro Villa

Abstract

This thesis is part of the SenSailor Project of the Facultat de Nàutica de Barcelona in which an unmanned sailboat is being built to perform scientific research in the ocean in a cost-effective manner. The thesis contributions to the SenSailor Project include the reconfiguration, communication and commissioning of sensors for environment research and navigation guidance, and the development and implementation of a machine learning model based on Deep Reinforcement Learning to actuate over the rudder and sail aileron. A virtual software environment has been developed to train and test the Artificial Intelligence model, which uses the reward of reducing the distance to a given way point as a tuning parameter. In addition, a LIDAR and a camera have been installed and configured to add computer vision for future collision avoidance navigation maneuvers. All the devices are powered by solar panels and batteries with low power techniques for self-sustainability. Finally, Wi-Fi and Bluetooth communication protocols have been developed to control the SenSailor navigation system by either the Artificial Intelligence or by human intervention.

Key words: Self-manned vehicles, autonomous sailboat, Q-learning, JSON, Python Turtle, LIDAR, Internet of Things.

Dedication

This thesis project is dedicated to God the Father and God the Mother Who help me all the way with Their invisible hands to blessed paths and allowed me to meet so many wonderful persons in the journey of learning.

Acknowledgements

Many thanks to my thesis tutors Rosa Fernández and José Lázaro for their guidance and support that encouraged me to exploit my potential through this work and their kindness that promotes the best working environment for a successful project development. Also, a special recognition to the UPC Automatic Control Department and to the UPC Barcelona Faculty of Nautical Studies for providing us with the tools to work with the edge of technology for projects that will certainly help the world to be a better place to live and explore. I am grateful to Carles Vallès, Joan Manich and Norbert Geraldo for their help during this project. Their effort to make the project go forward, certainly will be reflected in great success. This thesis couldn’t be done without the support of my father and mother that even from faraway, will give everything to help me achieve my dreams as if they were theirs. Certainly, any success I have, would not be possible without your love.

Revision history and approval record

Revision Date Purpose

0 21/11/2020 Document creation

1 22/01/2020 Document revision

2 28/01/2020 Document revision

Written by: Reviewed and approved by:

Date 21/01/2021 Date 28/01/2021

Name Esteban Chacón Name Rosa Fernández Cantí

Position Project Author Position Project Supervisor

Reviewed and approved by:

Date 28/01/2021

Name Jose A. Lázaro

Position Project Co-advisor

Table of contents

Abstract ...... 1 Dedication ...... 2 Acknowledgements ...... 3 Revision history and approval record ...... 4 Table of contents ...... 5 List of Figures ...... 7 List of Tables ...... 9 1. Introduction ...... 10 1.1. Requirements and specifications ...... 11 1.2. Statement of purpose ...... 11 1.3. Previous work ...... 14 2. State of the art of the technology used in this thesis ...... 16 2.1. Hardware ...... 16 2.1.1. Raspberry pi ...... 16 2.1.2. Arduino ...... 17 2.1.3. ESP32 ...... 18 2.2. Software development ...... 18 2.2.1. Python ...... 18 2.2.2. Arduino IDE ...... 19 2.2.3. JSON...... 19 2.2.4. Notepad++ ...... 19 2.3. Artificial Intelligence ...... 20 2.3.1. Machine learning models ...... 21 2.3.1.1. Supervised Learning ...... 21 2.3.1.2. Unsupervised Learning ...... 21 2.3.1.3. Reinforcement Learning ...... 22 2.3.2. Turtle ...... 23 2.3.3. Artificial Intelligence for ships navigation ...... 24 3. Methodology / project development ...... 25 3.1. Hardware Design ...... 25 3.1.1. Computation and roles ...... 25 3.1.1.1. LIDAR ...... 26

3.1.1.2. Raspberry Pi NoIR Camera ...... 27 3.2. Software Design ...... 28 3.2.1. Developer tools ...... 28 3.2.2. Code design ...... 29 3.2.3. Data management ...... 30 3.2.3.1. Communication between Arduino and Raspberry Pi ...... 31 3.2.3.2. Communication between Raspberry Pi and GPS ...... 31 3.2.3.3. Communication between Raspberry Pi and ESP32 ...... 32 3.2.3.4. Data in the Raspberry Pi ...... 32 3.3. Power Management ...... 33 3.3.1. Raspberry Pi 4 Low Power Techniques ...... 34 3.3.2. Arduino ONE Low Power Techniques...... 34 3.3.3. ESP32 low power techniques ...... 34 3.4. Network Design ...... 34 3.5. Artificial Intelligence ...... 35 3.5.1. Model Development ...... 36 3.5.2. Simulation on Turtle Python ...... 38 3.5.3. Training the model with the GPU ...... 40 3.5.4. Machine vision development ...... 41 3.5.4.1. LIDAR ...... 41 3.5.4.2. Camera ...... 43 4. Results ...... 44 4.1. Sensor implementation ...... 44 4.2. Raspberry Pi Data Acquisition ...... 45 4.3. Machine Learning scores ...... 46 4.4. Computer Vision Software ...... 47 4.5. Arduino memory ...... 48 4.6. Power consumption ...... 48 5. Budget ...... 50 5.1. Sponsor ...... 51 6. Conclusions and future development ...... 52 Bibliography ...... 54 Glossary ...... 57 Appendix ...... 59

List of Figures

Figure 1-1. SenSailor at Facultad de Náutica de Barcelona ...... 10 Figure 1-2. Microtransat competition map on December 08, 2020 [2] ...... 11 Figure 1-3. Triangular sailing race [2] ...... 12 Figure 1-4. Sail around race [2] ...... 13 Figure 1-5. Collaborative scan race [2] ...... 13 Figure 1-6. Hide and seek race [2] ...... 13 Figure 1-7. SenSailor and its builders [1] ...... 14 Figure 1-8. Electronics box outside and inside [1] ...... 15 Figure 2-1. Raspberry Pi 4 model B+, 8 GB RAM ...... 17 Figure 2-2. Arduino uno Rev 3 [27] ...... 18 Figure 2-3. ESP32 chip ...... 18 Figure 2-4. JSON data parsed with Notepad++ ...... 19 Figure 2-5. Two distinct eras of computational power for training AI [22] ...... 21 Figure 2-6. Reinforcement Learning model [17] ...... 22 Figure 2-7. a) Basic turtle movement commands b) Complex shapes on Turtle [23] ...... 23 Figure 3-1. Servomotor connection at the sail [1] ...... 26 Figure 3-2. SenSailor RPLIDAR A1 ...... 27 Figure 3-3. SenSailor Pi NoIR Camera V2 [24] ...... 27 Figure 3-4. SenSailor wired connection from the GPS to Raspberry Pi ...... 28 Figure 3-5. Code file for Arduino ONE in Notepad++ ...... 28 Figure 3-6. Sensors message position and definition ...... 31 Figure 3-7. File to configure the GPS coordinates to navigate ...... 32 Figure 3-8. Sensors data on Raspberry Pi ...... 32 Figure 3-9. Tacking and research data on Raspberry Pi ...... 33 Figure 3-10. Raspberry Pi Static IP configuration seen from the laptop screen ...... 35 Figure 3-11. Machine learning data preparation [25] ...... 36 Figure 3-12. SenSailor Convolutional Neural Network model ...... 37 Figure 3-13. SenSailor turtle screen environment ...... 39 Figure 3-14. SenSailor human control interface keyboard layout ...... 40 Figure 3-15. Anaconda installation panel and console ...... 41 Figure 3-16. RPLIDAR full scan proprietary software ...... 42 Figure 3-17. RPLIDAR data dump ...... 42

Figure 3-18. RPLIDAR USB port selection a) On RoboStudio b) On Script ...... 43 Figure 3-19. Pi NoIR Camera connection and flex cable ...... 43 Figure 4-1. Testing SenSailor sensors on the dry ...... 44 Figure 4-2. SenSailor communication flow ...... 45 Figure 4-3. Raspberry Pi 4 scripts and information files ...... 45 Figure 4-4. Raspberry Pi data gathering and storage...... 46 Figure 4-5. Artificial Intelligence score after training ...... 46 Figure 4-6. RPLIDAR script running on Raspberry Pi 4 ...... 47 Figure 4-7. Camera with IR filter vs Pi camera without IR filter at nightfall ...... 47 Figure 4-8. Arduino UNO program compilation ...... 48 Figure 4-9. Sensors power consumption [1] ...... 48 Figure 5-1. ADVANCED INDUSTRIAL SOLUTIONS EUROPE logo ...... 51

List of Tables

Table 1. Devices development received ...... 15 Table 2. Notepad++ Supported Programming Languages ...... 20 Table 3. Unmanned Surface Vehicles navigation tasks ...... 24 Table 4. SanDisk Extreme PRO features ...... 25 Table 5. Message protocol between Arduino and Raspberry Pi 4 ...... 30 Table 6. Message intervals in communication between Raspberry Pi and Arduino ...... 30 Table 7. Devices implementation and status ...... 44 Table 8. RPLIDAR 360° scan times ...... 47 Table 9. Current consumption on Raspberry Pi 4 ...... 49 Table 10. Current consumption on ESP32 ...... 49 Table 11. Project development costs ...... 50 Table 12. Project components costs ...... 50 Table 13. Free and Paid software list ...... 51

1. Introduction

The Artificial Intelligence (AI) is reaching every corner of everybody’s life and depending on how is being used, it can save resources, time and prevent people from performing dangerous tasks that on previous decades only humans could perform. One key task is to navigate the ocean to gather data in order to know the state of the sea, to explore it, to check the effects of climate change and ozone layer status. For this purpose, the Unmanned Surface Vehicles (USVs) are chosen and being developed around the world for the relative low cost of making it, short development times and people’s safety. The ocean poses many dangers for human beings, since the human body is not adapted for long term stay in the ocean due to physical and psychological conditions. Furthermore, to sail the sea for long terms requires a lot of space, infrastructure, food and resources that in some data gathering cases result on unfeasibility. The UPC is developing a USVs for research in the sea, looking forward to contribute with innovation in naval technology. The project began with the construction of a 2 meters length sailboat named SenSailor that was developed as a bachelor project of the degree on Engineering in Naval Systems and Technology [1]. The purpose of this work is to continue with the development of the USV SenSailor in order to provide it with the necessary electronics and software to be able to navigate autonomously on the sea. Due to time constraints, the scope is limited to navigate from the current position to a certain coordinate for a wide range of wind conditions, but not yet intended for obstacle collision avoidance or other ship encounters maneuvers.

Figure 1-1. SenSailor at Facultad de Náutica de Barcelona

1.1. Requirements and specifications The boat is completely built and have already passed flotation and impermeability tests. Therefore, to achieve the goals of this project, the next requirements for the electronic and software parts have to be accomplished: • The sensors should be read correctly, independently from the other sensors and without interference among them. • The boat position should be received in real time and be able to communicate wirelessly when required. • The boat should be able to ship in an autonomous manner to a desired destination pointed by GPS (Global System Position) coordinates and it should be able to change the destination when requested. • Wireless connection must be established between the servomotor at the sail, and the control unit on-board. This requirement is due to sail rotation constraints which may entangle wired connections. • Data must be stored in an efficient and secure way, to be transmitted at any time. • Low power consumption techniques must be applied due to the limited power.

1.2. Statement of purpose The importance of developing the SenSailor vessel is to have a cost-effective surface vehicle to gather data of the ocean as well as the environment and perform oceanography tasks, while gaining knowledge and experience on naval technology. In order to test the capabilities of the boat on the naval and electronic field, the FNB is considering to participate in challenging competitions where these can be tested in a wide variety of aspects regarding naval maneuvers. This is a way to enhance the SenSailor features and, at the same time, prove that it will be reliable while being used for investigation research in the ocean. The chosen competition, one of the most challenging worldwide, is the Microtransat Challenge [2] in which USVs compete in a transatlantic race from a coast in Europe to a coast in North America, delimited by geo starting and end lines which are shown in Figure 1-2. At the time of writing this assertion, just one competitor finished successfully after 10 years of this challenge that started on 2010, among participants from universities, private individuals, companies, navy and army.

Figure 1-2. Microtransat competition map on December 08, 2020 [2]

The rules on the competition [3] will partially guide the development of this project are: • Sailing permits only wind powered propulsion. • Two levels of autonomy for participation o Autonomous division that does not allow waypoints to be changed or messages that may change its course like weather forecast or ice observation, but allows AIS (Automatic Identification System) transponder data to be processes on-board. o Unmanned division that allows any data to be sent to the boat. • Radio equipment must comply with International regulations. • Recommended but not required to equip the boat with a white night light to be on during darkness and be visible from all directions at a distance of at least 2 nautical miles. • The boat must avoid collisions, complying with COLREG (Convention on the International Regulations for Preventing Collisions at Sea, International Maritime Organization, 1972), and avoid shipping lanes by the use of obstacle detection sensors. • The competitor is required to provide the boat position to the competition organizers through web or email interface at least every 6 hours. In case the boat fails to transmit its position for more than 10 consecutive days or more than 15 days in total, it will be disqualified. In addition, a record on board of the positions must be kept on the boat to be delivered by email to Microtransat with records no less frequent than 6 hours. At least one record per hour is recommended. • Before starting the race, the competitor must choose a circular target area of 25 km in radius on the finish line as the only valid area to consider the race as finished. The angle of entry to the area is not relevant and a GPS log on the area will validate its position on the area. Since this project was developed during the 2020 global pandemic all the competitions were cancelled without notice of further competitions to be resumed. As a result, some of the specifications for this project are based on the latest global competition, the WRSC (World Robotics Sailing Championship) [3] held on Ningbo - China in summer 2019. The challenges for the WRSC competition are: 1. Fleet race. – Competition based on triangular sailing race course as shown on Figure 1-3, where scores are obtained when a tracking point is recorded in a radius on 5 meters around virtual markers in correct order.

Figure 1-3. Triangular sailing race [2]

2. Station keeping and avoidance. – Also known as virtual moorings, they consist in keeping the boat on a circle area of 15 meters around a virtual buoy during 5 consecutive minutes, at the end of the 5 minutes, the boat should be in a cruise radius smaller than 15 meters to be counted as valid. Then, the boat will navigate to a physical buoy and sail around it in the same way as virtual buoy, but during 3 minutes. Collisions with the physical buoy reduce the score.

Figure 1-4. Sail around race [2] 3. Collaborative area scanning. – Teams will split up a big area that is required to be scanned and each team should scan its area without interfering with the others for 15 minutes. Score will be added according to the delegated grids explored.

Figure 1-5. Collaborative scan race [2] 4. Hide and seek. – This a high-level task where the boat will race against another team. Each team is assigned a pair of circles where hide means that a boat will sail back and forward between the circles. Seek means to be able to detect a tag placed on the sail of the other ship while hiding; there are two tags, one placed at each side of the sail and the time for this challenge is 15 minutes.

Figure 1-6. Hide and seek race [2]

Rules on the competition: • Multiple attempts will be allowed, with sufficient time and good weather conditions for the different challenges. • The sailing boat must be controllable by a human in all events, through Wi-Fi (Wireless Fidelity), RC (Radio Control), among others, with the limitation of avoiding Wi-Fi boosters that exceeds the regulations on wireless signal emissions. • Scoring will be based on real time positioning tracking. • Data recorded should be done in SI (International System) units. Some challenges offer bonus points for recording specific data. • The organizers will provide a tracking device, however in case of failure, the competitor tracking device information will be accounted. The SenSailor is not a mere scale model of a ship. It is a fully functional sailboat that due to its size and technology is capable of navigate and perform research in the ocean. It is completely self-sufficient since it is power sustained by wind and solar energy and the objective is that with artificial intelligence it should be capable of sailing through the sea without human intervention. The project thus includes technology and techniques from the fields of software development, communications and information gathering. Aside from the modules and sensors for navigation, it was required the implementation of a LDR (Light Dependent Resistor), ultraviolet light and temperature sensor to start collecting that data during tests for maritime research.

1.3. Previous work The SenSailor sailboat was developed by the Barcelona Nautical Faculty as a TFG (Treball Final de Grau) done through 2019 and was finished by the beginning of 2020 by Joan Sastre and Carles Manich as the first sailboat construction step to be completed by future developments to achieve fully autonomy. The boat passed floatability and impermeability tests that were planned for autonomy at the sea, a part of it is that even if the boat is overturn, it has the capability to return to the normal navigation conditions. In addition, it was adapted to have electronic equipment that will be powered by solar energy and propulsion by wind force.

Figure 1-7. SenSailor and its builders [1]

The project was developed at the Espai Vela FNB facility and during the final stages of the SenSailor construction, after floatability and impermeability tests, the Espai Vela was hit by the storm Gloria, damaging the navigation related parts like the rudder and the sail. Once the storm sequels were overcome and it was possible to return safely to the Espai Vela, a bigger issue happened, the world pandemic by Covid19 completely shut down any possibility of repairing the boat or doing tests. However, the development did not end up due to the difficult situations and the project changed to the circuit and software development, but final tests on water couldn’t be performed. The following is the description of the state of the hardware and software developed. Therefore, the starting point for this thesis. The SenSailor and the wiring were placed carefully and always looking forward to support a situation where water may get into the interior of the SenSailor. All the electronics were placed in a hermetic box as shown on Figure 1-8 where all the sensors and servo motors were connected and managed by an Arduino UNO board. Although all the sensors where properly connected, the code developed by [1] for the Arduino was in a first stage of programming.

Figure 1-8. Electronics box outside and inside [1] When testing the different elements and the code of [1], the results obtained can be seen on Table 1 where even if some sensors were implemented and working, they presented delays and comunication conflicts between the Arduino and the sensors when alll devices were connected and running. Some devices like the GPS constanly sends data, exausting the resources when not handled correctly. Device Hardware installation Drivers and software Status Arduino UNO Implemented Implemented Working Arduino NANO Implemented Not implemented Not working Servo motor helm Implemented Implemented Working Servo motor Sail Implemented Implemented Working Wind vane Implemented but damaged Implemented Not working Anemometer Implemented Implemented Working Magnetometer Implemented Implemented Working GPS Implemented Implemented Working LDR Implemented Not implemented Not working Ultraviolet Sensor Implemented but damaged Not implemented Not working DS18B20 Temp sensor Partially Implemented Not implemented Not working Table 1. Devices development received

2. State of the art of the technology used in this thesis

The present chapter describes the software development used for endowing the autonomous sailboat to achieve the objectives previously mentioned. The software capabilities are directly limited by the hardware where they are executed. Therefore, the first topic is devoted to the three hardware devices used: • Raspberry Pi 4: selected for the code execution robustness by its operating system that allow multitasking, processing power, interfaces (HDMI, USB, Wi-Fi, Bluetooth and digital GPIOs) and machine learning support • Arduino Uno: selected for its hardware reliability, low power consumption and digital and analog inputs where are useful for the sensors required for this project that are mostly analog and not directly supported by the Raspberry Pi 4. • ESP32: selected for its low power consumption and Bluetooth connectivity required to control a servomotor in the sail with a wireless connection. All three devices are the latest hardware versions available at the moment. The second part is devoted to the software tools used for code development. Coding becomes complicated over time and maintaining a code free of bugs, readable and optimized requires a lot of time and patience; however, the tools used for this project allowed short coding times between different languages and hardware. Finally, the last part is dedicated to the Artificial Intelligence developed to achieve autonomous navigation on the SenSailor.

2.1. Hardware

2.1.1. Raspberry pi The Raspberry Pi is a computer with the size of a credit card that can run a Linux operating system, but the recommended Linux version for the Raspberry Pi is the one called “Raspberry Pi OS” that was previously known as “Raspbian”. This Linux version comes with preinstalled software to manage the GPIOs and specific drivers for its hardware. The Raspberry Pi was created with the purpose of giving young people access to low- cost computers, this is around 30$ at the time of its launch. Now, the model Raspberry Pi Cero v1.3 can be bought for 5.50 € in Europe. The latest available model in terms of specification is the Raspberry Pi 4 model B+ with a CPU Quad-core ARM Cortex-A72 64- bit at 1.5 Ghz with 8 GB of RAM, which is the one used in this project [4]. People are using the Raspberry Pi in diverse projects due to its small size, low power consumption and powerful computational and hardware I/O features. It is so powerful that it was used on a NASA information breach [5] and also a Tesla wireless key was bypassed [6]. The other option was the NVIDIA Jetson Nano, but it wasn’t chosen because it has lower computation power, less RAM and smaller developer community.

Figure 2-1. Raspberry Pi 4 model B+, 8 GB RAM

2.1.2. Arduino Arduino is an electronic platform, that started as a project in 2005 for students to learn electronics and programming development, but due to its size and versatility, it is used on many projects in education and enterprises. The Arduino consist on a microcontroller ATMega of 8-bit with multiple analogs and digital I/Os in a Reduced Instruction Set Computer (RISC) architecture [7]. For this project, it has been chosen the Arduino Uno model version 3 that was released for first time in 2010. The development of projects with this module has risen over the years and the drivers from third-party companies that connect their hardware with Arduino can be accessed through Arduino IDE, being the mayor hardware and software developer, the company Adafruit, not just for Arduino, but also for Raspberry Pi [8]. Arduino has one advantage over the Raspberry Pi, which is that Raspberry Pi does not support analog inputs on its GPIO. Since many sensors to be implemented on the boat return analog data, Arduino will work as a DAQ (Data Acquisition Hardware). This project requires: • 3 analog inputs for wind vane, ultraviolet and LDR sensors. • 1 digital input for the anemometer sensor. • 1 I2C connection for the magnetometer sensor. • 1 one-wire connection for the temperature sensor. • 2 serial connections, one to communicate with the Raspberry Pi 4 and one virtual serial connection to the GPS. • 1 digital output for the masthead light.

Figure 2-2. Arduino uno Rev 3 [27]

2.1.3. ESP32 ESP32 is a low-cost and low power consumption system on a chip that includes integrated Wi-Fi and Bluetooth modules, has low power consumption and it is compatible with Arduino IDE. These devices are widely used on IoT (Internet of Things) projects due to its specifications, small size and user-friendliness. There is also a huge community on the Internet that supports it [9]. The ESP32 connects to Wi-Fi networks, and it can work also as hotspot creating its own Wi-Fi network, featuring also an Integrated TCP/IP protocol stack that allows it to work as a web server [10]. Regarding the Bluetooth, it supports Bluetooth BLE that stands for “Bluetooth Low Energy” which is widely used for IoT applications [11] and fits this project.

Figure 2-3. ESP32 chip

2.2. Software development

2.2.1. Python Python is a code interpreter developed in 1989 for object-oriented programming language. It is portable and runs on Linux, macOS and Windows, being a versatile tool for scripting and software development and giving an intuitive paradigm of programming. Python has

a huge community that supports it by continuously creating libraries for automating processes, device driver’s setup and Artificial Intelligence development [12]. It is a powerful tool for software developers as it is very intuitive and easy to read and write language following a uniform syntax that improves coding productivity. For this project, it is used to create scrips to drive the sailboat and manage the information and communications. Python was chosen over C language, because Python is a high-level programing language that reduces the complexity in programming. It requires more memory and execution time, however, with the current hardware, memory and speed are not an issue.

2.2.2. Arduino IDE The Arduino IDE is an open-source software that has grown over the years not only to program Arduino devices but also as a common platform to program other microcontrollers know as third-party hardware. It is a free easy to use tool for building the Internet of Things and in the present project it is used to program the Arduino UNO and ESP32, speeding up the application design and code testing [13].

2.2.3. JSON JSON (JavaScript Object Notation) is a text-based format developed by the IETF (Internet Engineering Task Force) that is being implemented on multiple devices due to its lightweight interchange format. There are libraries and plugins for most languages and platforms. The best feature of this format is that it is read- and write-friendly for humans while for machines is also easy to parse and generate. On the Figure 2-4 the structure of the data can be clearly seen with one view [14].

Figure 2-4. JSON data parsed with Notepad++ The other advantage of JSON is the access to the information is also very intuitive, since the data can be accessed calling the name of the element required at each level, creating a path based in names where the data can be any type and in any order. For example, from the Figure 2-4, the URL can be acceded by Image[ Thumbnail[ Url ] ] which is a String data type. Integer, float and array types can be acceded just by changing the path names. This advantage will be seen when the data is transmitted to a server through satellite communication, handling the data in a fast way and by avoiding sending data sizes and types or control signals that will be expensive in satellite communications.

2.2.4. Notepad++ It is a free source code editor that is open source under GNU (General Public License). Supporting every popular programming language as seen on Table 2, with high execution

speed while having small program size and low computational resources required, it is the perfect tool for this project. On Table 2 the languages used for the software development are in bold, these five languages were used in twelve files created modified and debugged in parallel to develop the software in the laptop to be later uploaded in the different devices. A further explanation of how it was used is in section 3.2.1. Python and Arduino IDE have powerful compilers for writing, debugging and testing the code, however they are too specialized on their code support and when working in a project with different technologies and programming languages, it becomes necessary to use code editors The Notepad++ is chosen because is lightweight, free source code which main features used while coding were: code editing, searching and replacement, Auto-competition and function list [15].

ActionScript Ada ASN.1 ASP Assembly AutoIt AviSynth scripts BaanC batch files Blitz Basic C C# C++ Caml CMake Cobol CoffeeScript Csound CSS D Diff Erlang escript Forth Fortran FreeBASIC Gui4Cli Haskell HTML INI files Intel HEX Inno Setup scripts Java JavaScript JSON JSP KiXtart LaTeX LISP Lua Makefile Matlab MMIX Nimrod nnCron NSIS scripts Objective-C OScript Pascal Perl PHP PostScript PowerShell PureBasic Python R Rebol Registry script Resource file Ruby Rust Scheme Shell script Smalltalk SPICE SQL Swift S-Record Tcl Tektronix HEX TeX txt2tags Visual Basic Visual Prolog VHDL Verilog XML YAML Table 2. Notepad++ Supported Programming Languages

2.3. Artificial Intelligence The Artificial Intelligence (AI) technology requires 3 major components: • Algorithms. – The necessary steps to follow for the machines to learn, the modern models applied were developed long ago by persons like Alan Turing, Arthur Samuel and Frank Rosenblatt [28]. • Big computation. – Even with the strongest algorithms, without sufficient computational power, no real test can be done. As seen on Figure 2-5, the explosion on AI development was possible just a decade ago due to the nanotechnology, GPU evolution and technology price affordability. • Big data. – Machine learning algorithms are well known for being data hungry. For years, AI developments could not be made for the lack of digital data to be used, storage size and data accessing read speed.

Figure 2-5. Two distinct eras of computational power for training AI [22]

2.3.1. Machine learning models There are many types of models for machine learning, but they can be grouped according to [16] in three main categories:

2.3.1.1. Supervised Learning The Supervised Learning (SL) algorithms are good at finding relationships between the data. SL works by learning a function that will map an input to an output, based on examples of known input-output pairs; for this reason, the data must be labelled. The main algorithms are: • Regression • Random Forest • Nearest Neighbor

2.3.1.2. Unsupervised Learning The Unsupervised Learning (UL) algorithms are good at categorizing data, finding outliers in the data and perform data segmentation. UL works by finding patterns from data without having labelled data. The main algorithms are: • K-means • Gaussian Mixture Model • Principal Component Analysis

2.3.1.3. Reinforcement Learning The Reinforcement Learning (RL) algorithm learns from its environment by interacting with it and it is mostly used in robotics due to its advantage at decision-making. It works by doing an action based on input values, previous state, current state and receiving a reward or punishment depending on how the action retrieves a response with the environment. The main algorithms are: • Q-Learning: On-policy approach which is in deep explained in [17]. • State Action Reward State Action (SARSA): Off-policy approach which is in deep explained in [17]. Reinforcement Learning can solve the challenges that our boat will face effectively, so, let us see in deep how Reinforcement Learning works.

Figure 2-6. Reinforcement Learning model [17] The reward R is a scalar signal that feedbacks to the agent how an action is performing through the environment at a time step, Rt. This is called observation, so the agent will work on maximizing the sum of rewards [17]. It is important to notice that long-term rewards can be programmed to be pursued over short-term rewards; this is used on scenarios where an action can have serious long-term consequences. The agent and the environment interact with each other in a discrete sequence of time steps. On each timestep, the agent receives the environment state St and selects an action At. Then in the next timestep the agent observes the consequence of its action

St+1 and the resulting reward which leads to a new state, Rt+1. All the sequences that an agent experience during a life time is called “history”. Hence, the agent could select actions according to the history, looking the sequence of actions that leads to the highest reward. However, the history could grow indefinitely, making it infeasible for most of real-world problems. Instead of the history, it is used the state information as a function of history; this is done by Markov states where a state gathers all the useful information so far acknowledged and throw away the entire history, since the current states have sufficient information to characterize the future. The agent is made out of the following components. 2.3.1.3.1. A policy It is a mapping from states to possibilities to select an action, a, for a given state, s. The policy denoted as at state s can be defined as deterministic or stochastic:

• Deterministic policy: (1)

• Stochastic policy: (2)

In a deterministic policy, if the state St=s is observed, then the action At=a is taken, whereas in a stochastic policy it is defined a probability level to take the action a given the state s. 2.3.1.3.2. A Value function The value function is a function that shows how much reward is expected when taking an action at a particular state. It can be seen as a reward prediction to assess the goodness or badness of states and it is defined by:

(3) where Rt+1, Rt+2,… are the reward values for different time samples at the state s and ɣ (varying between 0 and 1) is a penalization to reduce rewards as the time step goes by to force the algorithm choose policies that requires fewer steps to achieve the goal. 2.3.1.3.3. A Model The model is an agent representation of the environment that makes decisions regarding what should be done next and its denoted by a transition function P defined by:

] (4)

And a reward function for reward prediction when an action is associated to a given state, defined by:

(5)

2.3.2. Turtle Turtle is a Python library for graphical interfaces and allows creating virtual environments for Artificial Intelligence, its graphical primitives are intended for object-oriented and procedure-oriented programming. It is based on Tkinter package for underlying graphics to create Graphics User Interfaces (GUI) on Unix and Windows platforms. On Figure 2-7, it can be seen how simple moments directed by mathematic functions can create very complex forms. Turtle is perfect for this project because it starts with a screen size defined by say AxB. Then the screen is defined by rectangular coordinates starting at the center of the screen with coordinates 0,0. From there an object can be moved as desired through functions. Therefore, it is optimal and fast to code to create the environment for a boat that sails with latitude and longitude as X and Y coordinates respectively. Also, the LIDAR sensor implemented for the project returns as data, angle and radius to the surroundings; this data is converted from polar to rectangular coordinates, mapping the surrounding area.

Figure 2-7. a) Basic turtle movement commands b) Complex shapes on Turtle [23]

2.3.3. Artificial Intelligence for ships navigation As the world is more globalized, the commerce grows and improved security becomes a priority, artificial intelligence is developed in the nautical field by companies, governments and universities to make navigation through the ocean safe and efficient. [20] Artificial intelligence is preferred for collision avoidance situations because algorithms, unlike people, do not get nervous or tired in these situations. Furthermore, encounters between vehicles run by artificial intelligence shows an overwhelming performance compared with human collision avoidance maneuvers. However, for a successful avoidance, communication is important and NANET (Nautical Ad-hoc Network) networks are found to have great potential [18]. These networks use vacant radio frequencies around 700 MHz for a range of 20 nautical miles using Wi-Fi protocol for communication. A recent study of [19] shows the tasks that the artificial intelligence has to solve to reach the goal of full autonomy. The tasks are show on Table 3.

Navigation Localization Path planning Collision avoidance Reference Tracking Path following Station keeping Exploration Object recognition Detection and cuing Object tracking Simultaneous localization and mapping Oceanography Effect delivery Kinetic strike Nonkinetic strike Manipulation Countermeasures Evasion Fault tolerance Cyber resilience Deception Resource management Power management Table 3. Unmanned Surface Vehicles navigation tasks Since the tasks are different, machine learning algorithms perform well in certain tasks and bad on others. The scope of the current project is more focused on navigation, then for this kind of tasks, Neural Networks (NN) or variants of it are used according to [20]. The disadvantage of Neural Networks is that they require a lot of data, training and power, but yet they are optimal for learning any kind of functions. On the other hand, Reinforcement Learning is used in many projects nowadays after showing its efficacy by, for instance, playing at Go. For this reason, in this project a mix between Reinforcement Learning and Deep Neural Networks (DNN) has been used by generating the machine learning model with RL and the DNN is used to find the parameters for the RL. This mixed model is also known as Deep Reinforcement Learning according to [17].

3. Methodology / project development

The following project was done during the coronavirus pandemic, for that reason the access to the boat was limited and so was the testing in real water operation. For this reason, the testing of the onboard sensors and actuators as well as their communication was performed in dry conditions and, for the navigation part, a virtual environment for testing the machine learning algorithm was developed. Taking this into account, the project was developed as follows in this chapter. The electric diagram and the wire connections in the boat can be seen on the Appendix.

3.1. Hardware Design

3.1.1. Computation and roles To achieve a complete autonomous driving on the SenSailor, it was chosen the Raspberry Pi 4 model with 8 GB of RAM to run a DNN (Deep Neural Network) in a Reinforecemet Learning model. Moreover, to secure that the system withstand extreme environments at the sea, it was chosen as memory for the Raspberry Pi the SanDisk Extreme PRO of 32 GB which is the microSD card with the highest memory size recommended to run the Raspberry Pi OS. It is posible to use 64 GB or more size memory, but requires to be formated as exFAT filesystem and the CPU only has support for FAT16 and FAT32, so the system may lose fiability and for the current development the 32 GB is enough. The SanDisk Extreme PRO features are on Table 4. It is important to point out that the main characteristics considered are the Writing and Read speed to run fast the OS and the shock prove feature to overcome the wave abatement.

Specification Value Memory Storage 32 Speed Class 10 Read Speed 100MB/s Write Speed 90MB/s Operating temperature -25°C to 85°C Storage temperature -40°C to 85°C Water proof Yes Shock proof Yes Temperature proof Yes X-Ray proof Yes Table 4. SanDisk Extreme PRO features The downside of the Raspberry Pi is that it is designed to work with digital signals and many sensors used for this project are analog. For this reason, the Arduino UNO that was previously implemented will manage all the sensors to gather, process and send the sensors data to the Raspberry Pi. As for the servo motors of the helm that was managed by the Arduino, it will be handled directly by the Raspberry Pi, since the AI running in the Raspberry Pi is now in charge of the navigation. The current sail has navigation limits due to the wiring required to power and control the servomotor at the sail. The sail con rotate only 180 degrees, lossing maneuverability

agaist the wind, this phisical limitation can be better understood looking at the Appendix and Figure 3-1. Therefore, a complete wireless solution was required for the servomotor. The solution chosen was the ESP32, a microcoltroler with integrated Wi-Fi and Bluetooth connectivity that operates in the temperature range of –40°C to +125°C with low power consuption. Thankfuly, it is posible to power the ESP32 and the servomotor with a litium battery and a solar panel.

Figure 3-1. Servomotor connection at the sail [1] Without the wiring issue at the sail, a new sail is to be crafted to be 360 degrees rotation free as a future project, so the final position of the servomotor, the ESP32, the lithium battery and a small solar panel of 22 x 18 centimeters in the new sail are to be defined in order to best fit and protected these elements agaist the weather. The SenSailor depends on the sensors to be autonomous and having the processing power of the Raspberry Pi 4, increase the type of sensors that can be connected. For this project it was aquired 2 new sensors widely used and supported on autonomous vehicles.

3.1.1.1. LIDAR The LIDAR was chosen for this project due to its accuracy and speed to scan the surroundings in real time by emmiting a laser beam and mesuring the returning time of the beam with a recepto. The model used is the RPLIDAR A1 from SLAMTEC with a distance range of 0.15 to 12 meters, an angular range of 360 degrees with a resolution of 1 degree at a scan rate of 5 Hz. In other words, a full 360 scan every 0.2 seconds. As shown on Figure 3-2, the LIDAR is placed in the front deck, however, a hermetic transparent cover is still pending to be designed to protect the LIDAR from the saltwater without interfering with the laser. The LIDAR will be used for short range sensing.

Figure 3-2. SenSailor RPLIDAR A1

3.1.1.2. Raspberry Pi NoIR Camera The camera used for this project is the Pi NoIR Camera V2, a camera designed for Raspberry Pi boards with a Sony IMX219 8-megapixel sensor having a key feature for the project, the IR (Infrared) filter is removed allowing to take better images at night while the loss of image quality at daylight is irrelevant to the computer vision models. The Figure 3-3 shows a comparison between Pi Camera models. The camera is intended to work together with the LIDAR, improving the sensing at short and long range by image recognition algorithms based on border detection.

Figure 3-3. SenSailor Pi NoIR Camera V2 [24]

The sensors and the wiring position configuration in the SenSailor were kept as received, but the terminals are spliced with jumper cables to connect directly the sensors with the Arduino and the Raspberry Pi as seen on Figure 3-4, where the end-to-end wiring of the GPS is shown as an example. Some sensor signals where changed from the original pin at the Arduino and it is widely described in the software design section.

Figure 3-4. SenSailor wired connection from the GPS to Raspberry Pi

3.2. Software Design

3.2.1. Developer tools Technology development speed increases according to Moore’s Law. A key part is the tools used for fast coding that were certainly needed for this project due to the limitations by the coronavirus pandemic and the complexity of working with the several languajes and devices required for this project. Notepad++ was mainly used in this project for its versatily to work with different programming laguanges. Even that the Arduino has its own IDE, the Notepad++ has more features that enhance the coding process as listed below. • Find and replace.- Not only variable names, but complete string sequences. This is useful to avoid changing parts of the code that may not intend to be changed. • The view of the code file can be duplicated, allowing fast modifications side to side, error correction for misspelling and modular code development. • Sintax highlighting, for quick string view and search. Just by selecting a string sequence, the program will highlight the same sequence everywhere in the code.

Figure 3-5. Code file for Arduino ONE in Notepad++

3.2.2. Code design The code for reading the sensors was incomplete, complex and hard to maintain. Since the project needs to be optimized and further development will be needed from other researchers after this thesis, the best approach is always the simple approach, therefore, the names and comments in code were changed to English and the code was improved following the recommendations of the book [21], whose main code techniques are: • Improved readability by spacing in a symmetric way. • Naming variables and functions using camel case with clear and significant names, not just using letters or names that doesn’t represent its function clearly. • Variables and functions will have the same prefix or suffix as their module’s name. • With good names, commenting becomes unnecessary in most of the cases and when needed just short comments are more than enough. The techniques applied can be seen on Figure 3-5. Moreover, the code was modified to be modular, making it easier to maintain. The code order is the following: 1. Include libraires. 2. Code preprocessor enables a. Enable/disable debug logs. 3. Constants definitions a. Device pinout definitions. b. Messaging definitions. c. Timers definitions. d. Sensors definitions and configurations. 4. Global variable definitions a. Variable initialization. b. Data initialization. c. Sensor limits. 5. Program initialization a. Initialization functions. b. Initialization configurations. 6. Main program a. Update timers. b. Read and process all sensors data. c. Update data on messaging structures. d. Perform actions. 7. Function definitions

The program is designed in a way that all the definitions are shown in the first part, so the functioning of the program is managed at the heading of the file, to avoid going back and forth constantly in the code as it initially was in the previous code. The sensors are read independently and the information is stored in a data array, without interfering with each other by using individual timer for each sensor. Any delay was used to avoid dead processing times, allowing the controllers to be always ready to send and receive data from the sensors and other devices. Therefore, the sensor data sent to the Raspberry Pi is taken from the data array, not directly from the sensors allowing the mentioned independence.

3.2.3. Data management The major issue of having multiple sensors with different technologies, relay on how to handle asyncronicity. Some sensors can be read periodically, but others can return data at different times according to events. The solution was to read the data from the sensors by the Arduino and keep it into global access variables to later save the data into arrays as strings at the end of every program cycle in the following way: all the data is transformed into strings and added to a message structure defined on Table 5 in the positional order described on Figure 3-6. With this aproach the messages are always ready to be sent with the latest known data meanwhile reading the sensors without affecting the comunications Size 2 Bytes 1 Byte Variable Role Data Size Sensor Data Type Number Letter Variable Send as String Table 5. Message protocol between Arduino and Raspberry Pi 4 The program sends the data of a different sensor, one at a time from the Arduino to the Raspberry Pi over serial link every 50ms as shown in Table 6 where after sending all the sensors information from the data array, the communication stops during an Idle time allowing the Arduino to save power and the Raspberr Pi to process the data received, before having new data and also avoiding to send the same data continuously. The message block plus the Idle time sum 1 second, this is because the GPS position is usually updated each second, therofore, its importance for the navigation system became the reference for comunication interval. The program currently is supporting eleven sensors and adding new sensors only requires that definitions and functions are added next to the last sensor code, having the posibility to add 9 more sensors before reaching the 1 second time for a new block.

Message block Idle time Message block Idle time A B C D E F G H I J K A B C D E F G H I J K

50 ms 550 ms 450 ms 550 ms 450 ms

1 second 1 second Table 6. Message intervals in communication between Raspberry Pi and Arduino

3.2.3.1. Communication between Arduino and Raspberry Pi The messaging works as follows: 1. The sensors data is read by the Arduino, processed and saved in variables.. 2. The Arduino constructs the messages according to message protocol on Table 5, and the definitions on Figure 3-6 where the position translates into a letter.

Figure 3-6. Sensors message position and definition 3. The Arduino sends the constructed message to the Raspberry Pi over serial communication one at a time interval. When all the data is sent, it is considered as a data block according to Table 6. 4. The Raspberry listens the serial link and receives a string message from the Arduino every time interval. 5. The Raspberry reads the message looking for a string letter and transforms the preceding numbers to a int value in order to know the size of the message. 6. The Raspberry saves the data after the first letter into a file using JSON format as seen on Figure 3-7, based on the first letter received and according to the sensor, the string data is transformed to int, float or kept as string. 7. The Raspberry knows when a complete data block is received based on the last letter “K” defined on Figure 3-6 and if a message is required to be sent from the Raspberry to the Arduino, it will do it after each data block received. The message protocol does not consider a CRC (Cyclic Redundancy Check) implementation, since the processing and energy spent on CRC is not justified. The code has data definitions and the comunication between all devices are collition free.

3.2.3.2. Communication between Raspberry Pi and GPS The Raspberry Pi is the device in charge of driving the boat by moving the helm and the sail throught the servo motors, based on the information of the environment and the coordinates to go. The GPS module has the advantage of being able to calculate the angle course and the distance to a given latitude and longitude coordinates from its current location. Since the Arduino is the device connected to the GPS, the Raspberry Pi communicates with the Arduino to update the GPS coordinates to the destination point. This action is performed by setting the field “isUpdated” with 0 at the file in the Raspberry Pi named coordData.txt

as seen on Figure 3-7 and placing the coordinates required, preceding a “w” which is a message identifier for the Arduino to know that is a GPS configuration. Since this message has a fixed length, the message data size field is not required.

Figure 3-7. File to configure the GPS coordinates to navigate When a message to change the GPS coordinates destination point is received by the Arduino, it parces the message based on the “w”, reading the first coordinate as latitude until the “:” character and next, the second coordinate as longitude. Then, it sends it to the GPS and finally, sends to the raspberry the exact same message received, so the Raspberry Pi knows that the information was updated correctly and change the field “isUpdated” with 1 to finalize the GPS destination point update process.

3.2.3.3. Communication between Raspberry Pi and ESP32 The Raspberry Pi communicates with the ESP32 through bluetooth communication. The library “Bluetooth Serial” was installed on both devices that creates a virtual serial communication over bluetooth links, allowing also to control the servomotor with a smartphone that supports Android, iOS and Windows Phone 8 plataforms. The ESP32 generates a Bluetooth link with the device name SenSailor. The raspberry is already configured to connect directly to SenSailor when bluetooth is enabled. Then, the Raspberry Pi sends the position that the servomotor should have. Due to phisical limitations at the sail, the servomotor only works in the rage of 45 to 135 degrees, this limitation is programmed on the ESP32, any number or character received othside the working range is discarded.

3.2.3.4. Data in the Raspberry Pi All data is saved on the Raspberry as txt files with a JSON format, because JSON is optimal for data storage and transfer, since it is simultaneously machine and human readable friendly, sending and storing data as a sequence of bytes, separating and structuring the data according to commas, colons, braces and brackets. The program is designed to have only one function to write or modify data on a specific file and other fuctions or programs to handle reading and computation of the data to avoid access problems and maintain a modularity aproach.

Figure 3-8. Sensors data on Raspberry Pi

Two files keeps the information of the sensors: • sensorData.txt: stores all the information received from the Arduino as seen on Figure 3-8, following the posicional order of Figure 3-6. The Machine learning algorithm uses this file to learn and navigate to the destination point. • sensorsInfo.txt: stores the information for boat tracking and ocean environment research as seen on Figure 3-9; Notepad++ has the collapse object level feature that works perfectly with JSON for data visualization. The logs date and time are appended every 30 seconds time interval and its configurable. Logs are saved using UTC (Universal Time Coordinated) standard time in order not to depend on the time zone of the boat position, making it easier to calculate and graphic the boat position in world maps and process the sensor data in cloud databases, taking into account that JSON support is currently implemented in all databases and programming languajes.

Figure 3-9. Tacking and research data on Raspberry Pi

3.3. Power Management The SenSailor is powered by a 12 volts battery which is charged by a solar panel. The charge is managed by a solar charge controller circuit for longer battery life. The ESP32 that controls the servomotor at the sail also uses an independent battery also charged by solar power. However, since sunlight is not assured, therefore saving power is crucial for long term navigation periods. The primary configurations to reduce the power consumption are the following

3.3.1. Raspberry Pi 4 Low Power Techniques • The raspberry starts in console mode which consumes less power than GUI desktop interface, but if desktop interface is required, a dummy HDMI connector to be hable to connect to the Raspberry with a WiFi connection with a SSH communication protocol. • The communication between the Raspberry Pi and the ESP32 to control the sail through the servomotor is chosen to be bluetooth. This was chosen over Wi-Fi communication, because the Wi-Fi link consumes more energy, since the communication process includes encription and error correction. The power consumption difference between Wi-Fi and bluetooth can be up to 10 times. • From the Raspberry or the ESP32 side, the bluetooth can be disabled to save power; then, when bluetooth is enabled in both devices, they connect instantaneously.

3.3.2. Arduino ONE Low Power Techniques • Set GPIOs as inputs. • By removing Arduino power led , the current lowers by 7 mA. • The Arduino has a Linear Regulator that consumes more power as the difference of voltage to be regulated grows. The initial design had connected 12v to the Arduino Vin pin causing power loss. Now, the arduino is powered through the USB serial cable.

3.3.3. ESP32 low power techniques • Reduce clock speed from 160 MHz to 80 MHz reduces the energy consumption by 20%. • Since the Wi-Fi will not be used, it is turned off by software. • ADC is also turn off. • Set GPIOs as inputs. • Configurable leds turn off, the ESP32 has power on led that cannot be turned off.

3.4. Network Design An important part of the development for this project is to take the control of the SenSailor wirelessly. The SenSailor was previously able only to control the servo motors with a Radio control. This is a very limited solution, since only the servos can be moved without knowing the position and any data aquired by the sensors could be retrieved. The Raspberry Pi OS comes with the VNC (Virtual Network Computing) program preinstalled, which allows remote graphical desktops over IP networks. Therefore, a static IP address was configured on the Raspberry Pi asuming that an Android device will be in charge of controlling it, due to its low cost and availability. Android devices can work as Access Points and generate an IP network with the address range 192.168.42.0/24. Then, on the Rapberry Pi, it was decided to use the address ending in 250 as shown in Figure 3-10 because it is less likely that this address is used, because the first 100 numbers are normally asigned by access points through DHCP.

This PAN (Personal Area Network) can also be replicated by other devices like a laptop or the ESP32. For the current project, it was found optimal to create the network with the cellphone and conect the Raspberry Pi and the laptop to the cellphone PAN, because the cellphone can do tetering also known as phone-as-mode, which consists of sharing the Network Operator internet connection with the wirless PAN generated, as result, both the Raspberry and the laptop will have internet to connect to a server or update a module as needed.

Figure 3-10. Raspberry Pi Static IP configuration seen from the laptop screen The Raspberry Pi starts the Desktop interface if detects a monitor connected to the HDMI port and when not, it starts in console mode. To access the console mode, a user needs to connect to the wireless PAN and use an SSH console to access the Raspberry Pi. The hostname and password are the default for Raspberry Pi, but they should be changed for cybersecurity reasons when the SenSailor performs a task. For the desktop interface, the user can connect a monitor or a dummy HDMI plug to trick the Raspberry Pi and then access using the VNC application and control the Raspberry through Wi-Fi link.

3.5. Artificial Intelligence Machine learning algorithms learn from data, therefore, in projects where the environment is constantly changing, having accurate and correctly prepared data is necessary. On Figure 3-11 it can be seen how Tesla, which is one of the leading companies for self- driving vehicles, focus more on the data than the AI model and algorithm. This is because a model can learn well overtime if the data set is well prepared; the feature extractions require less layers and the model decision become more accurate, as a result, when new data sets are used or when tested in real environments the model performs well. On the other hand, deep and powerful models will perform poorly, overfitting to the data with which was trained when set on real environments.

Figure 3-11. Machine learning data preparation [25] Due to the limited access to the boat during the pandemic and the impossibility to test on the water because of the damages caused by storm Gloria, it was decided to use Reinforcement Learning algorithms that consist in active agents and models that allows the agent learn from its environment. The model will learn from a simulated environment how to drive the boat with enough flexibility to do transfer learning and be able to retrain the model when set in the water to learn from real environments.

3.5.1. Model Development The chosen model is a Deep Reinforcement Learning model that will learn control policies from the input sensors and it is known as AGENT in the RL models. Then to improve the quality of the feature representation, a CNN (Convolutional Neural Network) model will extract the features from the raw data gathered by the sensors and the outcome of the RL model will be driven by the future reward strategy. The CNN model consists on the following: • 6 inputs • 2 hidden layers of 64 neurons • 5 outputs The Deep Neural Network is a fully connected model as shown in Figure 3-12 were all the neurons are interconnected resulting in 4933 parameters with relu activation that will be trained based on rewards in order to learn how to navigate to a destination point by receiving as inputs: the wind angle, the SenSailor state including the state of its actuators and the destination information relative to the current SenSailor position, and as outputs: do nothing, move the helm or move the alerion (indirectly moves the sail). The future reward is designed to reach the destination point without focusing in the speed to reach it. Neither at the research missions or most of the competitions intended to participate requires short times. The destination goal is more imperative. The neural network uses a DQN (Deep Q-Networks) algorithm with stochastic gradient descent to find the best weights for the RL model. This DQN model used for this project was developed by the MIT which license agreement is placed in the project code, it is a

tested model for learning actions based on environment observations and rewards, being needed just to modify the action and state space with the sensor data and actuators.

Figure 3-12. SenSailor Convolutional Neural Network model

The algorithm for training the model works as follows: Initialize/Load weights Episode_Number = 0 while Episode_Number < max episodes do Reset environment Run 1 episode and return number_step if goal reached train model Episode_Number ++ The aim of this project is to get the SenSailor to sail the ocean, by navigating to one point after another until reaching all destination points. Thus, to reach this goal, the sailboat bow should be pointing to the current destination coordinates and the sail should make an angle towards wind that generates the most pushing force to make the SenSailor go forward, except for the case when the wind comes frontwards, in which case the boat will navigate in bowline. About the reward, on every training episode the boat will learn a set of actions to minimize the distance between the SenSailor and the destination point and the training episode is considered done when the distance is less than 1 meter. Changing the sailboat orientation is performed by the helm which is controlled by the servomotors and the sail should make an angle towards wind controlled by the alerion to make the sailboat go forward. Therefore, the reward is accumulated first according to the difference between the previous and the current distance to the destination and also according to

the angle between the bow and the destination point is reduced on inverse proportion as seen on equation (7) and an additional reward is given when the angle difference is 0° with a 5° tolerance. The distance and the angle to the destination is provided by the GPS module that has been already built-in functions that give this information.

(6)

(7) The reward also increases directly proportional to the speed according to equation (7), where the rwdSFactor is smaller than the rwdCFactor in (8) to make it a priority the correct course over the speed and the speed reward (penalization) becomes negative if the current distance to the destination is greater than the previous distance.

(8) During a training episode, a maximum of 3000 steps are allowed to achieve the destination course being one step defined as the set of actions performed before observing the environment at each iteration and every extra step has a penalization to avoid the trained model moving the helm frenetically to win a reward. It can be seen on equations 6, 7 and 8 that they are always multiplied by a factor, this is to manage how certain behaviors affects the total reward, focusing on distance and angle course over the speed for this project. If the destination is not reached after step limit, the environment is reset and the model is not updated. The machine learning model is created with Keras and TensorFlow libraries.

3.5.2. Simulation on Turtle Python To train the AGENT, it is necessary an ENVIRONMENT where the it can learn based on observations and rewards. The Python library used to create the graphic representation of the environment with a graphical library the one known as Turtle, which was chosen because of its integrated features and simplicity to create environments for RL agents. The environment models the sailboat movement response (heading and speed) while navigating in the ocean with the helm and the alerion angle as actions and the wind direction and sailboat course as inputs. On Turtle, figures can be drawn, then move and rotate them based on XY coordinates in a screen object measured in pixels. There are build-in functions that rotates the figure based on angles and turn directions right or left which fits perfectly with the behavior a boat has on the water and the XY coordinates match the latitude and longitude coordinates used for navigation. The environment created for this project is shown on Figure 13. The elements in the screen object and their relationship with the SenSailor and its sensor are the following: • The boat in the center represents the actual boat direction according to the data received by the magnetometer. The image of a sailboat is read from an image file and rotate it according to the simulated/real position. The rotation leaves a black square that is useful to have a good view of the rotated angle. • The big blue arrow is the apparent wind direction which is the wind perceived by the boat as a resultant wind between the true wind direction and the wind

generated by the sailboat course and speed. The apparent wind direction is obtained by the vane. • The grey line represents the course to the destination coordinates. The final distance and angle are shown in the screen object. Since the distance can go off the screen, the journey will be split in sections of 10 meters; 40 pixels represent 10 meters as shown in Figure 3-13. • The boat in the upper right side of the screen is the relative view where the wind is rotated until reaching 270 degrees and represented with the small blue arrow, which is the usual view of the wind as a reference to position the sail. As many degrees the wind is rotated, the same number the reference sailboat will rotate.

Figure 3-13. SenSailor turtle screen environment • The green arrow represents the alerion angle, not to be mistaken with the sail angle. The alerion angle is physically restricted from 45 to 135 degrees and the software that controls the servo motor of the alerion limits the movement to this range also. • The brown arrow represents the helm angle that is also physically restricted from 45 to 135 degrees and the software that controls the servo motor of the helm limits the movement within this range also. • At the upper part of Figure 3-13, in numbers are the angles described previously to allow a better understanding of how the model is learning. Dense Neural Networks become back boxes for the developers and also in the real scenarios where the sail boat is on the water, it is very difficult to see if the angles are taking the different parts and how the SenSailor is responding to commands. For this

reason, the environment created in Turtle has a human and AI interface that are designed to be modified once in a determined function making easier for developers to code. The human interface is controlled using the keyboard according to Figure 3-14 where it can be used to control the virtual or the real boat, the inputs and outputs signals can be switched from virtual to real ones in the code that runs the environment by changing the variable source value in an internal function named configHumanControl(). Turn sail alerion to left Turn sail alerion to right

Turn helm to left Turn Helm to Right Move the boat on screen (optional) Figure 3-14. SenSailor human control interface keyboard layout The AI interface can be configured to reward or punish certain actions on the function step(); in addition, it can be choose the variables sources to be virtual or real. The sailboat is animated with the sailboat direction, helm and alerion turning movements. Additionally, it moves forward only when the sailboat reduces distance to the destination. The intention was not to fully animate the boat movement, but to highlight the actions when the machine learning model is doing actions to achieve the destination point. The environment and the machine learning model are programmed in different files, allowing independence from each other. The learning model can be easily changed by another model or the virtual environment can be improved or removed. In the case of the SenSailor is performing long term missions, the graphical interface will not be required and power can be saved.

3.5.3. Training the model with the GPU The model uses a Deep Neural Network that requires a lot of computation to find the parameters used in the Reinforcement Learning model to drive the SenSailor. The Raspberry Pi 4 comes with a GPU, however, currently there is no support to run machine earning models on it, however, it can be done with the CPU at a lower rate. The Neural Compute Stick V2 from Intel was acquired to perform the online learning on the Raspberry Pi, but development time was short to integrate this hardware with the software already developed. To speed up the process and ensure the model and the environment were working correctly, it was both the agent and the environment run on a laptop with a relative more powerful GPU than the Raspberry and using the python platform ANACONDA, where the Keras and TensorFlow GPU integrated libraries were installed as shown on Figure 3-15.

Figure 3-15. Anaconda installation panel and console The code that runs on the Raspberry Pi and on the laptop is the same. The file DQN_agentBoat.py runs the environment that is coded in file boat.py during a hundred episodes, each one with a maximum of a thousand steps, after that, the learned parameters are saved in a file and the program enters in an Idle mode. The actions based on angles defines the rewards, but to avoid exploiting reward values after long scenarios, the rewards are multiplied by factors equal or below 0.1 to make the calculations up to 3 digit numbers after thousand of steps. This technique improve the data procesing and lower the memory usage In mathematical terms, the program looks for the optimal action value function to be the maximum achievable value following a strategy after a steps sequence s’ and action a’ over all posible actions. This is the Bellman equation (9) where r is the reward value and γ a factor that reduces its value on every step to force the algorithm to reduce the number of steps taken to achieve the best sequence action pair, optimizing the number of actions required to achive the desired goal.

(9)

3.5.4. Machine vision development The SenSailor initially had sensors only for navigation, so it does not take into account encounters with other ships or obstacles. The aim of this thesis is to make the sailboat able to navigate autonomously from one point to another with a clear line of sight. However, initial development is done for the future researchers to have the tools and the software to design the strategies and algorithms for encounters with ships and obstacles. In the following lines, the devices acquired for computer vision and their implementation with the SenSailor are described.

3.5.4.1. LIDAR The LIDAR is a product of Slamtec and it was acquired to improve the short-range obstacle sensing. This technology is widely used on autonomous vehicles. The model used is the RPLIDAR A1 that comes with its own drivers, SDK and executable programs. The application for Windows and Linux is the Slamtec RoboStudio seen on Figure 3-16 where the surrounding area is completely scanned. The maximum scan range is 12

meters, but if an object is placed at a distance less than 50 centimeters, the object will not be showed in the plot. The motor speed of 344 RPM is fixed and cannot be configured, but newer models can be changed as desired.

Figure 3-16. RPLIDAR full scan proprietary software The data acquired can be stored in a txt file with the data separated by spaces. Using a spreadsheet, the data can be displayed and processed in an easier way as seen on Figure 3-17, where the distance is measured in millimeters.

Figure 3-17. RPLIDAR data dump

To have more control over the data, processing and for the code to integrate better with the scripts already developed. A Python script was created where the angle and the distance are saved on a matrix and plotted using a Turtle screen. The drivers for the RPLIDAR used are from Adafruit libraries which were chosen over the Slamtec drivers because this last one had the drivers implemented in C++ with little support, while Adafruit libraries are widely used in robotic projects and have a huge community that support it.

Figure 3-18. RPLIDAR USB port selection a) On RoboStudio b) On Script To connect to the RPLIDAR with the Slamtec software or with the created script in Python, the USB port where the LIDAR is connected with 115200 of baud rate like Figure 3-18.

3.5.4.2. Camera The Raspberry Pi 4 comes with a specific camera port to connect the Pi NoIR Camera V2. The camera can take images day and night with an 8-megapixel camera. This device will be used for long and middle range obstacle detection which along with the LIDAR and computer vision algorithms based on machine learning will make it possible to navigate the world. The camera should be placed at the tip of the mast, with a crystal chamber to protect it without compromising the image capture. To help the design of the chamber and the positioning, it was acquired a 2 meters extension Flex Ribbon Cable for Raspberry Pi. The code was created to take pictures and keep it on a configurable folder to be retrieved by a computer vision algorithm as needed.

Figure 3-19. Pi NoIR Camera connection and flex cable

4. Results

4.1. Sensor implementation The SenSailor hardware and software implementation for navigation to a destination point is configured and tested. Each sensor was tested individually and then all sensors and devices were connected at the same time. The results presented in the Table 7 considering that only the wind vane could not be tested with the actual device due to delays with the device acquisition, however the test was made with a potentiometer in the range of its specification.

Device Hardware installation Drivers and software Status Raspberry Pi 4 Implemented Implemented Working Arduino UNO Implemented Implemented Working ESP32 Implemented Implemented Working Servo motor Helm Implemented Implemented Working Servo motor Sail Implemented Implemented Working Wind vane Simulated with potentiometer Implemented Working Anemometer Implemented Implemented Working Magnetometer Implemented Implemented Working GPS Implemented Implemented Working LDR Implemented Implemented Working Ultraviolet Sensor Implemented Implemented Working DS18B20 Temp sensor Implemented Implemented Working Pi NoIR Camera V2 Implemented Implemented Working LIDAR Implemented Implemented Working Table 7. Devices implementation and status All the testing was done with the SenSailor at the Espai Vela on the dry as seen on Figure 4-1. Tests could not be performed over the water because impermeability and waterproofness could not be guaranteed and the electronic component will be damaged.

Figure 4-1. Testing SenSailor sensors on the dry

4.2. Raspberry Pi Data Acquisition The SenSailor project is designed to be completely modular, not only in the code, but also at the hardware and communications level. All devices can communicate in a full duplex way, except for analog sensors and servomotors as described on Figure 4-2.

Figure 4-2. SenSailor communication flow All sensors, motors and devices can be controlled from the Raspberry Pi 4 through Python scripts which are seen on Figure 4-3. The code is developed in order to work with a dynamic folder location, to run the code from any location without errors.

Figure 4-3. Raspberry Pi 4 scripts and information files The Raspberry Pi operating system was backed up in a file of extension .img with a size of 32 GB. It contains the complete installation state and files required for the SenSailor. The advantage of having an operating system is the possibility of running different programs in parallel with the use of threads, which is already being used running a script for reading the serial link with the Arduino, other script to save information logs and a script to run the AI to navigate. The data is received from the Arduino by the Raspberry Pi and saved in sensorData.txt when a cycle of data gathering from all sensors is received, while every period of time the position and sensor information for research is saved in the file SensorsInfo.txt and the text “Now” is printed on console. All this process is showed on Figure 4-4. This image

was taken from the screen of a laptop connected with the Raspberry Pi through a PAN created by an Android cellphone.

Figure 4-4. Raspberry Pi data gathering and storage

4.3. Machine Learning scores The training was performed on laptop and in the Raspberry Pi, the results are shown in the Figure 4-5 where 4933 parameters in 4 dense and fully connected layers were trained during 100 episodes, where shows the sailboat reaches the destination more and more often with the restriction of 3000 steps per episode. The model can keep learning and improving with more episodes and easing the steps restrictions, however to train the model on a real environment is next step and the current tests in a virtual enviroment is considered enough to show the model potencial to learn.

Figure 4-5. Artificial Intelligence score after training

4.4. Computer Vision Software The script for connecting to the LIDAR and retrieve its data gives the same results as using the RoboStudio. On Figure 4-6, it can be seen a 360-degree scan and, in the lower part, the device information.

Figure 4-6. RPLIDAR script running on Raspberry Pi 4 After several tests using software timers, full scans were performed and the results are described on Table 8. The computation cycle difference between using or not the graphical interface is 800 times faster when comparing the average times.

Full 360 Time in seconds scan Min Avrg Max With GUI 0.8 1.2 1.45 Without GUI 0.0011 0.0015 0.0026 Table 8. RPLIDAR 360° scan times The script for controlling the Pi NoIR Camera was tested with and with minimum light. The images taken are displayed below.

Figure 4-7. Camera with IR filter vs Pi camera without IR filter at nightfall

4.5. Arduino memory The Arduino UNO program was developed in order to optimize the dynamic memory over the program memory by the use of constants instead of variables, global variable reutilization (mostly in timer variables), buffers were avoided when possible and nested functions were created to split local function variables. The dynamic memory can be easily overflowed during a program execution, being hard to predict and fix when occurs. According to Adafruit, a company that specializes on creating sensors and develop libraries for the Arduino, the dynamic memory is considered the most important commodity of the Arduino [26]. The results are shown on Figure 4-8, where in general the Arduino has enough memory to keep adding sensors as needed.

Figure 4-8. Arduino UNO program compilation

4.6. Power consumption The power consumption from sensors was previously calculated on [1] where the consumption from the sensor is the same and according to Figure 4-9, with the addition of the Raspberry Pi 4, Pi Noir camera and LIDAR device. Hours/Day Device Units Voltage [v] Current [A] Power [W] [Wh/day] activation Raspberry Pi 4 + Arduino + LIDAR + 1 5 1.24000 6.20000 24.00 148.80 Camera UV sensor 1 5 0.00700 0.03500 24.00 0.84 Termometer 1 5 0.00150 0.00750 24.00 0.18 Vane 1 12 0.06000 0.72000 24.00 17.28 Anemometer 1 0 0.00000 0.00000 24.00 0.00 GPS 1 3.6 0.06700 0.24120 24.00 5.79 LORA 1 5 0.12000 0.60000 2.00 1.20 Masthead light 1 12 0.00800 0.09600 6.86 0.66 Helm srvomotors 2 6 0.45000 5.40000 12.00 64.80 Magnetometer 1 2.5 0.00028 0.00070 24.00 0.02 Total: 239.56 Figure 4-9. Sensors power consumption [1]

The main modification done to the electric scheme is that the Arduino is not connected to the battery and it does not power any sensor. All the sensors are connected to the battery or the respective voltage converter according to its specification, while the Arduino UNO is powered by the Raspeberry Pi through the USB cable that also works for communication. The Pi Noir camera and the :LIDAR are also directly powered by the Raspberry Pi. As a result, to understand the new power consumtion, several tests were perfomerd which results are displayed in table 9.

Current consumption in Bluetooth Wi-Fi Arduino Camera LIDAR All devices connected Agent and environment Amperes Always ON ON OFF Always ON ON OFF ON OFF and Program runnning Learning OFF 0.56 0.56 0.51 0.85 0.60 0.56 1.887 1.09 1.97 0.75 0.563 Raspberry Pi 4 with GUI

0.48 0.48 0.43 0.785 0.52 0.48 1.05 0.847 1.24 0.67 0.483 Raspberry Pi 4 with console 0.05 0.04 0.203 0.187 ON/OFF difference NA NA NA Table 9. Current consumption on Raspberry Pi 4 All measurements were done in amperes and in two modes: • With GUI: The Raspberry Pi started with the graphical Desktop and connected to a laptop with the VNC program to see the program behavior in a more convenient way and quickly make any necessary adjustments. This mode is useful for short time missions guided by a researcher closer to the SenSailor. • Without GUI: The Raspberry Pi started with in console mode and connected to a laptop with the PuTTY program which is a serial terminal that connects with SSH to the Raspberry to make adjustments to the program while testing it in low power consumption mode. This mode is useful for long time missions with/without being guided by a researcher closer to the SenSailor. The 12 V battery in the SenSailor can provide 25 Ah (Amperes per Hour), as a result, 300 Wh is available to power the electronic system where the devices and sensors consumes 239.56 Wh leaving power to add more sensors. The ESP32 which controls the servomotor at the sail is separated from the 12v battery. It is powered by a Lithium battery of 3.7 volts of 7.4 watts per hour, energized by a sun panel. The device Bluetooth is always turned ON, this is taken as standby mode, since no action is performed. When an order to move the alerion is received from the Raspberry Pi, it moves to that angle and returns to standby. The consumption is showed in Table 10.

Current (mA) Voltage (v) Power (W) Stand by 61 3.7 0.2257 Moving servo 126 3.7 0.4662 Table 10. Current consumption on ESP32

5. Budget

The costs of implementing this thesis project is just a part of the total cost of the complete project. Material and development are still pending to ensure floatability, navigability and protection against corrosion and humidity. However, regarding the electronic part, the development, sensors and devices are integrated, having left only connectors and cables that will not mean a high cost for the continuation and tuning of the SenSailor. Regarding the costs considered as a Junior Engineer receiving a minimum salary according to the school regulations, as student from a Master Degree the work time is 10€/hour plus fees and social insurance, the total wage per hour is 13€. Therefore, just the development cost is detailed in Table 11.

Project Stage Hours Cost Formation and research 300 h 3,900 € Network and GUI Design 200 h 2,600 € Software development 600 h 7,800 € Total 14,300 € Table 11. Project development costs The components acquired for this project were provided by the UPC Barcelona Faculty of Nautical Studies (FNB) and the sponsor mentioned in next section according to Table 12. Component Sponsor Units Cost Lithium Battery 3.7V BFN 1 19.99 € Adafruit solar panel 5.6W BFN 1 54.45 € Solar Panel 50W BFN 1 83.80 € Lithium battery charger ADINSOL 1 22.78 € OkdoPi case for Raspberry Pi BFN 1 4.39 € Dual Battery Solar Charge Controller BFN 1 57.73 € Raspberry Pi 4 8GB BFN 1 88.90 € Pi Noir Cam BFN 1 29.99 € USB power bank BFN 1 18.94 € SanDisk Extreme PRO 32 GB BFN 1 11.99 € RPLIDAR BFN 1 112.98 € Dummy HDMI ADINSOL 1 7.99 € Flex Pi Camera cable ADINSOL 1 7.99 € Waterproof connectors ADINSOL 1 12.99 € UV sensor ADINSOL 1 6.64 € Total 541.55 €

Table 12. Project components costs

It is important to mention that the present project was done entirely with free and open- source software which are alternatives to very costly programs for project development. Next, there is a list of them and the paid solution that costs from hundreds to thousands of euros each. Free Software Paid Software Description Notepad ++ Source Insight Source code editor. KDiff3 Beyond Compare Directory and file comparator and merge software. FreeCAD AutoCAD Computer-aided design modeler software. OBS Camtasia Audio and video recorder for computer screen. OpenShot Movie Maker Audio and video editor. LibreOffice Microsoft Office Office suit for document creation and modification. Table 13. Free and Paid software list

5.1. Sponsor ADVANCED INDUSTRIAL SOLUTIONS EUROPE SL is a company based in Andorra and a branch office in Barcelona. The company works in different sectors of the industry, developing advanced technological solutions through their engineers’ team with focus on sustainability development. The company kindly contribute to the SenSailor project with components listed below for an estimated value of 60 €. • Lithium battery charger • Ultraviolet sensor • Dummy HDMI • Waterproof connectors • Flex cable of 200 centimeters for Pi camera

Figure 5-1. ADVANCED INDUSTRIAL SOLUTIONS EUROPE logo

6. Conclusions and future development

This thesis project was concluded successfully. As a result of this project, the SenSailor boat is fully instrumented and comes with a machine learning algorithm to perform totally autonomous navigation in a free-obstacles scenario. The instrumentation, communication issues and data gathering software have been successfully corrected, improved and tested in real, dry outdoor conditions. The AI model has been successfully trained and tested in a virtual environment developed in this same project as well showing that the model learned the basic actions to navigate to a destination. The project has been a very challenging and complete one where up to 5 different software environments and languages have been used. Several decisions have to been taken to connect everything, avoiding interference, keeping low power consumption in a scalable design to keep adding sensors and devices.

And the particular results are as follows:

Objectives Related Result conclusions Sections

Implement the sensors, devices 3.1, 3.2, All the sensors work correctly, without and data management for an 4.1 and 4.2 interfering with each other and the data is autonomous sailboat. stored in JSON format, including real time position tracking by GPS.

Design a wireless communication 3.2and 3.4 Wi-Fi and Bluetooth communication were system to connect all SenSailor developed to control all the devices of the parts. SenSailor with a laptop or a smartphone.

Apply techniques to make the 3.3 and 4.6 Low power techniques and optimized code were SenSailor self-sustainable from applied, allowing to have margin to add 9 new an energy point of view for short- sensors with the current communication and long-term research missions protocol. in the ocean.

Design and train a machine 3.5, 4.3 A Deep Reinforcement Learning model and a learning algorithm for navigation and 4.4 virtual environment were developed for based on destination points. autonomous drive point to point, also integrating a LIDAR and a camera for navigation by computer vision.

The SenSailor project will continue with new works. As future developments to be introduced in these works, we can point out the following: • There should be a logic that will make the AI to learn from the current environment the necessary procedures to recover the navigation if new circumstances appear and this training should be done only during daylight since the process consumes power and the solar cells will be able to charge the battery if needed. • Develop a robust algorithm for the AI to the use of LIDAR and computer vision to avoid obstacles according to the COLREGs maneuvers. • Develop a communication protocol for sending data in an optimal way through satellite communication. Since sending data over satellite communication is very expensive, it is recommended to use messaging based on delta encoding. • Security on the wireless communication should be revised. Bluetooth is a wireless technology that is vulnerable to man in the middle and phishing attacks. • The positional system should no rely only on GPS (United States satellites) information, the current positioning module at SenSailor can be configured to obtain the position also from Galileo (Europe satellites) or GLONASS (Russia satellites) positional systems.

Bibliography

[1] J. Sastre, C. Manich. "Disseny i construcció d’un dron de navegació automàtica a vela". Bachelor thesis, Facultat de Nàutica de Barcelona, Universitat Politècnica de Catalunya, Barcelona, Spain, 2020. [2] The Microtransat Challenge [Accessed: December 14th, 2020]. Available on: https://www.microtransat.org/ [3] The World Robotics Sailing Championship and The International Robotic Sailing Conference, Ningbo - China, held on 2019 [Accessed: December 12th, 2020]. Available on: https://www.roboticsailing.org/2019/rules/ [4] The Raspberry Pi Foundation [Accessed: 14 December 2020]. Available on: https://www.raspberrypi.org/documentation/faqs/ [5] Raspberry Pi used to steal data from Nasa lab [Accessed: 5 January 2021]. Available on: https://www.bbc.com/news/technology-48743043 [6] Belgian security researcher hacks Tesla with Raspberry Pi [Accessed: 5 January 2021]. Available on: https://www.computerweekly.com/news/252492564/Belgian-security-researcher- hacks-Tesla-with-Raspberry-Pi [7] J M Hughes. Arduino A Technical Reference A Handbook for Technicians Engineers and Makers In a Nutshell 1st ed. O'Reilly Media, 2016. [8] Adafruit [Accessed: 15 December 2020]. Available on: https://www.adafruit.com/about [9] Espressif Systems, “ESP32 Series datasheet” [Accessed: 12 December 2020]. Available on: https://www.espressif.com/sites/default/files/documentation/esp32_datasheet_en.pdf. [10] The Internet of Things with ESP32 [Accessed: 12 December 2020]. Available on: http://esp32.net/

[11] Bluetooth SIG, Intro to Bluetooth Low Energy [Accessed: 12 December 2020]. Available on: https://www.bluetooth.com/bluetooth-resources/intro-to-bluetooth-low-energy/ [12] Python.org [Accessed: 14 December 2020]. Available on: https://docs.python.org/3/faq/general.html#general-information [13] Arduino Software (IDE) [Accessed: 14 December 2020]. Available on: https://www.arduino.cc/en/Guide/Environment [14] The JavaScript Object Notation (JSON) Data Interchange Format IETF RFC 8259, December 2017. [15] Notepad++ User Manual [Accessed: 14 December 2020]. Available on: https://npp-user-manual.org/docs/programing-languages/ [16] Machine Learning Algorithms Comparison [Accessed: 24 December 2020]. Available on: https://medium.com/fintechexplained/machine-learning-algorithm-comparison- f14ce372b855 [17] Richard S. Sutton and Andrew G. Barto Reinforcement Learning: An Introduction 1st ed. The MIT Press, Cambridge, MA, US, 2017. [18] E. Kulbiej & P. Wołejsza Naval Artificial Intelligence, Maritime University of Szczecin, Szczecin, Poland, 2017 [19] Bradley Martin, Danielle C. Tarraf, Thomas C. Whitmore, Jacob DeWeese, Cedric Kenney, Jon Schmid, Paul DeLuca Advancing Autonomous Systems. An Analysis of Current and Future Technology for Unmanned Maritime Vehicles RAND Corporation, Santa Monica, US, 2019 [20] Zhixiang Liu, Youmin Zhang ∗, Xiang Yu, Chi Yuan Unmanned surface vehicles: An overview of developments and challenges, Quebec, Canada, 2016 [21] Max Kanat-Alexander Code Simplicity O’Reilly, CA, US, 2012. [22] AI and Compute [Accessed: 28 December 2020]. Available on: https://openai.com/blog/ai-and-compute/ [23] Intro to Python [Accessed: 28 December 2020]. Available on: https://www.slideshare.net/JayCoskey/intro-to-python-high-school-unit-3

[24] Raspberry Pi No-IR Camera Module V2 8MP 1080P IR Sensitive Webcam RPI [Accessed: 28 December 2020]. Available on: https://labists.com/products/raspberry-pi-webcam-rpi-camera-module-official

[25] Andrej Karpathy, Building the Software 2.0 Stack, 2018 [Accessed: 2 January 2021]. Available on: https://karpathy.ai/ [26] Optimizing SRAM [Accessed: 26 January 2021]. Available on: https://learn.adafruit.com/memories-of-an-arduino/optimizing-sram [27] Introduction to Arduino Uno [Accessed: 5 January 2021]. Available on: https://www.theengineeringprojects.com/2018/06/introduction-to-arduino-uno.html [28] A Short History of Machine Learning [Accessed: 3 January 2021] Available on: https://www.forbes.com/sites/bernardmarr/2016/02/19/a-short-history-of-machine- learning-every-manager-should-read/?sh=30ef962615e7

Glossary

Ad Hoc From the Latin word that means "for this” Ah Amperes per Hour AI Artificial Intelligence BT Bluetooth COLREGs The International Maritime Organisation's Rules for Prevention of Collisions at Sea CPU Central Processing Unit CNN Convolutional Neural Network CRC Cyclic Redundancy Check DAQ Data Acquisition Hardware DHCP Dynamic Host Configuration Protocol DNN Deep Neural Network DQN Deep Q-Networks or Deep Q Learning FNB Facultad de Náutica de Barcelona GB Giga Bytes Ghz Giga Hertz GNU General Public License GPS Global Positioning System GPU Graphics Processing Unit GUI Graphical User Interface HDMI High-Definition Multimedia Interface IDE Integrated Development Environment IETF Internet Engineering Task Force IoT Internet of Things IR Infrared JSON JavaScript Object Notation LDR Light Dependent Resistor LIDAR Laser Imaging Detection and Ranging ML Machine Learning NANET Nautical Ad-hoc Network NN Neural Network NoIR No Infrared

OS Operating System PCB Printed Circuit Board RAM Random Access Memory RFC Request for Comments RISC Reduced Instruction Set Computer RL Reinforcement Learning SARSA State Action Reward State Action SDK Software Development Kit SSH Secure Shell SI International System of Units SIG Special Interest Group USB Universal Serial Bus USV Unmanned Surface Vehicles UTC Universal Time Coordinated UV Ultraviolet VNC Virtual Network Computing WRSC World Robotic Sailing Championship

Appendix

SenSailor Electric Scheme upper part

SenSailor Electric Scheme lower part

SenSailor wiring