Target Detection, Tracking and Avoidance System for Low-cost UAVs using AI-Based Approaches Vinorth Varatharasan1, Alice Shuang Shuang Rao1, Eric Toutounji1, Ju-Hyeon Hong1, Hyo-Sang Shin1 Abstract— An onboard target detection, tracking and avoid- been developed in recent years and based on Convolutional ance system has been developed in this paper, for low-cost Neural Networks (CNNs). There are YOLO [1], SSD [2], UAV flight controllers using AI-Based approaches. The aim of R-CNN [3] and Faster R-CNN [4] with outstanding per- the proposed system is that an ally UAV can either avoid or track an unexpected enemy UAV with a net to protect itself. formance in both accuracy and frame rates. Hossain et In this point of view, a simple and robust target detection, al. [5] compared onboard embedded GPU systems, off-board tracking and avoidance system is designed. Two open-source GPU ground station, and onboard GPU constraint systems tools were used for the aim: a state-of-the-art object detection for target detection from a UAV in terms of frame rates technique called SSD and an API for MAVLink compatible and accuracy performance. Due to transmission capacity systems called MAVSDK. The MAVSDK performs velocity control when a UAV is detected so that the manoeuvre is done constraint and the limited onboard computing power, Intel simply and efficiently. The proposed system was verified with Movidius Neural Compute Stick (NCS) using the Intel Software in the loop (SITL) and Hardware in the loop (HITL) Movidius Neural Compute SDK (NCSDK) is introduced to simulators. The simplicity of this algorithm makes it innovative, accompany with Raspberry Pi while Mobilenet-SSD (5FPS) and therefore it should be used in future applications needing outperforms YOLO (1FPS). robust performances with low-cost hardware such as delivery drone applications. When detecting a UAV, a depth distance is primordial to I. INTRODUCTION obtain the distance between the UAV and the target. There Unmanned aerial vehicles (UAVs) have a growing impact are multiple active detection sensors, such as radar, LiDAR; on society. From their first use for war-fighting in 1849, however, the cost and deficiency in calibration with the nowadays they are used everywhere: surveillance, delivery, camera lead us to the passive detection via stereo vision. defence, agriculture, and so on. Therefore, their introduction To obtain the depth of the stereo vision, the most effective into our daily life arouses safety issues. Today, fully au- approach is based on supervised learning in CNNs, which tonomous UAVs are much more efficient than piloted UAVs, requires an abundant training set for a promising result but also riskier. One important requirement is the ability to applied in a single picture. Aguilar et al. [6] proposed a sense and avoid any obstacle in the environment. novel unsupervised technique that first calculates disparity in Target detection, tracking and avoidance system for UAVs an RGB image via CNNs and then depth by the geometrical are being developed in different areas: in defence as relationship with disparity. weapons, in civil as delivery drones, etc. Furthermore, Anti- UAV Defense Systems (AUDS) has been enhanced to counter new threats of UAVs (e.g. Gatwick Airport UAV incident in Finally, after a target is detected, it can be avoided or December 2018). Thus, target tracking is also an essential tracked by performing an adapted manoeuvre. In [7], a ability to be developed for UAVs. survey on different manoeuvre approaches was done: a BAE Systems sponsored an inter-university UAV Swarm geometric approach [8], an optimised trajectory approach [9], competition to simulate the military-world conditions in a a bearing-angle based approach [10], a force-field ap- arXiv:2002.12461v1 [cs.CV] 27 Feb 2020 game of offence and defence using UAVs. The mission proach [11], and so on. The advantage of geometric based statement was: “Create a novel and innovative solution with approaches compared to other approaches like probabilistic respect to defending against a swarm of UAVs”. and optimisation based algorithms is that they require less In this UAV swarm, two essential requirements had to be computational power. respected: • The potential collisions must be detected. In this paper, our objectives are to obtain a simple but • The ally UAVs must be able to either track or avoid effective approach to the detection, tracking and avoidance enemy UAVs (track the enemy when our agent has a problems with low computational power. The paper starts net and avoid the enemy after it has released the net). with an overview of the overall architecture, followed by Before a target is avoided or tracked, it has first to be a focus on the detection, tracking and avoidance method- detected. Many state-of-the-art detection algorithms have ologies. The proposed system is verified with the software in the loop (SITL) and hardware in the loop (HITL). Their 1 School of Aerospace, Transport and Manufac- performances are assessed in the results and discussions turing, Cranfield University, Cranfield MK43 0AL, UK (email: [email protected], results. Finally, the main outcomes are summarised in the [email protected]) conclusion. II. PROPOSED METHODOLOGY A. Architecture The global architecture for the Guidance, Navigation and Control (GNC) system is represented in Figure 1. The Ground Control Station (GCS) integrates the graphic user interface (GUI) allowing a user to set the mission parameters Fig. 2. Stadiametric rangefinding and a task control algorithm which is running to create way- points. The whole ground segment can send the commands to mainly taken from the Internet (Google Images, already- the flight control computer (FCS) via a WiFi network using made datasets, etc.). For the different sets, the following ratio the MAVLink protocol. was followed: training set (64%), validation set (16%) and Also, each MAVLink message is bypassed by the FCS test set (20%). For example, the most important category is and forwarded to the companion computer using a serial the UAV class and therefore, 2; 000 UAV pictures, labelled connection. The companion computer calculates commands by ourselves, were included in the training dataset. The for tracking and collision avoidance using the detection model was trained within TensorFlow in 200; 000 steps, results from the onboard detection system and sends the which achieved over 95% accuracy for the test dataset commands to the FCS. The detection algorithm is running (different images from the training dataset), with the mean on the companion computer with a vision camera. To access average precision metric. Using OpenVino Model Optimizer the data from the serial port on the onboard computer side, developed by Intel, TensorFlow model is converted into IR a MAVLink library with APIs for C++, MAVSDK, has been format, which could be interpreted by Neural Compute SDK used. (NCSDK) and run on the onboard computer. Hence there are two tracking modes: the waypoints track- ing mode using GCS’ commands and the target tracking Since the major challenge of using a camera alone is the mode using the companion computer’s commands. The GNC fact that no direct depth information is obtained since the system determines an appropriate tracking mode according angle subtended is the same. However, if we can classify to the onboard target detection results. For example, when what an object is (e.g. a UAV), and thus a look-up table can detected the target, the GNC system is switching the tracking be used, and therefore we can use a stadiametric rangefinding mode to target tracking, and vice versa. method. Every trained class is inside the look-up table and hence, if an object is detected, the rangefinding algorithm B. Detection algorithm will estimate its depth. The system requirements of the detection algorithm are In other words, in Figure 2, three parameters are known: essentially based on the following three aspects: • The distance x, which is the mean size of both small 1) Real-time (2 FPS according to the physical restrictions and big UAVs. from CA/tracking algorithms). • The angle subtended (on the detector, this is occupying 2) Accuracy higher than 70% for 50 meters. for example 50 pixels across). 3) Depth distance estimation: Error within one meter for • The instantaneous field of view (IF OV ) of the detector. 5 meters away detection (considering net size). Therefore, the subtended angle of the target is 50∗IF OV . Considering the requirements of real-time, Mobilenet-SSD Then, basic trigonometry can be used to find the depth of brings the best accuracy trade-off within the fastest detectors. the object. This method requires a good classification of Every object that would be likely to be avoided had to be UAVs. In order to obtain the depth estimation function, the included in the training dataset, and for that, images were linear relationships between the observed size in the camera Fig. 1. GNC system architecture Algorithm 1 Target positioning algorithm using the detection TABLE I results and depth estimation algorithm PROS AND CONS OF THE OFFBOARD VELOCITY AND POSITION CONTROL 1: if a UAV is detected AND the Velocity control Position control probability is more than a specified Controllability Better Worse threshold then Guidance reactivity Faster Slower Complexity 2 1 2: WI Initial width 3: HI Initial height 4: (xmin; xmax; ymin; ymax) Coordinates of the de- inputs to the flight control stack using the Offboard 5: tection bounding-box (multiplied by WI and HI ) xmax−xmin ymax−ymin mode. 6: centre = (xmin + 2 ; ymin + 2 ) 2 WI 2 HI 7: centremodified = ( ∗(centre[0]− ); ∗( − The task allocation algorithm is the default one: the WI 2 HI 2 centre[1])) UAV follows a pre-planned mission which depends on its (x −x )(y −y ) 8: size = max min max min role (attack or defence).