Sharkeye: Real-Time Autonomous Personal Shark Alerting Via Aerial Surveillance
Total Page:16
File Type:pdf, Size:1020Kb
University of Wollongong Research Online Faculty of Engineering and Information SMART Infrastructure Facility - Papers Sciences 1-1-2020 Sharkeye: Real-Time Autonomous Personal Shark Alerting via Aerial Surveillance Robert A. Gorkin III University of Wollongong, [email protected] Kye Adams University of Wollongong, [email protected] Matthew J. Berryman University of Wollongong, [email protected] Sam Aubin Wanqing Li University of Wollongong, [email protected] See next page for additional authors Follow this and additional works at: https://ro.uow.edu.au/smartpapers Part of the Engineering Commons, and the Physical Sciences and Mathematics Commons Recommended Citation Gorkin III, Robert A.; Adams, Kye; Berryman, Matthew J.; Aubin, Sam; Li, Wanqing; Davis, Andrew R.; and Barthelemy, Johan, "Sharkeye: Real-Time Autonomous Personal Shark Alerting via Aerial Surveillance" (2020). SMART Infrastructure Facility - Papers. 300. https://ro.uow.edu.au/smartpapers/300 Research Online is the open access institutional repository for the University of Wollongong. For further information contact the UOW Library: [email protected] Sharkeye: Real-Time Autonomous Personal Shark Alerting via Aerial Surveillance Abstract While aerial shark spotting has been a standard practice for beach safety for decades, new technologies offer enhanced opportunities, ranging from drones/unmanned aerial vehicles (UAVs) that provide new viewing capabilities, to new apps that provide beachgoers with up-to-date risk analysis before entering the water. This report describes the Sharkeye platform, a first-of-its-kind project to demonstrate personal shark alerting for beachgoers in the water and on land, leveraging innovative UAV image collection, cloud- hosted machine learning detection algorithms, and reporting via smart wearables. To execute, our team developed a novel detection algorithm trained via machine learning based on aerial footage of real sharks and rays collected at local beaches, hosted and deployed the algorithm in the cloud, and integrated push alerts to beachgoers in the water via a shark app to run on smartwatches. The project was successfully trialed in the field in Kiama, ustrA alia, with over 350 detection events recorded, followed by the alerting of multiple smartwatches simultaneously both on land and in the water, and with analysis capable of detecting shark analogues, rays, and surfers in average beach conditions, and all based on ~1 h of training data in total. Additional demonstrations showed potential of the system to enable lifeguard- swimmer communication, and the ability to create a network on demand to enable the platform. Our system was developed to provide swimmers and surfers with immediate information via smart apps, empowering lifeguards/lifesavers and beachgoers to prevent unwanted encounters with wildlife before it happens. Keywords via, surveillance, alerting, aerial, shark, personal, autonomous, real-time, sharkeye: Disciplines Engineering | Physical Sciences and Mathematics Publication Details Gorkin III, R., Adams, K., Berryman, M. J., Aubin, S., Li, W., Davis, A. R. & Barthelemy, J. (2020). Sharkeye: Real-Time Autonomous Personal Shark Alerting via Aerial Surveillance. Drones, 4 (2), 18-1-18-17. Authors Robert A. Gorkin III, Kye Adams, Matthew J. Berryman, Sam Aubin, Wanqing Li, Andrew R. Davis, and Johan Barthelemy This journal article is available at Research Online: https://ro.uow.edu.au/smartpapers/300 drones Article Sharkeye: Real-Time Autonomous Personal Shark Alerting via Aerial Surveillance Robert Gorkin III 1,*, Kye Adams 2 , Matthew J Berryman 1,3,4 , Sam Aubin 5 , Wanqing Li 6, Andrew R Davis 2 and Johan Barthelemy 1 1 SMART Infrastructure Facility, University of Wollongong, Wollongong 2522, Australia; [email protected] (M.J.B.); [email protected] (J.B.) 2 Centre for Sustainable Ecosystem Solutions and School of Earth Atmospheric and Life Sciences, University of Wollongong, Wollongong 2522, Australia; [email protected] (K.A.); [email protected] (A.R.D.) 3 Across the Cloud Pty Ltd., Wollongong 2500, Australia 4 Centre for Bioinformatics and Biometrics, National Institute for Applied Statistics Research Australia, University of Wollongong, Wollongong 2522, Australia 5 Sharkmate, Wollongong 2500, Australia; [email protected] 6 Advanced Multimedia Research Lab, University of Wollongong, Wollongong 2522, Australia; [email protected] * Correspondence: [email protected] Received: 16 March 2020; Accepted: 24 April 2020; Published: 4 May 2020 Abstract: While aerial shark spotting has been a standard practice for beach safety for decades, new technologies offer enhanced opportunities, ranging from drones/unmanned aerial vehicles (UAVs) that provide new viewing capabilities, to new apps that provide beachgoers with up-to-date risk analysis before entering the water. This report describes the Sharkeye platform, a first-of-its-kind project to demonstrate personal shark alerting for beachgoers in the water and on land, leveraging innovative UAV image collection, cloud-hosted machine learning detection algorithms, and reporting via smart wearables. To execute, our team developed a novel detection algorithm trained via machine learning based on aerial footage of real sharks and rays collected at local beaches, hosted and deployed the algorithm in the cloud, and integrated push alerts to beachgoers in the water via a shark app to run on smartwatches. The project was successfully trialed in the field in Kiama, Australia, with over 350 detection events recorded, followed by the alerting of multiple smartwatches simultaneously both on land and in the water, and with analysis capable of detecting shark analogues, rays, and surfers in average beach conditions, and all based on ~1 h of training data in total. Additional demonstrations showed potential of the system to enable lifeguard-swimmer communication, and the ability to create a network on demand to enable the platform. Our system was developed to provide swimmers and surfers with immediate information via smart apps, empowering lifeguards/lifesavers and beachgoers to prevent unwanted encounters with wildlife before it happens. Keywords: UAV; blimp; shark spotting; machine learning; wearables 1. Introduction Managing shark–human interactions is a key social and environmental challenge requiring a balance between responsibilities of providing beachgoer safety and the responsibility to maintain shark populations and a healthy marine ecosystem. Sharks pose some limited risk to public safety [1]; however, there is a recognized social and environmental responsibility to develop strategies that effectively keep people safe and ensure the least harm to the environment [2]. This process is still evolving, particularly in countries with yearly shark interactions. For instance, in Australia, The Shark Drones 2020, 4, 18; doi:10.3390/drones4020018 www.mdpi.com/journal/drones Drones 2020, 4, 18 2 of 17 Meshing (Bather Protection) Program NSW [3] is a hazard mitigation strategy in place since 1937 which is known to cause harm to vulnerable species [4] and is listed as a Key Threatening Process by the NSW Scientific Committee with ecological and economic costs [5]. With the range of emerging technologies offering new potential to mitigate harm to people and sharks alike, there are new ways to manage interactions without such high costs. Aerial surveillance has historically been used to help identify sharks in beach zones and warn beachgoers in order to avoid shark incidents. Australia started one of the earliest “shark patrols” known through the Australian Aerial Patrol in the Illawarra in 1957 [6]; currently, across the country, 1000s of km are patrolled by airplanes and helicopters on any weekend. For decades, aerial patrols have absolutely helped mitigate shark interactions, predominantly by alerting lifeguards/lifesavers to enact beach closures [7], and further are essential for a range of safety and rescue purposes. However, a recent study suggested the ability to reliably detect sharks from planes and helicopters has shown limited effectiveness, with only 12.5% and 17.1% of shark analogues spotted for fixed-wing and helicopter observers, respectively [8]. Though the assessment of such studies has been debated [9], there is certainly agreement that new technologies could assist expansion of aerial shark spotting and communication of threats to lead to better outcomes for humans and sharks alike. The recent expansion of unmanned aerial vehicles (UAVs), often known as drones, offers new abilities to enhance traditional aerial surveillance. In recent years alone, drones have been used to evaluate beaches and coastal waters [10–12] and monitor and study wildlife [13–15], from jellyfish [16], sea turtles [17–19], salt-water crocodiles [20], dolphins and whales [21–24], and manatees [25] to rays [26] and sharks [27–30]. This expansion in surveillance is likely related to the market release of consumer drones with advanced camera imaging; for instance, the DJI Phantom introduced in 2016. Whether rotor-copter or fixed wing, rapid development of commercial drones means more flying “eyes” can be deployed in more locations for conservation and management [31]. Using these methods for shark detection again shows great promise; drones are now a part of several lifeguard programs in Australia in multiple states [32] and are continuously being evaluated for reliability [33]. Other UAVs are also being considered [34]; a review by Bryson and Williamson (2015) into the use of UAVs