22nd Australasian Fluid Mechanics Conference AFMC2020 Brisbane, , 7-10 December 2020 https://doi.org/10.14264/767544c

Assessment of Surf Amenity using Computer Vision with Convolutional Neural Networks to Track Wave Pockets M. Thompson1, E. Watterson2 and T. E. Baldock1 1School of Civil Engineering, University of Queensland, Brisbane QLD 4072, Australia 2Bluecoast Consulting Engineers, 1 Bruce St, NEWCASTLE, NSW 2300, Australia

Abstract This new and valuable approach to quantifying surf amenity uses video footage and computer vision techniques, such as the Surf amenity can be significantly affected by coastal convolutional neural network (CNN). The Wave Pocket infrastructure and is a growing consideration for local Tracking (WPT) software is presented in this paper, together governments. Quantitative assessment of the surf amenity of with its application at two field sites. artificial reefs is also growing in interest. Surf amenity has been quantified using metrics derived from the surfable region near the breaking edge of a wave, defined as the ‘wave pocket’. Wave pocket tracking software has been developed to track Field Measurements potential rides using computer vision. This software Two locations on the Gold (QLD) were used for enabled a surf amenity assessment of the Rainbow (QLD) developing and testing the WPT software; Palm artificial ‘superbank’ during Cyclone Eusi and the Palm Beach artificial and Rainbow Bay. A professional grade filming drone was reef. The produced data revealed new insights into the surf used to film the Palm Beach on the 30/12/2019. amenity of natural and artificial surf locations. Figure 1 below shows a snapshot of the view of easterly waves Keywords breaking on the artificial reef during the low tide of 0.23m (above LAT) and a significant wave height 퐻푠=1.4m and peak Surf amenity; experimental techniques and facilities; field period 푇푝=10s, measured at the nearby Palm Beach Wave validation; artificial reef design; convolutional neural networks; Buoy. wave breaking.

Introduction Surfing has long been a major recreational activity and is a significant consideration for local governments in terms of tourism and local economies [10,2]. However, coastal infrastructure often influences the local morphology of a beach, which in turn has led to diminished surf amenity at some famous surfing locations [13]. A prime example is at Kirra, just north of Rainbow Bay (QLD) where the instalment of rock walls at the Tweed Heads caused changes in the Figure 1. Video frame of breaking waves at Palm Beach artificial reef at 5:29pm on the 30th of December 2019, 9 minutes of footage was of . The world-class surfing break of Kirra in turn had collected (footage Bluecoast Consulting Engineers). diminished surf amenity. Regular sand pumping and dredging has been implemented by the government to remedy this Additional film footage from the Rainbow Bay lookout on the situation which has produced the famous surfing ‘superbank’. Gold Coast was collected during swell conditions from Cyclone A more recent example of government action to maintain surf Eusi, 14/02/2020 (Figure 2). The lookout is positioned south- amenity is the Palm Beach artificial reef, which was constructed west of the famous ‘superbank’ point break, with an elevation primarily as a defence against beach erosion but also to provide of approximately 20m above sea level. The filming was a new surf amenity. performed with a conventional video camera during a high tide of 1.6m, with easterly waves of 퐻푠=2m, 푇푝=12s. The industry has adapted to the increased value of surf amenity when designing coastal infrastructure. A range of numerical wave models that simulate wave breaking processes are used to assess how surf amenity is affected by proposed designs [9,15]. For this modelling to be effective there must be a means of validation, which in this case requires a form of quantification of surf amenity. Surf amenity can be quantified from observations, with metrics defined such as the peel angle [8]. Such surf amenity assessments are effective as a time-averaged measure, however there remains a need for surf amenity quantification on a wave by wave basis. Previous work has attempted this by tracking bores and surfers themselves [5,4]. This paper presents a new approach to Figure 2. Video frame of breaking waves at Rainbow Bay during quantifying surf amenity, by tracking the breaking edge of a Cyclone Eusi at 1:11pm on the 14th of February 2020, 9 minutes of wave, the surfable region defined as the ‘wave pocket’. footage was collected (footage Michael Thompson). Algorithm connected layers was found to be sufficient for this task. Following the classification of a wave pocket, it needs to be The WPT approach involves tracking all the wave pockets assigned to an ID. A cost function was formed from distance, visible in a video to produce rectified surf amenity metrics. This time and image correlation calculations between stored wave was implemented in software using Python, with the OpenCV pockets and assigned wave pockets. The cost function was used library for general computer vision tasks including rectification to determine if an unassigned wave pocket belonged to a stored and the TensorFlow library for training and using the CNN. (previously identified) wave pocket track or was a completely Identifying and tracking objects is a known problem in new one. This assignment process was similar to that described computer vision. For the case of the WPT software, the main in the SORT process [1]. Wave pocket tracks with too few process involved detecting movement, identifying wave associated frames were not included in the output data, this pockets and tracking identified wave pockets. A snapshot of the helped reduce the possibility of tracking whitecaps in the case WPT software functioning is shown in Figure 3. Each wave of a windy day. pocket was assigned an ID (e.g. 116) and the trajectory of the centre of each wave pocket bounding box was stored according to its ID. The wave pocket tracks extracted from the footage could then be rectified to a map. A plane representing the mean water surface is required in spatial coordinates as well as in the images, to allow rectification of the video images into a plan view. Figure 5 illustrates how the video frame from Figure 2 can be rectified to the mean water surface by homography [6], such that wave pockets identified in the video could be translated to latitude and longitude coordinates. Figure 3. Wave pockets identified at Rainbow Bay during Cyclone Eusi by the WPT software.

The detection of movement was achieved using background subtraction [11]. This involves taking the difference between consecutive frames to identify regions of movement. Each region of movement then requires classification, as a wave pocket or not a wave pocket. Traditional computer vision methods could not be used for classification due to variation in water quality, wave pocket size, foaminess, objects within the wave pocket, as well as varying brightness from weather conditions. Consequently, a state-of-the-art deep learning classification method was adopted, a CNN [7] which uses labelled images. Deep learning has recently been used for remote sensing in coastal research for surface elevation Figure 5. Figure 2 rectified to a Nearmaps image of Rainbow Bay measurements [3]. (background from Nearmaps 19/11/2019).

For Rainbow Bay, rectification was achieved by paddling out To implement a CNN for wave pocket classification, over 1400 with a GPS device on a on a calm day with a similar images were labelled manually as either wave pockets or not tide level, with simultaneous filming from the same position as wave pockets. These images were extracted from beach footage used for the Cyclone Eusi data. By matching the position of the collected from a range of Gold Coast in different GPS device in the video footage with GPS coordinates, weather conditions. The beaches included Currumbin Alley, rectification could be achieved. For Palm Beach, the drone Duranbah, Palm Beach, Palm Beach artificial reef, Rainbow footage was rectified using the GPS locations of marker buoys Bay and . The varying conditions included cloud deployed on the day, but data was not obtained simultaneously. cover, sunshine, glare, large waves, small waves, foamy waves, This technique proved to be lower in accuracy due to the drift clear water, rough water, surfed waves and non-surfed waves. of the buoys. Whilst time-averaging buoy positions was used to This variation in labelled images is very important to allow the partially remedy this, simultaneous collection of GPS location CNN model to generalise when being trained. Figure 4 below and images is preferable. shows a few of the images used for training the CNN model.

Following the rectification of the wave pocket tracks data, these key metrics were defined to assess surf amenity: ride rate (rides/min), average ride duration (s), average ride length (m) and maximum ride length (m). The individual wave pocket tracks represent potential ‘rides’. Ride length was an approximate measure given by the Euclidean distance between Figure 4. Wave pockets (green) and not wave pockets (red) used for the starting point and finishing point of a wave pocket track. training the CNN model, classification by Michael Thompson. Therefore, if there is curvature the ride length metrics are potentially underestimated in length and hence a conservative With these labelled images a CNN model was trained as a measure. A greater value of each metric indicates greater surf classifier for wave pockets, which takes an image as input and amenity. Maps of the wave pocket tracks were produced, returns a confidence value of whether the image is a wave together with histograms of ride durations. These metrics and pocket. A lightweight CNN architecture, VGGNet [12], with figures are presented in the following results and discussion only two VGGNet style convolutional layers and two fully section. Results and Discussion The WPT software was run for the Palm Beach artificial reef

drone footage and the Rainbow Bay lookout footage. Table 1

below shows the key metrics produced. It is important to ) 6 recognise that surf amenity is largely reliant on the wave conditions. Since filming was completed during different wave conditions, Palm Beach artificial reef and the Rainbow Bay ‘superbank’ metrics are not directly comparable. Each set of

results were treated separately.

N Bearing (m × 10 × (m Bearing N - Metric Palm Beach Rainbow Bay S Artificial Outer Surf Reef* Zone

Wave Conditions** 1.4m, E, 10s, 2.0m, E, 12s, W-E Bearing (m × 105) Tide 0.23m Tide 1.6m Figure 7. Wave pocket tracks in the outer surf zone of Rainbow Bay Ride Rate (rides/min) 10 36 during Cyclone Eusi (background from Nearmaps 19/11/2019). Wave pocket tracks below the orange line were excluded to produce this Avg. Ride Duration (s) 5 10 figure and the results, such to focus on the outer surf zone and exclude smaller tracks on secondary bars. Avg. Ride Length (m) 46 72 Max. Ride Length (m) 153 454 The Palm Beach artificial reef results in Table 1 reveal high surf amenity given the wave conditions. However, due to the lower Table 1. Key metrics for surf amenity assessment from wave pocket rectification accuracy of the drone footage there is less tracks. *Metrics for tracks that were within the vicinity of the reef only. confidence in the positioning of the tracks on the image in ** (m), Peak Wave Direction and (s) from the Palm Beach Wave 퐻푠 푇푝 Figure 8. The spatial metrics produced also would be influenced Buoy, Tide from willyweather.com.au. by a lower rectification accuracy. The temporal metrics are still accurate and very insightful. A ride rate of 10 rides/min as well Rainbow Bay provided exceptional surf amenity during the as average ride duration of 5s shows desirable surf amenity for Cyclone Eusi swell event with a high ride rate of 36 rides/min. relatively smaller wave conditions in the vicinity of the reef. The average ride length of 72m with average ride duration of 10s reveals why the ‘superbank’ is regarded as a world class surfing location. A maximum ride length of 454m is rarely seen even at other world class surfing locations [14]. This maximum recorded ride length was only within field of view of the video therefore was likely even longer. The histogram in Figure 6 shows that many wave pockets lasted less than 15 seconds, hence wave selection by skilled surfers is important. There were a significant number of longer duration waves, with a wide variation in duration.

Figure 8. Wave pocket tracks at the Palm Beach artificial reef from drone footage. Yellow dots represent permanent buoys, vicinity of reef within red box (background from Nearmaps 02/12/2019).

Figure 8 above shows the wave pocket tracks on the reef and off the reef in the field of view of the drone. The positioning of the tracks relative to one another is still valid for analysis despite lower rectification accuracy. The concentration of wave pocket tracks in the vicinity of the reef are higher than those Figure 6. Ride durations histogram for Rainbow Bay Tracks. outside the reef. Additionally, the direction of the wave pocket tracks are more consistent in the vicinity of the reef than outside Figure 7 at the top of the right column shows the ride tracks the reef. This reveals surf amenity improvement for this region map coloured according to ride duration. The longest duration of Palm Beach due to the artificial reef. rides in red (>20s) broke consistently along the outer surf zone.

Shorter duration rides had greater spatial variation. Figure 7 also reveals an issue with the WPT software that could be The validity of the wave pocket tracks was judged qualitatively improved. The breaking waves at the start of the point break by observing the WPT software running. A video of the (circled in blue) had a small pixel footprint due to the Rainbow Bay WPT software running can be observed using this perspective of the camera and the distance to that location, link https://youtu.be/t2xzxaBteUU. Furthermore, track lengths therefore the WPT software struggled to identify that section of and durations were found to be realistic given the respective wave pocket tracks. dimensions of the surf breaks (i.e. the Palm Beach artificial reef and the Rainbow Bay ‘superbank’). GPS fitness watch data from surfers is currently being investigated as a method to quantitatively validate the WPT software. The success of the WPT approach for evaluating surf amenity [3] den Bieman, J., de Ridder, M., and van Gent, M. (2020). has led to plans for running the software on live footage. This Deep learning video analysis as measurement technique in would enable long term surf amenity assessments (> 1 day) and physical models. Coastal Engineering, 158, 103689 (DOI: provide new insights into quantifying surf amenity changes 10.1016/j.coastaleng.2020.103689) according to tides and differing wave conditions. A possible application for industry is to collect WPT data at a location of [4] Freeston, B. (2020). Real-time smart monitoring and proposed coastal infrastructure. WPT data would be collected prediction of recreational use and environmental data from during a time of year with optimal surfing conditions before ocean facing CCTV cameras - Surfzone.ai. Retrieved 9 infrastructure is built. Following the construction of the coastal April 2020, from https://www.surfzone.ai/ infrastructure, further WPT data could be collected in the next [5] Fung, J. (2017). Vision-Based Near- Wave Tracking optimal surfing conditions to quantify any changes to surf and Recognition for High Elevation and Aerial Video amenity. Such a system would benefit the industry, since it will Cameras. Los Angeles, California. From produce new knowledge of how coastal infrastructure designs https://github.com/citrusvanilla/multiplewavetracking_py affect surf amenity. This new knowledge could be used to ensure that future coastal infrastructure designs enhance rather [6] Holland, K., Holman, R., Lippmann, T., Stanley, J., and than diminish surfing breaks cherished by local communities. Plant, N. (1997). Practical use of video imagery in nearshore oceanographic field studies. IEEE Journal Of Oceanic Engineering, 22(1), 81-92 (DOI: The WPT approach provides a unique way to validate 10.1109/48.557542). numerical wave models that simulate wave breaking. There is [7] LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998). also potential to use the WPT approach to define the outer bar Gradient-based learning applied to document recognition. in large swell conditions when other measurement methods are in Proc IEEE, 86(11), 1-46 (DOI: 10.1109/5.726791) not possible. The Cyclone Eusi results in Figure 7 revealed this capability. At present, the software is also being applied to [8] Lewis, J., Hunt, S., and Evans, T. (2015). Quantification analyse physical model studies being used for artificial reef Of Surfing Amenity For Beach Value And Management. design. in Proc. Coastal Conference, Forster, Australia. Retrieved 9 April 2020, from https://www.coastalconference.com/2015/papers2015/Ja Conclusion mes%20Lewis.pdf A novel automated technique to extract surf amenity data has [9] Mortensen, S. (2015). Protecting and enhancing surf been developed. Wave pockets could be tracked from video amenity. Retrieved 8 April 2020, from footage using a combination of computer vision techniques, https://www.dhigroup.com/global/news/2015/4/protecting including the deep learning technique of convolutional neural -and-enhancing-surf-amenity networks. By rectification, wave pocket tracks were found for [10] Mucha, N. (2020). Gold Coast World Surfing Reserve | 8th the Palm Beach artificial reef and the Rainbow Bay World Surfing Reserve. Retrieved 8 April 2020, from ‘superbank’. The WPT approach proved successful for http://www.goldcoastworldsurfingreserve.com/ quantifying surf amenity at these locations in varying wave climates, including a cyclonic swell. Surf amenity metrics of a [11] Piccardi, M. (2004). Background subtraction techniques: a ride rate of 36 rides/min and an average ride length of 72m review. in Proc. IEEE International Conference on reveal new insights into why the Rainbow Bay point break is Systems, Man and Cybernetics, The Hague, Netherlands, 4 suitably named the ‘superbank’. A small clip of the WPT (DOI: 10.1109/ICSMC.2004.1400815) software functioning for Rainbow Bay can be seen with this [12] Simonyan, K., and Zisserman, A. (2015). Very Deep YouTube link (https://youtu.be/t2xzxaBteUU). Please note that Convolutional Networks for Large-Scale Image the footage of the software running has been sped up 4× due to Recognition. in Proc. International Conference on limited processing speeds on a laptop. Learning Representations 2015. San Diego, USA. Retrieved from https://arxiv.org/abs/1409.1556 Acknowledgements [13] Smith, J. (2016). Another Fine Wave Destroyed! (This Time, In Bali). Retrieved 8 April 2020, from Thank you to Heiko Loehr, James Lewis and Kyle Byblow for https://stabmag.com/news/another-perfect-wave- helping with the Palm Beach artificial reef monitoring exercise. destroyed-this-time-in-bali/ [14] SurferToday.com, n.d. The longest waves for surfing in the References world. Retrieved 13 July 2020, from https://www.surfertoday.com/surfing/the-longest-waves- [1] Bewley, A., Ge, Z., Ott, L., Ramos, F. and Upcroft, B. for-surfing-in-the-world (2016). Simple Online and Realtime Tracking. in Proc. IEEE International Conference on Image Processing [15] Weppe, S., and Shand, T. (2019). Modelling Surf Break 2016. Phoenix, USA (DOI: 10.1109/ICIP.2016.7533003) Wave Mechanics with – an application to Mangamaunu Point Break (Kaikōura, New Zealand). in [2] Black, K., and Mead, S. (2001). Design of the Gold Coast Proc. Australasian & Ports 2019 Conference, Reef for Surfing, Public Amenity and Coastal Protection: Hobart, Australia, 1218-1225. Retrieved 15 April 2020, Surfing Aspects. Journal Of Coastal Research, Special from Issue(29), 115-130. https://search.informit.com.au/fullText;dn=80120483548 2267;res=IELENG