geosciences

Article Results from the First Phase of the Seafloor Backscatter Processing Software Inter-Comparison Project

Mashkoor Malik 1,2,* , Alexandre C. G. Schimel 3 , Giuseppe Masetti 2 , Marc Roche 4, Julian Le Deunf 5 , Margaret F.J. Dolan 6, Jonathan Beaudoin 7, Jean-Marie Augustin 8, Travis Hamilton 9 and Iain Parnum 10

1 NOAA Office of Ocean Exploration and Research, Silver Spring, MD 20910, USA 2 Center for Coastal and Ocean Mapping, University of New Hampshire, Durham, NH 03824, USA 3 National Institute of Water and Atmospheric Research (NIWA), Greta Point, Wellington 6021, New Zealand 4 Federal Public Service Economy of Belgium (FPSE), 1210 Brussels, Belgium 5 Service Hydrographique et Océanographique de la Marine (SHOM), 29200 Brest, France 6 Geological Survey of Norway (NGU), 7491 Trondheim, Norway 7 QPS B.V. Fredericton, Fredericton, NB E3B 1P9, Canada 8 Service Acoustique Sous-marine et Traitement de l’Information, IFREMER, 29280 Plouzane, France 9 Teledyne CARIS, Fredericton, NB E3B 2L4, Canada 10 Centre for Marine Science and Technology, Curtin University, Perth 2605, Australia * Correspondence: [email protected]; Tel.: +1-301-734-1012



Received: 15 November 2019; Accepted: 11 December 2019; Published: 16 December 2019


Abstract: Seafloor backscatter mosaics are now routinely produced from multibeam echosounder data and used in a wide range of marine applications. However, large differences (>5 dB) can often be observed between the mosaics produced by different software packages processing the same dataset. Without transparency of the processing pipeline and the lack of consistency between software packages raises concerns about the validity of the final results. To recognize the source(s) of inconsistency between software, it is necessary to understand at which stage(s) of the data processing chain the differences become substantial. To this end, willing commercial and academic software developers were invited to generate intermediate processed backscatter results from a common dataset, for cross-comparison. The first phase of the study requested intermediate processed results consisting of two stages of the processing sequence: the one-value-per-beam level obtained after reading the raw data and the level obtained after radiometric corrections but before compensation of the angular dependence. Both of these intermediate results showed large differences between software solutions. This study explores the possible reasons for these differences and highlights the need for collaborative efforts between software developers and their users to improve the consistency and transparency of the backscatter data processing sequence.

Keywords: Acoustic backscatter processing; Multibeam Echo Sounders; seafloor mapping

Geosciences 2019, 9, 516; doi:10.3390/geosciences9120516 www.mdpi.com/journal/geosciences Geosciences 2019, 9, 516 2 of 23

1. Introduction Commercial Multibeam echosounders (MBES) were designed in the 1970’s [1] for the purpose of bathymetry data acquisition, but it is only in the past two decades that software packages generally became available to process seafloor backscatter data (henceforth, backscatter). The earliest software packages were developed and privately used by academics [2,3]. As backscatter started proving important in seafloor characterization studies [4–7], the user base expanded, and several commercial, proprietary software packages became available. Today, backscatter is collected by a broad range of users for a variety of applications, including scientists and resource managers to assess and quantify seafloor resources (sediment, geology, habitats, etc.), by hydrographic and military agencies to determine seafloor type, and by coastal zone managers for infrastructure planning. Most of these end-users rely on commercial software for data processing [8]. Due to their commercial nature, these software packages are often closed source, and very limited information is available about their proprietary data processing routines and algorithms. Processing backscatter data involves applying various and complex environmental and sensor-specific corrections to the raw level recorded by the system [9]. Those corrections have been well studied [6,7,9,10], but neither the details of each correction nor the order in which they are applied has ever been standardized [9–11]. This lack of standards for data processing and metadata, combined with the need for commercial software manufacturers to protect their intellectual property, resulted in software being developed mostly independently. Recent comparisons of the backscatter products obtained from processing the same datasets with different software highlighted differences in the results [8,12–14]. The approach adopted during these comparisons included comparing the backscatter end-results in the form of mosaics as obtained from various software packages. Having recognized that different software likely process backscatter differently, the challenge remains for the end-users to assess which, if any, of the processing methodology is most accurate. This challenge is further compounded by the lack of standards for backscatter data acquisition [10,15]. The uncertainty in backscatter results due to the hardware and environment have only recently begun to be recognized [15,16], and uncertainty standards still need to be developed in the manner that they were developed for multibeam bathymetry data over a decade ago [17]. With the goal of improving consistency among backscatter data acquisition and processing methodologies, the Backscatter Working Group (BSWG) was established in 2013 under the auspices of the GeoHab (Marine Geological and Biological Habitat Mapping) association. The BSWG compiled its guidelines in a report published in 2015 [11]. Among other recommendations, the BSWG encouraged comparative tests of processing software packages using common data sets [10]. As an outcome of this recommendation, the Backscatter Software Inter-Comparison Project (BSIP) was launched during the GeoHab 2019 conference. The long-term objective of the Backscatter Inter-comparison Project is to understand the reasons for the differences between the end-results obtained from a common dataset by various backscatter processing tools. The results in this paper represent the first phase of this project. Since comparing the end-results of the processing solutions does not allow for understanding the root causes of the discrepancies, the developers of commonly-used software were invited to provide a set of intermediate stages from the processing of a common dataset. This approach allows a comparison of intermediate corrections without requiring software developers to disclose the details of their proprietary algorithms. A recent survey of backscatter end-users [8] identified the most-commonly used backscatter data processing software packages to date: Hydrographic Information Processing System and Sonar Information Processing System (HIPS and SIPS) by Teledyne CARIS, the Fledermaus Geocoder Toolbox (FMGT) by QPS, Geocoder by the University of New Hampshire (UNH), Hypack by the Xylem, MB-System by the Monterey Bay Aquarium Research Institute, and Sonarscope by IFREMER. The developers of FMGT [18], HIPS and SIPS [19], SonarScope [20], and the MB-Process (a data processing research tool by Curtin University) agreed to participate in this study. This paper describes the results of the first phase of the project and the lessons learned. Geosciences 2019, 9, 516 3 of 23

2. Materials and Methods

2.1. Selection of Test Backscatter Data Five datasets were selected for this study (Table1), representing a range of shallow- and deep-water MBES: Kongsberg EM 2040, EM 3002, EM 710, EM 302, and Teledyne Reson SeaBat 7125. These datasets do not represent an exhaustive list of commercially available MBES but were opportunistically chosen because data from these surveys were publicly available. The test datasets were collected by different agencies (Table1). The list of individual data files used during this study is provided as AppendixA (Table A1). The common datasets used in this project are publicly available at https://bswg.github.io/bsip/.

Table 1. Datasets used during the study.

Echosounder Data Model Depth Vessel Acquisition Agency Location Weather Date (Nominal Range Software Frequency) Kwinte EM 2040 12 April RV Simon Stevin SIS FPS Economy reference area Calm 23–26 m (300 kHz) 2016 (Belgium) Carre Renard EM 3002 area, HSL Guillemot SIS SHOM Calm 13 Jan 2010 18–22 m (300 kHz) Brest Bay, France Carre Renard EM 710 14 Feb BH2 Borda SIS SHOM area, Brest Bay, Calm 18–22 m (70–100 kHz) 2013 France Johnston Atoll EM 302 NOAA Ship 17 July SIS NOAA near Hawaii, Rough ~3000 m (30 kHz) Okeanos Explorer 2017 USA Shallow Reson SeaBat survey 29 July 7125 HMSMB Owen PDS2000 Plymouth, UK Calm <10 m common 2014 (400 kHz) dataset 2015

2.2. Selection of Intermediate Processed Backscatter Levels A template backscatter data processing pipeline and nomenclature were recently proposed for adoption to assist standardizing backscatter data processing [9]. In this theoretical pipeline, the various stages of radiometric and geometric corrections are chronologically ordered, and the intermediate backscatter levels obtained between each stage are named (BL0 through to BL4), providing a sequence of intermediate results (Figure1). However, since each software package applies these corrections in different orders, most of these specific outputs cannot be produced without significantly modifying the data processing code. For the current study, after discussion and agreement with software developers, it was concluded that a phased approach would be most effective. In this first phase, only the intermediate levels that can be provided without significantly altering the code (i.e., BL0 and BL3) were considered. GeosciencesGeosciences2019 2019, 9,, 516 9, x FOR PEER REVIEW 4 of4 of24 23

Figure 1. Visual workflow of the backscatter data processing pipeline (adapted from Figure 1 in [9]), Figure 1. Visual workflow of the backscatter data processing pipeline (adapted from Figure 1 in [9]), resulting in the two common backscatter products—angular response curves and mosaic. Only the BL0 resulting in the two common backscatter products—angular response curves and mosaic. Only the and BL3 intermediate outputs were requested from software developers during the current study. BL0 and BL3 intermediate outputs were requested from software developers during the current study. 2.2.1. BL0: The Backscatter Level as Read in The Raw Files 2.2.1. BL0: The Backscatter Level as Read in The Raw Files The first stage of backscatter processing consists of reading the raw backscatter data recorded in the MBESThe first raw stage data of files. backscatter For both processing Kongsberg consists and of Teledyne reading Resonthe raw systems, backscatter the data raw recorded data format in organizesthe MBES the raw collected data files. information For both intoKongsberg several an typesd Teledyne of data Reson units, systems, known asthe datagrams, raw data format and the structureorganizes of eachthe collected datagram information type is described into several in types format of specificationsdata units, known made as publiclydatagrams, available and the by structure of each datagram type is described in format specifications made publicly available by the the manufacturers [21,22]. Not only are backscatter data typically available in different datagrams, manufacturers [21,22]. Not only are backscatter data typically available in different datagrams, but but the formats, the intermediate calculations applied, and the output resolution may have changed the formats, the intermediate calculations applied, and the output resolution may have changed over over the years. For example, in Kongsberg systems, the backscatter data are available in both the the years. For example, in Kongsberg systems, the backscatter data are available in both the “one- “one-value-per-beam” and “several-samples-per-beam” formats in two different datagrams (“Depth” value-per-beam” and “several-samples-per-beam” formats in two different datagrams (“Depth” datagramdatagram for for the the former former and and “Seabed “Seabed Image” Image” datagramdatagram for the later). later). In In November November 2005, 2005, the the “Depth” “Depth” datagramdatagram was was superseded superseded by by the the “XYZ “XYZ 88”88” datagram,datagram, and the “Seabed “Seabed Image” Image” datagram datagram was was supersededsuperseded by by the the “Seabed “Seabed Image Image 89” 89” datagram, datagram, withwith bothboth newer datagrams datagrams upgrading upgrading the the data data resolutionresolution from from 0.5 0.5 dB dB to to 0.10.1 dB [21]. [21]. Although Although Kongsberg Kongsberg released released KMALL KMALL format format in 2017 in [23], 2017 this [23 ], thisformat format has has not not been been adopted adopted widely widely by by the the proces processingsing software software packages packages and and was was not not considered considered duringduring this this study. study. For For the the Reson Reson system, system, datagrams datagrams withwith multiple samples samples per per beam beam data data are are referred referred to asto “snippets”.as “snippets”. With With the the aim aim of usingof using the the same same raw raw data, data, software software developers developers were were requested requested to startto thestart processing the processing with the with Seabed the Seabed Image Image/ Snippets / Snippets data asdata the as basis the basis to calculate to calculate BL0. BL0.

2.2.2.2.2.2. BL BL3: The3: The Backscatter Backscatter Level Level After After RadiometricRadiometric Corrections but but Before Before Compensation Compensation for for AngularAngular Dependence Dependence

Typically,Typically, several several radiometric radiometric corrections correctio arens appliedare applied to the to raw the data raw (BL data0) after(BL0 they) after are they extracted are fromextracted the file. from Schimel the file. et al.Schimel [9] suggest et al. [9] the suggest following the following three classifications: three classifications (i) Corrections : (i) Corrections for Gains appliedfor Gains during applied Reception during R (CorGR),eception (CorGR), (ii) Corrections (ii) Corrections for propagation for propagation through through Water Water column column and interactionand interaction with Seafloor with S andeafloor (CorWS) and (CorWS) and (iii) and Corrections (iii) Cor forrections Mechanical for Mechanical Properties Properties of the transducer of the (CorMP).transducer This (CorMP). is not the approachThis is not that the hasapproach been historically that has been taken historically in different taken software in different implementations; software someimplementations; software may applysome software all corrections may apply in bulk, all corrections others may incombine bulk, others several, may orcombine apply several, only partial or corrections,apply only or partial apply correctionscorrections, inor di applyfferent corrections orders. Therefore, in different this orders. study Therefore, could only this request study the could levels only request the levels before and after all radiometric corrections (BL0 and BL3, see Figure 2). BL3, before and after all radiometric corrections (BL0 and BL3, see Figure2). BL 3, the backscatter level correctedthe backscatter for radiometric level corrected corrections, for radiometric as a function correcti ofons, the as incident a function angle, of the is theincident “angular angle, response is the curve”, that is one of the two backscatter outputs commonly produced. Further corrections would Geosciences 2019, 9, x FOR PEER REVIEW 5 of 24

“angularGeosciences 2019response, 9, 516 curve”, that is one of the two backscatter outputs commonly produced. Further5 of 23 corrections would need to be applied to BL3 to obtain a backscatter mosaic, including the flattening of the backscatter angular dependence. need to be applied to BL3 to obtain a backscatter mosaic, including the flattening of the backscatter 2.3.angular Data dependence.Processing by Software Developers

2.3. DataSoftware Processing developers by Software provided Developers the results as an ASCII text file in the format requested (Table 2). One of the software packages already had some variations of the ASCII export built into their Software developers provided the results as an ASCII text file in the format requested (Table2). processing routine, while for others, the ASCII export was developed as a result of this request. One of the software packages already had some variations of the ASCII export built into their processing routine, while for others, the ASCII export was developed as a result of this request. Table 2. Requested variables to be included in the ASCII export files for this study.

Column # Table 1 2. Requested 2 variables 3 to be included 4 in the ASCII 5 export 6 files for this 7 study. 8 Column # 1 2 3 4 5 6 7 8 Seafloor Value Ping Time Seafloor Incidence Value Ping # BeamPing # Time Latitude Longitude BL BL Ping # Beam # Latitude Longitude BL0 0 Incidence 3 BL3 reported reported (Unix time)(Unix time) angle (BL3) angle (BL3) Software developers were given the option to in includeclude additional columns as desired. The details of the data processing, as implemented by software developers for this project, are outlined in the following sections.

2.3.1. CARIS CARIS SIPS SIPS Backscatter Backscatter Processing Processing Workflow Workflow The backscatter processing implementation in in CA CARISRIS SIPS is a continuation of its bathymetric processing workflowworkflow and is aimed towards creating a backscatter mosaic. SIPS supports supports data sources sources from Reson and Kongsberg systems in their three recordrecord modes: Side scan (o (onlynly applicable to Reson systems), beam average intensities, and snippets. Two Two separate backscatter processing engines are available withinwithin SIPS: SIPS: Geocoder Geocoder and and SIPS SIPS backscatter backsc processingatter processing engine. engine. As the existingAs the SIPSexisting workflow SIPS workflowdid not allow did not end-users allow end-users to extract to BL extract0 and BLBL3,0 theseand BL data3, these were data extracted were extracted by the SIPS by softwarethe SIPS softwaredevelopers developers themselves. themselves. The following The correctionsfollowing corrections and settings and were settings selected: were Processing selected: Engine: Processing SIPS; Engine:Source DataSIPS; Type: Source Time Data series; Type: Slant Time Range series; Correction, Slant Range Beam Correction, Pattern Correction; Beam Pattern Angular Correction; Variation AngularGains, Adaptive; Variation AVG Gains, size Adaptive; filter, 200 AVG samples. size filter, As of200 the samples. release As of theof the CARIS release SIPS of the 11.1.3 CARIS (released SIPS 11.1.3March (released 2019), end-users March 2019), have theend-users ability have to export the abi thelity intermediate to export the processing intermediate stages processing utilized in stages SIPS utilizedprocessing in engineSIPS processing accessed through engine ‘Advanced accessed through Settings’ and‘Advanced by designating Settings’ a ‘Correctionsand by designating Text Folder’ a ‘Correctionswhere an ASCII Text file Folder’ is stored where that an contains ASCII file results is stored of intermediate that contains processing results of intermediate stages. (Figure processing2). stages. (Figure 2).

Figure 2. CARIS SIPS mosaicmosaic creationcreation tooltool showing showing the the advanced advanced settings settings where where a foldera folder can can be setbe set for forexport export of the of textthe text file thatfile containsthat contains intermediate intermed backscatteriate backscatter processed processed levels. levels. CARIS CARIS SIPS ver.SIPS 11.1.3 ver. 11.1.3(Released (Released March March 2019). 2019).

Geosciences 2019, 9, 516 6 of 23

Geosciences 2019, 9, x FOR PEER REVIEW 6 of 24 2.3.2.Geosciences FMGT Backscatter2019, 9, x FOR PEER Processing REVIEW Workflow 6 of 24 2.3.2. FMGT Backscatter Processing Workflow The backscatter data processing in the QPS software suite is implemented in a separate toolbox: 2.3.2.The FMGT backscatter Backscatter data Processing processing Workflow in the QPS software suite is implemented in a separate toolbox: Fledermaus Geocoder Toolbox (FMGT). A notable factor in this implementation is that all the survey FledermausThe backscatter Geocoder data Toolbox processing (FMGT). in the A notableQPS software factor suite in this is implementedimplementation in a isseparate that all toolbox: the survey parametersparametersFledermaus are are readGeocoder read directly directly Toolbox from from (FMGT). the the survey surveyA notable line line factor files. files. in TheThe this processingimplementationprocessing parameters is that all used usedthe survey included included “Tx/“Tx/RxRxparameters Power Power Gain are Gain read Correction”, Correction”, directly from “Apply “Apply the survey Beam Beam Patternline Pattern files. Correction”, Correction”, The processing andand parameters “Keep data data used for for ARAincluded ARA (Angle (Angle RangeRange“Tx/Rx analysis)”. analysis)”. Power Gain Backscatter Backscatter Correction”, Range Range “Apply was was Beam selected selected Pattern basedbased Correction”, onon thethe minimum and “Keep and anddata maximum maximumfor ARA (Angle value value of of backscatterbackscatterRange analysis)”. from from “calibrated” “calibrated” Backscatter backscatter backscatterRange was with withselected beam beam based angle angle on cut cuttheo offffminimumbetween between and 00°◦ andmaximum 90°. 90◦. Export Export value ofof of BL BL0 0 andand BLbackscatter3 BLdata3 data are fromare available available “calibrated” through through backscatter export export ‘ASCIIwith ‘ASCII beam ARA ARA anglebeam beam cut detail’detail’off between (Figure 0° 3and3).). 90°. Export of BL0 and BL3 data are available through export ‘ASCII ARA beam detail’ (Figure 3).

FigureFigureFigure 3. ( a3. )3. ( The a(a) )The The QPS QPS QPS FMGT FMGTFMGT settings settings that thatthat are areare availableavailable available in in in version version version 7.8.0 7.8.07.8.0 (released (released December December December 2016). 2016). 2016). To To To enableenableenable export export export of theof of the the ARA ARA ARA Beam BeamBeam detail, detail, flag flagflag ‘Keep ‘Keep ‘Keep da data datata for forfor ARA ARAARA analysis’ analysis’ in inthe in the theprocessing processing processing parameters. parameters. parameters. (b) The(b()b )The ‘ASCII The ‘ASCII ‘ASCII ARA ARAARA Beam BeamBeam Detail’ Detail’Detail’ export exportexport is available isis availaavaila throughbleble through through the contextual the the contextual contextual display display display on the on mainonthe themain window. main window.window. 2.3.3. SonarScope Data Processing Workflow 2.3.3.SonarScope2.3.3. SonarScope SonarScope is a Data researchData ProcessingProcessing tool developed Workflow by the department of Underwater Acoustics at IFREMER. SonarScopeSonarScopeSonarScope is available is is a a research forresearch free undertool developed an academic by by the the non-commercial department department of of Underwater Underwater use license. Acoustics Acoustics This toolat IFREMER. at is IFREMER. developed in MatlabSonarScopeSonarScope as a laboratory isis availableavailable tool forfor aimed free atunder research anan academ andacadem development,icic non-commercialnon-commercial rather thanuse use license. production. license. This This SonarScopetool tool is is developed in Matlab as a laboratory tool aimed at research and development, rather than production. candeveloped handle a variety in Matlab of as MBES a laboratory formats. tool SonarScope aimed at research implemented and development, a new backscatter rather than data production. processing SonarScope can handle a variety of MBES formats. SonarScope implemented a new backscatter data methodologySonarScope concurrently can handle a withvariety this of study. MBES Aformats. detailed SonarScope analysis of implemented various processing a new backscatter stages based data on processing methodology concurrently with this study. A detailed analysis of various processing theprocessing sonar equation methodology [7] is provided concurrently in this updatedwith this workflowstudy. A anddetailed exported analysis as anof HTMLvarious summary processing file stages based on the sonar equation [7] is provided in this updated workflow and exported as an stages based on the sonar equation [7] is provided in this updated workflow and exported as an with graphicalHTML summary displays file of with the graphical various corrections.displays of the An various ASCII corrections. output file An is alsoASCII produced output file that is also contains severalHTMLproduced fields summary describing that contains file with the several correctionsgraphical fields displays describing (Figure of4). thethe corrections various corrections. (Figure 4). An ASCII output file is also produced that contains several fields describing the corrections (Figure 4).

Figure 4. The interface in the SonarScope to select the export of .csv and .html files that provide details of the various corrections applied to produce processed backscatter results. SonarScope version 20190702_R2017b (released 2 July 2019).

Geosciences 2019, 9, x FOR PEER REVIEW 7 of 24

Figure 4. The interface in the SonarScope to select the export of .csv and .html files that provide details of the various corrections applied to produce processed backscatter results. SonarScope version 20190702_R2017b (released 2 July 2019). Geosciences 2019, 9, 516 7 of 23 2.3.4. MB Process Data Processing and SONAR2MAT Data Conversion 2.3.4.MB MB Process Process is Data a proprietary Processing backscatter and SONAR2MAT data processing Data Conversion tool coded in Matlab and developed and used by Curtin University Centre for Marine Science and Technology (CMST) and Geoscience MB Process is a proprietary backscatter data processing tool coded in Matlab and developed Australia (GA) researchers to process Kongsberg (.all) files and Reson MBES (files saved as XTF) [24]. and used by Curtin University Centre for Marine Science and Technology (CMST) and Geoscience CMST-GA MB Process is available to download for free from Australia (GA) researchers to process Kongsberg (.all) files and Reson MBES (files saved as XTF) [24]. https://cmst.curtin.edu.au/products/multibeam-software/ (last accessed September 2019). As this CMST-GA MB Process is available to download for free from https://cmst.curtin.edu.au/products/ study used Reson (.s7k) files, the converter SONAR2MAT [25] was used to convert the (.s7k) data multibeam-software/ (last accessed September 2019). As this study used Reson (.s7k) files, the first to MATLAB (.mat) data files. SONAR2MAT converter supports a variety of MBES data formats converter SONAR2MAT [25] was used to convert the (.s7k) data first to MATLAB (.mat) data files. and is available to download for free from https://cmst.curtin.edu.au/products/sonar2mat-software/ SONAR2MAT converter supports a variety of MBES data formats and is available to download for free (last accessed Sept 2019). The script was used to calculate the mean for each beam (i.e., BL0) from the from https://cmst.curtin.edu.au/products/sonar2mat-software/ (last accessed Sept 2019). The script converted snippets data packet (7028) using the samples that fall within +/- 5 dB around the bottom was used to calculate the mean for each beam (i.e., BL0) from the converted snippets data packet detect echo level. The corrections applied followed Parnum and Gavrilov [26], and required other (7028) using the samples that fall within +/ 5 dB around the bottom detect echo level. The corrections converted data packets including settings− (7000), bathymetry (7027), and beam geometry (7004), to applied followed Parnum and Gavrilov [26], and required other converted data packets including produce BL3 data. Data were then exported in to the ASCII format specified in Table 2 except for settings (7000), bathymetry (7027), and beam geometry (7004), to produce BL3 data. Data were then beam depth. exported in to the ASCII format specified in Table2 except for beam depth.

3. Results The ASCII filesfiles obtainedobtained forfor eacheach softwaresoftware didifferffereded inin bothboth formatformat andand contents.contents. A summarysummary of thethe contentscontents ofof thethe ASCIIASCII filesfiles isis providedprovided inin AppendixAppendixB B (Table (Table A2 B1).). TheThe availabilityavailability of of the the results results on on pingping/beam/beam basisbasis mademade itit convenientconvenient toto comparecompare datadata fromfrom eacheach software.software. Data inter-comparison was conducted based on the ping/beam number, BL0, BL3, and incidence angle. conducted based on the ping/beam number, BL0, BL3, and incidence angle.

3.1. Flagged Invalid Beams The numbernumber of beamsbeams flaggedflagged asas “invalid”“invalid” byby eacheach softwaresoftware waswas didifferentfferent (Figure(Figure5 ).5). ForFor thethe Kongsberg EM 302 data,data, FMGTFMGT showedshowed almostalmost nono flaggedflagged beamsbeams whilewhile bothboth SIPSSIPS andand SonarScopeSonarScope showed aa large large number number of beamsof beams flagged. flagged. These These differences differences were foundwere tofound be related to be to related each software’s to each disoftware’sfferent choice different of dealing choice withof dealing soundings with withsoundings invalid with bottom invalid detection. bottom detection. Kongsberg’s Kongsberg’s “XYZ 88” datagram“XYZ 88” providesdatagram information provides information about ‘detection about information’ ‘detection thatinformation’ specifies amongthat specifies other things among whether other thethings beam whether had a valid the bottombeam had detection a valid or bottom not. The detection beams with or invalidnot. The bottom beams detections, with invalid however, bottom can bedetections, assigned however, interpolated can backscatter be assigned to interpolated provide continuous backscatter backscatter to provide across continuous all beams backscatter (see note 4 p.44across [21 all]). FMGTbeams has(see implemented note 4 p.44 the[21]). strategy FMGT to ha uses implemented the beams with the invalid strategy bottom to detection,use the beams while with SIPS andinvalid SonarScope bottom detection, utilize only while the beams SIPS and that SonarScope have a valid utilize bottom only detection the beams information that have available. a valid bottom For the purposesdetection ofinformation comparison, available. only the beams For thatthe werepurposes considered of comparison, valid by all softwareonly the packages beams werethat used.were considered valid by all software packages were used.

Figure 5. Percentage of beams flagged as “invalid” by each software of the EM 302 data. Geosciences 2019, 9, x FOR PEER REVIEW 8 of 24

Figure 5. Percentage of beams flagged as “invalid” by each software of the EM 302 data. Geosciences 2019, 9, x FOR PEER REVIEW 8 of 24 Geosciences 2019, 9, 516 8 of 23 3.2. Comparison of BL0 and BL3 Figure 5. Percentage of beams flagged as “invalid” by each software of the EM 302 data. The software provided results whose patterns were qualitatively comparable but whose relative 3.2.3.2. Comparison Comparison of of BL BL0 0andand BL BL3 3 levels were often very different (BL0 in Figure 6 and BL3 in Figure 7). The mean values of BL0 and BL3 were computedTheThe software software for providedeach provided beam results results and whoseping whose and patterns patterns showed were were that qualitatively qualitatively the differences comparable comparable between but but the whose whose tools relative relativecould be largerlevelslevels than were were 5 oftendB often (Figure very very different di8).ff Iterent was (BL (BL also0 in0 in Figureevident Figure 6 th6and andese BL differences BL3 in3 Figurein Figure 7).are 7The ).not The mean uniform mean values valuesacross of BL ofthe0 BLand swath.0 BLand3 A pair-wisewereBL3 werecomputed comparison computed for each for revealed eachbeam beam and that ping and the pingand differen showed and showedces that were thatthe more differences the di pronouncedfferences between between for the the tools the outer tools could couldbeams be comparedlargerbe larger than to than 5near-nadir dB 5 (Figure dB (Figure beams 8). 8It). was It(Figure was also also evident9). evident these these differences di fferences are arenot notuniform uniform across across the theswath. swath. A pair-wiseA pair-wise comparison comparison revealed revealed that that the the differen differencesces were were more more pronounced pronounced for for the the outer outer beams beams comparedcompared to to near-nadir near-nadir beams beams (Figure (Figure 99).).

Figure 6. Plots showing BL0 results from (a) CARIS SIPS, (b) FMGT, and (c) SonarScope for the EM Figure 6. Plots showing BL0 results from (a) CARIS SIPS, (b) FMGT, and (c) SonarScope for the EM 302Figure302 data. data. 6. Plots showing BL0 results from (a) CARIS SIPS, (b) FMGT, and (c) SonarScope for the EM 302 data.

Figure 7. Plots showing BL results from (a) CARIS SIPS, (b) FMGT, and (c) SonarScope for the EM FigureFigure 7. 7. Plots Plots showing showing BLBL33 3resultsresults from ((aa)) CARISCARIS SIPS, SIPS, ( b(b) )FMGT, FMGT, and and (c )( cSonarScope) SonarScope for for the the EM EM 302302302 data. data. data.

Geosciences 2019, 9, x FOR PEER REVIEW 9 of 24

GeosciencesGeosciences 20192019,, 99,, 516x FOR PEER REVIEW 99 ofof 2324

Figure 8. Plots showing BL0 (a, c) and BL3 (b, d) from CARIS SIPS, FMGT, and SonarScope for the EM Figure 8. Plots showing BL (a,c) and BL (b,d) from CARIS SIPS, FMGT, and SonarScope for the EM 302 data. The plots on top (0a, b) show the3 average over the entire survey line for all pings reported at 302 data. The plots on top (a,b) show the average over the entire survey line for all pings reported at Figureeach beam. 8. Plots The showing lower plots BL0 (a,c, cd)) andshow BL the3 (b, average d) from of CARIS all beams SIPS, for FMGT, each ping.and SonarScope for the EM each302 data. beam. The The plots lower on plotstop (a, (c b,d)) show show the the average average over of all the beams entire for su eachrvey ping. line for all pings reported at each beam. The lower plots (c, d) show the average of all beams for each ping.

Figure 9. Mean and standard deviation of pair-wise differences in BL0 and BL3 from different software, Figure 9. Mean and standard deviation of pair-wise differences in BL0 and BL3 from different software, as a function of beam number for the EM 302 data. (a) Mean of the BL0 differences, (b) standard as a function of beam number for the EM 302 data. (a) Mean of the BL0 differences, (b) standard deviation of the BL differences (c) Mean of the BL differences, and (d) standard deviation of the Figure 9. Mean and 0standard deviation of pair-wise differences3 in BL0 and BL3 from different software, deviation of the BL0 differences (c) Mean of the BL3 differences, and (d) standard deviation of the BL3 BL3 differences. asdifferences. a function of beam number for the EM 302 data. (a) Mean of the BL0 differences, (b) standard 3.3. Comparisondeviation of ofthe Reported BL0 differences Incidence (c) AnglesMean of the BL3 differences, and (d) standard deviation of the BL3 3.3. Comparisondifferences. of Reported Incidence Angles CARIS SIPS reported incidence angles were positive and ranged from 0◦ to 80◦ while FMGT and SonarScope3.3. ComparisonCARIS reported SIPS of reportedReported the incidence Incidenceincidence angle Angles angles with were a range positive from and80 rangedto 80 with from port 0° to swath 80° while incidence FMGT angles and − ◦ ◦ reportedSonarScope as negativereported numbersthe incidence (Figures angle 10 withand 11 a ).range Topographically-related from −80° to 80° with variations port swath in incidenceincidence CARIS SIPS reported incidence angles were positive and ranged from 0° to 80° while FMGT and

SonarScope reported the incidence angle with a range from −80° to 80° with port swath incidence

GeosciencesGeosciences2019 2019,,9 9,, 516 x FOR PEER REVIEW 1010 ofof 2324 Geosciences 2019, 9, x FOR PEER REVIEW 10 of 24 angles reported as negative numbers (Figures 10 and 11). Topographically-related variations in angles reported as negative numbers (Figures 10 and 11). Topographically-related variations in anglesincidence are clearlyangles visibleare clearly in the visible output in of the SIPS, output FMGT, of SIPS, and SonarScope, FMGT, and suggesting SonarScope, that suggesting seafloor slope that incidence angles are clearly visible in the output of SIPS, FMGT, and SonarScope, suggesting that wasseafloor considered slope was while considered computing while the seafloorcomputing incidence the seafloor angle. incidence However, angle. slight variationsHowever, inslight the seafloor slope was considered while computing the seafloor incidence angle. However, slight incidencevariations angle in the are incidence noticeable angle that are may noticeable be related that to the may di ffbeerences related in to the the cleaning differences or smoothing in the cleaning of the variations in the incidence angle are noticeable that may be related to the differences in the cleaning DTMor smoothing used to correctof the DTM for seafloor used to slope. correct for seafloor slope. or smoothing of the DTM used to correct for seafloor slope.

Figure 10. Plot showing the incidence angle reported for each beam by (a) CARIS SIPS, (b) FMGT, FigureFigure 10.10. PlotPlot showingshowing thethe incidenceincidence angle reported for each beam by (a)) CARISCARIS SIPS,SIPS, ((b)) FMGT,FMGT, and (c) SonarScope for the EM 302 data. andand ((cc)) SonarScopeSonarScope forfor thethe EM EM 302 302 data. data.

Figure 11. Comparison of empirical probability density function (pdf) of (a) BL0,(b) BL3, and (c) Figure 11. Comparison of empirical probability density function (pdf) of (a) BL0, (b) BL3, and (c) IncidenceFigure 11. angle Comparison from the of three empirical software probability packages fordensity the EM function 302 data. (pdf) of (a) BL0, (b) BL3, and (c) Incidence angle from the three software packages for the EM 302 data. Incidence angle from the three software packages for the EM 302 data. 3.4. Comparison of Corrections Applied For BL3 Processing 3.4. Comparison of Corrections Applied For BL3 Processing 3.4. ComparisonThe difference of Corrections between BL Applied3 and BLFor0 BLwas3 Processing computed for each software solution in order to obtain the totalThe correction difference factor between applied BL3 and in the BL radiometric0 was computed correction for each stage software (Figures solution 12 and in 13 order). These to obtain show The difference between BL3 and BL0 was computed for each software solution in order to obtain the total correction factor applied in the radiometric correction stage (Figures 12 and 13). These show thatthe total each correction software applies factor applied very di ffinerent the radiometric processing correction corrections stage to arrive (Figures at BL 123 .and In the 13). case These of SIPS,show thethat correction each software appears applies as an along-trackvery differen stripest processing pattern, corrections which would to implicatearrive at BL beam3. In pattern the case correction of SIPS, that each software applies very different processing corrections to arrive at BL3. In the case of SIPS, the correction appears as an along-track stripes pattern, which would implicate beam pattern (Sectionthe correction 2.3.1). Inappears the case as of an FMGT, along-track the correction stripe iss reminiscentpattern, which of the would incidence implicate angle. beam In the pattern case of correction (section 2.3.1). In the case of FMGT, the correction is reminiscent of the incidence angle. In SonarScope,correction (section the correction 2.3.1). In increases the case somewhatof FMGT, regularlythe correction away is from reminiscent nadir. Without of the incidence the knowledge angle. ofIn the case of SonarScope, the correction increases somewhat regularly away from nadir. Without the thethe intermediatecase of SonarScope, stages between the correction BL0 and increases BL3 (BL2A soandmewhat BL2B regularly—See Figure away2), thesefrom interpretationsnadir. Without are the notknowledge definitive. of the intermediate stages between BL0 and BL3 (BL2A and BL2B—See Figure 2), these knowledge of the intermediate stages between BL0 and BL3 (BL2A and BL2B—See Figure 2), these interpretations are not definitive. interpretations are not definitive.

Geosciences 2019, 9, x FOR PEER REVIEW 11 of 24

GeosciencesGeosciences2019 2019,,9 9,, 516 x FOR PEER REVIEW 1111 ofof 2324

Figure 12. Plots showing the total radiometric corrective factor (BL3−BL0) from (a) CARIS SIPS, (b) Figure 12. Plots showing the total radiometric corrective factor (BL3 BL0) from (a) CARIS SIPS, FMGT, and (c) SonarScope for the EM 302 data. − (Figureb) FMGT, 12. andPlots (c )showing SonarScope the fortotal the radiometric EM 302 data. corrective factor (BL3−BL0) from (a) CARIS SIPS, (b) FMGT, and (c) SonarScope for the EM 302 data.

Figure 13. PlotPlot showing showing the the average average corrective corrective factor factor (BL (BL3−BL0BL) per) perbeam beam over over 50 pings 50 pings (pings (pings # 100– # 3− 0 100–150)150) for the for EM the 302 EM data. 302 data. Figure 13. Plot showing the average corrective factor (BL3−BL0) per beam over 50 pings (pings # 100– 3.5. Summary 150) for the of Differences DiEMff erences302 data. Between Software forfor DiDifferentfferent SonarSonar TypesTypes 3.5. SummaryIn thethe previousprevious of Differences sections,sections, Between the didifferences Softwarefferences for betweenbetween Different SIPS,Sonar FMGT,Types andand SonarScopeSonarScope processingprocessing an EM302 data file were explored. In this section, the results of BL , BL , and incidence angle for other EM302 data file were explored. In this section, the results of BL00, BL33, and incidence angle for other In the previous sections, the differences between SIPS, FMGT, and SonarScope processing an sonar types are summarized.summarized. The results show that EM 710 (Figures 14 and 1515),), EM 3002 (Figures 1616 EM302 data file were explored. In this section, the results of BL0, BL3, and incidence angle for other and 1717),), EM EM 2040 2040 (Figures (Figures 18 and18 and19), and19), SeaBatand SeaBat 7125 (Figures 7125 (Figures 20 and 2120) alsoand present 21) also large present differences. large sonar types are summarized. The results show that EM 710 (Figures 14 and 15), EM 3002 (Figures 16 differences. and 17), EM 2040 (Figures 18 and 19), and SeaBat 7125 (Figures 20 and 21) also present large differences.

Geosciences 2019, 9, x FOR PEER REVIEW 12 of 24

GeosciencesGeosciences 20192019,, 99,, x 516 FOR PEER REVIEW 1212 of of 24 23

Figure 14. Comparison of empirical probability density function (pdf) of (a) BL0, (b) BL3 and (c) incidenceFigure 14.angleComparison from FMGT, of empiricalCARIS and probability Sonar Scope density for the function EM 710 (pdf) data. of (a) BL0,(b) BL3 and (c) Figureincidence 14. angleComparison from FMGT, of empirical CARIS and probability Sonar Scope density for thefunction EM 710 (pdf) data. of (a) BL0, (b) BL3 and (c) incidence angle from FMGT, CARIS and Sonar Scope for the EM 710 data.

Figure 15. Plots showing BL0 (a,c) and BL3 (b,d) from CARIS SIPS, FMGT, and SonarScope for the EM Figure 15. Plots showing BL0 (a, c) and BL3 (b, d) from CARIS SIPS, FMGT, and SonarScope for the Figure710 data. 15. ThePlots plots showing on top BL (a0, b(a,) showc) and the BL average3 (b, d) overfrom theCARIS entire SIPS, survey FMGT, line and for all SonarScope pings reported for the at EM 710 data. The plots on top (a, b) show the average over the entire survey line for all pings reported EMeach 710 beam. data. The The lower plots plotson top (c (,da,) b show) show the the average average of over all beams the entire for each survey ping. line SonarScope for all pings BL reported3 results at each beam. The lower plots (c, d) show the average of all beams for each ping. SonarScope BL3 atwere each clipped beam.for The the lower pings plots where (c,there d) show was the no referenceaverage of DTM all beams available. for each ping. SonarScope BL3 resultsresults were were clipped clipped for for the the pings pings where where therethere was nono referencereference DTM DTM available. available.

Geosciences 2019, 9, x FOR PEER REVIEW 13 of 24

GeosciencesGeosciences 20192019,, 99,, 516x FOR PEER REVIEW 13 of 2324

Figure 16. Comparison of empirical probability density function (pdf) of (a) BL0, (b) BL3, and (c) incidenceFigure 16.angleComparison results from ofCARIS empirical SIPS, FMGT, probability and Sonar density Scope function for the (pdf) EM of3002 (a )data BL0 file.,(b ) BL3, andFigure (c) incidence16. Comparison angle results of empirical from CARIS probability SIPS, FMGT,density and function Sonar Scope(pdf) of for ( thea) BL EM0, 3002(b) BL data3, and file. (c) incidence angle results from CARIS SIPS, FMGT, and Sonar Scope for the EM 3002 data file.

Figure 17. Plots showing BL0 (a,c) and BL3 (b,d) from CARIS SIPS, FMGT, and SonarScope for the EM Figure3002 data.17. Plots The showing plots on top BL (0a (,a,b) c show) and the BL average3 (b, d) from over theCARIS entire SIPS, survey FMGT, line for and all SonarScope pings reported for at the Figure 17. Plots showing BL0 (a, c) and BL3 (b, d) from CARIS SIPS, FMGT, and SonarScope for the EMeach 3002 beam. data. The The lower plots plots on (ctop,d) show(a, b) the show average the average of all beams over for the each enti ping.re survey SonarScope line BLfor allresults pings EM 3002 data. The plots on top (a, b) show the average over the entire survey line for all3 pings were clipped for the pings where there was no reference DTM available. reportedreported at ateach each beam. beam. The The lower lower plots plots ( (c,c, d d)) showshow the average ofof all all beam beams sfor for each each ping. ping. SonarScope SonarScope BL3 results were clipped for the pings where there was no reference DTM available. BL3 results were clipped for the pings where there was no reference DTM available.

Geosciences 2019, 9, x FOR PEER REVIEW 14 of 24

Geosciences 2019, 9, 516 14 of 23 Geosciences 2019, 9, x FOR PEER REVIEW 14 of 24

Figure 18. Comparison of the empirical probability density function (pdf) of (a) BL0, (b) BL3, and (c)

incidenceFigure angle 18. resultsComparison from CARIS of the empirical SIPS, FMGT, probability and Sonar density Scope function for the (pdf) EM of 2040 (a) BL data.0,(b ) BL3, Figure 18. Comparison of the empirical probability density function (pdf) of (a) BL0, (b) BL3, and (c) and (c) incidence angle results from CARIS SIPS, FMGT, and Sonar Scope for the EM 2040 data. incidence angle results from CARIS SIPS, FMGT, and Sonar Scope for the EM 2040 data.

0 3 Figure 19. Plots showingshowing BLBL0 ((aa,,c )c) and and BL BL3 (b (,b,d) d from) from CARIS CARIS SIPS, SIPS, FMGT FMGT and and SonarScope SonarScope for thefor EMthe EM 2040 data. The plots on top (a, b) show the average over the entire survey line for all pings Figure2040 19. data.Plots The showing plots on BL top0 (aa,,b c)) show and theBL average3 (b, d) overfrom the CARIS entire surveySIPS, lineFMGT for alland pings SonarScope reported at for the reported at each beam. The lower plots (c, d) show the average of all beams for each ping. EM 2040each data. beam. The The lowerplots plotson top (c,d )(a, show b) theshow average the ofaverage all beams over for eachthe enti ping.re survey line for all pings reported at each beam. The lower plots (c, d) show the average of all beams for each ping.

Geosciences 2019, 9, x FOR PEER REVIEW 15 of 24 Geosciences 2019, 9, 516 15 of 23 Geosciences 2019, 9, x FOR PEER REVIEW 15 of 24

0 3 Figure 20. ComparisonComparison of ofthe the empirical empirical probability probability density density function function (pdf) (pdf)of (a) ofBL (a, )(b BL) BL0,(, band) BL (c3), andincidenceFigure (c) incidence20. angle Comparison results angle resultsfromof the FMGT empirical from, FMGT, CARIS probability CARIS, ,and Curtin anddensity Curtin University function University MB (pdf) Process MBof (a Process) BLfor0 ,the (b for) SeaBatBL the3, SeaBatand 7125 (c) 7125data.incidence data. angle results from FMGT, CARIS ,and Curtin University MB Process for the SeaBat 7125 data.

Figure 21. Plots showing BL0 (a,c) and BL3 (b,d) from CARIS SIPS, FMGT, and Curtin University MB Figure 21. Plots showing BL0 (a, c) and BL3 (b, d) from CARIS SIPS, FMGT, and Curtin University MB Process for the SeaBat 7125 data. The plots on top (a,b) show the average over the entire survey line for ProcessFigure 21. for Plots the SeaBat showing 7125 BL data.0 (a, c The) and plots BL3 (onb, dtop) from (a, b CARIS) show SIPS,the average FMGT, overand Curtinthe entire University survey lineMB all pings reported at each beam. The lower plots (c,d) show the average of all beams for each ping. forProcess all pings for the reported SeaBat at 7125 each data. beam. The The plots lower on plotstop ( a,(c, b d)) show show the the averageaverage overof all the beams entire for survey each ping. line for all pings reported at each beam. The lower plots (c, d) show the average of all beams for each ping. The pairwise differences for both BL0 and BL3 results differ considerably among processing The pairwise differences for both BL0 and BL3 results differ considerably among processing solutions for the example files from all sonar models. The mean differences (except for the SeaBat 7125 solutionsThe pairwisefor the example differences files forfrom both all sonarBL0 and mode BL3ls. results The mean differ differences considerably (except among for theprocessing SeaBat data file) ranged from ~2 dB to ~10 dB with standard deviations of up to 8 dB. For the Seabat 7125 data 7125solutions data forfile) the ranged example from files ~2 dBfrom to ~10all sonar dB with mode standardls. The deviationsmean differences of up to (except 8 dB. Forfor the SeabatSeaBat file, the mean of the difference between FMGT and MB Process results was <1 dB, but the difference 7125 datadata file,file) the ranged mean from of the ~2 difference dB to ~10 betweendB with FMGTstandard and deviations MB Process of resultsup to 8 wasdB.

3.6. Relative Importance of Difference in Raw Data Reading (BL0) Compared to Radiometric Correction (BL3- BL3.6.0) Relative Importance of Difference in Raw Data Reading (BL0) Compared to Radiometric Correction (BL3- BL0)

Geosciences 2019, 9, 516 16 of 23

3.6. Relative Importance of Difference in Raw Data Reading (BL0) Compared to Radiometric Correction (BL3-BL0) The results presented above showed that software solutions provided levels that differ both at the initial raw data reading stage (BL0) and after the radiometric correction have been applied (BL3). A few possible reasons for differences in BL0 will be discussed in Section 4.2. The question arises as to whether the difference in the end results (BL3) is due mostly to the difference in data reading (BL ) or in radiometric corrections applied (BL BL ). To assess which of 0 3− 0 the two sources of differences contributes the most to the difference in end results, the absolute value of the ratio between the difference in radiometric correction and the difference in raw data reading (Equation (1), considering two software solutions A and B) was calculated: h i h i BLA BLA BLB BLB 3 0 3 0 γ = − h − i − . (1) BLA BLB 0 − 0 Small γ values (tending towards zero) indicate that the difference in end results is mostly due to differences in raw data reading. Conversely, large γ values (tending towards infinity) indicate that the difference in end results is mostly due to the difference in radiometric corrections. A γ value of 1 indicates that both sources of difference contribute equally to the difference in end results. In Table3, we report for each dataset and each pair of software that could be, thus, compared, the median γ value and its interquartile range. In the case of Kongsberg systems, the median γ value is almost always less than 1, indicating that for most datasets and most software comparisons, the difference in data reading has more influence on the difference in the end results than radiometric corrections. Only the SIPS/FMGT comparison on the EM710 dataset shows a median γ larger than 1, indicating, in this case, that difference in radiometric corrections have a (slightly) larger influence. The same analysis applied to the SeaBat 7125 data produced many different results. MB Process and FMGT read the SeaBat 7125 raw data very similarly, leading to a very large median γ value that confirms that the difference in end results is almost entirely due to the difference in radiometric corrections. However, SIPS reads the data differently than the two other software, and the difference in end result seems to be mostly due to this difference in data reading when compared with FMGT (median γ of 0.71), but mostly due to difference in radiometric processing when compared with MB Process (median γ of 2.27). Note that the interquartile ranges are often quite large, indicating that to obtain the future goal of consistent end results both of the two sources of difference will need to be addressed but the most important source of the differences in the end-results (except for MB Process and FMGT’s approach to processing SeaBat 7125 data), currently is simply due to the original choice of the starting value (BL0).

Table 3. Median of γ and interquartile range Q3–Q1 (in parentheses), computed for various pairs of software processing the same dataset.

FMGT/ SonarScope/ MB MB SIPS/FMGT SonarScope SIPS Process/FMGT Process/SIPS EM302 0.66 (1.23) 0.57 (1.12) 0.39 (0.87) - - EM710 1.19 (2.28) 0.63 (1.02) 0.80 (1.68) - - EM3002 0.63 (1.43) 0.04 (0.13) 0.94 (2.1) - - EM2040 0.3 (0.73) 0.04 (0.11) 0.31 (0.76) - - SeaBat7125 0.71 (0.04) - - 55.98 (148.9) 2.27(0.16) Geosciences 2019, 9, 516 17 of 23

4. Discussion MBES backscatter data are increasingly used to provide information about the nature of the seabed, in resource management projects, to assess the potential environmental impacts of human activities on the seabed, and for monitoring and managing marine habitats [8,10]. In many of such projects, it is often required to merge backscatter data from several sources, which often use different data processing and analysis software packages (e.g., EU national monitoring programs in relation with the EU Marine Strategic Framework Directive [27], Seamap Australia—a national seafloor habitat classification scheme [28] and Marine AREA database for Norwegian waters: MAREANO [29]). In this context, the quality control of the data and final products have important regulatory and legal implications. It is incumbent upon government agencies and scientific institutions to recognize that software packages used to process the raw data into useable products also impact the interpretation of these products and thus should be accredited for quality level [30]. There is a lot to gain for all the parties involved, to develop quality control approaches for the algorithms, and reach a level of standardization sufficient to merge the products from different software packages. The comparative analysis of intermediate software results, as developed in this paper, is a first step in the direction of processing standardization. We acknowledge that this study has been limited to analysis of data from a selection of commonly used MBES, and only a few backscatter processing packages. Future work of the BSIP/BSWG will incorporate data from a wider range of MBES, will facilitate the analysis of data logged in a variety of formats including Kongsberg’s new KMALL format for which software manufacturers are only starting to develop stable solutions at this time and hope to engage more backscatter processing software packages.

4.1. Importance of Accurate, Transparent, and Consistent Software Solutions in Science The software solutions provide critical functionality to support data acquisition, processing, analysis, and visualization for nearly all the scientific disciplines, including benthic studies [31,32]. The choice of processing software is a critical decision. Software solutions may be chosen based on several criteria including accuracy, transparency, consistency, ease of use, price, fit for the specific processing needs, computing resources requirements, and compatibility with other tools being used by an organization and project partners. The determination of accuracy, transparency, and consistency of software solutions requires detailed testing that is beyond the scope of a single study such as the present one [33]. However, the unexplained differences between the backscatter results processed by the tools that are widely used by scientists is a concern shared by end-users of backscatter data, agencies funding data acquisition and processing; and software solution providers [34,35]. Hook and Kelley [33] identified a lack of quality control and means of comparing software output to expected and correct results as a critical challenge to assess a software package. The current study compares some of the intermediate processing results of non-transparent processing chains in an attempt to highlight which parts of these processing chains differ the most. Only four software solution providers participated in this study, but it is expected that future efforts will include other software packages. One very positive development has been that through this study and the cooperation of the software manufacturers, each of the three commercial software packages that were studied (QPS, CARIS, and SonarScope) now have the functionality to export intermediate results that will enable future end-users to be able to assess the processing chain themselves.

4.2. Why Do Different Approaches to Reading Raw Data Exist and Which One is Correct? The results indicate that the raw data in the form of seabed image/snippets is read differently by various software to create what is termed as ‘beam averaged backscatter’ and was referred during this study as BL0. The impetus to compute beam averaged backscatter value stems from the need to reduce the statistical uncertainty of seafloor backscatter [6,15]. Through the commercial development of MBES, different approaches have been taken for the collection and provision of backscatter data, Geosciences 2019, 9, 516 18 of 23 and these differences may offer some explanation for the discrepancies found. For instance, historically, the approaches taken to compute a single representative value per beam from recorded snippets differed based on:

1. Choice of central tendency, i.e., mean, median, or some other measure; 2. Choice of how the backscatter samples are selected to compute a measure of central tendency, e.g., use all the samples within a beam vs. using some threshold around the bottom detect to obtain a subset of samples vs. some other variations to choose samples; 3. Choice of the calculation method. MBES samples provided by sonar manufacturers represent backscatter strength in dB. These samples can be directly used to compute their central tendency, or they can be first converted into linear domain before calculating averages, and then the computed average converted back to a logarithmic scale.

For the purposes of this study, the software vendors were not required to disclose the details of their processing steps. The discussions over the course of this study with software developers indicated that this information might not be readily disclosed as the software developers are limited by non-disclosure agreements with hardware manufacturers from openly disclosing the internal processing of hardware. The information about the computation of BL0 for various software that could be obtained during the study is summarized in Table4. The impact of these various choices will result in differences in the reported results depending on the specific data set and range of the recorded backscatter values. These differences are the most likely reasons the BL0 values reported for various tools were different. A recommendation to use one or the other approach based on rigorous analysis is beyond the scope of the current study, but further investigation into this issue should be prioritized in close collaboration with hardware manufacturers as well as software developers.

Table 4. Disclosed information by software packages to compute BL0. The information is produced here with the permission from the software packages.

Curtin Univ. CARIS SIPS FMGT Sonar Scope MB Process Kongsberg systems: Reson Systems: Use all of the full-time series Use the snippet sample Reson and Kongsberg samples recorded within a associated with the bottom systems: beam to compute an average detection. Divide the stored Identify all the samples value. By default, samples are value by 65536 (to convert that fall within 5 dB Reson systems: ± first converted to energy from 2 byte to floating point) around the bottom detect Calculate the mean of (linear domain) before before applying the 20log10. echo level and compute samples that fall within computing average and an average of these 5 dB around the bottom returned in dB. The new ± Kongsberg systems: qualifying samples using detect echo level. release (2019) provides the Fit a curve to snippet samples the amplitude values in option to compute this value using a moving window (size dB as reported in the in dB, energy, median, or 11 samples). Report the max datagram. amplitude. The new default value of the fit curve. method is now in amplitude.

4.3. Need for Adoption of Metadata Standards While MBES bathymetry data has long been subject to standards of accuracy [17], quantified uncertainties [36], and validated processing sequences, MBES backscatter mosaics are often considered qualitative products. The long-standing obstacle here is the complexity of the logistics of calibrating MBES backscatter data, and this situation has delayed the development and applications of the usage of this data-type [11]. The shift from a qualitative treatment of seafloor backscatter products such as backscatter mosaics to that of repeatable quantitative measurements may not be complete until feasible calibration procedures are developed, agreed upon, and routinely implemented. In the meantime, however, additional tools need to be made available to end-users to analyze the impact of their choices of parameters and algorithms in their backscatter data processing routine. Compilation of results from Geosciences 2019, 9, 516 19 of 23 multisource multibeam echo sounders (e.g., [37]) and for multi-frequency systems (e.g., [38]) indicate the growing demand for consistent processing methodology. The ability to identify the reason(s) of differences in the processed results is, therefore, an essential component to understand if the differences in repeat or adjacent surveys are due to the seafloor changes, acquisition differences, or merely due to post-processing differences. This study reinforces the need for comprehensive metadata to accompany processed results [11]. In the absence of estimates of the accuracy or uncertainty of a data product (as is the case with MBES bathymetry), metadata provide the backscatter users with the minimum sufficient information to replicate the final product if necessary, and correct issues that may be discovered. Metadata also has an essential role in providing information to end-users (e.g., a geologist interpreting seabed sediment type) who may not be actively involved in, or have an in-depth knowledge of, backscatter processing yet whose perception of the data is influenced by the data provenance from acquisition through processing. The development and implementation of a standard metadata format for backscatter data products by the community (involving sonar manufacturers, software developers, and the users of this hardware and software across industry, academia, and government organizations) should, therefore, be a priority.

4.4. Collaboration between Backscatter Stakeholders Our study reveals that much of the difference in backscatter results can be linked to a lack of communication between end-users, sonar manufacturers, and software developers. The current state where the results from different software packages are not reliable is a result of the independent evolution of methodology without considering end-users needs to be able to achieve consistent backscatter results irrespective of which software tool they use. This study has been conducted under the umbrella of the GEOHAB Backscatter Working Group (BSWG), which has been organized to provide a platform for academic, commercial, and government entities to collaborate to address challenges in backscatter processing. Although the calls for such collaborations have been numerous [39–41], collaborations focused on a specific data type (MBES backscatter) are rare. The lessons learned from the collaboration, which made the current study possible include:

The collaboration works well if all the stakeholders can communicate. BSWG provided an effective • communication platform that facilitated the discussions. Different entities may have different end goals in mind while collaborating on such projects. • The framework of a successful collaboration depends on finding common goals. For example, in this case, the common goal was an improvement in the consistency of backscatter results, which motivated all stakeholders to agree to work closely. For other similar efforts, e.g., efforts to standardize seafloor backscatter segmentation and characterization, the identification of a common goal may not be very clear due to multiple divergent needs of end-users or desire to protect commercial interests. Challenges of navigating proprietary restrictions both for multibeam echosounder software • and hardware manufacturers are very real and may hamper successful collaboration between stakeholders [42].

5. Conclusions The applications of seafloor backscatter data are expanding. To support such an expansion, there is a critical need for an increased output consistency among various software packages or, at least, a clear explanation for differences among software solutions. Hence, the progress made in this study was due to the cooperation of the software providers. For instance, during this study, significant differences were encountered between the outputs of several popular backscatter software packages, but through collaboration, a better understanding of where these differences were introduced in the processing pipeline was achieved. This study adapted the standard pipeline and nomenclature proposed by Schimel et al. [9] to produce the results from backscatter intermediate processing stages. However, Geosciences 2019, 9, 516 20 of 23 the data from these intermediate processing stages are currently not produced consistently by all the software developers. Therefore, the active participation of software developers will be critical to make appropriate changes in the software packages to enable the export of results from intermediate processing stages while expanding this approach to other software packages. Two intermediate processing levels were assessed during this study: the level read from the raw data files (BL0) and the level after radiometric corrections but before the removal of angular dependence (BL3). Software developers applied the required changes in their processing methodologies and provided data in Beam – Ping configuration with BL0 and BL3 reported for each beam along with incidence angle. Both BL0 and BL3 showed differences as high as >10 dB between the software packages. The differences in BL0 indicate that closed source software has adopted different approaches to read and reduce the raw data. These differences suggest this stage as one of the major causes of the observed differences in the final products. The observed discrepancy between BL0 calls for standardization of processing at this early stage of backscatter processing as well as more transparency from software providers to describe their computation choices. Critical choices of BL0 computation that should be targeted for developing a standard includes: (a) the choice of computation method for central tendency, i.e., mean or median; (b) the selection of samples used to compute BL0, and; (c) the choice of linear or logarithmic domain for computation. This study has shown the applicability and usefulness of the availability of intermediate processing stages for the inter-comparison of proprietary software without requiring the software vendors to disclose their proprietary algorithms. Hence, although the scope of this study has been limited to understand the differences between the specific software package results, it adds weight to the argument of why it is critical for various sonar manufacturers, commercial, and academic software developers, and end-users from diverse domains to work together to develop methods that can improve the consistency of backscatter processing. It is evident from this, and several previous studies that accepted protocols to test and compare software processing results are desired. This study offers a first step towards the implementation of previously proposed processing protocols. As software developers start to offer the results from other intermediate processing stages, it can be envisioned that data test benches can be developed to aid end-users in evaluating various processing options currently available in processing tools [43].

Author Contributions: Conceptualization—M.M., A.C.G.S., G.M., M.R. and M.F.J.D.; Data curation—M.M., G.M, J.L.D., J.B., J.-M.A., T.H. and I.P.; Formal analysis—M.M., A.C.G.S. and G.M.; Methodology—M.M., A.C.G.S., G.M., M.R., J.L.D. and M.F.J.D.; Project administration—M.M., A.C.G.S., G.M., M.R., J.L.D. and M.F.J.D.; Software—J.B., J.-M.A., T.H. and I.P.; Supervision—M.M.; Writing—original draft—M.M.; Writing—review & editing—M.M., A.C.G.S., G.M., M.R., J.L.D., M.F.J.D., J.B., J.-M.A., T.H. and I.P. Funding: This research received no external funding Acknowledgments: Backscatter Working Group (BSWG) co-chair Xavier Lurton and Geoffrey Lamarche facilitated early discussions to setup the Backscatter Software Inter-comparison Project under the auspices of BSWG. We thank two anonymous reviewers whose comments greatly improved this manuscript. Conflicts of Interest: The following authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper: Mashkoor Malik, Alexandre C. G. Schimel, Giuseppe Masetti, Marc Roche, Julian Le Deunf, Margaret F.J. Dolan. The following authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Travis Hamilton is employed at Teledyne CARIS which produces HIPS and SIPS, a software package used in this inter-comparison study; Jonathan Beaudoin is employed at QPS which produces FMGT, a software package used in this inter-comparison study; Jean-Marie Augustin is employed at IFREMER which produces SonarScope, a software package used in this inter-comparison study; and Iain Parnum is employed at Curtin University which produces MB Process, a software package used in this inter-comparison study. Disclaimer: The scientific results and conclusions, as well as any views or opinions expressed herein are those of the author(s) and do not necessarily reflect the views of the National Ocean and Atmospheric Administration (NOAA); the Department of Commerce; the National Institute of Water and Atmospheric Research (NIWA), the University of New Hampshire, the Federal Public Service Economy of Belgium (FPSE), the Service Hydrographique et Océanographique de la Marine (SHOM), the Geological Survey of Norway (NGU), QPS, CARIS Teledyne, IFREMER or Curtin University. Mention of a commercial company or product does not constitute an endorsement by host agencies of author(s). The comparison results have been developed in close collaboration with CARIS, QPS, Geosciences 2019, 9, 516 21 of 23

IFREMER and Curtin University and published with their permission. These comparison results were focused only on developing intermediate backscatter processing stages and did not address software usability, bathymetry processing and other features that prospective users of these software packages may also want to consider.

Appendix A

Table A1. List of the data files used during this study.

Sonar Type Data File EM 302 0213_20170717_112534_EX1706_MB.all EM 710 0002_20130214_091514_borda.all EM 3002 0009_20100113_121654_guillemot.all EM 2040 0005_20160412_104116_SimonStevin.all SeaBat 7125 20140729_082527_SMB Owen.s7k

Appendix B

Table A2. Number of columns, and relevant column names in the ASCII files exported from each software.

Software SonarScope FMGT CARIS SIPS MB Process # columns 31 12 17 11 Time stamp (Unix Time) Time UTC Ping Time Timestamp Ping Time Ping # Ping Ping Number Ping Ping Number Beam # Beam Beam Number Beam Beam Number Beam location (Lat/Long) Latitude/Longitude Latitude/Longitude Longitude/Latitude Longitude/Latitude Beam location (E/N) GeoX/GeoY Easting/Northing Easting/Northing Easting/Northing Beam depth BathyRT Depth Depth (Not Provided) Incidence angle IncidenceAngles True Angle IncidentAngle Incidence Angle BS as read from data files ReflecSSc Backscatter Value BL0 Backscatter value (BL0) BS processed angular Corrected Corr Backscatter SSc_Step1 BL3 response (BL3) Backscatter Value Value All except Data processed All except EM 3002 All Only SeaBat 7125 SeaBat 7125

References

1. de Moustier, C. State of the art in swath bathymetry survey systems. Int. Hydrogr. Rev. 1988, 65, 25–54. 2. Hughes Clarke, J.E.; Mayer, L.A.; Wells, D.E. Shallow-water imaging multibeam sonars: A new tool for investigating seafloor processes in the coastal zone and on the continental shelf. Mar. Geophys. Res. 1996, 18, 607–629. [CrossRef] 3. de Moustier, C. Beyond bathymetry: Mapping acoustic backscattering from the deep seafloor with Sea Beam. J. Acoust. Soc. Am. 1986, 79, 316–331. [CrossRef] 4. Mayer, L.A. Frontiers in Seafloor Mapping and Visualization. Mar. Geophys. Res. 2006, 27, 7–17. [CrossRef] 5. Anderson, J.T.; Holliday, D.V.; Kloser, R.; Reid, D.; Simard, Y.; Brown, C.J.; Chapman, R.; Coggan, R.; Kieser, R.; Michaels, W.L.; et al. ICES Acoustic Seabed Classification of Marine Physical and Biological Landscapes; ICES Cooperative Research Report No. 286; International Council for the Exploration of the Sea Conseil, International pour l’Exploration de la Mer: Copenhagen, Denmark, August 2007. 6. Jackson, D.R.; Richardson, M.D. High-Frequency Seafloor Acoustics; Springer Science & Business Media: New York, UK, 2007; ISBN 978-0-387-34154-5. 7. Lurton, X. An Introduction to Underwater Acoustics: Principles and Applications, 2nd ed.; Springer: Berlin/Heidelberg, Germany, 2010; ISBN 978-3-540-78480-7. Geosciences 2019, 9, 516 22 of 23

8. Lucieer, V.; Roche, M.; Degrendele, K.; Malik, M.; Dolan, M.; Lamarche, G. User expectations for multibeam echo sounders backscatter strength data-looking back into the future. Mar. Geophys. Res. 2018, 39, 23–40. [CrossRef] 9. Schimel, A.C.G.; Beaudoin, J.; Parnum, I.M.; Le Bas, T.; Schmidt, V.; Keith, G.; Ierodiaconou, D. Multibeam sonar backscatter data processing. Mar. Geophys. Res. 2018, 39, 121–137. [CrossRef] 10. Lurton, X.; Lamarche, G.; Brown, C.; Lucieer, V.; Rice, G.; Schimel, A.C.G.; Weber, T.C. Backscatter Measurements by Seafloor-Mapping Sonars—Guidelines and Recommendations. Geohab report. May 2015. Available online: http: //geohab.org/wp-content/uploads/2013/02/BWSG-REPORT-MAY2015.pdf (accessed on 10 November 2019). 11. Lamarche, G.; Lurton, X. Recommendations for improved and coherent acquisition and processing of backscatter data from seafloor-mapping sonars. Mar. Geophys. Res. 2018, 39, 5–22. [CrossRef] 12. Dufek, T. Backscatter Analysis of Multibeam Sonar Data in the Area of the Valdivia Fracture Zone using Geocoder in CARIS HIPS&SIPS and IVS3D Fledermaus. Master’s Thesis, Hafencity University Hamburg, Hamburg, Germany, 2012. 13. Roche, M.; Degrendele, K.; Mol, L.D. Constraints and limitations of MBES Backscatter Strength (BS) measurements for monitoring the seabed. Surveyor and geologist point of view. In Proceedings of the GeoHab (Maine Geological and Biological Habitat Mapping), Rome, Italy, 6–10 May 2013; p. 21. 14. Roche, M.; Degrendele, K.; Vrignaud, C.; Loyer, S.; Le Bas, T.; Augustin, J.-M.; Lurton, X. Control of the repeatability of high frequency multibeam echosounder backscatter by using natural reference areas. Mar. Geophys. Res. 2018, 39, 89–104. [CrossRef] 15. Malik, M.; Lurton, X.; Mayer, L. A framework to quantify uncertainties of seafloor backscatter from swath mapping echosounders. Mar. Geophys. Res. 2018, 39, 151–168. [CrossRef] 16. Malik, M. Sources and impacts of bottom slope uncertainty on estimation of seafloor backscatter from swath sonars. Geosciences 2019, 9, 183. [CrossRef] 17. Hare, R.; Godin, A.; Mayer, L.A. Accuracy Estimation of Canadian Swath (Multibeam) and Sweep (Multitransducer) Sounding Systems; Canadian Hydrographic Service Internal Report; Canadian Hydrographic Service: Ottawa, ON, Canada, 1995. 18. FMGT—Fledermaus Geocoder Toolbox 7.8.0; Quality Positioning Services BV (QPS): Zeist, The Netherlands, 2018. 19. Teledyne Computer Aided Resource Information System (CARIS) HIPS and SIPS; Teledyne CARIS Inc.: Fredericton, NB, Canada, 2019. 20. Augustin, J. SonarScope®software on-line presentation. Available online: http://flotte.ifremer.fr/fleet/ Presentation-of-the-fleet/Logiciels-embarques/SonarScope (accessed on 6 June 2019). 21. Kongsberg Inc. Kongsberg Multibeam Echo Sounder EM Datagram Formats. Rev. W. Available online: https://www.kongsberg.com/globalassets/maritime/km-products/product-documents/160692_em_ datagram_formats.pdf (accessed on 15 November 2019). 22. Teledyne Reson Teledyne Reson Data Format Definition Document. 7k Data Format, Volume 1, Version 3.01. Available online: https://www.teledyne-pds.com/wp-content/uploads/2016/11/DATA-FORMAT- DEFINITION-DOCUMENT-7k-Data-Format-Volume-I-Version-3.01.pdf (accessed on 20 June 2019). 23. Kongsberg Inc. KMALL Datagram description Rev:F. Available online: https://www.kongsberg.com/ maritime/support/document-and-downloads/software-downloads/ (accessed on 15 June 2019). 24. Gavrilov, A.; Duncan, A.; McCauley, R.; Parnum, I.; Penrose, J.; Siwabessy, P.; Woods, A.J.; Tseng, Y.-T. Characterization of the seafloor in Australia’s coastal zone using acoustic techniques. In Proceedings of the 1st International Conference on Underwater Acoustic Measurements, Heraklion, Greece, 28 June–1 July 2005; pp. 1075–1080. 25. Parnum, I.M.; Tyler, E.; Miles, P. Software for rapid visualisation and analysis of multibeam echosounder water column data. In Proceedings of the ICES Symposium on Marine Ecosystem Acoustics, Nantes, France, 25–28 May 2015. 26. Parnum, I.M.; Gavrilov, A.N. High-frequency multibeam echo-sounder measurements of seafloor backscatter in shallow water: Part 1—Data acquisition and processing. Underw. Technol. 2011, 30, 3–12. [CrossRef] 27. European Parliament; Council of the European Union. Directive 2008/56/EC of the European Parliament and of the Council of 17 June 2008 Establishing a Framework for Community Action in the Field of Marine Environmental Policy (Marine Strategy Framework Directive) (Text with EEA Relevance); 2008. Available online: http://www.legislation.gov.uk/eudr/2008/56/2019-10-31# (accessed on 10 November 2019). Geosciences 2019, 9, 516 23 of 23

28. Lucieer, V.; Walsh, P.; Flukes, E.; Butler, C.; Proctor, R.; Johnson, C. Seamap Australia—A National Seafloor Habitat Classification Scheme; Institute for Marine and Antarctic Studies (IMAS), University of Tasmania: Hobart TAS, Australia, 2017. 29. Buhl-Mortensen, L.; Buhl-Mortensen, P.; Dolan, M.F.J.; Holte, B. The MAREANO programme—A full coverage mapping of the Norwegian off-shore benthic environment and fauna. Mar. Biol. Res. 2015, 11, 4–17. [CrossRef] 30. Manzella, G.; Griffa, A.; de la Villéon, L.P. Report on Data Management Best Practice and Generic Data and Metadata Models. V.2.1 [Deliverable 5.9]; Ifremer for JERICO-NEXT Project: Issy-les-Moulineaux, France, 2017. 31. Idaszak, R.; Tarboton, D.G.; Yi, H.; Christopherson, L.; Stealey,M.J.; Miles, B.; Dash, P.; Couch, A.; Spealman, C.; Horsburgh, J.S.; et al. HydroShare—A Case Study of the Application of Modern Software Engineering to a Large Distributed Federally-Funded Scientific Software Development Project. In Software Engineering for Science; Taylor & Francis CRC Press: Boca Raton, FL, USA, 2017; pp. 253–270. 32. Hannay, J.E.; MacLeod, C.; Singer, J.; Langtangen, H.P.; Pfahl, D.; Wilson, G. How do scientists develop and use scientific software? In Proceedings of the 2009 ICSE Workshop on Software Engineering for Computational Science and Engineering, Washington, DC, USA, 23 May 2009; pp. 1–8. 33. Hook, D.; Kelly, D. Testing for trustworthiness in scientific software. In Proceedings of the 2009 ICSE Workshop on Software Engineering for Computational Science and Engineering, Washington, DC, USA, 23 May 2009; pp. 59–64. 34. Howison, J.; Deelman, E.; McLennan, M.J.; Ferreira da Silva, R.; Herbsleb, J.D. Understanding the scientific software ecosystem and its impact: Current and future measures. Res. Eval. 2015, 24, 454–470. [CrossRef] 35. Carver, J.; Hong, N.P.C.; Thiruvathukal, G.K. Software Engineering for Science; Taylor & Francis CRC Press: Boca Raton, FL, USA, 2017; ISBN 978-1-4987-4385-3. 36. Calder, B.R.; Mayer, L.A. Automatic processing of high-rate, high-density multibeam echosounder data. Geochem. Geophys. Geosyst. 2003, 4, 1048. [CrossRef] 37. Hughes Clarke, J.E.; Iwanowska, K.K.; Parrott, R.; Duffy, G.; Lamplugh, M.; Griffin, J. Inter-calibrating multi-source, multi-platform backscatter data sets to assist in compiling regional sediment type maps: Bay of Fundy. In Proceedings of the Canadian Hydrographic and National Surveyors’ Conference, Victoria Conference Centre, Victoria, BC, Canada, 5–8 May 2008; p. 23. 38. Feldens, P.; Schulze, I.; Papenmeier, S.; Schönke, M.; Schneider von Deimling, J. Improved interpretation of marine sedimentary environments using multi-frequency multibeam backscatter data. Geosciences 2018, 8, 214. [CrossRef] 39. Pendleton, L.H.; Beyer, H.; Estradivari; Grose, S.O.; Hoegh-Guldberg, O.; Karcher, D.B.; Kennedy, E.; Llewellyn, L.; Nys, C.; Shapiro, A.; et al. Disrupting data sharing for a healthier ocean. ICES J. Mar. Sci. 2019, 76, 1415–1423. [CrossRef] 40. Franken, S.; Kolvenbach, S.; Prinz, W.; Alvertis, I.; Koussouris, S. CloudTeams: Bridging the gap between developers and customers during software development processes. Procedia Comput. Sci. 2015, 68, 188–195. [CrossRef] 41. Mackenzie, B.; Celliers, L.; de Freitas Assad, L.P.; Heymans, J.J.; Rome, N.; Thomas, J.; Anderson, C.; Behrens, J.; Calverley, M.; Desai, K.; et al. The role of stakeholders in creating societal value from coastal and ocean observations. Front. Mar. Sci. 2019, 6, 137. [CrossRef] 42. Legendre, P. Reply to the comment by Preston and Kirlin on “Acoustic seabed classification: Improved statistical method”. Can. J. Fish. Aquat. Sci. 2003, 60, 1301–1305. [CrossRef] 43. Masetti, G.; Augustin, J.M.; Malik, M.; Poncelet, C.; Lurton, X.; Mayer, L.A.; Rice, G.; Smith, M. The Open Backscatter Toolchain (OpenBST) project: Towards an open-source and metadata-rich modular implementation. In Proceedings of the US Hydro, Biloxi, MS, USA, 19–21 March 2019. [CrossRef]

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).