Journal of Marine Science and Engineering

Article Assessing the Accuracy of Underwater Photogrammetry for Archaeology: A Comparison of Structure from Motion Photogrammetry and Real Time Kinematic Survey at the East Key Construction Wreck

Anne E. Wright 1,* , David L. Conlin 1 and Steven M. Shope 2

1 National Park Service Submerged Resources Center, Lakewood, CO 80228, USA; [email protected] 2 Sandia Research Corporation, Mesa, AZ 85207, USA; [email protected] * Correspondence: [email protected]



Received: 17 September 2020; Accepted: 19 October 2020; Published: 28 October 2020


Abstract: The National Park Service (NPS) Submerged Resources Center (SRC) documented the East Key Construction Wreck in Dry Tortugas National Park using Structure from Motion photogrammetry, traditional archaeological hand mapping, and real time kinematic GPS (Global Positioning System) survey to test the accuracy of and establish a baseline “worst case scenario” for 3D models created with NPS SRC’s tri-camera photogrammetry system, SeaArray. The data sets were compared using statistical analysis to determine accuracy and precision. Additionally, the team evaluated the amount of time and resources necessary to produce an acceptably accurate photogrammetry model that can be used for a variety of archaeological functions, including site monitoring and interpretation. Through statistical analysis, the team determined that, in the worst case scenario, in its current iteration, photogrammetry models created with SeaArray have a margin of error of 5.29 cm at a site over 84 m in length and 65 m in width. This paper discusses the design of the survey, acquisition and processing of data, analysis, issues encountered, and plans to improve the accuracy of the SeaArray photogrammetry system.

Keywords: Structure from Motion photogrammetry; multi-image photogrammetry; 3D modeling; RTK; underwater archaeology; cultural resource management; National Parks

1. Introduction In recent years, the use of photogrammetry to record submerged archaeological sites has developed into a standard documentation tool [1–13]. Photogrammetry allows archaeologists to record information in great detail, while leaving submerged heritage in situ for future study, and it allows the non-diving public to experience underwater cultural heritage without needing to physically visit the site. This mitigates potential disturbances caused by human interference and provides experiences for those without the skills or means to visit underwater cultural heritage sites in person. In archaeological practice, 3D models give archaeologists the ability to extract data from sites and artifacts as new questions arise, without the need to make repeat visits to a physical location. It also decreases the need for expensive and time-consuming artifact conservation processes and allows archaeologists to digitally “preserve” and catalogue artifacts and sites, as well as share data with other researchers. The techniques, hardware, and software used by heritage professionals to collect and process photogrammetric data are changing rapidly, allowing for greater accuracy, reliability, and detail.

J. Mar. Sci. Eng. 2020, 8, 849; doi:10.3390/jmse8110849 www.mdpi.com/journal/jmse J. Mar. Sci. Eng. 2020, 8, 849 2 of 20

Faced with the vast management responsibility for more than 1.4 million hectares of submerged lands, the National Park Service Submerged Resources Center (NPS SRC) and its partner organization, Marine Imaging Technologies (MiTech), developed a multi-camera, diver-operated photogrammetry platform called SeaArray that allows the user to capture hectares of seafloor in a single photogrammetry model. While the models created by the SeaArray system are compelling in their detail, NPS SRC wanted a clear understanding of how closely they reflected the spatial patterning and archaeological reality of the sites themselves, as the models are of limited scientific use if it is unknown how accurate the data are and what limitations there are in documentation. NPS SRC documented the East Key Construction Wreck in Dry Tortugas National Park using Structure from Motion (SfM) photogrammetry, traditional hand mapping, and Real Time Kinematic (RTK) survey to test and establish a baseline “worst case scenario” of accuracy for 3D models created with SeaArray, as well as to evaluate the amount of time and resources necessary to capture an acceptably accurate photogrammetry model of a submerged archaeological site. The resulting photogrammetry model had a specific issue known as the “bowling effect,” a scooping or bowling in the center of the model. This error was caused by a lack of camera calibration coupled with homogenous underwater terrain. Ultimately, the team found through statistical analysis that, in the worst case, this photogrammetry model created with SeaArray has a margin of error of 5.29 cm. Underwater archaeologists use photogrammetry to document submerged archaeological sites three dimensionally. There are two different types of photogrammetry utilized by archaeologists: “Structure from Motion” photogrammetry (abbreviated as “SfM” and commonly referred to as “Multi-Image” photogrammetry) and “Stereo-Vision” photogrammetry. Structure from Motion photogrammetry is the type most commonly used by underwater archaeologists and is the focus of this paper. SfM creates a 3D model from overlapping images by comparing large data sets to identify matching features shared by images. Once matching features within the images have been identified, a three-dimensional structure of the objects or landscapes is reconstructed via complex calculations of camera lens optics and, via that, camera position relative to the objects photographed. Underwater archaeologists have readily adopted SfM photogrammetry as a site-documentation tool [1–14], but relatively few studies have been done to assess the overall accuracy of the resulting models, particularly on underwater archaeological sites. An initial study conducted by Balletti et al. [2] showed promise, having found accuracy to 6 cm over a site of up to 36 m long by comparing photogrammetry to RTK data. A separate study by Skarlatos et al. [9], conducted using trilateration, suggests that photogrammetry models may be up to three times more accurate than traditional trilateration measurements taken in 2D. Parks Canada has offered perhaps the most robust test of accuracy to date using underwater photogrammetry to record the site of HMS Erebus [3]. After creating a model of the site depicting a 32 by 8.8 m area in Agisoft Photoscan consisting of over 125 million points, the Parks Canada team georectified the model using multibeam echo sounder (MBES) data and compared the model with eighteen reference points from digital surface model (DSM) data. Ultimately, Parks Canada found differences ranging from 0.4 cm and 2 cm between DSM and MBES data but confirmed problematic depth measurements in the DSM data with discrepancies between 1.8 cm and 35 cm. Agisoft Metashape Professional’s internal tool estimated the measurement precision of the model to be approximately 0.5 cm on average. Several studies have been conducted assessing the accuracy of terrestrial photogrammetry survey using aerial drones and compared to RTK survey. Using ten ground control points, Barry and Coakley [15] found that aerial drone photogrammetry survey was reliable within 41 mm horizontally and 68 mm vertically with a 1 cm ground sample distance. Uysal et al. [16] found a digital elevation model created with aerial drone photogrammetry to be accurate to 6.62 cm from an altitude of 60 m using 30 ground control points. These studies are the most comparable to the research presented here, J. Mar. Sci. Eng. 2020, 8, 849 3 of 20 although they were both conducted in a terrestrial setting. To the authors’ knowledge, Balletti et al. [2] offJ.ers Mar. the Sci.only Eng. 2020 comparable, 8, x FOR PEER study REVIEW conducted underwater using RTK to date. 3 of 19 Due to the ability of SeaArray to document large swaths of seafloor in one SfM model, NPS SRC wishedDue to testto the the ability accuracy of SeaArray of photogrammetry to document large models swaths of of large seafloor submerged in one SfM archaeological model, NPS SRC sites, suchwished as the to Easttest the Key accuracy Construction of photogrammetry Wreck. In addition models to of its large size, submerged this site was arch selectedaeological because sites, such it is in shallowas the waterEast Key with Construction proximity to Wreck. land that In addition makes it to possible its size, to this conduct site was RTK selected survey andbecause it has it readilyis in distinguishableshallow water featureswith proximity that are to not land subject that makes to movement it possible due to conduct to wave RTK or current survey action.and it has readily distinguishable features that are not subject to movement due to wave or current action. Historical Context 1.1. Historical Context Dry Tortugas National Park is located 109 km west of Key West, at the furthermost tip of the systemDry of lowTortugas reefs National and islands Park that is located comprise 109 thekm Floridawest of Keys.Key West, The at park the consistsfurthermost of seven tip of smallthe islandssystem within of low a managementreefs and islands area that of 259 comprise square kilometersthe Florida [ 17Keys.]. The The Dry park Tortugas, consists named of seven for thesmall lack ofislands fresh water within and a management its abundance area of seaof 259 turtles, square sit kilometers at the edge [17] of. the The major Dry Tortugas, route of maritime named for ingress the andlack egress of fresh for water the Gulf and of its Mexico—the abundance narrowof sea turtles, Straits sit of at Florida—which the edge of the separate major route the southernmostof maritime extentingress of theand Florida egress Keysfor the from Gulf the northernof Mexico—the shoreof narrow the island Straits ofCuba. of Florida—which At the center ofseparate the park the lies thesouthernmost huge “Third extent System” of the structure Florida Keys of Fort from Jeff theerson, northern built shore in the of first the halfisland of of the Cuba. nineteenth At the center century. Theof constructionthe park lies ofthe a massivehuge “Third fort inSystem this isolated” structure and of inhospitable Fort Jefferson, area bu isilt a testament in the first to half the strategicof the importancenineteenth ofcentury. the nearby The harbors.construction The of proximity a massive of fo thert Dryin this Tortugas isolated to and the inhospitable Straits of Florida area wasis a a factortestament in their to regionalthe strategic political importance and military of the nearby importance harbors. but The was proximity (and is) also of the a factor Dry Tortugas in their threat to the to Straits of Florida was a factor in their regional political and military importance but was (and is) also passing maritime commerce. “Any ships traveling the more than 1200 miles of the United States Gulf a factor in their threat to passing maritime commerce. “Any ships traveling the more than 1200 miles coastline will pass close to the Tortugas. The Dry Tortugas pose a serious navigational hazard and of the United States Gulf coastline will pass close to the Tortugas. The Dry Tortugas pose a serious have been the site of hundreds of marine casualties [17].” navigational hazard and have been the site of hundreds of marine casualties [17].” One such casualty was a 19th century sailing vessel that wrecked while carrying building materials One such casualty was a 19th century sailing vessel that wrecked while carrying building for the construction of the fort. Known as the East Key Construction Wreck, the remains of this vessel materials for the construction of the fort. Known as the East Key Construction Wreck, the remains of arethis located vessel in are approximately located in approximately three meters of three water, mete 8 kmrs east of ofwater, Fort Je8 ffkmerson, east as of illustrated Fort Jefferson, in Figure as 1. Theillustrated most prominent in Figurefeatures 1. The most of the prominent site are thefeatures remains of the of site its cargo:are the graywacke remains of its (a hard,cargo: metamorphic graywacke sandstone)(a hard, metamorphic paving slabs, sandstone) graywacke paving flagstones, slabs, and graywacke hundreds flagstones, of rounded and cement hundreds barrels of (asrounded seen in Figurecement2), whichbarrels formed(as seen when in Figure the wooden 2), which barrels formed of when dry-packed the wooden cement barrels hardened of dry-packed after the ship cement sank. Inhardened 1935, Fort after Jeff ersonthe ship and sank. the In surrounding 1935, Fort Jeffer watersson becameand the asurrounding National Monument waters became and a partNational of the UnitedMonument States’ and National a part of Park the System.United States’ In 1992, Nation theal National Park System. Monument In 1992, was the elevatedNational toMonument a National Park—thewas elevated highest to a designationNational Park—the for a unit highest of the desi Nationalgnation Park for a System. unit of the National Park System.

FigureFigure 1. 1. Map showing showing location location of ofEast East Key Key Constructi Constructionon Wreck Wreck relative relative to Garden to Garden Key and Key Fort and FortJefferson. Jefferson.

J. Mar. Sci. Eng. 2020, 8, 849 4 of 20 J. Mar. Sci. Eng. 2020, 8, x FOR PEER REVIEW 4 of 19

Figure 2. An National Park Service (NPS) archaeologist swims over cement barrels on the East Key Construction Wreck.

2. Materials Materials and and Methods

2.1. Real Time Kinematic (RTK) Survey and Measurements 2.1. Real Time Kinematic (RTK) Survey and Measurements Real time kinematic (RTK) GPS surveying is a method used to enhance the accuracy of GPS measurements. Corrective Corrective factors factors known known as as differe differentialntial corrections corrections are are applied applied to to the satellite measurements in in real real time time from from a a second second receiver receiver known known as as a abase base or or reference reference station station occupying occupying a knowna known location location on onthe the earth. earth. In practice, In practice, atmosphe atmosphericric distortion, distortion, multipathing multipathing of radio of radio signals, signals, and instrumentand instrument errors errors in things, in things, like the like GPS the GPSsatellite satellite clock, clock, introduce introduce distortions distortions in one in oneor more or more of the of satellitethe satellite signals signals in the in theconstellation constellation of ofGPS GPS satellit satelliteses that that is isused used to to generate generate a aposition position fix fix for for a surveyor. These cumulative cumulative errors errors reduce reduce the the accura accuracycy of of a a GPS GPS position position to to approximately approximately one one meter. meter. Applying real-time, didifferentialfferential corrections fromfrom a base station occupying a known terrestrial location can reduce these errors to a few centimeters. RTK survey requires a reference or base station receiver to occupy a location, location, commonly commonly referred referred to as a a survey survey control control point, point, that that has has an an already already known known position. position. The The differen differencesces between the position reported forfor thisthis survey survey control control point point by by the the receiver receiver and and the knownthe known position position are broadcast are broadcast to a second to a secondreceiver receiver or “rover” or to“rover” correct to cumulative correct cumulative errors in theerrors rover’s in the observations. rover’s observations. Utilizing RTK Utilizing techniques RTK techniquesand equipment, and equipment, the NPS archeology the NPS archeology team was ableteam to was reduce able theto reduce uncertainty the uncertainty in observed in positionsobserved positionsfrom a few from meters a few to meters a few centimetersto a few centimeters (or less) in(or 3 less) dimensions—this in 3 dimensions—this represents represents the current the current state of statethe art of for the accurate art for GPSaccurate measurements GPS measurements in a remote, in o ffa shore,remote, marine offshore, environment marine environment and provides and the providesbest data the available best data for available true real worldfor true locations real world of points locations within of points the East within Key the wreck East site. Key wreck site. NPS archeologistsarcheologists established established a networka network of two of primarytwo primary datums datums and eleven and sub-datums eleven sub-datums (hereafter (hereafterreferred to referred as markers) to as for markers) a total offor thirteen a total of control thirteen points control spread points out spread within out the within East Key the shipwreck East Key shipwrecksite, indicated site, inindicated red in Figure in red3 in. The Figure points 3. The were points labeled were A labeled through A Mthrough with points M with A points and B A set and as Bpermanent set as permanent brass datums brass atdatums the ends at ofthe the ends site of and the the site other and eleventhe other markers eleven denoted markers by denoted temporary by temporaryorange plastic orange stakes. plastic These stakes. markers These were markers used to testwere spatial used dimensions to test spatial of the dimensions SfM model of and the also SfM to modeladd spatial and also control. to add The spatial permanent control. (A The and permanent B) datums (A were and set B) datums on the outer were marginsset on the of outer the site margins at the ofnorth the site and at south the north ends, and while south the ends, other while markers the were other set markers throughout were set the throug site andhout were the not site set and up were in a notgrid set or up geometric in a grid pattern. or geometric pattern.

J. Mar. Sci. Eng. 2020, 8, 849 5 of 20 J. Mar. Sci. Eng. 2020, 8, x FOR PEER REVIEW 5 of 19

Figure 3.3. SiteSite planplan showing showing Structure Structure from from Motion Motion (SfM) (SfM) model model boundary, boundary, Real TimeReal Time Kinematic Kinematic (RTK) (RTK)markers, markers, and NPS and hand-mapped NPS hand-mapped site plan site from plan 1990. from 1990.

RTK observations were taken with the rover receiverreceiver set up on a tripod over the datum and markers on the wreck site, as shown in Figure4 4a.a. AA basebase stationstation antennaantenna/receiver/receiver was set up over a known control point at Fort Jefferson Jefferson (as seen in Figure 44b)b) whichwhich generatedgenerated correctionalcorrectional informationinformation to broadcast broadcast via via a a Trimble Trimble TDL TDL 450 450 UHF UHF radio radio to to th thee rover rover on on the the site. site. The The equipment equipment used used for the for basethe base was was a Trimble a Trimble NetR9 NetR9 receiver receiver connected connected to toa aZephyr Zephyr 3 3 GNSS GNSS (Global Navigation Satellite System) antenna. Rover Rover operations operations at at the the wreck wreck si sitete used used a Trimble a Trimble TDL TDL 450 450 UHF UHF radio radio mounted mounted in thein the survey/dive survey/dive boat boat operating operating as a as radio a radio repeater repeater that that rebroadcast rebroadcast differential differential corrections corrections from from the basethe base station station to a Trimble to a Trimble R8 model R8 model 3 GNSS 3 GNSS receiver receiver equipped equipped with a with short-range a short-range internal internal UHF radio. UHF Aradio. weighted A weighted 2 m, Hixon 2 m, Mfg. Hixon aluminum Mfg. aluminum tripod was tripod set up was and set centered up and centeredon a selected on a point, selected and point, then aluminumand then aluminum extension extension poles were poles added were until added the until top theextension top extension pole was pole less was than less than1 m 1below m below the surfacethe surface of the of thewater. water. Additional Additional stability stability was was added added to tothe the setup setup by by attaching attaching guy guy lines lines to to this this top extension pole and anchored to a solid feature on the seafloor.seafloor. This minimized excessive movement of the GNSS receiver caused by the effect effect of wave actionaction on the tripodtripod/extension/extension pole setup. Vertical Vertical control forfor thethe tripodtripod was was via via reference reference to to a suitea suite of of traditional traditional surveyor’s surveyor’s bubble bubble levels levels attached attached to the to thetripod tripod and and extension extension poles. poles. Once Once the tripodthe tripod was setwas up, set leveled, up, leveled, and secured, and secured, the Trimble the Trimble R8 GNSS R8 GNSSreceiver receiver was mounted was mounted on top ofon the top last of 1the m extensionlast 1 m extension pole by the pole surveyor by the insurveyor the boat in and the handed boat and off handedto the archeologist, off to the whoarcheologist, then mounted who thethen last mounte extensiond the pole last on theextension top of thepole tripod on /extensionthe top of poles, the tripod/extension poles, as shown in Figure 4c. A series of positional measurements were then taken and averaged to provide centimeter accurate positions for each point on the wreck site.

J. Mar. Sci. Eng. 2020, 8, 849 6 of 20 as shown in Figure4c. A series of positional measurements were then taken and averaged to provide J. Mar. Sci. Eng. 2020, 8, x FOR PEER REVIEW 6 of 19 centimeter accurate positions for each point on the wreck site.

(b)

(c)

(a)

FigureFigure 4. (a 4.) An (a) NPS An archaeologistNPS archaeologist inspects inspects the tripod the from tripod below. from (b )below. GPS base (b) stationGPS base at Fort station Jefferson. at Fort (c) AnJefferson. NPS diver (c) An mounts NPS diver the GPS mounts receiver the toGPS the receiver tripod. to the tripod.

2.2. Hand Measurement Methodology 2.2. Hand Measurement Methodology In additionIn addition to settingto setting up up and and taking taking GPS GPS location location measurements measurements for for a system a system of datumsof datums and and markers,markers, the the NPS NPS archeology archeology team team made made direct direct measurements measurements by handby hand of prominentof prominent features features and and artifactsartifacts on theon the East East Key Key Construction Construction Wreck Wreck site site to compare to compare to the to the same same measurements measurements taken taken in the in the photogrammetryphotogrammetry model. model. This This was donewas usingdone traditionalusing traditional fiberglass fiberglass tape measures tape measures and recording and resultsrecording on underwaterresults on underwater slates with waterproofslates with paper. waterproof Measurements paper. Measuremen were taken tots the were nearest taken centimeter. to the nearest 2.3.centimeter. SeaArray Methodology

2.3.To SeaArray meet the NPSMethodology mandate of resource stewardship and protection of submerged sites, the National Park Service Submerged Resources Center, in partnership with Marine Imaging Technologies (MiTech), To meet the NPS mandate of resource stewardship and protection of submerged sites, the developed a multi-camera photogrammetry platform named SeaArray, shown in Figure5. SeaArray is National Park Service Submerged Resources Center, in partnership with Marine Imaging a self-propelled diver-operated three camera array for high resolution image capture that has expanded Technologies (MiTech), developed a multi-camera photogrammetry platform named SeaArray, the boundaries of underwater photogrammetry capabilities from areas of meters to areas of hectares. shown in Figure 5. SeaArray is a self-propelled diver-operated three camera array for high resolution The initial impetus for the development of SeaArray was to maximize accuracy with underwater image capture that has expanded the boundaries of underwater photogrammetry capabilities from 3D models utilizing a multiple camera acquisition strategy and to increase the area covered during areas of meters to areas of hectares. data collection while maintaining high-resolution visualizations, all within the restrictions of time The initial impetus for the development of SeaArray was to maximize accuracy with underwater spent underwater and the diver’s available gas supply. The multi-camera system has exceeded initial 3D models utilizing a multiple camera acquisition strategy and to increase the area covered during expectations and has been used to survey large 250 m by 125 m areas with pixel-level clarity. With over data collection while maintaining high-resolution visualizations, all within the restrictions of time 340,000 images generated to date, the SeaArray averages 97% image alignment using extremely large spent underwater and the diver’s available gas supply. The multi-camera system has exceeded initial data sets and is capable of using nearly 30,000 images for a single 3D visualization.1 expectations and has been used to survey large 250 m by 125 m areas with pixel-level clarity. With over 340,000 images generated to date, the SeaArray averages 97% image alignment using extremely large data sets and is capable of using nearly 30,000 images for a single 3D visualization.1 1 A high percentage of aligned photos creates models with higher accuracy and larger amounts of detail.

1 A high percentage of aligned photos creates models with higher accuracy and larger amounts of detail.

J. Mar. Sci. Eng. 2020, 8, 849 7 of 20 J. Mar. Sci. Eng. 2020, 8, x FOR PEER REVIEW 7 of 19

Figure 5. A diver operates SeaArray.

SeaArray isis comprised comprised of of three three 45.7 45.7 MP MP Nikon Niko Z7n mirrorless Z7 mirrorless cameras cameras fitted withfitted 14 with mm 2.814 Rokinonmm 2.8 lensesRokinon housed lenses in housed Marine in Imaging Marine Imaging Technologies’ Techno customlogies’ underwatercustom underwater camera housingscamera housings and camera and controller.camera controller. The imaging The imaging system system is engineered is engineered on a modular, on a modular, carbon carbon fiber frame fiber frame with pivoting with pivoting arms builtarms aroundbuilt around a SubGravity a SubGravity/Bonex/Bonex diver propulsion diver propul vehiclesion (DPV).vehicle The (DPV). system The is system capable is of capable capturing of 10,800capturing images 10,800 per images hour. per The hour. camera The control camera housing control includes housing real-time includes HDMIreal-time (high HDMI definition (high multimediadefinition multimedia interface) camera interface) monitoring camera and monitoring switching, and ISO switching, control, shutter ISO control, independentshutter control, still imageindependent capture, still and image a vacuum capture, leak and detection a vacuum system. leak detection The Marine system. Imaging The Marine Arduino Imaging based softwareArduino operatesbased software the camera operates controls the camera and shutter controls releases and shu simultaneouslytter releases simultaneously with capabilities with for capabilities sequential for or simultaneoussequential or simultaneous image capturing image with capturing a customizable with a capturecustomizable rate, synchronized capture rate, camerasynchronized settings, camera and a batterysettings, life and indicator. a battery life indicator. The design of SeaArray as a multiple camera arrayarray requires the issue of overlap to be addressed on twotwo levels—individuallevels—individual camera camera and and lane lane overlap. overlap. The The “primary” “primary” camera camera for navigation for navigation is the centralis the cameracentral camera in the three-camera in the three-camera geometry. geometry. The overall The goaloverall in imagegoal in acquisition image acquisition is to achieve is to achieve the Agisoft the MetashapeAgisoft Metashape recommended recommended 60% side 60% image side overlap image and overlap 80% forwardand 80% image forward overlap. image To overlap. accomplish To thisaccomplish on the individual this on the camera individual level, camera overall level, coverage overall is largely coverage determined is largely bydetermined reciprocal by lane reciprocal spacing thatlane containsspacing that 60% contains side overlap, 60% side while overlap, the adjustable while the automated adjustable shutter automated release shutter ensures release 80% forwardensures overlap80% forward through overlap speed through of travel speed and aof typical travel and data a collection typical data rate collection of 1 frame rate per of second. 1 frame Collectively,per second. theCollectively, three-camera the three-camera alignment often alignment offers dioftenffering offers overlap differing on aoverlap single on object a single modeled. object Duringmodeled. a reciprocalDuring a reciprocal transect, thetransect, central the camera central may camera be collecting may be collecting imagery atimagery a 60% at overlap, a 60% overlap, but the inboard but the camerainboard maycamera be atmay 90% be overlap,at 90% overlap, while the while outboard the outboard camera camera may be may closer be closer to 30%. to 30%. Regarding Regarding lane overlap,lane overlap, which which consists consists of a collection of a collection of three of simultaneously three simultaneously generated generated images, images, the goal the is again goal tois achieveagain to 60%achieve side 60% overlap side andoverlap 80% and forward 80% overlap.forward overlap. The benefit The of benefit SeaArray of SeaArray is that multiple is that multiple cameras withincameras each within lane each create lane additional create additional overlap, which overlap, in turn which creates in turn more creates data pointsmore data for the points software for the to generate.software to This generate. results This in a higherresults numberin a higher of tie number points of within tie points the model. within the model. Underwater conditionsconditions are are consistently consistently variable variable from from one siteone to site the next.to the Water next. visibility, Water visibility, available light,available and light, contour and or contour relief of theor re objectlief of being the object modeled being are factorsmodeled when are consideringfactors when the considering distance to the subjectdistance or to height the subject from the or seafloorheight from that thethe diver seafloor must that maintain the diver while must operating maintain SeaArray. while operating Since the operationalSeaArray. Since goal the of theoperational system is goal to gather of the assystem dense is of to a gather data cloud as dense as possible, of a data NPS cloud SRC as typicallypossible, attemptsNPS SRC to typically operate 2–3attempts m from to the operate seafloor 2–3 or m subject, from the even seafloor in good or water subject, visibility. even Thisin good allows water the teamvisibility. to not This simply allows map the large team areas to not of simply seafloor map but tolarg doe soareas with of aseafloor saturation but ofto datado so points with a that saturation enables theof data end points model that to be enables visualized the end in great model detail. to be visualized in great detail. Advancements in modeling algorithms and the abilityability of Agisoft MetaShape to generate point locations and measured distances with very little opticaloptical testing or calculations was a known benefitbenefit when the development of SeaArray began. The intent was to build an imaging system that was field ready that required little optical engineering or testing that could deliver a highly accurate and robust

J. Mar. Sci. Eng. 2020, 8, 849 8 of 20 when the development of SeaArray began. The intent was to build an imaging system that was field ready that required little optical engineering or testing that could deliver a highly accurate and robust point cloud of underwater data. The team relies on the software to interpret the required optical corrections and angles (with the exception of camera calibration). NPS SRC and Marine Imaging Technologies, in partnership with the Department of Design Studies at the University of Wisconsin, developed custom workflows to generate models using Agisoft Metshape Professional (currently version 1.6.3) specific to field project resources, processing availability and time, and end model visualization needs. For the East Key Construction Wreck 3D model, the team created the model by aligning 100% of the 12,942 images collected from SeaArray’s three cameras in Agisoft Metashape. This initial photo alignment process resulted in a sparse cloud containing over 14.5 million tie points, which the team reprocessed by using the gradual selection and camera optimization tools to clean the model of spurious outlying points. This initial resulting sparse point cloud model fell victim to what is known as the “bowling effect.” The bowling effect, which causes a scooping or bowling in the center of a model, can sometimes occur in 3D models rendered from largely homogenous terrain, like the sea floor at the East Key Construction Wreck, when they are generated without proper camera calibration. The seafloor of the East Key Construction Wreck is made up of low-lying artifacts, patchy coral, and sand, all of a similar color palette and height off of the seafloor. The Agisosft Metashape User Manual Version 1.6 [18] suggests that lens calibration can be skipped in common workflows, but may be useful if alignment results are unstable [18]. The issue of lens calibration on the SeaArray system had been discussed and attempted by the authors. The area of optical coverage of each individual camera within the SeaArray system, resulting in the 14 mm (115.7 degree field of view) combined with a fixed focal distance, made SeaArray camera calibration difficult in water. SeaArray camera calibration was first attempted with an initial 2 m 1 m calibration grid, but it was not large enough to fill the frame while maintaining a standard × operational distance of 2 m–3 m. A larger calibration grid has been designed for future calibration of SeaArray cameras. Due to these difficulties, NPS SRC purposely tested SeaArray without calibrating the cameras in order to test the system at its most basic level, or its “worst case scenario.” If the bowling effect occurs in a model created without proper camera calibration, Agisoft recommends correcting this distortion through a camera optimization procedure based on ground control points or camera coordinates after the photo alignment process. During this process, Agisoft adjusts the 3D model based on geodetic parameters introduced by the ground control point data [19]. The team corrected the bowling effect using this methodology, but this “corrected” version of the model is not discussed here, as the primary goal for the study was to test the difference between RTK data and an uncorrected, uncalibrated SeaArray photogrammetry model. Next, the team generated a 1.2-billion-point dense cloud from the sparse cloud in Agisoft Metashape Professional. The model creation process within Metashape Professional was stopped at the dense cloud step, without continuing to a mesh or textured model, as a high-quality dense cloud is a much more accurate method of visualizing sites. At this point, the software is working only with actually observed data and has not yet interpolated any points, as occurs in the creation of a mesh or textured model. The dense cloud model was exported into Viscore software for data analysis. Viscore is a point-based visual analytics environment created for the purpose of conducting analysis on large point-cloud data sets by computer scientists of the Cultural Heritage Engineering Initiative (CHEI) at the University of California San Diego. The software allows for quick access and analysis of data [20]. A quicker response time when loading large data sets for model manipulation makes Viscore advantageous over Agisoft when conducting point-cloud data analysis. Once each version of the model was imported into Viscore, the team scaled the dense clouds based upon three, one-meter scaled targets placed on the seafloor during the data collection process. The position of the markers was then set within Viscore. Using a measurement tool within the software, J. Mar. Sci. Eng. 2020, 8, 849 9 of 20

the team measured distances between the markers and recorded them in a Microsoft Excel spreadsheet. J. Mar.The Sci. Viscore Eng. 2020 measurement, 8, x FOR PEER REVIEW process is shown in Figure6. 9 of 19

(b)

(a)

FigureFigure 6. (a) 6. Measurement(a) Measurement of distances of distances between between markers markers in Viscore. in Viscore. (b) Blow-up (b) Blow-up of artifact of artifact measure. measure.

Then,Then, using using the themeasurement measurement tool toolin Viscore, in Viscore, individual individual artifact artifact measurements measurements were were taken taken to to matchmatch those those taken taken using using traditional hand-mappinghand-mapping methods. methods. These These measurements measurements were were also recordedalso recordedin a Microsoft in a Microsoft Excel Excel matrix matrix for comparison. for comparison. Once Once all data all data were were recorded recorded in a in Microsoft a Microsoft Excel Excel matrix, matrix,analysis analysis of the of datathe data was was conducted conducted using using functions functions in Excel. in Excel.

3. Results3. Results

3.1. 3.1.Difference Difference Between between RTK RTK and andSfM SfM Model Model TableTable 1 displays1 displays the themeasurements measurements from from marker marker to marker to marker calculated calculated using using both both RTK RTK and and SfM SfM modelmodel data. data. These These data data sets setswere were used used to generate to generate the absolute the absolute value value differences differences between between RTK RTK and and SfMSfM values values displayed displayed in the in thethird third column. column.

Table 1. This table illustrates the calculated absolute value difference between marker measurements Table 1. This table illustrates the calculated absolute value difference between marker measurements taken by RTK and by SfM model. taken by RTK and by SfM model.

MarkerMarker to toMarker Marker RTK RTKMeasure Measure SfM Measure SfM Measure Absolute Absolute Value Difference Value Diff erence A Ato toB B60.398 60.398 59.628 59.6280.770 0.770 A Ato toC C 31.799 31.799 31.695 31.6950.104 0.104 A to D 49.979 49.566 0.413 A Ato toF D 51.125 49.979 50.440 49.5660.685 0.413 A toA G to F 49.187 51.125 48.271 50.4400.916 0.685 A to H 44.581 44.116 0.465 A to G 49.187 48.271 0.916 A to I 37.162 37.010 0.152 AA to toJ H 26.515 44.581 26.455 44.1160.060 0.465 A toA K to I 18.524 37.162 18.472 37.0100.052 0.152 A to L 10.772 10.682 0.090 A toA M to J 15.078 26.515 14.919 26.4550.159 0.060 B Ato C to K 38.872 18.524 38.990 18.4720.188 0.052 B to D 23.048 23.242 0.194 A to L 10.772 10.682 0.090 B to F 9.730 9.817 0.087 B to G 17.053 17.100 0.047 B to H 15.998 16.081 0.083 B to I 25.974 26.143 0.169 B to J 36.002 36.090 0.088 B to K 41.970 41.776 0.194 B to L 50.392 49.930 0.462

J. Mar. Sci. Eng. 2020, 8, 849 10 of 20

Table 1. Cont.

Marker to Marker RTK Measure SfM Measure Absolute Value Difference A to M 15.078 14.919 0.159 B to C 38.872 38.990 0.188 B to D 23.048 23.242 0.194 B to F 9.730 9.817 0.087 B to G 17.053 17.100 0.047 B to H 15.998 16.081 0.083 B to I 25.974 26.143 0.169 B to J 36.002 36.090 0.088 B to K 41.970 41.776 0.194 B to L 50.392 49.930 0.462 B to M 47.618 47.286 0.332 C to D 20.419 20.444 0.025 C to F 32.590 32.612 0.022 C to G 36.846 36.699 0.147 C to H 27.417 27.472 0.055 C to I 19.579 13.627 0.048 C to J 10.293 10.264 0.029 C to K 21.196 21.210 0.014 C to L 26.891 26.840 0.051 C to M 17.249 17.056 0.193 D to F 21.557 21.642 0.085 D to G 29.915 29.900 0.015 D to H 20.502 20.564 0.062 D to I 14.011 14.015 0.004 D to J 23.611 23.612 0.001 D to K 34.390 34.308 0.082 D to L 42.389 42.165 0.224 D to M 35.194 34.943 0.251 F to G 9.404 9.360 0.044 F to H 6.553 6.538 0.015 F to I 19.042 19.105 0.063 F to J 27.881 27.883 0.002 F to K 32.616 32.374 0.242 F to L 10.933 40.485 0.448 F to M 38.954 38.617 0.337 G to H 10.040 9.957 0.083 G to I 23.682 23.626 0.056 G to J 29.785 29.628 0.157 G to K 31.003 30.566 0.437 G to L 29.569 37.885 0.984 G to M 39.121 38.574 0.547 J. Mar. Sci. Eng. 2020, 8, 849 11 of 20

Table 1. Cont.

Marker to Marker RTK Measure SfM Measure Absolute Value Difference H to I 13.994 14.015 0.071 H to J 21.686 21.734 0.048 H to K 26.081 25.935 0.146 H to L 34.436 34.138 0.298 H to M 32.492 32.273 0.219 I to J 10.810 10.861 0.051 I to K 20.535 20.544 0.009 I to L 28.753 28.697 0.056 I to M 22.906 22.818 0.088 J to K 11.726 11.767 0.041 J to L 18.968 18.956 0.012 J to M 12.156 12.027 0.129 K to L 8.607 8.501 0.106 K to M 10.004 9.878 0.126 L to M 10.705 10.749 0.044 Average Difference 0.174327653

The team found that the average absolute value difference between the distance measurements calculated using RTK data and the distance measurements calculated using the uncorrected and uncalibrated photogrammetry model was 0.174 m, with a maximum difference of 0.916 m and a minimum of 0.001 m. This average difference was calculated using absolute value because as distance is a simple measure that describes the distance between two objects. It does not matter whether the difference between the markers calculated using RTK data or Agisoft data was positive or negative, only that these values differ from one another. Additionally, a mathematical average that combines positive and negative numbers would give an artificially low value for average difference. AppendixA is a scatter plot that illustrates the di fference between the RTK calculated distances and distances calculated with the uncalibrated SfM model. The use of a scatter plot shows the large degree of agreement between the two data sets and indicates that, even though there are several seemingly large differences in measurements, this difference is negligible when the 84 m 65 m total × size of the site is taken into account.

3.2. Statistical Analysis The team then conducted statistical analysis to determine a standard deviation and 95% confidence interval for the data set. The RTK GPS data was compared to the Agisoft data to determine the three-dimensional positioning differences (errors). A total of 65 error pairs were used. Error directionality was not included in this analysis. A histogram of these errors is shown in Figure7. The mean of these 65 errors is 12.11 cm as seen by the black vertical line in Figure8. The di fferences are skewed negatively − indicating that the Agisoft data is shorter than the RTK data. This may be caused by the bowling effect within the Agisoft processing. This also shows that the errors are not random with a zero mean. The standard deviation, σ, of the data is σ = 21.76 cm. The resulting 95% confidence interval is:

CI = ( 17.4, 6.82) cm. 95 − −

This 95% confidence interval indicates a spherical error of radius R95 = 5.29 cm. J.J. Mar.Mar. Sci.Sci. Eng.Eng. 20202020,, 88,, xx FORFOR PEERPEER REVIEWREVIEW 11 11 ofof 1919 seeminglyseemingly largelarge differencesdifferences inin measurements,measurements, thisthis didifferencefference isis negligiblenegligible whenwhen thethe 8484 mm ×× 6565 mm totaltotal sizesize ofof thethe sitesite isis takentaken intointo account.account.

3.2.3.2. StatisticalStatistical AnalysisAnalysis TheThe teamteam thenthen conductedconducted statisticalstatistical analysisanalysis toto determinedetermine aa standardstandard deviationdeviation andand 95%95% confidenceconfidence intervalinterval forfor thethe datadata set.set. TheThe RTKRTK GPSGPS datadata waswas comparedcompared toto thethe AgisofAgisoftt datadata toto determinedetermine thethe three-dimensionalthree-dimensional positioningpositioning differencesdifferences (errors).(errors). AA totaltotal ofof 6565 ererrorror pairspairs werewere used.used. ErrorError directionalitydirectionality waswas notnot includedincluded inin thisthis analysis.analysis. AA histogramhistogram ofof thesethese errorserrors isis shownshown inin FigureFigure 7.7. TheThe meanmean ofof thesethese 6565 errorserrors isis −−12.1112.11 cmcm asas seenseen byby thethe blackblack verticalvertical lineline inin FigureFigure 8.8. TheThe differencesdifferences areare skewedskewed negativelynegatively indicatingindicating thatthat thethe AgisoftAgisoft datadata isis shortershorter thanthan thethe RTKRTK data.data. ThisThis maymay bebe causedcaused byby thethe bowlingbowling effecteffect withinwithin thethe AgisoftAgisoft processing.processing. ThisThis alsoalso showshowss thatthat thethe errorserrors areare notnot randomrandom withwith aa zerozero mean.mean. TheThe standardstandard deviation,deviation, σσ,, ofof thethe datadata isis 𝜎𝜎 21.7621.76 .cm .cm TheThe resultingresulting 95%95% confidenceconfidence intervalinterval is:is:

𝐶𝐼𝐶𝐼 17.4, 17.4, 6.82 6.82 cmcm.. J. Mar. Sci. Eng. 2020, 8, 849 12 of 20 ThisThis 95%95% confidenceconfidence intervalinterval indicatesindicates aa sphericalspherical errorerror ofof radiusradius 𝑅𝑅 = = 5.295.29 cm.cm.

FigureFigure 7.7. HistogramHistogram showingshowing thethe distribudistributiondistributiontion ofof thethe 6565 errorerror pairs.pairs.

.. Figure 8. The horizontal axis shows the distribution of the 65 error pairs. The vertical line represents FigureFigure 8.8. TheThe horizontalhorizontal axisaxis showsshows thethe didistributionstribution ofof thethe 6565 errorerror pairpairs.s. TheThe verticalvertical lineline representsrepresents the mean of 12.11 cm. thethe meanmean ofof − −12.1112.11 cm.cm. 3.3. Difference between Artifact Hand Measurements and Artifact SfM Measurements

The team measured five cement slabs, a pig iron bar, ten individual cement barrels in a line, and the line of barrels as a whole by hand on the site of the East Key Construction Wreck. These same measurements were taken in Viscore and compared in Table2.

Table 2. Artifact hand measurements and artifact SfM model measurements.

Meaurement 1 (cm) Meaurement 2 (cm) Meaurement 3 (cm) Hand Viscore Abs. ∆ Hand Viscore Abs. ∆ Hand Viscore Abs. ∆ Slab #82 94 94.56 0.56 234 234.1 0.1 — — — Slab #83 166 164.2 1.8 40 39.953 0.05 — — — Barrel #1 67 66.051 0.949 — — 0 67 66.051 0.949 Barrel #2 62 63.197 1.2 74 73.703 0.297 136 136.9 0.9 Barrel #3 65 64.7 0.3 141 140.8 0.2 206 205.5 0.5 Slab #85 234 233.5 0.5 92 92.018 0.02 — — — Slab #86 255 252.4 2.6 42 41.507 0.49 — — — Slab #88 185 183.3 1.7 32 30.692 1.31 — — — J. Mar. Sci. Eng. 2020, 8, 849 13 of 20

Table 2. Cont.

Meaurement 1 (cm) Meaurement 2 (cm) Meaurement 3 (cm) Hand Viscore Abs. ∆ Hand Viscore Abs. ∆ Hand Viscore Abs. ∆ Pig Iron Bar #90 10 10.05 0.05 10 9.906 0.09 73 72.622 0.38 Barrel Line Length #84 710 709.7 0.3 — — — — — — Ave. Abs. Difference 1.072857 0.343333 0.38 Overall Barrel Width (cm) Barrell Origin (cm) Barrel Terminus (cm) Hand Viscore Abs. ∆ Hand Viscore Abs. ∆ Hand Viscore Abs. ∆ Barrel #4 63 63.9 0.9 207 207.1 0.1 270 271 1 Barrel #5 69 65.8 3.2 278 281.4 3.4 347 347.2 0.2 Barrel #6 66 64.3 1.7 348 348.8 0.8 414 413.1 0.9 Barrel #7 67 65.1 1.9 430 430.5 0.5 497 495.6 1.4 Barrel #8 66 65.4 0.6 504 503.9 0.1 570 569.3 0.7 Barrel #9 65 66 1 573 571.6 1.4 638 637.6 0.4 Barrel #10 66 63.3 2.7 646 647.4 1.4 712 710.7 1.3 Ave. Abs. Difference 1.714286 1.1 0.842857

When comparing the measurements of artifacts (the longest of which was 712 cm) taken by hand and by uncalibrated SfM model, the team found a negligible average difference of 928 cm, less than one centimeter, between the data sets.

4. Discussion

4.1. Accuracy and Precision Balletti et al. [2] and Boyer and Lockhart [3] confirm that photogrammetry models, when properly scaled, are an accurate recording tool for submerged cultural resources on underwater archaeological sites up to 30 m in length. This study of the East Key Construction Wreck indicates that large-scale areas, even up to 5460 sq. m, can be rendered with archaeologically acceptable accuracy and precision with the use of SeaArray, as well as may be even more accurate with proper camera calibration and georectification. Accuracy is defined as a measure of how close a value is to its true value and describes statistical bias. To test the accuracy of SeaArray photogrammetry models, the team compared measured distances generated from the RTK points and points on the SfM model. This comparison yielded a variety of errors ranging from 91.6 cm to 0.01 cm with an absolute value average error of 17.4 cm between the two data sets. The statistical analysis indicated a spherical error of radius of 5.29 cm at the 95% confidence level. The team also compared the traditional hand measurements of artifacts and features on the site and measurements of these same objects taken in Viscore on the SfM model. These measured distances between traditional mapping techniques and measurements taken from the SfM model in Viscore varied on average by 0.928 cm. Some of this difference can be attributed to the variation between the placement of the tape measure when hand measurements were taken and the placement of the mouse when measuring in Viscore. Some level of variation between users, and even by the same user, is unavoidable, whether the measure is being derived by fiberglass tape or mouse cursor. While the team took photos of tape measure placement during the hand measuring process and attempted to place the mouse at the same points when measuring on the Viscore model, there is undeniably some amount of variation based on location. Additionally, team members in the field rounded recorded measurements to the nearest centimeter, whereas Viscore measurements were presented in the software with at least two decimal places (e.g., a hand recorded measurement of 92 cm versus a Viscore measurement of 92.01 cm). This resulted in additional differences between the traditional tape measurements and those derived from the Viscore model. We doubt that these small measurement differences are mathematically or archaeologically meaningful. J. Mar. Sci. Eng. 2020, 8, x FOR PEER REVIEW 13 of 19

While the team took photos of tape measure placement during the hand measuring process and attempted to place the mouse at the same points when measuring on the Viscore model, there is undeniably some amount of variation based on location. Additionally, team members in the field rounded recorded measurements to the nearest centimeter, whereas Viscore measurements were presented in the software with at least two decimal places (e.g., a hand recorded measurement of 92 cm versus a Viscore measurement of 92.01 cm). This resulted in additional differences between the J. Mar. Sci. Eng. 2020, 8, 849 14 of 20 traditional tape measurements and those derived from the Viscore model. We doubt that these small measurement differences are mathematically or archaeologically meaningful. TheThe difference difference in inthe the results results when when measuring measuring indi individualvidual key key features features measured measured by by hand hand or orby by SfMSfM are are muchmuch smaller smaller than than the dithefference difference in results in whenresults comparing when marker-to-markercomparing marker-to-marker measurements. measurements.This partly reflects This partly the di ffreflectserence the between difference the small- between and the large-scales small- and in large-scales which these in measurements which these measurementsare made. The are largest made. single The featurelargest measurementsingle feature wasmeasurement 7.1 m, whereas was the7.1 largestm, whereas marker-to-marker the largest marker-to-markermeasurement was measurement 60.398 m. We was predict 60.398 that m. error We in predict both types that of error measurements in both types will of be measurements correspondingly willless be on correspondingly sites smaller than less the on Eastsites Key smaller Construction than the Wreck,East Key where Construction all measurements Wreck, where may beall in measurementssmaller increments. may be in smaller increments. WhenWhen compared compared to tothe the similar similar underwater underwater study study co conductednducted by by Balletti Balletti et etal. al. [2], [2 ],who who found found photogrammetryphotogrammetry models models to to be be accurate accurate up up to to6 cm 6 cm ov overer a site a site of of36 36m mlong long when when compared compared with with RTK, RTK, a sphericala spherical error error of ofradius radius of of5.29 5.29 cm cm at atthe the 95% 95% co confidencenfidence level level indicates indicates similar similar results. results. This This study study alsoalso yields yields similar similar results results to tothe the stud studyy conducted conducted by by Uysal Uysal et etal. al. [16], [16 ],who who found found a adigital digital elevation elevation modelmodel created created with with aerial aerial drone drone photogrammetry photogrammetry to tobe be accurate accurate to to6.62 6.62 cm cm from from an an altitude altitude of of60 60 mm usingusing 30 30 ground ground control control points. points. While While this this study study did did not not find find results results as asaccurate accurate as asthose those found found by by BarryBarry and and Coakley Coakley [15], [15 who], who found found that that aerial aerial drone drone photogrammetry photogrammetry survey survey was wasreliable reliable within within 41 mm41 mmhorizontally horizontally and and68 mm 68 mmvertically vertically with with a 1 acm 1 cmground ground sample sample distance, distance, the the many many variables variables of of conductingconducting photogrammetric photogrammetric survey survey underwate underwaterr likely likely account account for for these these discrepancies. discrepancies.

4.2.4.2. The The Bowling Bowling Effect Effect TheThe bowling bowling effect effect in inthe the uncorrected uncorrected data data was was caused caused by by relatively relatively flat flat topography topography of ofthe the wreck wreck sitesite combined combined with with a a purposeful purposeful lack ofof groundground control control tie tie points points and and camera camera lens lens calibration. calibration. The The large largearea area of the of the wreck wreck and and survey survey area, area, coupled coupled with with the the homogenous, homogenous, low-lying low-lying topography topography and and lack lackof calibratedof calibrated cameras, cameras, resulted resulted in conditions in conditions conducive conducive to the to bowling the bowling effect. effect. Eventhough Even though bowling bowlingresulted resulted in skewed in skewed long-distance long-distance measurement measurem data,ent the data, difference the difference between between the smaller, the individualsmaller, individualartifact measurements artifact measurements taken on taken both theon correctedboth the corrected and uncorrected and uncorrected model data model sets ofdata 0.928 sets cm of is 0.928negligible. cm is negligible. This indicates This that,indicates had thethat, uncalibrated, had the uncalibrated, ungeorectified ungeorectified 3D model not3D fallenmodel victim not fallen to the victimbowling to the eff ect,bowling the di ffeffect,erence the in measurementsdifference in measurements between the RTK between and SfM the dataRTK sets and would SfM data have sets been wouldmuch have smaller. been much smaller. FigureFigure 9 9illustrates illustrates profile profile views views of of both both the the SfM SfM model model discussed discussed in in this this paper paper and and a amodel model createdcreated with with the the same same data set thatthat waswas corrected corrected through through the the integration integration of RTKof RTK ground ground control control points points(not discussed(not discussed in this in paper). this paper). It is apparent It is apparent from the imagesfrom the that images the bowling that the eff ectbowling is responsible effect is for responsiblesome error for in some distance error measurements in distance measurements from marker tofrom marker. marker to marker.

(a)

Figure 9. Cont.

J. Mar. Sci. Eng. 2020, 8, 849 15 of 20 J. Mar. Sci. Eng. 2020, 8, x FOR PEER REVIEW 14 of 19

(b)

Figure 9. Profile views of bowled uncalibrated model (a) versus model corrected through ground control process (b). Figure 9. Profile views of bowled uncalibrated model (a) versus model corrected through ground controlThe process purpose (b). of this study was to test the “worst case scenario” for 3D models created with SeaArray and to gain a sense of how much work was needed on a site to capture an acceptably accurate The purpose of this study was to test the “worst case scenario” for 3D models created with SfM model. In some ways, the presence of the bowling effect helped put this to the test by creating an SeaArray and to gain a sense of how much work was needed on a site to capture an acceptably even less than ideal scenario than initially imagined by the team for testing the accuracy and precision accurate SfM model. In some ways, the presence of the bowling effect helped put this to the test by of measurements taken with a SeaArray model. creating an even less than ideal scenario than initially imagined by the team for testing the accuracy and4.3. precision Photogrammetry of measurements Models vs. taken Hand with Mapping a SeaArray model.

4.3. PhotogrammetryHolt [21] conducted Models vs. a study Hand assessingMapping the accuracy and precision of underwater archaeological survey using traditional tape measurements of up to 20 m. He found that three-dimensional trilateration usingHolt a [21] tape conducted measure resulted a study in asse a positionssing the accuracy accuracy of and+/ 4.0precision cm. Of of the underwater 304 measurements archaeological collected − surveyas part using of his traditional study, 20% tape of themmeasurements were in errorof up (i.e., to they20 m. exceeded He found the that 4 cm three-dimensional positional accuracy trilaterationtaken as acceptable). using a tape This measure was surprising resulted andin a challenges position theaccuracy normal of assumption +/−4.0 cm. ofOf accuracy the 304 for measurementsunderwater surveys.collected Additionally, as part of his the study, author 20% found of them that were there in was error no (i.e. correlation, they exceeded between the the 4 size cm of positionala measurement accuracy error taken and as theacceptable). measurement This was length, surprising meaning and that challenges large errors the arenormal likely assumption to appear as of oftenaccuracy as small for underwater ones regardless surveys. of measurement Additionally, length. the author found that there was no correlation betweenIf the Holt’s size [21 of] a value measurement is considered error an and acceptable the measurement amount of error,length, when meaning applied that to large the data errors collected are likelyat the to appear East Key as Constructionoften as small Wreck,ones regardless then an acceptableof measurement measurement length. variation where the longest measurementIf Holt’s [21] is slightlyvalue is more considered than 60 man would acceptable be 12cm, amount putting ofthe error, East when Key Construction applied to Wreckthe data with collectedan average at the error East of Key 17.4 Construction cm outside theWreck, “acceptable” then an acceptable level of error. measurement However, thisvariation is a straight where linearthe longestinterpolation measurement of Holt’s is slightly values andmore ignores than 60 the m realitieswould be of 12 trying cm, putting to pull very the East long Key tape Construction measurements Wreckunderwater with an andaverage does error not take of 17.4 into cm account outside our the statistical “acceptable” analysis. level It of is error. likely However, that traditional this is tapea straightmeasurements linear interpolation for distances of Holt’s exceeding values the and 20 mignores used inthe Holt’s realities test of case trying would to pull generate very long larger tape error measurementsvalues. It is unlikely underwater that archaeologists and does not would take into be able account to map our a site statistical of this size analysis. by hand It withoutis likelymaking that traditionala similar tape level measurements of error when suchfor distances things as exceeding current, tape the measure20 m used slack in overHolt’s distance, test case and would objects generateobtruding larger into error the values. path of It the is tapeunlikely measure that arearch takenaeologists into account.would be able to map a site of this size by handThe without level ofmaking variation a similar seen level by Holt of error [21] alsowhen occurred such thin duringgs as current, the hand-measurement tape measure slack process over of distance,the East and Key objects Construction obtruding Wreck. into the Archaeologists path of the tape measured measure and are recordedtaken into the account. same line of cement barrelsThe level twice of while variation on site, seen once by whileHolt [21] taking also the occurred overall during length the of the hand-measurement line of barrels, and process once while of themeasuring East Key theConstruction distance from Wreck. barrel Archaeologists to barrel within measured the line, and shown recorded in Figure the 10 sameb. The line result of cement was two barrelsdifferent twice lengths. while on When site, measuringonce while thetaking total the length, overall the length archaeologists of the line recorded of barrels, 710 and cm. once In contrast,while measuringwhen measuring the distance the lengthfrom barrel of the to line barrel from within barrel the to barrel,line, shown the team in Figure recorded 10b. a The total result length was of 712two cm different(Table2 lengths.). This indicates When measuring a variation the of total two centimeterslength, the archaeologists between measurements recorded and,710 cm. again, In contrast, points to a whenlevel measuring of inherent the uncertainty length of the in measurements line from barrel taken to barrel, underwater the team even recorded over relatively a total length short distancesof 712 cmand (Table under 2). This almost indicates ideal conditions. a variation of two centimeters between measurements and, again, points to a level of inherent uncertainty in measurements taken underwater even over relatively short distances and under almost ideal conditions.

J. Mar. Sci. Eng. 2020, 8, 849 16 of 20 J. Mar. Sci. Eng. 2020, 8, x FOR PEER REVIEW 15 of 19 J. Mar. Sci. Eng. 2020, 8, x FOR PEER REVIEW 15 of 19

(a) (b) (a) (b) FigureFigure 10. 10.Line Line ofof barrelsbarrels measured as as depicted depicted in in Viscore Viscore (a ()a and) and as as measured measured by byhand hand (b). ( b). Figure 10. Line of barrels measured as depicted in Viscore (a) and as measured by hand (b). WithWith the the East East Key Key ConstructionConstruction Wreck, we we have have a a unique unique opportunity opportunity to tocompare compare traditional traditional documentationdocumentationWith the East resultsresults Key with withConstruction SfMSfM documentation.documentation. Wreck, we have NPS NPS a archaeologistsunique archaeologists opportunity and and studentsto studentscompare from fromtraditional Brown Brown UniversityUniversitydocumentation mapped mapped results the the site withsite by hand bySfM hand indocumentation. 1990 in over1990 aover span NPSa ofspan severalarchaeologists of several weeks, resulting weeks,and students resulting in a two-dimensional, from in aBrown two- University mapped the site by hand in 1990 over a span of several weeks, resulting in a two- traditionaldimensional, archaeological traditional sitearchaeological plan, shown site superimposed plan, shown oversuperimposed the photogrammetry over the photogrammetry model in Figure 11 modeldimensional, in Figure traditional 11 over thearchaeological traditional map.site plan, This shownmap was superimposed superimposed over on thethe photogrammetrySfM model and over the traditional map. This map was superimposed on the SfM model and adjusted to a best fit of adjustedmodel in to Figure a best 11fit overof features the traditional by eye—i.e. map., the This Brown map University was superimposed map was movedon the SfMuntil modelit had theand features by eye—i.e., the Brown University map was moved until it had the best apparent visual overlap bestadjusted apparent to a visualbest fit overlap of features with by the eye—i.e. SfM model., the Brown As can University be seen from map Figure was moved11, the untiltwo mapsit had did the with the SfM model. As can be seen from Figure 11, the two maps did not cohere well. This illustrates notbest cohere apparent well. visual This overlap illustrates with severalthe SfM deficiencies model. As canin bethe seen basemap from Figureand highlights 11, the two what maps SfM did several deficiencies in the basemap and highlights what SfM technology can contribute to traditional technologynot cohere can well. contribute This illustrates to tradit ionalseveral mapping. deficiencies It is apparent in the basemapfrom the SfMand overlay highlights that manywhat siteSfM mapping. It is apparent from the SfM overlay that many site features are missing from the hand-drawn featurestechnology are missingcan contribute from the to hand-drawntraditional mapping. map, an dIt thereis apparent is disagreement from the SfM between overlay the thatplacement many site of map,somefeatures and features thereare missing isondisagreement the from hand-drawn the hand-drawn between map andthe map, placementSfM an dmodel. there ofis The disagreement some point features of the between on comparison the the hand-drawn placement is not to mapof anddenigratesome SfM features model. the work Theon thedone point hand-drawn by of the the team comparison map in 1990, and is butSfM not rather model. to denigrate to contrastThe point the how work of muchthe done comparison the by techniques the team is not inand 1990,to buttechnologydenigrate rather to the contrastof SfMwork modeling howdone muchby havethe the team revolutionized techniques in 1990, but and documentation rather technology to contrast of ofSfM underwater how modeling much sites. the have techniques revolutionized and documentationtechnology of ofSfM underwater modeling have sites. revolutionized documentation of underwater sites.

Figure 11. NPS hand drawn site plan from 1990 superimposed over the SfM model orthophoto. FigureFigure 11. 11.NPS NPS hand hand drawn drawn sitesite planplan fromfrom 1990 supe superimposedrimposed over over the the SfM SfM model model orthophoto. orthophoto. It is, of course, understandable that the traditional map does not include many of the elements of theIt is,It site, is, of ofseafloor, course, course, understandableand understandable coral matrixthat inthat which thethe traditionaltraditio the sitenal is embedded. map map does does not Gathering not include include the many many data of on ofthe thewhich elements elements the ofhand-drawnof the the site, site, seafloor, seafloor, map is and basedand coralcoral requires matrix a significant in in which which the amount the site site is of embedded. is time embedded. and effort,Gathering Gathering and decisionsthe data the on data about which on what which the theishand-drawn hand-drawnimportant enough map map is basedto is map based requires and requires what a significant is a not significant are baamountlanced amount of against time of and timethe effort,time, and staff, eandffort, decisions and and other decisions about resources what about whatavailable.is important is important With enough SeaArray, enough to map tothe and map same what and considerationsis what not are is ba notlanced areapply, balanced against but thea againstsite time, the staff, the size time,and of otherthe sta ffEast resources, and Key other available. With SeaArray, the same considerations apply, but a site the size of the East Key

J. Mar. Sci. Eng. 2020, 8, 849 17 of 20 resources available. With SeaArray, the same considerations apply, but a site the size of the East Key Construction Wreck can be recorded in a matter of hours in much greater detail than it could with weeks of hand mapping. In the span of two hours, a team of two mapped the East Key Construction Wreck. It is demonstrably faster and arguably more accurate than traditional approaches. The resulting model is visually and interpretively compelling, and it offers a tremendous amount of utility for archeologists, resource managers, and the general public. Moreover, there are secondary benefits. In terms of operations, it is cheaper and less labor intensive, in that the map data can be gathered by a smaller team with less time in the water. In terms of product and utility, the thousands of images that are collected to create a model can be used repeatedly as new questions arise, and the data can be reprocessed in future years as hardware, software, and model and rectification algorithms advance. In some very real sense, the East Key Construction Wreck site has been “digitally preserved” for future generations of managers, archeologists, and visitors to Dry Tortugas National Park. While SfM technology promises to fundamentally change how some shipwrecks are mapped and interpreted, it entails some compromises and commitments that should be acknowledged. First and foremost is that researchers trade speed and ease of documentation for close-up detailed understanding of underwater sites. The traditional process of hand mapping and underwater drawing, while slow, imparts a tremendous amount of information to those engaged in it. Slow, careful examination of sites, features, and artifacts leads to more nuanced interpretations of site formation processes, use-life, and repairs that are keys to evidence-based scientific statements about the past. The use of SfM as a documentation tool does not replace valuable time spent underwater on a site by an archaeologist, but allows NPS SRC to better handle our mandate of protecting 1.4 million hectares of submerged land. NPS SRC has embraced SfM documentation techniques and technology for small- and large-scale baseline site documentation because of our vast mandate and relatively small team. While we acknowledge the archeological utility of hand mapping, we made a conscious decision after weighing both the pros and the cons, to move forward with SfM models as a key technology for site management and interpretation.

4.4. Lessons Learned and Future Research A pitfall of SfM photogrammetry is that because the models yield such visually compelling images, it is sometimes difficult to be skeptical of the accuracy of the results they produce. The work of marrying SfM models to RTK survey points as described in this article was specifically designed to both challenge our assumptions of accuracy and provide quantifiable data about the accuracy of the models produced. There are a number of ways in which NPS SRC is committed to improving the accuracy of SeaArray SfM models. Largely due to the appearance of the bowling effect in the East Key Construction Wreck data set and the potential for this issue to arise on other sites in the future, the team acknowledges that calibration of SeaArray’s cameras is an absolutely essential step to ensuring archaeologically accurate SfM models in the future. NPS SRC is, at time of publication, designing a 4.99 m 2.13 m camera × calibration grid for SeaArray that will be deployable in the field, and plans to use this grid prior to each SeaArray project. NPS SRC has also purchased an Underwater Information Systems (UWIS) underwater GPS system that will be integrated with SeaArray’s cameras to provide real-time GPS locations for SeaArray’s photos. Additionally, NPS SRC is conducting further research into the importance of EXIF data in the Agisoft Metashape alignment process, as well as lens distortion and pixel size specific to SeaArray cameras, and how this information is conveyed internally to Agisoft Metashape. It is possible that the results produced here could be reproduced with a single camera photogrammetry system, as opposed to SeaArray’s three cameras. However, use of a single camera increases the amount of time that must be spent underwater collecting data. The advantage of SeaArray’s three cameras is that the system collects three times the data in the same time frame it J. Mar. Sci. Eng. 2020, 8, 849 18 of 20 would take to fly a single camera system around the same site. The more overlapping images that are collected, the greater the chance of achieving successful alignment with excellent coverage.

5. Conclusions This study sought to determine the accuracy of 3D models created with the SeaArray platform by comparing measurements from an uncalibrated “worst case scenario” configuration to those derived from a state of the art RTK GPS survey of the same site, as well as to evaluate the amount of time and resources necessary to capture an acceptably accurate photogrammetry model of a submerged archaeological site. Through statistical analysis, the team determined that at worst-case, in its current iteration, the photogrammetry models created with SeaArray of the East Key Construction Wreck has a margin of error of 5.29 cm on a site over 84 m in length and 65 m in width. In short, based upon our statistical analysis, measurements taken from the SfM model will fall within 5.29 cm of the true value of these same measurements taken on the East Key Construction Wreck site in 95% of all cases. The establishment of a “worst case scenario” baseline allows NPS SRC to continue research on how to create photogrammetry models with the highest levels of accuracy given our equipment configurations. As the National Park Service moves forward with the use of photogrammetry as a tool for archaeological data collection and analysis, we can now state with a quantifiable degree of certainty that 3D models produced by SeaArray are an accurate representation of what lies underwater, but we will continue to improve this methodology. Models that are properly geo-rectified and scaled with calibrated cameras are obviously the ideal, but the case examined here also indicates that, even in a worst case, with uncalibrated lenses, no ground control, and a bowling effect caused by these two factors compounded by homogenous site topography, the resulting SfM model has considerable fidelity to the actual site itself. This has important implications for the ways in which we as archaeologists do archaeology. NPS must often conduct rapid archaeological documentation and site monitoring due to the large number of archaeological sites located in the 1.4 million hectares of submerged public lands in its care. Confidence in the archaeological accuracy of 3D models produced by SeaArray will enable NPS archaeologists to best use SfM photogrammetry to rapidly record sites in the field, while conducting much of the associated analysis at another time, from an office. This will allow for better management of NPS time and resources, and, ultimately, better management of our shared underwater heritage.

Author Contributions: A.E.W.—Lead author, digital data collection and analysis, data curation, writing—original draft. D.L.C.—Project conceptualization, data collection, manuscript revision and editing, preparation of ArcGIS maps. S.M.S.—Statistical analysis, review. All authors have read and agreed to the published version of the manuscript. Funding: This research was funded by the Cultural Resources Projects and Programs fund of the National Park Service. Acknowledgments: The data that was used in this article were gathered by a very talented team of NPS archeologists, surveyors, and photographers. SRC Deputy Chief Brett Seymour operated SeaArray and ran numerous iterations of Agisoft/Viscore processing to create the digital models used in the analysis. NPS GPS coordinator Tim Smith designed the RTK survey, configured and installed both the base station instrumentation and the rover units, and gathered and post-processed the resulting data. Bryce Sprecher of the University of Wisconsin has developed several Metashape Professional processing workflows that allowed the team to process the large SeaArray image data efficiently. NPS Archeologist Joshua Marano assisted with all field operations and RTK work. SRC Photographer Susanna Pershern documented the RTK survey process as it occurred. The authors would also like to acknowledge and thank the staff and senior management of Dry Tortugas National Park and Everglades National Park for supporting this work. In addition, the SRC recognizes our photogrammetry partners. Marine Imaging Technologies led in the engineering, fabrication, and continued technical support of SeaArray. University of California San Diego’s Cultural Heritage Engineering Initiative has proved the analytical and visualization software Viscore. Conflicts of Interest: The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, nor in the decision to publish the results. J. Mar. Sci. Eng. 2020, 8, 849 19 of 20 J. Mar. Sci. Eng. 2020, 8, x FOR PEER REVIEW 18 of 19

AppendixAppendix A

Difference in Measurements Between RTK and Uncalibrated Structure from Motion Model Points 70 RTK 60

50 Uncalibrated 40 SfM Model

30

20

10

0 A A A A A A A A A A A B B B B B B B B B B C C C C C C C C C D D D D D D D D F F F F F F F G G G G G G H H H H H I I I I J J J K K L to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to to B C D F G H I J K L M C D F G H I J K L M D F G H I J K L M F G H I J K L M G H I J K L M H I J K L M I J K L M J K L M K L M L M M

Figure A1. This scatter plot illustrates the difference between the RTK calculated distances and distances calculatedFigure A1. withThis thescatter uncalibrated plot illustrates SfM the model. difference Distances between are inthe meters. RTK calculated distances and distances calculated with the uncalibrated SfM model. Distances are in meters. References References 1. Aragón, E.; Munar, S.; Rodríguez, J.; Yamafune, K. Underwater photogrammetric monitoring techniques for

1. mid-depthAragón, E.; shipwrecks.Munar, S.; Rodríguez,J. Cult. Herit.J.; Yamafune,2018, 34 K., 255–260.Underwater [CrossRef photogrammetric] monitoring techniques for mid-depth shipwrecks. J. Cult. Herit. 2018, 34, 255–260. 2. Balletti, C.; Beltrame, C.; Costa, E.; Guerra, F.; Vernier, P. 3D reconstruction of marble shipwreck cargoes based 2. Balletti, C.; Beltrame, C.; Costa, E.; Guerra, F.; Vernier, P. 3D reconstruction of marble shipwreck cargoes on underwater multi-image photogrammetry. Digit. Appl. Archaeol. Cult. Herit. 2016, 3, 1–8. [CrossRef] based on underwater multi-image photogrammetry. Digit. Appl. Archaeol. Cult. Herit. 2016, 3, 1–8. 3.3. Boyer,Boyer, T.; Lockhart, B. B. Seeing Seeing Life Life in 3D: in 3D:Digital Digital Recording Recording of HMS of Erebus. HMS Erebus. Advis. Counc.Advis. Underw. Counc. Archaeol Underw. Archaeol. 20202020, in press. press. 4.4. Henderson,Henderson, J., J.; Pizarro, O., O.; Johnson-Roberson, Johnson-Roberson, M.; M.; Mahon, Mahon, I. Mapping I. Mapping Submerged Submerged Archaeological Archaeological Sites Sites using Stereo-Visionusing Stereo-Vision Photogrammetry. Photogrammetry.Int. Int. J. Naut.J. Naut. Archaeol. Archaeol. 2013, ,5858, ,243–256. 243–256. [CrossRef] 5.5. Liarokapis,Liarokapis, F., F.; Kouˇril,Kouřil, P.; Agrafiotis,Agrafiotis, S.; S.; Demesticha, Liarokapis, S.; ChmelChmelík,ík, J.; J.; Skarlatos, D. D. 3D 3D Modelling Modelling and and Mapping forMapping Virtual for Exploration Virtual Exploration of Underwater of Underwater Archaeology Archaeology Assets. Assets.Int. Arch. Int. Arch. Photogramm. Photogramm. Remote Remote Sens. Sens. Spat. Inf. Sci. Spat. Inf. Sci. 2017, 42, 425–431. 2017, 42, 425–431. [CrossRef] 6. McCarthy, J.; Benjamin, J. Multi-image Photogrammetry for Underwater Archaeological Site Recording: 6. McCarthy, J.; Benjamin, J. Multi-image Photogrammetry for Underwater Archaeological Site Recording: An Accessible, Diver-Based Approach. J. Marit. Archaeol. 2014, 9, 95–114. J. Marit. Archaeol. 2014 9 7. AnPacheco-Ruiz, Accessible, R.; Diver-Based Adams, J.; Pedrotti, Approach. F.; Grant, M.; Holmlund, J.; Bailey,, , C. 95–114. Deep sea [CrossRef archaeological] survey 7. Pacheco-Ruiz,in the Black Sea R.;– Robotic Adams, documentation J.; Pedrotti, F.;of 2,500 Grant, years M.; of Holmlund, human seafaring. J.; Bailey, Deep C. Sea Deep Res. seaPart. archaeological I Oceanogr. survey inRes. the Pap. Black 2019 Sea, 152 –, 1–16 Robotic. documentation of 2,500 years of human seafaring. Deep Sea Res. Part I Oceanogr. 8. Res.Yamafune, Pap. 2019 K.; Castro,, 152, 1–16. F. Multi-Image [CrossRef Photogrammetry] to Record and Reconstruct Underwater Shipwreck 8. Yamafune,Sites. J. Archaeol. K.; Castro, Method F.Theory Multi-Image 2017, 23, 703–725. Photogrammetry to Record and Reconstruct Underwater Shipwreck 9. Sites.Prado,J. E., Archaeol. Gómez-Ballesteros, Method Theory M., Cobo,2017 A.,, 23 Sánchez,, 703–725. F., [Rodriguez-Basalo,CrossRef] A., Arrese, B., Rodríguez-Cobo, 9. Prado,L. 3D Modeling E.; Gómez-Ballesteros, of Rio Miera Wreck M.; Cobo,Ship Merging A.; Sánchez, Optical F.; and Rodriguez-Basalo, Multibeam High Resolution A.; Arrese, Points B.; Rodr Cloud.íguez-Cobo, L. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2019, 42, 159–165. 3D Modeling of Rio Miera Wreck Ship Merging Optical and Multibeam High Resolution Points Cloud. 10. Kan, H., Katagiri, Y., Yoshizaki, S., Nagao, M., Ono, R. Assessment and Significance of a World War II Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2019, 42, 159–165. [CrossRef] battle site: Recording the USS Emmons using a High-Resolution DEM combining Multibeam Bathymetry 10. Kan,and SfM H.; Photogrammetry. Katagiri, Y.; Yoshizaki, Int. J. Naut. S.; Nagao, Archaeol. M.; 2018 Ono,, 47, R.267–280. Assessment and Significance of a World War II battle 11. site:Meredith-Williams, Recording the M.G., USS EmmonsHaussman,using N., Bailey, a High-Resolution G.N., King, G.C.P., DEM Alsharekh, combining A., Multibeam Al Ghamdi, Bathymetry S., Inglis, and SfM Photogrammetry.R.H. Mapping, modellingInt. J. Naut.and predicting Archaeol. prehistoric2018, 47, 267–280.coastal archaeology [CrossRef in] the southern Red Sea using 11. Meredith-Williams,new applications of digital-imaging M.G.; Haussman, techniques. N.; Bailey, World G.N.; Archaeol. King, 2014 G.C.P.;, 46, 10–24. Alsharekh, A.; Al Ghamdi, S.; Inglis, R.H. 12. Mapping,Wu, J., Bingham, modelling R.C., Ting, and S., predicting Yager, K., Wood, prehistoric Z.J., Gambin, coastal T., archaeology Clark, C.M. Multi-AUV in the southern motion Redplanning Sea using new applicationsfor archeological ofdigital-imaging site mapping andtechniques. photogrammetricWorld reconstruction. Archaeol. 2014 J. ,Field46, 10–24.Robot. 2019 [CrossRef, 36, 1250–1269.]

12. Wu, J.; Bingham, R.C.; Ting, S.; Yager, K.; Wood, Z.J.; Gambin, T.; Clark, C.M. Multi-AUV motion planning for archeological site mapping and photogrammetric reconstruction. J. Field Robot. 2019, 36, 1250–1269. [CrossRef] J. Mar. Sci. Eng. 2020, 8, 849 20 of 20

13. Repola, L.; Scotto di Carlo, N.; Signoretti, D.; Leidwanger, J. Virtual simulation of a late antique shipwreck at Marzamemi, Sicily: Integrated processes for 3D documentation, analysis and representation of underwater archaeological data. Archaeol. Prospect. 2018, 25, 99–109. [CrossRef] 14. Skarlatos, D.; Menna, F.; Nocerino, E.; Agrafiotis, P. Precision Potential of Underwater Networks for Archaeological Excavation Through Trilateration and Photogrammetry. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2019, 42, 175–180. [CrossRef] 15. Barry, P.; Coakley, R. Field accuracy test of RPAS photogrammetry. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2013, 40, 27–31. [CrossRef] 16. Uysal, M.; Toprak, A.S.; Polat, N. DEM generation with UAV Photogrammetry and accuracy analysis in Sahitler hill. Measurement 2015, 73, 539–543. [CrossRef] 17. Murphy, L.E.; Brewer, D.M.; Cockrell, W.A.; Day, F.E.; Davis, G.E.; Gould, R.A.; Jonsson, R.A.; Stone, P.A.; Sturges, W.; Souza, D.J.; et al. Dry Tortugas National Park Submerged Cultural Resources Assessment. U.S. Dep. Inter. Natl. Park Serv. 1993, 2, 297–313. 18. Agisoft Metashape User Manual. Available online: https://www.agisoft.com/pdf/metashape-pro_1_6_en.pdf (accessed on 15 April 2020). 19. Agisoft. The Mysterious Bowl Effect. Available online: https://www.agisoft.com/forum/index.php?topic= 600.0#:~{}:text=Re%3A%20The%20mysterious%20$’$bowl%20effect\T1\textquoteright&text=Hello% 20Patrice%2C,and%20fixed%20during%20photo%20alignment (accessed on 2 May 2020). 20. Kuester, F.; Cultural Heritage Engineering Initiative, University of California, San Diego, CA, USA. Personal communication, 2020. 21. Holt, P. An assessment of quality in underwater archaeological surveys using tape measurements. Int. J. Naut. Archaeol. 2003, 32, 246–251. [CrossRef]

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).