Vision-Based Shipwreck Mapping: on Evaluating Features Quality and Open Source State Estimation Packages A. Quattrini Li, A. Coskun, S. M. Doherty, S. Ghasemlou, A. S. Jagtap, M. Modasshir, S. Rahman, A. Singh, M. Xanthidis, J. M. O’Kane and I. Rekleitis Computer Science and Engineering Department, University of South Carolina Email: [albertoq,yiannisr,jokane]@cse.sc.edu, [acoskun,dohertsm,sherving,ajagtap,modasshm,srahman,akanksha,mariosx]@email.sc.edu Abstract—Historical shipwrecks are important for many rea- sons, including historical, touristic, and environmental. Cur- rently, limited efforts for constructing accurate models are performed by divers that need to take measurements manually using a grid and measuring tape, or using handheld sensors. A commercial product, Google Street View1, contains underwater panoramas from select location around the planet including a few shipwrecks, such as the SS Antilla in Aruba and the Yongala at the Great Barrier Reef. However, these panoramas contain no geometric information and thus there are no 3D representations available of these wrecks. This paper provides, first, an evaluation of visual features quality in datasets that span from indoor to underwater ones. Second, by testing some open-source vision-based state estimation packages on different shipwreck datasets, insights on open chal- Fig. 1. Aqua robot at the Pamir shipwreck, Barbados. lenges for shipwrecks mapping are shown. Some good practices for replicable results are also discussed. demonstrations. However, applying any of these packages on a I. INTRODUCTION new dataset has been proven extremely challenging, because Historical shipwrecks tell an important part of history and of two main factors: software engineering challenges, such at the same time have a special allure for most humans, as as lack of documentation, compilation, dependencies; and exemplified by the plethora of movies and artworks of the algorithmic limitations—e.g., special initialization motions for Titanic. Shipwrecks are also one of the top scuba diving monocular cameras, number of and sensitivity to parame- attractions all over the world, see Fig. 1. Many historical ters [5]. Also, most of them are usually developed and tested shipwrecks are deteriorating due to warm, salt water, human with urban settings in mind. interference, and extreme weather (frequent tropical storms). This paper analyzes first different feature detectors and Constructing accurate models of these sites will be extremely descriptors in several datasets taken from indoor, urban, and valuable not only for the historical study of the shipwrecks, underwater domains. Second, some open source packages for but also for monitoring subsequent deterioration. Currently, visual SLAM are evaluated. The main contribution of this limited mapping efforts are performed by divers that need to paper is to provide, based on this evaluation, insights on the take measurements manually using a grid and measuring tape, open challenges in shipwreck mapping so that when designing or using handheld sensors [1]—a tedious, slow, and sometimes a new mapping algorithm they are taken into consideration. dangerous task. Automating such a task with underwater The next section discusses research on shipwreck mapping. robots equipped with cameras—e.g., Aqua [2]—would be Section III presents an analysis of the visual feature quality. extremely valuable. Some attempts have been performed by Section IV shows qualitative results of some visual SLAM al- using underwater vehicles with expensive setup—e.g., Remote gorithms. Finally, Section V concludes the paper by discussing Operated Vehicles (ROV) [3], [4]. some insights gained by this methods evaluation. Autonomous mapping using visual data has received a lot of II. RELATED WORK attention in the last decade, resulting in many research papers and open source packages published, supported by impressive Different technologies have been used to survey shipwreck areas, including ROVs, AUVs, and diver held sensors. Nornes 1https://www.google.com/maps/streetview/#oceans et al. [6] acquired from an ROV stereo images off the coast of Trondheim Harbour, Norway, where M/S Herkules shipwreck of images from several datasets from different domains (see sank. A commercially available software has been used to Fig. 2), including: process the images to reconstruct a model of the shipwreck. indoor environment, collected with a Clearpath Husky • In [3] a deepwater ROV is adopted to map, survey, sample, and equipped with a camera; excavate a shipwreck area. Sedlazeck et al. [4], preprocessing outdoor urban environments, specifically the KITTI • images collected by an ROV and applying a Structure from dataset [17]; Motion based algorithm, reconstructs a shipwreck in 3D. The outdoor natural environments, in particular the Devon • images used for testing such an algorithm contained some Island rover navigation dataset [18]; structure and a lot of background, where only water was coral reefs, collected by a drifter equipped with a monoc- • visible. ular camera [19]; Other works use AUVs to collect datasets and build ship- and shipwrecks, collected with Aqua2 underwater robot • wrecks models. Bingham et al. [7] used the SeaBED AUV to equipped with front and back cameras. build a texture-mapped bathymetry of the Chios shipwreck site in Greece. Gracias et al. [8] deployed the Girona500 AUV for Figs. 3-7 show: (a) the average number of detected features, surveying the ship ‘La Lune’ off the coast of Toulon, France. (b) the number of inliers used for the estimated homographies, Bosch et al. [9] integrated an omnidirectional camera to an and (c) the number of images matched together. Each figure AUV to create underwater virtual tours. presents the above measure for the different datasets used, with different combinations of feature detectors and descriptors Integrating inertial and visual data helps better estimating 2 the pose of the camera, especially in underwater domain, available in OpenCV . Note that the number of features where images are not as reliable as in ground environments. detected is a single frame is not necessarily a measure of Hogue et al. [10] demonstrated shipwreck reconstruction with how good a feature is. Many features cannot be matched with the use of a stereo vision-inertial sensing device. Moreover, features in subsequent frames, thus they do not contribute structured lights can provide some extra information to recover to the robot localization and the environment reconstruction. structure information of the scene. In [11], structured light was Other features are not stable changing location over time. used to aid the reconstruction of high resolution bathymetric Indeed, even if some methods are able to find many features, maps of underwater shipwreck sites. the number of inliers is generally relatively low. Some of the Methods to process such datasets are becoming more and combinations of feature detector and descriptor extractor are more reliable. Campos et al. [12] proposed a method to recon- not present in the figures, because no feature could be found. struct underwater surface by using range scanning technology. In indoor environment, there are several combinations of Given raw point sets, smooth approximations of surfaces to feature detector/descriptor extractor that work well and the triangle meshes are performed. The method was tested on features are quite stable as can be observed by the low standard several datasets, including the ship La Lune. Williams et deviation in the number of images matched; see Fig. 3. al. [13] described techniques to merge multiple datasets, which In outdoor urban environments, the distribution of number include stereo images of a shipwreck off the coast of the Greek of detected features and the number of inliers is similar to island of Antikythera, collected during different expeditions. the one in indoor environment; see Fig. 3 and Fig. 4. This However, usually these works collect data by teleoperating similarity can be explained by the fact that both classes the robot and process the data offline. To automate the explo- of environments are quite rich in features and are similarly ration task real-time methods for localizing the robot and at structured. However, the number of matched images decreases the same time mapping the environment are necessary. in the urban environment compared to the results from the indoor ones. One of the reasons is that in urban environments III. VISUAL FEATURE QUALITY dynamic obstacles, such as cars and bikers, are often present in There are two main classes of visual SLAM techniques: the scene, violating the common assumption of static scenes. sparse and dense. Sparse visual SLAM utilizes selected fea- In outdoor natural environment datasets, while the number tures to track the motion of the camera and reconstruct the of features detected is high as there are many rocks in the scene. Dense visual SLAM uses segments of the image and scene, the number of inliers drops; see Fig. 5. The probability attempts to reconstruct as much of the scene as possible. of mismatches is higher, given the fact that the terrain with In [14] SURF features are used for localizing and mapping rocks does not have any distinctive feature. This happens also scenes in the underwater domain. Thus, it is important to in the coral reef dataset. The combinations of feature detector identify stable features to use. The quality of some feature and descriptor extractor have different distributions for the detectors, descriptors, and matchers in underwater domain is coral reef dataset compared to the results for the above water assessed by Shkurti et al. [15]. The influence of underwater ones. conditions such as blurring and illumination changes has been In shipwreck datasets, the features number varies a lot studied by Oliver et al. [16]. over images—i.e., high variance in the number of detected In the following, feature detectors and descriptors that are features—compared to the other datasets; see Fig. 7. One of available as open source implementation in OpenCV are tested using the default parameters. The tests are run on a subset 2http://opencv.org Fig.