Real-Time Marine Snow Noise Removal from Underwater Video Sequences

Real-time marine snow noise removal from underwater video sequences

Bogusław Cyganek Karol Gongola

Bogusław Cyganek, Karol Gongola, “Real-time marine snow noise removal from underwater video sequences,” J. Electron. Imaging 27(4), 043002 (2018), doi: 10.1117/1.JEI.27.4.043002. Journal of Electronic Imaging 27(4), 043002 (Jul∕Aug 2018)

Real-time marine snow noise removal from underwater video sequences

Bogusław Cyganek* and Karol Gongola AGH University of Science and Technology, Department of Electronics, Krakow, Poland

Abstract. Underwater images suffer from various degradation factors, such as blur, haze, color degradation, and marine snow. Marine snow is a type of noise, caused mostly by biological particles that fall into the ocean bottom, and which impedes proper object detection in underwater vision. A method for real-time marine snow removal from underwater color and monochrome video is presented. It is based on the proposed marine snow model, spatiotemporal patch analysis, and three-dimensional median filtering. The method was evaluated on a number of real underwater sequences endowed with the hand-annotated ground-truth data which were made available from the Internet. As shown by the experiments, the method attains high accuracy and performs in real time. © 2018 SPIE and IS&T [DOI: 10.1117/1.JEI.27.4.043002]

Keywords: marine snow filtering; underwater image enhancement; real-time image filtering; remotely operated underwater vehicle. Paper 180201 received Mar. 7, 2018; accepted for publication Jun. 11, 2018; published online Jul. 5, 2018.

1 Introduction colleague biologists. The phenomenon of marine snow, or “ ” 8 Underwater image acquisition and processing find broad later called organic aggregates by Riley, has found interest interest in such areas as underwater exploration, inspection in biological sciences. Their role in the ocean ecosystem, as of underwater constructions, and underwater navigation, to for example food conveying or for various organisms, has name a few. However, their processing puts much higher been understood and appreciated for years of their study. In this respect, an interesting reading is the paper by demands than processing of air space images due to under- 9 water physical conditions, such as contrast and color decay, Suzuki and Kato summarizing their studies in the 1950s on suspended materials in the sea near Hokkaido. On the light scattering, blur, haze, and various types of noise. They 10 cause image quality degradation and lead to loss of conveyed other hand, in the 1980s Orzech and Nealson conducted information. There are many methods which allow for filter- research on measuring bioluminescence of marine snow ing of these unwanted effects. However, there is a special and its effect on the optical properties of the sea, as reported in their paper. An absorbing overview of research on marine type of noise, called marine snow, which greatly affects qual- snow, from the biological point of view, is provided in the ity of the underwater images and is difficult to filter out. paper by Silver.11 On the other hand, in the recent years vari- Marine snow is an effect caused by light back scattering ous methods of underwater image analysis and enhancement from small organic and mineral particles and air bubbles. have been proposed. The books by Duntley12 and the one by When falling down to the water basin, particles grow, Jerlov13 discuss basic physical properties of light propaga- which manifest in images as bright spots of various shapes tion in water conditions. McGlamery14,15 conducted research and sizes, which to some extent resemble snowflakes, as into analysis and simulation of an underwater camera system shown in exemplary frames in Fig. 1. Only recently this and laid out theoretical foundations of the radiometric model phenomenon found interest among researchers in order to of underwater image formation. Then Jaffe16 proposed exten- develop efficient methods of its elimination.1,2 However, — sion aimed at the design of the subsea image acquisition sys- the problem is not a trivial one the particles can be quite tem with optimized contrast and minimized backscattering large, of different structural and lighting characteristics, effect. This way, the Jaffe–McGlamery underwater image that make their statistical properties significantly different formation model has been derived. Recently, Jaffe17 pub- from other types of noise encountered in digital images. lished a comprehensive description of underwater optical im- Because of this, the classical linear and nonlinear filtering aging in the context of physical, biological, technological, methods, such as the averaging or median filters, usually and historical aspects. In this paper, the fundamental limits cannot be used to remove such type of noise from the imposed by the water environment are discussed and related 3,4 images. Thus, there is a need for development of models to the recent technological achievements. A slightly simpli- and methods of marine snow filtering, such as the one pro- fied version of the Jaffe–McGlamery underwater image posed in this paper. formation model was proposed by Trucco and Olmos- The term “marine snow” was coined after the Beebe’s Antillon18 for the self-tuning image restoration filter. An observations in Bathysphere,5,6 conducted in the 1930s, overview of the underwater optical systems is provided in then published in the book by Carson,7 one of the Beebe’s the paper by Kocak et al.19 Similarly, in the paper by Schettini and Corchs,20 theory and recent methods of

Journal of Electronic Imaging 043002-1 Jul∕Aug 2018 • Vol. 27(4) Cyganek and Gongola: Real-time marine snow noise removal from underwater video sequences

Fig. 1 Examples of marine snow in real underwater scenes. Marine snow manifests as lighter oval or rectangular and usually fast-moving shapes. underwater image restoration and enhancement are video to remove marine snow. Last but not least, for quanti- described. On the other hand, a method for single underwater tative evaluation and further comparisons, we provide under- image enhancement was recently proposed by Chongyi water sequences with hand-annotated ground-truth marine et al.21 In their method, numerical optimization is used for snow particles. color cast removal, followed by the visibility and contrast A significant achievement would be construction of an restoration based on the inherent relationship of medium appropriate marine snow model which would reflect various transmission maps of three color channels. Another interest- physical and biological conditions of the particles and the ing group of algorithms for smoothing and enhancement of environment. Such a model would be useful also in measur- underwater images, based on partial differential equations, ing quality of the marine snow filtering. In this respect, Slade was proposed by Nnolim.22 In the recent paper by Sánchez- et al.24 explored effects of particle aggregation and disaggre- Ferreira et al.,23 an algorithm for underwater image restora- gation on their inherent optical properties. They empirically tion is proposed which operates by evolutionary estimation investigated the role that aggregation plays in determining of the parameters of the underwater image formation model properties of the particle light scattering in coastal waters. using two quality metrics. After that, Boffety and Galland25 proposed a phenomeno- However, as alluded to previously, there are only few logical marine snow model for optical underwater image works which directly address the problem of marine snow simulation especially aimed at underwater color restoration. removal. In the paper by Banerjee et al.,1 a variant of the They argue that the simple model obtained by generation of median filtering, based on probability of existence of marine a salt-and-pepper noise does not take into account various snow, is proposed. In this method, high luminance pixels are physical conditions, such as water absorption, particles detected in each patch extracted from the image. Marine shapes and sizes, and signal backscattering by the particles. snow is modeled by the probability of observing sparse num- Therefore, Boffety and Galland proposed a simplified ber of high-intensity pixels two times in a patch and its approach, in which macroparticles are assumed to behave doubled size version. If such a probability is high, then like white Lambertian scatterers with the spatial profile of the center pixel of a patch is replaced with the median their reflection coefficient being a Gaussian function. value of the patch. However, this method works only for rel- The rest of this paper is organized as follows. In Sec. 2, atively small particles of few pixels, whereas marine snow the principles of the proposed method are presented. We start can manifest as much larger structures, as will be discussed. with the marine snow particle model, presented in Sec. 2.1, A modification to this method was proposed by Farhadifard followed by a detailed description of the marine snow detec- et al.2 They also follow the idea of median filtering after tion in Sec. 2.2, and its filtering method described in Sec. 2.3. supervised noise detection by a multiscale patch-based Experimental results are presented in Sec. 3. This paper ends approach. However, both of the above methods are limited with conclusions in Sec. 4. since they operate exclusively in spatial domain and only on 2 Method Description single frames not considering temporal relations. On the other hand, when processing underwater videos, a more ver- In this section, the characteristics of marine snow will be pre- satile spatiotemporal analysis is possible, as will be shown. sented. Especially, interesting and not previously exploited In this paper, we propose an efficient method of marine snow feature is the fast movement behavior of the marine snow elimination, which relies on analysis of spatiotemporal three- particles. dimensional (3-D) patches, i.e., tube-like structures, rather than on flat two-dimensional (2-D) ones. Based on this 2.1 Marine Snow Particle Model idea, marine snow is defined as an outlier in a form of Sea or lake water contains a lot of organic detritus, such as a three layers in a temporal domain, with a bright inner dead plants, animal partitions, fecal matter, bubbles, and layer surrounded by relatively low-intensity outer ones. mineral particles. These are referred to as marine snow, However, special conditions are checked to leave larger because due to the light back scattering20 they look like light objects untouched and avoid false positive (FP) detec- snowflakes.5,7 Frequently, these are small particles, but it tions. Due to this strategy, only relatively small and sparse is not unusual to observe ones reaching several centimeters outliers are detected which are mostly due to marine snow. of diameter, which in an image manifest as blobs of even few This way detected marine snow pixels are then corrected thousands of pixels. Also, their shape can be very different with the 3-D median filter. As a last step in our processing and usually highly acentric. chain, a spatial 2-D median filtering follows. Its purpose is to Their size and shape are just the characteristics that make discard small outliers and other smaller particles and impul- marine snow much different from other types of noise, as sive noise, which remained after the first filtering stage. shown in Fig. 2. Therefore, in this case, standard noise filter- To the best of our knowledge, this is the first method that ing methods, such as the ones for the salt-and-pepper noise considers spatial and temporal information in an underwater removal, usually fail. Last but not least, a characteristic

Journal of Electronic Imaging 043002-2 Jul∕Aug 2018 • Vol. 27(4) Cyganek and Gongola: Real-time marine snow noise removal from underwater video sequences

be discussed, although our proposed method can operate with monochrome images as well. 3. Sparsity, which makes them possible to be distin- guished from larger white objects. 4. Usually, the particles move much faster than other object, so we decided to use video stream to check information on relative changes in consecutive 3-D patches. Fig. 2 Examples of marine snow particles from the Antarctica sequence. The particles can be of various sizes and dimensions, low color saturation, and usually very fast moving in front of the camera due to geometric relations of the light source and camera 2.2 Marine Snow Detection Algorithm plane. The proposed marine snow filtering relies on detection of marine snow particles in 3-D patches of size s × s × t created feature of marine snow is fast movement of the particles in around each pixel of the scene, as shown in Fig. 3. Spatial front of the camera. This effect is due to geometric relations dimensions are denoted by s, while t corresponds to the of light source and camera plane. In the proposed system, we temporal one, i.e., across the frames. Various dimensions exploit all these features for design of an efficient marine were tested but good results were obtained for regions snow filtering method. The aforementioned properties of with squared profile of size s in the range 1 to 7 and temporal marine snow are summarized as follows: dimension t ¼ 3, respectively. Each 3-D patch, as shown in Fig. 3, is processed in a chain 1. The particles are usually lighter than neighboring of rule-based classifiers, as shown in Fig. 4. Each module in objects. a chain performs simple assessment if a pixel surrounded by 2. Color of the particles is limited mostly to the grayscale a 3-D patch can represent marine snow. If so, then such region, i.e., they exhibit low color saturation, as a pixel is filtered, as will be described. compared to other colorful objects. Therefore, when The first step of the processing chain pertains exclusively processing color images, the RGB distance is com- to the color input images, denoted as I ¼ðIR;IG;IBÞ. In this puted to check this characteristic property, as will case, color properties of each 3-D patch are investigated. As alluded to previously, due to the light back scattering, marine snow pixels are characteristic of being close to the grayscale, i.e., they exhibit low color saturation. To verify this condition, the RGB color distance is computed at each pixel position p ¼ðu; vÞ as follows:

EQ-TARGET;temp:intralink-;e001;326;393 px DRGBðpÞ¼jIRðpÞ − IGðpÞj þ jIGðpÞ − IBðpÞj s I s n+1 þjIBðpÞ − IRðpÞj; (1) Pn+1 In Pn Pn-1 where IRðpÞ, IGðpÞ, and IBðpÞ denote red, green, and blue In-1 t color values at the pixel position p, respectively. Then at the central pixel px in the n’th frame of a 3-D patch (i.e., ¯ Fig. 3 Video is processed by analyzing 3-D patches of dimensions the middle patch in Fig. 3), its mean DRGB;nðpxÞ distance is s × s × t. computed as follows:

Fig. 4 Stages of the marine snow detection algorithm. Pixels are processed by the spatiotemporal analysis. For color images, the first block verifies color properties. For monochrome images, this step is omitted. Then the temporal analysis follows: the middle layer of a 3-D “sandwich”-like patch is checked to have significantly higher value than in the outer layers. Finally, in the last step, too small regions are rejected based on a threshold.

Journal of Electronic Imaging 043002-3 Jul∕Aug 2018 • Vol. 27(4) Cyganek and Gongola: Real-time marine snow noise removal from underwater video sequences

EQ-TARGET;temp:intralink-;e002;63;752 1 XN patch Qn chosen in Eq. (2) for computations of the average D¯ p Av D p D p ; RGB;nð xÞ¼ ½ RGB;nð iÞ ¼ N RGB;nð iÞ color distance Eq. (1). i¼1 The role of the last block in the algorithm in Fig. 4 is to R for pi ∈ QðpxÞ; (2) connect pixels into regions i (i.e., connected components) and compute the area of each Ri. The latter is measured as a where Av denotes the averaging operation and QðpxÞ stands number of pixels belonging to a particular region, as follows: p Q for a compact neighborhood of a pixel x in a patch . A R #R ; (8) 2 EQ-TARGET;temp:intralink-;e008;326;686 ð iÞ¼ i The summation parameter N ¼ q denotes a number of pixels in QðpxÞ, which in practice was experimentally where # Ri denotes a number of connected pixels in the i’th chosen to q 32, i.e., the processed regions are squares ¼ region. Then a thresholding is applied to reject too small of 32 × 32 pixels. Then if the following holds areas. That is, an i’th region is accepted only if the following D p − min D¯ p ; D¯ p c; (3) is fulfilled: EQ-TARGET;temp:intralink-;e003;63;628 RGB;nð xÞ ½ RGB;n−1ð xÞ RGB;nþ1ð xÞ < for a certain constant c (in experiments set in the range 0 to Algorithm 1 Marine snow detection and filtering procedure for color p video. For monochrome version, the algorithm starts from the step 7 70), then a pixel x is considered as potential marine snow k 1 and put for further processing. Otherwise, the procedure assuming only one color channel ¼ and all pixels belonging to the set of initial candidates K 1. stops. On the other hand, for monochrome images, all these steps, i.e., Eqs. (1)–(3), are omitted.

In the next verification step, as shown in Fig. 4, particles Input: A video stream Vin with at least three frames, indexed by are checked for their relation within a 3-D patch in order to f ∈ f1;:::g detect fast changing ones. If this is the case, then intensity in the middle layer at level n will be significantly different from Spatial s ∈ f1;:::;7g and temporal t ¼ 3 dimensions of a 3-D patch n − 1 n 1 intensities in the neighboring layers and þ ,as q 32 Q shown in Fig. 3. This relation between intensity in the layers Spatial dimension ¼ of patches was checked by measuring median, maximal, and average Constants c, d, a intensity relations, in order to determine the best and min more reliable function. They are defined as follows: Output: A marine snow filtered output video stream Vout Med P Med I p for p ∈ P ; (4) EQ-TARGET;temp:intralink-;e004;63;442 f ≥ 2 ð n;kÞ¼ f½ n;kð iÞg i n;k 1. For each frame number in Vin do:

n ∈ f − 1;f;f 1 Max P Max I p for p ∈ P ; (5) 2. Get set of frames fIng for f þ g; EQ-TARGET;temp:intralink-;e005;63;411 ð n;kÞ¼ f½ n;kð iÞg i n;k

3. For each pixel location p in {In} do: Av P Av I p for p ∈ P ; (6) EQ-TARGET;temp:intralink-;e006;63;386 ð n;kÞ¼ f½ n;kð iÞg i n;k D 4. Compute set of f RGB;nðpÞg in accordance with Eq. (1); where Pn;k denotes all pixels in the n’th layer and the k’th 5. For each patch Q around pixel location p do: color channel of a 3-D patch, i.e., k ∈ fR; G; Bg, and In;kðpiÞ stands for an intensity of a pixel pi. As already mentioned, P s ∈ 1;:::;7 6. If condition Eq. (3) holds, then add p to the set K 1 of spatial dimensions of n;k are f g, whereas the candidates; temporal one is set to t ¼ 3, i.e., always three frames are considered. In our approach, the condition for a patch to 7. For each pixel location p in K 1 do: be marine snow is defined as follows: 8. Select a 3-D patch PðpÞ of dimensions s × s × t; ∃∶I p − Ω P >d and I p − Ω P >d; EQ-TARGET;temp:intralink-;e007;63;283 kð xÞ ð n−1Þ kð xÞ ð n 1Þ k þ 9. For each color value k do: (7)

10. Compute max values Eq. (5) in the patch layers Pn−1;k and P where d is a threshold value, px is a central pixel of a patch nþ1;k of PðpÞ; Pn in the middle layer of the whole 3-D patch, whereas K ΩðPnÞ denotes one of the operators Eqs. (4)–(6) applied 11. If condition Eq. (7) holds, then add p to the set 2 of candidates; to the Pn layer of the 3-D patch, as shown in Fig. 3. Based on the conducted tests, best results were obtained 12. Compute set of connected components {Ri } (clusters) in K 2; for the maximal value [Eq. (5)] and these are reported in Sec. 3. In the case of monochrome images, denoted as I, 13. For each Ri ∈ fRi g do: only intensity values are taken, so Ik ≡ I. However, when R a R R processing color images, we also observed that good results 14. If area of i is below min then remove i from the set { i }; can be obtained taking exclusively the red channel IR. This is similar to processing monochrome images, but in this case 15. Call the marine snow filtering procedure in Algorithm 2 with we can assume that k ≡ R. Results of these three cases are parameters ({In}, {Ri }); discussed in a further part of this paper. Let us observe that 16. Add a frame returned from the Algorithm 2 to Vout. size of the patch Pn is different (lower) from the size of the

Journal of Electronic Imaging 043002-4 Jul∕Aug 2018 • Vol. 27(4) Cyganek and Gongola: Real-time marine snow noise removal from underwater video sequences

A R >a ; (9) Algorithm 2 Marine snow removal procedure. EQ-TARGET;temp:intralink-;e009;63;752 ð iÞ min where amin is a minimal particle area threshold, set in our Input: A set {In} of three frames experiments in the range 0 to 100 pixels. An influence of its value on the accuracy will be discussed in Sec. 3. The A set {Ri } of connected components of pixels step (9) can be omitted in some variants of the proposed method. However, in the presented system, the reason for Output: A marine snow filtered frame Iout this operation is twofold. First, small square regions can cor- respond to small impulsive noise which will be removed 1. Copy Iout ¼ In; from an image with help of a nonlinear postprocessing, in our system performed with the 2-D median filter. For 2. For each Ri ∈ fRi g do: color images, this is a vector version, and for monochrome, it is a scalar median filtering, as will be discussed. The sec- 3. For each pixel location p in Ri do: ond reason for rejecting small particles is to allow precise measurement of the accuracy with help of the manually 4. Select a 3-D patch PðpÞ of dimensions s × s × t; annotated particles. In other words, it is very difficult and inaccurate, or in some cases even not possible, to manually 5. For each color value k do: outline particles of very small size. Therefore, they were I^ omitted. 6. Compute median value n;k ðpÞ Eq. (4) in the joined patch layers Pn−1;k and Pn 1;k of PðpÞ; Algorithm 1 presents the pseudocode of the complete þ I^ marine snow filtering method adapted to processing of 7. Replace pixel Iout;k ðpÞ with n;k ðpÞ; video streams. After detection, marine snow affected pixels are filtered out, as presented in Sec. 2.3. Such situations are due to fast movement of large marine 2.3 Marine Snow Removal snow particles, which makes undetected their ends in a direc- In Sec. 2.2, a method for marine snow detection in under- tion of their movement. In such a case, Eq. (7) is not fulfilled water video was presented. This is a fast algorithm that oper- for these ends in one of the layers, n þ 1 or n − 1. On the ates exclusively with integer arithmetic. In the next step, to other hand, these two conditions need to be met simultane- get rid of the marine snow, values of the pixels detected as ously to avoid FPs, i.e., false detections of marine snow on marine snow need to be substituted with the surrounding white objects. ones which are assumed to be correct. Finally, the frames Finally, some of the aforementioned remnants and other are optionally filtered to remove small remnants and other impulsive image degradations can be removed with a median types of noise. The filtering chain of the proposed method filter, this time applied only to the 2-D frame In, as shown in is shown in Fig. 5. Fig. 5. In our experiments, the 3 × 3 up to 9 × 9 median For substitution of pixels pi, which were detected as filters for monochrome and color images were applied. For marine snow, a median filtering in the 3-D spatiotemporal a speed up, a modified algorithm, described in the book by domain is proposed. Namely, for each color channel k, Cyganek,3 was used. However, since it blurs some fine tex- we calculate the 3-D median and replace outliers In;kðpiÞ tures, such as the number “20” on a scuba diver’s tank in with median values, computed as follows: Fig. 6, the experimental results discussed in Sec. 3 are related to the system with no final 2-D median filtering. Also, I^ p Med P ∪ P ; (10) EQ-TARGET;temp:intralink-;e010;63;319 n;kð iÞ¼ ð n−1;k nþ1;kÞ other types of filters can be applied at this stage, as will be discussed. where, as previously, Pn;k denotes pixels in an n’th layer for a k’th color channel of a 3-D patch. Thus the 3-D median is computed in each of the 3-D patches, however, taking 3 Experimental Results P P only the n−1;k and nþ1;k layers. In other words, pixels The presented method was implemented in C++ in the ® of the middle layers Pn;k are omitted in this process. Microsoft Visual 2017 environment. Experiments were run Steps of the marine snow removal procedure are presented on a computer equipped with the Intel® i7-3630QM CPU in Algorithm 2. at 2.4 GHz processor, 16 GB RAM, and 64-bit operating Nevertheless, in some cases, the detection and, in effect, system Windows 10. marine snow filtering are not perfect. An example of this Evaluation of marine snow is difficult since, to the best of effect is shown in Fig. 6. our knowledge, there is no ground-truth data available.

Fig. 5 Marine snow filtering after detection by the algorithm shown in Fig. 4. The last step consists of an optional color or monochrome 2-D frame filtering.

Journal of Electronic Imaging 043002-5 Jul∕Aug 2018 • Vol. 27(4) Cyganek and Gongola: Real-time marine snow noise removal from underwater video sequences

Fig. 6 An example of marine snow detection and removal of a large and fast-moving particle. (a) Original image with an enlarged marine snow region. (b) The corresponding filtered region with some remnants, characteristic in a direction of particle movement.

Therefore, we made available two underwater sequences, the first one from the sea of Antarctica (Ant) [Antarctica sequence was taken in the waters of Port Lockroy of Antarctica (64°49′31″S 63°29′40″W).], and the second one from the lake Zakrzowek (Zak). [Zakrzówek is an artificial lake in Kraków, Poland (50°02′11″N 19°54′38″E).] These are real scuba diving sequences containing marine snow of various shape and size. To allow quantitative evaluation and further comparisons, marine snow particles in some frames from both sequences were manually annotated (with the tool from Cyganek and Socha26), which is one of the contributions of our paper. These sequences and the annotated data are available from the Internet.27 Figure 7 shows stages of marine snow filtering in two real underwater color videos. Original frames of Ant-1153 and Zak-4891 are shown in the first row [Figs. 7(a) and 7(b)]. In the next, the hand-annotated ground-truth data are shown [Figs. 7(c) and 7(d)]. Consecutive row shows original images with superimposed ground-truth marine snow particles [Figs. 7(e) and 7(f)]. Frames [Figs. 7(g) and 7(h)] show detected and ground-truth areas—green color corresponds to the true positives (TPs), red denotes false positives (FPs), and yellow false negatives (FNs), respectively. Finally, the output frames filtered by our method are shown in Figs. 7(i)–7(j). It is visible that both, small and large marine snow particles were removed, whereas object details are not affected and the rest of the image is Fig. 7 Examples of marine snow filtering from real underwater color not blurred. This is due to precise detection, since other videos. (a) and (b) Original images, (c) and (d) hand annotated image areas are not processed—especially interesting is to ground-truth data, (e) and (f) ground-truth on original frames, (g) and (h) correct (green) and incorrect (red) detections, (i) and (j) observe in Fig. 7(i) small light places of underwater stones our method filtered frames, for the Ant-1153 and Zak-4891 frames, which were correctly left untouched. respectively. Even large marine snow particles were removed Comparing our method with the ones proposed in litera- whereas objects details are still present. For comparison results of 1 ture is not easy since the code for these methods is not avail- the method by Banerjee et al. are shown in (k) and (l), respectively. able and there is no ground-truth data. Also, only our method operates on streams of color and monochrome videos, taking full information on particle motion, rather than on a frame- and remove. On the other hand, the method Banerjee et al.,1 to-frame basis. Hence, for comparison we implemented in and the one proposed by Farhadifard et al.2 are kinds of C++, the method proposed by Banerjee et al.1 and applied guided median filters. Therefore, they are faster than a it to the not filtered frames shown in Figs. 7(a) and 7(b), classical median filtering and they do not affect all objects in respectively. Results of filtering with the latter method are the scene. Therefore, it could be beneficial to apply one of visualized in Figs. 7(k) and 7(l), respectively. When compar- them in the last step of our chain (Fig. 5). This is left for ing the last two rows of Fig. 7, it is easy to notice that the further investigation, though. method of Banerjee et al. cannot properly filter out large On the other hand, Fig. 8 shows stages of operation on marine snow particles which our method was able to detect the entirely monochrome signals from the Antarctica

Journal of Electronic Imaging 043002-6 Jul∕Aug 2018 • Vol. 27(4) Cyganek and Gongola: Real-time marine snow noise removal from underwater video sequences

Fig. 9 Relation of the marine snow detection in color images, measured with the F value in respect to the constant c in Eq. (3).

1 1. the monochrome case (M), in which I ¼ 3 ðR þ G þ BÞ and I I Fig. 8 Examples of marine snow filtering in the monochrome images. 2. the red-channel case (R), i.e., ¼ R. (Left column) (a) Original frame Ant-1153, (c) its detected marine snow particles, and (e) filtered version. (Right column) (b) Original In all cases, the F value in respect to various parameters of frame Ant-1176, (d) its detected marine snow particles, and (f) filtered our algorithm was measured, based on the annotated ground- version. It is interesting to observe that white stones were correctly truth data.27 In the rest of this section, only plots of the F left untouched. value versus control parameters are provided and discussed. However, detailed values of TP, TN, FP, and FN as well as video. The left column of Fig. 8 corresponds to the frame P and R can be also accessed from the Internet.28 Ant-1153, i.e., the same as the left column in Fig. 7, A plot showing a relation of the marine snow detection for while the right one to Ant-1176. Comparing Fig. 7(g) with the two sequences Ant and Zak measured with the F value in Fig. 8(c), we notice more errors in the monochrome version. respect to the constant c in Eq. (3) is shown in Fig. 9. Let us Nevertheless, the final filtering results, shown in Figs. 8(e)– c F recall that Eq. (3) and its parameter relate to the first 8(f), are also of good quality. Relation of the value for the processing step shown in Fig. 4, which is responsible for two types of signals, i.e., color versus monochrome ones, is assuring close distance of a pixel color to the gray line discussed in further part of this paper. zone in the RGB color space. Naturally, in Fig. 9, only The prepared ground-truth data allow for quantitative full color input images were investigated. It can be observed evaluation of the proposed marine snow detection method. c 8 “ ” that in the case of Ant, there is a visible peek around ¼ . However, for real scenes, we still do not have true color However, in the case of Zak, value of F slowly increases with images, so it is impossible to quantitatively measure the increasing c, up to a certain plateau. Thus we notice that c quality of the whole filtering chain. Therefore, we present influences detection accuracy in the range of about 4%. On filtering results for visual evaluation. A possible method the other hand, the proposed method can operate also with here would be to generate artificial marine snow and then pure monochrome images. In this case, the c parameter is to compare the input and output signals. However, as alluded not used. to previously and to the best of our knowledge, the available In the next experiment, we investigated an influence of models are not sufficient to generate realistic underwater the spatial size s of a 3-D patch on accuracy. Figure 10 scenes with superimposed marine snow noise. Thus, this shows F accuracy of the marine snow detection in respect problem is left for further research. to the constant patch size dimension s (Fig. 3). Also in the Experimental evaluation based on the annotated frames case of the s value, there is an optimal value of the parameter relies on computation of the matching areas of the spots F. For Ant, this maximum appears for s ¼ 3, and for in the ground-truth and binary maps detected by the algo- Zak s ¼ 1. It is interesting to observe that color and the red- rithm, respectively. Then, the TP, true negative (TN), FP, channel version of Ant perform almost identically, whereas and FN parameters are easily computed based on a degree the monochrome version is about 5% lower. On the other R TP of overlapping areas. From these, the recall ¼ TPþFN, pre- hand, Zak performs almost identically for all three cases. P TP F 2PR cision ¼ TPþFP, and ¼ PþR measures are obtained. Their Figure 11 shows a relation of the marine snow detection values will be presented and analyzed. accuracy F in respect to the threshold value d in Eq. (7). Let In the following experiments, we tested accuracy of the us recall, that it conveys information on the distance between proposed algorithm and its dependence upon the control intensity of the central pixels in respect to the maximum parameters. Color and scalar versions of the input videos intensities gathered from the neighboring patches. were tested. In the latter scenario, the two particular cases Also in this experiment, there are well visible maxima in were investigated: the plots of F for different versions of the algorithm. In all

Journal of Electronic Imaging 043002-7 Jul∕Aug 2018 • Vol. 27(4) Cyganek and Gongola: Real-time marine snow noise removal from underwater video sequences

Fig. 10 Relation of the marine snow detection measured with Fig. 12 Relation of the marine snow detection measured with the F s F a the value in respect to the patch spatial dimension shown in value in respect to the constant min in Eq. (9). Fig. 3.

Fig. 11 Relation of the marine snow detection measured with the F value in respect to the threshold d in Eq. (7). cases, color helps in detection, although in the case of Zak, the differences are very small. Interestingly enough, only the Fig. 13 Examples of marine snow filtering from real underwater color red-channel version of Ant, named Ant-R, performs almost scenes for which there is no manually annotated ground-truth data. (a) and (c) Original and filtered frames Ant-1006, (b) and (d) original the same as its color counterpart. This stays in concordance and filtered frames Zak-4907. Even large marine snow particles were with our observation that the red channel in color underwater removed and object details are still present. For comparison, results of images provides more information on the presence of marine the method by Banerjee et al.1 are shown in (e) and (f), respectively. snow than other channels. It is obvious that when processing In these, the large particles were not correctly filtered out. only IR channel a much lower amount of data needs to be processed than in the case of full color information, at the Experiments with hand-annotated ground-truth data same time obtaining better results than for the monochrome allowed for determination of optimal parameters of the pro- case I. posed method. Once fixed, the method can be used for filter- Last measurements in Fig. 12 show relation of F to the ing of other underwater videos for which the ground-truth threshold on a minimal acceptable area of marine snow amin, data are not available. as used in Eq. (9). As alluded to previously, its role is to Figure 13 shows the effect of detection and filtering of the exclude regions that are too small to be included in the marine snow in two exemplary frames for which there are no hand-annotated ground-truth data. hand-annotated ground-truth data. Figure 13(a) shows an In practice, amin can be kept around 25 to 30 pixels, which original frame no. Ant-1006, whereas its filtered version is seems to be a fair compromise and which allows further fil- shown in Fig. 13(c). Similarly, Fig. 13(b) shows an original tering of remaining smaller particles by the standard median frame no. Zak-4907 and its marine snow filtered version in filter, as outlined in Sec. 2.3. Fig. 13(d). The last row in Fig. 13 shows results of filtering

Journal of Electronic Imaging 043002-8 Jul∕Aug 2018 • Vol. 27(4) Cyganek and Gongola: Real-time marine snow noise removal from underwater video sequences of the frames from Figs. 13(a) and 13(b) with the method The white box in the first pair of images in Fig. 15 outlines an of Banerjee et al.1 As visible, their method is not able to image area with the high density of marine snow particles. cope with large size particles of the marine snow. On the These are filtered out in the image shown in the right column other hand, with our method even very large, marine snow in Fig. 15. particles were successfully detected and then removed, as Although there is no manually annotated ground-truth described. At the same time, important details of the visible data available for the sequences in Figs. 14 and 15,itis objects remained unchanged. Nevertheless, for very fast- evident that the proposed method can cope with even large moving marine snow particles, there are some remnants. marine snow particles in various underwater conditions. This effect was already discussed. As already mentioned, for proper detection of marine Figures 14 and 15 show more videos acquired in different snow particles, all image areas need to be investigated. underwater conditions and filtered with the proposed However, 3-D median filtering affects relatively small method. Figure 14 shows a few frames from the video taken during a wreck diving in Mauritius [Mauritius Island (20°00’56.6”N, 57°30’60.0”E)]. Highly saturated colors are due to camera setup and artificial lighting. The left column shows original frames, whereas the right one contains frames filtered with our proposed method. The first pair has outlined areas with regions of highest density of the marine snow particles. These particles are almost entirely filtered out, as outlined in the right image, even for very large particles. Figure 15 shows pairs of the original–filtered images from the video acquired in the winter conditions in the Lake Jaworzno [Jaworzno, Poland (50°13’45.0”N, 19°18’41.0”E)].

Fig. 15 Marine snow filtering of the real underwater video Jaworzno (no ground-truth data available). Original images (left column), marine snow filtered output of our proposed method (right column). First pair (a-left) with outlined region of the most prominent marine snow particles. Even very large not circular particles are removed (a-right).

Table 1 Speed of execution for different resolutions and sequences. Timings in milliseconds, in parenthesis a number of frames per second. For the VGA resolution, real-time operation is achieved in pure software implementation.

Resolution\video Antarctica (frames/s) Zakrzowek

1920 × 1080 190 ms (5) —

Fig. 14 Marine snow filtering of the real underwater video from wreck 1280 × 720 79 ms (12) 56 ms (17) diving in Mauritius (no ground-truth data available). Original images (left column) marine snow filtered output of our proposed method 640 × 480 24 ms (41) 17 ms (58) (right column). First pair (a-left) with outlined regions of the most prominent marine snow particles. Even very large not circular par- 320 × 240 7 ms (142) 4 ms (250) ticles are removed (a-right).

Journal of Electronic Imaging 043002-9 Jul∕Aug 2018 • Vol. 27(4) Cyganek and Gongola: Real-time marine snow noise removal from underwater video sequences regions returned by the detector. Processing of only sparse 2. F. Farhadifard, M. Radolko, and U. Lukas, “Single image marine snow removal based on a supervised median filtering scheme,” in Proc. of the regions speeds up operations. Also in the software imple- 12th Int. Joint Conf. on Computer Vision, Imaging and Computer mentation, only integer arithmetic is used. This is especially Graphics Theory and Applications (VISIGRAPP)—VISAPP, Vol. 4, pp. 280–287 (2017). beneficial for small form factor systems, such as control units 3. B. Cyganek, Object Detection and Recognition in Digital Images: of ROV, and generally allows for real-time operation. Table 1 Theory and Practice, Wiley, Hoboken, New Jersey (2013). presents measured timings in milliseconds for various reso- 4. G. M. Quénot, J. Pakleza, and T. Kowalewski, “Particle image veloc- imetry with optical flow,” Exp. Fluids 25(3), 177–189 (1998). lutions of the input color video. In parenthesis, a number of 5. W. Beebe, Half Mile Down, Duell Sloan Pearce, New York (1951). frames per second is shown. From these, it can be noticed 6. “The Official William Beebe Web Site,” https://sites.google.com/site/ that the proposed method allows real-time operation for cwilliambeebe/Home/bathysphere (2018). 7. R. L. Carson, The Sea Around US, Oxford Press, New York (1951). VGA resolutions. 8. G. A. Riley, “Organic aggregates in seawater and the dynamics of their Further code optimization can lead to even larger speed formation and utilization,” Limnol. Oceanogr. 8, 372–381 (1963). 9. N. Suzuki and K. Kato, “Studies on suspended materials (marine snow) up, which is left for the future research. Also, a larger in the sea. Part 1. Sources of marine snow,” Bull. Fac. Fish. Hokkaido number of sequences from different basins is planned to Univ. 4(2), 132–137 (1953). be investigated. 10. J. K. Orzech and K. H. Nealson, “Bioluminescence of marine snow: its effect on the optical properties of the sea,” Proc. SPIE 0489, 100–106 (1984). 4 Conclusions 11. M. Silver, “Marine snow: a brief historical sketch,” Limnol. Oceanogr. Bull. 24,5–10 (2015). In this paper, a method for marine snow noise removal is 12. S. Q. Duntley, “Light in the sea,” J. Opt. Soc. Am. 53, 214–233 (1963). proposed. To the best of our knowledge, this is the first 13. N. G. Jerlov, Marine Optics, Elsevier Oceanography Series, 2nd ed., method which exploits full 3-D information encountered Elsevier, Amsterdam (1976). 14. B. L. McGlamery, “Computer analysis and simulation of underwater in underwater color and monochrome video signals. Due camera system performance,” Report SIO 75-2, Scripps Institute of to analyzing signal patches in the spatial and temporal Oceanography, University of California, San Diego (1975). dimensions, we are able to efficiently detect marine snow 15. B. L. McGlamery, “A computer model for underwater camera systems,” Proc. SPIE 0208, 221–231 (1979). particles. This is based on our observation of marine 16. J. S. Jaffe, “Computer modeling and the design of optimal underwater snow shape, their light reflection properties and their fast imaging systems,” IEEE J. Ocean. Eng. 15(2), 101–111 (1990). movement properties when observed in front of the camera 17. J. S. Jaffe, “Underwater optical imaging: the past, the present, and the prospects,” IEEE J. Oceanic Eng. 40(3), 683–700 (2015). plane. Based on these, we proposed a detector operating as a 18. E. Trucco and A. T. Olmos-Antillon, “Self-tuning underwater image chain of three processing blocks. If a marine snow particle is restoration,” IEEE J. Oceanic Eng. 31(2), 511–519 (2006). 19. D. M. Kocak et al., “A focus on recent developments and trends in detected, then it is corrected by the proposed 3-D median underwater imaging,” Mar. Technol. Soc. J. 42(1), 52–67 (2008). filter, which substitutes a marine snow pixel by a median 20. R. Schettini and S. Corchs, “Underwater image processing: state of value computed from the external 2-D layers of the 3-D the art of restoration and image enhancement methods,” EURASIP J. Adv. Signal Process. 2010, 746052 (2010). patch. Finally, the frames are optionally processed by the 21. L. Chongyi et al., “Single underwater image enhancement based on 2-D median filter which removes smaller particles. Tested color cast removal and visibility restoration,” J. Electron. Imaging with real color and monochrome underwater sequences, the 25(3), 033012 (2016). 22. U. A. Nnolim, “Smoothing and enhancement algorithms for underwater method shows high efficacy in filtering fast moving marine images based on partial differential equations,” J. Electron. Imaging snow particles, whereas retaining other objects visible in the 26(2), 023009 (2017). 23. C. Sánchez-Ferreira et al., “Multi-objective differential evolution scenes. The method was compared with the one proposed by ” 1 algorithm for underwater image restoration, in IEEE Congress on Banerjee et al., which operates exclusively on single frames. Evolutionary Computation (CEC), Sendai, pp. 243–250 (2015). It was shown that our method outperforms the latter espe- 24. W. Slade, E. Boss, and C. Russo, “Effects of particle aggregation and disaggregation on their inherent optical properties,” Opt. Express 19, cially in the areas of marine snow particle of large size, 7945–7959 (2011). which the method Banerjee et al.1 is not able to filter out. 25. M. Boffety and F. Galland, “Phenomenological marine snow model for To apply a quantitative analysis, some parts of the sequen- optical underwater image simulation: applications to color restoration,” in OCEANS—Yeosu, Yeosu, pp. 1–6 (2012). ces were manually annotated to provide ground-truth posi- 26. B. Cyganek and K. Socha, “A multi-tool for ground-truth stereo corre- tions of the real marine snow particles. Based on this, we spondence, object outlining and points-of-interest selection,” in VIGTA, First Int. Workshop on Visual Interfaces for Ground Truth Collection in were able to measure detection accuracy which for the men- Computer Vision Applications, Capri, Italy (2012). tioned sequences reached F ¼ 0.71 and F ¼ 0.61, respec- 27. B. Cyganek and K. Gongola, “AGH marine snow database,” http://home tively. All these sequences and the annotations were made .agh.edu.pl/~cyganek/AGH_MSD.zip, AGH University of Science and Technology (2018). available to other researchers for comparison, which is 28. B. Cyganek and K. Gongola, “Marine snow detector—detailed mea- one of the contributions of our work. Implementation of surements,” http://home.agh.edu.pl/~cyganek/MarineSnowDetector_ the proposed method requires exclusively integer arithmetic. DetailedMeasurements.pdf, AGH University of Science and Due to this, the method can operate in real time, as shown by Technology (2018). experiments. Bogusław Cyganek received his MSc degree in electronics in 1993, and then his MSc degree in computer science in 1996, from AGH Acknowledgments University of Science and Technology, Krakow, Poland. He obtained This work was supported by the Polish National Science his PhD cum laude in 2001 with a thesis on correlation of stereo Center NCN under the Grant No. 2014/15/B/ST6/00609. images, his DSc degree in 2011, and professorship in 2017. His research interests include computer vision, pattern recognition, and embedded systems. He is also a SCUBA diving instructor. References 1. S. Banerjee et al., “Elimination of marine snow effect from underwater Karol Gongola received an engineering degree in electronics image—an adaptive probabilistic approach,” in IEEE Students’ Conf. on from AGH University of Science and Technology, Krakow, Poland, Electrical, Electronics and Computer Science (SCEECS), pp. 1–4 in 2017. His interests include embedded systems and software (2014). programming.

Journal of Electronic Imaging 043002-10 Jul∕Aug 2018 • Vol. 27(4)