Real-time marine snow noise removal from underwater video sequences Bogusław Cyganek Karol Gongola Bogusław Cyganek, Karol Gongola, “Real-time marine snow noise removal from underwater video sequences,” J. Electron. Imaging 27(4), 043002 (2018), doi: 10.1117/1.JEI.27.4.043002. Journal of Electronic Imaging 27(4), 043002 (Jul∕Aug 2018) Real-time marine snow noise removal from underwater video sequences Bogusław Cyganek* and Karol Gongola AGH University of Science and Technology, Department of Electronics, Krakow, Poland Abstract. Underwater images suffer from various degradation factors, such as blur, haze, color degradation, and marine snow. Marine snow is a type of noise, caused mostly by biological particles that fall into the ocean bottom, and which impedes proper object detection in underwater vision. A method for real-time marine snow removal from underwater color and monochrome video is presented. It is based on the proposed marine snow model, spatiotemporal patch analysis, and three-dimensional median filtering. The method was evaluated on a number of real underwater sequences endowed with the hand-annotated ground-truth data which were made available from the Internet. As shown by the experiments, the method attains high accuracy and performs in real time. © 2018 SPIE and IS&T [DOI: 10.1117/1.JEI.27.4.043002] Keywords: marine snow filtering; underwater image enhancement; real-time image filtering; remotely operated underwater vehicle. Paper 180201 received Mar. 7, 2018; accepted for publication Jun. 11, 2018; published online Jul. 5, 2018. 1 Introduction colleague biologists. The phenomenon of marine snow, or “ ” 8 Underwater image acquisition and processing find broad later called organic aggregates by Riley, has found interest interest in such areas as underwater exploration, inspection in biological sciences. Their role in the ocean ecosystem, as of underwater constructions, and underwater navigation, to for example food conveying or for various organisms, has name a few. However, their processing puts much higher been understood and appreciated for years of their study. In this respect, an interesting reading is the paper by demands than processing of air space images due to under- 9 water physical conditions, such as contrast and color decay, Suzuki and Kato summarizing their studies in the 1950s on suspended materials in the sea near Hokkaido. On the light scattering, blur, haze, and various types of noise. They 10 cause image quality degradation and lead to loss of conveyed other hand, in the 1980s Orzech and Nealson conducted information. There are many methods which allow for filter- research on measuring bioluminescence of marine snow ing of these unwanted effects. However, there is a special and its effect on the optical properties of the sea, as reported in their paper. An absorbing overview of research on marine type of noise, called marine snow, which greatly affects qual- snow, from the biological point of view, is provided in the ity of the underwater images and is difficult to filter out. paper by Silver.11 On the other hand, in the recent years vari- Marine snow is an effect caused by light back scattering ous methods of underwater image analysis and enhancement from small organic and mineral particles and air bubbles. have been proposed. The books by Duntley12 and the one by When falling down to the water basin, particles grow, Jerlov13 discuss basic physical properties of light propaga- which manifest in images as bright spots of various shapes tion in water conditions. McGlamery14,15 conducted research and sizes, which to some extent resemble snowflakes, as into analysis and simulation of an underwater camera system shown in exemplary frames in Fig. 1. Only recently this and laid out theoretical foundations of the radiometric model phenomenon found interest among researchers in order to of underwater image formation. Then Jaffe16 proposed exten- develop efficient methods of its elimination.1,2 However, — sion aimed at the design of the subsea image acquisition sys- the problem is not a trivial one the particles can be quite tem with optimized contrast and minimized backscattering large, of different structural and lighting characteristics, effect. This way, the Jaffe–McGlamery underwater image that make their statistical properties significantly different formation model has been derived. Recently, Jaffe17 pub- from other types of noise encountered in digital images. lished a comprehensive description of underwater optical im- Because of this, the classical linear and nonlinear filtering aging in the context of physical, biological, technological, methods, such as the averaging or median filters, usually and historical aspects. In this paper, the fundamental limits cannot be used to remove such type of noise from the imposed by the water environment are discussed and related 3,4 images. Thus, there is a need for development of models to the recent technological achievements. A slightly simpli- and methods of marine snow filtering, such as the one pro- fied version of the Jaffe–McGlamery underwater image posed in this paper. formation model was proposed by Trucco and Olmos- The term “marine snow” was coined after the Beebe’s Antillon18 for the self-tuning image restoration filter. An observations in Bathysphere,5,6 conducted in the 1930s, overview of the underwater optical systems is provided in then published in the book by Carson,7 one of the Beebe’s the paper by Kocak et al.19 Similarly, in the paper by Schettini and Corchs,20 theory and recent methods of *Address all correspondence to: Bogusław Cyganek, E-mail: cyganek@agh .edu.pl 1017-9909/2018/$25.00 © 2018 SPIE and IS&T Journal of Electronic Imaging 043002-1 Jul∕Aug 2018 • Vol. 27(4) Cyganek and Gongola: Real-time marine snow noise removal from underwater video sequences Fig. 1 Examples of marine snow in real underwater scenes. Marine snow manifests as lighter oval or rectangular and usually fast-moving shapes. underwater image restoration and enhancement are video to remove marine snow. Last but not least, for quanti- described. On the other hand, a method for single underwater tative evaluation and further comparisons, we provide under- image enhancement was recently proposed by Chongyi water sequences with hand-annotated ground-truth marine et al.21 In their method, numerical optimization is used for snow particles. color cast removal, followed by the visibility and contrast A significant achievement would be construction of an restoration based on the inherent relationship of medium appropriate marine snow model which would reflect various transmission maps of three color channels. Another interest- physical and biological conditions of the particles and the ing group of algorithms for smoothing and enhancement of environment. Such a model would be useful also in measur- underwater images, based on partial differential equations, ing quality of the marine snow filtering. In this respect, Slade was proposed by Nnolim.22 In the recent paper by Sánchez- et al.24 explored effects of particle aggregation and disaggre- Ferreira et al.,23 an algorithm for underwater image restora- gation on their inherent optical properties. They empirically tion is proposed which operates by evolutionary estimation investigated the role that aggregation plays in determining of the parameters of the underwater image formation model properties of the particle light scattering in coastal waters. using two quality metrics. After that, Boffety and Galland25 proposed a phenomeno- However, as alluded to previously, there are only few logical marine snow model for optical underwater image works which directly address the problem of marine snow simulation especially aimed at underwater color restoration. removal. In the paper by Banerjee et al.,1 a variant of the They argue that the simple model obtained by generation of median filtering, based on probability of existence of marine a salt-and-pepper noise does not take into account various snow, is proposed. In this method, high luminance pixels are physical conditions, such as water absorption, particles detected in each patch extracted from the image. Marine shapes and sizes, and signal backscattering by the particles. snow is modeled by the probability of observing sparse num- Therefore, Boffety and Galland proposed a simplified ber of high-intensity pixels two times in a patch and its approach, in which macroparticles are assumed to behave doubled size version. If such a probability is high, then like white Lambertian scatterers with the spatial profile of the center pixel of a patch is replaced with the median their reflection coefficient being a Gaussian function. value of the patch. However, this method works only for rel- The rest of this paper is organized as follows. In Sec. 2, atively small particles of few pixels, whereas marine snow the principles of the proposed method are presented. We start can manifest as much larger structures, as will be discussed. with the marine snow particle model, presented in Sec. 2.1, A modification to this method was proposed by Farhadifard followed by a detailed description of the marine snow detec- et al.2 They also follow the idea of median filtering after tion in Sec. 2.2, and its filtering method described in Sec. 2.3. supervised noise detection by a multiscale patch-based Experimental results are presented in Sec. 3. This paper ends approach. However, both of the above methods are limited with conclusions in Sec. 4. since they operate exclusively in spatial domain and only on 2 Method Description single frames not considering temporal relations. On the other hand, when processing underwater videos, a more ver- In this section, the characteristics of marine snow will be pre- satile spatiotemporal analysis is possible, as will be shown. sented. Especially, interesting and not previously exploited In this paper, we propose an efficient method of marine snow feature is the fast movement behavior of the marine snow elimination, which relies on analysis of spatiotemporal three- particles. dimensional (3-D) patches, i.e., tube-like structures, rather than on flat two-dimensional (2-D) ones.