REMOVAL ON COLOR IMAGE SEQUENCES USING COUPLED ANISOTROPIC DIFFUSIONS AND NOISE-ROBUST MOTION DETECTION

P. Bourdon, B. Augereau, C. Olivier, and C. Chatellier IRCOM-SIC, UMR-CNRS 6615, Universite´ de Poitiers Batˆ SP2MI, Bvd M. et P. Curie, BP 30179 86962 Futuroscope Chasseuneuil Cedex, France (Europe) phone: +33 5 49 49 65 85, fax: +33 5 49 49 65 70 email: {bourdon, augereau, olivier, chatellier}@sic.sp2mi.univ-poitiers.fr

ABSTRACT 2.1 Isotropic diffusion In this paper, we propose a new approach for noise removal Let I(x,y) = I be a still, grayscale, image, represented by on color image sequences, based on coupled spatial and tem- a function of Ω ⊂ R2 → R that associates to a pixel (x,y) ∈ poral anisotropic diffusions. This approach consists in smoo- Ω its gray level I(x,y) ; Ω is the support of the image. We thing a sequence while keeping all discontinuities caused by define I (x,y) = I as a noisy version of I(x,y). A well-known edges and moving objects. Our approach is based on a low 0 0 solution to recover I(x,y) from I0(x,y) is to perform Gaussian level detection of moving areas in a noisy sequence, in order , which happens to be equivalent to the evolution to associate temporal diffusion to a spatial diffusion process, of the linear heat (or diffusion) equation (see [6]) : thus denoising the sequence more efficiently.

∂ 1. INTRODUCTION I(x,y,t) = div(∇I(x,y,t)) ∂t (1) I(x,y,0) = I (x,y) Because they allow noise removal while preserving  0 global characteristics such as edges, Partial Differential Equations-based denoising methods have become quite po- Equation (1) defines the diffusion process of pixel (x,y)’s pular in image processing over the last decade [1, 2, 3]. brightness around neighboring pixels (x±∂x,y±∂y), during Unfortunately, most of these methods tend to focus on still a time t (t ∈ [0,T]) which value is directly linked to the Gaus- images, while only a few have been proposed for image se- sian kernel variance. This process, called isotropic diffusion, quences processing [4, 5], even less when it comes to color is known to introduce blur, since it operates the same way in sequences. Of course still image processing techniques could all directions, smoothing both noise and edges. be applied to perform ”frame-by-frame” video processing, but more efficient algorithms are needed to better exploit the 2.2 Anisotropic diffusion redundancy between successive frames. A simple idea for PDE-based image sequence denoising would be to consider a In 1990, Perona and Malik [1] introduce anisotropic dif- sequence as a 3D object, and apply Perona-Malik’s PDE [1] fusion, a nonlinear process in which smoothing is only per- in a 3D space, using a 3D gradient to detect discontinuities. formed in low gradient areas (homogeneous areas), thus al- In this paper, we propose a color image sequence noise remo- lowing noise blurring with edge preservation : val method, which is based on this idea, except for one point : in our method, temporal and spatial diffusions both use speci- fic discontinuities detection. While the well-known gradient ∂ I(x,y,t) = div(c(k∇I(x,y,t)k)∇I(x,y,t)) norm is used to detect spatial discontinuities, we decided to ∂t (2) I(x,y,0) = I (x,y) use a better, noise-robust, descriptor for temporal disconti-  0 nuities detection. To make this descriptor noise-robust, the idea is to look for variations of statistical attributes of a local with c(.) a decreasing positive function, called ”diffusion neighborhood, instead of looking for variations of a single fonction”, which allows to define the strength of the smoo- pixel’s brightness. More details about this methods will be thing process for each gradient norm value. (2) can also be given in this article, along with experimental results. written as follows [7] : In section 2, we present the principles of PDE-based grayscale and color image denoising. In section 3, we will focus on image sequences, before introducing our coupled ∂I 00 Φ0(k∇Ik) ∂ = Φ (k∇Ik)Iξξ + ∇ Iηη spatial and temporal anisotropic diffusions method in section t k Ik (3) I(x,y,0) = I (x,y) 4. Experimental results will be provided in section 5. ( 0

2. PDE-BASED NOISE REMOVAL with Iξξ and Iηη the second-order directional derivatives res- pectively along the gradient direction and along its orthogo- In this part, we propose a simple approach of noise re- nal direction. This equation actually is Perona-Malik’s PDE moval in image processing. This will allow us to introduce 0 ∇ Φ (k∇Ik) the principles of isotropic and anisotropic diffusions for still (2), with c(k Ik) = k∇Ik . Deriche-Faugeras’ formulation grayscale and color images. (3) allows to define conditions on function Φ(.) :

481 Φ00(0) ≥ 0 et Φ0(0) ≥ 0 Φ0(s) Φ00 Φ00 lims→0 s = lims→0 (s) = (0)  Φ0(s) ∞ Φ00 ∞ (4)  lims→ (s) = 0, lims→ s = 0  Φ00(s)  ∞ lims→ Φ0(s) = 0 s   Equation (4) means that smoothing is performed in all directions (isotropic diffusion) for low gradient areas, while (a) Noisy sequence (b) Restored sequence it is only performed along the gradient’s orthogonal direction (anisotropic diffusion) for high gradient areas. FIG. 1 – 3D diffusion of a noisy sequence (frame n◦82)

2.3 Extension to color images In the case of color images, we define a vectorial function although a 3D gradient norm allows to detect discontinuities, its value won’t give any information about either spatial or ~I(x,y) : Ω ⊂ R2 → R3 which associates to pixel (x,y) ∈ Ω its temporal variations. This means homogeneous areas of one 3 component values in the Red-Green-Blue color space. To frame won’t be blurred if pixels of these areas present high perform anisotropic diffusion on color images, we can use temporal variations on several frames. DiZenzo’s gradient norm [8], which is based on differential geometry of surfaces. It consists in defining a multispectral In the next section, we are presenting a denoising method , associated to a vector field, to look for local variations which is based on this 3D anisotropic diffusion process, ex- in the image. The highest eigenvalue of the multispectral ten- cept in our method spatial and temporal discontinuities have sor then corresponds to the square norm of the gradient. their own detector.

3. IMAGE SEQUENCE DENOISING 4. COUPLED SPATIAL AND TEMPORAL DIFFUSIONS 3.1 Introduction In this section, we consider a color image sequence. Video restoration being a more recent research area than ~ Ω R3 R3 still image restoration, it is no surprise that most papers dea- I(x,y,z) : ⊂ → is a function which associates to ling with PDE-based noise removal tend to focus on still pixel (x,y,z) its component values in the RGB color space. image processing. Of course we could use still image de- Our method consists in performing simultaneously two non- noising techniques to perform frame-per-frame video res- linear 2D and 1D filtering operations : spatial and temporal toration. But in this case redundancies between successive anisotropic diffusions. Such process should independently frames wouldn’t be exploited, and those provide very use- define the diffusion of pixel (x,y,z)’s chromatic RGB spec- trum around neighboring pixels (x ± ∂x,y ± ∂y) on frame z, ful information for noise removal. In 1984, Dubois and Sabri ∂ [9] introduce a method in which static scenes of a video are as well as its diffusion around neighboring pixels z ± z on identified. then consists in performing tem- spatial localisation (x,y). In both cases, diffusion speed is poral averaging on those scenes, while dynamic scenes are determinated by medium-specific (space or time) disconti- kept unfiltered. Identifying static/dynamic scenes to perform nuities detectors. Spatial discontinuities are measured via Di- noise removal happens to be very efficient. Kornprobst suc- Zenzo’s color gradient norm, while we propose our own me- cessfully adapted this idea for PDE-based restoration [5]. thod to detect temporal discontinuities. This method consists in studying a neighborhood for each pixel and comparing 3.2 A study of 3D diffusion neighborhood statistics to an estimated model. We will ex- plain how this method turns out to be more noise-robust and Ω R3 R Let I(x,y,z) = I be a function of ⊂ → that as- useful than a simple, easily perturbable, 1D gradient norm. sociates to a pixel (x,y) localized in frame z its gray level I(x,y,z). I (x,y,z) being a noisy version of I(x,y,z), a basic 0 4.1 Diffusion equation idea to recover I(x,y,z) would be to extend Perona-Malik’s PDE (2) to 3D space : To differentiate spatial diffusion from temporal diffusion, we write our PDE as follows :

∂I(x,y,z,t) ∂ = div(c(k∇I(x,y,z,t)k)∇I(x,y,z,t)) ∂~I t (5) cs( ∇~I ) ∂ I(x,y,z,0) = I0(x,y,z) s x   ∂~I  ∇~ ∂~I  ∂t = div cs( I ) ∂y Figure 1 shows results obtained on noisy ”Foreman” se-  s (6)   ∇~ ∂~I  quence by this 3D anisotropic diffusion process. Although   ct ( I ) ∂z  it seems to give satisfying results, we can find a few draw-  t  ~I(x,y,z,0) =~I0( x,y, z)  backs to this method. Considering a video as a 3D object ac-   tually consists in considering the video’s temporal coordinate  as a third spatial coordinate : in this case, the video’s voxels with cs(.) the spatial diffusion function, ct (.) the temporal ∇~ would be evolving in a homogeneous and isotropic medium. diffusion function, k I ks the vectorial norm representing ∇~ This, of course, is a wrong assumption. Taking a closer look spatial discontinuities, and k I kt the vectorial norm repre- at Figure 1.b, we can notice noise residuals on a few areas : senting temporal discontinuities.

482 ∇~ As we said already, k I ks is calculated using DiZenzo’s attributes D1, D2, ..., DL, ∇~ gradient norm. The calculation of k I kt is explained in next paragraph. – finally, for each pixel ~I, a Mahalanobis [12] distance between vector ~D on frame z and its parametric model 4.2 Temporal discontinuities detection estimated on neighboring frames is calculated, and ∇~ 2 To detect temporal discontinuities in a noisy sequence, corresponds to k I kt . the easiest solution would be to compute a 1D gradient calcu- lated on coordinate z for each pixel of this sequence. Unfor- Let P be the number of frames used to estimate a parame- tunately, experimental results show that the high noise sensi- tric model for vector ~D (P is an even value). For each pixel bility of this method makes it unusable. (x,y), we define :

Suppose now that instead of studying the temporal evolu- +P/2 tion of a single pixel~I, we focus on a whole set of pixels (a.e. µ 1 Dl (z) = ∑ Dl(z + i), l = 1,...,L (8) a M ×M neighborhood, as shown on Figure 2), and decide to P i=−P/2, study statistical attributes of this set instead of studying raw i6=0 RGB pixel values. +P/2 σ 2 1 µ 2 Dl (z) = ∑ (Dl(z + i) − Dl (z)) (9) P i=−P/2, i6=0

1 +P/2 µ (Dk(z + i) − Dk (z)) cov(Dk,Dl)(z) = ∑ µ (10)

P × (Dl(z + i) − Dl (z)) ¡ i=−P/2, I (x, y, z) D (x, y, z) i6=0 (a) Single-pixel approach (b) Cluster approach The Mahalanobis distance between vector ~D and its pa- rametric model is defined by : FIG. 2 – Neighborhood study on one z frame 2 t −1 ∇~I = (~D − µ~ ) .C .(~D − µ~ ) (11) t D ~D D In our cluster approach, statistical attributes of pixel ~I(x,y,z) and surrounding neighborhood are defined in vec- µ with ~ and C~ respectively the mean and covariance ~ D D torial function D(x,y,z) : matrices of vector ~D’s parametric model.

D1(x,y,z) D2(x,y,z) ~ 5. EXPERIMENTAL RESULTS D(x,y,z) =  .  (7) . 5.1 Motion detection  D (x,y,z)   L    Figure 3 shows temporal discontinuities detection results In practice, statistical attributes D1, D2, ..., DL could be obtained on noisy ”Claire” color sequence (additive gaus- spatial means and variances, median, mimimum or maxi- sian noise) using a 1D gradient (Fig 3.b) and our statistics- mum values, etc. based method, using P = 4 and M = 3 as neighborhood pa- Assuming the presence of motion between two consecu- rameters (Fig 3.c). In this example, we used spatial means tive frames z and z + dz produces brightness intensity varia- (D1) and variances (D2) of brightness values Y(x,y,z) (Y = tions on pixels of a defined array, statistical attributes of this 0.299R + 0.587V + 0.114B) to define statistics vector ~D. As array may also be subject to variations. This way, both single- we can see, noise perturbates the temporal gradient calcula- pixel value and array statistics-based approaches can be used tion and makes it almost unusable for detection, but our me- to detect temporal discontinuities. thods seems to present more robustness : Fig 3.c only shows Suppose we are working on a noisy sequence, this actual motion elements, while noise hasn’t been detected as noise being characterized by a statistical model. In most such. cases, including natural scenes, this noise model in unlikely to change during sequence-time. This actually makes our statistics-based detection method noise-robust, at least sta- tionary noise-robust. We can find several motion detection methods based on cluster approaches in literature [10, 11]. Ours operates this way :

– on frame z, statistics vector ~D is calculated for each pixel~I and surrounding neighborhood, (a) Noisy sequence (b) 1D gradient (c) our method

– statistical attributes of a same neighborhood are also FIG. 3 – Motion detection on a noisy sequence (frame n◦16) calculated on surrounding frames z ± 1, z ± 2, etc. to obtain temporal means and variances/covariances for

483 5.2 Noise removal results have been achieved and shown on this paper, and open In this part, we present experimental results of our noise new perspectives in PDE-based image sequence processing. removal method. Temporal discontinuities detection statisti- Future works will include automatic diffusion/detection pa- cal attributes and parameters remain the same as in 5.1. We rameters estimation, that would lead to an unsupervised res- decided to use the following diffusion function : toration algorithm.

REFERENCES c (x) = c (x) = (1 + (x/k )2)e−(x/k j), j ∈ {s,t} (12) j j j [1] P. Perona and J. Malik, “Scale-space and edge detec- The purpose of using this diffusion function instead of tion using anisotropic diffusion,” IEEE transactions on Perona-Malik’s is to focus on noise removal properties only Pattern Analysis and Machine Intelligence, vol. 12, no. (Perona-Malik’s function performs both noise removal and 7, pp. 629–639, 1990. edge enhancement). [2] L. Alvarez, P-L. Lions, and J-M. Morel, “Image selec- Figures 4.a and 4.b show results obtained on noisy tive smoothing and by nonlinear diffu- ”Claire” color sequence, using diffusion parameters ks = 2.5, sion. ii,” SIAM Journal on Numerical Analysis, vol. 29, kt = 0.25, and t = 15. We can notice serious enhancement no. 3, pp. 845–866, 1992. between Fig 4.a and Fig 4.b. This enhancement is confirmed [3] J. Weickert, Anisotropic Diffusion in Image Processing, by PSNR values provided on Fig 4.c, which increase from Ph.D. thesis, Dept. of Mathematics, University of Kai- about 27dB to about 38dB. serslautern, Germany, January 1996. [4] L. Moisan, “A depth-compatible multiscale analysis of movies,” Tech. Rep. CMLA No. 9803, Centre de Mathematiques´ et de Leurs Applications, Ecole Nor- male Superieure´ de Cachan, 1998. [5] P. Kornprobst, R. Deriche, and G. Aubert, “A PDE based coupled method for image restoration and mo- tion segmentation,” in Proceedings of the 5th Euro- pean Conference on , Freiburg, Ger- many, June 1998, vol. II, pp. 548–562. [6] J.J. Koenderink, “The structure of images,” Biological (a) Noisy sequence (b) Restored sequence Cybernetics, vol. 50, no. 5, pp. 363–370, 1984. [7] R. Deriche and O. Faugeras, “Les EDP en Traitement "Claire" sequence restoration 40 des Images et Vision par Ordinateur,” Traitement du Signal, vol. 13, no. 6, 1996. 38 [8] S. Di Zenzo, “Note : A note on the gradient of a multi- 36 Noisy sequence image,” Computer Vision, Graphics, and Image Pro- Restored sequence cessing, vol. 33, no. 1, pp. 116–125, January 1986. 34 [9] E. Dubois and S. Sabri, “Noise reduction in image se- 32

PSNR(dB) quences using motion compensated temporal filtering,”

30 IEEE transactions on communications, vol. 32, no. 7, pp. 826–831, July 1984. 28 [10] H.J. Eghbali, “K-s test for detecting changes from land- 26 sat imagery data,” IEEE Transactions on Systems, Man, 1 21 41 61 81 101 121 141 Frame number and Cybernetics, vol. 9, no. 1, pp. 17–23, 1979. (c) PSNR per frame [11] Y.Z. Hsu, H.H. Nagel, and G. Rekers, “New likelihood test methods for change detection in image sequences,” FIG. 4 – Restoration of noisy ”Claire” sequence Computer Vision, Graphics, and Image Processing, vol. 26, pp. 73–106, 1984. [12] P.C. Mahalanobis, “On the generalized distance in 6. CONCLUSION statistics,” Proceedings of the National Institute of Sciences of India, vol. 2, no. 1, pp. 49–55, 1936. In this paper, we’ve presented a noise removal method for color image sequences, based on coupled spatial and tempo- ral anisotropic diffusions. To make it more efficient, we’ve also introduced a new, noise-robust, temporal discontinuities detection technique. The main advantages of our method is that it will work for most types of stationary noise (besides additive , it also gave good results on com- pression noise), and the fact that PDE formalism allows to provide fast yet stable algorithms. Promising experimental

484