Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI-20) Multichannel Color Image Denoising via Weighted Schatten p-norm Minimization Xinjian Huang , Bo Du∗ and Weiwei Liu∗ School of Computer Science, Institute of Artificial Intelligence and National Engineering Research Center for Multimedia Software, Wuhan University, Wuhan, China h [email protected], [email protected], [email protected] Abstract the-art denoising methods are mainly based on sparse repre- sentation [Dabov et al., 2007b], low-rank approximation [Gu The R, G and B channels of a color image gen- et al., 2017; Xie et al., 2016], dictionary learning [Zhang and erally have different noise statistical properties or Aeron, 2016; Marsousi et al., 2014; Mairal et al., 2012], non- noise strengths. It is thus problematic to apply local self-similarity [Buades et al., 2005; Dong et al., 2013a; grayscale image denoising algorithms to color im- Hu et al., 2019] and neural networks [Liu et al., 2018; age denoising. In this paper, based on the non-local Zhang et al., 2017a; Zhang et al., 2017b; Zhang et al., 2018]. self-similarity of an image and the different noise Current methods for RGB color image denoising can be strength across each channel, we propose a Multi- categorized into three classes. The first kind involves apply- Channel Weighted Schatten p-Norm Minimization ing grayscale image denoising algorithms to each channel in (MCWSNM) model for RGB color image denois- a channel-wise manner. However, these methods ignore the ing. More specifically, considering a small local correlations between R, G and B channels, meaning that un- RGB patch in a noisy image, we first find its nonlo- satisfactory results may be obtained. The second method is cal similar cubic patches in a search window with to transform the RGB color image into other color spaces [D- an appropriate size. These similar cubic patches are abov et al., 2007a]. This transform, however, may change the then vectorized and grouped to construct a noisy noise distribution of the original observation data and intro- low-rank matrix, which can be recovered using the duce artifacts. The third type of method involves making full Schatten p-norm minimization framework. More- use of the correlation information across each channel and over, a weight matrix is introduced to balance each conduct the denoising task on R, G and B channels simulta- channel’s contribution to the final denoising result- neously [Zhang et al., 2017a]. s. The proposed MCWSNM can be solved via the Similar to the case of grayscale images, noise from each alternating direction method of multipliers. Con- channel can be generally, regarded as additive white Gaussian vergence property of the proposed method are also noise. However, the noise levels of each channel are diverse theoretically analyzed . Experiments conducted on due to camera sensor characteristics and the imagery environ- both synthetic and real noisy color image dataset- ment, such as fog, haze, illumination intensity, etc. Moreover, s demonstrate highly competitive denoising perfor- the on-board processing steps in digital camera pipelines may mance, outperforming comparison algorithms, in- also introduce different noise strength assigned to different cluding several methods based on neural networks. channels [Nam et al., 2016]. This reveals the difficulties and challenges faced by RGB color image denoising. Intuitive- 1 Introduction ly, if we apply the grayscale image denoising algorithm to color images in a band-wise manner without considering the Noise corruption is inevitable during the image acquisition mutual information and noise difference between each chan- process and may heavily degrade the visual quality of an ac- nel, artifacts or false colors could be generated [Mairal et al., quired image. Image denoising is thus an essential prepro- 2008]. An example of this is presented in Fig.1. Here, we cessing step in various image processing and computer vi- apply a representative low-rank-based method, WSNM [Xie [ sion tasks Chatterjee and Milanfar, 2010; Ye et al., 2018; et al., 2016], and our proposed method, MCWSNM (see Sec- ] Wang et al., 2018b ; moreover, it is also an ideal test platform tion 2), to the Kodak PhotoCD Dataset1. It can be seen from for evaluating image prior models and optimization methods the figure that WSNM retains a large amount of noise and, [ ] Roth and Black, 2005 . As a result, image denoising re- to some extent, introduces some artifacts. Thus, the issue of mains a challenging yet fundamental problem. Early denois- how noise differences in each channel should be modeled is ing algorithms were mainly devised on the basis of filter and key to designing a good RGB color denoising algorithm. transformation [Dabov et al., 2007b], such as wavelet trans- One well-known color image denoising method is color form and curvelet transform [Starck et al., 2002]. State-of- ∗Co-corresponding Author. 1http://r0k.us/graphics/kodak/ 637 Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI-20) V 1 b V g 1 r … (a) Clean (b) Noisy (c) WSNM (d) MCWSNM V 2 … 3V V ଶ … Figure 1: Denoised results of ”kodim07” in the Kodak PhotoCD Dataset with σr = 50, σg = 5 and σb = 20. WSNM performs in a band-wise manner; MCWSNM jointly processes three channels V M < jointly and considers their noise differences simultaneously. (It is V M best to zoom in on these images on-screen). Figure 2: Illusion of the grouped local patches. block-matching 3D (CBM3D) [Dabov et al., 2007a]. This 2 Our Proposed Method is a transformation field method that may destroy the noise 2.1 Method Formulation distribution across each channel and introduce artifacts. The The image denoising task involves recovering a clean image method in [Zhu et al., 2016] concatenates similar patches of xc from its noisy observation data yc, where c = fr; g; bg RGB channels into a long vector. However, this concatena- is the index of the R, G and B channels. Here, we assume tion operation regards each channel equally and ignores the that the noisy image is corrupted by additive white Gaussian noise level differences across each channel. For a long time, noise nc with σc = (σr; σg; σb), where σr; σg; σb denote the low-rank theory has been widely used in many fields, such noise standard deviation in the R, G, B channels, respectively. [ et al. ] as recommendation system Wang , 2018a , community Mathematically, y = x + n . Low-rank-based denoising [ et al. et al. ] c c c search Fang , 2020b; Fang , 2020a , multi-out task methods utilize the low-rank property of the grouped non- [ et al. ] [ et al. ] Liu , 2019 and multi-label Liu , 2017 , etc. local similarity patches; this group procedure is illustrated in y Recently, based on the non-local self-similarity of an image Fig.2. Given a noisy RGB color image c, each local patch of size s × s × 3 is extracted and stretched into a vector and SVD, several low-rank-based denoising algorithms have 2 2 y = (yT ; yT ; yT )T 2 R3s y ; y ; y 2 Rs been proposed, such as weighted nuclear norm minimization r g b , where r g b are (WNNM) [Gu et al., 2017] and WSNM [Xie et al., 2016]. the corresponding patches in the R, G and B channels. For y M The nuclear norm and Schatten p-norm [Xie et al., 2016] are each vectorized local patch , we find its most similar y two surrogates of the rank function [Xu et al., 2017a]. In patches (including itself) by Euclidean distance in a proper M WNNM and WSNM, singular values are assigned different size search window around it. We then stack the similar weights using a weight vector. It has been proven that WNN- patches vector column by column to obtain the noisy patch R3s2×M M is a special case of WSNM and that WSNM outperform- matrix Y = X + N 2 , where X and N are the s WNNM. Multi-channel WNNM (MCWNNM) [Xu et al., corresponding clean and noise patch matrix, respectively. It 2017b] is an extension of WNNM from grayscale image to is clear that Y is a low-rank matrix and X can be recovered color image that operates by introducing a weight matrix to using a low-rank approximation algorithm. adjust each channel’s contribution to the final results based As a nonconvex surrogate of the rank function, the weight- Rm×n on noise level. ed Schatten p-norm of a matrix Z 2 is defined as minfm;ng 1 P p p kZkw;Sp = ( i=1 wiσi ) ; 0 < p < 1, where w = T In this paper, based on the low-rank property of the non- (w1; w2; : : : ; wminfm;ng) is a non-negative weight vector local self-similar patches and the different noise strength and σi is the i-th singular value of Z. The weighted nuclear across each channel, we propose a multi-channel weighted norm [Gu et al., 2017] is a special case of the weighted Schat- Schatten p-norm minimization (MCWSNM) model for RG- ten p-norm when the power p = 1. It has been proven that the B color image denoising. More specifically, considering a Schatten p-norm with 0 < p < 1 is a better approximation of small local RGB patch in a noisy image, we first find its non- rank function than the nuclear norm [Zhang et al., 2013]. local similar cubic patches in a search window with an appro- Intuitively, when denoising a noisy color image, if a chan- priate size. These similar cubic patches are then vectorized nel is heavily polluted, the contribution of this channel to the and concatenated to construct a noisy low-rank matrix; then final denoised results should be less, and vice versa.