https://doi.org/10.2352/ISSN.2470-1173.2019.14.COLOR-093 © 2019, Society for Imaging Science and Technology Analysis of illumination correction error in camera color space Minji Lee12, and Byung-Uk Lee1 1Department of Electronics Engineering, Ewha Womans University, 52 Ewhayeodae-gil, Seodaemun-gu, Seoul 03760, Republic of Korea 2Current Affiliation: LG Electronics, 10 Magokjungang 10-ro, Gangseo-gu, Seoul 07796, Republic of Korea Abstract and camera sensitivity function can be approximated to 3, the Achieving color constancy is an important step to support actual illumination can be expressed as a 3×3 illumination visual tasks. In general, a linear transformation using a 3× 3 correction matrix without error. In another study, Cheng et al. [8] illuminant modeling matrix is applied in the RGB color space of a found that the accuracy of illumination estimation varies with camera to achieve color balance. Most of the studies for color illumination, and the diagonal matrix transformation under certain constancy adopt this linear model, but the relationship of illumination conditions can estimate RGB illumination with small illumination and the camera spectral sensitivity (CSS) is only error. Cheng et al. also reported that this issue is related to intricate partially understood. Therefore, in this paper, we analyze linear relationship between the illumination spectrum and CSS. combination of the illumination spectrum and the CSS using In this paper, hyperspectral data under actual illumination is hyperspectral data that have much more information than RGB. analyzed, which enables an investigation beyond the limitations of After estimating the illumination correction matrix we elucidate 3D RGB color space. Through this work, we identify the reason for the accuracy dependence on illumination spectrum and the camera the observation in the Cheng’s work [8], why diagonal model is as sensor response, which can be applied to CSS. effective as the full matrix for some illumination spectrum. Unlike the Chong’s mathematical analysis [7], which evaluates the illumination correction matrix on the assumption that the camera Index Terms — Color constancy, illuminant estimation, sensitivity function can have low rank, this paper assumes a camera spectral sensitivity general situation where a camera sensitivity function is given, and identify the reasons for the varying accuracy of RGB illumination 1. INTRODUCTION AND RELEATED WORK estimates. The human visual system can compensate for the effect of The structure of this paper is as follows. In Section 2, we illumination, so that color of an object can be perceived describe existing models for estimating the illumination correction consistently even though illumination changes. However, the matrix in RGB color space using hyperspectral data. In Section 3, image acquired through a camera, which captures physical signal we analyze errors of a 3x3 RGB illumination estimation matrix. intensity of the receiving light, is influenced by the illumination Finally, the conclusions and discussions of this study follow in spectrum. Finding the actual reflectance of an object by Section 4. compensating for the color of such illumination has been studied in the field of computational color constancy [1]. Implementation of color constancy can be divided into two stages: first, spectrum of 2. ESTIMATION OF RGB ILLUMINATION the illumination is estimated, and then, color of the image is corrected based on the estimated lighting. For image correction, 2.1 Hyperspectral to RGB illumination modeling white balancing is generally applied using a 3×3 diagonal matrix We analyze existing methods of estimating the illumination [2]-[4]. White balance aims at recovering achromatic color from an correction matrix in the RGB color space with hyperspectral data, image taken under unknown illumination. Since the method of which enables an accurate description of color correction error. correcting illumination using the diagonal matrix does not correct Under Lambertian reflectance image spectrum can be modeled as a colors other than achromatic colors, there is a limitation in that it is multiplication of the spectral components of the illumination and not possible to achieve color correction for the entire color gamut the reflectance of the object. Let D represent a hyperspectral image presented in the image. with an m×n matrix, where m is the number of spectral bands and In order to improve the color correction performance of the n is the number of pixels in the image. D can be expressed as a diagonal matrix transformation model, narrowing the wavelength product of the reflectance R whose dimension is m×n and the width of the sensitivity function of each RGB filter of the camera illumination L which is an m× m diagonal matrix. by a linear transformation is proposed [5]. In addition, there is a color correction method that realizes color consistency by 퐃퐦×퐧 = 퐋퐦×퐦퐑퐦×퐧 (1) estimating 9 elements of a 3×3 full matrix considering the correlation between RGB color channels, and is generally more RGB image is a projection of the hyperspectral data to the effective than diagonal white balancing method [6][12]. T T T T three-dimensional color space using the CSS S = [sr sg sb] . The However, despite the widespread use of this linear RGB image D3 is 3×n matrix and the RGB reflectance R3 is a 3×n transformation model, most studies focus on how to implement the matrix. color constancy in the RGB color space; the underlying principles for color correction is not fully investigated yet. In contrast, Chong [7] proved that if the rank of the tensor of illumination, reflectance, IS&T International Symposium on Electronic Imaging 2019 Color Imaging XXIV: Displaying, Processing, Hardcopy, and Applications 093-1 ퟐ ퟐ Figure 1. Residual error ‖퐌퐒퐑 − 퐒퐋퐑‖푭 of 퐌퐝 and 퐌+ for SFU illuminants Figure 2. Residual error ‖푴푺푹 − 푺푳푹‖푭 of 푴풅 and 푴+ for black-body radiation Then we have We can also estimate the full matrix M+ which minimizes the error between SLR and MSR. The least squares method using the 퐃ퟑ = 퐒퐃 = 퐒퐋퐑 Moore-Penrose pseudo inverse can estimate M+ which minimizes the Frobenius norm of the difference between two matrices: `퐑ퟑ = 퐒퐑 (2) ‖ ‖ퟐ Here, D3 denotes the RGB image acquired by the camera, 퐚퐫퐠 퐦퐢퐧 퐌+퐒퐑 − 퐒퐋퐑 퐅 (5) 퐌+ while R3 represents the RGB image under ideal white illumination. Typical linear white balance algorithm approximates the lighting 퐌 = 퐒퐋퐑퐑퐓 퐒퐓(퐒퐑퐑퐓퐒퐓)−ퟏ (6) by a linear transformation using a 3×3 illumination matrix M in the + RGB color space. Please note that the illumination matrix M+ is influenced by the correlation of the reflectance RRT. 퐃ퟑ = 퐒퐋퐑 ≅ 퐌퐑ퟑ = 퐌퐒퐑 (3) In this paper, hyperspectral data of 400 nm to 720 nm corresponding to the visible light range is sampled with 10 nm Since the degree of freedom of M is 9 and the number of interval, therefore experiment is conducted using spectrum constraint is 3n, there is no illumination spectrum satisfying this information of 33 bands. Since the image is expressed as a product over-determined problem [9]. Therefore, SLR and MSR do not of the reflectance and the illumination spectrum, all the match because there is no unique solution to project the hyperspectral data are normalized to the maximum value of 1 to illumination perfectly onto the three dimensional space. facilitate the experiment and the result analysis. We use the reflectance of a 24-patch Macbeth chart, which is often used as 2.2 Estimation of the illumination matrix reference measurements in color calibration. The camera White balancing, which is commonly used in color constancy, sensitivity function for projecting hyperspectral data into the RGB calibrates color usually by a diagonal matrix. This method aims at color space uses the dataset measured by Jiang et al. [10]. The adjusting the illumination vector on the white reference in the RGB experiment of this work is based on Cheng et al. [8], which has color space of the camera to be aligned to the direction of R = G = 101 illuminant spectra measured in a variety of weather conditions B, which is an achromatic vector. Mathematically, this means that [11], the illumination correction matrix M of the RGB color space the scale of each color channel is independently adjusted by is estimated for this hyperspectral data, and the RMSE between multiplying the diagonal element having the weights for each RGB MSR and SLR is calculated to analyze the accuracy of illumination illumination, while the correlation among RGB colors of estimation. illumination is ignored. Diagonal element mdi for each channel The performance of the diagonal and full matrix model for i= r, g, b of RGB illumination matrix Md can be calculated using illumination is compared as in [8]. Figure 1 is a graph comparing camera sensitivity function and illumination spectrum. Where the accuracy of Md and M+ estimated using SFU illuminants and T wm×1 = [1, ⋯ ,1] is the m-dimensional ideal white illumination, two different camera sensitivity functions. Illumination spectrum is T and lm×1 = diag(L) = [l1, ⋯ , lm] is an m × 1 column vector sorted according to the CCT (correlated color temperature) and consisting of the diagonal components of the illumination matrix. CSS of Canon1D Mark III and Sony Nex5N was used. The Then the diagonal element is: horizontal axis of the graph shows the CCT of the SFU illuminants in the descending order of CCT as in [8], and the vertical axis 풔푻풍 represents the RMSE for each illumination. The result shows that 풎 = 풊 퐝풊 풔푻풘 the modeling RGB illumination using full matrix is better than the 풊 diagonal method for all the illuminants. This is because the diagonal matrix does not include correlation information between 풎 ퟎ ퟎ 퐝풓 color channels. However, even in the case of M , a perfect ퟎ 풎 ퟎ + 퐌퐝 = [ 퐝품 ] (4) illumination modeling in the RGB space is not possible.