A Recursive Edge Detector For Filter Array Image Baptiste Magnier, A. Aberkane, N. Gorrity

To cite this version:

Baptiste Magnier, A. Aberkane, N. Gorrity. A Recursive Edge Detector For Image. ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing, May 2020, Barcelona, France. pp.2543-2547, ￿10.1109/ICASSP40776.2020.9053743￿. ￿hal-02920074￿

HAL Id: hal-02920074 https://hal.mines-ales.fr/hal-02920074 Submitted on 3 Jun 2021

HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. A RECURSIVE EDGE DETECTOR FOR COLOR FILTER ARRAY IMAGE

B.Magnier A. Aberkane, N. Gorrity

IMT Mines Ales` Audensiel, 88 r. du Domeˆ Poleˆ R&D 6 av. de Claviere` 30319 Ales,` France 92100 Boulogne-Billancourt, France

ABSTRACT for edge detection: (i) Deriche-based Luminance (DL) tech- nique estimates partial derivative images of luminance thanks Most of embedded cameras use a single sensor to capture im- to Deriche recursive filters [8], so that a gradient image is ages through a color filter. They produce special images with computed, (ii) Deriche-based Color (DC) derivative method only one color component per pixel. Missing data are usu- also uses Deriche filters to compute partial derivatives of , ally estimated through a demosaicking process, but this takes and channels so that a structure tensor is applied undesirable computation time and may generate undesirable [9]. Despite the best results offered by DL, methods comput- color artifacts. Many embedded systems under real-time con- ing luminance derivatives from Bayer CFA were not studied straints could use this type of camera for purposes like edge deeply enough. Aberkane et al. [7] showed the failure to de- detection. In this paper, a new edge detection method is pro- tect isoluminant edges. Moreover, a raw image of luminance posed for the computation of partial derivative images. This could be directly captured with a camera. As a algorithm is tested on a large set of synthetic images where Bayer CFA image offers color data, calculating color deriva- edge ground truth is unquestionable. The exploitation of the tives is more relevant. This paper presents a new algorithm raw data of images allows not only to drastically reduce com- for the computation of color partial derivatives directly from putational time, but also to get edge detection results even Bayer CFA image, with a low computational complexity. The more precise than using certain demosaicking-based methods. proposed algorithm offers a better overall performance com- Index Terms— Edge detection, Bayer CFA, rotated filter. pared to edge detection schemes applied to democaicking im- age or to CFA image with fewer operations per pixel. 1. INTRODUCTION AND MOTIVATIONS The most of digital devices such as smartphones and tablets 2. DERICHE FILTERS AND COLOR GRADIENT embed single-CCD (CMOS) cameras. They use only one 2.1. The recursive Deriche filter charge-coupled device, to which a Color Filter Array (CFA) Deriche [10] created an optimal edge detection filter accord- is superposed [1]. CFAs are mosaics of color filters placed ing to Canny’s criteria [11]: unicity, good detection and good over each pixel sensor. Different configurations exist, the localization. This filter can be implemented recursively al- most widely used in literature remains the Bayer filter [2]. lowing reducing computation time [8]. Deriche used a gra- The disposition is arranged on a base of 2×2 patterns, favor- dient approach based on two separable filters for the com- ing the green channel, with 50% of green data(G), 25% of red putation of partial derivative images: (i) a smoothing filter (R) and 25% of blue (B). Each pixel of the CFA image only s (t) approximating a Gaussian function, (ii) a derivative fil- contains one color component out of three (see Fig.1). α ter d (t) which is the derivative of s (t): The inversion process of the subsampling corresponds to α α s (t) = m · (1 + α · |t|) e−α|t| (1) a demosaicking. This allows to visualize color images, so α s 2 −α·|t| the missing data can be estimated, creating a fully-defined dα(t) = −md · α · t · e , (2) RGB . Many demosaicking algorithms have been presented in literature [1, 3, 4, ?]. Despite their efficiency to where α represent the standard deviation of the filter and produce visually satisfying images, their failure to reconstruct high-frequency data by interpolations is detrimental to low- level analysis like edge detection. This was shown by Losson et al. [5] and confirmed by Breier et al. [6]. Moreover, those algorithms inevitably consume additional computation time. Aberkane et al. [7] showed that directly edge detection in those images is possible with a very close or even better precision than segmentation after demosaicking. They intro- (a) ICFA CFA image (b) I¨G green channel (c) I¨R,B , red and blue channels duced two approaches to use the raw data of CFA images Fig. 1: Decomposition of Bayer filter array. 2 (1−e−α) be gathered in a vector Drot(x, y): (ms, md) are two normalization parameters: md = α2·e−α −α 2   (1−e ) rot IX,α0 (x, y) and ms = −α −2α . Thus, the partial derivative im- D (x, y) = 1+2·α·e −e I 0 (x, y) age in x direction is obtained by: (i) applying the smoothing Y,α filter sα(t) vertically on the original image, (ii) processing The equivalent vector D(x, y) in the straight system is com- this smoothened image horizontally with the derivative filter puted by rotating the original coordinates system by an angle ◦ dα(t). Afterward, the partial derivative image in y direction φ = −45 . To respect spatial ratio between both straight√ and can be obtained using the same process perpendicularly. rotated systems, a homothecy must be done with a factor 2. These filters can be implemented recursively [10]. Coeffi- The required transformation matrix is easily calculated by: cients defining the deviation of both smoothing and derivative √  cos(φ) sin(φ) D(x, y) = 2 · · Drot(x, y) (4) functions sα(t) and dα(t) depend on a single real positive pa- − sin(φ) cos(φ) rameter α. High values of α (α>1) enhance edge localization G G RB RB quality, whereas small values of α (α<1) enhance edge de- Going through this process for (IX,α, IY,α) and (IX,α, IY,α) tection quality with poorer localization. G G RB gives the images of straight partial derivatives Ix,α, Iy,α, Ix,α RB and Iy,α in half pixels for the G and RB componants. These 2.2. Structure Tensor: Di Zenzo gradient partial derivatives in odd pixels (for green) and partial deriva- A basic approach to compute a gradient of a color image tives in even pixels (for others) must be estimated. Thereafter, is to combine first order derivatives issued of each chan- missing values are calculated by bilinear interpolation in the nel. The best known color gradient was introduced by Di direction perpendicular to the derivation direction. The fully- ˇG ˇG ˇRB ˇRB Zenzo [9] which uses the partial derivatives along x and y of defined partial derivative images Ix,α, Iy,α, Ix,α and Iy,α are each red, green and blue component. Those partial deriva- computed by: R G B n I (x, y) where P (x, y) is defined (not empty) tives are gathered in two vectors Ix = (Ix ,Ix ,Ix ) and Iˇx,α(x, y)= x,α R G B 1 [I (x, y − 1) + I (x, y + 1)] elsewhere. Iy = (I ,I ,I ). For each pixel, the gradient direction θ is 2 x,α x,α y y y (5) chosen so the first fundamental form dI2 is maximized:  I (x, y) where P (x, y) is defined (not empty) 2 2 2 ˇ y,α dI (θ) = f · cos (θ) + 2 · g · cos(θ) sin(θ) + h · sin (θ) (3) Iy,α(x, y)= 1 2 [Iy,α(x − 1, y) + Iy,α(x + 1, y)] elsewhere. (6) P k2 P k k where f=Ix ·Ix= k=R,G,B Ix , g=Ix ·Iy= k=R,G,B Ix Iy P k2 Color gradient is computed using the structure tensor (de- and h=Iy ·Iy= I . This structure tensor provides k=R,G,B y tailed in Section 2.2) by considering that red and blue deriva- in every pixel the gradient direction θ∗ = argmax dI2(θ) tive images are equal to the derivative images. and its tied norm |∇I| = |dI2 (θ∗) |. θ∈[−π,π] 3.1. Application to edge detection 3. EDGE DETECTION DIRECTLY ON CFA IMAGES Starting from Bayer [2] observation: green channel can be In both methods, Di Zenzo gradient (detailed in Section 2.2) assimilated to luminance, red and blue channels to chromi- is calculated from the estimated color partial derivative im- ˇR ˇR ˇG ˇG ˇB ˇB nance. SEDD (Smoothed component Estimation and Deriche ages Ix , Iy , Ix , Iy , Ix and Iy . Thus, edges can be ex- Derivative) computes a gradient by estimating bilinear inter- tracted by thresholding local maxima of the gradient norm in polations [7]. Here, we propose to decompose the CFA image the gradient direction [11]. Finally, to eliminate undesirable edges due, for example, to noise, the image of edges must be into two channels I¨G and I¨R,B which both contain half of all thresholded either with a classic thresholding process or by pixels arranged in a quincunx lattice (see Figs.1(b) and 1(c)). hysteresis [11]. The algorithm introduced in this paper process pixels in diag- onal directions. We introduce a rotated Deriche filter directly adapted from eq.√ (1) and (2). As diagonal pixels are separated by a distance of 2 instead of 1 in usual directions (the gap rotation mirror effect between each pixel of a color channel√ in a CFA image), the parameter α must be modified so α0 = 2α, [7]. ¨G Each channel I¨G and I¨R,B is diagonally smoothed us- (a) rotation of I G ing eq. 1. The diagonal derivatives images IX,α0 (x, y) and RB IX,α0 (x, y) are obtained by filtering the smoothed images us- ing eq. 2 in the perpendicular direction. Fig. 2 shows the rotation mirror effect rotation process for compute those diagonal derivatives. Those partial derivatives must be combined as with the (b) rotation of I¨R,B structure tensor using horizontal and vertical partial deriva- ¨G ¨R,B tives. The rotated partials derivatives of a pixel P (x, y) can Fig. 2: The rotation process of I and I . " h oetesgetto sqaie ssial.O the of detection On poor a to if suitable. finally, corresponds 1, contours, 0 as to to qualified close is score is a score contrary, segmentation evaluation the the more closer the the measure score; the dissimilarity Therefore, pixel a between [15]. distance Euclidian minimal [16], normalized for the is, measure [16], evaluation in described Thus, eetdcnormpo nimage an and pixel of truth map ground contour the detected to Let corresponding of map [15]. positions known contour true perfectly are the edges where computational the images high synthetic are very using Evaluations reported a time-consuming. it has makes visu- which but produces complexity and images, domain satisfying residual ally a in [14], (ARI) interpolates Interpolation Residual Adaptative perpendicular gradient. direction the the to in in consisting interpolation procedure bilinear demosaicking a making simple a is demosaick- image, to ing applied Adaptive [13], (ACPI) process. Interpolation demosaicking Plane after Color ap- images also color is filter on Deriche plied [12]. detection adaptive di- edge a for data with filter derivatives raw Deriche partial use image the to Derivative estimate propose to rectly image, Deriche images. CFA real to and applied and (SEDD), Estimation synthetic component on out Smoothed carried are Experiments | P F where D .EAUTO N XEIETLRESULTS EXPERIMENTAL EVALUATION AND 4. i.3 Fig. f rpadelreeto h g lsia dedtcin()Cascleg eeto i EDeg eeto iety()RCeg eeto directly detection edge RGC (j) directly detection edge SEDD (i) detection edge Classical (h) detection edge Classical (g) the of enlargement and Crop (f) rudtuh(mg netd fe CIdmsikn,atrAIdmsikn,o h ae F mg,o h ae F image. CFA Bayer the on image, CFA Bayer the on demosaicking, ARI after demosaicking, ACPI after inverted) (image truth ground c

• • • • N (Gt,Dc) | measure score 0.75 0.85 0.95 0.7 0.8 0.9 G 1 · | · | 0 p N ( Negative True ( Negative False ( Positive False ( Positive True X ∈ oprsnadeauto feg eeto using detection edge of evaluation and Comparison : t D ( ( standarddeviationoftheGaussiannoise( and G κ 1 c eoe h adnlt fast and set, a of cardinality the denotes P F t + 1 (a) D , 2 D κ , N c c κ = ) N F ai vlaini opsdo statistics: of composed is evaluation basic a , crs()TP/ (b) scores 3 P F 1 ) P F ∈ 4 · P F P T d P F ]0 N T N F G 2 = , 5 t + σ 1] ,cmu onsof points commun ), 1 N F ( ) ,suiu eetdegsof edges detected spurious ), p ,cmo o-depoints. non-edge common ), > N F 2 N F ,msigbudr onsof points boundary missing ), ) 6 ersn w cl parameters scale two represent =0 + TP/|G| t 0.4 0.5 0.6 0.7 0.8 0.9 1 0 · , | N F N G then 0 standarddeviationoftheGaussiannoise( t 1 | acltsastandardized a calculates or · I p X N ∈ oprn ie by pixel Comparing . 2 > P F G =1 t G 3 + 1 | N G t . t etereference the be 4 G | 0 κ dedetection edge p : N F t d 5 n set a and and σ A ) 1 · ( d 6 p D D 2 ) D FP D 0.5 1.5 2.5 c c 0 1 2 3 sthe is 0 x 10 ( c , c D p 4 , the ) standarddeviationoftheGaussiannoise( (7) c 1 A # , c ubro P(d) FP of Number (c) , α =2 2 h rdeteauto scmue sfollows: as computed is evaluation gradient in the chains contour of gra- the set the available concerns is measure evaluation angle, image last dient CFA the Bayer Also, original 3(e). the Fig. and 3(f) Fig. of pixels (TP/ edge of displayed number total the segmentation to relative better itives each For 16]. to [17, tied score highest the a gets map be contour The- must ideal measure a [11]. the of segmentation hys- compared, objectively binary by be a threshold to obtain oretically, a to then applied non-maxima, is sup- a teresis local after the extracted are of contours pression the that, For way. pervised in ned silsrtdi is4c n d,demosaick- (d), and Figs.4(c) in illustrated segmenta- as before Indeed, demosaicking tion. of instead features directly on computed is which edge channels. process to full ACPI contrary and image), ARI CFA after (the detection an pixels regarding empty computed with is the because TPs. directions and ent FPs of terms in the Finally, statistics extraction good edge also better proposed obtains a The method process. illustrates demosaicking right after top edge detected the than on (j) 3 Fig. of hrfr,rgrigteproposed the regarding Therefore, the ( of noise deviation Gaussian standard presented the are to scores according (a-d) These Figs.3 0. in to close remains direction directions two other the and 0 mates to belong of angles angles identical for 0 from ranges −→ linearly evaluation This c on centered window η E . 3 where 01 k p ( C N h ubro otu iesin pixels contour of number the and eetdegsaete oteniyiaewith image noisy the to tied are edges Detected . h betv eei ogttebs otu a nasu- a in map contour best the get to is here objective The eut rsne nFg4ilsrtsteitrs oextract to interest the illustrates Fig.4 in presented Results 4 D c upromohrsoe.h emnainaalbein available segmentation scores.The other outperform η , N 5 η −→ d σ ) d = ) k h as oiie F)adpretg ftu pos- true of percentage and (FP) positives false the , E(D , η) k c 6 0.05 0.15 0.25 ersnsacnorpxlblnigto belonging pixel contour a represents 0.1 0.2 o1frage htdfes oethat Note differs. that angles for 1 to standarddeviationoftheGaussiannoise( 0 RGC η 0 E(D , η) |C c 0.05 0.15 0.25 0.1 0.2 1] Once [18]. 0 G D 1 0 σ c 1 t | .Tegon rt mg saalbein available is image truth ground The ). 0 180 [0; de oteCAiaefrec channel. each for image CFA the to added ) standarddeviationoftheGaussiannoise( 1 ehdotismr rosfrtegradi- the for errors more obtains method · p E 2 p ∈C X ( , 2 D D D η −→ 3 c D d ◦ c η , c k 3 [ D d c twihtesprie evaluation supervised the which at k X ) n hnoedrcinapproxi- direction one when and h rdetoinainof orientation gradient the c 4 ∈ ie,a es iesprchain), per pixels 2 least at (i.e., crs()Oiia mg 800 image Original (e) scores 4 W scetd considering created, is 5  180 5 1 σ ) W − RGC 6 ◦

6 σ iu h eta pixel. central the minus , h vlaino these of evaluation the , ) | 90 ◦

RGC SEDD RI ACPI | − ehd h scores the method, 90 −→ η p ◦ − η −→ η −→ G d d W t k × k C σ 3 a , | | r also are 800 D and d 4 =  k c the , /c and × −→ η . k 3 p , ing creates undesirable colors, especially for edges or rup- Algorithms ACPI [13] ARI [20] SEDD [7] RGC ture points. Each result is displayed with the same thresh- old. Thus, obtained edges with the proposed technique are demosaicking 30 1779 0 0 straighter regarding the vertical boards whereas contours of the details in the gray object on the right are well extracted Deriche smoothing 90 90 40 30 whereas demosaicking methods create a checkerboard effect. Deriche derivation 96 96 96 42

4.1. Computational complexity Total 216 1965 136 72 For real-time constraints, less time-consuming algorithms are Table 1: Number of elementary operations per pixel required preferable. This parts reports complexity of the compared al- for the different color partial derivative computation methods. gorithms: ACPI, ARI, SEDD and RGC using and the same color edge detection process. Their complexity only differ in Y smoothened images. The diagonal derivative filter requires (i) demosaicking process for ACPI and ARI, and (ii) partial 2 · 16 operations per pixel. Vector conversion in eq. (4) takes color derivative computation for SEDD and RGC. two multiplications and one addition per coordinates, which Tab. 1 shows the computational complexity of those algo- makes 2 · 3 operations per pixel. Interpolation of missing rithms. The complexity is evaluated as the number of required derivative values in eq. (5) and (6) takes 2 operations per co- elementary operations per pixel: addition, subtraction, multi- ordinate in half pixels of both channels, which is equivalent plication and division, each one being weighted similarly, as 2·2 to 2 · 2 = 4 operations per pixel. Hence derivation step of detailed in [7]. The expressions of the recursive Deriche (re- RGC takes 2 · 21 operations per pixel. ARI demosaicking is cursive of order 2) equations require seven additions and eight far more complex because its complexity as 1245 operations multiplications per pixel for the smoothing filter, seven addi- to estimate each missing green value and 534 operations to tions and nine multiplications per pixel for the derivative filter estimate a red or blue value. Thus, RGC allows to dramat- [10]. So 15 operations for left to right, and 15 from right to ically reduce the computational complexity. The number of left (causal + anti-causal), i.e., 30 operations for the smooth- elementary operations required is (i) three times lower than ing filter (for each (RGB) channels, i.e., 90 operations). In ACPI, (ii) almost twenty times lower than RI, and (iii) twice the same way, 96 operations are required per pixels for the lower than SEDD which is the method with no demosaicking derivative filter using a full color image (after demosaicking). proposed in [7] that offered the best edge quality with the In RGC, a diagonal Deriche smoothing filter is processed lowest complexity among color derivative methods. on the CFA image, which takes 15 operations for each X and 5. CONCLUSION Aberkane et al. in [7] made a first investigation of gradient- oriented edge detection methods that exploit raw CFA images and bypass the demosaicking step, with the main goal of re- ducing computational time. Their best color partial deriva- (a) Color image [19] (b) Bayer CFA image tive computation method SEDD still offered an average per- formance lower than ACPI, which uses standard Deriche fil- ters after a simple but efficient demosaicking process. This work aimed to propose other CFA-based multi-channel partial derivative computation methods with better performances and lower computational time than those using demosaicking. A (c) Demosaicked image with ACPI [13] (d) Demosaicked image with ARI [20] new approach is presented in this paper, computing the partial derivatives of the green channel and the union of red and blue channels. This innovative approach is the use of diagonal De- riche filters also implementable with standard Deriche filters on a rotated image. This method is tested on a dataset of syn- (e) Classical edge detection in (c) (f) Classical edge detection in (d) thetic images. and the influence the Gaussian noise outper- α=2.01 α=2.01 after ACPI demosaicking, after ARI demosaicking, forms other methods, even though the gradient and its direc- tion are estimated on images containing empty pixels. Even- tually computational complexity showed that RGC allows to considerably reduce the number of elementary operations per pixel compared to demosaicking process. Several applica- (g) SEDD edge detection on (b), α=2.01 (h) RGC edge detection on (b), α=2.01 tions may be investigated using the proposed method, such Fig. 4: Comparison of classical edge detection using color that develop a demosaicking technique using contour orienta- Deriche filter and detection directly on Bayer CFA images. tions [21] or up-sample images [22]. 6. REFERENCES [16] H. Abdulrahman, B. Magnier, and P. Montesinos, “From contours to ground truth: How to evaluate edge [1] R. Lukac and K. N. Plataniotis, “Color filter arrays: De- detectors by filtering,” J. of WSCG, vol. 25, no. 2, pp. sign and performance analysis,” IEEE TCE, vol. 51, no. 133–142, 2017. 4, pp. 1260–1267, 2005. [17] B. Magnier, “Edge detection evaluation: A new nor- [2] B. E. Bayer, “Color imaging array,” in U.S. Patent malized figure of merit,” in IEEE ICASSP, 2019, pp. 3971065, 1976. 2407–2411.

[3] D. Menon and G. Calvagno, “Color Image Demosaick- [18] B. Magnier, H. Abdulrahman, and P. Montesinos, “A ing: An Overview,” Image Communication, vol. 26, no. review of supervised edge detection evaluation methods 8-9, pp. 518–533, Oct. 2011. and an objective comparison of filtering gradient com- putations using hysteresis thresholds,” Journal of Imag- [4] H. Maˆıtre, From photon to pixel: the digital camera ing, vol. 4, no. 6, pp. 74, 2018. handbook, John Wiley & Sons, 2017. [19] B. Magnier, “An objective evaluation of edge detection [5] T. Ehret and G. Facciolo, “A study of two cnn demo- methods based on oriented half kernels,” ICISP, pp. 80– saicking algorithms,” Image Processing On Line, vol. 2, 89, 2018. pp. 220–230, 2019. [20] Eastman Kodak and Various photographers, “Kodak [6] O. Losson, L. Macaire, and Y. Yang, “Comparison of Photo CD PCD0992, Access Software & Photo Sam- color demosaicing methods,” Advances in Imaging and pler, Final version 2.0,” [CD-ROM, Part No. 15-1132- Electron Physics, vol. 162, pp. 173–265, July 2010. 01], 1991.

[7] M. Breier, C. Haas, W. Li, and D. Merhof, “Color filter [21] D. Kiku, Y. Monno, M. Tanaka, and M. Okutomi, “Be- arrays revisited – Evaluation of Bayer pattern interpola- yond : Residual interpolation for color tion for industrial applications,” in IEEE INDIN, 2016, image demosaicking,” IEEE TIP, vol. 25, no. 3, pp. pp. 52–57. 1288–1300, 2016.

[8] A. Aberkane, O. Losson, and L. Macaire, “Edge de- [22] P. Getreuer, “Image demosaicking with contour sten- tection from Bayer color filter array image,” J. of Elec. cils,” Image Processing On Line, vol. 2, pp. 22–34, Imag., vol. 27, no. 1, pp. 011006, Jan. 2018. 2012.

[9] R. Deriche, “Fast algorithms for low-level vision,” IEEE [23] J. Seiler, M. Jonscher, M. Schoberl,¨ and A. Kaup, “Re- TPAMI, vol. 12, no. 1, pp. 78–87, 1990. sampling images to a regular grid from a non-regular subset of pixel positions using frequency selective re- [10] S. Di Zenzo, “A note on the gradient of a multi-image,” construction,” IEEE TIP, vol. 24, no. 11, pp. 4540– CVGIP, vol. 33, no. 1, pp. 116 – 125, 1986. 4555, 2015.

[11] R. Deriche, “Using canny’s criteria to derive a recur- sively implemented optimal edge detector,” IJCV, vol. 1, no. 2, pp. 167–187, 1987.

[12] J. Canny, “A computational approach to edge detection,” IEEE TPAMI, pp. 679–698, 1986.

[13] A. Aberkane, Edge detection in CFA images, Ph.D. the- sis, Universite´ Lille 1 Sciences et Technologies, 2017.

[14] J. F. Hamilton and J. E. Adams, “Adaptive color plan interpolation in single sensor color electronic camera,” US patent 5,629,734, to Eastman Kodak Co., Patent and Trademark Office, Washington D.C., 1997.

[15] Yusuke Monno, Daisuke Kiku, Masayuki Tanaka, and Masatoshi Okutomi, “Adaptive residual interpolation for color and multispectral image demosaicking,” Sen- sors, vol. 17, no. 12, 2017.