Sensor Interpixel Correlation Analysis and Reduction for Color Filter Array High Dynamic Range Image Reconstruction

Home , Metamerism (color), Telecentric lens

Sensor interpixel correlation analysis and reduction for color filter array high dynamic range image reconstruction

Mikael Lindstrand

The self-archived postprint version of this journal article is available at Linköping University Institutional Repository (DiVA): http://urn.kb.se/resolve?urn=urn:nbn:se:liu:diva-154156

N.B.: When citing this work, cite the original publication. Lindstrand, M., (2019), Sensor interpixel correlation analysis and reduction for color filter array high dynamic range image reconstruction, Color Research and Application, , 1-13. https://doi.org/10.1002/col.22343

Original publication available at: https://doi.org/10.1002/col.22343 Copyright: This is an open access article under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License, which permits use and distribution in any medium, provided the original work is properly cited, the use is noncommercial and no modifications or adaptations are made.

Received: 10 April 2018 Revised: 3 December 2018 Accepted: 4 December 2018 DOI: 10.1002/col.22343

RESEARCH ARTICLE

Sensor interpixel correlation analysis and reduction for color filter array high dynamic range image reconstruction

Mikael Lindstrand1,2

1gonioLabs AB, Stockholm, Sweden Abstract 2Image Reproduction and Graphics Design, Campus Norrköping, ITN, Linköping University, High dynamic range imaging (HDRI) by bracketing of low dynamic range (LDR) Linköping, Sweden images is demanding, as the sensor is deliberately operated at saturation. This exac- Correspondence erbates any crosstalk, interpixel capacitance, blooming and smear, all causing inter- gonioLabs AB, Stockholm, Sweden. pixel correlations (IC) and a deteriorated modulation transfer function (MTF). Email: [email protected] Established HDRI algorithms exclude saturated pixels, but generally overlook Funding information gonioLabs AB, Stockholm, Sweden IC. This work presents a calibration method to estimate the affected region from saturated pixels for a color filter array (CFA) sensor, using the native CFA as a matched filter. The method minimizes color crosstalk given a set of candidates for proximity regions, and requires no special setup. Results are shown for a 21-bit HDR output image with improved color fidelity and reduced noise. The calibration reduces IC in the LDR images and is performed only once for a given sensor. The improvement is applicable to any HDRI algorithm based on CFA image bracketing, irrespective of sensor technology. Generalizations to subsaturated and supersaturated pixels are described, facilitating a suggested irradiance-exposure dependent point spread function charge repatriation strategy.

KEYWORDS blooming, color filter array, crosstalk, high dynamic range, Interpixel correlation, saturation

1 | INTRODUCTION likely) several pixels, and an underestimation of irradiance in the output of the source pixel. These overestimations and Because interpixel correlation (IC) is sparsely treated in the underestimations lower the signal-to-noise ratio (SNR) of the body of previous work, with individual papers generally cover- detected signal and worsen the modulation transfer function ing only selective relevant aspects, the introduction begins with (MTF) of the sensor. Note, for imaging fidelity enhancement an overview of the problem. The introduction continues with operations in general SNR and MTF are often opposite aspects, IC causes and remedies and finally ends pointing to the contri- one is improved on the expense of the other. However, for low bution of this work. noise applications in general, minimizing IC results both improved SNR and MTF, which makes the minimization of IC particularly relevant when aiming for high fidelity imaging. 1.1 | Sensor interpixel correlation The term IC is not yet an established concept within the IC in an image sensor is an undesired statistical dependence imaging science community. It is used only in a more infor- between sensor pixel outputs. In a sensor of nonzero IC, an mal manner, without a clear definition. Within this work, we increase of the irradiance in exactly one source pixel results in define IC as an aggregate of six established distortion con- an overestimation of irradiance in the output of one or (most cepts from the image sensor literature: optical crosstalk,

This is an open access article under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License, which permits use and distribution in any medium, provided the original work is properly cited, the use is non-commercial and no modifications or adaptations are made. © 2019 The Authors. Color Research & Application published by Wiley Periodicals, Inc. Color Res Appl. 2019;1–13. wileyonlinelibrary.com/journal/col 1 2 LINDSTRAND charge diffusion crosstalk, the so-called “brighter-fatter collected,2 and the nonlinear brighter-fatter effect, which is a effect”, blooming, interpixel capacitance, and smear. These radiant energy dependent widening of the point spread func- concepts represent different laws of physics; they may tion (PSF).3 include significant elements of nonlinearities and represent Blooming is caused by charge overflow from a full-well stochastic as well as deterministic correlation processes, all pixel into neighborhood pixel(s).4 In Figure 1, we symbolize of which present challenges for analysis and reduction (dec- this by an avalanche diode, DB, which is nonconducting in orrelation) of IC. It is worth noting that this work aims at the back direction (no blooming) up to the avalanche voltage being sensor technology neutral, even though the described (blooming radiant energy), above which the diode is con- concrete implementation has some technology specific ducting (blooming). Note that the diode model in this illus- details. No single technology in current use suffers all six tration is conceptual and only describes blooming from p0 to distortion concepts mentioned above, but every technology p1, that is, cases where the photon is entering pixel p0. may suffer from a majority of them. Therefore, IC has a Interpixel capacitance5 is a coupling in CMOS and complex, multidimensional character. hybrid CMOS pixel arrays employing source follower pixel In Figure 1, two neighboring pixels are shown, with the amplifiers.6 photon entry at pixel p0. Photons and charges ending up in Smear is due to both electrical and optical effects. Its the intermediate region, i0, only reduce the photon transfer main cause is a lateral diffusion of charges between the (sensor efficiency), while photons and charges crossing over photosensor and the CCD shift register, but it is also caused to the neighboring pixel, p1, also contribute to the IC. by light piping underneath or directly through the light Optical crosstalk may either be circuit-related scattering shields that may cover these registers.7–9 Although interpixel in the air-sensor surface boundary causing a photon to depart capacitance and smear are caused by different physical pro- from the intended optical path and enter another pixel,1 or cesses related to two different technology platforms (hybrid optical diffusion or other lateral redistribution of photons CMOS and CCD, respectively), they are jointly symbolized within the sensor epitaxial layer or bulk, causing the photon here by one coupling capacitance Cc. In literature, however, 2 5 to generate charge in the wrong pixel. Cc is usually related to interpixel capacitance only. Charge diffusion crosstalk has two causes: the linear dif- If the sensor is equipped with a color filter array (CFA) fusion of charge within the sensor from the pixel where the and/or a microlens array, those may be seen as an integral charge is generated to another pixel where the charge is part of the sensor, and their influence on IC are thus seen as sensor-caused effects. This includes reflection, refraction and optical scattering as well as effects from possible void space between layers on top of the sensor.10 IC according to the definition in this work characterizes only sensor-caused effects. Distortions before the sensor as caused by for example, the lens system and sensor housing veiling glare11 are therefore not considered as IC. The IC generally decays strongly with pixel distance: components of IC such as the brighter-fatter effect may drop at a r-2.5 power-law rate,12 r being the distance between two neighboring pixels. The consequence is therefore small for conventional imaging with moderate dynamic range (DR) and contrast. For calibrated and radiometrically accurate sensors, and for HDRI and potentially high contrast imaging, the lateral extent of the IC may however influence the entire sensor,13 which is of significant importance for the SNR and MTF.

1.2 | Implications of operating the sensor in saturation The decorrelation of IC is a challenge already in conven-

FIGURE 1 Illustration of intended and unintended collection of photons tional nonsaturated imaging, in part because several different and charges in a model sensor, not drawn to scale. The sensor pixel p0 (left) laws of physics contribute to the IC as described above. The is the photon entry position and the intended charge collection target, while IC generally increases with charge accumulation, with the p1 (right) is a nearby pixel. The small coupling capacitance Cc symbolizes possible exception of interpixel capacitance where a non- interpixel capacitance and smear. The avalanche diode, DB, symbolizes linear reduction of the coupling may occur with increased pixel blooming. The intermediate area i0 potentially contains circuitry for 5,14,15 example, anti-blooming. The vertical position of the photoelectric charge accumulation. An increase has also been 16 conversion is a function of a probabilistic process with a spectrally reported, which complicates the issue further, but the inter- dependent distribution pixel capacitance definitely shows a nonlinear dependence LINDSTRAND 3 on charge accumulation. Therefore, HDRI by bracketing, donor pixels to nearby recipient pixels are present, they will where the sensor is deliberately operated also in saturation, be reduced but not eliminated. A systematic overestimation makes decorrelation of IC more challenging. Because of error will remain in the output, because the proximity effect nonlinear effects, conventional linear deconvolution will fail is always considered positive: only the recipient pixels are to provide effective decorrelation of IC. considered, while the saturated pixels are excluded. The blending in the algorithm's case 2:3 and the estimation in 1.3 | Previous work case 3:3, hence also the blending in case 3:4 will all overesti- mate a true (unaffected) output value. It is established knowledge that there is an interpixel spatial A generalized mosaicing method23 fusing data from mul- dependence, and the effort to reduce it using conventional tiple views through an optical filter with spatially varying linear signal processing methods is far from new.17 The cited density also takes an ambitious approach in avoiding satura- work concerns astronomy imaging of poorly sampled point tion proximity effects. Such effects may result in abrupt sources using data matching by profile fitting of precharac- changes in the certainty function controlling the construction terized small variable size PSF pixel masks to the measured of a maximum likelihood (ML) pyramid, thereby causing response. This facilitates super-resolution in determining the artificial seams in the result. An introduced fuzzy classifica- position of the centroid and better accuracy for the bright- tion rule for “saturation-associated” pixels mitigates these ness. The idea of preanalyzed multiple PSF traces as a func- seams. “Low” and “high” saturation control and separate tion of sub-pixel position of a point source is similar to the isolated high radiant energy pixels having high certainty present work, since the detailed PSF analysis provides infor- from “saturation-associated” pixels with low certainty. This mation on how the neighborhood is affected by a local bright certainty measure defines the pixel weight controlling a spot. It is also suggested that the mask may be applied for 23 zeroing “the saturated pixels and their (likely) contaminated feathering effect that mitigates the seam. The ML pyramid neighbors”.17 It remains unclear how to estimate the radiant assumes pixels to be independent (note 13 in the reference), power of saturated pixels without exposure bracketing, but subsaturation crosstalk, which is present in all physical declipping18 or other techniques. This is important, because sensors, makes that independence assumption invalid. This “ ” the extent of the contaminated neighborhood increases with means that the effect of saturation-associated pixels will the radiant energy impinging on the saturated pixel. Poorer not be fully eliminated. In addition, due to the introduced “ ” results were reported for cases having saturated pixels.17 control parameter low saturation , high contrast regions Another study19 takes a more explicit and straight- affected by saturation proximity effects causes pixel values “ ” forward approach in specifically limiting the effect of below low saturation to remain undetected, which will blooming. Around a saturated pixel, a circular neighborhood cause errors in the result. is excluded, having a radius of three pixels motivated by A generic image fusion approach, as described already in 24 empirics. The empirical knowledge of the three pixel dis- 1993 , may have a potential also for the decorrelation of tance is of limited general applicability, for example, if the IC. The algorithm uses a local contrast dependent choice of sensor equipment, the HDRI implementation or the dynamic either pixel selection or averaging. Minor generalizations of ranges do not closely resemble those in the study. A similar the neighborhood analysis and of the process steps, most study20 excludes the region affected by a saturated pixel by importantly a saturation rejection, would make it applicable a two-pixel erosion around any saturated pixel value and for IC and HDR reconstruction with improved fidelity. studying cross-histograms identifying the outlier nonlinear- A method to avoid vignetting caused by a change of 25 ities next to high-contrast regions. Although more general aperture may similarly be effective for the decorrelation of and based on the cross-histogram findings, the rationale for IC, with minor modifications. The method compares pixel the exactly two pixel wide erosion remains undescribed. radiant energies for a set of LDR images with varying expo- An existing method for HDRI lens flare removal21 does sures irrespective of position, whereby outliers are identified not in a strict sense meet the definitions to be included in this and excluded. With only small changes in the algorithm, it study of IC effects, as lens flare is a pre-sensor defect. How- may instead detect IC. The challenge would be to define ever, the mitigation method is relevant also for the decorrela- effective outlier criteria. tion of IC and will be discussed further in section 5.3. Early attempts to model and reduce color crosstalk sim- HDR reconstruction has been described22 based on a plified the problem by handling for example, only one color 5 × 5 pixel neighborhood adaptive filtering of LDR sam- channel of the tri-chromatic CFA.26 However, the continu- ples, taking into account if the centroid is saturated and let- ing trend of reduced pixel pitch requires more effective and ting the number of saturated pixels in the neighborhood robust methods for color crosstalk reduction. A generaliza- influence the blending in order to “limit the proximity effects tion of a shading correction algorithm achieves a generalized (e.g., leakage or pixel cross-talk on the sensor)”. The algo- calibration of not only color crosstalk, but also minor mis- rithm is commendably ambitious in its analysis and decision matches due to micro-lens and imaging lens defects, reduc- rules. However, where proximity effects from saturated ing color noise.27 It is possible to separate a quantum 4 LINDSTRAND efficiency (QE) characterization of a CMOS sensor with a made. Presented is an image comparison, before and after IC CFA into effects from the color filters and the bare sensor reduction, only. However, the visual comparison leaves no QE.28 This includes color crosstalk due to pixel neighbors of doubt concerning the notable and substantial improvement. a different color. Subsequent studies may then build upon the results to further The characterization principle is developed further by improve IC decorrelation, advisably including also more rigor- configuring tailor-made CFA having spatially repeating ous SNR and MTF improvement analyses. 5 × 5 pixel evaluation clusters, with each cluster center pixel in one color and the neighborhood pixels in a single different 1.5 | Contribution color.29 By using different two-color configurations, the more varied color configurations of a traditional CFA were This work presents and implements a calibration method that characterized. A monochromatic illuminator was used to facilitates the detection, analysis, and reduction of IC in the facilitate more detailed study of QE and the color crosstalk affected neighborhood of saturated pixels in the LDR CFA sen- dependence on the depth position within the sensor.29 sor images, thus improving the LDR SNR and, consequently, Color crosstalk models and simulations are generally also the HDR reconstruction. The calibration method is novel considered in the design, construction and verification of as no special setup is needed, facilitated utilizing the sensor new sensors.10 Modified CFA configurations have been native CFA as a matched filter for IC detection. presented, which reduce the color error and SNR deteriora- Reflecting on the results, concrete suggestions for gen- tion due to color crosstalk relative to a conventional Bayer eralization are described to estimate and reduce both sub- configuration.30 This in-depth analysis also includes sensi- saturation and supersaturation IC, along with a final tivity studies for sources of error, as well as characteriza- generalization to an irradiance-exposure dependent (IED) tions of metamerism. PSF, extending to supersaturation levels. Reducing the effects of interpixel capacitance by deconvolution is suggested in the literature,31 but even fairly low levels of noise in the input make the implementation 2 | PROBLEM FORMULATION challenging,32 as the deconvolution is a de-facto high-pass Given a CFA HDR capture device imaging a static scene with filtering in the presence of noise.33 negligible level of motion blur for the set of LDR exposures The radiant energy dependence of the PSF due to the needed, we seek a method that specifically reduces the IC brighter-fatter effect, has been reduced, but not eliminated, caused by saturation in the LDR images. Based on a defined by simulating reverse charge shifts at the pixel level.12 IC error metric, the method is able to estimate the error and the affected environment using an appropriate test target. 1.4 | Remaining challenges The given HDR reconstruction is evaluated in terms of The majority of published methods for decorrelation of IC the visually apparent differences between using and not are limited to non-saturation operation, while the remaining using IC decorrelation. To show the influence of the LDR ones ignore the non-saturation operation. An aggregation of improvements clearly, a deliberately simplistic HDR recon- methods to cover the full envelope of operation, saturation struction algorithm will be used, inspired by an early HDRI as well as non-saturation, appears nontrivial. With few algorithm20 but without the ad hoc IC treatment. 12,22–24,32 exceptions, the works described generally lack The method has a potential for generalization to estimate detailed descriptions and motivations for important algo- and exclude subsaturated as well as supersaturated IC- rithm decisions. This is understandable, as IC has not been affected environments. their focus but rather is included as a secondary problem. The need for further research on the analysis and decorrelation of IC is apparent. 3 | METHOD It is desirable, but unquestionably challenging, to generalize the PSF to an irradiance-exposure dependent (IED) A calibration method is introduced to estimate the affected PSF able to characterize the described IC effects at an appro- region from saturated pixels for a color filter array (CFA) priate high fidelity. The subject of IC analysis and reduction sensor, using the native CFA as a matched filter. The related to conventional as well as HDR imaging is described method minimizes color crosstalk given a set of candidates more thoroughly by the author.34 for proximity regions. Because the described LDR problem of optimizing the The method is general and requires no special setup. How- SNR and MTF is inherently difficult, simplifications and ever, to test and verify the method for a challenging input, an approximations, including nonconventional approaches, are existing goniophotometric measurement system was utilized, motivated to at least improve SNR and MTF performance and consisting of an illumination system, a cylinder-shaped sample to gain knowledge. With the same motivation, in this explor- holder and a camera. The setup is configured such that part of atory first step, no rigorous SNR improvement analysis is the image area provides specular reflection, while other parts LINDSTRAND 5 provide bulk reflections. These two reflection modes, including Figure 3. Note, an achromatic original reproduced under an in-between transition region, represent a large difference in demanding conditions in terms of dynamic range or contrast reflectance levels and require HDRI. The illumination system is indeed a significant test of a color imaging system. Only a is a halogen white light and a telecentric lens (Sill optics high performance imaging system (hardware and software) TC55). The imaging system consists of a telecentric lens (Sill will reproduce the achromatic original without color tint. optics T30/0.375), a near-IR blocking filter and a 12-bit CCD, Indeed, analyzing the two plain white areas on the left non-cooled, anti-blooming Bayer CFA camera (Vosskühler and right hand side of the barcode in Figure 3, banding and CCD1300-QCB) delivering raw, non-demosaiced pixel color tints are apparent in what should ideally be smooth information. achromatic gradients across the curved samples. This is largely due to IC contamination, which will be further dis- 3.1 | Sensor calibration cussed in section 3.3. Comment: note the inclined banding in the upper left corner in Figure 3 which is a consequence The sensor values were subtracted by the per-pixel bias of an unintended and undesirable illumination inhomogene- frames for the exposure durations used, in this work 1, 2, ity. An ideal illumination without this inhomogeneity would 4 … 1024 ms, 11 durations in all. Imaging a constant flat have caused a horizontal banding, similar to the other band- field scene with varying exposure durations, the sensor ings in the figure. This undesirable imperfection of the illu- response is plotted in Figure 2. The response is very close to mination do however not influence the statistical analysis of linear. This is the only calibration required to execute the the color data or the implied other conclusions described in proposed IC decorrelation algorithm. the following, as the purpose of the motif is to yield a HDR In general, it is preferable to also perform absolute cali- input of varying achromatic reflectance, which remains true bration of HDR irradiance35 information. Doing so would be despite the imperfect illumination. fully compatible with the work presented, but was not per- The RGB values for all the pixels within these two white formed. Hence, the experimental measurements are referred areas are plotted in Figure 4. The image in Figure 3, was to as “intensities”, where the general reasoning instead refers white balanced for presentation, but Figure 4 shows the to the radiometric values. unscaled sensor values. Given the homogeneous test targets and the linear sensor, an image free from IC contamination — 3.2 | HDRI IC contaminated would have shown a range of achromatic intensities plus Using a deliberately simplistic but functional algorithm for noise, that is, data points clustered around a straight line in HDRI, selecting the highest per-pixel nonsaturated value the 3D color space. This is not the case in Figure 4: streaks, from the set of exposures and a subsequent bilinear demosai- gaps, and irregularities are clearly visible in the point cloud. cing, two different test targets were imaged: a high-gloss plastic label and a matt uncoated office paper. The images 3.3 | Sensor native CFA—an IC matched filter contain a wide dynamic range from high radiant power spec- IC causes undesired modifications of sensor pixel values. A ular reflections to low radiant power bulk reflections, see central finding in this work is that a CFA sensor may actually

104

103

102

101 sensor level [arb. scale] 100

10−1 100 101 102 103 sensor integration time [ms]

FIGURE 2 The sensor output versus exposure (integration time). Solid FIGURE 3 Image of the white areas of the test targets, mounted on a red—red channel; dashed green—green channel; dotted blue—blue channel. cylinder which results in a specular highlight requiring HDRI. The Dashed-dotted black—an ideal linear response (with an arbitrary offset) for measurement is performed with no IC reduction. Left: Glossy plastic label. reference Right: Matte office paper. Image width 21.8 mm, pixel pitch 17 μm 6 LINDSTRAND facilitate the detection and characterization of IC. Assuming a transition, that is, the edge. However, the effect occurs for any point source input, the source (donor) pixel will decline in value non-flat field, that is, any image with contrast. The experi- due to IC, while the neighboring (receiver) pixels are increasing ments presented below use the gradients of the white regions in value. The CFA causes the modification to potentially operate in Figure 3 as input, which give satisfactory results. across different color channels, depending on the CFA configu- The IC may extend beyond the closest four-connectivity ration. For a Bayer sensor, we note that the four-connectivity neighbors, somewhat reducing the selectivity of the matched neighbor receiver pixels are all of different colors than the donor filter, but assuming the IC is monotonically declining with pixel. Thus, the Bayer CFA provides something akin to a tem- distance in any given direction, the described matched filter plate matched filter for detecting and characterization of the characteristic will essentially remain valid. IC. The declining value and the set of increasing values all con- The combined CFA and spectral sensitivity differs between tribute to the deviation in pixel color vectors. IC can thus be color channels, causing the described matched filter characteris- detected as the donor pixel decline and the receiver pixels' value tic to be further improved. For an achromatic input, the most increase, both separately and as a combined effect, which makes sensitive channel will donate more than the other channels. If a for a sensitive filter. donor pixel is of the most sensitive channel, the 4-connectivity Generally, the color deviation will be at its strongest for a neighbors have lower sensitivities, and the receiving values are sub-pixel source (approximating a point source). For a bright amplified in post-processing. This causes IC caused color devi- area of larger spatial extent against a darker background, the ations to be amplified, which further facilitates the IC detection strongest color deviation occurs at the bright-to-dark and characterization.

FIGURE 4 Plot of the reconstructed HDR RGB intensities of the white sample areas of Figure 3, without IC reduction. The dashed–dotted line is the principal component (highest variance) of a principal component analysis derived by singular value decomposition, (A) 3D plot of RGB intensities. (B) Projection to R,G only (omitting the blue channel) (C) Projection to R,B LINDSTRAND 7

In Figure 4, the color data deviates substantially from the More formally: let A 2 Rm × n, m rows of test samples ideal achromatic straight line. Distinct cluster branches along (corresponding the pixels defining the previously described two the principal coordinate directions occur at a spacing of plain white areas on the left and right hand side of the barcode powers of two, corresponding to saturation levels of the inFigure3)ofan dimensional color space (RGB yield n =3). LDR images. Note, however, that all saturated LDR pixels Let B = round(A + E), B 2 Zm × n, E 2 Rm × n, be the infor- are already excluded, and the defects are due to secondary mation detected by the sensor, where the additive noise IC influence from saturated donor pixels to neighbors. E includes interpixel correlation (IC) and other sensor noise. Using SVD to decompose A A ¼ UΣVT ð Þ 3.4 | Defining and evaluating candidate regions for IC 1 influence where U and V are orthogonal matrices of sizes m × m and n × Σ α α … α To characterize the region of IC influence, we define candi- n respectively, and =diag( 1, 2, , n)isadiagonal matrix of descending order singular values of A such that α ≥ date regions of various shapes and extensions and evaluate 1 α ≥ … ≥ α . The number of nonzero singular values defines them in an optimization process. Choosing candidates 2 n the rank, r,ofA,wherer ≤ min (m, n). Ideally our monochro- deserves some consideration. The IC is a consequence of at matic test areas A have a first singular vector (1, 1, 1)×3-1/2 least six distortion components and different laws of physics, only, corresponding to achromatic and neutral gray (the factor as described in section 1.1. Without specific prior knowl- -1/2 3 for normalization). That is, α2 = α3 = 0, and rank(A)=1. edge it is difficult to foresee one particular shape and extent However, we detect B including a nonzero E, causing of the region being more likely than another. Instead, we rank(B) = 3 reflecting the detection is not monochromatic, choose candidates differing significantly in shape and size, but includes undesired color tints. By instead using SVD on to make a broader search. The goal is to have enough candi- T B, B ¼ UBΣBV , the singular value β1 and corresponding dates to cover a wide range of shapes with reasonable den- B (ie, first) singular vector of V defines the orientation of sity, such that the detected best candidate is close to the B maximum variance of B. By assuming that the orientation of global optimum. the maximum variance of B corresponds to the orientation of The optimization criterion is the minimization of the maximum variance of ideally our monochromatic test areas mean square color distortion in RGB space after HDR recon- A, we define an estimate of E: struction, as evaluated for each of the candidate regions of hi Xn 1=2 Figure 5. The distortion is measured as the orthogonal dis- E^ ¼ U Σ VT ¼ β ð Þ B Br B i 2 tance from the achromatic line, see Figure 4. The achromatic F i¼2 k•k Σ ðÞβ … β line, in turn, was characterized by linear regression analysis, where F is the Frobenious norm, Br is diag 0, 2, , n , using singular value decomposition (SVD) to determine the that is, singular value β1 set to zero and thereby keeping only principal component (highest variance). the residue of B, after elimination of the first singular vector

FIGURE 5 The evaluated candidate regions. The integer values below and to the left of each sub-figure denote the vertical and horizontal extent of the region. The value above each sub-figure is the estimated relative color distortion, compared to the null neighborhood at the top left, Ê/Ê0 where Ê0 is the estimate color distortion of the null neighborhood at the top left. The regions in gray rectangles are referred to in the text 8 LINDSTRAND energy36 (c.f. a residue of a principal component analysis of The IC-reduced HDR reconstruction is plotted in B). E and hence Ê are dependent on the candidate regions of Figure 6 (c.f. Figure 4) and imaged in Figure 7 Figure 5. Aiming for a small Ê, an unfavorably large elimi- (c.f. Figure 3). The relative sensitivity of the color channels nation region will cause the HDRI reconstruction to be inef- are compensated for in Figure 7 (as was done for Figure 3). fective, due to unnecessary pixel eliminations, hence Note that Figure 6 shows essentially a range of achromatic unnecessary high quantization and other sensor noise. A too intensities plus noise, that is, data points clustered around the small elimination region will cause an unnecessary high IC, achromatic straight line in the 3D color space. This is in con- and the IC will dominate the total error Ê. This optimization trast to Figure 4: where streaks, gaps and irregularities are is further explained in the next section 3.5. By evaluating Ê clearly visible in the point cloud. The two plain white areas on for each candidate region of Figure 5 we identify the region the left and right hand side of the barcode in Figure 7, show corresponding to the minimal Ê. smooth achromatic gradients across the curved samples. This is The candidates regions which were evaluated are shown in contrast to the banding and color tints apparent in Figure 3. in Figure 5. The patterns of white pixels define the neighbor- In Figure 8 the exposure duration for each position is hood to exclude around a saturated center pixel. The ordering, illustrated using the same motif as in Figures 3 and 7. The left-to-right, top-to-bottom, is in order from worst to best. The highest radiant power specular reflections are imaged with null 1 × 1 neighborhood, where only the saturated pixel itself the shortest exposures. is excluded, is at the top left. The error measures in the figure In Figure 9, the IC reduced HDR reconstruction (Figure 7) are the mean square deviations from the achromatic line, in is shown again, using different intensity scale factors to illus- relation to the same error metric for the null neighborhood. trate in print the range of intensities captured better. Among the regions tested, the most effective to reduce Notice that, the bluish tint in the lower part of Figure 9E, the color distortion was the 5 × 5 square, in the lower right and to some degree also in Figure 9F, corresponds to signifi- corner in Figure 5, circumscribed by a gray rectangle. This cant amplifications. This indication of further potential region was used in the following calculations. This is not improvement is the topic of next section and do not detract necessarily the optimum. A more thorough search might find from the verified main novel work of this study. a better candidate. However, the error is roughly the same for similar regions, and the 5 × 5 square is sufficient for the 5 | GENERALIZATION OF IC REDUCTION algorithm to improve the HDR result significantly. The estimated IC neighborhood is an average, based on all 3.5 | IC reduction saturated pixels under the illumination and exposure condi- The IC related errors are reduced by first excluding pixels tions represented in the set of LDR images. Hence the analy- around saturated pixels within the best candidate region sis did not address irradiance-exposure dependent (IED) according to section 3.4, and then performing the simplistic IC. An IED IC neighborhood determined by a more rigorous analysis and estimation would reasonably result in a smaller HDR reconstruction as before. Where a pixel in one LDR IC neighborhood for lower radiant energies and a larger image is discarded, its irradiance will instead be estimated neighborhood for higher radiant energies, because most from a shorter exposure, replacing a sample with IC contami- underlying causes of IC are exacerbated with increasing nation by a sample with higher quantization and read noise. radiant energy. With such refined characterization, it would The objective is to find the best middle ground: a region small be possible to achieve better IC reduction while discarding enough not to discard useful values, but large enough not to fewer pixels. The generalizations are suggestions for future keep values strongly affected by IC. Optimizing on global work to perform such refined characterization, using experi- SNR is a reasonable criterion to find the best middle ground. ence gained in the course of the current work. First, we describe how to characterize IED IC neighborhoods, initially for subsaturation, then for supersaturation. 4 | RESULTS Then, a more advanced generalization is suggested to replace “ ” The results are twofold. First, we estimated a neighborhood the crude, qualitative keep or discard strategy for LDR pixels with an IED PSF, and perform a potentially more efficient dec- of IC influence, determining where a saturated pixel reduces orrelation of the IC by a quantitativechargerepatriation the SNR in the HDR reconstruction and should be excluded. strategy. Pixels outside that neighborhood increase the SNR and should be included. The error measure was the color distortion in achromatic (white) image areas. Second, the esti- 5.1 | Generalization to multilevel subsaturation IC mated IC neighborhood and pixel exclusion were applied in reduction an HDRI reconstruction to illustrate the visual impact of the In order to generalize the findings to multilevel subsaturation pixel exclusion on the result HDR image. IC neighborhoods, the candidate regions would need to be LINDSTRAND 9

FIGURE 6 Plot of the RGB intensities for HDR reconstruction with IC reduction, otherwise identical to Figure 4 evaluated for a set of subsaturated irradiant exposures or nonlinear IC effects may require analysis at many different sensor output digital numbers (DN), coarsely or finely DN levels, but the characterization may be fully auto- spaced depending on ambition, using interpolation to mated. Of course, such more detailed characterizations cover intermediate levels. Properly characterizing the would require more and better tailored calibration data, but still no special setup.

1024

512

256

128

FIGURE 7 HDR image reconstructed with IC reduction, otherwise FIGURE 8 Illustration of the exposure durations used: 1, 2, 4 … 1024 ms. identical to Figure 3 (11 different exposure durations) 10 LINDSTRAND

5.3 | Generalization to an irradiance-exposure dependent PSF charge repatriation strategy The binary decision to either keep a LDR pixel value for further processing or to discard it for being too contaminated by distortion and noise, may be refined further by addressing two residual sources of error. First, signal will be lost in the elimination of the IC affected neighborhood. This signal may be sal- vaged if the neighborhood is instead characterized and processed. Second, pixels outside the neighborhood are processed as if free from IC, which is not correct as IC also influences more distant pixels. This influence may be reduced if the IC effects for more distant pixels are characterized and used to process data differently. Generalizing the multilevel IC neighborhood into an Irradiance-Exposure Dependent (IED) PSF hence has a potential of gaining SNR within the IC neighborhood as well as outside this neighborhood. This quantitative characterization requires a special test motif and a more controlled collection of larger amounts of calibration data. Measuring a radiant point source, it is possible to characterize the detailed shape of the IC influence, similarly to a conventional PSF characterization. For a CFA sensor, the influence of an imperfect point source may be FIGURE 9 HDR image reconstructed with IC reduction, rendered with six reduced by spectral matching of the source to the CFA chan- different intensity scale factors: (A) 0.72, (B) 5; (C) 50, (D) 100; (E) 2000, nels to evoke a response predominantly in a single color (F) 20 000 channel. This is another example of the CFA being used to facilitate the IC analysis. Pixel-individual variations may motivate pixel-individual A multilevel subsaturation IC characterization can be characterization or, less ambitiously, at least characterizing described as a 3D aggregate of individual 2D subsaturation different sets of pixels. Multiple characterizations may be IC neighborhoods. performed in parallel, provided that the IC influences from different point sources do not interfere. One possible test target would be a light blocking foil with an array of pinholes. 5.2 | Generalization to multilevel supersaturation IC Using this set of point sources and making sure they hit reduction pixels of all color channels with a dense enough distribution To generalize the method to multilevel supersaturation IC on the sensor, the multiple different subsaturation excitations neighborhoods, the candidate regions would need to be eval- described in section 5.1 and the multiple different supersatu- uated for a set of supersaturated levels with the same clipped ration excitations described in section 5.2 together facilitate DN. The level of irradiance exposure of a saturated pixel an IED PSF characterization. may be estimated from a shorter LDR exposure of the pixel The IED PSF opens up for what may be named a with nonsaturated values. If saturated values appear in the “charge repatriation strategy”, with the objective of decorre- shortest exposure, declipping18 may be applied. The reported lation of multilevel subsaturation and supersaturation range extension potential for declipping is about 5σ for the IC. The purpose of the repatriation is similar to a linear PSF noise in the signal. Past this level, clipping is beyond salva- deconvolution, that is, to undo the impact of the PSF, but a tion using the described approach. repatriation strategy for decorrelation of IC would also con- The resulting multilevel supersaturation IC neighbor- sider the nonlinearity and the supersaturation aspects of IC. hood may be represented as a 3D structure and aggregated A published method for HDRI lens flare method21 may with the 3D multilevel subsaturation IC neighborhood, inspire a design of a repatriation strategy. The method is described in section 5.1, to a general multilevel IC neighbor- restricted to subsaturation correlations, performing a PSF analy- hood. This multilevel IC neighborhood defines, for the char- sis for one exposure and assuming the effect is linear, which is acterized range of pixel irradiance exposure below and reasonable in the given context. In addition, the PSF is assumed above saturation, the spatial extent of pixels to exclude to radially symmetric,21 whereas the IED PSF in the present work reduce the joint negative effect of IC distortion, quantization can attain arbitrary shape. Nevertheless, an arbitrary shaped noise and read noise. IED PSF should not add further challenges compared to the LINDSTRAND 11 algorithm referred to,21 besides the main problem of incorporat- not immediately obvious. Janesick comments on page 144 in ing the irradiance exposure dependence. his reference book38 that pixel crosstalk at saturation acts as an The IED PSF characterization may be facilitated by HDRI averaging filter, causing an apparent reduction of noise. This is exposure bracketing. As each position is sampled at different important not only when estimating the noise level, but also exposures, this may be described as an overdetermined equa- when characterizing sensor gain by the commonly used photon tion system, albeit with noise and IC. The relative influence transfer curve method, or performing any other noise- of read noise, quantization noise and IC distortion will vary dependent characterizations. Although more pronounced at sat- with irradiant exposure, such that the noise will dominate the uration, the effect is present also at subsaturation. low irradiant exposures and the IC distortion will dominate Cross-histogram (joint histogram, 2D-histogram) analy- the high irradiant exposures. This will cause sensor IC to sis may be useful to gain a deeper understanding of IC. The increase with exposure, resulting in a characteristic and pre- discrepancy between the sensor output relations for different dictable IC related color tint that facilitates the IED PSF char- exposures,20,39–41 referred to as “widening ridge” in cross- acterization and optimizations of repatriation, and hence a histogram terms, illustrates one aspect of IC, albeit not potential SNR improvement in the HDR reconstruction. explicitly identified as a correlation effect in the references. Interpixel charge transfer effects have been introduced to 6 6 | DISCUSSION explain image deterioration in x-ray astronomy. Although lacking references by others, the effect may be of relevance The most effective region to reduce the color distortion was to refine the characterization of the IC into its sensor-caused the 5 × 5 square, among the candidate regions tested. This subcomponents, such as for example, charge transfer. may indicate that the IC to a considerable degree stem from Although the interpixel capacitance and the PSF widen- crosstalk (charge diffusion or light scattering), prone to be ing as described in section 1.3 may be mitigated by other more circular-symmetric in distribution, than from bloom- methods,12,31,32 our presented method adds the ability to ing, prone to be more column oriented. If true, it would indi- reduce the IC effect also of saturated pixels in a controlled cate the anti-blooming functionality of the CCD sensor manner, hence potentially widening the useful range of irra- being effective up to the radiant exposures evaluated, as diance exposures. blooming is often a challenge for this sensor technology if Established HDRI algorithms implicitly reduce aspects of not equipped with anti-blooming functionality. IC, but run the risk of losing or distorting information as a nega- The approximately circular footprint is also in agreement tive side effect. The complexity of IC at subsaturation as well as with previous results for one aspect of IC, the brighter-fatter supersaturation, having different physical causes and exhibiting 12 effect, where the closest neighbors show an anisotropic nonlinearities, makes it particularly problematic to adapt the correlation but the further reaches of the effect are isotropic. workarounds of established HDRI work to mitigate an identified The minor deviation from an isotropic shape in the current distortion at its source. A tailor-made IC reduction method that work may be a consequence of the configuration of the follows the best practice in signal processing and operates Bayer mosaic. Omitting the corners of the 5 × 5 pixel directly on the systematic errors in the LDR data set will have a square, the lower right corner in Figure 5, circumscribed by higher potential of effectiveness in terms of reducing a gray rectangle, the optimal IC neighborhood in this study, correlation-related errors. This in comparison with a less specific corresponds to a more isotropic shape, the lower left corner signal processing methods of established HDR reconstruction in Figure 5, circumscribed by a gray rectangle. However, algorithms. This, irrespective of whether the less specific algo- this would also keep pixels of the same color channel as the rithms are based on design and optimization of weight func- sensor position causing the saturation, which may be the rea- tions, operating jointly with the HDR reconstruction, or applied son for the inferior performance compared to the 5 × 5 square. Also, the data set used in this study was fairly small, as post-processing. Consequently, the present work shows a and the optimization was performed on all saturated pixels substantial decorrelation of the IC even with fairly simple irrespective of irradiant exposure. More experiments would means, by attacking the specific problem at the source in a man- be needed to determine the definite optimum. ner clearly motivated by analysis. In addition to the three generalizations presented, section 5, Inspiration for methods to reduce IC may be found in a detailed characterization of IC may also be resolved in terms other fields than HDR imaging, as related further to by the 34 of position in the sensor focal plane, similarly to crosstalk37 author. and PSF17 characterizations. The optimization may further be The presented work may be applicable to a wider field of performed for each color channel individually, as it is reason- HDRI, including both exposure fusion and radiant power able to assume the IC being spectrally dependent due to contri- calibrated applications, and also to non-HDRI applications, butions from spectrally dependent optical crosstalk.4 for example, Wide Dynamic Range sensors and conven- There are other findings worth mentioning which are rele- tional imaging. In sensor design, IC resilience is always an vant to the understanding of IC, although their applicability is optimization with competing characteristics. 12 LINDSTRAND

The algorithm introduced in the current work is a prepro- ORCID cessing strategy to mitigate IC in the LDR images. Established Mikael Lindstrand https://orcid.org/0000-0002-2537-2703 CFA HDR reconstruction algorithms that either do not consider IC, or perform the IC reduction less effectively, may adopt the proposed method as a preprocessor without a need to otherwise REFERENCES modify the HDR algorithm, yielding a higher SNR irrespective [1] Agranov G, Berezin V, Tsai RH. Crosstalk and microlens study in a color of the HDR reconstruction algorithm used. CMOS image sensor. IEEE Trans Electron Devices. 2003;50(1):4-11. The algorithm presented has been implemented in an [2] Kavaldjiev D, Ninkov Z. Subpixel sensitivity map for a charge-coupled industrial security document imaging application42 with device sensor. Opt Eng. 1998;37(3):948-954. [3] Antilogus P, Astier P, Doherty P, Guyonnet A, Regnault N. The brighter- HDR requirements. fatter effect and pixel correlations in CCD sensors. J Instrum. 2014;9: C03048. https://doi.org/10.1088/1748-0221/9/03/c03048. [4] Janesick JR. Scientific Charge-Coupled Devices. Vol PM83. Bellingham, 6.1 | Limitations of the presented work WA, USA: SPIE Press; 2001. DOI: https://doi.org/10.1117/3.374903. The rescaling of the sensitivity of the color channels is per- [5] Moore AC, Ninkov Z, Forrest WJ. Quantum efficiency overestimation and deterministic cross talk resulting from interpixel capacitance. Opt Eng. formed for the illustration purpose of Figure 3 and Figure 7 2006;45(7):076402. https://doi.org/10.1117/1.2219103. only, but not for the singular value decomposition with [6] Simms LM. Hybrid CMOS SiPIN detectors as astronomical imagers [PhD respect to the achromatic line. Depending on the application, Dissertation]. Stanford, USA: Department of Applied Physics, Stanford University; 2010. http://pqdtopen.proquest.com/pubnum/3395872.html? it may be desirable instead to perform a rescaling first, for FMT=AI, Accessed May 7, 2018. example, a white balancing based on visual importance of [7] Stevens EG, Lee YR, Burkey BC. The effects of smear on Antiblooming the color channels. It may have been more pedagogical to protection and dynamic range of interline CCD image sensors. IEEE Trans perform such white balancing prior to the optimization, Electron Devices. 1992;39:2508-2514. [8] Teranishi N, Ishihara Y. Smear reduction in the interline CCD image sen- because in this manuscript the application is for visualiza- sor. IEEE Trans Electron Devices. 1987;34:1052-1056. tion, demonstrating the algorithm in images, which moti- [9] Tanabe A, Kudoh Y, Kawakami Y, et al. Dynamic range improvement by vates a white balance calibrated optimization. However and Narrow-Channel effect suppression and smear reduction Technologies in Small Pixel IT-CCD image sensors. IEEE Trans Electron Devices. 2000; most importantly: whichever calibration target that best suit 47:1700-1706. the application at hand, the principal benefit and the algo- [10] Agranov G, Mauritzson R, Barna S, et al. Super Small, Sub 2μm Pixels For rithm presented remains otherwise unchained. Novel CMOS Image Sensors. In: International Image Sensor Workshop; – 1 June 7 10, 2007; Ogunquit Maine, USA. http://www.imagesensors.org/ Spectral crosstalk, caused by chromatic imperfections in Past%20Workshops/2007%20Workshop/2007%20Papers/079%20Agranov the CFA resulting also non-intended colors to pass, is neither %20et%20al.pdf, Accessed May 7, 2018. detected as IC nor reduced by the present algorithm. Like- [11] ISO. Optics and Optical Instruments - Veiling Glare of Image Forming Sys- wise, irradiance spectral variations within the spectral pass- tems - Definitions and Methods of Measurement. Vol 9358. Geneva, Swit- zerland: ISO; 1994. bands of each of the color channel of the color filter array, [12] Gruen D, Bernstein GM, Jarvis M, et al. Characterization and correction of will influence the diffusion crosstalk, hence IC, but will not charge-induced pixel shifts in DECam. J Instrum. 2015;10(5):C05032. be detected using the suggested method, including the gener- https://doi.org/10.1088/1748-0221/10/05/C05032. [13] Du H, Voss KJ. Effects of point-spread function on calibration and radio- alizations. However, this influence is (a) limited to one (opti- metric accuracy of CCD camera. Appl Opt. January 20, 2004;43(3): cal crosstalk) of the six described components of IC only, 665-670. and (b) generally very limited on the overall optical crosstalk [14] Cheng L. Interpixel capacitive coupling [M. Sc. Thesis]. Rochester, United States, Rochester Institute of Technology; 2009. http://scholarworks.rit.edu/ component, given the context described. Therefore, this limi- theses/2871, Accessed May 7, 2018. tation is likely of practical importance to only a most [15] Donlon K, Ninkov Z, Baum S, Cheng L. Modeling of hybridized infrared extreme and niche application. arrays for characterization of interpixel capacitive coupling. Opt Eng. 2017; 56(2):024103. https://doi.org/10.1117/1.oe.56.2.024103. [16] Giardino G, Sirianni M, Birkmann SM, et al. Noise properties and signal- dependent interpixel crosstalk of the detectors of the near-infrared spectro- ACKNOWLEDGMENT graph of the James Webb space telescope. Opt Eng. 2013;52(3):034001. https://doi.org/10.1117/1.oe.52.3.034001. The author would like to thank Stefan Gustavson, Sasan [17] Howell SB, Koehn B, Bowell E, Hoffman M. Detection and measurement Gooran, Jonas Unger, and Reiner Lenz, all at ITN, Linköp- of poorly sampled point sources imaged with 2-D arrays. Astron J (NY). ing University, for insightful comments and valuable feed- 1996;112:1302-1311. back improving the earlier versions of this article, and Stefan [18] Foi A. Clipped noisy images: heteroskedastic modeling and practical denoising. Signal Process. 2009;89:2609-2629. Gustavson for assistance in preparing the final text. [19] Tsin Y, Ramesh V, Kanade T. Statistical Calibration of CCD Imaging Pro- cess. In: International Conference on Computer Vision; 7–14 July, 2001; Vancouver, BC, Canada. DOI: 10.1109/iccv.2001.937555. [20] Madden BC. Extended Intensity Range Imaging: University of Pennsyl- CONFLICT OF INTEREST vania, Department of Computer & Information Science;1993. Mikael Lindstrand is the sole owner of gonioLabs AB and Paper 248. http://repository.upenn.edu/cis_reports/248, Accessed May gonioLabs sole proprietorship, and has related patents US 7, 2018. [21] Reinhard E, Ward G, Pattanaik SN, Debevec P. High Dynamic Range 6147750, and SE 532553 (family; also AU, CA, CN, HK, Imaging; Acquisition, Display, and Image-Based Lighting. San Francisco, US, and IN) issued to gonioLabs sole proprietorship. CA, USA: Morgan Kaufmann Publisher; 2006. LINDSTRAND 13

[22] Tocci MD, Kiser C, Tocci N, Sen P. A Versatile HDR video production [39] Mann S, Picard RW. On being ‘undigital’ with digital cameras: extending system. In: ACM SIGGRAPH; 7–11 August, 2011; Vancouver, British dynamic range by combining differently exposed pictures. In: Imaging Sci- Columbia, Canada. DOI: https://doi.org/10.1145/1964921.1964936. ence & Technology Conference; 7–11 May, 1995; Cambridge, MA, USA. [23] Schechner YY, Generalized Mosaicing: High dynamic range in a wide field http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.21.3141, of view. Int J Comput Vis. 2003;53:245-267. Accessed May 7, 2018. [24] Burt PJ, Kolczynski RJ. Enhanced image capture through fusion. In: Inter- [40] Robertson MA, Borman S, Stevenson RL. Estimation-theoretic approach to national Conference on Computer Vision; 11–14 May, 1993; Berlin, Ger- dynamic range enhancement using multiple exposures. J Electron Imaging. many DOI: https://doi.org/10.1109/ICCV.1993.378222. 2003;12:219-228. [25] Mitsunaga T, Nayar SK. Radiometric self calibration. In: Computer Society [41] Battiato S, Castorina A, Mancuso M. High dynamic range imaging for digi- Conference on Computer Vision and Pattern Recognition; June 23–25, tal still camera: an overview. J Electron Imaging. 2003;12:459-469. 1999; Fort Collins, CO, USA. DOI: https://doi.org/10.1109/CVPR.1999. [42] Lindstrand M. Spatially and angularly resolved high dynamic range reflec- 786966. tance measurements for forensic document inspection. Optical Document [26] Li W, Ogunbona P, Shi Y, Kharitonenko I. Modelling of color cross-talk in Security - the Conference on Optical Security and Counterfeit Deterrence; CMOS image sensors. In: IEEE International Conference on Acoustics, January 23–25. USA: San Francisco, CA; 2008. Speech, and Signal Processing (ICASSP); 13–17 May, 2002; Orlando, FL, USA. DOI: https://doi.org/10.1109/icassp.2002.5745428. AUTHOR BIOGRAPHY [27] Getman A, Uvarov T, Han YI, Kim B, Ahn JC, Lee YH. Crosstalk, color tint and shading correction for small pixel size image sensor. In: Interna- tional Image Sensor Workshop; 7–10 June, 2007; Ogunquit, ME, USA. MIKAEL LINDSTRAND is the founder and sole owner of http://www.imagesensors.org/Past%20Workshops/2007% gonioLabs AB. He received an M.S. degree in computer sci- 20Workshop/2007%20Papers/042%20Getman%20et%20al.pdf, Accessed ence and engineering, 1996, and a Ph.D. in computer science August 30, 2018. [28] Wu Y, Cizdziel PJ, Rhodes HE. Decoupling photo collection efficiency and from Linköping University, Sweden, 2018. For more than color crosstalk from the quantum efficiency spectrum for the CMOS image two decades he has developed and implemented image anal- sensor pixel development. In: Digital Photography V; January 18, 2009; ysis measurement and characterization tools in the context of San Jose, CA, USA. DOI: https://doi.org/10.1117/12.805743. [29] Vaillant J, Mornet C, Decroux T, Hérault D, Schanen I. Characterization of human visual perception-based evaluation. Since 2007, after pixel crosstalk and impact of Bayer patterning by quantum efficiency mea- founding gonioLabs, the work is focused on measurement surement. In: Digital Photography VII; 23 January, 2011; San Francisco methods and equipment for perceptually meaningful charac- Airport, California, USA. DOI: https://doi.org/10.1117/12.879603. [30] Anzagira L, Fossum ER. Color filter array patterns for small-pixel image terization of optically variable image devices as generally sensors with substantial cross talk. J Opt Soc Am A 2015;32:28-34. critically features of security documents such as travel docu- [31] Finger G, Dorn R, Meyer M, Mehrgan L, Moorwood AFM, Stegmeier J. ments (Passports) and bank notes. Such features, often based Interpixel capacitance in large format CMOS hybrid arrays. In: High on diffractive or interference optics, typically exhibit high Energy, Optical, and Infrared Detectors for Astronomy II; 24–31 May, 2006; Orlando, Florida , USA. DOI: https://doi.org/10.1117/12.671828. chromatic contrast and high dynamic range of reflectance. [32] McCullough P. Inter-pixel capacitance: prospects for deconvolution: Space Therefore these materials are especially demanding in terms Telescope Science Institute;2008. Instrument Science Report WFC3 of interpixel correlation, dynamic range capability and mod- 2008–26. http://www.stsci.edu/hst/wfc3/documents/ISRs/WFC3-2008-26. pdf, Accessed May 7, 2018. ulation transfer function of the image sensor. Currently he [33] Starck JL, Pantin E, Murtagh F. Deconvolution in astronomy: a review. doubles as a consultant data scientist in assignments “Artifi- Publ Astron Soc Pac. 2002;114(800):1051-1069. https://doi.org/10. cial intelligence ability development” and “Analyst for pre- 1086/342606. ” [34] Lindstrand M. Sensor Interpixel correlation analysis and reduction: a dictive analysis at two Swedish Government agencies. review. J Imag Sci Technol. 2019;63(1). https://doi.org/10.2352/J. ImagingSci.Technol.2019.63.1.010502. [35] Debevec PE, Malik J. Recovering high dynamic range radiance maps from photographs. In: SIGGRAPH; 3–8 August, 1997; Los Angeles, CA, USA. DOI: https://doi.org/10.1145/258734.258884. [36] Konstantinides K, Yao K. Statistical analysis of effective singular values in How to cite this article: Lindstrand M. Sensor inter- matrix rank determination. IEEE Trans Acoust Speech Signal Process. 1988;36(5):757-763. https://doi.org/10.1109/29.1585. pixel correlation analysis and reduction for color filter [37] Hirakawa K. Cross-talk explained. In: IEEE International Conference on array high dynamic range image reconstruction. Color Image Processing; 12–15 October, 2008; San Diego, CA, USA. DOI: Res Appl. 2019;1–13. https://doi.org/10.1002/col. https://doi.org/10.1109/icip.2008.4711845. [38] Janesick JR. Photon Transfer. Vol PM170. Bellingham, Washington, USA: 22343 SPIE Press; 2007. DOI: https://doi.org/10.1117/3.725073.