Fibonacci Exposure Bracketing for High Dynamic Range Imaging
Mohit Gupta Daisuke Iso Shree K. Nayar Columbia University Columbia University Columbia University New York, NY 10027 New York, NY 10027 New York, NY 10027 [email protected] [email protected] [email protected]
Abstract
Exposure bracketing for high dynamic range (HDR) imaging involves capturing several images of the scene at different exposures. If either the camera or the scene moves during capture, the captured images must be registered. Large exposure differences between bracketed images lead to inaccurate registration, resulting in artifacts such as ghosting (multiple copies of scene objects) and blur. We present two techniques, one for image capture (Fibonacci exposure bracketing) and one for image registration (gen- Exponential bracketing and Fibonacci bracketing and eralized registration), to prevent such motion-related arti- conventional registration generalized registration facts. Fibonacci bracketing involves capturing a sequence Figure 1. (Left) HDR image computed using conventional regis- of images such that each exposure time is the sum of the tration has blur and ghosting due to camera motion. Each of the previous N(N > 1) exposures. Generalized registration three candle flames (inset) has multiple copies. (Right) HDR im- involves estimating motion between sums of contiguous sets age computed using the proposed approach. Zoom in for details. of frames, instead of between individual frames. Together, 1 the two techniques ensure that motion is always estimated However, since the frames have different exposure times , between frames of the same total exposure time. This results they have differentamounts of motion blur and noise. These differences cannot be removed by normalizing the image in HDR images and videos which have both a large dynamic 2 range and minimal motion-related artifacts. We show, by re- intensities by their exposure times . Because of this, even sults for several real-world indoor and outdoor scenes, that after normalization, image features are not preserved across the proposed approach significantly outperforms several ex- frames, and motion information cannot be computed reli- isting bracketing schemes. ably. This results in artifacts, such as ghosting (multiple copies of scene objects), blur and distortions. An example is shown in Figure 1. Such artifacts often negate the quality 1. Introduction enhancementthat is broughtabout in the image by capturing a wide dynamic range. This presents a fundamental trade- High dynamic range (HDR) imaging is the process of off - while a large difference in image exposures is required capturing scenes with a larger intensity range than what to capture a wide intensity dynamic range, it also results in conventional sensors can capture. Because HDR images strong motion-related artifacts. faithfully capture details in both dark and bright parts of the In this paper, we present new exposure bracketing and scene, they are desirable in surveillance, astronomy, med- image registration techniques for handling scene and cam- ical imaging, and more recently, even consumer photogra- era motion while also capturing a wide dynamic range phy. Exposure bracketing [9, 2] is the most popular tech- (DR). The key idea is to compute motion information be- nique for HDR digital imaging. The basic idea is to capture tween sums of contiguous sets of frames. We call this gen- multiple images of the same scene with different exposures. eralized registration. This is different from conventional While each captured image has a low dynamicrange(LDR), registration, where motion is computed between individual a single HDR image is generated by merging the exposure- frames. We propose an exposure bracketing scheme called bracketed LDR frames. Because of its ease of implemen- Fibonacci bracketing where the exposure times follow the tation, bracketing for HDR is now available as a standard Fibonacci property, i.e., each exposure time is the sum of feature in most digital cameras, including cell-phones. the previous N(N > 1) exposure times. Together, Fi- bonacci bracketing and generalized registration ensure that Despite its simplicity, exposure bracketing is not used in several real-world scenarios because it is prone to errors 1In this paper, image exposure is changed only by varying the camera when there is scene or camera motion. In order to compen- shutter-time. Variations in camera aperture and gain have also been used sate for motion, the bracketed images are registered before for capturing HDR images [14, 5]. 2In all the simulations and experiments in the paper, wherever neces- merging into the HDR image. Registration is performed sary, the captured images were normalized by their exposures before mo- by computing motion information between adjacent frames. tion estimation in order to maintain brightness constancy between them.
1 motion is always estimated between frames of the same to- handle dynamic camera and scenes. tal exposure. The exposure times in a Fibonacci sequence Post-processing for Ghost Removal: In order to re- grow exponentially, thus capturing a large DR as well. Fig- move the ghosting artifacts in HDR images, several post- ure 1 (Right) shows an image computed using the proposed processing techniques have been proposed [7, 3]. These techniques. It captures the wide DR of the scene and has methods attenuate the contribution of pixels belonging to negligible motion-related artifacts. moving objects in the final HDR image. While these ap- Hardware Prototype: For generalized registration, a proaches reduce ghosting, the moving objects may not have sensor that allows exposure bracketing with a negligible HDR content if the inter-frame motion is large. Recently, inter-frame time-gap is required. Although most current Sen et al.[15] proposed creating the HDR image by enforc- cameras support exposure bracketing, there is often a large ing its consistency with the bracketed images in a patch- inter-frame gap (50 − 200ms). We implemented our tech- based optimization procedure. Our focus is different than niques on a machine vision camera which is triggered exter- the above post-processing techniques - it is on acquiring im- nally using a micro-controller based circuit. This hardware ages so that ghosting artifacts can be prevented. The above setup achievesa small inter-frame gap of 0.1ms while expo- techniques can be used in a complementary fashion to our suretimes are changedfrom oneframeto the next. We show approach in order to remove any residual artifacts. results for several real-world scenes, both indoors and out- Hardware Modifications: Several approaches have been doors, captured during different times of the day and hav- proposed to increase the DR by making hardware (opti- ing a wide range of motion characteristics. For the same cal and electronic) modifications to the camera. These in- time-budget, Fibonacci bracketing and generalized registra- clude using an array of neutral density filters [11] to spa- tion produce images of significantly higher quality as com- tially modulate light before reaching the sensor, splitting the pared to existing techniques. We also extend our techniques light inside the camera using beam-splitters [19], and plac- to capture HDR video at up to 15 fps while adapting the ing optical filters in front of the camera [13]. These systems bracketing sequence to scene brightness and motion. requiring hardware modifications are often expensive and Scope and Contributions: Our contributions are tech- inaccessible to consumers. For most consumer cameras, es- niques for image capture and registration that mitigate ar- pecially the point-and-shoot and cell-phone ones, exposure tifacts due to inter-frame motion and different amount of bracketing remains the cheapest and the most viable HDR motion blur between frames. These techniques do not re- imaging option. duce motion blur caused by intra-frame motion. We use existing deblurring methods to reduce blurring in the brack- 3. What are Good Exposure Bracketing eted frames. Our techniques are robust to non-linear cam- Schemes for HDR Imaging? era intensity response and small bit-depths, making them especially attractive for use in inexpensive cell-phone and Given a time-budget T for acquiring a single HDR im- point-and-shoot cameras. The proposed approach does not age, an exposure bracketing sequence is defined as a set of require any modifications to the optics. Because of its sim- frame exposures E = {e1,e2,...,eK } such that: plicity, our method is especially suited for implementation K on compact cell-phone cameras, for which, low-light and ei = T − (K − 1)δ , (1) low-dynamic-range are known problems. This makes our Xi=1 work particularly pertinent as cell-phone cameras are ex- where δ is the inter-frame time gap due to sensor read- pected to dominate consumer imaging in the next five years. out delay. The captured exposure-bracketed LDR frames {f1,f2,...,fK} constitute the exposure stack. The max- 2. Related Work imum number of frames K is constrained by the max- imum frame rate of the camera. For example, for an Exposure Bracketing: One of the most widely used F = 300 frames-per-second camera and a time budget of bracketing schemes for HDR imaging is the exponential T = 120ms, a maximum of K = F T = 36 LDR frames scheme [9, 2], where a sequence of images with exponen- 1000∗ can be captured for a single HDR image. tially increasing shutter times are used. Kang et al.[6] pro- Given a time-budget, there are infinite possible brack- posed a bracketing sequence of alternating short and long eting sequences. Which bracketing sequence achieves the exposures. In this approach, an HDR image is computed by highest quality HDR image? The dynamic range achieved registering three LDR images. Later, Zhang et al.[21] pro- by a bracketing scheme is given as [11]: posed capturing and registering a sequence of several very I e short and same exposure images. We propose using expo- DR = log max max , (2) sure times that have the Fibonacci property, i.e., each expo- Imin emin sure is the sum of previous N(N > 1) exposures. where emax and emin are the maximum and the minimum Recently, there has been a lot of work in devising scene- exposures in the bracketing scheme, respectively. Imax (de- adaptive exposure bracketing techniques [5]. These tech- termined by the sensor’s full well capacity) and Imin (de- niques attempt to maximize the signal-to-noise-ratio of the termined by the sensor read noise) are the maximum and final HDR image by adapting the bracketing sequence to the minimum signals detectable by the sensor. From Eq. 2, it is scene’s brightness distribution. All these techniques assume clear that in order to maximize the dynamic range, a brack- the scene and camera to be static. In contrast, our goal is to eting scheme should have a large range of exposures so that emax Registration Error (pixels) 4 5ms 10ms 15ms 20ms
the ratio is maximized. The LDR frames in the ex- emin posure stack captured using such bracketing sequences will 20 4 3 have large differences in exposures. 15 3 On the other hand, if there is camera or scene motion, 2 large differences in LDR frame exposures can lead to in- 10 2 1 Exposure x128 times correct registration. Why does this happen? This is be- 5 1
cause image registration techniques work best when both (pixels) Error Registration 0 5 10 15 20 0 5 10 15 20 25 the source and the targetimages have similar features. How- (ms) Image Target of Exposure Exposure of Source Image (ms) Exposure Time (ms) ever, because of different exposure times, images have dif- (a) (b) 400 ferent amount of motion blur and noise, and hence, image Figure 2. Illustration of the iso-exposure advantage. image patches were extracted from HDR images. Images were simulated features are not preserved. Although normalizing the im- assuming the scene to be a translating patch. For each patch, two ages with their exposure times maintains brightness con- images with different exposure times were generated, normalized, stancy between them, it does not remove the differences in and optical flow was computed between them. The difference in motion blur. This results in incorrect motion estimation. the computed flow and the ground truth flow gives the registration This is illustrated in Figure 2. Images are simulated as- error. (c) 2D plot of the mean errors. The exposures of the source suming the scene to be translating images patches of size and the target images are plotted on the X and Y axes, respectively. 64 × 64 pixels. For each patch, two images with different (d) Error plots for four different target exposures. Images with the exposure times are generated, using an affine image noise same exposures have the minimum registration errors. model. Image intensities are normalized by their exposures, Let the exposure bracketed frames be and then dense optical flow is computed between them. The {f ,...,f ,f ,...,f }, with exposure times difference in the estimated flow and the ground truth flow 1 i i+1 K {e ,...,e ,e ,...,e }. In order to compute flow gives the registration error. As shown, images with the same 1 i i+1 K oi+1 between f and f , we first make two adjacent, exposures have the minimum registration errors. i i i+1 contiguous sets of frames around fi: Qualitative comparison of existing bracketing schemes: s Si = {fi ns+1,fi ns+2,...,fi} , (3) The exponential scheme [2] achieves good DR as the ex- − − t posures grow exponentially. The alternating scheme [6] Si = {fi+1,fi+2,...,fi+nt } , (4) uses long and short exposures (ratio between the exposures where the superscripts s and t stand for source and target. s t is 16). This scheme achieves a moderate DR. Both expo- Number of frames in Si and Si is ns and nt, respectively. s t nential and alternating schemes are prone to registration The generalized frames Fi and Fi for both sets are defined errors due to large differences between consecutive expo- as the sum of the individual frames: sures. The burst-of-short-exposures scheme [21] results in s Fi = fi ns+1 + fi ns+2 + ... + fi , (5) the smallest increase in the DR as all the exposures are the − − t same. Since the burst scheme uses images of the same ex- Fi = fi+1 + fi+2 + ... + fi+nt . (6) posure, it is robust to registration errors. Suppose the inter-frame time gap δ between consecutive Thus, in the context of exposure bracketing for HDR frames is negligible and the camera has a linear intensity re- s imaging, there is a fundamental tradeoff between the dy- sponse. Then, the frame Fi is the same as the single frame namic range and registration accuracy. To capture a large s Fi that the camera would have captured had it exposed for dynamic range, it is important to use a large range of ex- 3 t the sum of exposure times ei ns+1+ei ns+2+...+ei . Fi posures. However, large differences in exposures of images f − − is related to F t in a similar manner. Let the flow between can result in strong registration artifacts. How can we cre- i F s and F t be o˜[i+1,i+nt] (generalized flow). The subscript ate a bracketing scheme that achieves high dynamic range i i f [i ns+1,i] − while minimizing the likelihood of registration artifacts? and the superscript denote the first and the last frames in the s t sets Si and Si , respectively. Assuming that the flow vectors 4. Generalized Registration for Exposure are linear within the duration of capture of the generalized s t i+1 4 Bracketed Image Sequences frames Fi and Fi , the flow oi is computed as : e + e Consider a sequence of exposure bracketed images that oi+1 = i i+1 o˜[i+1,i+nt] . (7) i i+nt [i ns+1,i] are to be registered using optical flow. Our key idea is that − ej instead of directly estimating the flow between individual j=i ns+1 −P frames, we estimate flow between sums of two contiguous [3] Two examples with ns = 2,nt = 1 o˜[1,2] and sets of frames. We call the flow between sums of frames as generalized flow and the process of estimating general- [3,4] ns = 1,nt = 2 o˜[5] are illustrated in Figure 3. The ized flow as generalized registration. Flow between indi- 3 s vidual frames is then computed by scaling the generalized The standard deviation of the effective read noise for i is √ns times F flow. This is illustrated in Figure 3. In the next section, that of fs; the photon noise is the same for s and fs. Fi Fi Fi we will show that with the correct choice of exposure se- 4In general, if a large number of frames are added to create the general- quence, generalized registration allows computing flow be- ized frames, the flows within a generalized frame may not be linear, espe- cially for very long exposures. As shown in the next section, our approach tween sums of frames having the same total exposure, while requires adding only a few frames (1 3). This allows approximating the also achieving a high dynamic range. flows by linear vectors for a wide range− of scenes and motions. Conventional Registration Generalized Registration
͵ ͵ ሾ͵ሿ ሾ͵ǡͶሿ ଶ ସ ሾଵǡଶሿ ሾହሿ
ଵ ଶ ଷ ଷ ସ ହ ݂ଵ ݂ ଶ ݂ଷ ݂ସ ݂ହ ݂ ݂ ݂ ݂ ݂ ݂ ͵ ͵ ሾ͵ሿ ሾ͵ǡͶሿ ଶ ସ ሾଵǡଶሿ ሾହሿ
frame exposure time frame exposure time ݁ଵ݁ଶ ݁ଷ ݁ସ ݁ହ ݁ଵ ݁ଶ ݁ଷ ݁ଷ ݁ସ ݁ହ Figure 3. Conventional versus generalized registration. fi are the bracketed LDR frames, ei are the exposure times and o are the flows between frames. (Left)݁ ൌIn conventional registration, flow is computed between individual݁ frames.ൌ The flows are then used to register all the frames to a reference frame. Differences in exposures of the frames result in registration artifacts. (Right) We propose generalized registration where flow is computed between summations of frames. By choosing the exposure times appropriately, generalized registration ensures that the flow is always computed between sums of frames with the same total exposure times. Flows between individual frames are computed by scaling the generalized flows o˜. individual flows are computed as o3 e2+e3 o[3] and The next higher order is (2, 1). Order (2, 1) iso-exposure 2 = e1+e2+e3 ˜[1,2] [3,4] property is achieved if every exposure e is equal to the o3 e3+e4 o . i 4 = e3+e4+e5 ˜[5] sum of two previous exposures, i.e., ei = ei 1 + ei 2. This is the property of the Fibonacci sequence of− numbers,− 5. Fibonacci Exposure Bracketing a series well studied in number theory [8]. The sequence In this section, we propose an exposure bracketing 1, 2, 3, 5, 8, 13,... is the canonical Fibonacci sequence. We scheme that exploits generalized registration to ensure that call the bracketing scheme with exposures forming a Fi- optical flow is always computed between frames of the bonacci sequence as Fibonacci exposure bracketing. same exposure time. To formalize this, we define the iso- What is the dynamic range achieved by Fibonacci exposure property for an exposure sequence: bracketing? The DR is determined by the ratio of consec- utive exposures [11]. The ratios of consecutive numbers in Definition 1 An exposure sequence {e ,e ,...,e } has 1+√5 1 2 K a Fibonacci sequence approaches (in the limit) φ = 2 , order (ns,nt) iso-exposure property if ∀i ∈ [2 ...K − 1], the golden ratio. This is a well-known result in number the- i i+nt ory [8]. Thus, Fibonacci sequences behave like exponential there exist n and n such that e e . s t j = j sequences with a growth factor G φ. While it may appear j=i ns+1 j=i+1 = − P P that the DR of Fibonacci bracketing (DRfib) is small due If an exposure sequence has order (ns,nt) iso-exposure to a relatively small growth factor, it turns out that DRfib s property, it is possible to make generalized frames Fi and is always within a small additive constant of the maximum t Fi (using Eqs. 5 and 6) for every pair of adjacent frames fi achievable dynamic range. s t and fi+1 so that Fi and Fi have the same total exposure. Lemma 1 For any given time-budget T , the dynamic range Since the flow between fi and fi+1 is computed by first achieved by Fibonacci bracketing DRfib is within 1.39 estimating the flow between F s and F t (Eq. 7), the iso- i i stops of the maximum achievable dynamic range DRmax. exposure property ensures that the flow is always estimated between frames of the same exposure. Proof 1 Consider an exposure sequence where the ratio of 1+√5 How should the parameters ns and nt be chosen? Both successive exposures is φ = 2 . This sequence follows ns and nt should be as small as possible so that (a) flow the Fibonacci property. Suppose the time-budget T is equal vectors within the generalized frames can be assumed to be to the sum of all exposures. Then, T is the sum of an expo- s t linear, and (b) the gaps created in Fi and Fi due to inter- nential series with emin as the first (minimum) exposure: frame gap δ are minimized. Moreover, ns − nt should be K K s φ − 1 φ minimized to ensure that the effective noise of both Fi and T = emin The maximum dynamic range Dmax for a time-budget T Conventional Registration 0.8 is achieved with two exposures, emin and T − emin: 1.5 0.6 Imax T − emin Imax T 1 DR = log (a)LDRImage (b)HDRImage(alternating (c)HDRImage(burst of short (d) HDR Image (Fibonacci (Bestexposure) exposures)[6] exposures)[21] bracketing+gen. registration) Figure 7. Comparison between Fibonacci and two existing bracketing schemes. (Top) Church on a cloudy day. (Bottom) Indoor birthday party. Both scenes have large dynamic range (105 − 106). (a) The best exposed LDR image has saturated regions (sky, candles). (b) The alternating (long and short exposure) scheme suffers from strong registration artifacts (church) and can not reconstruct mid-tones of the scenes faithfully (flowers, table). (c) Images captured using the burst (of short exposures) scheme have a low dynamic range, resulting in low quality in the dark regions. (d) HDR images created using Fibonacci bracketing and generalized registration. computed sufficiently fast on commodity hardware: the Flea3 camera for determining the bracketing sequence. Since image analysis and capture steps are parallelized and Gk = 1+ 1 − Iˆk 1 1 − Mˆ k 1 , (13) − − the total capture time for each bracketing sequence is about 60ms, our system captures HDR video at 15 fps. See the where Iˆk 1 is the median intensity of the previous frame − project web-page [1] for the videos and more results. and Mˆ k 1 is the mean motion between two previous − frames. Motion is computed by computing correlation be- 8. Discussion and Limitations tween 1-D projections of the two frames along rows and columns. Both Iˆk 1 and Mˆ k 1 are normalized to lie in the In order to be used widely in consumer cameras, HDR range [0, 1]. We used− a Miro− M310 camera (see Figure 5) techniques should handle motion. Ours is the first expo- for capturing exposure bracketed images. A Point-Grey sure bracketing scheme that is designed to deal with dy- Flea3 camera was used for ‘scene-metering’ - intensity and namic camera and objects. The proposed approach is robust motion information was computed on images captured by to non-linearities in the camera response functions and low Figure 8. Results of tribonacci bracketing and generalized reg- istration for the scenes shown in Figure 7. HDR Image (exponential) HDR Image (Fibonacci) Figure 10. Evaluating the effect of non-linear camera response. A non-linear response was emulated by applying a γ-curve (γ = 2.2) on the input LDR frames. Because of differences between ex- posures of consecutive frames, there are strong distortions for ex- ponential (bushes, walking path and people at the bottom) scheme. In contrast, the proposed approach is robust to non-linear response. [5] S. W. Hasinoff, F. Durand, and W. T. Freeman. Noise- optimal capture for high dynamic range photography. In IEEE CVPR, 2010. 1, 2 [6] S. B. Kang, M. Uyttendaele, S. Winder, and R. Szeliski. High dynamic range video. ACM Trans. Graph., 22(3), 2003. 2, 3, 6, 7 Fibonaccibracketing+ Fibonaccibracketing+ [7] E. A. Khan, A. O. Akyuz, and E. Reinhard. Ghost removal Conventional registration Generalized registration in high dynamic range images. In IEEE ICIP, 2006. 2 Figure 9. Comparison between conventional and generalized [8] M. Livio. The Golden Ratio: The Story of Phi, the World’s registration. The same LDR images (11 frames of a Fibonacci Most Astonishing Number. Broadway Books, 2002. 4 bracketing sequence) were used for both cases. Conventional reg- [9] S. Mann and R. W. Picard. On being ‘undigital’ with digital istration does not exploit the iso-exposure property of Fibonacci cameras: Extending dynamic range by combining differently bracketing. This results in incorrect registration and strong ghost- exposed pictures. In Proc. of IST, 1995. 1, 2 ing artifacts. Generalized registration produces a ghost-free image. [10] T. Mertens, J. Kautz, and F. V. Reeth. Exposure fusion. In Proc. Pacific Graphics, 2007. 5 sensor bit-depth, and require minimal modifications to use [11] S. K. Nayar and T. Mitsunaga. High dynamic range imaging: with existing sensors. Thus, our techniques are particularly Spatially varying pixel exposures. In CVPR, 2000. 2, 4 suitable for implementation on inexpensive image sensors [12] E. Reinhard, M. Stark, P. Shirley, and J. Ferwerda. Photo- such as ones used in cell-phone cameras. graphic tone reproduction for digital images. In ACM SIG- The technique of Fibonacci bracketing+generalized reg- GRAPH, 2002. 6 istration should be seen as a general ‘pre-conditioning’ step [13] M. Rouf, R. Mantiuk, W. Heidrich, M. Trentacoste, and in HDR imaging, that enhances the accuracy of existing C. Lau. Glare encoding of high dynamic range images. In image-alignment/optical flow methods. Note that it is not IEEE CVPR, 2011. 2 a new alignment technique in itself. [14] D. Schleicher and B. G. Zagar. High dynamic range imag- ing by varying exposure time, gain and aperture of a video Limitations: While the proposed approach significantly camera. In Proc. of IEEE Instrumentation and Measurement mitigates registration artifacts, it may not completely re- Technology Conference, 2010. 1 move them. Our method shares the limitations of dense [15] P. Sen, N. K. Kalantari, M. Yaesoubi, S. Darabi, D. B. Gold- optical flow techniques (e.g., aperture problem), and hence man, and E. Shechtman. Robust patch-based HDR recon- may not perform reliably for textureless regions, occlusions struction of dynamic scenes. ACM Trans. Graph., 31(6), and in the presence of highly non-rigidmotion (such as fluid 2012. 2, 6 motion). For extremely fast motions, or large inter-frame [16] W. R. Spickerman and R. N. Joyner. Binet’s formula for the time gaps, our technique may not produce a good result. recursive seq. of order K. Fibonacci Quart., 22, 1984. 5 In order to remove the residual artifacts, one of the post- [17] D. Sun, S. Roth, and M. J. Black. Secrets of optical flow processing techniques discussed in Section 2 may be used. estimation and their principles. In IEEE CVPR, 2010. 6 [18] M. Tallon, J. Mateos, S. Babacan, R. Molina, and A. K. Kat- References saggelos. Space-variant blur deconvolution and denoising in the dual exposure problem. Information Fusion, 2012. 6 [1] Project webpage. http://www.cs.columbia.edu/CAVE/ projects/FibonacciHDR/. 5, 6, 7 [19] M. Tocci, C. Kiser, N. Tocci, and P. Sen. A versatile HDR video prodction system. ACM Trans. Graph., 30(4), 2011. 2 [2] P. E. Debevec and J. Malik. Recovering high dynamic range radiance maps from photographs. In ACM SIGGRAPH, [20] G. Ward. Fast, robust image registration for compositing 1997. 1, 2, 3 high dynamic range photographs from handheld exposures. Journal Of Graphics Tools, 8, 2003. 6 [3] O. Gallo, N. Gelfand, W. Chen, M. Tico, and K. Pulli. [21] L. Zhang, A. Deshpande, and X. Chen. Denoising versus Artifact-free HDR imaging. In IEEE ICCP, 2009. 2 deblurring: HDR techniques using moving cameras. In IEEE [4] M. D. Grossberg and S. K. Nayar. What is the space of cam- CVPR, 2010. 2, 3, 4, 6, 7 era response functions? In IEEE CVPR, 2003. 5 Supplementary Technical Report for Paper ID 0714 In this technical report, we provide derivations supporting the content in the paper submission titled “Fibonacci exposure bracketing for high dynamic range imaging”. There are two sections: • Section 1 provides derivation of the normalized intensity errors (for con- ventional and generalized registration) due to non-linear camera response. This was discussed in Section 5, between lines 463-484 of the main paper. • Section 2 shows a comparison for evaluating the effect of different sensor bit-depths on the results. 1 Derivation of the Average Intensity Difference Due to Non-linear Camera Response Several digital sensors have non-linear intensity response, especially most con- sumer point-and-shoot cameras, cell-phone cameras and web-cams. Some typ- ical camera responses are shown in Figure 1 (a). In this section, we compare the effect of non-linearities in camera response on conventional and Fibonacci bracketing based registration. Consider two exposure bracketed frames fi and fi+1, with exposures ei and ei+1. Let I be the image irradiance of a scene point P . The corresponding image intensities in the two images are: Ei = C(I) (1) Ei+1 = C (RI) (2) C . R ei+1 where ( ) is the camera’s response curve, and = ei is the ratio of expo- sures. Optical flow techniques assume that the intensity of a scene point in two images being registered remains the same (after scaling by the exposure times). However, because of the non-linear response, the intensities in the two images may be different, leading to registration errors. The normalized difference in intensities for a scene point with irradiance I, a sensor with response curve C and the exposure ratio R is given as: |R C(I) − C(RI)| D (C,I,R) = (3) conv R C(I) 1 1 2 Generalized Registration Conventional Registration 0.8 1.5 0.6 1 0.4 0.5 0.2 Intensity Difference Intensity Image Intensity (0-1) Intensity Image 0 0 0 0.2 0.4 0.6 0.8 1 0 5 10 15 20 Image Irradiance (0-1) Camera Response Function Index (a) Camera response functions (b) Normalized intensity errors Figure 1: Robustness of Fibonacci bracketing to non-linear camera re- sponse functions. (a) 20 response functions (from [1]). (b) Average intensity difference between pixels in source and target images for each response function. Differences for Fibonacci bracketing based generalized registration is always less than that of conventional registration. where the subscript conv refers to conventional registration. Clearly, the large the difference, the higher the probability of registration error. In order to evalu- ate the effect of different response functions, we compute the average error over the exposure ratio and the irradiance: |R C(I) − C(RI)| Dconv(C) = dR dI . (4) Z Z R C(I) R I For a given response curve C, the above integral is computed numerically. The limits of I are between 0 and 1, and the limits of R are between 2 and 16 (corresponding to exponential bracketing schemes with different growth factors). The maroon-colored bars in Figure 1 (b) illustrate Dconv(C) for 20 different re- sponse functions. Strongly non-linear curves result in large intensity difference, thus increasing the probability of registration errors. In Fibonacci bracketing and generalized registration, flow is computed be- tween a frame and sum of two previous frames. Let the three exposures be e, 2 1+√5 Rfib × e and Rfib × e, where Rfib = 2 is the approximate ratio between consecutive exposures. Optical flow is computed between the sum of first two frames and the third frame. The normalized difference in intensities for a scene point with irradiance I is given as: 2 |C(I)+ C(RfibI) − C(RfibI)| Dfib(C, I) = . (5) C(I)+ C(RfibI) The average error for a response function G is given as: 2 2 |C(I)+ C(RfibI) − C(RfibI)| Dfib(C) = dI . (6) Z C(I)+ C(RfibI) I The blue-colored bars in Figure 1 (b) illustrate Dfib(C), and are signifi- cantly smaller than the bars for conventional registration. This makes Fibonacci bracketing highly robust to non-linear camera response functions. Thus, with the proposed approach, it is possible to achieve high-quality results without cali- brating the camera’s response curve. This is especially useful in exposure-fusion based approaches, where exposure bracketed images are directly merged into a high-quality image without computing an intermediate HDR image [3]. 2 Evaluating the Effect of Sensor Bit-Depth The M310 camera that we used for our experiments has a 14 bits sensor. Cheap sensors, especially cell-phone sensors have lower bit-depths (10 bits). In order to evaluate the results for different bit-depths, we emulated an 10 bits camera by clipping the lower 4 bits from the captured images. The resulting HDR reconstructions for different schemes are shown in Figure 2. The performances of both the burst and the alternating schemes degrade significantly as the bit-depth decreases. This is because both these schemes capture only low to moderate dynamic range. Fibonacci bracketing maintains high signal-to-noise-ratio even for low bit-depths, and thus can be used with low-quality sensors. For more results, please see the supplementary image gallery. References [1] Michael D. Grossberg and Shree K. Nayar. What is the space of camera response functions? In Proc. IEEE CVPR, 2003. [2] Sing Bing Kang, Matthew Uyttendaele, Simon Winder, and Richard Szeliski. High dynamic range video. ACM Trans. Graph., 22(3), 2003. [3] Tom Mertens, Jan Kautz, and Frank Van Reeth. Exposure fusion. In Proc. Pacific Graphics, 2007. [4] Li Zhang, Alok Deshpande, and Xin Chen. Denoising versus deblurring: HDR techniques using moving cameras. In Proc. IEEE CVPR, 2010. 3 14 bits sensor 10 bits sensor Burst scheme Alternating scheme Fibonacci scheme [4] [2] [This paper] Figure 2: Evaluating the effect of sensor bit-depth. HDR results of dif- ferent bracketing schemes using sensors with bit-depths of 14 bits (top) and 10 bits (bottom). Performance of the burst and the alternating schemes degrades significantly as the bit-depth decreases. Fibonacci bracketing maintains high signal-to-noise-ratio even for low bit-depths, and thus can be used with cheap sensors. 4