L2 Norm Optimization of Tone Mapping for Two Layer Lossless Coding of HDR Images

Masahiro Iwahashi∗,Taichi Yoshida∗, and Hitoshi Kiya† ∗ Department of Electrical Engineering, Nagaoka University of Technology, Nagaoka, Niigata, Japan E-mail: { iwahashi, yoshida }@vos.nagaokaut.ac.jp † Department of Information and Communication Systems, Tokyo Metropolitan University, Hino, Tokyo, Japan

Abstract—This paper optimizes a tone mapping function in II.EXISTING METHOD a backward compatible lossless coding of high dynamic range (HDR) images. The mapping function is designed so that it A. High Dynamic Range Images maximizes quality of decoded low dynamic range (LDR) images. In this paper, we deal with HDR images in the ‘Open EXR’ An optimization procedure for a lossy coding system is extended format [2] as an example. In this floating point data format, a to a lossless system under the condition that a user can specify pixel value xH of an input HDR image is expressed as arbitrary tone mapping function to produce an LDR image from { the original HDR image. Superiority of the proposed method on −10 xE −14 xS xM × 2 × 2 × (−1) , xE = 0 the rate distortion curves of the LDR image was confirmed to x = − − , H × 10 × xE 15 × − xS ̸ be up to 7.5 dB in PSNR improvement for an image sample. (1 + xM 2 ) 2 ( 1) , xE = 0 (1) I.INTRODUCTION where x , x and x are mantissa, exponent and sign of Recently, high dynamic range (HDR) images have been M E S the pixel. Those are given as integers in the range of x ∈ attracting researchers’ attention since it can express extremely M [0, 210), x ∈ [0, 25), and x = 0, 1. It also defines infinite wider dynamic range of pixel values than standard low dy- E S and not a number. However those are omitted below without namic range (LDR) images [1]. As it requires huge memory loss of generality. space to be stored, data compression of HDR images is one of important issues. So far, various lossless coding algorithms B. Lossy Coding System e.g. RLE, PIZ and LogLuv have been reported [2]–[4]. Those Fig. 1 illustrates a backward compatible two layer coding were extended to lossy coding for further compression [5]. system. Its encoder generates two kinds of bit streams b In the international standardization activities [6], it is also s1 and b as compressed data of the input HDR image x . The required to have backward compatibility with the standard s2 H system has backward compatibility as the output LDR image coding algorithm such as JPEG and JPEG 2000 developed y can be reconstructed with a standard decoder from the for LDR images [7]. L bit streams b . The system also reconstructs the output HDR A backward compatible two layer coding system outputs s1 image y combining with another bit streams b . Note that two kinds of bit streams [8]. One of them contains compressed H s2 both of yL and y contain quantization errors since the system data for decoding an LDR image. Combining with another H is composed of ‘lossy’ encoders and decoders. bit stream, the input HDR image is decoded. The two layer In a previous paper [18], the function h, which maps an coding system has been developed also for video signals HDR pixel value x to an LDR pixel value x , was optimized [9], [10]. Recently, quality of the HDR image was improved H L according to the histogram of pixel values in the input HDR introducing piecewise linear modeling of the tone mapping image. It was designed so that variance of the residue e [11]. Replacing the ratio image with a low pass filtered image, H was minimized. It contributes to reduce data size of the bit coding efficiency was significantly improved [12]. stream b . However, image quality of the LDR image was However, those previously reported systems are based on s2 not considered. Subsequently, a constraint of the LDR image lossy coding of HDR images. quality was added in [17] to the optimization procedure. As a Unlike those previous reports, we deal with lossless coding result, coding performance was significantly improved in lossy of HDR images. It is well known that extremely long bit depth coding of the LDR image. makes the coding problem difficult [13]. Introducing a lossless h version of the lossy logarithmic range reduction [5], lossless Fig. 2 illustrates a model of Fig. 1 for optimization of ex. e two layer systems were proposed [14]–[16]. Even though a The quantization error Q is added by the lossy encoder and h user can use arbitrary tone mapping function to produce an decoder. In [18], the function ex was optimized as ˆ 2 LDR from the HDR image, there is no room to optimize hex = arg min ||eH || (2) it according to the input image. In this paper, we introduce hex one more function and determine its shape based on the L2 for norm optimization in [17], so that quality of the LDR image −1 − is increased. eH = hex (eQ + hex(xH )) xH (3)

h h-1 h-1 㽢C 㾂C 㾂C

lossy lossy decoder decoder LDR LDR LDR x lossy lossy y x lossy lossy y L encoder decoder L Q encoder decoder L bs1 bs1 bit stream bit stream

Fig. 1: A two layer backward compatible ‘lossy’ coding Fig. 3: The existing method in [14]. It reconstructs HDR image system. The encoder outputs two kinds of bit streams bs1 and xH without any loss. The reversible logarithmic mapping fex bs2. The decoder reconstructs an LDR image yL from bs1, was introduced to reduce the range of HDR pixel values. and an HDR image yH from bs1 and bs2. Both of yL and yH The mapping gex is additionally introduced to compensate the contain quantization errors. effect of fex.

HDR e HDR - H + + = + xH yH xH eP system in Fig. .1. Therefore, we introduce an optimization procedure to this existing method in Sec. .III. e h h-1 P h-1 Qnt. error LDR III.PROPSED METHOD e + y =x +e Q L L Q A. Proposed Lossless Coding System 2 Fig. 2: The mapping h was optimized so that ||eH || was In this paper, we develop a backward compatible lossless minimized to reduce data size of bs2 in [18]. A constraint of coding of HDR images expressed in a floating point data 2 ||eQ|| was added to consider quality of yL in [17]. format via introducing the optimization procedure in [17] to the existing method in Sec. II-C. The proposed method is illustrated in Fig. 4. The scaling constant C in Fig. 3 is so that data size of bs2 in Fig. 1 was reduced. On the contrary, replaced with a function h. We optimize h for a given input 2 in [17], a constraint of ||eQ|| with a given constant CQ was HDR image, shown in Sec. III-C. added as The mapping from integers to a rational number in (1) is denoted as x = F (x ), where x = [x , x , x ]. The ˆ || ||2 || ||2 H DH D D M E S hex = arg min eH s.t. eQ = CQ (4) proposed function f is defined as hex b −1 so that data size of s1 in Fig. 1 was reduced maintaining xN = f(xH ) = FDN (FDH (xH )), (6) quality of the reconstructed LDR image. where C. Lossless Coding System F (x ) = (x + 210(x − C )) × (−1)xS . Fig. 3 illustrates the lossless coding system reported in DN D M E MinE [14]. A reversible logarithmic mapping xN = fex(xH ) was A constant CMinE is the minimum of xE in a given HDR introduced to reduce the range of HDR pixel values. Its inverse −1 image. Since the mapping f in (6) is reversible between xH f reconstructs the rational number xH from the integer xN ex and xN , both of range reduction and lossless coding of HDR without any loss. Applying a lossless encoder to the residue images become possible [13]–[16]. Note that f approximates integer eH , the HDR pixel value xH , which is exactly the a logarithmic function as same as the input value, is reproduced from bs1 and bs2. 10 − − Similarly to the system in Fig. 1, the LDR pixel value yL f(xH ) = 2 (log2 xH + 15 CMinE δ) (7) is reconstructed from bs1 with a lossy decoder. Note that a mapping gex was additionally introduced to compensate the for effect of fex. It was designed as 1 + ϵ ∈ × −10 ∈ ( ( )) δ = log2 [0, 0.086], ϵ = xM 2 [0, 1). −1 −1 2ϵ gex(xQ) = T fex C xQ for xL = T (xH ) (5) We also introduce the function g defined as for a given tone mapping function T . A constant C was ( ( )) x x −1 −1 determined to reduce the range of N so that Q meets g(xQ) = T f h (xQ) for xL = T (xH ) (8) requirement of the maximum bit depth of the lossy encoder. Since all of the functions fex, gex and a constant C are fixed, so that an LDR image xL, which is tone mapped with a given this lossless system cannot be optimum unlike the lossy coding tone mapping function T , is reconstructed. Residue HDR HDR C. Solution to the Optimization Problem xN eN lossless bs2 lossless -1 xH f - + f xH ′ encoder decoder Let L(xN , h, h ) be the Lagrangian of (12) as ( ) -1 -1 ′ 2 h h h ′ T L(xN , h, h ) = ′ × ′ eQ PN , (13) lossy f h decoder LDR and its Euler-Lagrange equation is derived as x lossy lossy Q g yL d ∂L ∂L encoder b decoder − s1 ′ = 0. (14) bit stream dxN ∂h ∂h Consequently, we have Fig. 4: The proposed method. The scaling constant C in ′ ∝ ′ 2/3 ′ −2/3 1/3 the existing method [14] is replaced with a function h. It is h (T ) (f ) PN , (15) optimized according to histogram of pixel values in the input and then we define h(xN ) as image. ∫ xN ′ 2/3 ′ −2/3 1/3 h(xN ) = C (T ) (f ) P dx. (16) HDR x e y HDR N N N N -1 −∞ xH f - + f xH A scaling parameter C is determined so that (16) satisfies the h h-1 h-1 boundary condition for the minimum and the maximum of pixel values in the image signal. Finally, the optimized func- Qnt. error LDR tion h was derived for a given tone mapping T . Performance eQ + g yL=xL+eL yQ=xQ+eQ of the proposed method is evaluated in Sec. IV. Note that PN in (12) can be replaced with PH which Fig. 5: This paper optimizes h considering f and g so that denotes PDF of the HDR image xH as 2 ||eL|| is minimized to increase quality of the LDR image yL. dxH PH PN dxN = PH dxH or PN = PH = ′ . (17) dxN f

B. Formulation of the Proposed Optimization Problem However we use PN rather than PH to avoid unstable exper- Similarly to [17], we optimize h to increase quality of imental results due to the histogram sparseness [19], [20]. the reconstructed LDR image. Fig.5 illustrates a model of IV. RESULTS AND DISCUSSIONS Fig.4 to determine h for a given tone mapping T in (5). The optimization procedure is formulated as Fig. 6 illustrates examples of decoded LDR images yL in our experiments. We used the Hill function as the tone ˆ 2 h = arg min ||eL|| , (9) h mapping function T [21]. The left column of Fig. 7 shows the LDR image quality where 2 index ||eL|| measured with the peak-signal to noise ratio − eL = g (eQ + h(f(xH ))) xL. (PSNR) in the vertical axis, versus lossy coding error index || ||2 A closed form solution to this problem is described as below. eQ in the horizontal axis. Comparing to the case without h, Substituting (8) into (9), we have it was observed that the proposed optimization in (16) gives ( ( )) better LDR image quality by approximately 3.5 dB for Mt. −1 −1 − eL = T f h (eQ + h(f(xH ))) T (xH ). (10) Tam West, 3.8 dB for Desk, and 2.0～2.7 dB for Still Life, Applying the Taylor expansion, it can be approximated as respectively. The right column of Fig. 7 indicates the LDR 2 ′ image quality index ||e || in the vertical axis, versus bit rate ( ( )) T L e = T f −1 h−1(e ) ≃ T ′(f −1)′(h−1)′e = e , of the image x in lossy coding (i.e., data size index of the bit L Q Q f ′ × h′ Q Q stream b ) in the horizontal axis. Superiority of the proposed (11) s1 method in respect of LDR image quality was observed to be where 1.7, 2.6 and 7.5 dB at the maximum, and 0.9, 1.8 and 5.3 ′ dT (xH ) ′ df(xH ) ′ dh(xN ) dB at the minimum for Mt. Tam West, Desk and Still Life, T = , f = , and h = . dxH dxH dxN respectively. Therefore, the norm in (9) becomes In this paper, variance of the quantization error in the ∫ ( ) LDR image was considered in the optimization explained in ∞ T ′ 2 || ||2 Sec. III-C. It is also possible to minimize the error in the eL = ′ ′ eQ PN dxN , (12) −∞ f × h 2 2 HDR images replacing ||eL|| in (14) with ||eN || . Note that where PN denotes the probability density function (PDF) of reduction of bit rates (data size of the bit streams) were xN . We know that this approximation is ineffective for large not explicitly considered in this paper. A bit rate constrained noise in dark pixels. However, note that f in (7) reliefs this approach with iteration can be utilized [22]. Discussions on difﬁculty comparing to the case without f. other norm such as the L inﬁnity can be found in [23]. Mt.Tam West

䕿 min eL 䕿 min eL 㽢 min eN 㽢 min eN (a) Mt. Tam West (b) Still Life (c) Dest w/o h w/o h

Fig. 6: Examples of decoded LDR images.

(a) Mt. Tim West V. CONCLUSION A tone mapping function h was introduced to a two layer lossless coding of HDR images. The function was designed so that it maximizes quality of decoded LDR images according to histogram of pixel values of the input image signal. The 䕿 min e 䕿 min eL L min e 㽢 min eN 㽢 N function can be used as an option for customization of the w/o h w/o h system. Applying the proposed method to HDR images in a ﬂoating point data format, effectiveness of our optimization was conﬁrmed in PSNR improvement. Since our discussion is limited to monochrome images, it should be extended to color (b) Still Life images in the future.

ACKNOWLEDGMENT This work was supported by JSPS KAKENHI Grant Num- ber 26289117. 䕿 min eL 䕿 min eL 㽢 min eN 㽢 min eN REFERENCES w/o h w/o h [1] S. Daly, “Digital images and human vision.” MIT Press, 1993, ch. The Visible Differences Predictor: An Algorithm for the Assessment of Image Fidelity, pp. 179–206. [2] R. Bogart, F. Kainz, and D. Hess, “OpenEXR image file format,” ACM (c) Desk SIGGRAPH, Sketches & Applications, 2003. [3] E. Reinhard, G. Ward, S. Pattanaik, and P. Debevec, High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting. Mor- Fig. 7: LDR image quality and Rate distortion curve in the gan Kaufmann, 2010. base layer. [4] A. Motra and H. Thoma, “An adaptive logluv transform for high dynamic range video compression,” in Proc. IEEE Int. Conf. Image Process., 2010, pp. 2061–2064. [5] R. Xu, S. N. Pattanaik, and C. E. Hughes, “High-dynamic-range still- [15] ——, “Two layer lossless coding of HDR images,” in Proc. IEEE Int. image encoding in JPEG 2000,” IEEE Comput. Graphics Appl., vol. 25, Conf. Acoustics Speech Signal Process., 2013, pp. 1340–1344. pp. 57–64, Nov. 2005. [16] C. Y. Ping, M. Iwahashi, and H. Kiya, “Lossless bit depth scalable [6] ISO/IEC JTC 1/SG 29 /WG 1 N6147, JPEG Part1: Call for proposals coding for floating point images,” in Proc. Int. Workshop Adv. Image for JPEG HDR. Technol., 2013, pp. 169–174. [7] ISO/IEC 15444-1, Information technology - JPEG 2000 image coding [17] A. Koz and F. Dufaux, “Optimized tone mapping with LDR image system: Core coding system. quality constraint for backward-compatible high dynamic range image [8] G. Ward and M. Simmons, “JPEG-HDR: a backwards-compatible, high and video coding,” in Proc. Int. Conf. Image Process., 2013, pp. 1762– dynamic range extension to JPEG,” in Proc. ACM SIGGRAPH 2006 1766. Courses, 2006. [18] Z. Mai, H. Mansour, R. Mantiuk, P. Nasiopoulos, R. Ward, and W. Hei- [9] R. Mantiuk, A. Efremov, K. Myszkowski, and H.-P. Seidel, “Backward drich, “Optimizing a tone curve for backward-compatible high dynamic compatible high dynamic range MPEG video compression,” ACM Trans. range image and video compression,” IEEE Trans. Image Process., Graph., vol. 25, no. 3, pp. 713–723, 2006. vol. 20, no. 6, pp. 1558–1571, 2011. [10] M. Winken, D. Marpe, H. Schwarz, and T. Wiegand, “Bit-depth scalable [19] M. Iwahashi, H. Kobayashi, and H. Kiya, “Lossy compression of sparse video coding,” in Proc. IEEE Int. Conf. Image Process., vol. 1, 2007, histogram image,” in Proc. IEEE Int. Conf. Acoustics Speech Signal pp. 5–8. Process., 2012, pp. 1361–1364. [11] I. R. Khan, “Two layer scheme for encoding of high dynamic range [20] ——, “Fine rate control and high SNR coding for sparse histogram images,” in Proc. IEEE Int. Conf. Acoustics Speech Signal Process., images,” in Proc. Picture Coding Symp., 2012, pp. 205–209. 2008, pp. 1169–1172. [21] E. Reinhard, M. Stark, P. Shirley, and J. Ferwerda, “Photographic tone [12] T. Jinno, M. Okuda, and N. Adami, “New local tone mapping and reproduction for digital images,” ACM Trans. Graph., vol. 21, no. 3, pp. two-layer coding for HDR images,” in Proc. IEEE Int. Conf. Acoustics 267–276, 2002. Speech Signal Process., 2012, pp. 765–768. [22] N. S. Jayant and P. Noll, Digital Coding of Waveforms, Principles and [13] H. Kikuchi, W. Otake, and M. Iwahashi, “Bit rate reduction of enhance- Applications to Speech and Video. Prentice-Hall, Englewood Cliffs NJ, ment layer in bit-depth scalable coding,” in Proc. Int. Symp. Intell. Signal 1984. Process. Commun. Syst., 2009, pp. 264–267. [23] M. Iwahashi and H. Kiya, “Error equalization for high quality ldr images [14] M. Iwahashi and H. Kiya, “Efficient lossless bit depth scalable coding in backward compatible hdr image coding,” in Proc. Asia-Pacific Signal for HDR images,” in Proc. Asia-Pacific Signal Inf. Process. Association Inf. Process. Association Annual Summit Conf., 2013, pp. 1–4. Annual Summit Conf., 2012, pp. 1–4.