Wavelet Transform and JPEG2000 Yao Wang Polytechnic School of Engineering, New York University © Yao Wang, 2016 EL-GY 6123: Image and Video Processing 1 Lecture Outline • Introduction • Multi-resolution representation of images: Gaussian and Laplacian pyramids • Wavelet transform through Iterated Filterbank Implementation • Basic Ideas in JPEG2000 Codec • JPEG vs. JPEG2000 • Scalability: why and how © Yao Wang, 2016 EL-GY 6123: Image and Video Processing 2 Multi-Resolution Representation (aka Pyramid Representation) © Yao Wang, 2016 EL-GY 6123: Image and Video Processing 3 Use of Pyramid Representations • Recall HBMA: uses pyramid representation to speed up motion estimation • Many other applications – Feature extraction across scales (SIFT) – Extracting faces of different sizes – … © Yao Wang, 2016 EL-GY 6123: Image and Video Processing 4 © Yao Wang, 2016 EL-GY 6123: Image and Video Processing 5 Pyramid is a redundant representation • A pyramid (either Guassian or Laplacian) includes an image of the original size plus additional smaller images • How many samples in all levels? • Original image (level J-0) NxN • Next level (level J-1): N/2xN/2 • Level J-l (l=0 to J): N/(2^l) x N/(2^l) • Total # samples =N^2 \sum_{l=0}^{J} (1/4^l)=N^2(1-(1/4)^(J-1))/ (1-1/4) \approx 4/3 N^2 • However, with Laplacian pyramid, many samples are close to zero except at the top level. • Wavelet transform is similar to Laplacian pyramid but does not incur additional pixels (non-redundant representation). © Yao Wang, 2016 EL-GY 6123: Image and Video Processing 6 Wavelet vs. Pyramid vs. Subband Decomposition • Pyramid is a redundant transform (more samples than original) • Wavelet is a non-redundant multi-resolution representation • There are many ways to interpret wavelet transform. Here we describe the generation of discrete wavelet transform using the tree-structured subband decomposition (aka iterated filterbank) approach – 1D 2-band decomposition – 1D tree-structured subband decomposition (discrete wavelet transform) – Harr wavelet as an example – Extension to 2D by separable processing © Yao Wang, 2016 EL-GY 6123: Image and Video Processing 7 Two Band Filterbank s(n) u(n) q(n) r(n) t(n) v(n) © Yao Wang, 2016 EL-GY 6123: Image and Video Processing 8 What does the filter bank do? • When H0 and G0 are ideal Low-Pass and H1 and G1 are ideal High-Pass filters h0: Lowpass filter (0-¼), y0: a low-passed and then down-sampled version of x (Sampling theorem tells us we can down-sample after bandlimiting) h1: Highpass filter (1/4-1/2), y1: a high-passed and then down-sampled version of x (Sampling theorem also works in this case) g0: interpolation filter for low-pass subsignal g1: interpolation filter for high-pass subsignal • Can still reach perfect reconstruction even if these filters are not ideal low-pass/high-pass filters! – When the filters h0,h1, g0, g1 are designed appropriately, – X^=X (perfect reconstruction filterbank) © Yao Wang, 2016 EL-GY 6123: Image and Video Processing 9 DTFT of signals after downsampling and upsampling Down&sampling!by!factor!of!2: 1⎛ ⎛ u⎞ ⎛ u 1⎞ ⎞ x (m) x(2m) X (u) X X d = ⇔ d = ⎜ ⎜ ⎟ + ⎜ − ⎟ ⎟ 2⎝ ⎝ 2⎠ ⎝ 2 2⎠ ⎠ Up&sampling!by!factor!of!2: ⎧ ⎛ m⎞ ⎪ x⎜ ⎟ , m = even xu(m)= ⎨ ⎝ 2 ⎠ ⇔ Xu(u)= X(2u) ⎪ 0, otherwise ! ⎩⎪ Conceptual proof by doing sampling on continuous signal under 2 different sampling rates. © Yao Wang, 2016 EL-GY 6123: Image and Video Processing 10 Perfect Reconstruction Conditions x(n)*h0(n)⇔ X(u)H0(u) xl(n)= down(x(n)*h0(n))⇔ Xl(u)= (X(u/2)H0(u/2)+ X(u/2−1/2)H0(u/2−1/2))/2 up(xl(n))⇔ Xl(2u)= (X(u)H0(u)+ X(u−1)H0(u−1))/2 x(n)= up(x (n))* g (n)+up(x (n))* g (n) l 0 h 1 ⇔ X(u)= (X(u)H0(u)G0(u)+ X(u−1)H0(u−1)G0(u))/2 +(X(u)H1(u)G1(u)+ X(u−1)H1(u−1)G1(u))/2 = X(u)(H0(u)G0(u)+ H1(u)G1(u))/2 + X(u−1)(H (u−1)G (u)+ H (u−1)G (u))/2 0 0 1 1 To!guarantee!X(u)= X(u),!we!need H0(u)G0(u)+ H1(u)G1(u)= 2 H (u−1)G (u)+ H (u−1)G (u)= 0!!(To!remove!aliasing!component!) ! 0 0 1 1 © Yao Wang, 2016 EL-GY 6123: Image and Video Processing 11 Perfect!reconstruction!condition: H (u)G (u) H (u)G (u) 2 0 0 + 1 1 = Perfect reconstruction H (u−1)G (u)+ H (u−1)G (u)= 0 0 0 1 1 conditions The!second!equation!(aliasing!cancelation)!can!be!guaranteed!by!requiring n G (u) H (u 1) g (n) 1 h (n) 0 = 1 − ⇔ 0 = (− ) 1 n+1 G (u) H (u 1) g (n) 1 h (n) 1 = − 0 − ⇔ 1 = (− ) 0 (Quadrature!Mirror!Condition) To!guarantee!perfect!reconstruction,!the!filters!must!satisfy!the!biorthogonality!condition: < hi(2n− k), gj (k)>= δ(i − j)δ(n) One!has!freedom!to!design!both!g0(n), g1(n),!which!can!have!different!length. A!more!strict!condition!requires!orthonality!between!g0(n), g1(n)!: < gi(n), gj (n+2m)>= δ(i − j)δ(m) which!yields n g (n) 1 g (L 1 n), 1 = (− ) 0 − − h0(n)= g0(L−1− n) n n h (n) g (L 1 n) 1 g (n) 1 h (L 1 n) 1 = 1 − − = (− ) 0 = (− ) 0 − − One!only!has!freedom!to!design!g (n),!filter!length!L!must!be!even!and!all!filters!have!same!length. ! 0 © Yao Wang, 2016 EL-GY 6123: Image and Video Processing 12 Haar Filter (Simplest Orthogonal Wavelet Filter) h0 : averaging, [1,1]/ 2; h1: difference, [1,-1]/ 2; g0 = [1,1]/ 2; g1 = [-1,1]/ 2 Input sequence : [x1,x2, x3,x4,....] Analysis(Assuming samples outside the boundaries are 0. remember to flip the filter when doing convolution) s = x *h0 = [s0,s1, s2, s3, s4,...], s0 = (x1+ 0)/ 2,s1 = (x2+ x1)/ 2,s2 = (x3+ x2)/ 2,s3 = (x4+ x3)/ 2... y0 = s ↓ 2 = [s1, s3,...,] t = x *h1 = [t0, t1,t2,t3,t4,...], t0 = [x1- 0]/ 2, t1 = [x2- x1]/ 2, t2 = [x3- x2]/ 2, t3 = [x4- x3]/ 2,... y1= t ↓ 2 = [t1, t3,...] Synthesis: u = y0 ↑ 2 = [0,s1,0,s3,...] q = u *g0 = [q1,q2,q3,q4,...],q1 = (s1+ 0)/ 2 = (x1+ x2)/2, q2 = (0+ s1)/ 2 = (x1+ x2)/2, q3 = (s3+ 0)/ 2 = (x3+ x4)/2 v = y1↑ 2 = [0,t1,0, t3,...] r = v*g1 = [r1,r2,r3,r4,...],r1= (-t1+ 0)/ 2 = (x1− x2)/2, r2 = (-0+ t1)/ 2 = (-x1+ x2)/2, r3 = (-t3+ 0)/ 2 = (x3− x4)/2, xˆ = q + r = [q1+ r1,q2 + r2,....] = [x1,x2, x3,...] Note with Haar wavelet, the lowpass subband essentially takes the average of every two samples, L=(x1+x2)/sqrt(2), and the highpass subband takes the difference of every two samples, H=(x1-x2)/ sqrt(2). For synthesis, you take the sum of the lowpass and high pass signal to recover first sample A=(L +H)/sqrt(2), and you take the difference to recover the second sample B=(L-H)/sqrt(2). © Yao Wang, 2016 EL-GY 6123: Image and Video Processing 13 Common Wavelet Filters • Haar: simplest, orthogonal, not very good in energy compaction • Daubechies 8/8: orthogonal • Daubechies 9/7: bi-orthogonal, most commonly used if numerical reconstruction errors are acceptable • LeGall 5/3: bi-orthogonal, integer operation, can be implemented with integer operations only, used for lossless image coding • Differ in energy compaction capability © Yao Wang, 2016 EL-GY 6123: Image and Video Processing 14 © Yao Wang, 2016 EL-GY 6123: Image and Video Processing 15 MATLAB example >> [ca,cd]=dwt(y,'db4'); >> z=idwt(ca,cd,'db4'); >> wy=[ca,cd]; >> subplot(3,1,1),plot(y), title('Original sequence'); >> subplot(3,1,2),plot(wy), title('Wavelet transform: left=low band, righ=high band'); >> subplot(3,1,3),plot(y), title('Reconstructed sequence'); © Yao Wang, 2016 EL-GY 6123: Image and Video Processing 16 Iterated Filter Bank From [Vetterli01] © Yao Wang, 2016 EL-GY 6123: Image and Video Processing 17 MATLAB example >> [ca,cd]=dwt(y,'db4'); [caa,cad]=dwt(ca,'db4'); [caaa,caad]=dwt(caa,'db4'); wy1=[ca,cd]; wy2=[caa,cad,cd]; >> wy3=[caaa,caad,cad,cd]; >> subplot(4,1,1),plot(y),title('Or iginal Signal'); >> subplot(4,1,2),plot(wy1),title(' 1-level Wavelet Transform'); >> subplot(4,1,3),plot(wy2),title(' 2-level Wavelet Transform'); >> subplot(4,1,4),plot(wy3),title(' 3-level Wavelet Transform'); © Yao Wang, 2016 EL-GY 6123: Image and Video Processing 18 Discrete Wavelet Transform = Iterated Filter Bank © Yao Wang, 2016 EL-GY 6123: Image and Video Processing 19 Temporal-Frequency Domain Partition © Yao Wang, 2016 EL-GY 6123: Image and Video Processing 20 Wavelet Transform vs. Fourier Transform • Fourier transform: – Basis functions cover the entire signal range, varying in frequency only • Wavelet transform – Basis functions vary in frequency (called “scale”) as well as spatial extend • High frequency basis covers a smaller area • Low frequency basis covers a larger area • Non-uniform partition of frequency range and spatial range • More appropriate for non-stationary signals © Yao Wang, 2016 EL-GY 6123: Image and Video Processing 21 Haar Wavelet: Analysis Note that the assumed high pass filter in this example has a factor “-1” difference from our previous example. Similarly the synthesis filter is off by the same factor. Both are OK. © Yao Wang, 2016 EL-GY 6123: Image and Video Processing 22 Haar Wavelet: Synthesis © Yao Wang, 2016 EL-GY 6123: Image and Video Processing 23 How to Apply Filterbank to Images? 2D decomposition is accomplished by applying the 1D decomposition along rows of an image first, and then columns.