Discrete Geometrical Image Processing using the Contourlet Transform
Minh N. Do
Department of Electrical and Computer Engineering University of Illinois at Urbana-Champaign
Joint work with Martin Vetterli (EPFL and UC Berkeley), Arthur Cunha, Yue Lu, Jianping Zhou (UIUC), and Duncan Po (Mathworks). Outline
1. Motivation
2. Discrete-domain construction using filter banks
3. Contourlets and directional multiresolution analysis
4. Contourlet approximation and directional vanishing moments
5. Applications
1. Motivation 1 What Do Image Processors Do for Living?
Compression: At 158:1 compression ratio...
1. Motivation 2 What Do Image Processors Do for Living?
Denoising (restoration/filtering)
Noisy image Clean image
1. Motivation 3 What Do Image Processors Do for Living?
Feature extraction (e.g. for content-based image retrieval)
Images Signatures
Feature extraction
Similarity Measurement
Query Feature extraction
N best matched images
1. Motivation 4 Fundamental Question: Efficient Representation of Visual Information
A random image A natural image
Natural images live in a very tiny part of the huge “image space” (e.g. R512×512×3)
1. Motivation 5 Mathematical Foundation: Sparse Representations Fourier, Wavelets... = construction of bases for signal expansions:
f = cnψn, where cn = hf, ψni. n X
Non-linear approximation: fˆM = cnψn, where IM = indexes of M most significant components. ∈ nXIM
Sparse (efficient) representation: f ∈ F can be (nonlinearly) ˆ 2 −α approximated with few components; e.g. kf − fM k2 ∼ M .
1. Motivation 6 Wavelets and Filter Banks
analysis synthesis y1 x1 Mallat H1 2 2 G1 x + xˆ
H0 2 2 G0 y0 x0
2 G1 Daubechies 2 G1 +
+ 2 G0
2 G0
1. Motivation 7 The Success of Wavelets
• Wavelets provide a sparse representation for piecewise smooth signals.
• Multiresolution, tree structures, fast transforms and algorithms, etc.
• Unifying theory ⇒ fruitful interaction between different fields.
1. Motivation 8 Fourier vs. Wavelets
Non-linear approximation: N = 1024 data samples; keep M = 128 coefficients
Original
40
20
0
−20 0 100 200 300 400 500 600 700 800 900 1000 Using Fourier (of size 1024): SNR = 22.03 dB
40
20
0
−20 0 100 200 300 400 500 600 700 800 900 1000 Using wavelets (Daubechies−4): SNR = 37.67 dB
40
20
0
−20 0 100 200 300 400 500 600 700 800 900 1000
1. Motivation 9 Is This the End of the Story?
1. Motivation 10 Wavelets in 2-D
• In 1-D: Wavelets are well adapted to abrupt changes or singularities.
• In 2-D: Separable wavelets are well adapted to point-singularities (only). But, there are (mostly) line- and curve-singularities...
1. Motivation 11 The Failure of Wavelets
Wavelets fail to capture the geometrical regularity in images.
1. Motivation 12 Edges vs. Contours
Wavelets cannot “see” the difference between the following two images:
• Edges: image points with discontinuity
• Contours: edges with localized and regular direction [Zucker et al.]
1. Motivation 13 Goal: Efficient Representation for Typical Images with Smooth Contours
smooth smooth
regions boundary
Key: Exploring the intrinsic geometrical structure in natural images.
⇒ Action is at the edges!
1. Motivation 14 And What The Nature Tells Us...
• Human visual system: – Extremely efficient: 107 bits −→ 20-40 bits (per second). – Receptive fields are characterized as localized, multiscale and oriented.
• Sparse components of natural images (Olshausen and Field, 1996):
Search for Sparse Code
16 x 16 patches from natural images
1. Motivation 15 Wavelet vs. New Scheme
Wavelets Newscheme For images:
• Wavelet scheme... see edges but not smooth contours.
• New scheme... requires challenging non-separable constructions.
1. Motivation 16 Recent Breakthrough from Harmonic Analysis: Curvelets [Cand`es and Donoho, 1999]
• Optimal representation for functions in R2 with curved singularities.
2 2 ˆ 2 3 −2 For f ∈ C /C : kf − fM k2 ∼ (log M) M
• Key idea: parabolic scaling relation for C2 curves: u = u(v)
w v u
width ∝ length2
l
1. Motivation 17 “Wish List” for New Image Representations
• Multiresolution ... successive refinement
• Localization ... both space and frequency
• Critical sampling ... correct joint sampling
• Directionality ... more directions
• Anisotropy ... more shapes
Our emphasis is on discrete framework that leads to algorithmic implementations.
1. Motivation 18 Outline
1. Motivation
2. Discrete-domain construction using filter banks
3. Contourlets and directional multiresolution analysis
4. Contourlet approximation and directional vanishing moments
5. Applications
2. Discrete-domain construction using filter banks 19 Challenge: Being Digital!
Pixelization:
Digital directions:
2. Discrete-domain construction using filter banks 20 Proposed Computational Framework: Contourlets
In a nutshell: contourlet transform is an efficient directional multiresolution expansion that is digital friendly!
contourlets = multiscale, local and directional contour segments
• Contourlets are constructed via filter banks and can be viewed as an extension of wavelets with directionality ⇒ Inherit the rich wavelet theory and algorithms.
• Starts with a discrete-domain construction that is amenable to efficient algorithms, and then investigates its convergence to a continuous-domain expansion.
• The expansion is defined on rectangular grids ⇒ Seamless transition between the continuous and discrete worlds.
2. Discrete-domain construction using filter banks 21 Discrete-Domain Construction using Filter Banks
Idea: Multiscale and Directional Decomposition
• Multiscale step: capture point discontinuities, followed by... • Directional step: link point discontinuities into linear structures.
2. Discrete-domain construction using filter banks 22 Analogy: Hough Transform in Computer Vision
Inputimage Edgeimage “Hough”image
=⇒ =⇒
Challenges:
• Perfect reconstruction.
• Fixed transform with low redundancy.
• Sparse representation for images with smooth contours.
2. Discrete-domain construction using filter banks 23 Multiscale Decomposition using Laplacian Pyramids
decomposition reconstruction
• Reason: avoid “frequency scrambling” due to (↓) of the HP channel.
• Laplacian pyramid as a frame operator → tight frame exists.
• New reconstruction: efficient filter bank for dual frame (pseudo-inverse).
2. Discrete-domain construction using filter banks 24 Directional Filter Banks (DFB)
• Feature: division of 2-D spectrum into fine slices using tree-structured filter banks. ω2 (π, π)
0 1 2 3 7 4
6 5 ω 5 6 1
4 7
3 2 1 0
(−π, −π)
• Background: Bamberger and Smith (’92) cleverly used quincunx FB’s, modulation and shearing.
• We propose: – a simplified DFB with fan FB’s and shearing – using DFB to construct directional bases
2. Discrete-domain construction using filter banks 25 Multidimensional Sampling
Example. Downsampling by M: xd[n]= x[Mn]
↓ R3 ↓ Q1
1 0 1 1 R = Q = = R D R 3 −1 1 1 −1 1 0 0 3
2. Discrete-domain construction using filter banks 26 Simplified DFB: Two Building Blocks
• Frequency splitting by the quincunx filter banks (Vetterli’84).
y0 Q Q
x + xˆ y1 Q Q
• Shearing by resampling
2. Discrete-domain construction using filter banks 27 Multichannel View of the Directional Filter Bank H0 G0 y0 S0 S0
H1 G1 y1 S1 S1 x + xˆ
H2l−1 G2l−1 y2l−1 S2l−1 S2l−1
Use two separable sampling matrices:
2l−1 0 0 ≤ k < 2l−1 (“near horizontal” direction) " 0 2# S = k 2 0 l−1 l l−1 2 ≤ k < 2 (“near vertical” direction) "0 2 #
2. Discrete-domain construction using filter banks 28 General Bases from the DFB
An l-levels DFB creates a local directional basis of l2(Z2):
(l) (l) gk [·− Sk n] 0≤k<2l, n∈Z2 n o (l) • Gk are directional filters:
• Sampling lattices (spatial tiling): 2 2l−1
2 2l−1
2. Discrete-domain construction using filter banks 29 Discrete Contourlet Transform
Combination of the Laplacian pyramid (multiscale) and directional filter banks (multidirection) ω2 (π, π)
bandpass (2,2) directional subbands ω1
bandpass image directional subbands (−π, −π)
Properties: + Flexible multiscale and directional representation for images (can have different number of directions at each scale!) + Tight frame with small redundancy (< 33%) + Computational complexity: O(N) for N pixels thanks to the iterated filter bank structures
2. Discrete-domain construction using filter banks 30 Wavelets vs. Contourlets
50 50
100 100
150 150
200 200
250 250 50 100 150 200 250 50 100 150 200 250
2. Discrete-domain construction using filter banks 31 Examples of Discrete Contourlet Transform
2. Discrete-domain construction using filter banks 32 Critically Sampled (CRISP) Contourlet Transform [LuD:03]
D M D ω2 (π, π) D 45 6 7 8 9
12 3 D
ω1 0 A
3 2 1
9 87 6 5 4 B (−π, −π)
• Directional bandpass: 3 × 2n (n = 1, 2,...) directions at each level.
• Refinement of directionality is achieved via iteration of two filter banks.
2. Discrete-domain construction using filter banks 33 Contourlet Packets
• Adaptive scheme to select the “best” tree for directional decomposition.
ω2 (π, π)
ω1
(−π, −π)
• Contourlet packets: obtained by altering the depth of the DFB decomposition tree at different scales and orientations – Allow different angular resolution at different scale and direction – Rich set of contourlets with variety of support sizes and aspect ratios – Include wavelet(-like) transform
2. Discrete-domain construction using filter banks 34 Nonsubsampled Contourlet Transform [Zhou, Cunha, ...] H0(z) G0(z) H0(z) G0(z)
H1(z) G1(z) H1(z) G1(z) Less stringent filter bank condition → design better filters.
H0(z^D) H0 H1(z^D) X H0(z^D) H1 H1(z^D)
Key: “`a trous” algorithm = efficient filtering “with holes” using the equivalent sampling matrices
2. Discrete-domain construction using filter banks 35 Outline
1. Motivation
2. Discrete-domain construction using filter banks
3. Contourlets and directional multiresolution analysis
4. Contourlet approximation and directional vanishing moments
5. Applications
3. Contourlets and directional multiresolution analysis 36 Multiresolution Analysis w2
Wj
Vj−1 = Vj ⊕ Wj, w1 2 2 L (R )= Wj. j∈Z Vj M
Vj−1
Vj has an orthogonal basis {φj,n}n∈Z2, where
−j −j φj,n(t) = 2 φ(2 t − n).
Wj has a tight frame {µj−1,n}n∈Z2 where
(i) n k µj−1,2 + i = ψj,n, i = 0,..., 3.
3. Contourlets and directional multiresolution analysis 37 Directional Multiresolution Analysis
ω2 2j+lj−2 (l ) (l ) j j ρ n W j,k, j,k 2j ω1 Vj Wj l 2 j −1 (lj) Vj−1 Wj = Lk=0 Wj,k
(lj) (l) Wj,k has a tight frame ρj,k,n where n∈Z2 n o (l) (l) (l) (l) j−1 (l) ρj,k,n(t)= gk [m − Sk n] µj−1,m(t) = ρj,k(t − 2 Sk n). m Z2 X∈ DFB basis LP frame | {z } | {z } Theorem (Contourlet Frames) [DoV:03]. (lj) 2 R2 ρj,k,n l is a tight frame of L ( ) for finite lj. j∈Z, 0≤k<2 j , n∈Z2 n o
3. Contourlets and directional multiresolution analysis 38 Sampling Grids of Contourlets
w l
w l
(a) (b) w/4 l/2
w/4 l/2
(c) (d)
3. Contourlets and directional multiresolution analysis 39 Connection between Continuous and Discrete Domains
2 2 Theorem [DoV:04] Suppose x[n]= hf,φL,ni, for an f ∈ L (R ), and
discrete contourlet transform (lj) x −→ (a , d ) l J j,k j=1,...,J; k=0,...,2 j −1
Then (lj) (lj) aJ[n]= hf,φL+J,ni and dj,k [n]= hf, ρL+j,k,ni
Digital images are obtained by: x˜[n]=(f∗ϕ¯)(T n)= hf,ϕ(·−T n)i
For T = 2L, with prefiltering we can get x[n] from x˜[n]
3. Contourlets and directional multiresolution analysis 40 Contourlet Features
• Defined via iterated filter banks ⇒ fast algorithms, tree structures, etc.
• Defined on rectangular grids ⇒ seamless translation between continuous and discrete worlds.
• Different contourlet kernel functions (ρj,k) for different directions.
• These functions are defined iteratively via filter banks.
• With FIR filters ⇒ compactly supported contourlet functions.
3. Contourlets and directional multiresolution analysis 41 Outline
1. Motivation
2. Discrete-domain construction using filter banks
3. Contourlets and directional multiresolution analysis
4. Contourlet approximation and directional vanishing moments
5. Applications
4. Contourlet approximation and directional vanishing moments 42 Contourlets with Parabolic Scaling
l j j lj+j Support size of the contourlet function ρj,k: width ≈ 2 and length ≈ 2 To satisfy the parabolic scaling: width ∝ length2, simply set: the number of directions is doubled at every other finer scale.
u = u(v)
ω2 (π, π) w v u
ω1
l
(−π, −π)
4. Contourlet approximation and directional vanishing moments 43 Supports of Contourlet Functions
LP DFB Contourlet
* =
* =
Key point: Each generation doubles spatial resolution as well as angular resolution.
4. Contourlet approximation and directional vanishing moments 44 Contourlet Approximation
ΠL ΠL
see a wavelet
Desire: Fast decay as contourlets turn away from the discontinuity direction Key: Directional vanishing moments (DVMs)
4. Contourlet approximation and directional vanishing moments 45 Geometrical Intuition
θj,k,n At scale 2j (j ≪ 0): d n 2j/2 j,k, j width ≈ 2 j/2 2 length ≈ 2 dj,k,n #directions ≈ 2−j/2
2j
−3j/4 3 |hf, ρj,k,ni| ∼ 2 · dj,k,n j j/2 −1 −j/2 dj,k,n ∼ 2 / sin θj,k,n ∼ 2 k˜ for k˜ = 1,..., 2 3j/4˜−3 =⇒ |hf, ρj,k,˜ ni| ∼ 2 k
4. Contourlet approximation and directional vanishing moments 46 How Many DVMs Are Sufficient? ω2
d2 ω1 α d
Sufficient if the gap to a direction with DVM:
α . d ∼ 2j/2k˜−1 for k˜ = 1,..., 2−j/2
This condition can be replaced with fast decay in frequency across directions. It is still an open question if there is an FIR filter bank that satisfies the sufficient DVM condition
4. Contourlet approximation and directional vanishing moments 47 Examples of Frequency Responses
8 directions: along the line ω2 = π/2
16 directions: along the line ω2 = π/2
Left: PKVA filters (size 41 × 41). Middle: New filters (size 11 × 11). Right: CD filters (size 9 × 9)
4. Contourlet approximation and directional vanishing moments 48 Experiments with Decay Across Directions using Near Ideal Frequency Filters
slope = −3 slope = −3
4. Contourlet approximation and directional vanishing moments 49 Nonlinear Approximation Rates
2 2 C boundary C regions
Under the (ideal) sufficient DVM condition
3j/4˜−3 |hf, ρj,k,˜ ni| ∼ 2 k
−j/2˜ with number of coefficients Nj,k˜ ∼ 2 k. Then
ˆ(contourlet) 2 3 −2 kf − fM k ∼ (log M) M
ˆ(Fourier) 2 −1/2 ˆ(wavelet) 2 −1 Note: kf − fM k ∼ O(M ) and kf − fM k ∼ O(M )
4. Contourlet approximation and directional vanishing moments 50 Non-linear Approximation Experiments
Image size = 512 × 512. Keep M = 4096 coefficients.
Original image Wavelets: Contourlets: PSNR = 24.34 dB PSNR = 25.70 dB
4. Contourlet approximation and directional vanishing moments 51 Detailed Non-linear Approximations
Wavelets M = 4 M = 16 M = 64 M = 256
Contourlets M = 4 M = 16 M = 64 M = 256
4. Contourlet approximation and directional vanishing moments 52 Filter Bank Design Problem
(l) (l) ρj,k(t)= ck [m]φj−1,m(t) m Z2 X∈ (l) ρj,k(t) has an L-order DVM along direction (u1,u2) (l) u2 −u1 L ⇔ Ck (z1, z2)=(1 − z1 z2 ) R(z1, z2) ω2