A Friendly Guide to the Frame Theory and Its Application to Signal Processing

Home , Frame (linear algebra)

Minh N. Do

Department of Electrical and Computer Engineering University of Illinois at Urbana-Champaign

www.ifp.uiuc.edu/∼minhdo [email protected] A Basic Problem

Consider the following linear inverse problem:

Ax = b PSfrag replacements where A ∈ Rm×n is ﬁxed, b ∈ Rm is given, and x ∈ Rn is unknown.

x b bˆ xˆ A + S

∆b

Examples: Deconvolution, computerized tomography, transform coding,... Possible noise due to: model mismatch, measurement and/or transmission error, quantization, thresholding,...

1 Two Basic Questions

PSfrag replacements Ax = b

x b bˆ xˆ A + S

∆b

We have two questions:

1. Can we reconstruct x in a numerically stable way from b?

2. Which is the “optimal” reconstruction algorithm in the presence of noise?

2 Linear Inverse Problem Ax = b: First Question

Question: Can we reconstruct x in a numerically stable way from b?

Answer: It depends on the condition number of A:

σ (A) κ(A) = 1 . σn(A)

The smaller κ(A) (κ(A) ≥ 1), the more stable (or well-conditioned) the problem is.

Intuition: σ1 and σn are the largest and smallest singular values of A. Thus, for all x: σnkxk2 ≤ kAxk2 ≤ σ1kxk2

That means A should behavior modestly with respect to 2-norm!

3 Linear Inverse Problem Ax = b: Second Question

Question: When the system A is overcomplete, there are many (inﬁnite) ways to reconstruct x from b. Which one is “optimal”?

Answer: Use the pseudo-inverse A† = (AT A)−1AT

xˆ = A†b

Special properties of the pseudo-inverse:

• A† provides the least-squares solution ⇒ Eliminates the inﬂuence of errors orthogonal to the range of A.

• A† has a minimum spectral norm among all left inverse of A ⇒ Recovers x and but doesn’t blow up the noise.

4 Frames: Generalize to Hilbert Spaces

Consider A as a linear operator, A : Rn → Rm, then

Ax = b ⇐⇒ bi = hx, aii, i = 1, 2, . . . , m

T where ai are the rows of A.

Def: A sequence {φk}k∈Γ in a Hilbert space H is a frame if there exist two constants (frame bounds) α > 0 and β < ∞ such that for any x ∈ H

2 2 2 αkxk ≤ |hx, φki| ≤ βkxk . kX∈Γ

Best case: α = β =⇒ tight frame

Signiﬁcance: {φk}k∈Γ is a frame ⇐⇒ one can recover x ∈ H from {hx, φki}k∈Γ.

5 Dual Frame

Frame operator: A : H → l2(Γ)

(Ax)k = hx, φki, for k ∈ Γ.

† ∗ −1 ∗ Pseudo inverse: A = (A A) A exists and bounded because {φk}k∈Γ is a frame.

Result: Reconstruction using pseudo inverse is related to a dual frame

† x = A Ax = hx, φkiφ˜k kX∈Γ

∗ −1 where the dual frame is deﬁned as φ˜k = (A A) φk.

−1 Easiest case: Tight frame (α = β), φ˜k = α φk.

6 Iterative Frame Reconstruction Algorithm

Both pseudo-inverse and dual frame computations need the inversion of A∗A, where ∗ A Ax = hx, φkiφk kX∈Γ

2 ∗ ∗ Consider R = I − α+β A A, then because α · I ≤ A A ≤ β · I β − α kRk ≤ ≤ 1. β + α

Thus, 2 2 ∞ (A∗A)−1 = (I − R)−1 = Ri α + β α + β Xi=0 Iterative reconstruction: 2 xn = xn + (hx, φki − hxn , φki)φk −1 α + β −1 kX∈Γ

7 Application to Generalized Sampling

• Sampled data: s[k] = hx, φki, where φk is the point spreading function (PSF) of the sensoring device at location tk.

• “Sampling theorem”: Function x(t) ∈ H can be recovered in a

numerically stable way from samples s[k] if and only if {φk}k∈Γ is a frame of H.

• Classical sampling: H = BL([−π, π]) and {φk} = {sinc(t − k)}k∈Z

8 Laplacian Pyramid: Burt Adelson, 1983

9 Why Laplacian Pyramid Instead of Orthogonal Filter Banks?

(2,2) (2,2) Wavelet FB Laplacian Pyramid

(2,2)

Even in higher dimensions, the Laplacian pyramid (LP) only generates one isometric detailed signal at each level.

highpass (HP)

LP has no “frequency scrambling” −π π due to downsamplingPSfragof thereplacementshighpass channel: downsampled HP −π π

10 PSfrag replacements Decomposition in the Laplacian Pyramid

x p d H M M G _ +

H G

Coarse: c = Hx Residual: d = x − GHx = (I − GH)x.

Combining gives c H = x. d I − GH y A | {z } | {z } 11 PSfragUsualreplacementsReconstruction in the Laplacian Pyramid

c xˆ M G +

c xˆ = G I . d S1 y | {z } | {z } Note that S1A = I (perfect reconstruction) for any H and G. But... what about noisy pyramids: yˆ = y + e ?

The most serious disadvantage of the LP for coding applications [Simoncelli & Adelson, 1991]: “...the errors from highpass subbands of a multilevel LP do not remain in these subbands but appear as broadband noise in the reconstructed signal...”

12 Frame Analysis

• LP is a frame operator (A) with redundancy.

• It admits an inﬁnite number of left inverses.

• Let S be an arbitrary left inverse of A,

xˆ = Syˆ = S(y + e) = x + Se.

• The optimal left inverse (minimizing kSk) is the pseudo-inverse of A:

A† = (AT A)−1AT .

• If the noise is white, then among all left inverses, the pseudo-inverse minimizes the reconstruction MSE.

• But... reconstruction using the pseudo-inverse might be computationally expensive, unless we have a tight frame.

13 A Tight Frame Case

Orthogonal ﬁlters:

hg[•], g[• − Mn]i = δ[n], and h[n] = g[−n], or H = GT .

Theorem. The Laplacian pyramid with orthogonal ﬁlters is a tight frame.

Proof: Under the orthogonality condition:

p[n] = hx[•], g[• − MkPSfrag]i g[n −replacementsMk]. x kX∈Zd c[k] Using the Pythago|rean theo{z rem: } d V kxk2 = kpk2 + kdk2 = kck2 + kdk2. p

14 Inspiring New Reconstruction Filter Bank

As a result, pseudo-inverse of A is simply its transpose

T H A† = AT = = G I − GGT . I − GGT

So the optimal reconstruction is

PSfrag replacements xˆ = A†y = Gc + (I − GGT )d = G(c − Hd) + d.

x d xˆ p H M _ + M G +

15 General Cases PSfrag replacements Consider the following ﬁlter bank for reconstruction c

x d xˆ p H M _ + M G +

Theorem. 1. It is an inverse of the LP if and only if H and G are biorthogonal ﬁlters, or GH is a projector.

2. It is the pseudo-inverse if and only if GH is an orthogonal projector.

Recall: A linear operator P is a projector if P 2 = P . Furthermore, if P = P T then P is an orthogonal projector.

16 PSfrag replacements PSfrag replacements

H ComparingHTwo Reconstruction Methods G G x xc

pc xˆ pd xˆ M G + H M _ + M G + d

Usual reconstruction: x1 = Gc + d PSfrag replacements New reconstruction: x2 = Gc + (I − GH) d

P˜W | {z }

W dˆ xˆ2 xˆ1 P˜W dˆ d x V p

17 Laplacian Pyramid as an Oversampled Filter Bank PSfrag replacements

c H M M G

d K 0 F x 0 M M 0 xˆ +

d|M|−1 K|M|−1 M M F|M|−1

[1] For the usual reconstruction method, synthesis ﬁlters Fi are all-pass [1] −ki (delay) ﬁlters: Fi (z) = z

[1] For the proposed reconstruction method, synthesis ﬁlters Fi are high-pass [2] −ki M ﬁlters: Fi (z) = z − G(z)Hi(z ).

18 Multilevel Laplacian Pyramids

xˆ PSfrag replacements 2 G + 2 G +

2 F 0 2 F 0

2 F 1 2 F 1

Comparing frequency responses of equivalent synthesis ﬁlters (REC-1 vs. REC-2)

3.5 3.5

3 3

2.5 2.5

2 2

1.5 1.5 Magnitude Magnitude

1 1

0.5 0.5

0 0 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 Normalized Frequency (×π rad/sample) Normalized Frequency (×π rad/sample)

19 PSfrag replacements Experimental Results

usual xˆ1 rec. x LP y yˆ + dec. new xˆ2 e rec.

With additive uniform white noise in [0, 0.1] (non-zero mean)...

usual new rec. rec. SNR = SNR = 17.42 dB 6.28 dB

20 Summary

• Frames are a powerful tool... – Generalizes matrix inversions for general (possible inﬁnite dimensional) vector spaces. – Generalizes bases for overcomplete (redundant) systems.

• Framing pyramids lead to... – New reconstruction algorithm with signiﬁcant improvement over the usual method. – Complete characterization of left inverses and the pseudo-inverse.

• Frames are everywhere... – Give me a linear operator with a bounded inverse, I’ll frame it! – If you have to deal with an overcomplete system, consider the frame theory!