Discrete Fourier and Transforms: Mathematical Microscopes for Signal Processing

Roe Goodman

Rutgers Math Club

October 15, 2014

Roe Goodman Discrete Fourier and Wavelet Transforms Continuous (calculus) to Discrete (linear algebra) analog signal f (t)(a ≤ t ≤ b) −→ digital signal v T v = Q[f (t0), f (t1),..., f (tN−1)] Q = quantize entries (# bits) tj = a + j∆t with ∆t = (b − a)/N (sample at rate N)

Sine Wave and 2−bit Quantization Sine Wave and 4−bit Quantization 1 1

0.8 0.8

0.6 0.6

0.4 0.4

0.2 0.2

0 0

−0.2 −0.2

−0.4 −0.4

−0.6 −0.6

−0.8 −0.8

−1 −1 0 10 20 30 40 50 60 70 80 90 100 0 10 20 30 40 50 60 70 80 90 100  ç more bits gives better sound but needs more memory space Linear algebra model: ∼ N digital signal ←→ vector in complex vector space = C Roe Goodman Discrete Fourier and Wavelet Transforms

Music engraving by LilyPond 2.14.2—www.lilypond.org Periodic Discrete Signals and Transforms

2 Fix integer N ≥ 2 ` [Z/NZ] = vector space of N-periodic signals φ : Z → C with φ(k + N) = φ(k) for all k ∈ Z PN−1 inner product hφ, ψi = k=0 φ(k)ψ(k)(z = complex conj.) 2 N T ` [Z/NZ] ←→ C by φ ←→ x = [φ(0), ··· , φ(N − 1)] Transform Method N Choose invertible matrix A = [v0, ··· , vN−1](basis for C ) ∗ ∗ −1 Let v0 ,..., vN−1 be rows of A (dual basis) −1 T ∗ A-transform X = A x = [c0, ··· , cN−1] , entries ck = vk x Inversion formula( ?) x = AX = c0v0 + ··· + cN−1vN−1 −1 T ∗ T Example A = A (Orthonormal basis) ⇐⇒ vk = v k 2 2 ⇐⇒ energy preserved: (x, x) = (X , X ) = |c0| + ··· + |cN−1| Transform Problem Find the best transform for expanding a particular type of signal x • Large percentage of the coefficients ck in( ?) are small • Replace small ck by 0 to get compressed signalx ˜ with small percentage of nonzero coefficients (good approximation to x)

Roe Goodman Discrete Fourier and Wavelet Transforms Shift Operator and Sampled Waves

2 N Shift operator For φ ∈ ` [Z/NZ] Sφ(k) = φ(k − 1) (S = I ) Problem Find eigenvectors√ for S Let ω = e2πi/N (i = −1) Then ωN = 1 and 1, ω, ω2, . . . , ωN−1 are the solutions to zN = 1 (roots of unity)

N = 8: ↑ Imaginary axis ω2 = i ...... t...... 2 2 3 ...... 2πi/8 π π ω ...... ω .....t ..... t..... = e = cos 8 + i sin 8 ...... 2 ...... π .. 4 ...... 8 ...... 8 . ω = −1 ...... ω ...... =1 .t ...... t ...... −→ Real axis ...... 5 t...... t 7 ...... ω ...... ω ...... t...... ω6 = −i kp 2 Set φp(k) = ω for k, p = 0,..., N − 1 Then φp ∈ ` [Z/NZ] φp = discrete sample of continuous wave fp (frequency p): 2πpti fp(t) = e = cos(2πpt) + i sin(2πpt)(t = k/N) ( low freq. wave when p ≈ 0 (mod N) ωp ≈ 1 φp ←→ high freq. wave when p ≈ N/2 (mod N) ωp ≈ −1

Roe Goodman Discrete Fourier and Wavelet Transforms Discrete

2 Fourier Basis {φ0, φ1, ... , φN−1} for ` [Z/NZ] −p −p • Eigenvectors for S: Sφp = ω φp (eigenvalue = ω ) ( N if p ≡ q mod N • Orthogonality: hφp, φqi = 0 else 2 Discrete Fourier transform(DFT) of φ ∈ ` [Z/NZ] PN−1 −kp 2 φb(p) = hφp, φi = k=0 ω φ(k) for φ ∈ ` [Z/NZ]  Fourier synthesis: φ = (1/N) φb(0) φ0 + ··· + φb(N − 1) φN−1 Matrix Description (calculated by FFT “Fast Fourier Transform”) N N φp ←→ Ep ∈ C (p = 0,..., N − 1) Fourier basis for C −(j−1)(k−1) Fourier matrix FN = [E0, ··· , EN−1](j, k entry ω )  1 1 1 1   1 −i −1 i  N = 4: F4 =    1 −1 1 −1  1 i −1 −i N If φ ←→ y ∈ C then φb ←→ Y = FN y and y = (1/N)F N Y

Roe Goodman Discrete Fourier and Wavelet Transforms Frequency Analysis

Powergraph of Train Whistle Train Whistle 1 peak at 900 890 Hz peak at 0.8 1170 Hz 800 0.6 700 0.4 peak frequency ratios approx. 600 0.2 3:5 peak at 4:5 710 Hz 3:4 0 500

−0.2 400

−0.4 300

−0.6 200

−0.8 100 −1 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 0 time (sec.) 600 800 1000 1200 1400 1600 Frequency (Hz) frequency peaks near 710Hz, 890Hz, 1170Hz (N = 12, 880) analog signal to synthesize peak frequencies f (t) = 0.45 sin(2π∗710t)+0.95 sin(2π∗890t)+0.93 sin(2π∗1170t)

What is missing from this model?   ç   ç    (quarter note, rest, half note) d minor chord (whole note) Need time& frequency information!

Roe Goodman Discrete Fourier and Wavelet Transforms

Music engraving by LilyPond 2.14.2—www.lilypond.org Music engraving by LilyPond 2.14.2—www.lilypond.org Wavelet Approach   sk (0)  sk (1)  Given: Digital signal s =   length N = 2k (level k) k  .   .  sk (N −1) Transform (pyramid algorithm)   (sk )even • Downsample sk −→ with (sk )odd     sk (0) sk (1)  sk (2)   sk (3)  (s ) =  ,(s ) =   (length N/2) k even  .  k odd  .   .   .  sk (N −2) sk (N −1) 1  • Trend (level k − 1) sk−1 = 2 (sk )even + (sk )odd 1  • Detail (level k − 1) dk−1 = 2 (sk )even − (sk )odd • Iterate on trend sk−1 −→ [sk−2, dk−2], sk−2 −→ · · ·

Roe Goodman Discrete Fourier and Wavelet Transforms Multiresolution Analysis

level−three signal s level−two trend s level−two detail d Example 3 2 2 60 50 15

50 45 10 40 40 35 5 30 30 0 20 25 10 −5 20

0 15 −10 0 1 2 3 4 5 6 7 0 1 2 3 0 1 2 3

s [0] Three-scale analysis pyramid: 0

d [0] level 0 0 s [0] 1 s [1] 1

d [0] level 1 1 d [1] 1 s [0] s [1] 2 2 s [2] s [3] 2 2

d [2] d [3] d [0] d [1] 2 2 level 2 2 2 s [4] s [7] s [0] s [1] s [2] s [3] 3 s [5] s [6] 3 3 3 3 3 3 3 Three-scale multiresolution analysis s3 −→ [s0, d0, d1, d2]

Roe Goodman Discrete Fourier and Wavelet Transforms Multiresolution Synthesis and Haar Basis   s0 (3) d0 Three-scale multiresolution synthesis s3 = W   d1 Haar three-scale synthesis matrix (8 × 8): d2 (3) 4 2 4 6 W = [u0, v0, v1, S v1, v2, S v2, S v2 S v2](S = shift) 1  1   1   1  1  1   1  −1         1  1  −1  0          1  1  −1  0  u0 =   v0 =   v1 =   v2 =   1 −1  0   0          1 −1  0   0          1 −1  0   0  1 −1 0 0 trend coarse detail medium detail fine detail Differences between detail vectors for Haar basis and Fourier basis: • localized in time (multiple time scales) • time shifted and rescaled samples of one function (wavelet)

Roe Goodman Discrete Fourier and Wavelet Transforms Mathematical Microscope

Wave + Step Function + Noise One-scale Haar transform

1 1

0.5 0.5

0 0

−0.5 −0.5 trend s detail d 9 9

−1 −1

256 512 768 1,024 256 512 768 1,024 Two−scale Haar transform Three−scale Haar transform

1 1

0.5 0.5

0 0 −0.5 trend s detail d −0.5 trend s detail d 8 8 7 7 −1 −1 128 256 384 512 64 128 192 256 10 Pyramid algorithm: x −→ [s7, d7, d8, d9](N = 2 ) There are many other mathematical microscopes!

Roe Goodman Discrete Fourier and Wavelet Transforms Feature Extraction and Signal Compression

Three-scale LeGall wavelet transform (used in JPEG 2000 lossless algorithm) 9 signal −→ [s6, d6, d7, d8](N = 2 = 512)

Signal with Pops and Static Three−Scale Wavelet Coefficients 1.5 5

s6 0

1 −5 10 20 30 40 50 60 2

0.5 d6 0

−2 0 10 20 30 40 50 60 1

d7 0 −0.5

−1 20 40 60 80 100 120 2 −1 d8 0

−1.5 −2 0 50 100 150 200 250 300 350 400 450 500 550 50 100 150 200 250 • Trend s6 shows the low frequency signal content • Time location of pops is clear at each level of detail • Most of the detail coefficients are small (noise)

Roe Goodman Discrete Fourier and Wavelet Transforms Multiresolution Decomposition

Three level wavelet synthesis of signal: Let Dj = inverse wavelet transform of details dj (j = 8, 7, 6) S6 = inverse wavelet transform of trend s6 512 Then original signal = D8 + D7 + D6 + S6 (each term in C )

2

1 D8 0

−1 0 100 200 300 400 500 600 2

1 D7 0

−1 0 100 200 300 400 500 600 0.2

D6 0

−0.2 0 100 200 300 400 500 600 2 S 6 0

−2 0 100 200 300 400 500 600

Roe Goodman Discrete Fourier and Wavelet Transforms Wavelet Filtering and Compression

Filtering by RemovingFiltering Waveletby Removing Details Wavelet and Noise Details and Noise 2 2

1 1

0 0

−1 −1 original signal original signal −2 −2 0 0100 100200 200300 300400 400500 500600 600 Threshold Compression2 2 1 1 • 97% of detail coefficients in d6, d7 and d8 are less than 0.2. 0 0 • Replace each such small coefficient by 0 to get d˜6, d˜7, d˜8 −1 −1 filtered signal filtered signal • Calculate inverse−2 wavelet−2 transforms De6, De7, De8 and add to S6 0 0100 100200 200300 300400 400500 500600 600

2 2

1 1 S6 + De6 + De7 + De8: 0 0

−1 −1 filtered signal withfiltered pops signal with pops −2 −2 0 0100 100200 200300 300400 400500 500600 600 Noise filtered out but pops still in (6 : 1 compression of signal) (Not possible using Fourier basis)

Roe Goodman Discrete Fourier and Wavelet Transforms Wavelet Analysis of Images

W = one-scale wavelet analysis matrix X = image matrix WXW T = wavelet transform (256 × 256 eight-bit matrix) (partitioned matrix)

Original Lena Image One-scale Wavelet Transform trend vertical 128×128 details horizontal diagonal details details

Roe Goodman Discrete Fourier and Wavelet Transforms Multiscale Image Transforms and Edge Detection

Pyramid algorithm for two-scale transform of image matrix • Keep the 3 one-scale detail matrices • Make a wavelet transform of the trend matrix Fig. 11: Sum of Details of Two−Scale Multiresolution Representation

50

100

150

200

250 50 100 150 200 250 Two-scale wavelet transform Inverse transform of details (omit level-two trend) Famous Cartoon (mathematician to engineer after seeing machine) “It works in practice, but does it work in theory?” Problem How do we build wavelet transforms? Roe Goodman Discrete Fourier and Wavelet Transforms Linear Time-Invariant Filters

N−1 Given: signal φ and polynomial f (z) = a0 + a1z + ··· + aN−1z Form moving average of the signal (as for Haar trend and detail)

a0φ(k) + a1φ(k − 1) + ··· + aN−1φ(k − N + 1) = (f (S)φ)(k) N−1 where S = shift operator and f (S) = a0 + a1S + ··· + aN−1S 2 Definition The linear transformation f (S) on ` [Z/NZ] is called a filter, and f (S)φ is the filtered signal −p 2πi/N Example On Fourier basis Sφp = ω φp (ω = e ) Hence −p −p(N−1) −p f (S)φp = (a0 + a1ω + ··· + aN−1ω )φp = f (ω )φp −p • φp is an eigenvector for f (S) with eigenvalue λ = f (ω ) PN−1 PN−1 −p • If φ = p=0 cpφp then f (S)φ = p=0 f (ω )cpφp • Low pass filter: f (S) attenuates high frequencies (p ≈ N/2) ⇐⇒ f (−1) = 0 ⇐⇒ f (z) = (z + 1)mg(z)(m ≥ 1) • High pass filter: f (S) attenuates low frequencies (p ≈ 0) ⇐⇒ f (1) = 0 ⇐⇒ f (z) = (z − 1)mg(z)(m ≥ 1)

Roe Goodman Discrete Fourier and Wavelet Transforms Filter Banks and Discrete Wavelet Transforms

2 Construct a wavelet analysis transform of φ ∈ ` [Z/NZ](N even) using lowpass and highpass filters followed by downsampling

• Choose a polynomial h0(z) with h0(−1) = 0 to get a lowpass filter H0 = h0(S) and calculate H0φ (length N) • Choose a polynomial h1(z) with h1(1) = 0 to get a highpass filter H1 = h1(S) and calculate H1φ (length N) 2 • Introduce downsampling operator D on ψ ∈ ` [Z/NZ]: N (Dψ)(k) = ψ(2k)(Dψ period N/2), ψ ←→ v ∈ C 2 D 2 N/2 ` [Z/NZ] −→ ` [Z/(N/2)Z], Dψ ←→ veven ∈ C • Calculate trend= DH0φ and detail= DH1φ (lengths = N/2) trend(φ) • Define a wavelet transform W φ = detail(φ) Important: length(φ) = length(W φ), so W ←→ N × N matrix Perfect Reconstruction (PR) Problem Is W invertible? Energy Preservation Problem Does W preserve the energy of a signal? (Important for signal compression)

Roe Goodman Discrete Fourier and Wavelet Transforms PR Low Pass & High Pass Filter Pairs

Algebraic condition for PR

(??) h0(z)h1(−z) − h0(−z)h1(z) is a nonzero monomial Examples • Haar transform h0(z) = 1 + z and h1(z) = 1 − z 2 2 Here h0(z)h1(−z) − h0(−z)h1(z) = (1 + z) − (1 − z) = 4z • LeGall transform (= CDF(2,2) transform) 2 2 2 h0(z) = (1 + z) (1 − 4z + z ) and h1(z) = (1 − z) • CDF(p,q) transforms (Ingrid Daubechies & collaborators, 1992) Given positive integers p, q with p + q even, there exits g(z) so q p h0(z) = (1 + z) g(z) and h1(z) = (1 − z) satisfy( ??) (explicit formula for g(z) with binomial coefficients) • CDF(9,7) filters used in JPEG 2000 image compression algorithm In these examples only the Haar transform preserves energy

Roe Goodman Discrete Fourier and Wavelet Transforms Energy Preserving Wavelet Transforms

Theorem Suppose the polynomial h0(z) satisfies h0(−1) = 0 and −1 −1 (]) h0(z)h0(z ) + h0(−z)h0(−z ) = 2 −1 Define h1(z) = zh0(−z ) and let H0 = h0(S),H1 = h1(S). Then H0,H1 are the low pass & high pass filters for an energy-preserving PR wavelet transform. Remark Equation( ]) ⇐⇒ system of quadratic equations for the coefficients of h0(z) Examples √ • Haar transform (normalized) Take h0(z) = (1 + z)/ 2 Then (])= 1 {(1 + z)(1 + z−1) + (1 − z)(1 − z−1)} = 2 2 √ 2 3 • Daub4 transform√ Take h0(√z) = (a + bz √+ cz + dz )/√ 32 where a = 1 + 3, b = 3 + 3, c = 3 − 3, d = 1 − 3 Then h(−1) = a − b + c − d = 0 and( ]) is satisfied • Daub2k transform (any positive integer k) Find complex roots of polynomial used in CDF(k,k) transform (No explicit formula)

Roe Goodman Discrete Fourier and Wavelet Transforms