The Discrete Fourier Transform: Hayden Borg The Discrete Fourier Transform: From Hilbert Spaces to the FFT Hayden Borg University of Puget Sound May 2021 Hilbert Spaces and Hand-waving The Discrete Fourier Hilbert spaces generalize Euclidean spaces. Transform: What ”defines” a Euclidean space: Hayden Borg Vector space with the dot product Calculus We can generalize the dot product as the inner product Inner Product Define the inner product to be a bilinear functional acting on two elements of a vector space h~x; ~yi which is: Conjugate symmetric: h~x; ~yi = h~y; ~xi Linear in its first argument: hax~1 + bx~2; ~yi = ahx~1; ~yi + bhx~2; ~yi Positive definite: h~x; ~xi > 0 for all ~x 6= ~0 and h~x; ~xi = 0 if and only if ~x = ~0 Hilbert Space and Hand-waving: Inner Product Spaces The Discrete Fourier Transform: A vector space V equipped with an inner product is an Inner Hayden Borg Product space. Norm We define the norm of a vector in an Inner Product space to be jj~vjj2 = h~v; ~vi. We have some familiar looking results: Theorem: Parallelogram Law 2 2 2 2 For v~1; v~2 2 V , jj~x + ~yjj + jj~x − ~yjj = 2jj~xjj + 2jj~yjj Theorem: Pythagoras For ~x; ~y 2 V such that h~x; ~yi = 0, jj~x + ~yjj2 = jj~xjj2 + jj~yjj2 Hilbert Spaces and Hand-waving: Metric Spaces The Discrete Fourier A set M with metric µ is a Metric space if for all Transform: m1:m2; m3 2 M: Hayden Borg Identity of Indiscernibles: µ(s1; s2) = 0 if and only if s1 = s2 Symmetric: µ(s1; s2) = µ(s2; s1) Triangle Inequality: µ(s1; s3) ≤ µ(s1; s2) + µ(s2; s3) µ(s1; s2) is the "distance" between s1 and s2. Completeness A Cauchy sequence in Metric space M is a sequence fmi g for i ≥ 1 such that for every > 0 there exists an N such that µ(ml ; mk ) < for l; k > N. A Metric space M is complete if every Cauchy sequence in M converges to a point in M Hilbert Spaces and Hand-waving The Discrete Fourier Transform: Hayden Borg Theorem Given an Inner Product space and v~1; v~2 2 V , the identity of indiscernibles, the triangle inequality, and symmetry hold for µ defined: µ(v~1; v~2) = jjv~2 − v~1jj. An Inner Product space which is also a complete Metric space is called a Hilbert space. We now have: Vector space with inner product Calculus Hilbert Spaces and Hand-waving: Central Result The Discrete Fourier Transform: Hayden Borg Let fy~i g be an orthonormal set in Hilbert space H and ~v 2 H P such that ~v = ak y~k . Then, ak = h~v; y~k i k Proof P P h~v; y~j i = h ak y~k ; y~j i = ak hy~k ; y~j i = ak k k Hilbert Spaces and Hand-waving The Discrete Fourier Transform: 2 Hayden Borg Example: R 2 p1 3 2 " p1 # B = f6 p1 7 ; 2 g is an orthonormal basis of 2. 4 2 5 − p1 R 0 2 " 1 # " 1 # " 1 # " 1 # 4 4 p p 4 p p = h ; 2 i 2 + h ; 2 i 2 8 8 p1 p1 8 − p1 − p1 2 2 2 2 " 1 # " 1 # 12 p 4 p 6 − 2 = p 2 − p 2 = p1 − p1 6 + 2 2 2 2 2 Hilbert Spaces and Hand-waving: L2([0; T ]) The Discrete The set of all complex valued functions with real input Fourier Transform: T R 2 Hayden Borg integrable on the interval [0; T ] such that jf (x)j dx < 1. 0 Inner Product T hf (x); g(x)i = R f (x)g(x)dx 0 1 i 2π kx Let's examine f p e T jk 2 g. T Z First, T T Z Z 2 1 i 2π kx 2 1 jp e T j dx = p dx T T 0 0 = 1 < 1 Hilbert Spaces and Hand-waving: L2([0; T ]) The Discrete Fourier Transform: Next, Hayden Borg Z 1 i 2π nx 1 i 2π mx 1 T i 2π nx i 2π mx hp e T ; p e T i = 0 e T e T dx T T T Z 1 T i 2π (n−m)x = 0 e T dx T ( 1 R T T 0 1dx for n = m = 1 2π(n−m) 0 i2π(n−m) (e − e ) for n 6= m ( 1 for n = m = 0 for n 6= m 1 i 2π kx So, f p e T jk 2 g is orthonormal. T Z L2([0; T ]) and Complex Fourier Series The Discrete Fourier Transform: 2 Hayden Borg We can rewrite any f 2 L ([0; T ]) using our central result: P i 2π kx f (x) = ak e T k2Z Where: i 2π kx ak = hf ; e T i T Z −i 2π kx = f (x)e T dx 0 This is the complex Fourier series. Complex Fourier Series: Example ( 1 The Discrete 1 for 0 ≤ x < Fourier Square wave period 1: f (x) = 2 Transform: 1 −1 for 2 ≤ x < 1 Hayden Borg First we find the ak : Z 1 −i2πkx ak = f (x)e dx 0 1 Z 2 Z 1 = e−i2πkx dx − e−i2πkx dx 1 0 2 −1 −1 = (e−iπk − 1) − (1 − e−iπk ) i2πk i2πk −1 = (e−ikπ − 1) inπ ( 0 if k is even = 2i − nπ if k is odd Fourier Series: Applications The Discrete Fourier Transform: Hayden Borg Wide variety of applications: Solving PDEs Probability theory and statistics NMR, IR, etc. spectroscopy X-ray crystallography MRI Image and signal processes Audio engineering Fourier Transform The Discrete Fourier Transform: Generalization to aperiodic functions Hayden Borg 2 Need to consider: L (R) Definition: Fourier Transform 1 f^(!) = R f (x)ei2π!x dx −∞ And 1 f (x) = R f^(!)ei2π!x d! −∞ i2π!x Orthonormal Basis: fe j! 2 Rg Discrete Fourier Transform The Discrete Fourier Transform: Hayden Borg Digital signal and discretized data are common. Consider a function f sampled uniformly N times over an interval [0; T ]. T T T That's at: 0; N ; 2 N ;:::; (N − 1) N \Package" in a vector: 2 3 f0 6 f1 7 f ! ~v = 6 7 6 . 7 4 . 5 fN−1 Discrete Fourier Transform The Discrete Fourier Transform: Hayden Borg Sample basis vectors at the same points. Roots of Unity −i 2π The primitive Nth root of unity is !N = e N 2 0·k 3 !N 1·k 6 !N 7 i 2π kx 6 2·k 7 T 6 !N 7 e ! e~k = 6 7 6 . 7 4 . 5 (N−1)·k !N Note: We only need e~0; e~1;::: e~N−1 Discrete Fourier Transform The Discrete Fourier Transform: We'll call ~v^ the transform of ~v. Hayden Borg ^ [~v]k = ak Central result Let fy~i g be an orthonormal set in Hilbert space H and ~v 2 H P such that ~v = ak y~k . Then, ak = h~v; y~k i k `Do' the Fourier series on our sampled function: ^ ∗ [~v]k = h~v; e~k i = e~k ~v Then, ^ ∗ ~v = [e~0je~1j ::: je~N−1] ~v The Discrete Fourier Transform The Discrete Fourier Transform: Hayden Borg So we define the DFT to be the linear transformation N N T : C ! C defined by the matrix vector product: 2 0·0 0·1 0·(N−1) 3 !N !N :::!N 6 1·0 1·1 1·(N−1) 7 6!N !N :::!N 7 T (~v) = 6 . 7 ~v 6 . .. 7 4 . 5 N·0 N·1 N·(N−1) !N !N :::!N Call this matrix F. Theorem The matrix U = p1 F is unitary. N The Discrete Fourier Transform The Discrete Fourier Transform: The DFT is primarily used to go from the `time' to the Hayden Borg `frequency' domain Spectral analysis Spectroscopy Filtering MRI Spatial information Artifacts Audio recording and engineering Can also be used in data compression JPEG mp3 Cooley-Tukey Algorithm The Discrete Fourier Transform: Determining F and calculating the matrix vector product is Hayden Borg O(n2). We can exploit some symmetries to make this more efficient. Danielson-Lanczos Lemma N−1 m ^ P The DFT for N = 2 for some m 2 N,[~v]i = [~v]k [FN ]ik , k=0 may be rewritten ^ N [~v]i = [F N ~veven]i + [D N F N ~vodd]i for 0 ≤ i ≤ − 1 2 2 2 2 ^ N [~v]i = [F N ~veven]i − [D N F N ~vodd]i for < i < N − 1 2 2 2 2 Cooley-Tukey Algorithm The Discrete Fourier Transform: Hayden Borg Theorem k Given an N−point DTF, FN , where N = 2 where k 2 N. Then, " #" # I N D N F N even-odd FN = 2 2 2 . I N −D N F N permutation 2 2 2 Then the FFT may be calculated Calculate the !N (O(N)) Recursively apply this decomposition log2 N times This recursion gives log2 N operations for each `slot' and there are N slots so we have O(N log N) Conclusions The Discrete Fourier Transform: Hayden Borg 1 Euclidean spaces can be generalized to Hilbert spaces 2 Square-integrable functions are vectors in the Hilbert 2 space L (R) and can be expressed as a linear combination of basis vectors 3 The Fourier series and Fourier Transform are vector decomposition with the special basis fei2π!x g 4 The DFT can `do' the Fourier Transform on discrete data and can be represented as a matrix vector product 5 The DFT can be more efficiently calculated using the Cooley-Tukey Algorithm References The Discrete Halmos, P.