Fast Fourier Transform Fast Fourier Transform Applications. ■ Perhaps single algorithmic discovery that has had the greatest practical impact in history. ■ Optics, acoustics, quantum physics, telecommunications, systems theory, signal processing, speech recognition, data compression. ■ Progress in these areas limited by lack of fast algorithms. History. ■ Cooley-Tukey (1965) revolutionized all of these areas. ■ Danielson-Lanczos (1942) efficient algorithm. ■ Runge-König (1924) laid theoretical groundwork. ■ Gauss (1805, 1866) describes similar algorithm. ■ Importance not realized until advent of digital computers. Jean Baptiste Joseph Fourier (1768-1830) 2 Polynomials: Coefficient Representation Polynomials: Point-Value Representation Degree n polynomial. Degree n polynomial. = + + 2 +L+ n−1 ■ Uniquely specified by knowing p(x) at n different values of x. p(x) a0 a1 x a2 x an−1 x − {}K = = + + 2 +L+ n 1 (x0 , y0 ), (x1, y1 ), , (xn-1, yn−1 ) , where yk p(xk ) q(x) b0 b1 x b2 x bn−1 x Addition: O(n) ops. + = + + + +L+ + n−1 y = p(x) p(x) q(x) (a0 b0 ) (a1 b1 )x (an−1 bn−1 )x Evaluation: O(n) using Horner's method. = + + +L+ + L p(x) a0 (x a1 x(a2 x(an−2 x(an−1 )) )) yj Multiplication (convolution): O(n2). j x × = + + +L+ j +L+ 2n−2 j x p(x) q(x) (a0b0 ) (a0b1 a1b0 )x ( ∑ akb j−k )x (an−1bn−1 )x k=0 3 4 Polynomials: Point-Value Representation Best of Both Worlds Degree n polynomial. Can we get "fast" multiplication and evaluation? {}K = (x0 , y0 ), (x1, y1 ), , (xn-1, yn−1 ) , where yk p(xk ) Representation Multiplication Evaluation {}K = (x0 , z0 ), (x1 , z1 ), , (xn-1 , zn−1 ) , where zk q(xk ) coefficient O(n2) O(n) point-value O(n) O(n2) Addition: O(n). FFT O(n log n) O(n log n) {}+ + K + (x0 , y0 z0 ), (x1 , y1 z1 ), , (xn-1, yn−1 zn−1 ) ! Yes! Convert back and forth between two representations. Multiplication: O(n), but need 2n points. K a0,a1, ,an-1 coefficient multiplication {}K c0,c1,K,c2n-2 (x , y z ), (x , y z ), , (x , y − z − ) K O(n2 0 0 0 1 1 1 2n-1 2n 1 2n 1 b0,b1, ,bn-1 ) evaluation interpolation 2 O(n log n) O(n log n) Evaluation: O(n ) using Lagrange's formula. FFT inverse FFT ∏ (x − x ) − j n 1 ≠ p(x) = ∑ y j k K k p(x0), p(x1), , p(x2n-1) point-value multiplication K = ∏ (x − x ) r(x0),r(x1), ,r (x2n-1) k 0 k j K O(n) j ≠k q(x0),q(x1), ,q(x2n-1) 5 6 Converting Between Representations: Naïve Solution Fast Interpolation: Key Idea Evaluation (coefficient to point-value). Key idea: choose {x0, x1, . , xn-1 } to make computation easier! 1 n-1 ■ Given a polynomial p(x) =a0 + a2x + . + an-1 x , choose n ■ Set xk =xj? distinct points {x0, x1, . , xn-1 } and compute yk = p(xk), for each k using Horner's method. 2 ■ O(n ). 2 L n−1 a y 1 x0 x0 x0 0 0 Interpolation (point-value to coefficient). 2 L n−1 1 x1 x1 x1 a1 y1 − ■ Given n distinct points {x0, x1, . , xn-1 } and yk = p(xk), compute 2 L n 1 a = y 1 x2 x2 x2 2 2 the coefficients {a0, a1, . , an-1 } by solving the following linear M M M O M M M system of equations. 2 L n−1 1 xn−1 xn−1 x0 an−1 yn−1 ■ Note Vandermonde matrix is invertible iff xk are distinct. 3 ■ O(n ). 2 L n−1 a y 1 x0 x0 x0 0 0 2 L n−1 1 x1 x1 x1 a1 y1 − 2 L n 1 a = y 1 x2 x2 x2 2 2 M M M O M M M 2 L n−1 1 xn−1 xn−1 xn−1 an−1 yn−1 7 8 Fast Interpolation: Key Idea Fast Interpolation: Key Idea Key idea: choose {x0, x1, . , xn-1 } to make computation easier! Key idea: choose {x0, x1, . , xn-1 } to make computation easier! ■ Set xk =xj? ■ Set xk = xj? 2 2 2 2 ■ Use negative numbers: set xk = -xj so that (xk ) = (xj ) . ■ Use negative numbers: set xk = -xj so that (xk ) = (xj ) . – set xk = -xn/2 + k – set xk = -xn/2 + k ωk ω th ■ Use complex numbers: set x k = where is n root of unity. 2 3 n−1 – (x )2 = (-x )2 1 17 (17) (17) M (17) k n/2 + k − a y – (x )4 = (-x )4 1 5 (5)2 (5)3 L (5)n 1 0 0 k n/4 + k a y – (x )8 = (-x )8 M M M M L M 1 1 k n/8 + k = a2 y2 − 1 1 1 1 L 1 1 − 17 (−17)2 (−17)3 M (−17)n 1 M M y a 0 ω1 ω2 ω 3 L ω n−1 0 − − 2 − 3 L − n−1 1 1 5 ( 5) ( 5) ( 5) a − y − y1 a1 n 1 n 1 2 4 6 2(n−1) M M M M L M y 1 ω ω ω L ω a 2 = 2 3 6 9 3(n−1) y3 1 ω ω ω L ω a3 M M M M M O M M – E = p (x2) = a + a 172 + a 174 + a 176 + . + a 17n-2 even 0 2 4 6 n-2 n−1 2(n−1) 3(n−1) (n−1)(n−1) 2 3 5 7 n - 1 yn−1 1 ω ω ω L ω an−1 – O = x podd(x ) = a117 + a317 + a517 + a717 + . + an-117 – y0 = E + O, yn/2 = E - O 9 10 Roots of Unity Roots of Unity: Properties An nth root of unity is a complex number z such that zn = 1. L1: Let ω be the principal nth root of unity. If n > 0, then ωn/2 = -1. 2π i / n th 2π i / n n/2 π i ■ ω = e = principal n root of unity. ■ Proof: ω = e ⇒ω = e = -1. (Euler's formula) i t ■ e = cos t + i sin t. 2 ω ν th th ■ i = -1. L2: Let n > 0 be even, and let and be the principal n and (n/2) k 2 k k roots of unity. Then (ω ) = ν . ■ There are exactly n roots of unity: ω , k = 0, 1, . , n-1. k 2 (2k)2π i / n (k) 2π i / (n / 2) k ■ Proof: (ω ) = e = e = ν . ω2 = i L3: Let n > 0 be even. Then, the squares of the n complex nth roots of ω3 ω1 unity are the n/2 complex (n/2) th roots of unity. th th ■ Proof: If we square all of the n roots of unity, then each (n/2) ω4 = -1 ω0 = 1 root is obtained exactly twice since: – L1 ⇒ωk + n / 2 = - ωk – thus, (ωk + n / 2) 2 = (ωk ) 2 – L2 ⇒ both of these = ν k ω5 ω7 – ωk + n / 2 and ωk have the same square ω6 = -i 11 12 Divide-and-Conquer FFT Algorithm Given degree n polynomial p(x) = a + a x1 + a x2 + . + a xn-1. 0 1 2 n-1 FFT (n, a , a , a , . , a ) th 0 1 2 n-1 ■ Assume n is a power of 2, and let ω be the principal n root of unity. if (n == 1) // n is a power of 2 ■ Define even and odd polynomials: return a0 1 2 3 n/2 - 1 – peven(x) := a0 + a2x + a4x + a6x + . + an-2 x ω ← e 2π i / n – p (x) := a + a x1 + a x2 + a x3 + . + a xn/2 - 1 odd 1 3 5 7 n-1 ← 2 2 (e0,e1,e2,...,en/2-1) FFT(n/2, a0,a2,a4,...,an-2) – p(x) = peven(x ) + x podd(x ) ← (d0,d1,d2,...,dn/2-1) FFT(n/2, a1,a3,a5,...,an-1) ■ Reduces problem of for k = 0 to n/2 - 1 y ← e + ωk d O(n) complex multiplies – evaluating degree n polynomial p(x) at ω0, ω1, . , ωn-1 k k k y ← e - ωk d if we pre-compute ωk. to k+n/2 k k – ω0 2 ω1 2 ωn-1 2 evaluating two degree n/2 polynomials at: ( ) , ( ) , . , ( ) . return (y0,y1,y2,...,yn-1) ⇒ th ■ L3 peven(x) and podd(x) only evaluated at n/2 complex (n/2) roots of unity. T (n) = 2T (n / 2) + O(n) ⇒ T (n) = O(nlogn) 13 14 Recursion Tree Proof of Correctness Proof of correctness. Need to show y = p(ωk ) n−1 k ω k = ∑ ω k j ω p( ) a j for each k = 0, . , n-1, where is the principal j =0 n th root of unity. a0, a1, a2, a3, a4, a5, a6, a7 ⇒ω ω0 ■ Base case. n = 1 = 1. Algorithm returns y0 = a0 = a0 . ■ Induction step. Assume algorithm correct for n / 2. – let ν be the principal (n/2) th root of unity νk ω2k – ek = peven( )= peven( ) by Lemma 2 a0, a2, a4, a6 a1, a3, a5, a7 νk ω2k – dk = podd( ) = podd( ) by Lemma 2 2 2 – recall p(x) = peven(x ) + x podd(x ) = + ω k = − ω k a0, a4 a2, a6 a1, a5 a3, a7 y k ek dk yk +n / 2 ek dk = ω 2k +ω k ω 2k = ω 2k − ω k ω 2k peven( ) podd( ) peven( ) podd( ) k = p(ω ) = ω 2k + ω k +n / 2 ω 2k peven( ) podd( ) + + + p (ω 2k n ) + ω k n / 2 p (ω 2k n ) a0 a4 a a a a a a even odd 2 6 1 5 3 7 + p(ω k n / 2 ) "bit-reversed" order 15 16 Best of Both Worlds Inverse FFT Can we get "fast" multiplication and evaluation? Forward FFT: given {a0, a1, .