Fast Fourier Transform

Fast Fourier Transform Applications. ■ Perhaps single algorithmic discovery that has had the greatest practical impact in history.

■ Optics, acoustics, quantum physics, telecommunications, systems theory, signal processing, speech recognition, data compression.

■ Progress in these areas limited by lack of fast algorithms.

History.

■ Cooley-Tukey (1965) revolutionized all of these areas.

■ Danielson-Lanczos (1942) efficient algorithm.

■ Runge-König (1924) laid theoretical groundwork.

■ Gauss (1805, 1866) describes similar algorithm.

■ Importance not realized until advent of digital computers.

Jean Baptiste Joseph Fourier (1768-1830)

2

Polynomials: Coefficient Representation Polynomials: Point-Value Representation

Degree n polynomial. Degree n polynomial.

= + + 2 +L+ n−1 ■ Uniquely specified by knowing p(x) at n different values of x. p(x) a0 a1 x a2 x an−1 x − {}K = = + + 2 +L+ n 1 (x0 , y0 ), (x1, y1 ), , (xn-1, yn−1 ) , where yk p(xk ) q(x) b0 b1 x b2 x bn−1 x

Addition: O(n) ops. + = + + + +L+ + n−1 y = p(x) p(x) q(x) (a0 b0 ) (a1 b1 )x (an−1 bn−1 )x

Evaluation: O(n) using Horner's method. = + + +L+ + L p(x) a0 (x a1 x(a2 x(an−2 x(an−1 )) ))

yj Multiplication (convolution): O(n2).

j x × = + + +L+ j +L+ 2n−2 j x p(x) q(x) (a0b0 ) (a0b1 a1b0 )x ( ∑ akb j−k )x (an−1bn−1 )x k=0

3 4 Polynomials: Point-Value Representation Best of Both Worlds

Degree n polynomial. Can we get "fast" multiplication and evaluation? {}K = (x0 , y0 ), (x1, y1 ), , (xn-1, yn−1 ) , where yk p(xk ) Representation Multiplication Evaluation {}K = (x0 , z0 ), (x1 , z1 ), , (xn-1 , zn−1 ) , where zk q(xk ) coefficient O(n2) O(n) point-value O(n) O(n2) Addition: O(n). FFT O(n log n) O(n log n) {}+ + K + (x0 , y0 z0 ), (x1 , y1 z1 ), , (xn-1, yn−1 zn−1 ) ! Yes! Convert back and forth between two representations.

Multiplication: O(n), but need 2n points. K a0,a1, ,an-1 coefficient multiplication c {}K c0, 1,K,c2n-2 (x , y z ), (x , y z ), , (x , y − z − ) K O(n2 0 0 0 1 1 1 2n-1 2n 1 2n 1 b0,b1, ,bn-1 )

evaluation interpolation 2 O(n log n) O(n log n) Evaluation: O(n ) using Lagrange's formula. p x FFT inverse FFT x ∏ ( −x ) n−1y p x = ≠ K p r ( ) ∑ j x j p(x0), ( 1), , (x 2n-1) point-value multiplication x k ∏ k( − x ) q x r(x ), ( ),K,r (x ) k=0 q 0 1 2n-1 ≠ q(x0), ( 1),K, (x2n-1) O(n) k j j k 5 6

Converting Between Representations: Naïve Solution Fast Interpolation: Key Idea

Evaluation (coefficient to point-value). Key idea: choose {x0, x1, . . . , xn-1 } to make computation easier! 1 n-1 ■ Given a polynomial p(x) =a0 + a2x + . . . + an-1 x , choose n ■ Set xk =xj? distinct points {x0, x1, . . . , xn-1 } and compute yk = p(xk), for each k x using Horner's method. x n 2 x x ■ O(n ). x n a x x − 1 x 2 L 1   y   0 0 n 0  a 0   0  Interpolation (point-value to coefficient).  2 L x −1 y 1 1 1 1 a 1   1   x −      ■ Given n distinct points {x , x , . . . , x } and y = p(x ), compute n 2 L 1 = y 0 1 n-1 k k 1 2 xn2 2   2   2  the coefficients {a , a , . . . , a } by solving the following linear  xn  M M 0 1 n-1 M M M O M a    system of equations.    n    x 2 L −1 y x 1 −1 −1 0   −1  n−1 ■ n Note Vandermonde matrixx is invertiblex iff xk are distinct. 3 x a ■ n O(n ). x x − 1 x 2 L 1   y   0 0 n 0  a 0   0  2 x −1 1 L  a 1  y 1   1 1 1  xn 2 L −1   = y  1 2 xn2 2   2   2   xn  M M M M M O nM a       n    2 L −1 1 −1 −1 −1   −1  yn−1

7 8 Fast Interpolation: Key Idea Fast Interpolation: Key Idea

Key idea: choose {x0, x1, . . . , xn-1 } to make computation easier! Key idea: choose {x0, x1, . . . , xn-1 } to make computation easier!

■ Set xk =xj? ■ Set xk = xj? 2 2 2 2 ■ Use negative numbers: set xk = -xj so that (xk ) = (xj ) . ■ Use negative numbers: set xk = -xj so that (xk ) = (xj ) . – set xk = -xn/2 + k – set xk = -xn/2 + k n k th ■ Use complex numbers: set x = ω where ω is n root of unity. n k y 2 2  2 3 −1  a – (x ) = (-x ) 1 17 (17) (17) M (17)  k n/2 + k −   y  y– (x )4 = (-x )4 1 5 (5)2 (5)3 L (5) 1  a 0   0  k n/4ω + k     y  y– (x )8 = (-x )8 ω  M M M M L Mn  a 1 1 k n/8 + k   =   ω ω  n  2 y 2 y n 2 3 −1      1 1 ω 1 1 L 1  1 − 17 (−17) (−17) M (−17)  M M    ω  a  a     0  1 2 ω 3 Lω −1  0   − − 2 − 3 L − −1   n     1 ω n  a 1 5 ( 5) ( 5) ( 5)   −   y −   y 1     1    1 n 1 n 2 4 6 ω 2( −1) M M M M L M    1 ω L  a     2  = ωn n  2   3 n6 9 3( −1)   3   1 ω n L ω  a 3      M  M M M ω M O n M  M – E = p (x2) = a + a 172 + a 174 + a 176 + . . . + a 17n-2    n    even 0 2 4 6 n-2   −1 2( −1) 3( −1) ( −1)( −1)   2 3 5 7 n - 1  −1  1 L ω  an−1 – O = x podd(x ) = a117 + a317 + a517 + a717 + . . . + an-117  

– y0 = E + O, yn/2 = E - O

9 10

Roots of Unity Roots of Unity: Properties

An nth root of unity is a complex number z such that zn = 1. L1: Let ω be the principal nth root of unity. If n > 0, then ωn/2 = -1. 2π i / n th 2π i / n n/2 π i ■ ω = e = principal n root of unity. ■ Proof: ω = e ⇒ω = e = -1. (Euler's formula) i t ■ e = cos t + i sin t. 2 ω ν th th ■ i = -1. L2: Let n > 0 be even, and let and be the principal n and (n/2) k 2 k k roots of unity. Then (ω ) = ν . ■ There are exactly n roots of unity: ω , k = 0, 1, . . . , n-1. k 2 (2k)2π i / n (k) 2π i / (n / 2) k ■ Proof: (ω ) = e = e = ν .

ω2 = i L3: Let n > 0 be even. Then, the squares of the n complex nth roots of ω3 ω1 unity are the n/2 complex (n/2) th roots of unity. th th ■ Proof: If we square all of the n roots of unity, then each (n/2) ω4 = -1 ω0 = 1 root is obtained exactly twice since: – L1 ⇒ωk + n / 2 = - ωk – thus, (ωk + n / 2) 2 = (ωk ) 2 – L2 ⇒ both of these = ν k ω5 ω7 – ωk + n / 2 and ωk have the same square ω6 = -i

11 12 Divide-and-Conquer FFT Algorithm

Given degree n polynomial p(x) = a + a x1 + a x2 + . . . + a xn-1. 0 1 2 n-1 FFT (n, a , a , a , . . . , a ) th 0 1 2 n-1 ■ Assume n is a power of 2, and let ω be the principal n root of unity. if (n == 1) // n is a power of 2 ■ Define even and odd polynomials: return a0 1 2 3 n/2 - 1 – peven(x) := a0 + a2x + a4x + a6x + . . . + an-2 x ω ← e 2π i / n – p (x) := a + a x1 + a x2 + a x3 + . . . + a xn/2 - 1 odd 1 3 5 7 n-1 ← 2 2 (e0,e1,e2,...,en/2-1) FFT(n/2, a0,a2,a4,...,an-2) – p(x) = peven(x ) + x podd(x ) ← (d0,d1,d2,...,dn/2-1) FFT(n/2, a1,a3,a5,...,an-1)

■ Reduces problem of for k = 0 to n/2 - 1 y ← e + ωk d O(n) complex multiplies – evaluating degree n polynomial p(x) at ω0, ω1, . . . , ωn-1 k k k y ← e - ωk d if we pre-compute ωk. to k+n/2 k k – evaluating two degree n/2 polynomials at: (ω0)2, (ω1)2, . . . , (ωn-1)2 . return (y ,y ,y ,...,y ) 0 T 1n 2 n-1 T n O n th ■ L3 ⇒ p (x) and p (x) only evaluated at n/2 complex (n/2) roots T n even odd O n of unity. ( ) = 2 ( / 2) + ( ) ⇒ ( ) = ( logn)

13 14

Recursion Tree Proof of Correctness ω ωk p k n−1 Proof of correctness. Need to show yk = p( ) = ω ( ) ∑ a k for each k = 0, . . . , n-1, where ω is the principal j j =j 0 n th root of unity. a0, a1, a2, a3, a4, a5, a6, a7 ⇒ω ω0 ■ Base case. n = 1 = 1. Algorithm returns y0 = a0 = a0 .

■ Induction step. Assume algorithm correct for n / 2. – let ν be the principal (n/2) th root of unity νk ω2k – ek = peven( )= peven( ) by Lemma 2 a0, a2, a4, a6 a1, a3, a5, a7 νk ω2k – dk = podd( ) = podd( ) by Lemma 2 – recall p(x) = p (x2) + x p (x2) y k ω even odd k k e y k n ωd k k k = + k e ω k a0, a4 a2, a6 a , a a3, a7 p ω + = − 1 5 ωk / 2 d = 2 + k ω 2k =p k 2 − k 2 peven( ) podd( ) evenω ( ) odd( ) ω k = ( ) = p k2 + p + / 2 2 evenω( ) k n ω odd( ) 2 +ω + / 2 2 + p ( k n ) + p k( ) a0 a4 a a a a a a evenω k odd 2 6 1 5 3 7 k + n ω p(ω n / 2 ) ω p k n "bit-reversed" order ω

15 16 Best of Both Worlds y Inverse FFT y ω Can we get "fast" multiplication and evaluation? Forwardy FFT: given {a , aω , . . . , a } , compute {y , y , . . . , y } . ω 0 1 n-1 0 1 n-1 y ω  1 1 ω 1 1 L n 1  a Representation Multiplication Evaluation  0   ω   0    1 2 ω 3 ω −1    1 ω L n  a coefficient O(n2) O(n)  y 1   1   n   2 4 6 ω 2( −1)    1 n ω L n a point-value O(n) O(n2)  2  =  ω   2   3 ω n6 9 ω 3( −1)   3  1 n L a 3  FFT O(n log n) O(n log n)       M M M M ω M O n M M    n    a    −1 2( −1) 3( −1) ( −1)( −1)     −  1 L ω a −  ! 1   n 1 Yes! Convert back and forth between two representations. a ω a Inverse FFT: given {y0, yω1, . . . , yn-1 } compute {a0, a1, . . . , an-1 }. ω a ,a ,K,a coefficient multiplication c a ω 0 1 n-1 c , ,K,c ω n −1 2 0 1 2n-2  1 1 1 1 L 1  y b ,b ,K,b O(n )  0   ω   0  0 1 n-1   1 2 ω 3 ω −1    1 ω L n  y an1   1  evaluation  2 4 6 ω 2( −1)  O(n log n) interpolation O(n log n)   1 n ω L n y  FFT inverse FFT  2   ω   2  = n − p    3 ω 6 n9 L ω 3( 1)  y  x 3  1  3 K p x r x   ω n   p(x0),q( 1), , ( 2n-1) point-value multiplication K r M  M M M M O M n  M x r(x0), ( 1), , (x2n-1)     q(x ), ( ),K,q(x ) O(n)    −1 2( −1) 3( −1) L ω ( −1)( −1)    0 1 2n-1  −1  1   yn−1 17 18

Inverse FFT Inverse FFT: Proof of Correctness

ω-1 th ω th Great news: same algorithmω as FFT, except use as "principal" n Summation lemma. Let be a primitive n root of unity. Then root of unity (and divide by n).ω n ω k ω n−1  ≡ 0 mod n n  1 1 ω 1 1 L 1  ∑ ω k j =   ω n  1 2 ω 3 ω ( −1) =j 0  0 otherwise  1 ω L   n  2 4 ω 6 Lω 2( −1)  1  ωk = ω ■ If k is a multiple of n then = 1. F  n3 6 9 3n( −1)  1 ω L ω th k  n  ■ Each n root of unity ω is a root of ω n  M M M M O M  xn - 1 = (x - 1) (1 + x + x2 + . . . + xn-1), ω n   ( −1) 2( −1) 3( −1) L ω ( −1)(n−1)  if ωk ≠ 1 we have: 1 + ωk + ωk(2) + . . . + ωk(n-1) = 0.  1 ω  ω F F n ω n n n ω i i ω−1 ω − ′  1 1 1 1 L 1  Claim: F and F -1 are inverses. −1 j i  ω n  n n ()′ = ∑ ij −1 −2 ω −3 ω −( −1) n  1 ω L  j =0  n  −2 −4 ω −6 Lω −2( −1) F − 1  1 ω  n−1 1 = n = n1 ω ( − ′)  − ω − − − n −  ∑ i i n 3 n6 9 L ω 3( 1) j  1  =j 0  M M M ω Mn O M  n  i = ′  −( −1) −2( −1) −3( −1) L ω −( −1)(n−1)  1 if i  1  =  0 otherwise

19 20 Inverse FFT: Algorithm Best of Both Worlds

Can we get "fast" multiplication and evaluation? IFFT (n, a0, a1, a2, . . . , an-1) if (n == 1) // n is a power of 2 Representation Multiplication Evaluation return a0 coefficient O(n2) O(n) ω ← -2πi/n e point-value O(n) O(n2) ← (e0,e1,e2,...,en/2-1) FFT(n/2, a0,a2,a4,...,an-2) ← FFT O(n log n) O(n log n) (d0,d1,d2,...,dn/2-1) FFT(n/2, a1,a3,a5,...,an-1)

for k = 0 to n/2 - 1 ! Yes! Convert back and forth between two representations. ← ωk yk (ek + dk ) / n ← ωk yk+n/2 (ek - dk ) / n K a0,a1, ,an-1 coefficient multiplication c return (y ,y ,y ,...,y ) c0, 1,K,c2n-2 0 1 2 n-1 K O(n2 b0,b1, ,bn-1 )

evaluation O(n log n) interpolation O(n log n) FFT inverse FFT p x p(x ), ( ),K,p (x ) point-value multiplication r x 0 q x1 2n-1 r(x ), ( ),K,r (x ) q 0 1 2n-1 q(x0), ( 1),K, (x2n-1) O(n)

21 22

Integer Arithmetic

Multiply two n-digit integers: a = an-1 . . . a1a0 and b = bn-1 . . . b1b0.

■ Form two degree n polynomials.

■ Note: a = p(10), b = q(10). p x n−1 q x n−1 = a = b ( ) ∑ x j ( ) ∑ x j j j =j 0 =j 0

■ Compute product using FFT in O(n log n) steps. r x p x ■ Evaluate r(10) = a × b. ( ) = ( )× q(x)

■ Problem: O(n log n) complex arithmetic steps.

Solution.

■ Strassen (1968): carry out arithmetic to suitable precision. ε – T(n) = O(n T(log n)) ⇒ T(n) = O(n log n (log log n)1+ )

■ Schönhage-Strassen (1971): use modular arithmetic. – T(n) = O(n log n log log n)

23