Fast Fourier Transform: Applications Convolution and FFT Applications. ! Optics, acoustics, quantum physics, telecommunications, control systems, signal processing, speech recognition, data compression, image processing. ! DVD, JPEG, MP3, MRI, CAT scan. ! Numerical solutions to Poisson's equation. The FFT is one of the truly great computational developments of this [20th] century. It has changed the face of science and engineering so much that it is not an exaggeration to say that life as we know it would be very different without the FFT. -Charles van Loan Algorithm Design by Éva Tardos and Jon Kleinberg • Copyright © 2005 Addison Wesley • Slides by Kevin Wayne 2 Fast Fourier Transform: Brief History Polynomials: Coefficient Representation Gauss (1805, 1866). Analyzed periodic motion of asteroid Ceres. Polynomial. [coefficient representation] A(x) = a + a x + a x2 + + a xn"1 0 1 2 L n"1 Runge-König (1924). Laid theoretical groundwork. B(x) = b +b x +b x2 + + b xn"1 0 1 2 L n"1 Danielson-Lanczos (1942). Efficient algorithm. ! Add: O(n) arithmetic operations. Cooley-Tukey (1965). Monitoring nuclear tests in Soviet Union and ! A(x)+ B(x) = (a +b )+(a +b )x + + (a +b )xn"1 tracking submarines. Rediscovered and popularized FFT. 0 0 1 1 L n"1 n"1 Evaluate: O(n) using Horner's method. ! Importance not fully realized until advent of digital computers. A(x) = a +(x(a + x(a + + x(a + x(a )) )) 0 1 2 L n"2 n"1 L Multiply (convolve): O(n2) using brute force. ! 2n#2 i i A(x)" B(x) = $ ci x , where ci = $ a j bi# j i =0 j =0 3 4 ! Polynomials: Point-Value Representation Polynomials: Point-Value Representation Fundamental theorem of algebra. [Gauss, PhD thesis] A degree n Polynomial. [point-value representation] polynomial with complex coefficients has n complex roots. A(x) : (x0, y0 ), K, (xn-1, yn"1) B (x) : (x0, z0 ), K, (xn-1, zn"1) Corollary. A degree n-1 polynomial A(x) is uniquely specified by its evaluation at n distinct values of x. Add: O(n) arithmetic operations. ! A(x)+ B(x) : (x , y + z ), , (x , y + z ) 0 0 0 K n-1 n"1 n"1 y Multiply: O(n), but need 2n-1 points. ! A(x) " B(x) : (x , y " z ), , (x , y " z ) 0 0 0 K 2n-1 2n#1 2n#1 Evaluate: O(n2) using Lagrange's formula. yj = A(xj) ! (x " x ) n"1 $ j A x y j#k xj ( ) = % k x k=0 $(xk " x j ) j#k 5 6 ! Converting Between Two Polynomial Representations Converting Between Two Polynomial Representations: Brute Force n-1 Tradeoff. Fast evaluation or fast multiplication. We want both! Coefficient to point-value. Given a polynomial a0 + a1 x + ... + an-1 x , evaluate it at n distinct points x0, ... , xn-1. Representation Multiply Evaluate Coefficient O(n2) O(n) 2 y # 2 n"1 & a O(n ) for matrix-vector multiply # 0 & 1 x0 x0 L x0 # 0 & 2 % ( Point-value O(n) O(n ) % y ( 2 n"1 % a ( % 1 ( % 1 x1 x1 L x1 ( % 1 ( y % 2 n"1 ( a % 2 ( = 1 x2 x2 L x2 % 2 ( % ( % ( % ( % M ( % M M M O M ( % M ( 2 n"1 $% y '( % 1 x x x ( $% a '( 3 Goal. Make all ops fast by efficiently converting between two n"1 $ n"1 n"1 L n"1 ' n"1 O(n ) for Gaussian elimination representations. Vandermonde matrix is invertible iff xi distinct ! Point-value to coefficient. Given n distinct points x0, ..., xn-1 and values a0, a1,K, an-1 (x0, y0 ), K, (xn"1, yn"1) n-1 y0, ..., yn-1, find unique polynomial a0 + a1 x + ... + an-1 x that has given coefficient point-value values at given points. representation representation ! ! 7 8 Coefficient to Point-Value Representation: Intuition Coefficient to Point-Value Representation: Intuition n-1 n-1 Coefficient to point-value. Given a polynomial a0 + a1 x + ... + an-1 x , Coefficient to point-value. Given a polynomial a0 + a1 x + ... + an-1 x , evaluate it at n distinct points x0, ... , xn-1. evaluate it at n distinct points x0, ... , xn-1. Divide. Break polynomial up into even and odd powers. Divide. Break polynomial up into even and odd powers. 2 3 4 5 6 7 2 3 4 5 6 7 ! A(x) = a0 + a1x + a2x + a3x + a4x + a5x + a6x + a7x . ! A(x) = a0 + a1x + a2x + a3x + a4x + a5x + a6x + a7x . 2 3 2 3 ! Aeven(x) = a0 + a2x + a4x + a6x . ! Aeven(x) = a0 + a2x + a4x + a6x . 2 3 2 3 ! Aodd (x) = a1 + a3x + a5x + a7x . ! Aodd (x) = a1 + a3x + a5x + a7x . 2 2 2 2 ! A(-x) = Aeven(x ) + x Aodd(x ). ! A(-x) = Aeven(x ) + x Aodd(x ). 2 2 2 2 ! A(-x) = Aeven(x ) - x Aodd(x ). ! A(-x) = Aeven(x ) - x Aodd(x ). Intuition. Choose two points to be ±1. Intuition. Choose four points to be ±1, ±i. ! A(-1) = Aeven(1) + 1 Aodd(1). ! A(-1) = Aeven(-1) + 1 Aodd( 1). ! A(-1) = Aeven(1) - 1 Aodd(1). Can evaluate polynomial of degree ! n ! A(-1) = Aeven(-1) - 1 Aodd(-1). Can evaluate polynomial of degree ! n at 2 points by evaluating two polynomials at 4 points by evaluating two polynomials ! A(-i) = Aeven(-1) + i Aodd(-1). of degree ! !n at 1 point. of degree ! !n at 2 points. ! A(-i) = Aeven(-1) - i Aodd(-1). 9 10 Discrete Fourier Transform Roots of Unity n-1 th n Coefficient to point-value. Given a polynomial a0 + a1 x + ... + an-1 x , Def. An n root of unity is a complex number x such that x = 1. evaluate it at n distinct points x0, ... , xn-1. Fact. The nth roots of unity are: "0, "1, …, "n-1 where " = e 2# i / n. k th k n 2# i k / n n # i 2k 2k Key idea: choose xk = " where " is principal n root of unity. Pf. (" ) = (e ) = (e ) = (-1) = 1. Fact. The !nth roots of unity are: $0, $1, …, $n/2-1 where $ = e 4# i / n. # & # y0 & 1 1 1 1 L 1 # a0 & 2 2 k k % 1 2 3 n"1 ( Fact. " = $ and (" ) = $ . % y ( 1 ) ) ) ) % a ( % 1 ( % L ( % 1 ( % y ( % 1 )2 ) 4 )6 )2(n"1) ( % a ( 2 L 2 "2 = $1 = i % ( = % 3 6 9 3(n"1) ( % ( y3 1 ) ) ) L ) a3 % ( % ( % ( "3 "1 % M ( % M M M M O M ( % M ( % ( % n"1 2(n"1) 3(n"1) (n"1)(n"1)( % ( $ yn"1' $ 1 ) ) ) L ) ' $a n"1' "4 = $2 = -1 n = 8 "0 = $0 = 1 Discrete Fourier transform Fourier matrix F ! n "5 "7 "6 = $3 = -i 11 12 Fast Fourier Transform FFT Algorithm n-1 th Goal. Evaluate a degree n-1 polynomial A(x) = a0 + ... + an-1 x at its n roots of unity: "0, "1, …, "n-1. fft(n, a0,a1,…,an-1) { Divide: break polynomial up into even and odd powers. if (n == 1) return a0 2 (n-1)/2 ! Aeven(x) = a0 + a2x + a4x + … + an/2-2 x . ! 2 (n-1)/2 (e ,e ,…,e ) % FFT(n/2, a ,a ,a ,…,a ) Aodd (x) = a1 + a3x + a5x + … + an/2-1 x . 0 1 n/2-1 0 2 4 n-2 2 2 (d0,d1,…,dn/2-1) % FFT(n/2, a1,a3,a5,…,an-1) ! A(x) = Aeven(x ) + x Aodd(x ). th for k = 0 to n/2 - 1 { Conquer. Evaluate degree Aeven(x) and Aodd(x) at the !n roots of k 2#ik/n " % e 0 1 n/2-1 unity: $ , $ , …, $ . k yk+n/2 % ek + " dk k yk+n/2 % ek - " dk Combine. } k+n k k k ! A(" ) = A ($ ) + " A ($ ), 0 ! k < n/2 even odd $k = ("k)2 = ("k+n)2 ! k+n k k k return (y ,y ,…,y ) A(" ) = Aeven($ ) - " Aodd($ ), 0 ! k < n/2 0 1 n-1 } "k+n = -"k 13 14 FFT Summary Recursion Tree Theorem. FFT algorithm evaluates a degree n-1 polynomial at each of th the n roots of unity in O(n log n) steps. a0, a1, a2, a3, a4, a5, a6, a7 assumes n is a power of 2 perfect shuffle Running time. T(2n) = 2T(n) + O(n) & T(n) = n log2 n. a0, a2, a4, a6 a1, a3, a5, a7 a0, a4 a2, a6 a1, a5 a3, a7 O(n log n) 0 n#1 a0, a1, , an-1 ( , y ), , ( , y ) K " 0 K " n#1 a0 a4 a2 a6 a1 a5 a3 a7 coefficient point-value 000 100 010 110 001 101 011 111 representation representation ! ! "bit-reversed" order 15 16 Point-Value to Coefficient Representation: Inverse DFT Inverse FFT Goal. Given the values y0, ... , yn-1 of a degree n-1 polynomial at the n Claim. Inverse of Fourier matrix is given by following formula. 0 1 n-1 n-1 points " , " , …, " , find unique polynomial a0 + a1 x + ... + an-1 x that has given values at given points. $ 1 1 1 1 L 1 ' & 1 #1 #2 #3 #(n#1) ) & " " " L " ) #2 #4 #6 #2(n#1) "1 1 & 1 " " " L " ) # a0 & # 1 1 1 1 L 1 & # y0 & Gn = & #3 #6 #9 #3(n#1) ) % 1 2 3 n"1 ( n 1 " " " L " % a ( 1 ) ) ) ) % y ( & ) % 1 ( % L ( % 1 ( & ) 2 4 6 2(n"1) M M M M O M % a2 ( % 1 ) ) ) L ) ( % y2 ( & #(n#1) #2(n#1) #3(n#1) #(n#1)(n#1)) = % 3 6 9 3(n 1) ( % 1 " " " L " ( % a ( 1 ) ) ) ) " % y ( % 3 ( % L ( % 3 ( % M ( % M M M M O M ( % M ( % ( % n"1 2(n"1) 3(n"1) (n"1)(n"1)( % ( $a n"1' $ 1 ) ) ) L ) ' $ yn"1' Consequence.! To compute inverse FFT, apply same algorithm but use "-1 = e -2# i / n as principal nth root of unity (and divide by n).