Convolution and FFT Applications

Convolution and FFT Applications. ! Optics, acoustics, quantum physics, telecommunications, control systems, signal processing, speech recognition, data compression, image processing.

! DVD, JPEG, MP3, MRI, CAT scan.

! Numerical solutions to Poisson's equation.

The FFT is one of the truly great computational developments of this [20th] century. It has changed the face of science and engineering so much that it is not an exaggeration to say that life as we know it would be very different without the FFT. -Charles van Loan

Fast Fourier Transform: Brief History Polynomials: Coefficient Representation

Gauss (1805, 1866). Analyzed periodic motion of asteroid Ceres. Polynomial. [coefficient representation] A(x) = a + a x + a x2 + + a xn"1 0 1 2 L n"1 Runge-König (1924). Laid theoretical groundwork. B(x) = b +b x +b x2 + + b xn"1 0 1 2 L n"1 Danielson-Lanczos (1942). Efficient algorithm. ! Add: O(n) arithmetic operations. Cooley-Tukey (1965). Monitoring nuclear tests in Soviet Union and ! A(x)+ B(x) = (a +b )+(a +b )x + + (a +b )xn"1 tracking submarines. Rediscovered and popularized FFT. 0 0 1 1 L n"1 n"1

Evaluate: O(n) using Horner's method. ! Importance not fully realized until advent of digital computers. A(x) = a +(x(a + x(a + + x(a + x(a )) )) 0 1 2 L n"2 n"1 L

Multiply (convolve): O(n2) using brute force. ! 2n#2 i i A(x)" B(x) = $ ci x , where ci = $ a j bi# j i =0 j =0

3 4

! Polynomials: Point-Value Representation Polynomials: Point-Value Representation

Fundamental theorem of algebra. [Gauss, PhD thesis] A degree n Polynomial. [point-value representation] polynomial with complex coefficients has n complex roots. A(x) : (x0, y0 ), K, (xn-1, yn"1)

B (x) : (x0, z0 ), K, (xn-1, zn"1) Corollary. A degree n-1 polynomial A(x) is uniquely specified by its evaluation at n distinct values of x. Add: O(n) arithmetic operations. ! A(x)+ B(x) : (x , y + z ), , (x , y + z ) 0 0 0 K n-1 n"1 n"1 y

Multiply: O(n), but need 2n-1 points. ! A(x) " B(x) : (x , y " z ), , (x , y " z ) 0 0 0 K 2n-1 2n#1 2n#1

Evaluate: O(n2) using Lagrange's formula. yj = A(xj) ! (x " x ) n"1 $ j A x y j#k xj ( ) = % k x k=0 $(xk " x j ) j#k 5 6

Converting Between Two Polynomial Representations Converting Between Two Polynomial Representations: Brute Force

n-1 Tradeoff. Fast evaluation or fast multiplication. We want both! Coefficient to point-value. Given a polynomial a0 + a1 x + ... + an-1 x ,

evaluate it at n distinct points x0, ... , xn-1. Representation Multiply Evaluate

Coefficient O(n2) O(n) 2 y # 2 n"1 & a O(n ) for matrix-vector multiply # 0 & 1 x0 x0 L x0 # 0 & 2 % ( Point-value O(n) O(n ) % y ( 2 n"1 % a ( % 1 ( % 1 x1 x1 L x1 ( % 1 ( y % 2 n"1 ( a % 2 ( = 1 x2 x2 L x2 % 2 ( % ( % ( % ( % M ( % M M M O M ( % M ( 2 n"1 $% y '( % 1 x x x ( $% a '( 3 Goal. Make all ops fast by efficiently converting between two n"1 $ n"1 n"1 L n"1 ' n"1 O(n ) for Gaussian elimination representations.

Vandermonde matrix is invertible iff xi distinct !

Point-value to coefficient. Given n distinct points x0, ..., xn-1 and values a0, a1,K, an-1 (x0, y0 ), , (xn"1, yn"1) n-1 K y0, ..., yn-1, find unique polynomial a0 + a1 x + ... + an-1 x that has given coefficient point-value values at given points. representation representation ! ! 7 8 Coefficient to Point-Value Representation: Intuition Coefficient to Point-Value Representation: Intuition

n-1 n-1 Coefficient to point-value. Given a polynomial a0 + a1 x + ... + an-1 x , Coefficient to point-value. Given a polynomial a0 + a1 x + ... + an-1 x ,

evaluate it at n distinct points x0, ... , xn-1. evaluate it at n distinct points x0, ... , xn-1.

Divide. Break polynomial up into even and odd powers. Divide. Break polynomial up into even and odd powers. 2 3 4 5 6 7 2 3 4 5 6 7 ! A(x) = a0 + a1x + a2x + a3x + a4x + a5x + a6x + a7x . ! A(x) = a0 + a1x + a2x + a3x + a4x + a5x + a6x + a7x . 2 3 2 3 ! Aeven(x) = a0 + a2x + a4x + a6x . ! Aeven(x) = a0 + a2x + a4x + a6x . 2 3 2 3 ! Aodd (x) = a1 + a3x + a5x + a7x . ! Aodd (x) = a1 + a3x + a5x + a7x . 2 2 2 2 ! A(-x) = Aeven(x ) + x Aodd(x ). ! A(-x) = Aeven(x ) + x Aodd(x ). 2 2 2 2 ! A(-x) = Aeven(x ) - x Aodd(x ). ! A(-x) = Aeven(x ) - x Aodd(x ).

Intuition. Choose two points to be ±1. Intuition. Choose four points to be ±1, ±i.

! A(-1) = Aeven(1) + 1 Aodd(1). ! A(-1) = Aeven(-1) + 1 Aodd( 1).

! A(-1) = Aeven(1) - 1 Aodd(1). Can evaluate polynomial of degree ! n ! A(-1) = Aeven(-1) - 1 Aodd(-1). Can evaluate polynomial of degree ! n at 2 points by evaluating two polynomials at 4 points by evaluating two polynomials ! A(-i) = Aeven(-1) + i Aodd(-1). of degree ! !n at 1 point. of degree ! !n at 2 points. ! A(-i) = Aeven(-1) - i Aodd(-1).

9 10

Discrete Fourier Transform Roots of Unity

n-1 th n Coefficient to point-value. Given a polynomial a0 + a1 x + ... + an-1 x , Def. An n root of unity is a complex number x such that x = 1.

evaluate it at n distinct points x0, ... , xn-1. Fact. The nth roots of unity are: "0, "1, …, "n-1 where " = e 2# i / n. k th k n 2# i k / n n # i 2k 2k Key idea: choose xk = " where " is principal n root of unity. Pf. (" ) = (e ) = (e ) = (-1) = 1.

Fact. The !nth roots of unity are: $0, $1, …, $n/2-1 where $ = e 4# i / n. # & # y0 & 1 1 1 1 L 1 # a0 & 2 2 k k % 1 2 3 n"1 ( Fact. " = $ and (" ) = $ . % y ( 1 ) ) ) ) % a ( % 1 ( % L ( % 1 ( % y ( % 1 )2 ) 4 )6 )2(n"1) ( % a ( 2 L 2 "2 = $1 = i % ( = % 3 6 9 3(n"1) ( % ( y3 1 ) ) ) L ) a3 % ( % ( % ( "3 "1 % M ( % M M M M O M ( % M ( % ( % n"1 2(n"1) 3(n"1) (n"1)(n"1)( % ( $ yn"1' $ 1 ) ) ) L ) ' $a n"1' "4 = $2 = -1 n = 8 "0 = $0 = 1

Discrete Fourier transform Fourier matrix F ! n "5 "7

"6 = $3 = -i

11 12 Fast Fourier Transform FFT Algorithm

n-1 th Goal. Evaluate a degree n-1 polynomial A(x) = a0 + ... + an-1 x at its n roots of unity: "0, "1, …, "n-1.

fft(n, a0,a1,…,an-1) {

Divide: break polynomial up into even and odd powers. if (n == 1) return a0 2 (n-1)/2 ! Aeven(x) = a0 + a2x + a4x + … + an/2-2 x .

! 2 (n-1)/2 (e ,e ,…,e ) % FFT(n/2, a ,a ,a ,…,a ) Aodd (x) = a1 + a3x + a5x + … + an/2-1 x . 0 1 n/2-1 0 2 4 n-2 2 2 (d0,d1,…,dn/2-1) % FFT(n/2, a1,a3,a5,…,an-1) ! A(x) = Aeven(x ) + x Aodd(x ).

for k = 0 to n/2 - 1 { Conquer. Evaluate degree A (x) and A (x) at the !nth roots of even odd "k % e2#ik/n 0 1 n/2-1 unity: $ , $ , …, $ . k yk+n/2 % ek + " dk k yk+n/2 % ek - " dk Combine. } k+n k k k ! A(" ) = A ($ ) + " A ($ ), 0 ! k < n/2 even odd $k = ("k)2 = ("k+n)2 ! k+n k k k return (y ,y ,…,y ) A(" ) = Aeven($ ) - " Aodd($ ), 0 ! k < n/2 0 1 n-1 } "k+n = -"k

13 14

FFT Summary Recursion Tree

Theorem. FFT algorithm evaluates a degree n-1 polynomial at each of th the n roots of unity in O(n log n) steps. a0, a1, a2, a3, a4, a5, a6, a7 assumes n is a power of 2

perfect shuffle Running time. T(2n) = 2T(n) + O(n) & T(n) = n log2 n.

a0, a2, a4, a6 a1, a3, a5, a7

a0, a4 a2, a6 a1, a5 a3, a7

O(n log n) 0 n#1 a0, a1,K, an-1 (" , y ), , (" , y ) 0 K n#1 a0 a4 a2 a6 a1 a5 a3 a7 coefficient point-value 000 100 010 110 001 101 011 111 representation representation ! ! "bit-reversed" order 15 16 Point-Value to Coefficient Representation: Inverse DFT Inverse FFT

Goal. Given the values y0, ... , yn-1 of a degree n-1 polynomial at the n Claim. Inverse of Fourier matrix is given by following formula. 0 1 n-1 n-1 points " , " , …, " , find unique polynomial a0 + a1 x + ... + an-1 x that has given values at given points. $ 1 1 1 1 L 1 ' & 1 #1 #2 #3 #(n#1) ) & " " " L " ) #2 #4 #6 #2(n#1) "1 1 & 1 " " " L " ) # a0 & # 1 1 1 1 L 1 & # y0 & Gn = & #3 #6 #9 #3(n#1) ) % 1 2 3 n"1 ( n 1 " " " L " % a ( 1 ) ) ) ) % y ( & ) % 1 ( % L ( % 1 ( & ) 2 4 6 2(n"1) M M M M O M % a2 ( % 1 ) ) ) L ) ( % y2 ( & #(n#1) #2(n#1) #3(n#1) #(n#1)(n#1)) = % 3 6 9 3(n 1) ( % 1 " " " L " ( % a ( 1 ) ) ) ) " % y ( % 3 ( % L ( % 3 ( % M ( % M M M M O M ( % M ( % ( % n"1 2(n"1) 3(n"1) (n"1)(n"1)( % ( $a n"1' $ 1 ) ) ) L ) ' $ yn"1' Consequence.! To compute inverse FFT, apply same algorithm but use "-1 = e -2# i / n as principal nth root of unity (and divide by n).

-1 Inverse DFT Fourier matrix inverse (Fn) !

17 18

Inverse FFT: Proof of Correctness Inverse FFT: Algorithm

Claim. Fn and Gn are inverses. Pf. ifft(n, a0,a1,…,an-1) { n$1 n$1 1 k j $ j k " 1 (k$k ") j & 1 if k = k " if (n == 1) return a0 Fn Gn k k " = % # # = % # = ' ( ) n n 0 otherwise j=0 j=0 ( (e0,e1,…,en/2-1) % FFT(n/2, a0,a2,a4,…,an-2) summation lemma (d0,d1,…,dn/2-1) % FFT(n/2, a1,a3,a5,…,an-1) ! Summation lemma. Let " be a principal nth root of unity. Then for k = 0 to n/2 - 1 { "k % e-2#ik/n k yk+n/2 % (ek + " dk) / n n #1 & n if k % 0 mod n k $ " k j = ' yk+n/2 % (ek - " dk) / n 0 otherwise j =0 ( }

Pf. return (y0,y1,…,yn-1) k ! } If k is a! multiple of n then " = 1 & sums to n. th k n 2 n-1 ! Each n root of unity " is a root of x - 1 = (x - 1) (1 + x + x + ... + x ). k k k(2) k(n-1) ! if " ' 1 we have: 1 + " + " + . . . + " = 0 & sums to 0. !

19 20 Inverse FFT Summary Polynomial Multiplication

Theorem. Inverse FFT algorithm interpolates a degree n-1 polynomial Theorem. Can multiply two degree n-1 polynomials in O(n log n) steps. given values at each of the nth roots of unity in O(n log n) steps.

assumes n is a power of 2

coefficient representation coefficient representation a0, a1,K, an-1 c , c , , c b , b , , b 0 1 K 2n-2 0 1 K n-1

! ! FFT O(n log n) inverse FFT O(n log n) O(n log n) 0 n#1 a0, a1,K, an-1 (" , y ), , (" , y ) 0 K n#1 O(n log n) point-value A(x0 ),K, A(x2n-1) point-value multiplication coefficient C(x ), C(x ), , C(x ) representation representation B(x ), , B(x ) 0 1 K 2n-1 0 K 2n-1 O(n) ! ! 21 22 ! !

FFT in Practice Integer Multiplication

Fastest Fourier transform in the West. [Frigo and Johnson] Integer multiplication. Given two n bit integers a = an-1 … a1a0 and

! Optimized C library. b = bn-1 … b1b0, compute their product c = a ( b.

! Features: DFT, DCT, real, complex, any size, any dimension.

! Won 1999 Wilkinson Prize for Numerical Software. Convolution algorithm. A(x) = a + a x + a x2 + + a xn"1 0 1 2 L n"1 ! Portable, competitive with vendor-tuned code. ! Form two polynomials. 2 n"1 B(x) = b0 +b1x +b2x +L+ bn"1x ! Note: a = A(2), b = B(2).

Implementation details. ! Compute C(x) = A(x) ( B(x). ! ! Instead of executing predetermined algorithm, it evaluates your ! Evaluate C(2) = a ( b. ! hardware and uses a special-purpose compiler to generate an ! Running time: O(n log n) complex arithmetic steps. optimized algorithm catered to "shape" of the problem.

! Core algorithm is nonrecursive version of Cooley-Tukey radix 2 FFT. Schönhage-Strassen (1971). O(n log n log log n) bit operations.

! O(n log n), even for prime sizes. In practice. GMP (GNU Multiple Precision Arithmetic Library) proclaims to be "the fastest bignum library on the planet." It uses Reference: http://www.fftw.org brute force, Karatsuba, and FFT, depending on the size of n.

23 24