1.10 trigonometric and FFT (CSE) convention: In this chapter, i = √ 1 with i2 = 1, that is, not an index. Numbering of indices of vectors starts at 0. − −

1.10.1 trigonometric interpolation

n 1 − ijx Definition 1.38 The p : R C, p(x)= cje , cj C are called trigono- → j=0 ∈ metric polynomials of degree n 1. P − goal: given distinct knots xj [0, 2π), j =0,...,n 1 and values yj, j =0,...,n 1 solve: ∈ − −

find trigonometric polyomial p of degree n 1 s.t.p(xj)= yj, j =0,...,n 1 (1.33) − − Remark 1.39 (i) The trigonometric x p(x) is a 2π-periodic . The 7→ coefficients cj are its Fourier coefficients.

(ii) The (continuous) Fourier transform is an important tool in signal processing, e.g., when analyzing audio signals. In the simplest setting, a signal is assumed to be periodic (over a given time interval (0, T )) and writing it as a decomposes the signal into different frequency components. These components are then analyzed or modified (e.g., low pass or high pass filters). For T =2π, the Fourier series is simply the representation

2π ∞ ixj 1 ixj f(x)= a e , a = f(x)e− dx, (1.34) j j 2π j= 0 X−∞ Z

and aj are the Fourier coefficients. In order to avoid evaluating the integrals, one could proceed as follows: 1) sample the signal in the points xj; 2) approximate f by its trigono- metric interpolant p; 3) interpret the Fourier coefficients of p as (good) approximations to the Fourier coefficients of f.3

(iii) For convenience, we work in the present section with the complex exponentials eixj. How- ever, the Euler identities eix = cos x + i sin x show eixj = cos(jx)+ i sin(jx) so that one could also formulate the problem in terms of “true” trigonometric polynomials involving sin(kx) and cos(kx). Also the FFT discussed below can be formulated for these functions (“discrete cosine transform” and “discrete sine transform”).

3 The procedure described only suggests how to get the coefficients aj for j 0 since the interpolating n−1 ijx ≥ polynomial is of the form p(x) = j=0 cj e . It would therefore be more natural to seek the interpolating n−1 ijx c e real valued a− a polynomial of the form j=−(n−1)Pj . For signals, one has j = j so that one is lead to seek the interpolating polynomial of the form 1 (p(x)+ p(x)) for a trigonometric polynomial p. For real-valued f, P 2 this p is the solution of (1.33).

21 Theorem 1.40 Let xj [0, 2π), j = 0,...,n 1 be distinct. Then (1.33) is uniquely ∈ n 1 n − solvable for each sequence (yj) − C . j=0 ∈ n 1 ixj − ijx Proof: Set zj := e , j =0,...,n 1. Then the zj are distinct. The ansatz p(x)= ci e − j=0 yields the linear system of equations: P

0 1 n 1 z0 z0 ... z0 − c0 y0 0 1 n 1 z1 z1 ... z1 − c1 y1  . . .   .  =  .  (1.35) . . . . .  0 1 n 1       zn 1 zn 1 ... zn−1   cn 1   yn 1   − − −   −   −   e      =:V =:c =:y | {z } | {z } | {z } V is a so-called Vandermonde matrix with det V = (zk zj) =0 ✷ 0 j

n j ixj which satisfies ωn = 1. J note: ωn = e− K The matrix V of (1.35) is easily inverted under the assumption (1.36):

n Theorem 1.41 Assume (1.36) and (1.37). Let y := (y0,...,yn 1)⊤ C be given and n 1 − ∈ − ijx p(x)= cj e be the solution to (1.33). Set j=0 P j k n 1 · − Vn := ωn j,k=0 J “ DFT matrix” K (1.38) Then: 

n 1 1 1 − j k (i) n Vn y = c J i.e., ck = n ωn· yj K j=0 P 1 1 1 − 1 H 1 (ii) √n Vn symmetric and unitary i.e., √n Vn = √n Vn = √n Vn     n 1 n 1 jk − jk − (iii) Vn = ωn j,k=0 = ωn− j,k=0 Proof: ad (iii): X  1 ad (ii): Let vj, j =0,...,n 1 be the columns of Vn. Then: − √n n 1 H 1 − jk jk vk vk = n ωn− ωn = 1 • j=0 P

22 k = l : • 6 n 1 n 1 H 1 − jk lj 1 − l k j geometr. vk vl = ωn− ωn = ωn− = n n series j=0 j=0 X X l k n nl k 1 1 ω − 1 1 (ω ) − (1.37) = − n = − n = 0 since ωn = 1 n 1 ωl k n 1 ωl k n − n−  − n− ad(i): For the equidistant points xj, j = 0,...,n 1, given by (1.36), the linear system of − 1 (ii) 1 1 1 − 1 1 equations (1.35) has the form Vnc = y c = Vn− y = Vn y = Vny = ⇒ √n √n √n √n 1 ✷ n Vny.  

Exercise 1.42 Show the formula for the geometric series by mulitplying out the right-hand side: n 1 xn+1 = (1 x) xi, x =1. − − 6 i=0 X

Definition 1.43 The linear map

n n n : C C F → y0 . y = . Vn y  .  7→ yn 1  −    is called the discrete Fourier transform (DFT) of length n. 1 The inverse − is called IDFT (inverse discrete Fourier transform). Fn Remark 1.44 Theorem 1.41 yields

1 1 1 1 − y = Vny = Vny = n(y) (1.39) Fn n n nF

1.10.2 FFT observation: The matrix Vn is fully populated. A naive realization of the DFT therefore 2 requires O(n ) arithmetic operations. Exploiting the special structure of Vn leads to the Fast Fourier transform (FFT), which only needs O(n log n) arithmetic operations.

2 i π n Lemma 1.45 Let n =2m, ω = e± n . Let (y0,...,yn 1) C . Then the terms − ∈ n 1 − kj αk := yj ω k =0,...,n 1 − j=0 X

23 with ξ := ω2 and l =0,...,m 1 can be computed as follows: − m 1 − jl α2l = gj ξ with gj := yj + yj+m j=0 X m 1 − jl j α l = hjξ with hj := (yj yj m) ω j =0,...,m 1 2 +1 − + − j=0 X Proof: Since ωnl = 1 we get

n 1 n 1 2 − − lj lj l j n 2 2 n 2 ( + 2 ) α2l = yj ω = yj ω + yj+ 2 ω = j=0 j=0 X X n 1 n 1 2 − 2 − lj ln lj 2 n 2 n = ω (yj + yj+ 2 ω )= ω (yj + yj+ 2 ) j=0 j=0 X X n Since ω 2 = 1 we have: − n 1 − (2l+1)j α2l+1 = yj ω = j=0 X n 1 2 − l j l j n (2 +1) n (2 +1)( + 2 ) = ω yj + yj+ 2 ω = j=0 Xn 2 1 − n 2lj j ln j 2 = ω yj ω + yj n ω ω ω = + 2 · j=0 X n 1  2 − j 2lj = (yj yj n ) ω ω − + 2 j=0 X ✷

Lemma 1.45 shows that the computation of yˆ = (ˆy0,..., yˆn 1)⊤ := n(y) can be reduced to − n n F the computation of 2 (g) and 2 (h). With n =2m we haveF F

m 1 (ˆy0, yˆ2,..., yˆn 2)⊤ = m(g) , g =(yj + yj+m) − − j=0 F j m 1 (ˆy1, yˆ3,..., yˆn 1)⊤ = m(h) , h = ((yj yj+m) ωn) − − F − j=0 This yields

Algorithm 1.46 (FFT) p n Input: n =2 , p N0, y =(y0,...,yn 1)⊤ C ∈ − ∈ Output: yˆ = (ˆy0,..., yˆn 1)= n(y) − F if n =1 then

24 yˆ0 := y0 else −2πi ω := e n n m := 2 m 1 m 1 − − (gj)j=0 := (yj + yj+m)j=0 m 1 j m 1 (hj) − := ((yj yj m) ω ) − j=0 − + j=0 (ˆy0, yˆ2,..., yˆn 2) := F F T (m, g) − (ˆy1, yˆ3,..., yˆn 1) := F F T (m, h) end if − return (yˆ)

1 Similarly for the Inverse Fourier Transform: (ˇy0,..., yˇn 1) := n− (y) (cf. first equation in − (1.39)): F 1 1 m 1 (ˇy0, yˇ2,..., yˇn 2)⊤ = −n (g) , g =(yj + yj+m) − − 2 F 2 j=0 1 1 j m 1 (ˇy1, yˇ3,..., yˇn 1)⊤ = −n (h) , h = (yj yj+m)ωn − − 2 F 2 − j=0  Algorithm 1.47 (IFFT) p n Input: n =2 , p N0, y =(y0,...,yn 1)⊤ C 1 ∈ − ∈ Output: yˇ = − (y) Fn if n =1 then yˇ0 := y0 else 2πi ω := e n n m := 2 m 1 1 m 1 − − (gj)j=0 := 2 (yj + yj+m)j=0 m 1 1 j m 1 (hj) − := ((yj yj m) ω ) − j=0 2 − + j=0 (ˇy0, yˇ2,..., yˇn 2) := IFFT (m, g) − (ˇy1, yˇ3,..., yˇn 1) := IFFT (m, h) end if − return (yˆ)

p Cost of the FFT: Denote by A(n) the cost of the call of F F T (n, y) and let n =2 , p N0. Then: ∈ A(n) 2A(n/2) + C n (1.40) ≤ computation of g, h |{z}

25 and thus: (1.40) n A(n) 2 A + C n = ≤ 2   (1.40) p 1 p = 2 A 2 − + C 2 ≤ (1.40) p 2 p 1 p 2 2A 2− + C 2 − + C 2 = ≤    (1.40) 2 p 2 p = 2 A 2 − +2 C 2 ≤ (1.40) 2 p 3 p 2 p 2 2A 2− + C 2 − +2 C 2 = ≤   3 p 3 p = 2 A 2 − +3 C 2 ... ≤ ≤ 2p A 20 + pC 2p = ≤  = n A(1) + (log n) C n  2 ≤ n log n C′ mit C′ = C + A(1) ≤ · 2 · slide 9b

1.10.3 Properties of the DFT

The DFT appears very prominently when one is trying to compute efficiently the convolution of two sequences, which is defined in the following definition.

Z Cn Definition 1.48 (i) A sequence f =(fj)j Z is called n-periodic, if fj+n = fj j . per ∈ denotes the space of the n-periodic sequences. ∀ ∈

(ii) The DFT n is defined by: F n n n : C C F per → per n 1 − jk (fj)j Z ωn fj ∈ 7→ j=0 !k Z P ∈ n Since ωn = 1 the DFT n is well-defined; J i.e., n((fj)j Z) is again an n-periodic ∈ sequence K F F

(iii) the convolution is defined by: ∗ : Cn Cn Cn ∗ per × per → per n 1 − (f, g) (f g)k := fk j gj k Z 7→ ∗ j=0 − ! ∀ ∈ P (iv) the pointwise multiplication is defined by: · : Cn Cn Cn · per × per → per (f, g) (f g)k := fk gk k Z 7→ · · ∀ ∈ 26 Remark 1.49 The DFT of Def. 1.43 coincides with the definition of the DFT of Def. 1.48, n 1 − if one extends the finite sequence (fj)j=0 n-periodically.

n Theorem 1.50 For f , g C let ˆf := n(f), ˆg := n(g) be the Fourier transformations. ∈ per F F Then:

n n (i) n : C C is linear. F per → per n 1 1 1 − jk (ii) n− (f) = n ωn fj F j=0 !k Z P ∈ (iii) (convolution theorem)

f g = n(f g) = ˆf gˆ ∗ F ∗ · Proof: Exercise d ✷

1.10.4 application: fast convolution of sequence

Cn Example 1.51 Let f , g per. The naive evaluation of the convolution h := f g costs O(n2) operations. It is more∈ efficient to proceed with Theorem 1.50: ∗

1.) compute ˆf and gˆ usingFFT cost: O(n log n) 2.) compute hˆ := ˆf gˆ cost: O(n) · 1 3.) compute h = − hˆ usingIFFT cost: O(n log n) Fn   The convolution of finite (non-periodic) sequences is defined slightly differently, namely, for two N 1 N 1 N 1 finite sequences (fj)j=0− , (gj)j=0− , its convolution is given by the sequence (cj)j=0− with entries

j

cj = fj kgk. (1.41) − k X=0 N 1 The sequence (cj)j=0− can also be computed with the aid of the FFT:

N 1 N 1 − − Example 1.52 let (fj)j=0 , (gj)j=0 be finite sequences. j N 1 goal: compute (hj)j=0− given by hj = fj k gk, j =0,...,N 1 k=0 − − N 1 N 1 idea: periodize the two sequences (fj)j=0P− and (gj)j=0− , so that Example 1.51 is applicable. p n Procedure: Choose an n 2N of the form n =2 for a p N and define f′ , g′ C by ≥ ∈ 0 ∈ per

fj for j =0,...,N 1 f ′ := − j 0 for j = N,...,n 1  −

gj for j =0,...,N 1 g′ := j 0 for j = N,...,n − 1  −

27 Then:

f ′ =0 for N n j 1 (1.42) j − ≤ ≤ − g′ =0 for N j n 1 (1.43) j ≤ ≤ − For k 0,...,N 1 we have: ∈{ − } n 1 N 1 k N 1 k − (1.43) − − (f′ g′)k = fk′ j gj′ = fk′ j gj = fk′ j gj + fk′ j gj = fk j gj ∗ − − − − − j=0 j=0 j=0 j=k+1 j=0 X X X =fk−j X =0 by (1.42) X The convolution of non-periodic sequence arises,|{z} for example, when|{z} polynomials are multiplied.

N 1 j N 1 j Example 1.53 Let polynomials π(x)= − fjx and π (x)= − gjx of degree N 1 be j=0 2 j=0 − given. Then the product π1π2 is a polynomial of degree 2N 2 given by P − P 2(N 1) j − j π1(x)π2(x)= hjx , cj = fj kgk, − j=0 k X X=0 where we implicitly assume that fk = gk =0 for k N,..., 2N 2 . Hence, Example 1.52 is applicable. ∈{ − } An application that exemplifies the use of the FFT in connection with the computation of the convolution of sequences is the multiplication of very large numbers. Example 1.54 (multiplication of numbers with many digits) The fast realization of the multiplication of numbers with many digits is nowadays done by FFT4. Consider the multipli- cation of two integers with n digits that are written as

n n j j x = fjb , y = gjb , j=0 j=0 X X where b N (e.g., b = 10) and the coefficients (“digits”) satisfy fj, gj 0,...,b 1 . We ∈ 2n j ∈ { − } seek the representation of z = xy in the form z = j=0 cjb with cj 0,...,b 1 . This is very similar to Example 1.53, and a formal multiplication yields ∈ { − } P 2n j j xy = hjb , hj = fj kgk, − j=0 k X X=0 where we again assumed that fj =0= gj for j n +1,..., 2n . The sequence (hj)j can be calculated with cost O(n log n) using the FFT as∈ described { in Example} 1.52. The sought coeffi- cients (cj)j of z are obtained from the sequence (hj)j by one more sweep through the sequence with cost O(n) that ensures that the coefficients cj satisfy cj 0,...,b 1 . The following ∈ { − } loop overwrites the hj with the sought cj:

for j =0:2n do

4This is also a building block of arbitrary precision arithmetic

28 if hj b then ⊲ carrying over is necessary ≥ hj := hj hj/b b − ⌊ ⌋ hj+1 := hj+1 + hj/b end if ⌊ ⌋ end for

n n Example 1.55 (solving linear systems with circulant matrices) A matrix C C × is called circulant, if it has the form ∈

c0 cn 1 c2 c1 − ··· c1 c0 cn 1 c2  . − . .  C = . c c .. . .  1 0   .. ..   cn 2 . . cn 1   − −   cn 1 cn 2 c1 c0   − − ···    T Introduce the vector c := (c0,...,cn 1) . Observe that the matrix-vector product Cx is a − convolution, i.e., the entries yj of the vector y = Cx are given by

n 1 − yj = cj kxk, − k X=0 n 1 Cn n 1 where we view the sequence (cj)j=0− as an element of per (i.e., extend the sequence (cj)j=0− periodically). That is, (Cx)j =(c x)j, j =0,...,n 1. ∗ − Hence, given b Cn, the linear system of equations Cx = b can also be written as ∈ c x = b. (1.44) ∗

Solving for x can be achieved with the FFT. To that end, write c = n(c), x = n(x), b = F F n(b) and observe: F b b b 1. Applying DFT on both sides of (1.44) gives by the convolution theorem cjxj = bj, j = 0,...,n 1. − b b b 2. Hence, xj = bj/cj.

n 1 3. an inverse DFT of x =(x ) − gives x. b b b j j=0 Hence, the work to solve Cx = x is 2 FFTs of length n and n divisions. b b Remark: a much more common type matrices that can be treated with similar techniques are Toeplitz matrices. →

29