Orthogonal Functions∗

May 9, 2003

1 Introduction

A particularly important class of applications of linear algebra within math- ematics arises from treating functions as vectors, with the rules • (f + g)(x) = f(x) + g(x) • (αf)(x) = αf(x), where α is a scalar. R b • f · g = a f(x)g(x) dx. Here, the interval [a, b] over which we integrate is regarded as fixed within any particular discussion. Two important examples are

1. [a, b] = [0, 1] and the functions in question are taken to be polynomials. 2. [a, b] = [0, 2π] and the functions are periodic, e.g. trigonometric poly- nomials.

A set of functions on [a,b] is linearly independent if no linear combination of those functions is identically 0 on [a,b]. In particular, the powers of x are linearly independent on any interval, as are the functions sin(nx), cos(nx) and 1 on [0, 2π], for n and m positive integers. However, the latter set has an additional important property: they are orthogonal. That is Z 2π Z 2π Z 2π sin(nx) cos(mx) dx = sin(nx) sin(mx) dx = cos(nx) cos(mx) dx = 0 0 0 0 ∗Preliminary version: subject to later expansion

1 unless m = n, in which case

Z 2π Z 2π Z 2π sin(nx) cos(nx) dx = 0; sin2(nx) dx = cos2(nx) dx = π. 0 0 0 The powers of x are not orthogonal on any interval. However if we confine our attention to any particular interval, such as [0, 1], we can use the Gram- Schmidt orthogonalization algorithm to produce . Suppose we have orthogonal functions {fi}0≤i≤n, and a function g = Pn i=0 aifi, which is a linear combination of the functions fi. Then

Z b n Z b Z b X 2 g(x)fj(x) dx = ai fi(x)fj(x) dx = aj fj(x) dx, a i=0 a a so that R b a g(x)fj(x) dx aj = , R b 2 a fj(x) dx and the coefficients aj can be recovered by integration.

2 Complex Exponentials and the Discrete Fourier Transform

In dealing with periodic functions on [0, 2π], it is particularly useful to con- sider the complex exponential functions einx as n varies over the integers. From the power series

∞ X xn x2 x3 ex = = 1 + x + + + ··· , n 2 3 0 x2 x4 x6 cos(x) = 1 − + − + ··· , 2 4 6 and x3 x5 x7 sin(x) = x − + − + ··· , 3 5 7 we recover the identity

eix = cos(x) + i sin(x),

2 which is also consistent with the trigonometric addition formulae. Because we are considering functions that take complex values (although only as functions of the real variable x), we are, for the first time, considering a complex . This means that scalars are allowed to be complex numbers, and also requires an adjustment to the dot product. We recall that if z = x + iy, thenz ¯ = x − iy, and set

Z 2π f · g = f(x)g(x) dx. 0 With this definition, einx · eimx = 2π if n = m, and 0 otherwise, since inx −inx P inx e = e . In particular, if f(x) = ane , then

Z 2π imx −imx f · e = f(x)e = 2πam, 0 and we can recover the coefficients am by integration, as in the strictly real case. Let us now consider how to best approximate the coefficients am for 0 ≤ 2πk |m| ≤ N − 1 when we know the values f( N ) for a real valued function f. From the fact that f is real values, it follows that a−m = am for all m, so that we need only determine the coefficients am for m ≥ 0. The next observation is that we can approximate the that gives us 2πam by the Riemann sum N−1 2π X 2πk −2πimk f( )e N , N N k=0 so that N−1 N−1 1 X 2πk −2πimk 1 X 2πk mk a ≈ f( )e N = f( )W , m N N N N N k=0 k=0

−2πi t where we define WN to be e N . If we set v to be the vector whose k h com- 2π(k−1) ponent is f( N ) and MN to be the complex N × N symmetric matrix th (m−1)(k−1) whose (m, k) entry is WN , the components of MN v are approxima- tions to 2πNam 0 ≤ m ≤ N − 1. This approximation is called the discrete Fourier transform. It tends to be rather crude for values of m close to N.

3 3 The Fast Fourier Transform

In order to appreciate what follows, one must think of N as very large (per- haps as large as 106), so that the N 2 multiplications that are necessary to compute the discrete Fourier transform will take significant time. We will see now that there is a trick that allows us to do the computation with many fewer multiplications if N is a power of 2. The assumption is that multipli- cation is much more time consuming than addition, so that it is much more efficient to compute a(b + c + d) then ab + ac + ad. It will be more appropriate for what follows to shift the components and 2π(k th indices so that they go from 0 to N − 1, vk = f( N ) and the (m, k) (m)(k) component of MN is WN . We make the observation that, for N even, N 2 2 W = W N , and W = −1. We write veven for the vector formed from the N 2 N even components of v taken in order, and vodd for the vector formed from the odd components of v, also taken in order. From this it follows that the first N N 2 components of MN v are the components of the vector of length 2 whose t N j h component, (ending with j = 2 − 1) is

j (M N veven)j + W (M N vodd)j, 2 N 2

N th while the (j + 2 ) component of MN v (starting with j = 0) is

j (M N veven)j − W (M N vodd)j. 2 N 2 If we write µ(N) for the minimum number of multiplications required to compute MN v, it follows from the foregoing that, for N even, N N µ(N) ≤ 2µ( ) + . 2 2 We are not counting the multiplications required to compute the powers of WN , because these can be stored and used repeatedly, and we are really interested in the speed for which the computation can be carried out for fixed N and constantly changing v. Since 1 1 M = , 2 1 −1 it follows that µ(2) ≤ 1 and, from the foregoing by induction, that µ(2p) ≤ p 2p−1.

4 4 Problems

1. Let f(x) be the function on [0, 1] defined by 1 1 f(x) = x, 0 ≤ x ≤ ; f(x) = 1 − x, ≤ x ≤ 1. 2 2 Find the orthogonal projection of f on the space of quartic polynomials. Begin by finding an orthogonal for the space of quartic polynomi- als by applying Gram-Schmidt to the functions {1, x, x2, x3, x4}. You are encouraged to use MATLAB for this purpose.

2. Let f(x) = π − |π − x| for 0 ≤ x ≤ 2π.

1 R 2π −imx (a) Find the Fourier coefficients am = 2π 0 f(x)e dx of f by direct integration for |m| ≤ 4. π (b) Use the values of f at multiples of 2 and the discrete Fourier transform to approximate the same coefficients.

5