Mathematics of Information Hilbert Spaces and Linear Operators

Chair for Mathematical Information Science

Verner Vlačić Sternwartstrasse 7 CH-8092 Zürich

Mathematics of Information Hilbert spaces and linear operators

These notes are based on [1, Chap. 6, Chap. 8], [2, Chap. 1], [3, Chap. 4], [4, App. A] and [5].

1 Vector spaces

Let us consider a field F, which can be, for example, the field of real numbers R or the field of complex numbers C.A vector space over F is a set whose elements are called vectors and in which two operations, addition and multiplication by any of the elements of the field F (referred to as scalars), are defined with some algebraic properties. More precisely, we have

Definition 1 (Vector space). A set X together with two operations (+, ·) is a vector space over F if the following properties are satisfied: (i) the first operation, called vector addition: X × X → X denoted by + satisfies

• (x + y) + z = x + (y + z) for all x, y, z ∈ X (associativity of addition) • x + y = y + x for all x, y ∈ X (commutativity of addition)

(ii) there exists an element 0 ∈ X , called the zero vector, such that x + 0 = x for all x ∈ X

(iii) for every x ∈ X , there exists an element in X , denoted by −x, such that x + (−x) = 0.

(iv) the second operation, called scalar multiplication: F × X → X denoted by · satisﬁes • 1 · x = x for all x ∈ X • α · (β · x) = (αβ) · x for all α, β ∈ F and x ∈ X (associativity for scalar multiplication) • (α+β)·x = α·x+β ·x for all α, β ∈ F and x ∈ X (distributivity of scalar multiplication with respect to ﬁeld addition) • α·(x+y) = α·x+α·y for all α ∈ F and x, y ∈ X (distributivity of scalar multiplication with respect to vector addition).

We refer to X as a real vector space when F = R and as a complex vector space when F = C.

Examples. 1. C is both a real and a complex vector space. N 2. The set F , {(x1, x2, . . . , xN ): xk ∈ F} of all N-tuples forms a vector space over F.

3. The set F[x] of all polynomials with coeﬃcients in F is a vector space over F.

4. The space FZ of all sequences of F is a vector space over F. M×N 5. The set F of all matrices of size M × N with entries in F forms a vector space over F under the laws of matrix addition and scalar multiplication.

6. If X is an arbitrary set and Y an arbitrary vector space over F, the set F(X , Y) of all functions X → Y is a vector space over F under pointwise addition and multiplication.

A subspace of a vector space X over F is a subset of X which is itself a vector space over F (with respect to the same operations). One can easily verify that a subset Y of X is a subspace using the following proposition.

Proposition 1.1. Let X be a vector space over F and Y a nonempty subset of X . Then Y is a subspace of X if it contains 0 and if it is stable under linear combinations, that is,

α · y + β · z ∈ Y

for all α, β ∈ F and y, z ∈ Y.

Examples. 1. R and iR are subspaces of the real vector space C. N 2. The set of N-tuples (x1, x2, . . . , xN−1, 0) with xk ∈ R is a subspace of R . p 3. The set ` (Z), p ∈ [1, ∞], of all complex-valued sequence {uk}k∈Z which satisfy

P+∞ p k=−∞ |uk| < ∞, if p ∈ [1, ∞), supk∈Z |uk| < ∞, if p = ∞,

is a subspace of CZ.

4. If X is an arbitrary set, Y a vector space over F, and Z a subspace of Y, then the set F(X , Z) of functions X → Z is a subspace of F(X , Y).

5. The space Cn[a, b] of all complex-valued functions with continuous derivatives of order 0 6 k 6 n on the closed, bounded interval [a, b] of the real line is a subspace of F([a, b], C). 6. If (S, Σ, µ) is a measure space, we let Lp(S, Σ, µ) denote the space of all measurable functions mapping S to C whose absolute value raised to the p-power is µ-integrable, that is, Z p p L (S, Σ, µ) = f : S → C measurable: |f| dµ < ∞ , S two elements of Lp(S, Σ, µ) being considered as equivalent if they diﬀer only on p a set whose measure is zero. L (S, Σ, µ) is a subspace of F(S, C). For simplicity, p N we often use the notation L (S) when S is a subset of R , Σ is the Borel σ-algebra over S, and µ the Lebesgue measure on S. In this case, two functions are equivalent when they are equal except possibly on a set of Lebesgue measure 1 N zero (e.g., a ﬁnite or countable set of points of R ).

2 Theorem 1 (Intersection of vector spaces). The intersection of any collection of subspaces of a vector space X over F is again a subspace of X .

Deﬁnition 2. If S and T are linear subspaces of a vector space X with S ∩ T = {0}, then we deﬁne the direct sum S ⊕ T by

S ⊕ T = {x + y | x ∈ S, y ∈ T }.

2 Note that the union of two subspaces is in general not a subspace. Take for example X = R . 2 The lines xR and yR, with x = (1, 0) and y = (0, 1), are both subspaces of R , but xR ∪ yR is a not subspace, given that x + y = (1, 1) ∈/ xR ∪ yR.

Definition 3 (Subspace spanned by S). Let S be a (possibly infinite) subset of a vector space X over F. The subspace spanned by S is the intersection of all subspaces containing S. It is the smallest subspace containing S. It is denoted by span(S) and may be written the set of all finite combinations of elements of S, that is,

( k ) X span(S) , λ`x` : k ∈ N, x` ∈ S, λ` ∈ F . `=1

Examples. 1. If S is empty, then the subspace spanned by S is {0}.

2. In the real vector space C, we have the following:

• the subspace spanned by {1} is R, • the subspace spanned by {i} is iR, • the subspace spanned by {1, i} is C. N 3. In F[x], the subspace spanned by {1, x, . . . , x } is the space FN [x] of all polynomials whose degree is less than N.

Deﬁnition 4 (Linear independence). Let X be a vector space over F. A ﬁnite set {x1, x2, . . . , xN } of vectors of X is said to be linearly independent if for every λ1, λ2, . . . , λN ∈ F, the equality

λ1x1 + λ2x2 + ... + λN xN = 0

2 implies that λ1 = λ2 = ... = λN = 0. An inﬁnite set S of vectors of X is linearly independent if every ﬁnite subset of S is linearly independent.

Examples. 1. Any set of vectors containing the zero vector is linearly dependent.

2. The real vector space C, the set {1, i} is linearly independent. 2 N 3. The basic monomials {1, x, x , . . . , x } form a linearly independent set of F[x]. 4. The set of trigonometric functions {1, cos(x), sin(x), cos2(x), cos(x) sin(x), sin2(x)} is linearly dependent.

1Note, however, that there exist uncountable sets (e.g., the Cantor set) of Lebesgue measure zero.

3 Deﬁnition 5 (Dimension). A vector space X is N-dimensional if there exists N linearly independent vectors in X and any N + 1 vectors in X are linearly dependent.

Definition 6 (Finite-dimensional space). A vector space X is finite-dimensional if X is N- dimensional for some integer N. Otherwise, X is infinite dimensional.

N Examples. 1. The spaces F and FN [x] are ﬁnite-dimensional (their dimension is N).

n p 2. The space F(X , X ), F[x], C [a, b], ` (Z) are inﬁnite-dimensional.

3 Deﬁnition 7 (Basis). A basis of a vector space X over F is a set of linearly independent and spanning vectors of X .

3 T T T Examples. 1. The set {ek}k=1, where e1 = [1 0 0] , e2 = [0 1 0] , and e3 = [0 0 1] , 3 forms a basis for R . 2 N 2. The set {1, x, x , . . . , x } forms a basis for FN [x]. 2 3 3. The set {1, x, x , x ,...} forms a basis for F[x].

Theorem 2. Let X be a ﬁnite dimensional vector space. Any set of linearly independent vectors can be extended to a basis of X .

2 Inner products and norms

From here, we will assume that F = R or F = C.

Deﬁnition 8 (Norm). Let X be a vector space over F. A norm on X is a function which maps X to R and satisﬁes the following properties:

(i) for all x ∈ X , we have kxk > 0 and kxk = 0 implies x = 0 (positivity)

(ii) for all x ∈ X and for all α ∈ F, we have kαxk = |α| kxk (homogeneity)

(iii) for all x, y ∈ X , we have kx + yk 6 kxk + kyk (triangle inequality).

Deﬁnition 9. A vector space X over F is a normed vector space (or a pre-Banach space) if it is equipped with a norm k·k.

3This type of basis is sometimes referred to as an algebraic basis or a Hamel basis to highlight the difference with Schauder bases in infinite-dimensional Banach spaces, and orthonormal bases in infinite-dimensional Hilbert space. In finite dimensions, all these concepts coincide (cf. notes on bases in infinite-dimensional spaces for more details).

4 Examples. 1. We can provide the space C0[a, b] with the norm

Z b kfk = |f(t)|dt. a

N 2. We can deﬁne a norm on F as follows:

N !1/2 X 2 kxk2 = |xk| . k=1

p 3. If we write u = {uk}k∈Z, then the following deﬁnes a norm on ` (Z), p ∈ [1, ∞],

P+∞ p1/p kukp = k=−∞ |uk| , p ∈ [1, ∞), kuk∞ = supk∈Z |uk|, p = ∞.

4. We can deﬁne the following norm on Lp(S, Σ, µ), p ∈ [1, ∞):

Z 1/p p kfkp = |f| dµ . S

Deﬁnition 10 (Inner product). Let X be a vector space over F. An inner product (or scalar product) over F is a function X × X → F such that, for all x, y ∈ X and α ∈ F, the following holds (i) hy, xi = hx, yi (symmetry)

(ii) hαx, yi = α hx, yi (sesquilinearity if F = C, bilinearity if F = R) (iii) hx + y, zi = hx, zi + hy, zi (sesquilinearity if F = C, bilinearity if F = R)

(iv) hx, xi ∈ R and hx, xi > 0 (positivity) (v) hx, xi = 0 implies that x = 0 (positivity).

The inner product on a vector space X induces a norm on X . But note that not all norms come from an inner product (see Problem 3 of Homework 1).

Theorem 3. Let X be a vector space. If h·, ·i is an inner product on X , then, kxk = phx, xi is a norm on X fulﬁlling the parallelogram law:

2 kxk2 + 2 kyk2 = kx + yk2 + kx − yk2 , x, y ∈ X .

On the other hand, every norm kxk on X fulﬁlling the parallelogram law is the induced norm of exactly one inner product, which is deﬁned as 1 hx, yi = (kx + yk2 − kx − yk2), = (1) 4 F R 1 hx, yi = (kx + yk2 − kx − yk2 + i kx + iyk2 − i kx − iyk2), = . (2) 4 F C Formulas (1) and (2) are called polarization formulas.

5 Deﬁnition 11 (Inner product space). A vector space X over F is an inner product space or a pre-Hilbert space if it is equipped with an inner product h·, ·i.

Theorem 4 (Cauchy-Schwarz inequality). Let X be an inner product space. Then, we have

|hx, yi| 6 kxk kyk , x, y ∈ X with equality if and only if x and y are linearly dependent.

In what follows, we will always assume that for an inner product space, k · k is the norm induced by the corresponding inner product.

N Examples. 1. The space C can be equipped with the inner product

N X hx, yi , xkyk, k=1

N where x = (x1, x2, . . . , xN ) and y = (y1, y2, . . . , yN ) are in C . The norm induced is the one given in the previous set of examples.

2 2. The space ` (Z) of square summable sequences is an inner product space. If we 2 write u = {uk}k∈Z and v = {vk}k∈Z, the inner product on ` (Z) and the norm induced are

+∞ !1/2 X X 2 hu, vi , ukvk and kuk2 , |uk| . k∈Z k=−∞

3. When (S, Σ, µ) is a measure space, the space L2(S, Σ, µ) is an inner product space if it is equipped with the inner product: Z hf, gi , fg dµ S with the induced norm Z 1/2 2 kfk2 , |f| dµ . S 4. The space C[0, 1] of all continuous functions on [0, 1] can be equipped with the inner product Z 1 hf, gi = f(x)g(x)dx. 0

Deﬁnition 12 (Convergence of a sequence). Let X be a normed space equipped with the norm k·k. A sequence {xn}n∈N of elements of X converges to x if for every ε > 0 there exists an integer N(ε) such that kxn − xk 6 ε whenever n > N(ε).

Theorem 5 (Norm is continuous). Let X be a normed vector space. Then, k · k is continuous, i.e., if a sequence {xn}n∈N of elements of X converges to x ∈ X , then kxnk converges to kxk.

6 Theorem 6 (Inner product is continuous). Let X be an inner product space. Then, h·, ·i is continuous, i.e., if two sequences {xn}n∈N and {yn}n∈N of elements of X converge to x ∈ X and y ∈ X , respectively, then hxn, yni converges to hx, yi.

3 Banach and Hilbert spaces

Deﬁnition 13 (Cauchy sequence). Let X be a normed space equipped with the norm k·k.A sequence {xn}n∈N of elements of X is called a Cauchy sequence if for every ε > 0 there exists an integer N(ε) such that kxn − xmk 6 ε whenever n, m > N(ε).

Theorem 7. Every convergent sequence is Cauchy.

Deﬁnition 14 (Complete space). A normed space X is complete if every Cauchy sequence in X converges in X .

Deﬁnition 15 (Banach space). A Banach space is a complete normed space.

Deﬁnition 16 (Hilbert space). A Hilbert space is a complete inner product space.

Deﬁnition 17. A subset S ⊆ X of a normed space X is called a closed set if it contains all limit points, i.e., if {xn}n∈N is a sequence of points xn ∈ S with limn→∞ xn = x ∈ X , then x ∈ S. We write S for the smallest closed subset of X containing S in the following sense: if A is any other closed subset of X such that A ⊃ S, then A ⊃ S. We say S is the closure of S in X .

2 2 Examples. 1. Consider R with norm k·k2. The open disk D = {x ∈ R | kxk2 < 1} is not a closed set (consider the sequence {(1 − 1/n, 0)}n∈N). The closure of D 2 is the closed disk D = {x ∈ R | kxk2 6 1}. 2 2. Let X = L [0, 1] be the space of square-integrable functions [0, 1] → C with the inner product Z 1 hf, gi = f(x)g(x)dx, (3) 0 and let S = C[0, 1] ⊂ X be the set of all continuous functions [0, 1] → C. Then 2πin · S = X . To see this, ﬁx a function f ∈ X and let fˆn = f, e denote its n-th PN ˆ 2πin · Fourier coeﬃcient. Then we have kf − fN k → 0, where fN = n=−N fne is the N-th partial Fourier sum. But, since fN is a linear combination of continuous 2πinx functions x → e , fN is itself continuous, for each N ∈ N. Therefore f ∈ S. Since f was arbitrary, we obtain X ⊂ S. But S ⊂ X in general, so S = X , as desired.

3. To say that a set S is closed only makes sense relative to the space X that S is a subset of. To see this, let X1 = C[0, 1] be the space of all continuous functions 2 from [0, 1] to C with the inner product (3), and let X2 = L [0, 1] be the space of all square-integrable functions f : [0, 1] → C with the same inner product. Let 1 S = {f ∈ C[0, 1] : f(x) = 0 for x ∈ [0, 2 ]} and note that S is a subset of both X1 and X2. However, S is closed in X1, but it is not closed in X2. Indeed, suppose

7 {fn}n∈N is a sequence in S that converges to some f ∈ X1 = C[0, 1]. Then

Z 1 Z 1 Z 1 2 2 2 2 2 2 |f(x)| dx = |f(x)−fn(x)| dx 6 |f(x)−fn(x)| dx = kf −fnk → 0, 0 0 0

1 as n → ∞, so f(x) = 0, for all x ∈ [0, 2 ]. Since we have assumed f ∈ C[0, 1], we have f ∈ S and so S is closed in X1. To see that S is not closed in X2, consider the following sequence of functions:  0 x ∈ [0, 1 ],  2 1 1 1 1 fn(x) := n x − 2 x ∈ ( 2 , 2 + n ],  1 1 1 x ∈ ( 2 + n , 1].

Note that fn ∈ S for all n ∈ N. Moreover we have

Z 1 Z 1 + 1 1 2 2 n 1 |fn(x) − [ 1 ,1](x)| dx 6 1 dx = → 0, 2 1 n 0 2

as n → ∞, so fn → 1 1 in X2. But 1 1 ∈/ S, so S is not closed in X2. [ 2 ,1] [ 2 ,1] 4. Since the concept of closedness depends on the ambient space, so does the concept of closure. In the above example, the closure of S in X1 is the set S itself, whereas 2 1 the closure of S in X2 is the set S = {f ∈ L [0, 1] : f(x) = 0 for x ∈ [0, 2 ]} ) S.

Deﬁnition 18. Let A ⊂ B be subsets of a normed space X . We say that A is dense in B if A = B. Equivalently, A is dense in B if, for every x ∈ B and every > 0, there exists a y ∈ A such that kx − yk < .

Theorem 8 (Completion of normed space). Let X be a normed space with norm k·k. Then, there exists a Banach space Xb with norm k·kc such that the following properties hold.

•X is a subspace of Xb;

• On X , k·kc = k·k;

• Xb is the closure of X .

Theorem 9 (Completion of inner product spaces). Let X be an inner product space with inner product h·, ·i . Then, there exists a Hilbert space Xb with inner product h·d, ·i such that the following properties hold.

•X is a subspace of Xb;

• On X , h·d, ·i = h·, ·i;

• Xb is the closure of X .

Furthermore, Xb is unique up to linear isometries.

8 Examples. 1. The set of rational numbers Q is an inner product space with inner product equal to pq for p, q ∈ Q. This space is not complete.

2. The set of real numbers R is a Hilbert space space with inner product equal to rs for r, s ∈ R. N 3. C is a Hilbert space. 2 4. ` (Z) is a Hilbert space. 5. When (S, Σ, µ) is a measure space, L2(S, Σ, µ) is a Hilbert space.

6. The space C[0, 1] equipped with the inner product

Z 1 hf, gi = f(x)g(x)dx. 0 is not a Hilbert space, because it is not complete. The completion of C[0, 1] is L2([0, 1]).

4 Orthogonality in Hilbert spaces

Deﬁnition 19. Two vectors x and y in an inner product space X are called orthogonal if hx, yi = 0. In this case we write x ⊥ y if . Two sets U, V ⊆ X are called orthogonal if hx, yi = 0 for all x ∈ U and y ∈ V. We write U ⊥ V if the sets X , Y are orthogonal.

Deﬁnition 20. Let S be a nonempty subset of the inner product space X . We deﬁne the orthogonal complement S⊥ of S as S⊥ = {x ∈ X : x ⊥ S}.

Lemma 1. Let S be a nonempty subset of the inner product space X . Then S⊥ is a closed linear subspace of X .

Proof. Follows from the linearity and continuity of the inner product.

The following theorem expresses one of the fundamental geometrical properties of Hilbert spaces. While the result may appear obvious (see Figure 1), the proof is not trivial.

Theorem 10 (Projection on closed subspaces). Let S be a closed subspace of a Hilbert space X . Then, the following properties hold.

(a) For each x ∈ X there is a unique closest point y ∈ S such that

kx − yk = min kx − zk. z∈S

(b) The point y ∈ S closest to x ∈ X is the unique element of S such that the “error” (x−y) ∈ S⊥.

Corollary 4.1. Let S be a closed subspace of a Hilbert space X . Then X = S ⊕ S⊥ and S⊥⊥ = S.

9 x S

Figure 1: y ∈ S is the point in the closed subspace S closest to x and the unique point such that the “error” (x − y) ∈ S⊥.

Lemma 2. Let S be a subset of a Hilbert space X . Then, the following properties hold.

(a) S⊥ = span(S)⊥;

(b) S⊥⊥ = span(S);

Proof. • Proof of (a): span(S)⊥ ⊆ S⊥ follows from S ⊆ span(S). It remains to show that S⊥ ⊆ span(S)⊥, which follows from the linearity and continuity of the inner product.

• Proof of (b): Follows from (a) and Corollary (4.1).

Proof of (c): Follows from (a) and Corollary (4.1).

Corollary 4.1 and Lemma 2 tell us that subspaces of Hilbert spaces behave analogously to subspaces of ﬁnite-dimensional vector spaces familiar to us. In these results the completeness of X is crucial, as illustrated by the following example:

Example. Let X = C[0, 1] equipped with the inner product

Z 1 hf, gi = f(x)g(x)dx. 0 As we have already seen, X is an inner product space, but not a Hilbert space. Let 1 S = {f ∈ C[0, 1] : f(x) = 0 for x ∈ [0, 2 ]} and note that this is a closed subspace of ⊥ 1 X . Its orthogonal complement is S = {f ∈ C[0, 1] : f(x) = 0 for x ∈ [ 2 , 1]}, but X 6= S ⊕ S⊥.

10 5 Orthonormal bases in Hilbert spaces

Deﬁnition 21 (Orthogonal/orthonormal set). A subset S of nonzero vectors of a Hilbert space X is orthogonal if any two distinct elements in S are orthogonal. An orthogonal set S is called orthonormal if kxk = 1 for all x ∈ S.

2iπnt Examples. 1. The set {t 7→ e }n∈Z of complex exponentials is orthonormal in L2[0, 1]. √ N−1 2iπkn/N 2. The set {en}n=0 , with en[k] = e / N for 0 6 k 6 N − 1, is an orthonor- N mal set in C . 3. Considering the space Z +∞ 2 2 Lw(R) = f : R → R: |f(x)| w(x)dx < ∞ −∞ equipped with the inner product Z +∞ hf, gi = f(x)g(x)w(x)dx −∞

−x2/2 with w(x) = e . We deﬁne the polynomial Hn, n ∈ N such that n d 2 2 e−x /2 = (−1)nH (x)e−x /2. dxn n

The polynomials Hn, n ∈ N, are the Hermite polynomials and they an orthogonal 2 set in Lw(R). 4. A function that is a sum of ﬁnitely many periodic functions is said to be quasiperiodic. If the ratios of the periods of the terms in the sum are rational, then the sum is itself periodic, but if at least one of the ratios is irrational, then the sum is not periodic. For example,

f(t) = eit + eiπt

is quasiperiodic but not periodic. Let X be the space of quasiperiodic functions f : R → C of the form n X iωkt f(t) = ake , n ∈ N, ak ∈ C, ωk ∈ R. k=1 Then X is a vector space. Furthermore, X is an inner product space with inner product

1 Z T hf, gi = lim f(t)g(t). (4) T →∞ T −T The set of functions

iωt {e | ω ∈ R} (5)

11 is orthonormal in X . Note that X is an inner product space but not complete. The completion of X is the space of L2-almost periodic functions and consists of equivalence classes of functions of the form

∞ X iωkt f(t) = ake , ak ∈ C, ωk ∈ R, k=1

P∞ 2 where k=1 |ak| < ∞. The set in (5) is an uncountable orthogonal subset of this Hilbert space.

We next introduce the concept of unconditional convergence, which allows us to deﬁne sums of uncountable number of terms.

Definition 22 (Unconditional convergence). Let {xα ∈ X | α ∈ I} be an indexed subset of a normed space X , where I may be countable or not. For each finite subset J of I, we define the partial sum X SJ = xα. α∈J P The unordered sum α∈I xα converges to x ∈ X , written as X x = xα, α∈I if for every ε > 0 there exists a finite subset Jε ⊆ I such that kSJ − xk < ε for all finite subsets J ⊆ I containing Jε.

Note that unconditional convergence is independent of permutations of the index set I.

Example. Let {xα ∈ X | α ∈ I} be an indexed set of non-negative real numbers xα > 0 and set n X o M = sup xα | J ⊆ I, J ﬁnite . α∈J P • If M = ∞, then α∈J xα is arbitrarily large for suﬃciently large J . Therefore, P α∈I xα does not converge unconditionally.

• Suppose that 0 6 M < ∞. Then, for each ε > 0 there exists a ﬁnite set Jε ⊆ J such that X M − ε < xα 6 M. α∈Jε

It follows that for each ﬁnite set J ⊆ I containing Jε we have X M − ε < xα 6 M, (6) α∈J

showing that the unordered sum converges to M.

12 P Definition 23. An unordered sum α∈I xα is Cauchy if for every ε > 0 there exists a finite subset Jε ⊆ I such that kSKk < ε for all finite sets K ⊆ I \ Jε.

Proposition 5.1. An unordered sum in a Banach space converges if and only if it is Cauchy. If an unordered sum in a Banach space converges then {α ∈ I | xα 6= 0} is at most countable.

Lemma 3. Let S = {xα ∈ X | α ∈ I} be an indexed orthogonal subset of a Hilbert space X . P P 2 Then, the sum α∈I xα converges unconditionally to some x ∈ X if and only if α∈I kxαk < ∞. In that case, we have

2 X X 2 xα = kxαk . (7) α∈I α∈I Proof. For every ﬁnite set K ⊆ I we have

2 X X X 2 xα = hxα, xβi = kxαk . α∈K α,β∈K α∈K

P P 2 Therefore, α∈I xα is Cauchy if and only if α∈I kxαk is Cauchy. Thus, one of the sums converges unconditionally if and only if the other does. Finally, (7) follows from the continuity of the norm and the fact that the sum is the limit of a sequence of ﬁnite partial sums.

Proposition 5.2 (Bessel’s inequality). Let X be a Hilbert space, S = {xα ∈ X | α ∈ I} an orthonormal subset of X , and x ∈ X . Then

P 2 2 (a) α∈I | hx, xαi | 6 kxk ; P (b) xS , α∈I hx, xαi xα converges unconditionally; ⊥ (c) x − xS ∈ S .

Proof. Let J ⊆ I be a ﬁnite set. It follows immediately from the orthogonality that

2 X 2 2 X 2 | hx, xαi | = kxk − x − hx, xαi xα 6 kxk . (8) α∈J α∈J

Therefore, (see Example 5) (a) holds. Furthermore, (a) implies (b) because of Lemma 3. Finally, the statement (c) means that hx − xS , xαi = 0 for all α ∈ I, which is true by the continuity of P the inner product and the fact that x − β∈K hx, xβixβ ⊥ xα for all ﬁnite subsets K of I and all α ∈ A.

Deﬁnition 24 (Closed linear span). Given a subset S of a Hilbert space X , we deﬁne the closed linear span [S] of S by n X X o [S] = cxx | cx ∈ F, cxx converges unconditionally . x∈S x∈S

Lemma 3 implies that the closed linear span [S] of an orthonormal subset S = {xα ∈ X | α ∈ I} of a Hilbert space X can be written as

n X X 2 o [S] = cαxα | cα ∈ F, |cα| < ∞ . α∈I α∈I

13 Theorem 11. If S = {xα ∈ X | α ∈ I} is an orthonormal subset of a Hilbert space X , then the following conditions are equivalent:

(a) hx, xαi = 0 for all α ∈ I implies that x = 0; P (b) x = α∈I hx, xαi xα for all x ∈ X ; 2 P 2 (c) kxk = α∈I | hx, xαi | for all x ∈ X ; (d) [S] = X ; (e) S is a maximal orthonormal set, i.e. it has the property that if S0 ⊃ S is another orthonormal set, then necessarily S0 = S.

Proof. • (a) implies (b): (a) states that S⊥ = {0}. Therefore, Proposition 5.2 implies (b); • (b) implies (c): Follows from Lemma 3; • (c) implies (d): (c) implies that S⊥ = {0}. Therefore, we also have [S]⊥ = {0}; P • (d) implies (e): (d) implies that x = α∈I cαxα for all x ∈ X . Therefore, if x ⊥ xα for all α ∈ I, then x = 0.

• (e) implies (a): Suppose for a contradiction that there exists an x ∈ X such that hx, xαi = 0 for all α ∈ I, but x 6= 0. Then S ∪ {x/kxk} is an orthonormal set and is a strict superset of S.

Deﬁnition 25. An orthonormal subset S = {xα ∈ X | α ∈ I} of a Hilbert space X is complete if it satisﬁes any of the equivalent conditions (a)–(e) in Theorem 11. A complete orthonormal subset of X is called orthonormal basis of X .

Condition (a) is the easiest to verify. Condition (b) is used most often. Condition (c) is called Parseval’s identity. Condition (d) simply expresses completeness. Condition (e) can be used to prove that every Hilbert space has an orthonormal basis using Zorn’s lemma. If a Hilbert space X has an orthonormal basis S = {xα ∈ X | α ∈ I}, then it is isomorphic to the sequence space l2(I). In what follows, we are mostly interested in Hilbert spaces that have a countable orthonormal bases, which are called separable.

2iπnt Examples. 1. The set {t 7→ e }n∈Z of complex exponentials forms an orthonormal basis for L2[0, 1].

2. The set {Hn}n∈N of Hermite polynomials forms an orthogonal basis for Lw(R). 2 √ It can be transformed into an orthonormal basis by noting that kHnk = 2πn! for all n ∈ N.

6 Bounded linear operators on a Hilbert space

Deﬁnition 26. Let X , Y be vector spaces over F. A linear operator (or linear transformation, or linear map) from X to Y is a function T: X → Y that satisﬁes T(αx + βy) = αTx + βTy

14 for all x, y ∈ X and α, β ∈ F. The set R(T) = {Tx: x ∈ X } is called the range of T. The dimension of R(T) is called the rank. The set N (T) = {x ∈ X : Tx = 0} is called the null space or kernel of T.

Example. N M If X is N-dimensional with basis {xn}n=1 and Y is M-dimensional with basis {ym}m=1, then T is completely determined by its matrix representation U = {Un,m} 16n6N with 16m6M respect to these two bases:

M X Txn = Un,mym, n = 1, 2,...,N. m=1

PN Therefore, if x = n=1 αnxn and y = Tx, we have

M N X X y = βmym with βm = Um,nαn. m=1 n=1

N M That is, if α = (α1, . . . , αN ) ∈ F and β = (β1, . . . , βM ) ∈ F are the coeﬃcient N M vectors of x and y with respect to the bases {xn}n=1 and {ym}m=1, respectively, then we have the matrix-vector relation: β = Uα.

Proposition 6.1. N (T) is a subspace of X and R(T) is a subspace of Y. Furthermore, if T is continuous then N (T) is closed.

Deﬁnition 27 (Bounded linear operators). A linear operator T: X → Y is bounded if there exists a constant K > 0 such that

kTxkY 6 K kxkX . (9) The set of bounded linear operators T: X → Y is denoted by B(X , Y).

Theorem 12. Let X , Y be normed spaces. Then, B(X , Y) is a normed space with norm

kTk = sup (kTxkY ). kxkX =1

This is called the operator norm of an operator between two normed vector spaces. Furthermore, if Y is a Banach space, then B(X , Y) is a Banach space.

Proposition 6.2. Let X , Y be normed spaces and T: X → Y a linear operator. The following properties are equivalent.

(a) T is continuous;

(b) kTk < ∞;

Proof.

15 ∞ (a) implies (b): Suppose that (b) does not hold. Then, there exists a sequence {xn}n=1 of elements xn ∈ X with kxnkX = 1 such that kTxnkY > n. Set yn = xn/n. Then, yn → 0, but kTynkY > 1. Therefore, T is not continuous as limn→∞ Tyn 6= T0 = 0; (b) implies (c): Obvious for x = 0. For x 6= 0, we have

x kTxkY = kxkX T 6 kTk kxkX ; kxkX Y

Lemma 4. Let X , Y be normed spaces and T: X → Y a linear operator. Then,

kTxkY 6 kTk kxkX , x ∈ X .

Proof. Obvious for x = 0. For x 6= 0, we have

x kTxkY = kxkX T 6 kTk kxkX . kxkX Y

Lemma 5. If T1 and T2 are linear operators between normed spaces for which the range of T2 is contained in the domain of T1, we can consider the composed operator T1T2. If T1 and T2 are bounded, then also T1T2 is bounded and

kT1T2k 6 kT1k kT2k .

Proof.

kT1T2k = sup (kT1(T2x)k) kxk=1

6 sup (kT1k k(T2x)k) kxk=1

= kT1k kT2k .

Deﬁnition 28. Let X , Y be a normed space with norms k·kX and k·kY , and let T : X → Y be a linear operator. We say that T is a linear isometry if kTxkY = kxkX , for all x ∈ X .

6.1 (Orthogonal) projections

Deﬁnition 29. A projection on a vector space X is a linear operator P: X → X such that 2 P = P.

Theorem 13. Let X be a vector space.

(a) If P : X → X is a projection, then X = R(P) ⊕ N (P);

(b) If X = S ⊕ T , then there exists a unique projection P: X → X such that S = R(P) and T = N (P), called the projection onto S along T .

16 Proof. We ﬁrst show that x ∈ R(P) if and only if x = Px. Clearly, Px ∈ R(P). Suppose that x ∈ R(P). Then x = Pz for some z ∈ X and, therefore, Px = PPz = Pz = x. Now we prove (a). Let x ∈ N (P) ∩ R(P). Then x = Px and Px = 0, which implies that x = 0. Therefore, N (P) ∩ R(P) = {0}. Furthermore, we can decompose each x ∈ X as x = Px + (I − P)x with Px ∈ R(P) and (I − P)x ∈ N (P). Therefore, X = R(P) ⊕ N (P). Now we show that (b) holds. Each x ∈ X has a unique decomposition x = s + t with s ∈ S and t ∈ T , and P (x) = s deﬁnes the required projection.

Example. Let n x o S = | x ∈ ⊆ 2 0 R R n x o T = | x ∈ ⊆ 2. x R R

2 Then, S and T are linear subspaces of R with S ∩ T = {0}. Furthermore, we can 2 decompose each vector in R as x x − y y = + . y 0 y | {z } |{z} ∈S ∈T

2 Therefore, R = S ⊕ T . Deﬁne the operator

2 2 P: R → R x x − y 7→ . y 0

2 This operator is linear and P = P. Therefore, P is a projection.

Deﬁnition 30. An orthogonal projection on a Hilbert space X is a linear operator P: X → X 2 such that P = P and

hPx, yi = hx, Pyi , x, y ∈ X .

Theorem 14. Let X be a Hilbert space.

⊥ (a) If P: X → X is an orthogonal projection, then R(P) is closed and R(P) = N (P). Therefore, X = R(P) ⊕ N (P) is an orthogonal direct sum;

(b) If S is a closed subspace, then there exists a unique orthogonal projection P: X → X such ⊥ that S = R(P) and S = N (P).

Proof. We ﬁrst prove (a). Let P: X → X be an orthogonal projection. According to Property (a) in Theorem 13, X = R(P) ⊕ N (P). If x = Pz ∈ R(P) and y ∈ N (P), we have

hx, yi = hPz, yi = hz, Pyi = 0.

17 ⊥ Therefore, R(P) = N (P) and, in particular, a closed subspace. Now we show that (b) holds. Let S be a closed subspace of X . Then, Corollary 4.1 implies ⊥ that X = S ⊕ S . We deﬁne the projection P by

⊥ Px = s, where x = s + t with s ∈ S, t ∈ S .

⊥ Then, R(P) = S and N (P) = S . Now let x1, x2 ∈ X and xi = si +ti be the unique decomposition ⊥ of xi with si ∈ S and ti ∈ S , i = 1, 2. Then, we have hx1, Px2i = hx1, P(s2 + t2)i = hx1, s2i = hs1 + t1, s2i = hs1, s2i = hs1, s2 + t2i = hPx1, s2 + t2i .

Therefore, P is an orthogonal projection. The uniqueness follows from Theorem 13(b).

Example. Let n x o S = | x ∈ ⊆ 2. 0 R R

2 Then, S is a (closed) linear subspaces of R . Furthermore, we have n 0 o S⊥ = | y ∈ . y R

Deﬁne the operator

2 2 P: R → R x x 7→ . y 0

2 This operator is linear, P = P, and

T !T x1 x2 x1 x2 x1 x2 2 P = x1x2 = P , , ∈ R . y1 y2 y1 y2 y1 y2

⊥ Therefore, P is an orthogonal projection with S = R(P) and S = N (P).

Lemma 6. Let P be a non-zero orthogonal projection. Then kPk = 1.

Proof. Recall that

kPk = sup (kPxk). kxk=1

If x 6= 0, the Cauchy-Schwarz inequality implies that

hPx, Pxi hx, Pxi kPxk = = 6 kxk . (10) kPxk kPxk

Therefore, kPk 6 1. For Px 6= 0, we have kP(Px)k = kPxk, which implies that kPk > 1.

18 Proposition 6.3. Let S = {xα : α ∈ I} be an orthonormal system (but not necessarily an ONB) in a Hilbert space X . Let V = [S] be the closed linear span of S. Then the orthogonal projection PV is given explicitly by X PV x = hx, xαixα. α∈I Proof. The set S can be extended to an ONB B of X (if X is separable, then this can be done by applying the Gram-Schmidt orthogonalization procedure, or in the general case, by using Zorn’s Lemma). Thus we can enumerate B = {xα : α ∈ J } by an index set J containing I. Now, given an arbitrary x ∈ X , deﬁne s, t ∈ X according to X X s = hx, xαixα, t = hx, xαixα. α∈I α∈J \I Then, by Theorem 11, we have X X X x = hx, xαixα = hx, xαixα + hx, xαixα = s + t. α∈J α∈I α∈J \I Moreover, * + X X hs, ti = hx, xαixα, hx, xβixβ α∈I β∈J \I X = hx, xαihx, xβi hxα, xβi = 0. α∈I | {z } β∈J \I =0, as α6=β

Therefore, by Theorem 14, we have PV x = s, as desired.

7 The dual of a Hilbert space

Deﬁnition 31. Let X be a Hilbert space. An element of B(X , F) is called a bounded linear functional.

Example. Let X be a Hilbert space. For every y ∈ X , the mapping

X → F (11) x 7→ hx, yi (12)

is a bounded linear functional, which we denote by fy.

Theorem 15 (Riesz representation theorem). Let f : X → F be a bounded linear functional on a Hilbert space X . Then there exists a unique yf ∈ X such that f(x) = hx, yf i for all x ∈ X .

Therefore, the mapping

θX : X → B(X , F) (13) y 7→ fy = h·, yi

19 is a bijection. In the case of F = C, this mapping is antilinear (as can be seen from θX (αy) = h·, αyi = α h·, yi = θX (y)). Furthermore, the mapping θ is an isometry:

kfyk = sup |fy(x)| = sup |hx, yi| = kyk, y 6= 0. kxk=1 kxk=1

What is more, B(X , F) is a Hilbert space with inner product

hfx, fyi = hy, xi .

Theorem 16. Let X , Y be Hilbert spaces and T ∈ B(X , Y) a bounded linear operator. Then, ∗ there exists a unique operator T ∈ B(Y, X ) such that ∗ hTx, yi = hx, T yi , x ∈ X , y ∈ Y (14) ∗ ∗∗ with kT k = kTk. Furthermore, we have T = T.

Proof. We ﬁrst prove the existence. Let y ∈ Y be arbitrary but ﬁxet. The mapping x 7→ hTx, yi is a bounded linear functional on X . Therefore, according to Riesz’s representation theorem, there exists a unique zy ∈ X such that

hTx, yi = hx, zyi , x ∈ X . ∗ We set T y = zy. ∗ We now show that T is unique. Suppose that ∗ ∗ hTx, yi = hx, T1yi = hx, T2yi , x ∈ X , y ∈ Y Then, we have

∗ ∗ hx, (T1 − T2)yi = 0, x ∈ X , y ∈ Y.

∗ ∗ ⊥ ∗ ∗ Therefore, (T1 − T2)y ∈ X = {0} for all y ∈ Y, which implies that T1 − T2 = 0 and, in turn, ∗ ∗ that T1 = T2 ∗ The linearity of T follows from the uniqueness and the linearity of the inner product. Furthermore, we have

∗ ∗ kT k = sup kT yk kyk=1

= sup kzyk kyk=1 = sup khT·, yik (15) kyk=1 = sup sup |hTx, yi| kyk=1 kxk=1 6 sup sup kTxk kyk (16) kyk=1 kxk=1 = kTk , (17) where (15) follows from the fact that the mapping θ in (13) is an isometry and in (16) we applied ∗∗ the Cauchy-Schwarz inequality. T = T follows immediately from (14). Furthermore, we have ∗∗ ∗ kTk = kT k 6 kT k 6 kTk , ∗ which shows that kT k = kTk.

20 ∗ Deﬁnition 32. Let X , Y be Hilbert spaces and T ∈ B(X , Y). The operator T ∈ B(Y, X ) in Theorem 16 is called the adjoint of T ∈ B(X , Y).

Deﬁnition 33. Let X be a Hilbert space and T ∈ B(X , X ). Then, T is

∗ 1. self-adjoint if T = T ;

−1 ∗ 2. unitary if T is invertible and T = T ;

∗ ∗ 3. normal if TT = T T.

Examples. 1. Every orthogonal projection P: X → X is a self-adjoint operator.

2. If T: X → X is a unitary operator then it is a linear isometry. Indeed,

2 ∗ 2 kTxk = hTx, Txi = hT Tx, xi = hId x, xi = kxk , for all x ∈ X .

The converse is not true, that is, a linear isometry may not be unitary. To see this, consider the Hilbert space X = {x = (x , x ,... ) ∈ N : P |x |2 < ∞} with 1 2 R n∈N n inner product hx, yi = P x y , and let (x , x , x ,... ) = (0, x , x ,... ) be n∈N n n T 1 2 3 1 2 the right-shift operator. Then T is a linear isometry, but it is not invertible, so it cannot be unitary.

Lemma 7. Let X be a Hilbert space and T self-adjoint. Then, we have

kTk = sup {|hTx, xi| : kxk = 1} .

Proof. Let

α = sup {|hTx, xi| : kxk = 1} .

2 From |hTx, xi| 6 kTxk kxk 6 kTk kxk it follows that α 6 kTk. It remains to show that kTk 6 α. It follows from the Cauchy Schwarz inequality that

|hTx, yi| 6 kTxk kyk , x, y ∈ X with equality if and only if Tx and y are colinear. Therefore, we can rewrite

kTxk = sup {|hTx, yi| : kyk = 1} , (18) which implies that

kTk = sup (kTxk) kxk=1 = sup {|hTx, yi| : kxk = 1, kyk = 1} = sup {|hTx, yi| : kxk = 1, kyk = 1, hTx, yi ∈ R} . (19)

Definition 34. Let X be a Hilbert space. A self-adjoint operator T ∈ B(X , X ) is 1. positive semidefinite if hTx, xi > 0 for all x ∈ X . In this case, we write T > 0. 2. positive definite if it is positive semidefinite and hTx, xi = 0 implies that x = 0. In this case, we write T > 0.

Deﬁnition 35. Let X be a Hilbert space and T ∈ B(X , X ), T > 0. We call a self-adjoint 2 operator A ∈ B(X , X ) a square root of T if T = A .

Proposition 7.1. Let X be a Hilbert space and T ∈ B(X , X ), T > 0. Then, T has a unique positive deﬁnite square root.

Deﬁnition 36. An operator T: X → Y is invertible if it is surjective and injective.

A Proof of Theorem 10

Proof of Theorem 10. We ﬁrst prove the existence of y in (a). Set

d = inf{kx − zk | z ∈ S}. (20)

From the deﬁnition of d, there exists a sequence {yn}n∈N of elements yn ∈ S such that

lim kx − ynk = d. (21) n→∞ Suppose that this sequence is Cauchy. Since X is complete, there is a y ∈ X such that limn→∞ yn = y, and since S is closed, we have y ∈ S. Finally, it follows from the continuity of k · k that kx − yk = limn→∞ kx − ynk = d. It remains to show that {yn}n∈N is Cauchy. The parallelogram law applied to (x − ym) and (x − yn) implies that

2 2 2 2 2kx − ynk + 2kx − ymk = k2x − ym − ynk + kym − ynk 2 2 = 4kx − (ym + yn)/2k + kym − ynk 2 2 > 4d + kym − ynk , (22)

22 where we used (20) and the fact that (ym + yn)/2 ∈ S in the last step. Furthermore, (21) implies that for each ε > 0 there exists an N(ε) such that kx − ynk 6 d + ε. Combining this with (22), we ﬁnd that

2 2 2 kym − ynk 6 4(d + ε) − 4d = 4ε(2d + ε).

Therefore, {yn}n∈N is Cauchy. Now let x ∈ X and y ∈ S be the unique closest point from (a). Then, for each λ ∈ F and z ∈ S, we have

2 2 kx − yk 6 kx − y + λzk = kx − yk2 + |λ|2kzk2 + hx − y, λzi + hλz, x − yi = kx − yk2 + |λ|2kzk2 + 2<(λ¯ hx − y, zi).

It follows that

¯ 2 2 −2<(λ hx − y, zi) 6 |λ| kzk , λ ∈ C, z ∈ S.

iϕz We can write hx − y, zi = |hx − y, zi| e (with ϕz ∈ {0, π} in the case of F = R). Therefore, for λ = −εeiϕz , we get

2 2 |hx − y, zi| 6 εkzk , ε > 0, z ∈ S.

Taking the limit ε → 0 gives

|hx − y, zi| = 0, z ∈ S.

Now we prove the uniqueness in (a). Suppose y1 and y2 are elements of S such that ⊥ kx − y1k = kx − y2k = d. Then since x − y1 ∈ S and y1 − y2 ∈ S, we have

2 2 2 2 d = kx − y2k = kx − y1k + ky1 − y2k 2 2 = d + ky1 − y2k ,

so y1 = y2. ⊥ Finally, we prove the uniqueness in (b). Let y1 and y2 be points in S such that x − y1 ∈ S , ⊥ x − y2 ∈ S . Since x − y1 ⊥ y1 − z for all z ∈ S, we have

2 2 2 2 d = inf kx − zk = inf kx − y1k + ky1 − zk z∈S z∈S 2 2 = kx − y1k + inf ky1 − zk z∈S 2 = kx − y1k .

Therefore kx − y1k = d, and by the same argument kx − y2k = d. By the proof of uniqueness in (a) it follows that y1 = y2.

23 B Proof of Theorem 15

Proof. If N (f) = X we have yf = 0. Suppose that N (f) 6= X . Since f is continuous, N (f) is closed. Therefore, according to Corollary 4.1, we can decompose X = N (f) ⊕ N (f)⊥. Since N (f) 6= X , there exists a non-zero z ∈ N (f)⊥. We can rewrite any x ∈ X as

f(x) f(x) x = x − z + z f(z) f(z) | {z } ∈N (f)

Therefore for any x ∈ X there exist α ∈ F and n ∈ N (f) such that

x = αz + n, α.

Taking the inner product with z implies that

hx, zi α = . kzk2

Evaluating f on x = αz + n gives

f(z) hx, zi f(x) = αf(z) = . kzk2

2 Therefore, we can set zf = (f(z)/ kzk )z. To prove uniqueness, suppose that

f(x) = hx, y1i = hx, y2i , x ∈ X .

In particular, we have

f(y1 − y2) = hy1 − y2, y1i = hy1 − y2, y2i , which implies that ky1 − y2k = 0. Therefore, y1 = y2.

C Spectrum of self-adjoint operators

Deﬁnition 37. Let X be a Hilbert space and T: X → X . The set

σ(T) = {λ ∈ F | (T − λI) is not invertible} is called the spectrum of T.

We can split the spectrum into σ = σ1 ∪ σ2, where

σ1(T) = {λ ∈ F | (T − λI) is not injective)} σ2(T) = {λ ∈ F | (T − λI) is not surjective)}.

Lemma 8. Let X be a Hilbert space and T ∈ B(X , X ) self-adjoint. Then, we have

(a) For all x ∈ X we have hTx, xi ∈ R and σ1(T) ⊆ R;

24 (b) if λ∈ / σ1(T), then R(T − λI) = X ;

(c) if there exists an A > 0 such that k(T − λI)xk > Akxk for all x ∈ X , then λ∈ / σ(T). Proof. • Proof of (a): Let x ∈ X . Then

hTx, xi = hx, Txi = hTx, xi ∈ R.

Now let λ1 ∈ σ1(T). Then there exists an x ∈ X , x 6= 0, such that Tx = λx. Then 2 λkxk = hTx, xi ∈ R, so λ ∈ R.

⊥ ⊥ • Proof of (b): We show that R(T − λI) = {0}. Let x ∈ R(T − λI) . Then, we have

0 = h(T − λI)y, xi

= y, (T − λI)x , y ∈ X .

Therefore, (T − λI)x = 0. If x 6= 0, this implies that λ ∈ σ1(T) ⊆ R and, in turn, that λ ∈ σ1(T), which is a contradiction. Therefore, x = 0.

• Proof of (c): it follows immediately that λ∈ / σ1(T). Therefore, it remains to show that λ∈ / σ2(T). According to (b), it remains to show that R(T − λI) is closed. Let {yn = (T − λI)xn}n∈N be a sequence in R(T − λI) such that y = limn→∞ yn. Then, we have

kyn − ymk = k(T − λI)(xn − xm)k > A k(xn − xm)k .

Therefore, {xn}n∈N is a Cauchy sequence in a Hilbert space and, therefore, converges to x. By the continuity of T − λI, we have y = (T − λI)x ∈ R(T − λI).

Lemma 9. Let X be a Hilbert space and T ∈ B(X , X ) self-adjoint. Then σ(T) ⊆ R.

Proof. Let λ = α + iβ ∈ C with β > 0. We have to show that λ∈ / σ(T). We have

2 h(T − λI)x, xi = hTx, xi −λ kxk , x ∈ X . | {z } |{z} ∈R ∈R This implies that

2 2i Im(h(T − λI)x, xi) = (λ − λ) kxk , x ∈ X = −2iβ kxk2 , x ∈ X and, in turn, that

kxk k(T − λI)xk > |h(T − λI)x, xi| 2 > |β| kxk , x ∈ X , where we used the Cauchy-Schwarz inequality in the ﬁrst step. Therefore, Lemma 8 implies that λ∈ / σ(T).

25 Theorem 17. Let T ∈ B(X , X ) self-adjoint and set

m = inf hTx, xi and M = sup hTx, xi . kxk=1 kxk=1

Then σ(T) ⊆ [m, M]. Proof. First note that x x m kxk2 kxk2 , M kxk2 , x ∈ X . 6 Tkxk kxk 6 | {z } hTx,xi Suppose that λ = m − c with c > 0. Then, the Cauchy-Schwarz inequality implies that

kxk k(T − λI)xk > |h(T − λI)x, xi| > h(T − λI)x, xi 2 > kxk (m − λ) = c kxk2 , x ∈ X .

Therefore, Lemma 8 implies that λ∈ / σ(T). Now suppose that λ = M + c with c > 0. Then, the Cauchy-Schwarz inequality implies that

kxk k(T − λI)xk > |h(T − λI)x, xi| > − h(T − λI)x, xi 2 > kxk (λ − M) = c kxk2 , x ∈ X .

Again, Lemma 8 implies that λ∈ / σ(T).

D Pseudo-inverse of a bounded linear operator

Let X and Y be Hilbert spaces. It is often desirable to ﬁnd some kind of “inverse” for an operator T: X → Y that is not invertible in the strict sense.

Theorem 18 (Inverse mapping theorem). Let X , Y be Banach spaces. If T ∈ B(X , Y) is −1 invertible, then its inverse T ∈ B(Y, X ). Consequently, T(S) is closed if and only if S is closed.

Theorem 19. Let X , Y be Hilbert spaces and T ∈ B(X , Y). Then, we have ∗ ⊥ R(T) = N (T ) (23) ∗ ⊥ N (T) = R(T ) . (24)

Proof. We ﬁrst show (23). If y ∈ R(T), there exists a z ∈ X such that y = Tz. Then, we have ∗ ∗ hy, y˜i = hTz, y˜i = hz, T y˜i = 0, y˜ ∈ N (T ),

∗ ⊥ ∗ ⊥ ∗ ⊥ which implies that R(T) ⊆ N (T ) . Since N (T ) is closed, it follows that R(T) ⊆ N (T ) . ⊥ On the other hand, if y ∈ R(T) , we have ∗ 0 = hTx, yi = hx, T yi , x ∈ X ,

26 ∗ ⊥ ∗ which implies that T y = 0. This means that R(T) ⊆ N (T ). By taking the complement of ∗ ⊥ ⊥⊥ ∗ this relation we get N (T ) ⊆ R(T) = R(T). To prove (24), we ﬁrst apply (23) to T , use ∗∗ T = T, and apply complements.

An equivalent formulation of this theorem is that if T ∈ B(X , Y) for two Hilbert spaces X , Y, then Y can be written as the orthogonal direct sum

∗ Y = R(T) ⊕ N (T ).

The equation y = Tx has a solution x ∈ X if and only if y ∈ R(T). Therefore, if T ∈ B(X , Y) ∗ has closed range, the equation y = Tx has a solution x ∈ X if and only if y ⊥ N (T ).

Theorem 20. Let X , Y be Hilbert spaces and T ∈ B(X , Y). Then, R(T) is closed if and only ∗ if R(T ) is closed.

∗ Proof. We show that R(T ) closed implies that R(T) is closed. We have the following situation:

∗ T ⊥ T ∗ X −→ (R(T) ⊕ R(T) ) −→ R(T ) ⊆ Y. Using (23), if follows that the mapping

∗ U: R(T) → R(T ) y 7→ T ∗(y)

∗ ∗ is a bounded invertible linear operator. Since R(R ) is closed in X which is complete, R(T ) is itself complete and hence a Hilbert space in its own right. Therefore, the adjoint

∗ ∗ U : R(T ) → R(T)

∗−1 −1∗ exists and is a bounded invertible linear operator (with inverse U = U ). We now show that R(T) ⊆ R(T), which implies that R is closed. Let y ∈ R(T) and denote by P the orthogonal ∗ ∗ projection onto R(T). Then, y = U (x) for some x ∈ R(T ), and we have

hy, zi = hy, Pzi ∗ = hU x, Pzi = hx, UPzi ∗ = hx, T Pzi ∗ = hx, T zi = hTx, zi , z ∈ Y.

Therefore, y = Tx ∈ R(T). The following lemma gives a condition that ensures the existence of a right-inverse.

Lemma 10. Let X , Y be Hilbert spaces and T ∈ B(X , Y) with closed range R(T). Then there † exists a unique operator T ∈ B(X , Y) satisfying

† ⊥ N (T ) = R(T) (25) † ⊥ R(T ) = N (T) (26) † TT y = y, y ∈ R(T). (27)

27 Proof. We have the orthogonal direct sums

⊥ X = N (T) ⊕ N (T) ⊥ Y = R(T) ⊕ R(T) .

† To prove the existence of T , we consider the linear operator

⊥ = | ⊥ : N ( ) → R( ). Te T N (T) T T

It is clear that Te is linear and bounded. Te is injective because if Tex = 0, it follows that ⊥ x ∈ N (T) ∩ N (T) = {0}. To show that Te is surjective, let y ∈ R(T). Then, there exists ⊥ an x ∈ X such that y = T(x). We can write x = u + v with u ∈ N (T) and v ∈ N (T) . −1 Therefore, y = T(u + v) = T(u) + T(v) = T(u) = Te(u) ∈ R(Te). It follows that Te exists and −1 ⊥ † Te ∈ B(R(T), N (T) ) by Theorem 18. We deﬁne the operator T as

† −1 T (y) = Te (y), y ∈ R(T) (28) † ⊥ T (y) = 0, y ∈ R(T) . (29)

† † It follows immediately that T has the desired properties. To prove uniqueness, suppose that T1 † and T2 fulﬁll Properties (25)–(27) and let y ∈ Y. We can decompose y = w + z with w ∈ R(T) ⊥ and z ∈ R(T) . Then, we have

† † † † −1 Ti (y) = Ti (w) + Ti (z) = Ti (w) = Te (w), i = 1, 2.

† † Therefore, T1 = T2.

Deﬁnition 38 (Pseudo-inverse). Let X , Y be Hilbert spaces and T ∈ B(X , Y) with closed † range R(T). The unique operator T ∈ B(X , Y) satisfying Properties (25)–(27) is called the † pseudo-inverse of T .

† Proposition D.1 (Properties of T ). Let X , Y be Hilbert spaces and T ∈ B(X , Y) with closed range R(T). Then, the following properties hold.

† (a) The orthogonal projection of Y onto R(T) is given by TT .

† † (b) The orthogonal projection of X onto R(T ) is given by T T.

∗ ∗ † † ∗ (c) T has closed range and (T ) = (T ) .

† † ∗ ∗ −1 (d) On R(T), the operator T is given explicitly by T = T (TT ) .

(e) If T is surjective, then given y ∈ Y, the equation Tx = y has a unique solution of minimal † norm, namely x = T y.

Proof. • (a) Follows from Theorem 14 and Properties (25) and (27);

† † ⊥ † • (b): Property (26) implies that T Tx = 0 for all x ∈ R(T ) . Now let x ∈ R(T ). Then, † ⊥ x = T y for some y ∈ Y. Decomposing y = u + v with u ∈ R(T) and v ∈ R(T) , Property † † † † † (25) implies that x = T u. Therefore, Property (26) implies that T Tx = T TT u = T u = x. † † Theorem 14 now shows that T T is an orthogonal projection onto R(T );

28 ∗ ∗† • (c): It follows from Theorem 20 that T has closed range. This implies that T is well defined and is the unique operator satisfying ∗† ∗ ⊥ N (T ) = R(T ) ∗† ∗ ⊥ R(T ) = N (T ) ∗ ∗† ∗ T T x = x, x ∈ R(T ). †∗ Furthermore, it follows from (25)–(27) and (23) together with (24), that T satisfies the ∗ † † ∗ same properties. Therefore, (T ) = (T ) ; ∗ ⊥ ∗ ⊥ • It follows from R(T) = N (T ) and N (T) = R(T ) that the operator U : R(T) → R(T), ∗ y 7→ TT y is invertible and the operator ∗ ∗ −1 ⊥ T (TT ) on R(T) and 0 on R(T) fulfills (25)–(27);

† † • x = T y is a solution of y = Tx, i.e., y = TT y. Suppose that y = Tz for some z ∈ X and † † ⊥ † set e = T y − z. It follows that e ∈ N (T). Since T y ∈ N (T) , we have e ⊥ T y, which 2 † 2 † 2 2 implies that kzk = T y − e = T y + kek , which is minimal for e = 0.

E Application to frame theory

∞ Let {fk}k=0 be a frame with frame bounds 0 < A 6 B < ∞ for a Hilbert space X . It has been seen in the lecture that the frame operator S: X → X P∞ x 7→ k=0 hx, fki fk −1 ∞ is bounded, invertible, self-adjoint, and positive, and that {S fk}k=0 is a frame with frame −1 −1 ∞ bounds 0 < B 6 A < ∞. In fact, {Tfk}k=0 is a frame for a larger class of operators T, as stated in the following theorem.

∞ Theorem 21. Let X be a Hilbert spaces and T ∈ B(X , X ) with closed range R(T) and {fk}k=0 be a frame with ∞ 2 X 2 2 A kxk 6 |hx, fki| 6 B kxk , x ∈ X . k=0 ∞ ∞ Then, {Tfk}k=0 is a frame in span{Tfk}k=0 with ∞ † −2 2 X 2 2 2 ∞ A T kyk 6 |hy, Tfki| 6 B kTk kyk , y ∈ span{Tfk}k=0. (30) k=0 Proof. In order to prove Theorem 21, we make use of the following lemma, which shows that it is enough to check the frame condition on a dense set.

∞ Lemma 11. Suppose that {fk}k=0 is a sequence of elements in X and there exist constants 0 < A B < ∞ such that 6 ∞ 2 X 2 2 A kxk 6 |hx, fki| 6 B kxk (31) k=0 for all x in a dense subset of X . Then (31) holds for all x ∈ X .

29 The upper frame bound in (30) follows from

∞ ∞ X 2 X ∗ 2 ∗ 2 2 2 |hy, Tfki| = |hT y, fki| 6 B kT yk 6 B kTk kyk , y ∈ X . k=0 k=0

∞ Now we prove that the lower frame bound in (30) holdes for all y ∈ span{Tfk}k=0. We know † that the operator TT is the orthogonal projection onto R(T) and, in particular, it is self-adjoint. ∞ Let y ∈ span{Tfk}k=0. Then, y ∈ R(T), and we obtain that

† † ∗ † ∗ ∗ y = (TT )y = (TT ) y = (T ) T y.

This gives

2 † ∗ 2 ∗ 2 kyk 6 k(T ) k kT yk (32) † 2 ∗ 2 = k(T )k kT yk . (33)

Furthermore, we have

∞ ∞ 1 X 1 X k ∗yk2 |h ∗y, f i|2 = |hy, f i|2 . (34) T 6 A T k A T k k=0 k=0

Plugging (34) into (32) gives

† 2 ∞ T X kyk2 |hy, f i|2 , 6 A T k k=0

∞ which shows that the lower frame condition is satisﬁed for all y ∈ span{Tfk}k=0. Using Lemma 11, ∞ ∞ we can conclude that {Tfk}k=0 forms a frame for span{Tfk}k=0.

In the special case where T is a surjective bounded operator, T has closed range. Furthermore, ∞ every y ∈ Y can be written as y = Tx, where x ∈ X . Since {fk}k=0 is a frame, one has

∞ X −1 x = x, S fk fk, k=0 and it follows that ∞ X −1 y = x, S fk Tfk, k=0 ∞ which shows that Y = span{Tfk}k=0. This result leads to the following corollary.

∞ Corollary E.1. Assume that {fk}k=0 is a frame for X with frame bounds 0 < A 6 B < ∞ ∞ and that T: X → Y is a bounded surjective operator. Then {Tfk}k=0 is a frame for Y with frame † −2 2 bounds A T , B kTk .

As a consequence, if we have at our disposal a frame, we can construct many other frames by applying surjective operators to it.

30 References

[1] J. K. Hunter and B. Nachtergaele, Applied Analysis. Singapore: World Scientiﬁc Publishing, 2005.

[2] S. B. Damelin and W. Miller, The Mathematics of Signal Processing, ser. Cambridge texts in applied mathematics. Cambridge, UK: Cambridge University Press, 2012.

[3] E. M. Stein and R. Sharachi, Real Analysis: Measure Theory, Integration, and Hilbert Spaces, ser. Princeton lectures in analysis. Princeton, NJ, USA: Princeton University Press, 2005.

[4] O. Christensen, An Introduction to Frames and Riesz Bases, ser. Applied and Numerical Harmonic Analysis. Boston, MA, USA: Birkhäuser, 2003.

[5] C. Heil, “Operators on Hilbert spaces,” Functional analysis lecture notes, February 2006.