Outline Vector Spaces Hilbert Spaces Matrices Linear Operators

Functional Analysis Review

Lorenzo Rosasco –slides courtesy of Andre Wibisono

9.520: Statistical Learning Theory and Applications

February 13, 2012

L. Rosasco Functional Analysis Review Outline Vector Spaces Hilbert Spaces Matrices Linear Operators

1 Vector Spaces

2 Hilbert Spaces

3 Matrices

4 Linear Operators

L. Rosasco Functional Analysis Review n Example: R , space of polynomials, space of functions.

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Vector Space

A vector space is a set V with binary operations

+: V × V → V and · : R × V → V

such that for all a, b ∈ R and v, w, x ∈ V: 1 v + w = w + v 2 (v + w) + x = v + (w + x) 3 There exists 0 ∈ V such that v + 0 = v for all v ∈ V 4 For every v ∈ V there exists −v ∈ V such that v + (−v) = 0 5 a(bv) = (ab)v 6 1v = v 7 (a + b)v = av + bv 8 a(v + w) = av + aw

L. Rosasco Functional Analysis Review Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Vector Space

A vector space is a set V with binary operations

+: V × V → V and · : R × V → V

such that for all a, b ∈ R and v, w, x ∈ V: 1 v + w = w + v 2 (v + w) + x = v + (w + x) 3 There exists 0 ∈ V such that v + 0 = v for all v ∈ V 4 For every v ∈ V there exists −v ∈ V such that v + (−v) = 0 5 a(bv) = (ab)v 6 1v = v 7 (a + b)v = av + bv 8 a(v + w) = av + aw

n Example: R , space of polynomials, space of functions.

L. Rosasco Functional Analysis Review 1 hv, wi = hw, vi

2 hav + bw, xi = ahv, xi + bhw, xi 3 hv, vi > 0 and hv, vi = 0 if and only if v = 0. v, w ∈ V are orthogonal if hv, wi = 0. Given W ⊆ V, we have V = W ⊕ W⊥, where W⊥ = { v ∈ V | hv, wi = 0 for all w ∈ W }. 1/2 1/2 Cauchy-Schwarz inequality: hv, wi 6 hv, vi hw, wi .

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Inner Product

An inner product is a function h·, ·i: V × V → R such that for all a, b ∈ R and v, w, x ∈ V:

L. Rosasco Functional Analysis Review v, w ∈ V are orthogonal if hv, wi = 0. Given W ⊆ V, we have V = W ⊕ W⊥, where W⊥ = { v ∈ V | hv, wi = 0 for all w ∈ W }. 1/2 1/2 Cauchy-Schwarz inequality: hv, wi 6 hv, vi hw, wi .

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Inner Product

An inner product is a function h·, ·i: V × V → R such that for all a, b ∈ R and v, w, x ∈ V:

1 hv, wi = hw, vi

2 hav + bw, xi = ahv, xi + bhw, xi 3 hv, vi > 0 and hv, vi = 0 if and only if v = 0.

L. Rosasco Functional Analysis Review Given W ⊆ V, we have V = W ⊕ W⊥, where W⊥ = { v ∈ V | hv, wi = 0 for all w ∈ W }. 1/2 1/2 Cauchy-Schwarz inequality: hv, wi 6 hv, vi hw, wi .

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Inner Product

An inner product is a function h·, ·i: V × V → R such that for all a, b ∈ R and v, w, x ∈ V:

1 hv, wi = hw, vi

2 hav + bw, xi = ahv, xi + bhw, xi 3 hv, vi > 0 and hv, vi = 0 if and only if v = 0. v, w ∈ V are orthogonal if hv, wi = 0.

L. Rosasco Functional Analysis Review 1/2 1/2 Cauchy-Schwarz inequality: hv, wi 6 hv, vi hw, wi .

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Inner Product

An inner product is a function h·, ·i: V × V → R such that for all a, b ∈ R and v, w, x ∈ V:

1 hv, wi = hw, vi

2 hav + bw, xi = ahv, xi + bhw, xi 3 hv, vi > 0 and hv, vi = 0 if and only if v = 0. v, w ∈ V are orthogonal if hv, wi = 0. Given W ⊆ V, we have V = W ⊕ W⊥, where W⊥ = { v ∈ V | hv, wi = 0 for all w ∈ W }.

L. Rosasco Functional Analysis Review Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Inner Product

An inner product is a function h·, ·i: V × V → R such that for all a, b ∈ R and v, w, x ∈ V:

1 hv, wi = hw, vi

2 hav + bw, xi = ahv, xi + bhw, xi 3 hv, vi > 0 and hv, vi = 0 if and only if v = 0. v, w ∈ V are orthogonal if hv, wi = 0. Given W ⊆ V, we have V = W ⊕ W⊥, where W⊥ = { v ∈ V | hv, wi = 0 for all w ∈ W }. 1/2 1/2 Cauchy-Schwarz inequality: hv, wi 6 hv, vi hw, wi .

L. Rosasco Functional Analysis Review A norm is a function k · k: V → R such that for all a ∈ R and v, w ∈ V:

1 kvk > 0, and kvk = 0 if and only if v = 0 2 kavk = |a| kvk 3 kv + wk 6 kvk + kwk

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Norm

Can define norm from inner product: kvk = hv, vi1/2.

L. Rosasco Functional Analysis Review Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Norm

A norm is a function k · k: V → R such that for all a ∈ R and v, w ∈ V:

1 kvk > 0, and kvk = 0 if and only if v = 0 2 kavk = |a| kvk 3 kv + wk 6 kvk + kwk Can define norm from inner product: kvk = hv, vi1/2.

L. Rosasco Functional Analysis Review A metric is a function d: V × V → R such that for all v, w, x ∈ V:

1 d(v, w) > 0, and d(v, w) = 0 if and only if v = w 2 d(v, w) = d(w, v) 3 d(v, w) 6 d(v, x) + d(x, w)

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Metric

Can define metric from norm: d(v, w) = kv − wk.

L. Rosasco Functional Analysis Review Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Metric

A metric is a function d: V × V → R such that for all v, w, x ∈ V:

1 d(v, w) > 0, and d(v, w) = 0 if and only if v = w 2 d(v, w) = d(w, v) 3 d(v, w) 6 d(v, x) + d(x, w) Can define metric from norm: d(v, w) = kv − wk.

L. Rosasco Functional Analysis Review An orthonormal basis is a basis that is orthogonal (hvi, vji = 0 for i 6= j) and normalized (kvik = 1).

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Basis

B = {v1,..., vn} is a basis of V if every v ∈ V can be uniquely decomposed as

v = a1v1 + ··· + anvn for some a1,..., an ∈ R.

L. Rosasco Functional Analysis Review Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Basis

B = {v1,..., vn} is a basis of V if every v ∈ V can be uniquely decomposed as

v = a1v1 + ··· + anvn for some a1,..., an ∈ R. An orthonormal basis is a basis that is orthogonal (hvi, vji = 0 for i 6= j) and normalized (kvik = 1).

L. Rosasco Functional Analysis Review Outline Vector Spaces Hilbert Spaces Matrices Linear Operators

1 Vector Spaces

2 Hilbert Spaces

3 Matrices

4 Linear Operators

L. Rosasco Functional Analysis Review Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Hilbert Space, overview

Goal: to understand Hilbert spaces (complete inner product spaces) and to make sense of the expression

f = hf, φiiφi, f ∈ H ∞ i=1 X Need to talk about:

1 Cauchy sequence

2 Completeness

3 Density

4 Separability

L. Rosasco Functional Analysis Review (xn)n∈N is a Cauchy sequence if for every  > 0 there exists N ∈ N such that kxm − xnk <  whenever m, n > N. Every convergent sequence is a Cauchy sequence (why?)

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Cauchy Sequence

Recall: limn→ xn = x if for every  > 0 there exists N ∈ N such that kx − xnk <  whenever n > N. ∞

L. Rosasco Functional Analysis Review Every convergent sequence is a Cauchy sequence (why?)

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Cauchy Sequence

Recall: limn→ xn = x if for every  > 0 there exists N ∈ N such that kx − xnk <  whenever n > N. ∞ (xn)n∈N is a Cauchy sequence if for every  > 0 there exists N ∈ N such that kxm − xnk <  whenever m, n > N.

L. Rosasco Functional Analysis Review Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Cauchy Sequence

Recall: limn→ xn = x if for every  > 0 there exists N ∈ N such that kx − xnk <  whenever n > N. ∞ (xn)n∈N is a Cauchy sequence if for every  > 0 there exists N ∈ N such that kxm − xnk <  whenever m, n > N. Every convergent sequence is a Cauchy sequence (why?)

L. Rosasco Functional Analysis Review Examples:

1 Q is not complete. 2 R is complete (axiom). n 3 R is complete. 4 Every finite dimensional normed vector space (over R) is complete.

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Completeness

A normed vector space V is complete if every Cauchy sequence converges.

L. Rosasco Functional Analysis Review Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Completeness

A normed vector space V is complete if every Cauchy sequence converges. Examples:

1 Q is not complete. 2 R is complete (axiom). n 3 R is complete. 4 Every finite dimensional normed vector space (over R) is complete.

L. Rosasco Functional Analysis Review Examples:

n 1 R 2 Every finite dimensional inner product space.

3 2 `2 = {(an)n=1 | an ∈ R, n=1 an < } ∞ ∞1 2 4 L2([0, 1]) = {f:[0, 1] → RP| f(x) dx < } 0 ∞ R ∞

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Hilbert Space

A Hilbert space is a complete inner product space.

L. Rosasco Functional Analysis Review Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Hilbert Space

A Hilbert space is a complete inner product space. Examples:

n 1 R 2 Every finite dimensional inner product space.

3 2 `2 = {(an)n=1 | an ∈ R, n=1 an < } ∞ ∞1 2 4 L2([0, 1]) = {f:[0, 1] → RP| f(x) dx < } 0 ∞ R ∞

L. Rosasco Functional Analysis Review Examples:

1 Q is dense in R. n n 2 Q is dense in R . 3 Weierstrass approximation theorem: polynomials are dense in continuous functions (with the supremum norm, on compact domains).

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Density

Y is dense in X if Y = X.

L. Rosasco Functional Analysis Review Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Density

Y is dense in X if Y = X. Examples:

1 Q is dense in R. n n 2 Q is dense in R . 3 Weierstrass approximation theorem: polynomials are dense in continuous functions (with the supremum norm, on compact domains).

L. Rosasco Functional Analysis Review Examples:

1 R is separable. n 2 R is separable.

3 `2, L2([0, 1]) are separable.

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Separability

X is separable if it has a countable dense subset.

L. Rosasco Functional Analysis Review Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Separability

X is separable if it has a countable dense subset. Examples:

1 R is separable. n 2 R is separable.

3 `2, L2([0, 1]) are separable.

L. Rosasco Functional Analysis Review Examples:

1 Basis of `2 is (1, 0, . . . , ), (0, 1, 0, . . . ), (0, 0, 1, 0, . . . ),...

2 Basis of L2([0, 1]) is 1, 2 sin 2πnx, 2 cos 2πnx for n ∈ N

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Orthonormal Basis

A Hilbert space has a countable orthonormal basis if and only if it is separable. Can write:

f = hf, φiiφi for all f ∈ H. ∞ i=1 X

L. Rosasco Functional Analysis Review Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Orthonormal Basis

A Hilbert space has a countable orthonormal basis if and only if it is separable. Can write:

f = hf, φiiφi for all f ∈ H. ∞ i=1 X

Examples:

1 Basis of `2 is (1, 0, . . . , ), (0, 1, 0, . . . ), (0, 0, 1, 0, . . . ),...

2 Basis of L2([0, 1]) is 1, 2 sin 2πnx, 2 cos 2πnx for n ∈ N

L. Rosasco Functional Analysis Review Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Maps

Next we are going to review basic properties of maps on a Hilbert space.

functionals: Ψ : H → R linear operators A : H → H, such that A(af + bg) = aAf + bAg, with a, b ∈ R and f, g ∈ H.

L. Rosasco Functional Analysis Review Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Representation of Continuous Functionals

Let H be a Hilbert space and g ∈ H, then

Ψg(f) = hf, gi , f ∈ H

is a continuous linear functional. Riesz representation theorem The theorem states that every continuous linear functional Ψ can be written uniquely in the form,

Ψ(f) = hf, gi

for some appropriate element g ∈ H.

L. Rosasco Functional Analysis Review Outline Vector Spaces Hilbert Spaces Matrices Linear Operators

1 Vector Spaces

2 Hilbert Spaces

3 Matrices

4 Linear Operators

L. Rosasco Functional Analysis Review m×n > n×m If A ∈ R , the transpose of A is A ∈ R satisfying > > > > m n hAx, yiR = (Ax) y = x A y = hx, A yiR n m for every x ∈ R and y ∈ R . A is symmetric if A> = A.

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Matrix

m n Every linear operator L: R → R can be represented by an m × n matrix A.

L. Rosasco Functional Analysis Review A is symmetric if A> = A.

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Matrix

m n Every linear operator L: R → R can be represented by an m × n matrix A.

m×n > n×m If A ∈ R , the transpose of A is A ∈ R satisfying > > > > m n hAx, yiR = (Ax) y = x A y = hx, A yiR n m for every x ∈ R and y ∈ R .

L. Rosasco Functional Analysis Review Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Matrix

m n Every linear operator L: R → R can be represented by an m × n matrix A.

m×n > n×m If A ∈ R , the transpose of A is A ∈ R satisfying > > > > m n hAx, yiR = (Ax) y = x A y = hx, A yiR n m for every x ∈ R and y ∈ R . A is symmetric if A> = A.

L. Rosasco Functional Analysis Review Symmetric matrices have real eigenvalues. Spectral Theorem: Let A be a symmetric n × n matrix. n Then there is an orthonormal basis of R consisting of the eigenvectors of A. Eigendecomposition: A = VΛV>, or equivalently, n > A = λivivi . i=1 X

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Eigenvalues and Eigenvectors

n×n n Let A ∈ R . A nonzero vector v ∈ R is an eigenvector of A with corresponding eigenvalue λ ∈ R if Av = λv.

L. Rosasco Functional Analysis Review Spectral Theorem: Let A be a symmetric n × n matrix. n Then there is an orthonormal basis of R consisting of the eigenvectors of A. Eigendecomposition: A = VΛV>, or equivalently, n > A = λivivi . i=1 X

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Eigenvalues and Eigenvectors

n×n n Let A ∈ R . A nonzero vector v ∈ R is an eigenvector of A with corresponding eigenvalue λ ∈ R if Av = λv. Symmetric matrices have real eigenvalues.

L. Rosasco Functional Analysis Review Eigendecomposition: A = VΛV>, or equivalently, n > A = λivivi . i=1 X

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Eigenvalues and Eigenvectors

n×n n Let A ∈ R . A nonzero vector v ∈ R is an eigenvector of A with corresponding eigenvalue λ ∈ R if Av = λv. Symmetric matrices have real eigenvalues. Spectral Theorem: Let A be a symmetric n × n matrix. n Then there is an orthonormal basis of R consisting of the eigenvectors of A.

L. Rosasco Functional Analysis Review Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Eigenvalues and Eigenvectors

n×n n Let A ∈ R . A nonzero vector v ∈ R is an eigenvector of A with corresponding eigenvalue λ ∈ R if Av = λv. Symmetric matrices have real eigenvalues. Spectral Theorem: Let A be a symmetric n × n matrix. n Then there is an orthonormal basis of R consisting of the eigenvectors of A. Eigendecomposition: A = VΛV>, or equivalently, n > A = λivivi . i=1 X

L. Rosasco Functional Analysis Review Singular system:

> 2 Avi = σiui AA ui = σiui > > 2 A ui = σivi A Avi = σivi

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Singular Value Decomposition

m×n Every A ∈ R can be written as A = UΣV>, m×m m×n where U ∈ R is orthogonal, Σ ∈ R is diagonal, n×n and V ∈ R is orthogonal.

L. Rosasco Functional Analysis Review Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Singular Value Decomposition

m×n Every A ∈ R can be written as A = UΣV>, m×m m×n where U ∈ R is orthogonal, Σ ∈ R is diagonal, n×n and V ∈ R is orthogonal. Singular system:

> 2 Avi = σiui AA ui = σiui > > 2 A ui = σivi A Avi = σivi

L. Rosasco Functional Analysis Review m×n The Frobenius norm of A ∈ R is v v u m n umin{m,n} u 2 u 2 kAkF = t aij = t σi. i=1 j=1 i=1 X X X

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Matrix Norm

m×n The spectral norm of A ∈ R is q q > > kAkspec = σmax(A) = λmax(AA ) = λmax(A A).

L. Rosasco Functional Analysis Review Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Matrix Norm

m×n The spectral norm of A ∈ R is q q > > kAkspec = σmax(A) = λmax(AA ) = λmax(A A).

m×n The Frobenius norm of A ∈ R is v v u m n umin{m,n} u 2 u 2 kAkF = t aij = t σi. i=1 j=1 i=1 X X X

L. Rosasco Functional Analysis Review Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Positive Definite Matrix

m×m A real symmetric matrix A ∈ R is positive definite if

t m x Ax > 0, ∀x ∈ R .

A positive definite matrix has positive eigenvalues.

Note: for positive semi-definite matrices > is replaced by >.

L. Rosasco Functional Analysis Review Outline Vector Spaces Hilbert Spaces Matrices Linear Operators

1 Vector Spaces

2 Hilbert Spaces

3 Matrices

4 Linear Operators

L. Rosasco Functional Analysis Review A linear operator L: H1 → H2 is bounded if there exists C > 0 such that

kLfkH2 6 CkfkH1 for all f ∈ H1.

A linear operator is continuous if and only if it is bounded.

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Linear Operator

An operator L: H1 → H2 is linear if it preserves the linear structure.

L. Rosasco Functional Analysis Review A linear operator is continuous if and only if it is bounded.

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Linear Operator

An operator L: H1 → H2 is linear if it preserves the linear structure.

A linear operator L: H1 → H2 is bounded if there exists C > 0 such that

kLfkH2 6 CkfkH1 for all f ∈ H1.

L. Rosasco Functional Analysis Review Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Linear Operator

An operator L: H1 → H2 is linear if it preserves the linear structure.

A linear operator L: H1 → H2 is bounded if there exists C > 0 such that

kLfkH2 6 CkfkH1 for all f ∈ H1.

A linear operator is continuous if and only if it is bounded.

L. Rosasco Functional Analysis Review A bounded linear operator L: H1 → H2 is compact if the image of the unit ball in H1 has compact closure in H2.

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Adjoint and Compactness

The adjoint of a bounded linear operator L: H1 → H2 is a ∗ bounded linear operator L : H2 → H1 satisfying ∗ hLf, giH2 = hf, L giH1 for all f ∈ H1, g ∈ H2. L is self-adjoint if L∗ = L. Self-adjoint operators have real eigenvalues.

L. Rosasco Functional Analysis Review Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Adjoint and Compactness

The adjoint of a bounded linear operator L: H1 → H2 is a ∗ bounded linear operator L : H2 → H1 satisfying ∗ hLf, giH2 = hf, L giH1 for all f ∈ H1, g ∈ H2. L is self-adjoint if L∗ = L. Self-adjoint operators have real eigenvalues.

A bounded linear operator L: H1 → H2 is compact if the image of the unit ball in H1 has compact closure in H2.

L. Rosasco Functional Analysis Review Eigendecomposition:

L = λihφi, ·iφi. ∞ i=1 X

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Spectral Theorem for Compact Self-Adjoint Operator

Let L: H → H be a compact self-adjoint operator. Then there exists an orthonormal basis of H consisting of the eigenfunctions of L, Lφi = λiφi

and the only possible limit point of λi as i → is 0.

∞

L. Rosasco Functional Analysis Review Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Spectral Theorem for Compact Self-Adjoint Operator

Let L: H → H be a compact self-adjoint operator. Then there exists an orthonormal basis of H consisting of the eigenfunctions of L, Lφi = λiφi

and the only possible limit point of λi as i → is 0.

Eigendecomposition: ∞

L = λihφi, ·iφi. ∞ i=1 X

L. Rosasco Functional Analysis Review