L2 Space of Random Functions L2 Space of Random Functions Marek Dvoˇr´Ak

L2 space of random functions L2 space of random functions Marek Dvoˇr´ak

Hilbert Spaces Definition of Hilbert Space Marek Dvoˇrák Orthogonality Compact operators Matematicko-fyzikáln´ıfakulta Univerzity Karlovy Riesz representation theorem Adjoint operators Singular values Hilbert-Schmidt operators Stochastic Modeling in Economics and Finance Space L2 in more detail 11.11.2013 Integral operator Covariance operator Asymptotic properties in L2 Estimation of mean and covariance functions 1/56 Outline

L2 space of 1 Introduction to Hilbert Spaces random functions Deﬁnition of Hilbert Space

Marek Dvoˇr´ak Orthogonality 2 Compact linear operators Hilbert Spaces Deﬁnition of Riesz representation theorem Hilbert Space Orthogonality Adjoint operators Compact Singular values operators Riesz Hilbert-Schmidt operators representation theorem 2 Adjoint 3 Space L in more detail operators Singular values Integral operator Hilbert-Schmidt operators Covariance operator 2 Space L2 in Asymptotic properties in L more detail Integral operator 4 Estimation of mean and covariance functions Covariance operator Estimation of mean Asymptotic properties in L2 Estimation of covariance Estimation of 5 Estimating eigenelements of the covariance operator mean and covariance functions 2/56 Outline

L2 space of random 1 Introduction to Hilbert Spaces functions Deﬁnition of Hilbert Space Marek Dvoˇr´ak Orthogonality

Hilbert Spaces 2 Compact linear operators Deﬁnition of Hilbert Space Riesz representation theorem Orthogonality Adjoint operators Compact Singular values operators Hilbert-Schmidt operators Riesz representation 2 theorem 3 Space L in more detail Adjoint operators Integral operator Singular values Covariance operator Hilbert-Schmidt 2 operators Asymptotic properties in L Space L2 in more detail 4 Estimation of mean and covariance functions Integral operator Estimation of mean Covariance operator Estimation of covariance Asymptotic properties in L2 5 Estimating eigenelements of the covariance operator Estimation of mean and covariance functions 3/56 Hilbert Space

L2 space of random functions

Marek Dvoˇr´ak Deﬁnition

Hilbert Spaces Let H be a complex vector space. An inner product on H is a Deﬁnition of C C Hilbert Space function , : H H such that a, b , x, y, z H Orthogonality h· ·i × → ∀ ∈ ∀ ∈ Compact ax + by, z = a x, z + b y, z , operators h i h i h i Riesz x, y = y, x , representation h i h i theorem 2 2 Adjoint x := x, x 0 with equality x =0 iﬀ x = 0. operators k k h i≥ k k Singular values Hilbert-Schmidt A complete inner product space H is called Hilbert space (= operators HS). Space L2 in more detail Integral operator Throughout the talk: separable Hilbert spaces (= spaces with Covariance operator Asymptotic countable orthonormal bases). properties in L2 Estimation of mean and covariance functions 4/56 Properties of the inner product

L2 space of random functions

Marek Dvoˇr´ak

Hilbert Spaces Deﬁnition of Hilbert Space Inner product , Orthogonality h· ·i 1 fulﬁlls Cauchy-Schwarz inequality: Compact operators Riesz representation x, y H : x, y x y , theorem ∀ ∈ |h i|≤k k · k k Adjoint operators Singular values 2 is a continuous mapping H H C. Hilbert-Schmidt operators × → Space L2 in more detail Integral operator Covariance operator Asymptotic properties in L2 Estimation of mean and covariance functions 5/56 Examples

L2 space of random functions Cn Marek Dvoˇrák 1. Standard inner product on for x =(x1,..., xn), y =(y1,..., yn) Hilbert Spaces Definition of Hilbert Space n Orthogonality x, y := xi yi Compact h i operators i=1 Riesz X representation theorem 2 2 Adjoint 2. Space L = L ([0, 1]) of measurable real-valued function f operators 1 2 Singular values defined on [0, 1] satisfying f (t)dt < with the inner Hilbert-Schmidt 0 ∞ operators product Space L2 in R 1 more detail Integral operator f , g = f (t) g(t)dt. Covariance h i 0 operator Z Asymptotic properties in L2 Estimation of mean and covariance functions 6/56 Orthogonality

L2 space of random functions

Marek Dvoˇrák Definition Hilbert Spaces Definition of Hilbert Space x, y H are orthogonal (x y) iff x, y = 0. Orthogonality ∈ ⊥ h i Compact U H is orthogonal iff x, y U, x = y: x, y = 0. operators ⊂ ∀ ∈ 6 h i Riesz representation U, V H are orthogonal (U V ) iff x U, y V : theorem ⊂ ⊥ ∀ ∈ ∀ ∈ Adjoint operators x, y = 0. Singular values h i Hilbert-Schmidt An orthonormal basis U of HS is an orthogonal set of operators Space L2 in elements ui U such that ui = 1. more detail ∈ k k Integral operator Covariance operator Asymptotic properties in L2 Estimation of mean and covariance functions 7/56 Parseval identity

L2 space of random functions Lemma (Parseval identity) Marek Dvoˇr´ak In the separable HS H with inner product , and orthonormal Hilbert Spaces h· ·i Deﬁnition of basis ej , for every x H holds the following: Hilbert Space { } ∈ Orthogonality ∞ Compact 2 2 operators x, ej = x . Riesz h i k k representation j=1 theorem X Adjoint operators Singular values Hilbert-Schmidt operators Remark Space L2 in more detail This is analogy to the Pythagorean theorem: The sum of the Integral operator Covariance operator squares of the components of a vector with respect to some Asymptotic properties in L2 orthonormal basis is equal to the squared length of the vector. Estimation of mean and covariance functions 8/56 Outline

L2 space of random 1 Introduction to Hilbert Spaces functions Deﬁnition of Hilbert Space Marek Dvoˇr´ak Orthogonality

L2 space of random functions

Marek Dvoˇr´ak

Hilbert Spaces Deﬁnition of Hilbert Space Let H = H( , ) be a sep. HS with inner product that Orthogonality h· ·i generates the norm and let H be the space of Compact k · k L⊂ operators bounded linear operators on H with the norm Riesz representation theorem Adjoint operators Ψ L := sup Ψ(x) . Singular values k k x∈H: kxk≤1 k k Hilbert-Schmidt operators Space L2 in more detail Integral operator Covariance operator Asymptotic properties in L2 Estimation of mean and covariance functions 10/56 Riesz representation theorem

L2 space of random functions Marek Dvoˇr´ak Theorem

Hilbert Spaces If Ψ is bounded linear operator on a HS H then there exists Deﬁnition of Hilbert Space some g H such that for every f H Orthogonality ∈ ∈ Compact operators Ψ(f )= f , g . Riesz h i representation theorem Adjoint Moreover Ψ = g where the left norm is an operator norm operators k k k k Singular values and g is a HS-norm. Hilbert-Schmidt operators k k Space L2 in more detail Remark Integral operator Covariance Holds for separable and non-separable HS. operator Asymptotic properties in L2 Estimation of mean and covariance functions 11/56 (Self) adjoint operator

L2 space of random Definition functions Let Ψ : H1 H2 is bounded linear operator. The adjoint Marek Dvoˇrák → operator is defined as follows: For each y H1 the linear Hilbert Spaces ∈ Definition of functional Ψ(x), y is bounded on H1. Hence, according to Hilbert Space h i Orthogonality Riesz, for every y H there exists unique z H such that ∈ 2 ∈ 1 Compact operators Riesz Ψ(x), y = x, z x H1. representation h i h i∀ ∈ theorem Adjoint ∗ ∗ operators Define an operator Ψ : H2 H1 : Ψ (y) := z. Such Singular values ∗ → Hilbert-Schmidt operator Ψ is called adjoint. operators Space L2 in more detail Remark Integral operator Covariance ∗ operator It holds that Ψ(x), y = x, Ψ (y) x H1 and y H2. If Ψ Asymptotic h i h∗ i ∀ ∈ ∈ properties in L2 is acting on H and Ψ Ψ which means that ≡ Estimation of Ψ(x), y = x, Ψ(y) x, y H then Ψ is called self-adjoint. mean and h i h i ∀ ∈ covariance functions 12/56 Preliminary for singular values

L2 space of random functions Definition Marek Dvoˇrák Positive-definiteness: Ψ(x), x 0. h i≥ Hilbert Spaces Eigenelements: Ψ(vj )= λj vj . Definition of Hilbert Space Orthogonality Lemma Compact operators ∗ Riesz Let Ψ: H1 H2 be a compact operator. Then, Ψ Ψ is a representation → theorem compact positive definite self-adjoint operator on H1. Adjoint operators Singular values Hilbert-Schmidt operators Proof. Space L2 in Ψ∗Ψ(x), x = Ψ(x), Ψ(x) = Ψ(x) 2 0 more detail h i h i k k ≥ ⇒ Integral operator positive-definite. Covariance operator ∗ ∗ Asymptotic Ψ Ψ(x), y = Ψ(x), Ψ(y) = x, Ψ Ψ(y) properties in L2 h i h i h i ⇒ self-adjoint. Estimation of mean and covariance functions 13/56 Singular values

L2 space of random functions Definition Marek Dvoˇrák We say that operator Ψ is of finite rank if number of nonzero ∗ Hilbert Spaces eigenvalues of the operator Ψ Ψ is finite. Definition of Hilbert Space Orthogonality Definition Compact operators By definition, for j = 1, 2, ... the j-th singular value or j-th Riesz representation theorem s-number of the operator Ψ is the number Adjoint operators Singular values Hilbert-Schmidt ∗ operators sj (Ψ) := λj (Ψ Ψ) Space L2 in q more detail Integral operator where Covariance ∗ ∗ operator λ1(Ψ Ψ) λ2(Ψ Ψ) ... Asymptotic ≥ ≥ 2 properties in L is non-increasing sequence of eigenvalues of the operator Ψ∗Ψ. Estimation of mean and covariance functions 14/56 Important characteristic of compact operators

L2 space of random functions

Marek Dvoˇrák This theorem serves as a definition in the Horváth’s book:

Hilbert Spaces Theorem (Gohberg et al., Theorem VI.1.1) Deﬁnition of Hilbert Space Orthogonality A compact operator Ψ on HS H admits representation of the Compact form operators ν(Ψ) Riesz representation theorem Ψ(x)= sj (Ψ) x, vj fj , x H (1) Adjoint h i ∈ operators j=1 Singular values X Hilbert-Schmidt operators where ν(Ψ) = # j = 1, 2, ... : sj (Ψ) = 0 and Space L2 in ν(Ψ) ν(Ψ){ 6 } more detail vj j=1 , fj j=1 are orthonormal bases in H and the series in Integral operator { } { } Covariance (1) converges in the operator norm if ν(Ψ) = . operator ∞ Asymptotic properties in L2 Estimation of mean and covariance functions 15/56 Important characteristic of compact operators

L2 space of random functions Marek Dvoˇr´ak Theorem (cont.)

Hilbert Spaces Deﬁnition of Conversely, if Hilbert Space ν Orthogonality Υ(y)= λj y, vj fj , Compact h i operators j=1 Riesz X representation theorem ν , ν λ Adjoint where vj j=1 fj j=1 are orthonormal bases in H and j is operators { } { } Singular values non-increasing sequence of positive numbers which converges Hilbert-Schmidt operators to zero if ν = , then Υ is a compact operator and sj (Υ) = λj 2 ∞ Space L in are the non-zero s-numbers of Υ. more detail Integral operator Covariance operator Representation (1) is called singular value decomposition. Asymptotic properties in L2 Estimation of mean and covariance functions 16/56 Proof

L2 space of random ﬁrst part. functions Marek Dvoˇr´ak First part complicated due to the construction of the

Hilbert Spaces orthonormal bases. Deﬁnition of Hilbert Space Sketch: Orthogonality 1. From spectral theorem of self-adjoint operators: Compact ∗ ν(Ψ) 2 operators v1, v2, ... orthonormal system: Ψ Ψ= s (Ψ) , vj vj Riesz j=1 j representation ∃ h· i theorem −1 Adjoint 2. Deﬁne fj := sj Ψ(vj ) and show orthonormalityP of fj . operators { } Singular values ν(Ψ) Hilbert-Schmidt 3. Representation of x: x = j=1 x, vj vj + u, where operators ∗ h i Space L2 in u Ker(Ψ Ψ) more detail ∈ P Integral operator 4. Ψ(vj )= sj fj Covariance n operator , Asymptotic 5. Show convergence Ψn Ψ: For Ψn := j=1 sj vj fj : properties in L2 →2 2 h· i (Ψ Ψn)(x) supj>n sj x . Estimation of k − k≤ { } · k k P mean and covariance functions 17/56 Proof

L2 space of random functions second part. Marek Dvoˇrák 1. If ν < then Υ is of finite rank and hence compact. Hilbert Spaces ∞ n Definition of 2. If ν = then put Υn := j=1 λj , vj fj and show that Hilbert Space ∞ h· i Orthogonality Υn Υ. Compact → P operators 3. Since Υn is finite then Υ is compact. Riesz representation ∗ n theorem 4. It holds that Υ = j=1 λj , fj vj hence Adjoint h· i operators Singular values P ν Hilbert-Schmidt operators ∗ 2 Υ Υ= λj , vj fj Space L2 in h· i more detail j=1 Integral operator X Covariance operator df ∗ 2 Asymptotic and thus sj (Υ) = λj (Υ Υ) = λj = λj . properties in L2 Estimation of p q mean and covariance functions 18/56 Trace class operators

L2 space of random functions

Marek Dvoˇr´ak Deﬁnition

Hilbert Spaces For a Hilbert space H we define Definition of Hilbert Space ∞ Orthogonality := A : H H : A compact and sj (A) < . Compact S → ∞ operators j=1 Riesz n o representation X theorem Adjoint operators The elements of are called trace class operators. Singular values S Hilbert-Schmidt operators Definition (Hilbert-Schmidt operator) Space L2 in more detail Integral operator A compact linear operator Ψ on a Hilbert space is said to be a Covariance ∗ operator Hilbert-Schmidt operator if Ψ Ψ is a trace-class operator. Asymptotic properties in L2 Estimation of mean and covariance functions 19/56 Hilbert-Schmidt operators

L2 space of random Lemma functions The space of Hilbert-Schmidt operators is a separable HS Marek Dvoˇr´ak S with the scalar product Hilbert Spaces Deﬁnition of Hilbert Space ∞ Orthogonality Ψ , Ψ := Ψ (ei ), Ψ (ei ) (2) Compact h 1 2iS h 1 2 i operators i=1 Riesz X representation theorem Adjoint where ei is an arbitrary orthonormal base. operators { } Singular values Hilbert-Schmidt operators Remark Space L2 in more detail Value (2) does not depend on the choice of ei since Integral operator { } Covariance operator ∞ Asymptotic 2 properties in L Ψ(ei )= λj ei , vj fj , x H. Estimation of h i ∈ mean and j=1 covariance X functions 20/56 Hilbert-Schmidt operators

L2 space of random functions Lemma Marek Dvoˇr´ak 2 ! 2 It holds that Ψ = Ψ, Ψ = Ψ(ei ), Ψ(ei ) = λ . k kS h iS i h i j j Hilbert Spaces Deﬁnition of P P Hilbert Space Orthogonality Proof. Compact ∞ ∞ ∞ operators 2 Riesz Ψ S = λj ei , vj fj , λk ei , vk fk = representation k k h i h i theorem i=1 j=1 k=1 Adjoint X D X X E operators ∞ ∞ ∞ Singular values Hilbert-Schmidt operators = λj λk ei , vj ei , vk fj , fk = h ih ih i Space L2 in i=1 j=1 k=1 more detail X X X ∞ ∞ ∞ Integral operator Covariance 2 2 2 operator = λj ei , vj = λj . Asymptotic h i 2 i=1 j=1 j=1 properties in L X X X Estimation of mean and covariance functions 21/56 Hilbert-Schmidt operators

L2 space of random functions Definition (Horváth’s book) Marek Dvoˇrák

Hilbert Spaces A compact operator (admitting SVD) is called Hilbert-Schmidt Deﬁnition of ∞ 2 Hilbert Space operator, iﬀ j=1 λj < . Orthogonality ∞ Compact P operators Riesz representation theorem Adjoint operators Singular values Hilbert-Schmidt operators Space L2 in more detail Integral operator Covariance operator Asymptotic properties in L2 Estimation of mean and covariance functions 22/56 Hilbert-Schmidt operators

L2 space of random functions Definition (Horváth’s book) Marek Dvoˇrák

Hilbert Spaces A compact operator (admitting SVD) is called Hilbert-Schmidt Definition of ∞ 2 Hilbert Space operator, iff j=1 λj < . Orthogonality ∞ Compact P operators Remark Riesz representation theorem Ψ= λj , vj fj is: Adjoint j operators h· i Singular values compact operator , iff: SVD possible and λj 0 Hilbert-Schmidt P → operators ∞ λ < 2 trace-class operator , iff: is compact and j=1 j Space L in ∞ more detail Hilbert-Schmidt operator , iff Ψ∗Ψ is trace-class, i.e. Integral operator P Covariance ∞ 2 operator j=1 λj < Asymptotic ∞ 2 properties in L P Estimation of mean and covariance functions 22/56 Theorem (Properties of Hilbert-Schmidt operators)

L2 space of random functions Theorem (Gohberg et al., Theorem VIII.2.1.) Marek Dvoˇr´ak For a compact linear operator Ψ the following statements are

Hilbert Spaces equivalent: Deﬁnition of Hilbert Space Orthogonality Ψ is a Hilbert-Schmidt operator. 2 Compact j Ψ(ej ) < , for (some, any) orthonormal basis ej . operators k k ∞ { } Riesz 2 representation Pj k Ψ(ej ), ek < for (some, any) orthonormal theorem |h i| ∞ Adjoint basis ej . operators P P{ } Singular values ∞ 2 Hilbert-Schmidt sj (Ψ) < . operators j=1 ∞ Space L2 in more detail P Integral operator Proof. Covariance ∞ ∞ ∞ ∞ operator 2 Asymptotic ∗ ∗ 2 2 λj (Ψ Ψ) = h(Ψ Ψ)(ej ), ej i = kΨ(ej )k = hΨ(ej ), ek i . properties in L X X X XX j=1 j=1 j=1 j,k=1 Estimation of mean and covariance functions 23/56 Symmetric positive-deﬁnite H.-S. operators

L2 space of random functions

Marek Dvoˇrák Ψ symmetric and positive-definite admits decomposition Hilbert Spaces Definition of ∞ Hilbert Space Orthogonality Ψ(x)= λk x, vk vk x H h i ∈ Compact k=1 operators X Riesz representation theorem where vk are orthonormal and which are the eigenfunctions of Adjoint operators Ψ, i.e. Singular values ∞ Hilbert-Schmidt operators Ψ(vj )= λk vj , vk vk = λj vj . Space L2 in h i more detail k=1 Integral operator X Covariance operator Asymptotic properties in L2 Estimation of mean and covariance functions 24/56 Outline

L2 space of random 1 Introduction to Hilbert Spaces functions Deﬁnition of Hilbert Space Marek Dvoˇr´ak Orthogonality

L2 space of random functions Consider space L2 = L2([0, 1]) of measurable real-valued Marek Dvoˇrák functions f defined on [0, 1] satisfying 1 f 2(t)dt < with 0 ∞ Hilbert Spaces the inner product Definition of Hilbert Space R Orthogonality 1 Compact operators f , g = f (t) g(t)dt Riesz h i 0 representation Z theorem Adjoint operators and the induced norm Singular values Hilbert-Schmidt operators 1 2 2 Space L in f := f (t)dt. more detail k k s 0 Integral operator Z Covariance operator 2 Asymptotic Note: The space L is a separable Hilbert space. properties in L2 Estimation of mean and covariance functions 26/56 Definition of integral operator in L2

L2 space of random functions Definition Marek Dvoˇrák Integral operator Ψ L2 is defined by Hilbert Spaces Definition of 1 Hilbert Space 2 Orthogonality Ψ(x)(t) := ψ(t, s) x(s)ds, x L , Compact 0 ∈ operators Z Riesz representation where ψ( , ) is a real kernel. theorem · · Adjoint operators Singular values Hilbert-Schmidt Remark operators Space L2 in This is a generalization of matrix-vector multiplication: more detail Integral operator n Covariance operator Au Asymptotic ( )i = aij uj 2 properties in L j=1 Estimation of X mean and covariance functions 27/56 Relation with Hilbert-Schmidt operators

L2 space of random Lemma functions Integral operators Ψ are symmetric and positive Marek Dvoˇrák Hilbert-Schmidt operators, iff ψ L2 [0, 1] [0, 1] , i.e. Hilbert Spaces ∈ × Definition of Hilbert Space 1 1 Orthogonality ψ2(s, t) ds dt < Compact 0 0 ∞ operators Z Z Riesz representation theorem Proof. Adjoint operators ∞ 2 Singular values Choose an orthonormal basis ϕj j=1 of L ([0, 1]) and define Hilbert-Schmidt { } operators ai,j (t, s) := ϕi (t)ϕj (s) which is an orthonormal basis of Space L2 in L2([0, 1] [0, 1]). For proving Hilbert-Schmidt-property: more detail × Integral operator Ψ(ϕ ),ϕ 2 < . .. We have Covariance i,j i j operator |h i| ∞ Asymptotic properties in L2 P 1 1 Estimation of Ψ(ϕi ),ϕj = ψ(t, s)ϕi (s) ϕj (t)ds dt = mean and h i 0 0 covariance Z Z functions 28/56 Proof of the theorem

L2 space of random Proof cont. functions Marek Dvoˇr´ak 1 1

Hilbert Spaces = ψ(t, s) aj,i (t, s)ds dt = ψ, aj,i Deﬁnition of 0 0 h i Hilbert Space Z Z Orthogonality where the last inner product is on the space L2 [0, 1] [0, 1] . Compact × operators Then ∞ ∞ Riesz 2 representation 2 theorem Ψ(ϕi ),ϕj = ψ . Adjoint h i k k operators i=1 j=1 Singular values X X Hilbert-Schmidt operators Space L2 in more detail The Hilbert-Schmidt norm of the integral operator is Integral operator Covariance operator 1 1 Asymptotic 2 2 2 properties in L Ψ S = ψ (t, s)ds dt. Estimation of k k 0 0 mean and Z Z covariance functions 29/56 Covariance operator

L2 space of random 2 functions Let X = X (t), t [0, 1] be a random element of L { ∈ } Marek Dvoˇr´ak equipped with the Borel σ-algebra.

Hilbert Spaces Deﬁnition of Deﬁnition (square integrability) Hilbert Space Orthogonality X is integrable if Compact operators Riesz 1 representation theorem 2 Adjoint E X := E X (t)dt < . operators k k "s 0 # ∞ Singular values Z Hilbert-Schmidt operators 2 Space L in 2 more detail Then according to Riesz, there exists unique function µ L Integral operator ∈ Covariance such that operator 2 Asymptotic E y, X = y,µ for any y L . properties in L2 h i h i ∈ Estimation of mean and covariance functions 30/56 Covariance operator

L2 space of random functions Definition (Covariance operator) Marek Dvoˇrák If X is square-integrable and EX = 0, the covariance operator Hilbert Spaces CX of X is defined as Definition of Hilbert Space Orthogonality 2 C(y) := CX (y):=E X , y X , y L . Compact h i ∈ operators Riesz h i representation theorem Adjoint 2 operators If y lies in L then Singular values Hilbert-Schmidt operators 1 Space L2 in CX (y)(t) = E X , y X (t) =E X (s)y(s)ds X (t) = more detail h i " 0 · # Integral operator h i Z Covariance 1 1 operator Asymptotic = E X (t) X (s) y(s)ds =: c(t, s) y(s)ds properties in L2 0 · 0 · Estimation of Z Z mean and covariance functions 31/56 Properties of Covariance operator

L2 space of random Lemma functions If X is a square-integrable random curve then the covariance Marek Dvoˇrák operator CX is symmetric and positive definite. Hilbert Spaces Definition of Hilbert Space Proof. Orthogonality Compact c(t, s)= c(s, t) hence symmetric and operators Riesz representation 1 1 theorem Adjoint CX (y), y = c(t, s) y(t)y(s)ds dt = operators h i Singular values Z0 Z0 Hilbert-Schmidt 1 1 operators Space L2 in = E X (t)X (s) y(t)y(s)ds dt = more detail Z0 Z0 Integral operator 1 Covariance 2 operator . Asymptotic = E X (t) y(t)dt 0 2 ≥ properties in L " Z0 # Estimation of mean and covariance functions 32/56 Properties of Covariance operator

L2 space of random Remark functions The converse is not true: Not every symmetric positive-definite Marek Dvoˇrák operator in L2 is the covariance operator. Hilbert Spaces Definition of Hilbert Space Let λj , vj , j N eigenvalues and eigenfunctions of CX . Then Orthogonality ∈ Compact CX (vj )= λj vj and operators Riesz representation theorem λj = λj vj , vj = CX (vj ), vj = E X , vj X , vj = Adjoint h i h i h i operators Singular values 2 D E Hilbert-Schmidt = E X , vj . operators h i Space L2 in h i more detail By Parseval: Integral operator Covariance ∞ ∞ operator Asymptotic 2 2 2 λj = E X , vj =E X < . properties in L h i k k ∞ Estimation of j=1 j=1 mean and X X h i h i covariance functions 33/56 Properties of Covariance operator

L2 space of random functions

Marek Dvoˇr´ak

Hilbert Spaces Definition of Hilbert Space Orthogonality Lemma Compact operators Operator C (L2) is a covariance operator iff it is symmetric, Riesz representation ∈L ∞ theorem positive-definite and its eigenvalues satisfy j=1 λj < . Adjoint ∞ operators Singular values P Hilbert-Schmidt operators Space L2 in more detail Integral operator Covariance operator Asymptotic properties in L2 Estimation of mean and covariance functions 34/56 Central Limit Theorem

L2 space of random functions Theorem (Bosq (2000), Theorem 2.7) Marek Dvoˇr´ak Suppose that Xi , i N is a sequence of iid mean zero { ∈ } Hilbert Spaces random elements in a separable HS H such that Deﬁnition of Hilbert Space E[ X 2] < . Then, as N , Orthogonality k 1k ∞ → ∞ Compact operators N Riesz 1 d representation theorem Xi Z, Adjoint √N · → operators i=1 Singular values X Hilbert-Schmidt operators where Z is a Gaussian random element with the covariance Space L2 in more detail operator Integral operator Covariance operator CZ (x)= E Z, x Z = E X1, x X1 Asymptotic h i h i properties in L2 Estimation of mean and covariance functions 35/56 Law of Large Numbers

L2 space of random functions

Marek Dvoˇrák Theorem (Bosq (2000), Theorem 2.4) Hilbert Spaces Definition of Suppose that Xi , i N is a sequence of iid mean zero Hilbert Space { ∈ } Orthogonality random elements in a separable HS H such that Compact 2 operators E[ X1 ] < . Then µ = EX1 is uniquely defined by Riesz k k ∞ representation µ, x = E X , x , and, as N , theorem h i h i → ∞ Adjoint operators Singular values N Hilbert-Schmidt 1 a.s. operators Xi µ. N · → Space L2 in i=1 more detail X Integral operator Covariance operator Asymptotic properties in L2 Estimation of mean and covariance functions 36/56 Outline

L2 space of random 1 Introduction to Hilbert Spaces functions Deﬁnition of Hilbert Space Marek Dvoˇr´ak Orthogonality

L2 space of random functions

Marek Dvoˇr´ak

Hilbert Spaces Deﬁnition of Assumptions 1: Hilbert Space 2 Orthogonality The observations X1, X2, ..., XN are iid in L and have the Compact operators same distribution as X , which is assumed to be square Riesz representation integrable. theorem Adjoint operators We usually observe a sample consisting of N random curves Singular values Hilbert-Schmidt X ,..., X . Each curve is a realization of a random function X . operators 1 N Space L2 in more detail Integral operator Covariance operator Asymptotic properties in L2 Estimation of mean and covariance functions 38/56 Estimators

L2 space of random functions

Marek Dvoˇr´ak µ(t) = E X (t) , N Hilbert Spaces 1 Deﬁnition of µ(t) := Xi (t), Hilbert Space N Orthogonality i=1 Compact X operators c(bt, s) = E X (t) µ(t) X (s) µ(s) , Riesz representation − − theorem h N i Adjoint operators 1 Singular values c(t, s) := Xi (t) µ(t) Xi (s) µ(s) , Hilbert-Schmidt N − − i operators X=1 Space L2 in more detail b CX = E (X µ) , (bX µ) , b Integral operator − · − Covariance h N i operator Asymptotic 1 2 CX (x) := (Xi µ) , x (Xi µ) properties in L N − − Estimation of i=1 mean and X D E covariance b b b functions 39/56 Asymptotic result for mean

L2 space of random Lemma functions If the Assumptions 1 hold then Eµ = µ and Marek Dvoˇr´ak E µ µ 2 = (N−1). Hilbert Spaces k − k O Deﬁnition of b Hilbert Space Proof. Orthogonality b 2 Compact i and for almost t [0, 1]: EXi (t)= µ(t) and Eµ = µ in L . operators ∀ ∈ N N Riesz 2 1 1 representation E µ µ = E (Xi µ) , (Xj µ) = theorem k − k N − N b − Adjoint i=1 j=1 operators D X X E Singular values N N Hilbert-Schmidt b 1 operators µ, µ = 2 E Xi Xj = 2 N · h − − i Space L in i=1 j=1 more detail h X X i Integral operator N N Covariance 1 operator = E Xi µ, Xj µ = Asymptotic N2 · h − − i properties in L2 i=1 j=1 X X Estimation of mean and covariance functions 40/56 Asymptotic result for mean

L2 space of random functions Proof. Marek Dvoˇr´ak N N Hilbert Spaces 1 Deﬁnition of = 2 E Xi µ, Xj µ = Hilbert Space N · h − − i Orthogonality i=1 j=1 X X Compact N operators 1 2 Riesz = E Xi µ = representation 2 theorem N · k − k Adjoint i=1 operators X Singular values 1 2 Hilbert-Schmidt = 2 N E X µ = operators N · · k − k 2 Space L in 1 2 more detail = E X µ . Integral operator N · k − k Covariance operator Asymptotic properties in L2 Estimation of mean and covariance functions 41/56 Orthogonality

L2 space of random functions

Marek Dvoˇr´ak Lemma Hilbert Spaces 2 Deﬁnition of If X1, X2 L independent square integrable and EX1 = 0 then Hilbert Space ∈ Orthogonality E[ X1, X2 ] = 0. Compact h i operators Riesz representation Proof. theorem Adjoint operators Singular values 1 Hilbert-Schmidt operators E X1(t) X2(t)dt = 0. 0 Space L2 in Z more detail h i Integral operator Covariance operator Asymptotic properties in L2 Estimation of mean and covariance functions 42/56 Operator CX is Hilbert-Schmidt

L2 space of random Lemma functions Function c( , ) L2([0, 1] [0, 1]) (df: c2(t, s) < ). Marek Dvoˇr´ak · · ∈ × ∞ Hilbert Spaces Proof. R R Deﬁnition of Hilbert Space Orthogonality 1 1 2 Compact c (t, s)ds dt = operators Z0 Z0 Riesz 1 1 representation 2 theorem = E X (t) µ(t) X (s) µ(s) ds dt Adjoint − − ≤ operators Z0 Z0 Singular values 1 1 h i Hilbert-Schmidt 2 2 operators = E X (t) µ(t) E X (s) µ(s) ds dt = − · − Space L2 in Z0 Z0 more detail 2 1 Integral operator 2 Covariance = E X (t) µ(t) dt = operator − Asymptotic 0 ! 2 Z h i properties in L 2 1 Estimation of 2 2 2 2 mean and = E X (t) µ(t) dt = E X µ < . covariance 0 − ! − ∞ functions Z 43/56 h i

L2 space of random functions Marek Dvoˇr´ak Assume that the observations have mean zero, i.e.

Hilbert Spaces N Deﬁnition of Hilbert Space 1 Orthogonality c(t, s) = Xi (t) Xi (s) N · Compact i=1 operators X Riesz N representation b 1 theorem CX1,...,XN (x) = Xi , x Xi Adjoint N · h i operators i=1 Singular values X Hilbert-Schmidt b 1 operators C (x)(t) = c(t, s) x(s) ds. Space L2 in X1,...,XN more detail Z0 Integral operator Covariance b operator b Asymptotic properties in L2 Estimation of mean and covariance functions 44/56 Operator CX is Hilbert-Schmidt

L2 space of random b functions Theorem Marek Dvoˇrák If Assumptions 1 holds, E X 4 < and EX = 0 then k k ∞ 2 4 Hilbert Spaces E C S E X , Definition of k k ≤ k k Hilbert Space 2 −1 4 Orthogonality E C C N E X k − kS ≤ · k k Compact b operators Riesz b representation part 1. theorem Adjoint operators N 2 Singular values 2 1 2 Hilbert-Schmidt C = Xi , Xi X , X = operators k kS N · h ·i ≤ h ·i S Space L2 in i=1 S more detail X b ∞ ∞ Integral operator 2 2 2 4 Covariance = X , ej X X X , ej = X . operator h i ≤ k k · h i k k Asymptotic j=1 j=1 properties in L2 X X Estimation of mean and covariance functions 45/56 Sufficient condition for Hilbert-Schmidt operator

L2 space of random part 2. functions 2 Marek Dvoˇrák ∞ N 2 1 C C S = Xi , ej Xi E X , ej X = Hilbert Spaces k − k N · h i − h i Definition of j=1 i=1 Hilbert Space X X Orthogonality b ∞ N 1 Compact = Xi , ej Xi E[ Xi , ej Xi ] , operators N h i − h i Riesz j=1 i=1 representation D theorem X X Adjoint N operators 1 Singular values , Xk , ej Xk E[ Xk , ej Xk ] = Hilbert-Schmidt N h i − h i operators k=1 2 X E Space L in ∞ N N more detail 1 Integral operator = Xi , ej Xi E[ Xi , ej Xi ] Covariance N2 h i − h i · operator j=1 i=1 k=1 Asymptotic X X X n o properties in L2 Estimation of Xk , ej Xk E[ Xk , ej Xk ] mean and · h i − h i covariance n o functions 46/56 Sufficient condition for Hilbert-Schmidt operator

L2 space of random functions part 2. Marek Dvoˇr´ak ∞ N 1 2 Hilbert Spaces 2 E C C S = E Xi , ej Xi E Xi , ej Xi = Deﬁnition of k − k N2 h i − h i Hilbert Space j=1 i=1 Orthogonality X X ∞ Compact b 1 2 operators = E X , ej X E X , ej X Riesz N h i − h i ≤ representation j=1 theorem X Adjoint ∞ operators 2 Singular values 1 Hilbert-Schmidt E X , ej X = operators ≤ N h i j=1 Space L2 in X more detail ∞ Integral operator 1 2 2 1 4 Covariance = E X X , ej = E X . operator N · k k · |h i| N · k k Asymptotic j=1 properties in L2 h X i Estimation of mean and covariance functions 47/56 Conclusion of the theorem

L2 space of random functions

Marek Dvoˇr´ak

Hilbert Spaces Remark Deﬁnition of Hilbert Space 2 −1 4 Orthogonality We have proved that E C C S N E X . It means that Compact k − k ≤ · k k operators 1 1 Riesz b 2 representation −1 4 theorem E c(t, s) c(t, s) dt ds N E X 0. Adjoint 0 0 − ≤ · k k → operators " Z Z # Singular values h i Hilbert-Schmidt operators i.e. covarianceb is mean-squared consistent estimator. Space L2 in more detail Integral operator Covariance operator Asymptotic properties in L2 Estimation of mean and covariance functions 48/56 Outline

L2 space of random 1 Introduction to Hilbert Spaces functions Deﬁnition of Hilbert Space Marek Dvoˇr´ak Orthogonality

L2 space of random functions Marek Dvoˇr´ak Assume: λ1 >λ2 >...>λp >λp+1 >...,

Hilbert Spaces Eigenelements: CX (vj )= λj vj , Deﬁnition of Hilbert Space vj eigenfunction a vj , a = 0 also eigenfunction, Orthogonality ⇒ · 6 vj usually normalized, i.e. vj = 1, Compact k k operators even though the sign of vj cannot be determined. Riesz representation theorem Let vj be an estimate of vj then it is not sure that cj vj are Adjoint operators close to vj where Singular values Hilbert-Schmidt b cj := sign( vj , vj ). b b operators h i Space L2 in more detail In addition, cj cannot be computed from data . Examples: Integral operator b b ⇒ Covariance see Johnston and Lu (2009): ”On consistency and sparsity for operator Asymptotic principal componentb analysis in high dimensions”. properties in L2 Estimation of mean and covariance functions 50/56 Estimated eigenelements

L2 space of Deﬁne random functions 1

Marek Dvoˇr´ak C(vj ) := c(t, s) vj (s)ds = λj vj (t) j = 1,..., N. Z0 Hilbert Spaces Deﬁnition of b b Hilbert Space Theoremb (Dauxoisb et al.b (1982)) b Orthogonality 4 Compact Suppose that E X < , EX = 0, Assumption 1 hold and operators k k ∞ Riesz λ1 >λ2 >...>λp >λp+1. Then for each j = 1,..., p representation theorem Adjoint operators 2 Singular values lim sup N E cj vj vj < Hilbert-Schmidt N→∞ · k − k ∞ operators 2 Space L in and b b more detail 2 Integral operator lim sup N E λj λj < . Covariance N · | − | ∞ operator →∞ Asymptotic properties in L2 b Estimation of Key tools for proving: If operators C and C are close then also mean and covariance its eigenelements are close. 51/56 functions b Eigenvalues are close

L2 space of random Lemma (Gohberg et al., Corollary VI.1.6) functions

Marek Dvoˇr´ak Suppose C, K compact operators with SVD ∞ ∈L ∞ C(x)= j=1 λj x, vj fj and K(x)= j=1 γj x, uj gj . Then Hilbert Spaces h i h i Deﬁnition of Hilbert Space P j N : γj λj PK C L. Orthogonality ∀ ∈ − ≤ k − k Compact operators Riesz Operator K represents the role of C in the proof of the th. representation theorem Adjoint Proof. operators Singular values b Hilbert-Schmidt Based on Theorem VI.1.5 in Gohberg et al.(1990): operators 2 λj (C) = min C Q Q (H), rank(Q) j 1 . Take Space L in {k − k ∈L ≤ − } more detail rank(Q) j 1. Then λj C Q C K + K Q , Integral operator ≤ − ≤ k − k ≤ k − k k − k Covariance operator so λj C K is a lower bound for K Q when Q runs Asymptotic − k − k k − k 2 over all operators of rank (j 1). Hence λj C K γj properties in L ≤ − − k − k≤ Estimation of and hence λj γj C K . Now interchange roles of C and mean and − ≤ k − k covariance K. functions 52/56 Eigenfunctions are close

L2 space of random functions Lemma Marek Dvoˇrák Assume C is symmetric with C(vj )= λj vj , i.e. in SVD: vj = fj , Hilbert Spaces and with eigenvalues λ1 >λ2 >...>λp >λp+1. Let Definition of Hilbert Space C, K are compact and K has SVD as before. Denote Orthogonality ′ ∈L Compact vj := cj vj and cj := sign( uj , vj ). Then operators h i Riesz representation theorem ′ 2√2 Adjoint uj vj K C L, operators k − k≤ αj k − k Singular values Hilbert-Schmidt operators where α = λ λ and α = min(λ λ , λ λ ), j 2. Space L2 in 1 1 2 j j−1 j j j+1 more detail − − − ≥ Integral operator Covariance operator Proof. Asymptotic properties in L2 Straightforward, see Horváth, page 41, Lemma 2.3. Estimation of mean and covariance functions 53/56 Proof of the theorem

L2 space of random functions Theorem:

Marek Dvoˇr´ak 2 2 lim sup N E cj vj vj < , lim sup N E λj λj < . Hilbert Spaces N→∞ · k − k ∞ N→∞ · | − | ∞ Deﬁnition of Hilbert Space Orthogonality b b b Compact Proof. operators Riesz representation We know: theorem Adjoint 2 2 2 operators γj λj C C L C C S Singular values − ≤ k − k ≤ k − k 2 Hilbert-Schmidt const 2 2 operators cj vj vj C C L C C S 2 − ≤b · k −b k ≤ k − k Space L in 2 more detail Now, according to the previous theorem, E C C S 0. Integral operator b b b bk − k → Covariance operator It can also be shown, that estimated eigenelements are Asymptotic b properties in L2 asymptotically normal. Estimation of mean and covariance functions 54/56 Summary of the talk

L2 space of random functions

Marek Dvoˇr´ak

Hilbert Spaces Most important things to remember: Deﬁnition of Hilbert Space Compact operators admit SVD. Orthogonality Compact Under Assumptions: operators X iid. with same distribution as a square integrable X , Riesz i representation 4 theorem EX =0, E X < , Adjoint k k ∞ operators λ1 >...>λp >... Singular values Hilbert-Schmidt Covariance operator is consistent. operators Space L2 in Estimated eigenelements converge to the true values. more detail Integral operator Covariance operator Asymptotic properties in L2 Estimation of mean and covariance functions 55/56 References

L2 space of random functions

Marek Dvoˇrák Gohberg I. et al. (1990): “Classes of Linear Operators Hilbert Spaces Definition of vol.I”, Birkhäuser Verlag. Munich. Hilbert Space Orthogonality Horváth et al.: “Inference for Functional Data with Compact operators Applications”. Preprint. Riesz representation theorem Adjoint operators Singular values Hilbert-Schmidt operators Thank you Space L2 in more detail for attention Integral operator Covariance operator Asymptotic properties in L2 Estimation of mean and covariance functions 56/56