Markov chains on measurable spaces Lecture notes Dimitri Petritis Master STS mention mathématiques Rennes UFR Mathématiques Preliminary draft of April 2012 ©2005–2012 Petritis, Preliminary draft, last updated April 2012 Contents 1 Introduction1 1.1 Motivating example.............................1 1.2 Observations and questions........................3 2 Kernels5 2.1 Notation...................................5 2.2 Transition kernels..............................6 2.3 Examples-exercises.............................7 2.3.1 Integral kernels...........................7 2.3.2 Convolution kernels........................9 2.3.3 Point transformation kernels................... 11 2.4 Markovian kernels............................. 11 2.5 Further exercises.............................. 12 3 Trajectory spaces 15 3.1 Motivation.................................. 15 3.2 Construction of the trajectory space................... 17 3.2.1 Notation............................... 17 3.3 The Ionescu Tulce˘atheorem....................... 18 3.4 Weak Markov property........................... 22 3.5 Strong Markov property.......................... 24 3.6 Examples-exercises............................. 26 4 Markov chains on finite sets 31 i ii 4.1 Basic construction............................. 31 4.2 Some standard results from linear algebra............... 32 4.3 Positive matrices.............................. 36 4.4 Complements on spectral properties.................. 41 4.4.1 Spectral constraints stemming from algebraic properties of the stochastic matrix........................ 41 4.4.2 Spectral constraints stemming from convexity properties of the stochastic matrix........................ 43 5 Potential theory 49 5.1 Notation and motivation.......................... 49 5.2 Martingale solution to Dirichlet’s problem............... 51 5.3 Asymptotic and invariant σ-algebras.................. 58 5.4 Triviality of the asymptotic σ-algebra.................. 59 6 Markov chains on denumerably infinite sets 65 6.1 Classification of states........................... 65 6.2 Recurrence and transience........................ 69 6.3 Invariant measures............................. 71 6.4 Convergence to equilibrium........................ 75 6.5 Ergodic theorem.............................. 77 6.6 Semi-martingale techniques for denumerable Markov chains.... 79 6.6.1 Foster’s criteria........................... 79 6.6.2 Moments of passage times.................... 84 7 Boundary theory for transient chains on denumerably infinite sets 85 7.1 Motivation.................................. 85 iii 7.2 Martin boundary.............................. 86 7.3 Convexity properties of sets of superharmonic functions....... 91 7.4 Minimal and Poisson boundaries..................... 93 7.5 Limit behaviour of the chain in terms of the boundary........ 95 7.6 Markov chains on discrete groups, amenability and triviality of the boundary................................... 100 8 Continuous time Markov chains on denumerable spaces 103 8.1 Embedded chain.............................. 103 8.2 Infinitesimal generator........................... 103 8.3 Explosion................................... 103 8.4 Semi-martingale techniques for explosion times........... 103 9 Irreducibility on general spaces 105 9.1 Á-irreducibility............................... 105 9.2 c-sets..................................... 109 9.3 Cycle decomposition............................ 114 10 Asymptotic behaviour for Á-recurrent chains 121 10.1 Tail σ-algebras................................ 121 10.2 Structure of the asymptotic σ-algebra.................. 123 11 Uniform Á-recurrence 127 11.1 Exponential convergence to equilibrium................ 127 11.2 Embedded Markov chains......................... 130 11.3 Embedding of a general Á-recurrent chain............... 131 iv 12 Invariant measures for Á-recurrent chains 133 12.1 Convergence to equilibrium........................ 133 12.2 Relationship between invariant and irreducibility measures..... 134 12.3 Ergodic properties............................. 140 13 Quantum Markov chains 141 A Complements in measure theory 143 A.1 Monotone class theorems......................... 143 A.1.1 Set systems............................. 143 A.1.2 Set functions and their extensions................ 145 A.2 Uniformity induced by Egorov’s theorem and its extension..... 146 B Some classical martingale results 147 B.1 Uniformly integrable martingales.................... 147 B.2 Martingale proof of Radon-Nikodým theorem............. 148 B.3 Reverse martingales............................ 149 C Semi-martingale techniques 151 C.1 Integrability of the passage time..................... 151 References 152 Index 156 v 1 Introduction 1.1 Motivating example The simplest non-trivial example of a Markov chain is the following model. Let X {0,1} (heads 1, tails 0) and consider two coins, one honest and one biased Æ giving tails with probability 2/3. Let ½ outcome of biased coin if Xn 0 Xn 1 Æ Å Æ outcome of honest coin if X 1. n Æ For y X and ¼ (y) P(X y), we have 2 n Æ n Æ ¼n 1(y) P(Xn 1 y honest)P(honest) P(Xn 1 y biased)P(biased), Å Æ Å Æ j Å Å Æ j yielding µ ¶ P00 P01 (¼n 1(0),¼n 1(1)) (¼n(0),¼n(1)) , Å Å Æ P10 P11 with P 2/3, P 1/3, and P P 1/2. 00 Æ 01 Æ 10 Æ 11 Æ Iterating, we get ¼ ¼ P n n Æ 0 where ¼ must be determined ad hoc (for instance by ¼ (x) ± for x X). 0 0 Æ 0x 2 In this formulation, the probabilistic problem is translated into a purely al- gebraic problem of determining the asymptotic behaviour of ¼n in terms of the n asymptotic behaviour of P , where P (Px y )x,y X is a stochastic matrix (i.e. P Æ 2 Px y 0 for all x and y and y X Px y 1). Although the solution of this prob- ¸ 2 Æ lem is an elementary exercise in linear algebra, it is instructive to give some de- µ1 a a ¶ tails. Suppose we are in the non-degenerate case where P ¡ with Æ b 1 b ¡ 0 a,b 1. Compute the spectrum of P and left and right eigenvectors: Ç Ç ut P ¸ut Æ Pv ¸v. Æ 1 2 1 0 Figure 1.1: Dotted lines represent all possible and imaginable trajectories of heads and tails trials. Full line represents a particular realisation of such a se- quence of trials. Note that only integer points have a significance, they are joined by line segments only for visualisation purposes. Since the matrix P is not normal in general, the left (resp. right) eigenvectors cor- responding to different eigenvalues are not orthogonal. However we can always normalise ut v ± . Under this normalisation, and denoting by E v ut , ¸ ¸0 Æ ¸¸0 ¸ Æ ¸ ­ ¸ we get ¸ u¸ v¸ E¸ µ ¶ µ ¶ µ ¶ b 1 1 b a ¸1 1 a b Æ a 1 Å b a µ 1 ¶ µ a ¶ µ a a¶ ¸ 1 (a b) 1 ¡ 2 Æ ¡ Å 1 b a b b b ¡ ¡ Å ¡ We observe that for ¸ and ¸0 being eigenvalues, E E ± E , i.e. E are spec- ¸ ¸0 Æ ¸¸0 ¸ ¸ tral projectors. The matrix P admits the spectral decomposition X P ¸E¸, Æ ¸ specP 2 where specP {1,1 (a b)} {¸ C : ¸I P is not invertible}. Now E being Æ ¡ Å Æ 2 ¡ ¸ orthogonal projectors, we compute immediately, for all positive integers n, n X n n n P ¸ E¸ ¸1 E¸1 ¸2 E¸2 Æ ¸ specP Æ Å 2 n and since ¸1 1 while ¸2 1, we get that limn P E¸ . Æ j j Ç !1 Æ 1 Applying this formula for the numerical values of the example, we get that limn ¼n (0.6,0.4). !1 Æ 1.2. Observations and questions 3 1.2 Observations and questions – In the previous example, a prominent role is played by the stochastic ma- trix P. In the general case, P is replaced by an operator, known as stochas- tic kernel that will be introduced and studied in chapter2. – An important question, that has been eluded in this elementary introduc- tion, is whether exists a probability space (­,F ,P) sufficiently large as to carry the whole sequence of random variables (Xn)n N. This question will 2 be positively answered by the Ionescu Tulce˘atheorem and the construc- tion of the trajectory space of the process in chapter3. – In the finite case, motivated in this chapter and fully studied in chapter4, the asymptotic behaviour of the sequence (Xn) is fully determined by the spectrum of P, a purely algebraic object. In the general case (countable or uncountable), the spectrum is an algebraic and topological object defined as specP {¸ C : ¸I P is not invertible}. Æ 2 ¡ This approach exploits the topological structure of measurable functions on X; a thorough study of the general case is performed, following the line of spectral theory of quasi-compact operators, in [14]. Although we don’t follow this approach here, we refer the reader to the previously mentioned monograph for further study. – Without using results from the theory of quasi-compact operators, we can 1 P n study the formal power series (I P)¡ 1 P (compare with the above ¡ Æ n 0 definition of the spectrum). This series canÆ be given both a probabilis- tic and an analytic significance realising a profound connection between harmonic analysis, martingale theory, and Markov chains. It is this latter approach that will be developed in chapter5. – Quantum Markov chains are objects defined on a quantum probability space. Now, quantum probability can be thought as a non-commutative extension of classical probability where real random variables are replaced by self-adjoint operators acting on a Hilbert space. Thus, once again the spectrum plays an important rôle in this context. Exercise 1.2.1 (Part of the paper of December 2006) Let (Xn) be the Markov 0 Pn 1 chain defined in the example of section 1.1, with X {0,1}. Let ´ (A) ¡ 1 (X ) Æ n Æ k 0 A k for A X. Æ ⊆ P 0 1. Compute Eexp(i x X tx ´n({x})) as a product of matrices and vectors, for 2 t R. x 2 2. What is the probabilistic significance of the previous expectation? 0 3. Establish a central limit theorem for ´n({0}). Pn 1 0 4. Let f : X R. Express S (f ) ¡ f (X ) with the help of ´ (A), with ! n Æ k 0 k n A X. Æ ⊆ 4 2 Kernels This chapter deals mainly with the general theory of kernels.