LTCC: Stochastic Processes Maria De Iorio Department of Statistical Science University College London [email protected] 1 / 42 Outline 1 Examples of CTMC Poisson Process Birth Process 2 Time reversibility 3 Birth-death process 2 / 42 Examples of CTMC Poisson Process The Poisson process with rate λ Let fN(t): t ≥ 0g be a Markov process, where N(t) represents the number of events (or births) in the interval (0; t], for example claim times to an insurance company, arrival times of customers in a shop,::: Let the transition probabilities of fN(t)g be: the probability of one occurrence in the time interval( t; t + h) is λh + o(h): pi i+1(h) = λh + o(h) the probability of more than one occurrence is o(h): pi j (h) = o(h); for j 6= i; i + 1 This implies that the probability of no events in( t; h + t) is1 − λh + o(h): pi i (h) = 1 − λh + o(h) the number of occurrences in( t; t + h) is independent of the occurrences in (0; t]. 3 / 42 Examples of CTMC Poisson Process The state space is S = f0; 1; 2;:::g. Then P(N(t + h) = k j N(t) = i) = λh + o(h); k = i + 1 P(N(t + h) = k j N(t) = i) = o(h); k > i + 1 P(N(t + h) = k j N(t) = i) = 0; k < i The Poisson process has generator 0 −λλ 00 ::: 1 B 0 −λλ 0 ::: C B C Q = B 00 −λλ::: C : @ . A . 4 / 42 Examples of CTMC Poisson Process A Poisson process N(t) describes the number of occurrences of some event up to time t, i.e. in the interval (0; t]. We assume N(0) = 0, so that the process stays in state 0 for a random time τ1, before jumping to state 1 and staying there a random time τ2, then jumping to state 2, etc. The intervals τ1; τ1;::: are independent, identically distributed (i.i.d. ) exponential variables with common parameter λ. Corollary 1 (holding times) Poisson process, rate λ has holding times iid∼ Expon(λ) Proof From Theorem on Holding Times , Ti ∼ Expon(−qii ). Want λ = −qii . 5 / 42 Examples of CTMC Poisson Process This property provides an alternative definition of the Poisson process as a sequence of point events where the intervals between successive events are i.i.d. exponential variables. Pr It follows that the time Tr = i=1 τi to the rth event has a gamma distribution, Gamma(r; λ). By the Central Limit Theorem, for large r, this can be approximated by the normal distribution N(r/λ, r/λ2). 6 / 42 Examples of CTMC Poisson Process The Poisson process has the property of independent increments, that numbers of events in non-overlapping intervals (and, more generally, disjoint sets) are independent. It follows that the interval from an arbitrary time point until the next event also has an exponential distribution with parameter λ. The chain is not irreducible, since, e.g., pij (t) ≡ 0 if j < i. Each state is in a separate transient class. There is no invariant distribution; the only solution of πQ = 0 is π0 = π1 = ::: = 0. 7 / 42 Examples of CTMC Poisson Process Recall Forward equations: P0(t) = P(t)Q We are interested in p0j (t) = P(j events in (0; t] j N(0) = 0). The forward equations are 0 0 p00(t) = −λp00(t); p0j (t) = λp0;j−1(t) − λp0j (t) for j = 1; 2;::: which are most easily solved using generating functions. 8 / 42 Examples of CTMC Poisson Process Let N(t) X j G(s; t) = E(s j N(0) = 0) = s p0j (t) j=0 be the probability generating function of N(t)( jsj ≤ 1). Then 1 @G(s; t) @ X = sj p (t) @t @t 0j j=0 1 X 0 j = p0j (t)s j=0 1 1 X j X j = −λp00(t) + λ p0;j−1(t)s − λ p0j (t)s j=1 j=1 = −λG(s; t) + λsG(s; t) = −λ(1 − s)G(s; t): The initial condition is G(s; 0) ≡ 1, so the solution of this equation is G(s; t) = e−λ(1−s)t , which is the probability generating function of a Poisson variable with mean λt. 9 / 42 Examples of CTMC Poisson Process It follows that −λt j p0j (t) = e (λt) =j! for j = 0; 1; 2;:::: This result provides another means of defining the Poisson process, as a point process for which, for all t and τ, N(t) has a Poisson distribution with mean λt, Corollary 2 (E N(t) = var N(t) = λt) d N(t) the `rate' at which events occur is E = λ dt and the numbers of events in (0; t] and( t; t + τ] are independent variables. There are many further characterisations of the Poisson process. 10 / 42 Examples of CTMC Poisson Process An important property of the Poisson process is that, given the number of events in an interval, these events are independently and uniformly distributed over the interval. Theorem 3 fN(t)g Poisson. GivenN (t) = n, the arrival times iid Θ1 < : : : < Θn ∼ Unif (0; t). 11 / 42 Examples of CTMC Poisson Process Sketch Proof The joint density of the arrival timesΘ 1:n := Θ1;:::; Θn with n arrivals at t can be written in terms of the (iid Expon) inter-arrival (holding) times T0:n := T0;:::; Tn, namely (see figure on board): fΘ1:n = θ1:n; N(t) = ng = fΘ1 = θ1;:::; Θn = θn; N(t) = ng = fT0 = θ1; T1 = θ2 − θ1;:::; Tn−1 = θn − θn−1; Tn > t − θng : Now f (Θ1:n = θ1:njN(t) = n) f (Θ = θ ; N(t) = n) = 1:n 1:n P(N(t) = n) f (T = θ ; T = θ − θ ;:::; T = θ − θ ; T > t − θ ) = 0 1 1 2 1 n−1 n n−1 n n P(N(t) = n) n−1 n! Y n! = eλt e−λ(t−θn) λe−λ(θi+1−θi ) = (λt)n tn | {z } i=0 | {z } f (Tn>t−θn) f (Ti =θi+1−θi ) ;with θ0:=0 |{z} | {z } joint pdf of ordered unif r.v.s 1=P(N(t)=n) which is the joint density of the order statistics of n independent variables, each uniformly distributed on (0; t). 12 / 42 Examples of CTMC Poisson Process We have reviewed homogeneous Poisson Processes where the intensity, λ, is constant across time. In practice we often meet inhomogeneous Poisson Processes where the intensity, λ(t), varies with time. This may be due to covariates which vary with time. In that case, the number of events that occur in( t1; t2] will have a Poisson distribution with meanΛ( t2) − Λ(t1) where Z ti Λ(ti ) = λ(t)dt 0 13 / 42 Examples of CTMC Birth Process Birth Process A birth process with intensities λ0; λ1;::: is a process fN(t); t ≥ 0g taking values in S = f0; 1; 2;:::g such that N(0) = 0; if s < t then N(s) ≤ N(t) 8 < λnh + o(h) if m = 1 PN(t + h) = n + m j N(t) = n) = o(h) if m > 1 : 1 − λnh + o(h) if m = 0 if s < t then N(t) − N(s) is independent of all arrivals prior to s. 14 / 42 Examples of CTMC Birth Process From the definition of the process it is clear that : qii = −λi ; qi;i+1 = λi ; qij = 0 if j < i or j > i The generator is then given 0 1 −λ0 λ0 000 ··· B 0 −λ1 λ1 00 ··· C Q = B C B 00 −λ2 λ2 0 ··· C @ . A . 15 / 42 Examples of CTMC Birth Process Special Cases Poisson Process: λn = λ for all n Simple birth: λn = nλ. This models the growth of a population in which each living individual may give birth to a new individual with probability λh + o(h) in the interval( t; t + h). No individuals may die. The number M of births in the interval( t; t + h) satisfies n P(M = m j N(t) = n) = (λh)m(1 − λh)n−m + o(h) m 8 < 1 − nhλ + o(h) if m = 0 = nhλ + o(h) if m = 1 : o(h) if m > 1 Simple birth with immigration: λn = nλ + ν. This models a simple birth process which experiences immigration at constant rate ν from elsewhere. 16 / 42 Time reversibility Consider a stochastic process fX (t)g (−∞ < t < 1 or t = :::; −2; −1; 0; 1; 2;:::). The process X (t) is reversible if its stochastic behaviour remains the same when the direction of time is reversed. Definition 4 A Markov chain is reversible if d X (t1);:::; X (tn) = X (τ − t1);:::; X (τ − tn) 8 t1;:::; tn; τ; n Setting τ equal to0, it follows that( X (t1); :::; X (tn)) and (X (−t1); :::; X (−tn)) have the same joint distribution. 17 / 42 Time reversibility Definition 5 fX (t)g is stationary if d X (t1);:::; X (tn) = X (t1 + τ);:::; X (tn + τ) 8 t1;:::; tn; τ; n Theorem 6 A reversible process is stationary Proof d X (t1);:::; X (tn) = X (τ − t1);:::; X (τ − tn) reversib. Set τ = 0 d = X (−t1);:::; X (−tn) Replace ti by ti + τ in the def. d X (t1 + τ);:::; X (tn + τ) = X (−t1);:::; X (−tn) + putting 2 eq together d X (t1);:::; X (tn) = X (t1 + τ);:::; X (tn + τ) 18 / 42 Time reversibility Note that if a Markov chain is reversible it must be stationary.