E2.204: Stochastic Processes and Queuing Theory Spring 2019 Course Instructor: Prof. Vinod Sharma ([email protected]) Teaching Assistants: Panju and Ajay Indian Institute of Science, Bangalore Contents 1 Poisson Processes4 1.1 Introduction to stochastic processes............................4 1.2 Poisson process.......................................5 1.2.1 Definition......................................5 1.2.1 (Contd.) Poisson Processes: Definition......................7 1.2.2 Properties of Poisson processes.......................... 10 1.2.3 Generalization of Poisson processes........................ 11 1.3 Problems.......................................... 13 2 Renewal Theory and Regenerative Processes 14 2.1 Renewal Process: Introduction............................... 14 2.2 Limit Theorems....................................... 16 2.2.2 Blackwell’s Theorem................................ 18 2.2.3 Renewal Equation................................. 19 2.2.4 Renewal theorems................................. 21 2.3 Regenerative Processes................................... 25 2.4 Problems.......................................... 28 3 Discrete Time Markov Chains 30 3.1 Markov Chains: Definitions................................ 30 3.2 Class Properties of Transience and Recurrence....................... 31 3.3 Limiting distributions of Markov chains.......................... 32 3.2 (Contd.) Limiting distributions of Markov chains..................... 33 3.4 Tests for transience, null recurrence and positive recurrence................ 35 3.5 Reversible Markov Chains................................. 35 3.6 Example: M/GI/1 queue.................................. 37 3.7 Rate of convergence to the stationary distribution..................... 37 3.8 Problems.......................................... 40 4 Continuous-Time Markov Chains 41 4.1 Introduction......................................... 41 4.2 Strong Markov property, Minimal construction...................... 42 4.3 Chapman Kolmogorov equations.............................. 44 4.4 Irreducibility and Recurrence................................ 46 4.5 Time Reversibility..................................... 49 4.6 Birth-Death process..................................... 50 4.6.1 Reversibility of Birth-Death process........................ 51 4.6.2 Examples...................................... 51 1 4.7 Problems.......................................... 53 5 Martingales 55 5.1 Introduction......................................... 55 5.2 Optional Sampling Theorem................................ 57 5.2 (Contd.) Optional Sampling Theorem:Example...................... 59 5.3 Martingale inequalities................................... 59 5.4 McDiarmid’s Inequality:Applications........................... 62 5.5 Martingale Convergence Theorem............................. 63 5.6 Applications to Markov chain................................ 65 5.7 Problems.......................................... 69 6 Random Walks 70 6.1 Definitions.......................................... 70 6.2 Ladder Heights, Maxima, GI=GI=1 Queue......................... 71 6.2 (Contd.) Ladder Epochs................................... 73 7 Queuing Theory 75 7.1 GI/GI/1 Queue....................................... 75 7.2 Palm Theory, PASTA.................................... 79 7.2.1 Rate conservation laws............................... 82 7.2.2 PASTA....................................... 82 7.3 Product-form Networks................................... 83 7.3.1 M=M=1 queue:................................... 83 7.3.2 Tandem Queues................................... 83 7.3.3 Open Jackson Networks.............................. 84 7.3.4 Closed queueing networks............................. 85 7.4 Product-Form Networks: Quasireversible networks.................... 86 7.4.1 Quasireversible Queues............................... 86 7.4.2 Networks of Quasireversible Queues........................ 88 7.4.2 Networks of quasireversible queues (contd.).................... 90 7.5 Problems.......................................... 93 2 Course structure • Two sessions per week. Tuesdays and Thursdays between 5:15 p.m. to 6:45 p.m. • One assignment per topic. There will be a quiz based on each assignment. Tutorial and quiz sessions will be held by TAs on Saturdays 10-11 a.m. after each topic is finished. • One mid-term exam. • One final exam. Reference Books [1] S. Assmussen, ”Applied Probability and Queues”, 2nd ed., Springer, 2003. [2] B. Hajek, ”Random Processes for Engineers”, Cambridge Univesity press, 2015. [3] S. Karlin and H.M. Taylor, ”A First Course in Stochastic Processes”, 2nd ed., 1975. [4] S.M. Ross, ”Stochastic Processes”,2nd ed., Wiley, 1996. [5] J. Walrand, ”An introduction to Queueing Netwoorks”, Printice Hall, 1988. 3 Chapter 1 Poisson Processes Lecture 1 Course E2.204: Stochastic Processes and Queueuing Theory (SPQT) Spring 2019 Instructor: Vinod Sharma Indian Institute of Science, Bangalore 1.1 Introduction to stochastic processes Review: Let (W;s;P) be a probability space. A measurable mapping X : W ! R is called a random variable (r.v.). X(w) for w 2 W is called a realization of X. FX (x) = P[X ≤ x] is called the distribution function of r.v. X. fX (x) = dF(x)=dx is called the probability density function of X. The probability den- R sity function may not always exist. E[X] = xdFX (x) is the expectation of X. When probability density R of X exists E[X] = x f (x)dx. Stochastic processes: fXt : t 2 Rg, where Xt is a r.v. is called a continuous time stochastic process. fXn : n 2 Ng, where Xn is a r.v. is called a discrete time stochastic process. The function t 7! Xt (w) is called a sample path of the stochastic process. For each w 2 W, Xt (w) is a function of t. Ft is the distribution of Xt . An analogous definition holds for discrete time stochastic processes. A stochastic process is described by the joint distribution of (Xt1 ;Xt2 ;:::;Xtn ) for any −¥ < + t1 < t2 < ::: < tn and n 2 N . A stochastic process fXt g is said to be stationary if for any 0 ≤ t1 < t2 < ··· < tn, the joint distribution of (Xt1 ;Xt2 ;:::;Xtn ) is identical to the joint distribution of (Xt1+t ;Xt2+t ;:::;Xtn+t ) for any t 2 R.A stochastic process fXt g is said to have independent increments if (Xt2 − Xt1 );(Xt3 − Xt2 );:::;(Xtn − Xtn−1 ) are independent. If joint distribution of (Xtn+t −Xtn−1+t ), (Xtn−2+t −Xtn−3+t );:::;(Xt2+t −Xt1+t ) does not depend on t, then fXt g is said to have stationary increments. If fXt g has both stationary and independent increments, it is called a stationary independent increment process. Point process: A stochastic process fNt ;t ≥ 0g with N0 = 0, Nt a non-negative integer, non-decreasing with piece-wise constant sample paths is called a point process. Nt counts the number of points or ’ar- rivals’ in the interval (0;t]. th th n Let An denote the interarrival time between n and (n−1) arrival. Let S0 = 0 and Sn = ∑k=1 Ak;8n ≥ th 1. Then Sn denotes the time instant of the n arrival. Nt = maxfn : Sn ≤ tg. A point process with at most one arrival at any time is called a simple point process. Mathematically, a simple point process fNt g is 4 described by following constraints for all t: PfNt+h − Nt ≥ 2g = o(h): f (x) Here, the notation o(g(x)) means a class of functions such that if f (x) 2 o(g(x)), then limx!0 g(x) = 0. 1.2 Poisson process 1.2.1 Definition In the following we customarily take N0 = 0. A point process Nt is Poisson if any of the following conditions hold. Definition [1]: −lx 1. fAk;k ≥ 1g are independent and exponentially distributed with parameter l: PfAk ≤ xg = 1−e . If l = 0, A1 = ¥ w.p.1. and Nt = 0 8t. If l = ¥, A1 = 0 w.p.1 and Nt = ¥ 8t. Thus, we restrict to 0 < l < ¥. In this range for l, Nt is guaranteed to be simple because PfAk = 0g = 0 8k. Definition [2]: 1. Nt is simple. 2. Nt has stationary independent increments. Definition [3]: 1. Nt has independent increment. 2. For s < t, (l(t − s))n fN − N = ng = e−l(t−s) P t s n! Definition [4]: 1. Nt has stationary and independent increments. 2. (a) PfNt+h − Nt = 1g = lh + o(h) (b) PfNt+h − Nt = 0g = 1 − lh + o(h) (c) PfNt+h − Nt ≥ 2g = o(h) We will show below that these definitions are equivalent. We need the following important charateri- zation of exponential distribution. 5 Exponential r.v. is memoryless: Let X be an exponential r.v. PfX > t + s;X > tg PfX > t + sjX > tg = PfX > tg fX > t + sg = P PfX > tg e−l(t+s) = e−lt = e−ls = PfX > sg If X is interarrival time, this property of an exponential r.v. indicates that the remaining time till next arrival does not depend on time t since last arrival. Thus, the term memoryless is used. + Theorem 1.2.1. Exponential distribution is the unique distribution on R with the memoryless property. + Proof. If a r.v. X on R is memoryless, we show that X must be exponential. If X is memoryless, we have for all t;s ≥ 0, PfX > t + sg = PfX > tgPfX > sg. Let f (t) = PfX > tg. We have the functional equation f (t + s) = f (t) f (s) (1.1) Taking t = s, we get f (2t) = f 2(t). By repeated application of Eq 1.1, m times, we get f (mt) = f m(t) 1 for positive integer m. Equivalently, we have f (t=m) = f m (t). Again by repeated application of Eq 1.1 n n m r n times, f ( mt) = f (t) for any positive integers m and n. So, we have f (rt) = f (t) for any positive rational number r. We know that 0 ≤ f (t) ≤ 1 since 1 − f is probability distribution. So, we can write f (1) = e−l for some l ≥ 0. Therefore we have, f (r) = f (r ×1) = f r(1) = e−lr for any positive rational number r. For any x 2 R, there is a sequence of rationals rn # x. Since f is right continuous, f (rn) ! f (x). In other words, for any x 2 R, f (x) = lim f (rn) rn!x = lim e−lrn rn!x = e−lx − x Thus, PfX > xg = e l and X is an exponential random variable. Now we show that definitions [1 − 4] for Poisson process given above are equivalent. Proposition 1.2.2. Definition [3] =) Definition [2]. Proof. We need to show that Nt has stationary increments if Nt has independent increments and Nt − Ns is Poisson distributed with mean l(t − s). Stationarity follows directly from the definition since the distribution of number of points in an interval depends only in the length of interval.