DOCSLIB.ORG

E2.204 Stochastic Processes and Queueing Theory

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
E2.204 Stochastic Processes and Queueing Theory E2.204: Stochastic Processes and Queuing Theory Spring 2019 Course Instructor: Prof. Vinod Sharma ([email protected]) Teaching Assistants: Panju and Ajay Indian Institute of Science, Bangalore Contents 1 Poisson Processes4 1.1 Introduction to stochastic processes............................4 1.2 Poisson process.......................................5 1.2.1 Definition......................................5 1.2.1 (Contd.) Poisson Processes: Definition......................7 1.2.2 Properties of Poisson processes.......................... 10 1.2.3 Generalization of Poisson processes........................ 11 1.3 Problems.......................................... 13 2 Renewal Theory and Regenerative Processes 14 2.1 Renewal Process: Introduction............................... 14 2.2 Limit Theorems....................................... 16 2.2.2 Blackwell’s Theorem................................ 18 2.2.3 Renewal Equation................................. 19 2.2.4 Renewal theorems................................. 21 2.3 Regenerative Processes................................... 25 2.4 Problems.......................................... 28 3 Discrete Time Markov Chains 30 3.1 Markov Chains: Definitions................................ 30 3.2 Class Properties of Transience and Recurrence....................... 31 3.3 Limiting distributions of Markov chains.......................... 32 3.2 (Contd.) Limiting distributions of Markov chains..................... 33 3.4 Tests for transience, null recurrence and positive recurrence................ 35 3.5 Reversible Markov Chains................................. 35 3.6 Example: M/GI/1 queue.................................. 37 3.7 Rate of convergence to the stationary distribution..................... 37 3.8 Problems.......................................... 40 4 Continuous-Time Markov Chains 41 4.1 Introduction......................................... 41 4.2 Strong Markov property, Minimal construction...................... 42 4.3 Chapman Kolmogorov equations.............................. 44 4.4 Irreducibility and Recurrence................................ 46 4.5 Time Reversibility..................................... 49 4.6 Birth-Death process..................................... 50 4.6.1 Reversibility of Birth-Death process........................ 51 4.6.2 Examples...................................... 51 1 4.7 Problems.......................................... 53 5 Martingales 55 5.1 Introduction......................................... 55 5.2 Optional Sampling Theorem................................ 57 5.2 (Contd.) Optional Sampling Theorem:Example...................... 59 5.3 Martingale inequalities................................... 59 5.4 McDiarmid’s Inequality:Applications........................... 62 5.5 Martingale Convergence Theorem............................. 63 5.6 Applications to Markov chain................................ 65 5.7 Problems.......................................... 69 6 Random Walks 70 6.1 Definitions.......................................... 70 6.2 Ladder Heights, Maxima, GI=GI=1 Queue......................... 71 6.2 (Contd.) Ladder Epochs................................... 73 7 Queuing Theory 75 7.1 GI/GI/1 Queue....................................... 75 7.2 Palm Theory, PASTA.................................... 79 7.2.1 Rate conservation laws............................... 82 7.2.2 PASTA....................................... 82 7.3 Product-form Networks................................... 83 7.3.1 M=M=1 queue:................................... 83 7.3.2 Tandem Queues................................... 83 7.3.3 Open Jackson Networks.............................. 84 7.3.4 Closed queueing networks............................. 85 7.4 Product-Form Networks: Quasireversible networks.................... 86 7.4.1 Quasireversible Queues............................... 86 7.4.2 Networks of Quasireversible Queues........................ 88 7.4.2 Networks of quasireversible queues (contd.).................... 90 7.5 Problems.......................................... 93 2 Course structure • Two sessions per week. Tuesdays and Thursdays between 5:15 p.m. to 6:45 p.m. • One assignment per topic. There will be a quiz based on each assignment. Tutorial and quiz sessions will be held by TAs on Saturdays 10-11 a.m. after each topic is finished. • One mid-term exam. • One final exam. Reference Books [1] S. Assmussen, ”Applied Probability and Queues”, 2nd ed., Springer, 2003. [2] B. Hajek, ”Random Processes for Engineers”, Cambridge Univesity press, 2015. [3] S. Karlin and H.M. Taylor, ”A First Course in Stochastic Processes”, 2nd ed., 1975. [4] S.M. Ross, ”Stochastic Processes”,2nd ed., Wiley, 1996. [5] J. Walrand, ”An introduction to Queueing Netwoorks”, Printice Hall, 1988. 3 Chapter 1 Poisson Processes Lecture 1 Course E2.204: Stochastic Processes and Queueuing Theory (SPQT) Spring 2019 Instructor: Vinod Sharma Indian Institute of Science, Bangalore 1.1 Introduction to stochastic processes Review: Let (W;s;P) be a probability space. A measurable mapping X : W ! R is called a random variable (r.v.). X(w) for w 2 W is called a realization of X. FX (x) = P[X ≤ x] is called the distribution function of r.v. X. fX (x) = dF(x)=dx is called the probability density function of X. The probability den- R sity function may not always exist. E[X] = xdFX (x) is the expectation of X. When probability density R of X exists E[X] = x f (x)dx. Stochastic processes: fXt : t 2 Rg, where Xt is a r.v. is called a continuous time stochastic process. fXn : n 2 Ng, where Xn is a r.v. is called a discrete time stochastic process. The function t 7! Xt (w) is called a sample path of the stochastic process. For each w 2 W, Xt (w) is a function of t. Ft is the distribution of Xt . An analogous definition holds for discrete time stochastic processes. A stochastic process is described by the joint distribution of (Xt1 ;Xt2 ;:::;Xtn ) for any −¥ < + t1 < t2 < ::: < tn and n 2 N . A stochastic process fXt g is said to be stationary if for any 0 ≤ t1 < t2 < ··· < tn, the joint distribution of (Xt1 ;Xt2 ;:::;Xtn ) is identical to the joint distribution of (Xt1+t ;Xt2+t ;:::;Xtn+t ) for any t 2 R.A stochastic process fXt g is said to have independent increments if (Xt2 − Xt1 );(Xt3 − Xt2 );:::;(Xtn − Xtn−1 ) are independent. If joint distribution of (Xtn+t −Xtn−1+t ), (Xtn−2+t −Xtn−3+t );:::;(Xt2+t −Xt1+t ) does not depend on t, then fXt g is said to have stationary increments. If fXt g has both stationary and independent increments, it is called a stationary independent increment process. Point process: A stochastic process fNt ;t ≥ 0g with N0 = 0, Nt a non-negative integer, non-decreasing with piece-wise constant sample paths is called a point process. Nt counts the number of points or ’ar- rivals’ in the interval (0;t]. th th n Let An denote the interarrival time between n and (n−1) arrival. Let S0 = 0 and Sn = ∑k=1 Ak;8n ≥ th 1. Then Sn denotes the time instant of the n arrival. Nt = maxfn : Sn ≤ tg. A point process with at most one arrival at any time is called a simple point process. Mathematically, a simple point process fNt g is 4 described by following constraints for all t: PfNt+h − Nt ≥ 2g = o(h): f (x) Here, the notation o(g(x)) means a class of functions such that if f (x) 2 o(g(x)), then limx!0 g(x) = 0. 1.2 Poisson process 1.2.1 Definition In the following we customarily take N0 = 0. A point process Nt is Poisson if any of the following conditions hold. Definition [1]: −lx 1. fAk;k ≥ 1g are independent and exponentially distributed with parameter l: PfAk ≤ xg = 1−e . If l = 0, A1 = ¥ w.p.1. and Nt = 0 8t. If l = ¥, A1 = 0 w.p.1 and Nt = ¥ 8t. Thus, we restrict to 0 < l < ¥. In this range for l, Nt is guaranteed to be simple because PfAk = 0g = 0 8k. Definition [2]: 1. Nt is simple. 2. Nt has stationary independent increments. Definition [3]: 1. Nt has independent increment. 2. For s < t, (l(t − s))n fN − N = ng = e−l(t−s) P t s n! Definition [4]: 1. Nt has stationary and independent increments. 2. (a) PfNt+h − Nt = 1g = lh + o(h) (b) PfNt+h − Nt = 0g = 1 − lh + o(h) (c) PfNt+h − Nt ≥ 2g = o(h) We will show below that these definitions are equivalent. We need the following important charateri- zation of exponential distribution. 5 Exponential r.v. is memoryless: Let X be an exponential r.v. PfX > t + s;X > tg PfX > t + sjX > tg = PfX > tg fX > t + sg = P PfX > tg e−l(t+s) = e−lt = e−ls = PfX > sg If X is interarrival time, this property of an exponential r.v. indicates that the remaining time till next arrival does not depend on time t since last arrival. Thus, the term memoryless is used. + Theorem 1.2.1. Exponential distribution is the unique distribution on R with the memoryless property. + Proof. If a r.v. X on R is memoryless, we show that X must be exponential. If X is memoryless, we have for all t;s ≥ 0, PfX > t + sg = PfX > tgPfX > sg. Let f (t) = PfX > tg. We have the functional equation f (t + s) = f (t) f (s) (1.1) Taking t = s, we get f (2t) = f 2(t). By repeated application of Eq 1.1, m times, we get f (mt) = f m(t) 1 for positive integer m. Equivalently, we have f (t=m) = f m (t). Again by repeated application of Eq 1.1 n n m r n times, f ( mt) = f (t) for any positive integers m and n. So, we have f (rt) = f (t) for any positive rational number r. We know that 0 ≤ f (t) ≤ 1 since 1 − f is probability distribution. So, we can write f (1) = e−l for some l ≥ 0. Therefore we have, f (r) = f (r ×1) = f r(1) = e−lr for any positive rational number r. For any x 2 R, there is a sequence of rationals rn # x. Since f is right continuous, f (rn) ! f (x). In other words, for any x 2 R, f (x) = lim f (rn) rn!x = lim e−lrn rn!x = e−lx − x Thus, PfX > xg = e l and X is an exponential random variable. Now we show that definitions [1 − 4] for Poisson process given above are equivalent. Proposition 1.2.2. Definition [3] =) Definition [2]. Proof. We need to show that Nt has stationary increments if Nt has independent increments and Nt − Ns is Poisson distributed with mean l(t − s). Stationarity follows directly from the definition since the distribution of number of points in an interval depends only in the length of interval.
Load more

Recommended publications
  • Poisson Processes Stochastic Processes
    Poisson Processes Stochastic Processes UC3M Feb. 2012 Exponential random variables A random variable T has exponential distribution with rate λ > 0 if its probability density function can been written as −λt f (t) = λe 1(0;+1)(t) We summarize the above by T ∼ exp(λ): The cumulative distribution function of a exponential random variable is −λt F (t) = P(T ≤ t) = 1 − e 1(0;+1)(t) And the tail, expectation and variance are P(T > t) = e−λt ; E[T ] = λ−1; and Var(T ) = E[T ] = λ−2 The exponential random variable has the lack of memory property P(T > t + sjT > t) = P(T > s) Exponencial races In what follows, T1;:::; Tn are independent r.v., with Ti ∼ exp(λi ). P1: min(T1;:::; Tn) ∼ exp(λ1 + ··· + λn) . P2 λ1 P(T1 < T2) = λ1 + λ2 P3: λi P(Ti = min(T1;:::; Tn)) = λ1 + ··· + λn P4: If λi = λ and Sn = T1 + ··· + Tn ∼ Γ(n; λ). That is, Sn has probability density function (λs)n−1 f (s) = λe−λs 1 (s) Sn (n − 1)! (0;+1) The Poisson Process as a renewal process Let T1; T2;::: be a sequence of i.i.d. nonnegative r.v. (interarrival times). Define the arrival times Sn = T1 + ··· + Tn if n ≥ 1 and S0 = 0: The process N(t) = maxfn : Sn ≤ tg; is called Renewal Process. If the common distribution of the times is the exponential distribution with rate λ then process is called Poisson Process of with rate λ. Lemma. N(t) ∼ Poisson(λt) and N(t + s) − N(s); t ≥ 0; is a Poisson process independent of N(s); t ≥ 0 The Poisson Process as a L´evy Process A stochastic process fX (t); t ≥ 0g is a L´evyProcess if it verifies the following properties: 1.
    [Show full text]
  • Stochastic Processes and Applications Mongolia 2015
    NATIONAL UNIVERSITY OF MONGOLIA Stochastic Processes and Applications Mongolia 2015 27th July - 7th August 2015 National University of Mongolia Ulan Bator Mongolia National University of Mongolia 1 1 Basic information Venue: The meeting will take place at the National University of Mongolia. The map below shows the campus of the University which is located in the North-Eastern block relative to the Government Palace and Chinggis Khaan Square (see the red circle with arrow indicating main entrance in map below) in the very heart of down-town Ulan Bator. • All lectures, contributed talks and tutorials will be held in the Room 320 at 3rd floor, Main building, NUM. • Registration and Opening Ceremony will be held in the Academic Hall (Round Hall) at 2nd floor of the Main building. • The welcome reception will be held at the 2nd floor of the Broadway restaurant pub which is just on the West side of Chinggis Khaan Square (see the blue circle in map below). NATIONAL UNIVERSITY OF MONGOLIA 2 National University of Mongolia 1 Facilities: The main venue is equipped with an electronic beamer, a blackboard and some movable white- boards. There will also be magic whiteboards which can be used on any vertical surface. White-board pens and chalk will be provided. Breaks: Refreshments will be served between talks (see timetable below) at the conference venue. Lunches: Arrangements for lunches will be announced at the start of the meeting. Accommodation: Various places are being used for accommodation. The main accommodation are indi- cated on the map below relative to the National University (red circle): The Puma Imperial Hotel (purple circle), H9 Hotel (black circle), Ulanbaatar Hotel (blue circle), Student Dormitories (green circle) Mentoring: A mentoring scheme will be running which sees more experienced academics linked with small groups of junior researchers.
    [Show full text]
  • The Maximum of a Random Walk Reflected at a General Barrier
    The Annals of Applied Probability 2006, Vol. 16, No. 1, 15–29 DOI: 10.1214/105051605000000610 c Institute of Mathematical Statistics, 2006 THE MAXIMUM OF A RANDOM WALK REFLECTED AT A GENERAL BARRIER By Niels Richard Hansen University of Copenhagen We define the reflection of a random walk at a general barrier and derive, in case the increments are light tailed and have negative mean, a necessary and sufficient criterion for the global maximum of the reflected process to be finite a.s. If it is finite a.s., we show that the tail of the distribution of the global maximum decays exponentially fast and derive the precise rate of decay. Finally, we discuss an example from structural biology that motivated the interest in the reflection at a general barrier. 1. Introduction. The reflection of a random walk at zero is a well-studied process with several applications. We mention the interpretation from queue- ing theory—for a suitably defined random walk—as the waiting time until service for a customer at the time of arrival; see, for example, [1]. Another important application arises in molecular biology in the context of local com- parison of two finite sequences. To evaluate the significance of the findings from such a comparison, one needs to study the distribution of the locally highest scoring segment from two independent i.i.d. sequences, as shown in [8], which equals the distribution of the maximum of a random walk reflected at zero. The global maximum of a random walk with negative drift and, in par- ticular, the probability that the maximum exceeds a high value have also been studied in details.
    [Show full text]
  • Renewal Theory for Uniform Random Variables
    California State University, San Bernardino CSUSB ScholarWorks Theses Digitization Project John M. Pfau Library 2002 Renewal theory for uniform random variables Steven Robert Spencer Follow this and additional works at: https://scholarworks.lib.csusb.edu/etd-project Part of the Mathematics Commons Recommended Citation Spencer, Steven Robert, "Renewal theory for uniform random variables" (2002). Theses Digitization Project. 2248. https://scholarworks.lib.csusb.edu/etd-project/2248 This Thesis is brought to you for free and open access by the John M. Pfau Library at CSUSB ScholarWorks. It has been accepted for inclusion in Theses Digitization Project by an authorized administrator of CSUSB ScholarWorks. For more information, please contact [email protected]. RENEWAL THEORY FOR UNIFORM RANDOM VARIABLES A Thesis Presented to the Faculty of California State University, San Bernardino In Partial Fulfillment of the Requirements for the Degree Master of Arts in Mathematics by Steven Robert Spencer March 2002 RENEWAL THEORY FOR UNIFORM RANDOM VARIABLES A Thesis Presented to the Faculty of California State University, San Bernardino by Steven Robert Spencer March 2002 Approved by: Charles Stanton, Advisor, Mathematics Date Yuichiro Kakihara Terry Hallett j. Peter Williams, Chair Terry Hallett, Department of Mathematics Graduate Coordinator Department of Mathematics ABSTRACT The thesis answers the question, "How many times must you change a light-bulb in a month if the life-time- of any one light-bulb is anywhere from zero to one month in length?" This involves uniform random variables on the - interval [0,1] which must be summed to give expected values for the problem. The results of convolution calculations for both the uniform and exponential distributions of random variables give expected values that are in accordance with the Elementary Renewal Theorem and renewal function.
    [Show full text]
  • Extinction, Survival and Duality to P-Jump Processes
    Markov branching processes with disasters: extinction, survival and duality to p-jump processes by Felix Hermann and Peter Pfaffelhuber Albert-Ludwigs University Freiburg Abstract A p-jump process is a piecewise deterministic Markov process with jumps by a factor of p. We prove a limit theorem for such processes on the unit interval. Via duality with re- spect to probability generating functions, we deduce limiting results for the survival proba- bilities of time-homogeneous branching processes with arbitrary offspring distributions, un- derlying binomial disasters. Extending this method, we obtain corresponding results for time-inhomogeneous birth-death processes underlying time-dependent binomial disasters and continuous state branching processes with p-jumps. 1 Introduction Consider a population evolving according to a branching process ′. In addition to reproduction events, global events called disasters occur at some random times Z(independent of ′) that kill off every individual alive with probability 1 p (0, 1), independently of each other.Z The resulting process of population sizes will be called− a branching∈ process subject to binomial disasters with survival probability p. ProvidedZ no information regarding fitness of the individuals in terms of resistance against disasters, this binomial approach appears intuitive, since the survival events of single individuals are iid. Applications span from natural disasters as floods and droughts to effects of radiation treatment or chemotherapy on cancer cells as well as antibiotics on popula- tions of bacteria. Also, Bernoulli sampling comes into mind as in the Lenski Experiment (cf. arXiv:1808.00073v2 [math.PR] 7 Jan 2019 Casanova et al., 2016). In the general setting of Bellman-Harris processes with non-lattice lifetime-distribution subject to binomial disasters, Kaplan et al.
    [Show full text]
  • Joint Pricing and Inventory Control for a Stochastic Inventory System With
    Joint pricing and inventory control for a stochastic inventory system with Brownian motion demand Dacheng Yao Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing, 100190, China; [email protected] Abstract In this paper, we consider an infinite horizon, continuous-review, stochastic inventory system in which cumulative customers' demand is price-dependent and is modeled as a Brownian motion. Excess demand is backlogged. The revenue is earned by selling products and the costs are incurred by holding/shortage and ordering, the latter consists of a fixed cost and a proportional cost. Our objective is to simultaneously determine a pricing strategy and an inventory control strategy to maximize the expected long-run average profit. Specifically, the pricing strategy provides the price pt for any time t ≥ 0 and the inventory control strategy characterizes when and how much we need to order. We show that an (s∗;S∗; p∗) policy is ∗ ∗ optimal and obtain the equations of optimal policy parameters, where p = fpt : t ≥ 0g. ∗ Furthermore, we find that at each time t, the optimal price pt depends on the current inventory level z, and it is increasing in [s∗; z∗] and is decreasing in [z∗; 1), where z∗ is a negative level. Keywords: Stochastic inventory model, pricing, Brownian motion demand, (s; S; p) policy, impulse control, drift rate control. arXiv:1608.03033v2 [math.OC] 11 Jul 2017 1 Introduction Exogenous selling price is always assumed in the classic production/inventory models; see e.g., Scarf(1960) and the research thereafter. In practice, however, dynamic pricing is one important tool for revenue management by balancing customers' demand and inventory level.
    [Show full text]
  • Central Limit Theorem for Supercritical Binary Homogeneous Crump-Mode
    Central limit theorem for supercritical binary homogeneous Crump-Mode-Jagers processes Benoit Henry1,2 Abstract We consider a supercritical general branching population where the lifetimes of individuals are i.i.d. with arbitrary distribution and each individual gives birth to new individuals at Poisson times independently from each others. The population counting process of such population is a known as binary homogeneous Crump-Jargers-Mode process. It is known that such processes converges almost surely when correctly renormalized. In this paper, we study the error of this convergence. To this end, we use classical renewal theory and recent works [17, 6, 5] on this model to obtain the moments of the error. Then, we can precisely study the asymptotic behaviour of these moments thanks to L´evy processes theory. These results in conjunction with a new decomposition of the splitting trees allow us to obtain a central limit theorem. MSC 2000 subject classifications: Primary 60J80; secondary 92D10, 60J85, 60G51, 60K15, 60F05. Key words and phrases. branching process – splitting tree – Crump–Mode–Jagers process – linear birth–death process – L´evy processes – scale function – Central Limit Theorem. 1 Introduction In this work, we consider a general branching population where individuals live and reproduce inde- pendently from each other. Their lifetimes follow an arbitrary distribution PV and the births occur at Poisson times with constant rate b. The genealogical tree induced by this population is called a arXiv:1509.06583v2 [math.PR] 18 Nov 2016 splitting tree [11, 10, 17] and is of main importance in the study of the model. The population counting process Nt (giving the number of living individuals at time t) is a binary homogeneous Crump-Mode-Jagers (CMJ) process.
    [Show full text]
  • Renewal and Regenerative Processes
    Chapter 2 Renewal and Regenerative Processes Renewal and regenerative processes are models of stochastic phenomena in which an event (or combination of events) occurs repeatedly over time, and the times between occurrences are i.i.d. Models of such phenomena typically focus on determining limiting averages for costs or other system parameters, or establishing whether certain probabilities or expected values for a system converge over time, and evaluating their limits. The chapter begins with elementary properties of renewal processes, in- cluding several strong laws of large numbers for renewal and related stochastic processes. The next part of the chapter covers Blackwell’s renewal theorem, and an equivalent key renewal theorem. These results are important tools for characterizing the limiting behavior of probabilities and expectations of stochastic processes. We present strong laws of large numbers and central limit theorems for Markov chains and regenerative processes in terms of a process with regenerative increments (which is essentially a random walk with auxiliary paths). The rest of the chapter is devoted to studying regenera- tive processes (including ergodic Markov chains), processes with regenerative increments, terminating renewal processes, and stationary renewal processes. 2.1 Renewal Processes This section introduces renewal processes and presents several examples. The discussion covers Poisson processes and renewal processes that are “embed- ded” in stochastic processes. We begin with notation and terminology for point processes that we use in later chapters as well. Suppose 0 ≤ T1 ≤ T2 ≤ ... are finite random times at which a certain event occurs. The number of the times Tn in the interval (0,t]is ∞ N(t)= 1(Tn ≤ t),t≥ 0.
    [Show full text]
  • Applications of Renewal Theory to Pattern Analysis
    Rochester Institute of Technology RIT Scholar Works Theses 4-18-2016 Applications of Renewal Theory to Pattern Analysis Hallie L. Kleiner [email protected] Follow this and additional works at: https://scholarworks.rit.edu/theses Recommended Citation Kleiner, Hallie L., "Applications of Renewal Theory to Pattern Analysis" (2016). Thesis. Rochester Institute of Technology. Accessed from This Thesis is brought to you for free and open access by RIT Scholar Works. It has been accepted for inclusion in Theses by an authorized administrator of RIT Scholar Works. For more information, please contact [email protected]. APPLICATIONS OF RENEWAL THEORY TO PATTERN ANALYSIS by Hallie L. Kleiner A Thesis Submitted in Partial Fulfillment of the Requirements for the Degree of Master of Science in Applied Mathematics School of Mathematical Sciences, College of Science Rochester Institute of Technology Rochester, NY April 18, 2016 Committee Approval: Dr. James Marengo Date School of Mathematical Sciences Thesis Advisor Dr. Bernard Brooks Date School of Mathematical Sciences Committee Member Dr. Manuel Lopez Date School of Mathematical Sciences Committee Member Dr. Elizabeth Cherry Date School of Mathematical Sciences Director of Graduate Programs RENEWAL THEORY ON PATTERN ANALYSIS Abstract In this thesis, renewal theory is used to analyze patterns of outcomes for discrete random variables. We start by introducing the concept of a renewal process and by giving some examples. It is then shown that we may obtain the distribution of the number of renewals by time t and compute its expectation, the renewal function. We then proceed to state and illustrate the basic limit theorems for renewal processes and make use of Wald’s equation to give a proof of the Elementary Renewal Theorem.
    [Show full text]
  • Exploiting Regenerative Structure to Estimate Finite Time Averages Via Simulation
    EXPLOITING REGENERATIVE STRUCTURE TO ESTIMATE FINITE TIME AVERAGES VIA SIMULATION Wanmo Kang, Perwez Shahabuddin* and Ward Whitt Columbia University We propose nonstandard simulation estimators of expected time averages over ¯nite intervals [0; t], seeking to enhance estimation e±ciency. We make three key assumptions: (i) the underlying stochastic process has regenerative structure, (ii) the time average approaches a known limit as time t increases and (iii) time 0 is a regeneration time. To exploit those properties, we propose a residual-cycle estimator, based on data from the regenerative cycle in progress at time t, using only the data after time t. We prove that the residual-cycle estimator is unbiased and more e±cient than the standard estimator for all su±ciently large t. Since the relative e±ciency increases in t, the method is ideally suited to use when applying simulation to study the rate of convergence to the known limit. We also consider two other simulation techniques to be used with the residual-cycle estimator. The ¯rst involves overlapping cycles, paralleling the technique of overlapping batch means in steady-state estimation; multiple observations are taken from each replication, starting a new observation each time the initial regenerative state is revisited. The other technique is splitting, which involves independent replications of the terminal period after time t, for each simulation up to time t. We demonstrate that these alternative estimators provide e±ciency improvement by conducting simulations of queueing models. Categories and Subject Descriptors: G.3 [Mathematics of Computing]: Probability and Statis- tics|renewal theory; I.6.6 [Computing Methodologies]: Simulation and Modelling|simula- tion output analysis General Terms: Algorithms, Experimentation, Performance, Theory Additional Key Words and Phrases: e±ciency improvement, variance reduction, regenerative processes, time averages 1.
    [Show full text]
  • Stochastic Models Laws of Large Numbers and Functional Central
    This article was downloaded by: [Stanford University] On: 20 July 2010 Access details: Access Details: [subscription number 731837804] Publisher Taylor & Francis Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37- 41 Mortimer Street, London W1T 3JH, UK Stochastic Models Publication details, including instructions for authors and subscription information: http://www.informaworld.com/smpp/title~content=t713597301 Laws of Large Numbers and Functional Central Limit Theorems for Generalized Semi-Markov Processes Peter W. Glynna; Peter J. Haasb a Department of Management Science and Engineering, Stanford University, Stanford, California, USA b IBM Almaden Research Center, San Jose, California, USA To cite this Article Glynn, Peter W. and Haas, Peter J.(2006) 'Laws of Large Numbers and Functional Central Limit Theorems for Generalized Semi-Markov Processes', Stochastic Models, 22: 2, 201 — 231 To link to this Article: DOI: 10.1080/15326340600648997 URL: http://dx.doi.org/10.1080/15326340600648997 PLEASE SCROLL DOWN FOR ARTICLE Full terms and conditions of use: http://www.informaworld.com/terms-and-conditions-of-access.pdf This article may be used for research, teaching and private study purposes. Any substantial or systematic reproduction, re-distribution, re-selling, loan or sub-licensing, systematic supply or distribution in any form to anyone is expressly forbidden. The publisher does not give any warranty express or implied or make any representation that the contents will be complete or accurate or up to date. The accuracy of any instructions, formulae and drug doses should be independently verified with primary sources. The publisher shall not be liable for any loss, actions, claims, proceedings, demand or costs or damages whatsoever or howsoever caused arising directly or indirectly in connection with or arising out of the use of this material.
    [Show full text]
  • Renewal Monte Carlo: Renewal Theory Based Reinforcement Learning Jayakumar Subramanian and Aditya Mahajan
    1 Renewal Monte Carlo: Renewal theory based reinforcement learning Jayakumar Subramanian and Aditya Mahajan Abstract—In this paper, we present an online reinforcement discounted and average reward setups as well as for models learning algorithm, called Renewal Monte Carlo (RMC), for with continuous state and action spaces. However, they suffer infinite horizon Markov decision processes with a designated from various drawbacks. First, they have a high variance start state. RMC is a Monte Carlo algorithm and retains the advantages of Monte Carlo methods including low bias, simplicity, because a single sample path is used to estimate performance. and ease of implementation while, at the same time, circumvents Second, they are not asymptotically optimal for infinite horizon their key drawbacks of high variance and delayed (end of models because it is effectively assumed that the model is episode) updates. The key ideas behind RMC are as follows. episodic; in infinite horizon models, the trajectory is arbitrarily First, under any reasonable policy, the reward process is ergodic. truncated to treat the model as an episodic model. Third, the So, by renewal theory, the performance of a policy is equal to the ratio of expected discounted reward to the expected policy improvement step cannot be carried out in tandem discounted time over a regenerative cycle. Second, by carefully with policy evaluation. One must wait until the end of the examining the expression for performance gradient, we propose a episode to estimate the performance and only then can the stochastic approximation algorithm that only requires estimates policy parameters be updated. It is for these reasons that of the expected discounted reward and discounted time over a Monte Carlo methods are largely ignored in the literature on regenerative cycle and their gradients.
    [Show full text]