IMA BOOTCAMP: STOCHASTIC MODELING

JASMINE FOO AND KEVIN LEDER

Note: much of the material in these bootcamp lecture notes is adapted from the book ‘Introduction to Probability Models’ by Sheldon Ross and lecture notes on the web by Janko Gravner for MAT135B.

1. Exponential random variables and their properties We begin with a review of exponential random variables, which appear a lot in mathe- matical models of real-world phenomena because of their convenient mathematical prop- erties. Definition 1. A continuous random variable X is said to have an with parameter λ, λ > 0, if its probability density function is given by ( λe−λx, x ≥ 0 f(x) = 0, x < 0 1.1. Moments. The mean of the exponential distribution, E[X] is given by Z ∞ 1 E[X] = xλe−λxdx = . −∞ λ The moment generating function φ(t) of the exponential distribution is Z ∞ λ φ(t) = E[etX ] = etxλe−λxdx = . 0 λ − t Using this we can derive the variance 2 2 2 d 1 1 V ar(X) = E[X ] − (EX) = 2 φ(t) − 2 = 2 . dt t=0 λ λ 1.2. Memoryless property. Exponential distributions have the memoryless property. Specifically, it satisfies the following: P (X > s + t|X > t) = P (X > s), for all s, t ≥ 0. To see this, note that the property is equivalent to P (X > s + t, X > t) = P (X > s) P (X > t) or P (X > s + t) = P (X > s)P (X > t). 1 2 JASMINE FOO AND KEVIN LEDER

For X exponentially distributed, we get e−λ(s+t) = e−λse−λt so the memoryless property is satisfied. It can be shown that the exponential distribution is the only distribution that satisfies this property. Example 1. A store must decide how much product to order to meet next month’s demand. The demand is modeled as an exponential random variable with rate λ. The product costs c dollars per unit, and can be sold at a price of s > c dollars per unit. How much should be ordered to maximize the store’s expected profit? Assume that any inventory left over at the end of the month is worthless. Solution. Let X be the demand. If the store orders t units of the product, the profit is P = s min(X, t) − ct In other words ( sX − ct, X < t P = (s − c)t, X ≥ t So E[P ] = (sE[X|X < t] − ct)P (X < t) + (s − c)tP (X ≥ t). To calculate this conditional , we note that 1/λ = E[X] = E[X|X < t]P (X < t) + E[X|X ≥ t]P (X ≥ t) = E[X|X < t](1 − eλt) + (t + EX)e−λt where in the last line we have used the memoryless property: conditioned on having exceeded t, the amount by which X exceeds t is distributed as an exponential variable with parameter λ. Thus we have 1 − (t + 1 )e−λt E[X|X < t] = λ λ 1 − e−λt Plugging this into the expression for E[P ] we have 1 − (t + 1 )e−λt E[P ] = (s λ λ − ct)(1 − e−λt) + (s − c)teλt 1 − e−λt = s/λ − se−λt/λ − ct To maximize the expected profits, we take the derivative and set se−λt − c = 0, obtaining that the maximal profit is obtained when 1 t = log (s/c). λ Definition 2. The hazard rate or failure rate of a random variable with cumulative distri- bution function F and density f is: f(t) h(t) = . 1 − F (t) IMA BOOTCAMP: STOCHASTIC MODELING 3

Consider the following related quantity: conditioned on X > t, what is the probability that X ∈ (t, t + dt)? P (X ∈ (t, t + dt), X > t) P (X ∈ (t, t + dt)) f(t)dt P (X ∈ (t, t + dt)|X > t) = = ≈ P (X > t) P (X > t) 1 − F (t) So, if we think of X as the time to failure of some object, the hazard rate can be interpreted as the conditional probability density of failure at time t, given that the object has not failed until time t. The failure rate of an exponentially distributed random variable is a constant: h(t) = λe−λteλt = λ

1.3. Sum and minimums of exponential random variables. Suppose Xi, i = 1 . . . n are independent identically distributed exponential random variables with parameter λ. It can be shown (by induction, for example), that the sum

X1 + X2 + ... + Xn has a Gamma distribution with parameters n and 1/λ, with pdf (λt)n−1 f(t) = λe−λt . (n − 1)! Next, let’s define Y to be the minimum of n independent exponential random variables, X1,...,Xn with parameters λ1, . . . , λn. n Y P (Y > x) = P (Xi > x) i=1 n Y = e−λix i=1 − Pn λ x = e ( i=1 i) Thus we see that the minimum, Y , is exponentially distributed with parameter equal to the sum of the parameters λ1 + λ2 ··· + λn.

2. Poisson Processes Definition 3. A counting process is a random process N(t), t ≥ 0 (e.g. number of events that have occurred by time t) such that (1) For each t, N(t) is a nonnegative integer (2) N(t) is nondecreasing in t (3) N(t) is right continuous. Thus N(t) − N(s) represents the number of events in (s, t]. Definition 4. A Poisson process is a random process with intensity (or rate) λ > 0 is a counting process N(t) such that 4 JASMINE FOO AND KEVIN LEDER

(1) N(0) = 0 (2) it has independent increments: if (s1, t1] ∩ (st, t2] = ∅ then N(t1) − N(s1) and N(t2) − N(s2) are independent (3) number of events in any interval of length t is Poisson(λt) Recall that a Poisson random variable Y with parameter λ takes values k = 0, 1, 2,... with probability λke−λ P (Y = k) = k! In particular, we have the probability mass function for the number of events in an interval of size h: (λh)k P (N(t + h) − N(t) = k) = e−λh , k = 0, 1, 2,... k! Now, if h → 0 we have that the number of arrivals in a small interval of size h satisfies P (N(h) = 1) = λh and P (N(h) ≥ 2) = o(h). Also, from the properties of a Poisson random variable we have that E[N(t)] = λt. 2.1. Motivation for the Poisson process. Suppose we have a store where the customers arrive at a rate λ = 4.5 customers per hour on average, and the store owner is interested to find the distribution of the average number X of customers who arrive during a particular time period of length t hours. The arrival rate can be recast as 4.5/3600 seconds per hour = 0.00125 per second. We can interpret this probabilistically by saying that during each second either 0 or 1 customers will arrive, and the probability during any single second is 0.00125. Under this setup the Binomial(3600t, 0.00125) distribution describes X, the number of customers who arrive during a time period of length t. Using the Poisson approximation to the Binomial we then have that X is approximated by a Poisson variable with mean 3600t ∗ 0.00125 = 4.5t = λt, which is consistent with our definition of a Poisson process. Recall that property 3 of a Poisson process is that the number of arrivals in any given time period of length t is Poisson(λt). In addition, by construction we have that the number of arrivals in non overlapping time intervals is independent. This example motivates the definition of a Poisson process, which is often used to model arrival processes of events with a constant average arrival rate. 2.2. Interarrival and waiting time distributions. Consider a Poisson process with rate λ and let T1 denote the time of the first event, and for n > 1 let Tn denote the elapsed time between the (n − 1)st and nth event. The sequence {Tn, n = 1, 2,...} is called the sequence of inter arrival times.

Proposition 1. Tn, n = 1, 2,... are independent identically distributed exponential random variables with parameter λ. IMA BOOTCAMP: STOCHASTIC MODELING 5

To see this, first consider T1. −λt P (T1 > t) = P (N(t) = 0) = e

Thus T1 is exponentially distributed with parameter λ. Then Z ∞ P (T2 > t) = P (T2 > t|T1 ∈ (s, s + ds))P (T1 ∈ (s, s + ds))ds 0 Z ∞ = P (no events in (s, s+t])λe−λsds 0 Z ∞ = e−λtλe−λsds 0 = e−λt

So, T2 is also exponentially distributed with parameter λ. From this argument we can also see that T2 is independent of T1 since conditioning on the value of T1 has no impact on the pdf of T2. Repeating the argument for n ≥ 2 gives the result. Intuitively, remember that the Poisson process has independent increments (so from any point onward it is independent of what happened in the past) and stationary increments (any increment of the same length has the same distribution), so the process has no memory. Hence exponential waiting times until the next arrival are expected. Next we consider the waiting time until the nth event, Sn, and its distribution. Note that n X Sn = Ti i=1 and since the Ti are iid exponential(λ) variables, the variable Sn is Gamma(n, λ). In other words, Sn has pdf (λt)n−1 f (t) = λe−λt , t ≥ 0. Sn (n − 1)! Note that we could also equivalently define a Poisson process by starting with iid ex- ponential inter arrival times with parameter λ, and then defining a counting process by saying that the nth event occurs at time

Sn = T1 + T2 + ... + Tn. The resulting counting process would be a Poisson process with rate λ.

3. Continuous time Markov Chains Next we consider a class of probability models that are widely used in modeling real- world applications: continuous-time Markov Chains. These stochastic processes take place in continuous time and have the property that, given the present state, the future is independent of the past. The Poisson process is an example of a continuous-time . 6 JASMINE FOO AND KEVIN LEDER

Definition 5. Suppose we have a continuous time {X(t), t ≥ 0} taking values in the nonnegative integers. The process X(t) is a continuous time Markov Chain if for all s, t ≥ 0 and nonnegative integers i, j, x(u), 0 ≤ u < s P (X(t + s) = j|X(s) = i, X(u) = x(u), 0 ≤ u < s} = P (X(t + s) = j|X(s) = i). If, additionally, P (X(t + s) = j|X(s) = i) is independent of s, then X is said to have homogeneous transition probabilities. In the following, we will always work with Markov chains with homogeneous transition probabilities. The Markovian property tells us that, no matter how long the chain has been in a particular state, the time it waits until transitioning to a new state has the same distribution (i.e. the waiting time between transitions has the memoryless property, and must thus be exponentially distributed). Thus, we can think of a continuous-time Markov chain as moving according to a discrete time Markov chain except that the time between jumps is exponentially distributed.

Example 2. A population of size N has It infected individuals, St susceptible individuals and Rt recovered/removed individuals. New infections occur at an exponential rate βItSt and infected individuals become removed/recovered at an exponential rate Y (i.e. the overall rate of leaving the infected state is YIt.) Here, the state of the system is represented by the number of people in each state at any time t. 3.1. Birth and Death processes. Let us consider a system whose state at any time is represented simply by the number of people in the system. Whenever there are n people in the system, new arrivals enter at an exponential rate λn and people leave at an exponential rate µn. This system is called a birth and death process, and it is a continuous time Markov chain with states 0, 1,.... The rates λn and µn are called the birth and death rates. This chain can only move up or down by one unit at a time. If the system is in state n > 0, the time it takes until the next arrival is exponentially distributed with parameter λn, and the time it takes until the next death is exponentially distributed with parameter µn. The time the system stays in state n is the minimum of these two exponentials, an exponential with rate λn + µn. If n = 0 then it takes an exponential(λ0) amount of time for the system to reach state 1. The transition probabilities Pi,j tell us the probability of moving from state i at the next transition if the system is in state j. Here we have

P0,1 = 1 and λi Pi,i+1 = λi + µi which is the probability that the next arrival/birth occurs before the next death, and can be derived from the properties of the exponential distribution. Lastly µi Pi,i−1 = λi + µi IMA BOOTCAMP: STOCHASTIC MODELING 7

Example 3. A Poisson process is a birth and death process with λn = λ and µn = 0 for all n ≥ 0. This is also called a pure birth process.

Example 4. A Yule process is a birth-death process with λn = λn and µn = 0, for all n ≥ 0. This models the scenario in which each member of the population can give birth to a new member and takes an exponential(λ) amount of time to give birth, but no deaths occur. This is also an example of a pure birth process.

Example 5. A linear birth-death process has λn = λn and µn = µn, for all n ≥ 0. This models the scenario in which each member of the population can give birth to a new member and takes an exponential(λ) amount of time to give birth. At the same time, each member can die at exponential rate µn. Example 6. Queueing system M/M/1. Suppose that customers arrive at a queue as a Poisson process with rate λ. If the queue is empty the customer goes directly into service; otherwise he/she goes to wait at the end of the queue. The server serves each customer in order of the queue, and the service times are independent exponential variables with parameter µ. This is the M/M/1 queueing system. It is a birth-death process with λn = λ for n ≥ 0 and µn = µ for n ≥ 1.

Let us return to a general birth and death process with birth and death rates λn and µn, where µ0 = 0. Let Ti be the time, starting from state i, that it takes the process to go to state i + 1, for any i ≥ 0. For i = 0, the only way for the system to reach state 1 is to get a birth, so 1 E[T0] = λ0 For i > 0 the time it takes depends on whether the first transition is a birth or a death. Let ( 1, if the first transition is to i + 1 Ii = 0, if the first transition is to i − 1 Now

E[Ti] = E[Ti|Ii = 1]P (Ii = 1) + E[Ti|Ii = 0]P (Ii = 0)   1 λi 1 µi = + + E[Ti−1] + E[Ti] λi + µi λi + µi λi + µi λi + µi 1 µi = + (E[Ti−1] + E[Ti]) λi + µi λi + µi

Here we have used the fact that the distribution of Ti conditioned on the first transition being to i + 1 or i − 1 is still exponential with parameter λi + µi. This can be seen by the following: P (T ∈ dt, I = 1) λ e−λite−µit i 1 1 −(λi+µi)t P (Ti ∈ dt|I1 = 1) = = = (λi + µi)e . λi λi λi+µi λi+µi 8 JASMINE FOO AND KEVIN LEDER

Solving we obtain 1 µi E[Ti] = + E[Ti−1], i ≥ 1. λi λi

This equation can be solved to get estimates recursively for E[T1],E[T2],....

3.2. Limiting probabilities. Definition 6. Let

Pij(t) ≡ P (X(t + s) = j|X(s) = i) denote the probability that a process in state i will be in state j t time units later. These quantities are called the transition probabilities of the continuous-time Markov chain. Example 7. Let us derive the transition probabilities for a pure birth process X(t) with birth rates λn, for n ≥ 0. Consider the quantity, for i < j, P (X(t) < j|X(0) = i) which is the probability that the process has not yet reached state j by the time t. We can think of the time before the process hits state j as the sum of the times the process has spent in each of the states k = 1, . . . , j − 1. Let Xi denote the amount of time the process spends in state k before moving to k + 1. We are essentially interested in the equivalent quantity

j−1 X P ( Xk > t). k=i

Now recall that the Xk are independent exponentials with with parameters λk. This quantity can be derived directly from the properties of the exponential distribution (we will not show it here); it can be shown that

j−1 j−1 j−1 λ X X Y l −λkt P (X(t) < j|X(0) = i) = P ( Xk > t) = e λl − λk k=i k=i l6=k We can also find

j j j λ X X Y l −λkt P (X(t) < j + 1|X(0) = i) = P ( Xk > t) = e λl − λk k=i k=i l6=k Thus we have, for i < j

j−1 j−1 j j λ λ X Y l −λkt X Y l −λkt Pij(t) = P (X(t) = j|X(0) = i) = e − e λl − λk λl − λk k=i l6=k k=i l6=k and of course −λit Pii(t) = P (Xi > t) = e IMA BOOTCAMP: STOCHASTIC MODELING 9

These transition probabilities are governed by a set of differential equations called the Kolmogorov forward and backward equations. In order to state them, we need to define a few quantities. For any pair of states i, j let vi be the rate at which the process makes a transition when in state i, and let qij = viPij. Then, qij are instantaneous transition rates; qij represents the rate, when in state i, at which the process makes a transition into state j. Note that X X vi = viPij = qij j j

Now we are ready to state the Kolmogorov backward equations. We will not prove that Pij satisfies these equations here, but the derivation can be found in Ross Chapter 6. For all states i, j and t ≥ 0, 0 X Pij(t) = qikPkj(t) − viPij(t) k6=i Example 8. The backwards equations for the pure birth process are 0 Pij(t) = λiPi+1,j(t) − λiPij(t)

Similarly, it can be shown that the Pij satisfy the Kolmogorov forward equations, given by: 0 X Pij(t) = qkjPik(t) − viPij(t) k6=i Next we will consider how to find the limiting probability that a continuous time Markov chain will be in state j: Pj = lim Pij(t), t→∞ where we are assuming that the limit exists and is independent of the initial state i. Let us consider the forward equations, taking a limit as t → ∞ and assuming that the limit and summation can be interchanged gives us   0 X lim Pij(t) = lim  qkjPik(t) − viPij(t) t→∞ t→∞ k6=i X = qkjPk − vjPj k6=i 0 Since Pij(t) is a bounded function it follows that if Pij(t) converges it must converge to zero. So X 0 = qkjPk − vjPj k6=j i.e. X (1) vjPj = qkjPk k6=j 10 JASMINE FOO AND KEVIN LEDER P Solving this system along with the constraint that j Pj = 1 allows us to solve for the limiting probabilities. This equation balances the rate at which the process enters and leaves state j. Example 9. Let us consider the limiting probabilities for a birth and death process. Using equation (1) we have

µj+1 λj−1 (λj + µj)Pj = (λj+1 + µj+1) Pj+1 + (λj−1 + µj−1) Pj−1 (λj+1 + µj+1) (λj−1 + µj−1) = µj+1Pj+1 + λj−1Pj−1 for j ≥ 1 and λ0P0 = µ1P1 for j = 0. This set of equations can be simplified to

λ0P0 = µ1P1

λ1P1 = µ2P2

λ2P2 = µ3P3 . . Solving this we obtain

λ0 P1 = P0 µ1 λ1 λ1λ0 P2 = P1 = P1 µ2 µ2µ1 . .

λn−1λn−2 ··· λ0 Pn = P0 µnµn−1 ··· µ1 P Combining with the fact that j Pj = 1 we obtain 1 P0 = 1 + P∞ λn−1λn−2···λ0 n=1 µnµn−1···µ1 and for n ≥ 1 λn−1λn−2 ··· λ0 Pn = µ µ ··· µ (1 + P∞ λn−1λn−2···λ0 ) n n−1 1 n=1 µnµn−1···µ1

4. Renewal Process Consider a lightbulb in a room. Each time it burns out we go and replace it, if we count the number of times we replace it then we arrive at a stochastic process called a ‘renewal process’. Each time the bulb is replaced we say a renewal has occurred. IMA BOOTCAMP: STOCHASTIC MODELING 11

Definition of Renewal process. Let {Xn}n≥1 be an i.i.d sequence of positive random variables. Define S0 = 0 and for n ≥ 1 set n X Sn = Xi i=1 then for t ≥ 0 define N(t) = max{n : Sn ≤ t}.

The process {N(t); t ≥ 0} is called a renewal process. The sequence {Sn}n≥1 are the times at which renewals occur and N(t) counts the number of renewals in the interval [0, t]. Since the i.i.d random variables Xi > 0 we know that µ = E[X1] > 0 and therefore by the Sn → ∞ as n → ∞. This implies that N(t) < ∞ for all t < ∞. Also know that N(∞) = lim N(t) = ∞, t→∞ since ∞ ! [ P (N(∞) < ∞) = P (Xn = ∞ some n) = P {Xn = ∞} = 0. n=1 Distribution of N(t) is obtained using the result that the two events

{N(t) ≥ n} ⇐⇒ {Sn ≤ t}. Therefore P (N(t) = n) = P (N(t) ≥ n) − P (N(t) ≥ n + 1)

= P (Sn ≤ t) − P (Sn+1 ≤ t).

If we can find CDF of Sn then we can find PMF of N(t). This can be difficult in general so often study mean of N(t) and asymptotic properties of N(t). The mean of renewal process is given by ∞ ∞ X X m(t) = E[N(t)] = P (N(t) ≥ n) = P (Sn ≤ t). n=1 n=1

If X1 has PDF f and CDF F then can show that m satisfies a renewal equation: Z t m(t) = F (t) + m(t − x)f(x)dx, 0 can occasionally solve to get explicit formula for m(t). Can also show that N(t) 1 (2) lim = . t→∞ t µ Example 10. Suppose customers arrive at a server according to a Poisson process with rate λ. If customer sees server is busy customer leaves, if server is free the customer is served. Service times are i.i.d random variables with mean µS. What is the rate of arrival for customers that stay? 12 JASMINE FOO AND KEVIN LEDER

Answer: Denote an arbitrary service time by S. By memoryless property of exponential distribution the time between arrival of staying customers is S + X, where X is an exponential random variable with mean 1/λ. Therefore the mean time between arrivals of customers that stay is 1 + µ λ µ = µ + 1/λ = S . S λ If N(t) is number of customer that stay that have arrived by time t then N(t) is a renewal process and from (2) we see that long run average arrival rate is 1/µ. 4.1. Renewal-Reward Processes. Let {N(t); t ≥ 0} be a renewal process with inter- arrival times {Xn}n≥1. At nth renewal we receive an award Rn, we assume that the sequence {(Xn,Rn)}n≥1 is an i.i.d sequence. The total reward earned by time t is given by

N(t) X R(t) = Rn. n=1 A good analogy is to think of a taxi-cab. They drive around and pick up a customer, at the end of the ride they are given a reward (note that the reward is correlated with the time between pickups). Renewal-Reward processes are useful because of the following theorem.

Theorem 11. If E[X1] < ∞ and E[R1] < ∞ then R(t) E[R] (i) limt→∞ t = E[X] . E[R(t)] E[R] (ii) limt→∞ t = E[X] . Terminology: Say a ‘cycle’ occurs with every renewal, then this theorem says long run average reward is E[reward during cycle] . E[length of cycle] Example 12. Lifetime of a car is a random variable with PDF f and CDF F . Mr. Smith has a policy to purchase a new car if (i) old one breaks down (ii) new one reaches T years of age.

A new car costs C1 and a breakdown costs him an extra C2. What is Mr. Smith’s long run average cost? Answer: From theorem, long run average cost is given by E[reward during cycle] E[C(T )] = . E[length of cycle] E[L(T )] IMA BOOTCAMP: STOCHASTIC MODELING 13

We first find expected cost during cycle, E[C(T )]. If X is lifetime of car during cycle then ( C , X > T C(T ) = 1 C1 + C2,X ≤ T. Therefore

E[C(T )] = C1P (X > T ) + (C1 + C2)P (X ≤ T ) = C1 + C2F (T ). We now calculate the length of the cycle: ( X,X ≤ T L(T ) = T, X > T. Therefore Z T Z ∞ Z T E[L(T )] = xf(x)dx + T f(x)dx = xf(x)dx + T (1 − F (T )). 0 T 0 We can now calculate long run average cost as C + C F (T ) 1 2 . R T 0 xf(x)dx + T (1 − F (T )) Example 13. Manufacturing process sequentially produces items. Goal is to minimize number of defective items shipped. Use following procedure for quality control: (i) Initially inspect every item. Do this until k consecutive acceptable items are found. (ii) Afterwards inspect each item with probability α ∈ (0, 1). Continue until defective is found. (iii) Return to (i). Each item is defective with probability q ∈ (0, 1). Want to answer two questions: (1) What fraction of items are inspected? (2) If defective items are removed, what fraction of remaining items are defective? Answer this question using renewal-reward theorem. Say cycles begin each time 100% inspection begins, and reward is number of items inspected in cycle. Then from renewal- reward theorem E[no. inspected in cycle] frac. of items inspected = . E[no. produced in cycle]

Let Nk be the number inspected until k consecutive successes. After Nk need a Geom(q) number of inspections to find defectives so

E[no. inspected in cycle] = E[Nk] + 1/q.

After Nk there will be Geom(αq) items produced until first defective found so

E[no. produced in cycle] = E[Nk] + 1/(αq). 14 JASMINE FOO AND KEVIN LEDER

It remains to find E[Nk]. Can set up a recursive equation to find that 1 1 1 (1/p)k − 1 E[N ] = + + ... + = , k p p2 pk q where p = 1 − q. Therefore (1/p)k frac. of items inspected = P = . I (1/p)k − 1 + 1/α We now answer what fraction of non-removed items are defective. First note that in a large number N of items roughly Nq are defective. Furthermore we inspect roughly NPI items, of which NqPI are defective. Therefore we have roughly Nq(1 − PI ) undiscovered defectives. After our inspection we will have N(1−qPI ) non-removed items. After N items we will have roughly the following fraction of defectives in our non-removed items

Nq(1 − PI ) N(1 − qPI ) If we send N → ∞ the approximations become exact and the long run proportion of non removed items that are defective is q(1 − P ) I . (1 − qPI )

5. Queueing Systems 5.1. Structure of Queueing Systems. Customers requiring service enter queueing sys- tem where they are either served or join a queue to wait for service. After service the customers may leave the system or may require additional service at another (or same) server. • Customers - entity that enters system requiring service. – Time of customer arrivals can be deterministic or governed by a stochastic process. If arrivals are stochastic, common model for arrivals times is a Poisson process. Sometimes use a general renewal process. – Customers may arrive as individuals or in various sized batches. Batch sizes can be deterministic or random. • Queue - line where customers wait. – Important property is finiteness of queue, is there a finite limit to the size of the queue? Always some upper bound on possible size of queue, but can we assume it is infinite? – Service discipline - how is the line ordered? Common choices - first come first serve (FCFS), last come first serve, random, or priority. • Service - what steps are necessary for customers to complete service and exit sys- tem? – Simplest scenario - a single server and customer exits. IMA BOOTCAMP: STOCHASTIC MODELING 15

∗ Important property is how long does the service time take? Often as- sume service times are i.i.d. random variables from a non-negative dis- tribution. Most common choice is exponential, used for simplicity not realism. – More complex scenarios - multiple servers connected in a network, called a queueing network. Example 14. Emergency room at a hospital has one doctor on staff, patients arrive at random. Customers are patients, and server is doctor. Possible models for service discipline are either FCFS or some form of priority discipline. Patients arrive according to a Poisson process with rate λ and service times are exponential with mean 1/µ. This is an example of the M/M/1 queue.

6. Queueing in Real Life Queueing is used in real world everyday. For example they are used in commercial service systems such as bank tellers, checkout stands at grocery stores, cafeteria, airport ticketing, etc. Also used in transportation: tollbooths, traffic lights, ships unloading at a dock, airplanes waiting to take off or land at airport, parking lots, etc. Also used to model systems inside companies: materials-handling systems, maintenance systems, inspection systems, etc. Below is a list of some notable examples. • In 1970’s Xerox had backlog for technicians to repair copying machines. Researchers used to see how to redeploy technicians to reduce waiting times. Recommendations resulted in a substantial decrease in customer waiting times and an increase in technician utilization. See Interfaces, November 1975 Part 2. • in 1980’s United Airlines needed to figure out how to assign employees at its 11 reservations offices and 10 largest airports. Each checkin counter at an airport was modeled as a queueing system, and queueing theory was used to find minimum number of employees at each counter. Finding improved distribution of employees resulted in annual savings of $6 million. (Interfaces, January 1986. • L.L. Bean used to do most of it’s business via phone orders based on its mail catalog. Call center where orders are placed can be viewed as a queuing system. Customers are callers trying to order things, and servers are telephone agents taking orders. L.L. Bean used queueing theory to analyze their call center to answer questions like 1) How many trunk lines should be provided for incoming calls? 2) How many agents should be employed at various times? 3) How many hold positions should be made available? • Queueing theory was used to analyze routing of air tankers in fighting forest fires of Ontario. In this setting customers are forest fires and servers are air tankers, see OR/MS Today Volume 40, Number 2. 6.1. Queueing: Fundamental Results. Interested in quantities for t > 0 • X(t) = # in system at time t 16 JASMINE FOO AND KEVIN LEDER

• Q(t) = # in queue at time t. These stochastic processes can be quite difficult to study, so often settle for long run averages (which we assume to exist): 1 Z t L = avg. no. of cust. in system = lim X(s)ds, t→∞ t 0 and 1 Z t LQ = avg. no. of cust. in queue = lim Q(s)ds. t→∞ t 0 Also interested in waiting times. Define

Wi = time cust. i spends in syst.

Wi,Q = time cust. i spends in queue. Again, settle for long run averages: n 1 X W = lim Wi n→∞ n i=1 n 1 X WQ = lim Wi,Q. n→∞ n i=1 Little’s Law is a very useful relationship that relates L and W . Define the arrival time of customer i by Ti and denote the number of customers that have arrived by time t by N(t) then N(t) N(t) 1 Z t 1 Z t X 1 X X(s)ds = 1 (s)ds ≈ W t t {Ti≤s≤Ti+Wi} t i 0 0 i=1 i=1 N(t) N(t) 1 X (3) = W . t N(t) i i=1 If customers arrive according to a renewal process with inter arrival times with mean 1/λ then Renewal theorem tells us that N(t) lim = λ. t→∞ t Therefore taking t → ∞ in (3) we see Little’s Law: (4) L = λW.

Can also show that LQ = λWQ and average no. cust. in service = λ ∗ E[S], where S is random service time. IMA BOOTCAMP: STOCHASTIC MODELING 17

Another way to derive Little’s Law is the following. Assume customers arriving at the system are forced to pay $ to the system then (5) average earning rate of system = λ × average amount customer pays. Little’s Law follows by charging customers $1 per unit time in the system, number in queue version follows by charging customers for time spent in queue.

6.2. PASTA Property. Long run fraction of time with k customers: 1 Z t Pk = lim 1{X(s)=k}ds. t→∞ t 0 What about perspective of arriving customers: n 1 X ak = lim 1{X(T )=k}. n→∞ n i i=1

Does ak = Pk? Example 15. Suppose a single server serves customers. All customers have service time of 1 minute. Time between arrivals is uniformly distributed between 1 and 2 minutes. If first customer arrives at t = 0 then they depart at t = 1. Second customer arrives at some time t ∈ (1, 2) and finds system empty. Third customer finds system empty. Therefore a0 = 1, since all customers find the system empty, however P0 6= 1 since system is not always empty. If arrival process is Poisson then can rule out this situation which is the PASTA Property (Poisson Arrivals See Time Averages).

Theorem 16. If the arrival process is a Poisson process and the sequences {ak}k≥1 and {Pk}k≥1 exist then Pk = ak for all k ≥ 1. Rough idea: Since Poisson process has independent increments knowing that an arrival occurred at time t tells us nothing about what happened prior to time t.

6.3. Exponential Models. Simplest queueing models have exponentially distributed ser- vice times and Poisson arrivals. Specifically, customers arrive at a single server according to a Poisson process with rate λ. Upon arrival customer enters service if server is available, if not customer joins the queue. After completing service the customer leaves the system. Service times are i.i.d exponential random variables with mean 1/µ. This system is called an M/M/1 queue, M is for Markovian (or memoryless), and 1 represents that there is a single server. Will find Pn for n ≥ 0. General idea: the rate at which the system enters state n is the same as the rate at which the system leaves state n. System enters state 0 at rate µP1, and leaves at rate λP0, so therefore λP0 = µP1. 18 JASMINE FOO AND KEVIN LEDER

For n ≥ 1, system leaves state n at rate (λ + µ)Pn and enters at rate λPn−1 + µPn+1. Thus for n ≥ 1 we have

(λ + µ)Pn = λPn−1 + µPn+1. Can iterate this system to obtain that λn P = P . n µ 0

Solve for P0 with condition that probabilities must some to 1: ∞ ∞  n X X λ P0 1 = P = P = , n 0 µ 1 − λ/µ n=0 n=0 or µ − λ P = 0 µ λn µ − λ P = n ≥ 1. n µ µ Note that these probabilities only exist if λ < µ, i.e., arrival rate is strictly less than service rate. If arrival rate is greater than service rate number of customers in system will grow with time and can’t define Pn. We can use {Pn}n≥1 to obtain the long run average number of customers in the system: ∞ ∞ n X X λ µ − λ L = nP = n n µ µ n=1 n=1 λ = . µ − λ We can now use Little’s Law to find the average wait: W = L/λ = 1/(µ − λ).

If we want instead WQ we use the fact that W = WQ + E[S] = WQ + 1/µ, this gives λ W = . Q µ(µ − λ)

6.3.1. Birth Death Queueing Models. M/M/1 queue is a special case of a class of models called birth-death queueing models. In these models the system state increases from n at rate λn and decreases from n at rate µn. Examples: • M/M/1 queue with arrivals at rate λ and service at rate µ.

λn = λ n ≥ 0

µn = µ n ≥ 1 IMA BOOTCAMP: STOCHASTIC MODELING 19

• M/M/1 queue with arrivals at rate λ, service at rate µ and balking. Specifically, if a customer see n customers present then they leave (balk) with probability 1 − αn

λn = λαn n ≥ 0

µn = µ n ≥ 1 • M/M/k queue with arrivals at rate λ, and k servers each with service rate µ. In this model arriving customers enter service if any of the k servers are free, otherwise they wait in line. λn = λ n ≥ 0. Total service rate: ( nµ, n ≤ k µn = . kµ, n > k Can find limiting probabilities for birth death queueing models using principle that rate in = rate out. In particular get the equations

λ0P0 = µ1P1

(λn + µn)Pn = λn−1Pn−1 + µn+1Pn+1 n ≥ 1. Solve these equations to get that

∞ !−1 X λ0λ1 ··· λn−1 P = 1 + , 0 µ µ ··· µ n=1 1 2 n and λ0λ1 ··· λn−1 Pn = P0. µ1µ2 ··· µn 6.4. General Queueing Systems. Poisson process is often a reasonable model arrival process, but exponential distribution is rarely appropriate for modeling service times. Therefore people often study the M/G/1 distribution, here G stands for general. First establish a result about work in system V (t) = time required to empty the system if no more arrivals = work in system = virtual waiting time.

Denote the sequence of service, arrival times, and time in queue by {Si}i≥1 , {Ti}i≥1 and {Wi,Q}i≥1 respectively. Then ∞ ∞ X X (6) V (t) = Si1{Ti

Integrate RHS of (6) over [0,T ]: " ∞ ∞ # Z T X X Si1{Ti

Since arrivals are Poisson can use the PASTA property to see that n Z t   1 X 1 2 W¯ Q = lim V (Ti) = lim V (s)ds = λ E[Si]W¯ Q + E[S ] . n→∞ n t→∞ 2 i=1 0

Solve the above equation for W¯ Q to get that λE[S2] W¯ = , Q 2(1 − λE[S]) known as Pollaczek-Khinchine formula. Can use Little’s Law to get value of average number in system, average wait time, and average number in queue:

W = W¯ Q + E[S]

LQ = λW¯ Q L = λW.