The geometric distribution So far, we have seen only examples of random variables that have a finite number of possible values. However, our rules of probability allow us to also study random variables that have a countable [but possibly infinite] number of possible values. The word “countable” means that you can label the possible values as 1,2,. . . . A theorem of Cantor states that the number of elements of the real line is uncountable. And so is the number of elements of any nonempty closed/open interval. Therefore, the condition that a random variable X has a countable number of possible values is a restriction. We say that X has the geometric distribution with parameter ! := 1−" if #−1 P{X = #} = " ! (# =1$ 2$ % % %)% We have seen this distribution before. Here is one way it arises naturally: Suppose we toss a !-coin [i.e., P(heads)=!, P(tails)=" =1− !] until the first heads arrives. If X denotes the number of tosses, then X has the Geometric(!) distribution. Example 1. Suppose X has the Geometric(!) distribution. Then 2 P{X ≤ 3} = P{X =1} + P{X =2} + P{X =3} = ! + !" + !" % Here is an alternative expression for P{X ≤ 3}: ∞ ∞ ! #−1 ! & P{X ≤ 3} =1− P{X ≥ 4} =1− " ! =1− ! " % #=4 &=3

Simple facts about geometric series then tell us that " 3 # " 3 P{X ≤ 3} =1− ! =1− " $ 1 − "

55 56 12 since ! =1− ". More generally, if ' is a positive integer ≥ 1, then ' ' '−1 " ' # ! #−1 ! #−1 ! & 1 − " ' P{X ≤ '} = " ! = ! " = ! " = ! =1− " $ #=1 #=1 &=0 1 − " since ! =1− ". Example 2. What is E(X) when X has the Geometric(!) distribution? Well, clearly, ∞ ∞ ! #−1 ! #−1 E(X)= #" ! = ! #" % #=1 #=1 #−1 ( # How can we calculate this? Note that #" = (" " . Therefore a standard fact from calculus [term-by-term differentiation of an infinite series] tells us that ∞ ∞ " # " # ( ! # ( ! # ( 1 1 1 E(X)=! " = ! " = ! = ! 2 = % (" #=1 (" #=0 (" 1 − " (1 − ") ! Note that we benefited greatly from studying the problem for a general variable ! [as opposed to ! =1/2, say, or some such choice]. In this way, we were able to use the rules of calculus in order to obtain the answer.

Example 3. What is Var(X) when X has the Geometric(!) distribution?2 We know that E(X)=1/!. Therefore, it suffices to compute E(X ). But ∞ ∞ 2 ! #−1 ! 2 #−1 E(X )= #" ! = ! # " % #=1 #=1

In order to compute this we can write the sum as ∞ ∞ ∞ ! 2 #−1 ! #−1 ! #−1 # " = #(# − 1)" + #" % #=1 #=1 #=1

We saw earlier that ∞ ∞ ! #−1 ( ! # 1 #" = " = 2 % #=1 (" #=0 ! Similarly, ∞ ∞ 2 ∞ ! #−1 ! #−2 ( ! # 2" #(# − 1)" = " #(# − 1)" = " 2 " = 3 % #=1 #=1 (" #=0 ! $∞ 2 #−1 −3 −2 Therefore, #=1 # " =2"! + ! , and hence 2 2" 1 2 − ! 2 − ! 1 " E(X )= 2 + = 2 % Var(X)= 2 − 2 = 2 % ! ! ! ! ! ! √ Equivalently, SD(X)= "/!. The Poisson distribution 57

The negative binomial distribution The negative binomial distribution is a generalization of the geometric [and not the binomial, as the name might suggest]. Let us fix an integer ) ) ≥ 1; then we toss a !-coin until the )th heads occur. Let X denote the total number of tosses. Example 4 (The negative binomial distribution). What is the distribution ) ) of X ? First of all, the possible values of X are )$ ) +1$) +2$ % % % . Now for all integers * ≥ ), ) P{X = *} = P {The first * − 1 tosses have ) − 1 heads, and the *th toss is heads} % Therefore, " # )−1 (*−1)−()−1) " # ) *−) ) * − 1 * − 1 P{X = *} = ! " × ! = ! " (* ≥ ))% ) − 1 ) − 1 This looks a little bit like a binomial probability, except the variable of the mass function is *. This distribution is called the negative binomial distribution with parameters () $ !). ) ) 1 Example 5. Compute E(X ) and Var(X ). Let G denote the number of 2 tosses required to obtain the first heads. Then define G to be the num- ) ber of additional tosses required to obtain the next heads . . . G to be the number of additional tosses required to obtain the next [and )th] 1 2 * head. Then, G $G $%%%$G are independent random variables each with the Geomteric(!) distribution. Moreover, ) 1 ) X = G + ··· + G % Therefore, ) 1 ) ) E(X )=E(G )+··· + E(G )= $ and ! √ ) 1 ) )" ) )" Var(X )=Var(G )+··· + Var(G )= 2 % SD(X )= % ! !

The Poisson distribution We say that a random variable X has the Poisson distribution with pa- rameter λ> 0 when −λ ' , λ P{X = '} = for ' =0$ 1$ 2$ % % % % '! Two questions arise: First, is the preceding a well-defined probability dis- tribution? And second, why this particular form? 58 12

∞ −λ ' $'=0 In order to answer the first question we. have to prove that , λ /'!= 1. But the Taylor’s expansion of -(.)=, tells us that ∞ ' λ ! λ , = % '=0 '! λ $∞ Divide by , in order to find that '=0 P{X = '} =1, as one would like. The next proposition [the law of rare events] shows how the Poisson distribution can be viewed as an approximation to the Binomial distribution when ! ∝ 1/*. Proposition 1 (Poisson). Suppose λ is a positive constant that does not depend on *. If X has the Binomial distribution with parameters * and λ/*, then −λ ' , λ P{X = '}≈ for all ' =0$ % % % $ *$ '! provided that * is large.

Proof. The exact expression we want follows: For all ' =0$ % % % $ *, " #" #' " #*−' * λ λ P{X = '} = 1 − ' * * " #' " #*−' *(* − 1) ··· (* − ' + 1) λ λ = 1 − % '! * * ' Because ' is held fixed, we have *(* − 1) ··· (* − ' + 1) ≈ * as * →∞. Therefore, " #*−' " #* 1 ' λ 1 ' λ P{X = '}≈ λ 1 − ≈ λ 1 − $ '! * '! * λ ' since (1 − * ) → 1 as * →∞[again because ' is fixed]. It suffices to prove that " #* λ −λ 1 − ≈ , as * →∞% * Equivalently, " # λ * ln 1 − ≈−λ as * →∞% * But this is clear because ! ln(1 − .) ≈ 1 − . if . ≈ 0, thanks to the Taylor expansion of ln(1 − .).

Now because Poisson(λ) is approximately Binomial (*$λ/*), one might imagine that the expectation of a Poisson(λ) is approximately the expec- tation of Binomial (*$λ/*) which is *λ/* = λ. And that the variance of The Poisson distribution 59

a Poisson(λλ) is approximately the variance of Binomial (*$λ/*) which is *(λ/*)(1 − * ) ≈ λ% Although the preceding “argument” is logically flawed, it does produced the correct answers. Let X have the Poisson(λ) distribution. In order to compute E(X) we proceed as follows: ∞ −λ ' ∞ −λ ' ∞ −λ ' ! , λ ! , λ ! , λ EX = ' = ' = '=0 '! '=1 '! '=1 (' − 1)! ∞ −λ &+1 ∞ & ! , λ −λ ! λ = = λ, = λ% &=0 &! &=0 &!

Similarly, ∞ −λ ' ∞ −λ ' ∞ −λ ' ∞ −λ ' 2 ! 2 , λ ! 2 , λ ! , λ ! , λ E(X )= ' = ' = '(' − 1) + ' '=0 '! '=1 '! '=1 '! '=1 '! ∞ −λ ' ! , λ = '(' − 1) + λ [from the computation for EX]. '=1 '!

Now ∞ −λ ' ∞ '−2 ! , λ −λ 2 ! λ '(' − 1) = , λ '(' − 1) '=1 '! '=1 '! 2 ∞ ' 2 −λ 2 ( ! λ −λ 2 ( λ 2 = , λ 2 = , λ 2 , = λ % (λ '=1 '! (λ Consequently, 2 2 √ E(X )=λ + λ % Var(X)=λ ⇔ SD(X)= λ% λ Theorem 1 (A central limit theorem). Let X have a Poisson distribution λ with parameter λ. Then the standardization of X has approximately a standard normal distribution. That is, for all −∞ ≤ / ≤ 0 ≤∞, % λ & X − λ P / ≤ √ ≤ 0 ≈ Φ(0) − Φ(/) when λ is large% λ