Introduction to Probability Ariel Yadin Lecture 7 1

Ariel Yadin

Lecture 7

1. Independence Revisited *** Dec. 6 *** 1.1. Some reminders. Let (Ω, F, P) be a probability space. Given a collection of subsets K ⊂ F, recall that the σ-algebra generated by K, is \ σ(K) = {G : G is a σ-algebra , K ⊂ G} , and this σ-algebra is the smallest σ-algebra containing K. σ(K) can be thought of as all possible information that can be generated by the sets in K.

Recall that a collection of events (An)n are mutually independent if for any ﬁnite number of these events A1,...,An we have that

P(A1 ∩ · · · ∩ An) = P(A1) ··· P(An).

We can also deﬁne the independence of families of events:

Deﬁnition 7.1. Let (Ω, F, P) be a probability space. Let (Kn)n be a collection of families of events in F. Then, (Kn)n are mutually independent if for any ﬁnite number of families from this collection, K1,..., Kn and any events A1 ∈ K1,A2 ∈ K2,...,An ∈ Kn, we have that

P(A1 ∩ · · · ∩ An) = P(A1) ··· P(An).

For example, we would like to think that (An) are mutually independent, if all the information from each event in the sequence is independent from the information from the other events in the sequence. This is the content of the next proposition.

Proposition 7.2. Let (An) be a sequence of events on a probability space (Ω, F, P). Then, (An) are mutually independent if and only if the σ-algebras (σ(An))n are mutually independent.

It is not diﬃcult to prove this by induction:

Proof. By induction on n we show that (σ(A1), . . . , σ(An), {An+1} ,...) are mutually independent. The base n = 0 is just the assumption. 1 2

So assume that (σ(A1), . . . , σ(An−1), {An} , {An+1} ,...) are mutually independent. Let n1 < n2 < . . . < nk < nk+1 < . . . < nm be a ﬁnite number of indices such that nk < n < nk+1. Let

Bj ∈ σ(Anj ) for j = 1, . . . , k, and Bj = Anj for j = k + 1, . . . , m. Let B ∈ σ(An). If B = ∅ then P(B1 ∩ · · · ∩ Bn ∩ B) = 0 = P(B1) ··· P(Bn) · P(B). If B = Ω then P(B1 ∩ · · · ∩ Bn ∩ B) =

P(B1 ∩ · · · ∩ Bn) = P(B1) ··· P(Bn), by the induction hypotheses. If B = An then this also holds c by the induction hypotheses. So we only have to deal with the case B = An. In this case, for

X = B1 ∩ · · · ∩ Bn,

P(X ∩ B) = P(X \ (X ∩ An)) = P(X) − P(X ∩ An) = P(X)(1 − P(An)) = P(X) P(B),

where we have used the induction hypotheses to say that X and An are independent. Since this holds for any choice a a ﬁnite number of events, we have the induction step. ut

However, we will take a more windy road to get stronger results... (Corollary 7.8)

2. π-systems and Independence

Deﬁnition 7.3. Let K be a family of subsets of Ω.

• We say that K is a π-system if ∅ ∈ K and K is closed under intersections; that is, for all A, B ∈ K, A ∩ B ∈ K. • We say that K is a Dynkin system (or λ-system) if K is closed under set complements

and countable disjoint unions; that is, for any A ∈ K and any sequence (An) in K of c S pairwise disjoint subsets, we have that A ∈ K and n An ∈ K.

The main goal now is to show that probability measures are uniquely deﬁned once they are deﬁned on a π-system. This is the content of Theorem 7.7.

Proposition 7.4. If F is Dynkin system on Ω and F is also a π-system on Ω, then F is a σ−algebra.

Proof. Since F is a Dynkin system, it is closed under complements, and Ω = ∅c ∈ F.

Let (An) be a sequence of subsets in F. Set B1 = A1, C1 = ∅, and for n > 1,

n−1 [ Cn = Aj and Bn = An \ Cn. j=1 3

Tn−1 c c Since F is a π-system and closed under complements, Cn = ( j=1 Aj) ∈ F, and so Bn = c An ∩ Cn ∈ F. Since (Bn) is a sequence of pairwise disjoint sets, and since F is a Dynkin system, [ [ An = Bn ∈ F. n n ut

Proposition 7.5. If (Dα)α is a collection of Dynkin systems (not necessarily countable). Then T *** leave as ex- D = α Dα is a Dynkin system. ercise ***

Proof. Since ∅ ∈ Dα for all α, we have that ∅ ∈ D. c c If A ∈ D, then A ∈ Dα for all α. So A ∈ Dα for all α, and thus A ∈ D.

If (An)n is a countable sequence of pairwise disjoint sets in D, then An ∈ Dα for all α and all S S n. Thus, for any α, n An ∈ Dα. So n An ∈ D. ut

Lemma 7.6 (Dynkin’s Lemma). If a Dynkin system D contains a π-system K, then σ(K) ⊂ D.

Proof. Let \ F = {D0 : D0 is a Dynkin system containing K} .

So F is a Dynkin system and K ⊂ F ⊂ D. We will show that F is a σ−algebra, so σ(K) ⊂ F ⊂ D. Suppose we know that F is closed under intersections (which is Claim 3 below). Since ∅ ∈ K ⊂ F, we will then have that F is a π-system. Being both a Dynkin system and a π−system, F is a σ−algebra. Thus, to show that F is a σ−algebra, it suﬃces to show that F is closed under intersections. Note that F is closed under complements (because all Dynkin systems are). Claim 1. If A ⊂ B are subsets in F, then B \ A ∈ F.

Proof. If A, B ∈ F, then since F is a Dynkin system, also Bc ∈ F. Since A ⊂ B, we have that A, Bc are disjoint, so A ∪ Bc ∈ F and so B \ A = (Ac ∩ B) = (A ∪ Bc)c ∈ F. ut

Claim 2. For any K ∈ K, if A ∈ F then A ∩ K ∈ F.

Proof. Let E = {A : A ∩ K ∈ F}.

Let A ∈ E and (An) be a sequence of pairwise disjoint subsets in E. Since K ∈ F and A ∩ K ∈ F, by Claim 1 we have that Ac ∩ K = K \ (A ∩ K) ∈ F. So Ac ∈ E. 4

Since (An ∩ K)n is a sequence of pairwise disjoint subsets in F, we get that

[ \ [ An K = (An ∩ K) ∈ F. n n So we conclude that E is a Dynkin system. Since K is closed under intersections, E contains K. Thus, by deﬁnition F ⊂ E. So for any A ∈ F we have that A ∈ E, and A ∩ K ∈ F. ut

Claim 3. For any B ∈ F, if A ∈ F then A ∩ B ∈ F.

Proof. Let E = {A : A ∩ B ∈ F}.

Let A ∈ E and (An) be a sequence of pairwise disjoint subsets in E. Since B ∈ F and A ∩ B ∈ F, by Claim 1 we have that Ac ∩ B = B \ (A ∩ B) ∈ F. So Ac ∈ E.

Since (An ∩ B)n is a sequence of pairwise disjoint subsets in F, we get that

[ \ [ An B = (An ∩ B) ∈ F. n n So we conclude that E is a Dynkin system. By Claim 2, K is contained in E. So by deﬁnition, F ⊂ E. ut

Since F is closed under intersections, this completes the proof. ut

The next theorem tells us that a probability measure on (Ω, F) is determined by it’s values on a π-system generating F.

Theorem 7.7 (Uniqueness of Extension). Let K be a π−system on Ω, and let F = σ(K) be the σ−algebra generated by K. Let P,Q be two probability measures on (Ω, F), such that for all A ∈ K, P (A) = Q(A). Then, P (B) = Q(B) for any B ∈ F.

Proof. Let D = {A ∈ F : P (A) = Q(A)}. So K ⊂ D. We will show that D is a Dynkin system, and since it contains K, the by Dynkin’s Lemma it must contain F = σ(K). Of course P (Ω) = 1 = Q(Ω), so Ω ∈ D. If A ∈ D, then P (Ac) = 1 − P (A) = 1 − Q(A) = Q(Ac), so Ac ∈ D.

Let (An) be a sequence of pairwise disjoint sets in D. Then,

[ X X [ P ( An) = P (An) = Q(An) = Q( An). n n n n S So n An ∈ D. ut 5

Corollary 7.8. Let (Ω, F, P) be a probability space. Let (Πn)n be a sequence of π-systems, and let Fn = σ(Πn). Then, (Πn)n are mutually independent if and only if (Fn)n are mutually independent.

Proof. We will prove by induction on n that for any n ≥ 0, the collection (F1, F2,..., Fn, Πn+1, Πn+2,..., ) are mutually independent.

For n = 0 this is the assumption. For n > 1, let n1 < n2 < . . . < nk < nk+1 < . . . < nm be a ﬁnite number of indices such that nk < n < nk+1. Let Aj ∈ Fnj , j = 1, . . . , k and

Aj ∈ Πnj , j = k + 1, . . . , m.

For any A ∈ Fn, if P(A1 ∩ · · · ∩ Am) = 0 then A is independent of A1 ∩ · · · ∩ Am. So assume that P(A1 ∩ · · · ∩ Am) > 0.

For any A ∈ Fn deﬁne the probability measure

P (A) := P(A|A1 ∩ · · · ∩ Am).

By induction, the collection (F1,..., Fn−1, Πn, Πn+1,...) are mutually independent, so P (A) =

P(A) for any A ∈ Πn. Since Πn is a π-system generating Fn, we have by Theorem 7.7 that

P (A) = P(A) for any A ∈ Fn. Since this holds for any choice of a ﬁnite number of events

A1,...,Am, we get that the collection (F1,..., Fn−1, Fn, Πn+1,...) are mutually independent. ut *** Dec. 8 ***

Corollary 7.9. Let (Ω, F, P) be a probability space, and let (An)n be a sequence of mutually independent events. Then, the σ-algebras (σ(An))n are mutually independent.

3. Independent Random Variables

Let X : (Ω, F) → (R, B) be a random variable. Recall that in the deﬁnition of a random variable we require that X is a measurable function; i.e. for any Borel set B ∈ B we have X−1(B) ∈ F. We want to deﬁne a σ-algebra that is all the possible information that can be infered from X.

Deﬁnition 7.10. Let (Ω, F, P) be a probability space and let (Xα)α∈I :Ω → R be a collection of random variables. The σ-algebra generated by (Xα)α∈I is deﬁned as

−1 σ(Xα : α ∈ I) := σ( Xα (B): α ∈ I,B ∈ B ). 6

−1 Note that for a random variable X :Ω → R, the collection X (B): B ∈ B is a σ-algebra, so σ(X) = X−1(B): B ∈ B . We now deﬁne independent random variables.

Deﬁnition 7.11. Let (Xn)n, (Yn)n be a collection of random variables on some probability

space (Ω, F, P). We say that (Xn)n are mutually independent if the σ-algebras (σ(Xn))n are mutually independent.

We say that (Xn)n are independent of (Yn)n if the σ-algebras σ((Xn)n) and σ((Yn)n) are independent.

*** keep this for An important property (that is a consequence of the π-system argument above): later *** Proposition 7.12. Let (Xn)n be a collection of random variables on (Ω, F, P). (Xn) are mutu-

ally independent, if for any ﬁnite number of random variables from the collection, X1,...,Xn,

and any real numbers a1, a2, . . . , an, we have that

P[X1 ≤ a1,X2 ≤ a2,...,Xn ≤ an] = P[X1 ≤ a1] · P[X2 ≤ a2] ··· P[Xn ≤ an].

Proof. Let X1,...,Xn be a ﬁnite number of random variables. Deﬁne two probability measure on the Borel sets of Rn: For any B ∈ Bn let

P1(B) = P((X1,...,Xn) ∈ B) and P2(B) = P(X1 ∈ π1B) ··· P(Xn ∈ πnB),

where πj is the projection onto the j-th coordinate. Since these two measure are identical on the π-system

{(−∞, a1] × · · · × (−∞, an]: a1, a2, . . . , an ∈ R} ,

n and since this π-system generates all Borel sets on R , we get that P1 = P2 for all Borel sets on Rn. −1 Thus, if Aj ∈ σ(Xj), j = 1, . . . , n, then for all j, Aj = Xj (Bj) for some Borel set on R, so

P(A1∩· · ·∩An) = P((X1,...,Xn) ∈ B1×· · ·×Bn) = P(X1 ∈ B1) ··· P(X1 ∈ Bn) = P(A1) ··· P(An).

Example 7.13. We toss two dice. Y is the number on the ﬁrst die and X is the sum of both dice. Here the probability space is the uniform measure on all pairs of dice results, so Ω = {1, 2,..., 6}2. 7

What is σ(X)?

σ(X) = {{(x, y) ∈ Ω: x + y ∈ A,A ⊂ {2,..., 12}}} .

(We have to know that all subsets of {2,..., 12} are Borel sets. This follows from the fact that S {r} = n(r − 1/n, r].) On the other hand,

σ(Y ) = {{(x, y) ∈ Ω: x ∈ A,A ⊂ {1,..., 6}}} .

Now note 1 [X = 7,Y = 3] = = [X = 7] · [Y = 3]. P 36 P P This is mainly due to P[X = 7] = 1/6. Similarly, for any y ∈ {1,..., 6}, we have that the events {X = 7} and {Y = y} are independent. Are X and Y independent random variables? NO! For example, 1 5 1 [X = 6,Y = 3] = 6= · = [X = 6] · [Y = 3]. P 36 36 6 P P For independence we need all information from X to be independent of the information from Y ! 4 5 4

4. Second Borel Cantelli Lemma

We conclude this lecture with

Lemma 7.14 (Second Borel-Cantelli Lemma). Let (An) be a sequence of mutually independent P events. If n P(An) = ∞ then P(lim supn An) = P(An i.o.) = 1.

c Proof. It suﬃces to show that P(lim inf An) = 0. For ﬁxed m > n note that by independence

m m \ c Y P( Ak) = (1 − P(Ak)). k=n k=n

Tm c T c Since limm k=n Ak = k≥n Ak, we have that

m \ c Y Y X ( Ak) = lim (1 − (Ak)) = (1 − (Ak)) ≤ exp − (Ak) , P m→∞ P P P k≥n k=n k≥n k≥n where we have used 1 − x ≤ e−x. 8

P P T c Since n P(An) = ∞, we have that k≥n P(Ak) = ∞ for all ﬁxed n. Thus, P( k≥n Ak) = 0 for all ﬁxed n, and

c [ \ c \ c \ c (lim inf An) = ( Ak) = (lim Ak) = lim ( Ak) = 0. P P P n n→∞ P n k≥n k≥n k≥n ut

Example 7.15. A biased coin is tossed inﬁnitely many times, all tosses mutually independent. What is the probability that the sequence 01010 appears inﬁnitely many times?

Let p be the probability the coin’s outcome is 1. Let an be the outcome of the n-th toss. let An be the event that an = 0, an+1 = 1, an+2 = 0, an+3 = 1, an+4 = 0. Since the tosses are mutually independent, we have that the sequence of events (A5n+1)n≥0 are mutually independent. Since −2 −3 P∞ P(A5n+1) = p (1 − p) > 0, we have that n=0 P(A5n+1) = ∞. So the Borel-Cantelli Lemma tells us that P(A5n+1 i.o.) = 1. Since {A5n+1 i.o.} implies {An i.o.} we are done. 4 5 4