Probability Theory Basics from Measure Theory

Probability Theory 1 Math A 9343 Chapter 1 Introductory Measure Theory (Lecture 01)

Mohamed I. Riffi

Department of Mathematics Islamic University of Gaza

September 16, 2015

Mohamed I. Riffi Probability Theory 1 Math A 9343 2 Basics from Measure Theory Sets Collections of Sets Generators

Probability Theory Basics from Measure Theory Outline

1 Probability Theory: An Introduction

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Collections of Sets Generators

Probability Theory Basics from Measure Theory Outline

1 Probability Theory: An Introduction

2 Basics from Measure Theory

Mohamed I. Riffi Probability Theory 1 Math A 9343 Collections of Sets Generators

Probability Theory Basics from Measure Theory Outline

1 Probability Theory: An Introduction

2 Basics from Measure Theory Sets

Mohamed I. Riffi Probability Theory 1 Math A 9343 Generators

Probability Theory Basics from Measure Theory Outline

1 Probability Theory: An Introduction

2 Basics from Measure Theory Sets Collections of Sets

Mohamed I. Riffi Probability Theory 1 Math A 9343 Probability Theory Basics from Measure Theory Outline

1 Probability Theory: An Introduction

2 Basics from Measure Theory Sets Collections of Sets Generators

Mohamed I. Riffi Probability Theory 1 Math A 9343 Probability Theory Basics from Measure Theory Introduction

Probability Space Let us introduce the probability space or probability triple (Ω, F, P), where Ω is the sample space; F is the collection of events; P is a probability measure.

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators Set Operations

Set Operations

Let A, A1, A2,... and B, B1, B2,... be sets. Union: A ∪ B = {x : x ∈ A or x ∈ B}; Intersection: A ∩ B = {x : x ∈ A and x ∈ B}; Complement: Ac = {x : x 6∈ A}; c Difference: A r B = A ∩ B ; Symmetric difference: A 4 B = (A B) ∪ (B r A). n n Also we use ∪k=1Ak for unions and ∩k=1Ak intersections.

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators Some Additional Terminology

Some Additional Terminology the empty set: φ; subset: A is a subset of B, A ⊂ B, if x ∈ A ⇒ x ∈ B; disjoint: A and B are disjoint if A ∩ B = φ; power set: P(Ω) = {A : A ⊂ Ω};

{An, n ≥ 1} is non-decreasing, An %, if A1 ⊂ A2 ⊂ · · · ;

{An, n ≥ 1} is non-increasing, An &, if A1 ⊃ A2 ⊃ · · · .

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators The De Morgan Formulas

The De Morgan Formulas n !c n [ \ c Ak = Ak k=1 k=1 n !c n \ [ c Ak = Ak k=1 k=1

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators Limits of Sets

Limits of Sets Definition (2.1)

Let {An, n ≥ 1} be a sequence of subsets of Ω. We define ∞ ∞ [ \ A∗ = lim inf An = Am n→∞ n=1 m=n ∞ ∞ ∗ \ [ A = lim sup An = Am n→∞ n=1 m=n

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators Proposition

Proposition (2.1)

Let {An, n ≥ 1} be a sequence of subsets of Ω.

(i) If A1 ⊂ A2 ⊂ A3 ··· , then

∞ [ lim An = An. n→∞ n=1

(ii) If A1 ⊃ A2 ⊃ A3 ··· , then

∞ \ lim An = An. n→∞ n=1

Mohamed I. Riffi Probability Theory 1 Math A 9343 Definition ∗ ∗ If the sets A∗ and A agree, then A = A∗ = A = limn→∞ An and we say A = lim An = lim sup An = lim inf An;

in other words, we say that An converges to A and write An → A.

Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Proof of Proposition 2.1. Exercise 2.2

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Proof of Proposition 2.1. Exercise 2.2

Definition ∗ ∗ If the sets A∗ and A agree, then A = A∗ = A = limn→∞ An and we say A = lim An = lim sup An = lim inf An; in other words, we say that An converges to A and write An → A.

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Example Let’s actually go from what we know about the real numbers R. What’s an example where we have the lim sup and lim inf are different? How about the sequence:

1 (−1)n+1 a = + n 2 2

Then the lim sup an = 1 and the lim inf an = 0.

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

How can we use this to make a simple example for the case of sets? Example

Let A1 = A3 = A5 = ··· = A and A2 = A4 = A6 = ··· = B where B is a proper subset of A. Then lim sup An = A and lim inf An = B.

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Set Relations Let A be a non-empty collection of subsets of Ω, and consider the following set relations: (a) A ∈ A ⇒ Ac ∈ A; (b) A, B ∈ A ⇒ A ∪ B ∈ A; (c) A, B ∈ A ⇒ A ∩ B ∈ A; (d) A, B ∈ A, B ⊂ A =⇒ A \ B ∈ A; S∞ (e) A1, A2,... ∈ A =⇒ n=1 An ∈ A; S∞ (f) An ∈ A, n ≥ 1, Ai ∩ Aj = φ, i 6= j =⇒ n=1 An ∈ A; T∞ (g) An ∈ A, n ≥ 1, =⇒ n=1 An ∈ A; S∞ (h) An ∈ A, n ≥ 1, An % =⇒ n=1 An ∈ A; T∞ (i) An ∈ A, n ≥ 1, An & =⇒ n=1 An ∈ A.

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Definition (2.2) A collection F of events is called an algebra or a field if Ω ∈ A and (a) It is closed under complementation:

A ∈ F ⇒ Ac ∈ F

(b) It is closed under taking finite union:

A, B ∈ F ⇒ A ∪ B ∈ F.

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Corollary It follows from these two conditions that algebras are also closed under finite intersections:

A, B ∈ F ⇒ A ∩ B ∈ F.

Proof. De Morgan’s Law states (A ∩ B)c = Ac ∪ Bc . Thus, A ∩ B = (Ac ∪ Bc )c

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Example Let F = {∅, Ω}. This is the trivial algebra. In some sense, it is the “poorest” algebra. At the opposite extreme let F = P(Ω) the power set of Ω. This is the “richest” algebra.

The true life is between these extremes.

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Definition (2.2) A collection F of events is called a σ-algebra or a σ-field if Ω ∈ A and (a) It is closed under complementation:

A ∈ F ⇒ Ac ∈ F

(e) It is closed under taking countable unions:

∞ [ A1, A2,... ∈ F =⇒ An ∈ F. n=1

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Discrete Probabilities You should not worry too much about σ-algebras when working in discrete probability. Then, you can just take the power set for your algebra.

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Definition Suppose F is a σ-algebra on Ω. Let A1, A2,... ⊆ R. Then ∞ ∞ \ [ lim sup An := Ak n=1 k=n

= {ω ∈ Ω: ∀ n ∃ k ≥ n such that ω ∈ Ak }

= {ω ∈ Ω: ω belongs to infinitely many An’s} = “Infinitely many events occur”.

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Definition Suppose F is a σ-algebra on Ω. Let A1, A2,... ⊆ R. Then ∞ ∞ [ \ lim inf An := Ak n=1 k=n

= {ω ∈ Ω: ∃ n such that ∀ k ≥ n, ω ∈ Ak }

= {ω ∈ Ω: ω belongs to all except finitely many An’s}

= “All except finitely many events An occur”.

Mohamed I. Riffi Probability Theory 1 Math A 9343 Solution From now on, whenever I say that A is an event, I implicitly mean that A ∈ F.

Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Question Is A in the σ-algebra?

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Question Is A in the σ-algebra?

Solution From now on, whenever I say that A is an event, I implicitly mean that A ∈ F.

Mohamed I. Riffi Probability Theory 1 Math A 9343 Solution This turns out to always be true, as long as Ω 6= ∅. Indeed, take {a} ∩ {a}c = ∅ ∈ F.

Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Question Are we assuming Ω ∈ F?

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Question Are we assuming Ω ∈ F?

Solution This turns out to always be true, as long as Ω 6= ∅. Indeed, take {a} ∩ {a}c = ∅ ∈ F.

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Definition (2.2) Let A be a collection of subsets of Ω. A is a monotone class (m.c.) if the following properties hold: (h) ∞ [ An ∈ A, n ≥ 1, An % =⇒ An ∈ A; n=1 (i) ∞ \ An ∈ A, n ≥ 1, An & =⇒ An ∈ A n=1

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Definition (2.2) Let A be a collection of subsets of Ω. A is a π-system if the following property holds: (c) A is closed under taking finite intersection:

A, B ∈ A ⇒ A ∩ B ∈ A.

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Definition (2.2) Let A be a collection of subsets Ω. A is a Dynkin system if Ω ∈ A, and the following properties hold: (d) A, B ∈ A, B ⊂ A =⇒ A \ B ∈ A; (h) ∞ [ An ∈ A, n ≥ 1, An % =⇒ An ∈ A. n=1

Mohamed I. Riffi Probability Theory 1 Math A 9343 Remark (2.2) The definition of a Dynkin system varies. One alternative, in addition to the assumption that Ω ∈ A, is that the following properties hold: (a) A is closed under complementation:

A ∈ A ⇒ Ac ∈ A

(f) ∞ [ An ∈ A, n ≥ 1, Ai ∩ Aj = φ, i 6= j =⇒ An ∈ A. n=1

Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Remark (2.1) Dynkin systems are also called λ-systems.

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Remark (2.1) Dynkin systems are also called λ-systems.

Remark (2.2) The definition of a Dynkin system varies. One alternative, in addition to the assumption that Ω ∈ A, is that the following properties hold: (a) A is closed under complementation:

A ∈ A ⇒ Ac ∈ A

(f) ∞ [ An ∈ A, n ≥ 1, Ai ∩ Aj = φ, i 6= j =⇒ An ∈ A. n=1

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Theorem (2.1) The following connections hold: 1 Every algebra is a π-system. 2 Every σ-algebra is an algebra. 3 An algebra is a σ-algebra if and only if it is a monotone class. 4 Every σ-algebra is a Dynkin system. 5 A Dynkin system is a σ-algebra if and only if it is π-system. 6 Every Dynkin system is a monotone class. 7 Every σ-algebra is a monotone class. 8 The power set of any subset of Ω is a σ-algebra on that subset. 9 The intersection of any number of σ-algebras, countable or uncountable, is, again, a σ-algebra.

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Theorem (2.1 cont.)

10 The countable union of a non-decreasing sequence of σ-algebras is an algebra, but not necessarily a σ-algebra. 11 If A is a σ-algebra, and B ⊂ Ω, then B ∩ A = {B ∩ A : A ∈ A} is a σ-algebra on B. 0 0 0 0 12 If Ω and Ω are sets, A a σ-algebra on Ω and T :Ω −→ Ω a mapping, then T −1(A0) = {T −1(A0): A0 ∈ A0} is a σ-algebra on Ω.

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Exercise 2.5 (a) Prove the above statements. (b) Find two σ-algebras, the union of which is not an algebra (only very few elements in each suffice). (c) Prove that if, for the infinite set Ω, A consists of all A ⊂ Ω, such that either A or Ac is finite, then A is an algebra, but not a σ-algebra.

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Generating σ-algebras Start with an arbitrary collection A of subsets of Ω. We want the smallest σ-algebra containing A. That’s what we actually do: We look at all σ-algebras that contain A, and we pick the smallest. Choose the smallest one. Call it σ(A), the σ-algebra generated by A.

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Example If Ω = R, then A might consist of all intervals.

Example Again, let Ω = R and A = {(a, b): −∞ < a < b < ∞}, the collection of open intervals. Then σ(A) is called the Borel σ-algebra, denoted by R. Elements A ∈ R are called Borel sets. The pair (R, R) is called Borel space.

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Remark Note, the arbitrary intersection of σ-algebras is again a σ-algebra. Thus, we can actually define \ σ(A) = G. G is a σ-algebra ⊃ A In other words, let F ∗ = {σ-algebras ⊃ A}. Then the smallest σ-algebra containing A is \ G, G∈F ∗ and is unique since we have intersected all σ-algebras containing A.

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Definition (2.3) Let A be a collection of subsets of Ω. The smallest σ-algebra containing A, σ(A), is called the σ-algebra generated by A. The smallest Dynkin system containing A, D(A), is called the Dynkin system generated by A. The smallest monotone class containing A, M(A), is called the monotone class generated by A. In each case A is called the generator of the actual collection.

Mohamed I. Riffi Probability Theory 1 Math A 9343 Remark (2.4)

Let {An, n ≥ 1} be σ-algebras. Even though the union need not S∞ be a σ-algebra, σ( n=1 An), that is, the σ-algebra generated by {An, n ≥ 1}, always exists.

Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Remark (2.3) The σ-algebra generated by A, σ(A), is also called the minimal σ-algebra containing A. Similarly for the other collections.

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Remark (2.3) The σ-algebra generated by A, σ(A), is also called the minimal σ-algebra containing A. Similarly for the other collections.

Remark (2.4)

Let {An, n ≥ 1} be σ-algebras. Even though the union need not S∞ be a σ-algebra, σ( n=1 An), that is, the σ-algebra generated by {An, n ≥ 1}, always exists.

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Exercise 2.6 Prove that (i) If A = A, a single set, then σ(A) = σ(A) = {φ, A, Ac , Ω}. (ii) If A is a σ-algebra, then σ(A) = A.

Mohamed I. Riffi Probability Theory 1 Math A 9343 Example

Let Ω = R and A = [0, 1]. We obtain the Borel σ-algebra R|[0,1] on [0, 1] by restriction on A. We can (alternatively) do this by thinking of [0, 1] as a metric space and doing a direct construction.

Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Remark Everything can be carried over into higher dimensions. In fact, it is n true for all metric spaces. In particular, it is true for R . Here, R is the σ-algebra generated by open sets. Also, it easily generalizes to intervals Ω = [0, 1] (or any other interval). Why? By restriction.

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Remark Everything can be carried over into higher dimensions. In fact, it is n true for all metric spaces. In particular, it is true for R . Here, R is the σ-algebra generated by open sets. Also, it easily generalizes to intervals Ω = [0, 1] (or any other interval). Why? By restriction.

Example

Let Ω = R and A = [0, 1]. We obtain the Borel σ-algebra R|[0,1] on [0, 1] by restriction on A. We can (alternatively) do this by thinking of [0, 1] as a metric space and doing a direct construction.

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Theorem (2.2) Let A be an algebra. Then M(A) = σ(A).

Proof. Since a σ-algebra is a monotone class, σ(A) is a monotone class containing A. Hence M(A) ⊂ σ(A), by the minimality of M(A). To prove that σ(A) ⊂ M(A), it suffices to show that M(A) is a σ-algebra containing A. So, let

E1 = {B ∈ M(A): B ∪ C ∈ M(A) for all C ∈ A} c E2 = {B ∈ M(A): B ∈ M(A)}.

We need to show that E1 = E2 = M(A).

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Proof Cont. Let

E3 = {B ∈ M(A): B ∪ C ∈ M(A) for all C ∈ M(A)}

Let C ∈ A. For every B ∈ M(A), B ∪ C ∈ M(A), since E1 = M(A). This proves that A ⊂ E3. Now, by the minimality again, M(A) = E3, since E3 is a monotone class. Hence properties (a) and (b) hold for M(A).

Mohamed I. Riffi Probability Theory 1 Math A 9343 Theorem (2.3 The monotone class theorem) If A is a π-system on Ω, then

D(A) = σ(A).

Corollary (2.2) If A is a σ-algebra, then

M(A) = D(A) = σ(A) = A.

Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Corollary (2.1) If A is an algebra and G a monotone class containing A, then

G ⊃ σ(A).

Mohamed I. Riffi Probability Theory 1 Math A 9343 Corollary (2.2) If A is a σ-algebra, then

M(A) = D(A) = σ(A) = A.

Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Corollary (2.1) If A is an algebra and G a monotone class containing A, then

G ⊃ σ(A).

Theorem (2.3 The monotone class theorem) If A is a π-system on Ω, then

D(A) = σ(A).

Mohamed I. Riffi Probability Theory 1 Math A 9343 Sets Probability Theory Collections of Sets Basics from Measure Theory Generators

Corollary (2.1) If A is an algebra and G a monotone class containing A, then

G ⊃ σ(A).

Theorem (2.3 The monotone class theorem) If A is a π-system on Ω, then

D(A) = σ(A).

Corollary (2.2) If A is a σ-algebra, then

M(A) = D(A) = σ(A) = A.

Mohamed I. Riffi Probability Theory 1 Math A 9343