Theory of Probability Measure theory, classical probability and stochastic analysis Lecture Notes by Gordan Žitkovi´c Department of Mathematics, The University of Texas at Austin Contents Contents 1 I Theory of Probability I 4 1 Measurable spaces 5 1.1 Families of Sets . 5 1.2 Measurable mappings . 8 1.3 Products of measurable spaces . 10 1.4 Real-valued measurable functions . 13 1.5 Additional Problems . 17 2 Measures 19 2.1 Measure spaces . 19 2.2 Extensions of measures and the coin-toss space . 24 2.3 The Lebesgue measure . 28 2.4 Signed measures . 30 2.5 Additional Problems . 33 3 Lebesgue Integration 36 3.1 The construction of the integral . 36 3.2 First properties of the integral . 39 3.3 Null sets . 44 3.4 Additional Problems . 46 4 Lebesgue Spaces and Inequalities 50 4.1 Lebesgue spaces . 50 4.2 Inequalities . 53 4.3 Additional problems . 58 5 Theorems of Fubini-Tonelli and Radon-Nikodym 60 5.1 Products of measure spaces . 60 5.2 The Radon-Nikodym Theorem . 66 5.3 Additional Problems . 70 1 6 Basic Notions of Probability 72 6.1 Probability spaces . 72 6.2 Distributions of random variables, vectors and elements . 74 6.3 Independence . 77 6.4 Sums of independent random variables and convolution . 82 6.5 Do independent random variables exist? . 84 6.6 Additional Problems . 86 7 Weak Convergence and Characteristic Functions 89 7.1 Weak convergence . 89 7.2 Characteristic functions . 96 7.3 Tail behavior . 100 7.4 The continuity theorem . 102 7.5 Additional Problems . 103 8 Classical Limit Theorems 106 8.1 The weak law of large numbers . 106 8.2 An “iid”-central limit theorem . 108 8.3 The Lindeberg-Feller Theorem . 110 8.4 Additional Problems . 114 9 Conditional Expectation 115 9.1 The definition and existence of conditional expectation . 115 9.2 Properties . 118 9.3 Regular conditional distributions . 121 9.4 Additional Problems . 128 10 Discrete Martingales 130 10.1 Discrete-time filtrations and stochastic processes . 130 10.2 Martingales . 131 10.3 Predictability and martingale transforms . 133 10.4 Stopping times . 135 10.5 Convergence of martingales . 137 10.6 Additional problems . 139 11 Uniform Integrability 142 11.1 Uniform integrability . 142 11.2 First properties of uniformly-integrable martingales . 145 11.3 Backward martingales . 147 11.4 Applications of backward martingales . 149 11.5 Exchangeability and de Finetti’s theorem (*) . 150 11.6 Additional Problems . 156 Index 158 2 Preface These notes were written (and are still being heavily edited) to help students with the graduate courses Theory of Probability I and II offered by the Department of Mathematics, University of Texas at Austin. Statements, proofs, or entire sections marked by an asterisk ( ) are not a part of the syllabus ∗ and can be skipped when preparing for midterm, final and prelim exams. GORDAN ŽITKOVIC´ Austin, TX December 2010. 3 Part I Theory of Probability I 4 Chapter 1 Measurable spaces Before we delve into measure theory, let us fix some notation and terminology. denotes a subset (not necessarily proper). • ⊆ A set A is said to be countable if there exists an injection (one-to-one mapping) from A • into N. Note that finite sets are also countable. Sets which are not countable are called uncountable. For two functions f : B C, g : A B, the composition f g : A C of f and g is given • → → ◦ → by (f g)(x)= f(g(x)), for all x A. ◦ ∈ An n N denotes a sequence. More generally, (Aγ)γ Γ denotes a collection indexed by the •{ } ∈ ∈ set Γ. 1.1 Families of Sets Definition 1.1 (Order properties) A (countable) family An n N of subsets of a non-empty set S is { } ∈ said to be 1. increasing if A A for all n N, n ⊆ n+1 ∈ 2. decreasing if A A for all n N, n ⊇ n+1 ∈ 3. pairwise disjoint if A A = for m = n, n ∩ m ∅ 4. a partition of S if An n N is pairwise disjoint and nAn = S. { } ∈ ∪ We use the notation An A to denote that the sequence An n N is increasing and A = nAn. ր { } ∈ ∪ Similarly, An A means that An n N is decreasing and A = nAn. ց { } ∈ ∩ 5 CHAPTER 1. MEASURABLE SPACES Here is a list of some properties that a family of subsets of a nonempty set S can have: S (A1) , ∅∈S (A2) S , ∈S (A3) A Ac , ∈ S ⇒ ∈S (A4) A, B A B , ∈ S ⇒ ∪ ∈S (A5) A, B , A B B A , ∈S ⊆ ⇒ \ ∈S (A6) A, B A B , ∈ S ⇒ ∩ ∈S (A7) A for all n N A , n ∈S ∈ ⇒∪n n ∈S (A8) A , for all n N and A A implies A , n ∈S ∈ n ր ∈S (A9) An , for all n N and An n N is pairwise disjoint implies nAn , ∈S ∈ { } ∈ ∪ ∈S Definition 1.2 (Families of sets) A family of subsets of a non-empty set S is called an S 1. algebra if it satisfies (A1),(A3) and (A4), 2. σ-algebra if it satisfies (A1), (A3) and (A7) 3. π-system if it satisfies (A6), 4. λ-system if it satisfies (A2), (A5) and (A8). Problem 1.3 Show that: 1. Every σ-algebra is an algebra. 2. Each algebra is a π-system and each σ-algebra is an algebra and a λ-system. 3. A family is a σ-algebra if and only if it satisfies (A1), (A3), (A6) and (A9). S 4. A λ-system which is a π-system is also a σ-algebra. 5. There are π-systems which are not algebras. 6. There are algebras which are not σ-algebras (Hint: Pick all finite subsets of an infinite set. That is not an algebra yet, but sets can be added to it so as to become an algebra which is not a σ-algebra.) 7. There are λ-systems which are not π-systems. Definition 1.4 (Generated σ-algebras) For a family of subsets of a non-empty set S, the inter- A section of all σ-algebras on S that contain is denoted by σ( ) and is called the σ-algebra generated A A by . A 6 CHAPTER 1. MEASURABLE SPACES Remark 1.5 Since the family 2S of all subsets of S is a σ-algebra, the concept of a generated σ- algebra is well defined: there is always at least one σ-algebra containing - namely 2S. σ( ) A A is itself a σ-algebra (why?) and it is the smallest (in the sense of set inclusion) σ-algebra that contains . In the same vein, one can define the algebra, the π-system and the λ-system generated A by . The only important property is that intersections of σ-algebras, π-systems and λ-systems A are themselves σ-algebras, π-systems and λ-systems. Problem 1.6 Show, by means of an example, that the union of a family of algebras (on the same S) does not need to be an algebra. Repeat for σ-algebras, π-systems and λ-systems. Definition 1.7 (Topology) A topology on a set S is a family τ of subsets of S which contains ∅ and S and is closed under finite intersections and arbitrary (countable or uncountable!) unions. The elements of τ are often called the open sets. A set S on which a topology is chosen (i.e., a pair (S,τ) of a set and a topology on it) is called a topological space. Remark 1.8 Almost all topologies in these notes will be generated by a metric, i.e., a set A S ⊂ will be open if and only if for each x A there exists ε > 0 such that y S : d(x,y) < ε A. ∈ { ∈ } ⊆ The prime example is R where a set is declared open if it can be represented as a union of open intervals. Definition 1.9 (Borel σ-algebras) If (S,τ) is a topological space, then the σ-algebra σ(τ), generated by all open sets, is called the Borel σ-algebra on (S,τ). Remark 1.10 We often abuse terminology and call S itself a topological space, if the topology τ on it is clear from the context. In the same vein, we often speak of the Borel σ-algebra on a set S. Example 1.11 Some important σ-algebras. Let S be a non-empty set: 1. The set = 2S (also denoted by (S)) consisting of all subsets of is a σ-algebra. S P S 2. At the other extreme, the family = ,S is the smallest σ-algebra on S. It is called the S {∅ } trivial σ-algebra on . S 3. The set of all subsets of S which are either countable or whose complements are countable S is a σ-algebra. It is called the countable-cocountable σ-algebra and is the smallest σ-algebra on S which contains all singletons, i.e., for which x for all x S. { }∈S ∈ 4. The Borel σ-algebra on R (generated by all open sets as defined by the Euclidean metric on R), is denoted by (R). B Problem 1.12 Show that the (R)= σ( ), for any of the following choices of the family : B A A 1. = all open subsets of R , A { } 2. = all closed subsets of R , A { } 7 CHAPTER 1. MEASURABLE SPACES 3. = all open intervals in R , A { } 4. = all closed intervals in R , A { } 5. = all left-closed right-open intervals in R , A { } 6.