Lecture Notes: Probability and Random Processes at KTH for sf2940 Probability Theory Edition: 2017 Timo Koski Department of Mathematics KTH Royal Institute of Technology Stockholm, Sweden 2 Contents Foreword 9 1 Probability Spaces and Random Variables 11 1.1 Introduction...................................... .......... 11 1.2 Terminology and Notations in Elementary Set Theory . .............. 11 1.3 AlgebrasofSets.................................... .......... 14 1.4 ProbabilitySpace.................................... ......... 19 1.4.1 Probability Measures . ...... 19 1.4.2 Continuity from below and Continuity from above . .......... 21 1.4.3 Why Do We Need Sigma-Fields? . .... 23 1.4.4 P - Negligible Events and P -AlmostSureProperties . 25 1.5 Random Variables and Distribution Functions . ............ 25 1.5.1 Randomness?..................................... ...... 25 1.5.2 Random Variables and Sigma Fields Generated by Random Variables ........... 26 1.5.3 Distribution Functions . ...... 28 1.6 Independence of Random Variables and Sigma Fields, I.I.D. r.v.’s . ............... 29 1.7 TheBorel-CantelliLemmas ............................. .......... 30 1.8 ExpectedValueofaRandomVariable . ........... 32 1.8.1 A First Definition and Some Developments . ........ 32 1.8.2 TheGeneralDefinition............................... ....... 34 1.8.3 The Law of the Unconscious Statistician . ......... 34 1.8.4 Three Inequalities for Expectations . ........... 35 1.8.5 Limits and Integrals . ...... 37 1.9 Appendix: lim sup xn and lim inf xn .................................. 37 1.9.1 Sequencesofrealnumbers . .. .. .. .. .. .. .. .. .. .. .. ......... 37 1.9.2 lim sup xn ............................................ 38 1.9.3 lim inf xn ............................................. 38 1.9.4 Properties, The Limit of a Sequence . ......... 38 1.10 Appendix: lim sup An and lim inf An .................................. 39 1.11 Appendix: Combinatorics of Counting and Statistics of Particles in Cells . 40 1.12Exercises ........................................ ......... 42 1.12.1 Easy Drills . .... 42 1.12.2 Measures, Algebras and Sigma Fields . ......... 42 1.12.3 Random Variables and Expectation . ......... 44 3 4 CONTENTS 2 Probability Distributions 47 2.1 Introduction...................................... .......... 47 2.2 Continuous Distributions . .......... 49 2.2.1 Univariate Continuous Distributions . ......... 49 2.2.2 Continuous Bivariate Distributions . ......... 57 2.2.3 Mean, Variance and Covariance of Linear Combinations of R.V.s . ............ 59 2.3 Discrete Distributions . .......... 59 2.3.1 Univariate........................................ ..... 59 2.3.2 Bivariate Discrete Distributions . ......... 64 2.4 Transformations of Continuous Distributions . ............... 65 2.4.1 The Probability Density of a Function of Random Variable . .......... 65 2.4.2 Change of Variable in a Joint Probability Density . ........ 67 2.5 Appendix: Decompositions of Probability Measures on the Real Line ............... 71 2.5.1 Introduction ..................................... ...... 71 2.5.2 Decompositions of µ on (R, ) ................................. 72 B 2.5.3 Continuous, Discrete and Singular Random Variables . ........... 74 2.6 Exercises ......................................... ........ 75 2.6.1 Distribution Functions . ...... 75 2.6.2 Univariate Probability Density Functions . ......... 76 2.6.3 Multivariate P.d.f.’s . .... 81 2.6.4 ExpectationsandVariances . ......... 87 2.6.5 Additional Exercises . ....... 88 3 Conditional Probability and Expectation w.r.t. a Sigma Field 91 3.1 Introduction...................................... .......... 91 3.2 Conditional Probability Densities and Conditional Expectations . ................. 91 3.3 Conditioning w.r.t. an Event . ......... 94 3.4 Conditioning w.r.t. a Partition . ......... 95 3.5 Conditioning w.r.t. a Random Variable . ......... 96 3.6 A Case with an Explicit Rule for Conditional Expectation . ............ 97 3.7 Conditioning w.r.t. a σ -Field ..................................... 98 3.7.1 Properties of Conditional Expectation . .......... 99 3.7.2 An Application of the Properties of Conditional Expectation w.r.t. a σ-Field . 101 3.7.3 EstimationTheory.................................. ......102 3.7.4 TowerPropertyandEstimationTheory . ..........103 3.7.5 Jensen’s Inequality for Conditional Expectation . ............103 3.8 Exercises ......................................... ........104 3.8.1 Easy Drills . 104 3.8.2 Conditional Probability . ......104 3.8.3 Joint Distributions & Conditional Expectations . ..........106 3.8.4 Miscellaneous . .109 3.8.5 Martingales....................................... .112 CONTENTS 5 4 Characteristic Functions 117 4.1 OnTransformsofFunctions . .. .. .. .. .. .. .. .. .. .. .. ...........117 4.2 Characteristic Functions: Definition and Examples . ...............119 4.2.1 Definition and Necessary Properties of Characteristic Functions ..............119 4.2.2 Examples of Characteristic Functions . ..........122 4.3 Characteristic Functions and Moments of Random Variables . ................127 4.4 Characteristic Functions of Sums of Independent Random Variables ................128 4.5 Expansions of Characteristic Functions . ...............131 4.5.1 ExpansionsandErrorBounds . ........131 4.5.2 A Scaled Sum of Standardised Random Variables (Central Limit Theorem) . .132 4.6 AnAppendix:ALimit ................................... ......133 4.6.1 A Sequence of Numbers with the Limit ex ..........................133 4.6.2 Some Auxiliary Inequalities . ......134 4.6.3 Applications . 135 4.7 Exercises ......................................... ........136 4.7.1 Additional Examples of Characteristic Functions . ............136 4.7.2 Selected ExamQuestionsfrom the PastDecades . ............137 4.7.3 Various Applications of the Characteristic Function . ............138 4.7.4 Mellin Transform in Probability . .139 5 Generating Functions in Probability 143 5.1 Introduction...................................... ..........143 5.2 Probability Generating Functions . ...........143 5.3 Moments and Probability Generating Functions . .............147 5.4 Probability Generating Functions for Sums of Independent RandomVariables . .148 5.5 Sums of a Random Number of Independent Random Variables . ...............149 5.6 TheProbabilityofanEvenNumberofSuccesses . ..............151 5.7 MomentGeneratingFunctions . ...........154 5.7.1 Definition and First Properties . ........154 5.7.2 M.g.f. is really an Exponential Moment Generating Function, E.m.g.f ! ..........157 5.8 Exercises ......................................... ........158 5.8.1 Probability Generating Functions . ........158 5.8.2 MomentGeneratingFunctions . ........159 5.8.3 Sums of a Random Number of Independent Random Variables . ............161 5.8.4 Various Additional Generating Functions in Probability . ...........163 5.8.5 TheChernoffInequality .............................. .......164 6 Convergence of Sequences of Random Variables 165 6.1 Introduction...................................... ..........165 6.2 Definitions of Modes of Convergence, Uniqueness of the Limit . ................167 6.3 RelationsbetweenConvergences . .............169 6.4 SomeRulesofComputation ............................. .........172 6.5 AsymptoticMomentsandPropagationofError . ..............174 6.6 ConvergencebyTransforms . .............177 6.6.1 Theorems on Convergence by Characteristic Functions . ...............177 6.6.2 ConvergenceandGeneratingFunctions . ...........179 6 CONTENTS 6.6.3 CentralLimitTheorem ............................... ......179 6.7 AlmostSureConvergence ............................. ...........180 6.7.1 Definition......................................... 180 6.7.2 Almost Sure Convergence Implies Convergence in Probability . .............180 6.7.3 A Summary of the General Implications between Convergence Concepts and One Special Implication ........................................... 181 6.7.4 TheStrongLawofLargeNumbers . .........182 6.8 Exercises ......................................... ........183 6.8.1 Convergence in Distribution . ........183 6.8.2 CentralLimitTheorem ............................... ......187 6.8.3 Convergence in Probability . ......188 6.8.4 ProofofTheorem6.7.2 ............................... ......190 6.8.5 Almost Sure Convergence, The Interrelationship Between Almost Sure Convergence and Mean Square Convergence, Criteria for Almost Sure Convergence ..............190 7 Convergence in Mean Square and a Hilbert Space 193 7.1 Convergence in Mean Square; Basic Points of View . .............193 7.1.1 Definition......................................... 193 7.1.2 The Hilbert Space L (Ω, , P).................................193 2 F 7.2 Cauchy-Schwartz and Triangle Inequalities . ..............194 7.3 PropertiesofMeanSquareConvergence . ...............194 7.4 Applications........................................ ........196 7.4.1 MeanErgodicTheorem ............................... ......196 7.4.2 MeanSquareConvergenceofSums . ..........196 7.4.3 Mean Square Convergence of Normal Random Variables . .............197 7.5 Subspaces, Orthogonality and Projections in L (Ω, , P)......................198 2 F 7.6 Exercises ......................................... ........201 7.6.1 MeanSquareConvergence . .. .. .. .. .. .. .. .. .. .. .. ........201 7.6.2 Optimal Estimation as Projection on Closed Linear Subspaces in L (Ω, , P) ......201 2 F 8 Gaussian Vectors 205 8.1 Multivariate Gaussian Distribution . ..........205