Probability with Engineering Applications ECE 313 Course Notes Bruce Hajek Department of Electrical and Computer Engineering University of Illinois at Urbana-Champaign January 2017 c 2017 by Bruce Hajek All rights reserved. Permission is hereby given to freely print and circulate copies of these notes so long as the notes are left intact and not reproduced for commercial purposes. Email to [email protected], pointing out errors or hard to understand passages or providing comments, is welcome. Contents 1 Foundations 3 1.1 Embracing uncertainty . .3 1.2 Axioms of probability . .6 1.3 Calculating the size of various sets . 10 1.4 Probability experiments with equally likely outcomes . 13 1.5 Sample spaces with infinite cardinality . 15 1.6 Short Answer Questions . 20 1.7 Problems . 21 2 Discrete-type random variables 25 2.1 Random variables and probability mass functions . 25 2.2 The mean and variance of a random variable . 27 2.3 Conditional probabilities . 32 2.4 Independence and the binomial distribution . 34 2.4.1 Mutually independent events . 34 2.4.2 Independent random variables (of discrete-type) . 36 2.4.3 Bernoulli distribution . 37 2.4.4 Binomial distribution . 38 2.5 Geometric distribution . 41 2.6 Bernoulli process and the negative binomial distribution . 43 2.7 The Poisson distribution{a limit of binomial distributions . 45 2.8 Maximum likelihood parameter estimation . 47 2.9 Markov and Chebychev inequalities and confidence intervals . 50 2.10 The law of total probability, and Bayes formula . 53 2.11 Binary hypothesis testing with discrete-type observations . 60 2.11.1 Maximum likelihood (ML) decision rule . 61 2.11.2 Maximum a posteriori probability (MAP) decision rule . 62 2.12 Reliability . 67 2.12.1 Union bound . 67 2.12.2 Network outage probability . 67 2.12.3 Distribution of the capacity of a flow network . 70 2.12.4 Analysis of an array code . 72 iii iv CONTENTS 2.12.5 Reliability of a single backup . 74 2.13 Short Answer Questions . 74 2.14 Problems . 76 3 Continuous-type random variables 95 3.1 Cumulative distribution functions . 95 3.2 Continuous-type random variables . 100 3.3 Uniform distribution . 103 3.4 Exponential distribution . 104 3.5 Poisson processes . 107 3.5.1 Time-scaled Bernoulli processes . 107 3.5.2 Definition and properties of Poisson processes . 108 3.5.3 The Erlang distribution . 112 3.6 Linear scaling of pdfs and the Gaussian distribution . 113 3.6.1 Scaling rule for pdfs . 113 3.6.2 The Gaussian (normal) distribution . 115 3.6.3 The central limit theorem and the Gaussian approximation . 119 3.7 ML parameter estimation for continuous-type variables . 124 3.8 Functions of a random variable . 125 3.8.1 The distribution of a function of a random variable . 125 3.8.2 Generating a random variable with a specified distribution . 135 3.8.3 The area rule for expectation based on the CDF . 137 3.9 Failure rate functions . 138 3.10 Binary hypothesis testing with continuous-type observations . 140 3.11 Short Answer Questions . 146 3.12 Problems . 148 4 Jointly Distributed Random Variables 161 4.1 Joint cumulative distribution functions . 161 4.2 Joint probability mass functions . 163 4.3 Joint probability density functions . 165 4.4 Independence of random variables . 175 4.4.1 Definition of independence for two random variables . 175 4.4.2 Determining from a pdf whether independence holds . 176 4.5 Distribution of sums of random variables . 178 4.5.1 Sums of integer-valued random variables . 179 4.5.2 Sums of jointly continuous-type random variables . 181 4.6 Additional examples using joint distributions . 184 4.7 Joint pdfs of functions of random variables . 189 4.7.1 Transformation of pdfs under a linear mapping . 189 4.7.2 Transformation of pdfs under a one-to-one mapping . 191 4.7.3 Transformation of pdfs under a many-to-one mapping . 195 4.8 Correlation and covariance . 196 CONTENTS v 4.9 Minimum mean square error estimation . 205 4.9.1 Constant estimators . 205 4.9.2 Unconstrained estimators . 205 4.9.3 Linear estimators . 206 4.10 Law of large numbers and central limit theorem . 211 4.10.1 Law of large numbers . 212 4.10.2 Central limit theorem . 214 4.11 Joint Gaussian distribution . 217 4.11.1 From the standard 2-d normal to the general . 218 4.11.2 Key properties of the bivariate normal distribution . 219 4.11.3 Higher dimensional joint Gaussian distributions . 222 4.12 Short Answer Questions . 223 4.13 Problems . 225 5 Wrap-up 239 6 Appendix 241 6.1 Some notation . 241 6.2 Some sums . 242 6.3 Frequently used distributions . 242 6.3.1 Key discrete-type distributions . 242 6.3.2 Key continuous-type distributions . 243 6.4 Normal tables . 245 6.5 Answers to short answer questions . 247 6.6 Solutions to even numbered problems . 248 vi CONTENTS Preface A key objective of these notes is to convey how to deal with uncertainty in both qualitative and quantitative ways. Uncertainty is typically modeled as randomness. We must make decisions with partial information all the time in our daily lives, for instance when we decide what activities to pursue. Engineers deal with uncertainty in their work as well, often with precision and analysis. A challenge in applying reasoning to real world situations is to capture the main issues in a mathematical model. The notation that we use to frame a problem can be critical to understanding or solving the problem. There are often events, or variables, that need to be given names. Probability theory is widely used to model systems in engineering and scientific applications. These notes adopt the most widely used framework of probability, namely the one based on Kol- mogorov's axioms of probability. The idea is to assume a mathematically solid definition of the model. This structure encourages a modeler to have a consistent, if not completely accurate, model. It also offers a commonly used mathematical language for sharing models and calculations. Part of the process of learning to use the language of probability theory is learning classifications of problems into broad areas. For example, some problems involve finite numbers of possible alternatives, while others concern real-valued measurements. Many problems involve interaction of physically independent processes. Certain laws of nature or mathematics cause some probability distributions, such as the normal bell-shaped distribution often mentioned in popular literature, to frequently appear. Thus, there is an emphasis in these notes on well-known probability distributions and why each of them arises frequently in applications. These notes were written for the undergraduate course, ECE 313: Probability with Engineering Applications, offered by the Department of Electrical and Computer Engineering at the University of Illinois at Urbana-Champaign. The official prerequisites of the course ensure that students have had calculus, including Taylor series expansions, integration over regions in the plane, the use of polar coordinates, and some basic linear algebra. The author gratefully acknowledges the students and faculty who have participated in this course through the years. He is particularly grateful to Professor D. V. Sarwate, who first intro- duced the course, and built up much material for it on the website. B. Hajek January 2017 vii CONTENTS 1 Organization Chapter 1 presents an overview of the many applications of probability theory, and then explains the basic concepts of a probability model and the axioms commonly assumed of probability models. Often probabilities are assigned to possible outcomes based on symmetry. For example, when a six sided die is rolled, it is usually assumed that the probability a particular number i shows is 1/6, for 1 ≤ i ≤ 6: For this reason, we also discuss in Chapter 1 how to determine the sizes of various finite sets of possible outcomes. Random variables are introduced in Chapter 2 and examined in the context of a finite, or countably infinite, set of possible outcomes. Notions of expectation (also known as mean), variance, hypothesis testing, parameter estimation, multiple random variables, and well known probability distributions{Poisson, geometric, and binomial, are covered. The Bernoulli process is considered{it provides a simple setting to discuss a long, even infinite, sequence of event times, and provides a tie between the binomial and geometric probability distributions. The focus shifts in Chapter 3 from discrete-type random variables to continuous-type random variables. The chapter takes advantage of many parallels and connections between discrete-type and continuous-type random variables. The most important well known continuous-type distribu- tions.