DOCSLIB.ORG

Topic 4: Multivariate Random Variables

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Topic 4: Multivariate Random Variables Topic 4: Multivariate random variables Joint, marginal, and conditional pmf ² Joint, marginal, and conditional pdf and cdf ² Independence ² Expectation, covariance, correlation ² Conditional expectation ² Two jointly Gaussian random variables ² ES150 { Harvard SEAS 1 Multiple random variables In many problems, we are interested in more than one random ² variables representing di®erent quantities of interest from the same experiment and the same sample space. { Examples: The tra±c loads at di®erent routers in a network, the received quality at di®erent HDTVs, the shuttle arrival time at di®erent stations. These random variables can be represented by a random vector X that ² assign a vector of real number to each outcome s in the sample space ­. X = (X1; X2; : : : ; Xn) It prompts us to investigate the mutual coupling among these random ² variables. { We will study 2 random variables ¯rst, before generalizing to a vector of n elements. ES150 { Harvard SEAS 2 Joint cdf The joint cdf of two random variables X and Y speci¯es the ² probability of the event X x Y y f · g \ f · g F (x; y) = P [X x; Y y] X;Y · · The joint cdf is de¯ned for all pairs of random variables. Properties of the joint cdf: ² { Non-negativity: F (x; y) 0 X;Y ¸ { Non-decreasing: If x x and y y , 1 · 2 1 · 2 then F (x ; y ) F (x ; y ) X;Y 1 1 · X;Y 2 2 { Boundedness: lim FX;Y (x; y) = 1 x;y!1 lim FX;Y (x; y) = lim FX;Y (x; y) = 0 x!¡1 y!¡1 ES150 { Harvard SEAS 3 { Marginal cdf's: which are the individual cdf's FX (x) = lim FX;Y (x; y) y!1 The marginal cdf's can be obtained from the joint cdf, but usually not the reverse. In general, knowledge of all marginal cdf's is insu±cient to specify the joint cdf. { Rectangle formula: P [a < X b; c < Y d] · · = F (b; d) F (b; c) F (a; d) + F (a; c) X;Y ¡ X;Y ¡ X;Y X;Y ES150 { Harvard SEAS 4 Discrete random variables { Joint pmf Consider two discrete random variables X and Y . ² They are completely speci¯ed by their joint pmf, which speci¯es the ² probability of the event X = x; Y = y f g pX;Y (xk; yj) = P (X = xk; Y = yj) The probability of any event A is given as ² P ((X; Y ) A) = p (x ; y ) 2 X;Y k j (xkX;yj )2A By the axioms of probability ² p (x ; y ) 0 X;Y k j ¸ 1 1 pX;Y (xk; yj) = 1 kX=1 Xj=1 ES150 { Harvard SEAS 5 Marginal and conditional pmf's From the joint pmf P (x; y), we can calculate the individual pmf ² X;Y pX (x) and pY (y), which are now referred to as the marginal pmf 1 pX (xi) = pX;Y (xi; yj) Xj=1 1 Similarly pY (yk) = i=1 pX;Y (xi; yk). { The marginal pmfPis an one-dimensional pmf. { In general, knowledge of all marginal pmf's is insu±cient to specify the joint pmf. Sometimes we know one of the two random variables, and we are ² interested in the probability of the other one. This is captured in the conditional pmf pX;Y (xi; yk) pXjY (X = xi Y = yk) = j pY (yk) provided p (y ) = 0. Otherwise de¯ne p (x y ) = 0. Y k 6 XjY j k ES150 { Harvard SEAS 6 Example: The binary symmetric channel Consider the following binary communication channel Z 0; 1 2 f g X 0; 1 Y 0; 1 2 f g 2 f g The bit sent is X Bern(p); 0 p 1 » · · The noise is Z Bern(²); 0 ² 0:5 » · · The bit received is Y = (X + Z) mod 2 = X Z © where X and Z are independent. Find the following probabilities: 1. pY (y); 2. p (x y); XjY j 3. P [X = Y ], the probability of error. 6 ES150 { Harvard SEAS 7 Jointly continuous random variables { Joint, marginal, and condition pdf Two random variables X and Y are jointly continuous if the ² probability of any event involving (X; Y ) can be expressed as an integral of a probability density function P [(X; Y ) A] = fX;Y (x; y) dx dy 2 Z ZA { fX;Y (x; y) is called the joint probability density function { It is possible to have two continuous random variables that are not jointly continuous. For jointly continuous random variables ² x y FX;Y (x; y) = fX;Y (u; v) dv du Z¡1 Z¡1 @2F (x; y) f (u; v) = X;Y X;Y @x@y Since FX;Y (x; y) is non-decreasing, the joint density is always ES150²{ Harvard SEAS 8 non-negative f (x; y) 0 X;Y ¸ But fX;Y (x; y) is NOT a probability measure (it can be > 1). By the axioms of probability, ² 1 1 fX;Y (x; y) dx dy = 1 Z¡1 Z¡1 The marginal pdf can be obtained from the joint pdf as ² 1 fX (x) = fX;Y (x; y)dy Z¡1 The conditional pdf is given as ² fX;Y (x; y) fXjY (x y) = j fY (y) Example: Consider two jointly Gaussian random variables with the ² joint pdf 1 x2 2½xy + y2 fX;Y (x; y) = exp ¡ 2¼ 1 ½2 ½¡ 2(1 ½2) ¾ ¡ ¡ ES150 { Harvard SEAS p 9 { The marginal pdf of X is Gaussian 2 1 ¡x =2 fX (x) = e p2¼ { The conditional pdf of X given Y is also Gaussian 1 (x ½y)2 fX (x y) = exp ¡ j 2¼(1 ½2) ½¡ 2(1 ½2) ¾ ¡ ¡ p ES150 { Harvard SEAS 10 Independence Independence between 2 random variables X and Y means ² P (X B; Y C) = P (X B) P (Y C) 2 2 2 2 Two random variables X and Y are independent if and only if ² FX;Y (x; y) = FX (x)FY (y) Equivalently, in terms of the joint and conditional densities ² { For discrete r.v.'s: pX;Y (xi; yk) = pX (xi)pY (yk) p (x y ) = p (x ) XjY ij k X i { For jointly continuous r.v.'s: fX;Y (x; y) = fX (x)fY (y) f (x y) = f (x) XjY j X ES150 { Harvard SEAS 11 X and Y independent implies ² E[XY ] = E[X]E[Y ] The reverse is not always true. That is, E[XY ] = E[X]E[Y ] does not necessarily imply X and Y are independent. Example: Consider discrete RVs X and Y such that ² 1 P [X = 1] = P [X = 2] = and Y = X2 § § 4 Then E[X] = 0 and E[XY ] = 0, but X and Y are not independent. Find pY (1), pY (4), pXY (1; 4). ES150 { Harvard SEAS 12 Expectation, covariance, correlation Let g(x; y) be a function of two random variables X and Y . The ² expectation of g(X; Y ) is given by g(xi; yk)pX;Y (xi; yk) discrete X; Y 8 E[g(X; Y )] = Xk Xi > 1 1 < g(x; y)fX;Y (x; y) dx dy jointly continuous X; Y Z¡1 Z¡1 > The covarianc:e between 2 random variables is de¯ned as ² cov(X; Y ) = E X E[X] Y E[Y ] ¡ ¡ = E[£XY ] E[X¤£]E[Y ] ¤ ¡ { If E[X] = 0 or E[Y ] = 0 then cov(X; Y ) = E[XY ]. { Covariance is analogous to the variance of a single random variable: cov(X; X) = var(X). E[XY ] is the correlation between the two random variables. ² { By the Cauchy-Schwatz inequality E[XY ] E[X2]E[Y 2] j j · ES150 { Harvard SEAS p 13 X and Y are said to be (note the di®erence in terminology here) ² { orthogonal if E[XY ] = 0 { uncorrelated if cov(X; Y ) = 0 The correlation coe±cient is de¯ned as ² cov(X; Y ) ½X;Y = X Y where X = var(X) and Y = var(Y ) are the standard deviations. Properties: p p { Bounded: 1 ½ 1 ¡ · X;Y · { If ½X;Y = 0, X and Y are uncorrelated { If X and Y are independent, then ½X;Y = 0. Independence implies uncorrelation, but not the reverse. For Gaussian random variables X and Y , however, if they are ² uncorrelated, then they are also independent. ES150 { Harvard SEAS 14 Conditional expectation The conditional expectation of X given Y is de¯ned as ² { For discrete r.v.'s: E X Y = y = x p (x y ) j k i XjY ij k £ ¤ Xi { For jointly continuous r.v.'s: 1 E X Y = y = x f (x y)dx j Z XjY j £ ¤ ¡1 Similarly for the conditional expectation of a function of X, given Y ² g(x )p (x y) for discrete X; Y i XjY ij 8 i E [g(X) Y = y] = X1 j > < g(x) fXjY (x y)dx for jointly continuous X; Y Z¡1 j > :> The law of conditional expectation ² E[g(X)] = E E[g(X) Y ] Y j ES150 { Harvard SEAS £ ¤ 15 For discrete r.v.'s E[g(X)] = E g(X) y p (y ) j k Y k Xk £ ¤ For continuous r.v.'s 1 E[g(X)] = E g(X) y fY (y)dy Z j ¡1 £ ¤ This law is very useful in calculating the expectation. Example: Defects in a chip. ² { The total number of defects on a chip is X Poisson(®). » { The probability of ¯nding an defect in the memory is p. Find the expected number of defects in the memory. ES150 { Harvard SEAS 16 Two jointly Gaussian random variables The joint pdf of two jointly Gaussian r.v.'s X and Y is 2 2 1 (x¡¹X ) (x¡¹X )(y¡¹Y ) (y¡¹Y ) exp 2 2 2½ + 2 2(1¡½ ) XY X Y ¡ XY X ¡ Y fX;Y (x; y) = n h io 2¼ 1 ½2 X Y ¡ XY p The pdf is a function only of ¹ , ¹ , , , and ½ .
Load more

Recommended publications
  • Measure-Theoretic Probability I
    Measure-Theoretic Probability I Steven P.Lalley Winter 2017 1 1 Measure Theory 1.1 Why Measure Theory? There are two different views – not necessarily exclusive – on what “probability” means: the subjectivist view and the frequentist view. To the subjectivist, probability is a system of laws that should govern a rational person’s behavior in situations where a bet must be placed (not necessarily just in a casino, but in situations where a decision must be made about how to proceed when only imperfect information about the outcome of the decision is available, for instance, should I allow Dr. Scissorhands to replace my arthritic knee by a plastic joint?). To the frequentist, the laws of probability describe the long- run relative frequencies of different events in “experiments” that can be repeated under roughly identical conditions, for instance, rolling a pair of dice. For the frequentist inter- pretation, it is imperative that probability spaces be large enough to allow a description of an experiment, like dice-rolling, that is repeated infinitely many times, and that the mathematical laws should permit easy handling of limits, so that one can make sense of things like “the probability that the long-run fraction of dice rolls where the two dice sum to 7 is 1/6”. But even for the subjectivist, the laws of probability should allow for description of situations where there might be a continuum of possible outcomes, or pos- sible actions to be taken. Once one is reconciled to the need for such flexibility, it soon becomes apparent that measure theory (the theory of countably additive, as opposed to merely finitely additive measures) is the only way to go.
    [Show full text]
  • Probability and Statistics Lecture Notes
    Probability and Statistics Lecture Notes Antonio Jiménez-Martínez Chapter 1 Probability spaces In this chapter we introduce the theoretical structures that will allow us to assign proba- bilities in a wide range of probability problems. 1.1. Examples of random phenomena Science attempts to formulate general laws on the basis of observation and experiment. The simplest and most used scheme of such laws is: if a set of conditions B is satisfied =) event A occurs. Examples of such laws are the law of gravity, the law of conservation of mass, and many other instances in chemistry, physics, biology... If event A occurs inevitably whenever the set of conditions B is satisfied, we say that A is certain or sure (under the set of conditions B). If A can never occur whenever B is satisfied, we say that A is impossible (under the set of conditions B). If A may or may not occur whenever B is satisfied, then A is said to be a random phenomenon. Random phenomena is our subject matter. Unlike certain and impossible events, the presence of randomness implies that the set of conditions B do not reflect all the necessary and sufficient conditions for the event A to occur. It might seem them impossible to make any worthwhile statements about random phenomena. However, experience has shown that many random phenomena exhibit a statistical regularity that makes them subject to study. For such random phenomena it is possible to estimate the chance of occurrence of the random event. This estimate can be obtained from laws, called probabilistic or stochastic, with the form: if a set of conditions B is satisfied event A occurs m times =) repeatedly n times out of the n repetitions.
    [Show full text]
  • Random Variable = a Real-Valued Function of an Outcome X = F(Outcome)
    Random Variables (Chapter 2) Random variable = A real-valued function of an outcome X = f(outcome) Domain of X: Sample space of the experiment. Ex: Consider an experiment consisting of 3 Bernoulli trials. Bernoulli trial = Only two possible outcomes – success (S) or failure (F). • “IF” statement: if … then “S” else “F” • Examine each component. S = “acceptable”, F = “defective”. • Transmit binary digits through a communication channel. S = “digit received correctly”, F = “digit received incorrectly”. Suppose the trials are independent and each trial has a probability ½ of success. X = # successes observed in the experiment. Possible values: Outcome Value of X (SSS) (SSF) (SFS) … … (FFF) Random variable: • Assigns a real number to each outcome in S. • Denoted by X, Y, Z, etc., and its values by x, y, z, etc. • Its value depends on chance. • Its value becomes available once the experiment is completed and the outcome is known. • Probabilities of its values are determined by the probabilities of the outcomes in the sample space. Probability distribution of X = A table, formula or a graph that summarizes how the total probability of one is distributed over all the possible values of X. In the Bernoulli trials example, what is the distribution of X? 1 Two types of random variables: Discrete rv = Takes finite or countable number of values • Number of jobs in a queue • Number of errors • Number of successes, etc. Continuous rv = Takes all values in an interval – i.e., it has uncountable number of values. • Execution time • Waiting time • Miles per gallon • Distance traveled, etc. Discrete random variables X = A discrete rv.
    [Show full text]
  • Determinism, Indeterminism and the Statistical Postulate
    Tipicality, Explanation, The Statistical Postulate, and GRW Valia Allori Northern Illinois University [email protected] www.valiaallori.com Rutgers, October 24-26, 2019 1 Overview Context: Explanation of the macroscopic laws of thermodynamics in the Boltzmannian approach Among the ingredients: the statistical postulate (connected with the notion of probability) In this presentation: typicality Def: P is a typical property of X-type object/phenomena iff the vast majority of objects/phenomena of type X possesses P Part I: typicality is sufficient to explain macroscopic laws – explanatory schema based on typicality: you explain P if you explain that P is typical Part II: the statistical postulate as derivable from the dynamics Part III: if so, no preference for indeterministic theories in the quantum domain 2 Summary of Boltzmann- 1 Aim: ‘derive’ macroscopic laws of thermodynamics in terms of the microscopic Newtonian dynamics Problems: Technical ones: There are to many particles to do exact calculations Solution: statistical methods - if 푁 is big enough, one can use suitable mathematical technique to obtain information of macro systems even without having the exact solution Conceptual ones: Macro processes are irreversible, while micro processes are not Boltzmann 3 Summary of Boltzmann- 2 Postulate 1: the microscopic dynamics microstate 푋 = 푟1, … . 푟푁, 푣1, … . 푣푁 in phase space Partition of phase space into Macrostates Macrostate 푀(푋): set of macroscopically indistinguishable microstates Macroscopic view Many 푋 for a given 푀 given 푀, which 푋 is unknown Macro Properties (e.g. temperature): they slowly vary on the Macro scale There is a particular Macrostate which is incredibly bigger than the others There are many ways more to have, for instance, uniform temperature than not equilibrium (Macro)state 4 Summary of Boltzmann- 3 Entropy: (def) proportional to the size of the Macrostate in phase space (i.e.
    [Show full text]
  • Is the Cosmos Random?
    IS THE RANDOM? COSMOS QUANTUM PHYSICS Einstein’s assertion that God does not play dice with the universe has been misinterpreted By George Musser Few of Albert Einstein’s sayings have been as widely quot- ed as his remark that God does not play dice with the universe. People have naturally taken his quip as proof that he was dogmatically opposed to quantum mechanics, which views randomness as a built-in feature of the physical world. When a radioactive nucleus decays, it does so sponta- neously; no rule will tell you when or why. When a particle of light strikes a half-silvered mirror, it either reflects off it or passes through; the out- come is open until the moment it occurs. You do not need to visit a labora- tory to see these processes: lots of Web sites display streams of random digits generated by Geiger counters or quantum optics. Being unpredict- able even in principle, such numbers are ideal for cryptography, statistics and online poker. Einstein, so the standard tale goes, refused to accept that some things are indeterministic—they just happen, and there is not a darned thing anyone can do to figure out why. Almost alone among his peers, he clung to the clockwork universe of classical physics, ticking mechanistically, each moment dictating the next. The dice-playing line became emblemat- ic of the B side of his life: the tragedy of a revolutionary turned reaction- ary who upended physics with relativity theory but was, as Niels Bohr put it, “out to lunch” on quantum theory.
    [Show full text]
  • Topic 1: Basic Probability Definition of Sets
    Topic 1: Basic probability ² Review of sets ² Sample space and probability measure ² Probability axioms ² Basic probability laws ² Conditional probability ² Bayes' rules ² Independence ² Counting ES150 { Harvard SEAS 1 De¯nition of Sets ² A set S is a collection of objects, which are the elements of the set. { The number of elements in a set S can be ¯nite S = fx1; x2; : : : ; xng or in¯nite but countable S = fx1; x2; : : :g or uncountably in¯nite. { S can also contain elements with a certain property S = fx j x satis¯es P g ² S is a subset of T if every element of S also belongs to T S ½ T or T S If S ½ T and T ½ S then S = T . ² The universal set ­ is the set of all objects within a context. We then consider all sets S ½ ­. ES150 { Harvard SEAS 2 Set Operations and Properties ² Set operations { Complement Ac: set of all elements not in A { Union A \ B: set of all elements in A or B or both { Intersection A [ B: set of all elements common in both A and B { Di®erence A ¡ B: set containing all elements in A but not in B. ² Properties of set operations { Commutative: A \ B = B \ A and A [ B = B [ A. (But A ¡ B 6= B ¡ A). { Associative: (A \ B) \ C = A \ (B \ C) = A \ B \ C. (also for [) { Distributive: A \ (B [ C) = (A \ B) [ (A \ C) A [ (B \ C) = (A [ B) \ (A [ C) { DeMorgan's laws: (A \ B)c = Ac [ Bc (A [ B)c = Ac \ Bc ES150 { Harvard SEAS 3 Elements of probability theory A probabilistic model includes ² The sample space ­ of an experiment { set of all possible outcomes { ¯nite or in¯nite { discrete or continuous { possibly multi-dimensional ² An event A is a set of outcomes { a subset of the sample space, A ½ ­.
    [Show full text]
  • Measure Theory and Probability
    Measure theory and probability Alexander Grigoryan University of Bielefeld Lecture Notes, October 2007 - February 2008 Contents 1 Construction of measures 3 1.1Introductionandexamples........................... 3 1.2 σ-additive measures ............................... 5 1.3 An example of using probability theory . .................. 7 1.4Extensionofmeasurefromsemi-ringtoaring................ 8 1.5 Extension of measure to a σ-algebra...................... 11 1.5.1 σ-rings and σ-algebras......................... 11 1.5.2 Outermeasure............................. 13 1.5.3 Symmetric difference.......................... 14 1.5.4 Measurable sets . ............................ 16 1.6 σ-finitemeasures................................ 20 1.7Nullsets..................................... 23 1.8 Lebesgue measure in Rn ............................ 25 1.8.1 Productmeasure............................ 25 1.8.2 Construction of measure in Rn. .................... 26 1.9 Probability spaces ................................ 28 1.10 Independence . ................................. 29 2 Integration 38 2.1 Measurable functions.............................. 38 2.2Sequencesofmeasurablefunctions....................... 42 2.3 The Lebesgue integral for finitemeasures................... 47 2.3.1 Simplefunctions............................ 47 2.3.2 Positivemeasurablefunctions..................... 49 2.3.3 Integrablefunctions........................... 52 2.4Integrationoversubsets............................ 56 2.5 The Lebesgue integral for σ-finitemeasure.................
    [Show full text]
  • 1 Probability Measure and Random Variables
    1 Probability measure and random variables 1.1 Probability spaces and measures We will use the term experiment in a very general way to refer to some process that produces a random outcome. Definition 1. The set of possible outcomes is called the sample space. We will typically denote an individual outcome by ω and the sample space by Ω. Set notation: A B, A is a subset of B, means that every element of A is also in B. The union⊂ A B of A and B is the of all elements that are in A or B, including those that∪ are in both. The intersection A B of A and B is the set of all elements that are in both of A and B. ∩ n j=1Aj is the set of elements that are in at least one of the Aj. ∪n j=1Aj is the set of elements that are in all of the Aj. ∩∞ ∞ j=1Aj, j=1Aj are ... Two∩ sets A∪ and B are disjoint if A B = . denotes the empty set, the set with no elements. ∩ ∅ ∅ Complements: The complement of an event A, denoted Ac, is the set of outcomes (in Ω) which are not in A. Note that the book writes it as Ω A. De Morgan’s laws: \ (A B)c = Ac Bc ∪ ∩ (A B)c = Ac Bc ∩ ∪ c c ( Aj) = Aj j j [ \ c c ( Aj) = Aj j j \ [ (1) Definition 2. Let Ω be a sample space. A collection of subsets of Ω is a σ-field if F 1.
    [Show full text]
  • Sample Space, Events and Probability
    Sample Space, Events and Probability Sample Space and Events There are lots of phenomena in nature, like tossing a coin or tossing a die, whose outcomes cannot be predicted with certainty in advance, but the set of all the possible outcomes is known. These are what we call random phenomena or random experiments. Probability theory is concerned with such random phenomena or random experiments. Consider a random experiment. The set of all the possible outcomes is called the sample space of the experiment and is usually denoted by S. Any subset E of the sample space S is called an event. Here are some examples. Example 1 Tossing a coin. The sample space is S = fH; T g. E = fHg is an event. Example 2 Tossing a die. The sample space is S = f1; 2; 3; 4; 5; 6g. E = f2; 4; 6g is an event, which can be described in words as "the number is even". Example 3 Tossing a coin twice. The sample space is S = fHH;HT;TH;TT g. E = fHH; HT g is an event, which can be described in words as "the first toss results in a Heads. Example 4 Tossing a die twice. The sample space is S = f(i; j): i; j = 1; 2;:::; 6g, which contains 36 elements. "The sum of the results of the two toss is equal to 10" is an event. Example 5 Choosing a point from the interval (0; 1). The sample space is S = (0; 1). E = (1=3; 1=2) is an event. Example 6 Measuring the lifetime of a lightbulb.
    [Show full text]
  • Probability Theory: STAT310/MATH230; September 12, 2010 Amir Dembo
    Probability Theory: STAT310/MATH230; September 12, 2010 Amir Dembo E-mail address: [email protected] Department of Mathematics, Stanford University, Stanford, CA 94305. Contents Preface 5 Chapter 1. Probability, measure and integration 7 1.1. Probability spaces, measures and σ-algebras 7 1.2. Random variables and their distribution 18 1.3. Integration and the (mathematical) expectation 30 1.4. Independence and product measures 54 Chapter 2. Asymptotics: the law of large numbers 71 2.1. Weak laws of large numbers 71 2.2. The Borel-Cantelli lemmas 77 2.3. Strong law of large numbers 85 Chapter 3. Weak convergence, clt and Poisson approximation 95 3.1. The Central Limit Theorem 95 3.2. Weak convergence 103 3.3. Characteristic functions 117 3.4. Poisson approximation and the Poisson process 133 3.5. Random vectors and the multivariate clt 140 Chapter 4. Conditional expectations and probabilities 151 4.1. Conditional expectation: existence and uniqueness 151 4.2. Properties of the conditional expectation 156 4.3. The conditional expectation as an orthogonal projection 164 4.4. Regular conditional probability distributions 169 Chapter 5. Discrete time martingales and stopping times 175 5.1. Definitions and closure properties 175 5.2. Martingale representations and inequalities 184 5.3. The convergence of Martingales 191 5.4. The optional stopping theorem 203 5.5. Reversed MGs, likelihood ratios and branching processes 209 Chapter 6. Markov chains 225 6.1. Canonical construction and the strong Markov property 225 6.2. Markov chains with countable state space 233 6.3. General state space: Doeblin and Harris chains 255 Chapter 7.
    [Show full text]
  • 1. Review of Probability We Start out with a Quick Review of Probability Theory
    STATISTICS CHRISTIAN REMLING 1. Review of probability We start out with a quick review of probability theory. A probability measure P (or just probability, in short) is a function on subsets of a sample space Ω with the following properties: (1) 0 ≤ P (A) ≤ 1, P (Ω) = 1 ; (2) P (A [ B) = P (A) + P (B) if A \ B = ; The subsets A; B ⊆ Ω are often called events in this context. Say you roll a fair die. Then you would describe this random ex- periment with the help of the sample space Ω = f1; 2;:::; 6g and the probability measure P (fjg) = 1=6 for j = 1; 2;:::; 6. Strictly speaking, I haven't specified a full probability measure yet: so far, P (A) has been defined only for one element subsets of Ω, not for general A. However, additivity (= property (2)) of course forces us to set P (A) = jAj=6 for arbitrary A ⊆ Ω, and this P does have properties (1), (2). More generally, this procedure lets us define probability measures on arbitrary finite or countable sample spaces Ω = f!1;!2;:::g: suppose P we are given numbers pn, 0 ≤ pn ≤ 1, with pn = 1. Then X (1.1) P (A) = pn n:!n2A is a probability measure. The dice example above is of this type, with pn = 1=6 for n = 1; 2;:::; 6. Such a finite or countable Ω is called a discrete sample space. Exercise 1.1. Show that conversely, every probability P on a discrete sample space is of the type (1.1) for suitable pn's.
    [Show full text]
  • Probability Spaces Lecturer: Dr
    EE5110: Probability Foundations for Electrical Engineers July-November 2015 Lecture 4: Probability Spaces Lecturer: Dr. Krishna Jagannathan Scribe: Jainam Doshi, Arjun Nadh and Ajay M 4.1 Introduction Just as a point is not defined in elementary geometry, probability theory begins with two entities that are not defined. These undefined entities are a Random Experiment and its Outcome: These two concepts are to be understood intuitively, as suggested by their respective English meanings. We use these undefined terms to define other entities. Definition 4.1 The Sample Space Ω of a random experiment is the set of all possible outcomes of a random experiment. An outcome (or elementary outcome) of the random experiment is usually denoted by !: Thus, when a random experiment is performed, the outcome ! 2 Ω is picked by the Goddess of Chance or Mother Nature or your favourite genie. Note that the sample space Ω can be finite or infinite. Indeed, depending on the cardinality of Ω, it can be classified as follows: 1. Finite sample space 2. Countably infinite sample space 3. Uncountable sample space It is imperative to note that for a given random experiment, its sample space is defined depending on what one is interested in observing as the outcome. We illustrate this using an example. Consider a person tossing a coin. This is a random experiment. Now consider the following three cases: • Suppose one is interested in knowing whether the toss produces a head or a tail, then the sample space is given by, Ω = fH; T g. Here, as there are only two possible outcomes, the sample space is said to be finite.
    [Show full text]