2. Basic Concepts of Probability Theory The following basic concepts will be presented. First, set theory is used to specify the sample space and the events of a random experiment. Second, the axioms of probability specify rules for computing the probabilities of events. Third, the notion of conditional probability allows us to determine how partial information about the outcome of an experiment affects the probabilities of events. Conditional probability also allows us to formulate the notion of “independence” of events and of experiments. Finally, we consider “sequential” random experiments that consist of performing a sequence of simple random subexperiments. We show how the probabilities of events in these experiments can be derived from the probabilities of the simpler subexperiments. Throughout the book it is shown that complex random experiments can be analyzed by decomposing them into simple subexperiments. 2.1 Specifying Random Experiments A random experiment is specified by stating an experimental procedure and a set of one or more measurements or observations. An unambiguous statement of exactly what is measured or observed. It may involve more than one measurement or observation. It may consist of a sequence of experiments and measurements Example 2.1 Experiment E1: Select a ball from an urn containing balls numbered 1 to 50. Note the number of the ball. Experiment E2: Select a ball from an urn containing balls numbered 1 to 4. Suppose that balls 1 and 2 are black and that balls 3 and 4 are white. Note the number and color of the ball you select. Experiment E3: Toss a coin three times and note the sequence of heads and tails. Experiment E4: Toss a coin three times and note the number of heads. Experiment E5: Count the number of voice packets containing only silence produced from a group of N speakers in a 10-ms period. Notes and figures are based on or taken from materials in the textbook: Alberto Leon-Garcia, “Probability, Statistics, and Random Processes For Electrical Engineering, 3rd ed.”, Pearson Prentice Hall, 2008, ISBN: 013-147122-8. 1 of 55 Experiment E6: A block of information is transmitted repeatedly over a noisy channel until an error-free block arrives at the receiver. Count the number of transmissions required. Experiment E7: Pick a number at random between zero and one. Experiment E8: Measure the time between page requests in a Web server. Experiment E9: Measure the lifetime of a given computer memory chip in a specified environment. Experiment E10: Determine the value of an audio signal at time Experiment E11: Determine the values of an audio signal at times and Experiment E12: Pick two numbers at random between zero and one. Experiment E13: Pick a number X at random between zero and one, then pick a number Y at random between zero and X. Experiment E14: A system component is installed at time t=0. For t≥0 let X(t)=1 as long as the component is functioning, and X(t)=0 let after the component fails. 2.1.1 The Sample Space The sample space S of a random experiment is defined as the set of all possible outcomes. Sample spaces may be discrete. Sample spaces may be continuous. Sample spaces may be a multi-dimensional composed both discrete and continuous subspaces. The sample space S can be specified compactly by using set notation. It can be visualized by drawing tables, diagrams, intervals of the real line, or regions of the plane. Two basic ways to specify a set: 1. List all the elements, separated by commas, inside a pair of braces. For example: A 1,2,3,,13 2. Give a property that specifies the elements of the set. For example: A x :xisanintegersuch that0 x 3 Sample space examples for the experiments of example 2.1 are shown in Example 2.2. Notes and figures are based on or taken from materials in the textbook: Alberto Leon-Garcia, “Probability, Statistics, and Random Processes For Electrical Engineering, 3rd ed.”, Pearson Prentice Hall, 2008, ISBN: 013-147122-8. 2 of 55 Some more interesting sample spaces are shown in the following figure. Experiment E7: Pick a number at random between zero and one. Experiment E9: Measure the lifetime of a given computer memory chip in a specified environment. Experiment E12: Pick two numbers at random between zero and one. Experiment E13: Pick a number X at random between zero and one, then pick a number Y at random between zero and X. 2.1.2 Events We are usually not interested in the occurrence of specific outcomes, but rather in the occurrence of some event, typically a subset of the sample spaces. We say that A is a subset of B if every element of A also belongs to B. Two events of special interest are the certain event, S, which consists of all outcomes and hence always occurs, and the impossible or null event, Ø, which contains no outcomes and hence never occurs. Notes and figures are based on or taken from materials in the textbook: Alberto Leon-Garcia, “Probability, Statistics, and Random Processes For Electrical Engineering, 3rd ed.”, Pearson Prentice Hall, 2008, ISBN: 013-147122-8. 3 of 55 2.1.3 Review of Set Theory A set is a collection of objects and will be denoted by capital letters: S, A, B, … We define U as the universal set that consists of all possible objects of interest in a given setting or application. In the context of random experiments we refer to the universal set as the sample space. A set A is a collection of objects from U, and these objects are called the elements or points of the set A and will be denoted by lowercase letters. We use the notation: x A or y A read as “x is an element of A” and “y is not an element of A”. We say A is a subset of B if every element of A also belongs to B, that is, if x A implies x B . We say that “A is contained in B” and we write: A B Venn diagrams of set operations and relations. Notes and figures are based on or taken from materials in the textbook: Alberto Leon-Garcia, “Probability, Statistics, and Random Processes For Electrical Engineering, 3rd ed.”, Pearson Prentice Hall, 2008, ISBN: 013-147122-8. 4 of 55 Equality We say sets A and B are equal if they contain the same elements. Since every element in A is also in B, then x A implies x B so A B. Similarly every element in B is also in A, so x B implies x A and so B A. Therefore: A B if and only if A B and B A The standard method to show that two sets, A and B, are equal is to show that A B and B A. A second method is to list all the items in A and all the items in B, and to show that the items are the same. Sum or Union The union of two sets A and B is denoted by A B and is defined as the set of outcomes that are either in A or in B, or both: A B x :x Aor x B The sum or union of sets results in a set that contains all of the elements that are elements of every set being summed. S A1 A2 A3 AN Laws for Unions A B B A A A A A A A S S A B A, if B A Notes and figures are based on or taken from materials in the textbook: Alberto Leon-Garcia, “Probability, Statistics, and Random Processes For Electrical Engineering, 3rd ed.”, Pearson Prentice Hall, 2008, ISBN: 013-147122-8. 5 of 55 Products or Intersection The intersection of two sets A and B is denoted by A B and is defined as the set of outcomes that are in both A and B: A B x :x Aand x B The product or intersection of sets results in a set that contains all of the elements that are present in every one of the sets. S Laws for Intersections A B B A A A A A A S A A B B, if B A Mutually Exclusive or Disjoint Sets Two sets are said to be disjoint or mutually exclusive if their intersection is the null set, A B NOTE: The intersection of two disjoint sets is a set … the null set. Notes and figures are based on or taken from materials in the textbook: Alberto Leon-Garcia, “Probability, Statistics, and Random Processes For Electrical Engineering, 3rd ed.”, Pearson Prentice Hall, 2008, ISBN: 013-147122-8. 6 of 55 Complement The complement of a set A is denoted by AC and is defined as the set of all elements not in A: AC x :x A Laws for Complement S S A A A B,if B A A B,if B A Differences The relative complement or difference of sets A and B is the set of elements in A that are not in B: A B x :x Aand x B A B A B C Laws for Differences A B B B A A A A A A A A A A S S A A S A AC Notes and figures are based on or taken from materials in the textbook: Alberto Leon-Garcia, “Probability, Statistics, and Random Processes For Electrical Engineering, 3rd ed.”, Pearson Prentice Hall, 2008, ISBN: 013-147122-8. 7 of 55 Properties of Sets Commutative properties: A B B A and A B B A Associative properties: A B C A B C and A B C A B C Distributive properties: A B C A B A C and A B C A B A C DeMorgan’s rules: A B C AC BC and A BC AC BC Notes and figures are based on or taken from materials in the textbook: Alberto Leon-Garcia, “Probability, Statistics, and Random Processes For Electrical Engineering, 3rd ed.”, Pearson Prentice Hall, 2008, ISBN: 013-147122-8.