Chapter 2 Conditional Probability and Independence
Total Page:16
File Type:pdf, Size:1020Kb
CHAPTER 2 CONDITIONAL PROBABILITY AND INDEPENDENCE INTRODUCTION This chapter introduces the important concepts of conditional probability and statistical independence. Conditional probabilities arise when it is known that a certain event has occurred. This knowledge changes the probabilities of events within the sample space of the experiment. Conditioning on an event occurs frequently, and understanding how to work with conditional probabilities and apply them to a particular problem or application is extremely important. In some cases, knowing that a particular event has occurred will not effect the probability of another event, and this leads to the concept of the statistical independence of events that will be developed along with the concept of conditional independence. 2-1 CONDITIONAL PROBABILITY The three probability axioms introduced in the previous chapter provide the foundation upon which to develop a theory of probability. The next step is to understand how the knowledge that a particular event has occurred will change the probabilities that are assigned to the outcomes of an experiment. The concept of a conditional probability is one of the most important concepts in probability and, although a very simple concept, conditional probability is often confusing to students. In situations where a conditional probability is to be used, it is often overlooked or incorrectly applied, thereby leading to an incorrect answer or conclusion. Perhaps the best starting point for the development of conditional probabilities is a simple example that illustrates the context in which they arise. Suppose that we have an electronic device that has a probability pn of still working after n months of continuous operation, and suppose that the probability is equal to 0.5 that the device will still be working after one year (n = 12). The device is then 45 46 CHAPTER 2 CONDITIONAL PROBABILITY AND INDEPENDENCE 1 Ω 1 A B ∪ A B ∩ A B AB ∪ A B (A B)c ∩ 1 ∪ AB (A B)c A B ∪ ∪ A B ∩ ABC (A B)c 1 ∪ Conditioning event 1 A B ∪ Figure 2-1: IllustrationA B of conditioningA B by an event A. Any outcome not in A and any event C that is mutually∩ exclusive of∪A becomes an impossible event. AB A B ∩ (A B)cAB put into operation and∪ after one(A yearB it is)c still working. The question then is “What is the probability that the device∪ will continue working for another n months.” These are known as conditional probabilities because they are probabilities that are conditioned on the event that n 12. With this specific example in≥ mind, let us now look at conditional probability in a more general context. Suppose that we have an experiment with a sample space Ω with probabilities defined on the events in Ω. If it is given that event A has occurred, then the only outcomes that are possible are those that are in A, and any outcomes that are not in A will have a probability of zero. Therefore, it is necessary to adjust or scale the probability of each elementary event within A so that the probability of event A is equal to one. A picture illustrating the effect of conditioning is given in Figure 2-1. There are three observations worth noting at this point: 1. If the probability of an event A is P A , and if it is given that A has occurred, f g then the probability of A becomes equal to one (Axiom 2). In other words, since the only outcomes that are possible are those that are in A, then A has effectively become the new sample space or the new certain event. 2. Conditioning by A will not change the relative probabilities between the experimental outcomes in A. For example, if the probability of the elementary event ! A is equal to the probability of the elementary event i 2 Copyright 2012, M. H. Hayes 2-1 CONDITIONAL PROBABILITY 47 ! A, then conditioning by A will not change this relationship. In other j 2 words, ! and ! will still be equally likely outcomes, P ! = P ! . i j f ig f jg 3. For any event C that is mutually exclusive of the conditioning event, A C = \ ?, the conditional probability of C will be equal to zero. In other words, given that A has occurred, if there are no outcomes in C that are also in A, then P C = 0. f g Important Concept Conditioning an experiment on an event A effectively changes the sample space from Ω to the conditioning event, A since any outcomes not in A will have a probability of zero. To make this a bit more concrete, consider the experiment of rolling a fair die. With a sample space consisting of six equally likely events with a probability of 1/6 for each possible outcome, suppose that the experiment is performed and we are told that the roll of the die is even (we know nothing else about the outcome, only that it is even). How does this information (conditioning) change the probabilities of the remaining events in the sample space? It should be clear that the new information (that the outcome of the roll of the die is even) should not change the relative probabilities of the remaining events, so the remaining outcomes should still be equally likely. Since only three possible outcomes remain, then their conditional probabilities should be equal to one third. Note that this also makes the probability of the conditioning event (the new sample space) equal to one. Thus, the probability that a two is rolled given that the roll resulted in an even number is equal to 1/3, P roll a two, given that the roll is even = 1=3 f g If we define the event A = Roll is even f g and the event B = Roll a two f g then this conditional probability of B given A is written as follows P B A = 1=3 f j g Copyright 2012, M. H. Hayes 48 CHAPTER 2 CONDITIONAL PROBABILITY AND INDEPENDENCE Note that this is not the same as the probability that we roll a two and that the roll is even, which we know is equal to one sixth. A more interesting example of conditional probability is given in the (in)famous Monte Hall problem that may be stated as follows. Monte Hall, a famous game show host, invites you to the stage and explains that behind one of the three large doors behind you there is an expensive sports car, and behind the other two there are small consolation prizes of little value. He tells you that if you select the door that hides the sports car, it is yours to keep. After selecting one of the doors, Monte proceeds to open one of the two remaining doors to show you that the car is not behind that door, and tells you that the car is either behind the door that you selected or behind the other remaining door. Monte then gives you the option to change your selection and choose the other door. The question is Would your chances of winning the car increase, decrease, or remain the same if you were to change your mind, and switch doors? Before your selection was made, it is clear that the car is equally likely to be behind any one of the three doors, so the probability that the car is behind the door that you selected is initially equal to 1/3. So now the question is: What is the probability that the car is behind the door that you selected given that it is not behind the door that was opened by Monte?1 For now, you are asked to think about this problem, and see if you can come up with the correct strategy to maximize your odds of winning the car. The Monte Hall problem is developed in one of the problems at the end of the chapter, which you should be able to solve once you become familiar with conditional probabilities. Having introduced the concept of conditional probability, we now look at how conditional probabilities are found. Let Ω be a sample space with events A and B, and suppose that the probability of event B is to be determined when it is given that event A has occurred, i.e., P B A . Given A, all outcomes in Ω that are not in A become impossible events andf willj g have a probability of zero, and the probability of each outcome in A must be scaled. The scaling factor is 1=P A since this will f g make the probability of event A equal to one, as it must be since it is given that A has occurred. To find the probability of the event B given A, we first find the set of all outcomes that are in both B and A, B A, because any outcome not in B A \ \ will be equal to zero. The probability of this event, P B A , after it is scaled by f \ g 1=P A , is the conditional probability. f g 1Note that this problem would be different if Monte eliminated one of the doors before you make your choice. Copyright 2012, M. H. Hayes 2-1 CONDITIONAL PROBABILITY 49 Conditional Probability Let A be any event with nonzero probability, P A > 0. For any event f g B, the conditional probability of B given A, denoted by P B A , is f j g P B A P B A = f \ g (2.1) f j g P A f g Although it will not be done here, Eq.