Finite Probability Spaces Lecture Notes L´aszl´oBabai April 5, 2000 1 Finite Probability Spaces and Events Definition 1.1 A finite probability space is a finite set Ω = together with a function Pr : Ω R+ such that 6 ; ! 1. ! Ω; Pr(!) > 0 8 2 2. Pr(!) = 1: X ! Ω 2 The set Ω is the sample space and the function Pr is the probability distribution. The elements ! Ω are called atomic events or elementary events. An event is a subset of Ω. For A 2Ω, we define the probability of A to be Pr(A) := Pr(!): In particular, for X ⊆ ! A atomic events we have Pr( ! ) = Pr(!); and Pr( ) = 0, Pr(Ω) =2 1. The trivial events are those with probability 0 orf 1,g i. e. and Ω. ; ; The uniform distribution over the sample space Ω is defined by setting Pr(!) = 1= Ω for every ! Ω. With this distribution, we shall speak of the uniform probability spacej j over Ω. In a2 uniform space, calculation of probabilities amounts to counting: Pr(A) = A = Ω . j j j j Exercise 1 In the card game of bridge, a deck of 52 cards are evenly distributed among four players called North, East, South, and West. What sample space does each of the following questions refer to: (a) What is the probability that North holds all the aces? (b) What is the probability that each player holds one of the aces? { These questions refer to uniform probability spaces. Calculate the probabilities. Observation 1.2 Pr(A B) + Pr(A B) = Pr(A) + Pr(B): [ \ Definition 1.3 Events A and B are disjoint if A B = : \ ; 1 k Consequence 1.4 Pr(A1 Ak) i=1 Pr(Ai), and equality holds if and only if the Ai are pairwise disjoint. [···[ ≤ P Definition 1.5 If A and B are events and Pr(B) > 0 then the conditional probability of Pr(A B) A relative to B, written Pr(A B), is given by Pr(A B) = \ : j j Pr(B) We note that B can be viewed as a sample space with the probabilities being the conditional probabilities under condition B. Note that Pr(A B) = Pr(A B) Pr(B). \ j Exercise 2 Prove: Pr(A B C) = Pr(A B C) Pr(B C) Pr(C): \ \ j \ j Exercise 3 We roll three dice. What is the probability that the sum of the three numbers we obtain is 9? What is the probability that the first die shows 5? What is the conditional probability of this event assuming the sum of the numbers shown is 9? { What is the probability space in this problem? How large is the sample space? A partition of Ω is a family of pairwise disjoint events H1;:::;Hm covering Ω: Ω = H1 ::: Hk;Hi Hj = : (1) [ [ \ ; We usually assume that each event in the partition is nontrivial. Exercise 4 Prove: given a partition (H1;:::;Hk) of Ω, we have k Pr(A) = Pr(A H ) Pr(H ): (2) X i i i=1 j for any event A. The significance of this formula is that the conditional probabilities are sometimes easier to calculate than the left hand side. Definition 1.6 Events A and B are independent if Pr(A B) = Pr(A) Pr(B): \ Exercise 5 If Pr(B) > 0 then: A and B are independent Pr(A B) = Pr(A): () j Exercise 6 Prove: if A and B are independent events then A¯ and B are also independent, where A¯ = Ω A. n 2 Exercise 7 (a) If we roll a die, are the following events independent: \the number shown is odd"; \the number shown is prime"? (b) Let us consider a uniform probability space over a sample space whose cardinality is a prime number. Prove that no two non-trivial events can be independent. Note that the trivial events are independent of any other events, i. e. if a trivial event is added to a collection of independent events, they remain independent. The events A and B are said to be positively correlated if Pr(A B) > Pr(A) Pr(B). They are negatively correlated if Pr(A B) < Pr(A) Pr(B). \ \ Exercise 8 Are the two events described in Exercise 3 positively, or negatively correlated, or independent? Exercise 9 Prove: two events A; B are positively (negatively) correlated if and only if Pr(B A) > Pr(B) (Pr(B A) < Pr(B), resp.). j j Definition 1.7 Events A1;:::;Ak are independent if for all subsets S 1; : : : ; k ; we have ⊆ f g Pr( i SAi) = Pr(Ai): (3) 2 Y \ i S 2 Note that if k 3, then the statement that events A1;:::;Ak are independent is stronger than pairwise independence.≥ For example, pairwise independence does not imply triplewise independence. For added emphasis, independent events are sometimes called fully independent, or mutually independent, or collectionwise independent. Exercise 10 Construct 3 events which are pairwise independent but not collectionwise inde- pendent. What is the smallest sample space for this problem? (See the end of this section for more general problems of this type.) Exercise 11 Prove: if the events A; B; C; D; E are independent then the events A B, C D, and E are independent as well. Formulate a general statement, for n events groupedn [ into blocks. Exercise 12 We have n balls colored red, blue, and green (each ball has exactly one color and each color occurs at least once). We select k of the balls with replacement (independently, with uniform distribution). Let A denote the event that the k balls selected have the same color. Let pr denote the conditional probaility that the first ball selected is red, assuming condition A. Define pb and pg analogously for blue and green outcomes. Assume p1 + p2 = p3. Prove: k 2: Show that k = 2 is actually possible. ≤ 3 Exercise 13 (Random graphs) Consider the uniform probability space over the set of all the n 2(2) graphs with a given set V of n vertices. (a) What is the probability that a particular pair of vertices is adjacent? Prove that these n events are independent. (b) What is the probability 2 that the degrees of vertex 1 and vertex 2 are equal? Give a closed-form expression. (c) If pn denotes the probability calculated in part (b), prove that pnpn tends to a finite positive limit and determine its value. (c) How are the following two events correlated: A:\vertex 1 has degree 3"; B:\vertex 2 has degree 3"? Asymptotically evaluate the ratio Pr(A B)= Pr(A). j In exercises like the last one, one often has to estimate binomial coefficients. The following result comes handy: Stirling's formula. n! (n=e)np2πn: (4) ∼ Here the notation refers to asymptotic equality: for two sequences of numbers an and bn ∼ we say that an and bn are asymptotically equal and write an bn if limn an=bn = 1. ∼ !1 To \evaluate a sequence an asymptotically" means to find a simple expression describing a function f(n) such that an f(n). Striling's formula is such an example. While such \asymptotic formulae" are excellent∼ at predicting what happens for \large" n, they do not tell how large is large enough. A stronger, non-asymptotic variant, giving useful results for specific values of n, is the following: n n! = (n=e) p2πn(1 + θn=(12n)); (5) where θn 1. j j ≤ Exercise 14 Evaluate asymptotically the binomial coefficient 2n . Show that 2n c 4n=pn n n where c is a constant. Determine the value of c. ∼ · We mention some important asymptotic relations from number theory. Let π(x) denote the number of all prime numbers x, so π(2) = 1, π(10) = 4, etc. The Prime Number Theorem of Hadamard and de la≤ Vall´ee-Poussin asserts that π(x) x= ln x: (6) ∼ Another important relation estimates the sum of reciprocals of prime numbers. The sum- mation below extends over all primes p x. ≤ 1=p ln ln x: (7) X p x ∼ ≤ 4 In fact a stronger result holds: there exists a number B such that lim 0 1=p ln ln x1 = B: (8) x X − !1 @p x A ≤ (Deduce (7) from (8).) Exercise 15 Assuming 100-digit integers are \large enough" for the Prime Number Theorem to give a good approximation, estimate the probability that a random integer with at most 100 decimal digits is prime. (The integer is drawn with uniform probability from all positive integers in the given range.) Exercise 16 Construct a sample space Ω and events A1;:::;An ( n 2) such that Pr(Ai) = 8 ≥ 1=2, every n 1 of the Ai are independent, but the n events are not independent. − Exercise 17 (*) Let 1 k n 1. (a) Construct a sample space Ω and n events such that every k of these n events≤ are≤ independent;− but no k + 1 of these events are independent. (b) Solve part (a) under the additional constraint that each of the n events have probability 1/2. (Hint. Take a k-dimensional vector space W over a finite field of order q n. Select n vectors from W so that any k are linearly independent. Let W be the sample space.)≥ Exercise 18 Suppose we have n independent nontrivial events. Prove: Ω 2n. j j ≥ Exercise 19 (Small sample space for pairwise independent events.) (a) For n = 2k 1, construct a probability space of size n+1 with n pairwise independent events each of probability− 1=2.