Lenstra Notes.Pdf

Home , Conditional probability, Outcome (probability), Sample space

DEPARTMENT OF MATHEMATICS

UNIVERSITY OF CALIFORNIA, BERKELEY C

Probability theory

H.W. Lenstra, Jr.

These notes contain material on probability for Math 55, Discrete mathematics. They

were written to supplement sections 3.6 and 3.7 of C. L. Liu, Elements of discrete math-

ematics McGraw-Hill, second edition, 1985, but they can b e used in conjunction with

other textb o oks as well. Novemb er 1988.

1. Sample spaces, events and conditional probabilities.

A sample space is a nite or countable set S together with a function

P : S ! [0; 1] = fy 2 R :0 y 1g;

such that the following condition is satis ed:

P x=1:

x2S

One thinks of S as the set of all p ossible outcomes of an exp eriment, with P x equal to

the probability that the outcome is x.We call P the probability function.

In manyinteresting cases S is nite and P x=1=S for all x 2 S .

Let S b e a sample space, with probability function P .Anevent is a subset A of S ,

and the probabilityofanevent A is given by

P A= P x:

x2A

If A, B are twoevents, and P B 6= 0, then the conditional probability of A given B is

de ned by

P A \ B

: P AjB =

P B

Examples of sample spaces, events and conditional probabilities, and exercises on them,

can b e found in many textb o oks on discrete mathematics. 1

2. Indep endence.

Twoevents A, B are called indep endent if P A \ B =P A P B . This means the same

as P AjB =P A if P B 6= 0, so twoevents are indep endent if the o ccurrence of one

of them do es not make the other more likely or less likely to o ccur.

Example 1. Let S = f1; 2; 3; 4; 5; 6gf1; 2; 3; 4; 5; 6g, the set of outcomes of rolling

two dice, with each outcome having probability . Consider the events

A = the rst die equals 3,

B = the second die equals 4,

C = the total equals 7,

D = the total equals 6,

more precisely

A = f3gf1; 2; 3; 4; 5; 6g;

B = f1; 2; 3; 4; 5; 6g f4g;

C = f1; 6; 2; 5; 3; 4; 4; 3; 5; 2; 6; 1g;

D = f1; 5; 2; 4; 3; 3; 4; 2; 5; 1g:

= P A P B . Also A and C are Here A and B are indep endent, since P A \ B =

indep endent, and B and C are indep endent. However, A and D are not indep endent, since

1 1 5 1

P A \ D = , whereas P A P D = , which is smaller than . Likewise B and

36 6 36 36

D are not indep endent. Are C and D indep endent?

Warning.Anytwo of the three events A, B , C are indep endent, but nevertheless the

three events A, B , C cannot b e considered indep endent, since the o ccurrence of anytwoof

them implies the o ccurrence of the third one. For three events A, B , C to b e indep endent

one requires not only that anytwo are indep endent, but also that P A \ B \ C =P A

P B P C . For more than three events one has to lo ok at even more combinations, see

for example Exercise 25 a. 2

3. Random variables.

It frequently happ ens that one is not interested in the actual outcome of the exp eriment,

but in a certain function of the outcome. For example, if one throws two dice one is often

just interested in the total score; or if one tosses a coin n times, it may b e that one is

just interested in the numb er of heads app earing, or in the longest consecutive sequence

of tails. These are examples of random variables.

Let, generally, S b e a sample space, with probability function P .Arandom variable

is a function f : S ! R.Ifr is a real numb er in the image of f , then the probability that

f assumes the value r is de ned by

P f = r = P x:

x2S; f x=r

This is the same as the probability of the event f x=r . The exp ectation or exp ected

value,ormean of a random variable f is de ned by

f xP x: E f =

x2S

This may b e thought of as the \average" value of f if one rep eats the exp eriment a great

numb er of times. An alternative formula for E f is

E f = r P f = r :

r 2f S

To prove this, notice that for each r 2 f S wehave

f xP x=r P f = r ;

x2S; f x=r

by the de nition of P f = r . Now sum this over r 2 f S .

, and let f : S ! R b e the inclusion Example 2. Let S = f0; 1g, with P 0 = P 1 =

map so f 0 = 0, f 1 = 1. Then E f = .

Example 3. Let S = f1; 2; 3; 4; 5; 6g, with P x= for each x 2 S throwing one die,

and let f : S ! R again b e the inclusion map. Then E f = 1+2+3+4+5+6=6= 3 .

If f , g are two random variables, then f + g is also a random variable; namely,f + g x

is de ned to b e f x+g x, for all x 2 S . One has

E f + g =E f +E g : 3

This follows directly from the de nition.

Example 4. Let S , P b e as in Example 1 rolling two dice. If f is the value of the

rst die so f a; b = a, and g is the value of the second die g a; b = b, then

. The total is f + g , so the exp ectation of the total score equals E f =E g =3

E f + g = E f +E g =7.

4. Events as random variables.

Let S b e a sample space, with probability function P . Let A b e an event; so this is a

subset of S . It gives rise to a random variable f that is de ned by

1 if x 2 A,

f x=

0 if x=2 A.

This random variable assumes only the values 0 and 1; and any random variable that

assumes only the values 0 and 1 arises in this way from some event. The probabilityof

the event A is the same as the exp ectation E f of the corresp onding random variable f .

Example 5. Let p 2 R,0

with probability function given by P 1 = p, P 0 = q . This sample space can b e thought

of as the set of outcomes of tossing a coin that is not fair unless p = . The probability

of the event A = f1g is the same as the exp ectation of the inclusion map see Example 2,

namely p.

Example 6. Rep eat the exp eriment from the previous example n times; so S must then

n k nk

b e replaced by f0; 1g , with x 2 S having probability p q if x contains exactly k 1's

any x 2 S is a sequence of n 0's and 1's. For each i =1,2, :::, n, let A b e the event

that the i-th toss of the coin gives a 1, and let f b e the corresp onding random variable;

so if x =x ;x ;:::; x , then f x=x . Each A has probability p, so each f has

1 2 n i i i i

exp ectation p.

Now let f = f + f + + f . Then f x is the total numb er of 1's in x, and

1 2 n

E f =E f + f + + f =E f +E f + + E f =np:

1 2 n 1 2 n

So the exp ected numb er of 1's is np.We can also verify this by a direct calculation. First

one notes that

k nk

p q ; P f = k =

b ecause there are elements x 2 S that have exactly k 1's, and each such x has proba-

k nk

bility p q .Now

n n

X X

n 1 n

k 1 nk n1 k nk

p q = npp + q = np: p q = np k E f =

k 1 k

k =1 k =0 4

A random variable f : S ! R is said to b e a binomial random variable,ortohavea binomial

distribution, if there exists an integer n 0 and a real number p,0

, then we see not surprisingly that the holds for all k ,0 k n.Ifwe take p =

average numb er of elements of a subset of a set of n elements equals n.

Example 7. Ten men went to a party and checked their hats when they arrived. The

hats were randomly returned to them when they departed. Wewant to know the exp ected

numb er of men that get their own hats back. In this example, S is the set of bijective

functions of a set M of 10 men to itself, all these functions having the same probability,

whichis1=S =1=10! = 1=3628800 =0:0000002755732. The bijective function that is the

outcome of the exp eriment maps i to j if i receives j 's hat. The numb er of men that receive

their own hats back is given by the random variable f : S ! R that sends each bijective

function b: M ! M to the numb er of elements that it xes: f b=fi 2 M : bi=ig.

Wewant to calculate the exp ectation of f . As in the previous example, this exp ectation

can b e written as the sum of the probabilities of 10 events A , where A is the event that

i i

the i-th man gets his own hat back. It is not hard to see that A = 9! for each i,so

1 1

P A = and E f =10 = 1: on the average, one man gets his own hat back! This

10 10

remains true for any p ositive numb er of men instead of 10.

Example 8. Consider the exp eriment of sho oting at a target until there is a hit. Let p

b e the probability of hitting if one sho ots once, with 0

the probability of missing. Assume that all shots are indep endent. What is the exp ected

numb er of shots? Here S = fh; mh; mmh; mmmh; : : :g, where the n-th element has

probability q p one factor q for each miss, and p for the hit; you may wish to check

that all these numb ers add up to 1. Let f : S ! R count the numb er of shots; so f maps

the n-th elementof S to n. The exp ectation of f is

E f = nq p:

n=1

Using the formula

nx =

1 x

n=1

for jxj < 1 one nds that

1 p

= : E f =

1 q p

This answer is intuitively clear: if one needs k shots for a hit, on the average, then a hit

o ccurs once every k times, so p =1=k and k =1=p. 5

P A , One can also obtain this result by using Exercise 19 to write E f =

k =1

where A is the event that one sho ots at least k times, which happ ens with probability

k 1

q .

we see that a fair coin has to b e tossed twice, on the average, b efore the With p =

rst head comes up.

5. Indep endence of random variables.

Let f , g : S ! R be two random variables. We call f and g indep endent if for each

r 2 f S and each r 2 g S one has

1 2

P fx 2 S : f x=r and g x=r g=P f = r P g = r :

1 2 1 2

In other words, the events f x=r and g x=r are indep endent, for each r 2 f S and

1 2 1

each r 2 g S . It means that the value of the rst random variable gives no information

ab out the value of the second random variable.

This notion generalizes the notion of indep endence of events that wesaw earlier:

twoevents are indep endent if and only if the two corresp onding random variables are

indep endent.

Example 9. Let S b e as in Example 1 rolling two dice. The rst die more precisely: the

random variable that gives the value of the rst die and the second die are indep endent.

The rst die and the total score are not indep endent.

If f , g are two random variables, then f g is also a random variable; namely,f g xis

de ned to b e f xg x, for all x 2 S . If in addition f and g are indep endent, then one has

E f g =E f E g :

Pro of:

X X

r r P f = r P g = r E f g = f xg xP x=

1 2 1 2

r ;r

x2S

1 2

X X

= r P f = r r P g = r = E f E g :

1 1 2 2

r r

1 2 6

6. Standard deviation and variance.

Example 10. Let S = f1; 2; 3; 4; 5; 6gf1; 2; 3; 4; 5; 6g, with eachi; j 2 S having

probability , as in Example 1. Let the random variables f , t: S ! R b e de ned by

f i; j =2i twice the value of the rst die and t i; j = i + j the total score. Both

random variables have exp ectation 7. However, the value of f is usually much further

removed from the exp ectation 7 than the value of t. To make this quantitative, one

usually considers the standard deviation.

Let, generally, S b e a sample space, with probability function P . Let f : S ! R be a

random variable. The standard deviation of f is de ned by

1=2

P x : f x E f f =

x2S

, then Equivalently, if the random variable h: S ! R is de ned by hx= f x E f

E h: f =

So this is the square ro ot of the exp ectation of the square of the deviation of f from its

exp ected value. At rst sight this de nition seems unnecessarily complicated: why not

simply take the exp ected value of the deviation itself ? This can b e done, and it yields

jf x E f jP x the mean deviation of f . However, it turns out that f =

x2S

f is far simpler to use than f , b ecause it avoids the non-di erentiable function j j.

The square of the standard deviation is called the variance of f , and denoted by V f :

V f = f x E f P x=E h:

x2S

This is the exp ectation of the squared deviation. A useful formula for the variance is:

2 2

V f =E f E f :

Pro of:

V f = f x E f P x

x2S

X X X

2 2

f xP x+E f P x f x P x 2E f =

x2S x2S x2S

2 2 2 2

= E f 2E f E f +E f = E f E f : 7

Example 10 continued. We calculate the variance of the function f .Wehave

364 1

2 2 2 2 2 2 2

2 +4 +6 +8 +10 +12 = =60 E f = ;

6 6

and E f = 7, so

2 2 2

7 =11 ; f = 11 =3:416: V f =60

3 3 3

We can calculate the variance of t in a similar straightforward way, but it is easier to apply

another formula.

Let, generally, S b e a sample space with probability function P , and let f , g : S ! R be

two indep endent random variables. Then wehave

V f + g = V f +V g :

The pro of uses the relation E fg=E f E g , whichwe proved earlier for indep endent

random variables:

2 2

V f + g = E f + g E f + g

2 2 2 2

= E f +2E fg+E g E f 2E f E g E g

2 2 2 2

= E f E f + E g E g = V f +V g :

Example 10 continued. We use this formula to calculate the variance of t. We

write t = t + t , where t is the value of the rst die so t i; j = i and t is the

1 2 1 1 2

value of the second die. From t = f and the calculation we did for f we see that

1 35

V t = V f = . Since t and t are indep endent, it follows that

1 1 2

2 12

5 5

5 V t=V t +V t =5 ; t= =2:415:

1 2

6 6

Example 11. Let p 2 R,0

5 so P 1 = p, P 0 = q . Wesaw in Example 5 that the inclusion map f : S ! R has

exp ectation p.Wehave f = f ,so

2 2 2

V f =E f E f = p p = pq ; f = pq :

k nk

q .To calculate V f , Example 12. Let S , f b e as in Example 6; so P f = k = p

we write f = f + f + + f .Wenowhave the formula

1 2 n

V f =V f +V f + + V f ;

1 2 n 8

which is generally valid provided that anytwoofthe f are indep endent the pro of is

analogous to what we did ab ove for the sum of two indep endent random variables. By

Example 11, each f has variance V f =pq ,so

i i

V f =npq ; f = npq :

These formulas apply to any binomial random variable f .Ifwe take p = , then we see

that the standard deviation of the size of a random subset of a set of n elements equals

7. Chebyshev's inequality.

Chebyshev's inequality expresses that it is not likely that a random variable assumes a

value that is far from its exp ected value. The precise formulation is as follows.

Let S b e a sample space with probability function P , and let f : S ! R b e a random

variable. Then for any p ositive real number r wehave

V f

P jf x E f jr :

Here the left hand side is the probability of the event A = fx 2 S : jf x E f jr g,as

usual.

Pro of. Let A b e as just de ned. Wehave

P x V f = f x E f

x2S

X X

2 2

P x: P x+ f x E f = f x E f

x2S A x2A

The second sum is obviously 0. Each term in the rst is r P x, b ecause of the

de nition of A. Therefore

2 2

V f r P x=r P A:

x2A

This implies Chebyshev's inequality. 9

8. Generating functions.

Let S b e a sample space, and f : S ! R a random variable with the prop erty that the

image of f is contained in the set of non-negativeintegers. The generating function g of f

is de ned by

g z = P f = k z :

k =0

Here z is a real variable with jz j1. From

P f = k =1

k =0

it follows that the series converges, and that g 1=1.

The generating function can b e used to calculate the exp ectation E f and the vari-

ance V f off . Namely, supp ose that the radius of convergence of the p ower series de ning

g is larger than 1. Then the derivative g = dg =dz is given by

k 1 0

kP f = k z ; g z =

k =1

for jz j1. Putting z =1 we nd the following formula for the exp ectation:

E f =g 1:

Similarly, one can prove

00 0 0 2

V f =g 1 + g 1 g 1 ;

see Exercise 22.

Example 13. Let p b e a real numb er, 0

sample space S = f0; 1g b e as in Example 6; so an element x 2 S that has exactly k 1's

k nk

has probability p q , where q =1 p. Let the random variable f also b e as in Example

6; so f x is the numb er of 1's in x.Wehave

k nk

P f = k = p q ;

so the generating function g is given by

k nk n

pz q =pz + q : g z =

k =0 10

Notice that indeed g 1=1.Wehave

0 n1 00 2 n2

g z =nppz + q ; g z =nn 1p pz + q

and therefore

E f =g 1 = np;

00 0 0 2 2 2 2

V f =g 1 + g 1 g 1 = nn 1p + np n p = npq ;

in agreement with what we found in Example 6 and Example 12.

Exercises.

1. Andrew, Beatrix and Charles are playing with a crown. If Andrew has the crown, he

throws it to Charles. If Beatrix has the crown, she throws it to Andrew or to Charles,

with equal probabilities. If Charles has the crown, he throws it to Andrew or to Beatrix,

with equal probabilities.

At the b eginning of the game the crown is given to one of Andrew, Beatrix and

Charles, with equal probabilities. What is the probability that, after the crown is thrown

once, Andrew has it? that Beatrix has it? that Charles has it?

2. Fromashued deckof52playing cards one card is drawn. Let A b e the event that

an ace is drawn, B the event that a diamond is drawn, and C the event that a red card

heart or diamond is drawn. Whichtwoof A, B , and C are indep endent? What if a joker

is added to the deck?

3. Let S b e a sample space, and let A, B S be two indep endentevents. Let A = S A

0 0 0 0

and B = S B . Prove that A and B are indep endentevents. Are A and B indep endent?

And A and A ?

4. A random numb er is drawn from f1; 2; 3; ::: ; 899; 900g; eachnumb er has probability

. Are the events \the numb er is divisible by 3" and \the numb er is divisible by5"

900

indep endent? What is the probability that the numb er has no factor in common with 15?

5. In a lottery 10,000 tickets are sold for $ 1 each. There are ve prizes: $ 5,000 once,

$ 700 once, $ 100 three times. What is the exp ected value of a ticket?

6. Four b oxes contain red balls, white balls and blue balls:

the rst b ox contains 13 red balls, 11 white balls and 6 blue balls;

the second b ox contains 13 blue balls, 11 red balls and 6 white balls; 11

the third b ox contains 13 white balls, 11 blue balls and 6 red balls;

the fourth b ox contains 10 red balls, 10 white balls and 10 blue balls.

One p erforms the following exp eriment: rst a b oxisdrawn at random giving equal

probability to all four b oxes. Next 12 balls are drawn at random from this b ox.

a Prove that the exp ected numb er of red balls that are drawn is equal to the exp ected

numb er of white balls that are drawn.

b What is the exp ected numb er of blue balls that are drawn?

Hint :avoid doing any calculations.

. De ne 7. Let S = f1; 2; 3; :::; 899; 900g, and let each elementof S have probability

900

the random variables f , g : S ! R by

f x = remainder of x up on division by3,

g x = remainder of x up on division by4.

a Prove that f and g are indep endent.

b Calculate E f + g , E fg and V f + g .

8. For this exercise, you have to know that Martians come in three sexes: male, emale

and female.

A Martian triple is going to have nine children. At each birth, the probability for

each sex is . What is the exp ected numb er of emales among the children? What is the

variance?

9. The Collected Novels of Mrs. Kip Vandegril have b een published in ten volumes. Some-

one checks out ve of these volumes from the lo cal library,chosen at random, without

knowing that Mrs. Vandegril wrote only venovels, each o ccupying twovolumes. Calcu-

late, for each i = 0, 1, 2, the probability that he can read i complete novels. What is the

exp ected numb er of complete novels he can read? And the variance?

10. Let p b e a real numb er, 0

sample space S = fh; mh; mmh; mmmh; : : :g, where the n-th element has probability

q p. Let further the random variable f : S ! R map the n-th variable to n. In Example

8wesaw that E f =1=p. Prove that V f =q=p .

11. Let f b e a binomial random variable, as in Example 6. Calculate the probability that

the value assumed by f is even, and the probability that the value assumed by f is o dd. 12

12. Fifty-twocho colate letters, two for each letter of the alphab et, are distributed at

random among fty-twochildren, one each. Can you calculate the exp ected number of

children whose rst name starts with the letter that they get, without knowing the names

of the children? And the variance?

13. Prove that at least 75 of all subsets of a set of 100 elements have cardinalities lying

between 41 and 59 inclusive.

14. Let S b e a sample space, and f : S ! R a random variable. De ne the random variable

2 2

g : S ! R by g x=jf x E f j. Prove that V g = f f , and that f f .

Under which conditions is f = f ?

15. Can the equality sign hold in Chebyshev's inequality?

16. Let S b e a sample space, A S an event, and f : S ! R the corresp onding random

variable; so f x=1forx 2 A and f x = 0 for x=2 A. Let the variance V AofA be

0 0

de ned as the variance of f . Prove that V A=P A P A , where A = S A.

17. Twenty distinct numb ers are arranged in a list in a random order, such that all 20!

orderings are equally likely. Going down the list, one marks every numb er that is larger

than all earlier numb ers on the list. In particular, the rst numb er of the list is marked.

Prove that, for each i =1; 2; ::: ; 20, the probability that the i-th numb er on the list is

marked is 1=i. What is the exp ectation of the numb er of marked numb ers on the list?

18. Let the sample space b e as in Example 7.

a What is more likely: that no man gets his own hat back, or that exactly one man gets

his own hat back? Do es your answer change if the numb er of men is di erent from

10?

b What is the probability that exactly nine men get their own hats back?

c Let A b e the event that the i-th man and the j -th man get their own hats back.

What is the probabilityof A , for i 6= j ? Is the event that the i-th man gets his own

hat back indep endent from the event that the j -th man gets his own hat back?

d In Example 7 wesaw that the exp ected numb er of men that get their own hats back

equals 1. Prove that the standard deviation of the numb er of men that get their own

hats back is also 1.

19. Let f b e a random variable assuming only non-negativeinteger values, and denote by

A the event that f x k . Prove that E f = P A .

k k

k =1 13

20. A street p erformer hides your coin under one of three cups. You want to get it back,

but you don't have the faintest idea under which cup it is. The rules of the game require

that, rst, you p oint at one of the cups; next, to help you, the p erformer lifts one of

the other two cups, showing that there is no coin under it; nally,you must determine

your choice: stick to the cup you chose initially,orchange to the last one. What is your

strategy?

21. Klaas owes Karel an amountof x Dutch guilders, where 0 x 1; but all he has is

one Dutch guilder, and neither has change. They decide to settle the debt by playing the

following game.

i First they make sure that 0 x . If this is not the case, it is achieved by letting

the coin change hands, so that the debt x is changed into 1 x owed by the other

p erson. If now x = 0 then the game is over, else they pass to ii.

ii The p erson having the coin ips it. If Queen Beatrix comes up, he p o ckets the coin,

and the game is over. If the other side comes up, his debt is doubled i. e., x is replaced

by2x. Now the game is started all over again at i.

Prove that this is a fair game, in the sense that the exp ected amount of money that Karel

receives is x guilders, for the initial value of x.You may supp ose that the coin is fair; this

is guaranteed by Queen Beatrix.

Hint : write x in binary.

22. Let f b e a random variable with the prop erty that the image of f is contained in

the set of non-negativeintegers, with generating function g . Prove the formula V f =

00 0 0 2

g 1 + g 1 g 1 . You may assume that the radius of convergence of g is larger

than 1.

23. Let p b e a real numb er, 0

Exercise 10, the sample space S = fh; mh; mmh; mmmh; : : :g, where the n-th element

has probability q p. Let the random variable f : S ! R map the n-th variable to n.

What is the generating function of f ?Verify the result of Example 8 that E f =1=p,

and the result of Exercise 10 that V f =q=p .

24. Let S b e a sample space, and f , f : S ! R two indep endent random variables.

1 2

Assume that the images of f and f are contained in the set of non-negativeintegers, and

1 2

denote by g , g the generating functions of f , f . Let f b e the random variable f + f .

1 2 1 2 1 2

Prove that the generating function g of f is given by

g z =g z g z :

1 2 14

25. Let the sample space S b e as suggested in Exercise 17 so S = 20!, and denote, for

1 i 20, by A the event that the i-th numb er on the list is marked.

a Prove that the events A , A , ::: , A are indep endent in the sense that

1 2 20

\ Y

A = P A P

i i

i2I i2I

for every subset I of f1; 2; :::; 20g.

b Calculate the variance of the random variable that indicates how manynumb ers are

marked by the pro cedure describ ed in Exercise 17.

c Prove that the generating function of the random variable from b is given by

z + i:

20!

i=0

26. Eachbox of the cereal Flakey contains, inside the package, a small plastic letter. Only

the six letters A, E , F , K , L, Y o ccur, and with each purchase of a b ox they all have the

same probability .

a Prove that the exp ectation of the number of boxes of Flakey that one needs to buy

until one has two distinct letters equals 1 + .

b Prove that the exp ectation of the number of boxes that one needs to buy until one

1 1 1 1 1

+ + + + =14:7. has all six letters equals 6 1 +

2 3 4 5 6

c Calculate the generating function of the random variable from b.

27. Let the sample space S and the random variable f : S ! R b e as in Example 7 and

n n n

Exercise 18, but with n men instead of 10.

a Let 0 k n. Prove that

1 1

: P f = k =

k ! i!

i=0

b Denote the generating function of f by g . Prove that for anytwo real numb ers z , t

n n

with jtj < 1 one has

tz 1

: g z t =

1 t

n=0 15