1.3 Probability theory 1.3.1 Basics Probability theory starts with a non-definable notion of experiment, which has possible outcomes. The set of all possible outcomes is called the sample space and usually denoted by Ω. In all the problems dealing with probabilities the first step if to identify the sample space. Example 1.5. An experiment consists in tossing a coin three times, here is the sample space: Ω = fHHH;HHT;HTH;HTT;THH;THT;TTH;TTT g; 3 and an outcome is !i = (a1; a2; a3); ai 2 fH; T g. Obviously, jΩj = 2 = 8. Example 1.6. Consider an experiment of choosing a graph randomly from the set of all graphs on 4 vertices and with 3 edges. The sample space here is G(n; m), where n = 4 and m = 3. All the elementary events are presented in the figure below. Note that we consider that the vertices of the graph are distinguishable (labeled). How many non-isomorphic graphs are here? Example 1.7. If you are asked to specify the probability space for the experiment that consists in picking n balls our of an urn containing m balls, you should ask the follow-up questions whether we care about the order of the balls or not, and which sampling procedure is used. The answer crucially depends on these important details. Start with an urn that contains M distinguishable balls, and perform sampling with replace- ment (i.e., after each drawing we return the ball into the urn). If an outcome of our experiment 2 4 2 4 2 4 2 4 2 4 1 3 1 3 1 3 1 3 1 3 2 4 2 4 2 4 2 4 2 4 1 3 1 3 1 3 1 3 1 3 2 4 2 4 2 4 2 4 2 4 1 3 1 3 1 3 1 3 1 3 2 4 2 4 2 4 2 4 2 4 1 3 1 3 1 3 1 3 1 3 ( 4 ) G jGj (2) Figure 1.6: The sample space for (4; 3), here = 3 = 20 18 is a sample of n balls, then what is jΩj? To answer this question we need to distinguish between ordered and unordered samples, whether we care about the exact order the balls appear or not. For the ordered sample we have that !i = (a1; : : : ; an), where each aj can take any values out of M. Hence here jΩj = M n. If, however we consider the unordered samples: Ω = f! : ! = fa1; : : : ; ang; ai = 1;:::;Mg; then the answer is not straightforward to come( up with) (except that it should be smaller than n j j M+n−1 M ). Let us prove that N(M; n) := Ω = n in this case. I use induction. First, note that for k ≤ M ( ) k N(k; 1) = k = : 1 ( ) k+n−1 ≤ Now assume that N(k; n) = k for k M, I need to show that this formula continues to hold when n is replaced with n + 1. For an unordered sample we can always assume that it is arranged as a1 ≤ a2 ≤ ::: ≤ an ≤ an+1: We have that the number of the unordered samples with a1 = 1 is N(M; n), with a1 = 2 is N(M − 1; n), etc, with a1 = M is N(1; n) = 1. Hence, N(M; n + 1) = N(M; n) + N(M − 1; n) + :::; +N(1; n) = ( ) ( ) ( ) M + n − 1 M + n − 2 n = + + ::: + = (( n ) ( n )) (( n ) ( )) M + n M + n − 1 M + n − 1 M + n − 2 = − + − + ((n + 1 ) ( n +)) 1 n + 1 n + 1 n + 1 n ::: + − = ( n +) 1 n + 1 M + n = : n + 1 Here we used the fact that ( ) ( ) ( ) k + 1 k k = + : l l l − 1 If we need to perform the sampling without replacement, we need n ≤ M. For the ordered samples one has (M)n := jΩj = M(M − 1) ::: (M − n + 1): Note that if n = M, we obtain here permutations of the set of balls, the total number of which is M! := 1 · 2 · ::: · M (and of course 0! := 1). For the unordered samples we do not bother about the order of the balls in our sample, hence here ( ) (M) M(M − 1) ::: (M − n + 1) M! M jΩj = n = = = : n! n! n!(M − n)! n 19 Example 1.8. Distribution of n objects in M cells (think about distribution of n particles among M energy states). Assume that we assign numbers 1; 2;:::;M to the cells and 1; 2; : : : ; n to the balls. If all the balls are distinguishable, then putting n balls into M cells amounts to having an ordered sample (a1; : : : ; an), where ai is the number of the cell into which the ith ball was put. However, if we do not distinguish the balls, then an outcome in an unordered sample fa1; : : : ; ang, where ai is the number of the cell into which an object is put at the step i. Hence we get a bijection ordered samples $ distinguishable objects unordered samples $ indistinguishable objects In an analogous way we get sampling with replacement $ a cell may get any number of balls sampling without replacement $ a cell can get only one ball per cell Hence we calculated the sizes of the sample spaces in four cases for out example! Problem 1.16. Give a combinatorial prove for the number of outcomes in the case of putting n indistinguishable balls among M cells such that any cell may contain any number of balls. The next important thing to specify is the set of events F. An event A 2 F is a subset of Ω, for our case finite Ω F is usually taken as the power set 2Ω, i.e., the set of all subsets of Ω. The events are sets, and we can do usual set operation with them (taking the complement, union, intersection, difference). In the jargon of probability theory, if A; B 2 F, the event A \ B reads \both A and B occurred", the event A [ B reads \either A or B or both occurred", the event A n B reads \event A occurred and B did not", the event \A := Ω n A" means \not A occurred". For example the event A for the graph to be not connected in case G(4; 3) consists of four outcomes (see the figure above). Given the sample space and the set of events, finally to specify the probability space one needs the probability measure P: F! R, for which the following axioms hold: 1. P fAg 2 [0; 1] for any A 2 F and P fΩg = 1. 2. If fAi 2 F : i 2 Ig is a countable set of pairwise disjoint events, then ( ) [ X P Ai = P fAig : i2I i2I The triple (Ω; F; P) is called the probability space. When we dealt with the Ramsey numbers, we used the classical probability model that assign to each event A the probability jAj P fAg = ; jΩj and each outcome !i 2 Ω has the probability 1 P f! g = : i jΩj 20 The classical probability model is also called the uniform probability space on Ω. Now we can return to Example 1.8 and ask which probability space one should pick to solve one or another problem. This question is not as obvious as might seem from the first view. For example, a simple question would be what is more probable: to observe 11 or 12 points if two dices are tossed. To answer this question we first must decide whether the outcomes (5; 6) and (6; 5) are considered different. If they are different (and we talk about ordered samples) then 2 P f11 points is observedg = ; 36 where jΩj = 36 and 1 P f12 points is observedg = : 36 However, if we think that (5; 6) and (6; 5) are the same outcome, then f g ( 1 ) f g P 11 points is observed = 6+2−1 = P 12 points is observed : 2 Those who played dice probably know that 11 points are observed somewhat more frequently than 12, hence the first approach is what we need to use not to contradict the nature. However, if we move, e.g, to the realm of physics particles, then other sample spaces have to be chosen. Consider the statistical physics problem to describe the (random) distribution of particles in some region, subdivided into smaller ones. It would be natural to assume that any configuration of particles has to have the probability M −n, where n is the number of particles and M is the number of subregions, in physics this is called Maxwell{Boltzmann statistics. Now we know from experiments that this statistics does not apply to any known type of particles! Actually, photons,( for) example, satisfy Bose{Einstein statistics, when the probability of any configuration M+n−1 −1 is n (i.e, the particles are indistinguishable and any subregion can accommodate more than one particle),( ) and protons obey Fermi{Dirac statistics, with the probability of any config- M −1 uration is n (the particles are indistinguishable and any subregion can accommodate only one particle). Note that the general problem of the probability theory is not to figure out how to assign probabilities to the outcomes, but, given the probabilities of outcomes, to present a framework to infer the probabilities of more complex events. In general, for our finite sample space Ω we can define the probability of event A as X P fAg = P f!ig ; !i2A and the axioms above follow from this definition.