M2S1 Lecture Notes

Home , Bernoulli distribution, Bernoulli trial, Binomial distribution, Central moment, Cumulant, Geometric distribution

M2S1 Lecture Notes

G. A. Young http://www2.imperial.ac.uk/ ayoung ∼ September 2011 ii Contents

1 DEFINITIONS, TERMINOLOGY, NOTATION 1 1.1 EVENTS AND THE SAMPLE SPACE ...... 1 1.1.1 OPERATIONS IN SET THEORY ...... 2 1.1.2 MUTUALLY EXCLUSIVE EVENTS AND PARTITIONS ...... 2 1.2 THE σ-FIELD ...... 3 1.3 THE PROBABILITY FUNCTION ...... 4 1.4 PROPERTIES OF P(.): THE AXIOMS OF PROBABILITY ...... 5 1.5 CONDITIONAL PROBABILITY ...... 6 1.6 THE THEOREM OF TOTAL PROBABILITY ...... 7 1.7 BAYES’ THEOREM ...... 7 1.8 COUNTING TECHNIQUES ...... 8 1.8.1 THE MULTIPLICATION PRINCIPLE ...... 8 1.8.2 SAMPLING FROM A FINITE POPULATION ...... 8 1.8.3 PERMUTATIONS AND COMBINATIONS ...... 9 1.8.4 PROBABILITY CALCULATIONS ...... 10

2 RANDOM VARIABLES & PROBABILITY DISTRIBUTIONS 13 2.1 RANDOM VARIABLES & PROBABILITY MODELS ...... 13 2.2 DISCRETE RANDOM VARIABLES ...... 14 2.2.1 PROPERTIES OF MASS FUNCTION fX ...... 14 2.2.2 CONNECTION BETWEEN FX AND fX ...... 15 2.2.3 PROPERTIES OF DISCRETE CDF FX ...... 15 2.3 CONTINUOUS RANDOM VARIABLES ...... 15 2.3.1 PROPERTIES OF CONTINUOUS FX AND fX ...... 16 2.4 EXPECTATIONS AND THEIR PROPERTIES ...... 19 2.5 INDICATOR VARIABLES ...... 20 2.6 TRANSFORMATIONS OF RANDOM VARIABLES ...... 21 2.6.1 GENERAL TRANSFORMATIONS ...... 21 2.6.2 1-1 TRANSFORMATIONS ...... 24 2.7 GENERATING FUNCTIONS ...... 26 2.7.1 MOMENT GENERATING FUNCTIONS ...... 26 2.7.2 KEY PROPERTIES OF MGFS ...... 26 2.7.3 OTHER GENERATING FUNCTIONS ...... 28 2.8 JOINT PROBABILITY DISTRIBUTIONS ...... 29 2.8.1 THE CHAIN RULE FOR RANDOM VARIABLES ...... 33 2.8.2 CONDITIONAL EXPECTATION AND ITERATED EXPECTATION . . . 33 2.9 MULTIVARIATE TRANSFORMATIONS ...... 34 2.10 MULTIVARIATE EXPECTATIONS AND COVARIANCE ...... 37 2.10.1 EXPECTATION WITH RESPECT TO JOINT DISTRIBUTIONS ...... 37 2.10.2 COVARIANCE AND CORRELATION ...... 37 2.10.3 JOINT MOMENT GENERATING FUNCTION ...... 39 2.10.4 FURTHER RESULTS ON INDEPENDENCE ...... 39 2.11 ORDER STATISTICS ...... 40

iii iv CONTENTS

3 DISCRETE PROBABILITY DISTRIBUTIONS 41

4 CONTINUOUS PROBABILITY DISTRIBUTIONS 45

5 MULTIVARIATE PROBABILITY DISTRIBUTIONS 51 5.1 THE MULTINOMIAL DISTRIBUTION ...... 51 5.2 THE DIRICHLET DISTRIBUTION ...... 51 5.3 THE MULTIVARIATE NORMAL DISTRIBUTION ...... 52

6 PROBABILITY RESULTS & LIMIT THEOREMS 55 6.1 BOUNDS ON PROBABILITIES BASED ON MOMENTS ...... 55 6.2 THE CENTRAL LIMIT THEOREM ...... 56 6.3 MODES OF STOCHASTIC CONVERGENCE ...... 57 6.3.1 CONVERGENCE IN DISTRIBUTION ...... 57 6.3.2 CONVERGENCE IN PROBABILITY ...... 58 6.3.3 CONVERGENCE IN QUADRATIC MEAN ...... 59

7 STATISTICAL ANALYSIS 61 7.1 STATISTICAL SUMMARIES ...... 61 7.2 SAMPLING DISTRIBUTIONS ...... 61 7.3 HYPOTHESIS TESTING ...... 63 7.3.1 TESTING FOR NORMAL SAMPLES - THE Z-TEST ...... 63 7.3.2 HYPOTHESIS TESTING TERMINOLOGY ...... 64 7.3.3 THE t-TEST ...... 65 7.3.4 TEST FOR σ ...... 65 7.3.5 TWO SAMPLE TESTS ...... 66 7.4 POINT ESTIMATION ...... 68 7.4.1 ESTIMATION TECHNIQUES I: METHOD OF MOMENTS ...... 68 7.4.2 ESTIMATION TECHNIQUES II: MAXIMUM LIKELIHOOD ...... 69 7.5 INTERVAL ESTIMATION ...... 70 7.5.1 PIVOTAL QUANTITY ...... 70 7.5.2 INVERTING A TEST STATISTIC ...... 71 CHAPTER 1 DEFINITIONS, TERMINOLOGY, NOTATION

1.1 EVENTS AND THE SAMPLE SPACE

Definition 1.1.1 An experiment is a one-off or repeatable process or procedure for which (a) there is a well-defined set of possible outcomes (b) the actual outcome is not known with certainty.

Definition 1.1.2 A sample outcome, ω, is precisely one of the possible outcomes of an experiment.

Definition 1.1.3 The sample space, Ω, of an experiment is the set of all possible outcomes.

NOTE : Ω is a set in the mathematical sense, so set theory notation can be used. For example, if the sample outcomes are denoted ω1, ..., ωk, say, then

Ω = ω1, ..., ωk = ωi : i = 1, ..., k , { } { } and ωi Ω for i = 1, ..., k. ∈ The sample space of an experiment can be

- a FINITE list of sample outcomes, ω1, ..., ωk { } - a (countably) INFINITE list of sample outcomes, ω1, ω2, ... { } - an INTERVAL or REGION of a real space, ω : ω A Rd ∈ ⊆ Definition 1.1.4 An event, E, is a designated collection of sample outcomes. Event E occurs if the actual outcome of the experiment is one of this collection. An event is, therefore, a subset of the sample space Ω.

Special Cases of Events The event corresponding to the collection of all sample outcomes is Ω. The event corresponding to a collection of none of the sample outcomes is denoted . ∅ i.e. The sets and Ω are also events, termed the impossible and the certain event respectively, and for any event∅ E, E Ω. ⊆

1 2 CHAPTER 1. DEFINITIONS, TERMINOLOGY, NOTATION

1.1.1 OPERATIONS IN SET THEORY

Since events are subsets of Ω, set theory operations are used to manipulate events in probability theory. Consider events E,F Ω. Then we can reasonably concern ourselves also with events obtained from the three basic set⊆ operations:

UNION E F “E or F or both occur” INTERSECTION E ∪ F “both E and F occur” ∩ COMPLEMENT E0 “E does not occur”

Properties of Union/Intersection operators

Consider events E,F,G Ω. ⊆ COMMUTATIVITY E F = F E E ∪ F = F ∪ E ∩ ∩ ASSOCIATIVITY E (F G) = (E F ) G E ∪ (F ∪ G) = (E ∪ F ) ∪ G ∩ ∩ ∩ ∩ DISTRIBUTIVITY E (F G) = (E F ) (E G) E ∪ (F ∩ G) = (E ∪ F ) ∩ (E ∪ G) ∩ ∪ ∩ ∪ ∩

DE MORGAN’S LAWS (E F )0 = E0 F 0 ∪ ∩ (E F )0 = E0 F 0 ∩ ∪

1.1.2 MUTUALLY EXCLUSIVE EVENTS AND PARTITIONS Definition 1.1.5 Events E and F are mutually exclusive if E F = , that is, if events E and F cannot both occur. If the sets of sample outcomes represented∩ by E ∅and F are disjoint (have no common element), then E and F are mutually exclusive.

Definition 1.1.6 Events E1, ..., Ek Ω form a partition of event F Ω if ⊆ ⊆ (a) Ei Ej = for i = j, i, j = 1, ..., k k ∩ ∅ 6 (b) Ei = F , i=1 so thatS each element of the collection of sample outcomes corresponding to event F is in one and only one of the collections corresponding to events E1, ...Ek. 1.2. THE σ-FIELD 3

Figure 1.1: Partition of Ω

6 In Figure 1.1, we have Ω = Ei i=1 S

Figure 1.2: Partition of F Ω ⊂

6 In Figure 1.2, we have F = (F Ei), but, for example, F E6 = . i=1 ∩ ∩ ∅ S 1.2 THE σ-FIELD

Events are subsets of Ω, but need all subsets of Ω be events? The answer is negative. But it suffices to think of the collection of events as a subcollection of the set of all subsets of Ω. This subcollection should have the following properties: A

(a) if A, B then A B and A B ; ∈ A ∪ ∈ A ∩ ∈ A (b) if A then A ; ∈ A 0 ∈ A 4 CHAPTER 1. DEFINITIONS, TERMINOLOGY, NOTATION

(c) . ∅ ∈ A A collection of subsets of Ω which satisfies these three conditions is called a field. It follows A from the properties of a field that if A1,A2,...,Ak , then ∈ A k Ai . i ∈ A [=1 So, is closed under finite unions and hence under finite intersections also. To see this notethat A if A1,A2 , then ∈ A

A0 ,A0 = A0 A0 = A0 A0 0 = A1 A2 . 1 2 ∈ A ⇒ 1 ∪ 2 ∈ A ⇒ 1 ∪ 2 ∈ A ⇒ ∩ ∈ A This is fine when Ω is a finite set, but we require slightly more to deal with the commonsituation when Ω is infinite. We require the collection of events to be closed under the operation oftaking countable unions, not just finite unions.

Definition 1.2.1 A collection of subsets of Ω is called a σ field if it satisfies the following conditions: A −

(I) ; ∅ ∈ A (II) if A1,A2,... then ∞ Ai ; ∈ A i=1 ∈ A (III) if A then A . S ∈ A 0 ∈ A To recap, with any experiment we may associate a pair (Ω, ), where Ω is the set of all possible outcomes (or elementary events) and is a σ field of subsetsA of Ω, which contains all the events in whose occurrences we may be interested.A So,− from now on, to call a set A an event is equivalent to asserting that A belongs to the σ field in question. −

1.3 THE PROBABILITY FUNCTION

Definition 1.3.1 For an event E Ω, the probability that E occurs will be written P (E). ⊆ Interpretation: P (.) is a set-function that assigns “weight” to collections of possible outcomes of an experiment. There are many ways to think about precisely how this assignment is achieved;

CLASSICAL : “Consider equally likely sample outcomes ...” FREQUENTIST : “Consider long-run relative frequencies ...” SUBJECTIVE : “Consider personal degree of belief ...” or merely think of P (.) as a set-function.

Formally, we have the following definition.

Definition 1.3.2 A probability function P (.) on (Ω, ) is a function P : [0, 1] satisfying: A A → 1.4. PROPERTIES OF P(.): THE AXIOMS OF PROBABILITY 5

(a) P ( ) = 0,P (Ω) = 1; ∅

(b) if A1,A2,... is a collection of disjoint members of , so that Ai Aj = from all pairs i, j with i = j, then A ∩ ∅ 6 ∞ ∞ P Ai = P (Ai). i ! i [=1 X=1 The triple (Ω, ,P (.)), consisting of a set Ω, a σ-field of subsets of Ω and a probability function P (.) on (Ω, )A is called a probability space. A A

1.4 PROPERTIES OF P(.): THE AXIOMS OF PROBABILITY

For events E,F Ω ⊆ 1. P (E ) = 1 P (E). 0 − 2. If E F , then P (E) P (F ). ⊆ ≤ 3. In general, P (E F ) = P (E) + P (F ) P (E F ). ∪ − ∩ 4. P (E F ) = P (E) P (E F ) ∩ 0 − ∩ 5. P (E F ) P (E) + P (F ). ∪ ≤ 6. P (E F ) P (E) + P (F ) 1. ∩ ≥ − NOTE : The general addition rule 3 for probabilities and Boole’s Inequalities 5 and 6 extend to more than two events. Let E1, ..., En be events in Ω. Then

n n n P Ei = P (Ei) P (Ei Ej) + P (Ei Ej Ek) ... + ( 1) P Ei i ! i − i

i 1 − 0 Fi = Ei Ek ∩ ! k[=1 n n for i = 2, 3, ..., n. Then F1,F2, ...Fn are disjoint, and Ei = Fi, so i=1 i=1 n n S n S P Ei = P Fi = P (Fi). i ! i ! i [=1 [=1 X=1 6 CHAPTER 1. DEFINITIONS, TERMINOLOGY, NOTATION

Now, by property 4 above

i 1 − P (Fi) = P (Ei) P Ei Ek , i = 2, 3, ..., n, − ∩ k=1 S i 1 − = P (Ei) P (Ei Ek) − ∩ k=1 S and the result follows by recursive expansion of the second term for i = 2, 3, ...n.

NOTE : We will often deal with both probabilities of single events, and also probabilities for intersection events. For convenience, and to reflect connections with distribution theory that will be presented in Chapter 2, we will use the following terminology; for events E and F

P (E) is the marginal probability of E

P (E F ) is the joint probability of E and F ∩

1.5 CONDITIONAL PROBABILITY

Definition 1.5.1 For events E,F Ω the conditional probability that F occurs given that E occurs is written P(F E), and is⊆ defined by | P (E F ) P (F E) = ∩ , | P (E) if P(E) > 0.

NOTE: P (E F ) = P (E)P (F E), and in general, for events E1, ..., Ek, ∩ | k P Ei = P (E1)P (E2 E1)P (E2 E1 E2)...P (Ek E1 E2 ... Ek 1). − i ! | | ∩ | ∩ ∩ ∩ \=1 This result is known as the CHAIN or MULTIPLICATION RULE.

Definition 1.5.2 Events E and F are independent if

P (E F ) = P (E), so that P (E F ) = P (E)P (F ). | ∩

Extension : Events E1, ..., Ek are independent if, for every subset of events of size l k, indexed ≤ by i1, ..., il , say, { } l l P Ei = P (Ei ).  j  j j j \=1 Y=1   1.6. THE THEOREM OF TOTAL PROBABILITY 7

1.6 THE THEOREM OF TOTAL PROBABILITY

THEOREM

Let E1, ..., Ek be a (finite) partition of Ω, and let F Ω. Then ⊆ k P (F ) = P (F Ei)P (Ei). i | X=1 PROOF

E1, ..., Ek form a partition of Ω, and F Ω, so ⊆

F = (F E1) ... (F Ek) ∩ ∪ ∪ ∩ k k = P (F ) = P (F Ei) = P (F Ei)P (Ei), ⇒ i=1 ∩ i=1 | writing F as a disjoint union and usingP the definition Pof a probability function.

Extension: The theorem still holds if E1,E2, ... is a (countably) infinite a partition of Ω, and F Ω, so that ⊆ ∞ ∞ P (F ) = P (F Ei) = P (F Ei)P (Ei), i ∩ i | X=1 X=1 if P(Ei) > 0 for all i.

1.7 BAYES’ THEOREM

THEOREM Suppose E,F Ω, with P (E),P (F ) > 0. Then ⊆ P (F E)P (E) P (E F ) = | . | P (F ) PROOF

P (E F )P (F ) = P (E F ) = P (F E)P (E), so P (E F )P (F ) = P (F E)P (E). | ∩ | | |

Extension: If E1, ..., Ek are disjoint, with P (Ei) > 0 for i = 1, ..., k, and form a partition of F Ω, then ⊆ P (F Ei)P (Ei) P (Ei F ) = | . | k P (F Ej)P (Ej) j | X=1

NOTE: in general, P (E F ) = P (F E). | 6 | 8 CHAPTER 1. DEFINITIONS, TERMINOLOGY, NOTATION

1.8 COUNTING TECHNIQUES

This section is included for completeness, but is not examinable.

Suppose that an experiment has N equally likely sample outcomes. If event E corresponds to a collection of sample outcomes of size n(E), then

n(E) P (E) = , N so it is necessary to be able to evaluate n(E) and N in practice.

1.8.1 THE MULTIPLICATION PRINCIPLE

If operations labelled 1, ..., r can be carried out in n1, ..., nr ways respectively, then there are

r ni = n1...nr i Y=1 ways of carrying out the r operations in total.

Example 1.1 If each of r trials of an experiment has N possible outcomes, then there are N r possible sequences of outcomes in total. For example: (i) If a multiple choice exam has 20 questions, each of which has 5 possible answers, then there are 520 different ways of completing the exam. (ii) There are 2m subsets of m elements (as each element is either in the subset, or not in the subset, which is equivalent to m trials each with two outcomes).

1.8.2 SAMPLING FROM A FINITE POPULATION Consider a collection of N items, and a sequence of operations labelled 1, ..., r such that the ith operation involves selecting one of the items remaining after the first i 1 operations have been − carried out. Let ni denote the number of ways of carrying out the ith operation, for i = 1, ..., r. Then there are two distinct cases; (a) Sampling with replacement : an item is returned to the collection after selection. Then r ni = N for all i, and there are N ways of carrying out the r operations. (b) Sampling without replacement : an item is not returned to the collection after selected. Then ni = N i + 1, and there are N(N 1)...(N r + 1) ways of carrying out the r operations. e.g. Consider− selecting 5 cards from 52. Then− −

(a) leads to 525 possible selections, whereas

(b) leads to 52.51.50.49.48 possible selections.

NOTE : The order in which the operations are carried out may be important e.g. in a raffle with three prizes and 100 tickets, thedraw 45, 19, 76 is different from 19, 76, 45 . { } { } 1.8. COUNTING TECHNIQUES 9

NOTE : The items may be distinct (unique in the collection), or indistinct (of a unique type in the collection, but not unique individually). e.g. The numbered balls in the National Lottery, or individual playing cards, are distinct. However when balls in the lottery are regarded as “WINNING” or “NOT WINNING”, or playing cards are regarded in terms of their suit only, they are indistinct.

1.8.3 PERMUTATIONS AND COMBINATIONS Definition 1.8.1 A permutation is an ordered arrangement of a set of items. A combination is an unordered arrangement of a set of items.

RESULT 1 The number of permutations of n distinct items is n! = n(n 1)...1. − RESULT 2 The number of permutations of r from n distinct items is n! P n = = n(n 1)...(n r + 1) (by the Multiplication Principle). r (n r)! − − −

If the order in which items are selected is not important, then

RESULT 3 The number of combinations of r from n distinct items is the binomial coefficient n n! Cn = = (as P n = r!Cn). r r r!(n r)! r r − -recall the Binomial Theorem, namely

n n (a + b)n = aibn i. i − i X=0 Then the number of subsets of m items can be calculated as follows; for each 0 j m, choose a subset of j items from m. Then ≤ ≤

m m Total number of subsets = = (1 + 1)m = 2m. j j X=0

If the items are indistinct, but each is of a unique type, say Type I, ..., Type κ say, (the so-called Urn Model) then

RESULT 4 The number of distinguishable permutations of n indistinct objects, comprising ni items of type i for i = 1, ..., κ is n! . n1!n2!...nκ!

Special Case : if κ = 2, then the number of distinguishable permutations of the n1 objects of type I, and n2 = n n1 objects of type II is − n n! Cn2 = . n1!(n n1)! − 10 CHAPTER 1. DEFINITIONS, TERMINOLOGY, NOTATION

n RESULT 5 There are Cr ways of partitioning n distinct items into two “cells”, with r in one cell and n r in the other. −

1.8.4 PROBABILITY CALCULATIONS Recall that if an experiment has N equally likely sample outcomes, and event E corresponds to a collection of sample outcomes of size n(E), then n(E) P (E) = . N

Example 1.2 A True/False exam has 20 questions. Let E = “16 answers correct at random”. Then

20 Number of ways of getting 16 out of 20 correct 16 P (E) = = = 0.0046. Total number of ways of answering 20 questions 220

Example 1.3 Sampling without replacement. Consider an Urn Model with 10 Type I objects and 20 Type II objects, and an experiment involving sampling five objects without replacement. Let E=“precisely 2 Type I objects selected” We need to calculate N and n(E) in order to calculate P(E). In this case N is the number of ways of choosing 5 from 30 items, and hence 30 N = . 5 To calculate n(E), we think of E occurring by first choosing 2 Type I objects from 10, and then choosing 3 Type II objects from 20, and hence, by the multiplication rule, 10 20 n(E) = . 2 3 Therefore 10 20 2 3 P (E) = = 0.360. 30 5 This result can be checked using a conditional probability argument; consider event F E, where F = “sequence of objects 11222 obtained”. Then ⊆

5 F = Fij i=1 T where Fij = “type j object obtained on draw i” i = 1, ..., 5, j = 1, 2. Then 10 9 20 19 18 P (F ) = P (F11)P (F21 F11)...P (F52 F11,F21,F32,F42) = . | | 30 29 28 27 26 1.8. COUNTING TECHNIQUES 11

Now consider event G where G = “sequence of objects 12122 obtained”. Then 10 20 9 19 18 P (G) = , 30 29 28 27 26 i.e. P (G) = P (F ). In fact, any sequence containing two Type I and three Type II objects has this 5 probability, and there are such sequences. Thus, as all such sequences are mutually exclusive, 2 10 20 5 10 9 20 19 18 2 3 P (E) = = 2 30 29 28 27 26 30 5 as before.

Example 1.4 Sampling with replacement. Consider an Urn Model with 10 Type I objects and 20 Type II objects, and an experiment involving sampling five objects with replacement. Let E = “precisely 2 Type I objects selected”. Again, we need to calculate N and n(E) in order to calculate P(E). In this case N is the number of ways of choosing 5 from 30 items with replacement, and hence N = 305. To calculate n(E), we think of E occurring by first choosing 2 Type I objects from 10, and 3 Type II objects from 20 in any order. Consider such sequences of selection Sequence Number of ways

11222 10.10.20.20.20 12122 10.20.10.20.20 .. etc., and thus a sequence with 2 Type I objects and 3 Type II objects can be obtained in 102203 5 ways. As before there are such sequences, and thus 2 5 102203 2 P (E) = = 0.329. 305 Again, this result can be verified using a conditional probability argument; consider event F E, where F = “sequence of objects 11222 obtained”. Then ⊆

10 2 20 3 P (F ) = 30 30 as the results of the draws are independent. This result is true for any sequence containing two 5 Type I and three Type II objects, and there are such sequences that are mutually exclusive, 2 so 5 10 2 20 3 P (E) = . 2 30 30 12 CHAPTER 1. DEFINITIONS, TERMINOLOGY, NOTATION CHAPTER 2 RANDOM VARIABLES & PROBABILITY DISTRIBUTIONS

This chapter contains the introduction of random variables as a technical device to enable the general specification of probability distributions in one and many dimensions to be made. Thekey topics and techniques introduced in this chapter include the following:

EXPECTATION • TRANSFORMATION • STANDARDIZATION • GENERATING FUNCTIONS • JOINT MODELLING • MARGINALIZATION • MULTIVARIATE TRANSFORMATION • MULTIVARIATE EXPECTATION & COVARIANCE • SUMS OF VARIABLES • Of key importance is the moment generating function, which is a standard device for identification of probability distributions. Transformations are often used to transform a random variable or statistic of interest to one of simpler form, whose probability distribution is more convenient to work with. Standardization is often a key part of such simplification.

2.1 RANDOM VARIABLES & PROBABILITY MODELS

We are not always interested in an experiment itself, but rather in some consequence of its random outcome. Such consequences, when real valued, may be thought of as functions which map Ω to R, and these functions are called random variables.

Definition 2.1.1 A random variable (r.v.) X is a function X :Ω R with the property that → ω Ω: X(ω) x for each x R. { ∈ ≤ } ∈ A ∈ The point is that we defined the probability function P (.) on the σ field , so if A(x) = ω Ω: X(ω) x , we cannot discuss P (A(x)) unless A(x) belongs to . We− generallyA pay no attention{ ∈ to the technical≤ } condition in the definition, and just think ofA random variables as functions mapping Ω to R.

13 14 CHAPTER 2. RANDOM VARIABLES & PROBABILITY DISTRIBUTIONS

So, we regard a set B R as an event, associated with event A Ω if ⊆ ⊆ A = ω : X(ω) = x for some x B . { ∈ } A and B are events in different spaces, but are equivalent in the sense that P (X B) = P (A), ∈ where, formally, it is the latter quantity that is defined by the probability function. Attention switches to assigning the probability P (X B) for appropriate sets B R. ∈ ⊆ If Ω is a list of discrete elements Ω = ω1, ω2, ... , then the definition indicates that the events of { } interest will be of the form [X = b], or equivalently of the form [X b] for b R. For more general ≤ ∈ sample spaces, we will concentrate on events of the form [X b] for b R. ≤ ∈

2.2 DISCRETE RANDOM VARIABLES

Definition 2.2.1 A random variable X is discrete if the set of all possible values of X (that is, the range of the function represented by X), denoted X, is countable, that is

X = x1, x2, ..., xn [FINITE] or X = x1, x2, ... [INFINITE]. { } { }

Definition 2.2.2 PROBABILITY MASS FUNCTION

The function fX defined on X by

fX (x) = P [X = x], x X ∈ that assigns probability to each x X is the (discrete) probability mass function, or pmf. ∈ NOTE: For completeness, we define

fX (x) = 0, x / X, ∈ so that fX is defined for all x R Furthermore we will refer to X as the support of random ∈ variable X, that is, the set of x R such that fX (x) > 0. ∈

2.2.1 PROPERTIES OF MASS FUNCTION fX Elementary properties of the mass function are straightforward to establish using properties of the probability function. A function fX is a probability mass function for discrete random variable X with range X of the form x1, x2, ... if and only if { } (i) fX (xi) 0, (ii) fX (xi) = 1. ≥ These results follow as events [X = x1], [X = x2] etc. areXequivalent to events that partition Ω, that is, [X = xi] is equivalent to event Ai hence P [X = xi] = P (Ai), and the two parts of the theorem follow immediately.

Definition 2.2.3 DISCRETE CUMULATIVE DISTRIBUTION FUNCTION

The cumulative distribution function, or cdf, FX of a discrete r.v. X is defined by

FX (x) = P [X x], x R. ≤ ∈ 2.3. CONTINUOUS RANDOM VARIABLES 15

2.2.2 CONNECTION BETWEEN FX AND fX

Let X be a discrete random variable with range X = x1, x2, ... , where x1 < x2 < ..., and { } probability mass function fX and cdf FX . Then for any real value x, if x < x1, then FX (x) = 0, and for x x1, ≥

FX (x) = fX (xi) fX (xi) = FX (xi) FX (xi 1) i = 2, 3, ... ⇐⇒ − − xi x X≤ with, for completeness, fX (x1) = FX (x1) . These relationships follow as events of the form [X xi] ≤ can be represented as countable unions of the events Ai. The first result therefore follows from properties of the probability function. The second result follows immediately.

2.2.3 PROPERTIES OF DISCRETE CDF FX (i) In the limiting cases,

lim FX (x) = 0, lim FX (x) = 1. x x →−∞ →∞

(ii) FX is continuous from the right (but not continuous) on R that is, for x R, ∈ lim FX (x + h) = FX (x). h 0+ →

(iii) FX is non-decreasing, that is

a < b = FX (a) FX (b). ⇒ ≤ (iv) For a < b, P [a < X b] = FX (b) FX (a). ≤ −

The key idea is that the functions fX and/or FX can be used to describe the probability distribution of the random variable X. A graph of the function fX is non-zero only at the elements of X. A graph of the function FX is a step-function which takes the value zero at minus infinity, the value one at infinity, and is non-decreasing with points of discontinuity at the elements of X.

2.3 CONTINUOUS RANDOM VARIABLES

Definition 2.3.1 A random variable X is continuous if the function FX defined on R by

FX (x) = P [X x] ≤ for x R is a continuous function on R , that is, for x R, ∈ ∈ lim FX (x + h) = FX (x). h 0 → Definition 2.3.2 CONTINUOUS CUMULATIVE DISTRIBUTION FUNCTION The cumulative distribution function, or cdf, FX of a continuous r.v. X is defined by

FX (x) = P [X x], x R. ≤ ∈ 16 CHAPTER 2. RANDOM VARIABLES & PROBABILITY DISTRIBUTIONS

Definition 2.3.3 PROBABILITY DENSITY FUNCTION A random variable is absolutely continuous if the cumulative distribution function FX can be written x FX (x) = fX (t)dt Z−∞ for some function fX , termed the probability density function, or pdf, of X.

From now on when we speak of a continuous random variable, we will implicitly assume the absolutely continuous case, where a pdf exists.

2.3.1 PROPERTIES OF CONTINUOUS FX AND fX

By analogy with the discrete case, let X be the range of X, so that X = x : fX (x) > 0 . { }

(i) The pdf fX need not exist, but as indicated above, continuous r.v.’s where a pdf fX cannot be defined in this way will be ignored. The function fX can be defined piecewise on intervals of R . (ii) For the cdf of a continuous r.v.,

lim FX (x) = 0, lim FX (x) = 1. x x →−∞ →∞

(iii) Directly from the definition, at values of x where FX is differentiable, d fX (x) = FX (t) . dt { }t=x (iv) If X is continuous,

fX (x) = P [X = x] = lim [P (X x) P (X x h)] = lim [FX (x) FX (x h)] = 0. 6 h 0+ ≤ − ≤ − h 0+ − − → → (v) For a < b,

P [a < X b] = P [a X < b] = P [a X b] = P [a < X < b] = FX (b) FX (a). ≤ ≤ ≤ ≤ −

It follows that a function fX is a pdf for a continuous random variable X if and only if

∞ (i) fX (x) 0, (ii) fX (x)dx = 1. ≥ Z−∞ This result follows direct from definitions and properties of FX .

Example 2.1 Consider a coin tossing experiment where a fair coin is tossed repeatedly under identical experimental conditions, with the sequence of tosses independent, until a Head is obtained. For this experiment, the sample space, Ω is then the set of sequences ( H , TH , TTH , TTTH ...) with associated probabilities 1/2, 1/4, 1/8, 1/16, ... . { } { } { } { } Define discrete random variable X :Ω R, by X(ω) = x first H on toss x. Then −→ ⇐⇒ 1 x f (x) = P [X = x] = , x = 1, 2, 3, ... X 2 2.3. CONTINUOUS RANDOM VARIABLES 17 and zero otherwise. For x 1, let k(x) be the largest integer not greater than x, then ≥ k(x) 1 k(x) FX (x) = fX (xi) = fX (i) = 1 − 2 xi x i=1 X≤ X and FX (x) = 0 for x < 1.

Graphs of the probability mass function (left) and cumulative distribution function (right) are shown in Figure 2.1. Note that the mass function is only non-zero at points that are elements of X, and that the cdf is defined for all real values of x, but is only continuous from the right. FX is therefore a step-function.

PMF f(x) CDF F(x) 1.0 0.5 0.8 0.4 0.6 0.3 f(x) F(x) 0.4 0.2 0.1 0.2 0.0 0.0

0 2 4 6 8 10 0 2 4 6 8 10

x x

1 x 1 k(x) Figure 2.1: PMF fX (x) = , x = 1, 2,..., and CDF FX (x) = 1 . 2 − 2

Example 2.2 Consider an experiment to measure the length of time that an electrical component functions before failure. The sample space of outcomes of the experiment, Ω is R+, and if Ax is the event that the component functions for longer than x > 0 time units, suppose 2 that P(Ax) = exp x . − Define continuous random variable X :Ω R+, by X(ω) = x component fails at time x. Then, if x > 0, −→ ⇐⇒ 2 FX (x) = P [X x] = 1 P (Ax) = 1 exp x ≤ − − − 18 CHAPTER 2. RANDOM VARIABLES & PROBABILITY DISTRIBUTIONS and FX (x) = 0 if x 0. Hence if x > 0, ≤

d 2 fX (x) = FX (t) = 2x exp x , dt { }t=x − and zero otherwise.

Graphs of the probability density function (left) and cumulative distribution function (right) are shown in Figure 2.2. Note that both the pdf and cdf are defined for all real values of x, and that both are continuous functions.

PDF f(x) CDF F(x) 1.0 0.8 0.8 0.6 0.6 f(x) F(x) 0.4 0.4 0.2 0.2 0.0 0.0

0 1 2 3 4 5 0 1 2 3 4 5

x x

2 2 Figure 2.2: PDF fX (x) = 2x exp x , x > 0, and CDF FX (x) = 1 exp x , x > 0. {− } − {− }

Note that here x x FX (x) = fX (t)dt = fX (t)dt 0 Z−∞ Z as fX (x) = 0 for x 0, and also that ≤

∞ ∞ fX (x)dx = fX (x)dx = 1. 0 Z−∞ Z 2.4. EXPECTATIONS AND THEIR PROPERTIES 19

2.4 EXPECTATIONS AND THEIR PROPERTIES

Definition 2.4.1 For a discrete random variable X with range X and with probability mass function fX , the expectation or expected value of X with respect to fX is defined by

∞ EfX [X] = xfX (x) = xfX (x). x= x X X−∞ X∈ For a continuous random variable X with range X and pdf fX , the expectation or expected value of X with respect to fX is defined by

∞ EfX [X] = xfX (x)dx = xfX (x)dx. X Z−∞ Z

NOTE : The sum/integral may not be convergent, and hence the expected value may be infinite. It is important always to check that the integral is finite: a sufficient condition isthe absolute integrability of the summand/integrand, that is

x fX (x) < = xfX (x) = EfX [X] < , x | | ∞ ⇒ x ∞ X X or in the continuous case

∞ ∞ x fX (x)dx < = xfX (x)dx = Ef [X] < . | | ∞ ⇒ X ∞ Z−∞ Z−∞

Extension : Let g be a real-valued function whose domain includes X. Then

g(x)fX (x), if X is discrete,  x X E [g(X)] = X∈ fX    g(x)fX (x)dx, if X is continuous.  Z X   PROPERTIES OF EXPECTATIONS Let X be a random variable with mass function/pdf fX . Let g and h be real-valued functions whose domains include X, and let a and b be constants. Then

EfX [ag(X) + bh(X)] = aEfX [g(X)] + bEfX [h(X)], as (in the continuous case)

EfX [ag(X) + bh(X)] = [ag(x) + bh(x)]fX (x)dx Z X

= a g(x)fX (x)dx + b h(x)fX (x)dx Z X Z X

= aEfX [g(X)] + bEfX [h(X)]. 20 CHAPTER 2. RANDOM VARIABLES & PROBABILITY DISTRIBUTIONS

SPECIAL CASES :

(i) For a simple linear function

EfX [aX + b] = aEfX [X] + b.

2 (ii) Consider g(x) = (x EfX [X]) . Write μ =EfX [X] (a constant that does not depend on x). Then, expanding the− integrand

2 2 2 Ef [g(X)] = (x μ) fX (x)dx = x fX (x)dx 2μ xfX (x)dx + μ fX (x)dx X − − Z Z Z Z

2 2 2 2 2 = x fX (x)dx 2μ + μ = x fX (x)dx μ − − Z Z 2 2 = Ef [X ] Ef [X] . X − { X } Then 2 2 V arf [X] = Ef [X ] Ef [X] X X − { X }

is the variance of the distribution. Similarly, V arfX [X] is the standard deviation of the distribution. p (iii) Consider g(x) = xk for k = 1, 2, .... Then in the continuous case

k k EfX [g(X)] = EfX [X ] = x fX (x)dx, ZX k and EfX [X ] is the kth moment of the distribution. (iv) Consider g(x) = (x μ)k for k = 1, 2, .... Then − k k Ef [g(X)] = Ef [(X μ) ] = (x μ) fX (x)dx, X X − − ZX k and Ef [(X μ) ] is the kth central moment of the distribution. X − 2 (v) Consider g(x) = aX + b. Then V arfX [aX + b] = a V arfX [X],

2 V arf [g(X)] = Ef [(aX + b Ef [aX + b]) ] X X − X 2 = Ef [(aX + b aEf [X] b) ] X − X − 2 2 = Ef [(a (X Ef [X]) ] X − X 2 = a V arfX [X].

2.5 INDICATOR VARIABLES

A particular class of random variables called indicator variables are particularly useful. Let A be an event and let IA :Ω R be the indicator function of A, so that → 1, if ω A, I (ω) = A 0, if ω ∈ A . ∈ 0 2.6. TRANSFORMATIONS OF RANDOM VARIABLES 21

Then IA is a random variable taking values 1 and 0 with probabilities P (A) and P (A0) respectively. Also, IA has expectation P (A) and variance P (A) 1 P (A) . The usefulness lies in the fact that any discrete random variable X can be written{ as− a linear} combination of indicator random variables:

X = aiIAi , i X for some collection of events (Ai, i 1) and real numbers (ai, i 1). Sometimes we can obtain the expectation and variance of a random≥ variable X easily by expressing≥ it in this way, then using knowledge of the expectation and variance of the indicator variables IAi , rather than by direct calculation.

2.6 TRANSFORMATIONS OF RANDOM VARIABLES

2.6.1 GENERAL TRANSFORMATIONS

Consider a discrete/continuous r.v. X with range X and probability distribution described by mass/pdf fX , or cdf FX . Suppose g is a real-valued function defined on X. Then Y = g(X) is also an r.v. (Y is also a function from Ω to R). Denote the range of Y by Y. For A R, the event ⊆ [Y A] is an event in terms of the transformed variable Y . If fY is the mass/density function for Y ,∈ then fY (y),Y discrete,  y A P [Y A] = X∈ ∈    fY (y)dy, Y continuous. A  Z We wish to derive the probability distribution of random variable Y ; in order to do this, we first 1 consider the inverse transformation g− from Y to X defined for set A Y (and for y Y) by ⊆ ∈ 1 1 g− (A) = x X : g(x) A , g− (y) = x X : g(x) = y , { ∈ ∈ } { ∈ } 1 1 that is, g− (A) is the set of points in X that map into A, and g− (y) is the set of points in X that map to y, under transformation g. By construction, we have

1 P [Y A] = P [X g− (A)]. ∈ ∈ Then, for y R, we have ∈ fX (x),X discrete,

x Ay  X∈ FY (y) = P [Y y] = P [g(X) y] = ≤ ≤    fX (x)dx, X continuous, A  Z y  where Ay = x X : g(x) y . This result gives the “first principles” approach to computing the distribution{ ∈ of the new≤ variable.} The approach can be summarized as follows:

consider the range Y of the new variable; • consider the cdf FY (y). Step through the argument as follows • FY (y) = P [Y y] = P [g(X) y] = P [X Ay]. ≤ ≤ ∈ 22 CHAPTER 2. RANDOM VARIABLES & PROBABILITY DISTRIBUTIONS

Transformation: Y=sin(X) 1.0

y 0.5

0.0 x1 x2 Y = sin(X) -0.5 -1.0

0 1 2 3 4 5 6

X (radians)

Figure 2.3: Computation of Ay for Y = sin X.

Note that it is usually a good idea to start with the cdf, not the pmf or pdf.

Our main objective is therefore to identify the set

Ay = x X : g(x) y . { ∈ ≤ }

Example 2.3 Suppose that X is a continuous r.v. with range X (0, 2π) whose pdf fX is constant ≡ 1 f (x) = , 0 < x < 2π, X 2π and zero otherwise. This pdf has corresponding continuous cdf x F (x) = , 0 < x < 2π. X 2π

Consider the transformed r.v. Y = sin X. Then the range of Y , Y, is [ 1, 1], but the transformation is not 1-1. However, from first principles, we have −

FY (y) = P [Y y] = P [sin X y] . ≤ ≤

Now, by inspection of Figure 2.3, we can easily identify the required set Ay, y > 0 : it is the union of two disjoint intervals

1 1 Ay = [0, x1] [x2, 2π] = 0, sin− y π sin− y, 2π . ∪ ∪ − Hence 2.6. TRANSFORMATIONS OF RANDOM VARIABLES 23

Transformation: T=tan(X) 10 5 t 0 x1 x2 T = tan(X) -5 -10

0 1 2 3 4 5 6

X (radians)

Figure 2.4: Computation of Ay for T = tan X.

FY (y) = P [sin X y] = P [X x1] + P [X x2] = P [X x1] + 1 P [X < x2] ≤ ≤ ≥ { ≤ } { − }

1 1 1 1 1 1 1 = sin− y + 1 π sin− y = + sin− y, 2π − 2π − 2 π and hence, by differentiation, 1 1 fY (y) = . π 1 y2 − [A symmetry argument verifies this for y < 0.] p

Example 2.4 Consider transformed r.v. T = tan X. Then the range of T , T, is R, but the transformation is not 1-1. However, from first principles, we have, for t > 0,

FT (t) = P [T t] = P [tan X t] . ≤ ≤ Figure 2.4 helps identify the required set At: in this case it is the union of three disjoint intervals

π 3π 1 π 1 3π At = [0, x1] , x2 , 2π = 0, tan− t , π + tan− t , 2π , ∪ 2 ∪ 2 ∪ 2 ∪ 2 i i (note, for values of t < 0, the union will be of only two intervals, but the calculation proceeds identically). Therefore, π 3π FT (t) = P [tan X t] = P [X x1] + P < X x2 + P < X 2π ≤ ≤ 2 ≤ 2 ≤ h i

1 1 1 1 π 1 3π 1 1 1 = tan− t + π + tan− t + 2π = tan− t + , 2π 2π − 2 2π − 2 π 2 n o 24 CHAPTER 2. RANDOM VARIABLES & PROBABILITY DISTRIBUTIONS and hence, by differentiation, 1 1 f (t) = . T π 1 + t2

2.6.2 1-1 TRANSFORMATIONS

The mapping g(X), a function of X from X, is 1-1 and onto Y if for each y Y, there exists one ∈ and only one x X such that y = g(x). ∈ The following theorem gives the distribution for random variable Y = g(X) when g is 1-1.

Theorem 2.6.1 THE UNIVARIATE TRANSFORMATION THEOREM

Let X be a random variable with mass/density function fX and support X. Let g be a 1-1 function 1 from X onto Y with inverse g− . Then Y = g(X) is a random variable with support Y and Discrete Case : The mass function of random variable Y is given by

1 fY (y) = fX (g− (y)), y Y = y fY (y) > 0 , ∈ { | } 1 where x is the unique solution of y = g(x) (so that x = g− (y)). Continuous Case : The pdf of random variable Y is given by

1 d 1 fY (y) = fX (g− (y)) g− (t) , y Y = y fY (y) > 0 , dt t=y ∈ { | }

where y = g(x), provided that the derivative d g 1(t) dt − is continuous and non-zero on Y.

Proof. Discrete case : by direct calculation,

1 fY (y) = P [Y = y] = P [g(X) = y] = P [X = g− (y)] = fX (x)

1 where x = g (y), and hence fY (y) > 0 fX (x) > 0. − ⇐⇒ Continuous case : function g is either (I) a monotonic increasing, or (II) a monotonic decreasing function. Case (I): If g is increasing, then for x X and y Y, we have that ∈ ∈ 1 g(x) y x g− (y). ≤ ⇐⇒ ≤ Therefore, for y Y, ∈ 1 1 FY (y) = P [Y y] = P [g(X) y] = P [X g− (y)] = FX (g− (y)) ≤ ≤ ≤ 2.6. TRANSFORMATIONS OF RANDOM VARIABLES 25 and, by differentiation, because g is monotonic increasing,

d d d f (y) = f (g 1(y)) g 1(t) = f (g 1(y)) g 1(y) , as g 1(t) > 0. Y X − dt − t=y X − dt − t=y dt −

Case (II): If g is decreasing, then for x X and y Y we have ∈ ∈

1 g(x) y x g− (y) ≤ ⇐⇒ ≥ Therefore, for y Y, ∈ 1 1 FY (y) = P [Y y] = P [g(X) y] = P [X g− (y)] = 1 FX (g− (y)), ≤ ≤ ≥ − so

1 d 1 1 d 1 d 1 fY (y) = fX (g− (y)) g− (y) = fX (g− (y)) g− (t) as g− (t) < 0. − dt dt t=y dt

Definition 2.6.1 Suppose transformation g : X Y is 1-1, and is defined by g(x) = y for −→ x X. Then the Jacobian of the transformation, denoted J(y), is given by ∈ d J(y) = g 1(t) , dt − t=y 1 that is, the first derivative of g− evaluated at y = g(x). Note that the inverse transformation 1 g− : Y X has Jacobian 1/J(x). −→

NOTE :

(i) The Jacobian is precisely the same term that appears as a change of variable term in an integration.

(ii) In the Univariate Transformation Theorem, in the continuous case, we take the modulus of the Jacobian

(iii) To compute the expectation of Y = g(X), we now have two alternative methods of computation; we either compute the expectation of g(X) with respect to the distribution of X, or compute the distribution of Y , and then its expectation. It is straightforward to demonstrate that the two methods are equivalent, that is

EfX [g(X)] = EfY [Y ]

This result is sometimes known as the Law of the Unconscious Statistician. 26 CHAPTER 2. RANDOM VARIABLES & PROBABILITY DISTRIBUTIONS

IMPORTANT NOTE: Note that the apparently appealing “plug-in” approach that sets 1 fY (y) = fX g− (y) will almost always fail as the Jacobian term must be included. For example, if Y = eX so that X = log Y , then merely setting fY (y) = fX (log y) is insufficient, you must have 1 fY (y) = fX (log y) . × y

2.7 GENERATING FUNCTIONS

2.7.1 MOMENT GENERATING FUNCTIONS

Definition 2.7.1 For random variable X with mass/density function fX , the moment generating function, or mgf, of X, MX , is defined by

tX MX (t) = EfX [e ], if this expectation exists for all values of t ( h, h) for some h > 0, that is, ∈ − tx DISCRETE CASE MX (t) = e fX (x) X tx CONTINUOUS CASE MX (t) = e fX (x)dx Z where the sum/integral is over X.

NOTE : It can be shown that if X1 and X2 are random variables taking values on X with mass/density functions fX1 and fX2 , and mgfs MX1 and MX2 respectively, then

fX1 (x) fX2 (x), x X MX1 (t) MX2 (t), t ( h, h). ≡ ∈ ⇐⇒ ≡ ∈ − Hence there is a 1-1 correspondence between generating functions and distributions: this provides a key technique for identification of probability distributions.

2.7.2 KEY PROPERTIES OF MGFS (r) (i) If X is a discrete random variable, the rth derivative of MX evaluated at t, MX (t), is given by

r r (r) d d sx r tx MX (t) = MX (s) s=t = e fX (x) = x e fX (x) dsr { } dsr s=t nX o X and hence (r) r r MX (0) = x fX (x) = EfX [X ]. X 2.7. GENERATING FUNCTIONS 27

If X is a continuous random variable, the rth derivative of MX is given by dr M (r)(t) = esxf (x)dx = xretxf (x)dx X dsr X X Z s=t Z and hence (r) r r MX (0) = x fX (x)dx = EfX [X ]. Z

(ii) If X is a discrete random variable, then

∞ (tx)r M (t) = etxf (x) = f (x) X X r! X (r=0 ) X X X ∞ tr ∞ tr = 1 + xrf (x) = 1 + E [Xr]. r! X r! fX r=1 r=1 X nX o X The identical result holds for the continuous case.

(iii) From the general result for expectations of functions of random variables,

tY t(aX+b) t(aX+b) bt atX bt Ef [e ] Ef [e ] = MY (t) = Ef [e ] = e Ef [e ] = e MX (at). Y ≡ X ⇒ X X Therefore, if bt Y = aX + b, MY (t) = e MX (at)

Theorem 2.7.1 Let X1, ..., Xk be independent random variables with mgfs MX1 , ..., MXk respectively. Then if the random variable Y is defined by Y = X1 + ... + Xk,

MY (t) = MXi (t). i Y=1

Proof. For k = 2, if X1 and X2 are independent, integer-valued, discrete r.v.s, then if Y = X1 + X2, by the Theorem of Total Probability,

fY (y) = P [Y = y] = P [Y = y X1 = x1] P [X1 = x1] = fX2 (y x1) fX1 (x1) . x | x − X1 X1 Hence

tY ty ty MY (t) = EfY [e ] = e fY (y) = e fX2 (y x1) fX1 (x1) y y ( x − ) X X X1

t(x1+x2) = e fX2 (x2) fX1 (x1) (changing variables in the summation, x2 = y x1) x ( x ) − X2 X1

tx1 tx2 = e fX1 (x1) e fX2 (x2) = MX1 (t)MX2 (t), ( x )( x ) X1 X2 28 CHAPTER 2. RANDOM VARIABLES & PROBABILITY DISTRIBUTIONS and the result follows for general k by recursion. The result for continuous random variables follows in the obvious way.

Special Case : If X1, ..., Xk are identically distributed, then MX (t) MX (t), say, for all i, so i ≡ k k MY (t) = MX (t) = MX (t) . i { } Y=1

2.7.3 OTHER GENERATING FUNCTIONS

Definition 2.7.2 For random variable X, with mass/density function fX , the factorial moment or probability generating function, fmgf or pgf , of X, GX , is defined by

X X log t GX (t) = EfX [t ] = EfX [e ] = MX (log t), if this expectation exists for all values of t (1 h, 1 + h) for some h > 0. ∈ − Properties : (i) Using similar techniques to those used for the mgf, it can be shown that

r (r) d X r G (t) = GX (s) = Ef X(X 1)...(X r + 1)t − X dsr { }s=t X − − (r) = G (1) = Ef [X(X 1)...(X r + 1)], ⇒ X X − − where Ef [X(X 1)...(X r + 1)] is the rth factorial moment. X − − (ii) For discrete random variables, it can be shown by using a Taylor series expansion of GX that, for r = 1, 2, ..., G(r)(0) X = P [X = r]. r!

Definition 2.7.3 For random variable X with mass/density function fX , the cumulant generating function of X, KX , is defined by

KX (t) = log [MX (t)] , for t ( h, h) for some h > 0. ∈ − Moment generating functions provide a very useful technique for identifying distributions, but suffer from the disadvantage that the integrals which define them may not always be finite. Another class of functions which are equally useful and whose finiteness is guaranteed is described next.

Definition 2.7.4 The characteristic function, or cf, of X, CX , is defined by

itX CX (t) = EfX e . 2.8. JOINT PROBABILITY DISTRIBUTIONS 29

By definition

itx CX (t) = e fX (x)dx = [cos tx + i sin tx] fX (x)dx x X x X Z ∈ Z ∈

= cos txfX (x)dx + i sin txfX (x)dx x X x X Z ∈ Z ∈

= EfX [cos tX] + iEfX [sin tX] . We will be concerned primarily with cases where the moment generating function exists, and the use of moment generating functions will be a key tool for identification of distributions.

2.8 JOINT PROBABILITY DISTRIBUTIONS

Suppose X and Y are random variables on the probability space (Ω, ,P (.)). Their distribution A functions FX and FY contain information about their associated probabilities. But how do we describe information about their properties relative to each other? We think of X and Y as components of a random vector (X,Y ) taking values in R2, rather than as unrelated random variables each taking values in R.

Example 2.5 Toss a coin n times and let Xi = 0 or 1, depending on whether the ith toss is a tail or a head. The random vector X = (X1,...,Xn) describes the whole experiment. The total n number of heads is i=1 Xi.

The joint distributionP function of a random vector (X1,...,Xn) is P (X1 x1,...,Xn xn), ≤ ≤ a function of n real variables x1, . . . , xn.

For vectors x = (x1, . . . , xn) and y = (y1, . . . , yn), write x y if xi yi for each i = 1, . . . , n. ≤ ≤ Definition 2.8.1 The joint distribution function of a random vector X = (X1,...,Xn) on n n (Ω, ,P (.)) is given by FX : R [0, 1], defined by FX (x) = P (X x), x R . [Remember, formally,A X x means ω Ω:−→X(ω) x .] ≤ ∈ { ≤ } { ∈ ≤ } We will consider, for simplicity, the case n = 2, without any loss of generality: the case n > 2 is just notationally more cumbersome.

Properties of the joint distribution function.

The joint distribution function FX,Y of the random vector (X,Y ) satisfies: (i) lim FX,Y (x, y) = 0, x,y −→−∞

lim FX,Y (x, y) = 1. x,y −→∞ 30 CHAPTER 2. RANDOM VARIABLES & PROBABILITY DISTRIBUTIONS

(ii) If (x1, y1) (x2, y2) then ≤ FX,Y (x1, y1) FX,Y (x2, y2). ≤

(iii) FX,Y is continuous from above,

FX,Y (x + u, y + v) FX,Y (x, y), −→ as u, v 0+. −→

(iv) lim FX,Y (x, y) = FX (x) P (X x), y −→∞ ≡ ≤ lim FX,Y (x, y) = FY (y) P (Y y). x −→∞ ≡ ≤

FX and FY are the marginal distribution functions of the joint distribution FX,Y .

Definition 2.8.2 The random variables X and Y on (Ω, ,P (.)) are (jointly) discrete if (X,Y ) A takes values in a countable subset of R2 only.

Definition 2.8.3 Discrete variables X and Y are independent if the events X = x and Y = y are independent for all x and y. { } { }

2 Definition 2.8.4 The joint probability mass function fX,Y : R [0, 1] of X and Y is given by −→ fX,Y (x, y) = P (X = x, Y = y).

The marginal pmf of X, fX (x), is found from:

fX (x) = P (X = x) = P (X = x, Y = y) y X = fX,Y (x, y). y X

Similarly for fY (y).

The definition of independence can be reformulated as: X and Y are independent iff fX,Y (x, y) = fX (x)fY (y), for all x, y R. ∈

More generally, X and Y are independent iff fX,Y (x, y) can be factorized as the product g(x)h(y) of a function of x alone and a function of y alone.

Let X be the support of X and Y be the support of Y . Then Z = (X,Y ) has support Z = (x, y): { fX,Y (x, y) > 0 . In nice cases Z = X Y, but we need to be alert to cases with Z X Y. In } × ⊂(k) × general, given a random vector (X1,...,Xk) we will denote its range or support by X . 2.8. JOINT PROBABILITY DISTRIBUTIONS 31

Definition 2.8.5 The conditional distribution function of Y given X = x, FY X (y x) is defined by | | FY X (y x) = P (Y y X = x), | | ≤ | for any x such that P (X = x) > 0. The conditional probability mass function of Y given X = x, fY X (y x), is defined by | |

fY X (y x) = P (Y = y X = x), | | | for any x such that P (x = x) > 0.

Turning now to the continuous case, we define:

Definition 2.8.6 The random variables X and Y on (Ω, ,P (.)) are called jointly continuous if their joint distribution function can be expressed as A

x y FX,Y (x, y) = fX,Y (u, v)dvdu, u= v= Z −∞ Z −∞ 2 x, y R, for some fX,Y : R [0, ). ∈ −→ ∞

Then fX,Y is the joint probability density function of X,Y .

If FX,Y is ‘sufficiently differentiable’ at(x, y) we have

∂2 f (x, y) = F (x, y). X,Y ∂x∂y X,Y

This is the usual case, which we will assume from now on.

Then:

(i)

P (a X b, c Y d) = FX,Y (b, d) FX,Y (a, d) FX,Y (b, c) + FX,Y (a, c) ≤ ≤ ≤ ≤ d b − − = fX,Y (x, y)dxdy. Zc Za

If B is a ‘nice’ subset of R2, such as a union of rectangles,

P ((X,Y ) B) = fX,Y (x, y)dxdy. ∈ ZB Z

(ii) The marginal distribution functions of X and Y are:

FX (x) = P (X x) = FX,Y (x, ), ≤ ∞

FY (y) = P (Y y) = FX,Y ( , y). ≤ ∞ 32 CHAPTER 2. RANDOM VARIABLES & PROBABILITY DISTRIBUTIONS

Since x ∞ FX (x) = fX,Y (u, y)dy du, Z−∞ Z−∞ we see, differentiating with respect to x, that the marginal pdf of X is

∞ fX (x) = fX,Y (x, y)dy. Z−∞

Similarly, the marginal pdf of Y is

∞ fY (y) = fX,Y (x, y)dx. Z−∞

We cannot, as we did in the discrete case, define independence of X and Y in terms of events X = x and Y = y , as these have zero probability and are trivially independent. { } { } So,

Definition 2.8.7 X and Y are independent if X x and Y y are independent { ≤ } { ≤ } events, for all x, y R. ∈ So, X and Y are independent iff

FX,Y (x, y) = FX (x)FY (y), x, y R, ∀ ∈ or (equivalently) iff fX,Y (x, y) = fX (x)fY (y),

whenever FX,Y is differentiable at (x, y).

(iv) Definition 2.8.8 The conditional distribution function of Y given X = x, FY X (y x) or P (Y y X = x) is defined as | | ≤ | y fX,Y (x, v) FY X (y x) = dv, | | v= fX (x) Z −∞ for any x such that fX (x) > 0.

Definition 2.8.9 The conditional density function of Y , given X = x, fY X (y x), is defined by | | fX,Y (x, y) fY X (y x) = , | | fX (x)

for any x such that fX (x) > 0.

This is an appropriate point to remark that not all random variables are either continuous or discrete, and not all distribution functions are either absolutely continuous or discrete. Many practical examples exist of distribution functions that are partly discrete and partly continuous. 2.8. JOINT PROBABILITY DISTRIBUTIONS 33

Example 2.6 We record the delay that a motorist encounters at a one-way traffic stop sign. Let X be the random variable representing the delay the motorist experiences. There is a certain probability that there will be no opposing traffic, so she will be able to proceed without delay. However, if she has to wait, she could (in principle) have to wait for any positive amount of time. The experiment could be described by assuming that X has distribution function λx FX (x) = (1 pe− )I[0, )(x). This has a jump of 1 p at x = 0, but is continuous for x > 0: there is a probability− 1 ∞ p of no wait at all. − − We shall see later cases of random vectors, (X,Y ) say, where one component is discrete and the other continuous: there is no essential complication in the manipulation of the marginal distributions etc. for such a case.

2.8.1 THE CHAIN RULE FOR RANDOM VARIABLES

As with the chain rule for manipulation of probabilities, there is an explicit relationship between joint, marginal, and conditional mass/density functions. For example, consider three continuous random variables X1,X2,X3, with joint pdf fX1,X2,X3 . Then,

fX1,X2,X3 (x1, x2, x3) = fX1 (x1)fX2 X1 (x2 x1)fX3 X1,X2 (x3 x1, x2), | | | | so that, for example,

fX1 (x1) = fX1,X2,X3 (x1, x2, x3)dx2dx3 ZX2 ZX3

= fX1 X2,X3 (x1 x2, x3)fX2,X3 (x2, x3)dx2dx3 | | ZX2 ZX3

= fX1 X2,X3 (x1 x2, x3)fX2 X3 (x2 x3)fX3 (x3)dx2dx3. | | | | ZX2 ZX3

Equivalent relationships hold in the discrete case and can be extended to determine the explicit relationship between joint, marginal, and conditional mass/density functions for any number of random variables.

NOTE: the discrete equivalent of this result is a DIRECT consequence of the Theorem of Total Probability; the event [X1 = x1] is partitioned into sub-events [(X1 = x1) (X2 = x2) (X3 = x3)] ∩ ∩ for all possible values of the pair (x2, x3).

2.8.2 CONDITIONAL EXPECTATION AND ITERATED EXPECTATION

Consider two discrete/continuous random variables X1 and X2 with joint mass function/pdf fX1,X2 , and the conditional mass function/pdf of X1 given X2 = x2, defined in the usual way by

fX1,X2 (x1, x2) fX1 X2 (x1 x2) = . | | fX2 (x2) 34 CHAPTER 2. RANDOM VARIABLES & PROBABILITY DISTRIBUTIONS

Then the conditional expectation of g(X1) given X2 = x2 is defined by

g(x1)fX1 X2 (x1 x2),X1 DISCRETE. | |  x1 X1 E [g(X ) X = x ] = X∈ fX X 1 2 2  1| 2 |   g(x1)fX1 X2 (x1 x2)dx1,X1 CONTINUOUS, | |  Z X1  i.e. the expectation of g(X1) with respect to the conditional density of X1 given X2 = x2, (possibly giving a function of x2). The case g(x) x is a particular case. ≡

Theorem 2.8.1 THE LAW OF ITERATED EXPECTATION

For two continuous random variables X1 and X2 with joint pdf fX1,X2 ,

E [g(X )] = E E [g(X ) X = x ] . fX1 1 fX2 fX X 1 2 2 1| 2 | h i Proof

E [g(X )] = g(x )f (x )dx fX1 1 1 X1 1 1 Z X1

= g(x1) fX1,X2 (x1, x2)dx2 dx1 Z X1 Z X2

= g(x1) fX1 X2 (x1 x2)fX2 (x2)dx2 dx1 | | Z X1 Z X2

= g(x1)fX1 X2 (x1 x2)fX2 (x2)dx2dx1 | | Z X1 Z X2

= g(x1)fX1 X2 (x1 x2)dx1 fX2 (x2)dx2 | | Z X2 Z X1

= EfX X [g(X1) X2 = x2] fX2 (x2)dx2 1| 2 | Z X2 n o = E E [g(X ) X = x ] , fX2 fX X 1 2 2 1| 2 | h i so the expectation of g(X1) can be calculated by finding the conditional expectation of g(X1) given X2 = x2, giving a function of x2, and then taking the expectation of this function with respect to the marginal density for X2. Note that this proof only works if the conditional expectation and the marginal expectation are finite. This results extends naturally to k variables.

2.9 MULTIVARIATE TRANSFORMATIONS

Theorem 2.9.1 THE MULTIVARIATE TRANSFORMATION THEOREM

Let X = (X1, ..., Xk) be a vector of random variables, with joint mass/density function fX1,...,Xk . 2.9. MULTIVARIATE TRANSFORMATIONS 35

Let Y = (Y1, ..., Yk) be a vector of random variables defined by Yi = gi(X1, ..., Xk) for some functions gi, i = 1, ..., k, where the vector function g mapping (X1, ..., Xk) to (Y1, ..., Yk) is a 1-1 transformation. Then the joint mass/density function of (Y1, ..., Yk) is given by

DISCRETE fY1,...,Yk (y1, ..., yk) = fX1,...,Xk (x1, ..., xk),

CONTINUOUS fY ,...,Y (y1, ..., yk) = fX ,...,X (x1, ..., xk) J(y1, ..., yk) , 1 k 1 k | | 1 where x = (x1, ..., xk) is the unique solution of the system y = g(x), so that x = g− (y), and where J(y1, ..., yk) is the Jacobian of the transformation, that is, the determinant of the k k matrix whose (i, j)th element is ×

∂ 1 gi− (t) t =y ,...,t =y , ∂tj 1 1 k k 1 1 where gi− is the inverse function uniquely defined by Xi = gi− (Y1, ..., Yk). Note again the modulus.

Proof. The discrete case proof follows the univariate case precisely. For the continuous case, consider the equivalent events [X C] and [Y D], where D is the image of C under g. Clearly, ∈ ∈ P [X C] = P [Y D]. Now, P [X C] is the k dimensional integral of the joint density fX ,...,X ∈ ∈ ∈ 1 k over the set C, and P [Y D] is the k dimensional integral of the joint density fY1,...,Yk over the set D. The result follows by∈ changing variables in the first integral from x to y = g(x), and equating the two integrands. Note : As for single variable transformations, the ranges of the transformed variables must be considered carefully.

Example 2.7 The multivariate transformation theorem provides a simple proof of the convolution formula: if X and Y are independent continuous random variables with pdfs fX (x) and fY (y), then the pdf of Z = X + Y is

∞ fZ (z) = fX (w)fY (z w)dw. − Z−∞ Let W = X. The Jacobian of the transformation from (X,Y ) to (Z,W ) is 1. So, the joint pdf of (Z,W ) is fZ,W (z, w) = fX,Y (w, z w) = fX (w)fY (z w). − − Then integrate out W to obtain the marginal pdf of Z.

Example 2.8 Consider the case k = 2, and suppose that X1 and X2 are independent continuous random variables with ranges X1 = X2 = [0, 1] and pdfs given respectively by

fX (x1) = 6x1(1 x1), 0 x1 1, 1 − ≤ ≤ 2 fX (x2) = 3x , 0 x2 1, 2 2 ≤ ≤ and zero elsewhere. In order to calculate the pdf of random variable Y1 defined by

Y1 = X1X2, 36 CHAPTER 2. RANDOM VARIABLES & PROBABILITY DISTRIBUTIONS using the transformation result, consider the additional random variable Y2, where Y2 = X1 (note, as X1 and X2 take values on [0, 1], X1 X1X2 so Y1 Y2). ≥ ≤ The transformation Y = g(X) is then specified by the two functions

g1(t1, t2) = t1t2, g2(t1, t2) = t1,

1 and the inverse transformation X = g− (Y) (i.e. X in terms of Y) is

X1 = Y2,X2 = Y1/Y2, giving 1 1 g1− (t1, t2) = t2, g2− (t1, t2) = t1/t2.

Hence

∂ 1 ∂ 1 g1− (t1, t2) = 0, g1− (t1, t2) = 1, ∂t1 ∂t2 ∂ 1 ∂ 1 2 g2− (t1, t2) = 1/t2, g2− (t1, t2) = t1/t2, ∂t1 ∂t2 − and so the Jacobian J(y1, y2) of the transformation is given by

0 1 1/y y /y2 2 1 2 − so that J(y1, y2) = 1/y2. Hence, using the theorem −

fY ,Y (y1, y2) = fX ,X (y2, y1/y2) J(y1, y2) 1 2 1 2 × | | 2 = 6y2(1 y2) 3(y1/y2) 1/y2 − × × 2 2 = 18y (1 y2)/y , 1 − 2

(2) on the set Y = (y1, y2) : 0 y1 y2 1 , and zero otherwise. Hence { ≤ ≤ ≤ }

1 2 2 fY (y1) = 18y (1 y2)/y dy2 1 y1 1 − 2

R 2 1 = 18y [ 1/y2 log y2] 1 − − y1 2 = 18y ( 1 + 1/y1 + log y1) 1 −

= 18y1(1 y1 + y1 log y1), − for 0 y1 1, and zero otherwise. ≤ ≤ 2.10. MULTIVARIATE EXPECTATIONS AND COVARIANCE 37

2.10 MULTIVARIATE EXPECTATIONS AND COVARIANCE

2.10.1 EXPECTATION WITH RESPECT TO JOINT DISTRIBUTIONS (k) Definition 2.10.1 For random variables X1, ..., Xk with range X with mass/density function fX1,...,Xk , the expectation of g(X1, ..., Xk) is defined in the discrete and continuous cases by

... g(x1, ..., xk)fX1,...,Xk (x1, ..., xk),  XX1 XXk Ef [g(X , ..., Xk)] = X1,...,Xk 1    ... g(x1, ..., xk)fX1,...,Xk (x1, ..., xk)dx1...dxk.  Z X1 Z Xk   PROPERTIES (i) Let g and h be real-valued functions and let a and b be constants. Then, if fX fX ,...,X , ≡ 1 k

EfX [ag(X1, ..., Xk) + bh(X1, ..., Xk)] = aEfX [g(X1, ..., Xk)] + bEfX [h(X1, ..., Xk)].

(ii) Let X1, ...Xk be independent random variables with mass functions/pdfs fX1 , ..., fXk respectively. Let g1, ..., gk be scalar functions of X1, ..., Xk respectively (that is, gi is a function of Xi only for i = 1, ..., k). If g(X1, ..., Xk) = g1(X1)...gk(Xk), then

k E [g(X , ..., X )] = E [g (X )], fX 1 k fXi i i i Y=1 where E [g (X )] is the marginal expectation of g (X ) with respect to f . fXi i i i i Xi (iii) Generally, Ef [g(X1)] Ef [g(X1)], X ≡ X1 so that the expectation over the joint distribution is the same as the expectation over the marginal distribution. The proof is an immediate consequence of the fact that the marginal pdf fX1 is obtained by integrating the joint density with respect to x2, . . . , xk. So, whevever we wish, it is reasonable to denote the expectation as, say, E[g(X )], rather than E [g(X )] or E [g(X )]: we 1 fX1 1 fX 1 can ‘drop subscripts’.

2.10.2 COVARIANCE AND CORRELATION

Definition 2.10.2 The covariance of two random variables X1 and X2 is denoted Cov [X ,X ], and is defined by fX1,X2 1 2

Covf [X1,X2] = Ef [(X1 μ )(X2 μ )] = Ef [X1X2] μ μ , X1,X2 X1,X2 − 1 − 2 X1,X2 − 1 2 where μ = E [X ] is the marginal expectation of X , for i = 1, 2, and where i fXi i i

E [X X ] = x x f (x , x )dx dx , fX1,X2 1 2 1 2 X1,X2 1 2 1 2 ZZ that is, the expectation of function g(x1, x2) = x1x2 with respect to the joint distribution fX1,X2 . 38 CHAPTER 2. RANDOM VARIABLES & PROBABILITY DISTRIBUTIONS

Definition 2.10.3 The correlation of X and X is denoted Corr [X ,X ], and is defined 1 2 fX1,X2 1 2 by Cov [X ,X ] fX1,X2 1 2 CorrfX ,X [X1,X2] = . 1 2 V ar [X ]V ar [X ] fX1 1 fX2 2 If Cov [X ,X ] = Corr [X ,X ] = 0q then variables X and X are uncorrelated. fX1,X2 1 2 fX1,X2 1 2 1 2

Note that if random variables X1 and X2 are independent then

Covf [X1,X2] = Ef [X1X2] Ef [X1]Ef [X2] X1,X2 X1,X2 − X1 X2

= Ef [X1]Ef [X2] Ef [X1]Ef [X2] = 0, X1 X2 − X1 X2 and so X1 and X2 are also uncorrelated (the converse does not hold).

NOTES: (i) For random variables X1 and X2, with (marginal) expectations μ1 and μ2 respectively, and 2 2 (marginal) variances σ1 and σ2 respectively, if random variables Z1 and Z2 are defined by

Z1 = (X1 μ )/σ1,Z2 = (X2 μ )/σ2, − 1 − 2 then Z and Z are standardized variables. Then E [Z ] = 0, V ar [Z ] = 1 and 1 2 fZi i fZi i Corr [X ,X ] = Cov [Z ,Z ]. fX1,X2 1 2 fZ1,Z2 1 2

(ii) Extension to k variables: covariances can only be calculated for pairs of random variables, but if k variables have a joint probability structure it is possible to construct a k k matrix, C say, of covariance values, whose (i, j)th element is ×

Cov [X ,X ], fXi,Xj i j for i, j = 1, .., k, that captures the complete covariance structure in the joint distribution. If i = j, then 6 Cov [X ,X ] = Cov [X ,X ], fXj ,Xi j i fXi,Xj i j so C is symmetric, and if i = j,

Covf [Xi,Xi] V arf [Xi]. Xi,Xi ≡ Xi The matrix C is referred to as the variance-covariance matrix.

(iii) If random variable X is defined by X = a1X1 +a2X2 +...akXk, for random variables X1, ..., Xk and constants a1, ..., ak, then

k E [X] = a E [X ], fX i fXi i i X=1 k k i 1 − V ar [X] = a2V ar [X ] + 2 a a Cov [X ,X fX i fXi i i j fXi,Xj i j i=1 i=1 j=1 XT XTX = a Ca, a = (a1, . . . , ak) . 2.10. MULTIVARIATE EXPECTATIONS AND COVARIANCE 39

(iv) Combining (i) and (iii) when k = 2, and defining standardized variables Z1 and Z2,

0 V arf [Z1 Z2] = V arf [Z1] + V arf [Z2] 2Covf [Z1,Z2] ≤ Z1,Z2 ± Z1 Z2 ± Z1,Z2

= 1 + 1 2Corrf [X1,X2] = 2(1 Corrf [X1,X2]) ± X1,X2 ± X1,X2 and hence 1 Corrf [X1,X2] 1. − ≤ X1,X2 ≤

2.10.3 JOINT MOMENT GENERATING FUNCTION Definition 2.10.4 Let X and Y be jointly distributed. The joint moment generating function of X and Y is sX+tY MX,Y (s, t) = E(e ).

If this exists in a neighbourhood of the origin (0,0), then it has the same attractive properties as the ordinary moment generating function. It determines the joint distribution of X and Y uniquely and it also yields the moments:

∂m+n M (s, t) = E(XmY n). ∂sm∂tn X,Y s=t=0

Joint moment generating functions factorize for independent random variables. We have

MX,Y (s, t) = MX (s)MY (t) if and only if X and Y are independent. Note, MX (s) = MX,Y (s, 0), etc.

The definition of the joint moment generating function extends in an obvious way tothreeor more random variables, with the corresponding result for independence. For instance, the random variable X is independent of (Y,Z) if and only if the joint moment generating function sX+tY +uZ MX,Y,Z (s, t, u) = E(e ) factorizes as

MX,Y,Z (s, t, u) = MX (s)MY,Z (t, u).

2.10.4 FURTHER RESULTS ON INDEPENDENCE

The following results are useful:

(I) Let X and Y be independent random variables. Let g(x) be a function only of x and h(y) be a function only of y. Then the random variables U = g(X) and V = h(Y ) are independent.

(II) Let X1,...,Xn be independent random vectors. Let gi(xi) be a function only of xi, i = 1, . . . , n. Then the random variables Ui = gi(Xi), i = 1, . . . , n are independent. 40 CHAPTER 2. RANDOM VARIABLES & PROBABILITY DISTRIBUTIONS

2.11 ORDER STATISTICS

Order statistics, like sample moments, play a very important role in statistical inference. Let X1,...,Xn be independent, identically distributed continuous random variables, with cdf FX and pdf fX . Then order the Xi: let Y1 be the smallest of X1,...,Xn , Y2 be the second smallest of { } X1,...,Xn , ..., Yn be the largest of X1,...,Xn . Note that since we assume continuity, the chance{ of ties} is zero. It is customary to{ use the notation}

Yk = X(k), and then X(1),X(2),...,X(n) are known as the order statistics of X1,...,Xn. Two key results are:

Result A The order statistics have joint density

n n! fX (yi), y1 < y2 < . . . < yn. i Y=1

Result B The order statistic X(k) has density

n n k k 1 f k (y) = k fX (y) 1 FX (y) − FX (y) − . ( ) k { − } { } An informal proof of Result A is straightforward, using a symmetry argument based on the independent, identically distributed nature of X1,...,Xn. Result B follows on noting that the event X k y occurs if and only if at least k of the Xi lie in ( , y]. Recalling the Binomial distribution, ( ) ≤ −∞ this means that X(k) has distribution function

n n j n j F k (y) = FX (y) 1 FX (y) − . ( ) j { } { − } Xj=k The pdf follows on differentiating this cdf. CHAPTER 3 DISCRETE PROBABILITY DISTRIBUTIONS

Definition 3.1.1 DISCRETE UNIFORM DISTRIBUTION

X Uniform(n) ∼ 1 fX (x) = , x X = 1, 2, ..., n , n ∈ { } and zero otherwise.

Definition 3.1.2 BERNOULLI DISTRIBUTION

X Bernoulli(θ) ∼ x 1 x fX (x) = θ (1 θ) − , x X = 0, 1 , − ∈ { } and zero otherwise.

NOTE The Bernoulli distribution is used for modelling when the outcome of an experiment is either a “success” or a ‘failure”, where the probability of getting a success is equal to θ. Such an experiment is a ‘Bernoulli trial’. The mgf is

t MX (t) = (1 θ) + θe . −

Definition 3.1.3 BINOMIAL DISTRIBUTION

X Bin(n, θ) ∼

n x n x fX (x) = θ (1 θ) − , x X = 0, 1, 2, ..., n , n 1, 0 θ 1. x − ∈ { } ≥ ≤ ≤ NOTES 1. If X1, ..., Xk are independent and identically distributed (IID) Bernoulli(θ) random variables, and Y = X1 + ..., Xk, then by the standard result for mgfs,

k t k MY (t) = MX (t) = 1 θ + θe , { } − so therefore Y Bin(k, θ) because of the uniqueness of mgfs. Thus the binomial distribution is used to model the∼ total number of successes in a series of independent and identical experiments.

2. Alternatively, consider sampling without replacement from infinite collection, or sampling with replacement from a finite collection of objects, a proportion θ of which are of Type I, and the remainder are of Type II. If X is the number of Type I objects in a sample of n, X Bin(n, θ). ∼

41 42 CHAPTER 3. DISCRETE PROBABILITY DISTRIBUTIONS

Definition 3.1.4 POISSON DISTRIBUTION

X P oisson(λ) ∼ λ x e− λ fX (x) = , x X = 0, 1, 2, ... , λ > 0, x! ∈ { } and zero otherwise

NOTES 1. If X Bin(n, θ), let λ = nθ. Then ∼ t n t n λ(e 1)) t MX (t) = 1 θ + θe = 1 + − exp λ e 1 , − n −→ − as n , which is the mgf of a Poisson random variable. Therefore, the Poisson distribution arises−→ as the∞ limiting case of the binomial distribution, when n , θ 0 with nθ = λ constant (that is, for “large” n and “small” θ). So, if n is large, θ is small,−→ ∞ we can−→ reasonably approximate Bin(n, θ) by P oisson(λ).

2. Suppose that X1 and X2 are independent, with X1 P oisson(λ1), X2 P oisson(λ2), then if ∼ ∼ Y = X1 + X2, using the general mgf result for independent random variables,

t t t MY (t) = MX (t)MX (t) = exp λ1 e 1 exp λ2 e 1 = exp (λ1 + λ2) e 1 1 2 − − − so that Y P oisson(λ1 + λ2). Therefore, the sum of two independent Poisson random variables ∼ also has a Poisson distribution. This result can be extended easily; if X1, ..., Xk are independent random variables with Xi P oisson(λi) for i = 1, ..., k, then ∼ k k Y = Xi = Y P oisson λi . i ⇒ ∼ i ! X=1 X=1 3. THE POISSON PROCESS (This material is not examinable.)

The Poisson distribution arises as part of a larger modelling framework. Consider an experiment involving events (such as radioactive emissions) that occur repeatedly and randomly in time. Let X(t) be the random variable representing the number of events that occur in the interval [0, t], so that X(t) takes values 0, 1, 2, .... Informally, in a Poisson process we have the following properties. In the interval (t, t + h) there may or may not be events. If h is small, the probability of an event in (t, t + h) is roughly proportional to h; it is not very likely that two or more events occur in a small interval.

Formally, a Poisson process with intensity λ is a process X(t), t 0, taking values in 0, 1, 2,... such that ≥ { } (a) X(0) = 0 and if s < t then X(s) X(t), ≤ (b) λh + o(h), m = 1, P (X(t + h) = n + m X(t) = n) = o(h), m > 1,  | 1 λh + o(h), m = 0.  −  43

Here [definition] o(h) lim = 0. h 0 h −→

Then, if Pn(t) = P [X(t) = n] = P [n events occur in [0, t]] it can be shown that e λt(λt)n P (t) = − n n! (that is, the random variable corresponding to the number of events that occurs in the interval [0, t] has a Poisson distribution with parameter λt.)

Definition 3.1.5 GEOMETRIC DISTRIBUTION

X Geometric(θ) ∼ x 1 fX (x) = (1 θ) − θ, x X = 1, 2, ... , 0 θ 1. − ∈ { } ≤ ≤ NOTES 1. The cdf is available analytically as

x FX (x) = 1 (1 θ) , x = 1, 2, 3, ... − − 2. If X Geometric(θ), then for x, j 1, ∼ ≥ x+j 1 P [X = x + j, X > j] P [X = x + j] (1 θ) − θ x 1 P [X = x+j X > j] = = = − = (1 θ) − θ = P [X = x]. | P [X > j] P [X > j] (1 θ)j − − So P[X = x + j X > j] =P[X = x]. This property is unique (among discrete distributions) to the geometric distribution,| and is called the lack of memory property.

3. Alternative representations are sometimes useful:

x 1 fX (x) = φ − (1 φ) , x = 1, 2, 3... (that is, φ = 1 θ), − −

x fX (x) = φ (1 φ) , x = 0, 1, 2,.... − 4. The geometric distribution is used to model the number, X, of independent, identical Bernoulli trials until the first success is obtained. It is a discrete waiting time distribution.

Definition 3.1.6 NEGATIVE BINOMIAL DISTRIBUTION

X NegBin(n, θ) ∼

x 1 n x n fX (x) = − θ (1 θ) − , x X = n, n + 1, ... , n 1, 0 θ 1. n 1 − ∈ { } ≥ ≤ ≤ − 44 CHAPTER 3. DISCRETE PROBABILITY DISTRIBUTIONS

NOTES 1. If X Bin(n, θ), Y NegBin(r, θ), then for r n, P [X r] = P [Y n]. ∼ ∼ ≤ ≥ ≤ 2. The Negative Binomial distribution is used to model the number, X, of independent, identical Bernoulli trials needed to obtain exactly n successes. [The number of trials up to and including the nth success].

3. Alternative representation: let Y be the number of failures in a sequence of independent, identical Bernoulli trials that contains exactly n successes. Then Y = X n, and hence − n + y 1 n y fY (y) = − θ (1 θ) , y 0, 1, ... . n 1 − ∈ { } − 4. If Xi Geometric(θ), for i = 1, ...n, are i.i.d. random variables, and Y = X1 + ... + Xn, then Y NegBin∼ (n, θ) (result immediately follows using mgfs). ∼ 5. If X NegBin(n, θ), let n(1 θ)/θ = λ and Y = X n. Then ∼ − − n n nt θ 1 t MY (t) = e− MX (t) = = exp λ(e 1) , 1 et(1 θ) 1 λ (et 1) −→ − ( n ) − − − − as n , hence the alternate form of the negative binomial distribution tends to the Poisson distribution−→ ∞ as n with n(1 θ)/θ = λ constant. −→ ∞ −

Definition 3.1.7 HYPERGEOMETRIC DISTRIBUTION X HypGeom(N, R, n) for N R n ∼ ≥ ≥ N R R − n x x fX (x) = − , x X = max(0, n N + R), ..., min(n, R) , N ∈ { − } n and zero otherwise.

NOTES 1. The hypergeometric distribution is used as a model for experiments involving sampling without replacement from a finite population. Specifically, consider a finite population ofsize N, consisting of R items of Type I and N R of Type II: take a sample of size n without replacement, and let X be the− number of Type I objects on the sample. The mass function for the hypergeometric distribution can be obtained by using combinatorics/counting techniques. However the form of the mass function does not lend itself readily to calculation of moments etc..

2. As N,R with R/N = θ(constant), then −→ ∞

n x n x P [X = x] θ (1 θ) − , −→ x − so the distribution tends to a Binomial distribution. CHAPTER 4 CONTINUOUS PROBABILITY DISTRIBUTIONS

Definition 4.1.1 CONTINUOUS UNIFORM DISTRIBUTION

X Uniform(a, b) ∼ 1 fX (x) = , a x b. b a ≤ ≤ − NOTES 1. The cdf is x a FX (x) = − , a x b. b a ≤ ≤ −

2. The case a = 0 and b = 1 gives the Standard uniform.

Definition 4.1.2 EXPONENTIAL DISTRIBUTION

X Exp(λ) ∼ λx fX (x) = λe− , x > 0, λ > 0. NOTES 1. The cdf is λx FX (x) = 1 e− , x > 0. − 2. An alternative representation uses θ = 1/λ as the parameter of the distribution. This is sometimes used because the expectation and variance of the Exponential distribution are 1 1 Ef [X] = = θ, V arf [X] = . X λ X λ2

3. If X Exp(λ), then, for all x, t > 0, ∼ λ(x+t) P [X > x + t, X > t] P [X > x + t] e− λx P [X > x + t X > t] = = = λt = e− = P [X > x]. | P [X > t] P [X > t] e− Thus, for all x, t > 0, P [X > x + t X > t] = P [X > x] - this is known as the Lack of Memory Property, and is unique to the exponential| distribution amongst continuous distributions.

4. Suppose that X(t) is a Poisson process with rate parameter λ > 0, so that

e λt(λt)n P [X(t) = n] = − . n!

Let X1, ..., Xn be random variables defined by X1 = “time that first event occurs”, and, for i = 2, ..., n, Xi = “time interval between occurrence of (i 1)st and ith events”. Then X1, ..., Xn − 45 46 CHAPTER 4. CONTINUOUS PROBABILITY DISTRIBUTIONS are IID because of the assumptions underlying the Poisson process. So consider the distribution of X1; in particular, consider the probability P[X1 > x] for x > 0. The event [X1 > x] is λx equivalent to the event “No events occur in the interval (0,x]”, which has probability e− . But

λx FX (x) = P [X1 x] = 1 P [X1 > x] = 1 e− = X1 Exp(λ). 1 ≤ − − ⇒ ∼ 5. The exponential distribution is used to model failure times in continuous time. It is a continuous waiting time distribution, the continuous analogue of the geometric distribution.

6. If X Uniform(0, 1), and ∼ 1 Y = log(1 X), −λ − then Y Exp(λ). ∼ 7. If X Exp(λ), and ∼ Y = X1/α, for α > 0, then Y has a (two-parameter) Weibull distribution, and

α 1 λyα fY (y) = αλy − e− , y > 0.

Definition 4.1.3 GAMMA DISTRIBUTION

X Ga(α, β) ∼ βα f (x) = xα 1e βx, x > 0, α, β > 0, X Γ(α) − − where, for any real number α > 0, the gamma function, Γ(.) is defined by

∞ α 1 t Γ(α) = t − e− dt. Z0 NOTES 1. If X1 Ga(α1, β),X2 Ga(α2, β) are independent random variables, and ∼ ∼

Y = X1 + X2 then Y Ga(α1 + α2, β) (directly from properties of mgfs). ∼ 2. Ga(1, β) Exp(β). ≡

3. If X1, ..., Xn Exp(λ) are independent random variables, and ∼

Y = X1 + ... + Xn then Y Ga(n, λ) (directly from 1. and 2. ). ∼ 4. For α > 0, integrating by parts, we have that

Γ(α) = (α 1)Γ(α 1) − − and hence if α = 1, 2, ..., then Γ(α) = (α 1)!. A useful fact is that Γ( 1 ) = √π. − 2 47

5. Special Case : If α = 1, 2, ... the Ga(α/2, 1/2) distribution is also known as the chi-squared 2 distribution with α degrees of freedom, denoted by χα.

2 2 6. If X1 χn1 and X2 χn2 are independent chi-squared random variables with n1 and n2 degrees of∼ freedom respectively,∼ then random variable F defined as the ratio

X /n F = 1 1 X2/n2 has an F-distribution with (n1, n2) degrees of freedom.

Definition 4.1.4 BETA DISTRIBUTION

X Be(α, β) ∼

Γ(α + β) α 1 β 1 fX (x) = x − (1 x) − , 0 < x < 1, α, β > 0. Γ(α)Γ(β) −

NOTES 1. If α = β = 1, Be(α, β) Uniform(0, 1) ≡

2. If X1 Ga(α1, β), X2 Ga(α2, β) are independent random variables, and ∼ ∼ X Y = 1 X1 + X2 then Y Be(α1, α2) (using standard multivariate transformation techniques). ∼ 3. Suppose that random variables X and Y have a joint probability distribution such that the conditional distribution of X, given Y = y for 0 < y < 1, is binomial, Bin(n, y), and the marginal distribution of Y is beta, Be(α, β), so that

n x n x fX Y (x y) = y (1 y) − , x = 0, 1, ..., n, | | x −

Γ(α + β) α 1 β 1 fY (y) = y − (1 y) − , 0 < y < 1. Γ(α)Γ(β) −

Then the marginal distribution of X is given by

1 fX (x) = fX Y (x y)fY (y)dy | | Z0 n Γ(α + β) Γ(x + α)Γ(n x + β) = − , x = 0, 1, 2...n. x Γ(α)Γ(β) Γ(n + α + β) Note that this provides an example of a joint distribution of continuous Y and discrete X. 48 CHAPTER 4. CONTINUOUS PROBABILITY DISTRIBUTIONS

Definition 4.1.5 NORMAL DISTRIBUTION

X N(μ, σ2) ∼

1 1 2 fX (x) = exp (x μ) , x R, μ R, σ > 0. √ 2 −2σ2 − ∈ ∈ 2πσ NOTES 1. Special Case : If μ = 0, σ2 = 1, then X has a standard or unit normal distribution. Usually, the pdf of the standard normal is written φ(x), and the cdf is written Φ(x).

2. If X N(0, 1), and ∼ Y = σX + μ then Y N(μ, σ2). Re-expressing this result, if X N(μ, σ2), and Y = (X μ)/σ, then Y N(0∼, 1) (using transformation or mgf techniques).∼ − ∼

3. The Central Limit Theorem Suppose X1, ..., Xn are IID random variables with some mgf 2 MX , with EfX [Xi] = μ and V arfX [Xi] = σ that is, the mgf and the expectation and variance of the Xi’s are specified, but the pdf is not. Let the standardized random variable Zn be defined by

n Xi nμ − i=1 Zn = X √nσ2 and let Zn have mgf MZ . Then, as n , n −→ ∞ 2 MZ (t) exp t /2 , n −→ irrespective of the distribution of the Xi’s, that is, the distribution of Zn tends to a standard normal distribution as n tends to infinity. This theorem will be proved and explained in Chapter 6.

2 2 4. If X N(0, 1), and Y = X , then Y χ1, so that the square of a unit normal random variable∼ has a chi-squared distribution∼with 1 degree of freedom.

5. If X N(0, 1), and Y N(0, 1) are independent random variables, and Z is defined by Z = X/Y∼, then Z has a Cauchy∼ distribution 1 1 fZ (z) = , z R. π 1 + z2 ∈ 2 6. If X N(0, 1), and Y Ga(n/2, 1/2) for n = 1, 2, ... (so that Y χn), are independent random variables,∼ and T is∼ defined by ∼ X T = Y/n then T has a Student-t distribution with n degreesp of freedom, T St(n), ∼ n + 1 Γ 1/2 2 (n+1)/2 2 1 t − fT (t) = n 1 + , t R. Γ nπ n ∈ 2 49

Taking limiting cases of the Student-t distribution

n : St(n) N(0, 1), n 1 : St(n) Cauchy. −→ ∞ −→ −→ −→ 2 2 7. If X1 N(μ1, σ1) and X2 N(μ2, σ2) are independent and a, b are constants, then ∼ ∼ 2 2 2 2 T = aX1 + bX2 N(aμ + bμ , a σ + b σ ). ∼ 1 2 1 2 50 CHAPTER 4. CONTINUOUS PROBABILITY DISTRIBUTIONS CHAPTER 5 MULTIVARIATE PROBABILITY DISTRIBUTIONS

For purely notational reasons, it is convenient in this chapter to consider a random vector X as a T column vector, X = (X1,...,Xk) , say

5.1 THE MULTINOMIAL DISTRIBUTION

The multinomial distribution is a multivariate generalization of the binomial distribution. Recall that the binomial distribution arose from an infinite Urn model with two types of objects being sampled with replacement. Suppose that the proportion of “Type 1” objects in the urn is θ (so 0 θ 1) and hence the proportion of “Type 2” objects in the urn is 1 θ. Suppose that n objects are≤ sampled,≤ and X is the random variable corresponding to the number− of “Type 1” objects in the sample. Then X Bin(n, θ), and ∼ n x n x fX (x) = θ (1 θ) − , x 0, 1, 2, ..., n . x − ∈ { } Now consider a generalization; suppose that the Urn contains k + 1 types of objects (k = 1, 2, ...), with θi being the proportion of Type i objects, for i = 1, ..., k + 1. Let Xi be the random variable corresponding to the number of type i objects in a sample of size n, for i = 1, ..., k. Then the joint T distribution of vector X = (X1, ..., Xk) is given by

k+1 n! x1 xk xk+1 n! xi fX1,...,Xk (x1, ..., xk) = θ1 ....θk θk+1 = θi , x1!...xk!xk+1! x1!...xk!xk+1! i=1 Q where 0 θi 1 for all i, and θ1 + ... + θk + θk+1 = 1, and where xk+1 is defined by xk+1 = ≤ ≤ n (x1 + ... + xk). This is the mass function for the multinomial distribution which reduces to the − binomial if k = 1. It can also be shown that the marginal distribution of Xi is Bin(n, θi).

5.2 THE DIRICHLET DISTRIBUTION

The Dirichlet distribution is a multivariate generalization of the beta distribution. Recall that the beta distribution arose as follows; suppose that V1 and V2 are independent Gamma random variables with V1 Ga(α1, β),V2 Ga(α2, β). Then if X is defined by X = V1/ (V1 + V2), we have ∼ ∼ that X Be(α1, α2). Now consider a generalization; suppose that V1, ..., Vk+1 are independent ∼ Gamma random variables with Vi Ga(αi, β), for i = 1, ..., k + 1. Define ∼ Vi Xi = V1 + ... + Vk+1 T for i = 1, ..., k. Then the joint distribution of vector X = (X1, ..., Xk) is given by density Γ(α) α1 1 αk 1 αk+1 1 fX1,...,Xk (x1, ..., xk) = x1 − ...xk − xk+1 − , Γ(α1)...Γ(αk)Γ(αk+1)

51 52 CHAPTER 5. MULTIVARIATE PROBABILITY DISTRIBUTIONS for 0 xi 1 for all i such that x1 + ... + xk + xk+1 = 1, where α = α1 + ... + αk+1 and where ≤ ≤ xk+1 is defined by xk+1 = 1 (x1 + ... + xk). This is the density function which reduces to the beta − distribution if k = 1. It can also be shown that the marginal distribution of Xi is Beta(αi, α).

5.3 THE MULTIVARIATE NORMAL DISTRIBUTION

T The random vector X = (X1,...,Xk) has a multivariate normal distribution if the joint pdf is of the form:

k/2 1 1 1 T 1 fX (x1, ..., xk) = exp (x μ) Σ− (x μ) . 2π Σ 1/2 −2 − − | | Here x is the (column) vector of length k formed by x1, . . . , xk, μ is a (column) vector of length k, Σ is a k k symmetric, positive definite matrix [Σ =ΣT , xT Σx > 0 for all x = 0], and Σ denotes the determinant× of Σ. 6 | |

We write X Nk(μ, Σ). ∼ Properties

T 1. E[X] = μ: μ is the mean vector of X. If μ = (μ1, . . . , μk) , we have E[Xi] = μi. Further, Σ is the variance-covariance matrix of X, Σ = [σij], where σij = cov[Xi,Xj].

2. Since Σ is symmetric and positive definite, there exists a matrix Σ1/2 [the ‘square root of Σ’] 1/2 1/2 1/2 1/2 1/2 1/2 1/2 such that: (i) Σ is symmetric; (ii) Σ = Σ Σ ; (iii) Σ Σ− = Σ− Σ = I, the k k 1/2 1/2 1 × identity matrix, with Σ− = (Σ )− .

1/2 Then, if Z Nk(0,I), so that Z1,...Zk are IID N(0, 1), and X = μ + Σ Z, then X Nk(μ, Σ). ∼ 1/2 ∼ Conversely, if X Nk(μ, Σ), then Σ (X μ) Nk(0,I). ∼ − − ∼

3. A useful result is the following: if X Nk(μ, Σ) and D is a m k matrix of rank m k, then T ∼ × ≤ Y DX Nm(Dμ,DΣD ). A special case is where X Nk(0,I) and D is a k k matrix of full ≡ ∼ ∼ T × rank k, so that D is invertible: then Y = DX Nk(0,DD ). ∼ 4. Suppose we partition X as X X = a , X b with X1 Xm+1 . . Xa =  .  ,Xb =  .  . X X  m   k  We can similarly partition μ and Σ:    

μ Σ Σ μ = a , Σ = aa ab . μ Σ Σ b ba bb

Then if X Nk(μ, Σ) we have: ∼ 5.3. THE MULTIVARIATE NORMAL DISTRIBUTION 53

(I) The marginal distribution of Xa is Nm(μa, Σaa).

(II) The conditional distribution of Xb, given Xa = xa is

1 1 Xb Xa = xa Nk m(μb + ΣbaΣaa− (xa μa), Σbb ΣbaΣaa− Σab). | ∼ − − −

T (III) If a = (a1, . . . , ak) , then

k T T T a X aiXi N(a μ, a Σa). ≡ i ∼ X=1

(IV) V = (X μ)T Σ 1(X μ) χ2. − − − ∼ k 54 CHAPTER 5. MULTIVARIATE PROBABILITY DISTRIBUTIONS CHAPTER 6 PROBABILITY RESULTS & LIMIT THEOREMS

6.1 BOUNDS ON PROBABILITIES BASED ON MOMENTS

Theorem 6.1.1 If X is a random variable, then for non-negative function h, and c > 0, E [h(X)] P [h(X) c] fX . ≥ ≤ c Proof. (continuous case) : Suppose that X has density function fX which is positive for x X. ∈ Let = x X : h(x) c X. Then, as h(x) c on , A { ∈ ≥ } ⊆ ≥ A

EfX [h(X)] = h(x)fX (x)dx = h(x)fX (x)dx + h(x)fX (x)dx Z Z Z A A0 h(x)fX (x)dx cfX (x)dx = cP [X ] = cP [h(X) c] . ≥ ≥ ∈ A ≥ Z Z A A SPECIAL CASE I - THE MARKOV INEQUALITY : If h(x) = x r for r > 0, so | | r 1 r P [ X c] Ef [ X ] . | | ≥ ≤ c X | | SPECIAL CASE II - THE CHEBYCHEV INEQUALITY: Suppose that X is a random variable with expectation μ and variance σ2. Then taking h(x) = (x μ)2 and c = k2σ2, for k > 0, gives − P [ X μ kσ] 1/k2, | − | ≥ ≤ and setting = kσ gives P [ X μ ] σ2/2,P [ X μ < ] 1 σ2/2. | − | ≥ ≤ | − | ≥ −

Theorem 6.1.2 JENSEN’S INEQUALITY d2 Suppose that X is a random variable, and function g is convex so that g(t) = g (x) > 0, dt2 { }t=x 00 x, with Taylor expansion around expectation μ of the form ∀ 1 2 g(x) = g(μ) + (x μ)g0(μ) + (x μ) g00(x0), (6.1) − 2 − for some x0 such that x < x0 < μ. Then

Ef [g(X)] g(Ef [X]). X ≥ X 2 2 Proof. Taking expectations in (6.1), and noting that Ef [(X μ)] = 0, Ef (X μ) = σ , X − X − g (x0) 0, we have that 00 ≥ 1 2 Ef [g(X)] = g(μ) + 0 g0(μ) + σ g00(x0) g(μ) = g(Ef [X]), X × 2 × ≥ X 2 as σ , g00(x0) > 0.

55 56 CHAPTER 6. PROBABILITY RESULTS & LIMIT THEOREMS

6.2 THE CENTRAL LIMIT THEOREM

Theorem 6.2.1 Suppose X1, ..., Xn are i.i.d. random variables with mgf MX , with

2 EfX [Xi] = μ, VarfX [Xi] = σ , both finite. Let the random variable Zn be defined by n Xi nμ − i=1 Zn = X , √nσ2 and let Zn have mgf MZ . Then, as n , n −→ ∞ t2 MZ (t) exp , n −→ 2 irrespective of the form of MX .

Proof. First, let Yi = (Xi μ)/σ for i = 1, ..., n. Then Y1, ..., Yn are i.i.d. with mgf MY say, and by the elementary properties− of expectation,

EfY [Yi] = 0, V arfY [Yi] = 1, for each i. Using the power series expansion result for mgfs, we have that t2 t3 t4 M (t) = 1 + tE [Y ] + E [Y 2] + E [Y 3] + E [Y 4] + ... Y fY 2! fY 3! fY 4! fY t2 t3 t4 = 1 + + E [Y 3] + E [Y 4] + ... 2! 3! fY 4! fY

Now, the random variable Zn can be rewritten n 1 Xi μ Zn = − √n σ i X=1 and thus, again by a standard mgf result, as Y1, ..., Yn are independent, we have that

n 2 3 4 n t t 3 t 4 MZ (t) = MY (t/√n) = 1 + + Ef [Y ] + Ef [Y ]... . n 2n 6n3/2 Y 6n2 Y i=1 Y Thus, as n , using the properties of the exponential function, which give that if an a, then −→ ∞ → a n 1 + n ea, n → we have t2 MZ (t) exp . n −→ 2 INTERPRETATION: Sums of independent and identically distributed random variables have a limiting distribution that is Normal, irrespective of the distribution of the variables. 6.3. MODES OF STOCHASTIC CONVERGENCE 57

6.3 MODES OF STOCHASTIC CONVERGENCE

6.3.1 CONVERGENCE IN DISTRIBUTION

Definition 6.3.1 Consider a sequence Xn , n = 1, 2,..., of random variables and a { } corresponding sequence of cdfs, FX ,FX , ... so that for n = 1, 2,..., FX (x) =P[Xn x] . Suppose 1 2 n ≤ that there exists a cdf, FX , such that for all x at which FX is continuous,

lim FX (x) = FX (x). n n −→∞

Then the sequence Xn converges in distribution to the random variable X with cdf FX . This is denoted { } d Xn X, −→ and FX is the limiting distribution.

Convergence of a sequence of mgfs also indicates convergence in distribution. That is, if for all t at which MX (t) is defined, if as n , we have −→ ∞

MX (t) MX (t) n −→ d then Xn X. −→

Definition 6.3.2 The sequence Xn of random variables converges in distribution to the { } constant c if the limiting distribution of Xn is degenerate at c, that is,

d Xn X −→ and P[X = c] = 1, so that 0, x < c, F (x) = X 1, x c. ≥ This special type of convergence in distribution occurs when the limiting distribution is discrete, with the probability mass function only being non-zero at a single value. That is, if the limiting random variable is X, then

fX (x) = 1, x = c, and zero otherwise.

Theorem 6.3.1 The sequence of random variables Xn converges in distribution to c if and only if, for all > 0, { }

lim P [ Xn c < ] = 1. n −→∞ | − | This theorem indicates that convergence in distribution to a constant c occurs if and only if the probability becomes increasingly concentrated around c as n . −→ ∞ 58 CHAPTER 6. PROBABILITY RESULTS & LIMIT THEOREMS

6.3.2 CONVERGENCE IN PROBABILITY Definition 6.3.3 CONVERGENCE IN PROBABILITY TO A CONSTANT The sequence of random variables Xn converges in probability to the constant c, denoted { } P Xn c −→ if for all > 0

lim P [ Xn c < ] = 1, or, equivalently, lim P [ Xn c ] = 0, n n −→∞ | − | −→∞ | − | ≥ that is, if the limiting distribution of Xn is degenerate at c. Interpretation. Convergence in probability to a constant is precisely equivalent to convergence in distribution to a constant.

d P A very useful result is Slutsky’s Theorem which states that if Xn X and Yn c, where c is d d −→ d −→ a finite constant, then: (i) Xn + Yn X + c, (ii) XnYn cX, (iii) Xn/Yn X/c, if c = 0. −→ −→ −→ 6

Theorem 6.3.2 WEAK LAW OF LARGE NUMBERS 2 Suppose that Xn is a sequence of i.i.d. random variables with expectation μ and variance σ . { } Let Yn be defined by n 1 Y = X . n n i i X=1 Then, for all > 0, lim P [ Yn μ < ] = 1, n −→∞ | − | P that is, Yn μ, and thus the mean of X1,...,Xn converges in probability to μ. −→

Proof. Using the properties of expectation, it can be shown that Yn has expectation μ and variance σ2/n, and hence by the Chebychev Inequality,

σ2 P [ Yn μ ] 0, as n | − | ≥ ≤ n2 −→ −→ ∞ for all > 0. Hence P [ Yn μ < ] 1, as n | − | −→ −→ ∞ P and Yn μ. −→

Definition 6.3.4 CONVERGENCE TO A RANDOM VARIABLE The sequence of random variables Xn converges in probability to the random variable P { } X, denoted Xn X, if, for all > 0, −→

lim P [ Xn X < ] = 1, or, equivalently, lim P [ Xn X ] = 0. n n −→∞ | − | −→∞ | − | ≥ 6.3. MODES OF STOCHASTIC CONVERGENCE 59

6.3.3 CONVERGENCE IN QUADRATIC MEAN Definition 6.3.5 CONVERGENCE IN QUADRATIC MEAN The sequence of random variables Xn converges in quadratic mean (also called L2 { } qm convergence) to the random variable X, denoted Xn X, if −→ 2 E(Xn X) 0, − → as n . −→ ∞

Theorem 6.3.3 For the sequence Xn of random variables, { } (a) Convergence in quadratic mean to a random variable implies convergence in probability:

qm P Xn X = Xn X. −→ ⇒ −→ (b) Convergence in probability to a random variable implies convergence in distribution:

P d Xn X = Xn X. −→ ⇒ −→

qm Proof. The proof of (a) is simple. Suppose that Xn X. Fix > 0. Then, by Markov’s inequality, −→ 2 2 2 E(Xn X) P ( Xn X > ) = P ((Xn X) > ) − 0. | − | − ≤ 2 −→

Proof of (b). Fix > 0 and let x be a continuity point of FX . Then

FX (x) = P (Xn x, X x + ) + P (Xn x, X > x + ) n ≤ ≤ ≤ P (X x + ) + P ( Xn X > ) ≤ ≤ | − | = FX (x + ) + P ( Xn X > ). | − | Also,

FX (x ) = P (X x ) = P (X x , Xn x) + P (X x , Xn > x) − ≤ − ≤ − ≤ ≤ − FX (x) + P ( Xn X > ). ≤ n | − | Hence, FX (x ) P ( Xn X > ) FX (x) FX (x + ) + P ( Xn X > ). − − | − | ≤ n ≤ | − | Take the limit as n to conclude that −→ ∞

FX (x ) lim inf FXn (x) lim sup FXn (x) FX (x + ). − ≤ n ≤ n ≤ −→∞ −→∞ This holds for all > 0. Take the limit as 0 and use the fact that FX is continuous at x to conclude that −→ lim FX (x) = FX (x). n n −→∞ Note that the reverse implications do not hold. Convergence in probability does not imply convergence in quadratic mean. Also, convergence in distribution does not imply convergence in probability. 60 CHAPTER 6. PROBABILITY RESULTS & LIMIT THEOREMS CHAPTER 7 STATISTICAL ANALYSIS

7.1 STATISTICAL SUMMARIES

Definition 7.1.1 A collection of independent, identically distributed random variables X1, ..., Xn each of which has distribution defined by cdf FX (or mass/density function fX ) is a random sample of size n from FX (or fX ).

Definition 7.1.2 A function, T , of a random sample, X1, ..., Xn, that is, T = t(X1, ..., Xn) that depends only on X1, ..., Xn is a statistic. A statistic is a random variable. For example, the sample mean X + X + ... + X Xˉ = 1 2 n n is a statistic. But a statistic T need not necessarily be constructed from a random sample (the random variables need not be independent, identically distributed), but that is the case encountered most often. In many circumstances it is necessary to consider statistics constructed from a collection of independent, but not identically distributed, random variables.

7.2 SAMPLING DISTRIBUTIONS

Definition 7.2.1 If X1, ..., Xn is a random sample from FX , say, and T = t(X1, ..., Xn) is a statistic, then FT (or fT ), the cdf (or mass/density function) of random variable T , is the sampling distribution of T . This notion extends immediately to the case of a statistic T constructed from a general collection of random variables X1, ..., Xn.

2 EXAMPLE: If X1, ..., Xn are independent random variables, with Xi N(μ , σ ) for ∼ i i i = 1, ..., n, and a1, ..., an are constants, consider the distribution of random variable Y defined by

n Y = aiXi. i X=1 Using standard mgf results, the distribution of Y is derived to be normal with parameters

n n 2 2 2 μY = aiμi, σY = ai σi . i i X=1 X=1 Now consider the special case of this result when X1, ..., Xn are independent, identically 2 2 distributed with μi = μ and σi = σ , and where ai = 1/n for i = 1, ..., n. Then

n 1 σ2 Y = X = Xˉ N μ, . n i n i ∼ X=1

61 62 CHAPTER 7. STATISTICAL ANALYSIS

Definition 7.2.2 For a random sample X1, ..., Xn from a probability distribution, then the sample variance, S2, is the statistic defined by

n 2 1 2 S = Xi Xˉ . n 1 − i=1 − X

Theorem 7.2.1 SAMPLING DISTRIBUTION FOR NORMAL SAMPLES 2 If X1, ..., Xn is a random sample from a normal distribution, say Xi N(μ, σ ), then: ∼ (a) Xˉ is independent of Xi X, i = 1, ..., n ; (b) Xˉ and S2 are independent− random variables; (c) The random variable n 2 Xi Xˉ 2 (n 1)S i − − = X=1 σ2 σ2 has a chi-squared distribution with n 1 degrees of freedom. −

Proof. Omitted here.

Theorem 7.2.2 Suppose that X1, ..., Xn is a random sample from a normal distribution, say 2 Xi N(μ, σ ). Then the random variable ∼ Xˉ μ T = − S/√n has a Student-t distribution with n 1 degrees of freedom. −

Proof. Consider the random variables

√n(Xˉ μ) Z = − N(0, 1), σ ∼

2 (n 1)S 2 V = − 2 χn 1, σ ∼ − and Z T = , V n 1 r − and use the properties of the normal distribution and related random variables (NOTE 6, following Definition 4.1.5). Also, see EXERCISES 5, Q4 (b). 7.3. HYPOTHESIS TESTING 63

7.3 HYPOTHESIS TESTING

7.3.1 TESTING FOR NORMAL SAMPLES - THE Z-TEST We concentrate initially on random data samples that we can assume to have a normal distribution, and utilize the Theorem from the previous section. We will look at two situations, 2 namely one sample and two sample experiments. So, we suppose that X1, ..., Xn N μ, σ (one sample) and X , ..., X N(μ , σ2 ), Y , ..., Y N(μ , σ2 ) (two sample): in the∼ latter case 1 n X X 1 n Y Y we assume also independence∼ of the two samples. ∼

ONE SAMPLE Possible tests of interest are: μ = μ , σ = σ0 for some specified constants • 0 μ0 and σ0.

TWO SAMPLE Possible tests of interest are: μ = μ , σX = σY . • X Y 2 Recall from Theorem 7.2.1 that, if X1, ..., Xn N(μ, σ ) are the i.i.d. outcome random variables of n experimental trials, then ∼

2 2 ˉ σ (n 1) S 2 X N μ, and − 2 χn 1, ∼ n σ ∼ − ˉ 2 with X and S statistically independent. Suppose we want to test the hypothesis that μ = μ0, for some specified constant μ0, (for example, μ0 = 20.0) is a plausible model. More specifically, we want to test

H0 : μ = μ0, the NULL hypothesis, against H1 : μ = μ , the ALTERNATIVE hypothesis. 6 0

So, we want to test whether H0 is true, or whether H1 is true. In the case of a Normal sample, the distribution of Xˉ is Normal, and

σ2 Xˉ μ Xˉ N μ, = Z = − N (0, 1) , ∼ n ⇒ σ/√n ∼ where Z is a random variable. Now, when we have observed the data sample, we can calculate x, and therefore we have a way of testing whether μ = μ0 is a plausible model; we calculate x from x1, ..., xn, and then calculate x μ z = − 0 . σ/√n

If H0 is true, and μ = μ0, then the observed z should be an observation from an N(0, 1) distribution (as Z N(0, 1)), that is, it should be near zero with high probability. In fact, z should lie between∼ -1.96 and 1.96 with probability 1 α = 0.95, say, as − P [ 1.96 Z < 1.96] = Φ(1.96) Φ( 1.96) = 0.975 0.025 = 0.95. − ≤ − − − If we observe z to be outside of this range, then there is evidence that H0 is not true. So, basically, if we observe an extreme value of z, either H0 is true, but we have observed a rare event, or we prefer to disbelieve H0 and conclude that the data contains evidence against H0. Notice the asymmetry between H0 and H1. The null hypothesis is ‘conservative’, reflecting perhaps a current state of belief, and we are testing whether the data is consistent with that 64 CHAPTER 7. STATISTICAL ANALYSIS

Figure 7.1: CRITICAL REGION IN A Z-TEST (taken from Schaum’s ELEMENTS OF STATIS- TICS II, Bernstein & Bernstein.

hypothesis, only rejecting H0 in favour of the alternative H1 if evidence is clear i.e. when the data represent a rare event under H0. As an alternative approach, we could calculate the probability p of observing a z value that is more extreme than the z we did observe; this probability is given by

2Φ(z), z < 0, p = 2(1 Φ(z)), z 0. − ≥

If this p is very small, say p α = 0.05, then again there is evidence that H0 is not true. This approach is called significance≤ testing. In summary, we need to assess whether z is a surprising observation from an N(0, 1) distribution - if it is, then we reject H0. Figure 6.1 depicts the ”critical region” in a Z-test.

7.3.2 HYPOTHESIS TESTING TERMINOLOGY There are five crucial components to a hypothesis test, namely:

the TEST STATISTIC; • the NULL DISTRIBUTION of the statistic; • the SIGNIFICANCE LEVEL of the test, usually denoted by α; • the P-VALUE, denoted p; • CRITICAL VALUE(S) of the test. • 7.3. HYPOTHESIS TESTING 65

In the Normal example given above, we have that:

z is the test statistic; • The distribution of random variable Z if H0 is true is the null distribution; • α = 0.05 is the significance level of the test (choosing α = 0.01 gives a “stronger” test); • p is the p-value of the test statistic under the null distribution; • the solution CR of Φ(CR) = 1 α/2 gives the critical values of the test CR. These • critical values define the boundary− ofa critical region: if the value z is in± the critical region we reject H0.

7.3.3 THE t-TEST In practice, we will often want to test hypotheses about μ when σ is unknown. We cannot perform the Z-test, as this requires knowledge of σ to calculate the z statistic. Recall that we know the sampling distributions of Xˉ and S2, and that the two estimators are statistically independent. Now, from the properties of the Normal distribution, if we have independent random variables Z N(0, 1) and Y χ2, then we know that random variable T defined by ∼ ∼ ν Z T = Y/ν has a Student-t distribution with ν degrees of freedom.p Using this result, and recalling the sampling distributions of Xˉ and S2, we see that

Xˉ μ − σ/√n (Xˉ μ) T = = − tn 1 : (n 1)S2/σ2 S/√n ∼ − − s (n 1) − T has a Student-t distribution with n 1 degrees of freedom, denoted St(n 1), which does not depend on σ2. Thus we can repeat the− procedure used in the σ known case,− but use the sampling distribution of T rather than that of Z to assess whether the value of the test statistic is “surprising” or not. Specifically, we calculate the observed value

(x μ) t = − s/√n and find the critical values fora α = 0.05 test by finding the ordinates corresponding to the0.025 and 0.975 percentiles of a Student-t distribution, St(n 1) (rather than a N(0, 1)) distribution. −

7.3.4 TEST FOR σ The Z-test and t-test are both tests for the parameter μ. To perform a test about σ, say

H0 : σ = σ0,

H1 : σ = σ0, 6 66 CHAPTER 7. STATISTICAL ANALYSIS we construct a test based on the estimate of variance, S2. In particular, we saw from Theorem 7.2.1 that the random variable Q, defined by

2 (n 1)S 2 Q = − 2 χn 1, σ ∼ − if the data have an N(μ, σ2) distribution. Hence if we define test statistic value q by

(n 1)s2 q = − σ0

2 then we can compare q with the critical values derived from a χn 1 distribution; we look for the 0.025 and 0.975 quantiles - note that the chi-squared distribution− is not symmetric, so we need two distinct critical values.

7.3.5 TWO SAMPLE TESTS It is straightforward to extend the ideas from the previous sections to two sample situations where we wish to compare the distributions underlying two data samples. Typically, we consider 2 sample one, x1, ..., xnX , from a N(μX , σX ) distribution, and sample two, y1, ..., ynY , independently 2 from a N(μY , σY ) distribution, and test the equality of the parameters in the two models. 2 Suppose that the sample mean and sample variance for samples one and two are denoted (x, sX ) 2 and (y, sY ) respectively.

1. First, consider the hypothesis testing problem defined by

H0 : μX = μY , H1 : μ = μ , X 6 Y when σX = σY = σ is known, so the two samples come from normal distributions with the same, known, variance. Now, from the sampling distributions theorem we have, under H0 σ2 σ2 σ2 σ2 Xˉ N μ , , Yˉ N μ , , = Xˉ Yˉ N 0, + , ∼ X n ∼ Y n ⇒ − ∼ n n X Y X Y since Xˉ and Yˉ are independent. Hence by the properties of normal random variables

Xˉ Yˉ Z = − N(0, 1), 1 1 ∼ σ + n n r X Y

if H0 is true, giving us a test statistic z defined by x y z = − , 1 1 σ + n n r X Y which we can compare with the standard normal distribution. If z is a surprising observation from N(0, 1), and lies in the critical region, then we reject H0. This procedure is the Two Sample Z-Test. 7.3. HYPOTHESIS TESTING 67

2. If we can assume that σX = σY , but the common value, σ, say, is unknown, we parallel the one sample t-test by replacing σ by an estimate in the two sample Z-test. First, we obtain 2 an estimate of σ by “pooling” the two samples; our estimate is the pooled estimate, sP , defined by 2 2 2 (nX 1)sX + (nY 1)sY sP = − − , nX + nY 2 − which we then use to form the test statistic t defined by x y t = − . 1 1 s + P n n r X Y It can be shown that if H0 is true then t should be an observation from a Student-t distribution with nX + nY 2 degrees of freedom. Hence we can derive the critical values from the tables of the Student-− t distribution.

3. If σX = σY , but both parameters are known, we can use a similar approach to the one above6 to derive a test statistic z defined by x y z = − , σ2 σ2 X + Y snX nY

which has an N(0, 1) distribution if H0 is true.

4. If σX = σY , but both parameters are unknown, we can use a similar approach to the one above6 to derive a test statistic t defined by x y t = − . s2 s2 X + Y snX nY

The distribution of this statistic when H0 is true is not analytically available, but can be adequately approximated by a Student (m) distribution, where

(w + w )2 m = X Y , w2 w2 X + Y nX 1 nY 1 − − with 2 2 sX sY wX = , wY = . nX nY

Clearly, the choice of test we use depends on whether σX = σY or not. We may test this hypothesis formally; to test H0 : σX = σY , H1 : σX = σY , 6 we compute the test statistic 2 SX Q = 2 , SY 68 CHAPTER 7. STATISTICAL ANALYSIS which has as null distribution the F distribution with (nX 1, nY 1) degrees of freedom. This − − distribution can be denoted F (nX 1, nY 1), and its quantiles are tabulated. Hence we can look up the 0.025 and 0.975 quantiles− of this− distribution (the F distribution is not symmetric), and hence define the critical region. Informally, if the test statistic value q is very small or very large, then it is a surprising observation from the F distribution and hence we reject the hypothesis of equal variances.

HYPOTHESIS TESTING SUMMARY In general, to test a hypothesis H0, we consider a statistic calculated from the sample data. We derive mathematically the probability distribution of the statistic, considered as a random variable, when the hypothesis H0 is true, and compare the actual observed value of the statistic computed from the data sample with the hypothetical probability distribution. We ask the question “Is the value a likely observation from this probability distribution ”? If the answer is “No”, then reject the hypothesis, otherwise accept it.

7.4 POINT ESTIMATION

Definition 7.4.1 Let X1, ..., Xn be a random sample from a distribution with mass/density function fX that depends on a (possibly vector) parameter θ. Then fX1 (x1) = fX (x1; θ), so that

fX1,...,Xk (x1, ..., xk) = fX (xi; θ). i Y=1

A statistic T = t(X1, ..., Xn) that is used to represent or estimate a function τ(θ) of θ based on an observed sample of the random variables x1, ..., xn is an estimator, and t = t(x1, ..., xn) is an estimate,τ ˆ(θ), of τ(θ). The estimator T = t(X1,...,Xn) is said to be unbiased if E(T ) = θ, otherwise T is biased.

7.4.1 ESTIMATION TECHNIQUES I: METHOD OF MOMENTS

Suppose that X1, ..., Xn is a random sample from a probability distribution with mass/density function fX that depends on a vector parameter θ of dimension k, and suppose that a sample x1, ..., xn has been observed. Let the rth moment of fX be denoted μr, and let the rth sample moment, denoted mr be defined for r = 1, 2,... by

n 1 m = xr. r n i i X=1

Then mr is an estimate of μr, and n 1 M = Xr j n i i X=1 is an estimator of μr.

PROCEDURE : The method of moments technique of estimation involves matching the theoretical moments μr μr(θ) to the sample moments mr, r = 1, 2, . . . , l, for suitable l, and solving for θ. In most situations≡ taking l = k, the dimension of θ, suffices: we obtain k equations in the k elements of vector θ which may be solved simultaneously to find the parameter estimates. 7.4. POINT ESTIMATION 69

We may, however, need l > k. Intuitively, and recalling the Weak Law of Large Numbers, it is reasonable to suppose that there is a close relationship between the theoretical properties of a probability distribution and estimates derived from a large sample. For example, we know that, for large n, the sample mean converges in probability to the theoretical expectation.

7.4.2 ESTIMATION TECHNIQUES II: MAXIMUM LIKELIHOOD

Definition 7.4.2 Let random variables X1, ..., Xn have joint mass or density function, denoted fX1,...,Xk , that depends on a vector parameter θ = (θ1, ..., θk). Then the joint/mass density function considered as a function of θ for the (fixed) observed values x1, ..., xn of the variables is the likelihood function, L(θ):

L(θ) = fX1,...,Xn (x1, ..., xn; θ).

If X1, ..., Xn represents a random sample from joint/mass density function fX

n L(θ) = fX (xi; θ). i Y=1 Definition 7.4.3 Let L(θ) be the likelihood function derived from the joint/mass density k function of random variables X1, ..., Xn, where θ Θ R , say, and Θ is termed the parameter ∈ ⊆ space. Then for a fixed set of observed values x1, ..., xn of the variables, the estimate of θ termed the maximum likelihood estimate (MLE) of θ, θ, is defined by

θ = arg maxbL(θ). θ Θ ∈ b That is, the maximum likelihood estimate is the value of θ for which L(θ) is maximized in the parameter space Θ. DISCUSSION : The method of estimation involves finding the value of θ for which L(θ) is maximized. This is generally done by setting the first partial derivatives of L(θ) with respect to θj equal to zero, for j = 1, ..., k, and solving the resulting k simultaneous equations. But we must be alert to cases where the likelihood function L(θ) is not differentiable, or where the maximum occurs on the boundary of Θ! Typically, it is easier to obtain the MLE by maximising the (natural) logarithm of L(θ): we maximise l(θ) = log L(θ), the log-likelihood.

THE FOUR STEP ESTIMATION PROCEDURE: Suppose a sample x1, ..., xn has been obtained from a probability model specified by mass or density function fX (x; θ) depending on parameter(s) θ lying in parameter space Θ. The maximum likelihood estimate is produced as follows;

1. Write down the likelihood function, L(θ).

2. Take the natural log of the likelihood, collect terms involving θ.

3. Find the value of θ Θ, θ, for which log L(θ) is maximized, for example by differentiation. Note that, if parameter∈ space Θ is a bounded interval, then the maximum likelihood estimate may lie on the boundaryb of Θ. If the parameter is a k vector, the maximization involves evaluation of partial derivatives. 70 CHAPTER 7. STATISTICAL ANALYSIS

4. Check that the estimate θ obtained in STEP 3 truly corresponds to a maximum in the (log) likelihood function by inspecting the second derivative of log L(θ) with respect to θ. In the single parameter case, ifb the second derivative of the log -likelihood is negative at θ = θ, then θ is confirmed as the MLE of θ (other techniques may be used to verify that the likelihood is maximized at ˆθ). b b 7.5 INTERVAL ESTIMATION

The techniques of the previous section provide just a single point estimate of the value of an unknown parameter. Instead, we can attempt to provide a set or interval of values which expresses our uncertainty over the unknown value.

Definition 7.5.1 Let X = (X1,...,Xn) be a random sample from a distribution depending on an unknown scalar parameter θ. Let T1 = l1(X1,...,Xn) and T2 = l2(X1,...,Xn) be two statistics satisfying T1 T2 for which P (T1 < θ < T2) = 1 α, where P denotes probability when X has distribution specified≤ by parameter value θ, whatever− the true value of θ, and where 1 α − does not depend on θ. Then the random interval (T1,T2) is called a 1 α confidence interval − for θ, and T1 and T2 are called lower and upper confidence limits, respectively.

Given data, x = (x1, . . . , xn), the realised value of X, we calculate the interval (t1, t2), where t1 = l1(x1, . . . , xn) and t2 = l2(x1, . . . , xn). NOTE: θ here is fixed (it has some true, fixed but unknown value in Θ), and it is T1,T2 that are random. The calculated interval (t1, t2) either does, or does not, contain the true value of θ. Under repeated sampling of X, the random interval (T1,T2) contains the true value a proportion 1 α of the time. Typically, we will take α = 0.05 or α = 0.01, corresponding to a 95% or 99% confidence− interval.

If θ is a vector, then we use a confidence set, such as a sphere or ellipse, instead of an interval. So, C(X) is a 1 α confidence set for θ if P (θ C(X)) = 1 α, for all possible θ. − ∈ − The key problem is to develop methods for constructing a confidence interval. There are two general procedures.

7.5.1 PIVOTAL QUANTITY

Definition 7.5.2 Let X = (X1,...,Xn) be a random sample from a distribution specified by (scalar) parameter θ. Let Q = q(X; θ) be a function of X and θ. If Q has a distribution that does not depend on θ, then Q is defined to be a pivotal quantity.

If Q = q(X; θ) is a pivotal quantity and has a probability density function, then for any fixed 0 < α < 1, there will exist q1 and q2 depending on α such that P (q1 < Q < q2) = 1 α. Then, if − for each possible sample value x = (x1, . . . , xn), q1 < q(x1, . . . , xn; θ) < q2 if and only if l1(x1, . . . , xn) < θ < l2(X1, . . . , xn), for functions l1 and l2, not depending on θ, then (T1,T2) is a 1 α confidence interval for θ, where Ti = li(X1,...,Xn), i = 1, 2. − 7.5. INTERVAL ESTIMATION 71

7.5.2 INVERTING A TEST STATISTIC

A second method utilises a correspondence between hypothesis testing and interval estimation.

Definition 7.5.3 For each possible value θ0, let A(θ0) be the acceptance region of a test of H0 : θ = θ0, of significance level α, so that A(θ0) is the set of data values x such that H0 is accepted in a test of significance level α. For each x, define a set C(x) by

C(x) = θ0 : x A(θ0) . { ∈ } Then the random set C(X) is a 1 α confidence set. − 2 2 Example. Let X = (X1,...,Xn), with the Xi IID N(μ, σ ), with σ known. Both the above ˉ ˉ procedures yield a 1 α confidence interval for μ of the form (X zα/ σ/√n, X + zα/ σ/√n), − − 2 2 where Φ(zα/ ) = 1 α/2. 2 − We know that Xˉ μ − N(0, 1), σ/√n ∼ and so is a pivotal quantity. We have

Xˉ μ P ( zα/ < − < zα/ ) = 1 α. − 2 σ/√n 2 − Rearranging gives ˉ ˉ P (X zα/ σ/√n < μ < X + zα/ σ/√n) = 1 α, − 2 2 − which defines a 1 α confidence interval. −

If we test H0 : μ = μ against H1 : μ = μ a reasonable test has rejection region of the form 0 6 0 x : xˉ μ > zα/ σ/√n . So, H0 is accepted for sample points with xˉ μ < zα/ σ/√n, or, { | − 0| 2 } | − 0| 2

xˉ zα/ σ/√n < μ < xˉ + zα/ σ/√n. − 2 0 2

The test is constructed to have size (significance level) α, so P (H0 is accepted μ = μ0) = 1 α. So we can write | − ˉ ˉ P (X zα/ σ/√n < μ < X + zα/ σ/√n μ = μ ) = 1 α. − 2 0 2 | 0 −

This is true for every μ0, so ˉ ˉ P (X zα/ σ/√n < μ < X + zα/ σ/√n) = 1 α − 2 2 − is true for every μ, so the confidence interval obtained by inverting the test is the same asthat derived by the pivotal quantity method.