Lectures on Machine Learning (Fall 2017) Hyeong In Choi Seoul National University Lecture 4: Exponential family of distributions and generalized linear model (GLM) (Draft: version 0.9.2) Topics to be covered: • Exponential family of distributions • Mean and (canonical) link functions • Convexity of log partition function • Generalized linear model (GLM) • Various GLM models 1 Exponential family of distributions In this section, we study a family of probability distribution called the exponential family (of distributions). It is of a special form, but most, if not all, of the well known probability distributions belong to this class. 1 1.1 Definition Definition 1. A probability distribution (PDF or PMF) is said to belong to the exponential family of distributions (in natural or canonical form) if it is of the form. 1 P (y) = h(y)eθ·T (y); (1) θ Z(θ) m k where y = (y1; ··· ; ym) is a point in R ; and θ = (θ1; ··· ; θk) 2 R is a parameter called the canonical (natural) parameter; T : Rm ! Rk is R θ·T (y) a map T (y) = (T1(y); ··· ;Tk(y)); and Z(θ) = h(y)e dy is called the partition function, while its logarithm, A(θ) = log Z(θ); is called the log partition (cumulant) function. Remark. In this lecture and throughout this course, the \dot" notation as in θ · T (y) always means the inner (dot) product of two vectors. Equivalently, Pθ(y) can be written in the form Pθ(y) = exp[θ · T (y) − A(θ) + C(y)]; (2) where C(y) = log h(y). More generally, one sometimes introduces an extra parameter φ, called the dispersion parameter, to control the shape of Pθ(y) by θ · T (y) − A(θ) P (y) = exp + C(y; φ) : θ φ 1.2 Standing assumption In our discussion on the exponential family of distribution, we always assume the following. • In case Pθ(y) is a probability density function (PDF), it is assumed to be continuous as a function of y. It means that there is no singularity in the probability measure Pθ(y); • In case Pθ(y) is a probability mass function (PMF), there exists a range of discrete value of Pθ(y) that is the same for all θ and for all y. If Pθ(y) satisfies either condition, we say it is regular, which is always as- sumed throughout this course. 2 Remark. Sometimes people use the more general form of Pθ(y) to write 1 P (y) = h(y)eη(θ)·T (y): θ Z(θ) But most of the time the same result can be obtained without the use of the general form η(θ): So it is not much of a loss of generality to stick to our convention of using just θ: 1.3 Examples Let us now look at a few illustrative examples. (1) Bernoulli(µ) The Bernoulli distribution is perhaps the simplest in the exponential family. Let Y be a random variable taking its binary value in f0; 1g. Let µ = P [Y = 1]: Its distribution (in fact, the probability mass function, PMF) is then succinctly written as P (y) = µy(1 − µ)1−y: Then, P (y) = µy(1 − µ)1−y = exp[y log µ + (1 − y) log(1 − µ)] µ = exp y log + log(1 − µ) : 1 − µ µ Letting T (y) = y and θ = log and recalling the definitions of 1 − µ logit and σ functions given in Lecture 3, we have µ logit(µ) = log : 1 − µ Thus θ = logit(µ): (3) Its inverse function is the sigmoid function: µ = σ(θ); (4) where 1 σ(θ) = : 1 + e−θ 3 Therefore, we have A(θ) = − log(1 − µ) = log(1 + eθ): (5) Thus Pθ(y); written in canonical form as in (2), becomes θ Pθ(y) = exp[θ · y − log(1 + e )]: (2) Exponential distribution The exponential distribution is a distribution that models the indepen- dent arrival time. Its distribution (the probability density function, PDF) is given as −θy Pθ(y) = θe I(x ≥ 0): To put it in the exponential family form, we use the same θ as the canonical parameter and we let T (y) = −y and h(y) = I(y ≥ 0): Since 1 Z Z(θ) = = e−θy (y ≥ 0)dy; θ I it is already in the canonical form given as in (1). (3) Normal distribution The normal (Gaussian) distribution given by 1 (y − µ)2 P (y) = p exp − 2πσ2 2σ2 is the single most well known distribution. As far as its relation with the exponential family is concerned there are two views. (Here, this σ is a number, not the sigmoid function.) • 1st view (σ2 as a dispersion parameter) This is the case when the main focus is the mean. In this case, the variance σ2 is regarded as known or as a parameter that can be fiddled with as if known. Writing P (y) in the form 2 3 1 2 1 2 − y + yµ − µ 1 P (y) = exp 6 2 2 − log(2πσ2)7 ; θ 4 σ2 2 5 4 One can see right away it is in the form of the exponential family if we set θ = (θ1; θ2) = (µ, 1) 1 T (y) = (y; − y2) 2 φ = σ2 1 1 A(θ) = µ2 = θ2 2 2 1 1 C(y; φ) = − log(2πσ2): 2 • 2nd view (φ = 1) When both µ and σ are parameters to be treated as unknown, we take this point of view. In here, we set the dispersion parameter φ = 1: Writing out Pθ(y) we have 1 µ 1 1 P (y) = exp − y2 + y − µ2 − log σ − log 2π : θ 2σ2 σ2 2σ2 2 Thus it is easy to see the following: 1 T (y) = (y; y2) 2 µ 1 θ = (θ ; θ ) = ( ; − ) 1 2 σ2 σ2 2 1 2 1 θ1 1 A(θ) = 2 µ + log σ = − − log(−θ2) 2σ 2 θ2 2 1 C(y) = − log(2π): 2 1.4 Properties of exponential family The log partition function A(θ) plays a key role, so let us now look at it more carefully. First, since Z Z θ · T (y) − A(θ) P (y)dy = exp + C(y; φ) dy = 1; θ φ 5 taking rθ; we have Z θ · T (y) − A(θ) T (y) − r A(θ) exp + C(y; φ) θ dy = 0: φ φ Thus Z θ · T (y) − A(θ) Z θ · T (y) − A(θ) exp + C(y; φ) T (y)dy = exp + C(y; φ) r A(θ)dy φ φ θ Z θ · T (y) − A(θ) = r A(θ) exp + C(y; φ) dy: θ φ Since Z θ · T (y) − A(θ) exp + C(y; φ) dy = 1; φ we have the following: Proposition 1. Z rθA(θ) = Pθ(y)T (y)dy = E[T (Y )]; where Y is the random variable with distribution Pθ(y): Writing the component, we have @A Z θ · T (y) − A(θ) = exp + C(y; φ) Ti(y)dy: @θi φ Taking the second partial derivative, we have @2A 1 Z θ · T (y) − A(θ) @A(θ) = exp + C(y; φ) Tj(y) − Ti(y)dy @θi@θj φ φ @θj 1 Z θ · T (y) − A(θ) = exp + C(y; φ) T (y)T (y)dy φ φ i j @A(θ) Z θ · T (y) − A(θ) − exp + C(y; φ) Ti(y)dy : @θj φ 6 Using Proposition 1 once more, we have @2A 1 n o = E[Ti(Y )Tj(Y )] − E[Ti(Y )]E[Tj(Y )] @θi@θj φ 1 h i = E T (Y ) − E[T (Y )] T (Y ) − E[T (Y )] φ i i j j 1 = Cov(T (Y );T (Y )): φ i j Therefore we have the following result on the Hessian matrix of A: 2 2 @ A 1 1 Proposition 2. Dθ A = = Cov(T (Y );T (Y )) = Cov(T (Y )); @θi@θj φ φ where Y is the random variable with distribution Pθ(y): Here, Cov(T (Y );T (Y )) denotes the covariance matrix of T (Y ) with itself, which is sometimes called the variance matrix. Since the covariance matrix is always positive semi-definite, we have Corollary 1. A(θ) is a convex function of θ: Remark. In most exponential family models, Cov(T (Y )) is positive definite, in which case A(θ) is strictly convex. In this lecture and throughout the course, we always assume that Cov(T (Y )) is positive definite and therefore that A(θ) is strictly convex. 1.5 Maximum likelihood estimation (i) N (i) m Let D = fy gi=1 be a given data of IID samples, where y 2 R : Then its likelihood function L(θ) and the log likelihood function l(θ) are given by N Y θ · T (y(i)) − A(θ) L(θ) = exp + C(y(i); φ) φ i=1 N N 1 X X l(θ) = θ · T (y(i)) − NA(θ) + C(y(i); φ): φ i=1 i=1 7 Since A(θ) is strictly convex, l(θ) has a unique maximum at θb that is the unique solution of rθl(θ) = 0: Note that N 1 X r l(θ) = T (y(i)) − Nr A(θ) : θ φ θ i=1 So we have the following: Proposition 3. There is a unique θb that maximizes l(θ) at which N 1 X r A(θ)j = T (y(i)): θ θ=θb N i=1 1.5.1 Example: Bernoulli Distribution Let us apply the above argument to the Bernoulli distribution Bernoulli(µ) and see what it leads to.