Bayesian logistic regression and Laplace approximations So far we have only performed Bayesian inference in two particularly tractable situations: 1) a small discrete problem (the card game); and 2) “linear-Gaussian models”, where the observations are linear combinations of variables with Gaussian beliefs, to which we add Gaussian noise. For most models, we cannot compute the equations for Bayesian prediction exactly. Logistic regression will be our working example. We’ll look at how Bayesian predictions differ from regularized maximum likelihood. Then we’ll look at approximate computation of the integrals, starting with the Laplace approximation. Logistic regression As a quick review, the logistic regression model gives the probability of a binary label given a feature vector: w x P(y =1 x, w) = s(w>x) = 1/(1 + e− > ). j We usually add a bias parameter b to the model, making the probability s(w>x+b). Although often it’s dropped from the presentation, knowing that it can always be added back in, or included in w by including a constant element in the input features x. You’ll see various notations used for the training data . The model gives the probability of D (n) a vector of outcomes y associated with a matrix of inputs X (where the nth row is x >). Maximum likelihood fitting maximizes the probability: (n) (n) (n) (n) (n) P(y X, w) = ∏ s(z w>x ), where z = 2y 1, if y 0, 1 . j n − 2 f g For simplicity we’ll write this likelihood as P( w), even though really only the outputs y D j in the data are modelled. The inputs X are assumed fixed and known. Logistic regression is most frequently fitted by a regularized form of maximum likelihood. For example L2 regularization fits an estimate h i w∗ = arg max log P(y X, w) lw>w . w j − We find a setting of the weights that make the training data appear probable, but discourage fitting extreme settings of the weights, that don’t seem reasonable. (Usually the bias weight will be omitted from the regularization term.) Just as with simple linear regression, we can instead follow a Bayesian approach. The weights are unknown, so predictions are made considering all possible settings, weighted by how plausible they are given the training data. Bayesian logistic regression The posterior distribution over the weights is given by Bayes’ rule: P( w) p(w) p(w ) = D j ∝ P( w) p(w). j D P( ) D j D The normalizing constant is the integral required to make the posterior distribution integrate to one: Z P( ) = P( w) p(w) dw. D D j The figures below1 are for five different plausible sets of parameters, sampled from the pos- terior p(w ).2 Each figure shows the decision boundary s(w x)=0.5 for one parameter j D > vector as a solid line, and two other contours given by w x = 1. > ± 1. The two figures in this section are extracts from Figure 41.7 of MacKay’s textbook (p499). Murphy’s Figures 8.5 and 8.6 contain a similar illustration. 2. It’s not obvious how to generate samples from p(w ), and in fact it’s hard to do exactly. These samples were drawn approximately with a “Markov chain Monte Carlo”j D method. MLPR:w8a Iain Murray, http://www.inf.ed.ac.uk/teaching/courses/mlpr/2016/ 1 The axes in the figures above are the two input features x1 and x2. The model included a bias parameter, and the model parameters were sampled from the posterior distribution given data from the two classes as illustrated. The arrow, perpendicular to the decision boundary, illustrates the direction and magnitude of the weight vector. Assuming that the data are well-modelled by logistic regression, it’s clear that we don’t know what the correct parameters are. That is, we don’t know what parameters we would fit after seeing substantially more data. The predictions given the different plausible weight vectors differ substantially. The Bayesian way to proceed is to use probability theory to derive an expression for the prediction we want to make: Z P(y x, ) = p(y, w x, ) dw j D j D Z = P(y x, w) p(w ) dw. j j D That is, we should average the predictive distributions P(y x, w) for different parameters, j weighted by how plausible those parameters are, p(w ). Contours of this predictive j D distribution, P(y =1 x, ) 0.5, 0.27, 0.73, 0.12, 0.88 , are illustrated in the left panel below. j D 2 f g Predictions become certain less quickly when moving away from the decision boundary when further away from the training inputs. That’s because the different predictors above disagreed in regions far from the data. 10 10 5 5 (a) (b) 0 0 0 5 10 0 5 10 Again, the axes are the input features x and x . The right hand figure shows P(y =1 x, w ) 1 2 j ∗ for some fitted weights w∗. No matter how these fitted weights are chosen, the contours have to be linear. The parallel contours mean that the uncertainty of predictions falls at the same rate when moving away from the decision boundary, no matter how far we are from the training inputs. It’s common to describe L2 regularized logistic regression as MAP (Maximum a posteriori) estimation with a Gaussian (0, s2 ) prior on the weights. The “most probable”3 weights, N w coincide with an L2 regularized estimate: 1 w∗ = arg max [log p(w )] = arg max log P( w) 2 w>w w j D w D j − 2sw MAP estimation is not a “Bayesian” procedure. The rules of probability theory don’t tell us to fix an unknown parameter vector to an estimate. We could view MAP as an approximation 3. “Most probable” is problematic for real-valued parameters. Really we are picking the weights with the highest probability density. But those weights aren’t well-defined, because if we consider a non-linear reparameterization of the weights, the maximum of the pdf will be in a different place. That’s why I prefer to describe estimating the weights as “regularized maximum likelihood” or “penalized maximum likelihood” rather than MAP. MLPR:w8a Iain Murray, http://www.inf.ed.ac.uk/teaching/courses/mlpr/2016/ 2 to the Bayesian procedure, but the figure above illustrates that it is a crude one: the Bayesian predictions (left) are qualitatively different to the MAP ones (right). Unfortunately, we can’t evaluate the integral for predictions P(y x, ) in closed form. j D Making model choices for Bayesian logistic regression is also computationally challenging. The marginal probability of the data, P( ), is the marginal likelihood of the model, which D we might write as P( ) when we are evaluating some model choices (such as basis D j M M functions and hyperparameters). We also can’t evaluate the integral for P( ) in closed form. D The logistic regression posterior is sometimes approximately Gaussian We’re able to do some integrals involving Gaussian distributions. The posterior distribution over the weights p(w ) is not Gaussian, but we can make progress if we can approximate j D it with a Gaussian. First, I’ve contrived an example to illustrate how the posterior over the weights can look non-Gaussian. We have a Gaussian prior with one sigmoidal likelihood term. Here we assume we know the bias4 is 10, and we have one datapoint with y =1 at x = 20: − p(w) ∝ (w; 0, 1) N p(w ) ∝ (w; 0, 1) s(10 20w). j D N − We are now fairly sure that the weight isn’t a large positive value, because otherwise we’d have probably seen y =0. We (softly) slice off the positive region and renormalize to get the posterior distribution illustrated below: p(w) p(w ) |D −4 −2 0 2 4 w The distribution is asymmetric and so clearly not Gaussian. Every time we multiply the posterior by a sigmoidal likelihood, we softly carve away half of the weight space in some direction as less likely. While the posterior distribution has no neat analytical form, the distribution over plausible weights often does look Gaussian after many observations. As another example, I generated N = 500 labels, z(n) , from a logistic regression model f g with no bias and with w =1 at x(n) (0, 102). Then, ∼ N p(w) ∝ (w; 0, 1) N 500 p(w ) ∝ (w; 0, 1) ∏ s(wx(n)z(n)), z(n) 1 . j D N n=1 2 f± g The posterior now appears to be a beautiful bell-shape: p(w) p(w ) |D −4 −2 0 2 4 w Fitting a Gaussian distribution (using the Laplace approximation, next section) shows that the distribution isn’t quite Gaussian. but it’s close: 4. Perhaps we have many datapoints from which we have fitted the bias precisely, then we have one datapoint that has a novel feature turned on, and the example is showing the posterior over the weight that interacts with that one feature. MLPR:w8a Iain Murray, http://www.inf.ed.ac.uk/teaching/courses/mlpr/2016/ 3 p(w) p(w ) |D (w; w∗,1/H) N −4 −2 0 2 4 w The Laplace Approximation There are multiple ways that we could try to fit a distribution with a Gaussian form. For example, we could try to match the mean and variance of the distribution.