MCMC estimation in MLwiN Version 2.36 by William J. Browne Programming by William J. Browne, Chris Charlton and Jon Rasbash Updates for later versions by William J. Browne, Chris Charlton, Mike Kelly and Rebecca Pillinger Printed 2016 Centre for Multilevel Modelling University of Bristol ii MCMC Estimation in MLwiN version 2.36 © 2016. William J. Browne. No part of this document may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, for any purpose other than the owner's personal use, without the prior written permission of one of the copyright holders. ISBN: 978-0-903024-99-0 Printed in the United Kingdom First printing November 2004 Updated for University of Bristol, October 2005, January 2009, July 2009, August 2011, January 2012, September 2012, August 2014 January 2015 and March 2016. Contents Table of Contents viii Acknowledgements ix Preface to the 2009, 2011, 2012 and 2014 Editions xi 1 Introduction to MCMC Estimation and Bayesian Modelling1 1.1 Bayesian modelling using Markov Chain Monte Carlo methods1 1.2 MCMC methods and Bayesian modelling............2 1.3 Default prior distributions....................4 1.4 MCMC estimation........................5 1.5 Gibbs sampling..........................5 1.6 Metropolis Hastings sampling..................8 1.7 Running macros to perform Gibbs sampling and Metropolis Hastings sampling on the simple linear regression model... 10 1.8 Dynamic traces for MCMC.................... 12 1.9 Macro to run a hybrid Metropolis and Gibbs sampling method for a linear regression example.................. 15 1.10 MCMC estimation of multilevel models in MLwiN....... 18 Chapter learning outcomes....................... 19 2 Single Level Normal Response Modelling 21 2.1 Running the Gibbs Sampler................... 26 2.2 Deviance statistic and the DIC diagnostic........... 28 2.3 Adding more predictors...................... 29 2.4 Fitting school effects as fixed parameters............ 32 Chapter learning outcomes....................... 33 3 Variance Components Models 35 3.1 A 2 level variance components model for the Tutorial dataset. 36 3.2 DIC and multilevel models.................... 41 3.3 Comparison between fixed and random school effects..... 41 Chapter learning outcomes....................... 43 4 Other Features of Variance Components Models 45 4.1 Metropolis Hastings (MH) sampling for the variance compo- nents model............................ 46 4.2 Metropolis-Hastings settings................... 47 4.3 Running the variance components with Metropolis Hastings. 48 iii iv CONTENTS 4.4 MH cycles per Gibbs iteration.................. 49 4.5 Block updating MH sampling.................. 49 4.6 Residuals in MCMC....................... 51 4.7 Comparing two schools...................... 54 4.8 Calculating ranks of schools................... 55 4.9 Estimating a function of parameters............... 59 Chapter learning outcomes....................... 61 5 Prior Distributions, Starting Values and Random Number Seeds 63 5.1 Prior distributions........................ 63 5.2 Uniform on variance scale priors................. 63 5.3 Using informative priors..................... 64 5.4 Specifying an informative prior for a random parameter.... 67 5.5 Changing the random number seed and the parameter starting values............................... 68 5.6 Improving the speed of MCMC Estimation........... 71 Chapter learning outcomes....................... 72 6 Random Slopes Regression Models 73 6.1 Prediction intervals for a random slopes regression model... 77 6.2 Alternative priors for variance matrices............. 80 6.3 WinBUGS priors (Prior 2).................... 80 6.4 Uniform prior........................... 81 6.5 Informative prior......................... 82 6.6 Results............................... 83 Chapter learning outcomes....................... 83 7 Using the WinBUGS Interface in MLwiN 85 7.1 Variance components models in WinBUGS........... 86 7.2 So why have a WinBUGS interface ?.............. 94 7.3 t distributed school residuals................... 94 Chapter learning outcomes....................... 98 8 Running a Simulation Study in MLwiN 99 8.1 JSP dataset simulation study.................. 99 8.2 Setting up the structure of the dataset............. 100 8.3 Generating simulated datasets based on true values...... 104 8.4 Fitting the model to the simulated datasets.......... 109 8.5 Analysing the simulation results................. 112 Chapter learning outcomes....................... 113 9 Modelling Complex Variance at Level 1 / Heteroscedasticity115 9.1 MCMC algorithm for a 1 level Normal model with complex variation.............................. 117 9.2 Setting up the model in MLwiN................. 119 9.3 Complex variance functions in multilevel models........ 123 9.4 Relationship with gender..................... 127 9.5 Alternative log precision formulation.............. 130 CONTENTS v Chapter learning outcomes....................... 132 10 Modelling Binary Responses 133 10.1 Simple logistic regression model................. 134 10.2 Random effects logistic regression model............ 140 10.3 Random coefficients for area type................ 142 10.4 Probit regression......................... 144 10.5 Running a probit regression in MLwiN............. 146 10.6 Comparison with WinBUGS................... 147 Chapter learning outcomes....................... 155 11 Poisson Response Modelling 157 11.1 Simple Poisson regression model................. 159 11.2 Adding in region level random effects.............. 161 11.3 Including nation effects in the model.............. 163 11.4 Interaction with UV exposure.................. 165 11.5 Problems with univariate updating Metropolis procedures... 167 Chapter learning outcomes....................... 169 12 Unordered Categorical Responses 171 12.1 Fitting a first single-level multinomial model.......... 173 12.2 Adding predictor variables.................... 177 12.3 Interval estimates for conditional probabilities......... 179 12.4 Adding district level random effects............... 181 Chapter learning outcomes....................... 184 13 Ordered Categorical Responses 185 13.1 A level chemistry dataset..................... 185 13.2 Normal response models..................... 187 13.3 Ordered multinomial modelling................. 190 13.4 Adding predictor variables.................... 195 13.5 Multilevel ordered response modelling.............. 196 Chapter learning outcomes....................... 200 14 Adjusting for Measurement Errors in Predictor Variables 201 14.1 Effects of measurement error on predictors........... 202 14.2 Measurement error modelling in multilevel models....... 207 14.3 Measurement errors in binomial models............. 210 14.4 Measurement errors in more than one variable and misclassi- fications.............................. 214 Chapter learning outcomes....................... 215 15 Cross Classified Models 217 15.1 Classifications and levels..................... 218 15.2 Notation.............................. 219 15.3 The Fife educational dataset................... 219 15.4 A Cross-classified model..................... 222 15.5 Residuals............................. 225 15.6 Adding predictors to the model................. 227 vi CONTENTS 15.7 Current restrictions for cross-classified models......... 231 Chapter learning outcomes....................... 232 16 Multiple Membership Models 233 16.1 Notation and weightings..................... 234 16.2 Office workers salary dataset................... 234 16.3 Models for the earnings data................... 237 16.4 Fitting multiple membership models to the dataset...... 239 16.5 Residuals in multiple membership models............ 242 16.6 Alternative weights for multiple membership models...... 245 16.7 Multiple membership multiple classification (MMMC) models 246 Chapter learning outcomes....................... 247 17 Modelling Spatial Data 249 17.1 Scottish lip cancer dataset.................... 249 17.2 Fixed effects models....................... 250 17.3 Random effects models...................... 253 17.4 A spatial multiple-membership (MM) model.......... 254 17.5 Other spatial models....................... 257 17.6 Fitting a CAR model in MLwiN................. 257 17.7 Including exchangeable random effects............. 261 17.8 Further reading on spatial modelling.............. 262 Chapter learning outcomes....................... 263 18 Multivariate Normal Response Models and Missing Data 265 18.1 GCSE science data with complete records only......... 266 18.2 Fitting single level multivariate models............. 267 18.3 Adding predictor variables.................... 272 18.4 A multilevel multivariate model................. 273 18.5 GCSE science data with missing records............ 277 18.6 Imputation methods for missing data.............. 282 18.7 Hungarian science exam dataset................. 284 Chapter learning outcomes....................... 288 19 Mixed Response Models and Correlated Residuals 289 19.1 Mixed response models...................... 289 19.2 The JSP mixed response example................ 291 19.3 Setting up a single level mixed response model......... 293 19.4 Multilevel mixed response model................ 296 19.5 Rats dataset............................ 297 19.6 Fitting an autoregressive structure to the variance matrix... 300 Chapter learning outcomes....................... 303 20 Multilevel Factor Analysis Modelling 305 20.1 Factor analysis modelling..................... 305 20.2 MCMC algorithm........................