Bayesian analysis Outline General idea The method Introduction to Bayesian Analysis using Fundamental equation MCMC Stata Stata tools bayes: - bayesmh Linear regression Gustavo Sánchez bayesstats ess bayesgraph Multiple chains StataCorp LLC Postestimation Radom-effects probit Virtual|November 19, 2020 Random effects Convervence Swiss Stata Conference Bayesian predictions Summary References Bayesian analysis Outline Outline 1 Bayesian analysis: Basic concepts General idea • The general idea The method Fundamental • The method equation MCMC Stata tools 2 The Stata tools bayes: - bayesmh • The general command bayesmh Linear • regression The bayes prefix bayesstats ess • Postestimation commands bayesgraph Multiple chains • New in Stata 16 Postestimation • Multiple chains Radom-effects • probit Bayes predictions Random effects Convervence 3 A few examples Bayesian predictions • Linear regression Summary • Random-effects probit model References • Population mean Bayesian analysis The general idea Outline General idea The method Fundamental equation MCMC Stata tools bayes: - bayesmh Linear regression bayesstats ess bayesgraph Multiple chains Postestimation Radom-effects probit Random effects Convervence Bayesian predictions Summary References Bayesian analysis The general idea Outline General idea The method Fundamental equation MCMC Stata tools bayes: - bayesmh Linear regression bayesstats ess bayesgraph Multiple chains Postestimation Radom-effects probit Random effects Convervence Bayesian predictions Summary References Bayesian analysis Bayesian Analysis vs. Frequentist Analysis Outline General idea Frequentist Analysis Bayesian Analysis The method Fundamental equation MCMC • Estimates unknown fixed • Probability distributions for Stata tools parameters. unknown random bayes: - bayesmh parameters. Linear • The data come from a • The data are fixed. regression random sample bayesstats ess bayesgraph (hypothetically repeatable). Multiple chains • Postestimation • Uses data to estimate Combines data with prior unknown fixed parameters. beliefs to get updated Radom-effects probit probability distributions for Random effects the parameters. Convervence • p-values are conditional • It allows formulating Bayesian probabilistic statements for predictions probability statements that assume Ho to be true. the hypothesis of interest. Summary "Conclusions are based on the "Bayesian analysis answers References distribution of statistics derived questions based on the distribution from random samples, assuming of parameters conditional on the unknown but fixed parameters." observed sample." Bayesian analysis bayes Outline Stata’s convenient syntax: : General idea regress y x1 x2 x3 The method Fundamental equation MCMC bayes: regress y x1 x2 x3 Stata tools bayes: - bayesmh Linear regression logit y x1 x2 x3 bayesstats ess bayesgraph Multiple chains Postestimation bayes: logit y x1 x2 x3 Radom-effects probit Random effects Convervence Bayesian mixed y x1 x2 x3 || region: predictions Summary bayes: mixed y x1 x2 x3 || region: References Bayesian analysis Outline General idea The method Fundamental equation MCMC Stata tools bayes: - bayesmh Linear The method regression bayesstats ess bayesgraph Multiple chains Postestimation Radom-effects probit Random effects Convervence Bayesian predictions Summary References Bayesian analysis The method (Fundamental Equation) • Inverse law of probability (Bayes’ Theorem): Outline p (yjθ) p (θ) f (y; θ) π (θ) General idea p (θjy) = = The method p (y) f (y) Fundamental equation MCMC Where: Stata tools f (y; θ): probability density function for y given θ. bayes: - bayesmh π (θ): prior distribution for θ Linear regression • The marginal distribution of y, f(y), does not depend on θ; so bayesstats ess bayesgraph we can write the fundamental equation for Bayesian analysis: Multiple chains Postestimation Radom-effects p (θjy) / L (θ; y) π (θ) probit Random effects Convervence Where: Bayesian L (θ; y): likelihood function of the parameters given the data. predictions Summary References Bayesian analysis The method (Fundamental Equation) • Inverse law of probability (Bayes’ Theorem): Outline p (yjθ) p (θ) f (y; θ) π (θ) General idea p (θjy) = = The method p (y) f (y) Fundamental equation MCMC Where: Stata tools f (y; θ): probability density function for y given θ. bayes: - bayesmh π (θ): prior distribution for θ Linear regression • The marginal distribution of y, f(y), does not depend on θ; so bayesstats ess bayesgraph we can write the fundamental equation for Bayesian analysis: Multiple chains Postestimation Radom-effects p (θjy) / L (θ; y) π (θ) probit Random effects Convervence Where: Bayesian L (θ; y): likelihood function of the parameters given the data. predictions Summary References Bayesian analysis The method (Fundamental Equation) • Inverse law of probability (Bayes’ Theorem): Outline p (yjθ) p (θ) f (y; θ) π (θ) General idea p (θjy) = = The method p (y) f (y) Fundamental equation MCMC Where: Stata tools f (y; θ): probability density function for y given θ. bayes: - bayesmh π (θ): prior distribution for θ Linear regression • The marginal distribution of y, f(y), does not depend on θ; so bayesstats ess bayesgraph we can write the fundamental equation for Bayesian analysis: Multiple chains Postestimation Radom-effects p (θjy) / L (θ; y) π (θ) probit Random effects Convervence Where: Bayesian L (θ; y): likelihood function of the parameters given the data. predictions Summary References Bayesian analysis The method Outline • Let’s assume that both the data and the prior beliefs General idea are normally distributed: The method • 2 Fundamental The data: y ∼ N θ; σd equation MCMC • 2 The prior: θ ∼ N µp; σp Stata tools bayes: - bayesmh • Homework...: Doing the algebra with the fundamental Linear regression equation, we find that the posterior distribution would bayesstats ess bayesgraph be normal with (see, for example, Cameron & Trivedi Multiple chains 2005): Postestimation Radom-effects 2 probit • The posterior: θjy ∼ N µ, σ Random effects Convervence Bayesian Where: predictions 2 2 2 µ = σ Ny¯/σd + µp/σp Summary −1 References 2 2 2 σ = N/σd + 1/σp Bayesian analysis Example (Prior distributions) Outline General idea The method Fundamental equation MCMC Stata tools bayes: - bayesmh Linear regression bayesstats ess bayesgraph Multiple chains Postestimation Radom-effects probit Random effects Convervence Bayesian predictions Summary References Bayesian analysis Example (Posterior distributions) Outline General idea The method Fundamental equation MCMC Stata tools bayes: - bayesmh Linear regression bayesstats ess bayesgraph Multiple chains Postestimation Radom-effects probit Random effects Convervence Bayesian predictions Summary References Bayesian analysis Outline The method (MCMC) General idea • The previous example has a closed-form solution. The method Fundamental equation • What about the cases with non-closed solutions or MCMC Stata tools more complex distributions? bayes: - bayesmh • Integration is performed via simulation. Linear • We need to use intensive computational simulation regression bayesstats ess tools to find the posterior distribution in most cases. bayesgraph Multiple chains Postestimation • Markov chain Monte Carlo (MCMC) methods are the Radom-effects current standard in most software. Stata implements probit two alternatives: Random effects Convervence • Metropolis–Hastings (MH) algorithm Bayesian • Gibbs sampling predictions Summary References Bayesian analysis Outline General idea The method The method Fundamental • Links for Bayesian analysis and MCMC on our YouTube equation MCMC channel: Stata tools • Introduction to Bayesian statistics, part 1: The basic bayes: - bayesmh concepts. Linear regression bayesstats ess bayesgraph https://www.youtube.com/watch?v=0F0QoMCSKJ4&feature=youtu.be Multiple chains Postestimation • Radom-effects Introduction to Bayesian statistics, part 2: MCMC and probit the Metropolis–Hastings algorithm. Random effects Convervence Bayesian https://www.youtube.com/watch?v=OTO1DygELpY&feature=youtu.be predictions Summary References Bayesian analysis The method Outline • Monte Carlo simulation General idea The method Fundamental equation MCMC Stata tools bayes: - bayesmh Linear regression bayesstats ess bayesgraph Multiple chains Postestimation Radom-effects probit Random effects Convervence Bayesian predictions Summary References Bayesian analysis The method • Metropolis–Hastings simulation Outline • The trace plot illustrates the sequence of accepted General idea proposal states for a simulation with not enough burnin The method iterations. Fundamental equation MCMC Stata tools bayes: - bayesmh Linear regression bayesstats ess bayesgraph Multiple chains Postestimation Radom-effects probit Random effects Convervence Bayesian predictions Summary References Bayesian analysis The method • Metropolis–Hastings simulation Outline • The trace plot illustrates the sequence of accepted General idea proposal states for a simulation with enough burnin The method iterations. Fundamental equation MCMC Stata tools bayes: - bayesmh Linear regression bayesstats ess bayesgraph Multiple chains Postestimation Radom-effects probit Random effects Convervence Bayesian predictions Summary References Bayesian analysis The method Outline • We expect to obtain a stationary sequence when General idea convergence is achieved. The method Fundamental equation MCMC Stata tools bayes: - bayesmh Linear regression bayesstats ess bayesgraph Multiple chains Postestimation Radom-effects probit Random