DOCSLIB.ORG

Generalized Linear Models I

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Generalized Linear Models I Statistics 203: Introduction to Regression and Analysis of Variance Generalized Linear Models I Jonathan Taylor - p. 1/18 Today's class ● Today's class ■ Logistic regression. ● Generalized linear models ● Binary regression example ■ ● Binary outcomes Generalized linear models. ● Logit transform ■ ● Binary regression Deviance. ● Link functions: binary regression ● Link function inverses: binary regression ● Odds ratios & logistic regression ● Link & variance fns. of a GLM ● Binary (again) ● Fitting a binary regression GLM: IRLS ● Other common examples of GLMs ● Deviance ● Binary deviance ● Partial deviance tests 2 ● Wald χ tests - p. 2/18 Generalized linear models ● Today's class ■ All models we have seen so far deal with continuous ● Generalized linear models ● Binary regression example outcome variables with no restriction on their expectations, ● Binary outcomes ● Logit transform and (most) have assumed that mean and variance are ● Binary regression ● Link functions: binary unrelated (i.e. variance is constant). regression ● Link function inverses: binary ■ Many outcomes of interest do not satisfy this. regression ● Odds ratios & logistic ■ regression Examples: binary outcomes, Poisson count outcomes. ● Link & variance fns. of a GLM ● Binary (again) ■ A Generalized Linear Model (GLM) is a model with two ● Fitting a binary regression GLM: IRLS ingredients: a link function and a variance function. ● Other common examples of GLMs ◆ The link relates the means of the observations to ● Deviance ● Binary deviance predictors: linearization ● Partial deviance tests 2 ◆ ● Wald χ tests The variance function relates the means to the variances. - p. 3/18 Binary regression example ● Today's class ■ A local health clinic sent fliers to its clients to encourage ● Generalized linear models ● Binary regression example everyone, but especially older persons at high risk of ● Binary outcomes ● Logit transform complications, to get a flu shot in time for protection against ● Binary regression ● Link functions: binary an expected flu epidemic. regression ● Link function inverses: binary ■ In a pilot follow-up study, 50 clients were randomly selected regression ● Odds ratios & logistic and asked whether they actually received a flu shot. regression ● Link & variance fns. of a GLM ■ ● Binary (again) In addition, data were collected on their age and their health ● Fitting a binary regression GLM: IRLS awareness. ● Other common examples of GLMs ■ Here is the data. ● Deviance ● Binary deviance ● Partial deviance tests 2 ● Wald χ tests - p. 4/18 Binary outcomes ● Today's class ■ Suppose outcome Y is a 0-1 random variable, then ● Generalized linear models i ● Binary regression example ● Binary outcomes µ E Y π : ● Logit transform i = ( i) = i ● Binary regression ● Link functions: binary ■ regression Variance function ● Link function inverses: binary regression ● Odds ratios & logistic Var(Yi) = πi(1 − πi) = µi(1 − µi) regression ● Link & variance fns. of a GLM ● Binary (again) Variance is related to mean! ● Fitting a binary regression GLM: IRLS ■ A convenient way to model the dependence of Y on ● Other common examples of i GLMs ● Deviance covariates Xi1; : : : ; Xi;p−1 is through the logit transform. ● Binary deviance ● Partial deviance tests 2 p−1 ● Wald χ tests logit(πi) = β0 + βj Xij j=1 X - p. 5/18 Logit transform ● Today's class ■ Logit transform: ● Generalized linear models ● Binary regression example ● Binary outcomes π ● Logit transform logit π ; ● Binary regression ( ) = log 2 (−∞ +1) ● Link functions: binary 1 − π regression ● Link function inverses: binary ■ Inverse: regression x ● Odds ratios & logistic −1 e regression logit (x) = 2 (0; 1): ● Link & variance fns. of a GLM x ● Binary (again) 1 + e ● Fitting a binary regression ■ GLM: IRLS Derivative: ● Other common examples of GLMs ● Deviance d 1 1 ● Binary deviance logit(π) = = : ● Partial deviance tests dπ π(1 − π) V (π) 2 ● Wald χ tests Note: special relation between derivatives and variance function – more on this next lecture. - p. 6/18 Binary regression ● Today's class ■ Logistic regression model: ● Generalized linear models ● Binary regression example ● Binary outcomes logit(E(Yi)) = logit(πi) ● Logit transform ● Binary regression p−1 ● Link functions: binary regression ● Link function inverses: binary = β0 + βj Xij regression ● Odds ratios & logistic j=1 regression X ● Link & variance fns. of a GLM ■ ● Binary (again) Probit regression model: ● Fitting a binary regression GLM: IRLS ● Other common examples of p−1 GLMs −1 ● Deviance Φ (E(Yi)) = β0 + βj Xij ● Binary deviance ● Partial deviance tests j=1 2 ● Wald χ tests X where Φ is CDF of N(0; 1), i.e. Φ(t) = pnorm(t). ■ In each case, Var(Yi) = πi(1 − πi) but the regression model is different. ■ Here is an example - p. 7/18 Link functions: binary regression 6 4 logit probit 2 0 f(pi) −2 −4 −6 0.0 0.2 0.4 0.6 0.8 1.0 pi - p. 8/18 Link function inverses: binary regression 1.0 0.8 logit−inv probit−inv 0.6 f(pi) 0.4 0.2 0.0 −4 −2 0 2 4 pi - p. 9/18 Odds ratios & logistic regression ● Today's class ■ For any event A and any probability P ● Generalized linear models ● Binary regression example ● Binary outcomes P(A) ● Logit transform ODDS(A) = : ● Binary regression P A ● Link functions: binary 1 − ( ) regression ● Link function inverses: binary ■ regression In the logistic regression model with outcome Y ● Odds ratios & logistic regression ● ODDS Y : : : ; X x ; : : : Link & variance fns. of a GLM ( = 1j j = j + 1 ) βj ● Binary (again) = e ● Fitting a binary regression ODDS(Y = 1j : : : ; Xj = xj ; : : : ) GLM: IRLS ● Other common examples of GLMs is the (multiplicative) change in odds if variable X increases ● Deviance j ● Binary deviance by 1: βj is known as the ODDS RATIO for . ● Partial deviance tests e Xj 2 ● Wald χ tests ■ If Xj 2 f0; 1g is dichotomous, and P(Y = 1j : : : ) is small (rare event hypothesis) then group with Xj = 1 are approximately eβj more likely to have event, all other parameters being the same. - p. 10/18 Link & variance fns. of a GLM ● Today's class ■ If ● Generalized linear models ● Binary regression example k ● Binary outcomes E ● Logit transform ηi = g( (Yi)) = g(µi) = β0 + βj Xij ● Binary regression j=1 ● Link functions: binary regression X ● Link function inverses: binary then g is called the link function for the model. regression ● Odds ratios & logistic ■ regression If ● Link & variance fns. of a GLM E ● Binary (again) Var(Yi) = φ · V ( (Yi)) = φ · V (µi) ● Fitting a binary regression GLM: IRLS for φ > 0 and some function V , then V is the called variance ● Other common examples of GLMs ● Deviance function for the model. ● Binary deviance ■ ● Partial deviance tests “Canonical”reference: Generalized Linear Models, 2 ● Wald χ tests McCullagh and Nelder. - p. 11/18 Binary (again) ● Today's class ■ For a logistic model, ● Generalized linear models ● Binary regression example ● Binary outcomes g(µ) = logit(µ); V (µ) = µ(1 − µ): ● Logit transform ● Binary regression ● Link functions: binary ■ For a probit model, regression ● Link function inverses: binary regression −1 ● Odds ratios & logistic g(µ) = Φ (µ); V (µ) = µ(1 − µ): regression ● Link & variance fns. of a GLM ● Binary (again) ● Fitting a binary regression GLM: IRLS ● Other common examples of GLMs ● Deviance ● Binary deviance ● Partial deviance tests 2 ● Wald χ tests - p. 12/18 Fitting a binary regression GLM: IRLS ● Today's class ■ Algorithm ● Generalized linear models ● Binary regression example 1. Initialize: set µi = 0:999 or 0.001 depending on whether ● Binary outcomes ● Logit transform Yi = 1 or 0. ● Binary regression 0 ● Link functions: binary 2. Compute Z = g(µ ) + g (µ )(Y − µ ): regression i b i i i i ● Link function inverses: binary −1 0 2 regression 3. Use weights Wi = g (µi) V (µi) to regress Z onto X’s to ● Odds ratios & logistic regression get β using WLS.b b b ● Link & variance fns. of a GLM ● Binary (again) −1 b p b ● Fitting a binary regression 4. Compute µi = g β0 + Xijβj : GLM: IRLS j=1 ● Other common examples of b GLMs 5. Repeat steps 2-4 until convergence. ● Deviance b P b ● Binary deviance ■ b ● Partial deviance tests Approximate distribution 2 ● Wald χ tests β ∼ N(β; φ(XtW X)−1): ■ If φ has to be estimated, a simple choice is Pearson’s X 2: b c n 1 (Y − µ )2 φ = i i : n − p V (µ ) i=1 i X b b - p. 13/18 b Other common examples of GLMs ● Today's class ■ Standard multiple linear regression: g µ µ, Var µ ● Generalized linear models ( ) = ( ) = 1 ● Binary regression example ■ ● Binary outcomes Linear regression with variance tied to mean, for example: ● Logit transform 2 ● Binary regression g(µ) = µ, Var(µ) = µ . ● Link functions: binary ■ regression Poisson log-linear models: g(µ) = log(µ); Var(µ) = µ. ● Link function inverses: binary regression ● Odds ratios & logistic regression ● Link & variance fns. of a GLM ● Binary (again) ● Fitting a binary regression GLM: IRLS ● Other common examples of GLMs ● Deviance ● Binary deviance ● Partial deviance tests 2 ● Wald χ tests - p. 14/18 Deviance ● Today's class ■ Instead of least squares, models are fit on the basis of scaled ● Generalized linear models ● Binary regression example deviance, analogous to SSE when errors are not Gaussian. ● Binary outcomes ● Logit transform ■ ● Binary regression This is not completely clear from the IRLS algorithm, but it is ● Link functions: binary very close to the Newton-Raphson algorithm. Algorithm regression ● Link function inverses: binary often called Fisher scoring. regression ● Odds ratios & logistic ■ regression ● Link & variance fns. of a GLM ● Binary (again) DEV (µ, Y ) = −2 log L(µ, Y ) + −2 log L(Y; Y ) ● Fitting a binary regression GLM: IRLS ● Other common examples of where µ is a location estimator for Y (usually in an GLMs ● Deviance exponential family – more next lecture). ● Binary deviance ■ 2 ● Partial deviance tests If Y is Gaussian with independent N µ ; σ entries 2 ( i ) ● Wald χ tests n 1 DEV (µ, Y ) = (Y − µ )2 σ2 i i i=1 X - p.
Load more

Recommended publications
  • An Instrumental Variables Probit Estimator Using Gretl
    View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by Research Papers in Economics An Instrumental Variables Probit Estimator using gretl Lee C. Adkins Professor of Economics, Oklahoma State University, Stillwater, OK 74078 [email protected] Abstract. The most widely used commercial software to estimate endogenous probit models offers two choices: a computationally simple generalized least squares estimator and a maximum likelihood estimator. Adkins [1, -. com%ares these estimators to several others in a Monte Carlo study and finds that the 1LS estimator performs reasonably well in some circumstances. In this paper the small sample properties of the various estimators are reviewed and a simple routine us- ing the gretl software is given that yields identical results to those produced by Stata 10.1. The paper includes an example estimated using data on bank holding companies. 1 Introduction 4atchew and Griliches +,5. analyze the effects of various kinds of misspecifi- cation on the probit model. Among the problems explored was that of errors- in-variables. In linear re$ression, a regressor measured with error causes least squares to be inconsistent and 4atchew and Griliches find similar results for probit. Rivers and 7#ong [14] and Smith and 8lundell +,9. suggest two-stage estimators for probit and tobit, respectively. The strategy is to model a con- tinuous endogenous regressor as a linear function of the exogenous regressors and some instruments. Predicted values from this regression are then used in the second stage probit or tobit. These two-step methods are not efficient, &ut are consistent.
    [Show full text]
  • Generalized Linear Models and Generalized Additive Models
    00:34 Friday 27th February, 2015 Copyright ©Cosma Rohilla Shalizi; do not distribute without permission updates at http://www.stat.cmu.edu/~cshalizi/ADAfaEPoV/ Chapter 13 Generalized Linear Models and Generalized Additive Models [[TODO: Merge GLM/GAM and Logistic Regression chap- 13.1 Generalized Linear Models and Iterative Least Squares ters]] [[ATTN: Keep as separate Logistic regression is a particular instance of a broader kind of model, called a gener- chapter, or merge with logis- alized linear model (GLM). You are familiar, of course, from your regression class tic?]] with the idea of transforming the response variable, what we’ve been calling Y , and then predicting the transformed variable from X . This was not what we did in logis- tic regression. Rather, we transformed the conditional expected value, and made that a linear function of X . This seems odd, because it is odd, but it turns out to be useful. Let’s be specific. Our usual focus in regression modeling has been the condi- tional expectation function, r (x)=E[Y X = x]. In plain linear regression, we try | to approximate r (x) by β0 + x β. In logistic regression, r (x)=E[Y X = x] = · | Pr(Y = 1 X = x), and it is a transformation of r (x) which is linear. The usual nota- tion says | ⌘(x)=β0 + x β (13.1) · r (x) ⌘(x)=log (13.2) 1 r (x) − = g(r (x)) (13.3) defining the logistic link function by g(m)=log m/(1 m). The function ⌘(x) is called the linear predictor. − Now, the first impulse for estimating this model would be to apply the transfor- mation g to the response.
    [Show full text]
  • Generalized Linear Models (Glms)
    San Jos´eState University Math 261A: Regression Theory & Methods Generalized Linear Models (GLMs) Dr. Guangliang Chen This lecture is based on the following textbook sections: • Chapter 13: 13.1 – 13.3 Outline of this presentation: • What is a GLM? • Logistic regression • Poisson regression Generalized Linear Models (GLMs) What is a GLM? In ordinary linear regression, we assume that the response is a linear function of the regressors plus Gaussian noise: 0 2 y = β0 + β1x1 + ··· + βkxk + ∼ N(x β, σ ) | {z } |{z} linear form x0β N(0,σ2) noise The model can be reformulate in terms of • distribution of the response: y | x ∼ N(µ, σ2), and • dependence of the mean on the predictors: µ = E(y | x) = x0β Dr. Guangliang Chen | Mathematics & Statistics, San Jos´e State University3/24 Generalized Linear Models (GLMs) beta=(1,2) 5 4 3 β0 + β1x b y 2 y 1 0 −1 0.0 0.2 0.4 0.6 0.8 1.0 x x Dr. Guangliang Chen | Mathematics & Statistics, San Jos´e State University4/24 Generalized Linear Models (GLMs) Generalized linear models (GLM) extend linear regression by allowing the response variable to have • a general distribution (with mean µ = E(y | x)) and • a mean that depends on the predictors through a link function g: That is, g(µ) = β0x or equivalently, µ = g−1(β0x) Dr. Guangliang Chen | Mathematics & Statistics, San Jos´e State University5/24 Generalized Linear Models (GLMs) In GLM, the response is typically assumed to have a distribution in the exponential family, which is a large class of probability distributions that have pdfs of the form f(x | θ) = a(x)b(θ) exp(c(θ) · T (x)), including • Normal - ordinary linear regression • Bernoulli - Logistic regression, modeling binary data • Binomial - Multinomial logistic regression, modeling general cate- gorical data • Poisson - Poisson regression, modeling count data • Exponential, Gamma - survival analysis Dr.
    [Show full text]
  • Bayesian Inference Chapter 4: Regression and Hierarchical Models
    Bayesian Inference Chapter 4: Regression and Hierarchical Models Conchi Aus´ınand Mike Wiper Department of Statistics Universidad Carlos III de Madrid Master in Business Administration and Quantitative Methods Master in Mathematical Engineering Conchi Aus´ınand Mike Wiper Regression and hierarchical models Masters Programmes 1 / 35 Objective AFM Smith Dennis Lindley We analyze the Bayesian approach to fitting normal and generalized linear models and introduce the Bayesian hierarchical modeling approach. Also, we study the modeling and forecasting of time series. Conchi Aus´ınand Mike Wiper Regression and hierarchical models Masters Programmes 2 / 35 Contents 1 Normal linear models 1.1. ANOVA model 1.2. Simple linear regression model 2 Generalized linear models 3 Hierarchical models 4 Dynamic models Conchi Aus´ınand Mike Wiper Regression and hierarchical models Masters Programmes 3 / 35 Normal linear models A normal linear model is of the following form: y = Xθ + ; 0 where y = (y1;:::; yn) is the observed data, X is a known n × k matrix, called 0 the design matrix, θ = (θ1; : : : ; θk ) is the parameter set and follows a multivariate normal distribution. Usually, it is assumed that: 1 ∼ N 0 ; I : k φ k A simple example of normal linear model is the simple linear regression model T 1 1 ::: 1 where X = and θ = (α; β)T . x1 x2 ::: xn Conchi Aus´ınand Mike Wiper Regression and hierarchical models Masters Programmes 4 / 35 Normal linear models Consider a normal linear model, y = Xθ + . A conjugate prior distribution is a normal-gamma distribution:
    [Show full text]
  • Stochastic Process - Introduction
    Stochastic Process - Introduction • Stochastic processes are processes that proceed randomly in time. • Rather than consider fixed random variables X, Y, etc. or even sequences of i.i.d random variables, we consider sequences X0, X1, X2, …. Where Xt represent some random quantity at time t. • In general, the value Xt might depend on the quantity Xt-1 at time t-1, or even the value Xs for other times s < t. • Example: simple random walk . week 2 1 Stochastic Process - Definition • A stochastic process is a family of time indexed random variables Xt where t belongs to an index set. Formal notation, { t : ∈ ItX } where I is an index set that is a subset of R. • Examples of index sets: 1) I = (-∞, ∞) or I = [0, ∞]. In this case Xt is a continuous time stochastic process. 2) I = {0, ±1, ±2, ….} or I = {0, 1, 2, …}. In this case Xt is a discrete time stochastic process. • We use uppercase letter {Xt } to describe the process. A time series, {xt } is a realization or sample function from a certain process. • We use information from a time series to estimate parameters and properties of process {Xt }. week 2 2 Probability Distribution of a Process • For any stochastic process with index set I, its probability distribution function is uniquely determined by its finite dimensional distributions. •The k dimensional distribution function of a process is defined by FXX x,..., ( 1 x,..., k ) = P( Xt ≤ 1 ,..., xt≤ X k) x t1 tk 1 k for anyt 1 ,..., t k ∈ I and any real numbers x1, …, xk .
    [Show full text]
  • An Empirical Analysis of Factors Affecting Honors Program Completion Rates
    An Empirical Analysis of Factors Affecting Honors Program Completion Rates HALLIE SAVAGE CLARION UNIVERSITY OF PENNSYLVANIA AND T H E NATIONAL COLLEGIATE HONOR COUNCIL ROD D. RAE H SLER CLARION UNIVERSITY OF PENNSYLVANIA JOSE ph FIEDOR INDIANA UNIVERSITY OF PENNSYLVANIA INTRODUCTION ne of the most important issues in any educational environment is iden- tifying factors that promote academic success. A plethora of research on suchO factors exists across most academic fields, involving a wide range of student demographics, and the definition of student success varies across the range of studies published. While much of the research is devoted to looking at student performance in particular courses and concentrates on examina- tion scores and grades, many authors have directed their attention to student success in the context of an entire academic program; student success in this context usually centers on program completion or graduation and student retention. The analysis in this paper follows the emphasis of McKay on the importance of conducting repeated research on student completion of honors programs at different universities for different time periods. This paper uses a probit regression analysis as well as the logit regression analysis employed by McKay in order to determine predictors of student success in the honors pro- gram at a small, public university, thus attempting to answer McKay’s call for a greater understanding of honors students and factors influencing their suc- cess. The use of two empirical models on completion data, employing differ- ent base distributions, provides more robust statistical estimates than observed in similar studies 115 SPRING /SUMMER 2014 AN EMPIRI C AL ANALYSIS OF FA C TORS AFFE C TING COMPLETION RATES PREVIOUS LITEratURE The early years of our research was concurrent with the work of McKay, who studied the 2002–2005 entering honors classes at the University of North Florida and published his work in 2009.
    [Show full text]
  • On Assessing Binary Regression Models Based on Ungrouped Data
    Biometrics 000, 1{18 DOI: 000 000 0000 On Assessing Binary Regression Models Based on Ungrouped Data Chunling Lu Division of Global Health, Brigham and Women's Hospital & Department of Global Health and Social Medicine Harvard University, Boston, U.S. email: chunling [email protected] and Yuhong Yang School of Statistics, University of Minnesota, Minnesota, U.S. email: [email protected] Summary: Assessing a binary regression model based on ungrouped data is a commonly encountered but very challenging problem. Although tests, such as Hosmer-Lemeshow test and le Cessie-van Houwelingen test, have been devised and widely used in applications, they often have low power in detecting lack of fit and not much theoretical justification has been made on when they can work well. In this paper, we propose a new approach based on a cross validation voting system to address the problem. In addition to a theoretical guarantee that the probabilities of type I and II errors both converge to zero as the sample size increases for the new method under proper conditions, our simulation results demonstrate that it performs very well. Key words: Goodness of fit; Hosmer-Lemeshow test; Model assessment; Model selection diagnostics. This paper has been submitted for consideration for publication in Biometrics Goodness of Fit for Ungrouped Data 1 1. Introduction 1.1 Motivation Parametric binary regression is one of the most widely used statistical tools in real appli- cations. A central component in parametric regression is assessment of a candidate model before accepting it as a satisfactory description of the data. In that regard, goodness of fit tests are needed to reveal significant lack-of-fit of the model to assess (MTA), if any.
    [Show full text]
  • Application of Random-Effects Probit Regression Models
    Journal of Consulting and Clinical Psychology Copyright 1994 by the American Psychological Association, Inc. 1994, Vol. 62, No. 2, 285-296 0022-006X/94/S3.00 Application of Random-Effects Probit Regression Models Robert D. Gibbons and Donald Hedeker A random-effects probit model is developed for the case in which the outcome of interest is a series of correlated binary responses. These responses can be obtained as the product of a longitudinal response process where an individual is repeatedly classified on a binary outcome variable (e.g., sick or well on occasion t), or in "multilevel" or "clustered" problems in which individuals within groups (e.g., firms, classes, families, or clinics) are considered to share characteristics that produce similar responses. Both examples produce potentially correlated binary responses and modeling these per- son- or cluster-specific effects is required. The general model permits analysis at both the level of the individual and cluster and at the level at which experimental manipulations are applied (e.g., treat- ment group). The model provides maximum likelihood estimates for time-varying and time-invari- ant covariates in the longitudinal case and covariates which vary at the level of the individual and at the cluster level for multilevel problems. A similar number of individuals within clusters or number of measurement occasions within individuals is not required. Empirical Bayesian estimates of per- son-specific trends or cluster-specific effects are provided. Models are illustrated with data from
    [Show full text]
  • Variance Function Regressions for Studying Inequality
    Variance Function Regressions for Studying Inequality The Harvard community has made this article openly available. Please share how this access benefits you. Your story matters Citation Western, Bruce and Deirdre Bloome. 2009. Variance function regressions for studying inequality. Working paper, Department of Sociology, Harvard University. Citable link http://nrs.harvard.edu/urn-3:HUL.InstRepos:2645469 Terms of Use This article was downloaded from Harvard University’s DASH repository, and is made available under the terms and conditions applicable to Open Access Policy Articles, as set forth at http:// nrs.harvard.edu/urn-3:HUL.InstRepos:dash.current.terms-of- use#OAP Variance Function Regressions for Studying Inequality Bruce Western1 Deirdre Bloome Harvard University January 2009 1Department of Sociology, 33 Kirkland Street, Cambridge MA 02138. E-mail: [email protected]. This research was supported by a grant from the Russell Sage Foundation. Abstract Regression-based studies of inequality model only between-group differ- ences, yet often these differences are far exceeded by residual inequality. Residual inequality is usually attributed to measurement error or the in- fluence of unobserved characteristics. We present a regression that in- cludes covariates for both the mean and variance of a dependent variable. In this model, the residual variance is treated as a target for analysis. In analyses of inequality, the residual variance might be interpreted as mea- suring risk or insecurity. Variance function regressions are illustrated in an analysis of panel data on earnings among released prisoners in the Na- tional Longitudinal Survey of Youth. We extend the model to a decomposi- tion analysis, relating the change in inequality to compositional changes in the population and changes in coefficients for the mean and variance.
    [Show full text]
  • Flexible Signal Denoising Via Flexible Empirical Bayes Shrinkage
    Journal of Machine Learning Research 22 (2021) 1-28 Submitted 1/19; Revised 9/20; Published 5/21 Flexible Signal Denoising via Flexible Empirical Bayes Shrinkage Zhengrong Xing [email protected] Department of Statistics University of Chicago Chicago, IL 60637, USA Peter Carbonetto [email protected] Research Computing Center and Department of Human Genetics University of Chicago Chicago, IL 60637, USA Matthew Stephens [email protected] Department of Statistics and Department of Human Genetics University of Chicago Chicago, IL 60637, USA Editor: Edo Airoldi Abstract Signal denoising—also known as non-parametric regression—is often performed through shrinkage estima- tion in a transformed (e.g., wavelet) domain; shrinkage in the transformed domain corresponds to smoothing in the original domain. A key question in such applications is how much to shrink, or, equivalently, how much to smooth. Empirical Bayes shrinkage methods provide an attractive solution to this problem; they use the data to estimate a distribution of underlying “effects,” hence automatically select an appropriate amount of shrinkage. However, most existing implementations of empirical Bayes shrinkage are less flexible than they could be—both in their assumptions on the underlying distribution of effects, and in their ability to han- dle heteroskedasticity—which limits their signal denoising applications. Here we address this by adopting a particularly flexible, stable and computationally convenient empirical Bayes shrinkage method and apply- ing it to several signal denoising problems. These applications include smoothing of Poisson data and het- eroskedastic Gaussian data. We show through empirical comparisons that the results are competitive with other methods, including both simple thresholding rules and purpose-built empirical Bayes procedures.
    [Show full text]
  • Bayesian Methods: Review of Generalized Linear Models
    Bayesian Methods: Review of Generalized Linear Models RYAN BAKKER University of Georgia ICPSR Day 2 Bayesian Methods: GLM [1] Likelihood and Maximum Likelihood Principles Likelihood theory is an important part of Bayesian inference: it is how the data enter the model. • The basis is Fisher’s principle: what value of the unknown parameter is “most likely” to have • generated the observed data. Example: flip a coin 10 times, get 5 heads. MLE for p is 0.5. • This is easily the most common and well-understood general estimation process. • Bayesian Methods: GLM [2] Starting details: • – Y is a n k design or observation matrix, θ is a k 1 unknown coefficient vector to be esti- × × mated, we want p(θ Y) (joint sampling distribution or posterior) from p(Y θ) (joint probabil- | | ity function). – Define the likelihood function: n L(θ Y) = p(Y θ) | i| i=1 Y which is no longer on the probability metric. – Our goal is the maximum likelihood value of θ: θˆ : L(θˆ Y) L(θ Y) θ Θ | ≥ | ∀ ∈ where Θ is the class of admissable values for θ. Bayesian Methods: GLM [3] Likelihood and Maximum Likelihood Principles (cont.) Its actually easier to work with the natural log of the likelihood function: • `(θ Y) = log L(θ Y) | | We also find it useful to work with the score function, the first derivative of the log likelihood func- • tion with respect to the parameters of interest: ∂ `˙(θ Y) = `(θ Y) | ∂θ | Setting `˙(θ Y) equal to zero and solving gives the MLE: θˆ, the “most likely” value of θ from the • | parameter space Θ treating the observed data as given.
    [Show full text]
  • Generalized Linear Models
    Generalized Linear Models A generalized linear model (GLM) consists of three parts. i) The first part is a random variable giving the conditional distribution of a response Yi given the values of a set of covariates Xij. In the original work on GLM’sby Nelder and Wedderburn (1972) this random variable was a member of an exponential family, but later work has extended beyond this class of random variables. ii) The second part is a linear predictor, i = + 1Xi1 + 2Xi2 + + ··· kXik . iii) The third part is a smooth and invertible link function g(.) which transforms the expected value of the response variable, i = E(Yi) , and is equal to the linear predictor: g(i) = i = + 1Xi1 + 2Xi2 + + kXik. ··· As shown in Tables 15.1 and 15.2, both the general linear model that we have studied extensively and the logistic regression model from Chapter 14 are special cases of this model. One property of members of the exponential family of distributions is that the conditional variance of the response is a function of its mean, (), and possibly a dispersion parameter . The expressions for the variance functions for common members of the exponential family are shown in Table 15.2. Also, for each distribution there is a so-called canonical link function, which simplifies some of the GLM calculations, which is also shown in Table 15.2. Estimation and Testing for GLMs Parameter estimation in GLMs is conducted by the method of maximum likelihood. As with logistic regression models from the last chapter, the generalization of the residual sums of squares from the general linear model is the residual deviance, Dm 2(log Ls log Lm), where Lm is the maximized likelihood for the model of interest, and Ls is the maximized likelihood for a saturated model, which has one parameter per observation and fits the data as well as possible.
    [Show full text]