Statistics 203: Introduction to Regression and Analysis of Variance Nonlinear regression Jonathan Taylor - p. 1/11 Today’s class ● Today’s class ■ Nonlinear regression. ● Nonlinear regression models ● Weight loss data ■ ● What to do? Examples in R. ● Delta method ● Nonlinear regression ● Nonlinear regression: details ● Iteration & Distribution ● Confidence intervals ● Weight loss data - p. 2/11 Nonlinear regression models ● Today’s class ■ We have usually assumed regression is of the form ● Nonlinear regression models ● Weight loss data ● What to do? p−1 ● Delta method ● Nonlinear regression Yi = β0 + βj Xij + "i: ● Nonlinear regression: details X ● Iteration & Distribution j=1 ● Confidence intervals ● Weight loss data ■ Or, the regression function p−1 f(x; β) = β + β x 0 X j j j=1 is linear in beta. ■ Many real-life phenomena can be parameterized by non-linear regression functions. Example: ◆ Radioactive decay: half-life is a non-linear parameter − f(t; θ) = C · 2 t/θ: - p. 3/11 Weight loss data 180 170 160 150 Weight 140 130 120 110 0 50 100 150 200 250 Days - p. 4/11 What to do? ● Today’s class ■ In some cases, it is possible to linearize to get the original ● Nonlinear regression models ● Weight loss data model. ● What to do? ● Delta method ■ Suppose ● Nonlinear regression ● Nonlinear regression: details −Xi/θ ● Iteration & Distribution Yi = C · 2 "i; ● Confidence intervals ● Weight loss data then 0 ∗ log(Yi) = C − Xi/θ + "i : ∗ If "i have approximately the same variance, than it looks like original model. ■ What if −Xi/θ Yi = C · 2 + "i where the "i’s have the same variance? - p. 5/11 Delta method ● Today’s class ■ The delta method tells us that ● Nonlinear regression models ● Weight loss data ● 0 2 What to do? Var(f(Y )) ' f (E(Y )) Var(Y ): ● Delta method i i i ● Nonlinear regression ● Nonlinear regression: details ■ In this case ● Iteration & Distribution ● Confidence intervals 1 2 ● Weight loss data Var(log(Yi)) ' − σ : C22 2Xi/θ ■ Could use weighted least-squares if θ was known. ■ One possibility: IRLS. ◆ j Starting at some initial θ ; j ≥ 0, regress log(Yi) onto Xi j+1 solve for β1 , and set b b j+1 1 θ0 = − j+1 b β1 b - p. 6/11 Nonlinear regression ● Today’s class ■ Alternatively, ● Nonlinear regression models ● Weight loss data ● What to do? n ● Delta method −Xi/θ 2 ● Nonlinear regression (C; θ) = argmin (Yi − C2 ) : ● Nonlinear regression: details (C,θ) X ● Iteration & Distribution b b i=1 ● Confidence intervals ● Weight loss data ■ In general, given a possibly non-linear regression function f(x; θ); θ 2 Rp: we can try to solve n θ = argmin (Y − f(x ; θ))2 X i i b θ i=1 where xi is the i-th row of the design matrix. ■ Techniques: usually use a steepest descent approach instead of Newton-Raphson. - p. 7/11 Nonlinear regression: details ● Today’s class ■ 0 ● Nonlinear regression models Given an initial value θ ● Weight loss data ● What to do? b p ● Delta method 0 @f 0 0 ● Nonlinear regression f(xi; θ) ' f(xi; θ ) + (θi − θi ) = ηi (θ): ● Nonlinear regression: details X @θ b0 j (xi,θ)=(xi;θ ) ● Iteration & Distribution b j=1 ● Confidence intervals ● Weight loss data ■ In matrix form η0(θ) = !0 + Z0θ where 0 @f @ηi Zij = = @θj b0 @θj b0 (x,θ)=(xi;θ ) θ=θ p !0 = f(x ; θ ) − θ0Z0 i i 0 X i ij j=1 ■ The vector η0(θ) is our approximation to the regression surface, we want to choose θ to get as close to Y as possible. Project onto “tangent space.” - p. 8/11 Iteration & Distribution ● Today’s class ■ 1 ● Nonlinear regression models Want to choose θ as follows ● Weight loss data ● What to do? b n ● Delta method 1 0 0 2 ● Nonlinear regression θ = argmin (Yi − !i − Z θ) ● Nonlinear regression: details θ X ● Iteration & Distribution b i=1 ● Confidence intervals ● Weight loss data ■ Solution −1 θ1 = Z0 tZ0 Z0 t(Y − !0): b ■ Given θj, set b −1 θj+1 = Zj tZj Zj t(Y − !j): b ■ Repeat until convergence. ■ Approximate distribution of θ: b − θ ∼ N(θ; σ2(ZtZ) 1); b b b where b n σ2 = (Y − f(x ; θ))2=(n − p): X i i - p. 9/11 b i=1 b Confidence intervals ● Today’s class ■ If θ is restricted, say θ ≥ 0 the asymptotic confidence ● Nonlinear regression models ● Weight loss data intervals may be inaccurate (may overlap into negative ● What to do? ● Delta method numbers). ● Nonlinear regression ● Nonlinear regression: details ■ library(MASS) provides another method to obtain ● Iteration & Distribution ● Confidence intervals confidence intervals based on “inverting” an F -test. ● Weight loss data ■ Basic idea, confidence interval for θ1: for each fixed value θ1;0 we could compute the “extra sum of squares” between the unrestricted model and the model with θ1 fixed at θ1;0: SSEθ1=θ1;0 (θ2:p) − SSE(θ) F (θ ) = ∼ F − 1;0 bσ2 b 1;n p at least approximately under Hb0 : θ1 = θ1;0. ■ Or, T (θ1;0) = sign(θ − θ1;0)qF (θ1;0) ∼ tn−p: b b ■ Confidence interval: fθ1 : −tn−p;1−α/2 < T (θ1) < tn−p;1−α/2g: - p. 10/11 Weight loss data 180 170 160 150 Weight 140 130 120 110 0 50 100 150 200 250 Days - p. 11/11.