36-709: Advanced Statistical Theory I Spring 2019 Lecture 1: January 15 Lecturer: Alessandro Rinaldo Scribe: Elan Rosenfeld Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer: These notes have not been subjected to the usual scrutiny reserved for formal publications. They may be distributed outside this class only with the permission of the Instructor. 1.1 Recap of Parametric Statistical Models Definition 1.1 P = fPθ : θ 2 Θg (1.1) d s where Θ ⊆ R and Pθ is a probability distribution on R . 1.1.1 Example: Normal k k k Θ = f(µ, Σ) : µ 2 R ; Σ 2 S+g where S+ is the cone of PD k × k matrices. (Recall that a matrix M is PD () xT Mx > 0 for all x 2 Rk 6= 0) k(k+1) k2 3 Then Pθ ∼ N (µ, Σ) and dim(Θ) = k + 2 = 2 + 2 k. 1.1.2 Example: Linear Regression 2 n n×d d×1 Y ∼ N (Xβ; σ In) where Y 2 R , X 2 R , β 2 R , and σ > 0. 2 Model: Y = Xβ + where = (1; :::; n) i.i.d. from N (0; σ ). Observe X = (x1; :::; xn) i.i.d. from Pθ ~ 0 Goal: Draw inference on θ0. Important Assumption: P , θ0 are fixed as n ! 1. In high dimensional statistics we assume d ! 1 as n ! 1. In non-parametric statistics we assume P grows as n ! 1. 1.1.3 Master Theorem for Parametric Models (from Jon Wellner's Notes) Found at http://www.stat.cmu.edu/~arinaldo/Teaching/36709/S19/Wellner_Notes.pdf There is a mistake in the notes, find it for extra credit on HW1! 1-1 1-2 Lecture 1: January 15 1.1.3.1 Assumptions Let 1. pθ be the density for the distribution Pθ P 2. Ln(θjXn) = Πpθ(xi) be the likelihood function, `n(θjXn) = log Ln(θjXn) = log pθ(xi) ~ ~ ~ 3. rθ`n(θ) be the gradient of `n(θ), H`n(θ) be the Hessian of `n(θ) 4. I(θ) = −Ex[H`n(θjx)] be the Fisher Information Then under certain regularity conditions (smoothness, identifiability, ...) on P , let Θe n be a solution to the score equation r`n(θ) = 0 (i.e., the MLE). We have that: p • Θe n exists and Θe n −! Θ0 (WLLN) p d −1 • n(Θe n − Θ0) −!Nd(0;I (Θ0)) (CLT) d 2 `n(Θe njXn) • 2 log λn −! χ where λn = ~ (Wilk's Theorem) e d e `n(Θ0jXn) ~ p p T ^ d 2 • n(Θe n − Θ0) In(Θe n) n(Θe n − Θ0) −! χd (Wald Test) Some points of note: • These results are asymptotic! • Critical: Require P and number of parameters to be fixed as n ! 1 1.2 High-Dimensional Statistical Models Definition 1.2 A high-dimensional parametric statistical model is a sequence of parametric statistical models 1 fPngn=1 where for each n, the sample space has size Sn and the parameter space has dimension dn, where Sn, dn are allowed to grow with n. 1.2.1 Example: Linear Regression dn T 2 (Y1;X1); :::; (Yn;Xn) are n R.V.s in R × R such that Yi = Xi β + i where (1; :::; n) ∼ N (0; σ In) and β 2 Rdn . 1.3 Different Types of Parametric Models 1. Fixed-d models (what we've worked with before) Lecture 1: January 15 1-3 2. High-dimensional models (1a) dn is allowed to change but dn 2 o(n) xn xn (xn 2 o(yn) () 8 > 0: 9n0 s.t. 8n > n0: j j < [i.e., j j ! 0]) yn yn See work by Portnoy on these models (1b) dn n Not generally possible without additional structural assumptions (sparsity, data near a low- dimensional manifold, etc.) Recommended Reading / Class Sources [] M.J. Wainwright, \High-Dimensional Statistics: A Non-Asymptotic Viewpoint," Cambridge Series in Statistical and Probabilistic Mathematics, 2019. Note: This text has yet to be published! The author has generously provided Prof. Rinaldo with advance copies of some chapters, so do not distribute this material outside of the class [] R. Vershynin, \High-Dimensional Probability: An Introduction with Applications in Data Science," Cambridge Series in Statistical and Probabilistic Mathematics, 2018..