Detection & Estimation Lecture 2
Total Page:16
File Type:pdf, Size:1020Kb
3/4/2019 Detection & Estimation Lecture 2 MVUE Xiliang Luo 1 MVUE • Minimize the variance while being unbiased • Question: whether MVUE exists? • unbiased estimator with minimum variance for all values of the unknown parameter • Example: [Example 2.3] , 1 , 0 0∼,1 1∼ , 2 , 0 1 2 01 3 1 01 2 2 1 3/4/2019 MVUE • No known “turn‐the‐crank” procedure to produce the MVUE • We have discussed • Cramer‐Rao lower bound • Next, we will discuss • Rao‐Blackwell‐Lehmann‐Scheffe theorem • Best Linear Estimator 3 Efficient Estimator • An unbiased estimator attaining the CRLB is efficient • efficiently uses the observed data • CRLB can be evaluated always • efficient estimator may not exist • we may still find the MVUE when MVUE exits • Sufficient statistics • Rao‐Blackwell‐Lehmann‐Scheffe 4 2 3/4/2019 Sufficient Statistics • DC level in noise • • ∑ • 0 • Both are unbiased estimators • Question: • Which data samples are pertinent to the estimation? • Is there a set of data which is sufficient? 5 Sufficient Statistics • DC level in noise • • Data set 1: • Data set 2: • Data set 3: minimal sufficient statistic The data set that contains the least number of elements is called the minimal set! 6 3 3/4/2019 Sufficient Statistics • DC level in noise • • PDF of the observed data: • ; exp ∑ • Assume the following statistic has been observed: • ∑ • Then, we have the posteriori PDF • |; 7 Sufficient Statistics • DC level in noise • • Since is sufficient, this conditional PDF should not depend on A anymore! • ; 8 4 3/4/2019 Sufficient Statistics 9 Sufficient Statistics • Example: • Verify ∑ is a sufficient statistic for the DC level in noise: , ; ; ; | ; ; 1 1 ; exp ∑ 2 2 2 1 1 ; exp 2 2 10 5 3/4/2019 Neyman‐Fisher Factorization • Theorem: the PDF ; can be factorized as • ; ,⋅ if and only if (iff) is a sufficient statistic for . • Generalization: the PDF ; can be factorized as • ; ,…, ,⋅ iff ,…, are jointly sufficient statistics • | ,…, ;does not depend on . 11 Ronald Fisher Jerzy Neyman 1890‐1962, England 1894‐1981, Polish Statistician, Biologist Statistician, Mathematician Fisher Information NP‐test 12 6 3/4/2019 Neyman‐Fisher Factorization • Example: DC level in WGN 1 1 ; exp 2 2 1 1 1 exp 2 exp 2 2 2 13 Neyman‐Fisher Factorization • Example: Power of WGN 1 1 ; exp ⋅1 2 2 14 7 3/4/2019 Sufficient Statistics If X1, ...., Xn are independent and have a Poisson distribution with parameter λ, then the sum T(X) = X1 + ... + Xn is a sufficient statistic for λ. ,…,; ,…,; 15 Proof of NFF Theorem Two Useful Results: 1. , ; ; 2. If , the pdf of y can be written as: For scalar : where x,…,x are the solutions of 16 8 3/4/2019 Proof of NFF Theorem • Sufficient: we prove if the factorization holds, then is a sufficient statistic. ; ,⋅ ; , ; ; ; , ; Does not depend on 17 Proof of NFF Theorem • Necessary: we prove if is a sufficient statistic, then the factorization holds. ; ,⋅ , ; ; ; ; Since is a sufficient statistic, conditional PDF does not depend on : ; where 18 9 3/4/2019 Proof of NFF Theorem • Necessary: we prove if is a sufficient statistic, then the factorization holds. ; ,⋅ , ; ; ; ; ; ; 19 Find MVUE • Example: DC Level in WGN • Sufficient statistic: ∑ • Two ways to find the MVUE • 1. 0, | • Start from any unbiased estimator • Determine the conditional mean • Depends on the value of the sufficient statistic only 1 20 10 3/4/2019 Find MVUE , are jointly Gaussian with mean: , and covariance matrix: E , , then we can have: ≔ 0 ≔ 21 Find MVUE • Example: DC Level in WGN • Sufficient statistic: ∑ • Two ways to find the MVUE • 2. Find function such that is an unbiased estimator for 1 22 11 3/4/2019 RBLS Theorem • Rao‐Blackwell‐Lehmann‐Scheffe • If is an unbiased estimator of and is a sufficient statistic for , then is • 1. a valid estimator • 2. unbiased • 3. of lesser or equal variance for all value Additionally, if the sufficient statistic is complete, then is the MVUE! • a statistic is complete if there is only one function of the statistic that is unbiased 23 RBLS Theorem MVUE 24 12 3/4/2019 Rao‐Blackwell Theorem Calyampudi Radhakrishna Rao David Harold Blackwell (born 10 September 1920) (April 24, 1919 –July 8, 2010) an Indian‐born, naturalized American, Professor Emeritus of Statistics at the mathematician and statistician University of California, Berkeley Fisher’s student 25 Lehmann‐Scheffe Theorem Erich Leo Lehmann Henry Scheffé (20 November 1917 – (New York City, United States 12 September 2009) 11 April 1907 –Berkeley, California an American statistician USA, 5 July 1977) an American statistician 26 13 3/4/2019 Completeness • A sufficient statistic is complete if the following condition: ; 0, ∀ is satisfied only by the zero function or by 0 ∀. 27 Completeness • Example: DC level in WGN • ∑is complete • only one function such that • assume another function such that • 0,∀,≔ • ∼ , 1 exp 0 2 , ≔ 1 exp 0 2 • we can verify 0 • key: satisfy the condition for all A 28 14 3/4/2019 Completeness 1 exp 0 2 0,∀ ∗0 ≔exp 2 29 Completeness • Example: Incomplete • 00 • 0∼ , • a sufficient statistic is 0, which is the only available data • check the following condition ; 0,∀ sin2 30 15 3/4/2019 Bernoulli Model • ,…, : IID Bernoulli distribution with parameter • 1, 01 • Let be the number of 1’s observed in the samples • is has a binomial distribution with parameters , • is a complete statistic for ∈0,1 0 : 1 31 Procedure to find MVUE Use Neyman‐Fisher factorization theorem to find sufficient statistic Determine if the statistic is complete RBLS: Find function of the statistic that is unbiased or conditional mean of an arbitrary unbiased estimator! MVUE 32 16 3/4/2019 Proof of RBLS • Rao‐Blackwell‐Lehmann‐Scheffe • If is an unbiased estimator of and is a sufficient statistic for , then is • 1. a valid estimator • 2. unbiased • 3. of lesser or equal variance for all value Additionally, if the sufficient statistic is complete, then is the MVUE! • a statistic is complete if there is only one function of the statistic that is unbiased 33 Proof of RBLS • 1. a valid estimator • sufficient statistic new estimator is a function of observation only ; is a function of T only! 34 17 3/4/2019 Proof of RBLS • 2. unbiased estimator • property of conditional mean ; ;; ; 35 Proof of RBLS • 3. lower variance var +2 + var | | 0 36 18 3/4/2019 Mean of Uniform Noise • Observation: • ,0,…,1 • is IID noise, uniformly distributed over 0, • We want to estimate the mean: Option 1: Efficient: find the CRLB Option 2: MVUE: Neyman‐Fisher factorization and RBLS theorem 37 Mean of Uniform Noise 1 ; max ⋅min A sufficient statistic: max PDF: 0, 0 pdf: ,0 0, 2 1 38 19 3/4/2019 Mean of Uniform Noise 1 var • Sample mean: 12 1 • MVUE: var max 2 4 2 -1 10 sample mean MVUE -2 10 -3 10 1 2 3 4 5 6 7 8 9 10 39 One More Example • A single observation with the pdf 1 and we wish to find MVUE of exp log log1 1 1 1 log 1 var log 1 MVUE: 40 20 3/4/2019 Exponential Family Distributions • A single‐parameter exponential family is a set of probability distributions whose pdf can be expressed in the form exp ⋅ where ,,, are known functions. 41 Examples of Exponential Family • normal • exponential • gamma • chi‐squared • beta • Bernoulli 1⋅exp⋅ln ln1 1 • … 42 21 3/4/2019 Properties of Exponential Family • is a sufficient statistic of the distribution • Exponential families have a large number of properties that make them extremely useful for statistical analysis. • Exponential families have sufficient statistics that can summarize arbitrary amounts of iid data using a fixed number of values. 43 22.