Overdispersed Models for Claim Count Distribution
Total Page:16
File Type:pdf, Size:1020Kb
Load more
Recommended publications
-
Laws of Total Expectation and Total Variance
Laws of Total Expectation and Total Variance Definition of conditional density. Assume and arbitrary random variable X with density fX . Take an event A with P (A) > 0. Then the conditional density fXjA is defined as follows: 8 < f(x) 2 P (A) x A fXjA(x) = : 0 x2 = A Note that the support of fXjA is supported only in A. Definition of conditional expectation conditioned on an event. Z Z 1 E(h(X)jA) = h(x) fXjA(x) dx = h(x) fX (x) dx A P (A) A Example. For the random variable X with density function 8 < 1 3 64 x 0 < x < 4 f(x) = : 0 otherwise calculate E(X2 j X ≥ 1). Solution. Step 1. Z h i 4 1 1 1 x=4 255 P (X ≥ 1) = x3 dx = x4 = 1 64 256 4 x=1 256 Step 2. Z Z ( ) 1 256 4 1 E(X2 j X ≥ 1) = x2 f(x) dx = x2 x3 dx P (X ≥ 1) fx≥1g 255 1 64 Z ( ) ( )( ) h i 256 4 1 256 1 1 x=4 8192 = x5 dx = x6 = 255 1 64 255 64 6 x=1 765 Definition of conditional variance conditioned on an event. Var(XjA) = E(X2jA) − E(XjA)2 1 Example. For the previous example , calculate the conditional variance Var(XjX ≥ 1) Solution. We already calculated E(X2 j X ≥ 1). We only need to calculate E(X j X ≥ 1). Z Z Z ( ) 1 256 4 1 E(X j X ≥ 1) = x f(x) dx = x x3 dx P (X ≥ 1) fx≥1g 255 1 64 Z ( ) ( )( ) h i 256 4 1 256 1 1 x=4 4096 = x4 dx = x5 = 255 1 64 255 64 5 x=1 1275 Finally: ( ) 8192 4096 2 630784 Var(XjX ≥ 1) = E(X2jX ≥ 1) − E(XjX ≥ 1)2 = − = 765 1275 1625625 Definition of conditional expectation conditioned on a random variable. -
Probability Cheatsheet V2.0 Thinking Conditionally Law of Total Probability (LOTP)
Probability Cheatsheet v2.0 Thinking Conditionally Law of Total Probability (LOTP) Let B1;B2;B3; :::Bn be a partition of the sample space (i.e., they are Compiled by William Chen (http://wzchen.com) and Joe Blitzstein, Independence disjoint and their union is the entire sample space). with contributions from Sebastian Chiu, Yuan Jiang, Yuqi Hou, and Independent Events A and B are independent if knowing whether P (A) = P (AjB )P (B ) + P (AjB )P (B ) + ··· + P (AjB )P (B ) Jessy Hwang. Material based on Joe Blitzstein's (@stat110) lectures 1 1 2 2 n n A occurred gives no information about whether B occurred. More (http://stat110.net) and Blitzstein/Hwang's Introduction to P (A) = P (A \ B1) + P (A \ B2) + ··· + P (A \ Bn) formally, A and B (which have nonzero probability) are independent if Probability textbook (http://bit.ly/introprobability). Licensed and only if one of the following equivalent statements holds: For LOTP with extra conditioning, just add in another event C! under CC BY-NC-SA 4.0. Please share comments, suggestions, and errors at http://github.com/wzchen/probability_cheatsheet. P (A \ B) = P (A)P (B) P (AjC) = P (AjB1;C)P (B1jC) + ··· + P (AjBn;C)P (BnjC) P (AjB) = P (A) P (AjC) = P (A \ B1jC) + P (A \ B2jC) + ··· + P (A \ BnjC) P (BjA) = P (B) Last Updated September 4, 2015 Special case of LOTP with B and Bc as partition: Conditional Independence A and B are conditionally independent P (A) = P (AjB)P (B) + P (AjBc)P (Bc) given C if P (A \ BjC) = P (AjC)P (BjC). -
Probability Cheatsheet
Probability Cheatsheet v1.1.1 Simpson's Paradox Expected Value, Linearity, and Symmetry P (A j B; C) < P (A j Bc;C) and P (A j B; Cc) < P (A j Bc;Cc) Expected Value (aka mean, expectation, or average) can be thought Compiled by William Chen (http://wzchen.com) with contributions yet still, P (A j B) > P (A j Bc) of as the \weighted average" of the possible outcomes of our random from Sebastian Chiu, Yuan Jiang, Yuqi Hou, and Jessy Hwang. variable. Mathematically, if x1; x2; x3;::: are all of the possible values Material based off of Joe Blitzstein's (@stat110) lectures Bayes' Rule and Law of Total Probability that X can take, the expected value of X can be calculated as follows: P (http://stat110.net) and Blitzstein/Hwang's Intro to Probability E(X) = xiP (X = xi) textbook (http://bit.ly/introprobability). Licensed under CC i Law of Total Probability with partitioning set B1; B2; B3; :::Bn and BY-NC-SA 4.0. Please share comments, suggestions, and errors at with extra conditioning (just add C!) Note that for any X and Y , a and b scaling coefficients and c is our http://github.com/wzchen/probability_cheatsheet. constant, the following property of Linearity of Expectation holds: P (A) = P (AjB1)P (B1) + P (AjB2)P (B2) + :::P (AjBn)P (Bn) Last Updated March 20, 2015 P (A) = P (A \ B1) + P (A \ B2) + :::P (A \ Bn) E(aX + bY + c) = aE(X) + bE(Y ) + c P (AjC) = P (AjB1; C)P (B1jC) + :::P (AjBn; C)P (BnjC) If two Random Variables have the same distribution, even when they are dependent by the property of Symmetry their expected values P (AjC) = P (A \ B jC) + P (A \ B jC) + :::P (A \ B jC) Counting 1 2 n are equal. -
Conditional Expectation and Prediction Conditional Frequency Functions and Pdfs Have Properties of Ordinary Frequency and Density Functions
Conditional expectation and prediction Conditional frequency functions and pdfs have properties of ordinary frequency and density functions. Hence, associated with a conditional distribution is a conditional mean. Y and X are discrete random variables, the conditional frequency function of Y given x is pY|X(y|x). Conditional expectation of Y given X=x is Continuous case: Conditional expectation of a function: Consider a Poisson process on [0, 1] with mean λ, and let N be the # of points in [0, 1]. For p < 1, let X be the number of points in [0, p]. Find the conditional distribution and conditional mean of X given N = n. Make a guess! Consider a Poisson process on [0, 1] with mean λ, and let N be the # of points in [0, 1]. For p < 1, let X be the number of points in [0, p]. Find the conditional distribution and conditional mean of X given N = n. We first find the joint distribution: P(X = x, N = n), which is the probability of x events in [0, p] and n−x events in [p, 1]. From the assumption of a Poisson process, the counts in the two intervals are independent Poisson random variables with parameters pλ and (1−p)λ (why?), so N has Poisson marginal distribution, so the conditional frequency function of X is Binomial distribution, Conditional expectation is np. Conditional expectation of Y given X=x is a function of X, and hence also a random variable, E(Y|X). In the last example, E(X|N=n)=np, and E(X|N)=Np is a function of N, a random variable that generally has an expectation Taken w.r.t. -
Measurement and Uncertainty Analysis Guide
Measurements & Uncertainty Analysis Measurement and Uncertainty Analysis Guide “It is better to be roughly right than precisely wrong.” – Alan Greenspan Table of Contents THE UNCERTAINTY OF MEASUREMENTS .............................................................................................................. 2 RELATIVE (FRACTIONAL) UNCERTAINTY ............................................................................................................. 4 RELATIVE ERROR ....................................................................................................................................................... 5 TYPES OF UNCERTAINTY .......................................................................................................................................... 6 ESTIMATING EXPERIMENTAL UNCERTAINTY FOR A SINGLE MEASUREMENT ................................................ 9 ESTIMATING UNCERTAINTY IN REPEATED MEASUREMENTS ........................................................................... 9 STANDARD DEVIATION .......................................................................................................................................... 12 STANDARD DEVIATION OF THE MEAN (STANDARD ERROR) ......................................................................... 14 WHEN TO USE STANDARD DEVIATION VS STANDARD ERROR ...................................................................... 14 ANOMALOUS DATA ................................................................................................................................................ -
Chapter 3: Expectation and Variance
44 Chapter 3: Expectation and Variance In the previous chapter we looked at probability, with three major themes: 1. Conditional probability: P(A | B). 2. First-step analysis for calculating eventual probabilities in a stochastic process. 3. Calculating probabilities for continuous and discrete random variables. In this chapter, we look at the same themes for expectation and variance. The expectation of a random variable is the long-term average of the ran- dom variable. Imagine observing many thousands of independent random values from the random variable of interest. Take the average of these random values. The expectation is the value of this average as the sample size tends to infinity. We will repeat the three themes of the previous chapter, but in a different order. 1. Calculating expectations for continuous and discrete random variables. 2. Conditional expectation: the expectation of a random variable X, condi- tional on the value taken by another random variable Y . If the value of Y affects the value of X (i.e. X and Y are dependent), the conditional expectation of X given the value of Y will be different from the overall expectation of X. 3. First-step analysis for calculating the expected amount of time needed to reach a particular state in a process (e.g. the expected number of shots before we win a game of tennis). We will also study similar themes for variance. 45 3.1 Expectation The mean, expected value, or expectation of a random variable X is writ- ten as E(X) or µX . If we observe N random values of X, then the mean of the N values will be approximately equal to E(X) for large N. -
What Is This “Margin of Error”?
What is this “Margin of Error”? On April 23, 2017, The Wall Street Journal reported: “Americans are dissatisfied with President Donald Trump as he nears his 100th day in office, with views of his effectiveness and ability to shake up Washington slipping, a new Wall Street Journal/NBC News poll finds. “More than half of Americans—some 54%—disapprove of the job Mr. Trump is doing as president, compared with 40% who approve, a 14‐point gap. That is a weaker showing than in the Journal/NBC News poll in late February, when disapproval outweighed approval by 4 points.” [Skipping to the end of the article …] “The Wall Street Journal/NBC News poll was based on nationwide telephone interviews with 900 adults from April 17‐20. It has a margin of error of plus or minus 3.27 percentage points, with larger margins of error for subgroups.” Throughout the modern world, every day brings news concerning the latest public opinion polls. At the end of each news report, you’ll be told the “margin of error” in the reported estimates. Every poll is, of course, subject to what’s called “sampling error”: Evenly if properly run, there’s a chance that the poll will, merely due to bad luck, end up with a randomly‐chosen sample of individuals which is not perfectly representative of the overall population. However, using the tools you learned in your “probability” course, we can measure the likelihood of such bad luck. Assume that the poll was not subject to any type of systematic bias (a critical assumption, unfortunately frequently not true in practice). -
Lecture Notes 4 Expectation • Definition and Properties
Lecture Notes 4 Expectation • Definition and Properties • Covariance and Correlation • Linear MSE Estimation • Sum of RVs • Conditional Expectation • Iterated Expectation • Nonlinear MSE Estimation • Sum of Random Number of RVs Corresponding pages from B&T: 81-92, 94-98, 104-115, 160-163, 171-174, 179, 225-233, 236-247. EE 178/278A: Expectation Page 4–1 Definition • We already introduced the notion of expectation (mean) of a r.v. • We generalize this definition and discuss it in more depth • Let X ∈X be a discrete r.v. with pmf pX(x) and g(x) be a function of x. The expectation or expected value of g(X) is defined as E(g(X)) = g(x)pX(x) x∈X • For a continuous r.v. X ∼ fX(x), the expected value of g(X) is defined as ∞ E(g(X)) = g(x)fX(x) dx −∞ • Examples: ◦ g(X)= c, a constant, then E(g(X)) = c ◦ g(X)= X, E(X)= x xpX(x) is the mean of X k k ◦ g(X)= X , E(X ) is the kth moment of X ◦ g(X)=(X − E(X))2, E (X − E(X))2 is the variance of X EE 178/278A: Expectation Page 4–2 • Expectation is linear, i.e., for any constants a and b E[ag1(X)+ bg2(X)] = a E(g1(X)) + b E(g2(X)) Examples: ◦ E(aX + b)= a E(X)+ b ◦ Var(aX + b)= a2Var(X) Proof: From the definition Var(aX + b) = E ((aX + b) − E(aX + b))2 = E (aX + b − a E(X) − b)2 = E a2(X − E(X))2 = a2E (X − E(X))2 = a2Var( X) EE 178/278A: Expectation Page 4–3 Fundamental Theorem of Expectation • Theorem: Let X ∼ pX(x) and Y = g(X) ∼ pY (y), then E(Y )= ypY (y)= g(x)pX(x)=E(g(X)) y∈Y x∈X • The same formula holds for fY (y) using integrals instead of sums • Conclusion: E(Y ) can be found using either fX(x) or fY (y). -
Probabilities, Random Variables and Distributions A
Probabilities, Random Variables and Distributions A Contents A.1 EventsandProbabilities................................ 318 A.1.1 Conditional Probabilities and Independence . ............. 318 A.1.2 Bayes’Theorem............................... 319 A.2 Random Variables . ................................. 319 A.2.1 Discrete Random Variables ......................... 319 A.2.2 Continuous Random Variables ....................... 320 A.2.3 TheChangeofVariablesFormula...................... 321 A.2.4 MultivariateNormalDistributions..................... 323 A.3 Expectation,VarianceandCovariance........................ 324 A.3.1 Expectation................................. 324 A.3.2 Variance................................... 325 A.3.3 Moments................................... 325 A.3.4 Conditional Expectation and Variance ................... 325 A.3.5 Covariance.................................. 326 A.3.6 Correlation.................................. 327 A.3.7 Jensen’sInequality............................. 328 A.3.8 Kullback–LeiblerDiscrepancyandInformationInequality......... 329 A.4 Convergence of Random Variables . 329 A.4.1 Modes of Convergence . 329 A.4.2 Continuous Mapping and Slutsky’s Theorem . 330 A.4.3 LawofLargeNumbers........................... 330 A.4.4 CentralLimitTheorem........................... 331 A.4.5 DeltaMethod................................ 331 A.5 ProbabilityDistributions............................... 332 A.5.1 UnivariateDiscreteDistributions...................... 333 A.5.2 Univariate Continuous Distributions . 335 -
(Introduction to Probability at an Advanced Level) - All Lecture Notes
Fall 2018 Statistics 201A (Introduction to Probability at an advanced level) - All Lecture Notes Aditya Guntuboyina August 15, 2020 Contents 0.1 Sample spaces, Events, Probability.................................5 0.2 Conditional Probability and Independence.............................6 0.3 Random Variables..........................................7 1 Random Variables, Expectation and Variance8 1.1 Expectations of Random Variables.................................9 1.2 Variance................................................ 10 2 Independence of Random Variables 11 3 Common Distributions 11 3.1 Ber(p) Distribution......................................... 11 3.2 Bin(n; p) Distribution........................................ 11 3.3 Poisson Distribution......................................... 12 4 Covariance, Correlation and Regression 14 5 Correlation and Regression 16 6 Back to Common Distributions 16 6.1 Geometric Distribution........................................ 16 6.2 Negative Binomial Distribution................................... 17 7 Continuous Distributions 17 7.1 Normal or Gaussian Distribution.................................. 17 1 7.2 Uniform Distribution......................................... 18 7.3 The Exponential Density...................................... 18 7.4 The Gamma Density......................................... 18 8 Variable Transformations 19 9 Distribution Functions and the Quantile Transform 20 10 Joint Densities 22 11 Joint Densities under Transformations 23 11.1 Detour to Convolutions...................................... -
On the Method of Moments for Estimation in Latent Variable Models Anastasia Podosinnikova
On the Method of Moments for Estimation in Latent Variable Models Anastasia Podosinnikova To cite this version: Anastasia Podosinnikova. On the Method of Moments for Estimation in Latent Variable Models. Machine Learning [cs.LG]. Ecole Normale Superieure de Paris - ENS Paris, 2016. English. tel- 01489260v1 HAL Id: tel-01489260 https://tel.archives-ouvertes.fr/tel-01489260v1 Submitted on 14 Mar 2017 (v1), last revised 5 Apr 2018 (v2) HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. THÈSE DE DOCTORAT de l’Université de recherche Paris Sciences Lettres – PSL Research University préparée à l’École normale supérieure Sur la méthode des mo- École doctorale n°386 ments pour d’estimation Spécialité: Informatique des modèles à variable latentes Soutenue le 01.12.2016 On the method of moments for es- timation in latent variable models par Anastasia Podosinnikova Composition du Jury : Mme Animashree ANANDKUMAR University of California Irvine Rapporteur M Samuel KASKI Aalto University Rapporteur M Francis BACH École normale supérieure Directeur de thèse M Simon LACOSTE-JULIEN École normale supérieure Directeur de thèse M Alexandre D’ASPREMONT École normale supérieure Membre du Jury M Pierre COMON CNRS Membre du Jury M Rémi GRIBONVAL INRIA Membre du Jury Abstract Latent variable models are powerful probabilistic tools for extracting useful latent structure from otherwise unstructured data and have proved useful in numerous ap- plications such as natural language processing and computer vision. -
A Note on Confidence Interval Estimation and Margin of Error
Journal of Statistics Education, Volume 18, Number 1 (2010) A Note on Confidence Interval Estimation and Margin of Error Dennis Gilliland Vince Melfi Michigan State University Journal of Statistics Education Volume 18, Number 1 (2010), www.amstat.org/publications/jse/v18n1/gilliland.pdf Copyright © 2010 by Dennis Gilliland and Vince Melfi all rights reserved. This text may be freely shared among individuals, but it may not be republished in any medium without express written consent from the authors and advance notification of the editor. Key Words: Confidence interval estimation; Margin of error; Interpretations; Misinterpretations. Abstract Confidence interval estimation is a fundamental technique in statistical inference. Margin of error is used to delimit the error in estimation. Dispelling misinterpretations that teachers and students give to these terms is important. In this note, we give examples of the confusion that can arise in regard to confidence interval estimation and margin of error. 1. Introduction Confidence interval estimation is a widely used method of inference and margin of error is a commonly used term, and these occupy a large part of introductory courses and textbooks in Statistics. It is well-known that these concepts are often misused and misunderstood. Examples of incorrect interpretations from a variety of sources, some “authoritative,” are given in Thornton and Thornton (2004). Recent doctoral theses of Liu (2005) and Noll (2007) address teacher and teaching assistant understanding of the concepts. Misunderstandings arise for a variety of reasons, some as simple as confusing population parameters and sample statistics. Some of the confusion in regard to the meaning and interpretation of these terms stems from the lack of appreciation of the difference between a random variable (a function) and its realization 1 Journal of Statistics Education, Volume 18, Number 1 (2010) (evaluation).