DOCSLIB.ORG

A Method to Integrate and Classify Normal Distributions

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
A Method to Integrate and Classify Normal Distributions A method to integrate and classify normal distributions Abhranil Das ∗1,3,4 and Wilson S Geisler2,3,4 1Department of Physics, The University of Texas at Austin 2Department of Psychology, The University of Texas at Austin 3Center for Perceptual Systems, The University of Texas at Austin 4Center for Theoretical and Computational Neuroscience, The University of Texas at Austin July 29, 2021 Abstract performance (behavior) of humans, other animals, neural circuits or algorithms. They also serve as a starting point in developing Univariate and multivariate normal probability distribu- other models/theories that describe sub-optimal performance of tions are widely used when modeling decisions under uncer- agents. tainty. Computing the performance of such models requires integrating these distributions over specic domains, which To compute the performance predicted by such theories it is can vary widely across models. Besides some special cases necessary to integrate the normal distributions over specic do- where these integrals are easy to calculate, there exist no gen- mains. For example, a particularly common task in vision science eral analytical expressions, standard numerical methods or is classication into two categories (e.g. detection and discrim- software for these integrals. Here we present mathematical ination tasks). The predicted maximum accuracy in such tasks results and open-source software that provide (i) the proba- is determined by integrating normals over domains dened by bility in any domain of a normal in any dimensions with any a quadratic decision boundary.2,3 Predicted accuracy of some of parameters, (ii) the probability density, cumulative distribu- the possible sub-optimal models is determined by integrating over tion, and inverse cumulative distribution of any function of a other domains. normal vector, (iii) the classication errors among any num- Except for some special cases4,5 (e.g. multinormal probabilities ber of normal distributions, the Bayes-optimal discriminabil- ity index and relation to the operating characteristic, (iv) di- in rectangular domains, such as when two normals have equal mension reduction and visualizations for such problems, and covariance and the optimal classication boundary is at), there (v) tests for how reliably these methods may be used on given exists no general analytical expression for these integrals, and we data. We demonstrate these tools with vision research appli- must use numerical methods, such as integrating over a Cartesian cations of detecting occluding objects in natural scenes, and grid, or Monte-Carlo integration. detecting camouage. Since the normal distribution tails o innitely outwards, it is Keywords: multivariate normal, integration, classication, inecient to numerically integrate it over a nite uniform Carte- signal detection theory, Bayesian ideal observer, vision sian grid, which would be large and collect ever-reducing masses outwards, yet omit some mass wherever the grid ends. Also, if the normal is elongated by unequal variances and strong covariances, 1 Introduction or the integration domain is complex and non-contiguous, naïve integration grids will waste resources in regions or directions that The univariate or multivariate normal (henceforth called sim- have low density or are outside the domain. One then needs to arXiv:2012.14331v7 [stat.ML] 27 Jul 2021 ply ‘normal’) is arguably the most important and widely-used visually inspect and arduously hand-tailor the integration grid to probability distribution. It is frequently used because various t the shape of each separate problem. central-limit theorems guarantee that normal distributions will Monte-Carlo integration involves sampling from the multinor- occur commonly in nature, and because it is the simplest and most mal, then counting the fraction of samples in the integration do- tractable distribution that allows arbitrary correlations between main. This does not have the above ineciencies or problems, but the variables. has other issues. Unlike grid integration, the desired precision Normal distributions form the basis of many theories and mod- cannot be specied, but must be determined by measuring the els in the natural and social sciences. For example, they are the spread across multiple runs. Also, when the probability in the in- foundation of Bayesian statistical decision/classication theories 1 tegration domain is very small (e.g. to compute the classication using Gaussian discriminant analysis, and are widely applied in error rate or discriminability d0 for highly-separated normal dis- diverse elds such as vision science, neuroscience, probabilistic tributions), it cannot be reliably sampled without a large number planning in robotics, psychology, and economics. These theo- of samples, which costs resource and time (see the performance ries specify optimal performance under uncertainty, and are of- benchmark section for a comparison). ten used to provide a benchmark against which to evaluate the Thus, there is no single analytical expression, numerical ∗[email protected] method or standard software tool to quickly and accurately in- 1 tegrate arbitrary normals over arbitrary domains, or to compute dimensional sd, since it linearly scales the normal (like σ in 1d), classication errors and the discriminability index d0. Evaluat- and its eigenvectors and values are the axes of the 1 sd error el- ing these quantities is often simplied by making the limiting lipsoid. We rst invert this transform to standardize the normal: assumption of equal variance. This impedes the quick testing, z = S−1(x − µ). This decorrelates or ‘whitens’ the variables, comparison, and optimization of models. Here we describe a and transforms the integration domain to a dierent quadratic: mathematical method and accompanying software implementa- 0 0 tion which provides functions to (i) integrate normals with arbi- q~(z) = z Qe 2z + q~1z +q ~0 > 0; with trary means and covariances in any number of dimensions over Qe 2 = SQ2S; arbitrary domains, (ii) compute the pdf, cdf and inverse cdf of (2) q~ = 2SQ µ + Sq ; any function of a multinormal variable (normal vector), and (iii) 1 2 1 compute the performance of classifying amongst any number of q~0 = q(µ): normals. This software is available as a Matlab toolbox ‘Inte- Now the problem is to nd the probability of the standard normal grate and classify normal distributions’, and the source code is z in this domain. If there is no quadratic term Qe 2, q~(z) is nor- at github.com/abhranildas/IntClassNorm. mally distributed, the integration domain boundary is a at, and We rst review and assimilate previous mathematical results the probability is Φ( q~0 ), where Φ is the standard normal cdf.4 into a generalized chi-squared method that can integrate arbi- kq~1k 0 trary normals over quadratic domains. Then we present a novel Otherwise, say Qe 2 = RDR is its eigen-decomposition, where 0 ray-trace method to integrate arbitrary normals over any domain, R is orthogonal, i.e. a rotoreection. So y = R z is also standard and consequently to compute the distribution of any real-valued normal, and in this space the quadratic is: function of a normal vector. We describe how these results can q^(y) = y0Dy + b0y +q ~ (b = R0q~ ) be used to compute error rates (and other relevant quantities) 0 1 X 2 X for Bayes-optimal and custom classiers, given arbitrary priors = Diyi + biyi + bi0 yi0 +q ~0 and outcome cost matrix. We then present some methods to re- i i0 duce problems to fewer dimensions, for analysis or visualization. (i and i0 index the nonzero and zero eigenvalues) Next, we provide a way to test whether directly measured sam- 2 2 X bi X X bi ples from the actual distributions in a classication problem are = Di yi + + bi0 yi0 +q ~0 − 2Di 2Di close enough to normal to trust the computations from the tool- i i0 i box. After describing the methods and software toolbox with ex- X 02 = Di χ 2 + x; amples, we demonstrate their accuracy and speed across a variety 1;(bi=2Di) i of problems. We show that for quadratic-domain problems both the generalized chi-squared method and the ray-trace method are a weighted sum of non-central chi-square variables χ02, each with accurate, but vary in relative speed depending on the particular 1 degree of freedom, and a normal variable x ∼ N(m; s). So this 2 problem. Of course, for domains that are not quadratic, only the is a generalized chi-square variable χ~w;k;λ;m;s, where we merge ray-trace method applies. Finally, we illustrate the methods with the non-central chi-squares with the same weights, so that the two applications from our laboratory: modeling detection of oc- vector of their weights w are the unique nonzero eigenvalues cluding targets in natural scenes, and detecting camouage. among Di, their degrees of freedom k are the numbers of times the eigenvalues occur, and their non-centralities, and normal pa- rameters are: 2 Integrating the normal s 1 X 2 X 2 λj = 2 bi ; m = q(µ) − w.λ; s = bi0 : 2.1 In quadratic domains: the generalized chi- 4wj 0 i:Di=wj i square method The required probability, p χ~2 > 0, is now a 1d integral, com- Integrating the normal in quadratic domains is important for putable using, say, Ruben’s6 or Davies’7 methods. We use the computing the maximum possible classication accuracy. The Matlab toolbox ‘Generalized chi-square distribution’ that we de- problem is the following: given a column vector x ∼ N(µ; Σ), veloped (source code is at github.com/abhranildas/gx2), which nd the probability that can compute the generalized chi-square parameters correspond- 0 0 ing to a quadratic form of a normal vector, its statistics, cdf (using q(x) = x Q2x + q1x + q0 > 0: (1) three dierent methods), pdf, inverse cdf and random numbers. (Here and henceforth, bold upper-case symbols represent ma- Previous software implements specic forms of this theory for trices, bold lower-case symbols represent vectors, and regular particular quadratics such as ellipsoids.5 The method described lower-case symbols represent scalars.) here correctly handles all quadratics (ellipsoids, hyperboloids, This can be viewed as the multi-dimensional integral of the paraboloids and degenerate conics) in all dimensions.
Load more

Recommended publications
  • Moments of the Product and Ratio of Two Correlated Chi-Square Variables
    View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by Springer - Publisher Connector Stat Papers (2009) 50:581–592 DOI 10.1007/s00362-007-0105-0 REGULAR ARTICLE Moments of the product and ratio of two correlated chi-square variables Anwar H. Joarder Received: 2 June 2006 / Revised: 8 October 2007 / Published online: 20 November 2007 © The Author(s) 2007 Abstract The exact probability density function of a bivariate chi-square distribu- tion with two correlated components is derived. Some moments of the product and ratio of two correlated chi-square random variables have been derived. The ratio of the two correlated chi-square variables is used to compare variability. One such applica- tion is referred to. Another application is pinpointed in connection with the distribution of correlation coefficient based on a bivariate t distribution. Keywords Bivariate chi-square distribution · Moments · Product of correlated chi-square variables · Ratio of correlated chi-square variables Mathematics Subject Classification (2000) 62E15 · 60E05 · 60E10 1 Introduction Fisher (1915) derived the distribution of mean-centered sum of squares and sum of products in order to study the distribution of correlation coefficient from a bivariate nor- mal sample. Let X1, X2,...,X N (N > 2) be two-dimensional independent random vectors where X j = (X1 j , X2 j ) , j = 1, 2,...,N is distributed as a bivariate normal distribution denoted by N2(θ, ) with θ = (θ1,θ2) and a 2 × 2 covariance matrix = (σik), i = 1, 2; k = 1, 2. The sample mean-centered sums of squares and sum of products are given by a = N (X − X¯ )2 = mS2, m = N − 1,(i = 1, 2) ii j=1 ij i i = N ( − ¯ )( − ¯ ) = and a12 j=1 X1 j X1 X2 j X2 mRS1 S2, respectively.
    [Show full text]
  • A Multivariate Student's T-Distribution
    Open Journal of Statistics, 2016, 6, 443-450 Published Online June 2016 in SciRes. http://www.scirp.org/journal/ojs http://dx.doi.org/10.4236/ojs.2016.63040 A Multivariate Student’s t-Distribution Daniel T. Cassidy Department of Engineering Physics, McMaster University, Hamilton, ON, Canada Received 29 March 2016; accepted 14 June 2016; published 17 June 2016 Copyright © 2016 by author and Scientific Research Publishing Inc. This work is licensed under the Creative Commons Attribution International License (CC BY). http://creativecommons.org/licenses/by/4.0/ Abstract A multivariate Student’s t-distribution is derived by analogy to the derivation of a multivariate normal (Gaussian) probability density function. This multivariate Student’s t-distribution can have different shape parameters νi for the marginal probability density functions of the multi- variate distribution. Expressions for the probability density function, for the variances, and for the covariances of the multivariate t-distribution with arbitrary shape parameters for the marginals are given. Keywords Multivariate Student’s t, Variance, Covariance, Arbitrary Shape Parameters 1. Introduction An expression for a multivariate Student’s t-distribution is presented. This expression, which is different in form than the form that is commonly used, allows the shape parameter ν for each marginal probability density function (pdf) of the multivariate pdf to be different. The form that is typically used is [1] −+ν Γ+((ν n) 2) T ( n) 2 +Σ−1 n 2 (1.[xx] [ ]) (1) ΓΣ(νν2)(π ) This “typical” form attempts to generalize the univariate Student’s t-distribution and is valid when the n marginal distributions have the same shape parameter ν .
    [Show full text]
  • A Guide on Probability Distributions
    powered project A guide on probability distributions R-forge distributions Core Team University Year 2008-2009 LATEXpowered Mac OS' TeXShop edited Contents Introduction 4 I Discrete distributions 6 1 Classic discrete distribution 7 2 Not so-common discrete distribution 27 II Continuous distributions 34 3 Finite support distribution 35 4 The Gaussian family 47 5 Exponential distribution and its extensions 56 6 Chi-squared's ditribution and related extensions 75 7 Student and related distributions 84 8 Pareto family 88 9 Logistic ditribution and related extensions 108 10 Extrem Value Theory distributions 111 3 4 CONTENTS III Multivariate and generalized distributions 116 11 Generalization of common distributions 117 12 Multivariate distributions 132 13 Misc 134 Conclusion 135 Bibliography 135 A Mathematical tools 138 Introduction This guide is intended to provide a quite exhaustive (at least as I can) view on probability distri- butions. It is constructed in chapters of distribution family with a section for each distribution. Each section focuses on the tryptic: definition - estimation - application. Ultimate bibles for probability distributions are Wimmer & Altmann (1999) which lists 750 univariate discrete distributions and Johnson et al. (1994) which details continuous distributions. In the appendix, we recall the basics of probability distributions as well as \common" mathe- matical functions, cf. section A.2. And for all distribution, we use the following notations • X a random variable following a given distribution, • x a realization of this random variable, • f the density function (if it exists), • F the (cumulative) distribution function, • P (X = k) the mass probability function in k, • M the moment generating function (if it exists), • G the probability generating function (if it exists), • φ the characteristic function (if it exists), Finally all graphics are done the open source statistical software R and its numerous packages available on the Comprehensive R Archive Network (CRAN∗).
    [Show full text]
  • Gamma and Related Distributions
    GAMMA AND RELATED DISTRIBUTIONS By Ayienda K. Carolynne Supervisor: Prof J.A.M Ottieno School of Mathematics University of Nairobi A thesis submitted to the School of Mathematics, University of Nairobi in partial fulfillment of the requirements for the degree of Master of Science in Statistics. November, 2013 Declaration I, Ayienda Kemunto Carolynne do hereby declare that this thesis is my original work and has not been submitted for a degree in any other University. Sign: ---------------------------------------- Date................................ AYIENDA KEMUNTO CAROLYNNE Sign: ---------------------------------------- Date............................... Supervisor: J.A.M OTTIENO i Dedication I wish to dedicate this thesis to my beloved husband Ezekiel Onyonka and son Steve Michael. To you all, I thank you for giving me the support needed to persue my academic dream. ii Acknowledgements I am grateful to the all-mighty God who has watched over me all my life and more so, during my challenging academic times. It is my prayer that he grants me more strength to achieve all my ambitions and aspirations in life. I am thankful to the University of Nairobi, which not only offered me a scholarship, but also moral support during this study. It is my pleasant privilege to acknowledge with gratitude one and all from whom I received guidance and encouragement during hard times of my study. Special thanks to my supervisor Prof. J.A.M Ottieno, School of Mathematics, University of Nairobi for his moral and material support, encouragement through which I have managed to do my work. Also I cannot forget to appreciate his invaluable guidance, tolerance and constructive criticism without which I would not have accomplished my objectives.
    [Show full text]
  • Hand-Book on STATISTICAL DISTRIBUTIONS for Experimentalists
    Internal Report SUF–PFY/96–01 Stockholm, 11 December 1996 1st revision, 31 October 1998 last modification 10 September 2007 Hand-book on STATISTICAL DISTRIBUTIONS for experimentalists by Christian Walck Particle Physics Group Fysikum University of Stockholm (e-mail: [email protected]) Contents 1 Introduction 1 1.1 Random Number Generation .............................. 1 2 Probability Density Functions 3 2.1 Introduction ........................................ 3 2.2 Moments ......................................... 3 2.2.1 Errors of Moments ................................ 4 2.3 Characteristic Function ................................. 4 2.4 Probability Generating Function ............................ 5 2.5 Cumulants ......................................... 6 2.6 Random Number Generation .............................. 7 2.6.1 Cumulative Technique .............................. 7 2.6.2 Accept-Reject technique ............................. 7 2.6.3 Composition Techniques ............................. 8 2.7 Multivariate Distributions ................................ 9 2.7.1 Multivariate Moments .............................. 9 2.7.2 Errors of Bivariate Moments .......................... 9 2.7.3 Joint Characteristic Function .......................... 10 2.7.4 Random Number Generation .......................... 11 3 Bernoulli Distribution 12 3.1 Introduction ........................................ 12 3.2 Relation to Other Distributions ............................. 12 4 Beta distribution 13 4.1 Introduction .......................................
    [Show full text]
  • Handbook on Probability Distributions
    R powered R-forge project Handbook on probability distributions R-forge distributions Core Team University Year 2009-2010 LATEXpowered Mac OS' TeXShop edited Contents Introduction 4 I Discrete distributions 6 1 Classic discrete distribution 7 2 Not so-common discrete distribution 27 II Continuous distributions 34 3 Finite support distribution 35 4 The Gaussian family 47 5 Exponential distribution and its extensions 56 6 Chi-squared's ditribution and related extensions 75 7 Student and related distributions 84 8 Pareto family 88 9 Logistic distribution and related extensions 108 10 Extrem Value Theory distributions 111 3 4 CONTENTS III Multivariate and generalized distributions 116 11 Generalization of common distributions 117 12 Multivariate distributions 133 13 Misc 135 Conclusion 137 Bibliography 137 A Mathematical tools 141 Introduction This guide is intended to provide a quite exhaustive (at least as I can) view on probability distri- butions. It is constructed in chapters of distribution family with a section for each distribution. Each section focuses on the tryptic: definition - estimation - application. Ultimate bibles for probability distributions are Wimmer & Altmann (1999) which lists 750 univariate discrete distributions and Johnson et al. (1994) which details continuous distributions. In the appendix, we recall the basics of probability distributions as well as \common" mathe- matical functions, cf. section A.2. And for all distribution, we use the following notations • X a random variable following a given distribution, • x a realization of this random variable, • f the density function (if it exists), • F the (cumulative) distribution function, • P (X = k) the mass probability function in k, • M the moment generating function (if it exists), • G the probability generating function (if it exists), • φ the characteristic function (if it exists), Finally all graphics are done the open source statistical software R and its numerous packages available on the Comprehensive R Archive Network (CRAN∗).
    [Show full text]
  • Field Guide to Continuous Probability Distributions
    Field Guide to Continuous Probability Distributions Gavin E. Crooks v 1.0.0 2019 G. E. Crooks – Field Guide to Probability Distributions v 1.0.0 Copyright © 2010-2019 Gavin E. Crooks ISBN: 978-1-7339381-0-5 http://threeplusone.com/fieldguide Berkeley Institute for Theoretical Sciences (BITS) typeset on 2019-04-10 with XeTeX version 0.99999 fonts: Trump Mediaeval (text), Euler (math) 271828182845904 2 G. E. Crooks – Field Guide to Probability Distributions Preface: The search for GUD A common problem is that of describing the probability distribution of a single, continuous variable. A few distributions, such as the normal and exponential, were discovered in the 1800’s or earlier. But about a century ago the great statistician, Karl Pearson, realized that the known probabil- ity distributions were not sufficient to handle all of the phenomena then under investigation, and set out to create new distributions with useful properties. During the 20th century this process continued with abandon and a vast menagerie of distinct mathematical forms were discovered and invented, investigated, analyzed, rediscovered and renamed, all for the purpose of de- scribing the probability of some interesting variable. There are hundreds of named distributions and synonyms in current usage. The apparent diver- sity is unending and disorienting. Fortunately, the situation is less confused than it might at first appear. Most common, continuous, univariate, unimodal distributions can be orga- nized into a small number of distinct families, which are all special cases of a single Grand Unified Distribution. This compendium details these hun- dred or so simple distributions, their properties and their interrelations.
    [Show full text]
  • Multivariate Statistics
    Chapter 2 Multivariate Statistics The univariate normal distribution takes a central position in statistics. This situation is even more apparent in multivariate analysis, because the mul- tivariate normal distribution is one of the few analytically tractable multi- variate distributions (Eaton, 1983). Absolutely continuous multivariate dis- tributions on Rm can be defined by specifying the density function of the distribution with respect to the Lebesgue measure on Rm. The probability mass of such distributions cannot be concentrated in a null set. However, if the random vector takes values in, for example, a proper affine subspace, then this approach needs some modification. We will call these distributions singular distributions. Degenerated distributions give all their mass to just one point. Therefore, we call such distributions singular distributions. To incorporate singular distributions, it is convenient to define a multivariate distribution by specifying the distribution of each linear function of the ran- dom vector (see, e.g., Muirhead, 1982, theorem 1.2.2). We briefly repeat this discussion in subsection 2.1.1. It is often stated, however, that the density function of a normally distributed m × 1 vector X with singular covariance matrix ‘does not exist’. Whereas, indeed, the density does not exist with re- spect to the Lebesgue measure on Rm, the density function does exist on an affine subspace, a topic we will discuss in subsection 2.1.2. To be able to work with the density function of singular normally distributed random vec- tors in practice, the problem is often redefined on this affine subspace (see, e.g., definition 3.2.2 of Tong, 1990).
    [Show full text]
  • Chi Percent Point Function with Degrees of Freedom Parameter V
    Auxillary CHPPF CHPPF PURPOSE Compute the chi percent point function with degrees of freedom parameter v. DESCRIPTION The distribution of the positive square root of a variable having a chi-square distribution is a chi-distribution. The chi-distribution has the following probability density function: –x2 -------- 2 υ – 1 (), υ e x υ > f x = --------------------------υ x > 0, 0 (EQ Aux-67) ---– 1 υ 22 Γ --- 2 where Γ is the gamma function The percent point function is the inverse of the cumulative distribution function. The cumulative distribution sums the probability from 0 to the given x value. The percent point function takes a cumulative probability value and computes the corresponding x value. The input value should be between 0 and 1 and the degrees of freedom should be a positive real number. The chi percent point function is computed numerically. SYNTAX LET <y2> = CHPPF(<y1>,<v>) <SUBSET/EXCEPT/FOR qualification> where <y1> is a positive number, parameter, or variable in the range 0 to 1; <y2> is a variable or a parameter (depending on what <y1> is) where the computed chi ppf value is stored; <v> is a positive number, parameter, or variable that specifies the degrees of freedom; and where the <SUBSET/EXCEPT/FOR qualification> is optional. EXAMPLES LET A = CHPPF(0.9,10) LET A = CHPPF(A1,10) LET X2 = CHPPF(X1,10) NOTE 1 DATAPLOT uses a bisection method to compute the chi ppf value. The algorithm for the central beta distribution is given in the Kennedy and Gentle bool (see the REFERENCE section below). The algorithm for the chi-distribution is similar.
    [Show full text]
  • Probability Distribution Relationships
    STATISTICA, anno LXX, n. 1, 2010 PROBABILITY DISTRIBUTION RELATIONSHIPS Y.H. Abdelkader, Z.A. Al-Marzouq 1. INTRODUCTION In spite of the variety of the probability distributions, many of them are related to each other by different kinds of relationship. Deriving the probability distribu- tion from other probability distributions are useful in different situations, for ex- ample, parameter estimations, simulation, and finding the probability of a certain distribution depends on a table of another distribution. The relationships among the probability distributions could be one of the two classifications: the transfor- mations and limiting distributions. In the transformations, there are three most popular techniques for finding a probability distribution from another one. These three techniques are: 1 - The cumulative distribution function technique 2 - The transformation technique 3 - The moment generating function technique. The main idea of these techniques works as follows: For given functions gin(,XX12 ,...,) X, for ik 1, 2, ..., where the joint dis- tribution of random variables (r.v.’s) XX12, , ..., Xn is given, we define the func- tions YgXXii (12 , , ..., X n ), i 1, 2, ..., k (1) The joint distribution of YY12, , ..., Yn can be determined by one of the suit- able method sated above. In particular, for k 1, we seek the distribution of YgX () (2) For some function g(X ) and a given r.v. X . The equation (1) may be linear or non-linear equation. In the case of linearity, it could be taken the form n YaX ¦ ii (3) i 1 42 Y.H. Abdelkader, Z.A. Al-Marzouq Many distributions, for this linear transformation, give the same distributions for different values for ai such as: normal, gamma, chi-square and Cauchy for continuous distributions and Poisson, binomial, negative binomial for discrete distributions as indicated in the Figures by double rectangles.
    [Show full text]
  • Statistical Analysis
    statistical_analysis-WORKBOOK In [1]: ## CSS coloring for the dataframe tables from IPython.core.display import HTML css = open('../style/style-table.css').read() + open('../style/style-not ebook.css').read() HTML('<style>{}</style>'.format(css)) Out[1]: Statistical Analysis Index Exploratory Data Analysis Data Types Estimates of Location mean exercise exercise exercise median exercise Estimates of Variability Standard deviation and related estimates exercise Estimates based on percentiles Estimates of skewness and tailedness exercise Estimates of Distribution Boxplot exercise Frequency tables, density plots and histograms exercise exercise Correlation Data and Sampling distribution Common distributions Processing math: 100% file:///C/...ta_science_in_python/01.intrduction_to_python_for_data_science/04.statistical_analysis/statistical_analysis-WORKBOOK.html[18.09.2019 09:04:37] statistical_analysis-WORKBOOK Binomial distribution Exercise Exercise Poisson distribution Exercise Exponential distribution Exercise Uniform distribution Normal distribution Chi2 distribution Sampling Distribution of a Statistic Sampling Distribution of Means Exercise Exploratory Data Analysis [[1](#ref1)] Statistics has developed mostly in the past century. Probability theory was develped in the 17th to 19th century based on the work of Thomas Bayes, Pierre-Simon laplace and Carl Gauss. source: https://bit.ly/1LEZVtR Statistics is an applied science concerned with analysis and modeling of data. Francis Galton, Karl Pearson andRoland Fisher laying the pillars of moders statistics by introducing key ideas like experimental design and maximum likelihood estimation. These and many statistical concepts formed the foundations of moderen data science. The first step on any data science project is exploring the data. Exploratory Data Analysis (EDA) is a new area of statistics established by John W. Tukey, where he called fo rreformation of statistics in his paper The Future of Data Analysis.
    [Show full text]
  • Probability Theory with Simulations - Part-III Continuous Distributions in One-Dimension
    Probability Theory with Simulations - Part-III Continuous distributions in one-dimension - Andras Vetier 07 11 2017 Contents 1 Continuous random variables3 2 Distribution function3 3 *** Empirical distribution function5 4 Density function6 5 *** Histogram9 6 Uniform distributions 10 7 Distributions of some functions of random numbers 12 8 *** Arc-sine distribution 18 9 *** Cauchy distribution 19 10 *** Beta distributions 20 11 Exponential distribution 23 12 *** Gamma distribution 25 13 Normal distributions 27 14 *** Distributions derived from normal 30 1 15 ***Generating a random variable with a given continuous distribution 31 16 Expected value of continuous distributions 33 17 Expected value of a function of a continuous random variable 39 18 ***Median 42 19 Standard deviation, etc. 45 20 *** Poisson-processes 53 21 ***Transformation from line to line 55 2 1 Continuous random variables In this chapter, we start studying random variables which, contrary to discrete random variables, may take any numbers from an interval of the real line. Some examples: 1. X = the amount of time someone has to wait for the bus when he or she goes to work. The waiting time may be any positive number not exceeding a certain upper limit. 2. X = the weight of a new-born baby. The weight may be any number between certain lower and upper limits. 3. X = the height of a randomly chosen Hungarian man. The height may be any number between certain lower and upper limits. Obviously, when we observe such a random variable, that is, we check the time or measure the weight or the height, - we get the time rounded, for example, to minutes, possibly with some decimals, - we get the weight rounded, for example, to kilograms or grams, or to pounds, possibly with some decimals, - we get the height rounded, for example, to centimeters, or to inches, possibly with some decimals, as well.
    [Show full text]