DOCSLIB.ORG

Discrete Probability Distributions

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Discrete Probability Distributions Machine Learning Srihari Discrete Probability Distributions Sargur N. Srihari 1 Machine Learning Srihari Binary Variables Bernoulli, Binomial and Beta 2 Machine Learning Srihari Bernoulli Distribution • Expresses distribution of Single binary-valued random variable x ε {0,1} • Probability of x=1 is denoted by parameter µ, i.e., p(x=1|µ)=µ – Therefore p(x=0|µ)=1-µ • Probability distribution has the form Bern(x|µ)=µ x (1-µ) 1-x • Mean is shown to be E[x]=µ Jacob Bernoulli • Variance is var[x]=µ (1-µ) 1654-1705 • Likelihood of n observations independently drawn from p(x|µ) is N N x 1−x p(D | ) = p(x | ) = n (1− ) n µ ∏ n µ ∏µ µ n=1 n=1 – Log-likelihood is N N ln p(D | ) = ln p(x | ) = {x ln +(1−x )ln(1− )} µ ∑ n µ ∑ n µ n µ n=1 n=1 • Maximum likelihood estimator 1 N – obtained by setting derivative of ln p(D|µ) wrt m equal to zero is = x µML ∑ n N n=1 • If no of observations of x=1 is m then µML=m/N 3 Machine Learning Srihari Binomial Distribution • Related to Bernoulli distribution • Expresses Distribution of m – No of observations for which x=1 • It is proportional to Bern(x|µ) Histogram of Binomial for • Add up all ways of obtaining heads N=10 and ⎛ ⎞ m=0.25 ⎜ N ⎟ m N−m Bin(m | N,µ) = ⎜ ⎟µ (1− µ) ⎝⎜ m ⎠⎟ Binomial Coefficients: ⎛ ⎞ N ! ⎜ N ⎟ • Mean and Variance are ⎜ ⎟ = ⎝⎜ m ⎠⎟ m!(N −m)! N E[m] = ∑mBin(m | N,µ) = Nµ m=0 Var[m] = N (1− ) 4 µ µ Machine Learning Srihari Beta Distribution • Beta distribution a=0.1, b=0.1 a=1, b=1 Γ(a + b) Beta(µ |a,b) = µa−1(1 − µ)b−1 Γ(a)Γ(b) • Where the Gamma function is defined as ∞ Γ(x) = ∫u x−1e−u du a=2, b=3 0 a=8, b=4 • a and b are hyperparameters that control distribution of parameter µ • Mean and Variance Beta distribution as function of µ a ab E [µ] = var[µ] = For values of hyperparameters a and b a + b (a + b)2 (a + b +1) 5 Machine Learning Srihari Bayesian Inference with Beta • MLE of µ in Bernoulli is fraction of observations with x=1 – Severely over-fitted for small data sets • Likelihood function takes products of factors of the form µx(1-µ)(1-x) • If prior distribution of µ is chosen to be proportional to powers of µ and 1-µ, posterior will have same functional form as the prior – Called conjugacy • Beta has form suitable for a prior distribution of p(µ) 6 Machine Learning Srihari Bayesian Inference with Beta Illustration of one step in process • Posterior obtained by multiplying beta a=2, b=2 prior with binomial likelihood yields p(µ) p(µ | m,l,a,b) α µ m+a−1(1− µ)l+b−1 – where l=N-m, which is no of tails N=m=1, with x=1 p(x=1/µ)= – m is no of heads µ1(1-µ)0 • It is another beta distribution Γ(m + a + l + b) p(µ | m,l,a,b) = µ m+a−1(1− µ)l+b−1 Γ(m + a)Γ(l + b) a=3, b=2 – Effectively increase value of a by m and b by l p(µ/x=1) – As number of observations increases distribution becomes more peaked 7 Machine Learning Srihari Predicting next trial outcome • Need predictive distribution of x given observed D – From sum and products rule 1 1 p(x = 1| D) = p(x = 1, µ | D)dµ = p(x = 1| µ)p(µ | D)dµ = ∫0 ∫0 1 = µp(µ | D)dµ = E[µ | D] ∫0 • Expected value of the posterior distribution can be shown to be m + a p(x = 1| D) = m + a + l + b – Which is fraction of observations (both fictitious and real) that correspond to x=1 • Maximum likelihood and Bayesian results agree in the limit of infinite observations – On average uncertainty (variance) decreases with observed data 8 Machine Learning Srihari Summary of Binary Distributions • Single Binary variable distribution is represented by Bernoulli • Binomial is related to Bernoulli – Expresses distribution of number of occurrences of either 1 or 0 in N trials • Beta distribution is a conjugate prior for Bernoulli – Both have the same functional form 9 Machine Learning Srihari Sample Matlab Code Probability Distributions • Binomial Distribution: – Probability Density Function : Y = binopdf (X,N,P) returns the binomial probability density function with parameters N and P at the values in X. – Random Number Generator: R = binornd (N,P,MM,NN) returns n MM-by-NN matrix of random numbers chosen from a binomial distribution with parameters N and P • Beta Distribution – Probability Density Function : Y = betapdf (X,A,B) returns the beta probability density function with parameters A and B at the values in X. – Random Number Generator: R = betarnd (A,B) returns a matrix of random numbers chosen from the beta distribution with parameters A and B. Machine Learning Srihari Multinomial Variables Generalized Bernoulli and Dirichlet 11 Machine Learning Srihari Generalization of Binomial Histogram of Binomial for • Binomial N=10 and µ=0.25 – Tossing a coin – Expresses probability of no of successes in N trials • Probability of 3 rainy days in 10 days • Multinomial – Throwing a Die – Probability of a given frequency for each value • Probability of 3 specific letters in a string of N • Probability Calculator – http://stattrek.com/Tables/Multinomial.aspx 12 Machine Learning Srihari Generalized Bernoulli: Multinoulli • Bernoulli distribution x is 0 or 1 Bern(x|µ)=µ x (1-µ) 1-x • Discrete variable that takes one of K values (instead of 2) – Represent as 1-of-K scheme • Represent x as a K-dimensional vector • If x=3 then we represent it as x=(0,0,1,0,0,0)T K – Such vectors satisfy x = 1 ∑k=1 k – If probability of xk=1 is denoted µk then distribution of x is given by K x T p( | ) = k where = ( ,.., ) Generalized Bernoulli x µ ∏µk µ µ1 µK k=1 13 Machine Learning Srihari MLE of Multinoulli Parameters • Data set D of N indep observations x1,..xN th • where the n observation is written as [xn1,.., xnK] • Likelihood function has the form N K K K x x ∑n nk m p(D | ) nk ( ) k µ = ∏∏ µk = ∏ µk = ∏ µk n=1 k=1 k=1 k=1 • where mk=Σn xnk is the no. of observations of xk=1 • Maximum likelihood solution – Maximize ln p(D|µ) with Lagrangian constraint that the µk must sum to one, i.e., maximize K ⎛ K ⎞ mk ln µk + λ µk − 1 m µML = k ∑k ⎜ ∑ ⎟ k= 1N ⎝ k=1 ⎠ • Setting derivative wrt µ to zero k ML mk µk = N – which is fraction of N observations for which xk=1 14 Machine Learning Srihari Generalized Binomial Distribution • Multinomial distribution (with K-state variable) K ⎛ N ⎞ m Mult m m ..m | µ,N = µ k ( 1 2 K ) ⎜ m m ..m ⎟ ∏ k ⎝ 1 2 k ⎠ k=1 µ = 1 ∑k k – Where the normalization coefficient is the no of ways of partitioning N objects into K groups of size m ,m ..m 1 2 k • Given by ⎛ N ⎞ N ! = ⎜ m m ..m ⎟ ⎝ 1 2 k ⎠ m1 !m2 !..mk ! 15 Machine Learning Srihari Dirichlet Distribution Lejeune Dirichlet 1805-1859 • Family of prior distributions for parameters µk of multinomial distribution • By inspection of multinomial, form of conjugate prior is K α −1 p(µ | α) α µ k where 0 ≤ µ ≤ 1 and µ = 1 ∏ k k ∑k k k=1 • Normalized form of Dirichlet distribution K K Γ(α ) α −1 Dir(µ | α) = 0 µ k where α = α ∏ k 0 ∑ k Γ(α )...Γ(α ) k=1 k=1 1 k 16 Machine Learning Srihari Dirichlet over 3 variables αk=0.1 • Due to summation µ =1 constraint ∑k k – Distribution over Plots of Dirichlet =1 space of { } is distribution over the αk µk simplex for various confined to the settings of parameters simplex of αk dimensionality K-1 3 3 Γ(α ) α −1 =10 Dir(µ | α) = 0 µ k where α = α αk ∏ k 0 ∑ k – For K=3 Γ(α )...Γ(α ) k=1 k=1 1 3 17 Machine Learning Srihari Dirichlet Posterior Distribution • Multiplying prior by likelihood K p( | D, ) p(D | ) p( | ) αk +mk −1 µ α α µ µ α α∏ µk k =1 • Which has the form of the Dirichlet distribution p(µ | D,α) = Dir(µ |α + m) Γ(α + N) K 0 αk +mk −1 = ∏ µk Γ(α1 + m1)..Γ(α K + mK ) k =1 18 Machine Learning Srihari Summary of Discrete Distributions • Bernoulli (2 states) : Bern(x|µ)=µ x (1-µ) 1-x ⎛ ⎞ ⎛ N ⎞ m N −m ⎜ N ⎟ N ! – Binomial: Bin(m | N, µ) = ⎜ ⎟µ (1− µ) ⎜ ⎟ = ⎜ ⎟ ⎝⎜ m ⎠⎟ m!(N −m)! ⎝ m ⎠ • Generalized Bernoulli (K states): K p(x | µ) xk where µ ( ,.., )T = ∏ µk = µ1 µK k =1 ⎛ N ⎞ K Mult m m ..m | , N mk – Multinomial ( 1 2 K µ ) = ⎜ ⎟∏ µk ⎝m1m2..mk ⎠ k =1 • Conjugate priors: – Binomial is Beta Γ(a + b) Beta(µ | a,b) = µ a−1(1− µ)b−1 Γ(a)Γ(b) – Multinomial is Dirichlet Γ(α ) K K Dir(µ |α) = 0 µαk −1 where α = α ∏ k 0 ∑ k 19 Γ(α1)...Γ(α k ) k =1 k =1 Machine Learning Srihari Distributions: Landscape Discrete- Binary Bernoulli Binomial Beta Discrete- Multivalued Multinomial Dirichlet Continuous Gaussian Wishart Student’s-t Gamma Exponential Angular Von Mises Uniform 20 Machine Learning Srihari Distributions: Relationships Discrete- Conjugate Binary N=1 Prior Beta Binomial Bernoulli Continuous variable N samples of Bernoulli Single binary variable between {0,1] K=2 Discrete- Multinomial Large Conjugate Prior Multi-valued One of K values = Dirichlet N K-dimensional K random variables binary vector between [0.1] Continuous Student’s-t Gamma Wishart Exponential Generalization of ConjugatePrior of univariate Conjugate Prior of multivariate Special case of Gamma Gaussian robust to Gaussian precision Gaussian precision matrix Gaussian Outliers Infinite mixture of Gaussians Gaussian-Gamma Gaussian-Wishart Conjugate prior of univariate Gaussian Conjugate prior of multi-variate Gaussian Unknown mean and precision Unknown mean and precision matrix Angular Von Mises Uniform 21 .
Load more

Recommended publications
  • On Two-Echelon Inventory Systems with Poisson Demand and Lost Sales
    View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by Universiteit Twente Repository On two-echelon inventory systems with Poisson demand and lost sales Elisa Alvarez, Matthieu van der Heijden Beta Working Paper series 366 BETA publicatie WP 366 (working paper) ISBN ISSN NUR 804 Eindhoven December 2011 On two-echelon inventory systems with Poisson demand and lost sales Elisa Alvarez Matthieu van der Heijden University of Twente University of Twente The Netherlands1 The Netherlands 2 Abstract We derive approximations for the service levels of two-echelon inventory systems with lost sales and Poisson demand. Our method is simple and accurate for a very broad range of problem instances, including cases with both high and low service levels. In contrast, existing methods only perform well for limited problem settings, or under restrictive assumptions. Key words: inventory, spare parts, lost sales. 1 Introduction We consider a two-echelon inventory system for spare part supply chains, consisting of a single central depot and multiple local warehouses. Demand arrives at each local warehouse according to a Poisson process. Each location controls its stocks using a one-for-one replenishment policy. Demand that cannot be satisfied from stock is served using an emergency shipment from an external source with infinite supply and thus lost to the system. It is well-known that the analysis of such lost sales inventory systems is more complex than the equivalent with full backordering (Bijvank and Vis [3]). In particular, the analysis of the central depot is complex, since (i) the order process is not Poisson, and (ii) the order arrival rate depends on the inventory states of the local warehouses: Local warehouses only generate replenishment orders if they have stock on hand.
    [Show full text]
  • Distance Between Multinomial and Multivariate Normal Models
    Chapter 9 Distance between multinomial and multivariate normal models SECTION 1 introduces Andrew Carter’s recursive procedure for bounding the Le Cam distance between a multinomialmodelandits approximating multivariatenormal model. SECTION 2 develops notation to describe the recursive construction of randomizations via conditioning arguments, then proves a simple Lemma that serves to combine the conditional bounds into a single recursive inequality. SECTION 3 applies the results from Section 2 to establish the bound involving randomization of the multinomial distribution in Carter’s inequality. SECTION 4 sketches the argument for the bound involving randomization of the multivariate normalin Carter’s inequality. SECTION 5 outlines the calculation for bounding the Hellinger distance between a smoothed Binomialand its approximating normaldistribution. 1. Introduction The multinomial distribution M(n,θ), where θ := (θ1,...,θm ), is the probability m measure on Z+ defined by the joint distribution of the vector of counts in m cells obtained by distributing n balls independently amongst the cells, with each ball assigned to a cell chosen from the distribution θ. The variance matrix nVθ corresponding to M(n,θ) has (i, j)th element nθi {i = j}−nθi θj . The central limit theorem ensures that M(n,θ) is close to the N(nθ,nVθ ), in the sense of weak convergence, for fixed m when n is large. In his doctoral dissertation, Andrew Carter (2000a) considered the deeper problem of bounding the Le Cam distance (M, N) between models M :={M(n,θ) : θ ∈ } and N :={N(nθ,nVθ ) : θ ∈ }, under mild regularity assumptions on . For example, he proved that m log m maxi θi <1>(M, N) ≤ C √ provided sup ≤ C < ∞, n θ∈ mini θi for a constant C that depends only on C.
    [Show full text]
  • A Matrix-Valued Bernoulli Distribution Gianfranco Lovison1 Dipartimento Di Scienze Statistiche E Matematiche “S
    Journal of Multivariate Analysis 97 (2006) 1573–1585 www.elsevier.com/locate/jmva A matrix-valued Bernoulli distribution Gianfranco Lovison1 Dipartimento di Scienze Statistiche e Matematiche “S. Vianelli”, Università di Palermo Viale delle Scienze 90128 Palermo, Italy Received 17 September 2004 Available online 10 August 2005 Abstract Matrix-valued distributions are used in continuous multivariate analysis to model sample data matrices of continuous measurements; their use seems to be neglected for binary, or more generally categorical, data. In this paper we propose a matrix-valued Bernoulli distribution, based on the log- linear representation introduced by Cox [The analysis of multivariate binary data, Appl. Statist. 21 (1972) 113–120] for the Multivariate Bernoulli distribution with correlated components. © 2005 Elsevier Inc. All rights reserved. AMS 1991 subject classification: 62E05; 62Hxx Keywords: Correlated multivariate binary responses; Multivariate Bernoulli distribution; Matrix-valued distributions 1. Introduction Matrix-valued distributions are used in continuous multivariate analysis (see, for exam- ple, [10]) to model sample data matrices of continuous measurements, allowing for both variable-dependence and unit-dependence. Their potentials seem to have been neglected for binary, and more generally categorical, data. This is somewhat surprising, since the natural, elementary representation of datasets with categorical variables is precisely in the form of sample binary data matrices, through the 0-1 coding of categories. E-mail address: [email protected] URL: http://dssm.unipa.it/lovison. 1 Work supported by MIUR 60% 2000, 2001 and MIUR P.R.I.N. 2002 grants. 0047-259X/$ - see front matter © 2005 Elsevier Inc. All rights reserved. doi:10.1016/j.jmva.2005.06.008 1574 G.
    [Show full text]
  • The Exciting Guide to Probability Distributions – Part 2
    The Exciting Guide To Probability Distributions – Part 2 Jamie Frost – v1.1 Contents Part 2 A revisit of the multinomial distribution The Dirichlet Distribution The Beta Distribution Conjugate Priors The Gamma Distribution We saw in the last part that the multinomial distribution was over counts of outcomes, given the probability of each outcome and the total number of outcomes. xi f(x1, ... , xk | n, p1, ... , pk)= [n! / ∏xi!] ∏pi The count of The probability of each outcome. each outcome. That’s all smashing, but suppose we wanted to know the reverse, i.e. the probability that the distribution underlying our random variable has outcome probabilities of p1, ... , pk, given that we observed each outcome x1, ... , xk times. In other words, we are considering all the possible probability distributions (p1, ... , pk) that could have generated these counts, rather than all the possible counts given a fixed distribution. Initial attempt at a probability mass function: Just swap the domain and the parameters: The RHS is exactly the same. xi f(p1, ... , pk | n, x1, ... , xk )= [n! / ∏xi!] ∏pi Notational convention is that we define the support as a vector x, so let’s relabel p as x, and the counts x as α... αi f(x1, ... , xk | n, α1, ... , αk )= [n! / ∏ αi!] ∏xi We can define n just as the sum of the counts: αi f(x1, ... , xk | α1, ... , αk )= [(∑αi)! / ∏ αi!] ∏xi But wait, we’re not quite there yet. We know that probabilities have to sum to 1, so we need to restrict the domain we can draw from: s.t.
    [Show full text]
  • Discrete Probability Distributions Uniform Distribution Bernoulli
    Discrete Probability Distributions Uniform Distribution Experiment obeys: all outcomes equally probable Random variable: outcome Probability distribution: if k is the number of possible outcomes, 1 if x is a possible outcome p(x)= k ( 0 otherwise Example: tossing a fair die (k = 6) Bernoulli Distribution Experiment obeys: (1) a single trial with two possible outcomes (success and failure) (2) P trial is successful = p Random variable: number of successful trials (zero or one) Probability distribution: p(x)= px(1 − p)n−x Mean and variance: µ = p, σ2 = p(1 − p) Example: tossing a fair coin once Binomial Distribution Experiment obeys: (1) n repeated trials (2) each trial has two possible outcomes (success and failure) (3) P ith trial is successful = p for all i (4) the trials are independent Random variable: number of successful trials n x n−x Probability distribution: b(x; n,p)= x p (1 − p) Mean and variance: µ = np, σ2 = np(1 − p) Example: tossing a fair coin n times Approximations: (1) b(x; n,p) ≈ p(x; λ = pn) if p ≪ 1, x ≪ n (Poisson approximation) (2) b(x; n,p) ≈ n(x; µ = pn,σ = np(1 − p) ) if np ≫ 1, n(1 − p) ≫ 1 (Normal approximation) p Geometric Distribution Experiment obeys: (1) indeterminate number of repeated trials (2) each trial has two possible outcomes (success and failure) (3) P ith trial is successful = p for all i (4) the trials are independent Random variable: trial number of first successful trial Probability distribution: p(x)= p(1 − p)x−1 1 2 1−p Mean and variance: µ = p , σ = p2 Example: repeated attempts to start
    [Show full text]
  • STATS8: Introduction to Biostatistics 24Pt Random Variables And
    STATS8: Introduction to Biostatistics Random Variables and Probability Distributions Babak Shahbaba Department of Statistics, UCI Random variables • In this lecture, we will discuss random variables and their probability distributions. • Formally, a random variable X assigns a numerical value to each possible outcome (and event) of a random phenomenon. • For instance, we can define X based on possible genotypes of a bi-allelic gene A as follows: 8 < 0 for genotype AA; X = 1 for genotype Aa; : 2 for genotype aa: • Alternatively, we can define a random, Y , variable this way: 0 for genotypes AA and aa; Y = 1 for genotype Aa: Random variables • After we define a random variable, we can find the probabilities for its possible values based on the probabilities for its underlying random phenomenon. • This way, instead of talking about the probabilities for different outcomes and events, we can talk about the probability of different values for a random variable. • For example, suppose P(AA) = 0:49, P(Aa) = 0:42, and P(aa) = 0:09. • Then, we can say that P(X = 0) = 0:49, i.e., X is equal to 0 with probability of 0.49. • Note that the total probability for the random variable is still 1. Random variables • The probability distribution of a random variable specifies its possible values (i.e., its range) and their corresponding probabilities. • For the random variable X defined based on genotypes, the probability distribution can be simply specified as follows: 8 < 0:49 for x = 0; P(X = x) = 0:42 for x = 1; : 0:09 for x = 2: Here, x denotes a specific value (i.e., 0, 1, or 2) of the random variable.
    [Show full text]
  • Chapter 5 Sections
    Chapter 5 Chapter 5 sections Discrete univariate distributions: 5.2 Bernoulli and Binomial distributions Just skim 5.3 Hypergeometric distributions 5.4 Poisson distributions Just skim 5.5 Negative Binomial distributions Continuous univariate distributions: 5.6 Normal distributions 5.7 Gamma distributions Just skim 5.8 Beta distributions Multivariate distributions Just skim 5.9 Multinomial distributions 5.10 Bivariate normal distributions 1 / 43 Chapter 5 5.1 Introduction Families of distributions How: Parameter and Parameter space pf /pdf and cdf - new notation: f (xj parameters ) Mean, variance and the m.g.f. (t) Features, connections to other distributions, approximation Reasoning behind a distribution Why: Natural justification for certain experiments A model for the uncertainty in an experiment All models are wrong, but some are useful – George Box 2 / 43 Chapter 5 5.2 Bernoulli and Binomial distributions Bernoulli distributions Def: Bernoulli distributions – Bernoulli(p) A r.v. X has the Bernoulli distribution with parameter p if P(X = 1) = p and P(X = 0) = 1 − p. The pf of X is px (1 − p)1−x for x = 0; 1 f (xjp) = 0 otherwise Parameter space: p 2 [0; 1] In an experiment with only two possible outcomes, “success” and “failure”, let X = number successes. Then X ∼ Bernoulli(p) where p is the probability of success. E(X) = p, Var(X) = p(1 − p) and (t) = E(etX ) = pet + (1 − p) 8 < 0 for x < 0 The cdf is F(xjp) = 1 − p for 0 ≤ x < 1 : 1 for x ≥ 1 3 / 43 Chapter 5 5.2 Bernoulli and Binomial distributions Binomial distributions Def: Binomial distributions – Binomial(n; p) A r.v.
    [Show full text]
  • 556: MATHEMATICAL STATISTICS I 1 the Multinomial Distribution the Multinomial Distribution Is a Multivariate Generalization of T
    556: MATHEMATICAL STATISTICS I MULTIVARIATE PROBABILITY DISTRIBUTIONS 1 The Multinomial Distribution The multinomial distribution is a multivariate generalization of the binomial distribution. The binomial distribution can be derived from an “infinite urn” model with two types of objects being sampled without replacement. Suppose that the proportion of “Type 1” objects in the urn is θ (so 0 · θ · 1) and hence the proportion of “Type 2” objects in the urn is 1 ¡ θ. Suppose that n objects are sampled, and X is the random variable corresponding to the number of “Type 1” objects in the sample. Then X » Bin(n; θ). Now consider a generalization; suppose that the urn contains k + 1 types of objects (k = 1; 2; :::), with θi being the proportion of Type i objects, for i = 1; :::; k + 1. Let Xi be the random variable corresponding to the number of type i objects in a sample of size n, for i = 1; :::; k. Then the joint T pmf of vector X = (X1; :::; Xk) is given by e n! n! kY+1 f (x ; :::; x ) = θx1 ....θxk θxk+1 = θxi X1;:::;Xk 1 k x !:::x !x ! 1 k k+1 x !:::x !x ! i 1 k k+1 1 k k+1 i=1 where 0 · θi · 1 for all i, and θ1 + ::: + θk + θk+1 = 1, and where xk+1 is defined by xk+1 = n ¡ (x1 + ::: + xk): This is the mass function for the multinomial distribution which reduces to the binomial if k = 1. 2 The Dirichlet Distribution The Dirichlet distribution is a multivariate generalization of the Beta distribution.
    [Show full text]
  • Probability Distributions and Related Mathematical Constructs
    Appendix B More probability distributions and related mathematical constructs This chapter covers probability distributions and related mathematical constructs that are referenced elsewhere in the book but aren’t covered in detail. One of the best places for more detailed information about these and many other important probability distributions is Wikipedia. B.1 The gamma and beta functions The gamma function Γ(x), defined for x> 0, can be thought of as a generalization of the factorial x!. It is defined as ∞ x 1 u Γ(x)= u − e− du Z0 and is available as a function in most statistical software packages such as R. The behavior of the gamma function is simpler than its form may suggest: Γ(1) = 1, and if x > 1, Γ(x)=(x 1)Γ(x 1). This means that if x is a positive integer, then Γ(x)=(x 1)!. − − − The beta function B(α1,α2) is defined as a combination of gamma functions: Γ(α1)Γ(α2) B(α ,α ) def= 1 2 Γ(α1+ α2) The beta function comes up as a normalizing constant for beta distributions (Section 4.4.2). It’s often useful to recognize the following identity: 1 α1 1 α2 1 B(α ,α )= x − (1 x) − dx 1 2 − Z0 239 B.2 The Poisson distribution The Poisson distribution is a generalization of the binomial distribution in which the number of trials n grows arbitrarily large while the mean number of successes πn is held constant. It is traditional to write the mean number of successes as λ; the Poisson probability density function is λy P (y; λ)= eλ (y =0, 1,..
    [Show full text]
  • Lecture 6: Special Probability Distributions
    The Discrete Uniform Distribution The Bernoulli Distribution The Binomial Distribution The Negative Binomial and Geometric Di Lecture 6: Special Probability Distributions Assist. Prof. Dr. Emel YAVUZ DUMAN MCB1007 Introduction to Probability and Statistics Istanbul˙ K¨ult¨ur University The Discrete Uniform Distribution The Bernoulli Distribution The Binomial Distribution The Negative Binomial and Geometric Di Outline 1 The Discrete Uniform Distribution 2 The Bernoulli Distribution 3 The Binomial Distribution 4 The Negative Binomial and Geometric Distribution The Discrete Uniform Distribution The Bernoulli Distribution The Binomial Distribution The Negative Binomial and Geometric Di Outline 1 The Discrete Uniform Distribution 2 The Bernoulli Distribution 3 The Binomial Distribution 4 The Negative Binomial and Geometric Distribution The Discrete Uniform Distribution The Bernoulli Distribution The Binomial Distribution The Negative Binomial and Geometric Di The Discrete Uniform Distribution If a random variable can take on k different values with equal probability, we say that it has a discrete uniform distribution; symbolically, Definition 1 A random variable X has a discrete uniform distribution and it is referred to as a discrete uniform random variable if and only if its probability distribution is given by 1 f (x)= for x = x , x , ··· , xk k 1 2 where xi = xj when i = j. The Discrete Uniform Distribution The Bernoulli Distribution The Binomial Distribution The Negative Binomial and Geometric Di The mean and the variance of this distribution are Mean: k k 1 x1 + x2 + ···+ xk μ = E[X ]= xi f (xi ) = xi · = , k k i=1 i=1 1/k Variance: k 2 2 2 1 σ = E[(X − μ) ]= (xi − μ) · k i=1 2 2 2 (x − μ) +(x − μ) + ···+(xk − μ) = 1 2 k The Discrete Uniform Distribution The Bernoulli Distribution The Binomial Distribution The Negative Binomial and Geometric Di In the special case where xi = i, the discrete uniform distribution 1 becomes f (x)= k for x =1, 2, ··· , k, and in this from it applies, for example, to the number of points we roll with a balanced die.
    [Show full text]
  • 11. Parameter Estimation
    11. Parameter Estimation Chris Piech and Mehran Sahami May 2017 We have learned many different distributions for random variables and all of those distributions had parame- ters: the numbers that you provide as input when you define a random variable. So far when we were working with random variables, we either were explicitly told the values of the parameters, or, we could divine the values by understanding the process that was generating the random variables. What if we don’t know the values of the parameters and we can’t estimate them from our own expert knowl- edge? What if instead of knowing the random variables, we have a lot of examples of data generated with the same underlying distribution? In this chapter we are going to learn formal ways of estimating parameters from data. These ideas are critical for artificial intelligence. Almost all modern machine learning algorithms work like this: (1) specify a probabilistic model that has parameters. (2) Learn the value of those parameters from data. Parameters Before we dive into parameter estimation, first let’s revisit the concept of parameters. Given a model, the parameters are the numbers that yield the actual distribution. In the case of a Bernoulli random variable, the single parameter was the value p. In the case of a Uniform random variable, the parameters are the a and b values that define the min and max value. Here is a list of random variables and the corresponding parameters. From now on, we are going to use the notation q to be a vector of all the parameters: Distribution Parameters Bernoulli(p) q = p Poisson(l) q = l Uniform(a,b) q = (a;b) Normal(m;s 2) q = (m;s 2) Y = mX + b q = (m;b) In the real world often you don’t know the “true” parameters, but you get to observe data.
    [Show full text]
  • Chapter 3. Discrete Random Variables and Probability Distributions
    Chapter 3. Discrete random variables and probability distributions. ::::Defn: A :::::::random::::::::variable (r.v.) is a function that takes a sample point in S and maps it to it a real number. That is, a r.v. Y maps the sample space (its domain) to the real number line (its range), Y : S ! R. ::::::::Example: In an experiment, we roll two dice. Let the r.v. Y = the sum of two dice. The r.v. Y as a function: (1; 1) ! 2 (1; 2) ! 3 . (6; 6) ! 12 Discrete random variables and their distributions ::::Defn: A :::::::discrete::::::::random :::::::variable is a r.v. that takes on a ::::finite or :::::::::countably ::::::infinite number of different values. ::::Defn: The probability mass function (pmf) of a discrete r.v Y is a function that maps each possible value of Y to its probability. (It is also called the probability distribution of a discrete r.v. Y ). pY : Range(Y ) ! [0; 1]: pY (v) = P(Y = v), where ‘Y=v’ denotes the event f! : Y (!) = vg. ::::::::Notation: We use capital letters (like Y ) to denote a random variable, and a lowercase letters (like v) to denote a particular value the r.v. may take. Cumulative distribution function ::::Defn: The cumulative distribution function (cdf) is defined as F(v) = P(Y ≤ v): pY : Range(Y ) ! [0; 1]: Example: We toss two fair coins. Let Y be the number of heads. What is the pmf of Y ? What is the cdf? Quiz X a random variable. values of X: 1 3 5 7 cdf F(a): .5 .75 .9 1 What is P(X ≤ 3)? (A) .15 (B) .25 (C) .5 (D) .75 Quiz X a random variable.
    [Show full text]