A Lower Bound on the Differential Entropy of Log-Concave Random Vectors with Applications
Total Page:16
File Type:pdf, Size:1020Kb
Load more
Recommended publications
-
An Unforeseen Equivalence Between Uncertainty and Entropy
An Unforeseen Equivalence between Uncertainty and Entropy Tim Muller1 University of Nottingham [email protected] Abstract. Beta models are trust models that use a Bayesian notion of evidence. In that paradigm, evidence comes in the form of observing whether an agent has acted well or not. For Beta models, uncertainty is the inverse of the amount of evidence. Information theory, on the other hand, offers a fundamentally different approach to the issue of lacking knowledge. The entropy of a random variable is determined by the shape of its distribution, not by the evidence used to obtain it. However, we dis- cover that a specific entropy measure (EDRB) coincides with uncertainty (in the Beta model). EDRB is the expected Kullback-Leibler divergence between two Bernouilli trials with parameters randomly selected from the distribution. EDRB allows us to apply the notion of uncertainty to other distributions that may occur when generalising Beta models. Keywords: Uncertainty · Entropy · Information Theory · Beta model · Subjective Logic 1 Introduction The Beta model paradigm is a powerful formal approach to studying trust. Bayesian logic is at the core of the Beta model: \agents with high integrity be- have honestly" becomes \honest behaviour evidences high integrity". Its simplest incarnation is to apply Beta distributions naively, and this approach has limited success. However, more powerful and sophisticated approaches are widespread (e.g. [3,13,17]). A commonality among many approaches, is that more evidence (in the form of observing instances of behaviour) yields more certainty of an opinion. Uncertainty is inversely proportional to the amount of evidence. Evidence is often used in machine learning. -
On Relations Between the Relative Entropy and $\Chi^ 2$-Divergence
On Relations Between the Relative Entropy and χ2-Divergence, Generalizations and Applications Tomohiro Nishiyama Igal Sason Abstract The relative entropy and chi-squared divergence are fundamental divergence measures in information theory and statistics. This paper is focused on a study of integral relations between the two divergences, the implications of these relations, their information-theoretic applications, and some generalizations pertaining to the rich class of f-divergences. Applications that are studied in this paper refer to lossless compression, the method of types and large deviations, strong data-processing inequalities, bounds on contraction coefficients and maximal correlation, and the convergence rate to stationarity of a type of discrete-time Markov chains. Keywords: Relative entropy, chi-squared divergence, f-divergences, method of types, large deviations, strong data-processing inequalities, information contraction, maximal correlation, Markov chains. I. INTRODUCTION The relative entropy (also known as the Kullback–Leibler divergence [28]) and the chi-squared divergence [46] are divergence measures which play a key role in information theory, statistics, learning, signal processing and other theoretical and applied branches of mathematics. These divergence measures are fundamental in problems pertaining to source and channel coding, combinatorics and large deviations theory, goodness-of-fit and independence tests in statistics, expectation-maximization iterative algorithms for estimating a distribution from an incomplete -
Guaranteed Bounds on Information-Theoretic Measures of Univariate Mixtures Using Piecewise Log-Sum-Exp Inequalities
Preprints (www.preprints.org) | NOT PEER-REVIEWED | Posted: 20 October 2016 doi:10.20944/preprints201610.0086.v1 Peer-reviewed version available at Entropy 2016, 18, 442; doi:10.3390/e18120442 Article Guaranteed Bounds on Information-Theoretic Measures of Univariate Mixtures Using Piecewise Log-Sum-Exp Inequalities Frank Nielsen 1,2,* and Ke Sun 1 1 École Polytechnique, Palaiseau 91128, France; [email protected] 2 Sony Computer Science Laboratories Inc., Paris 75005, France * Correspondence: [email protected] Abstract: Information-theoretic measures such as the entropy, cross-entropy and the Kullback-Leibler divergence between two mixture models is a core primitive in many signal processing tasks. Since the Kullback-Leibler divergence of mixtures provably does not admit a closed-form formula, it is in practice either estimated using costly Monte-Carlo stochastic integration, approximated, or bounded using various techniques. We present a fast and generic method that builds algorithmically closed-form lower and upper bounds on the entropy, the cross-entropy and the Kullback-Leibler divergence of mixtures. We illustrate the versatile method by reporting on our experiments for approximating the Kullback-Leibler divergence between univariate exponential mixtures, Gaussian mixtures, Rayleigh mixtures, and Gamma mixtures. Keywords: information geometry; mixture models; log-sum-exp bounds 1. Introduction Mixture models are commonly used in signal processing. A typical scenario is to use mixture models [1–3] to smoothly model histograms. For example, Gaussian Mixture Models (GMMs) can be used to convert grey-valued images into binary images by building a GMM fitting the image intensity histogram and then choosing the threshold as the average of the Gaussian means [1] to binarize the image. -
Specific Differential Entropy Rate Estimation for Continuous-Valued Time Series
Specific Differential Entropy Rate Estimation for Continuous-Valued Time Series David Darmon1 1Department of Military and Emergency Medicine, Uniformed Services University, Bethesda, MD 20814, USA October 16, 2018 Abstract We introduce a method for quantifying the inherent unpredictability of a continuous-valued time series via an extension of the differential Shan- non entropy rate. Our extension, the specific entropy rate, quantifies the amount of predictive uncertainty associated with a specific state, rather than averaged over all states. We relate the specific entropy rate to pop- ular `complexity' measures such as Approximate and Sample Entropies. We provide a data-driven approach for estimating the specific entropy rate of an observed time series. Finally, we consider three case studies of estimating specific entropy rate from synthetic and physiological data relevant to the analysis of heart rate variability. 1 Introduction The analysis of time series resulting from complex systems must often be per- formed `blind': in many cases, mechanistic or phenomenological models are not available because of the inherent difficulty in formulating accurate models for complex systems. In this case, a typical analysis may assume that the data are the model, and attempt to generalize from the data in hand to the system. arXiv:1606.02615v1 [cs.LG] 8 Jun 2016 For example, a common question to ask about a times series is how `complex' it is, where we place complex in quotes to emphasize the lack of a satisfactory definition of complexity at present [60]. An answer is then sought that agrees with a particular intuition about what makes a system complex: for example, trajectories from periodic and entirely random systems appear simple, while tra- jectories from chaotic systems appear quite complicated. -
"Differential Entropy". In: Elements of Information Theory
Elements of Information Theory Thomas M. Cover, Joy A. Thomas Copyright 1991 John Wiley & Sons, Inc. Print ISBN 0-471-06259-6 Online ISBN 0-471-20061-1 Chapter 9 Differential Entropy We now introduce the concept of differential entropy, which is the entropy of a continuous random variable. Differential entropy is also related to the shortest description length, and is similar in many ways to the entropy of a discrete random variable. But there are some important differences, and there is need for some care in using the concept. 9.1 DEFINITIONS Definition: Let X be a random variable with cumulative distribution function F(x) = Pr(X I x). If F(x) is continuous, the random variable is said to be continuous. Let fix) = F’(x) when the derivative is defined. If J”co fb> = 1, th en fl x 1 is called the probability density function for X. The set where f(x) > 0 is called the support set of X. Definition: The differential entropy h(X) of a continuous random vari- able X with a density fix) is defined as h(X) = - f(x) log f(x) dx , (9.1) where S is the support set of the random variable. As in the discrete case, the differential entropy depends only on the probability density of the random variable, and hence the differential entropy is sometimes written as h(f) rather than h(X). 2.24 9.2 THE AEP FOR CONTZNUOUS RANDOM VARIABLES 225 Remark: As in every example involving an integral, or even a density, we should include the statement if it exists. -
Entropy Power Inequalities the American Institute of Mathematics
Entropy power inequalities The American Institute of Mathematics The following compilation of participant contributions is only intended as a lead-in to the AIM workshop “Entropy power inequalities.” This material is not for public distribution. Corrections and new material are welcomed and can be sent to [email protected] Version: Fri Apr 28 17:41:04 2017 1 2 Table of Contents A.ParticipantContributions . 3 1. Anantharam, Venkatachalam 2. Bobkov, Sergey 3. Bustin, Ronit 4. Han, Guangyue 5. Jog, Varun 6. Johnson, Oliver 7. Kagan, Abram 8. Livshyts, Galyna 9. Madiman, Mokshay 10. Nayar, Piotr 11. Tkocz, Tomasz 3 Chapter A: Participant Contributions A.1 Anantharam, Venkatachalam EPIs for discrete random variables. Analogs of the EPI for intrinsic volumes. Evolution of the differential entropy under the heat equation. A.2 Bobkov, Sergey Together with Arnaud Marsliglietti we have been interested in extensions of the EPI to the more general R´enyi entropies. For a random vector X in Rn with density f, R´enyi’s entropy of order α> 0 is defined as − 2 n(α−1) α Nα(X)= f(x) dx , Z 2 which in the limit as α 1 becomes the entropy power N(x) = exp n f(x)log f(x) dx . However, the extension→ of the usual EPI cannot be of the form − R Nα(X + Y ) Nα(X)+ Nα(Y ), ≥ since the latter turns out to be false in general (even when X is normal and Y is nearly normal). Therefore, some modifications have to be done. As a natural variant, we consider proper powers of Nα. -
January 28, 2021 1 Dealing with Infinite Universes
Information and Coding Theory Winter 2021 Lecture 6: January 28, 2021 Lecturer: Madhur Tulsiani 1 Dealing with infinite universes So far, we have only considered random variables taking values over a finite universe. We now consider how to define the various information theoretic quantities, when the set of possible values is not finite. 1.1 Countable universes When the universe is countable, various information theoretic quantities such as entropy an KL-divergence can be defined essentially as before. Of course, since we now have infinite sums in the definitions, these should be treated as limits of the appropriate series. Hence, all quantities are defined as limits of the corresponding series, when the limit exists. Convergence is usually not a problem, but it is possible to construct examples where the entropy is infinite. Consider the case of U = N, and a probability distribution P satisfying ∑x2N p(x) = 1. Since the sequence ∑x p(x) converges, usually the terms of ∑x p(x) · log(1/p(x)) are not much larger. However, we can construct an example using a the fact that ∑n≥2 1/(k · (log k) ) converges if an only if a > 1. Define C 1 1 p(x) = 8x ≥ 2 where lim = . 2 n!¥ ∑ 2 x · (log x) 2≤x≤n x · (log x) C Then, for a random variable X distributed according to P, C x · (log x)2 ( ) = · = H X ∑ 2 log ¥ . x≥2 x · (log x) C Exercise 1.1. Calculate H(X) when X be a geometric random variable with P [X = n] = (1 − p)n−1 · p 8n ≥ 1 1 1.2 Uncountable universes When the universe is not countable, one has to use measure theory to define the appropri- ate information theoretic quantities (actually, it is the KL-divergence which is defined this way). -
On the Entropy of Sums
On the Entropy of Sums Mokshay Madiman Department of Statistics Yale University Email: [email protected] Abstract— It is shown that the entropy of a sum of independent II. PRELIMINARIES random vectors is a submodular set function, and upper bounds on the entropy of sums are obtained as a result in both discrete Let X1,X2,...,Xn be a collection of random variables and continuous settings. These inequalities complement the lower taking values in some linear space, so that addition is a bounds provided by the entropy power inequalities of Madiman well-defined operation. We assume that the joint distribu- and Barron (2007). As applications, new inequalities for the tion has a density f with respect to some reference mea- determinants of sums of positive-definite matrices are presented. sure. This allows the definition of the entropy of various random variables depending on X1,...,Xn. For instance, I. INTRODUCTION H(X1,X2,...,Xn) = −E[log f(X1,X2,...,Xn)]. There are the familiar two canonical cases: (a) the random variables Entropies of sums are not as well understood as joint are real-valued and possess a probability density function, or entropies. Indeed, for the joint entropy, there is an elaborate (b) they are discrete. In the former case, H represents the history of entropy inequalities starting with the chain rule of differential entropy, and in the latter case, H represents the Shannon, whose major developments include works of Han, discrete entropy. We simply call H the entropy in all cases, ˇ Shearer, Fujishige, Yeung, Matu´s, and others. The earlier part and where the distinction matters, the relevant assumption will of this work, involving so-called Shannon-type inequalities be made explicit. -
Relative Entropy and Score Function: New Information–Estimation Relationships Through Arbitrary Additive Perturbation
Relative Entropy and Score Function: New Information–Estimation Relationships through Arbitrary Additive Perturbation Dongning Guo Department of Electrical Engineering & Computer Science Northwestern University Evanston, IL 60208, USA Abstract—This paper establishes new information–estimation error (MMSE) of a Gaussian model [2]: relationships pertaining to models with additive noise of arbitrary d p 1 distribution. In particular, we study the change in the relative I (X; γ X + N) = mmse PX ; γ (2) entropy between two probability measures when both of them dγ 2 are perturbed by a small amount of the same additive noise. where X ∼ P and mmseP ; γ denotes the MMSE of It is shown that the rate of the change with respect to the X p X energy of the perturbation can be expressed in terms of the mean estimating X given γ X + N. The parameter γ ≥ 0 is squared difference of the score functions of the two distributions, understood as the signal-to-noise ratio (SNR) of the Gaussian and, rather surprisingly, is unrelated to the distribution of the model. By-products of formula (2) include the representation perturbation otherwise. The result holds true for the classical of the entropy, differential entropy and the non-Gaussianness relative entropy (or Kullback–Leibler distance), as well as two of its generalizations: Renyi’s´ relative entropy and the f-divergence. (measured in relative entropy) in terms of the MMSE [2]–[4]. The result generalizes a recent relationship between the relative Several generalizations and extensions of the previous results entropy and mean squared errors pertaining to Gaussian noise are found in [5]–[7]. -
A Probabilistic Upper Bound on Differential Entropy Joseph Destefano, Member, IEEE, and Erik Learned-Miller
1 A Probabilistic Upper Bound on Differential Entropy Joseph DeStefano, Member, IEEE, and Erik Learned-Miller 1 Abstract— A novel, non-trivial, probabilistic upper bound on the entropy of an unknown one-dimensional distribution, given 0.9 the support of the distribution and a sample from that dis- tribution, is presented. No knowledge beyond the support of 0.8 the unknown distribution is required, nor is the distribution required to have a density. Previous distribution-free bounds on 0.7 the cumulative distribution function of a random variable given Upper bound a sample of that variable are used to construct the bound. A 0.6 simple, fast, and intuitive algorithm for computing the entropy bound from a sample is provided. 0.5 Index Terms— Differential entropy, entropy bound, string- 0.4 Empirical cumulative tightening algorithm 0.3 Lower bound I. INTRODUCTION 0.2 The differential entropy of a distribution [6] is a quantity 0.1 employed ubiquitously in communications, statistical learning, 0 physics, and many other fields. If is a one-dimensional 0 0.5 1 1.5 2 2.5 3 3.5 4 random variable with absolutely continuous distribution ¡£¢¥¤§¦ and density ¨©¢¥¤§¦ , the differential entropy of is defined to Fig. 1. This figure shows a typical empirical cumulative distribution (blue). The red lines show us the probabilistic bounds provided by Equation 2 on the be true cumulative. ¦ ¨©¢¥¤§¦¨©¢¤¦©¤ ¢ (1) For our purposes, the distribution need not have a density. II. THE BOUND If there are discontinuities in the distribution function ¡£¢¥¤§¦ , 1 ¤+- ¤. Given , samples, through , from an unknown dis- then no density exists and the entropy is ! . -
Cramér-Rao and Moment-Entropy Inequalities for Renyi Entropy And
1 Cramer-Rao´ and moment-entropy inequalities for Renyi entropy and generalized Fisher information Erwin Lutwak, Deane Yang, and Gaoyong Zhang Abstract— The moment-entropy inequality shows that a con- Analogues for convex and star bodies of the moment- tinuous random variable with given second moment and maximal entropy, Fisher information-entropy, and Cramer-Rao´ inequal- Shannon entropy must be Gaussian. Stam’s inequality shows that ities had been established earlier by the authors [3], [4], [5], a continuous random variable with given Fisher information and minimal Shannon entropy must also be Gaussian. The Cramer-´ [6], [7] Rao inequality is a direct consequence of these two inequalities. In this paper the inequalities above are extended to Renyi II. DEFINITIONS entropy, p-th moment, and generalized Fisher information. Gen- Throughout this paper, unless otherwise indicated, all inte- eralized Gaussian random densities are introduced and shown to be the extremal densities for the new inequalities. An extension of grals are with respect to Lebesgue measure over the real line the Cramer–Rao´ inequality is derived as a consequence of these R. All densities are probability densities on R. moment and Fisher information inequalities. Index Terms— entropy, Renyi entropy, moment, Fisher infor- A. Entropy mation, information theory, information measure The Shannon entropy of a density f is defined to be Z h[f] = − f log f, (1) I. INTRODUCTION R provided that the integral above exists. For λ > 0 the λ-Renyi HE moment-entropy inequality shows that a continuous entropy power of a density is defined to be random variable with given second moment and maximal T 1 Shannon entropy must be Gaussian (see, for example, Theorem Z 1−λ λ 9.6.5 in [1]). -
Chapter 5 Differential Entropy and Gaussian Channels
Chapter 5 Differential Entropy and Gaussian Channels Po-Ning Chen, Professor Institute of Communications Engineering National Chiao Tung University Hsin Chu, Taiwan 30010, R.O.C. Continuous sources I: 5-1 • Model {Xt ∈X,t∈ I} – Discrete sources ∗ Both X and I are discrete. – Continuous sources ∗ Discrete-time continuous sources ·Xis continuous; I is discrete. ∗ Waveform sources · Both X and I are continuous. • We have so far examined information measures and their operational charac- terization for discrete-time discrete-alphabet systems. In this chapter, we turn our focus to discrete-time continuous-alphabet (real-valued) sources. Information content of continuous sources I: 5-2 • If the random variable takes on values in a continuum, the minimum number of bits per symbol needed to losslessly describe it must be infinite. • This is illustrated in the following example and validated in Lemma 5.2. Example 5.1 – Consider a real-valued random variable X that is uniformly distributed on the unit interval, i.e., with pdf given by 1ifx ∈ [0, 1); fX(x)= 0otherwise. – Given a positive integer m, we can discretize X by uniformly quantizing it into m levels by partitioning the support of X into equal-length segments 1 of size ∆ = m (∆ is called the quantization step-size) such that: i i − 1 i q (X)= , if ≤ X< , m m m m for 1 ≤ i ≤ m. – Then the entropy of the quantized random variable qm(X)isgivenby m 1 1 H(q (X)) = − log2 =log2 m (in bits). m m m i=1 Information content of continuous sources I: 5-3 – Since the entropy H(qm(X)) of the quantized version of X is a lower bound to the entropy of X (as qm(X) is a function of X) and satisfies in the limit lim H(qm(X)) = lim log2 m = ∞, m→∞ m→∞ we obtain that the entropy of X is infinite.