The Interpretation of Spectral Entropy Based Upon Rate Distortion Functions
Total Page:16
File Type:pdf, Size:1020Kb
The Interpretation of Spectral Entropy Based Upon Rate Distortion Functions Jaewoo Jung Jerry D. Gibson Department of Electrical and Computer Eng. Department of Electrical and Computer Eng. University of California, Santa Barbara University of California, Santa Barbara Santa Barbara, CA 93106-9560, USA Santa Barbara, CA 93106-9560, USA Email: [email protected] Email: [email protected] Abstract— In 1960 Campbell derived a quantity that he called not hold[6]. Using a different development based upon the coefficient rate which is expressible in terms of the entropy of AEP for continuous random variables, Yang, et al showed the process power spectral density. Later, Yang, et al showed that the equivalent rate per dimension of a random process that the spectral entropy is proportional to the logarithm of the equivalent bandwidth of the smallest frequency band containing falls within a small region about the coefficient rate defined by most of the energy. Gibson et al also showed that for discrete Campbell. Additionally, they showed that the number of times time AR(1) sequences, Campbell’s coefficient rate and Shannon’s that a particular coefficient in the Karhunen-Loeve` expansion entropy rate power are equal but that the equality does not appears in the high energy set is proportional to the variance hold for higher order AR processes. In this paper, we derive of that coefficient[7]. a new expression for Campbell’s coefficient rate in terms of the parametrized version of the rate distortion function of a Gaussian In this paper, we investigate Campbell’s coefficient rate random process with a given power spectral density subject to within the context of rate distortion theory and develop two the MSE fidelity criterion. We also derive expressions for the significant results. First, based upon the classical expression entropy rate power and coefficient rate in terms of the slope of for the rate distortion function in terms of the parameter θ, the rate distortion function for the given source and for a source we derive a new expression for coefficient rate in terms of with flat power spectral density. the bandwidth occupied by the process as a function of θ. I. INTRODUCTION Second, we derive an expression for the spectral entropy in The coefficient rate of a random process was first derived terms of the ratio of the values of θ of a uniform PSD to and defined by Campbell in 1960[1]. Campbell considered an AR PSD for a given value of average distortion D, where the product of N sample functions of a random process, and this ratio is averaged over all values of distortion greater than showed, using an AEP(Asymptotic Equipartition Property)- the distortion associated with the minimum of the PSD. This like argument, that a Karhunen-Loeve` expansion of this prod- result is contrasted with a similar result for the entropy rate uct could be separated into two sets - one set with average power. power very close to that of the product and the other set The paper is organized as follows. Section II contains rate having very low average power. Asymptotically in the number distortion theory basics used in the remainder of the paper. of sample functions forming the product and in the support Section III considers the averaging of the parametrized rate interval of the process, he showed that the average number distortion function for a Gaussian source with respect to the of terms in the high power set approached a quantity that he MSE fidelity criterion over all values of the parameter θ and interpreted as a coefficient rate given by presents the resulting new expression for coefficient rate in h Z 1 i terms of the bandwidth as a function of θ. Section IV contains Q2 = exp ¡ S(f) log S(f)df ; (1) the derivation of new expressions for the spectral entropy in ¡1 terms of the ratio of the values of θ for a uniform PSD and an where we denote the quantity in the exponent as the spectral AR PSD at a given average distortion and in terms of the ratio entropy (we use Q2 for the coefficient rate and reserve Q1 for of the slopes of the rate distortion functions corresponding to Shannon’s entropy rate power[2],[3],[4]). these two PSDs. A related result for the entropy rate power Campbell did not investigate further the implications of co- is also derived. Section V contains conclusions and future efficient rate for signal processing or for source compression. research directions. Abramson also considered Campbell’s coefficient rate and noted that an approach to source compression using coefficient II. RATE-DISTORTION CURVE rate did not exist and was not immediately obvious[5]. Gibson, et al studied the relationship of coefficient rate For use in subsequent sections, we note the following and Shannon’s entropy rate power and proved that for first theorem. order autoregressive processes (AR(1)), these quantities are Theorem (Berger [2]) Let fXt; t = 0; §1; ¢ ¢ ¢; g be a equal, but that for higher order AR processes, equality does time-discrete stationary Gaussian source with spectral density function 3 X1 ¡jk2¼f S(f) = Áke : (2) 2.5 k=¡1 2 Then the MSE rate distortion function of fXtg has the θ parametric representation 1.5 Z 0:5 1 Dθ = min[θ; S(f)]df; (3) ¡0:5 0.5 and Z 0:5 1 S(f) 0 −0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.5 R(Dθ) = max[0; log ]df: (4) −xθ xθ 2 ¡0:5 θ The nonzero portion of the R(D) curve is generated as Fig. 1. Threshold θ and PSD of an AR(1) process (r=0.5) the parameter θ transverses the interval 0 · θ · ¢ = ess sup S(f). The entropy power, or entropy rate power, of a stationary where 2W (θ) is the bandwidth where the magnitude of the Gaussian sequence fXtg is PSD is greater than the threshold θ. In the case of the AR(1) process in Fig. 1, 2W (θ) = 2x . h Z0:5 i θ The first term of the right-hand side of Eq. (8) has the form Q1 = exp log S(f)df : (5) of the spectral entropy. If we multiply Eq. (8) by 2, ¡0:5 Z Z 0:5 a 1 The right side of Eq. (5) is the formula for the one-step ahead ¡ S(f) log S(f)df = ¡2E[R(Dθ)]+ 2W (θ)(log )dθ: prediction error of a stationary process with power spectral ¡0:5 0 θ density (PSD) S(f)[2]. The expected value of rate in Eq. (7) is ¯S(f) Z 0:5 n ¯ o III. EXPECTED VALUE OF RATE 1 ¯ E[R(Dθ)] = [log S(f)θ ¡ (θ log θ ¡ θ)]¯ df In transform coding, the actual bandwidth of a coded ¡0:5 2 ¯ Z 0 sequence, i.e., the number of coefficients that need to be 0:5 n1 o coded, depends on the distortion level θ. We assume that the = S(f) df: 2 distribution of θ is uniform over [0; a] where a is the maximum ¡0:5 Assuming the source is unit-variance Gaussian and value of the PSD, and calculate the expected value of R(Dθ) R 0:5 in the above Theorem as ¡0:5 S(f)df = 1, Z a 1 E[R(D )] = R(D )dθ (6) E[R(Dθ)] = : (9) θ θ 2 Z0 Z a n 0:5 1 S(f) o The expected value of the rate is always 1/2, regardless of the = max[0; log ]df dθ 2 θ shape of the PSD. Z0 ¡Z0:5 From the expected value of rate, we can derive the spectral 0:5 n a 1 S(f) o = max[0; log ]dθ df; entropy in another form as 2 θ Z Z ¡0:5 0 0:5 a 1 by changing the order of integration. ¡ S(f) log S(f)df = ¡1 + 2W (θ)(log )dθ: (10) ¡0:5 0 θ For a fixed f, the max function of the integrand is ( The coefficient rate Q2 is 1 S(f) Z a 1 S(f) log 0 · θ · S(f) 1 max[0; log ] = 2 θ Q2 = expf¡1 + 2W (θ)(log )dθg: (11) 2 θ 0 S(f) · θ · a: 0 θ The bandwidth 2W (θ) affects Q . We can rewrite Q as The range of integration can be divided into two parts, 2 2 Z a Z Z Z 1 0:5 n S(f) 1 S(f) a o expf 2W (θ)(log )dθg E[R(D )] = log dθ + 0dθ df 0 θ θ Q2 = Z : (12) ¡0:5 0 2 θ S(f) 1 1 expf (log )dθg Z Z 0 θ 0:5 n S(f) 1 o = [log S(f) ¡ log θ]dθ df (7) The limits of the integral in the denominator are the same as ¡0:5 0 2 those in the numerator in the case of uniform power spectral Z Z density, i.e.,a = 1 and 2W (θ) = 1. We infer from Eq. (12) 1 0:5 1 a 1 that the uniform power spectral density is a reference power = S(f) log S(f)df + 2W (θ)(log )dθ; (8) 2 ¡0:5 2 0 θ spectral density for Q2. IV. COEFFICIENT RATE AND ENTROPY POWER FROM The derivative of Dθ with respect to θ is RATE-DISTORTION CURVES @D @x @x θ = 2[x ¡ x ] + 2θ[ 2θ ¡ 1θ ] To investigate the physical meaning of the spectral entropy, @θ 2θ 1θ @θ @θ the coefficient rate and entropy power are derived from the @x @x +2S(x ) 1θ ¡ 2S(x ) 2θ : rate-distortion curves.