IEEE ~NSACnONS ON INFORMATION THRORY, VOL. m-20, NO. 4, JULY 1974 517 On the &-Entropy and the Rate-Distortion Function of Certain Non-Gaussian Processes
JACOB BINIA, MOSHE ZAKAI, FELLOW, IEEE AND JACOB ZIV, FELLOW, IEEE
Abstract--Let C = {5(t), 0 5 t I T} be a process with covariance and (2) to random processes or to infinite-dimensional function K(s,t ) and E jf t'(t) dt < co. It is proved that for every E > 0 random variables is impossible, since the results are based the E-entropy H,(T) satisfies on the existence of a probability density. Namely, the probability measure of 5 is required to be absolutely where & is a Gaussian process with the covarhmce K&t) and &‘;,(T) is continuous with respect to the Lebesguemeasure. In this the entropy of the measure induced by c (in function space) with respect paper we replace this requirement by a requirement of to that induced by &,. It is also shown that if ze,(T) < co, then, as E + 0 absolute continuity with respect to a Gaussian measure. This enables us to prove the following bounds on the s-entropy of a random process 5 = {c(t), 0 I t < T} : Furthermore, if there exists a Gaussian process g = {g(t); 0 5 t 5 T} such that y%(T) < co, then the ratio between H,(5) and H,(g) goes to one f4(5,) - qJ3 5 f&(C) 5 f&(5,) (3) as E goes to zero. Similar results are given for the rate-distortion function, and some particular examples are worked out in detail. Some cases for where 5, is the Gaussian processwith the same covariance which A$,(@ = 00 are discussed, and asymptotic bounds on H,(T), as that of 5 and &&(e) is the relative entropy of the measure expressed in terms of H&&J, are derived. induced by < with respectto that induced by 5, (cf., Section II). Furthermore, if Xc,(t) < co, then we show that for I. INTRODUCTION E-+0 HE e-ENTROPY (and its related normalized form, f&(r) = f&(5> - JqJ3 + 4). (4) the rate-distortion function) provides an important mathematicalT tool for the analysis of communication In fact, for a finite-dimensional random variable 5, the sources and systems. Given a communication system, the precedinglower bound on H,(t), as well as the a symptotic s-entropy of the source and the channel capacity yield a behavior (4) are stronger than previously known results lower bound on the minimum attainable distortion [l]. (the upper bound on H,(e), for N-dimensional random Let 5 be a real-valued random variable and denote by variables, is already known; [l, sec. 4.6.21. H,(c) the s-entropy (rate-distortion function) of 5, relative The results in (3) and (4) on H,(c) are expressedin terms to a mean-square-errorcriterion. A well-known result of of the s-entropy of a related Gaussian process H,(&) and Shannon [2], [3] states that, if the probability distribution the entropy of the measureof t; with respect to the measure of 5 possessesa density, then of the related Gaussian processt,. Results on the a-entropy of Gaussian processesare available in the literature [l], 1 tT2 [6]-[lo]. The entropy Xg(l) of 5 with respect to some h(5) + $ In -2nee2 5 K(t) 5 3 ln ~z (1) Gaussian processesg can be derived from known results on the Radon-Nikodym derivative of certain processeswith where r? denotes the variance of 5 and h(5) = -f P&C) * respectto certain Gaussianprocesses [ 1I]-[ 131.It is shown In P&X) dx. Furthermore, it was shown by Gerrish and that if there exists a Gaussian processg for which y10,(5)< Schultheiss[4] that as E -+ 0 co, then &$(<) < co. Therefore the bounds (3) also yield asymptotic results, for E + 0. Several examples are given, f&(t) = ME)+ 3 ln & + 41). and in one of these examples we show that if d{(t) = c(t) dt + p &v(t) (where w(t) is a standard Brownian Results of the type (1) and (2) have been extended for motion) then as E + 0 N-dimensional random variables, provided that the random variables possessa probability density [l], [4], and [5]. The purpose of this paper is to derive results of the type (1) and (2) for random processes.A direct extension of (1) which is a generalization of previously known results
Manuscript received June 5, 1973; revised January 2,. 1974. PI, WI. J. Binia is with the Armament Development Authority, Ministry of Section II is devoted to notation and certain preliminary Defense, Israel. results. The main results and examplesare given in Section M. Zakai and J. Ziv are with the Faculty of Engineering, Technion- Israel Institute of Technology, Israel. III, and their proofs are given in Section IV. 518 IEEE TRANSACTIONS ON INFORMATION THEORY, JULY 1974
II. PRELIMINARIES AND NOTATION Then we have Let P, and P, be two probability measures defined on a measurablespace (C&B)and let {Ei} be a finite B-measurable partition of R. The entropy &‘pzCP1) of P, with respect to Remarks: a) From now on, the discussion is limited to P, is defined by [14, ch. 21 the metrics of L2,12. Although the results will be stated for processesin L’[O,T], they hold for all processesdefined in &&(PI) = sup C P,(E,) In z spaces isometric to L’[O,T]. b) The random function E(t), 1 2 i 0 I t I Twill be assumedto be measurable, separable, and satisfying E JE (f’(t) dt < co. Also, since the a-entropy is where the supremum is taken over all partitions of R. Let independent of the process mean, we always assume P, and P, be the distributions of the random variables t and q (taking values in the same space (X,B,.)), respectively. E<(t) = 0. Lemma 2.2: Let c,&,, n = 1,2,* * . be a sequence of In this case the notation for the entropy will be random processessuch that Jq5) = G,(Py) = xP*(Pl). lim E ,,T [@(t) - 5,(t)]’ dt = 0. (5) Let 5 and r] be random variables with values in the n+cc s measurable spaces (X,B,) and (YJ,,), respectively. The mutual information 1(t,r) between these variables is Then for every E > 0 defined as [14, ch. 21 lim K(L) = f&(4>. (6) "-CC Z(t,v) = sup c PCq(Ei x Fj) In ‘@I(~’ ’ Fj) Proof: We first show that if c1 and c2 are random i,j P<(Ei>Pq(Fj) processessuch that where the supremum is taken over all partitions {Ei} of X and (Fj} of Y. E 0T [(l(t) - t2(t)12 dt 5 8’ (7) The s-entropy H,(r) of a random variable 5 taking values s in a metric space (X,&) with a metric p(x,y), x,y E X, is then defined as f&+&2) 5 W51) 5 fLdt2)~ & 2 8 > 0. (8) By the definition of the s-entropy, there exists a process YI where the infimum is taken over all joint distributions Ps, such that (defined on (X x X, B, x B,.)), with a fixed marginal E 0T [cl(t) - q(t)]’ dt I E2 (9) distribution P,;, such that s EP2(5,1) s E2. and The following lemma shows that the s-entropy is inde- f&(51) 2 z(tl,r) - 6 (10) pendent of the representation of a process e in different where 6 > 0 is arbitrary. Note that we can choose a version isometric spaces. of the preceding process VIsuch that r~and t2 are condition- Lemma 2.1: Let A be a linear, one-to-one and bi- ally independent, conditioned on ti ; therefore [14, sect. 3.41 measurable transformation from the metric space (X&J onto the metric space@,&) and let Pt; be the distribution G52,d 5 N52,51M = I
The proof of Lemma 2.1 follows directly from the trans- %+,(tz) 5 m2Pd s mdi9 5 m-1) + 6. formation properties, since such a transformation preserves This completes the proof of the left side of (8); the right the amount of information [15]. side of (8) follows by a similar argument. In particular, let 5 = {g(t), 0 I t I T} be a stochastic Equation (6) follows from (5), (8) and the continuity of process with E jz c2(t) dt < co. Let {pi} be a complete H,(t) as a function of E. orthonormal system in I?[O,T], and consider the following Remark: Let g = {t(t), 0 I t I T} be a process with transformation A from L’[O,T] to 1’: E jt t’(t) dt c co. Suppose that < is passed through a A: x(t) + (x1,x2, * . -1, x(t) E L’[O,T] linear filter with transfer function h(z), given by where 1 0 Denote by & the filter output in [O,T]. Then obviously & and is a quadratic-mean-continuousprocess and Note that since the entropy is always nonnegative [14], liiisoT[t(t) - &(t)] d=t0 2. Lemma 3.1 implies This result will allow us to assume throughout the proofs that 5 is mean-continuous,and the results will remain valid In the case that there exists a Gaussian processg such that without this assumption by Lemma 2.2. Xg(t) < co, we use Lemma 3.1 and the following known Finally, the rate-distortion function R<(D) of a process results about the behavior of the two equivalent Gaussian e = {c(t), 0 I t < co} is defined as ([l], [16]) processesF, and g. a) For equivalent Gaussian processes5 and &, it has been proved under some generalconditions [8] that where to’ = {C(t), 0 I t I; T}. Pinsker [17] proved, f&(5,) = K(g)> & -+ 0. under some general conditions, that the rate-distortion That is, H,(<,) and H,(g) are “asymptotically equal,” in the function R&o) of a stationary processt is well defined and sensethat their ratio goesto one as E goesto zero. finite for D > 0. b) Suppose that 5, and g are also strongly equivalent. The importance of the rate-distortion function in (Denote by K(s,t), 0 _< s,t 5 T, the covariance function information theory stems from the following facts: 1) it of a process t and consider the integral operator K defined provides general lower bounds on the performance of on the spaceL ’[O,T] by communication systems and 2) in some cases a coding T theorem holds to the effect that the performance bounds Kf(t) = K(vJf(4 ds, f E L2[O,T], t E [O,T]. derived from the rate-distortion function can be approached s 0 asymptotically by using appropriate delays [l]. Two Gaussian measures P, and P2, with covariance functions K,(s,t) and K,(s,t), respectively,are defined to be III. MAIN RESULTS AND EXAMPLES strongly equivalent if the operator Let 5 = {r(t), 0 I t I T} be a random process with a K = I _ K-1’2K K-112 covariance function K&s,t), 0 < t,s 5 T. Let 5, be a 2 1 2 Gaussianprocess with the samecovariance function. Let Pt; is of trace class, namely if xi IAil < co, where {&} are the and Peg be the measuresinduced by < and &, respectively,in eigenvaluesof K. [18].) function space (namely the real-valued, Borel-measurable Then it has been shown that [lo] space over [O,T]), and denote by y%&(t) the entropy of P, f&(5,) = H,(g) + o(l), E-+0 with respect to P<,. The following theorem extends the upper and lower bounds (Shannon, [2], [3]) on the which is tighter than the preceding asymptotic equality. s-entropy of a one-dimensionalrandom variable to the case Hence, the following theorem follows from Theorem 1, of a random process. Lemma 3.1, and the asymptotic behavior of the E-entropies of equivalent and strongly equivalent Gaussian processes. Theorem 1: @orevery E > 0 Theorem 2: If there exists a Gaussian process g such that yiG$(<)< co, then If &Jo c cc then as E --, 0 f&C3 = He(g), E -+ 0. (18) fm> = f&(4,) - ~&~ + 41). (14) If, in addition, the Gaussian processesc!$ and g are strongly Theorem 1 relates the s-entropy of a process 5 to the equivalent, then s-entropy of & and to Z<,(e). In the literature the absolute H,(t) = H,(g) + o(l), E -+ 0. (19) continuity of some non-Gaussian processeswith respect to Gaussian measures are discussed. Furthermore, the Example I: Let 4 be the solution of the stochastic s-entropy of these Gaussian processesis known for E + 0 equation (Cl], [6]-[lo]). Therefore, in the following we try to 5(t) = m + ’ dIt tgi = T tg(tMi(t) dt, i = 1,2;**. (28) Example 2: Let the process &,, be defined by s 0 Theorem 4: Let 4 be a mean-continuous process with t-z(t) = f Tt(s> ds, s 0 E j; r2(t) dt < co. 4 If Y%;<,,, .,GL)G,. - *,tL) = o(L) and if the eigen- values of the process satisfy L(t) = s’ L-t(s) ds, O R,(D) x ; 2r + 2 (2r+2)‘(2r+1) (p2c)1/(2r+l) 10, otherwise ( 7c ) and therefore in this case . (2r + l)-l/czr+i) 1 1’(2r+1). (27) 0D Gg,,. ~,s,L,G* . *9t3 = i$l z<,i(ti) = OCL)+ co2 L + 03. The E-Entropy of Certain Processes with Zc,(Q = co In the following we consider mean-continuous processes 3) Part II of b) is a generalization of a result of Kazi [ 191. g such that Xt,(r) = co. In this case there does not exist any Gaussian processg for which J$(<) -C co, and therefore the IV. PROOF OF RESULTS previous lower bounds on H,(t) are meaningless. Proof of Theorem I : By the remark to Lemma 2.2 we may Let K&s,t), the covariance function of t and of &, be assume, without loss of generality, that 5 is a mean- continuous in [O,T] x [O,T] and denote by {pi} and continuous process. Let 5i, i = 1,2,* * . be the Karhunen- BINIA et al. : &-ENTROPY AND RATE-DISTORTION 521 Loeve expansion coefficients of t, and let the eigenvalues By the convexity of In x, for all positive numbers xl, x2, iii = Eli2, i = 1,2; * * be arranged in nondecreasing, x1’, and x2’ such that xl + x2 = xl’ + x2’ and xl 2 monotonic order, From Lemmas 2.1 and 2.2 x1’ 2 x2! 2 x2 In L + In I 2 In L + In L . Xl x2 Xl’ x2’ Therefore, we have to prove the lower bound in (13) only for the finite-dimensional random variable 4 = (gl; * *,e,), Therefore, the infimum in (35) over all vectors (Ebb, * * * ,E~~) where E 1((51,** *,tj,’ * *,td9(V19*f‘ ,?j,” ‘9VL)) f&(tt,. . . ,td 2 Hd&t,* . &.) + NC,,. . .,t~> 2 r< Vi = ai4i + 5i where the infimum is taken over all distortion vectors Vgi = aitgi + ii, i = l;**,m (40) satisfying (33) and where ai = 1 - e/ii and ci, i = 1,. * -,m are independent h(5,;. .,td Gaussian random variables with Eti2 = ai0 and are independent of 4 and 5,. =-sP&l9 * * * ,xL) In p&xl, * * * ,xJ dx,, . * * ,k. Since by (40) Then, we also have E i$l (5i - rlJ2 + E i=$+l ti2 = e2 f&(51,.. .A,) 2 inf (i$l k ln3) we have by the definition of H,(51,42; * *) f&(41,52>. * *) 5 ut,,t,,* * .),(rll,‘. ~,rl,,w,*~ *)> + h(t,;.. ,tL) - i jl In 2di. (35) = W5d2;. *Milt,*. .,%?Jh (41) 522 IEEE TRANSACTIONS ON INFORMATION THEORY, JULY 1974 Observe that by (40) the variables (tm+ r; - *) and (vi,* - a,~,,) li’emark: The result (51) is derived in [20] for mutual are conditionally independent, conditioned on (cl, - - -,&,). information only; however, the proof given there goes over Therefore, as in (11) directly to the case of entropy. From (50) and (51) we obtain (49). Therefore, from (47)-(49) and by (40) H,(t) I f&(5,) - y;p,,tt) + o(l), E + 0 (52) which together with the left side of (13) yields (14). k In 271eQ. (43) Proof of Lemma 3.1: Let {tl; * *,t,} be an arbitrary finite set in [O,T]. First, let us assume, for simplicity, that Since the s-entropy of 5, is given by [7] the density functions pr,(xl, * * * ,x,) of (tg(tl), * * *,<,(t,)), Pgh * * *J,) of (s(h),* * ~,s(h,)) and P&, - * -A,,) of (&tl), * * *,((t,)) exist. Then (44) and since by (40) $ In 27WU& (45) we obtain from (31) and (41)-(45) H,(5) 5 H,tt,) - z(s,,, . . . ,qsm)h * * *A,>. (46) The upper bound on the right side of (13) follows at once from (46) since the entropy of one measure with respect to Observe that the function In pc,[(xl, * - * ,x,)/(p,(x,, - - . ,x,)1 another is always nonnegative [14]. is a quadratic form of the variables xl, * - *,x,. Therefore, we Note that both YE&, . . .,g,,j(rjl, * * *,y,) and m depend on may replace the second term on the right side of (53) with E, and m -+ co as E + 0. Since it follows that for any 6 > 0 there exists an integer N such Therefore that 2 (&(tl),. . . ,.&(tm))(5(tl)~* . *,5(4nN q,(4) 5 ~(&,,. . ~,~,.&l~* * *&) + 6. (47) = Yi4m,~ . . ,,(t,,,ttttl>~ * * *&L>> Let ~~ be sufficiently small such that m(t) 2 N, for 0 < E I Eo. Since - %3w . . ,gcr,,,(Sg(tl)~ * * - 9tg(4n)). (54) %bl, . .,~g,,h~~~ ~JLrl> 2 qqsl,. . ..~gN)hl~. * -A%) Since t,; * *,t, and m were arbitrary, (16) is proved. Also, by (15) and (16), both &‘$(E) and %g(&J are finite, and thus for m 2 N [14, ch. 21, (46) yields for E I e. & and g are equivalent processes [14, p. 1341. f&15> 5 f&(5,) - q,,gt,. . . .,&rt 3 * * * JN). (48) Finally, in the case where the distributions Pcyct,), . . *atm,,. P are not absolutely Finally, we show that (c&l), . . ~,e;,(L)) and PWt), . . . d7(Gd) continuous, we define another set of variables as follows. Let 5i, i = l;** ,m be independent Gaussian random variables with the same variance a2, which are independent By (40), for every E I ~~ we have [14, eq. 2.4.81 of 5, t,, and a and let t&A. . *,&,,N, (?,k>,- - -,~,k,)), and (J(tl); * *J(Q) be defined by 2 (Q71,~. . .tlg.v) iw * *Jhd s Z(&,,. . .,&.J,(l,. . .,&q)(tl>e* *Ldt,* * *7&v) tcti) = tcti) + 5i = Jq<,l,. . . &N)(L - * *&)* (50) Egtti) = tgcti> + 5i Since by (40) the random variables (vi,- * *,Q and s”tti) = dt3 + Ci9 i = I,**-,m. (%71~.” ,Q) converge (as E + 0) in the mean-square sense to the variables (tl, * - *,&) and (&r, * s *,JBN), respectively. Then, obviously, the distributions The measures induced by (vi, ; * *,qN) and (Q, * * *,Q) (on the real-valued N-dimensional Bore1 space) converge 4m . . ~.4(f*))~ &7(tl), . . ..Es(fm)) weakly, as E goes to zero, to the measures of (cl, * * *,&J and and hitI), . ~9ir(kJ)are absolutely continuous, and P” ttg1, * * *,&J, respectively, and we have [20] converges weakly to P as 6 goes to zero. The result of Lemma 3.1 follows now by the same arguments as in the limqqg,,. f .,qgN)h* * ~Jliv) 2 qee,,. . .,S,N,G - *A)* (51) e-0 proof of (49). BINL4 et al. : MNTROPY AND RATE-DISTORTION 523 We turn now to the proof of Theorem 4 (the proofs of for every L. Let E be small enough that 0 < a2 < l/M2 Theorems 2 and 3 follow directly from the discussion in Cg 1 li. We choose L and 0 satisfying Section III). (~~12 = Le, aL+l < 8 4 aL. (66) Proof of Theorem 4: a) By a property of mutual informa- Since E’/L I I,, the a-entropy of (&, * * *&,) is given by tion [14, eq. 2.2.71 and the definition of s-entropy, f&(t) = fat1,e2,** *> 2 4(S1,***9L) (55) for every L. Using Theorem 1 and our assumption on the asymptotic behavior of &&, . . .,5sJt1, * * s&), we get From (66) and (67) we obtain m7 2 fcb&l, * * *, E2 - jl min (0,&), An+l < e I 1,. Proofof Examples: We start with the proof of Example 1. Let the Gaussian processg be defined as Since J&(&J = C;= 1 4 In (A#) (and since by Theorem 1 OstlT H,(c) I H,(&)), the proof of part a) is completed. so> = g(O) + Pm whereg(0) = C&(O),and let P, and PB be the measures(in b) The proof of part I of b) is similar to that of part a). function space)induced by < and g, respectively. Since the &) = o(L), then there exists a II. If X(& . . .&&l, * * * asymptotic behavior of H,(g) is given by (22) [lo], we have constant M > 1 such that for every L only to show that Xg(<) < co. In this case the Radon- Nikodym derivative dPr/dPg is given by [ 121, [ 131 . . &,)(51~ * * .,&) I g In M2. (64) 3b. indP -, ln d% +J)1 Tt- ’ (tMdt)t Therefore, by (55) and Theorem 1 dP-, dPg,o, P2 so f&(t) 2 fUtgl,. * *,tgL) - 4 In M2 (65) + $ f- t(t) dw(t) - $1’ c2(t) dt (74) 0 0 524 IEEE TRANSACTIONS ON INFORMATTON THEORY, JULY 1974 (almost surely PC), where b(t) is defined by density Scf) %tt> = EPJtt) I tw9 z I t]. R(D) = kIlnmax [y, l] df ’ Taking the expectation of both sides of (74) with respect to Pe yields [14, theorem 2.4.21 D = min [S(f)@] dJ I s q(t) = ~g(O)MW It is easy to show that the asymptotic behavior of the rate- distortion function of a Gaussian process with spectral +LE = &t)((t) dt - 2/oT/2Wdf]1 - (75) B” [S 0 density satisfying (78) is given by (27). Thus the proof is completed. Since Err(t) - t(t)]” 2 0, (75) yields REFERENCES = C2(t) dt < co (76) 111T. Berger, Rate Distortion Theory: A Mathematical Basis for Data sgI~g Ct>(o)t5(0))++ E 0 Compression. Englewood Cliffs, N.J. : Prentice-Hall, 1970. s C. E. Shannon and W. Weaver, The Mathematical Theory of ECommunication.l Urbana. Ill. : Univ. of Illinois Press, 1949. which completes the proof of Example 1. [31 C. E. Shannon, “Coding.theorems for a discrete source with a Example 2 is a particular case of the following generaliza- fidelity criterion,” IRE Nat. Conv. Rec., pt. 4, pp. 142-163, 1959. tion: Let 5 be a random process and let g be a Gaussian r41 A. M. Gerrish and P. M. Schultheiss, “Information rates of non- Gaussian processes,” IEEE Trans. Znform. Theory, vol. IT-lo, process such that Xg(Q < co. Let [ and g be defined by pp. 265-271, Oct. 1964. r = A& # = Ag, where A is a measurable transformation 151 Y. N. Lin’kov, “Evaluation of &-entropy of random variables for small E,” Probl. Peredach. Inform., vol. 1, p. 12, Apr.-June 1965. such that J is a Gaussian process with E fz g2(t) dt < co. A. N. Kolmogorov, “On the Shannon theory of information [61transmission in the case of continuous signals,” IRE Trans. Inform. Then Theory, vol. IT-2, pp. 102-108, Dec. 1956. He@ = Hh7), E + 0. 171 M. S. Pinsker, “Gaussian sources,” Probl. Peredach. Inform., (77)vol. 14, pp. 59-100, 1963. Since we have &(t) 2 %&) [14, ch. 21, (77) follows S. Ihara, “On &-entropy of equivalent Gaussian processes,” PNagoyaI Math. J., vol. 37, pp. 121-130,197O. directly from Theorem 1. PI J. Binia, M. Zakai, and J. Ziv, “Bounds on the E-entropy of Wiener Now, the proof of Example 2 follows from (77), (76), and and RC processes,” IEEE Trans. Inform. Theory (Corresp.), vol. IT-19, pp. 359-362, May 1973. the result (24) for the Gaussian process defined by m - 1 DOIJ. Binia, “On the &-entropy of certain Gaussian processes,” integrations of the Wiener process [lo]. IEEE Trans. Inform. Theory, vol. IT-20, pp. 190-196, Mar. 1974. 1111A. V. Skorokhod, Studies in the Theory of Random Processes. Finally, we turn to the proof of Example 3. Let the Reading, Mass. : Addison-Wesley, 1965, ch. 4. Gaussian process g be as in the proof of Example 1. Then WI T. Kailath, “The structure of Radon-Nikodym derivatives with respect to Wiener and related measures,” Ann. Math. Statist., z$~([) < co, and therefore Z&u) < co, where g is the vol. 42, no. 3, pp. 1054-1067. output of the same linear filter G(jo) operating on the r131 T. E. Duncan, “Evaluation of likelihood functions,” Inform. Contr., vol. 13, pp. 62-74, 1968. process g. Therefore, by Theorem 3, r141 M. S. Pinsker, Information and Information Stability of Random Variables and Processes. New York: Holden-Dav. 1964. 4,(D) = R,(D), D --f 0. r151 A. N. Kolmogorov, “Theory of transmission of’information,” Amer. Math. Sot. Transl., ser. 2, vol. 33, pp. 291-321. It has been shown in [21], under some general conditions, R. G. Gallager, Information Theory and Reliable Communication. 1161New York: Wiley, 1968. that the asymptotic behavior of the power spectral density iI71 M. S. Pinsker, “Communication sources,” Probl. Peredach. of 5 is given by Inform., vol. 14, pp. 5-20, 1963. P81 A. M. Yaglom, “On the equivalence and perpendicularity of two Gaussian probability measures in function space,” in Proc. Symp. S&02) z5 p20F2, 101 + co. Time Series Analysis. New York: Wiley, 1963, pp. 327-346. 1191K. Kazi, “On the c-entropy of diffusion processes,” Ann. Inst. Therefore, the power spectral density of q satisfies Statist. Math., vol. 21, pp. 347-356, 1969. I. M. Gelfand and A. M. Yaglom, “Calculation of the amount of 533) = @3-&2’+2) + o(w-w+29* WinformationI about a random function contained in another such (78)function, ” Amer. Math. Sot. Transl., ser. 2, vol. 13, pp. 199-246. Using the known result (e.g., [16, ch. 91) on the rate- M. Zakai and J. Ziv, “Lower and upper bounds on the optimal m1filtering l error of certain diffusion processes,” IEEE Trans. Inform. distortion function of a Gaussian process with spectral Theory, vol. IT-18, pp. 325-331, May 1972. lds + b(t), OstsT expressH,(c) in terms of H,(g) and Xg(~), where g is any s 0 given Gaussian process for which Xg(t) < co. We start (20) with a lemma which enablesus to replaceXC,(~) with yig(Q or, more generally, let 5 = {i(t); 0 1. t < T} be a random in the lower bound on H,(t). processwith E Sg c2(t) dt < co, and let 4 be defined by Lemma 3.1: Let g = {g(t), 0 < t I T} be a Gaussian process. If 5(t) = 5(O) + ’ 56) ds + b(t)> 0 I t I T (21) -qg(5) < co (15) s 0 then g, and g are equivalent Gaussian processes(i.e., the where w is a standard Brownian motion. We assume that measuresPeg and PB are mutually absolutely continuous) l(O) = 0, or c(O)is a random variable (independentof [ and 520 IEEE TRANSACTIONS ON INFORMATION THEORY, JULY 1974 w) with density function such that Et2(0) < co, and that {&} the eigenfunctions and the eigenvalues of KS. We future increments {w(t) - w(s)}, 0 I s < t I T, are assume that the eigenvalues are arranged in nonincreasing, independent of the a-field generated by the past of [ and w. monotonic order. The variables (t1,t2; * a) and (c&,&~; * a) Then, as shown in Section IV, the asymptotic behavior of are defined through {4i(t)} by H,(t) is given by T ci = tXtMi(t) dt, 2T2p2 1 s 0 f&(5> = 72 & -+ 0. (22)