Appendix A Performance Comparison of Various Discrete Transforms This will not include the fast algorithms, separability, recursivity, orthogonality and fast algorithms (complexity of implementation). These topics are described else- where in detail (see Chapter 3). The focus is on their various properties. We will consider the random vector x is generated by I-order Markov process. When T x ¼ ðÞx0; x1; ...; xNÀ1 , ðx0; x1; ...; xNÀ1 are the N random variables) correlation matrix ½Rxx is generated by the I-order Markov process, hi jjÀkj ½Rxx ¼ r ; r ¼ adjacent correlation coefficient |ffl{zffl} |fflfflffl{zfflfflffl} ðÞNÂN ðÞNÂN j; k ¼ 0; 1; ...; N À 1 (A.1) The covariance matrix ½S in the data domain is mapped into the transform domain ½S~ as is Âà à S~ ¼ ½DOT ½S ½DOT T (A.2) |{z} |fflffl{zfflffl} |{z} |fflfflfflfflfflfflffl{zfflfflfflfflfflfflffl} ðÞ ðÞNÂN ðÞNÂN N N ðÞNÂN DOT stands for discrete orthogonal transform. Superscripts T and * denote trans- pose and complex conjugateÂà respectively. When DOT is KLT, S~ is a diagonal matrix as all the transform coefficients in the KLT domain are uncorrelated. For all the other DOTs, residual correlation (correlation left undone in the DOT domain) is defined as [G6, G10] ! ! XNÀ1 XNÀ1 XNÀ1 XNÀ1 1 2 ~ 2 1 2 ~ 2 r ¼ kkS À Snn ¼ jjSmn À Snn (A.3) N n¼0 N m¼0 n¼0 n¼0 XNÀ1 XNÀ1 2 2 1 2 where jjSjj is the HilbertÀSchmidt norm defined as kkS ¼ jjSmn . N m¼0 n¼0 Note that N is the size of discrete signal. Smn is the element of ½S in row m and column n (m, n ¼ 0, 1, ..., NÀ1). 317 318 Appendix A Performance Comparison of Various Discrete Transforms For a 2D-random signal such as an image assuming that row and column statistics are independent of each other, the variances of the ðÞN  N samples can be easily obtained. This concept is extended for computing the variances of the ðÞN  N transform coefficients. A.1 Transform Coding Gain ½S ¼ Correlation or covariance matrix in data domain (see Section 5.6) ðÞNÂN Âà à à À S~ ¼ ½A ½S ð½A TÞ (Note that ð½A T Þ ¼ ½A 1 for unitary transforms) ðÞNÂN ðÞNÂN ðÞNÂN ðÞNÂN ¼ Correlation or covariance matrix in transform domain The transform coding gain, GTC is defined as N -1 1 σ~ 2 å kk N k=0 Arithmetic Mean GTC = = æ N -1 ö1/ N Geometric Mean ç σ~ 2 ÷ çÕ kk ÷ è k=0 ø s~2 ð¼; ; ...; À Þ where kk is the variance of the kth transform coefficient k 0 1 N 1 .As the sum of the variances in any orthogonal transform domain is invariant (total energy is preserved), GTC can be maximized by minimizing the geometric mean [B23]. The lower bound on the gain is 1 (as seen in Fig. A.1), which is attained only if all the variances are equal. 16-point regular DCT 9 Proposed integer 8 DCTs for H.264 and 7 AVS in [LA14] TC G 6 Ma’s integer DCT 5 Wien-Kuo’s integer DCT 4 Coding gain 3 2 1 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 Correlation coefficient ρ Fig. A.1 Comparing coding gain of orthogonal transforms [LA14] # 2009 IEEE A.4 Rate versus Distortion (Rate-Distortion) [B6] 319 A.2 Variance Distribution in the Transform Domain It is desirable to have few transform coefficients with large variances (this implies the remaining coefficients will have small variances, as the sum of the variances is invariant). This variance distribution can be described graphically or in a tabular form for N ¼ 8, 16, 32, ...and r ¼ 0.9, 0.95, ..., etc. The compaction of the energy in few transform coefficients can be represented by the normalized basis restriction error [B6] defined as NPÀ1 s~2 kk ¼ J ðÞ¼r k m ; m ¼ 0; 1; ...; N À 1Eq: ð5.179) in½ B6 ð5:72Þ m NPÀ1 s~2 kk k¼0 s~2 where kk have been arranged in decreasing order. See Table 5.2 and Fig. 5.33 about variance distribution of transform coefficients (See also Table 5.2 and Fig. 5.18 in [B6]). A.3 Normalized MSE (see Figs. 5.35À5.39 and see also Figs. 5.21À5.23 in [B6]) PP 2 vk; l k; l2 stopband J ¼ s PPNÀ1 2 vk; l k; l¼0 Js ¼ Energy in stopband/total energy, vk; l ðk; l ¼ 0; 1; ...; N À 1Þ are the trans- form coefficients of an ðÞN  N image. A.4 Rate versus Distortion (Rate-Distortion) [B6] The rate distortion function RD is the minimum average rate (bits/sample) for coding a signal at a specified distortion D (mean square error) [B6]. Let x0; x1; ...; xNÀ1 be Gaussian random variables encoded independently and ^ x^0; x^1; ...; x^NÀ1 be their reproduced values. Xk and Xk k ¼ 0, 1, ..., N À 1 are the corresponding transform coefficients. Then the average mean square distortion is 320 Appendix A Performance Comparison of Various Discrete Transforms XNÀ1 hiXNÀ1 hiÀÁ 1 2 1 ^ 2 D ¼ ExðÞn À x^n ¼ EXk À Xk N n¼0 N k¼0 For a fixed average distortion D, the rate distortion function RD is XNÀ1 1 1 s~2 R ðyÞ ¼ max ; kk Eq: ð2:118Þ in½ B6 D 0 log2 y N k¼0 2 where threshold y is determined by solving XNÀ1 ÀÁ ðyÞ¼1 y; s~2 ; fs~2 gy fs~2 g : ð : Þ ½ D min kk min kk max kk Eq 2 119 in B6 kk kk N k¼0 Select a value for Û to get a point in the plot of versus . Develop RD versus D for various discrete transforms based on I-order Markov process given N and r adjacent correlation coefficient. Plot RD vs. D, for N ¼ 8, 16, 32, ...and r ¼ 0.9, 0.95, ... For I-order Markov process (Eq. 2.68 in [B6]): ½S ¼ s2 ¼ rjjÀkj ; ¼ ; ; ...; À jk jk j k 0 1 N 1 maximum achievable coding gain is ðÞ= ½S ÀÁÀðÞÀ = ðÞ¼r 1 N tr ¼ À r2 1 1 N GN = 1 ðÞdet½S 1 N where tr ¼ trace of the matrix, det ¼ determinant of the matrix (see Appendix C in [B9]). A.5 Residual Correlation [G1] While the KLT completely decorrelates a random vector [B6], other discrete transforms fall short of this. An indication of the extent of decorrelation can be gauged by the correlation left undone by the discrete transform. This can be measured by the absolute sum of the cross covariance in the transform domain i.e., A.5 Residual Correlation [G1] 321 Data domain ⇔ Transform domain ∼ [Σ] [Σ] Set off diagonal elements to zero ∼ [Σ]ˆ [Σkk] Âà Fig. A.2 The relation between ½S and S~ XNÀ1 XNÀ1 js~2 j ij (A.4) i¼0 j¼0 i6¼j for N ¼ 8, 16, 32, ...as a function of r (Fig. A.2). Âà à Given S~ ¼ ½A ½S ½A T (A.5) ðÞNÂN ðÞNÂN ðÞNÂN ðÞNÂN Âà à Âà ~ T ~ obtain S ¼ ½A Skk ½A (A.6) ðÞNÂN ðÞNÂN ðÞNÂN ðÞNÂN Âà S~ ÂÃwhere kk is a diagonal matrix whose diagonal elements are the same as those of S~ , i.e., Âà S~ ¼ s~2 ; s~2 ; ...; s~2 kk diag 00 11 ðNÀ1ÞðÞNÀ1 It should be recognized that the conjugate appears in (A.5) that is derived in (5.42a), whereas the 2-D discrete transform of ½S is ½A ½S ½A T defined in (5.6a) and has no conjugate. Thus (A.5) can be regarded as a separable two-dimensional unitary transform of ½S for purposes of computation. Plot residual correlation versus r for DCT, DFT, KLT and ST [B23]. Fractional correlation (correlation left undone by a transform – for KLT this is zero, as KLT diagonalizes a covariance or correlation matrix) is defined as kk½ÀS ½S 2 (A.7) kk½ÀS ½l 2 P P 2 ½ ðÞ kk½2 ¼ NÀ1 NÀ1 ½ where IN is an N N unit matrix and A j¼0 k¼0 A jk . Note that theÂà measuresÂà (A.3), (A.4) and (A.7) are zeros respectively for the KLT ~ ~ as S ¼ Skk . 322 Appendix A Performance Comparison of Various Discrete Transforms x + n X N xˆ + Inverse Corrupted Transform Filtered transform signal signal [G] Filter matrix n Additive noise Fig. A.3 Scalar Wiener filtering abc Identity Hadamard defUnitary DFT DST of type 1 DCT of type 2 Karhunen−Loéve Fig. A.4 Magnitude displays of Wiener filter matrices ½G for a vector length of 16 elements ðÞN ¼ 16 . Dark pixels represent zero values, light pixels represent one values, and gray pixels represent values in between. Signal-to-noise ratio is 0.3 and r ¼ 0:9. Dynamic ranges of the Wiener filter magnitudes in the figure are compressed via the logarithmic transformation defined in (5.26) A.6 Scalar Wiener Filtering Filter matrix ½G is optimized for a specific transform, such that the noise can be filtered (Fig. A.3)[G5]. Evaluate MSE ¼ Eðjjx À ^xjj2Þ for the discrete transforms (appearing in Fig. A.4, plus Haar and slant transforms defined in [B6]) for N ¼ 4, 8, 16, 32, and r ¼ 0.9 and 0.95. A.7 Geometrical Zonal Sampling (GZS) 323 Plot magnitude displays of various discrete transforms referring to Fig. A.4. Comparing the filter planes, the filter characteristic changes drastically for differ- ent unitary transforms. For the KLT the filtering operation is a scalar multiplica- tion, while for the identity transform most elements of the filter matrix are of relatively large magnitude.