508 11. Wiener Filtering
noisy random walk 8 observations y n 12 signal x 6 n prediction x n/n−1 4 Linear Prediction
2
0
−2
−4 0 50 100 150 200 250 300 time samples, n 12.1 Pure Prediction and Signal Modeling
In Sec. 1.17, we discussed the connection between linear prediction and signal modeling. Here, we rederive the same results by considering the linear prediction problem as a special case of the Wiener filtering problem, given by Eq. (11.4.6). Our aim is to cast the results in a form that will suggest a practical way to solve the prediction problem and hence also the modeling problem. Consider a stationary signal yn having a signal model = 2 −1 Syy(z) σ B(z)B(z ) (12.1.1)
as guaranteed by the spectral factorization theorem. Let Ryy(k) denote the autocorre- lation of yn : Ryy(k)= E[yn+kyn]
The linear prediction problem is to predict the current value yn on the basis of all the past values Yn−1 ={yi , −∞
of estimating yn from the related signal y1(n). The optimal estimation filter H(z) is given by Eq. (11.4.6), where we must identify xn and yn with yn and y1(n) of the present −1 notation. Using the filtering equation Y1(z)= z Y(z), we find that yn and y1(n) have the same spectral factor B(z)
= −1 = = 2 −1 Sy1y1 (z) (z )(z)Syy(z) Syy(z) σ B(z)B(z )
and also that = = 2 −1 Syy1 (z) Syy(z)z zσ B(z)B(z ) Inserting these into Eq. (11.4.6), we find for the optimal filter H(z) 2 −1 1 Syy (z) 1 zσ B(z)B(z ) H(z)= 1 = , or, 2 −1 2 −1 σ B(z) B(z ) + σ B(z) B(z ) +
1 H(z)= zB(z) (12.1.2) B(z) + 510 12. Linear Prediction 12.1. Pure Prediction and Signal Modeling 511
The causal instruction can be removed as follows: Noting that B(z) is a causal and The prediction error stable filter, we may expand it in the power series en/n−1 = yn − yˆn/n−1
−1 −2 −3 is identical to the whitening sequence driving the signal model (12.1.1) of y , indeed, B(z)= 1 + b1z + b2z + b3z +··· n n = − ˆ = − = − −1 The causal part of zB(z) is then E(z) Y(z) Y(z) Y(z) H(z)Y1(z) Y(z) H(z)z Y(z) −1 −2 −1 −2 −1 1 zB(z) + = [z + b1 + b2z + b3z +··· ]+= b1 + b2z + b3z +··· = 1 − z H(z) Y(z)= Y(z)= (z) B(z) −1 −2 −3 = z b1z + b2z + b3z +··· = z B(z)−1 Thus, in accordance with the results of Sec. 1.13 and Sec. 1.17 The prediction filter H(z) then becomes en/n−1 = yn − yˆn/n−1 = n (12.1.4) 1 1 H(z)= z B(z)−1 = z 1 − (12.1.3) B(z) B(z)
The input to this filter is y1(n) and the output is the prediction yˆn/n−1.
Example 12.1.1: Suppose that yn is generated by driving the all-pole filter
yn = 0.9yn−1 − 0.2yn−2 + n
by zero-mean white noise n. Find the best predictor yˆn/n−1. The signal model in this case −1 −2 is B(z)= 1/(1 − 0.9z + 0.2z ) and Eq. (12.1.3) gives Fig. 12.1.1 Linear Predictor. 1 z−1H(z)= 1 − = 1 − (1 − 0.9z−1 + 0.2z−2)= 0.9z−1 − 0.2z−2 B(z) An overall realization of the linear predictor is shown in Fig. 12.1.1. The indicated dividing line separates the linear predictor into the Wiener filtering part and the input The I/O equation for the prediction filter is obtained by part which provides the proper input signals to the Wiener part. The transfer function −1 −1 −2 from yn to en/n−1 is the whitening inverse filter Y(z)ˆ = H(z)Y1(z)= z H(z)Y(z)= 0.9z − 0.2z Y(z)
1 −1 and in the time domain A(z)= = 1 − z H(z) B(z) yˆn/n−1 = 0.9yn−1 − 0.2yn−2 which is stable and causal by the minimum-phase property of the spectral factorization Example 12.1.2: Suppose that (12.1.1). In the z-domain we have (1 − 0.25z−2)(1 − 0.25z2) Syy(z)= (1 − 0.8z−1)(1 − 0.8z) E(z)= (z)= A(z)Y(z)
ˆ Determine the best predictor yn/n−1. Here, the minimum phase factor is and in the time domain −2 1 − 0.25z ∞ B(z)= 1 − 0.8z−1 en/n−1 = n = amyn−m = yn + a1yn−1 + a2yn−2 +··· m=0 and therefore the prediction filter is
− − − The predicted estimate yˆ − = y − e − is 1 1 − 0.8z 1 0.8z 1 − 0.25z 2 n/n 1 n n/n 1 z−1H(z)= 1 − = 1 − = B(z) 1 − 0.25z−2 1 − .25z−2 yˆn/n−1 =− a1yn−1 + a2yn−2 +··· The I/O equation of this filter is conveniently given recursively by the difference equation These results are identical to Eqs. (1.17.2) and (1.17.3). The relationship noted above yˆn/n−1 = 0.25yˆn−2/n−3 + 0.8yn−1 − 0.25yn−2 between linear prediction and signal modeling can also be understood in terms of the 512 12. Linear Prediction 12.2. Autoregressive Models 513 gapped-function approach of Sec. 11.7. Rewriting Eq. (12.1.1) in terms of the prediction- These equations are known as the normal equations of linear prediction [915–928]. error filter A(z) we have They provide the solution to both the signal modeling and the linear prediction prob- 2 σ lems. They determine the model parameters {a ,a ,... ; σ2} of the signal y directly in S (z)= (12.1.5) 1 2 n yy −1 A(z)A(z ) terms of the experimentally accessible quantities Ryy(k). To render them computation- from which we obtain ally manageable, the infinite matrix equation (12.1.11) must be reduced to a finite one, σ2 A(z)S (z)= (12.1.6) and furthermore, the quantities Ryy(k) must be estimated from actual data samples of yy −1 A(z ) yn. We discuss these matters next. Since we have the filtering equation (z)= A(z)Y(z), it follows that = Sy(z) A(z)Syy(z) 12.2 Autoregressive Models and in the time domain { } ∞ In general, the number of prediction coefficients a1,a2,... is infinite since the pre- Rey(k)= E[nyn−k]= aiRyy(k − i) (12.1.7) dictor is based on the infinite past. However, there is an important exception to this; i=0 namely, when the process yn is autoregressive. In this case, the signal model B(z) is an all-pole filter of the type which is recognized as the gapped function (11.7.1). By construction, n is the orthog- onal complement of y with respect to the entire past subspace Y − ={y − ,k= n n 1 n k 1 1 } = B(z)= = (12.2.1) 1, 2,... , therefore, n will be orthogonal to each yn−k for k 1, 2,.... These are pre- −1 −2 −p A(z) 1 + a1z + a2z +···+apz cisely the gap conditions. Because the prediction is based on the entire past, the gapped function develops an infinite right-hand side gap. Thus, Eq. (12.1.7) implies which implies that the prediction filter is a polynomial ∞ = + −1 + −2 +···+ −p Rey(k)= E[nyn−k]= aiRyy(k − i)= 0 , for all k = 1, 2,... (12.1.8) A(z) 1 a1z a2z apz (12.2.2) i=0 The same result, of course, also follows from the z-domain equation (12.1.6). Both The signal generator for yn is the following difference equation, driven by the un- sides of the equation are stable, but since A(z) is minimum-phase, A(z−1) will be correlated sequence n : maximum phase, and therefore it will have a stable but anticausal inverse 1/A(z−1). y + a y − + a y − +···+a y − = (12.2.3) Thus, the right-hand side of Eq. (12.1.6) has no strictly causal part. Equating to zero all n 1 n 1 2 n 2 p n p n the coefficients of positive powers of z−1 results in Eq. (12.1.8). and the optimal prediction of yn is simply given by: The value of the gapped function at k = 0 is equal to σ2. Indeed, using the gap conditions (12.1.8) we find yˆ − =− a y − + a y − +···+a y − (12.2.4) n/n 1 1 n 1 2 n 2 p n p 2 = 2 = + + +··· σ E[n] E n(yn a1yn−1 a2yn−2 ) In this case, the best prediction of yn depends only on the past p samples {yn−1, = Ry(0)+a1Ry(1)+a2Ry(2)+···=Ry(0)= E[nyn] yn−2,...,yn−p}. The infinite set of equations (12.1.10) or (12.1.11) are still satisfied even though only the first p + 1 coefficients {1,a1,a2,...,ap} are nonzero. Using Eq. (12.1.7) with k = 0 and the symmetry property Ryy(i)= Ryy(−i),wefind The (p + 1)×(p + 1) portion of Eq. (12.1.11) is sufficient to determine the (p + 1) 2 = 2 = = + + +··· { 2} σ E[n] E[nyn] Ryy(0) a1Ryy(1) a2Ryy(2) (12.1.9) model parameters a1,a2,...,ap; σ : ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ Equations (12.1.8) and (12.1.9) may be combined into one: ··· 2 Ryy(0)Ryy(1)Ryy(2) Ryy(p) 1 σ ∞ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ R (1)R(0)R(1) ··· R (p − 1) ⎥ ⎢ a ⎥ ⎢ 0 ⎥ − = 2 ≥ ⎢ yy yy yy yy ⎥ ⎢ 1 ⎥ ⎢ ⎥ aiRyy(k i) σ δ(k) , for all k 0 (12.1.10) ⎢ ··· − ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ Ryy(2)Ryy(1)Ryy(0) Ryy(p 2) ⎥ ⎢ a2 ⎥ = ⎢ 0 ⎥ i=0 ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ (12.2.5) ⎢ ...... ⎥ ⎢ . ⎥ ⎢ . ⎥ which can be cast in the matrix form: ⎣ . . . . . ⎦ ⎣ . ⎦ ⎣ . ⎦ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ R (p) R (p − 1)R(p − 2) ··· R (0) a 0 R (0)R(1)R(2)R(3) ··· 1 σ2 yy yy yy yy p ⎢ yy yy yy yy ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ···⎥ ⎢ ⎥ ⎢ ⎥ ⎢ Ryy(1)Ryy(0)Ryy(1)Ryy(2) ⎥ ⎢ a1 ⎥ ⎢ 0 ⎥ Such equations may be solved efficiently by Levinson’s algorithm, which requires ⎢ ···⎥ ⎢ ⎥ ⎢ ⎥ ⎢ Ryy(2)Ryy(1)Ryy(0)Ryy(1) ⎥ ⎢ a2 ⎥ = ⎢ 0 ⎥ 2 3 ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ (12.1.11) O(p ) operations and O(p) storage locations to obtain the ais instead of O(p ) and ⎢ R (3)R(2)R(1)R(0) ···⎥ ⎢ a ⎥ ⎢ 0 ⎥ 2 ⎣ yy yy yy yy ⎦ ⎣ 3 ⎦ ⎣ ⎦ O(p ), respectively, that would be required if the inverse of the autocorrelation matrix ...... 514 12. Linear Prediction 12.3. Linear Prediction and the Levinson Recursion 515
{ 2} Ryy were to be computed. The finite set of model parameters a1,a2,...,ap; σ de- jω termines the signal model of yn completely. Setting z = e into Eq. (12.1.5)) we find a simple parametric representation of the power spectrum of the AR signal yn
2 2 σ σ Syy(ω)= = (12.2.6) 2 −jω −2jω −jωp2 A(ω) 1 + a1e + a2e +···+ape
In practice, the normal equations (12.2.5) provide a means of determining approx- { 2} Fig. 12.2.1 Yule-Walker Analysis Algorithm. imate estimates for the model parameters a1,a2,...,ap; σ . Typically, a block of length N of recorded data is available
spanned by the entire past {yn−i , 1 ≤ i<∞} is the same as the projection of yn onto y0,y1,y2,...,yN−1 the subspace spanned only by the past p samples; namely, {yn−i , 1 ≤ i ≤ p}. This is a consequence of the difference equation (12.2.3) generating y . There are many different methods of extracting reasonable estimates of the model n If the process y is not autoregressive, these two projections will be different. For parameters using this block of data. We mention: (1) the autocorrelation or Yule-Walker n any given p, the projection of y onto the past p samples will still provide the best linear method, (2) the covariance method, and (3) Burg’s method. There are also some varia- n prediction of y that can be made on the basis of these p samples. As p increases, more tions of these methods. The first method, the Yule-Walker method, is perhaps the most n and more past information is taken into account, and we expect the prediction of y obvious and straightforward one. In the normal equations (12.2.5), one simply replaces n to become better and better in the sense of yielding a smaller mean-square prediction the ensemble autocorrelations R (k) by the corresponding sample autocorrelations yy error. computed from the given block of data; that is, In this section, we consider the finite-past prediction problem and discuss its effi- N−1−k cient solution via the Levinson recursion [915–928]. For sufficiently large values of p,it 1 Rˆ (k)= y + y , for 0 ≤ k ≤ p (12.2.7) may be considered to be an adequate approximation to the full prediction problem and yy N n k n n=0 hence also to the modeling problem. = where only the first p + 1 lags are needed in Eq. (12.2.5). We must have, of course, Consider a stationary time series yn with autocorrelation function R(k) E[yn+kyn]. p ≤ N − 1. As discussed in Sec. 1.13, the resulting estimates of the model parameters For any given p, we seek the best linear predictor of the form { 2} aˆ1, aˆ2,...,aˆp; σˆ may be used now in a number of ways; examples include obtaining yˆn =− a1yn−1 + a2yn−2 +···+apyn−p (12.3.1) an estimate of the power spectrum of the sequence yn { } ˆ 2 ˆ 2 The p prediction coefficients a1,a2,...,ap are chosen to minimize the mean- ˆ σ σ Syy(ω)= = square prediction error ˆ 2 + −jω + −2jω +···+ −jωp2 A(ω) 1 aˆ1e aˆ2e aˆpe E = 2 E[en] (12.3.2) or, representing the block of N samples yn in terms of a few (i.e., p+1) filter parameters. where en is the prediction error 2 To synthesize the original samples one would generate white noise n of variance σˆ and send it through the generator filter whose coefficients are the estimated values; that en = yn − yˆn = yn + a1yn−1 + a2yn−2 +···+apyn−p (12.3.3) is, the filter 1 1 = B(z)ˆ = = Differentiating Eq. (12.3.2) with respect to each coefficient ai, i 1, 2,...,p, yields −1 −2 −p A(z)ˆ 1 + aˆ1z + aˆ2z +···+aˆpz the orthogonality equations The Yule-Walker analysis procedure, also referred to as the autocorrelation method = = of linear prediction [917], is summarized in Fig. 12.2.1. E[enyn−i] 0 , for i 1, 2,...,p (12.3.4)
which express the fact that the optimal predictor yˆn is the projection onto the span of { = } 12.3 Linear Prediction and the Levinson Recursion the past p samples; that is, yn−i ,i 1, 2,...,p . Inserting the expression (12.3.3) for en into Eq. (12.3.4), we obtain p linear equations for the coefficients In the last section, we saw that if the signal being predicted is autoregressive of order p p p, then the optimal linear predictor collapses to a pth order predictor. The infinite di- ajE[yn−jyn−i]= R(i − j)aj = 0 , for i = 1, 2,...,p (12.3.5) mensional Wiener filtering problem collapses to a finite dimensional one. A geometrical j=0 j=0 way to understand this property is to say that the projection of yn on the subspace 516 12. Linear Prediction 12.3. Linear Prediction and the Levinson Recursion 517
Using the conditions (12.3.4) we also find for the minimized value of Our objective is to construct the latter in terms of the former. We will use the ap- proach of Robinson and Treitel, based on gapped functions [925]. Suppose that the best p 2 2 predictor of order p, [1,ap1,ap2,...,app], has already been constructed. The corre- σ = E = E[e ]= E[enyn]= R(j)aj (12.3.6) e n sponding gapped function is j=0 ⎡⎛ ⎞ ⎤ p p Equations (12.3.5) and (12.3.6) can be combined into the (p+1)×(p+1) matrix equation ⎣⎝ ⎠ ⎦ gp(k)= E[ep(n)yn−k]= E apiyn−i yn−k = apiR(k − i) (12.3.8) ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ i=0 i=0 R(0)R(1)R(2) ··· R(p) 1 σ2 ⎢ ⎥ ⎢ ⎥ ⎢ e ⎥ ⎢ ··· − ⎥ ⎢ ⎥ ⎢ ⎥ It has a gap of length p as shown , that is, ⎢ R(1)R(0)R(1) R(p 1) ⎥ ⎢ a1 ⎥ ⎢ 0 ⎥ ⎢ ··· − ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ R(2)R(1)R(0) R(p 2) ⎥ ⎢ a2 ⎥ = ⎢ 0 ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ (12.3.7) ⎢ . . . . . ⎥ ⎢ . ⎥ ⎢ . ⎥ ⎣ ...... ⎦ ⎣ . ⎦ ⎣ . ⎦ R(p) R(p − 1) R(p − 2) ··· R(0) ap 0 gp(k)= 0 , for 1 ≤ k ≤ p which is identical to Eq. (12.2.5) for the autoregressive case. It is also the truncated version of the infinite matrix equation (12.1.11) for the full prediction problem. Instead of solving the normal equations (12.3.7) directly, we would like to embed this problem into a whole class of similar problems; namely, those of determining the These gap conditions are the same as the orthogonality equations (12.3.4). Using best linear predictors of orders p = 1, p = 2, p = 3,..., and so on. This approach will gp(k) we now construct a new gapped function gp+1(k) of gap p + 1. To do this, first lead to Levinson’s algorithm and to the so-called lattice realizations of linear prediction we reflect gp(k) about the origin; that is, gp(k)→ gp(−k). The reflected function has filters. Pictorially this class of problems is illustrated below a gap of length p but at negatives times. A delay of (p + 1) time units will realign this gap with the original gap. This follows because if 1 ≤ k ≤ p, then 1 ≤ p + 1 − k ≤ p. The reflected-delayed function will be gp(p + 1 − k). These operations are shown in the following figure
Since both gp(k) and gp(p + 1 − k) have exactly the same gap, it follows that so will any linear combination of them. Therefore, where [1,a11], [1,a21,a22], [1,a31,a32,a33],..., represent the best predictors of or- = − + − ders p = 1, 2, 3,...,respectively. It was necessary to attach an extra index indicating the gp+1(k) gp(k) γp+1gp(p 1 k) (12.3.9) order of the predictor. Levinson’s algorithm is an iterative procedure that constructs will have a gap of length at least p. We now select the parameter γp+1 so that gp+1(k) the next predictor from the previous one. In the process, all optimal predictors of lower acquires an extra gap point; its gap is now of length p + 1. The extra gap condition is orders are also computed. Consider the predictors of orders p and p + 1, below
gp+1(p + 1)= gp(p + 1)−γp+1gp(0)= 0 yn−p−1 yn−p ··· yn−2 yn−1 yn
app ··· ap2 ap1 1 which may be solved for gp(p + 1) ap+1,p+1 ap+1,p ··· ap+1,2 ap+1,1 1 γp+1 = gp(0) Evaluating Eq. (12.3.8) at k = p + 1, and using the fact that the value of the gapped ep(n)= yn + ap1yn−1 + ap2yn−2 +···+appyn−p function at k = 0 is the minimized value of the mean-squared error, that is, ep+1(n)= yn + ap+1,1yn−1 + ap+1,2yn−2 +···+ap+1,p+1yn−p−1 = 2 = = Ep E ep(n) E[ep(n)yn] gp(0) (12.3.10) 518 12. Linear Prediction 12.3. Linear Prediction and the Levinson Recursion 519 we finally find Taking inverse z-transforms, we find Δp γ + = (12.3.11) ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ p 1 E 1 1 0 p ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ap+1,1 ⎥ ⎢ ap1 ⎥ ⎢ app ⎥ where we set Δp = gp(p + 1) ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ a + ⎥ ⎢ a ⎥ ⎢ a − ⎥ ⎡ ⎤ ⎢ p 1,2 ⎥ ⎢ p2 ⎥ ⎢ p,p 1 ⎥ ⎢ ⎥ = ⎢ ⎥ − γ + ⎢ ⎥ (12.3.15) 1 ⎢ . ⎥ ⎢ . ⎥ p 1 ⎢ . ⎥ ⎢ ⎥ ⎢ . ⎥ ⎢ . ⎥ ⎢ . ⎥ ⎢ a ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ p ⎢ p1 ⎥ ⎣ a + ⎦ ⎣ a ⎦ ⎣ a ⎦ ⎢ ⎥ p 1,p pp p1 = + − = + − ⎢ ap2 ⎥ Δp apiR(p 1 i) R(p 1), R(p), R(p 1),...,R(1) ⎢ ⎥ (12.3.12) ap+1,p+1 0 1 = ⎢ . ⎥ i 0 ⎣ . ⎦ which can also be written as app
ap+1,i = api − γp+1ap,p+1−i , for 1 ≤ i ≤ p The coefficients γp+1 are called reflection, PARCOR,orSchur coefficients. This ter- minology will become clear later. Evaluating Eq. (12.3.9) at k = 0 and using gp(p + 1)= ap+1,p+1 =−γp+1 γp+1gp(0), we also find a recursion for the quantity Ep+1 = gp+1(0) R = −p −1 Introducing the reverse polynomial Ap (z) z Ap(z ), we may write Eq. (12.3.14) as Ep+1 = gp+1(0)= gp(0)−γp+1gp(p + 1)= gp(0)−γp+1 · γp+1gp(0), or, −1 R Ap+1(z)= Ap(z)−γp+1z A (z) (12.3.16) = − 2 p Ep+1 (1 γp+1)Ep (12.3.13) 2 Taking the reverse of both sides, we find This represents the minimum value of the mean-square prediction error E ep+1(n) for the predictor of order p + 1. Since both E and E + are nonnegative, it follows p p 1 −1 = −1 − p+1 − 2 Ap+1(z ) Ap(z ) γp+1z Ap(z) that the factor (1 γp+1) will be nonnegative and less than one. It represents the improvement in the prediction obtained by using a predictor of order p + 1 instead of a R = −(p+1) −1 = −(p+1) −1 − Ap+1(z) z Ap+1(z ) z Ap(z ) γp+1Ap(z) , or, predictor of order p. It also follows that γp+1 has magnitude less than one, |γp+1|≤1. R = −1 R − To find the new prediction coefficients, ap+1,i, we use the fact that the gapped func- Ap+1(z) z Ap (z) γp+1Ap(z) (12.3.17) tions are equal to the convolution of the corresponding prediction-error filters with the Equation (12.3.17) is, in a sense, redundant, but it will prove convenient to think autocorrelation function of yn : of the Levinson recursion as a recursion on both the forward, Ap(z), and the reverse, R × p Ap (z), polynomials. Equations (12.3.16) and Eq. (12.3.17) may be combined into a 2 2 gp(k)= apiR(k − i) ⇒ Gp(z)= Ap(z)Syy(z) matrix recursion equation, referred to as the forward Levinson recursion: i=0 p+1 −1 Ap+1(z) 1 −γp+1z Ap(z) = (forward recursion) (12.3.18) g + (k)= a + R(k − i) ⇒ G + (z)= A + (z)S (z) R − −1 R p 1 p 1,i p 1 p 1 yy Ap+1(z) γp+1 z Ap (z) i=0 The recursion is initialized at p = 0 by setting where Syy(z) represents the power spectral density of yn. Taking z-transforms of both sides of Eq. (12.3.9), we find = R = = = 2 A0(z) A0 (z) 1 and E0 R(0) E[yn] (12.3.19) −(p+1) −1 G + (z)= G (z)−γ + z G (z ), or, p 1 p p 1 p which corresponds to no prediction at all. We summarize the computational steps of −(p+1) −1 −1 the Levinson algorithm: Ap+1(z)Syy(z)= Ap(z)Syy(z)−γp+1z Ap(z )Syy(z ) where we used the fact that the reflected gapped function gp(−k) has z-transform 1. Initialize at p = 0 using Eq. (12.3.19). G (z−1), and therefore the delayed (by p + 1) as well as reflected gapped function p 2. At stage p, the filter Ap(z) and error Ep are available. −(p+1) −1 −1 gp(p + 1 − k) has z-transform z Gp(z ). Since Syy(z)= Syy(z ) because of the 3. Using Eq. (12.3.11), compute γp+1. symmetry relations R(k)= R(−k), it follows that Syy(z) is a common factor in all the 4. Using Eq. (12.3.14) or Eq. (12.3.18), determine the new polynomial A + (z). terms. Therefore, we obtain a relationship between the new best prediction-error filter p 1 5. Using Eq. (12.3.13), update the mean-square prediction error to E + . Ap+1(z)and the old one Ap(z) p 1 6. Go to stage p + 1. −(p+1) −1 Ap+1(z)= Ap(z)−γp+1z Ap(z ) (Levinson recursion) (12.3.14) 520 12. Linear Prediction 12.3. Linear Prediction and the Levinson Recursion 521
The iteration may be continued until the final desired order is reached. The depen- The function bkwlev is the inverse operation to frwlev. Its input is the vector of dence on the autocorrelation R(k) of the signal yn is entered through Eq. (12.3.11) and prediction-error filter coefficients [1,aM1,aM2,...,aMM] of the final order M, and its E0 = R(0). To reach stage p, only the p+1 autocorrelation lags {R(0), R(1),...,R(p)} output is the matrix L containing all the lower order prediction-error filters. The set of are required. At the pth stage, the iteration already has provided all the prediction fil- reflection coefficients are extracted from the first column of L. This function is based ters of lower order, and all the previous reflection coefficients. Thus, an alternative on the inverse of the matrix equation (12.3.18). Shifting p down by one unit, we write parametrization of the pth order predictor is in terms of the sequence of reflection Eq. (12.3.18) as −1 { } A (z) 1 −γ z A − (z) coefficients γ1,γ2,...,γp and the prediction error Ep p = p p 1 R − −1 R (12.3.22) Ap (z) γp z Ap−1(z) { } { } Ep,ap1,ap2,...,app Ep,γ1,γ2,...,γp Its inverse is A − (z) 1 1 γ A (z) One may pass from one parameter set to another. And both sets are equivalent p 1 = p p R 2 R (backward recursion) (12.3.23) to the autocorrelation set {R(0), R(1),...,R(p)}. The alternative parametrization of Ap−1(z) 1 − γp γpzz Ap (z) the autocorrelation function R(k) of a stationary random sequence in terms of the At each stage p, start with Ap(z) and extract γp =−app from the highest coefficient equivalent set of reflection coefficients is a general result [929,930], and has also been of Ap(z). Then, use Eq. (12.3.23) to obtain the polynomial Ap−1(z). The iteration begins extended to the multichannel case [931]. at the given order M and proceeds downwards to p = M − 1,M− 2,...,1, 0. If the process yn is autoregressive of order p, then as soon as the Levinson recursion The function rlev generates the set of autocorrelation lags {R(0), R(1), ..., R(M)} reaches this order, it will provide the autoregressive coefficients {a1,a2,...,ap} which from the knowledge of the final prediction-error filter AM(z) and final prediction error are also the best prediction coefficients for the full (i.e., based on the infinite past) EM. It calls bkwlev to generate all the lower order prediction-error filters, and then prediction problem. Further continuation of the Levinson recursion will produce nothing = it reconstructs the autocorrelation lags using the gapped function condition gp(p) p new—all prediction coefficients (and all reflection coefficients) of order higher than p a R(p − i)= 0, which may be solved for R(p) in terns of R(p − i), i = 1, 2,...,p, = i=0 pi will be zero, so that Aq(z) Ap(z) for all q>p. as follows: The four functions lev, frwlev, bkwlev, and rlev allow the passage from one pa- p rameter set to another. The function lev is an implementation of the computational R(p)=− apiR(p − i) , p = 1, 2,...,M (12.3.24) = sequence outlined above. The input to the function is the final desired order of the i 1 predictor, say M, and the vector of autocorrelation lags {R(0), R(1), ..., R(M)}. Its out- For example, the first few iterations of Eq. (12.3.24) will be: put is the lower-triangular matrix L whose rows are the reverse of all the lower order R(1) =− a11R(0) prediction-error filters. For example, for M = 4 the matrix L would be =− + ⎡ ⎤ R(2) a21R(1) a22R(0) 10000 ⎢ ⎥ =− + + ⎢ ⎥ R(3) a31R(2) a32R(1) a33R(0) ⎢ a11 1000⎥ ⎢ ⎥ L = ⎢ a22 a21 100⎥ (12.3.20) To get this recursion started, the value of R(0) may be obtained from Eq. (12.3.13). ⎢ ⎥ ⎣ a33 a32 a31 10⎦ Using Eq. (12.3.13) repeatedly, and E0 = R(0) we find a44 a43 a42 a41 1 = − 2 − 2 ··· − 2 EM (1 γ1)(1 γ2) (1 γM)R(0) (12.3.25) The first column of L contains the negatives of all the reflection coefficients. This fol- Since the reflection coefficients are already known (from the call to bkwlev) and EM lows from the Levinson recursion (12.3.14) which implies that the negative of the highest is given, this equation provides the right value for R(0). coefficient of the pth prediction-error filter is the pth reflection coefficient; namely, The function schur, based on the Schur algorithm and discussed in Sec. 12.10, is an alternative to lev. The logical interconnection of these functions is shown below. γp =−app ,p= 1, 2,...,M (12.3.21)
This choice for L is justified below and in Sec. 12.9. The function lev also produces the vector of mean-square prediction errors {E0,E1,...,EM} according to the recursion (12.3.13). The function frwlev is an implementation of the forward Levinson recursion (12.3.18) or (12.3.15). Its input is the set of reflection coefficients {γ1,γ2,...,γM} and its output is the set of all prediction-error filters up to order M, that is, Ap(z), p = 1, 2,...,M. Again, this output is arranged into the matrix L. 522 12. Linear Prediction 12.3. Linear Prediction and the Levinson Recursion 523
Example 12.3.1: Given the autocorrelation lags Sending these lags through the function lev we find the prediction-error filters:
= − −1 {R(0), R(1), R(2), R(3), R(4)}={128, −64, 80, −88, 89} A1(z) 1 0.8z
−1 −2 A2(z) = 1 − 0.889z + 0.111z Find all the prediction-error filters Ap(z) up to order four, the four reflection coefficients, = − −1 + −3 and the corresponding mean-square prediction errors. Below, we simply state the results A3(z) 1 0.875z 0.125z
obtained using the function lev: −1 −4 A4(z) = 1 − 0.857z + 0.143z −1 A1(z) = 1 + 0.5z Therefore, the fourth order prediction of yn given by Eq. (12.3.1) is −1 −2 A2(z) = 1 + 0.25z − 0.5z yˆn = 0.857yn−1 − 0.143yn−4 −2 −3 A3(z) = 1 − 0.375z + 0.5z
−1 −2 −3 −4 which gives yˆ5 = 0.857 − 0.143 = 0.714. A4(z) = 1 − 0.25z − 0.1875z + 0.5z − 0.5z The results of this section can also be derived from those of Sec. 1.8 by invoking The reflection coefficients are the negatives of the highest coefficients; thus, stationarity and making the proper identification of the various quantities. The data = = vector y and the subvectors y¯ and y˜ are identified with y yp+1(n), y¯ yp(n), and {γ1,γ2,γ3,γ4}={−0.5, 0.5, −0.5, 0.5} y˜ = yp(n − 1), where ⎡ ⎤ The vector of mean-squared prediction errors is given by ⎡ ⎤ ⎡ ⎤ yn ⎢ ⎥ yn yn−1 { }={ } ⎢ yn−1 ⎥ ⎢ ⎥ ⎢ ⎥ E0,E1,E2,E3,E4 128, 96, 72, 54, 40.5 ⎢ ⎥ ⎢ y − ⎥ ⎢ y − ⎥ ⎢ . ⎥ ⎢ n 1 ⎥ ⎢ n 2 ⎥ y (n)= ⎢ . ⎥ , y (n)= ⎢ ⎥ , y (n − 1)= ⎢ ⎥ (12.3.26) p+1 ⎢ . ⎥ p ⎢ . ⎥ p ⎢ . ⎥ Sending the above vector of reflection coefficients through the function frwlev would gen- ⎢ ⎥ ⎣ . ⎦ ⎣ . ⎦ ⎣ yn−p ⎦ erate the above set of polynomials. Sending the coefficients of A4(z) through bkwlev yn−p yn−p−1 yn−p−1 would generate the same set of polynomials. Sending the coefficients of A4(z) and E4 = 40.5 through rlev would recover the original autocorrelation lags R(k), k = 0, 1, 2, 3, 4. It follows from stationarity that the autocorrelation matrices of these vectors are independent of the absolute time instant n; therefore, we write The Yule-Walker method (see Sec. 12.2) can be used to extract the linear prediction = T = − − T = T parameters from a given set of signal samples. From a given length-N block of data Rp E[yp(n)yp(n) ] E[yp(n 1)yp(n 1) ], Rp+1 E[yp+1(n)yp+1(n) ]
y0,y1,y2,...,yN−1 It is easily verified that Rp is the order-p autocorrelation matrix defined in Eq. (12.3.7) and that the order-(p+1) autocorrelation matrix Rp+1 admits the block decompositions compute the sample autocorrelations {R(ˆ 0), R(ˆ 1),...,R(M)ˆ } using, for example, Eq. (12.2.7), ⎡ ⎤ ⎡ ⎤ R(0) R(1) ··· R(p + 1) R(p + 1) and send them through the Levinson recursion. The yw implements the Yule-Walker ⎢ ⎥ ⎢ ⎥ ⎢ R(1) ⎥ ⎢ . ⎥ method. The input to the function is the data vector of samples {y ,y ,...,y − } and ⎢ ⎥ ⎢ . ⎥ 0 1 N 1 R + = ⎢ ⎥ = ⎢ Rp . ⎥ p 1 ⎢ . ⎥ ⎢ ⎥ the desired final order M of the predictor. Its output is the set of all prediction-error fil- ⎣ . Rp ⎦ ⎣ R(1) ⎦ ters up to order M, arranged in the matrix L, and the vector of mean-squared prediction R(p + 1) R(p + 1) ··· R(1) R(0) errors up to order M, that is, {E0,E1,...,EM} It follows, in the notation of Sec. 1.8, that R¯ = R˜ = Rp and ρa = ρb = R(0), and Example 12.3.2: Given the signal samples ⎡ ⎤ ⎡ ⎤ R(1) R(p + 1) ⎢ ⎥ ⎢ ⎥ { }={ } = ⎢ . ⎥ = ⎢ . ⎥ y0,y1,y2,y3,y4 1, 1, 1, 1, 1 ra ⎣ . ⎦ , rb ⎣ . ⎦ R(p + 1) R(1) determine all the prediction-error filters up to order four. Using the fourth order predictor,
predict the sixth value in the above sequence, i.e., the value of y5. Thus, ra and rb are the reverse of each other. It follows that the backward predictors The sample autocorrelation of the above signal is easily computed using the methods of are the reverse of the forward ones. Therefore, Eq. (12.3.14) is the same as Eq. (1.8.40), Chapter 1. We find (ignoring the 1/N normalization factor): with the identifications
{R(ˆ 0), R(ˆ 1), R(ˆ 2), R(ˆ 3), {R(ˆ 4)}={5, 4, 3, 2, 1} a = ap+1 , b = bp+1 , ¯a = ˜a = ap , b¯ = b˜ = bp 524 12. Linear Prediction 12.4. Levinson’s Algorithm in Matrix Form 525
where for the unknowns {E3,a31,a32,a33}. The corresponding prediction-error filter is ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ 1 a + + −1 −2 −3 p 1,p 1 A3(z)= 1 + a31z + a32z + a33z ⎢ ⎥ ⎢ ⎥ 1 app ⎢ a + ⎥ ⎢ a + ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ p 1,1 ⎥ ⎢ p 1,p ⎥ ⎢ a ⎥ ⎢ . ⎥ ⎢ . ⎥ ⎢ . ⎥ ⎢ p1 ⎥ ⎢ . ⎥ a + = ⎢ . ⎥ , b + = ⎢ . ⎥ a = ⎢ ⎥ , b = ⎢ . ⎥ and the minimum value of the prediction error is E3. The solution is obtained in an p 1 ⎢ . ⎥ p 1 ⎢ . ⎥ p ⎢ . ⎥ p ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ . ⎦ ⎣ ap1 ⎦ iterative manner, by solving a family of similar matrix equations of lower dimensionality. ⎣ ap+1,p ⎦ ⎣ ap+1,1 ⎦ app 1 Starting at the upper left corner, ap+1,p+1 1
= R = R ¯ = ˜ = = = Symbolically, bp ap , bp+1 ap+1. We have Ea Eb Ep and γa γb γp+1. Thus, Eq. (12.3.15) may be written as a 0 a 0 = p − = p − ap+1 γp+1 γp+1 R (12.3.27) 0 bp 0 ap
The normal Eqs. (12.3.7) can be written for orders p and p + 1 in the compact form of Eqs. (1.8.38) and (1.8.12) the R matrices are successively enlarged until the desired dimension is reached (4×4in this example). Therefore, one successively solves the matrix equations 1 up R a = E u ,R+ a + = E + u + , u = , u + = (12.3.28) ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ p p p p p 1 p 1 p 1 p 1 p 0 p 1 0 R0 R1 R2 1 E2 R0 R1 1 E1 ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ [R ][1]= [E ], = , ⎣ R1 R0 R1 ⎦ ⎣ a21 ⎦ = ⎣ 0 ⎦ T 0 0 Recognizing that Eq. (12.3.12) can be written as Δp = a rb, it follows that the re- R1 R0 a11 0 p R2 R1 R0 a22 0 flection coefficient equation (12.3.11) is the same as (1.8.42). The rows of the matrix L defined by Eq. (12.3.20) are the reverse of the forward predictors; that is, the backward The solution of each problem is obtained in terms of the solution of the previous predictors of successive orders. Thus, L is the same as that defined in Eq. (1.8.13). The one. In this manner, the final solution is gradually built up. In the process, one also rows of the matrix U defined in Eq. (1.8.30) are the forward predictors, with the first finds all the lower order prediction-error filters. row being the predictor of highest order. For example, The iteration is based on two key properties of the autocorrelation matrix: first, ⎡ ⎤ the autocorrelation matrix of a given size contains as subblocks all the lower order 1 a a a a ⎢ 41 42 43 44 ⎥ autocorrelation matrices; and second, the autocorrelation matrix is reflection invariant. ⎢ ⎥ ⎢ 01a31 a32 a33 ⎥ ⎢ ⎥ That is, it remains invariant under interchange of its columns and then its rows. This U = ⎢ 00 1a21 a22 ⎥ ⎢ ⎥ interchanging operation is equivalent to the similarity transformation by the “reversing” ⎣ ⎦ 0001a11 matrix J having 1’s along its anti-diagonal, e.g., 00001 ⎡ ⎤ 0001 ⎢ ⎥ Comparing L with U, we note that one is obtained from the other by reversing its ⎢ 0010⎥ = J = ⎢ ⎥ (12.4.1) rows and then its columns; formally, U JLJ, where J is the corresponding reversing ⎣ 0100⎦ matrix. 1000
The invariance property means that the autocorrelation matrix commutes with the 12.4 Levinson’s Algorithm in Matrix Form matrix J − JRJ 1 = R (12.4.2) In this section, we illustrate the mechanics of the Levinson recursion—cast in matrix form—by explicitly carrying out a few of the recursions given in Eq. (12.3.15). The This property immediately implies that if the matrix equation is satisfied: objective of such recursions is to solve normal equations of the type ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ R R R R a b ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎢ 0 1 2 3 ⎥ ⎢ 0 ⎥ ⎢ 0 ⎥ ⎢ R R R R ⎥ ⎢ a ⎥ ⎢ b ⎥ R0 R1 R2 R3 1 E3 ⎢ 1 0 1 2 ⎥ ⎢ 1 ⎥ = ⎢ 1 ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ ⎣ ⎦ ⎣ ⎦ ⎢ R R R R ⎥ ⎢ a ⎥ ⎢ 0 ⎥ R2 R1 R0 R1 a2 b2 ⎢ 1 0 1 2 ⎥ ⎢ 31 ⎥ = ⎢ ⎥ ⎣ R2 R1 R0 R1 ⎦ ⎣ a32 ⎦ ⎣ 0 ⎦ R3 R2 R1 R0 a3 b3 R3 R2 R1 R0 a33 0 526 12. Linear Prediction 12.4. Levinson’s Algorithm in Matrix Form 527 then the following equation is also satisfied: Step 2 ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ R R R R a b ⎢ 0 1 2 3 ⎥ ⎢ 3 ⎥ ⎢ 3 ⎥ We wish to solve ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎢ R R R R ⎥ ⎢ a ⎥ ⎢ b ⎥ R R R 1 E ⎢ 1 0 1 2 ⎥ ⎢ 2 ⎥ = ⎢ 2 ⎥ 0 1 2 2 ⎣ ⎦ ⎣ ⎦ ⎣ ⎦ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ R2 R1 R0 R1 a1 b1 ⎣ R1 R0 R1 ⎦ ⎣ a21 ⎦ = ⎣ 0 ⎦ (12.4.4) R3 R2 R1 R0 a0 b0 R2 R1 R0 a22 0 The steps of the Levinson algorithm are explicitly as follows: Try an expression of the form: ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ 1 1 0 Step 0 ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ a21 ⎦ = ⎣ a11 ⎦ − γ2 ⎣ a11 ⎦ , with γ2 to be determined · = Solve R0 1 E0. This defines E0. Then enlarge to the next size by padding a zero, that a22 0 1 is, R0 R1 1 E0 Acting on both sides by the 3×3 autocorrelation matrix and using Step 1, we find = , this defines Δ0. Then, also R1 R0 0 Δ0 ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ E E Δ ⎢ 2 ⎥ ⎢ 1 ⎥ ⎢ 1 ⎥ R0 R1 0 Δ0 ⎣ 0 ⎦ = ⎣ 0 ⎦ − γ ⎣ 0 ⎦ , or, = , by reversal invariance 2 R1 R0 1 E0 0 Δ1 E1 These are the preliminaries to Step 1. E2 = E1 − γ2Δ1 , 0 = Δ1 − γ2E1 , or Δ 1 Step 1 = 1 = − 2 = γ2 ,E2 (1 γ2)E1 , where Δ1 R2,R1 E1 a11 We wish to solve R0 R1 1 E1 These define γ2 and E2. As a preliminary to Step 3, enlarge Eq. (12.4.4) to the next = (12.4.3) R1 R0 a11 0 size by padding a zero Try an expression of the form ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ R0 R1 R2 R3 1 E2 1 1 0 ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ = − ⎢ R1 R0 R1 R2 ⎥ ⎢ a21 ⎥ ⎢ 0 ⎥ γ1 ⎢ ⎥ ⎢ ⎥ = ⎢ ⎥ , this defines Δ2. Then, also a11 0 1 ⎣ R2 R1 R0 R1 ⎦ ⎣ a22 ⎦ ⎣ 0 ⎦ R3 R2 R1 R0 0 Δ2 R0 R1 Acting on both sides by and using the results of Step 0, we obtain ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ R1 R0 R R R R 0 Δ ⎢ 0 1 2 3 ⎥ ⎢ ⎥ ⎢ 2 ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ R R 1 R R 1 R R 0 R1 R0 R1 R2 a22 0 0 1 = 0 1 − 0 1 ⎢ ⎥ ⎢ ⎥ = ⎢ ⎥ , by reversal invariance γ1 , or, ⎣ R2 R1 R0 R1 ⎦ ⎣ a21 ⎦ ⎣ 0 ⎦ R1 R0 a11 R1 R0 0 R1 R0 1 R3 R2 R1 R0 1 E2 E1 E0 Δ0 = − γ1 , or, 0 Δ0 E0 Step 3
E1 = E0 − γ1Δ0 , 0 = Δ0 − γ1E0 , or We wish to solve ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ Δ0 = = − = − 2 = R0 R1 R2 R3 1 E3 γ1 ,E1 E0 γ1Δ0 (1 γ1)E0 , where Δ0 R1 ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ E0 ⎢ R R R R ⎥ ⎢ a ⎥ ⎢ 0 ⎥ ⎢ 1 0 1 2 ⎥ ⎢ 31 ⎥ = ⎢ ⎥ These define γ and E . As a preliminary to Step 2, enlarge Eq. (12.4.3) to the next (12.4.5) 1 1 ⎣ R2 R1 R0 R1 ⎦ ⎣ a32 ⎦ ⎣ 0 ⎦ size by padding a zero ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ R3 R2 R1 R0 a33 0 R R R 1 E ⎢ 0 1 2 ⎥ ⎢ ⎥ ⎢ 1 ⎥ Try an expression of the form: ⎣ R R R ⎦ ⎣ a ⎦ = ⎣ 0 ⎦ , this defines Δ . Then, also 1 0 1 11 1 ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ R2 R1 R0 0 Δ1 1 1 0 ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ a31 ⎢ a21 a22 R0 R1 R2 0 Δ1 ⎢ ⎥ = ⎥ − γ3 ⎢ ⎥ , with γ3 to be determined ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ a32 ⎦ ⎣ a22 ⎦ ⎣ a21 ⎦ ⎣ R1 R0 R1 ⎦ ⎣ a11 ⎦ = ⎣ 0 ⎦ , by reversal invariance a33 0 1 R2 R1 R0 1 E1 528 12. Linear Prediction 12.5. Autocorrelation Sequence Extensions 529
Acting on both sides by the 4×4 autocorrelation matrix and using Step 2, we obtain It follows from the Levinson recursion that all prediction-error filters of order greater
⎡ ⎤ ⎡ ⎤ ⎡ ⎤ than p will remain equal to the pth filter, Ap(z)= Ap+1(z)= Ap+2(z)=···. Therefore, E3 E2 Δ2 = ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ the corresponding whitening filter will be A(z) Ap(z), that is, an autoregressive model ⎢ 0 ⎥ ⎢ 0 ⎥ ⎢ 0 ⎥ ⎢ ⎥ = ⎢ ⎥ − γ ⎢ ⎥ , or, of order p. With the exception of the above autoregressive extension that leads to an ⎣ 0 ⎦ ⎣ 0 ⎦ 3 ⎣ 0 ⎦ all-pole signal model, the extendibility conditions |γp+i| < 1, i ≥ 1, do not necessarily 0 Δ2 E2 guarantee that the resulting signal model will be a rational (pole-zero) model.
E3 = E2 − γ3Δ2 , 0 = Δ2 − γ3E2 , or Example 12.5.1: Consider the three numbers {R(0), R(1), R(2)}={8, 4, −1}. The Levinson ⎡ ⎤ recursion gives {γ1,γ2}={0.5, −0.5} and {E1,E2}={6, 4.5}. Thus, the above numbers 1 Δ2 2 ⎢ ⎥ qualify to be autocorrelation lags. The corresponding prediction-error filters are γ = ,E= (1 − γ )E , where Δ = R ,R ,R ⎣ a21 ⎦ 3 E 3 3 2 2 3 2 1 ⎡ ⎤ ⎡ ⎤ 2 a 1 1 22 1 1 ⎢ ⎥ ⎢ ⎥ a1 = = , a2 = ⎣ a21 ⎦ = ⎣ −0.75 ⎦ Clearly, the procedure can be continued to higher and higher orders, as required a11 −0.5 a22 0.5 in each problem. Note that at each step, we used the order-updating Eqs. (1.8.40) in conjunction with Eq. (1.8.47). The next lag in this sequence can be chosen according to Eq. (12.5.1) R(3)= γ3E2 − a21R(2)+a22R(1) = 4.5γ3 − 2.75 12.5 Autocorrelation Sequence Extensions where γ3 is any number in the interval −1 <γ3 < 1 . The resulting possible values of In this section, we discuss the problem of extending an autocorrelation function and R(3) are plotted below versus γ3 . In particular, the autoregressive extension corresponds = =− the related issues of singular autocorrelation matrices. The equivalence between an to γ3 0, which gives R(3) 2.75. autocorrelation function and the set of reflection coefficients provides a convenient and systematic way to (a) test whether a given finite set of numbers are the autocorrelation lags of a stationary signal and (b) extend a given finite set of autocorrelation lags to arbitrary lengths while preserving the autocorrelation property. For a finite set of numbers {R(0), R(1),...,R(p)} to be the lags of an autocorre- lation function, it is necessary and sufficient that all reflection coefficients, extracted from this set via the Levinson recursion, have magnitude less than one; that is, |γi| < 1, for i = 1, 2,...,p, and also that R(0)> 0. These conditions are equivalent to the pos- itive definiteness of the autocorrelation matrix Rp. The proof follows from the fact The end-points, γp+1 =±1, of the allowed interval (−1, 1) correspond to the two that the positivity of Rp is equivalent to the conditions on the prediction errors Ei > 0, possible extreme values of R(p + 1): for i = 1, 2,...,p. In turn, these conditions are equivalent to E = R(0)> 0 and and, 0 + =± − + − +···+ through Eq. (12.3.13), to the reflection coefficients having magnitude less than one. R(p 1) Ep ap1R(p) ap2R(p 1) appR(1) { } The problem of extending a finite set R(0), R(1),...,R(p) of autocorrelation lags = − 2 = In this case, the corresponding prediction error vanishes Ep+1 (1 γp+1)Ep 0. is to find a number R(p+1) such that the extended set {R(0), R(1),...,R(p), R(p+1)} This makes the resulting order-(p + 1) autocorrelation matrix Rp+1 singular. The pre- is still an autocorrelation sequence. This can be done by parametrizing R(p + 1) in diction filter becomes either the symmetric (if γp+1 =−1) or antisymmetric (if γp+1 = 1) terms of the next reflection coefficient γ + . Solving Eq. (12.3.12) for R(p + 1) and p 1 combination using Eq. (12.3.11), we obtain a 0 = p + = + −1 R + = − + − +···+ ap+1 R ,Ap+1(z) Ap(z) z Ap (z) , or, R(p 1) γp+1Ep ap1R(p) ap2R(p 1) appR(1) (12.5.1) 0 ap − a 0 Any number γp+1 in the range 1 <γp+1 < 1 will give rise to an acceptable value for = p − = − −1 R ap+1 R ,Ap+1(z) Ap(z) z Ap (z) R(p+1) . The choice γp+1 = 0 is special and corresponds to the so-called autoregressive 0 ap or maximum entropy extension of the autocorrelation function (see Problem 12.16). If this choice is repeated to infinity, we will obtain the set of reflection coefficients In either case, it can be shown that the zeros of the polynomial Ap+1(z) lie on the unit circle, and that the prediction filter ap+1 becomes an eigenvector of Rp+1 with + + = {γ1,γ2,...,γp, 0, 0,...} zero eigenvalue; namely, Rp 1ap 1 0. This follows from the normal Eqs. (12.3.28) Rp+1ap+1 = Ep+1up+1 and Ep+1 = 0. 530 12. Linear Prediction 12.5. Autocorrelation Sequence Extensions 531
± Example 12.5.2: Consider the extended autocorrelation sequence of Example 12.5.1 defined unit circle coinciding with the sinusoids present in R, namely, z = e jω1 . The other two
by the singular choice γ3 =−1. Then, R(3)=−4.5 − 2.75 =−7.25. The corresponding eigenvectors of R are order-3 prediction-error filter is computed using the order-2 predictor and the Levinson ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ 1 1 0 0 recursion ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ √ ⎥ = ⎣ ⎦ = ⎣ ⎦ = ⎣ ⎦ = ⎣ ⎦ 1 1 0 1 c cos ω1 0.5 , d sin ω1 √3/2 ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ a ⎥ ⎢ −0.75 ⎥ ⎢ 0.5 ⎥ ⎢ −0.25 ⎥ cos 2ω1 −0.5 sin 2ω1 3/2 = ⎢ 31 ⎥ = ⎢ ⎥ − ⎢ ⎥ = ⎢ ⎥ a3 ⎣ ⎦ ⎣ ⎦ γ3 ⎣ − ⎦ ⎣ − ⎦ a32 0.5 0.75 0.25 √ a33 0 1 1 both belonging to eigenvalue λ = 3. Their norm is c = d = 3/2. The three eigen-
vectors a2, c, d are mutually orthogonal. It is easily verified that the matrix R may be rep- − −1 − It is symmetric about its middle. Its zeros, computed as the solutions of (1 0.25z resented in the form R = 2ccT + 2ddT, which, after normalizing c and d to unit norm, is −2 + −3 = + −1 − −1 + −2 = 0.25z z ) (1 z )(1 1.25z z ) 0 are recognized as the eigendecomposition of R, We can also express R in terms of its complex √ = † + ∗ T 5 ± j 39 sinusoidal components in the form R ss s s , where z =−1 ,z= ⎡ ⎤ 8 1 ⎢ ⎥ † ∗ − − s = c + jd = ⎣ ejω1 ⎦ , s = s T = 1,e jω1 ,e 2jω1 and lie on the unit circle. Finally, we verify that a3 is an eigenvector of R3 with zero e2jω1 eigenvalue: ⎡ ⎤ ⎡ ⎤ 84−1 −7.25 1 Example 12.5.4: Similarly, one can verify that the four autocorrelation lags {8, 4, −1, −7.25} ⎢ ⎥ ⎢ ⎥ ⎢ 484−1 ⎥ ⎢ −0.25 ⎥ of the singular matrix of Example 12.5.2 can be represented in the sinusoidal form R a = ⎢ ⎥ ⎢ ⎥ = 0 3 3 ⎣ −148 4⎦ ⎣ −0.25 ⎦ = jω1k + jω2k + jω3k = −7.25 −14 8 1 R(k) P1e P2e P3e , for k 0, 1, 2, 3
= = = Singular autocorrelation matrices, and the associated symmetric or antisymmetric where P1 8/13, P2 P3 96/13, and ωi correspond to the zeros of the prediction prediction filters with zeros on the unit circle, find application in the method of line filter a3, namely, √ √ spectrum pairs (LSP) of speech analysis [937]. They are also intimately related to the 5 + j 39 5 − j 39 jω1 jω2 jω3 e =−1 ,e = ,e = , so that, ω3 =−ω2 eigenvector methods of spectrum estimation, such as Pisarenko’s method of harmonic 8 8 retrieval, discussed in Sec. 14.2. This connection arises from the property that singular autocorrelation matrices (with nonsingular principal minors) admit a representation as The matrix itself has the sinusoidal representation a sum of sinusoidal components [938], the frequencies of which are given precisely by ⎡ ⎤ 1 the zeros, on the unit circle, of the corresponding prediction filter. This sinusoidal ⎢ jω ⎥ † † † ⎢ e i ⎥ R = P s s + P s s + P s s , where s = ⎢ ⎥ representation is equivalent to the eigen-decomposition of the matrix. The prediction 1 1 1 2 2 2 3 3 3 i ⎣ e2jωi ⎦ filter can, alternatively, be computed as the eigenvector belonging to zero eigenvalue. e3jωi The proof of these results can be derived as a limiting case; namely, the noise-free case, of the more general eigenvector methods that deal with sinusoids in noise. A direct Had we chosen the value γ3 = 1 in Example 12.5.2, we would have found the extended lag T proof is suggested in Problem 14.10. R(3)= 1.75 and the antisymmetric order-3 prediction-error filter a3 = [1, −1.25, 1.25, −1] , ⎡ ⎤ whose zeros are on the unit circle: 21−1 √ √ ⎢ ⎥ 1 + j 63 1 − j 63 Example 12.5.3: Consider the autocorrelation matrix R = ⎣ 12 1⎦. It is easily verified ejω1 = 1 ,ejω2 = ,ejω3 = −11 2 8 8 that the corresponding autocorrelation lags R(k) admit the sinusoidal representation with R(k) admitting the sinusoidal representation
jω1k −jω1k R(k)= 2 cos(ω1k)= e + e , for k = 0, 1, 2 R(k)= P1 + 2P2 cos(ω2k)= [8, 4, −1, 1.75], for k = 0, 1, 2, 3
where ω1 = π/3. Sending these lags through the Levinson recursion, we find {γ1,γ2}= where P1 = 24/7 and P2 = 16/7. {0.5, −1} and {E1,E2}={1.5, 0}. Thus, R singular. Its eigenvalues are {0, 3, 3}. The T T corresponding prediction filters are a1 = [1, −0.5] and a2 = [1, −1, 1] . It is easily
verified that a2 is an eigenvector of R with zero eigenvalue, i.e., Ra2 = 0. The corresponding −1 −2 eigenfilter A2(z)= 1 − z + z , is symmetric about its middle and has zeros on the 532 12. Linear Prediction 12.6. Split Levinson Algorithm 533
12.6 Split Levinson Algorithm or, −1 −1 Fp+2(z)= (1 + z )Fp+1(z)−αp+1z Fp(z) (12.6.4) The main computational burden of Levinson’s algorithm is 2p multiplications per stage, where αp+1 = (1 + γp+1)(1 − γp). In block diagram form arising from the p multiplications in Eq. (12.3.15) and in the computation of the inner product (12.3.12). Thus, for M stages, the algorithm requires