0000 –– FIRFIR FilteringFiltering ResultsResults ReviewReview && PracticalPractical ApplicationsApplications

• [p. 2] Mean square signal estimation principles • [p. 3] Orthogonality principle • [p. 6] FIR Wiener filtering concepts • [p.10] FIR Wiener filter equations • [p. 12] Wiener Predictor • [p. 16] Examples • [p. 24] Link between information signal and predictor behaviors • [p. 27] Examples • [p. 37] FIR Weiner filter and error surfaces • [p. 43] Application: Channel equalization • [p. 52] Application: Noise cancellation • [p. 56] Application: Noise cancellation with information leakage • [p. 59] Application: Spatial filtering • [p. 66] Appendices • [p. 70] References

08/29/12 EC4440.MPF -Section 00 1 ™ Mean Square Signal Estimation Distorted received signal s x Optimal processor d Transmitted signal best estimate of transmitted signal s (as a function of received signal x) Possible procedure: • conditional mean!, usually nonlinear Mean square estimation, i.e., in x [exception when x and s are jointly  2 normal Gauss Markov theorem] minimize ξ =−E{}sdx() • Complicated to solve, leads to dx ( )= Esx[ | ] •Restriction to Linear Mean Square (LMS), estimator of s is forced to be a linear function of (proof given in Appendix A) measurements x: →=dhx H •Solution via Wiener Hopf equations using orthogonality principle

08/29/12 EC4440.MPF -Section 00 2 ™ Orthogonality Principle Use LMS Criterion: estimate s by dhx = H where weights {hi} minimize MS error: 2 2  σ e =−E{sdx()}

Theorem: Let error e= s - d 2 hh minimizes the MSE quantity σ e if is chosen such that ** Eex{}ii==∀= Exe {} 0 , i 1," , N

i.e., the error ex is orthogonal to the observations i ,i= 1..., N used to compute the filter output. σ 2*= E se Corollary: minimum MSE obtained: e min { } where e is the minimum error obtained for the optimum filter vector. (Proof given in Appendix B)

08/29/12 EC4440.MPF -Section 00 3 P = 2  s dn()=+− hxn01 () hxn ( 1)

x(n)

x(n-1)

08/29/12 EC4440.MPF -Section 00 4 + x(n) dn () signal s(n) + filter noise w(n)

Typical Wiener Filtering Problems Form of Desired d(n) Problem Observations Signal x Filtering of x(n) = s(n)+w(n) d(n) = s(n) n n0 d(n) signal in noise n Prediction of x(n) = s(n)+w(n) d(n) = s(n+p); n0 d(n) n+p p signal in noise p > 0 n n n−q 0 q d(n) Smoothing of x(n) = s(n)+w(n) d(n) = s(n−q); signal in noise q > 0 n0 n n+1 1 Linear x(n) = s(n−1) d(n) = s(n) prediction

08/29/12 EC4440.MPF -Section 00 5 ™ FIR Wiener Filtering Concepts • Filter criterion used: minimization of mean square error between d(n) and dnˆ ( ) . d(n) What are we doing here? •  + x(n) dna f s(n) + filter - signal + noise w(n) e(n) We want to design a filter (in the generic sense can be: filter, smoother, predictor) so that: P−1 dna f=−∑ h*a kxnf a kf k=0 How d(n) is defined specifies the operation done: − filtering: d(n)=s(n) − predicting: d(n)=s(n+p) − smoothing: d(n)=s(n-p)

08/29/12 EC4440.MPF -Section 00 6

™ How to find hk? ˆ 2 Minimize the MSE: Edn{ ()− d ()n }

P−1 * H ∑ hxk () n−=khx k =0 T T hh==−+[]01,, hP− x⎣⎡ xn(),xn (P 1⎦⎤ ) Wiener filter is a ⇒ orthogonality principle applies

* ⇒−Exn{ () ie( n)} =∀=−0, i 0,..., P 1

⎧⎫P−1 * ⎪⎪⎡⎤* Exni⎨⎬()()−−⎢⎥ dn∑ hxnkk −=∀=− (0, i ) 0,..., P 1 ⎩⎭⎪⎪⎣⎦k=0 P−1 * ⇒−−rixd()∑ hRki k x −=∀=− (0, i ) 0,..., P 1 k=0

08/29/12 EC4440.MPF -Section 00 7 P−1 * ridx()= ∑ hRki k x −∀=− (, i 0,..., ) P 1 k=0 Matrix form: * i=0 ⇒ ⎡⎤rdx ()0 ⎡⎤RRxx(01) () " RP x( − 1) ⎡ h0 ⎤ ⎢⎥* ⎢⎥⎢ ⎥ r ()1 RR()−−10" () RP 2h ( ) i =1 ⇒ ⎢⎥dx =⎢⎥xx x⎢ 1 ⎥ ⎢⎥# ⎢⎥# ⎢ # ⎥ ⎢⎥⎢⎥⎢ ⎥ ⎣⎦# ⎣⎦⎢⎥RPxx()−+10 R ⎣hP−1⎦ ()

Note: different notation than in [Therrien, section 7.3]!

08/29/12 EC4440.MPF -Section 00 8 ™ Minimum MSE (MMSE) obtained when h is obtained from solving WH equations.

For best h obtained: σ 2*==−Ee2 E(() dn dne ()) emin {}min{} min

* = Edne{}()min () n

⎧ P−1 * ⎫ ⎪ ⎛⎞* ⎪ =−−Edn⎨ ()⎜⎟ dn ()∑ hxnkk (⎬ ) ⎪⎩⎭⎝⎠k=0 ⎪ P−1 =−Rhrkddx()0 ∑ k () k=0 σ 2 =−R 0 hrT edmin () dx

08/29/12 EC4440.MPF -Section 00 9 Summary: FIR Wiener Filter Equations

d(n)  + x(n) dna f s(n) + filter - signal + noise w(n) e(n) • FIR Wiener filter is a FIR filter such that:

P−1 ˆ * dn()= ∑ hxnk − k ( ) k=0 2 2 ⎡⎤ where σ =−Edn() dnˆ () is minimum. e ⎣⎦⎢⎥

08/29/12 EC4440.MPF -Section 00 10 • How d(n) is defined specifies the specific type of Wiener filter designed: filtering:

smoothing:

predicting:

** • → W-H eqs: ⎧ −1 ⎪ Rhxx==>= rdx hopt R rdx ⎨ σ2 =−RhrRrh(0)TT =− (0) • MMSE: ⎩⎪ edmin opt dx d dx opt

08/29/12 EC4440.MPF -Section 00 11 ™ One-step ahead Wiener predictor • tracking of moving series • forecasting of system behavior • data compression • telephone transmission

• W-H equations

−1 * hRropt = x dx P−1 ˆ * where dn()= ∑ hxnl −A ( ) A=0 () dn= ?

08/29/12 EC4440.MPF -Section 00 12 ™ Wiener predictor geometric interpretation: Assume a 1- step ahead predictor of length 2 (no additive noise)

x(n+1) e(n) x(n) dnˆ ( )= dnˆ ( ) en()= x(n-1)

e(n) is the error between true value x(n+1) and predicted value for x(n+1) based on predictor inputs x(n) and x(n-1) Æ represents the new information in x(n+1) which is not already contained in x(n) or x(n-1) Æ e(n) is called the innovation process corresponding to x(n)

08/29/12 EC4440.MPF -Section 00 13 Geometric interpretation, cont’ Assume x(n+1) only has NO new information (i.e., information in x(n+1) is that already contained in x(n) and x(n-1). Filter of length 2.

Plot xn(+ 1), dˆ ( n) , e( n)

x(n) dnˆ ( ) = en()= x(n-1)

08/29/12 EC4440.MPF -Section 00 14 Geometric interpretation, cont’ Assume x(n+1) only has new information (i.e., information in x(n+1) is that NOT already contained in x(n) and x(n-1). Filter of length 2.

Plot xn(+ 1), dˆ ( n) , e( n)

x(n) dnˆ ( ) = en()= x(n-1)

08/29/12 EC4440.MPF -Section 00 15 ™ Example 1: Wiener filter (filter case: d(n) = s(n) & white noise) Assume x(n) is defined by d(n)  + x(n) dna f s(n) + filter - signal + noise w(n) e(n) s(n), w(n) uncorrelated

w(n) white noise, zero mean Rw( n) = 2δ(n) |n| s(n) Rs (n) = 2 (0.8)

08/29/12 EC4440.MPF -Section 00 16 08/29/12 EC4440.MPF -Section 00 17 Filter Filter coefficients MMSE length 2 [0.405, 0.238] 0.81 3 [0.382, 0.2, 0.118] 0.76 4 [0.377, 0.191, 0.01, 0.06] 0.7537 5 [0.375, 0.188, 0.095, 0.049, 0.7509 0.029] 6 [0.3751, 0.1877, 0.0941, 0.0476, 0.7502 0.0249, 0.0146] 7 [0.3750, 0.1875, 0.0938, 0.0471, 0.7501 0.0238, 0.0125, 0.0073] 8 [0.3750, 0.1875, 0.038, 0.049, 0.75 0.0235, 0.0119, 0.0062, 0.0037

08/29/12 EC4440.MPF -Section 00 18 ™ Example 2: Application to Wiener filter (filter case: d(n) = s(n) & colored noise) s(n), w(n) uncorrelated, and zero-mean |n| w(n) noise with Rw (n) = 2 (0.5) |n| s(n) signal with Rs (n) = 2 (0.8)

08/29/12 EC4440.MPF -Section 00 19 08/29/12 EC4440.MPF -Section 00 20 Filter Filter coefficients MMSE length 2 [0.4156, 0.1299] 0.961 3 [0.4122, 0.0750, 0.0878] 0.9432 4 [0.4106, 0.0737, 0.0508 0.9351 0.0595] 5 [0.4099, 0.0730, 0.0499, 0.0344, 0.9314 0.0403] 6 [0.4095, 0.0728, 0.0495, 0.0338, 0.9297 0.0233, 0.0273] 7 [0.4094, 0.0726, 0.0493, 0.0335, 0.9289 0.0229, 0.0158, 0.0185] 8 [0.4093, 0.0726, 0.0492, 0.0334, 0.9285 0.0227, 0.0155, 0.0107, 0.0125]

08/29/12 EC4440.MPF -Section 00 21 ™ Example 3: 1-step ahead predictor AR (1) process RP x(n) defined as x(n) =x(n-1) +v(n) |a| < 1 Predictor a = 0.5 v(n) is white noise. 1-step predictor of length 2.

seed = 1024  xn()=+ axn12 −12 + ( axn − ) (+ 05. ) a = L O NM 0 QP

08/29/12 EC4440.MPF -Section 00 22 08/29/12 EC4440.MPF -Section 00 23 ™ Link between Predictor behavior & input signal behavior 1) Case 1: s(n) = process with correlation

Rks ()=+δ () k 0.5(δδ k −++ 1)0.5( k 1)

Investigate performances of N-step predictor as a function of changes N

2) Case 2: s(n) = process with correlation

||k Rks ()=< a ,||1 a Investigate performances of predictor as a function of changes in a

08/29/12 EC4440.MPF -Section 00 24 1) Case 1: s(n) = wss process with

Rks ()=+δ () k 0.5(δδ k −++ 1)0.5( k 1)

08/29/12 EC4440.MPF -Section 00 25 ||k 2) Case 2: s(n) = wss process with Rks ()= a ,||1 a< % EC3410 - MPF % Compute FIR filter coefficients for % a 1-step ahead predictor of length 2 % for correlation sequence of type R(k)=a^{|k|} % Compute and plot resulting MMSE value A=[-0.9:0.1:0.9]; for k0=1:length(A) a=A(k0); for k=1:3 rs(k)=a^(k-1); end Rs=toeplitz(rs(1:2)); rdx=[rs(2);rs(3)]; h(:,k0)=Rs\rdx; mmse(k0)=rs(1)-h(:,k0)'*rdx; end stem(A,mmse) xlabel('value of a') ylabel('MMSE(a)') title('MMSE(a)for1-step predictor of length 2, … for R_s(k)=a^{|k|}')

08/29/12 EC4440.MPF -Section 00 26 ™ Example 4: n Rnsa f = 208a . f s(n) = process with Rnw a f = 2δa nf w(n) = white noise, zero mean s(n), w(n) uncorrelated Design the 1-step ahead predictor of length 2. Compute MMSE.

08/29/12 EC4440.MPF -Section 00 27 1- step ahead Predictor Filter Length Coefficients MMSE Filter MMSE 2 [0.3238, 0.1905] 1.2381 0.81 3 [0.3059, 01.6, 0.0941] 1.2094 0.76 4 [0.3015, 0.1525, 0.0798, 1.2023 0.7537 0.0469] 5 [0.3004, 0.1506, 0.0762, 0.0762, 1.2006 0.7509 0.04, 0.0234] 6 [0.3001, 0.1502, 0.0753, 0.0381, 1.2001 0.7502 0.0199, 0.0199] 7 [0.3, 0.15, 0.0751, 0.0376, 1.2 0.7501 0.0190, 0.001, 0.0059] 8 [0.3, 0.15, 0.075, 0.0375, 1.2 0.75 0.0188, 0.0095, 0.0050, 0.003]

08/29/12 EC4440.MPF -Section 00 28 N-step ahead Predictor Filter Length 1-step ahead 2-step ahead Filter MMSE MMSE MMSE 2 1.2381 1.5124 0.81 3 1.2094 1.494 0.76 4 1.2023 1.4895 0.7537 5 1.2006 1.4884 0.7509 6 1.2001 1.4881 0.7502 7 1.2 1.4880 0.7501 8 1.2 1.4880 0.75

08/29/12 EC4440.MPF -Section 00 29 n ™ Example 5: Rns a f = 208a . f n s(n) = process with Rnwa f = 205a . f w(n) = wss noise, zero mean s(n), w(n) uncorrelated • Design the 1-step ahead predictor of length 2 • Design 1-step back smoother of length 2

08/29/12 EC4440.MPF -Section 00 30 08/29/12 EC4440.MPF -Section 00 31 1-step ahead predictor (Col. Noise) Length Coefficients MMSE 2 [0.3325, 0.1039] 1.3351 3 [0.3297, 0.06, 0.0703] 1.3237 4 [0.3285, 0.0589, 0.0406, 0.0476] 1.3185 5 [0.3279, 0.0584, 0.04, 0.0275, 1.3161 0.0322] 6 [0.3276, 0.0582, 0.0396, 0.0270, 1.315 0.0186, 0.0218] 7 [0.3275, 0.0581, 0.0394, 0.0268, 1.3145 0.0183, 0.0126, 0.0148] 8 [0.3275, 0.0581, 0.0394, 0.0267, 1.3142 0.018, 0.0124, 0.0085, 0.0100]

08/29/12 EC4440.MPF -Section 00 32 MMSE (Col. Noise) Length N-step ahead Predictor N-step back Smoother Filter 1-step 2-step 3-step 4-step 1-step 2-step 3-step 4-step 2 1.3351 1.5744 1.7276 1.8257 0.961 1.3351 1.5744 1.7276 0.961 3 1.3237 1.5672 1.7230 1.8227 0.925 0.9432 1.3237 1.5672 0.9432 4 1.3185 1.5638 1.7208 1.8213 0.9085 0.9085 0.9351 1.3185 0.9351 5 1.3161 1.5623 1.7199 1.8207 0.9009 0.8926 0.9009 0.9314 0.9314 6 1.315 1.5616 1.7194 1.8204 0.8975 0.8853 0.8853 0.8975 0.9297 7 1.3145 1.5613 1.7192 1.8203 0.8959 0.8819 0.8781 0.8819 0.9289 8 1.3142 1.5611 1.7191 1.8202 0.8952 0.8804 0.8748 0.8748 0.9285

08/29/12 EC4440.MPF -Section 00 33 Comments

08/29/12 EC4440.MPF -Section 00 34 ™ Example 6: s(n) and w(n) defined as before Rn= 208. n with w(n) zero mean, s a f a f n and s(n) and w(n) uncorrelated Rnwa f = 205a . f

Design the 3-step ahead predictor of length 2, and associated MMSE

08/29/12 EC4440.MPF -Section 00 35 08/29/12 EC4440.MPF -Section 00 36 ™ Wiener Filters and Error Surfaces * R x hropt = dx Recall hopt computed from

2 * 2 HHH σ e =−=−E{} dn() hx E{( dn() hx)( dn() − hx)}

HH T =+RhExxhhrd ()02{} −Real()dx

for real signals d(n), x(n)

2 TT σ ed=+−R ()02hRh x hrdx

using the fact that * (hxHH) = xh dnhx()HH= hdnx()

08/29/12 EC4440.MPF -Section 00 37 2 TT σed=+−R ()02hRh x hrdx

‰ For filter length P = 1 h = h0 x = x(n)

2 2 σed=+RhRhr()00 ()x 0200 − ()dx () ()

2 σe

2 σemin h(0) hopt (0)

08/29/12 EC4440.MPF -Section 00 38 2 TT σed=+−R ()02hRh x hrdx ‰ For filter length P = 2

TT h==−⎣⎦⎣⎡⎤⎡ h()0, h 1 () ; x xn() , xn ( 1 ⎦⎤ )

2 TT σ ed=+−RhRhhr()02 x dx

⎡⎤RRxx()01 ()⎡ h (0)⎤ =+Rhhd ()00,1⎣⎦⎡⎤ () ()⎢⎥⎢ ⎥ ⎣⎦RRxx()10 ( )⎣ h (1) ⎦

⎡⎤rdx ()0 − 20,1⎣⎦⎡⎤hh() ()⎢⎥ ⎣⎦rdx ()1

2 22 ⇒=σ e Ah0123(0110 ) + Ah () + Ah () + Ah ( )

++Ah4 ()()01 h Rd () 0

08/29/12 EC4440.MPF -Section 00 39 2 σ e

hopt (0 ) h (0 ) hopt (0 ) h (1) h (0 ) opt

hopt ()1 h (1) hopt depends on rdx and Rx h ()1 2 TT σ ed=+−RhRhhr(02) x dx 2 shape of σ e depends on Rx information: λ(Rx ) & eigenvectors moves 2 up and σ e down ∂σ e =−22Rhx rdx 2 ∂h Rx: specifies shape of σe (h)

rdx: specifies where the bowl is in the 3-d plane but doesn’t change the shape of the bowl Rhx = rdx Rd (0): moves bowl up and down in 3-d plane but doesn’t change shape or location of bowl

08/29/12 EC4440.MPF -Section 00 40 ‰Correlation matrix Eigenvalue Spread Impact on Error Surface Shape see plots Eigenvector Direction for 2 × 2 Toeplitz Correlation Matrix

⎡⎤RRxx(01) ( ) normalize⎡⎤ 1 a R x =→⎢⎥ ⎢⎥ ⎣⎦RRxx()10 ( ) correlation⎣⎦a 1

eigenvalues of Rx 2 2 ⎧1 − a ()10−−=⇒=λλa ⎨ ⎩1 + a eigenvectors

⎛⎞1 − λ a ⎛⎞u11 ⎜⎟⎜⎟= 0 ⎝⎠a 1 − λ ⎝⎠u12

⇒−(10λ ) uau11 += 12

λ1 =−11a ⇒ ( − 1 + au) 11+= au 12 0 ⎡⎤1 ⇒=−uu11 12 ⇒=u1 ⎢⎥ ⎣⎦−1 ⎡1⎤ λ 2 =+1 au ⇒2 =⎢ ⎥ ⎣1⎦ 08/29/12 EC4440.MPF -Section 00 41 ™ Error surface shape and eigenvalue ratios

a=0.1 a=0.99

a=0.5

08/29/12 EC4440.MPF -Section 00 42 ™ Application to Channel Equalization

Delay D v(n) d(n) = s(n-D) dnˆ s(n) z(n) x(n) ( ) + W(z) ⊕ H(z) −⊕ e(n) Goal: Implement the equalization filter H(z) as a stable causal FIR filter

• Goal: Recover s(n) by estimating channel distortion (applications in communications, control, etc.) • Information Available: xn()=+zn( ) v (n ) channel output additive noise due to sensors dn( )=− sn( D) s(n): original data samples • Assumptions: 1) v(n) is stationary, zero-mean, uncorrelated with s(n).

2) v(n) = 0 & D=0

08/29/12 EC4440.MPF -Section 00 43 Assume: W(zzz )=++ 0.2798−12 0.2798 −

Questions: 1) Assume v(n) = 0 & D=0. Identify the type of filter (FIR/IIR) needed to cancel channel distortions. Identify resulting H(z) 2) Identify whether the equalization filter is causal and stable.

3) Assume v(n) = 0 & D≠0. Identify resulting H2(z) in terms of H(z).

Delay D d(n) = s(n-D) dnˆ s(n) z(n) ( ) + W(z) H(z) −⊕ e(n)

08/29/12 EC4440.MPF -Section 00 44 08/29/12 EC4440.MPF -Section 00 45 08/29/12 EC4440.MPF -Section 00 46 08/29/12 EC4440.MPF -Section 00 47 -9 -7 -5 -3 -1 0 1 3 5 7

08/29/12 EC4440.MPF -Section 00 48 Assume D≠0

Delay D d(n) = s(n-D) dnˆ s(n) z(n) () + W(z) H2 (z) −⊕ e(n)

08/29/12 EC4440.MPF -Section 00 49 08/29/12 EC4440.MPF -Section 00 50 -5 0 5

08/29/12 EC4440.MPF -Section 00 51 ™ Application to Noise Cancellation

‰ Goal: Recover s(n) by compensating for the noise distortion while having only access to the related noise distortion signal v(n) (applications in communications, control, etc.)

Signal d(n)=s(n)+w(n) + en()=− dn () dn () source s(n) -

w(n) ™ Information Available: s(n)+w(n) & v(n) xn()= vn () Transformation White Of white noise Wiener dn () ™ Assumption: w(n) is noise into correlated filter stationary, zero-mean, source noise w(n) uncorrelated with s(n). A(z) v(n)

08/29/12 EC4440.MPF -Section 00 52 Assume: vn( )=−+ avn ( 1) wn ( ), a = 0.6, 2 wn ( ) white noise with variance σ w

sn ( )=+ sin(ω0 nφφ ), ~ π U [0,2 ]

Compute the FIR Wiener filter of length 2 and evaluate filter performances

08/29/12 EC4440.MPF -Section 00 53 08/29/12 EC4440.MPF -Section 00 54 Results Interpretation

08/29/12 EC4440.MPF -Section 00 55 ™ Application to Noise Cancellation (with information leakage)

Signal d(n)=s(n)+w(n) + en()=− dn () dn () source - s(n) w(n) ™ Information Available: K s(n)+w(n) & v(n)+Ks(n) xn()= vn ()+ Ksn () Transformation White Of white noise Wiener dn ()™ Assumption: w(n) is noise into correlated filter stationary, zero-mean, source noise uncorrelated with s(n). w(n) A(z) v(n)

08/29/12 EC4440.MPF -Section 00 56 08/29/12 EC4440.MPF -Section 00 57 Results Interpretation

08/29/12 EC4440.MPF -Section 00 58 ™ Application to Spatial Filtering Reference d(n)=s(n) signal e(n)

x1 (n) s(n) v1 (n) dn () x1 (n) ±1 x2 (n) BPSK Filtering v2 (n) x (n) Noisy 2 received signals

™ Information Available: ™ Assumption: v1(n), v2(n) ™ Goal: Denoise snapshot in time of received zero mean wss white noise received signal signal retrieved at two RPs independent of each antennas & reference signal other and of s(n).

08/29/12 EC4440.MPF -Section 00 59 ™ Application to Spatial Filtering, cont’

08/29/12 EC4440.MPF -Section 00 60 08/29/12 EC4440.MPF -Section 00 61 Example: Gain Pattern at filter output Example: N-element array, - desired signal

at θ0 =30°, - interferences

at θ1 = -20° θ2 = 40° - Noise power: 0.1

Array steering vector

2 waH ()θ A ()θ = w wwH

08/29/12 EC4440.MPF -Section 00 62 Application to Spatial Filtering, cont’ Did we gain anything by using multiple receivers? ™ Assumption: v(n) zero mean RP independent of s(n). s(n) x(n) Filtering Process ±1 ™ Goal: Compute filter coefficients, filter output, and BPSK v(n) MMSE.

08/29/12 EC4440.MPF -Section 00 63 08/29/12 EC4440.MPF -Section 00 64 08/29/12 EC4440.MPF -Section 00 65 Appendices

08/29/12 EC4440.MPF -Section 00 66 Appendix A: Derivation of proof for Mean Square estimate derivation (p. 3)  2 Proof: ξ =−Es{ dx()} T =−Esdxsdx () − () {}() ()

Define L (ss,  () x) : loss function ξ = ∫∫ Lsdx( ,, ()) f( sxf) () xdxdsM Bayes Rule =,∫∫⎡⎤Lsdx() () f() sxdsf() xdM x  ⎣⎦ N ≥ 0 K ξ is minimized if K is minimized for each value of x  Problem: find dx ( ) so that K is minimum ∂∂K 2 =−⎡ (s dx ()) fsxds()⎤ ∂∂dd  ⎣⎢ ∫ ⎦⎥ =−2 ∫ ()sdxfsxds − () () ∂ K =⇒00(sdxfsxds − ()) ()= ∂ d ∫ ⇒=∫∫s f() sxds d () xf() sxds ⇒=∫∫s fsxdsdx()  () fsxds() ⇒=×∫ sf() s x ds d () x 1 ⇒=Es [ | x ] d () x

08/29/12 EC4440.MPF -Section 00 67 ∂ 2 K For a minimum we need: 2 > 0 ∂d Note: ∂ 2 K =−(2) ∂s −dx fsxds / ∂ d 2 (∫()()()) ∂d =−(2)∫ −fsxds() =(2)×> 1 0

08/29/12 EC4440.MPF -Section 00 68 Appendix B: Proof of corollary of orthogonality principle Proof: H H   eshxs=− =−( hhhx +−) where h is the weight vector defined so that the orthogonality principle holds. Resulting error is called H eshx =−

H H 2 2 ⎧⎫⎛⎞  H σ e ==−+−Ee{} E⎨⎜ s hx( h h) x ⎟() ⎬ ⎩⎭⎝⎠ ⎧⎫⎛⎞  H H =+−Eehhx⎨⎜ ()⎟()⎬ ⎩⎭⎝⎠

2 HH2 ⎧⎫ H  H  =+−+−+−Ee⎨⎬() hh xe() hh x exhh() ⎩⎭

2 HH2  H ⎧⎫ H  =+−E {}ehhExeEhhxEexhh(){} +−⎨⎬() +{}() − ⎩⎭ 2  HH H H H ==−=Ee{} E{}{}() s hxe Ese − hExe{} H * ==Ese{} Ese {}

08/29/12 EC4440.MPF -Section 00 69 References

[1] C.W. Therrien, Discrete Random Signals and Statistical . [2] D. Manolakis, V. Ingle, S. Kogon, Statistical and Adaptive Signal Processing, Artech House, 2005. [3] S. Haykin, Theory, Prentice Hall 2002.

08/29/12 EC4440.MPF -Section 00 70