Spectral Estimation Examples from research of Kyoung Hoon Lee, Aaron Hastings, Don Gallant, Shashikant More, Weonchan Sung Herrick Graduate Students Estimation: Bias, Variance and Mean Square Error Let φ denote the thing that we are trying to estimate. Let φ ˆ denote the result of an estimation based on one data set with N pieces of information. Each data set used for estimation à a different estimate of φ. ˆ ˆ Bias: b ( φ ) = φ − E [ φ ] True value - the average of all possible estimates formed from N data points 2 2 Variance: σ = E[ (φˆ − E[φˆ]) ] Measure of the spread of the estimates about the mean of all estimates. 2 2 2 Mean Square Error: m.s.e. = E[ (φˆ −φ) ] = b +σ Estimation: Some definitions Estimate is consistent if, when we use more data to form the estimate, the mean square error is reduced. If we have two ways of estimating the same thing, we say that the estimator that leads to the smaller mean square error is more efficient than the other estimator. true estimates value φ = (a,b) bias xxxx b xx x mean of all x x x estimates a Examples 1 N Bias and variance of an estimate of the mean: X ,µˆ = ∑ Xn N n=1 ⎡ 1 N ⎤ 1 N 1 N E[µˆ] = E⎢ ∑ X ⎥ = ∑ E⎡X ⎤ = ∑ µ = µ (unbiased) N n N ⎣ n⎦ N ⎣⎢ n=1 ⎦⎥ n=1 n=1 Derivation ⎡ 2⎤ ⎡ 2⎤ ⎛⎛ N ⎞ ⎞ ⎛ N ⎞ ⎡ 2⎤ ⎢ 1 ⎥ ⎢ 1 ⎥ assuming that 2 ˆ ˆ ⎜⎜ ⎟ ⎟ ⎜ ⎟ σµˆ E⎢ µ − E[µ] ⎥ = E⎢ ∑ Xn − µ ⎥ = E⎢ ∑ Xn − µ ⎥ ⎣( ) ⎦ ⎜⎜ N ⎟ ⎟ ⎜ N ( )⎟ the samples X ⎢⎝⎝ n=1 ⎠ ⎠ ⎥ ⎢⎝ n=1 ⎠ ⎥ n ⎣ ⎦ ⎣ ⎦ are independent 1 ⎡ N N ⎤ = E⎢ X − µ X − µ ⎥ 2 ∑ ∑ ( m )( n ) of one another. N ⎣⎢n=1m=1 ⎦⎥ ⎧ 2 ⎫ 1 2 ⎡ ⎤ ⎡ ⎤ = ⎨ N − N E⎣(Xn − µ)(Xm − µ)⎦ + N E⎢(Xn − µ) ⎥⎬ N 2 ⎩( ) ⎣ ⎦⎭ Separate into terms ⎧ 2 ⎫ 1 2 ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ = ⎨ N − N E⎣(Xn − µ)⎦E⎣(Xm − µ)⎦ + N E⎢(Xn − µ) ⎥⎬ where n does not N 2 ⎩( ) ⎣ ⎦⎭ 1 ⎡ 2⎤ 1 2 equal m and where = N E⎢(Xn − µ) ⎥ = σ x N 2 ⎣ ⎦ N n=m Examples Biased Estimate of the variance of a set of N measurements: N 1 2 ∑ (Xn − µˆ) N n=1 Unbiased Estimates of the variance of a set of N measurements:: N N 1 2 1 2 ∑ (Xn − µˆ) and ∑ (Xn − µ) N −1n=1 N n=1 First estimate the mean, and use Special case where the mean is that estimate in this calculate known and doesn’t need to be (have lost 1 degree of freedom) estimated from the data Estimation of Autocovariance functions Two methods of estimating Rxx(τ) from T sec. of data. 1. Dividing by the integration time: T-|τ| Estimation was unbiased but had very high variance, particularly when τ is close to T. 2. Dividing by total time: T Estimation was biased (asymptotically unbiased). This was equivalent to multiplying first estimate by a triangular window (T-|τ|)/T. This window attenuates the high variance estimates. x(t) x(t) x(t+τ) T secs τ time Calculating the average value of [x(t) x(t+τ)] from T seconds of data. Estimation of Autocovariance functions Two methods of estimating Rxx(τ) from T sec. of data. 1. Dividing by the integration time: T-|τ| Estimation was unbiased but had very high variance, particularly when τ is close to T. 2. Dividing by total time: T Estimation was biased (asymptotically unbiased). This was equivalent to multiplying first estimate by a triangular window (T-|τ|)/T. This window attenuates the high variance estimates. x(t+τ) x(t) T secs x(t) x(t+ ) x(t) τ τ time x(t) x(t+ ) x(t+ ) x(t) τ τ x(t) τ τ τ τ Calculating the average value of [x(t) x(t+τ)] from T seconds of data. Estimation of Cross Covariance Same issues as for Auto-Covariance: Bigger τ less averaging for finite T. y(t-τ) x(t) x(t) T time τ y(t) T x(t) y(t+τ) time x(t) and y(t), zero mean, weakly stationary random processes. Average value of [x(t) y(t+τ)]. Additional problem: must make T large enough to accommodate system delays. Estimation of Covariance With fast computation of spectra, these are now more usually estimated by inverse Fourier transforming the power and cross spectral density estimates. Inverse transform of RAW PSD or CSD ESTIMATE equivalent to Method 2 for calculating covariance functions with triangular window for data of size Tr Power Spectral Density Estimation Definition: ⎡ X * X ⎤ S ( f ) = lim E⎢ T T ⎥ = +∞ R (τ)e− j2π f τ dτ. xx T → ∞ ∫−∞ xx ⎣⎢ T ⎦⎥ Estimation: 1. Could Fourier Transform the Autocorrelation Function estimate (not computationally efficient). 2. Could use the frequency domain definition directly. ⎡ * ⎤ ˆ XT XT Raw Estimate = Sxx ( f ) = ⎢ ⎥ ⎣⎢ T ⎦⎥ No averaging! Extremely poor variance characteristics. 2 Variance is S x x ( f ) and is unaffected by T, the length of data used. Power Spectral Density Estimation (Continued) Smoothed estimate from segment averaging. w(t) x(t) Ts time 1. Break signal up into Nseg segments, Tr seconds long. 2. For each segment: 1. Apply a window to smooth transition at ends of segments 2. Fourier Transform windowed segment à XT(f) 2 3. Calculate a raw power spectral density: |XTs | /Ts estimate 3. Average the results from each segment to get the smoothed estimate and do a power compensation for the window used. NSEG 1 ˆ 1 2 S!xx ( f ) = Sxx ( f ) wcomp = w (t)dt NSEG.wcomp ∑ i T ∫ i=1 Power Spectral Density Estimation (Continued) Smoothed estimate from segment averaging. w(t) x(t) Ts time Overlap: For some windows segment overlap makes sense. A Hann window, 50% overlap means that data de-emphasized in one windowed segment is strong emphasized in the next window (and vice versa). Bias: Note PSD estimate bias is controlled by the size of the window (Ts) which controls the frequency resolution (1/Ts). Larger window, smoother transitions à less power leakage à less bias Power Spectral Density (PSD) Estimation (Continued) We argue that the distribution of the smoothed PSD was related to that of a Chi-squared 2 random variable (χν ) with ν = 2.NSEG degrees of freedom, if Tr was large enough so we could ignore bias errors. Therefore: ⎡2.Nseg.S! ⎤ 4.Nseg2 xx ⎡ ! ⎤ Variance⎢ ⎥ = Variance Sxx = 2(2.Nseg) S 2 ⎣ ⎦ ⎣ xx ⎦ Sxx 2 ! Sxx and rearranging we showed that: Variance[Sxx ] = Nseg Therefore, we can control variance by averaging more segments. Note: shorter segments mean larger bias, so for a fixed T seconds of data, there is a trade-off between Segment Length (Tr), which controls the bias, and Number of Segments (NSEG), which controls the variance: T=Tr.NSEG. Cross Spectral Density (CSD) Definition: ⎡ * ⎤ lim XTYT +∞ − j2π f τ Sxy ( f ) = E⎢ ⎥ = ∫ Rxy (τ)e dτ. T → ∞ ⎢ T ⎥ −∞ ⎣ ⎦ Estimation: Could Fourier Transform the Cross-correlation function estimate (not computationally efficient). Could use the frequency domain definition directly. ⎡ * ⎤ ˆ XTYT Raw Estimate = Sxy ( f ) = ⎢ ⎥ ⎣⎢ T ⎦⎥ As with PSD, this has extremely poor variance characteristics, so – divide the time histories into segments, – generate a raw estimate from each segment, and – average to reduce variance and produce a smoothed estimate. Cross Spectral Density Estimation: Segment Averaging w(t) x(t) time Ts y(t) w(t) Ts time à Fourier Transform of Windowed Segments XT(f) & YT(f). * ˆ XTs( f )YTs( f ) Sxy ( f ) = Raw Estimate from ith segment = i Ts 1 Nseg S! ( f ) = Sˆ ( f ) Smoothed Estimate = xy ∑ xyi Nseg i=1 Issues with Cross Spectral Density Estimates 1. Reduce bias by choosing the segment length (Tr) as large as possible. (Bias greatest where the phase changes rapidly.) 2. Reduce variance by averaging many segments. 3. Might require a large amount of averaging to reduce noise effects: y (t) y(t) n(t) h(t) x(t) n(t) m = + = ∗ + x(t), n(t) zero mean, weakly stationary, uncorrelated random processes 2 Syy H( f ) Sxx ! ! ! ! SNR = = Sxy ≈ H( f )Sxx + Sxn → H( f )Sxx ym S S nyny nyny 1 ⎡ 1 ⎤ Var{Sxy} proportional to ⎢1+ ⎥ Nseg 2 ⎣⎢ γxy ⎦⎥ 4. Time delays between x and y cause problems, if the time delay (to) is greater than a small fraction of the segment length (Tr). Can estimate t0 and offset y segments, but need T+t0 seconds of data. Cross Spectral Density Estimation: Segment Averaging with System Delays w(t) x(t) time estimated Ts t y(t) 0 w(t) Ts time Offsetting y segements essentially removes most ofà the delay from the Fourier Transform of Windowed Segments XT(f) & YT(f). estimated frequency response function. Can put back delay effects in by multiplying estimate of H(f) by: ˆ e− j2π f t0 Coherence Function Estimation: Substitute in Smoothed Estimates of Spectral Densities Coherence takes values in the range 0 to 1. 2 ! 2 2 | Sxy | 2 | Sxy | Definition: γ = ; Estimate: γ! = xy xy ! ! SxxSyy SxxSyy – Substituting raw spectral density estimates into formula results in 1 A result where the coherence = 1 at all frequencies from measured signals should be treated with a high degree of suspicion. – Estimate highly sensitive to bias in spectral density estimates, which is particularly bad where the phase of the cross spectral density changes rapidly (at maxima and minima in |Sxy|). – COHERENCE à 0 because of: NOISE ON INPUT AND OUTPUT NONLINEARITY BIAS ERRORS IN ESTIMATION Example: System with Some Nonlinearities (cubic stiffness) and Noisy Measurements Nonlinearity causes spread of energy here, around 3x and 5x this frequency Nonlinear Mode Poor Poor SNRy SNRy Nonlineary causes broad dips in coherence function.