Signal Processing and Time-Series Analysis 1. Signal Processing A. Analytical Signals are recorded as: Spectra, chromatograms, voltammograms or titration curves (monitored in frequency, wavelength, time) B. Signal processing is used to distinguish between signal and noise. 1 Signal Processing and Time-Series Analysis 1. Signal Processing C. Methods of Evaluating Analytical Signals 1) Transformation 2) Smoothing 3) Correlation 4) Convolution 5) Deconvolution 6) Derivation 7) Integration Important as data is usually processed digitally 2 1 Signal Processing and Time-Series Analysis D. Digital smoothing and Filtering 1) Moving Average Filtering – smoothes data by replacing each data point with the average of the neighboring data points: 1 y (i) = [y(i + N) + +y(i + N -1) +...+ y(i - N)] s 2N +1 Where ys(i) is the smoothed value for the ith data point, N is the # of neighboring data points on either side of ys(i), and 2N+1 is the span (filter width). 3 Signal Processing and Time-Series Analysis D. Digital Smoothing and Filtering 1. Moving Average Filtering – Rules for selecting the most appropriate filter: • When applied repetitively, the largest smoothing effect (>95%) is observed in the first application (single smoothing usually sufficient). • Filter width should correspond to the full width at half maximu m of q band or a peak. à Too small a width results in unsatisfactory smoothing. à Too large of a width leads to distortion of the original data structure • Distortion of data structure is more severe in respect of the area than of the height of the peaks. à Filter width selected must be smaller if the height rather than the area is evaluated. 4 2 Signal Processing and Time-Series Analysis D. Digital Smoothing and Filtering 1. Moving Average Filtering Note: The influence of the filter-width on the distortion of the peaks can be quantified by means of the relative filter width, brelative: b filter b = relative b 0.5 Where bfilter is the filter width, and b0.5 is the full width at half maximum. 5 Signal Processing and Time-Series Analysis E. Savitzky-Golay Filter (Polynomial smoothing) à smoothing that seeks to preserve shapes of peaks -After deciding on the filter width, the filtered value for the kth data point is calculated from: * 1 å j k + j y k = NORM c y where NORM is a normalization factor obtained from the sum of the coefficients cj 6 3 Signal Processing and Time-Series Analysis F. Kalman Filter à Estimate the state of a system from measuring which contain random errors à Based on two models: 1) Dynamic System model (Process) x(k) = F x(k -1) + w(k -1) 2) Measurement Model y(k) = HT(k) x (h) + v(h) - where x = state vector, y = the measurement, F = system transition matrix and H = the measurement vector (matrix). - w = signal noise vector, v = measurement noise vector - h = denotes the actual measurement or time 7 Signal Processing and Time-Series Analysis F. Kalman Filter 1) only matrix operations allowed a) Dynamic System é1 0 Xn - 1ù éXn ù ê ú é~ Vk - 1ù ê ú = + ê ú ë yk û ê ú ë~ Yk - 1û ë0 1 Yn - 1û state state state noise transition 8 4 Signal Processing and Time-Series Analysis F. Kalman Filter b) Measurement Model éHx 0 Xnù Uk ~ Uk é ù = ê ú + é ù ëêVk ûú ê ú ëê~ Yk ûú ë 0 Hy Ynû Measurement measurement state noise matrix 9 Signal Processing and Time-Series Analysis G. Signal Derivatives à useful for eliminating background noise, determining peak position and improving the visual resolution of peaks. 10 5 Signal Processing and Time-Series Analysis G. Signal Derivatives Ex: Noise characteristics for derivatives of signals (y-signal around an observation point h). yh-2 yh-1 yh yh+1 yh+2 0.2 0.5 0.7 0.4 0.1 - can calculate filtered values and their standard deviations by means of the tabulated filter coefficients 11 Signal Processing and Time-Series Analysis G. Signal Derivatives Points 25 23 21 19 17 15 13 11 9 7 5 -5 322 54 204 144 18 42 0 -36 -4 387 63 249 189 27 87 9 9 -21 -3 422 70 284 224 34 122 16 44 14 -2 -2 447 75 309 249 39 147 21 69 39 3 -3 -1 462 78 324 264 42 162 24 84 54 6 12 0 447 79 329 269 43 167 25 89 59 7 17 +1 462 78 324 264 42 162 24 84 54 6 12 +2 447 75 309 249 39 147 21 69 39 3 -3 +3 422 70 284 224 34 122 16 44 14 -2 +4 387 63 249 189 27 57 9 9 -21 +5 322 54 204 144 18 42 0 -36 12 6 Signal Processing and Time-Series Analysis H. Transformations à useful for filtering of data, convolution and deconvolution of analytical signals, integration, background correction and reducing data points. 1) Fourier Transforms – integral transform that re-expresses a function x(k) = F x(k -1) + w(k -1) 2) Measurement Model y(k) = HT(k) x (h) + v(h) - where x = state vector, y = the measurement, F = system transition matrix and H = the measurement vector (matrix). - w = signal noise vector, v = measurement noise vector - h = denotes the actual measurement or time 13 Signal Processing and Time-Series Analysis Figure: Fourier Transform of a One Dimensional Signal Source: NikosDrakos, Computer Based Learning Unit, University of Leads & Kristian Sandberg, University of Colorado. 14 7 Signal Processing and Time-Series Analysis 3. Hadamard Transformation à Based on the Walsh Function in contrast to the sine and cosine functions of FT. y* = Hy where H is the (n x n) Hadaward matrix, y is the vector of the original n signal values and y* is the vector of the transformed n signal values The hth Hardamard Transform matrix H H é h-1 h-1 ù H = ê ú h êH - H ú ë h-1 h-1 û 15 Signal Processing and Time-Series Analysis 3. Hadamard Transformation Ex: Four data points are to be treated with HT. with n = 2h = 4 we have h=2. If we set H0 = 1 then we would obtain for the matrices H1 and H2: æ1 1 ö H = ç ÷ 1 ç ÷ è1 - 1ø æ1 1 1 1 ö ç ÷ ç ÷ æH H ö 1 - 1 1 - 1 ç 1 1 ÷ ç ÷ H = = ç ÷ 2 ç ÷ çH -H ÷ ç1 1 - 1 - 1 ÷ è 1 1 ø ç ÷ ç ÷ ç1 - 1 - 1 1 ÷ è ø 16 8 Signal Processing and Time-Series Analysis 3. Hadamard Transformation Transformation Equation (according to y*=Hy): * æ y ö æ1 1 1 1 öæ y ö ç 1 ÷ ç ÷ç 1 ÷ ç ÷ ç ÷ç ÷ ç y* ÷ 1 - 1 1 -1 y 2 ç ÷ç 2 ÷ ç ÷ = ç ÷ç ÷ ç * ÷ y ç1 1 -1 -1÷ç y ÷ ç 3 ÷ ç ÷ç 3 ÷ ç ÷ ç ÷ç ÷ ç y* ÷ ç1 - 1 -1 1 ÷ç y ÷ è 4 ø è øè 4 ø multiplication of the equations = transformed signal * y = y + y + y + y 1 1 2 3 4 * y = y + y + y + y …… and so on. 2 1 2 3 4 *Insert own #’s to transform your signals. 17 Signal Processing and Time-Series Analysis 3. Hadamard Transformation Advantages over FT a. Simple arithmetic operations (addition & subtraction) b. Faster algorithm c. Real (no imaginary transformations) Applications a. Signal filtering – suppresses high frequency noise or drift b. Convolution and Deconvolution – restoration of signal distorted by instrument function or overlapping signals c. Integration – of area (how does this differ from peak height?) d. Data reduction and background correction 18 9 Time-Series Analysis 4. Time-Series Analysis -characterization of a set of measurements as a function of time e.g. Phosphorus concentrations in rivers: 5.6 5.5 5.4 Downtime/ Filter 5.3 5.2 Downtime/ 5.1 Calibration Calibration 5.0 Concentration FRP (uM) Filter/ Filter/ 4.9 Calibration Calibration Calibration 4.8 11/07/0017:00 03:0012/07/00 13:00 23:0013/07/00 Date 19 Time-Series Analysis 1. Field-Based/Submersible Approaches 2. Laboratory-Based Approaches SC SFC - + SV1 SV2 A C B W W B Pump S 3. Chemometric Approach Experimental Design and Multivariate Data Analysis 20 10 Time-Series Analysis Temperature FRP Concentration 25 6 20 5.5 15 5 10 5 4.5 Air Temperature (oC) 0 4 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 Concentration FRP (uM) Sample Any Relationship? 21 Time-Series Analysis Mean monthly physico-chemical parameters of the River Frome from 1990 –1998. Month EpCO2 Alkalinity Calcium pH Conductivity Discharge FRP TP (m Eq L-1) (m Eq L-1) (m S) (m 3 s-1) (m M) (m M) January 10.56 3.94 4.77 8.32 514 8.40 4.38 7.38 February 12.61 4.01 4.80 7.75 507 8.03 4.04 6.88 March 10.72 4.11 4.90 7.86 521 4.87 3.44 5.97 April 9.87 4.03 4.86 7.89 508 4.24 2.87 5.54 May 9.43 4.14 4.84 7.91 515 6.33 3.38 6.22 June 8.33 4.13 4.81 7.96 515 5.61 4.67 8.23 July 11.10 4.07 4.82 7.83 514 5.71 5.34 7.88 August 11.32 4.12 4.82 7.84 510 6.84 5.47 8.87 September 13.67 3.98 4.65 7.74 514 7.37 5.70 9.40 October 13.82 3.91 4.62 7.73 511 6.90 5.70 8.44 November 12.53 3.72 4.50 7.75 505 6.92 4.89 7.88 December 10.22 3.84 4.62 7.83 489 7.09 4.36 8.29 Any Relationships? 22 11 Time-Series Analysis 8 6 Discharge FRP 7 5 ) -1 s FRP [uM] 3 6 4 Discharge (m 5 3 4 2 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec -- Month 23 Time-Series Analysis A.