Particle Filters and Finite Ensemble Kalman Filters in Large Dimensions: Theory, Applied Practice, and New Phenomena Andrew J. Majda Morse Professor of Arts and Science Department of Mathematics Founder of Center for Atmosphere Ocean Science (CAOS) Courant Institute, New York University and Principal Investigator, Center for Prototype Climate Modeling (CPCM) NYU – Abu Dhabi (NYUAD) CAOS colloquium September 12, 2018 Outline I Introduction: Kalman Filters and Finite Ensemble Kalman Filters for Complex Systems I Applied Practice, New Phenomena, and Math Theory for Finite Ensemble Kalman Filters I Preventing Catastrophic Filter Divergence using Adaptive Additive Inflation I State Estimation and Prediction using Clustered Particle Filters I High Dimensional Challenges for Analyzing Finite Ensemble Kalman Filters I Robustness and Accuracy of Finite Ensemble Kalman Filters in Large Dimensions 1 / 51 Outline Introduction: Kalman Filters and Finite Ensemble Kalman Filters for Complex Systems Applied Practice, New Phenomena, and Math Theory for Finite Ensemble Kalman Filters Preventing Catastrophic Filter Divergence using Adaptive Additive Inflation State Estimation and Prediction using Clustered Particle Filters High Dimensional Challenges for Analyzing Finite Ensemble Kalman Filters Robustness and Accuracy of Finite Ensemble Kalman Filters in Large Dimensions 2 / 51 Filtering the Turbulent Signals12 Filtering is a two-step process involving statistical prediction of the state variables through a forward operator followed by an analysis step at the next observation time which corrects this prediction on the basis of the statistical input of noisy observations of the system. 1Majda & Harlim, book Cambridge University Press, 2012; 2Majda, Harlim, & Gershgorin, review article, Discrete Contin. Dyn. Syst, 2010. 3 / 51 Filtering I Signal-observation system with random coefficients Signal: Xn+1 = AnXn + Bn + ξn+1; ξn+1 ∼ N (0; Σn) Observation: Yn+1 = HnXn+1 + ζn+1; ζn+1 ∼ N (0; Iq ) Goal: estimate Xn based on Y1;:::; Yn I An; Bn; Hn (stationary) sequence of random matrices and vectors. I An can be unstable sometimes. Hn can be on and off. 4 / 51 Weather forecast I Signal: Xn+1 = AnXn + Bn + ξn+1; Observation: Yn+1 = HXn+1 + ζn+1. I Weather forecast: Signal: atmosphere and ocean “follow" a PDE. Obs: weather station, satellite, sensors, .... I Main challenge: high dimension, d ∼ 106 − 108. 5 / 51 Kalman filter Use Gaussian: X j ∼ N (m ; R ) I n Y1:::n n n T I Forecast step: m^ n+1 = Anmn + Bn; Rbn+1 = AnRnAn + Σn. I Assimilation step: apply the Kalman update rule mn+1 = m^ n+1 + G(Rbn+1)(Yn+1 − Hnm^ n+1); Rn+1 = K(Rbn+1) T T −1 G(C) = CHn (Iq + HnCHn ) ; K(C) = C − G(C)HnC I Complexity: O(d3). Posterior at t = n Prior+Obs at t = n + 1 Posterior at t = n + 1 forecast assimilate 6 / 51 Random Kalman filters Turbulent barotropic Rossby wave equation: Random Kalman3 Filters are very useful as computationally cheap filtering for forecast and ω = β/k, E = k− k − filteringk models (Majda & Harlim book, 2012). SPEKF, DSS (Branicki & Majda, 2014, 2018) (a) 6 5 4 3 Figure: Spatial pattern for a 2 turbulent system of the 1 externally forced barotropic 0 240 245 250 255 260 265 270 Rossby wave equation with instability through intermittent (b) 2.5 negative damping. Note the 2 coherent wave train that 1.5 emerges during the unstable 1 regime. 0.5 0 −0.5 240 245 250 255 260 265 270 7 / 51 Sampling+Gaussian I Monte Carlo: use samples to represent a distribution: K (1) (K ) 1 X X ;:::; X ∼ p; δ (k) ≈ p: K X k=1 (k) K I Ensemble fXn gk=1 to represent N (X n; Cn) P (k) T Xn (1) (K ) SnSn X n = ; Sn = [∆X ; ··· ; ∆X ]; Cn = : K n n K − 1 T SnSn or through ensemble spread Cn = K −1 (1) (K ) Sn = [Xn − X n; ··· ; Xn − X n]; 8 / 51 Ensemble Kalman filter (EnKF) I Forecast step T (k) (k) (k) Sbn+1Sbn+1 Xb = AnX + Bn + ζ ; Cb + = n+1 n n+1 n 1 K − 1 I EAKF assimilation step, find Sn+1 = An+1Sbn+1 X n+1 = Xb n+1 + G(Cbn+1)(Yn+1 − HnXb n+1); Cn+1 = K(Cbn+1) I Complexity: O(dK 2). Posterior at t = n Prior+Obs at t = n + 1 Posterior at t = n + 1 (k) Xn+1 (k) (k) Xn+1 Xn forecast: M.C. assimilate b 9 / 51 Outline Introduction: Kalman Filters and Finite Ensemble Kalman Filters for Complex Systems Applied Practice, New Phenomena, and Math Theory for Finite Ensemble Kalman Filters Preventing Catastrophic Filter Divergence using Adaptive Additive Inflation State Estimation and Prediction using Clustered Particle Filters High Dimensional Challenges for Analyzing Finite Ensemble Kalman Filters Robustness and Accuracy of Finite Ensemble Kalman Filters in Large Dimensions 10 / 51 Application of Finite Ensemble KFs Application: I Successful weather forecast and oil reservoir management. I Recently been applied to deep neural networks. I K = 50 ensembles can forecast d = 106 dimensional systems. I Extreme saving: 1010 = dK 2 d3 = 1018. I Localization & inflation for successful filtering: needed practically to avoid filter divergence Major contributions (introduced by geoscientists): I G. Evensen, Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics. J Geophys Res Oceans, 1994. I P. L. Houtekamer, and H. L. Mitchell. Data assimilation using an ensemble Kalman filter technique. Mon. Weather Rev., 1998. I Gaspari, G., and S. Cohn. Construction of correlation functions in two and three dimensions. Quart. J. Roy. Meteor. Soc., 1999. Quart. J. Roy. Meteor. Soc., I J. L. Anderson. An ensemble adjustment Kalman filter for data assimilation. Mon. Weather Rev., 2001. I C. H. Bishop, B. J. Etherton, and S. J. Majumdar. Adaptive sampling with the ensemble transform kalman filter. Mon. Weather Rev., 2001. I E. Kalnay. Atmospheric modeling, data assimilation, and predictability. Cambridge university press, 2003. 11 / 51 New Phenomena: Catastrophic Filter Divergence Surprising pathological discovery in EnKF For filtering forced dissipative system with absorbing ball property (such as L-96 model), EnKF can explode to machine infinity in finite time! (Harlim and Majda 2008; Gottwald and Majda, NPG 2013) I Observations are typically sparse and infrequent as in oceanography I Ensemble filtering methods can suffer from catastrophic filter divergence with sparse and infrequent observations and small observation errors I Catastrophic filter divergence drives the filter prediction to machine infinity although the underlying system remains in a bounded set Rigorous math contirbutions Well-posedness of EnKF: D Kelly, KJ Law, and A Stuart. Well-posedness and accuracy of the ensemble Kalman filter in discrete and continuous time. Nonlinearity, 2014. • D Kelly, A J Majda, X Tong, Concrete Ensemble Kalman Filters with Rigorous Catastrophic Filter Divergence, PNAS 112 (34), pp. 10589–10594, 2015. 12 / 51 Rigorous nonlinear stability for finite ensemble Kalman filter (EnKF) (Xin Tong, Majda, Kelly, Nonlinearity 2015) Filter divergence – a potential flaw for EnKF: I Catastrophic filter divergence: the ensemble members diverging to infinity, I Lack of stability: the ensemble members being trapped in locations far from the true process. Finding practical conditions and modifications to rule out filter divergence with rigorous analysis: I Ruling out catastrophic filter divergence by establishing an energy principle for the filter ensemble. I Looking for energy principles inherited by the Kalman filtering scheme. I Looking for modification schemes of EnKF that ensures an energy principle and preserving the original EnKF performance (Xin Tong, Majda, Kelly, Comm. Math. Sci., 2015). I Verifying the nonlinear stability of EnKF through geometric ergodicity. Rigorous example of catastrophic divergence: I For filtering a nonlinear map with absorbing ball property (Kelly, Majda, Xin Tong, PNAS 2015). Outstanding problem: Why and when is there accuracy in mean for M ≤ N? 13 / 51 Outline Introduction: Kalman Filters and Finite Ensemble Kalman Filters for Complex Systems Applied Practice, New Phenomena, and Math Theory for Finite Ensemble Kalman Filters Preventing Catastrophic Filter Divergence using Adaptive Additive Inflation State Estimation and Prediction using Clustered Particle Filters High Dimensional Challenges for Analyzing Finite Ensemble Kalman Filters Robustness and Accuracy of Finite Ensemble Kalman Filters in Large Dimensions 14 / 51 Preventing catastrophic filter divergence using adaptive additive inflation Main reference I Y. Lee, A.J. Majda, and D. Qi, (2016) Preventing catastrophic filter divergence using adaptive additive inflation for baroclinic turbulence, Monthly Weather Review, 145(2), pp. 669-682. Catastrophic Filter Divergence and Adaptive Inflation I D Kelly, A J Majda, X Tong, (2015) Concrete Ensemble Kalman Filters with Rigorous Catastrophic Filter Divergence, PNAS 112 (34), pp. 10589–10594 I X. Tong, A J Majda, D Kelly, (2016) Nonlinear Stability of the Ensemble Kalman Filter with Adaptive Covariance Inflation, Commun. Math. Sci., 14 (5), pp. 1283–1313 15 / 51 Goal : I Demonstrate catastrophic filter divergence I Idealized baroclinic turbulence model for the ocean: two-layer quasigeostrophic equation I Adaptive inflation to prevent catastrophic filter divergence I Comparison between different reduced-order forecast models; importance of accurate parameterization of small scales 16 / 51 Ensemble Kalman Filter and Catastrophic Filter Divergence (review) Finite ensemble Kalman filter: Approximate the prior mean and covariance using an ensemble I Computationally cheap I Low dimensional ensemble state approximation for extremely high dimensional turbulent dynamical systems