ABSTRACT Title of dissertation: MACHINE LEARNING APPROACHES FOR DATA-DRIVEN ANALYSIS AND FORECASTING OF HIGH-DIMENSIONAL CHAOTIC DYNAMICAL SYSTEMS Jaideep Pathak Doctor of Philosophy, 2019 Dissertation directed by: Professor Edward Ott Department of Physics We consider problems in the forecasting of large, complex, spatiotemporal chaotic systems and the possibility that machine learning might be a useful tool for significant improvement of such forecasts. Focusing on weather forecasting as perhaps the most important example of such systems, we note that physics-based weather models have substantial error due to various factors including imperfect modeling of subgrid-scale dynamics and incomplete knowledge of physical processes. In this thesis, we ask if machine learning can potentially correct for such knowledge deficits. First, we demonstrate the effectiveness of using machine learning for model- free prediction of spatiotemporally chaotic systems of arbitrarily large spatial extent and attractor dimension purely from observations of the system's past evolution. We present a parallel scheme with an example implementation based on the reser- voir computing paradigm and demonstrate the scalability of our scheme using the Kuramoto-Sivashinsky equation as an example of a spatiotemporally chaotic system. We then demonstrate the use of machine learning for inferring fundamental properties of dynamical systems, namely the Lyapunov exponents, purely from ob- served data. We obtain results of unprecedented fidelity with our novel technique, making it possible to find the Lyapunov exponents of large systems where previously known techniques have failed. Next, we propose a general method that combines a physics-informed knowledge- based model and a machine learning technique to build a hybrid forecasting scheme. We further extend our hybrid forecasting approach to the difficult case where only partial measurements of the state of the dynamical system are available. For this purpose, we propose a novel technique that combines machine learning with a data assimilation method called an Ensemble Transform Kalman Filter (ETKF). MACHINE LEARNING APPROACHES FOR DATA-DRIVEN ANALYSIS AND FORECASTING OF HIGH-DIMENSIONAL CHAOTIC DYNAMICAL SYSTEMS by Jaideep Satyajit Pathak Dissertation submitted to the Faculty of the Graduate School of the University of Maryland, College Park in partial fulfillment of the requirements for the degree of Doctor of Philosophy 2019 Advisory Committee: Professor Edward Ott, Chair/Advisor Professor Michelle Girvan, Co-Advisor Professor Brian Hunt Professor Rajarshi Roy Professor Thomas Antonsen c Copyright by Jaideep Satyajit Pathak 2019 Dedication Dedicated to my mother, who first taught me math, and my father, who has always given me his unwavering support. ii Acknowledgments I owe a tremendous debt of gratitude to my advisor, Dr. Edward Ott, who has guided and mentored me with a lot of kindness and patience. I have enjoyed working on incredibly interesting and challenging projects under his very able guidance. I consider myself very fortunate to have had the opportunity to work with and learn from such a remarkable person. My co-advisor, Dr. Michelle Girvan has always made sure she was available for scientific and professional advice when I have needed it despite her many respon- sibilities. Her scientific acumen and work ethic is inspirational and I thank her for guiding my research. I would also like to thank Dr. Brian Hunt. Without his helpful insights, brilliant ideas, hard work and expertise, this thesis would not have been possible. Thanks to Dr. Rajarshi Roy, Dr. Bill Dorland, Dr. Tom Antonsen, and Dr. Daniel Lathrop for helpful discussions and feedback during various stages of my research. I have been very fortunate to have been part of a thriving culture of col- laboration within the graduate student body at the University of Maryland. My friends Sarthak Chandra, Sarah Burnett, Zhixin Lu, Alex Wikner and Joe Hart have been unwavering in their help, support and motivation every time I have needed it. Thanks are due to my collaborators Dr. Istvan Szunyogh, Dr. Garrett Katz and Troy Arcomano. I am very grateful to the team maintaining the Deepthought-2 High-Performance iii Computing (HPC) cluster which enabled a large part of my research. I thank the administrative staff at IREAP and the Department of Physics, particularly Josiland Chambers and Jessica Crossby. I owe my most profound thanks to my mother, Dr. Padmini Pathak and my father, Dr. Satyajit Pathak for their tremendous support through every stage of my life. Thanks to my sister, Dr. Priya Pathak for always being there for me. I would like to thank my girlfriend, Sasha Mehan, for all her love, support and encouragement. Thanks to Kaustubh, Ani and Arushi whose friendship has truly meant a lot to me through graduate school. iv Table of Contents Dedication ii Acknowledgements iii Table of Contentsv List of Tables viii List of Figures ix List of Abbreviations xiv 1 Introduction1 1.1 Overview..................................1 1.2 Model-Free Prediction of Large Spatiotemporally Chaotic Dynamical Systems..................................4 1.3 Using Machine Learning for Data-Driven Analysis of Chaotic Dynam- ical Systems................................4 1.4 Model-Assisted Prediction of Chaotic Dynamical Systems.......5 1.5 Reservoir Observers: Model-free Inference of Unmeasured Variables in Chaotic Dynamical Systems......................6 1.6 Reconstruction and Forecasting of Chaotic Dynamical Systems us- ing Partial Measurements, Imperfect Modeling and Machine Learning Assisted Data Assimilation........................7 2 Model-Free Prediction of Large Spatiotemporally Chaotic Systems: A Reser- voir Computing Approach9 2.1 Introduction................................9 2.2 Reservoir Computing Configuration................... 10 3 Using Machine Learning for Data-Driven Analysis of Dynamical Systems 27 3.1 Reservoir Computers, Short Term Prediction and Attractor Climate. 29 3.2 Climate Replication in the Lorenz System............... 33 3.3 Determining a Large Number of Lyapunov Exponents of a High Di- mensional Spatiotemporal Chaotic System from Data......... 42 3.3.1 Homogeneous KS System (µ = 0)................ 49 3.3.2 Inhomogeneous KS System (µ = 0:1).............. 52 v 3.3.3 Effect of Measurement Noise................... 53 3.3.4 Effect of Training Data Length................. 53 3.4 Discussion and Conclusion........................ 55 4 Model-Assisted Prediction of Chaotic Dynamical Systems 57 4.1 Introduction................................ 58 4.2 Prediction Methods............................ 60 4.2.1 Knowledge-Based Model..................... 62 4.2.2 Reservoir-Only Predictor..................... 63 4.2.3 Hybrid Scheme.......................... 65 4.3 Implementation.............................. 67 4.3.1 Reservoir-Only and Hybrid Implementations.......... 68 4.3.2 Training Reusability....................... 71 4.3.3 Assessments of Prediction Methods............... 72 4.4 Lorenz system............................... 74 4.5 Kuramoto-Sivashinsky equations..................... 78 4.6 Conclusions................................ 82 5 Reservoir observers: Model-free inference of unmeasured variables in chaotic system 84 5.1 Introduction................................ 85 5.2 Setup.................................... 86 5.3 Examples................................. 93 5.3.1 Kuramoto-Sivashinsky Equations................ 93 5.4 Conclusions................................ 96 5.5 Acknowledgment............................. 98 6 Reconstruction and Forecasting of Dynamical Systems using Partial Mea- surements, Imperfect Modeling and Machine Learning Assisted Data Assim- ilation 99 6.1 Introduction................................ 99 6.2 Method.................................. 100 6.2.1 Data Assimilation......................... 101 6.2.2 Kalman Filter: Linear Case................... 102 6.2.3 Kalman Filter: Nonlinear Case................. 103 6.3 Machine Learning Assisted Ensemble Transform Kalman Filtering.. 107 6.3.1 Reservoir Computer....................... 107 6.3.2 Algorithm............................. 108 6.3.2.1 Training......................... 109 6.3.2.2 Prediction........................ 110 6.4 Results................................... 113 6.4.0.1 Baseline ETKF Forecast:............... 113 6.4.0.2 ML-ETKF Forecast:.................. 114 6.4.0.3 RMS error........................ 114 6.4.0.4 Valid Time....................... 114 vi 6.4.1 Lorenz 63............................. 115 6.4.1.1 Results: Optimizing the Covariance Inflation.... 116 6.4.2 Kuramoto-Sivashinsky (KS) system............... 118 6.4.2.1 Results: Dependence on Model Error......... 119 Bibliography 121 vii List of Tables 2.1 Largest Lyapunov Exponent (Λmax) and Kaplan-Yorke Dimension (DKY ) of the attractor (λ = 100; µ = 0:01) along with the num- ber of parallel reservoirs (g) and the total number (NT ) of all nodes in the g reservoirs of the parallelized reservoir scheme used....... 24 3.1 Standard reservoir parameters used for a successful climate replica- tion of the Lorenz system (referred to in the text as the R1 reservoir). The R2 reservoir uses the same parameters with a different spectral radius, ρ = 1:45.............................. 34 3.2 Three largest Lyapunov exponents Λ1 ≥ Λ2 ≥ Λ3 for the Lorenz system (Eq. (3.5)), and for the reservoir set up in the configuration of Fig. 3.1(b) for R1 and R2. Since the reservoir system that we employ is a discrete time system, while the Lorenz system