UNIVERSITY OF THE AEGEAN PHDTHESIS Bayesian Methods & Estimation of Nonlinear Dynamical Systems Author: Supervisor: Konstantinos Kaloudis Spyridon J. Hatjispyros A thesis submitted in fulfillment of the requirements for the degree of Doctor of Philosophy in the Department of Statistics and Actuarial-Financial Mathematics Karlovassi, Greece April 2019 iii Bayesian Methods & Estimation of Nonlinear Dynamical Systems Konstantinos Kaloudis PHDTHESIS COMMITTEE Spyridon J. Hatjispyros (SUPERVISOR–Participant of the 3 and 7 member committee) Assoc. Professor, Dept. of Statistics and Actuarial-Financial Mathematics, University of the Aegean. Theodoros Nikoleris (Participant of the 3 and 7 member committee) Assistant Professor, Dept. of Economics, National and Kapodistrian University of Athens. Athanasios N. Yannacopoulos (Participant of the 3 and 7 member committee) Professor, Dept. of Statistics, Athens University of Economics and Business. Nikolaos Halidias (Participant of the 7 member committee) Assoc. Professor, Dept. of Statistics and Actuarial-Financial Mathematics, University of the Aegean. Theodoros Karakasidis (Participant of the 7 member committee) Professor, Dept. of Civil Engineering, University of Thessaly. John Tsimikas (Participant of the 7 member committee) Assoc. Professor, Dept. of Statistics and Actuarial-Financial Mathematics, University of the Aegean. Stavros Vakeroudis (Participant of the 7 member committee) Assistant Professor, Dept. of Statistics and Actuarial-Financial Mathematics, University of the Aegean. April 2019 I, KONSTANTINOS KALOUDIS, DECLARE THAT THE RESEARCH PRESENTED IN THIS THESIS IS MY OWN UNLESS OTHERWISE STATED. THIS THESIS HAS BEEN PREPARED USING THE PROGRAM LATEX AND THE STYLE MASTERDOC- TORALTHESIS WITH SOME MODIFICATIONS IN A TEXLIVE DISTRIBUTION.WRITING WAS MADE USING THE PROGRAM “TEXSTUDIO” ON THE LINUX FEDORA 26 OPERATING SYSTEM.THE PROGRAMMING LANGUAGE USED FOR THE DEVELOPMENT OF THE PRESENTED METHODS IS Julia.THE FIGURES WERE CREATED USING THE PROGRAMS R AND Matlab. v This thesis contains original work published or submitted for publication to international jour- nals. The following papers are included: 1. Christos Merkatas, Konstantinos Kaloudis, and Spyridon J Hatjispyros. A Bayesian non- parametric approach to reconstruction and prediction of random dynamical systems. Chaos: An Interdisciplinary Journal of Nonlinear Science, 27(6):063116, 2017. 2. Konstantinos Kaloudis, Spyridon J. Hatjispyros. A Bayesian nonparametric approach to dynamical noise reduction. Chaos: An Interdisciplinary Journal of Nonlinear Science, 28(6):063110, 2018. 3. Spyridon J. Hatjispyros, Konstantinos Kaloudis. On the stochastic approximation of the global stable manifold. Submitted for publication, 2019. This thesis has been funded by the “YPATIA” Scholarship Program for Doctoral Studies, of the University of the Aegean. vii Acknowledgements I want to express my warmest thanks to my supervisor Spyridon Hatjispyros, for his motiva- tion, guidance and support. Spyros has not only been my supervisor, but also a mentor and a friend. Thank you for everything. I express my gratitude to the University of the Aegean, for providing me with the “YPATIA” Scholarship Program for Doctoral Studies. I would also like to thank all the members of my committee, for their valuable comments on this work. It is my pleasure to thank Ass. Prof. Eleftherios Tachtsis and Ass. Prof. Stelios Xanthopoulos for their support and all the discussions we had at the “foyer” of the Provatari building. I am also grateful to my (academic) brother Christos M. and my beloved friends Alice S., Artemis B., Antonis M., Dimitris P., Dimitris Gk., Elena K., Kostas T., Mihalis S., Panos D., Pavlos D., Tassos Gk. for their encouragement and for always being there for me. Special thanks to Sofia for her patience, love and support. Finally, words are not enough to express my gratitude to my family, for their constant and unconditional support. This thesis is dedicated to them. ix Abstract This thesis is concerned with the interplay between Bayesian Statistics and Nonlinear Dynam- ical Systems. Specifically, the main goal of the thesis is the development of new Markov Chain Monte Carlo (MCMC) methods that have applications in the general field of nonlinear dynam- ics. The motivation for this approach is the decomposition of the modeling procedure into two interacting parts: the deterministic part and the stochastic noise process. Using this kind of modeling, we are able to capture a wide variety of phenomena, utilizing the complex behavior of the nonlinear part and the new characteristics emerging from the interaction with the noise process. The proposed methods are nonparametric, based on the Geometric Stick-Breaking process as a prior over the space of probability measures. An important aspect of this work is the relaxation of a very common assumption in the literature: the normality of the noise distribution. In Chapter 1 we present the basic notions and results of the Bayesian parametric and non- parametric framework, that are essential for the understanding of the methods developed in this thesis. We discuss the concept of exchangeability and the Theorem of de Finetti. Follow- ing, we introduce the notion of a Bayesian nonparametric model and define the most popular Bayesian nonparametric prior, the Dirichlet Process. We continue with the definition of the main Bayesian nonparametric prior for the scope of this thesis, the Geometric Stick-Breaking process and discuss the properties of Bayesian nonparametric Mixture models. Finally, the main MCMC sampling methods are presented and discussed. In Chapter 2, which is also introductory, we give the definitions and basic properties of dy- namical systems, both in the deterministic and the stochastic case. The concept of chaos is rigorously analyzed, along with the properties of chaotic systems, homoclinic tangencies and invariant manifolds. We continue with the presentation of the most popular in the literature noise-induced effects, as well as with the effects of dynamical noise on non-hyperbolic non- linear maps. We conclude with an explanation of the inefficiency of parametric reconstruction models under a non-Gaussian noise process. In Chapter 3 we propose a Bayesian nonparametric framework, the Geometric Stick Breaking Reconstruction (GSBR) model, suitable for the full reconstruction and prediction of dynamically- noisy corrupted time series, when the additive noise may exhibit significant departures from normality. In particular, any 0-mean symmetric density can be recovered, even in cases where x the size of the observed time series is small, hence statistical inference for the system is im- proved and reliable. With the proposed model, we have also shown that the associated quasi- invariant measure of the underlying random dynamical system, appears naturally as a predic- tion barrier, similarly as the invariant measure appears as the prediction barrier in the deter- ministic case. We have used the Geometric Stick Breaking process as a prior over the unknown noise density, showing that it yields almost indistinguishable results from the more commonly used, but computationally more expensive, Dirichlet Process prior. The GSBR model is also generalized in order to include arbitrary number of finite lag terms and finally extended in the multivariate case, where the noise process is modeled as an infinite mixture of multivariate Normal kernels with unknown precision matrices, using Wishart distributions. In Chapter 4, the thesis proceeds with the proposal of a new Bayesian nonparametric method, the Dynamic Noise Reduction Replicator (DNRR) model, suitable for noise reduction over a given chaotic time series, subjected to the effects of (the perhaps non-Gaussian) additive dy- namical noise. The DNRR model provides a highly accurate reconstruction of the dynamical equations, while in parallel it replicates the dynamics under reduced noise level dynamical perturbations. The advantages of this method are that the estimated noise-reduced orbit has approximately the same estimated deterministic part, while it evolves in a neighborhood of the original noisy orbit. The two orbits remain close even in the regions of the noise-induced pro- longations of the attractor, or in cases of perturbed multistable maps exhibiting noise-induced jumps. We were also able to relate the regions of primary homoclinic tangencies of the associ- ated deterministic system, with regions of persistent high determinism deviations. Further, in relating the random dynamical systems with their associated deterministic parts, in Chapter 5 we present an extention of the GSBR sampler, in order to provide a MCMC-based stochastic approximation of the global stable manifold. Specifically, we have introduced the Backward GSBR (BGSBR) model, in order to estimate past unobserved observations, namely performing prediction in reversed time. We have emphasized on the support of the poste- rior marginals of the unknown initial conditions, demonstrating that the union of the supports contain the associated deterministic stable manifold of the attractor. The BGSBR sampler can be applied multiple times over proper subsets of the noisy observations, each time generating posterior samples for the various initial conditions. Then the global stable manifold of the asso- ciated deterministic map can be stochastically approximated as the union of the supports of the posterior marginal distributions. The method is parsimonious and efficient both in invertible and non-invertible