University of Osnabruck¨ Doctoral Dissertation Time Series Analysis informed by Dynamical Systems Theory by Johannes Schumacher from Siegen A thesis submitted in fulfilment of the requirements for the degree of Dr. rer. nat. Neuroinformatics Department Institute of Cognitive Science Faculty of Human Sciences May 16, 2015 Dedicated to the loving memory of Karlhorst Dickel (1928 – 1997) whose calm and thoughtful ways have inspired me to become an analytical thinker. ABSTRACT This thesis investigates time series analysis tools for prediction, as well as detec- tion and characterization of dependencies, informed by dynamical systems theory. Emphasis is placed on the role of delays with respect to information processing in dynamical systems, as well as with respect to their effect in causal interactions between systems. The three main features that characterize this work are, first, the assumption that time series are measurements of complex deterministic systems. As a result, func- tional mappings for statistical models in all methods are justified by concepts from dynamical systems theory. To bridge the gap between dynamical systems theory and data, differential topology is employed in the analysis. Second, the Bayesian paradigm of statistical inference is used to formalize uncertainty by means of a con- sistent theoretical apparatus with axiomatic foundation. Third, the statistical mod- els are strongly informed by modern nonlinear concepts from machine learning and nonparametric modeling approaches, such as Gaussian process theory. Conse- quently, unbiased approximations of the functional mappings implied by the prior system level analysis can be achieved. Applications are considered foremost with respect to computational neuroscience but extend to generic time series measurements. v PUBLICATIONS In the following, the publications that form the main body of this thesis are listed with corresponding chapters. Chapter 5: Johannes Schumacher, Hazem Toutounji and Gordon Pipa (2014). An in- troduction to delay-coupled reservoir computing. Springer Series in Bio- /Neuroinformatics 4, Artificial Neural Networks – Methods and Applica- tions, P. Koprinkova-Hristova et al. (eds.). Springer International Publishing Switzerland 2015, 10.1007/978-3-319-09903-3_4 Johannes Schumacher, Hazem Toutounji and Gordon Pipa (2013). An analyt- ical approach to single-node delay-coupled reservoir computing. Artificial Neural Networks and Machine Learning – ICANN 2013, Lecture Notes in Computer Science Volume 8131, Springer, pp 26-33. Chapter 6: Hazem Toutounji, Johannes Schumacher and Gordon Pipa (2015). Homeo- static plasticity for single node delay-coupled reservoir computing. Neural Computation, 10.1162/NECO_a_00737. Chapter 7: Johannes Schumacher, Robert Haslinger and Gordon Pipa (2012). A statis- tical modeling approach for detecting generalized synchronization. Physical Review E, 85.5(2012):056215. Chapter 8: Johannes Schumacher, Thomas Wunderle, Pascal Fries and Gordon Pipa (2015). A statistical framework to infer delay and direction of information flow from measurements of complex systems. Accepted for publication in Neural Computation (MIT Press Journals). vii NOTHING. A monk asked, “What about it when I don’t understand at all?” The master said, “I don’t understand even more so.” The monk said, “Do you know that or not?” The master said, “I’m not wooden-headed, what don’t I know?” The monk said, “That’s a fine ’not understanding’.” The master clapped his hands and laughed. The Recorded Sayings of Zen Master Joshu [Shi and Green, 1998] ix CONTENTS i introduction 1 1 time series and complex systems 3 1.1 Motivation and Problem Statement3 2 statistical inference 9 2.1 The Bayesian Paradigm 10 2.1.1 The Inference Problem 11 2.1.2 The Dutch Book Approach to Consistency 12 2.1.3 The Decision Problem 17 2.1.4 The Axiomatic Approach of Subjective Probability 19 2.1.5 Decision Theory of Predictive Inference 23 3 dynamical systems, measurements and embeddings 25 3.1 Directed Interaction in Coupled Systems 25 3.2 Embedding Theory 26 4 outline and scientific goals 35 ii publications 39 5 an introduction to delay-coupled reservoir computing 41 5.1 Abstract 41 5.2 Introduction to Reservoir Computation 41 5.3 Single Node Delay-Coupled Reservoirs 42 5.3.1 Computation via Delayed Feedback 42 5.3.2 Retarded Functional Differential Equations 46 5.3.3 Approximate virtual node equations 48 5.4 Implementation and Performance of the DCR 51 5.4.1 Trajectory Comparison 52 5.4.2 NARMA-10 52 5.4.3 5-Bit Parity 53 5.4.4 Large Setups 53 5.4.5 Application to Experimental Data 55 5.5 Discussion 60 5.6 Appendix 61 6 homeostatic plasticity for delay-coupled reservoirs 65 6.1 Abstract 65 6.2 Introduction 65 6.3 Model 67 6.3.1 Single Node Delay-Coupled Reservoir 67 6.3.2 The DCR as a Virtual Network 69 6.4 Plasticity 71 6.4.1 Sensitivity Maximization 72 6.4.2 Homeostatic Plasticity 73 6.5 Computational Performance 74 6.5.1 Memory Capacity 75 xi xii contents 6.5.2 Nonlinear Spatiotemporal Computations 75 6.6 Discussion: Effects of Plasticity 77 6.6.1 Entropy 77 6.6.2 Virtual Network Topology 78 6.6.3 Homeostatic Regulation Level 79 6.7 Commentary on Physical Realizability 81 6.8 Conclusion 82 6.9 Appendix A: Solving and Simulating the DCR 83 6.10 Appendix B: Constraint Satisfaction 84 7 detecting generalized synchronization 87 7.1 Abstract 87 7.2 Introduction 87 7.3 Rössler-Lorenz system 90 7.4 Mackey-Glass nodes 92 7.5 Coupled Rössler systems 94 7.6 Phase Synchrony 94 7.7 Local field potentials in macaque visual cortex 98 7.8 Conclusion 99 8 d2 if 101 8.1 Abstract 101 8.2 Introduction 101 8.3 Methods 103 8.3.1 Embedding Theory 103 8.3.2 Statistical Model 107 8.3.3 Estimation of Interaction Delays 112 8.3.4 Experimental Procedures 117 8.4 Results 117 8.4.1 Logistic Maps 117 8.4.2 Lorenz-Rössler System 119 8.4.3 Rössler-Lorenz System 121 8.4.4 Mackey-Glass System 123 8.4.5 Local Field Potentials of Cat Visual Areas 124 8.5 Discussion 128 8.6 Supporting Information 132 8.6.1 Embedding Theory 132 8.6.2 Discrete Volterra series operator 137 8.6.3 Treatment of stochastic driver input in the statistical model under the Bayesian paradigm 140 iii discussion 143 9 discussion 145 9.1 Prediction 145 9.2 Detection and Characterization of Dependencies 148 iv appendix 151 a the savage representation theorem 153 b normal form of the predictive inference decision problem 155 c reservoir legerdemain 157 d a remark on granger causality 163 bibliography 165 LISTOFFIGURES Figure 1 Rössler system driving a Lorenz system 26 Figure 2 Embedding a one-dimensional manifold in two or three- dimensional Euclidean space 27 Figure 3 Equivalence of a dynamical system and its delay embed- ded counterpart 30 Figure 4 Exemplary trajectory of Mackey-Glass system during one t-cycle 44 Figure 5 Schematic illustration of a DCR 45 Figure 6 Illustration of a temporal weight matrix for a DCR 51 Figure 7 Comparison between analytical approximation and nu- merical solution for an input-driven Mackey-Glass sys- tem 52 Figure 8 Comparison on nonlinear tasks between analytical approx- imation and numerical solution for an input-driven Mackey- Glass system 54 Figure 9 Normalized data points of the Santa Fe data set 59 Figure 10 Squared correlation coefficient of leave-one-out cross-validated prediction with parametrically resampled Santa Fe train- ing data sets 60 Figure 11 Comparing classical and single node delay-coupled reser- voir computing architectures 68 Figure 12 DCR activity superimposed on the corresponding mask 70 Figure 13 Virtual weight matrix of a DCR 71 Figure 14 Memory capacity before and after plasticity 75 Figure 15 Spatiotemporal computational power before and after plas- ticity 76 Figure 16 Average improvement for different values of the regulat- ing parameter 79 Figure 17 Performance of 1000 NARMA-10 trials for regulating pa- rameter r values between 0 and 2 80 Figure 18 Identification of nonlinear interaction in a coupled Rössler- Lorenz system 91 Figure 19 Performance on generalized synchronized Mackey-Glass delay rings 93 Figure 20 Identification of nonlinear interaction between coupled Rössler systems 95 Figure 21 Identification of interaction between unidirectionally cou- pled Rössler systems in 4:1 phase synchronization 97 xiii xiv List of Figures Figure 22 Two macaque V1 LFP recordings x and y recorded from electrodes with different retinotopy 99 Figure 23 Functional reconstruction mapping 106 Figure 24 Delay Estimation 113 Figure 25 Delay-coupled logistic maps 118 Figure 26 Delay-coupled Lorenz-Rössler System 120 Figure 27 Delay-coupled Rössler-Lorenz System in Generalized Syn- chronization 122 Figure 28 Delay-coupled Mackey-Glass Oscillators 124 Figure 29 Exemplary LFP Time Series from Cat Visual Cortex Area 17 125 Figure 30 Connectivity Diagram of the LFP Recording Sites 126 Figure 31 Reconstruction Error Graphs for Cat LFPs 129 Figure 32 Reservoir Legerdemain trajectory and mask 161 LISTOFTABLES Table 1 DCR results on the Santa Fe data set 57 xv ACRONYMS DCR Delay-coupled reservoir RC Reservoir computing REG Reconstruction error graph GS Generalized synchronization DDE Delay differential equation LFP Local field potential xvi Part I INTRODUCTION This part motivates the general problem, states the scientific goals and provides an introduction to the theory of statistical inference, as well as to the reconstruction of dynamical systems from observed measure- ments. TIMESERIESANDCOMPLEXSYSTEMS 1 This thesis documents methodological research in the area of time series analy- sis, with applications in neuroscience. The objectives are prediction, as well as detection and characterization of dependencies, with an emphasis on delayed inter- actions. Three main features characterize this work. First, it is assumed that time series are measurements of complex deterministic systems.