From Nonlinearity to Causality: Statistical testing and inference of physical mechanisms underlying complex dynamics Milan Palu·s Institute of Computer Science, Academy of Sciences of the Czech Republic, Pod vod¶arenskou v·e·z¶³ 2, 182 07 Prague 8, Czech Republic [email protected]; http://www.cs.cas.cz/mp (Dated: January 12, 2008) Principles and applications of statistical testing as a tool for inference of underlying mechanisms from experimental time series are discussed. The computational realizations of the test null hypoth- esis known as the surrogate data are introduced within the context of discerning nonlinear dynamics from noise, and discussed in examples of testing for nonlinearity in atmospheric dynamics, solar cy- cle and brain signals. The concept is further generalized for detection of directional interactions, or causality in bivariate time series. Index 1. Introduction 5.3. But the atmosphere is not linear ... 2. Searching order in chaos 6. Nonlinearity in the sunspot numbers 2.1. Low-dimensional chaotic dynamics: detection 6.1. Amplitude-frequency correlation in nonlinear and characterization oscillators 2.2. Ubiquity of chaos challenged: surrogate data 6.2. Amplitude-frequency correlation in the testing sunspot cycle 3. Statistical testing 7. Statistical testing in the process of scienti¯c 3.1. Signi¯cance testing and hypothesis testing discovery 3.2. Null distribution of discriminating statistic 8. Some remarks on nonlinearity in the human EEG 3.3. The story of sunspots and senators as an 9. Inference of directional interactions or causality illustrative example in complex systems 4. Testing for nonlinearity 9.1. Asymmetry in coupling: test systems 4.1. Formulation of the problem 9.2. Asymmetry measures 4.2. The null hypothesis and the surrogate data 9.3. Asymmetric measures and causality 4.3. Information-theoretic functionals as 9.4. Inference of causality with the conditional discriminating statistics mutual information 4.4. The null hypothesis of nonlinearity tests and 9.5. Conditional mutual information as a its negations discriminating statistic 5. Testing for nonlinearity in atmospheric dynamics 9.6. Testing the direction of the cardiorespiratory 5.1. Lack of nonlinearity in geopotential heights interaction and in the temperature 9.7. Towards reliable inference of causality 5.2. Pressure data: What kind of nonlinearity do 10. Conclusion we observe? This is a preprint of an article whose ¯nal and de¯nitive form has been published in Contemporary Physics 48(6) (2007) 307 { 348. °c Taylor & Francis; available online at http://www.informaworld.com/smpp/content~content=a792784686 1. INTRODUCTION namics of various origin, thus it is not surprising that re- view papers in this journal discuss complex processes not only in traditional physical areas such as, for instance, One of the great challenges of contemporary physics the physics of atmosphere and climate [1, 2], but also and contemporary science in general is understanding of touch complexity in biological systems in general [3], in emergent phenomena in complex systems. Physicists try proteins [4], DNA [5, 6], or in the dynamics of the human to apply their ideas and tools in studies of complex dy- cardiovascular system [7]. Processing and evaluation of 2 complex biomedical signals and images is also a problem development in the area of nonlinear dynamics and the attracting the attention of physicists and the Contempo- theory of deterministic chaos, oriented to processing of rary Physics [8, 9]. Systems of many interacting compo- experimental data, in which the need of the statistical nents can be found not only in the traditional areas of sta- testing naturally emerged. tistical physics, but also at various levels of organization A description and explanation of the ways of physical of living tissues and organisms, from molecular structures thinking that have led to complicated theories and their to neuronal networks, from proteins, cells, to the human elegant formulations in the language of mathematics, brain, and further to interactions of human individuals instead of lecturing the chain of axioms, de¯nitions, or groups organizing themselves in social networks, work- theorems and corollaries, was the way how Professor ing and doing business in the globalized economy. From Jozef Kvasnica (1930{1992) used to give his memorable some contemporary physicists we can hear the opinion lectures on quantum mechanics and quantum ¯eld that economics might be `the next physical science' [10]. theory at the Faculty of Mathematics and Physics of Others assert that the contemplation and resolution of the Charles University in Prague. The author would questions at the interfaces of biology, mathematics, and like to dedicate this article to memory of Professor physics promise to lead to a greater understanding of the Jozef Kvasnica, head of department of mathematical natural world and to open new avenues for physics [11]. physics (1976{1986), a theoretical physicist and a great In the scienti¯c areas close to biology or to social educator [20] who was able to maintain a relatively sciences, the connection between experimental data liberal academic atmosphere at his department in 1980s and a theory is usually less straightforward than in Czechoslovakia. traditional physical sciences. The decision whether data support a proposed theory or a hypothesis usually In experimental studies of complex systems, usually it cannot be made just using simple methods of data is not possible to characterize the state of such systems evaluation and presentation. Sophisticated statistical by a single measurement or a set of measurements. In approaches should be used in order to distinguish many cases, however, it is possible to follow dynamics repetitive patterns from random e®ects and then to or evolution of a system by recording of some observ- infer possible physical mechanisms underlying an ob- able quantity (or a set quantities, or the same quantity served complex phenomenon. The traditional ¯eld of in di®erent spatial locations) by registering its values in mathematical statistics provides both a language and successive instants of time t1; t2; : : : ; tN . The collection a toolbox for dealing with the questions of inference of measurements fs(t)g is called a time series. The time that emerge in the search for `order in chaos', or, more series are the kind of data we will consider in this article. speci¯cally, in attempts to discern noise from complex Traditional statistical approaches to time series analysis dynamics generated by possibly deterministic and are based on the linear theory [21], although some exten- probably nonlinear physical (chemical, biological, social) sions counting for speci¯c types of nonlinearity have been processes. A formal framework that can help in asking developed [22, 23]. An independent approach to analysis and answering questions about possible mechanisms of nonlinear time series emerged in physics-related areas underlying experimental data has been developed in of nonlinear dynamics and the theory of deterministic the ¯eld of statistical testing. The statistical testing chaos. We will introduce some ideas of this approach is widely used in many scienti¯c ¯elds including some below. Inspirations from the chaos theory have been ex- areas of physics (see, e.g., [12{14, 16{18]), however, in plored in the statistical context of time series analysis as many physical areas it is underestimated or ignored. well [23, 24]. The underestimation occurs in two ways: A part of Having recorded a time series fs(t)g of some observ- physicists use it, however, without a deep understanding able quantity reflecting complex behaviour of some sys- of the principles and without serious mastering of the tem or process, one could ask questions about the nature available tools. This underestimation of the necessary of that system or process. Is it random, or is it deter- expertise level leads to many incorrect results and/or ministic and predictable? If it is not random, what are wrong interpretations of the statistical tests, giving the underlying mechanisms? Can we infer them from thus arguments to the proponents of the other kind of the data and express them in a form of a mathemati- underestimation { the belief that the statistical testing cal model with a predictive power? These questions are as a scienti¯c tool is useless (see, e.g., [19] and references quite general questions of science. We will describe how therein). Therefore it is desirable to discuss realistic researchers in the ¯eld of nonlinear dynamics and chaos possibilities of the statistical testing in a form accessible tried to cope with such questions and how the statisti- to a broad physical readership. cal testing has been introduced into the `search for order in chaos'. After an introduction of necessary concepts This article is an attempt to introduce the statisti- of statistical testing, namely the computational Monte cal testing to a general physical community. Instead of Carlo approaches based on the surrogate data techniques, starting with formal de¯nitions and/or rephrasing mate- we will discuss the problem of detection of nonlinearity rials from some of the many textbook of mathematical in time series. Starting with some theoretical considera- or applied statistics, we will briefly review the scienti¯c tions, we will continue our review with particular exam- 3 ples of testing for nonlinearity in atmospheric dynamics, solar cycle and in brain signals. From statistical infer- 1 ences about dynamics of a single process we will move n to a study of interactions between systems, in particular, x 0.5 we will demonstrate a statistical approach for detection of directional interactions, or causality from bivariate time 0 series. 1 n 2. SEARCHING ORDER IN CHAOS x 0 In 1991 Shu-yu Zhang published the Bibliography on Chaos [25] comprising 7167 titles from which more than 2700 papers contained the words `chaos', `chaotic', or 1 `strange attractor' in their title. n 0 The editor of the New York Times Book Review chose x as one of the best books of the year 1987 James Gleick's -1 Chaos: Making a New Science [26].