Santa Fe Institute Working Paper 11-08-XXX arxiv.org:1008.XXXX [nlin.CD] Toward a Physics of Pattern Focus Issue on \Randomness, Structure, and Causality: Measures of Complexity from Theory to Applications" James P. Crutchfield1, 2, ∗ and Jon Machta3, 2, y 1Complexity Sciences Center & Physics Department, University of California at Davis, One Shields Avenue, Davis, CA 95616 2Santa Fe Institute, 1399 Hyde Park Road, Santa Fe, NM 87501 3Physics Department University of Massachusetts, Amherst, MA 01003 (Dated: August 9, 2011) We introduce the contributions to this Focus Issue and describe their origin in a recent Santa Fe Institute workshop. Keywords: measures of complexity, computation theory, information theory, dynamical systems PACS numbers: 05.45.-a 89.75.Kd 89.70.+c 05.45.Tp The workshop on \Randomness, Structure, and The workshop's goal was to bring together researchers Causality: Measures of Complexity from Theory from a variety of fields to discuss structural and dynami- to Applications" was held at the Santa Fe Insti- cal measures of complexity appropriate for their field and tute in January 2011. This Focus Issue records the commonality between these measures. Some of the work that was presented at and stimulated by questions addressed were: this workshop. The contributions treat both fun- damental questions in the theory of complex sys- 1. Are there fundamental measures of complexity that tems and information theory and their applica- can be applied across disciplines or are measures of tion to a wide range of disciplines including biol- complexity necessarily tied to particular domains? 2. How is a system's causal organization, reflected in ogy, linguistics, computation, and dynamical sys- models of its dynamics, related to its complexity? tems. 3. Are there universal mechanisms at work that lead to increases in complexity or does complexity arise for qualitatively different reasons in different set- I. INTRODUCTION tings? 4. Can we reach agreement on general properties that all measures of complexity must have? In 1989, the Santa Fe Institute (SFI) hosted a 5. How would the scientific community benefit from a workshop|Complexity, Entropy, and the Physics of In- consensus on the properties that measures of com- formation|on fundamental definitions of complexity. plexity should possess? This workshop and the proceedings that resulted [1] stim- 6. Some proposed measures of complexity are difficult ulated a great deal of thinking about how to define com- to effectively compute. Is this problem inherent in plexity. In many ways|some direct, many indirect|the measures of complexity generally or an indication foundational theme of the workshop colored much of the of an unsuitable measure? evolution of complex systems science since then. Com- plex systems science has considerably matured as a field The Santa Fe Institute hosted 20 workshop partici- in the intervening decades. As a result, it struck us that pants in mid-January 2011. It turned out to be a stimu- it was time to revisit fundamental aspects of this nascent lating and highly interdisciplinary group with represen- field in a workshop. Partly, this was to take stock; but tation from physics, biology, computer science, social sci- it was also to ask what innovations are needed for the ence, and mathematics. An important goal was to under- coming decades, as complex systems ideas continue to stand the successes and difficulties in deploying complex- extend their influence in the sciences, engineering, and ity measures in practice. And so, participants came from humanities. both theory and experiment, with a particular emphasis on those who have constructively bridged the two. Since the 1989 SFI workshop, a number of distinct strands have developed in the effort to define and mea- ∗Electronic address: [email protected] sure complexity. Several of the well developed strands yElectronic address: [email protected] are based on: 2 • Predictive information and excess entropy [2{7], tional dynamics of single molecules [24] through model- • Statistical complexity and causal structure [8{10], ing subgrid structure in turbulent fluid flows [25] and new • Logical depth and computational complexity [11{ visualization methods for emergent flow patterns [26] to 15], and monitoring market efficiency [27] and the organization of • Effective complexity [16, 17]. animal social structure [28]. In this light, the intention While these measures are broadly based on information was to find relations between the practically motivated theory or the theory of computation, the full set of con- measures and the more general and fundamentally moti- nections and contrasts between them has not been suffi- vated measures. Can the practically motivated measures ciently well fleshed out. Some have sought to clarify the be improved by an appreciation of fundamental princi- relationship among these measures [7, 17{21] and one ples? Can fundamental definitions be sharpened by con- goal of the workshop was to foster this kind of compar- sidering how they interact with real-world data? ative work by bringing together researchers developing The workshop's goal was to re-ignite the efforts that various measures. began with Complexity, Entropy, and the Physics of In- A number of lessons devolved from these early efforts, formation workshop. A new level of rigor, in concepts though. Several come immediately to mind: and in analysis, is now apparent in how statistical me- chanics, information theory, and computation theory can 1. There are two quite different but complementary be applied to complex systems. The meteoric rise of both meanings of the term \complexity". The term is computer power and machine learning has led to new al- used both to indicate randomness and structure. gorithms that address many of the computational diffi- As workshop discussions repeatedly demonstrated, culties in managing data from complex systems and in these are diametrically opposed concepts. Conflat- estimating various complexity measures. Given progress ing them has led to much confusion. on all these fronts, the time was ripe to develop a much closer connection between fundamental theory and appli- 2. Moreover, a correct understanding of complexity cations in many areas of complex systems science. reveals that both are required elements of complex system. In particular, we now have a large number of cases demonstrating that structural complexity arises from the dynamical interplay of tendencies to II. OVERVIEW OF CONTRIBUTIONS TO THE order and tendencies to randomness. Organization FOCUS ISSUE in critical phenomena arising at continuous phase transitions is only one example|and a static ex- The Focus Issue reflects the work of a highly interdis- ample at that. Much of the work described in the ciplinary group of contributors representing engineering, Focus Issue addresses the interplay of randomness physics, chemistry, biology, neuroscience, cognition, com- and structure in complex systems. puter science, and mathematics. An important goal was to understand the successes and difficulties in deploying 3. Even if one concentrates only on detecting and these concepts in practice. Here is a brief preview of measuring randomness, one must know how the un- those contributions. derlying system is organized. The flip side is that A Geometric Approach to Complexity by Nihat Ay: missing underlying structure leads one to conclude Ay develops a thorough-going mathematical treatment that a process is more random than it really is. of the complexity question, from the point of view of 4. Elaborating on the original concept of extracting the relationship of whole versus the parts. This builds a \Geometry from a Times Series" [22], we now know bridge to the differential geometric approach to statisti- that processes do tell us how they are best repre- cal inference pioneered by Amari in artificial neural net- sented. This, in turn, lends credence to the original works. New here, Ay connects the approach to Markov call for “artificial science" [8]|a science that auto- processes, going beyond merely thinking of in terms of matically builds theories of natural systems. \nodes on a graph". Partial Information Decomposition as a Spatiotempo- The lessons echoed throughout the workshop and can be ral Filter by Benjamin Flecker, Wesley Alford, John seen operating in the Focus Issue contributions. Beggs, Paul Williams, and Randall Beer: The authors A second motivation for the workshop was to bring to- apply their new method of partial information decom- gether workers interested in foundational questions|who position to analyze the spacetime information dynam- were mainly from the physics, mathematics, and com- ics generated by elementary cellular automata. This re- puter science communities|with complex systems scien- frames recent efforts to develop a theory of local informa- tists in experimental, data-driven fields who have devel- tion storage, transfer, and modification. They show that oped quantitative measures of complexity, organization, prior approaches can be reinterpreted and recast into a and emergence that are useful in their fields. The range clearer form using partial information decomposition. In of data-driven fields using complexity measures is impres- particular, one that does not require arbitrarily selected sively broad: ranging from molecular excitation dynam- threshold for detection. Importantly, the decomposition ics [23] and spectroscopic observations of the conforma- suggests a new level of semantic analysis of what would 3 otherwise be mere syntactic