Causal Architecture, Complexity and Self-Organization in Time Series and Cellular Automata Cosma Rohilla Shalizi 4 May 2001 i To Kris, for being perfect ii Abstract All self-respecting nonlinear scientists know self-organization when they see it: except when we disagree. For this reason, if no other, it is important to put some mathematical spine into our floppy intuitive notion of self-organization. Only a few measures of self-organization have been proposed; none can be adopted in good intellectual conscience. To find a decent formalization of self-organization, we need to pin down what we mean by organization. The best answer is that the organization of a process is its causal architecture | its internal, possibly hidden, causal states and their interconnections. Computational mechanics is a method for inferring causal architecture | represented by a mathematical object called the -machine | from observed behavior. The -machine captures all patterns in the process which have any predictive power, so computational mechanics is also a method for pattern discovery. In this work, I develop computational mechanics for four increasingly sophisticated types of process | memoryless transducers, time series, transducers with memory, and cellular automata. In each case I prove the optimality and uniqueness of the -machine's representation of the causal architecture, and give reliable algorithms for pattern discovery. The -machine is the organization of the process, or at least of the part of it which is relevant to our measurements. It leads to a natural measure of the statistical complexity of processes, namely the amount of information needed to specify the state of the -machine. Self-organization is a self-generated increase in statistical complexity. This fulfills various hunches which have been advanced in the literature, seems to accord with people's intuitions, and is both mathematically precise and operational. iii Contents Abstract ii List of Figures xiii Notes on the Text and on Sources xiv 1 Introduction 1 1.1 Self-Organization . 1 1.2 Formalizing an Intuitive Notion . 3 1.3 The Strategy . 4 1.4 A Summary . 4 1.5 Historical Sketch . 6 1.5.1 Origins of the Concept . 6 1.5.2 Uses of the Concept . 7 1.5.3 History of Pattern Discovery and Computational Mechanics . 7 2 Measuring Pattern, Complexity & Organization 9 2.1 Organization . 9 2.2 Complexity Measures, or, the History of One-Humped Curves . 11 2.3 Patterns . 12 2.3.1 Algebraic Patterns . 12 2.3.2 Turing Mechanics: Patterns and Effective Procedures . 13 2.3.3 Patterns with Error . 14 2.3.4 Causation . 15 2.3.5 Synopsis of Pattern . 15 3 Memoryless Transducers 16 3.1 The Setup . 16 3.2 Effective States . 16 Definition 1 Effective States (Memoryless Transduction) . 16 Definition 2 Predictive Power of Effective States (Memoryless) . 17 Lemma 1 The Old Country Lemma (Memoryless) . 17 Definition 3 Prescient States (Memoryless) . 17 Lemma 2 Prescient States are Sufficient Statistics . 17 3.2.1 Minimality and Prediction . 17 Definition 4 Statistical Complexity of State Classes . 17 3.3 Causal States . 18 Definition 5 Causal State (Memoryless) . 18 3.3.1 Homogeneity . 18 Definition 6 Strict Homogeneity . 18 iv Definition 7 Weak Homogeneity . 18 Lemma 3 Strict Homogeneity of Causal States . 18 3.3.2 Optimalities and Uniqueness . 18 Theorem 1 Prescience and Sufficiency of Causal States (Memoryless) . 19 Lemma 4 Refinement Lemma (Memoryless) . 19 Theorem 2 Minimality of Causal States (Memoryless) . 20 Corollary 1 Causal States Are Minimal Sufficient . 20 Theorem 3 Uniqueness of Causal States (Memoryless) . 20 Definition 8 Statistical Complexity of Memoryless Transduction . 20 Theorem 4 Control Theorem (Memoryless) . 21 3.4 Other Approaches to Memoryless Transduction . 21 3.4.1 Graphical Models . 21 3.4.2 The Information-Bottleneck Method . 22 3.4.3 The Statistical Relevance Basis . 22 3.5 Summary . 22 4 Time Series 24 4.1 Paddling Around in Occam's Pool . 24 4.1.1 Processes . 24 Definition 9 Processes . 24 Definition 10 Stationarity . 25 4.1.2 The Pool . 25 Definition 11 Effective States (Time Series) . 25 4.1.3 Patterns in Ensembles . 26 Definition 12 Predictive Power of Effective States (Time Series) . 26 4.1.4 The Lessons of History . 27 Lemma 5 Old Country Lemma (Time Series) . 27 4.2 The Causal States . 27 Definition 13 Causal States of a Process . 28 4.2.1 Morphs . 28 Lemma 6 Independence of Past and Future Given Causal State . 29 Lemma 7 Strict Homogeneity of Causal States (Time Series) . 29 4.2.2 Causal State-to-State Transitions . 29 Definition 14 Causal Transitions . 29 Lemma 8 Transition Probabilities . 30 4.2.3 -Machines . 30 Definition 15 -Machine . 30 Lemma 9 -Machines Are Monoids . 30 Lemma 10 -Machines Are Deterministic . 31 Lemma 11 -Machines Are Markovian . 32 Definition 16 -Machine Reconstruction . 32 4.3 Optimalities and Uniqueness . 32 Theorem 5 Prescience of Causal States (Time Series) . 33 Corollary 2 Causal States Are Sufficient Statistics (Time Series) . 33 Definition 17 Prescient Rivals . 34 Theorem 6 Sufficiency and Determinism Imply Prescience . 34 Lemma 12 Refinement Lemma (Time Series) . 34 Theorem 7 Minimality of Causal States (Time Series) . 35 Corollary 3 Causal States Are Minimal Sufficient Statistics (Time Series) . 35 Definition 18 Statistical Complexity of a Process . 35 Theorem 8 Uniqueness of Causal States (Time Series) . 36 v Theorem 9 Minimal Internal Stochasticity of -Machines . 36 4.4 Bounds . 37 Definition 19 Excess Entropy . 37 Theorem 10 The Bounds of Excess . 37 Corollary 4 . 38 Lemma 13 Conditioning Does Not Affect Entropy Rate . 38 Theorem 11 Control Theorem (Time Series) . 38 4.5 The Physical Meaning of Causal States . 39 5 A Reconstruction Algorithm 42 5.1 Reconstructing States by Merging . 43 5.1.1 What's Wrong with Merging Methods? . 43 5.2 Reconstructing States by Splitting . ..