Detecting Emergent Processes in Cellular Automata with Excess Information
Total Page:16
File Type:pdf, Size:1020Kb
Detecting emergent processes in cellular automata with excess information David Balduzzi1 1 Department of Empirical Inference, MPI for Intelligent Systems, Tubingen,¨ Germany [email protected] Abstract channel – transparent occasions whose outputs are marginal- ized over. Finally, some occasions are set as ground, which Many natural processes occur over characteristic spatial and fixes the initial condition of the coarse-grained system. temporal scales. This paper presents tools for (i) flexibly and Gliders propagate at 1/4 diagonal squares per tic – the scalably coarse-graining cellular automata and (ii) identify- 4n ing which coarse-grainings express an automaton’s dynamics grid’s “speed of light”. Units more than cells apart cannot well, and which express its dynamics badly. We apply the interact within n tics, imposing constraints on which coarse- tools to investigate a range of examples in Conway’s Game grainings can express glider dynamics. It is also intuitively of Life and Hopfield networks and demonstrate that they cap- clear that units should group occasions concentrated in space ture some basic intuitions about emergent processes. Finally, and time rather than scattered occasions that have nothing to we formalize the notion that a process is emergent if it is bet- ter expressed at a coarser granularity. do with each other. In fact, it turns out that most coarse- grainings express a cellular automaton’s dynamics badly. The second contribution of this paper is a method for dis- Introduction tinguishing good coarse-grainings from bad based on the following principle: Biological systems are studied across a range of spa- tiotemporal scales – for example as collections of atoms, • Coarse-grainings that generate more information, rela- molecules, cells, and organisms (Anderson, 1972). How- tive to their sub-grainings, better express an automaton’s ever, not all scales express a system’s dynamics equally dynamics than those generating less. well. This paper proposes a principled method for identify- ing which spatiotemporal scale best expresses a cellular au- We introduce two measures to quantify the information gen- tomaton’s dynamics. We focus on Conway’s Game of Life erated by coarse-grained systems. Effective information, ei, and Hopfield networks as test cases where collective behav- quantifies how selectively a system’s output depends on its ior arises from simple local rules. input. Effective information is high if few inputs cause the Conway’s Game of Life is a well-studied artificial sys- output, and low if many do. Excess information, ξ, mea- tem with interesting behavior at multiple scales (Berlekamp sures the difference between the information generated by a et al., 1982). It is a 2-dimensional grid whose cells are up- system and its subsystems. dated according to deterministic rules. Remarkably, a suffi- With these tools in hand we investigate coarse-grainings ciently large grid can implement any deterministic compu- of Game of Life grids and Hopfield networks and show that tation. Designing patterns that perform sophisticated com- grainings with high ei and ξ capture our basic intuitions putations requires working with distributed structures such regarding emergent processes. For example, excess infor- as gliders and glider guns rather than individual cells (Den- mation distinguishes boring (redundant) from interesting nett, 1991). This suggests grid computations may be better (synergistic) information-processing, exemplified by blank expressed at coarser spatiotemporal scales. patches of grid and gliders respectively. The first contribution of this paper is a coarse-graining Finally, the penultimate section converts our experience procedure for expressing a cellular automaton’s dynamics with examples in the Game of Life and Hopfield networks at different scales. We begin by considering cellular au- into a provisional formalization of the principle above. tomata as collections of spacetime coordinates termed occa- Roughly, we define a process as emergent if it is better ex- sions (cell ni at time t). Coarse-graining groups occasions pressed at a coarser scale. into structures called units. For example a unit could be a The principle states that emergent processes are more 3 × 3 patch of grid containing a glider at time t. Units do than the sum of their parts – in agreement with many other not have to be adjacent to one another; they interact through approaches to quantifying emergence (Crutchfield, 1994; Tononi, 2004; Polani, 2006; Shalizi and Moore, 2006; Seth, pl aljdo(sl) . The do(−) is not included in the notation ex- 2010). Two points distinguishing our approach from prior plicitly to save space. However, it is always implicit when work are worth emphasizing. First, coarse-graining is scal- applying any Markov matrix. able: coarse-graining a cellular automaton yields another A Hopfield network over time interval [α; β] is an abstract cellular automaton. Prior works identify macro-variables automaton. Occasions are spacetime coordinates – e.g. vl = such as temperature (Shalizi and Moore, 2006) or centre- ni;t, cell i at time t. An edge connects vk ! vl if there is of-mass (Seth, 2010) but do not show how to describe a sys- a connection from vk’s cell to vl’s and the time coordinates tem’s dynamics purely in terms of these macro-variables. By are t − 1 and t respectively for some t. The mechanism is contrast, an emergent coarse-graining is itself a cellular au- given by Eq. (1). Occasions at t = α, with no incoming tomaton, whose dynamics are computed via the mechanisms edges, can be set as fixed initial conditions or noise sources. of its units and their connectivity (see below). Similar considerations apply to the Game of Life. Second, our starting point is selectivity rather than pre- Non-Markovian automata (whose outputs depend on in- dictability. Assessing predictability necessitates building a puts over multiple time steps) have edges connecting occa- model and deciding what to predict. Although emergent sions separated by more than one time step. variables may be robust against model changes (Seth, 2010), it is unsatisfying for emergence to depend on properties of Coarse-graining both the process and the model. By contrast, effective and Define a subsystem X of cellular automaton Y as a subgraph excess information depend only on the process: the mecha- containing a subset of Y ’s vertices and a subset of the edges nisms, their connectivity, and their output. A process is then targeting those vertices. We show how to coarse-grain X. emergent if its internal dependencies are best expressed at Definition (coarse-graining). Let X be a subsystem of Y . coarse granularities. The coarse-graining algorithm detailed below takes X ⊂ Y and data K as arguments, and produces new cellular au- Probabilistic cellular automata tomaton XK. Data K consists of (i) a partition of X’s occa- Concrete examples. This paper considers two main ex- sions VX = G[C[U1 [···[UN into ground G, channel G amples of cellular automata: Conway’s Game of Life and C and units U1 ::: UN and (ii) ground output s . Hopfield networks (Hopfield, 1982). Vertices of automaton XK, the new coarse-grained occa- The Game of Life is a grid of deterministic binary cells. A sions, are units: VX := fU1 ::: UN g. The directed graph cell outputs 1 at time t iff: (i) three of its neighbors outputted K of XK is computed in Step 4 and the alphabets Al of units 1s at t − 1 or (ii) it and two neighbors outputted 1s at t − 1. Ul are computed in Step 5. Computing the Markov matrices In a Hopfield network (Amit, 1989), cell nk fires with (mechanisms) of the units takes all five steps. probability proportional to The ground specifies occasions whose outputs are fixed: 2 3 the initial condition sG. The channel specifies unobserved 1 X p(n = 1jn ) / exp α · n (1) occasions: interactions between units propagate across the k;t •;t−1 4T jk j;t−15 j!k channel. Units are macroscopic occasions whose interac- tions are expressed by the coarse-grained automaton. Fig. 1 Temperature T controls network stochasticity. Attractors illustrates coarse-graining a simple automaton. fξ1; : : : ; ξN g are embedded into a network by setting the There are no restrictions on partitions. For example, al- PN µ µ connectivity matrix as αjk = µ=1(2ξj − 1)(2ξk − 1). though the ground is intended to provide the system’s ini- tial condition, it can contain any spacetime coordinates so Abstract definition. A cellular automaton is a finite di- that in pathological cases it may obstruct interactions be- rected graph X with vertices VX = fv1 : : : vng. Vertices tween units. Distinguishing good coarse-grainings from bad are referred to as occasions; they correspond to spacetime is postponed to later sections. coordinates in concrete examples. Each occasion vl 2 VX is equipped with finite output alphabet Al and Markov ma- Algorithm. Apply the following steps to coarse-grain: Q trix (or mechanism) pl(aljsl), where sl 2 Sl = k!l Ak, Step 1. Marginalize over extrinsic inputs. the combined alphabet of the occasions targeting vl. The External inputs are treated as independent noise sources; mechanism specifies the probability that occasion vl chooses we are only interested in internal information-processing. output al given input sl. The input alphabet of the entire au- An occasion’s input alphabet decomposes into a product tomaton X is the product of the alphabets of its occasions X Y nX Q Sl = Sl × Sl of inputs from within and without the Xin := l2V Al. The output alphabet is Xout = Xin. X system. For each occasion vl 2 VX , marginalize over exter- Remark. The input Xin and output Xout alphabets are dis- nal outputs using the uniform distribution: tinct copies of the same set. Inputs are causal interven- X X X Y nX Y nX tions imposed via Pearl’s do(−) calculus (Pearl, 2000).