Algorithmic Analysis of Irrational Rotations in a Single Neuron Model$

Journal of Complexity 19 (2003) 132–152 http://www.elsevier.com/locate/jco

Algorithmic analysis of irrational rotations in a single neuron model$

Hayato Takahashia,* and Kazuyuki Aiharab,c a Department of Statistical Science, The Graduate University for Advanced Studies, The Institute of Statistical Mathematics, 4-6-7 Minami-Azabu, Minato-ku, Tokyo 106-8569, Japan b Department of Mathematical Engineering and Information Physics, Graduate School of Engineering, University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan c CREST, Japan Science and Technology Corporation (JST), 4-1-8 Hon-Cho, Kawaguchi, Saitama 332-0012, Japan

Received 1 June 2002; accepted 11 September 2002

Abstract

We study computability of real-valued functions and the information needed for simulation of dynamical systems. In particular, we describe application of Kolmogorov complexity theory to computer simulation of irrational rotations in a single neuron model. We deduce the information needed for a parameter in simulating the dynamics by showing a difference in Kolmogorov complexity between a computable parameter and a non-computable parameter. Finally, we show that all trajectories generated by irrational rotations are non-computable iff its parameter is non-computable. r 2002 Elsevier Science (USA). All rights reserved.

Keywords: Kolmogorov complexity; Algorithmic information theory; Computability; Bayes code; MDL code; Nagumo–Sato model; Single neuron model; Irrational rotation; Sturmian sequence; Symbolic dynamics

1. Introduction

Although continuous analysis has been greatly progressing in mathematics and physics, there is just a paucity of studies (for example, see [25,30]) on computability

$Revised from ‘‘Asymptotic evaluation of Kolmogorov complexity of pulse sequences generated by a single neuron model’’. *Corresponding author. The present address: The Institute of Statistical Mathematics, 4-6-7 Minami- Azabu, Minato-ku, Tokyo, 106-8569, Japan. Tel.: 81-3-3446-1501; fax: 81-3-5421-8750. E-mail addresses: [email protected] (H. Takahashi), [email protected] (K. Aihara).

Nomenclature

N the set of natural numbers Q the set of rational numbers R the set of real numbers n f0; 1g the set of finite binary strings f0; 1gn the set of binary strings of length n N f0; 1g the set of infinite binary sequences o1:n ¼ o1o2?on; oiAf0; 1g; 1pipn: N oAf0; 1g ; o ¼ o1o2? KðxÞ the prefix Kolmogorov complexity of a string x KðxjyÞ the prefix Kolmogorov complexity of a string x with a given string y ðX; F; mÞ a probability measure space X ¼½0; 1Þ; F is Borel s algebra and m is a measure on ðX; FÞ r 0prp1 r r A0 ¼½0; 1 À rÞ; A1 ¼½1 À r; 1Þ Tr : X-X TrðxÞ¼x þ r mod 1 Sr ¼ Ar T À1Ar ? T Ànþ1Ar o1:n o1 - r o2 - - r on P ðo Þ¼ mðSr Þ r 1:n o1:n D ¼fðr; xÞjxASr ; 0 r 1g o1:n o1:n p p

Bo1:n ¼fr j Prðo1:nÞ40g

Lo1:n ¼ sup Bo1:n À inf Bo1:n of real-valued functions and information required to simulate dynamical systems [27]. These subjects are important when we solve analytical and physical problems with computers. In this paper, we explore a particular problem on irrational rotations and a single neuron model. Irrational rotations are a classical subject in dynamical systems theory, and the single neuron model examined in this paper is called the Nagumo–Sato model [23], which can potentially generate aperiodic behavior described with irrational rotations. We study the information on a parameter for simulation of irrational rotations by means of algorithms. The theory of Kolmogorov complexity, the complexity of a string measured in terms of a minimum length program that generates the string [8,15,18,28], plays an especially important role in our analysis. From Kolmogorov complexity of an observed sequence of a dynamical system, we know how much information is necessary to simulate the dynamics because Kolmogorov complexity is a minimum length program that generates the sequence. Therefore, if the asymptotic order of Kolmogorov complexity of an observed sequence of length n is f ðnÞ; where f ðnÞ is some increasing function of n; then the 134 H. Takahashi, K. Aihara / Journal of Complexity 19 (2003) 132–152 total bits for the initial value and the parameter values i.e., the total bits that are needed to simulate the dynamics are on the order of at least f ðnÞ (see also [10]). Although the dynamics of the Nagumo–Sato model [23] is almost always periodic (for details see [11,12] and the remark in Section 2), we discuss mainly its aperiodic behavior because in periodic behavior, the spike sequence is computable and the asymptotic value of Kolmogorov complexity with a given sequence length is constant. Moreover, aperiodic behavior has been observed in an electrophysiological experiment with squid giant axons under a condition corresponding to the Nagumo– Sato model (see [21] and Fig. 1). Although such aperiodic dynamics can be well described by a chaotic neuron model derived by replacing the Heaviside output function of the Nagumo–Sato model with a sigmoidal one, which can be extended to a chaotic neural network model [1,9], a kind of aperiodic structure of solutions is immanent in the Nagumo–Sato model itself as Sturmian sequences that are studied in this paper. The present paper is our first step toward analyzing more general dynamics. The prefix Kolmogorov complexity (see [8,18]), which is a variant of the originally defined Kolmogorov complexity [15,28], is represented by string x with a given length n as KðxjnÞ: The precise definition of the prefix Kolmogorov complexity is given in Section 5. We relate a measure on infinite sequences to the dynamics of the Nagumo–Sato model to evaluate Kolmogorov complexity. Aperiodic spike sequences of the Nagumo–Sato model are considered observed data of a trajectory of an irrational rotation. Such sequences generated by irrational rotations are called Sturmian sequences [3,4,22,31], which is an example of symbolic dynamics and described with a probability measure on infinite sequences. There are several known relations between Kolmogorov complexity and measures. For example, if a measure P on infinite sequences is ergodic [5], then limn-NKðxjnÞ=n ¼ h; PÀa.e. (almost everywhere with respect to P), where x is a string of length n and h is the entropy of P [6,32]. In this case we cannot derive the oðnÞ term of Kolmogorov complexity, and this cannot be applied to our case because

Fig. 1. A complicated response (the lower waveform) experimentally observed in a squid giant axon where the nerve membrane in the resting state is stimulated by periodic pulses (the upper waveform). H. Takahashi, K. Aihara / Journal of Complexity 19 (2003) 132–152 135 the order of Kolmogorov complexity of the Nagumo–Sato model is oðnÞ: The probability model we use is parametric, and MDL coding is generally used for parametric models [26]. However, MDL coding is not applicable to our model. The main reason is that the likelihood function of our model is not differentiable at maximum-likelihood estimate (see also Remark 6.2). If a probability measure P is computable (for the computability of measure, see [19] and Section 4), then Kolmogorov complexity KðxjnÞ is bounded above by Àlog PðxÞþc where PðxÞ is the probability of sequences with the initial segment x and constant c independent of P and x [19]. But if the measure is not computable, we cannot apply this property. In this paper we use Bayes coding [2] to give an asymptotic evaluation of the oðnÞ term of Kolmogorov complexity, particularly when the measure is not computable. First, we explore computability of the measure derived from the Nagumo–Sato model. Speciﬁcally, we show that computability of the model parameter is equivalent to computability of the derived measure. Second, we deals with the Kolmogorov complexity of spike sequences of the Nagumo–Sato model when the derived measure is not computable. This shows a difference in Kolmogorov complexity between computable parameters and non-computable parameters. Finally, we show that the parameter is non-computable iff all sequences generated by our model are non- computable. All technical proofs are given in the appendix.

2. The Nagumo–Sato model, Hata’s result, and Sturmian sequences

In this section, we briefly describe the Nagumo–Sato model and its characteristics. The Nagumo–Sato model is a special version of Caianiello’s neuronic equation [7]. This model deals with a single neuron stimulated by a periodic constant-strength input (see the upper waveform of Fig. 1). The refractory effect due to past firing of the neuron is assumed to decay exponentially with time. The dynamics of the model can be expressed as follows [23]: "# Xn r xnþ1 ¼ 1 A À a xnÀr=b À y ; ð1Þ r¼0 where 1½x is the Heaviside function, A is the strength of the input stimulus, a40; b41; and y is the threshold value. xn is the output of the neuron at time n: xn ¼ 0 and 1 represent non-firingP and firing of a output spike, respectively. Letting y ¼ 1 þðA À yÞ=ab À n x =br; (1) is rewritten as follows [23,11,12]: n ( r¼0 nÀr bðyn À cÞþ1ifynoc; ynþ1 ¼ f ðynÞ ð2Þ bðyn À cÞ if ynXc;

xnþ1 ¼ 1½yn À c;

AÀy 1 where b ¼ 1=b and c ¼ 1 À a ð1 À bÞ: We assume that 0oco1 because the neuron keeps firing if cp0 or resting if cX1: We also assume that f is a map from I ¼½0; 1Þ to itself because the dynamics with any initial value must fall into I after some 136 H. Takahashi, K. Aihara / Journal of Complexity 19 (2003) 132–152 iterations. To study the dynamics of the Nagumo–Sato model, it is sufficient to analyze the dynamics of f on I: To briefly explain the results obtained by Hata [11,12], we first define an average firing rate rðyÞ:

Deﬁnition 2.1. 1 XnÀ1 rðyÞ lim 1½ f jðyÞÀc: n-N n j¼0

Hata showed that rðyÞ is independent of y: In the following, we write r for rðyÞ: Hata also showed that

fxng is aperiodic 3r is irrational: The following theorem is useful for analysis of the dynamics of f :

Theorem 2.1 (Hata [11,12]). If r is irrational, f is topologically semiconjugate to an irrational rotation, i.e.,

Rrh ¼ hf ; where h is a continuous and monotonically increasing onto-map (but not necessarily one-to-one) and Rr : I-I; RrðxÞ¼x þ r mod 1:

With this theorem, the dynamics of the Nagumo–Sato model is written as follows: n xnþ1 ¼ 1½ f ðy0ÞÀc ( 1ifhðy0Þþrn ðmod 1ÞA½1 À r; 1Þ; ¼ ð3Þ 0 otherwise: From this expression, we see that it is enough to analyze irrational rotations for studying the aperiodic dynamics of the Nagumo–Sato model. Sequence fxng obtained from (3) is called the Sturmian sequence [3,4,22,31]. In the following, we explore Kolmogorov complexity of Sturmian sequences.

Remark 2.1. Fix b in (2). Let S ¼fcjfxng is aperiodicg; then mðSÞ¼0; where m is the Lebesgue measure on ½0; 1Þ [11,12]. In other words, the aperiodic solutions are not observable. But Kolmogorov complexity Kðx1?xnjnÞ of any periodic solution is simply constant order. Therefore, we concentrate on aperiodic cases in the following analysis.

? Remark 2.2. In [14], it is shown that a ﬁnite sequence x1x2 xn is [22]Pan initial segment of a Sturmian sequence iff x x ?x is ‘‘balanced’’, i.e. j iþL x À P 1 2 n i i jþL j xjjp1 holds for any positive integers i; j; L such that 1pipi þ Lpn; 1pjpj þ Lpn: The spike sequence 101011010101011010101011010101 shown in Fig. 1 is balanced and hence Sturmian. H. Takahashi, K. Aihara / Journal of Complexity 19 (2003) 132–152 137

3. Representation of dynamical systems as measures

Here, we introduce a method of representing dynamical systems as measures on the basis of symbolic dynamics. n First we introduce notations for finite strings and infinite sequences. We use f0; 1g to n denote the set of finite strings such as f0; 1g ¼fl; 0; 1; 00; 01; yg; where l is an empty N word, and we use f0; 1g to denote the set of infinite sequences consisting of zeros and N ones. For an infinite sequence o ¼ o1o2?Af0; 1g ; o1:n denotes o1o2?on: Next, we define a probability measure on infinite sequences. Let Gx fojo1:n ¼ xg: We call Gx a cylinder. The cylinder Gx is the set of infinite sequences that begins with a finite word x: If PðGxÞX0 and PðGxÞ¼PðGx0ÞþPðGx1Þ hold for all x and PðGlÞ¼1; we say that P is a measure on infinite sequences. We write PðxÞ for PðGxÞ: For example, if PðxÞ¼2Àn where n is the length of x; then P represents fair coin N 1 flipping, i.e., i.i.d. stochastic process on f0; 1g with Pð1Þ¼Pð0Þ¼2: Note that if a s-additive probability measure P is defined for all cylinders, then P is uniquely defined as a s-additive probability measure on the s-algebra generated by all cylinders (for more details, see [16]). Now we relate a dynamical system to a probability measure on infinite sequences. Let ðX; F; mÞ be a probability measure space. Let T : X-X be a measurable transformation on a state space X such that AAF implies T À1AAF: Let measurable sets A0; A1AF be a partition of X such as A0,A1 ¼ X and A0-A1 ¼ |: We define a probability measure on infinite sequences PT induced by T as PT ðo1:nÞ À1 ? Ànþ1 mðAo1 -T Ao2 - -T Aon Þ: Intuitively, PT ðo1:nÞ gives a probability of initial kÀ1 A y values x such that T ðxÞ Aok for k ¼ 1; ; n: If T is a measure-preserving À1 transformation on ðX; F; mÞ (i.e., 8AAF; mðT AÞ¼mðAÞ), then PT is a N stationary stochastic process on f0; 1g : If T is an ergodic transformation on ðX; F; mÞ (i.e., T is a measure-preserving transformation and T À1A ¼ A implies N mðAÞ¼0 or 1), then PT is an ergodic process on f0; 1g (see [5] for details). Such a measure m preserved by transformation is called an invariant measure. The probability measure PT represents the dynamics of T: Indeed, if T is a measure- preserving transformation on ðX; F; mÞ; then PT is a factor of T (but generally not isomorphic to T; for more details see [5, p. 19][24, p. 88]). For example if T : ½0; 1Þ-½0; 1Þ; TðxÞ¼2x mod 1; A0 ¼½0; 1=2Þ; A1 ¼½1=2; 1Þ and m is the uniform measure on ½0; 1Þ; then T is a measure-preserving transformation and PT is the i.i.d. 1 stochastic process with Pð1Þ¼Pð0Þ¼2: In the following, we consider not only invariant measures but also other general measures. From (3), in the case of the Nagumo–Sato model (or Sturmian sequences), let define notations as follows.

Deﬁnition 3.1.

ðX; F; mÞ is a probability measure space;

X ¼½0; 1Þ; F is Borel s algebra and m is a measure on ðX; FÞ; 138 H. Takahashi, K. Aihara / Journal of Complexity 19 (2003) 132–152

r r A0 ¼½0; 1 À rÞ; A1 ¼½1 À r; 1Þ;

Tr : X-X; TrðxÞ¼x þ r mod 1;

Sr ¼ Ar TÀ1Ar ? T Ànþ1Ar ; o1:n o1 - r o2 - - r on P ðo Þ¼mðSr Þ; r 1:n o1:n D ¼fðr; xÞjxASr ; 0 r 1g; o1:n o1:n p p where r is an irrational average ﬁring rate. We omit the index r when it is obvious from the context; namely, we use A ; A ; T; and S for Ar; Ar; T ; and Sr : 0 1 o1:n 0 1 r o1:n P ðo Þ¼mðSr Þ gives a probability measure of a spike sequence o when r 1:n o1:n 1:n xAX (or y0 of (3)) is randomly drawn according to m: Since Tr is ergodic with respect to uniform measure, if m is the uniform measure on X; then Pr is ergodic and has entropy of 0 (for more details, see [5]). We give an example of partition of X ð¼ ½0; 1ÞÞ by fSr g in Fig. 2. o1:n

S000

0 =1 mod1 A0

A 1− 3 1 S100

S001 1 −

1− 2 S010

Fig. 2. An example of partition of X by fSr g; where o1:n r r n ¼ 3; X ¼½0; 1Þ; A1 ¼½1 À r; 1Þ; A0 ¼½0; 1 À rÞ;

Tr : X-X; TrðxÞ¼x þ r mod 1;

r À1 À2 S100 ¼½1 À r; 1Þ-Tr ½0; 1 À rÞ-Tr ½0; 1 À rÞ¼½1 À r; 1Þ;

r À1 À2 S010 ¼½0; 1 À rÞ-Tr ½1 À r; 1Þ-Tr ½0; 1 À rÞ¼½1 À 2r; 1 À rÞ;

r À1 À2 S001 ¼½0; 1 À rÞ-Tr ½0; 1 À rÞ-Tr ½1 À r; 1Þ¼½1 À 3r; 1 À 2rÞ;

r À1 À2 S000 ¼½0; 1 À rÞ-Tr ½0; 1 À rÞ-Tr ½0; 1 À rÞ¼½0; 1 À 3rÞ; r r r r | and S011 ¼ S101 ¼ S110 ¼ S111 ¼ : H. Takahashi, K. Aihara / Journal of Complexity 19 (2003) 132–152 139

1 100

0.8 111 110 0.6 101 x 010 0.4 001 000 0.2 011

1/3 1/2 2/3 1

Fig. 3. An example of partition of ½0; 1Â½0; 1Þ by Do1:n for n ¼ 3: Each region partitioned by lines except vertical lines at r ¼ 1=3; 1=2; 2=3 represents the region Do1:n : If m in Deﬁnition 3.1 is uniform measure, the vertical lines at r ¼ 0; 1=3; 1=2; 2=3; 1 show the values of r that attain maximum values of Prðo1:3Þ (i.e., maximum-likelihood estimators).

Do1:n is the set of parameters and initial values that generate o1:n: Each region

Do1:n is represented as follows.

a| Lemma 3.1 (Berstel and Pocchiola [3] and Yasutomi [31]). If Do1:n ; Do1:n is one of the regions of ½0; 1Â½0; 1Þ partitioned by lines Àkr mod 1; 0pkpn; 0prp1:

We see that Sr ¼fx jðr; xÞAD g; i.e., the intersection of the vertical line at r o1:n o1:n with D is Sr ; if m is the uniform measure on ½0; 1Þ; the length of the intersection o1:n o1:n is Prðo1:nÞ: If we consider Prðo1:nÞ as a function of r; Prðo1:nÞ is called likelihood function. The r# that attains the maximum value of the likelihood function is called the maximum-likelihood estimator. The following lemma is a direct consequence of the lemma above.

Lemma 3.2. If m is the uniform measure on ½0; 1Þ; the followings hold: (a) the likelihood function Prðo1:nÞ is continuous and piecewise linear; (b) the absolute values of the slopes of the graph of the likelihood function are less than n; (c) Pr# ðo1:nÞX1=n:

See proof in the appendix. We give an example of partition of ½0; 1Â½0; 1Þ by

Do1:n for n ¼ 3inFig. 3. 140 H. Takahashi, K. Aihara / Journal of Complexity 19 (2003) 132–152

4. Computability of measure derived from the Nagumo–Sato model

In this section, we study the computability of the measure derived from the Nagumo–Sato model. If a measure is computable, then we can use a data compression technique such as the Shannon–Fano–Elias code. Hence, if the derived measure is computable, we can obtain an upper bound of Kolmogorov complexity. First, we give the definitions of computability of real numbers and probability measures. Informally, we say a real number r is computable if we can compute r with N any given precision. Also, we say a probability measure on f0; 1g is computable if we can compute the probability of any cylinder with any given precision, and a probability measure on R is computable if we can compute the probability of any interval having rationalffiffiffi endpoints with any given precision. p N For example, P2 and p are computable,P and a probability measure P on f0; 1g ; n n x nÀ x such as PðxÞ¼p i¼1 i ð1 À pÞ i¼1 i ; is computable if p ð0ppp1Þ is a computable real number, where x is a finite sequence, x ¼ x1?xn; and xiAf0; 1g [19, p. 300]. Also, the uniform measure on ½0; 1 is computable. Definition 4.1. A real number r is computable iff 1 (A 8kANjr À AðkÞjp ; k where A is a rational-valued computable function. N A probability measure P on f0; 1g is computable iff

n 1 (A 8xAf0; 1g 8kANjPðxÞÀAðx; kÞjp ; k where A is a rational-valued computable function. A probability measure m on ½0; 1Þ is computable iff 1 (A 8a; b ð0papbo1; a; bAQÞ8kANjmð½a; bÞ À Aða; b; kÞjp ; k where A is a rational-valued computable function.

For the precise meaning of rational-valued computable functions, see [19, p. 35]. Before proving our ﬁrst theorem, we need a condition on a measure m: Condition 4.1. A measure m is computable, where m is a measure given in Deﬁnition 3.1. Also mð½0; xÞ is a continuous function of x for 0pxo1; and mð½0; xÞomð½0; yÞ for 0pxoyo1:

For example, the uniform measure on ½0; 1Þ satisﬁes the condition. Now we state our ﬁrst result.

Theorem 4.1. Let Pr be a probability measure given in Deﬁnition 3.1. Under Condition 4.1, Pr is computable iff r is computable.

See proof in the appendix. H. Takahashi, K. Aihara / Journal of Complexity 19 (2003) 132–152 141

5. Kolmogorov complexity

In this section we give some definitions and known results about Kolmogorov complexity. First, we give the definition of the prefix Kolmogorov complexity introduced in n [8,18]. For x ¼ x1x2?xnAf0; 1g ; let jxjn: We use two for the logarithmic base. n We establish a one-to-one correspondence between f0; 1g and the natural numbers N as follows [32]: l20; 021; 122; 0023; 0124; y: The conditional n Kolmogorov complexity KAðxjyÞ of x; yAf0; 1g is defined as KðxjyÞ n n n minfjpjjAðp; yÞ¼xg; where A : f0; 1g Âf0; 1g -f0; 1g is a partial recursive function and the domain of AðÁ; yÞ is a prefix code (in [8], a different conditional complexity is defined). The unconditional Kolmogorov complexity is defined as KAðxÞKAðxjlÞ; where l is an empty word. The prefix Kolmogorov complexity KAðxjyÞ is called optimal if KAðxjyÞpKBðxjyÞþc for every partial recursive function B; where c is a constant independent of x and y: In [8,18] it is shown that an optimal prefix Kolmogorov complexity KAðxjyÞ exists. In the following, fix an optimal prefix Kolmogorov complexity KAðxjyÞ and let KðxjyÞ¼KAðxjyÞ: Note that if KA and KB n are optimal then jKAðxjyÞÀKBðxjyÞjpc for all x; yAf0; 1g where c is a constant independent of x and y: The next theorem describes known relations between Kolmogorov complexity and measures.

Theorem 5.1. (a) For any ergodic measure P; limn-NKðo1:njnÞ=n ¼ h; P-a.e. holds, where h is the entropy of P: (b) For any computable measure P; Kðo1:njnÞp À log Pðo1:nÞþc holds for all n and n o1:nAf0; 1g ; where c is a constant independent of n and o1:n: (c) For any measure P; let A ¼foj(c8n À log Pðo1:nÞÀlog n À 2 log log n À cpKðo1:njnÞg; then PðAÞ¼1: (d) For any measure P; EðÀlog Pðo1:nÞÞpEðKðo1:njnÞÞ holds where E means expectation.

Part (a) is an immediate consequence of [6,32]. For a proof of (b), see [19]. Proof of (c) is given in [2]. A similar result of (c) is given in [17]. For a proof of (d), see [19].

6. Kolmogorov complexity of the Nagumo–Sato model

In this section, we provide an asymptotic estimate of Kolmogorov complexity of the spike sequences of the Nagumo–Sato model. In the following, let o1:n be a spike sequence of the Nagumo–Sato model. First we consider the periodic case. In this case the average ﬁring rate r is rational. Since the spike sequence is computable, we have

Kðo1:njnÞpc; where c is a constant independent of n: 142 H. Takahashi, K. Aihara / Journal of Complexity 19 (2003) 132–152

Next we consider the aperiodic case. In this case, the average ﬁring rate r is irrational. If Pr is computable, we can easily obtain an upper bound of Kðo1:njnÞ by Theorem 5.1(b). Hence, we analyze Kðo1:njnÞ separately according to whether Pr is computable or not. Due to Theorem 4.1, under Condition 4.1, Pr is computable iff r is computable.

6.1. Computable case

First we note the following corollary, which is a direct consequence of Theorem 5.1(c), and we omit the proof.

Corollary 6.1 (Corollary of Theorem 5.1(c)). Let Pr be a probability measure given in Deﬁnition 3.1. If we set B ¼foj(c8n À log Prðo1:nÞÀlog n À 2 log log n À cpKðo1:njnÞg; then PrðBÞ¼1 for all r:

The lower bound stated above is tight under the condition described below.

Theorem 6.1. Let Pr be a probability measure given in Deﬁnition 3.1. Under Condition 4.1, if r is a computable irrational number then the following relations (a)–(d) hold. n (a) Kðo1:njnÞp À log Prðo1:nÞþc for all n and o1:nAf0; 1g ; where c is a constant independent of o1:n and n: (b) EðKðo1:njnÞÞplog n þ c for all n; where c is a constant independent of o1:n and n: a| (c) If So1:n ; then Kðo1:njnÞplog n þ 2 log log n þ c for all n; where c is a constant independent of o1:n and n: For any irrational r; the following relation (d) holds. (d) If we let Sr ¼fo jSr a|g; then there are at least ðn þ 1Þð1 À 2ÀcÞþ1 strings n 1:n o1:n r o1:nASn such that Kðo1:njnÞXlog n À c for all n:

See proof in the appendix.

6.2. Non-computable case

Let us deﬁne Z 1 P˜ðo1:nÞ¼ Prðo1:nÞ dFðrÞ; ð4Þ 0 where F is a distribution deﬁned on ½0; 1; i.e. F is a prior distribution of parameter r: The following corollary is a direct consequence of Theorem 5.1(c) and Fubini’s theorem [16], and we omit the proof.

Corollary 6.2 (Corollary of Theorem 5.1(c)). Let P˜ be a probability measure given in (4). H. Takahashi, K. Aihara / Journal of Complexity 19 (2003) 132–152 143

If we set A ¼foj(c8n À log P˜ðo1:nÞÀlog n À 2 log log n À cpKðo1:njnÞg; then PrðAÞ¼1 for almost every r with respect to F:

The lower bound above is tight for almost every r as follows:

Theorem 6.2. Let P˜ be a probability measure given in (4). Let F in (4) be continuous and computable in the sense of Deﬁnition 4.1; i.e., the probability of any single element set fxg; xA½0; 1 is zero, and FðxÞÀFðyÞ is computable for any rational numbers 0pxpyp1: Under condition 4.1, the followings hold: N (a) The probability measure Pon˜ f0; 1g is computable. 0 n (b) Kðo1:njnÞp À log P˜ðo1:nÞþc for all nAN and o1:nAf0; 1g ; 0 where c is a constant independent of n and o1:n:

See proof in the appendix.

6.2.1. Stationary case Now we consider stationary stochastic processes. Let a measure m in Deﬁnition 3.1 be the uniform measure on ½0; 1Þ: Also, we let a prior distribution F in (4) be the uniform measure for simplicity. Then m and F satisfy the condition of Theorem 6.2. In this case, Pr is ergodic, and its entropy is 0 (for more details see [5] and Section 3). Hence, from Theorem 5.1(a), we have

lim Kðo1:njnÞ=n ¼ 0; Pr-a:e: n-N

In the following, we study the asymptotic order of Kðo1:nÞ up to the oðnÞ term with Theorem 6.2. For this purpose, let us deﬁne

Bo1:n ¼frjPrðo1:nÞ40g; Lo1:n ¼ sup Bo1:n À inf Bo1:n : ð5Þ

Let r# be a real number r that attains the maximum value of Prðo1:nÞ; i.e. r# is a maximum-likelihood estimator. Note that r# is well deﬁned because Prðo1:nÞ is a continuous function of r on ½0; 1 (see Lemma 3.2). Also note that r# depends on o1:n: Approximating P˜ðo1:nÞ; we obtain the following.

˜ Theorem 6.3. Let P be a probability measure given in (4). Let Lo1:n be the length of the support set of the likelihood function as deﬁned in (5). Let m in Deﬁnition 3.1 and F in (4) be uniform measures on ½0; 1Þ and ½0; 1; respectively. If nX2; the followings hold:

(a) # ˜ # Prðo1:nÞLo1:n =2pPðo1:nÞpPrðo1:nÞLo1:n : (b) L XP# ðo Þ=nX1=n2: o1:n rP 1:n P nÀ1 nÀ1 (c) nþ o 2nÀ1À o i¼1 i i¼1 i Lo1:n p nðnÀ1Þ ; if on ¼ 1: Lo1:n p nðnÀ1Þ ; if on ¼ 0: In both cases, 2nÀ1 Lo1:n pnðnÀ1Þ: 144 H. Takahashi, K. Aihara / Journal of Complexity 19 (2003) 132–152

Corollary 6.3. Under the condition given above for Theorem 6.3, the followings hold: n # A (a) Kðo1:njnÞp À log Prðo1:nÞÀlog Lo1:n þ c for all n and o1:n f0; 1g ; where c is a constant independent of n and o1:n: # (b) If we set A ¼foj(c8n À log Prðo1:nÞÀlog Lo1:n À log n À 2 log log n À cpKðo1:njnÞg; then PrðAÞ¼1 for almost every r with respect to uniform measure. 0 0 (c) Kðo1:njnÞpÀ 2 log Pr# ðo1:nÞþlog n þ c pÀ log Pr# ðo1:nÞþ2 log n þ c p3 log 0 n 0 n þ c for all n and o1:nAf0; 1g ; where c is a constant independent of n and o1:n: (d) If we set B ¼foj(c8n À log Pr# ðo1:nÞÀ2 log log n À cpKðo1:njnÞg; then PrðBÞ¼1 for almost every r with respect to uniform measure. # # (e) EP˜ ðÀlog Prðo1:nÞÞþlogðnÀ 1ÞÀ1pEP˜ ðÀlog Prðo1:nÞÞ þ EP˜ ðÀlog Lo1:n Þ pEP˜ ˜ ðKðo1:njnÞÞ where EP˜ means expectation with respect to P.

See proof in the appendix.

Remark 6.1. Theorem 6.3(c) implies limn-NLo1:n ¼ 0 and the maximum-likelihood estimator always converges to parameter r; i.e. limn-Nr# ¼ r: This is a stronger property than strong consistency, i.e. limn-Nr# ¼ r; Pr-a.e.

Remark 6.2. The Nagumo–Sato model is parametric and so here we consider MDL code [26]. Let Qy be a parametric probability model. According to Rissanen [26], the length of MDL code is 1 # Àlog Qyðo1:nÞþ2 log n þ oðlog nÞ: ð6Þ By comparing (6) with Corollary 6.3(e), we see that MDL code does not hold for our model. This is because our model is not differentiable at maximum likelihood. In [13,14,29], it is shown that the shape of the graph of the likelihood function is always triangular.

7. Computability of sequences

In this section, we prove non-computability of sequences generated by the Nagumo–Sato model when a parameter is non-computable. Computability of sequences is defined as follows: Definition 7.1. An infinite sequence o is computable if there is a computable function A such that AðnÞ¼o1:n for all nAN:

Computability of sequences is characterized in terms of Kolmogorov complexity as follows [20]:

(c 8nANKðo1:njnÞoc iff o is computable: In [20], a different kind of complexity is used, but the proof holds for the preﬁx complexity used in this paper. In view of the results of Section 6, we see that almost H. Takahashi, K. Aihara / Journal of Complexity 19 (2003) 132–152 145 all sequences generated by an irrational rotation are non-computable. Moreover, we have the following result:

Theorem 7.1. Let Tr be an irrational rotation given in Deﬁnition 3.1. r is a non- computable real number iff all sequences o generated by Tr are non-computable.

See proof in the appendix.

8. Discussion

Comparing the results of Section 6.1 with these of Section 6.2, we observe that Kolmogorov complexity with a non-computable rotation number is larger than one with a computable rotation number. The increment of Kolmogorov complexity corresponds to the information on a non-computable rotation number needed to simulate the dynamics. Moreover by Theorem 7.1, we see that all sequences generated by an irrational rotation (i.e. Sturmian sequences) are non-computable when the parameter is non- computable. Although Kolmogorov complexity of such sequences is low in comparison to that of chaotic sequences, Theorem 7.1 is an important characteristic of the present model because many dynamics generally generate not only non- computable sequences but also computable sequences even if the parameters are not computable or the dynamics is chaotic. Note that, in general, it is hard to decide whether the observed sequence is computable or not.

Acknowledgments

We thank Prof. Takashi Tsuchiya of The Graduate University for Advanced Studies (Tokyo), Prof. Teturo Kamae (Osaka City University), Prof. Shin-ichi Yasutomi (Suzuka), Prof. Vale´ rie Berthe´ (Montpellier), and Prof. Shunji Ito (Tsuda College) for thoughtful discussions and comments.

Appendix

Proof of Lemma 3.1. Recall that Sr ¼ Ar TÀ1Ar ? T Ànþ1Ar : Since each o1:n o1 - r o2 - - r on interval TÀkAr ðk ¼ 0; y; n À 1Þ has end points of Àðk þ 1Þr mod 1 and r okþ1 Àkr mod 1; Sr is one of the intervals partitioned by Àk r mod 1; o1:n 1 Àk2r mod 1; k1; k2Af1; y; ng: Then, by moving r; we obtain the lemma. &

Proof of Lemma 3.2. Conditions (a) and (b) follow from Lemma 3.1. To prove (c), r let Rn ¼fl=t j 0plpn; 1ptpn; l; tANg: If rARn and the denominator of r is t; T is periodic with period t; i.e., mðSr Þ¼1=t for Sr a|: By Lemma 3.1, the o1:n o1:n maximum-likelihood estimator r#ARn and hence (c) holds. & 146 H. Takahashi, K. Aihara / Journal of Complexity 19 (2003) 132–152

We use the following lemma for the proof of Theorem 4.1.

Lemma A.1 (Li and Vitangi [19]). Let P be a measure on inﬁnite sequences. If there is n a rational-valued computable function A such that 8nAN8xAf0; 1g Aðx; nÞpPðxÞ and limn-NAðx; nÞ¼PðxÞ; then P is computable.

A proof is given in [19, p. 246, Example 4.3.2].

Proof of Theorem 4.1. First we show the ‘‘if’’ part. If r is computable, then for any tAN and any xAQ; TtðxÞ is computable, i.e., there is a rational-valued computable t 1 function A such that jAðx; t; kÞÀT ðxÞjpk: Since 1 À r is also computable, there is a 1 rational-valued computable function B such that jBðkÞÀ1 þ rjpk: 1 2 t A 3 A Àt So if kpAðx; t; kÞoBðkÞÀk then T ðxÞ A0ð x T A0Þ:

2 1 t A 3 A Àt And if BðkÞþkpAðx; t; kÞp1 À k then T ðxÞ A1ð x T A1Þ: Let x ¼ i ði ¼ 0; y; k À 1Þ; i k

t 1 2 S ¼ xi pAðxi; t; kÞoBðkÞÀ ; 0;k k k

t 2 1 S ¼ xi BðkÞþ pAðxi; t; kÞp1 À ; 1;k k k and S ¼ S0 S1 ? SnÀ1 : o1:n;k o1;k- o2;k- - on;k C Then So1:n;k So1:n ; infSo1:n ¼ limk-N min So1:n;k; and sup So1:n ¼ limk-N max So1:n;k: C As So1:n is connected, ½min So1:n;k; max So1:n;k So1:n ; thus by Condition 4.1, limk-Nmð½min So1:n;k; max So1:n;kÞ ¼ mðSo1:n Þ and mð½min So1:n;k; max So1:n;kÞ is computable. Hence from Lemma A.1, Prðo1:nÞ¼mðSo1:n Þ is computable. Now we show the ‘‘only if’’ part. If Pr is a computable function, then Prð0Þ is a computable real and there is a rational-valued computable function C such that A 1 8k NjCðkÞÀPrð0Þjpk: Also, from the ‘‘if’’ part proof above, we see that Pr is a computable function if r is a rational number, i.e., there is a rational-valued A A 1 computable function D such that 8k N8r QjDðr; kÞÀPrð0Þjpk: Since Prð0Þ¼ m½0; 1 À rÞ and by Condition 4.1, Prð0Þ is a continuous and strictly decreasing function of r: Therefore the following binary search algorithm can compute r for 1 any given precision s when r is not a rational number. Step 1: k :¼ 1; r0 :¼ 0; r00 :¼ 1: Step 2: k :¼ k þ 1: r0þr00 2 00 r0þr00 Step 3: if Dð 2 ; kÞþkpCðkÞ then r :¼ 2 : r0þr00 X 2 0 r0þr00 Step 4: if Dð 2 ; kÞ CðkÞþk then r :¼ 2 : 0 00 1 0 Step 5: if jr À r jps then output r and quit, otherwise go to step 2. H. Takahashi, K. Aihara / Journal of Complexity 19 (2003) 132–152 147

If r is a rational number then r is a computable real, and this completes the proof. &

We remark the following simple fact, before proof of Theorem 6.1 (b)–(d) which is used to derive Theorem 6.1. If we set Sr ¼fo jSr a|g and r is irrational, n 1:n o1:n then r jSn j¼n þ 1 for all n; ðA:1Þ r r r r r where jSn j is the number of elements in Sn ; because jSnþ1j¼jSn jþ1 and jS1 j¼2 (see Fig. 2 or [4]).

Proof of Theorem 6.1. (a) Due to Theorem 4.1, Pr is computable since r is computable. Hence the result follows by Theorem 5.1(b). (b) The result follows from (A.1), Jensen’s inequality, and (a) of this theorem. (c) If x is an interior point of Sr and r is computable, then from the proof of the o1:n ‘‘if’’ part of Theorem 4.1, there is a computable function Wðx; n; kÞ that computes o1:n for sufﬁciently large k; otherwise output error and quit. So the following r algorithm enumerates all o1:n in Sn when n is given. Step 1: k :¼ 1: Step 2: k :¼ k þ 1: Step 3: for i ¼ 1tok i if W ; n; k ¼ o then output o : k 1:n 1:n Step 4: go to Step 2. r For any k; 1pkpjSn j; we are able to compute the kth enumerated string o1:n without repetition. The result follows from (A.1). (d) We use the following fact [19]: for a positive integer c and each y; every ﬁnite set A with the cardinality d has at least dð1 À 2ÀcÞþ1 elements x with KðxjyÞXlog d À c: For the proof, see [19, p. 109, Theorem 2.2.1]. Applying the fact above to (A.1), we obtain the proof. &

r A ri ri i Proof of Theorem 6.2. (a) Let ri ¼ i=k for 0pipk À 1; k N and ½l ; h Þ¼So1:n : By Lemma 3.1, Sr C½lri þ a; hri þ bÞ ½0; 1Þ; jaj; jbj n=k for rA½r ; r þ 1=kÞ; and o1:n - p i i hence Sri WSr Cð½lri À n=k; lri þ n=k ½hri À n=k; hri þ n=kÞ ½0; 1Þ; ðA:2Þ o1:n o1:n , - c where AWB ¼ðA,BÞ-ðA-BÞ : Thus jP ðo ÞÀP ðo Þj ¼ jmðSri ÞÀmðSr Þj ri 1:n r 1:n o1:n o1:n mðSri WSr Þ p o1:n o1:n p mð½lri À n=k; lri þ n=k-½0; 1ÞÞ þ mð½hri À n=k; hri þ n=k-½0; 1ÞÞ; ðA:3Þ 148 H. Takahashi, K. Aihara / Journal of Complexity 19 (2003) 132–152 where the ﬁrst inequality follows from jmðAÞÀmðBÞjpmðAWBÞ for arbitrary measurable sets A and B (see [16, p. 260]), and the second inequality follows from (A.2). P

Let Aðo1:n; kÞ¼ i Pri ðo1:nÞðFðri þ 1=kÞÀFðriÞÞ: By Theorem 4.1, Aðo1:n; kÞ is computable, and by (A.3), Z 1

Aðo1:n; kÞÀ Prðo1:nÞ dFðrÞ 0 p maxfmð½lri À n=k; lri þ n=k-½0; 1ÞÞ i þ mð½hri À n=k; hri þ n=k-½0; 1ÞÞg: ðA:4Þ

r r r r Let Bðo1:n; kÞ¼maxifmð½l i À n=k; l i þ n=k-½0; 1ÞÞ þ mð½h i À n=k; h i þ n=k- ½0; 1ÞÞg: Bðo1:n; kÞ is computable because ri is a rational number and m is computable. Also limk-NBðo1:n; kÞ¼0 by the continuity of m: Let nðkÞ be the least integer such that Bðo ; kÞ 1=k: Therefore from (A.4), 8k; jAðo ; nðkÞÞ À R 1:n p 1:n 1 P ðo Þ dFðrÞj 1=k: Since nðkÞ is computable, Aðo ; nðkÞÞ is computable, 0 r 1:n R p 1:n 1 and hence 0 Prðo1:nÞ dFðrÞ is computable. (b) Apply (a) to Theorem 5.1(b). &

We prove the following lemma before the proof of Theorem 6.3 and Corollary 6.3.

Lemma A.2. If Sr a|; let n ¼ IT kxm ¼ Ix þ rkm for xASr ; where Ix þ rkm is o1:n k r o1:n the greatest integer that is less than or equal to x þ rk: Then

Xk n0 ¼ 0; and nk ¼ oi for 1pkpn À 1: i¼1

3 k A Proof. It can be easily seen that n0 ¼ 0 and nkþ1 À nk ¼ 1 Tr x A1 for 0pkpn À 1 (see Fig. 2). By noting that o ¼ 13TkxAA for xASr ; we have the k r 1 o1:n lemma. &

Proof of Theorem 6.3. (a) If we can show that Prðo1:nÞ is concave as a function of r on the support Bo1:n ; then the theorem holds immediately.

Now we prove that Prðo1:nÞ is concave on Bo1:n : Let x and y be any real numbers 0 such that xASr and yASr : If o ¼ 0 then from Lemma A.2, x þ rðk À 1ÞÀ o1:n o1:n k A r 0 A r0 0 nkÀ1 A0 ¼½0; 1 À rÞ and y þ r ðk À 1ÞÀnkÀ1 A0 ¼½0; 1 À r Þ: Hence for 0pap1; 0 ð1 À aÞðx þ rðk À 1ÞÀnkÀ1Þþaðy þ r ðk À 1ÞÀnkÀ1Þ A½0; ð1 À aÞð1 À rÞþað1 À r0ÞÞ

0 ð1ÀaÞrþar0 ¼½0; 1 Àð1 À aÞr À ar Þ¼A0 : H. Takahashi, K. Aihara / Journal of Complexity 19 (2003) 132–152 149

On the other hand, 0 ð1 À aÞðx þ rðk À 1ÞÀnkÀ1Þþaðy þ r ðk À 1ÞÀnkÀ1Þ 0 ¼ð1 À aÞx þ ay þðð1 À aÞr þ ar Þðk À 1ÞÀnkÀ1: Hence, 0 A ð1ÀaÞrþar0 ð1 À aÞx þ ay þðð1 À aÞr þ ar Þðk À 1ÞÀnkÀ1 A0 :

In the same way, if ok ¼ 1 then 0 A ð1ÀaÞrþar0 ð1 À aÞx þ ay þðð1 À aÞr þ ar Þðk À 1ÞÀnkÀ1 A1 : Therefore for all xASr ; yASr0 and 0 a 1; o1:n o1:n p p 0 ð1 À aÞx þ ayASð1ÀaÞrþar : o1:n 0 0 If we set ½l; hÞ¼Sr ; ½l0; h0Þ¼Sr ; and S ¼fð1 À aÞx þ ay j xASr ; yASr g; o1:n o1:n o1:n o1:n then S ¼½ð1 À aÞl þ al0; ð1 À aÞh þ ah0Þ: Since P ðo Þ¼mðSr Þ; where m is the r 1:n o1:n uniform measure, ð1ÀaÞrþar0 P 0 ðo Þ¼mðS Þ ð1ÀaÞrþar 1:n o1:n X mðSÞ¼ð1 À aÞðh À lÞþaðh0 À l0Þ

0 ¼ð1 À aÞmðSr ÞþamðSr Þ o1:n o1:n

¼ð1 À aÞPrðo1:nÞþaPr0 ðo1:nÞ:

Therefore Prðo1:nÞ is concave on the support. This completes the proof. (b) This is a direct consequence of Lemma 3.2(b) and (c). (c) Let x be a real number such that xASr : o1:n If on ¼ 1; then from Lemma A.2,

x þðn À 1ÞrA½nnÀ1 þ 1 À r; nnÀ1 þ 1Þ

3xA½nnÀ1 þ 1 À nr; nnÀ1 þ 1 þð1 À nÞrÞ; P nÀ1 where nnÀ1 ¼ i¼1 oi: Hence, Sr ¼ Sr ½n þ 1 À nr; n þ 1 þð1 À nÞrÞ o1:n o1:nÀ1 - nÀ1 nÀ1

C½0; 1Þ-½nnÀ1 þ 1 À nr; nnÀ1 þ 1 þð1 À nÞrÞ: Therefore, Sr a| ) 14n þ 1 À nr; 0 n þ 1 þð1 À nÞr o1:n nÀ1 o nÀ1

3 nnÀ1=noroðnnÀ1 þ 1Þ=ðn À 1Þ and P nÀ1 n þ i¼1 oi Lo pðnnÀ1 þ 1Þ=ðn À 1ÞÀnnÀ1=n ¼ : 1:n nðn À 1Þ 150 H. Takahashi, K. Aihara / Journal of Complexity 19 (2003) 132–152

In the same way, if on ¼ 0; then P nÀ1 2n À 1 À i¼1 oi Lo p : 1:n nðn À 1Þ In both cases, we have 2n À 1 Lo p ; 1:n nðn À 1Þ P nÀ1 because 0p i¼1 oipn À 1: &

Proof of Corollary 6.3. Observe that the uniform distribution F and the uniform measure m satisﬁes the condition of Theorem 6.2. Conditions (a) and (c) follow from Theorems 6.2 and 6.3. Conditions (b) and (d) follow from Corollary 6.2 and Theorem 6.3. Condition (e) follows from Theorems 5.1(d) and 6.3(a), (c). &

Proof of Theorem 7.1. First we prove that if there is a computable sequence o generated by Tr; then r is computable. Let o be such a sequence and let A be a computable function such that AðnÞ¼o1:n for all n: Let m in Deﬁnition 3.1 be the uniform measure. By Theorem 4.1, Pa is computable for rational a; and hence there is a rational-valued computable function B such that jBða; o1:n; kÞÀPaðo1:nÞjp1=k; for all rational aA½0; 1; and kAN: Since the likelihood function Pa is continuous by Lemma 3.2 and is not identically zero by assumption, there is a rational number a such that Paðo1:nÞ40: Therefore, if o1:n is given, the following algorithm computes aðo1:nÞ such that Paðo1:nÞðo1:nÞ40: Step 1: k :¼ 0: Step 2: k :¼ k þ 1: Step 3: for i ¼ 0tok: Begin i a :¼ k; If Bða; o1:n; kÞÀ1=k40 then output a and quit; End Step 4: go to Step 2. 2nÀ1 X By Theorem 6.3(c), jaðo1:nÞÀrjpnðnÀ1Þ for n 2: Let nðkÞ be the least integer such 2nÀ1 A that nðnÀ1Þp1=k for k N then nðkÞ is computable. Therefore, the function aðAðnðkÞÞÞ is computable and jaðAðnðkÞÞÞ À rjp1=k for all kAN; i.e. r is computable. Conversely, if r is a computable real, let the initial value of Tr be 0: Then it is easily seen that the generated sequence is computable i.e., the converse holds. This completes the proof. &

References

[1] K. Aihara, T. Takabe, M. Toyoda, Chaotic neural networks, Phys. Lett. A 144 (6,7) (1990) 333–340. H. Takahashi, K. Aihara / Journal of Complexity 19 (2003) 132–152 151

[2] A.R. Barron, Logically smooth density estimation, Ph.D. Dissertation, Dept. Elec. Eng., Stanford Univ., Stanford, CA, September 1985. [3] J. Berstel, M. Pocchiola, Random generation of ﬁnite Sturmian words, Discrete Math. 153 (1996) 29–39. [4] V. Berthe´ , Sequences of low complexity: automatic and Sturmian sequences, in: F. Blanchard, A. Maass, A. Nogueria (Eds.), Topics in Symbolic Dynamics and Applications, London Mathematical Lecture Notes, Vol. 279, Cambridge University Press, Cambridge, 2000. [5] P. Billingsley, Ergodic Theory and Information, Wiley, New York, 1965. [6] A.A. Brudno, Entropy and the complexity of the trajectories of a dynamical system, Trans. Moscow Math. Soc. 44 (2) (1983) 127–151. [7] E.R. Caianiello, Outline of a theory of thought process and thinking machines, J. Theoret. Biol. 1 (1961) 204–235. [8] G.J. Chaitin, A theory of program size formally identical to information theory, J. ACM 22 (1975) 329–340. [9] L. Chen, K. Aihara, Global searching ability of chaotic neural networks, IEEE Trans. Circuits Systems I Fund. Theory Appl. 46 (8) (1999) 974–993. [10] J. Ford, How random is a coin toss?, Phys. Today 36 (1983) 40–47. [11] M. Hata, Dynamics of Caianiello’s equation, J. Math. Kyoto Univ. 22 (1982) 155–173. [12] M. Hata, Chaos in Neural Network Models, Asakura, Tokyo, 1998 (in Japanese). [13] T. Kamae, private communication, August 2001. [14] T. Kamae, H. Takahashi, Statistical problems related to irrational rotations, preprint. [15] A.N. Kolmogorov, Three approaches to the quantitative deﬁnition of information, Problems Inform. Transmission 1 (1) (1965) 1–7. [16] A.N. Kolmogorov, S.V. Fomin, Introductory Real Analysis, Dover, New York, 1975 (revised English edition translated and edited by R.A. Silverman). [17] S.K. Leung-Yan-Cheong, T.M. Cover, Some equivalences between Shannon entropy and Kolmogorov complexity, IEEE Trans. Inform. Theory IT-24 (3) (1978) 331–338. [18] L.A. Levin, Laws of information conservation (nongrowth) and aspects of the foundation of probability theory, Probab. Inform. Transmission 10 (1974) 206–210. [19] M. Li, P.M.B. Vitanyi, An introduction to Kolmogorov Complexity and its applications, 2nd Edition, Springer, New York, 1997. [20] D.W. Loveland, A variant of the Kolmogorov concept of complexity, Inform. Control 15 (1969) 510–526. [21] G. Matsumoto, K. Aihara, Y. Hanyu, N. Takahashi, S. Yoshizawa, J. Nagumo, Chaos and phase locking in normal squid axons, Phys. Lett. A 123 (4) (1987) 162–166. [22] M. Morse, G.A. Hedlund, Symbolic dynamics II. Sturmian trajectories, Amer. J. Math. 62 (1940) 1–42. [23] J. Nagumo, S. Sato, On a response characteristic of a mathematical neuron model, Kybernetik 10 (1972) 155–164. [24] M. Pollicott, M. Yuri, Dynamical Systems and Ergodic Theory, Cambridge University Press, Cambridge, 1998. [25] M.B. Pour-El, J.I. Richards, Computability in Analysis and Physics, Springer, New York, 1988. [26] J. Rissanen, Universal coding, information, prediction, and estimation, IEEE Trans. Inform. Theory IT-30 (4) (1984) 629–636. [27] S. Smale, Some remarks on the foundations of numerical analysis, SIAM Rev. 32 (2) (1990) 211–220. [28] R.J. Solomonoff, A formal theory of inductive inference, Inform. Control 7 (1–22) (1964) 224–254. [29] H. Takahashi, Parametric models and Kolmogorov complexity, Ph.D. Thesis, The Graduate University for Advanced Studies, Tokyo, March 2002. [30] J.F. Traub, A.G. Werschulz, Complexity and Information, Cambridge University Press, Cambridge, 1998. 152 H. Takahashi, K. Aihara / Journal of Complexity 19 (2003) 132–152

[31] S. Yasutomi, The continued fraction expansion of a with mðaÞ¼3; Acta Arith. 84 (4) (1998) 337–374. [32] A.K. Zvonkin, L.A. Levin, The complexity of ﬁnite objects and the development of the concepts of information and randomness by means of the theory of algorithms, Russian Math. Surveys 25 (6) (1970) 83–124.