Evolutionary Game Dynamics and the Moran Model

U.U.D.M. Project Report 2020:39

William Norman

Examensarbete i matematik, 15 hp Handledare: Ingemar Kaj Examinator: Martin Herschend Juni 2020

Department of Mathematics Uppsala University

Evolutionary Game Dynamics and the Moran Model

William Norman

1 Contents

1 Introduction 3

2 Evolutionary Game Theory 3

3 The Replicator Dynamics 5 3.1 Two Strategies: Generic Cases ...... 7 3.2 Rock-Paper-Scissors ...... 8

4 The Moran Model 9 4.1 Fixation Probabilities and Times ...... 11 4.1.1 Fixation Probabilities ...... 11 4.1.2 Fixation Times ...... 14 4.2 The Moran Process and Weak Selection ...... 16 4.3 Moran Process with Constant Fitness ...... 21

5 Evolutionary Graph Theory and the Spatial Moran Model 23 5.1 The Isothermal Theorem ...... 24 5.2 Suppressing and Amplifying Selection ...... 26 5.2.1 Suppressors ...... 27 5.2.2 Ampliﬁers ...... 27

6 Conclusion 28

2 1 Introduction

Evolutionary game theory is a branch of game theory used for modelling populations under recurrent strategic interactions. Although originally intended for evolutionary biology the theory has found widespread applications such as solving social dilemmas, contributing to the understanding of evolution of language, financial risk decision-making and creating neural networks. Evolutionary game dynamics, which is the focus of this thesis, is the part of population modelling that describes the manner in which strategies spread among the population. A strategy can be interpreted in many different ways. In biology we may interpret a strategy as a gene, a phenotype or a species; in economics, an investment strategy; in social science, a behaviour. Evolutionary game dynamics act on populations. It is populations that evolve, not strategies. The aim of this thesis is to introduce the reader to the theory of evolutionary game dynamics by showcasing two commonly used dynamics. In section 2 we introduce basic concepts of evolutionary game theory. In section 3 we consider the deterministic replicator dynamics, a system of ordinary non- linear differential equations. In section 4 we consider stochastic dynamics in the form of birth-death processes, in particular, we consider the Moran process. In chapter 5 we develop a framework for analyzing how population structure influences evolution under the spatial Moran process.

2 Evolutionary Game Theory

Game theory is used to model strategic interactions among rational players acting to maximize payoffs by finding optimal strategies in a single game. Further, each player is assumed to have full information of the game rules. In contrast to the strict assumptions of game theory, evolutionary game theory adopts a philosophy based on survival of the fittest and trial and error. Individuals with bounded rationality in a population act to the best of their abilities with limited understanding of the world (using their strategy). Fur- ther, instead of a single game we consider repeated games called evolutionary games. After each game, the proportions of strategies in the population is updated according to payoff comparisons and the choice of game dynamics. This shift allows us to use the underlying framework of game theory and expand upon it to model evolutionary games between, for example, genes which evolve by trial-and-error rather than by a rational approach.

3 Definition 2.1. A payoff matrix U for a game with m strategies is a m × m matrix with real-valued entries called payoffs. Payoff uij represents the reward for an individual with strategy i interacting with an individual with strategy j.

Example 2.1. The following is a payoﬀ matrix for a (symmetric) two player game with strategies A and B

AB . A a b B c d

We interpret a and d as the payoffs of A and B self-interacting, respectively. b is the payoff of A interacting with B and c is the payoff of B interacting with A. In this thesis we consider only symmetric, normal-form games. Normal- form meaning that each game is completely determined by a payoff matrix. Symmetric meaning that payoffs are dependent solely on the employed strategy, not on who employs the strategy.

Deﬁnition 2.2. For a population with m strategies, the population state is a stochastic vector x = (x1, x2, ..., xm) where entry xi is the proportion of strategy i in the population.

It is standard practice in traditional game theory to analyze how rational players will behave through static solution concepts such as the Nash equilibrium. A Nash equilibrium is a game state in which no player has an incentive to switch strategy in the sense that no action results in an incremental beneﬁt assuming all other players stick to their strategy.

Deﬁnition 2.3. Let U be the payoﬀ matrix of a game. A population state x is a Nash equilibrium (NE) if for all other population states y

xT Ux ≥ yT Ux.

We call x a strict NE if the inequality is strict for all population states y 6= x. We call x a weak NE if the inequality holds for all states but there is an equality for some population state y 6= x.

4 Definition 2.4. Let U be the payoff matrix of a game. A population state x is a evolutionary stable state (ESS) if for all population states y 6= x either of the following conditions hold 1. xT Ux > yT Ux (strict NE condition), 2. xT Ux = yT Ux and xT Uy > yT Uy (stability against other strategies). Remark. From the definitions we get Strict NE =⇒ ESS =⇒ Weak NE.

3 The Replicator Dynamics

Assume that there is a large population with m strategies. The population is well-mixed or structureless, meaning that the probability that an individual using strategy i interacts with j is simply the proportion of strategy j. We define the population space, the set of possible population states, as m m Pm the m-simplex: ∆ = {x ∈ [0, 1] : i=1 xi = 1}. Let U be the payoff matrix for an evolutionary game. The expected payoff for an individual with strategy i interacting with a random individual in the population is given by m X (Ux)i = uijxj. (1) j=1 The average payoff for an individual chosen at random interacting with a random individual in the population is given by m m m T X X X x Ux = xi(Ux)i = xiuijxj. (2) i=1 i=1 j=1 Assume that proportions are differentiable with respect to time t and denote the derivative of xi(t) asx ˙i. This results in continuous evolution (change in proportions). Let evolution be governed by the continuous time replicator equation T x˙i = xi[(Ux)i − x Ux] i = 1, 2, ..., m. (3) Under the replicator dynamics, evolution is governed by m − 1 non-linear ordinary differential equations that encapsulates the notion of ’survival of the fittest’ as seen by T x˙i > 0 ⇐⇒ (Ux)i > x Ux, T x˙i = 0 ⇐⇒ (Ux)i = x Ux or xi = 0, (4) T x˙i < 0 ⇐⇒ (Ux)i < x Ux.

5 Further, the simplex is invariant under the replicator dynamics,5 meaning that trajectories starting inside the simplex remain inside the simplex forever. To determine the evolutionary outcome of an evolutionary game governed by the replicator dynamics we borrow concepts from the theory of dynamical systems.

Deﬁnition 3.1. A point x∗ is an equilibrium point of the diﬀerential equation x˙ = f(t, x) if f(t, x∗) = 0 for t ≥ 0.

Deﬁnition 3.2. Let d be the euclidean metric. An equilibrium point x∗ of x˙ = f(t, x) with initial condition x(0) = x0 is stable if ∀ > 0, ∃δ > 0 : ∗ ∗ d(x0, x ) < δ =⇒ d(x(t), x ) < , ∀t ≥ 0. An equilibrium point is unstable if it is not stable. Further, an equilibrium point is asymptotically stable if it is stable and the state of the system converges to x∗ as t → ∞.

Remark. For a time-invariant, non-linear system of ordinary diﬀerential equations, such as the replicator system of equations, we may determine the stability of an equilibrium point by considering the Jacobian matrix J(x) of the system evaluated at the equilibrium point; if J(x∗) has only strictly negative eigenvalues then x∗ is asymptotically stable; if at least one eigenvalue is positive then it is unstable. The following theorem connect equilibrium concepts of game theory and dynamical systems making deterministic games easier to analyze.

Theorem 3.1. 1,5 In an evolutionary game, the following is true for a population state x∗ under the replicator dynamics.

1. If x∗ is a stable equilibrium point then x∗ is a NE,

2. If x∗ is a NE then x∗ is a equilibrium point,

3. If x∗ is a ESS (or strict NE) then x∗ is asymptotically stable equilibrium point.

6 3.1 Two Strategies: Generic Cases Consider a game of two strategies A and B given by the payoﬀ matrix

AB . A a b B c d

Denote x = xA = 1 − xB. Evolution in this case is determined by the single replicator equation

T x˙ = x[(Ux)1 − x Ux] = x[ax + b(1 − x) − x(ax + b(1 − x)) − (1 − x)(cx + d(1 − x))] (5) = x(1 − x)[(a − b − c + d)x + b − d]. The equation has trivial equilibrium points x∗ = 0 and x∗ = 1. If also ∗ d−b [a > c ∧ d > b] or [a < c ∧ d < b] then x = a−b−c+d is also an equilibrium point in the open interval (0, 1). To determine the stability of the equilibrium points, consider the Jacobian of the system dx˙ d J(x) = = x(1 − x)[(a − b − c + d)x + b − d] dx dx (6) = 3x2(−a + b + c − d) + 2x(a − 2b − c + 2d) + b − d. Analyzing the stability of the equilibrium points we deduce the following. • J(0) = b − d =⇒ x∗ = 0 is stable if d > b and unstable if b > d. • J(1) = c − a =⇒ x∗ = 1 is stable if a > c and unstable if c > a.

d−b (a−c)(d−b) ∗ d−b • J( a−b−c+d ) = a−b−c+d =⇒ x = a−b−c+d is stable if [c > a ∧ b > d] and unstable when [a > c ∧ d > b]. Based on this analysis we arrive at the following generic two player game cases. • Dominance. A dominates B when [a > c ∧ b > d] that is when A is a strict NE and hence xA = 1 is asymptotically stable. All B individuals go extinct as long as there is at least one individual of strategy A. Similarly, B dominates A when [c > a ∧ d > b]. • Bistability. If [a > c ∧ d > b] then the interior equilibrium point ∗ d−b x = a−b−c+d is unstable and xA = 1, xA = 0 are both strict NE hence ∗ asymptotically stable equilibrium points. If xA(0) > x then A over- ∗ takes the population, if xA(0) < x then B overtakes the population.

7 • Coexistence. If [c > a ∧ b > d] then the interior equilibrium point ∗ d−b x = a−b−c+d is stable hence a NE. Neither xA = 1 nor xA = 0 are Nash equilibria. The evolutionary outcome is a mix of both strategies with proportions decided by x∗.

• Neutrality. If [a = c ∧ b = d] then the proportion of strategies remain constant. The whole set [0, 1] consists of Nash equilibria.

3.2 Rock-Paper-Scissors Consider a game of rock-paper-scissors (RPS). Here each strategy dominates one other strategy and get dominated by the yet another another, in other words the game exhibit cyclic dominance. Let U be the payoﬀ matrix

RPS R 0 -1 1 . P 1 0 -1 S -1 1 0

Let xR, xP , xS denote the proportions of individuals using the corresponding strategy. Note that

xT Ux = x (x − x ) + x (x − x ) + x (x − x ) R p S P S R S R P (7) = 0, which is the case for all zero-sum games (games where the gain of one individual is the loss of another). Thus the replicator system of equations can be written as           x˙ R xR 0 −1 1 xR xR(xS − xP ) x˙ P  = xP   1 0 −1 xP  = xP (xR − xS) (8) x˙ S xS −1 1 0 xS xS(xP − xR)

The proportion of a strategy increases if there are more opponents that the strategy can beat than can be beaten by. For this system (1, 0, 0), (0, 1, 0), (0, 0, 1) are unstable equilibrium points, further, (1/3, 1/3, 1/3) is a stable equilibrium point hence a NE, but it is not asymptotically stable equilibrium point hence not an ESS.

8 (a)

(b) (c)

Figure 1: Phase plot of proportions. RPS matrix was used unless stated otherwise. (a) Stable interior equilibrium point. (b) Asymptotically stable interior equilibrium point as (u31 is changed from −1 to −0.5). (c) Unstable interior equilibrium point as (u23 is changed from −1 to −1.5). Figures were created using tools on the website EvoLudo.

4 The Moran Model

The assumption of differentiable proportions implies that the replicator dynamics describe an ’infinite population’.6 It is not clear before analysis when the replicator dynamics gives a good approximation of a finite, stochastic population. We will now consider stochastic modelling. For simplicity, we consider the case of two strategies for the remainder of this thesis.

9 Definition 4.1. Given a payoff matrix U, the fitness function of strategy i is defined to be a positive, convex combination of background fitness and expected payoff denoted

fi(w) = 1 − w + w(Ux)i where w ∈ [0, 1] is called the intensity of selection. For increasing values of w, the payoff matrix has increasing influence on fitness.

• If w = 0 we call it neutral selection because the payoff matrix has no influence on fitness thus all strategies are deemed equal by selection.

• If 0 < w 1 we call it weak selection because the payoﬀ matrix has a weak inﬂuence on selection.

Remark. If payoffs are allowed to be negative then either w has to be chosen such that fitness is positive (weak selection guarantees that fitness is positive regardless of payoff matrix) or the fitness function can be redefined to fi = + exp [w(Ux)i] where w ∈ R . Weak selection is an evolutionary theory explaining the maintenance of different strategies in a large population where only small fitness differences exists. We may capture this mathematically by letting intensity of selection be a function of the population size such that w(N) → 0 as N → ∞.

Deﬁnition 4.2 (The Moran Process). At each time-step, one individual is selected to reproduce an exact copy at random proportional to ﬁtness. One is sampled uniformly random for death. Both events may occur to the same individual resulting in no change. The process is repeated at discrete time-steps until one strategy’s lineage succeeds in overtaking the population.

One may think of the Moran process as a random walk that is pulled in the direction of whichever strategy is more fit given the population state. For decreasing intensity of selection the Moran process drives evolution more by sheer randomness rather than by fitness. For neutral selection w = 0 the Moran process is reduced to a symmetric random walk. The Moran process is a time-homogeneous Markov process, specifically, for two strategies a one-dimensional Markov process. The number of individuals in a strategy may change by at most one individual at a time-step and the population size stays constant. We conclude that the process is a certain type of Markov chain called a birth-death process with tri-diagonal transition matrix.

10 4.1 Fixation Probabilities and Times For the replicator dynamics we used equilibrium concepts to analyze the outcome of an evolutionary game. For stochastic populations we answer the following two key questions:

1. What is the probability of a strategy becoming extinct or ﬁxated in a population?

2. How long does the process take?

First we will answer these questions for general birth-death processes with tri-diagonal transition matrices.

4.1.1 Fixation Probabilities For an evolutionary game of two strategies A and B. Let N be the size of the population and j be the number of A individuals and thus N − j the number of B individuals. Birth-death process of the sort considered has state space S = {0, 1, ..., N} with {0,N} being absorbent states (if the process reaches either state it will stay there forever) and all intermediate states {1, 2..., N − 1} being transient states (the process eventually never returns to these states). From the theory of Markov chains we get that the probability of the process reaching either absorbent state (in ﬁnite time) is 1 because the state space is ﬁnite. We conclude that co-existence is not possible in this model.

• Denote the ﬁxation probability of j A individuals as φj. That is the probability that j A individuals succeed in overtaking the population.

• Denote the transition probability that the number of A individuals increase from j to j + 1 by pj,j+1. Similarly the transition probability that the number of A individuals decrease from j to j − 1 by pj,j−1. For the process considered we can state the following facts about the transition probabilities

pj,j+1 > 0, pj,j−1 > 0, pj,j = 1 − pj,j−1 − pj,j+1 j ∈ {1, 2..., N − 1},

p0,0 = pN,N = 1,

pi,j = 0 else. (9)

11 Further, because a non-existent strategy can not reproduce and a ﬁxated strategy already is ﬁxated

φ = 0, 0 (10) φN = 1.

For the intermediate states j, by conditioning on the next step

φj = pj,j−1φj−1 + (1 − pj,j−1 − pj,j+1)φj + pj,j+1φj+1, (11) which may be rearranged to

pj,j−1 φj+1 − φj = (φj − φj−1). (12) pj,j+1

pj,j−1 Introducing the new notation yj = φj − φj−1, γj = we may rewrite pj,j+1 recursive relation (12) as

yj+1 = γjyj. (13)

Using this recursive relation we may derive

y1 = φ1 − φ0 = φ1, k−1 Y yk = φk − φk−1 = φ1 γi, k = 2, 3, ..., N − 1, i=1 (14) N−1 Y yN = φN − φN−1 = 1 − φ1 γi. i=1 From this we get the telescoping sum

j X y = (φ − φ ) + (φ − φ ) + ... + (φ − φ ) k 1 0 2 1 j j−1 (15) k=1

= φj − φ0 = φj, in particular

N X yk = (φ1 − φ0) + (φ2 − φ1) + ... + (φN − φN−1) (16) k=1

= φN − φ0 = 1.

12 Using equations (14) and (16)

N N k−1 N−1 k X X Y X Y 1 = yk = φ1 γi = φ1[1 + γi]. (17) k=1 k=1 i=1 k=1 i=1

Solving for φ1 we get an expression for the ﬁxation probability of a single A individual 1 φ = . (18) 1 PN−1 Qk 1 + k=1 i=1 γi Using (15) and (18) we may solve for the ﬁxation probability of any state j

j j k−1 0 j k−1 X X Y Y X Y φj = yk = φ1 γi = φ1 γi + φ1 γi k=1 k=1 i=1 i=1 k=2 i=1 | {z } =1 j k−1 j−1 k X Y X Y (19) = φ1(1 + γi) = φ1(1 + γi) k=2 i=1 k=1 i=1 1 + Pj−1 Qk γ = k=1 i=1 i . PN−1 Qk 1 + k=1 i=1 γi

Definition 4.3. Let ρA = φ1 and ρB = 1 − φN−1 (the probability that N − 1 A individuals fails to fixate) be the mutant fixation probabilities of strategy A and B respectively.

In the case of neutral selection: fA = fB =⇒ pj,j−1 = pj,j+1 =⇒ γj = i 1 for all intermediate states j =⇒ φi = N (same as risk of ruin for a simple random walker on {0, 1, ..., N} starting at j!). For this reason we are interested whether ρA > 1/N or not. A comparison of ρA and ρB is also of interest because it in the favor of which strategy the process spends more time, the system spends more time in whichever strategy’s corresponding ρ is greater (because that strategy needs

13 less invasion attempts to ﬁxate).4

PN−2 Qk 1 + γj ρ = 1 − φ = 1 − k=1 j=1 B N−1 PN−1 Qk 1 + k=1 j=1 γj PN−1 Qk PN−2 Qk 1 + γj 1 + γj = k=1 j=1 − k=1 j=1 PN−1 Qk PN−1 Qk 1 + k=1 j=1 γj 1 + k=1 j=1 γj =0 z }| { N−2 N−1 N−2 N−1 (20) QN−1 X Y X Y j=1 γj + γj − γj j=1 j=1 = k=1 k=1 PN−1 Qk 1 + k=1 j=1 γj N−1 Y = ρA γj. j=1

To compare the mutant ﬁxation probabilities we only need to calculate QN−1 j=1 γj. Deﬁnition 4.4.

• If ρA > 1/N (or if ρA < 1/N) then we say that A is advantageous (disadvantageous) to neutral selection. QN−1 • If ρB/ρA = j=1 γj < 1 (or if ρB/ρA > 1) we say that A is advantageous (disadvantageous) to B.

4.1.2 Fixation Times

Let Xn be the number of A individuals at time n. Denote the expected time until A ﬁxates or becomes extinct starting from state j as

tj = E[min{n ≥ 0 : Xn ∈ {0,N}|X0 = j}].

By deﬁnition t0 = tN = 0. For the intermediate states, conditioning on the next step

tj = 1 + pj,j−1tj−1 + (1 − pj,j−1 − pj,j+1)tj + pj,j+1tj+1, which may be rearranged to 1 tj+1 − tj = γj(tj − tj−1) − . (21) pj,j+1

14 pj,j−1 Denoting zj = tj − tj−1 and γj = we may rewrite recursive relation pj,j+1 (21) as 1 zj+1 = γjzj − . (22) pj,j+1 Using this recursive relation we may derive

z1 = t1 − t0 = t1,

zk = tk − tk−1 k−1 k−1 k−1 (23) Y X 1 Y = t1 γm − γm, k = 2, 3, ..., N − 1. pl,l+1 m=1 l=1 m=l+1

Consider the telescoping sum

N X zk = (tj+1 − tj) + (tj+2 − tj+1) + ... + ( tN −tN−1) |{z} (24) k=j+1 =0

= −tj.

For j = 1,

N N−1 k N−1 k k X X Y X X 1 Y t1 = − zk = −t1 γm + γm, (25) pl,l+1 k=2 k=1 m=1 k=1 l=1 m=l+1 which may be rewritten as

N−1 k k 1 X X 1 Y t1 = N−1 k γj P Q pl,l+1 1 + k=1 j=1 γj k=1 l=1 j=l+1 (26) N−1 k k X X 1 Y = φ1 γj. pl,l+1 k=1 l=1 j=l+1

Finally, using, (23), (24) and (26) we get an expression for the ﬁxation times of any state j

N N−1 k N−1 k k X X Y X X 1 Y tj = − zk = −t1 γm + γm. (27) pl,l+1 k=j+1 k=j m=1 k=j l=1 m=l+1

15 In the case of neutral selection when w = 0 the following holds γj = 1 =⇒ 1 l(N − l) Q γ = 1 =⇒ φ = and p = . Therefore, j j 1 N l,l+1 N 2

N−1 k large N X X N t = = NH z}|{≈ N ln N, (28) 1 l(N − l) N k=1 l=1 1 where H = PN is the Nth harmonic number. For the intermediate N k=1 k states

N−1 k N−1 k X Y X X N 2 t = −t γ + j 1 m l(N − l) k=j m=1 k=j l=1 N−1 k N−1 k X X N X X N 2 = −(N − j) + (29) l(N − l) l(N − l) k=1 l=1 k=j l=1 j−1 k X X N 2 = jNH − N l(N − l) k=1 l=1 We recall that co-existence is not possible in the Moran model due to intermediate states being transient. However, a sufficiently large expected fixation time may indicate co-existence. By similar calculations as above we may find the conditional expected A fixation times tj , the expected time it takes for j A-individuals to overtake the population given that it finally does. Here we only state the result

N−1 k k A X X φl Y t1 = γm pl,l+1 k=1 l=1 m=l+1 (30) N−1 k N−1 k k A A φ1 X Y X X φl 1 Y 7 tj = −t1 γm + γm. φj φj pj,j+1 k=j m=l k=j l=1 m=l+1

4.2 The Moran Process and Weak Selection For our choice of ﬁtness function we may obtain analytical results for the transition probabilities only in the case of weak selection by use of approximation.

16 Consider an evolutionary game with two strategies A and B. Let U be the following payoﬀ matrix

AB . A a b B c d

Assume that there are j A individuals and hence N − j B individuals in the population. Assume that individuals may not self-interact. Denote the payoﬀs of strategy A and B respectively as

j j − 1 N − j πA = (Ux)A = a + b, N − 1 N − 1 (31) j N − j − 1 πj = (Ux) = c + d. B B N − 1 N − 1 Fitness is set to j j fA(w) = 1 − w + wπA, j j (32) fB(w) = 1 − w + wπB. By considering the independent events that A is selected for reproduction and B sampled for death we find the transition probabilities j jfA(w) N − j pj,j+1 = j j , jfA(w) + (N − j)fB(w) N j (33) (N − j)fB(w) j pj,j−1 = j j . jfA(w) + (N − j)fB(w) N The quotient of transition probabilities reduces to a quotient of fitness functions j j pj,j−1 fB(w) 1 − w + wπB γj(w) = = j = j . (34) pj,j+1 fA(w) 1 − w + wπA We will now find an approximation for the fixation probabilities under weak selection. Consider the Taylor expansion of γj(w) at w = 0

∞ i (i) X w γj (0) γ (w) ≈ γ (0) + wγ0 (0) + . (35) j j j i! i=2 We evaluate the terms one by one. First, the constant term j fB(0) γj(0) = j = 1 (36) fA(0)

17 as ﬁtness is reduced to background ﬁtness. Second, the linear term

d f j (w) (f j )0(0)f j (0) − f j (0)(f j )0(0) wγ0 (0) = w B = w B A B A j dw j f 2 (0, j) fA(w) A (37) w = w(πj − πj ) ∼ . B A N For the higher order derivatives

∞ 2 X (i) w wiγ (0) ∼ w2(πj − πj ) ∼ . (38) j A B N i=2 With only the linear term of the Taylor expansion we have

j j γj(w) ≈ 1 − w(πA − πB). (39)

We may now approximate the product of γi,

k k Y Y j j γj(w) ≈ (1 − w(πA − πB)) j=1 j=1 k X j j 2 X j j = 1 − w (πA − πB) + O[w (πA − πB)] j=1 (40) k X j j ≈ 1 − w (πA − πB) j=1 k X (a − b − c + d) (−a + bN − dN + d) = 1 − w ( j + ). N − 1 N − 1 j=1

a−b−c+d −a+bN−dN+d Denote u = N−1 and v = N−1 . We may rewrite approximation (40) as

k k Y X (k + 1)k γ (w) ≈ 1 − w (uj + v) = 1 − w[u + vk] j 2 j=1 j=1 (41) u u = 1 − w[ k2 + ( + v)k]. 2 2 We may now derive an approximation for the mutant ﬁxation probability of

18 strategy A under weak selection 1 φ = 1 PN−1 Qk 1 + k=1 j=1 γj(w) 1 ≈ u u 1 + PN−1[1 − w( k2 + ( + v))k] k=1 2 2 1 = N(N − 1)(2N − 1) u N(N − 1) N − wu − w( + v) 12 2 2 1 w 2N − 1 = + ((a − b − c + d) − a − b − c + 3d + (2b − 2d)N) . N 4N 3 | {z } =Γ (42) We conclude that A is advantageous to neutral selection ⇐⇒ Γ > 0, (43) A is disadvantageous to neutral selection ⇐⇒ Γ < 0. Remark. In the limit of large populations γ(N) → a + 2b − c − 2d thus the above condition becomes very simple. For all states j 1 + Pj−1 Qk γ φ = k=1 i=1 i j PN−1 Qk 1 + k=1 i=1 γi u u 1 + Pj−1 [1 − w( k2 + ( + v)k)] k=1 2 2 ≈ u u 1 + PN−1[1 − w( k2 + ( + v)k)] k=1 2 2 j N − j j a − b − c + d −a + bN − dN + d = + Nw ( (N + j) + ). N N N 6(N − 1) 2(N − 1) (44) Now consider the mutant ﬁxation probability ratio

N−1 N−1 ρB Y X = γ ≈ 1 − w [πj − πj ] ρ j A B A j=1 j=1 u u = 1 − w[ (N − 1) + + v](N − 1) 2 2 w = 1 − [(a − b − c + d)(N − 1) − a − b − c + 3d + (2b − 2d)N] . 2 | {z } Ψ (45)

19 We conclude that A is advantageous to B ⇐⇒ Ψ > 0, (46) A is disadvantageous to B ⇐⇒ Ψ < 0.

Remark. In the limit of large populations Ψ(N) → a+b−c−d thus the above condition becomes very simple. In fact, it tells us that for large populations, being advantageous is equivalent to doing better on average. In the limit of large population sizes we notice a connection to the replicator dynamics d − b 1 x∗ = < ⇐⇒ a + b − c − d > 0. (47) a − b − c + d 2 We conclude that, d − b 1 ρ > ρ ⇐⇒ x∗ = < . (48) A B a − b − c + d 2 This equivalence shows a connection between advantageous strategies in the weak selection Moran process and having greater basins of attraction in the deterministic case of bistability. In the case when both a > c and b > d then A is the dominant strategy in the deterministic case which implies ρA > ρB. It is no surprise that there appears such a connections between the replicator dynamics and the Moran process with weak selection in the limit of large populations. Connections between the two dynamics have been studied and it has been shown that in the limit of large populations the deterministic replicator equation can be recovered from a stochastic differential equation arising from the Moran process.7 Interestingly, similar expressions for fixation probabilities as we have found here are obtained for a large variety of stochastic processes7 and fitness mappings under weak selection.2

20 4.3 Moran Process with Constant Fitness Assume that strategy A has ﬁtness r ∈ (0, ∞)\{1} and strategy B has ﬁtness 1. This assumption leads to the following simple transition probabilities rj N − j p = , j,j+1 rj + N − j N (49) N − j j p = . j,j−1 rj + N − j N The quotient of transition probabilities is simply

k pj,j−1 1 Y 1 γ = = =⇒ γ = . (50) i p r i rk j,j+1 i=1 The ﬁxation probabilities are

Pj−1 Qk 1 Pj−1 1 1 + k=1 i=1 r 1 + k=1 rk φj = = 1 + PN−1 Qk 1 1 + PN−1 1 k=1 i=1 r k=1 rk (51) 1 1 − rj = 1 . 1 − rN Specifically the mutant fixation probabilities take the form of the inverse of 1 the sum of finite geometric series with common ratio r and r respectively. 1 − 1 ρ = φ = r , A 1 1 − 1 rN (52) 1 − r ρ = 1 − φ = . B N−1 1 − rN

21 (a) (b)

(c)

Figure 2: RPS with Moran process as game dynamics showing proportions on the 3-simplex. (a) population: 1000, payoff matrix: RPS matrix from section 3.2, intensity of selection w = 1, time-steps before fixation: 5520. (b)  0 −1 0.8 population: 1000, payoff matrix: 0.7 0 −1, intensity of selection w = −1 3.4 0 0.5, time-steps before fixation: 8120. (c) Same case as in (b) but intensity of selection w = 0.05. This payoff matrix makes the interior equilibrium point asymptotically stable in the replicator dynamics. For the Moran model with the process trajectory is slightly pulled into the interior equilibrium point by selection. time-steps before fixation: 14660. Pictures were created using tools on the website EvoLudo.

22 5 Evolutionary Graph Theory and the Spatial Moran Model

In previous sections we assumed that the population be well-mixed. In section 3 this assumption influenced the expected payoffs and thus helped shape the replicator equation. In section 4 this assumption once again influenced the expected payoffs used in the definition of fitness and implicitly influenced which individuals may be selected for reproduction and sampled for death in the Moran process. In this section we will lose the assumption of the population being well-mixed and consider a framework for investigating the effects of population structure on selection in the Moran model. Consider a population of size N living on the vertices of a fixed directed graph, each vertex holds only one individual and no vertex is empty. Label the vertices i = 1, 2, ..., N corresponding to the individuals in the population. Let W be a stochastic matrix such that wij be the probability vertex j being replace by a copy of vertex i given that vertex i has been selected for reproduction. We call W the weight matrix. We encode the population structure in W as follows, if wij > 0 then there is a directed edge between vertex i and vertex j in the graph and we call vertex j a neighbor of vertex i. In this way we may define a graph by its population size N and matrix W that is G = (N,W ). 1 Remark. A graph G = (N,W ) where W fulfills wij = N for i, j = 1, 2, ...N is equivalent to a well-mixed population.

Deﬁnition 5.1 (The Spatial Moran process). At each time-step, one individual is selected to reproduce (an exact copy) at random proportional to ﬁtness. One of the selected individual’s neighbors are then sampled to be replaced by the copy with a probabilities given by the weight matrix W . The process is repeated at discrete time-steps until the mutant’s lineage overtakes the population or becomes extinct.

Assume that there is a homogeneous population of only B individuals on a graph. A mutation occurs such that a single A individual appears at a uniform random vertex. Does the graph structure inﬂuence the ﬁxation probability of this A mutant?

Definition 5.2. Let ρG be the fixation probability of a single mutant with fitness r ∈ (0, ∞) \{1} appearing at a uniform random vertex on graph G consisting of an otherwise homogeneous population with fitness 1.

23 Example 5.1. Consider a directed cycle graph with weight matrix

wi,i+1 = 1, i = 0, 1, ..., N − 1,

wN,0 = 1, (53)

wi,j = 0, else.

Let j be the number of mutants. Because the cluster of mutants may not break into separate parts and may only reproduce in one direction, we get the following transition probabilities r p = , j,j+1 N − j + rj 1 (54) p = . j,j−1 N − j + rj

This takes us back to the familiar case of (51) 1 p 1 1 − γ = j,j−1 = =⇒ ρ = r . (55) i p r G 1 j,j+1 1 − rN Interestingly, the spatial Moran process on the directed cycle graph yields the same mutant ﬁxation probability as the Moran process in the well-mixed population. We shall now investigate why.

5.1 The Isothermal Theorem Deﬁnition 5.3.

• A graph G is ρ-equivalent to the Moran process if

1 1 − r ρG = 1 . 1 − rN Remark. A graph is ρ-equivalent if the spatial Moran process on the graph yields the same mutant ﬁxation probability as the Moran process yields in a well-mixed population.

• The temperature of vertex j is the sum of all weights leading into vertex PN j denoted Tj = i=1 wij. Remark. The word temperature suggests that a vertex with higher temperature changes more often.

24 • If T1 = T2 = ... = TN we call the graph isothermal. Remark. The term isothermal is borrowed from thermodynamics. A graph being isothermal means that a vertex has the same eﬀect on its neighborhood as the neighborhood has on the vertex. Theorem 5.1 (The Isothermal Theorem). For a graph G = (N,W ) the following statements are equivalent: 1. G is isothermal,

2. G is ρ-equivalent to the Moran process,

3. the weight matrix W is doubly stochastic. Proof. First we show that (1 ⇐⇒ 3), then we show that (2 ⇐⇒ 3) and we are done. PN (3 =⇒ 1) : Assume W is doubly stochastic, that is i=1 wij = 1 for all vertices j. Trivially, the graph is isothermal. PN (1 =⇒ 3) : Assume that G is isothermal, that is i=1 wij = k for all vertices j in G and some constant k. Because W is a stochastic matrix we also have PN j=1 wij = 1. Now consider the following double sums

N N N X X X wij = 1 = N, i=1 j=1 i=1 (56) N N N X X X wij = k = Nk. j=1 i=1 j=1

PN PN PN PN Because ﬁnite sums are interchangeable i=1 j=1 wij = j=1 i=1 wij we PN PN conclude that k = 1 and so j=1 wij = i=1 wij = 1, that is W is doubly stochastic. T (1 ⇐⇒ 2) : Let v = (v1, v2, ..., vN ) be a binary vector where vi = 1 if there is a mutant at vertex i else vi = 0. The number of mutants in population is PN given by m = i=1 vi. Consider the transition probability that m increases by one,

PN PN r vi wij(1 − vj) p = i=1 j=1 , (57) m,m+1 rm + N − m Similarly, the transition probability that m decreases by one,

PN PN (1 − vi) wijvj p = i=1 j=1 . (58) m,m−1 rm + N − m

25 Remark. These transition probabilities are trickier than in the well-mixed case because who may be replaced by a copy is dependent on who is selected for reproduction. Now, consider the following chain of equivalences

1 1 − r ρG = 1 ⇐⇒ 1 − rN pm,m−1 1 γm = = ⇐⇒ (59) pm,m+1 r N N N N X X X X (1 − vi) wijvj = vi wij(1 − vj) i=1 j=1 i=1 j=1 which holds if and only if v is binary (as it is) and W is doubly stochastic.4

The most obvious type of isothermal graph G = (N,W ) are those with symmetric matrices W . Such graphs are bi-directed such that if a vertex i has an inﬂuence of vertex j, then vertex j has the same degree of inﬂuence of vertex i. It is seen that this type of graph is isothermal by noting that PN PN i=1 wij = j=1 wij = 1. This gives us the following simple corollary. Corollary 5.1.1. A graph G = (N,W ) with symmetric matrix W is ρ- equivalent.

5.2 Suppressing and Amplifying Selection The isothermal theorem implies that there exists graphs which are not ρ- equivalent.

1 1 − r Deﬁnition 5.4. Let ρ = 1 . 1 − rN

• If r > 1 and ρG > ρ or if 0 < r < 1 and ρG < ρ then G is an ampliﬁer of selection,

• If r > 1 and ρG < ρ or if 0 < r < 1 and ρG > ρ then G is a suppressor of selection.

26 5.2.1 Suppressors Any graph such that a single vertex has zero temperature and an edge leading 1 from the vertex to an otherwise complete graph fulfills ρG = N because the probability of the mutation happening in the root is 1/N, independent of fitness difference. If the mutation happens in the root it is guaranteed to overtake the population, if it does not happen in the root the mutant’s lineage will never overtake the whole population. Graphs with fixation probability 1/N completely suppress the effect of selection and are thus called strongest possible suppressors. Graphs with multiple roots does not allow a mutation’s lineage to completely overtake the population. Consider the following way to construct a graph G such that selection is suppressed but not completely if r > 1. Take a population and split it into two subpopulations. Let the first subpopulation correspond to a complete graph G1. Now let the second part of the population correspond to some arbitrary graph G2. Lastly, connect edges such that

• there is at least one edge leading from G1 to G2 but no edge from G2 to G1,

• every vertex in G2 is reachable from G1.

1 The constructed graph G fulﬁlls N < ρG < ρ(N) which makes the graph a suppressor of selection.4

5.2.2 Ampliﬁers Take for example the star structure. In a bi-directed graph, a center vertex is connected with peripheral vertices. The peripheral vertices are connected only with the center vertex. The weight matrix for such a graph with center vertex labeled k can be written as 1 w = if j ∈ {1, 2, 3, ..., N}\{k}, kj N − 1 (60) wjk = 1 if j ∈ {1, 2, 3, ..., N}\{k},

wij = 0 else.

It can be shown that for large N the fixation probability is amplified such that 1 − 1/r2 ρ = .3 A mutant with fitness r on the star is as likely to overtake G 1 − 1/r2N the population as a mutant with fitness r2 in a well-mixed population of the same size.

27 It is also possible to construct structures such as the super-star or the meta-funnel such that ( 1 if r > 1 ρ → (61) G 0 if 0 < r < 1, as N → ∞.3

6 Conclusion

This thesis has looked at the very surface of evolutionary game dynamics. We have investigated the commonly used replicator dynamics and Moran model to get a glimpse of both the deterministic and stochastic modelling paradigms. Further, we have seen a connection between the two dynamics. Lastly, we investigated how population structure influences selection in the Moran model and saw that the mutant fixation probability is invariant under graphs with a certain type of symmetry called isothermal graphs. We have left out one way of modelling evolving populations which is common, namely that of using stochastic differential equations. Further, We have only a single fitness function and a single stochastic process that drives evolution, namely the Moran process. Other such process are the Wright- Fisher process or pair-wise comparison processes for example. Just like with game theory, or any theory, some simplifying assumptions must be made. We wish to point out some that have been made implicitly such as: reducing multi-dimensional interactions to a one-dimensional payoff; all individuals using the same strategy have the same payoff; individuals are assumed to act independently. In closing, we wish to give a glimpse of how the assumptions we have made may be changed to capture different aspects of evolving populations. Other assumptions may make the model more realistic, for instance we may consider: an individual may employ multiple strategies, even a continuous set of strategies; dynamic population size; group interactions of different sizes; spontaneous mutations or revisions of strategy; non-symmetric games; payoff functions instead of constants; acquisition of information (games between paired individuals are not repetitive); evolution of dynamic graphs instead of fixed graphs and more. Evolutionary game theory and its dynamics are generally seen as a promising research area due to its generality.

28 References

[1] Cressman, R., Tao, Y. (2014). The replicator equation and other game dynamics. Proceedings of the National Academy of Sciences, 111(Supple- ment 3), 10810–10817. doi: 10.1073/pnas.1400823111

[2] Huang, F., Chen, X., Wang, L. (2018). The Equivalence Induced by Unifying Fitness Mappings in Frequency-Dependent Moran Process. 2018 IEEE 7th Data Driven Control and Learning Systems Conference (DD- CLS). doi: 10.1109/ddcls.2018.8516086

[3] Lieberman, E., Hauert, C., Nowak, M. A. (2005). Evolutionary dynamics on graphs. Nature, 433(7023), 312–316. doi: 10.1038/nature03204

[4] Nowak, M. A. (2006). Evolutionary Dynamics: Exploring the Equations of Life. Cambridge, MA: Belknap Press of Harvard University Press.

[5] Robinson, R. C. (2012). An Introduction to Dynamical Systems: Contin- uous and Discrete. Providence, RI: American Mathematical Society.

[6] Sigmund, K. (2011). Evolutionary game dynamics: American Mathemat- ical Society Short Course, January 4-5, 2011, New Orleans, Louisiana. Providence: American Mathematical Society.

[7] Traulsen, A., Hauert, C. (2010). Stochastic Evolutionary Game Dynam- ics. Reviews of Nonlinear Dynamics and Complexity Vol II, 25–61. doi: 10.1002/9783527628001.ch2