ARTICLE IN PRESS

Journal of Theoretical Biology 239 (2006) 172–182 www.elsevier.com/locate/yjtbi

The Maynard Smith model of sympatric

Sergey Gavriletsa,b,Ã

aDepartment of Ecology and , University of Tennessee, Knoxville, TN 37996, USA bDepartment of Mathematics, University of Tennessee, Knoxville, TN 37996, USA

Received 4 March 2005; received in revised form 23 July 2005; accepted 24 July 2005 Available online 20 October 2005

Abstract

The paper entitled ‘‘,’’ which was published by John Maynard Smith in 1966, initiated the development of mathematical models aiming to identify the conditions for sympatric speciation. A part of that paper was devoted to a specific two-locus, two-allele model of sympatric speciation in a population occupying a two-niche system. Maynard Smith provided some initial numerical results on this model. Later, Dickinson and Antonovics (1973) and Caisse and Antonovics (1978) performed more extensive numerical studies on the model. Here, I report analytical results on the haploid version of the Maynard Smith model. I show how the conditions for sympatric and parapatric speciation and the levels of resulting genetic divergence and reproductive isolation are affected by the strength of disruptive selection and nonrandom mating, recombination rate, and the rates of male and female dispersal between the niches. r 2005 Elsevier Ltd. All rights reserved.

Keywords: Sympatric; Speciation; Mathematical model; Maynard Smith

1. Introduction ‘‘Animal speciation and ’’ which forcefully argued against the importance of sympatric speciation in . John Maynard Smith has made a number of extremely By the mid-1960s, solid theoretical work on sympatric important contributions to evolutionary biology, many of speciation was needed to help clarify the arguments and which have also found various applications well outside conflicting evidence on sympatric speciation. evolutionary biology (see other papers in this issue). One of The first part of the paper analysed the conditions for the his earlier influential works was the paper entitled maintenance of genetic variation in a two-deme version of ‘‘Sympatric speciation’’ published by American Naturalist the Levene model (Levene, 1953) which Maynard Smith in 1966. generalized for the case of habitat choice by females. The There were two major stimulas for the paper in a few second part was devoted to the following question: given preceding years. The first was a series of laboratory that genetic variation is stably maintained under disruptive experiments with melanogaster by Thoday and selection, can reproductive isolation evolve between the his colleagues (Millicent and Thoday, 1961; Thoday and two ‘‘extreme’’ morphs, so that no low-fitness intermedi- Boam, 1959; Thoday and Gibson, 1962) that showed rapid ates are produced even when different individuals freely emergence of very strong reproductive isolation. These encounter each other? experiments suggested that sympatric speciation can Maynard Smith discussed four possible mechanisms proceed in a straightforward way. The second was the that can lead to the evolution of reproductive isolation publication of ’s (1963) monumental volume in the presence of gene flow: (1) ‘‘habitat selection,’’ by which he meant the tendency of organisms to mate in the same niche where they were born, (2) ‘‘pleiotropism,’’ ÃCorresponding author at: Department of Ecology and Evolutionary Biology, University of Tennessee, Knoxville, TN 37996, USA. Tel.: by which he meant the situations in which the alleles +1 865 974 8136; fax: +1 865 974 3067. that adapt the individuals to local conditions also E-mail address: [email protected]. control mating behavior, (3) ‘‘modifier genes,’’ by which

0022-5193/$ - see front matter r 2005 Elsevier Ltd. All rights reserved. doi:10.1016/j.jtbi.2005.08.041 ARTICLE IN PRESS S. Gavrilets / Journal of Theoretical Biology 239 (2006) 172–182 173 he meant the establishment of selectively neutral genes to the following scheme: that cause pleiotropism in the genes underlying local AA Aa aa , and (4) ‘‘assortative mating genes,’’ by which he meant genes that cause assortative mating in niche 1 1 1 1 s1 regardless of the genotype at the selected loci. I note that in niche 2 1 s2 1 s2 1: the majority of the later theoretical research on sympatric Here the coefficients s and s measure the strength of speciation has been done along these four lines first 1 2 viability selection within each niche (0 s ; s 1). That is, discussed by Maynard Smith. Maynard Smith also p 1 2p the dominant allele A is advantageous in niche 1, whereas proposed a specific model of sympatric speciation via the the recessive allele a is advantageous in niche 2. Maynard mechanism of ‘‘assortative mating genes’’ which will be the Smith assumed that surviving adults form a single focus of this paper. randomly mating population. After mating, each female In the years since 1966 sympatric speciation went lays a fraction ð1 þ HÞ=2 of her eggs in the niche where she through cycles of increased enthusiasm and skepticism was raised and lays the remaining fraction ð1 HÞ=2 in the accompanied by a continuous growth in both empirical other niche (0 H 1). The coefficient H measures the (reviewed by Coyne and Orr, 2004) and theoretical p p degree of habitat selection by females. H ¼ 0 implies no (reviewed by Gavrilets, 2004) knowledge. Unfor- habitat selection, as in the Levene model (Levene, 1953; tunately, most of the theoretical work remains based on Gavrilets, 2004, pp. 235–240). H ¼ 1 implies complete numerical simulations which significantly limits the habitat selection. Note that, alternatively, one can think of generality of theoretical conclusions (Gavrilets, 2003). mating as taking place within the niches, with males Given the exploding growth of interests in speciation, in migrating randomly between the niches prior to mating general, and in sympatric speciation, in particular, and and females either staying in the niche where they were given the controversies that keep remain associated born or going to the other niche with probabilities ð1 þ with sympatric speciation (as is apparent from a compar- HÞ=2andð1 HÞ=2, respectively. ison of major conclusions on sympatric speciation of The first question to ask is whether spatially hetero- Coyne and Orr, 2004 and Gavrilets, 2004 and those geneous selection maintains genetic variation in the DS of Dieckmann et al., 2004) there is a growing need locus. For simplicity, assume that the sizes of the two of a solid quantitative theory of speciation. Although populations are the same. Then, as shown by Maynard recently a number of simple models of sympatric speciation Smith (1966),ifH ¼ 0 (i.e. females lay eggs in the two have been solved analytically (Gavrilets, 2004), obtaining niches at random), genetic variation is maintained if analytical results still remains a major theoretical s2 s1 challenge. Below I present analytical results for a 1o o1. simplified version of the model introduced by Maynard s1s2 Smith. A part of these results were briefly introduced in If H ¼ 1 (females always lay eggs in the niche where they Gavrilets (2004, Chapter 10). themselves were raised), then genetic variation is main- I will define sympatric speciation as the emergence of tained if new from a population where mating is random 3 s2 s1 with respect to the birthplace of the mating partners o o3. (Gavrilets, 2003, 2004). Note that, implicitly, mating is 2 s1s2 allowed to be nonrandom with respect to, for example, These inequalities show that maintaining genetic variation genotype, phenotype, and culturally inherited traits. This requires sufficiently strong selection (i.e. large s1 and s2); definition is actually implied in most existing mathematical increasing the degree of habitat choice by females H models of sympatric speciation. increases the range of conditions resulting in stable . At the polymorphic equilibrium the locally advanta- 2. The Maynard Smith model geous alleles (i.e. allele A in niche 1 and allele a in niche 2) have higher frequencies than the locally deleterious alleles Consider a diploid population with discrete nonoverlap- (i.e. allele a in niche 1 and allele A in niche 2). However, ping generations inhabiting two distinct niches. Assume because mating is random, offspring with locally deleter- that density-dependent factors operating indepen- ious genotypes will be constantly produced at relatively dently within each niche maintain population sizes high frequencies which will reduce the average fitness of at constant levels that do not depend on the average each population. fitness of the populations. This is the case of soft Assume next that there is also a nonrandom mating selection (e.g. Christiansen, 1975; DeMeeus et al., 1993; (NM) locus with alleles B and b controlling assortative Wallace, 1968). mating. Let the rate of recombination between the DS and 1 Let individuals differ with regard to the disruptive NM locus be r (0prp2). We will suppose that mating selection (DS) locus with alleles A and a controlling occurs according to the O’Donald model with dominance fitness (viability) in a local environment according (O’Donald, 1960, 1980; Gavrilets, 2004, pp. 287–290). That ARTICLE IN PRESS 174 S. Gavrilets / Journal of Theoretical Biology 239 (2006) 172–182 is, individuals BB and Bb have the same phenotype which selection, and weak assortative mating. The second out- is recognizably different from the phenotype of individuals come is promoted by low migration, strong selection, and bb. With probability a, an individual mates with another strong assortative mating. This outcome is interpreted as (a individual that has the same phenotype (0pap1). With step towards) speciation that is sympatric or parapatric, 1 probability 1 a, the individual is engaged in random depending on whether male migration rate is m ¼ 2 or 1 mating. mo2. For the special case of H ¼ 1 (i.e. complete habitat Overall, Dickinson and Antonovics (1973) confirmed selection by females), a ¼ 1 (i.e. no mating between Maynard Smith’s conclusion that sympatric speciation is 1 genotypes BB=Bb and bb is allowed), and r ¼ 2 (i.e. the theoretically possible. However, they also showed that the loci are unlinked), Maynard Smith presented a numerical maintenance of genetic variation under disruptive selection example illustrating that an initial difference in the does not necessarily lead to the evolution of reproductive frequencies of NM alleles between the niches gets isolation. Some additional conditions have to be satisfied progressively amplified. One NM allele gets associated as well. In particular, for sympatric speciation to occur (i) with the locally advantageous allele A in niche 1, and the assortative mating has to be strong enough and (ii) the other NM allele gets associated with the locally advanta- joint strength of selection and assortative mating has to be geous allele a in niche 2. As a result of this, the individuals large enough. raised in a niche will tend to mate together. The eventual Later, Caisse and Antonovics (1978) extended the two- outcome of the evolutionary dynamics is the increase in deme results of Dickinson and Antonovics (1973) to the frequencies of two reproductively isolated genotypes case of 10 demes arranged along a line. In one version of AA=BB and aa=bb (or AA=bb and aa=BB) and the their model, the rate of male migration between demes complete disappearance of all intermediate genotypes. This decreases exponentially with the distance between them. outcome is naturally interpreted as sympatric speciation. The other version assumes that males migrated only [Note that although in Maynard Smith’s simulations between the neighboring demes (i.e. as in the stepping- females always lay eggs in the niche where they were born, stone model). The strength of selection acting on the DS speciation is sympatric according to our definition because locus increased linearly from deme 1 to deme 10 or males move randomly between the niches.] Although undertook a step change between the two deme in the Maynard Smith did acknowledge that the parameter middle of the array (see Endler, 1977 for a review of earlier combination he used was very favorable for speciation work on spatially heterogeneous selection). Most of the but unrealistic, he thought that more realistic conditions results given by Caisse and Antonovics (1978) are for would merely slow down the speciation process but not unlinked loci and a ¼ 0:9, that is, very strong assortative prevent it completely. According to his 1966 paper, the mating. Slow convergence towards equilibria and the crucial process in sympatric speciation is the establishment relatively short duration of the simulations (typically 400 of a stable polymorphism under disruptive selection rather generations) make interpretation of their results difficult than the subsequent evolution of reproductive isolation, because there is no guarantee that they correspond to which he believed was likely (p. 649). equilibrium rather than transient states. Overall, however, Thorough numerical analysis of the Maynard Smith their results appear to be quite compatible with those of model was undertaken by Dickinson and Antonovics Dickinson and Antonovics. In particular, strong selection (1973) as a part of their study of a number of different (s40:1) and high levels of assortative mating (a40:4) are models. They assumed that the alleles at the DS locus act necessary for a divergence at the NM locus. additively (so that in each niche, the fitness of hetero- zygotes is the average of the fitnesses of homozygotes), and 3. The haploid version of the Maynard Smith model allowed for a limited migration of males between the niches at rate m. The females did not migrate. This model was To get a better understanding of the interactions between designed to reflect plant situations, where there is pollen disruptive selection and nonrandom mating, let us consider transfer between sessile organisms and seed flow is a haploid version of the Maynard Smith model. Let the DS negligible. Dickinson and Antonovics numerically studied locus control viability according to the following sym- the progress towards speciation during the first 400 metric scheme: generations as a function of the strength of selection for local adaptation s (which was assumed to be equal in both Aa niches), the strength of assortative mating a, and the male viability in niche 1 1 1 s (1) migration rate m (see their Fig. 5). The progress toward viability in niche 2 1 s 1; speciation was measured by the difference DNM in the frequency of an NM allele between the niches. Depending where coefficient s measures the strength of selection within on parameter values, the system evolved to a state with no each niche (0psp1). Note that this type of spatially differentiation in the NM locus (i.e. DNM ¼ 0) or to a state heterogeneous selection always maintains genetic variation 1 with some differentiation in the NM locus (i.e. jDNMj40). at the DS locus with both alleles having frequency 2 in both The first outcome is promoted by large migration, weak niches. Let the NM locus control assortative mating ARTICLE IN PRESS S. Gavrilets / Journal of Theoretical Biology 239 (2006) 172–182 175 according to the O’Donald model. That is, with probability Note that the frequency of allele B among mating males in a, an individual mates with another individual that has niche i is exactly the same allele (B or b) at the NM locus. With Y ¼ y þ y , (5) probability 1 a, the individual mates irrespective of i i;1 i;3 genotype. and that of alleles b is 1 Y i. Assume that migration takes place after selection but The frequency of matings in niche i between females with before mating and that the sizes of the populations are the genotype k and males with genotype l can be written as same and constant (i.e. soft selection). Let males migrate dkl between the niches with probability m per generation, and P ðk lÞ¼x y 1 a þ a . (6) i i;k i;l Y females with probability f per generation. Note that Zl parameter f corresponds to parameter H in the original This expression implies that each female mates randomly Maynard Smith model. In the special case numerically with probability 1 a and assortatively with probability a. studied by Maynard Smith, females always lay eggs in the Here variable Y Zl is the population frequency of the allele niche they were born (f ¼ 0) whereas males disperse present at the NM locus of males with genotype l. 1 randomly (m ¼ 2). Genotype frequencies among offspring born in niche i in In this model, there are four different genotypes and two the next generation are different niches. Thus, the population genetic state has to X 0 1 be characterized by 2 ð4 1Þ¼6 independent variables. p ¼ Piðk lÞRðk; l ! jÞ, (7) i;j W In spite of this complexity, analytical progress is possible k;l by utilizing the symmetry of the model. where Rðk; l ! jÞ is the probability that k l mating results in offspring with genotype j (e.g. Nagylaki, 1992, Eq. 8.9; see also BuP¨rger, 2000), and W is the normalizing 3.1. Dynamic equations 0 factor (such that j pi;j ¼ 1). To analyse the dynamic equations (2)–(7) it is convenient Let the frequencies of four genotypes AB; Ab; aB, and ab to use a transformation proposed by Karlin and Feldman among offspring in the ith niche be zi;1; zi;2; zi;3; zi;4. Assume (1970). Specifically, let that the viability of the jth genotype in the ith niche is wi;j ui;1 ¼ zi;1 þ zi;2 þ zi;3 þ zi;4, (8a) (i ¼ 1; 2; j ¼ 1; 2; 3; 4). Let us define indicator variables Zi showing the presence or absence of allele B at genotype i: ui;2 ¼ zi;1 zi;2 þ zi;3 zi;4, (8b) Z1 ¼ Z3 ¼ 1; Z2 ¼ Z4 ¼ 0. Let us also define indicator variables dij showing whether genotypes i and j have the ui;3 ¼ zi;1 þ zi;2 zi;3 zi;4, (8c) same allele at the NM locus. The values of dij for different pairs of genotypes can be represented by a matrix: ui;4 ¼ zi;1 zi;2 zi;3 þ zi;4 (8d) 0 1 1010 be new variables describing the genetic state of the B C B 0101C population in niche i. Note that ui;1 ¼ 1, that the d ¼ B C. @ 1010A frequencies of alleles A and B are 1 þ u 0101 z ¼ z þ z ¼ i;3 , (9a) i;A i;1 i;2 2 The genotype frequencies among surviving offspring in 1 þ u niche i are z ¼ z þ z ¼ i;2 , (9b) i;B i;1 i;3 2 s wi;j zi;j ¼ zi;j, (2) and that linkage disequilibrium is wi P ui;4 ui;2ui;3 where w ¼ w z is the average fitness in niche i. Di ¼ zi;1zi;4 zi;2zi;3 ¼ . (9c) i j i;j i;j 4 After migration, the genotype frequencies in mating females are In this model, the population always evolves to an equilibrium state where the locally advantageous allele has s s x1;j ¼ð1 f Þz1;j þ fz2;j, (3a) a higher frequency than the locally deleterious allele. With regard to other features, there are two qualitatively s s different regimes. In the first regime, the population x2;j ¼ð1 f Þz þ fz , (3b) 2;j 1;j evolves to a line of neutrally stable equilibria at which and the genotype frequencies in mating males are u2;1 ¼ u2;2 ¼ U 2, (10a) s s y1;j ¼ð1 mÞz1;j þ mz2;j, (4a) u3;1 ¼u3;2 ¼ U 3, (10b)

s s y2;j ¼ð1 mÞz2;j þ mz1;j. (4b) u4;1 ¼u4;2 ¼ U 2U 3, (10c) ARTICLE IN PRESS 176 S. Gavrilets / Journal of Theoretical Biology 239 (2006) 172–182 where U2 is an arbitrary number between 1 and 1, and 3.2. Basic case: no female migration, random male U 3 is a positive solution of a certain quadratic equation. migration, and no linkage The above conditions imply that the frequencies of the alleles at the NM locus in both niches are the same (and Assume that females never migrate (f ¼ 0), males 1 arbitrary), the frequencies of the locally advantageous disperse randomly between the niches (i.e. m ¼ 2) and the 1 alleles (i.e. allele A in niche 1 and allele a in niche 2) are the loci are unlinked (i.e. r ¼ 2). This case corresponds to the same and uniquely defined by parameters, and that both one analysed by Maynard Smith (1966). local populations are at linkage equilibrium. At the line of equilibria (10), U 2 is an arbitrary number In the second regime, the population evolves to one of between 1 and 1, and U 3 is given by a positive solution of two ‘‘speciation equilibria’’ with strong differentiation in a quadratic equation both loci between the niches and linkage disequilibrium 2 s 1 within each niche: U2 þ U ¼ 0. (12) 3 2s 3 2 u2;1 ¼u2;2 ¼ V 2, (11a) The frequency of the locally advantageous allele at the DS 1 3 locus is ð1 þ U 3Þ=2. This frequency grows from 2 to 4 as the u3;1 ¼u3;2 ¼ V 3, (11b) strength of selection for local adaptation s increases from 0 to 1. The difference in the frequencies of an DS allele u4;1 ¼ u4;2 ¼ V 4, (11c) between the niches is equal to U 3. This difference grows from 0 to 1 as s increases from 0 to 1 (see Fig. 1(a)). where V ; V are positive and their values as well as the 2 3 4 The line of equilibria exists always. It is locally stable if absolute value of V 2 are uniquely defined by parameters. qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi The difference between the equilibria is which NM allele (B ð3sÞ2 þ 4ð1 sÞ 3s or b) is associated with the locally advantageous DS allele. 1 aoac þ , (13) We will start analysing system (2)–(7) with the simplest 2 4 case. Some details of the derivations are outlined in that is, if the intensity of assortative mating is smaller than Appendix. A Maple file with more details can be down- a critical value ac. Fig. 1(b) illustrates the dependence of ac loaded from the author’s web page. on s.

0.5 1

0.4 0.8

0.3 0.6 3 α U 0.2 0.4

0.1 0.2

0 0 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 (a)s (b) s

1 1 0.8 0.8 0.6 0.6 2 3 V 0.4 V 0.4 0.2 0.2 0 0 1 1 1 1 0.5 0.5 α 0.5 α 0.5 s s (c)0 0 (d) 0 0

Fig. 1. Equilibria in the basic model. (a) The value of U3 at the line of equilibria. (b) The critical value of a for the existence of a pair of speciation equilibria. (c) The values of V 3 at a speciation equilibrium. (d) The values of V 2 at a speciation equilibrium. ARTICLE IN PRESS S. Gavrilets / Journal of Theoretical Biology 239 (2006) 172–182 177

At the speciation equilibria (11), increasing s and a. Fig. 1(d) illustrates the level of between- rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi niche divergence in the DS locus achieved at speciation ð1 þ aÞð2a2 þ 3sa 2a sÞ V ¼ , (14a) equilibria. At the speciation equilibria the difference in the 2 4as allele frequencies in the DS locus is equal to jV 3j. Note that s þ a 1 at a speciation equilibrium jV 3j can reach the maximum V 3 ¼ , (14b) possible value of 1 whereas at the line of equilibria the s 1 maximum value was 2. 1 V 4 ¼ V 2. (14c) The critical value ac decreases from 1 to 2 as s grows The speciation equilibria are feasible if a is larger than the from 0 to 1. That is, sympatric speciation requires sufficiently strong assortative mating and selection for critical value ac defined by Eq. (13). That is, the speciation equilibria exist only when the line of equilibria (10) is local adaptation. The population switches from the first unstable. Numerical simulations suggest that no other regime to the second ‘‘because’’ this results in increasing its average fitness. In this model, the gain in the average fitness equilibria are stable in this model. That is, for aoac the ð Þ system evolves to the line of equilibria whereas for can be written as a ac =8. Quantitatively similar behavior is observed in some other models of sympatric a4ac, (15) speciation (Gavrilets, 2004). However, in other models (e.g. the system evolves to a speciation equilibrium. Gavrilets, 2005; Kirkpatrick and Ravigne´, 2002; Kisdi and Following Maynard Smith (1966) and Dickinson and Geritz, 1999) whether or not the population evolves to the Antonovics (1973), one can characterize the resulting speciation equilibrium depends also on initial conditions. degree of reproductive isolation by the difference in the frequencies of an NM allele between the niches. At the 3.3. Effects of linkage speciation equilibria this difference is equal to jV 2j. Fig. 1(c) illustrates the level of between-niche divergence The results can be generalized for the case of linked loci. in the NM locus achieved at equilibrium. One can see that It is well appreciated that close linkage increases both the once the condition for sympatric speciation is satisfied, the plausibility and the level of genetic divergence between equilibrium value of jV 2j approaches 1 very rapidly with sympatric populations (e.g. Felsenstein, 1981; Gavrilets,

1

0.8 r=0.5 0.6 α 0.4 r=0.1

0.2 r=0.02 r=0.004 0 0 0.2 0.4 0.6 0.8 1 (a) s

1 1 0.8 0.6 3 2 0.5 V V 0.4 0.2 1 1 0 0 0.8 1 1 0.6 0.5 0.8 s 0.6 0.4 s 0.5 0.4 0.2 α α 0.2 (b)0 0 (c) 0 0

Fig. 2. Effects of recombination rate r on the possibility of sympatric speciation. (a) Critical value of a for four different values of r. (b) The value of V 2 at speciation equilibria with r ¼ 0:1. (c) The value of V 3 at speciation equilibria with r ¼ 0:1. ARTICLE IN PRESS 178 S. Gavrilets / Journal of Theoretical Biology 239 (2006) 172–182

2004; Udovic, 1980). Fig. 2 illustrates these effects for the quickly shrinks with increasing f. Figs. 3(c) and (d) show Maynard Smith model. In particular, Fig. 2(a) shows that that increasing female migration also decreases the result- increasing recombination rate r dramatically increases the ing levels of divergence in both loci. range of combinations of s and a leading to sympatric speciation. Figs. 2(b) and (c) show that once conditions for 3.5. Effects of restrictions on male migration sympatric speciation are satisfied the level of divergence in both the NM and DS loci grows quickly. 1 Let f ¼ 0 but m and r be between 0 and 2. This is the case of parapatric speciation. On the line of equilibria the value 3.4. Effects of female migration of U 3 is given by a positive solution of the quadratic equation: Let m ¼ 1 but f and r be between 0 and 1. Note that this 2 2 2 2 s is still the case of sympatric speciation (because males U þ m U3 ð1 mÞ¼0. (17) 3 s disperse randomly). On the line of equilibria U 3 is given by a positive solution of the quadratic equation Obviously, decreasing the rate of male migration m increases the frequency of the locally advantageous allele. 2 s 1 2 This effect is illustrated in Fig. 4(b). Note that u3 does not U 3 þð1 þ 2f Þ U 3 þ f ¼ 0. (16) 2s 2 depend on the recombination rate. As before U 3 also gives the difference in the frequency of The pair of speciation equilibria (11) are defined by an allele in the DS locus between the niches (see Fig. 3(b)). much more cumbersome equations. Fig. 4(a) illustrates the Increasing female migration rate f decreases the degree of conditions for existence of these equilibria. Interestingly, divergence between the niches in the DS locus with U 3 the effect of reduced male migration is much less significant 1 approaching zero as f approaches 2. Note that U 3 does not than that of reduced recombination (compare Fig. 4(a) depend on the recombination rate. with Fig. 2(a)). Figs. 4(c) and (d) illustrate the level of The effects of female migration rate f on the possibility between-niche divergence in the NM and DS loci achieved of sympatric speciation are illustrated in Fig. 3(a). The at equilibrium. One can see that once the conditions range of parameter values resulting in sympatric speciation for sympatric speciation are satisfied, the equilibrium

1 f=0.3 0.9

0.8 f=0.2 0.7 f=0.1 0.6 0.5 0.45 f=0.0 0.4 0.5

α 0.35 0.3 3 0.25 0.4 U 0.2 0.15 0.3 0.1 0.5 0.1 0 0.2 0

0.1 0.5 0.25 s 0 f 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.5 0 (a)s (b)

1 0.8 0.6 2 3 V V 0.4 0.2 1 0 0.8 1 0.6 s 0.8 0.4 α 0.6 s 0.4 0.2 α 0.2 (c) (d) 0 0

Fig. 3. Effects of female migration rate f on the possibility of sympatric speciation. (a) The critical values of a for the existence of a pair of speciation equilibria for four different values of f. (b) The value of U 3 at the line of equilibria. (c) The value of V 2 at a speciation equilibrium with f ¼ 0:1. (d) The value of V 3 at a speciation equilibrium with f ¼ 0:1. ARTICLE IN PRESS S. Gavrilets / Journal of Theoretical Biology 239 (2006) 172–182 179

1

0.8

m=0.5 1 0.6 m=0.1 α

3 0.5 0.4 U m=0.02

m=0.004 0 1.0 0.2 0 0.5 0.25 s 0 m 0 0.2 0.4 0.6 0.8 1 0.5 0 (a)s (b)

1 1 0.8 0.6 3 2

0.5 V

V 0.4 0.2 1 1 0 0 0.8 1 1 0.6 0.8 0.5 0.4 s 0.6 0.5 0.4 0.2 s α α 0.2 (c) 0 0 (d) 0 0

Fig. 4. Effects of male migration rate m on the possibility of parapatric speciation. (a) Critical value of a for four different values of m. The curves corresponding to m ¼ 0:02 and m ¼ 0:004 are very close to each other. (b) The value of U3 at the line of equilibria. (c) The value of V 2 at the speciation equilibrium with m ¼ 0:1. (d) The value of V 3 at the speciation equilibrium with m ¼ 0:1.

values of jV 2j and V 3 approach 1 very rapidly with the than in the standard model with no differences increasing s and a. (Bodmer, 1965).

3.6. Equal migration of sexes 4. Discussion

Finally, let us assume that the male and female migration The results on the haploid version of the Maynard Smith rates are arbitrary but equal (f ¼ m). In this case, on the model presented above put earlier conclusions, which were based exclusively on numerical simulations, on firmer line of equilibria the value of U 3 is given by a positive solution of the quadratic equation theoretical grounds. Here I have shown that in the model studied sympatric speciation requires sufficiently strong 2 2 s nonrandom mating and sufficiently strong and spatially U 3 þ 2m U 3 ð1 2mÞ¼0. (18) s heterogeneous selection for local adaptation. Close linkage Obviously, decreasing the rate of migration increases the between the disruptive selection locus and the nonrandom frequency of the locally advantageous allele. mating locus greatly increases the range of parameter As for speciation equilibria (11), it turns out that they do values resulting in sympatric speciation (see Fig. 2(a)). not exist. This suggests that the assumption about unequal Restrictions on male migration promote (parapatric) migration rates is crucial for the possibility of speciation in speciation but the overall effect is not particularly great the haploid version of the Maynard Smith model. (There is (see Fig. 4(a)). The explanation is that decreasing male of course a possibility that some nonsymmetric equilibria migration rate decreases the opportunities for hybridiza- exist and are stable but I was not able to find them.) I do tion which, in turn, reduces the ‘‘incentive’’ to evolve not have an intuitive explanation of this result, however, stronger reproductive isolation. These conclusions are well somewhat similar effects have been observed in other appreciated in theoretical speciation research (Gavrilets, models with sex differences. For example, it is well known 2004). A less appreciated feature of the Maynard Smith that the structure of equilibria is much more complex in model is the importance of the assumption that females’ one-locus, two-allele models with differential selection in movement between the niches is restricted. Allowing for ARTICLE IN PRESS 180 S. Gavrilets / Journal of Theoretical Biology 239 (2006) 172–182 females to migrate and increasing the rate of female nonrandom mating increases. Intuitively one expects this migration quickly decreases the opportunity for speciation range to shrink (because each individual locus will be (see Fig. 3(a)). Also, speciation in (the haploid version of) subject to weaker selection whereas recombination will the Maynard Smith model appears to require that the rates become more effective in destroying within-population of male and female migration are different. If one allows differentiation). Finding conditions for speciation in multi- for equal migration of both sexes, then speciation does not locus models represents a formidable mathematical seem to be possible, even when migration and recombina- challenge. tion rates are small and selection and assortative mating are very strong. Therefore, although speciation in the Acknowledgements Maynard Smith model is sympatric, spatial subdivision and restrictions on migration are crucial for its success. I thank Max Shpak and reviewers for helpful comments An important biological question concerns the levels of of the manuscript. This work was supported by National reproductive isolation and of gene flow between the Institutes of Health (Grant GM56693) and by National emerging clusters of genotypes. Theoretical results (see Science Foundation (Grant DEB-0111613). Figs. 1–4) show that once the conditions for speciation are satisfied, genetic divergence between the clusters quickly Appendix A approaches the maximum possible value with increasing the strength of selection s and the strength of nonrandom A.1. Equilibria in the basic case mating a. However, unless s and a are very close to one, hybridization will be ongoing. As a result, no genetic Let f ¼ 0; m ¼ 1; r ¼ 1. I will start by looking for divergence in neutral markers will be maintained between 2 2 symmetric equilibria satisfying conditions u2;1 ¼ u2;2; u3;1 ¼ the sympatric or parapatric clusters of genotypes. u3;2 and u4;1 ¼u4;2. Dropping the second subscript for Where comparable, the analytical results on the haploid simplicity of notation, the equations for a change per version of the Maynard Smith model appear to be generation in u2 and u3 are qualitatively similar to the earlier numerical results on the original diploid model (Caisse and Antonovics, 1978; sðu4 u2u3Þ Du2 ¼ , (19a) Dickinson and Antonovics, 1973). Whether the differences 2 s þ su3 in male and female migration rates in the diploid model are 2 as important as in the haploid model is currently unknown. 2su3 þð2 sÞu3 s Du3 ¼ . (19b) One major implicit limitation of the Maynard Smith 2ð2 s þ su3Þ model is the absence of costs of being choosy. In the model, From the first equation it follows that at equilibrium all organisms have equal mating success no matter how u4 ¼ u2u3. Assuming this, the equation for u4 becomes many (or few) preferred mates exist in their local 2 populations. Costs of being choosy have been recently u2ð2su3 þð2 sÞu3 s Du4 ¼ . (19c) identified as a major factor opposing sympatric speciation 2ð2 s þ su3Þ (Gavrilets, 2004) whose importance is similar to that of Thus, at equilibrium u2 is defined by Eq. (12), u2 is recombination (Felsenstein, 1981). Another important arbitrary, and u4 ¼ u2u3. This set of conditions defines a limitation is the assumption that selection is soft, so that, line of equilibria (10). the contribution of a local population to the overall Next let us look for symmetric equilibria satisfying offspring pool does not depend on the average fitness of the conditions u2;1 ¼u2;2; u3;1 ¼u3;2 and u4;1 ¼ u4;2. Drop- local population. Introducing costs of being choosy and ping the second subscript for simplicity of notation, the using hard selection instead of soft selection are expected to equation for a change per generation in u2 is make the maintenance of genetic variation and sympatric speciation much more difficult. 2su2u3 sð1 þ aÞu4 þð2 sÞð1 aÞu2 Du2 ¼ . (20a) In the models considered here I assumed that migration 2½ð2 s þ su3Þ takes place after selection. Migration occurring before Solving the right-hand side of the last equation for u4 and /selection will impose a positive correlation substituting the resulting equation into an equation for a between offspring fitness and place of origin, reinforcing change per generation in u3, one finds that the speciation process.

2 2 2 2 sð1 þ aÞ ½2su3 þð2 sÞu3 sþ4au2½sðs 2Þu3 þ 2s 2 s 2as þ 2a Du2 ¼ 2 . (20b) 2½ð2 s þ su3Þsð1 þ aÞ

A critical theoretical question is how the range of The right-hand side of the last equation can be used to parameter values resulting in speciation will be affected if express u2 is a function of u3. Substituting the resulting the number of loci underlying local adaptation and expression for u2 into an equation for a change per ARTICLE IN PRESS S. Gavrilets / Journal of Theoretical Biology 239 (2006) 172–182 181 generation in u4, one finds that Finally, given V 3 and V 2,

2u2ðsu3 þ 1 a sÞðsu3 þ 1 a þ asÞ V 2f2sV 3 þð2 sÞ½1 a þ 2f ð1 þ aÞg Du4 ¼ . (20c) V 4 ¼ . (22e) sð1 þ aÞð2 s þ su3Þ sð1 þ aÞð1 2f Þ

Of the two solutions for u3 the last equation only is The speciation equilibria exist whenever the solutions of feasible. the above equations are biologically meaningful (i.e. Solutions for other parameter values are identified using 0pjV 2j; V 3; V 4p1). similar procedures. A.4. Speciation equilibria when f ¼ 0 1 A.2. The eigenvalues of the stability matrix when r ¼ 2; f ¼ 0; m ¼ 1 Straightforward calculations show that V 3 is given by a 2 positive solution of the quadratic Let us choose a point on the line of equilibria. 2 2 s ð1 mÞV 3 þ B1V 3 þ B0 ¼ 0 (23a) Straightforward calculations show that the eigenvalues of the stability matrix at this point do not depend on its satisfying 0pV 3p1. Here, coordinates and are B1 ¼ sð2 sÞmð1 þ r mÞð1 aÞ, (23b)

4ð1 sÞ 2 l1 ¼ 2 , (21a) B0 ¼ rmð1 aÞ½s þ 2ð1 aÞð1 sÞ ½2 sð1 u3Þ s2ð1 mÞ½1 mð1 aÞ. ð23cÞ ð1 sÞð1 þ aÞ l2 ¼ , (21b) Given V 3, the value of V 2 can be found from ½ sð u Þ2 2 1 3 2 2 2 sV 3 þ mð2 sÞV 3 sð1 mÞ V 2 ¼ s½1 mð1 aÞ 2 , ð1 sÞð1 þ aÞ C2u3 þ C1u3 C0 l3 ¼ , (21c) 2 sð1 u3Þ (23d) where 2ð1 sÞ l4 ¼ 2 , (21d) 2 2 ½2 sð1 u3Þ C2 ¼ s ð1 2mÞ , which all are always between 0 and 1, C1 ¼ sð2 sÞm½1 4mð1 mÞð1 aÞ,

1 þ a 2 l ¼ , (21e) C0 ¼ s ð1 mÞf1 4m½1 mð1 aÞg 5 2 sð1 u Þ 3 4m2að1 aÞð1 sÞ. which is always nonnegative, and Finally, given V 3 and V 2, l6 ¼ 1. (21f) V 2½sV 3 þ mð2 sÞð1 aÞ u4 ¼ . (23e) The line of equilibria is locally stable if l5o1 which is the s½1 mð1 aÞ same as ao1 sð1 u3Þ. The speciation equilibria exist whenever the solutions of the above equations are biologically meaningful (i.e. 1 A.3. Speciation equilibria when m ¼ 2 0pjV 2j; V 3; V 4p1).

Straightforward calculations show that V 3 is given by a References positive solution of the quadratic 2s2V 2 þ A V þ A ¼ 0, (22a) Bodmer, W.F., 1965. Differential fertility in models. 3 1 3 0 Genetics 51, 411–424. where Bu¨rger, R., 2000. The Mathematical Theory of Selection, Recombination and Mutation. Wiley, Chichester. A1 ¼ð2 sÞs½ð1 aÞð1 þ 2rÞþ2f ð1 þ aÞ, (22b) Caisse, M., Antonovics, J., 1978. Evolution of reproductive isolation in clinical populations. Heredity 40, 371–384. 2 2 Christiansen, F., 1975. Hard and soft selection in a subdivided population. A0 ¼s ð1 þ aÞþ2rð1 aÞ½s þ 2ð1 sÞð1 aÞ Am. Nat. 109, 11–16. þ 2f ð1 þ aÞ½s2 þ 4rð1 sÞð1 aÞ. ð22cÞ Coyne, J., Orr, H.A., 2004. Speciation. Sinauer Associates, Sunderland, MA. Given V 3, the value of V 2 can be found from DeMeeus, T., Michalakis, Y., Renaud, F., Olivieri, I., 1993. Polymorph- ism in heterogeneous environments, evolution of habitat selection and sð1 þ aÞ2ð1 2f Þ½2sV 2 þð2 sÞð1 þ 2f ÞV sð1 2f Þ sympatric speciation—soft and hard selection models. Evol. Ecol. 7, 2 ¼ 3 3 V 2 2 . 175–198. 4afsð2 sÞV 3 þ s þ 2ð1 sÞ½1 a þ 2f ð1 þ aÞg Dickinson, H., Antonovics, J., 1973. Theoretical considerations of (22d) sympatric divergence. Am. Nat. 107, 256–274. ARTICLE IN PRESS 182 S. Gavrilets / Journal of Theoretical Biology 239 (2006) 172–182

Dieckmann, U., Doebeli, M., Metz, J.A.J., Tautz, D., 2004. Adaptive Maynard Smith, J., 1966. Sympatric speciation. Am. Nat. 100, 637–650. Speciation. Cambridge University Press, Cambridge. Mayr, E., 1963. Animal Species and Evolution. Belknap Press, Cam- Endler, J.A., 1977. Geographic Variation, Speciation and Clines. bridge, MA. Princeton University Press, Princeton. Millicent, E., Thoday, J.M., 1961. Effects of disruptive selection. IV. Gene Felsenstein, J., 1981. Skepticism towards Santa Rosalia, or why are there flow and divergence. Heredity 16, 199–217. so few kinds of animals? Evolution 35, 124–138. Nagylaki, T., 1992. Introduction to Theoretical Population Genetics. Gavrilets, S., 2003. Models of speciation: what have we learned in 40 Springer, Berlin. years? Evolution 57, 2197–2215. O’Donald, P., 1960. Assortative mating in a population in which two Gavrilets, S., 2004. Fitness Landscapes and the Origin of Species. alleles are segregating. Heredity 15, 389–396. Princeton University Press, Princeton, NJ. O’Donald, P., 1980. Genetic Models of Sexual Selection. Cambridge Gavrilets, S., 2005. ‘‘Adaptive speciation’’: it is not that simple. Evolution University Press, Cambridge. 59, 696–699. Thoday, J.M., Boam, T.B., 1959. Effects of disruptive selection. II. Karlin, S., Feldman, M.W., 1970. Linkage and selection: two-locus Polymorphism and divergence without isolation. Heredity 13, 205–218. symmetric viability model. Theor. Popul. Biol. 1, 39–71. Thoday, J.M., Gibson, J.B., 1962. Isolation by disruptive selection. Kirkpatrick, M., Ravigne´, V., 2002. Speciation by natural and sexual Nature 193, 1164–1166. selection: models and experiments. Am. Nat. 159, S22–S35. Udovic, D., 1980. Frequency-dependent selection, disruptive selection, Kisdi, E., Geritz, S.A.H., 1999. Adaptive dynamics in allele space: and the evolution of reproductive isolation. Am. Nat. 116, evolution of genetic polymorphism by small mutations in a hetero- 621–641. geneous environment. Evolution 53, 993–1008. Wallace, B., 1968. Polymorphism, population size, and genetic load. In: Levene, H., 1953. Genetic equilibrium when more than one ecological Lewonton, R.C. (Ed.), Population Biology and Evolution. Syracuse niche is available. Am. Nat. 87, 331–333. University Press, Syracuse, pp. 87–108.