Copyright 0 1994 by the Society of America

Epistasis Can Facilitate the of Reproductive Isolation by Peak Shifts: A Two-Locus Two-Allele Model

Andreas Wagner,* Giinter P. Wagner* and Philippe Similiont

“Yale University, Department of Biology, Center for Computational Ecology, New Haven, Connecticut 06520-8104 and Yale University, Department for Applied Physics, Becton Center, New Haven, Connecticut 06520-2157 Manuscript received October 13, 1993 Accepted for publication July 8, 1994

ABSTRACT The influence of on the evolution of reproductive isolation by peak shifts is studied in a two-locus two-allele model of a quantitative genetic character under stabilizing selection. Epistasis is in- troduced by a simple multiplicative term in the function that maps gene effects onto genotypic values. In the model with only additive effects on the trait, the probability of a peak shift and the amount of reproductive isolation are always inverselyrelated, i. e., the higherthe peak shift rate, thelower the amount of reproductive isolation caused by the peak shift. With epistatic characters there is no consistent rela- tionship between these two values. Interestingly, there are cases where both transition rates as well as the amount of reproductive isolation are increased relative to the additive model. This effect has two main causes: a shift in the location of the transition point, and the hybrids between the two alternative optimal genotypes have lower average fitnessin the epistatic case. A review of the empirical literature shows that the fitness relations resulting in higher peak shift rates and more reproductive isolation are qualitatively the same as those observed for genes causing hybrid inferiority.

OR sexually reproducing populations, between alternative optimal genotypes aswell as the F consistsin the acquisition of genetic differences amount of reproductive isolationbetween those that cause reproductive isolation of one such population genotypes. from other populations. Of all the possible modes of Considerable effort has been devoted to obtaining es- speciation genetic differentiation of isolated popula- timates of the peak-shift rates by modeling specific evo- tions seems to be prevalent in animals [MAYR(1942) ; see lutionary scenarios (BARTONand CHARLESWORTH 1984; also COYNE(1992) for a recentreview]. In this report, we BARTONand ROUHANI1987; LANDE 1985). In principle, consider a scenario in which a small population gets two kinds of models can be distinguished: In “pheno- separated from the main population without undergo- typic” models distinct phenotypes can occupylocal ing a founder event or several crash/flush cycles. We maxima in the fitness landscape. In “genotypic”models, further assume that genetic differentiation is mainly on the other hand,different combinations of genes may caused by random drift, i.e., there is no difference in lead to the same, optimal phenotype. The model studied selection onthe main population and thedaughter below is an example for the latter kind of models. population. In this case, genetic differences are either Assumptions about modes of interactions between neutral and therefore irrelevant for speciation, or they gene effects are essential for genotypic models. The constitute transitions among adaptively equivalent geno- most elementary of these assumptions is that geneshave types. In the latter case, the population has to cross an linear (additive) effects on the genotypic valueof a adaptive valley (a “peak shift” occurs) (WRIGHT1952). quantitative character. Peak-shift scenarios under the as- Such a transition event will cause a reproductive barrier sumption of purely additive gene effects have been ex- between the source population and its isolate. This is tensively studied (LANDE 1985;FOLEY 1987;BURGER 1988; because not all of the possible hybrid and back cross CHARLESWORTHand ROUHANI1988; BARTON1989a). phenotypes can be adaptively optimal, otherwise the dif- BARTON(1989a), for example, showed that peak shifts ference would be neutral. can occur at an appreciable rate, although the amount Genetic analysis suggests that initial stages of specia- of reproductive isolation obtained perpeak shift is small. tion may be caused by a small number (mostly pairs) of The overall rate of accumulation of reproductive isola- interacting loci (ORR1987, 1989;ORR and COYNE1989). tion has been estimated to be of theorder of the Therefore, peak shifts among alternative genotypes are rate. likely to involve onlyfew genes, if genetic differentiation To what extent the slow pace of evolution predicted is due to drift. We model this process in the framework by additive models carries over to epistatic models is of a two-locus two-allele model. We will ask how the ge- unclear [for areview, see BARTON(1989b) 1. The discus- netic architecture influences both, the rateof transition sion is mostly framed in terms of the contrast between

Genetics 138 533-545 (October, 1994) 534 A. Wagner, G. P.and Wagner P. Similion

type I architectures, where many genes have approxi- and Y, with two alleles each: X, and X, at locus X and Yo mately additive effects,and type I1 architectures, where and Y, at locus Y. Contributions to G by the underlying there are few genes with strongly epistatic effects and variables which are influenced by locus X are denoted many “modifier” loci. The results are difficult to gen- by the lowercase letters x,, x,, x, for genotypes &Xo, eralize, since there are countless many ways to model XoX,and XIXI, respectively. An analogous notation is epistatic gene interactions. In the present paperwe con- used for contributions yk+, of the genotypes YkYl at centrate on anepistatic model of a quantitative charac- the locus Y. The genotypic values for the underlying ter under stabilizing selection. The relative strength of variables are arbitrarily chosen to be x, = yo = 3, epistasis can be adjusted by changes in one parameter x, = y1 = 2 and x, = y2 = 1. Additionally, we suppose (see below). Thus, aone parameter class of models with that thereexists a smooth function g( x,y) of the under- different degrees of epistasis in the quantitative char- lyingvariables x and y which determines the geno- acter is defined. This class includes a purely additive typic value G, G := g( x, y). In this paper wewill be case. A member of this class which shows a high rate concerned onlywith two special cases ofsuch a function, of evolution of reproductive isolation is analyzed in alinear (additive) one, and a “mildly” non-linear detail and compared to the additivecase. A compari- (epistatic) one. son of this member to data about genescausing hybrid The linear functiong, is given by g, (x, y ) := e, + cIx + inferiority suggests that it is representative of many cg, c,, c, and c, being arbitrary real numbers. To specify genes involved in theemergence of reproductive the coefficients ci in the above functions, we introduce isolation. the following three assumptions. First, since we want to study the transition between alternative genetic adap- THE MODEL tations, it is assumed that two complementary double We consider a diploid randomly mating population homozygous genotypes have the same genotypic value with overlapping generations thatis either dioecious or zero, i. e., g, ( x,, y,) : = 0 and g, (x,, yo) := 0. Further, we monoeciouswith the same pattern of selection acting on assume that the contributions of the loci are symmetrical, both sexes. Heritable fitness differences areassumed to i.e., 9 = I+ These assumptions specify the additive geno- be due to the variation in one quantitative character P. type phenotype map up to a multiplicative constant: No genotype-environment interaction is assumed to be present, such thatthe genotypic value G andthe g,(x, y) := c (x + y - 4). (44 environmental contribution E are independent: Since fitness onlydepends on the product of st the con- P= G+E. (1) stant c can be chosen arbitrarily withoutloss of generality The environmental effect E is supposed to be normally asc= 1. distributed with mean zero and variance one. As a modelof the epistatic genotype phenotype map x, Selection is supposed to bestabilizing with P = 0 as g2( y) we use a modification of GIMELFARB’S(1989) the optimal phenotype. More precisely, we assign to additive/multiplicative model, which one obtains by each phenotypicvalue Pa fitness value mp(P) , defined adding a multiplicative term to the additive model: as g,(x, y) = g,(s y) + 4x9 97 m,(P) = 1 - SPZ, (2) where where s denotes a measure for the strength of stabi- lizing selection. However, since the environmental ef- v(x, y) = c, xy + e,. fect E is assumed to be superimposed onto G, it is necessary to evaluate the mean fitnessm( G)of a geno- Since we want to maintain that the two complemen- type with genotypic value G. By using (2) and the tary double homozygous genotypes have the same ge- above assumptions about the distribution of E, %( G) notypic value zero in order to have the same optimal becomes genotypes in both models, the interaction termu( x, y) has tobe zero for these two genotypes too. This specifies its coefficients up to a multiplicative con- stant: u( x, y) = b( xy - 3). In the resulting model for the epistatic character = 1 - s - sG2. g,(x, y) := c’(x + y - 4) + b (xy - 3) Because the selection dynamics with overlapping gen- erations is invariant to the addition of a constant to the strength of epistasis is determined by the ratio P = the fitness, we can ignore the term s in %( G). b/c’ . It is thus convenient to rewrite g2(x,y) as Furthermore, we assume that thegenotypic value G of g,(x, y) := c’[(x + y - 4) + P (q- 3)]. (4b) the quantitative character under consideration is deter- mined solely by the genetic composition of two loci X As in the linear model, e‘ only enters the fitness as set*, E pistasis and Speciation 535 Speciation and Epistasis

(CROWand K~MURA1970).

p, = -dP1 = p,(m(J) - -(j) m ) - + P(A+ - (54 dt a h

= -dA = &(X$) - djj) + rD + p(p, + p4 - 2A) (5b) p;, dt

p; =-=p(dP3 m3(j)- djj) + + p(p, + p, - 2~3) (5c) dt ,

dP4 p, = 2 = P4(my' - &'I) - rD + I*.(&+ p, - 2p4), (54

where p,, p,, p,, p, denote the frequencies of gametes (4 &Yo, &Y,, X,Yo, X, Y,, respectively. Linkage disequilib- yoyo YOYl YlYl Mz YOYl yoyo rium is denoted by D := p,p4 - p& and the recombi- XoXo 1 - 4~(1+3P)' 1 - s(1 + 3P)' 1 nation rate between loci is denoted by r. Forward and XoX1[1 - s(l+ 3P)Z 1 - spz 1 - $1 +P)Z backward ateach locus are assumed to occur XlXl 1 1 - ~(l+ P)z 1 - 4~(1+P)' I at the same frequency, p. Furthermore, (4 4 m(i): = mWp Mz YoyoYoYl YlYl x kl 1 (6) XoXo 1-36s 1-9s 1= 1 x.x,[l-gs 1-T 1-; is the marginal mean fitness of the gamete pk,and XIXl 1 1"; 1-483 4 FIGURE 1.-Genotype and fitness values. (a) Genotype val- ~(j)= m(j) ues, G,, for an additive character G, resulting from (4a), or x k Pk (7) from (4b) with p = 0 (c = 1). (b) Genotype values, G2,for an k= 1 epistatic model resultingfrom (4b) (c = 1). (c)Fitness matrix M2resulting from(4b) ( c = 1). (d)Fitness matrix M2,as in (c), represents the mean fitness of the population. but for p = -%. See text for details. Certain restrictions apply to the usage of equations (5) (CROWand KIMURA 1970; CHARLESWORTH1970). First, selection has to be weak with no dominance in the and can be treatedas an arbitrary constant c' = 1without loss ofgenerality. The factor /3 defines a class ofmodels death rates and small differences in the birth rates. where theamount epistasisof in the geno- Furthermore it is assumed that the interaction be- type-phenotype mapdepends on the parameter p tween mutation and selection can be neglected, al- (N. BARTON,personal communication). lowing for an additive contribution of the change of The resulting matrices for genotypic values, G, for the pi due to mutation to the total change of p,. Also, in a continuous time model the parameter ris in factthe prod- additive model (p = 0) and G, (p # 0) for the epistatic model, are shown in Figure la and lb,respectively. Ap uct ofthe recombination hction and the birth rate of the double heterozygote. Therefore, r 0.5 need not neces plying (3) to the genotype matrices G, := (g',") and 5 G2 := (g';)), and setting the maximum fitness equal to sarily hold. one, we obtain aset of fitness matrices denoted as Mk = The results are presentedin four parts: first, the struc- ture of the fitness matrices resulting from the assump (ml,k)),whereml,k) = 1 - s,(g',))>' (kE(1, 21, i,j€{O,1, 2)). M2 is depicted in Figure IC. tion of epistasison the characterwill be discussed,which Note that we use here a coeffkient sk instead of s to leads to the identification of a special case that is ana- specify the intensity of stabilizing selection. Since the lyzed in detail; second, the amountof reproductive iso- additive and the epistatic character model lead to dif- lation between alternative optimal genotypes is calcu- ferent average allelic effects, they also lead to different lated and compared to the additivecase; third,a selection intensities at the genic level. When these two deterministic analysisof the epistatic model sets the models are going to be compared, these differences stage for a numerical estimation of the rates of peak have to be taken into account. Since there are several shifts betweenalternative genotypes, which is the fourth ways to correct for these effects, we will maintain dif- part of the results. ferent coefficients for the selection on the additive and the epistatic character. RESULTS For investigating the selection forces that act on Qualitativecomparison between additive and epi- thegamete frequencies wewill use the following static character models:Epistasis isintroduced as a mul- standardmodel of ordinarydifferential equations tiplicative term in the function g specifying the genetic 536 A. Wagner, G. P. Wagner and P. Similion effects on the variation of the quantitative character. E g, (xty) := (x + y - 4) gZ(x,y) := [(x + y - 4) + - 311. The parameter p regulates the strengthof epistasisat the character level such that theadditive model can be con- sidered a special case (p = 0) of the epistatic model. Epistasis on the level of the quantitative character is to be distinguished from epistasis in fitness. Whereas in the additive model there are only additive effects of the ge- netic level on the quantitative character, the quadratic fitness function we use to model stabilizing selection in- troduces negative epistasis infitness, E < 0. The measure for the amountof epistasiswe use here is the coefficient FIGUREZ.-Coef€icient of epistasis in fitness E as a function of epistasis in fitness, E := rn, - m2 - m, + m4, intro- of p, The coefficient of epistasis infitness, E (CROWand KIMURA 1970), defined as - - is shown as a function duced by CROWand KIMURA (1970). (Note that this E m, m2 m3 + m, of p in the center of the simplex ( p, = p2 = p3 = p, = ?A). It differs from the E in (l).)The epistatic model, on the is closest to zero for /3 = -0.4. Note that, since the fitness other hand, incorporatesepistasis alsoon the level ofthe function used is nonlinear, E for a completely additive char- character, such that an increase in I p I leads to an in- acter (p = 0) is not equal to zero. crease in epistasison this level.The outcome in terms of the amount of epistasis in fitness is no longer a mono- reach the alternative genotype can be less deep than the fitness valleyin which the hybrids are located. Below, we tonic function of p, as canbe seen from Figure 2, which will use the special case of g2( y) with = -% as ex- shows the coefficient of epistasisE in fitness as a function x, p ample for the deterministic analysis and the Monte- of the parameter p. It indicates that for /3 = 0 (additive Carlo simulations. This case will allowus to demonstrate character) epistasis for fitness is strong (E# 0) and that most clearly the difference between the additive char- it approaches zero from below, as p approaches -0.4 acter case and the epistatic character case. The fitness (for p, = X). For p > -0.4 it deviates from zero again. matrix M2 for p = -% is shown in Figure Id. For positive values of I E I always increases. 0, One further remarkseems appropriate. The existence For all values of /3 the homozygous genotypes two of genotypes with high fitness between the alternative &&Yl Yl and XIX, YoYo have the highest fitness mZ2 = homozygous genotypes does not imply that there is no m33= 1. Further, in the case of an additive character, the adaptive valley separating them. For instance, in the case double heterozygous genotype also has maximal fitness. of an additive character under stabilizing selection, the This is the genotype of the F, hybrid between the alter- double heterozygous genotype has maximal fitness. native optimal genotypes. Whenever p is unequal to However, there exists an adaptive valley between the two zero, the double heterozygous genotype is less fit than optimal homozygous genotypes, because the offspring the two optimal homozygous genotypes. of double heterozygous individuals have a high segre- The fitness matrix in the additive character model is gation variance and therefore low average fitnessunder symmetric. In particular, m22 = m3, = 1 and ml, = m44 stabilizing selection. The existence of intermediate < 1 holds. As epistasis is introduced, this symmetry of m,, genotypes with high fitness alone is thereforenot and m44 does in general not hold. For /3 E (- ?4, O), m44 sufficient to predict how easy an alternative optimum < mll and for /3 < -%, m44 > m,,. Interestingly, for fJ= genotype can be reached. -%, mll = m44 as in the additive case, and the fitness The amount of reproductive isolation: In this section landscape for this parameter value resembles the fitness it is discussed how epistasis in the character influences landscape with an additive quantitative character. Most the amountof reproductive isolation between the alter- importantly, the fitness of mIl or mq4 (depending on P) native optimal genotypes. Since our model is based on can be larger than thefitness of the doubleheterozygous viability selection, only the evolution of postzygotic iso- genotype (Figure 3). Consider, for example, the differ- lation can be considered here. In this case, the amount ence between m44and mI4,m,, - m14.This value is larger of gene flow between partially isolated populations de- than zero in the interval -2 5 p 5 -%; it is maximal pends on the intensity of selection against hybrids. We for p = "$4. Similarly, there exist valuesfor p for which analyze the fitness of the Fl hybrids and the genotypes m,, is larger than the fitness of the double heterozygote that result from back crosses B into one of the parental genotype. Hence, depending on /3 there may be geno- populations. These values directly measure the selection types intermediate to the two alternative optimal geno- against the most frequent hybrid genotypes (F, and B) . types, which havehigher fitness than theF, hybrid. This An alternative would be to use the mean fitness of a means that the fitness valley which has to be crossed to hybrid populationinterbreeding among themselves Epistasis and Speciation 537

crosses is

1.51 and in the epistatic case it is

In both cases, recombination between the two interact- ing loci increases the amountof reproductive isolation, P because it producesunbalanced genotypes. Further- more, if s, = s, the mean fitness of the back crossgeno- types is lower in the case of an epistatic character than for an additive character, whenever /3 > 0 or /3 < -S/(l + 9r). For /3 = -% the mean fitness of the back cross FIGURE 3.4omparisonof fitness ofF, hybrid ( m29)with m,, genotypes is and m44.The difference between m,, or mW, whichever is larger, and the fitness,?,, of the F, hybrid between the two ?qB)=l-% -+- optimal genotypes, Le., ma%(m,,, m,,) - ?s, is shown as a (: ‘:3 function of p. Positive values indicate that thereare genotypes intermediate between the two alternative optimal genotypes which is much less than for additive characters, even if which have higher fitness than the F, hybrids. These are the s, is larger than s, by a factor of four to five. cases where the population has to cross an adaptive valley Both the Fl hybrids and theback crossesare less fit in which is less deep than the fitness of the hybrids. For ad- the epistatic case than in the additive model. Conse- ditive characters, this value is zero. In the case of p = -%, m,, is maximal compared to the hybrid fitness, and the dis- quently, for a wide range of values of the ratio sl/s2, the crepancy between the adaptivevalley and the hybrid fitness amount of reproductive isolation caused by peak shifts is largest. This is also the case which is analyzed in detail is higher in the epistatic character model than in the below. additive character model. Deterministic analysis: The fitness matrix M, gener- (BARTON1989a). However, hybrids usually do not exist ated by the genotype matrix G, is a special case of the in isolated populations, but are part of a hybrid zone fitness matrix of the so-called symmetric viabilitymodel, interbreeding with the two parental populations. Gene which has found considerable attention since it has been flow between parental populationsis caused by matings first investigated by WRIGHT(1952). Rigorous analytical between hybrids and members of the parental popula- work in the case ofp = 0 has been carried out by KARLIN tions (back crosses). We therefore think that the fitness and FELDMAN(1970) as well as by BODMERand FELSEN- of hybrid genotypes is a more adequate measure of re- STEIN (1967).More recently, BURGER(1989) has carried productive isolation than the mean fitness of a isolated out a global analysis of exactly the dynamics generated hybrid population. by MI using (5). Since his results for the special case of Because the two alternative adaptive genotypes are p = 0 are of interest to our analysis and since they will thedouble homozygous genotypes X,X,YoYo and be used inthe sequel, we shallbriefly recapitulate %XoY,Y,, the F, hybrids are the double heterozygous them using the notation introduced above. First, it has genotypes X,,X,YoY,. For the additive character model, been shown that all orbits in the linear model converge these genotypes have always the same genotypic values to the plane p, = p,. Second, there exists an unstable as the optimal ones. In the epistatic case, the mean fit- equilibrium point F1 in this plane, given by ness in the F, generation is 1 - s$‘, because the epistatic effects cause a displacement from the optimum of the fj1=lj4=?4+D1, fj*=fj3=?4-D1, mean phenotype of the F, hybrids. The mean fitness of the back crossesB with one of the parental populations where depends on the recombination rate,because the F, hy- brids produce all four possible gametes by recombina- SI 1 tion. Consequently, four genotypes have to be taken into L)’=--ivw+; account to estimate selection against the backcross Orbits starting in the interiorof the manifold defined by genotypes with,say, thepopulation consisting of p, = p, converge to F1. Third, two locally stable “vertex” X,X, YoYo genotypes: equilibria, F2(F3), existwith coordinates j, = 1 (b3= 1). l-r l-r r r Their domainof attraction is the setwhere p, > p, ( p, < ?E@)= -2 mzZ+~mz,+pi+p. p,) . The above results imply that theequilibrium state of the population is influenced by its initial state. Transi- For the additive character case the mean fitness of back tions from the region of S where p, < p, holds, to the 538 A. Wagner, G. P. Wagner and P. Similion

(4 cross it. They spend more time in the vicinity of p, = 1 P4” than those in the additive model. This becomes evident by numerically integrating (5) for different initial con- ditions, as shown in Figure4 for the additive and for the epistatic case, respectively. Although the final outcome for both models in the absence of mutation will be the same (in that the stable equilibria in this case reduce to = or = Pg=l p, 1 p, l), these differences in the dynamics of the two models are importantfor the stochastic analysis carried out below, since it is the location of the saddle points within the plane p2 = p3,as well as the path of the trajectories that define the region where peak shifts take place and the above differences are responsible for the P4= 1 difference between the additive and the epistatic model. Peakshift probabilities estimated by Mont4arlo simulations: Stochastic perturbations of the determin- istic dvnamical systemdefined in (5)were used to com- pare transition rates for the additive and the epistatic model. The differential equations used are identical to those in (5), except for an additivestochastic term 5 := (q,. . . , v4), Le., they are given by (6j5/6t),,,, := FIGURE4.-Phase diagrams for epistatic and additive model. (6j5/6t),,, + 5, where (6j3/6t)delrepresents the right Numerical integrationof (5) for a series of initial conditions close hand side of (5).The stochastic term 5 represents the to the intersectionof the plane wheref+ = A (gray shading) and effects of . It is generated a pseudo- the plane wherep, = 0 in the domain f+ > py 5 = 0.001; r = 0.01; as p = 0; (a) Additive model; (b) Epistatic model; In the additive random multidimensional Gaussian deviatethat has the case, the trajectoriesapproach the (invisible)planep, = p4 before exact co-variance matrix, ie., the one corresponding to converging to f+ = I, whereas in the epistatic case trajectories are a multinomial distribution approaching pi = 1 before convergingto f+ = 1 very close to the edge where pL + pi = 1. (Vi) = 0 region where p, > p, can only be expectedif there is also /1 a stochastic componentthat influences the gamete frequencies, such as genetic drift. Analysis of the dynamics of (5) with p = 0 and the fitness matrix M, with p = -% (Figure Id), i.e., the epistatic case, reveals dynamicsthat arequalitatively dif- where N, is the effective population size. This Gaussian ferent from the additive case. Here we only present a approximation is adequate provided the componentsof summary ofa moredetailed analysis which carriedis out $are nottoo small. Becausethe equations used here are in the APPENDIX. In the epistatic case there also exist two continuouwalued approximation of a discrete 5, allele locally stable vertex equilibria p2 = 1 ( p, = 1) with the fixation has to be enforced for small pi. In order to rep set p, > p,(p, < p,) as domain of attraction, and one resent the effects of mutations correctly, the following unstable equilibrium point within the plane p, = p,. The algorithm was used: if pi E [0,1/2N,], it is reset to 1/2N, \atter tq-dibiurn i3 a sa&d’~e, i. t., it has a WD- witkpm~~a~~ity23cg~a~~t~Owithp~~a~l~~1-?X$,. dimensional stable manifold that is located within that After any such fixation step, normalization of gamete plane p, = p,. Thus, the number ofequilibria and their frequencies to a sum of 1 was performed. stability properties arevery similar in the additive and in For estimates of shift times betweenp, > p, and p2 < the epistatic model. However, an important difference p, a population was initialized at the optimum genotype between the additive and the epistatic model is that the ( p, = 1) and let to evolve until genetic drift caused it to respective equilibrium points in p, = p, are “semi- cross the plane of symmetry corresponding to the valley qmmeuical (A # A) in &e episatic case (see MPEW hat sepamtes rhr two basjns oFattmct;on.c ;n the .*J.*p DIX), but “symmetrical” ( pl = p4) in the additive case tive landscape ( p, = p,). Crossing time and crossing (BORGER1989). In fact, the saddle point within the p2 = coordinates in the simplex were recorded and the S~S- p, plane is shifted towards the vertex p4 = 1 in the epi- tem was reinitialized. The following results werederived static case.A further, qualitative difference can be seen from the statistical distribution of crossing events thus in the path that trajectories leading to the vertex equi- obtained. libria take in the two models. Trajectories in the epistatic Simulationswere carried out with large or small popu- model do in general not approach theplane p, = p4 but lation sizes ( Nr5, > 1 or Nrs, < 1) in the regime of small Epistasis and Speciation 5.79

300

200

100

0 ...... , 0 0.2 0.4 0.6 0.8 1

500

400

300

200 FIGURE 6.4omparison peakof shift frequencies. Transi- tion frequency r normalized to mutation rate p as a function 100 of N,s,. (a) Additive model: sI = 0.156; r = 0.5. (b) Epistatic model: s2 = 0.156; r = 0.5. Standard errors for all valuesgiven 0 are 2%. Note that the exponential approximation of r as a 0 0.2 0.4 0.6 0.8 1 function of N,s, only holds for small mutation rates. FIGURE5.4omparison of transition point distributions. Histograms of the frequency p4 at the crossing point ( p2 = p,). (a) Additive model: N,s, = 2.34; r = 0.5;N,p = 0.06. (b) Epi- static model: Nts2= 15.6; r = 0.5; N,p = 0.4. Standard errors sk within an order of magnitude while keeping N,s, and for all values given are 2%. Note that inthe epistatic model the p/sk constant (results not shown). transitions occur at much higher frequencies of p4 than in the In Figures 6, a and b, for the additive and theepistatic additive model (see text for explanation). case, respectively,crossing times Tare plotted as a func- tion of effectivepopulation size N,, selection coefficient mutation rates ( N,p << 1). Histograms of the distribu- sk, and mutation rate p. More precisely, crossing rates tion of p,coordinates of the crossing points are pre- normalized to the mutation rate, r/p, are shown. sented in Figure 5, a and b, for the additive and the Our numerical results show that the transition fre- epistatic character, respectively. They show that, pro- quency is wellapproximated by an exponentialfunction vided that Nrsk islarge enough, the trajectories tend to of N,sk. One obtains for the additive model cross the dividing plane p2 = ps nearthe unstable rl/p - O.Gexp(-0.96N,sl) equilibrium points. Note that the locations of the saddle points are very different forthe two models, as predicted with a mean fitness at the saddle point of 6")= 1 - above. 0.85~~.In the epistatic case one finds It follows fromexamination of the Kolmogorov- r2/p- 1.2exp(-0.2Nf%) Fokker-Planck equation (EWENS1979) which describes the evolution of the probability distribution of the popu- and a mean fitness at the saddle point of d2) - 1 - lation state that the shift frequency r = 1/T is a homG 0.23~~.Note the dependence of r on the mutation rate geneous function of order (+ 1) in the four arguments (Figure 6b). For small NIsk,r is proportional to p. How- (p,s, l/Nr, r).One can deduce from this symmetry that ever, for larger values ofN,s, r is proportional to a power r/p is a functionof three argumentsonly, for instance, of p higher than one: accordingto the analytical results ( N,s, p/sk, sJr). If r is much larger than sk one can ar- given by BARTONand ROUHANI (1987), r should be gue that recombinationwill enforce linkage equilibrium proportional to p2 for very large values of N,s. but will not furtheraffect the solution,since the regime The ratio of the transition probabilities R is will have entered the asymptotic region where sk/r is r2 nearly null. Under this condition, the shift time nor- R = - = 2.0 exp(N,(0.96sl - 0.2%))> 1. malized to themutation ratewill depend asymptotically rl on two arguments only, N,sk and p/sk. We were able to As mentioned above, the effect of a genesubstitution is verify numerically that this asymptotic regime was different in the additive andthe epistatic character reached, since r/p was not affected by varying N,, p and model. Therefore, the intensity of stabilizing selection 540 A. Wagner, G. P.and Wagner P. Similion has to bescaled such that thetransition probabilities can under stabilizing selection influence the rate of accu- be compared. Below we willdiscuss three ways ofscaling, mulation of reproductive isolation by peak shifts in a one with s1 = sp,one where the average effect of a gene small and isolated, but stable population. More specifi- substitution is held constant and one where the fitness cally, we are interested in the influence of epistatic in- at thetransition point is kept thesame for thetwo mod- teractions among quantitative trait loci on the rate of els. The ratio of the transition rates R implies that fors2 evolution of reproductive isolation. A class of two-locus < 4.8~~peak shifts are more likely in the epistatic char- two-allele models was analyzed, for which the degree of acter case than in the additive case. The exact magni- epistasis among quantitative trait loci was adjustable via tude of the advantage depends on how one scales the a parameter, p. p = 0 results in a completely additive selection intensities on the additive and the epistatic character, whereas deviations from /3 = 0 yield continu- character and what values of N,s are reasonable. ously varyingdegrees of epistasis.The value /3 = - %, as Values for the intensity of stabilizing selection have discussed above, allows us to illustrate most clearly our been compiled by ENDLER(1986). He uses the effects of central claim, namely that epistasis may allow for acon- stabilizing selection on phenotypic variance to measure siderable increase in peak shift rate, in comparison to an the intensity of stabilizing selection additive trait. We therefore focused on a comparison of the cases P = 0 and p = -%, referring to them in the Vh - VP j = ~ following asthe additive model and the epistatic model, VP respectively. It has to be pointed out that this property where V, and V,, denote thephenotypic variance before of increased peak shift rates is not aproperty of epistatic and after selection, respectively. In terms of our model interactions in and by themselves, but ratherof a certain and using V, = 1 one obtains type of epistasis. Cases are conceivable where epistasis will decrease peakshift rates. However, it will be argued -j sk= . below that there is empirical evidence in favor of the I + VP(1 + 11. fitness scenario presented here. Assuming a heritability of about 0.5, one obtains The impact on speciation of the architecture of the genetic system has been discussed mainlywith reference -j to the contrast between the so-called type I and type I1 s, = - 5j + 3' architectures (TEMPLETON1982; BARTON and CHARLES- WORTH, 1984; CARSON and TEMPLETON1984; BARTON Given the values ofj compiled by ENDLER, = 0.1 is quite sk 1989b). Type I architecture assumes many genes with realistic in wild populations. This selection coefficient small and approximately additive effects, while type I1 easily leads to values for Neskthat areof the orderof 10. architecture assumes strong epistasis betweengenes and If selection on the epistatic character is as strong as on genes with unequal effects (TEMPLETON 1982).The im- the additive character, s1 = sq, then the predicted ratio plications of type I architecture for the accumulation of peak shift probabilities easily exceeds three orders of of reproductive isolation arequite well understood magnitude. (BARTONand CHARLESWORTH1984; BARTON 1986, Another way to adjust for the differences in the epi- 1989a,b), andessentially state that slow accumulation of static and the additive character model is to hold the small contributions to postzygotic isolation is the most average effect on the character of a mutation atone of likely mode of speciation with typeI architecture.While the loci constant. For p = -%, the appropriate adjust- there is ample evidence for the involvement of epistati- ment is sp = 0.64~~= s. Under these conditions, the ratio cally interacting genes in species differences [see reviews of the transition probabilities exceeds four orders of by CARSONand TEMPLETON(1984) and COWE(1992) 1, magnitudes as Nes exceeds a value of 11. Finally, it is the importance of epistasisfor the speciation process is possible to adjust the selection intensity such that the controversial. It has been shown that certain type I1 mod- mean fitness at the saddle point is the same for the epi- els predict approximately the same rate of accumulation static and the additive character (s2 = 3.7~~).This is the of reproductive isolation as the type I model (BARTON most conservative assumption. Under these conditions, 1989b). Below, it will be discussed how the higher rate the ratio still exceeds one order of magnitude. Hence, of divergence, predicted by the epistatic model analyzed for moderately large populations and realistic strengths here, is related to the genetic architecture assumed in of stabilizing selection (ENDLER1986) epistasis in the genotype phenotype map is expected tohave a substan- the model. The genetic architecture of the trait considered in the tial influence on the rate of peak shifts as compared to epistatic model results from two assumptions. First, it is an additive character. assumed that the effect of the two genes on the pheno- typic characters is mediated by their independent effects DISCUSSION on two underlying physiological or developmental traits. The question considered in this contribution is: how Second, it is assumed that these underlying variables in- does thegenetic architecture of a quantitative character teract nonlinearily to produce thephenotypic character, Epistasis and Speciation 541 which leads to epistatic interactions between the genes. fit genotypes to reach thealternative genetic . In theepistatic model analyzed here, we assume the low- The fitness of the intermediate stages in a peak shift is est possible degree of nonlinearity, namely a mixed sec- independent of the fitness of the inter-population hy- ond orderterm of a polynomial. Two differences to the brids or the backcrosses between the F, and one of the additive model in the genetic architecture are thereby parental populations. The existence of such a chain of introduced. First, the double heterozygote genotype, relatively fit genotypes provides a qualitative explana- i.e., the hybrids between the two alternative tion of themathematical results presented in this paper. &X,Y, Y, and X,X, YOYO,are not optimal. In theadditive For instance, in our epistatic model the saddle point in case the doubleheterozygote has the same phenotype as the plane p, = p, is displaced towards the vertex p, = 1 the two alternative homozygote genotypes due to the of the simplex, as opposed to the additive model where mathematical properties of additivity. The second dif- the correspondingfixed point occupies a location close ference is the effect of a gene-substitution at one of the to the center of the simplex. The displacement of this optimal genotypes. In the additive model used for com- fixed point in the epistatic model is due to much stron- parison, the allele substitution at either locus has the ger selection against the gamete X, Yoas compared to the same effect on the phenotypeand is thus a simple model gamete X,Y,. (In the additive model, selection against of type I architecture. In the epistatic model, effects of these two gametes is equally strong.) This can be seen substitutions at the X and the Y locus are not thesame, from the lower fitness of the genotype XoX,Y,Y, relative representing a type I1 architecture. For /3 = -%, if one to the fitness X,X,Y,Y, (Figure Id). Hence, selection considers the effect of allele substitutions at the geno- pushes the population towards higher frequencies p, of type &&Y,Y,, the substitution of X, by X, would have X, Y,,ie., towards the vertex p, = 1 of the simplex. The an effect of -Vi on the characterp, while a substitution location of the saddle point within p, = p3 is important, at the Y locus would have the effect 3. Hence, Y may be since peak shifts occur in its neighborhood (Figure 5). called the major locus and Xtheminor locus in this case. Its displacement in the epistatic case avoids the produc- However, the role of the major and the minor locus tion of genotypes with very lowfitness, such as X,,& YOYO, switches weif considerthe alternative genotype duringthe peak shift. Metaphorically speaking, the X,X,Y,Y,. In this case the X locus has a much greater population shifting between peaks followsa path around effect than the Y locus. Hence the nonlineargenotype- the adaptive valley. phenotype map induces a context dependent type I1 The presence of context dependent type I1 architec- architecture of the trait The role of the major and the p. ture is detectable by the genetic techniques used by ORR minor locus depends on thegenotype we consider. This and COYNE(1989) for theanalysis of species hybrids. The contextdependent type I1 architecture of a quantitative prediction is that theepistatic interaction between pairs trait resembles, if combined with stabilizing selection, of locicausing postzygotic isolation leads to an inversion the phenomenon of complementary deleterious alleles of the relative magnitude of effects ofgene substitutions [see, e.g., SAUNDERS(1952)l: in the simplest case the two at the differentloci in the two parental genotypes. Epi- populations have the complementary two-locus geno- types LIL1l$, and llllL2L2,where in the hybrids each static models have been used by DOBZHANSKY(1937) and genotype containing both L, and L, alleles are lethal or MULLER(1942) to show that there is no necessity for have reduced viability,while the genotypes 1, l,L,l,, crossing an adaptive valley during the evolution of in- L, I, 1212 and 1, I, 41, are less affected or not atall affected. compatible genotypes. The model assumes epistatic This context dependent type I1 architecture is the major interaction between two loci A and B which diverge in- difference of the presentepistatic model to the epistatic dependently in two isolated populations from A,A,B,B, models criticized by BARTON(1989b). In BARTON’Smodel to A,A,B,B, and A1A1B2B2,respectively. Reproductive the identity of minor loci and major loci is predeter- isolation occurs if the alleles A, and B, are incompatible. mined by the model and epistasis only defines which In this case no adaptive valley hasto be crossed to evolve gene combination is “coadapted.” Due to the multipli- reproductive isolation. The epistatic model analyzed in cative interaction between the primary gene effects in this paper is similar to theDOBZHANSKY-MULLER model, as our model, however, these roles are context dependent. it involves epistatic interactions and does not require the The key feature of the epistatic model considered populations to cross an adaptive valley that is as deep as here is that a chain of genotypes with high fitness exists the fitness of the inter-population hybrids. The differ- that connects the two alternative optimal genotypes. In ences are that,in our model, the two gene substitutions our notation, these genotypes are X,&Y,Y,, X,X,y,y,, required to get reproductive isolation take place in the and X,X,Y,Y, (see Figure Id). These genotypes differ same population and that the build-up of reproductive only by one allele substitution per stepand connect the isolation occurs by peak shifts. alternative genotypes X,X,Y,Y, and X,X,YoY,. All these Both, the DOBZHANSKY-MULLERmodel as well as the genotypes are much morefit than the doubleheterozy- present model assume that reproductive isolation is pri- gote or the other genotypes, e.g., the &&,YOYO geno- marily due to a few epistatic loci. Not much is known type. Hence, peak shifts can trace this chain of relatively about the genetic basis of reproductive isolation, but 542 A. Wagner, G. P.and Wagner P. Similion

there are several examples that demonstrate strongep context-dependent type I1 architecture, in which a chain istasis among a limited number of loci causing reduced of viable genotypes connects alternative optimal states, hybrid fitness and hybrid break down(WIEBE 1934; while F1 hybrids between these optimal genotypes are SAUNDERS 1952; HERMSEN1963; OKA1974; MACNAIRand inviable or lethal. This ultimately allows a much higher CHRISTIE 1983;CHRISTIE and MACNAIR1984; ORR1987, rate of accumulation of postzygotic isolation than for 1989; ORRand COYNE1989). Furthermore, it has been other genetic systems. However, the question remains demonstrated in five cases that the loci involved act as whether atall peak shifts are a reasonable model for the complementary deleterious/lethal genes with fitness re- evolution of reproductive isolation. This question has to lations qualitatively similar to our case p = -4/3 (WIEBE be raised, since the rate of transitions in the epistatic 1934; SAUNDERS1952; CHRISTIEand MACNAIR1984; model is still only of the order of the mutation rate ANDERS et al. 1984; TEMPLETONet al. 1985). (see RESULTS). Of particular interest is tumorigenesis in hybrids be- To evaluate the plausibility ofthe peak shift model as tween Xiphophorus maculatus and Xiphophorus helleri an explanation of speciation we use the data of COMVE (GORDON1927; ~USSLER1928; KOSSWIC1928) [for re- and ORR(1989) on the degree of reproductive isolation view see ANDERS et al. (1984)], because its molecular between closelyrelated species of Drosophila. They have basis is wellunderstood in comparison to other systems. shown that the degreeof postzygotic isolationincreases This form of hybrid inviability is caused by the interac- approximately linearly with the NEI'Sgenetic distance D. tion between a dominant oncogenTu, which is a tyrosine Postzygotic isolation reaches avalue of 1 (total isolation) kinase gene, and a regulatory gene R (WIITBRODTet al. among species pairs with an average D of 1 (COYNEand 1989). Melanoma formation occurs whenever the Tu al- ORR1989, Tab. 2). NEI (1987) suggests that in many lele is separated by recombination from its regulator species D = 1 corresponds on the average to a diver- gene at theR locus. This system exhibits the same kind gence time of 5,000,000 years, whichis also in agreement of context dependent type I1 architecture that is char- with molecular, biogeographic and paleontological data acteristic of the modelanalyzed here. Startingwith geno- for Drosophila (POWELLand DESALLE1994). Taking into type TuTuRRa substitution of the R allele has a major account the generation time of Drosophila, this easily effect on viability while a substitution of Tuhas at most leads to a number of generations in excess of107. Hence, a minor effect. In contrast, if we start with the comple- in Drosophila, postzygotic isolation evolves during a pe- mentary genotype that is fully viable,t,t,rr, a substitution riod of more than lo' generations in allopatry. at the Tu locus, replacing tu by Tu, has a major effect, The additive model can not be compared to this data while a substitution of r by R has only minor effects. since it always predicts optimal phenotypes for the F1 Hence, epistasis that results from interactions between hybrids. Our epistatic model assumes that each peak regulatory genes and primary genes area realization of shift reduces thefitness of the F1 hybrids by about - sbp2. the model analyzed in this report. With our value of p = -4/3 and a sq value of 0.05 (see Another example for which the molecular basis of a above for estimates), it takes about 10 peak shifts to context dependent type I1 architecture is known, is the reach total postzygotic isolation. Thus, the question is abnormal abdomen syndrome (aa) of Drosophila mer- how large the transition probability has to be in order catorum (TEMPLETONet al. 1985). Two loci interact in to expect about 10 transitions in lo7 generations? OB causing the aa syndrome. One is a 5-kb insertion into viously, anytransition probability of the orderof 1O"j is some copies of the 28s rRNA genes, the second is an sufficient to reach total postzygotic isolation withinthe allele at a different locus responsible for differential rep time frame observed for Drosophila. Assuming a muta- lication of the noninsertedcopies of the 28s rRNA genes tion rate at the relevant lociof abouttothe during polytenization. The aa phenotype requires the predictions of our epistatic simulation model are com- presence of inserts in the rRNA genes as well asa failure patible with the Drosophila data. Even though one to replicate the noninserted rRNA genes differentially. should not puttoo much weight on the exact numerical There aretwo optimal homozygous genotypes, one with values, the calculations show that the model is in quali- the insertions and the genefor differential replication tative agreement with observed rates of speciation. of the noninserted genes and one with neither the in- The reason why low transition rates are compatible sertion nor the differential replication allele. However, with the observed rates of divergence is that a peak shift the genotype without insertions and with the differential model has to assume the existence of severalpairs of loci replication gene is also viable. Here, again, two alter- able to undergo a peak shift to an alternative optimal native genotypes areproducing imbalanced hybrids genotype, each contributing a fraction of the total re- (with the aa syndrome), but arenevertheless connected productive isolation. In our example we have to assume by a chain of viable genotypes. that there are atleast 10 independent pairs of interact- In summary, according to the results presented here, ing loci contributing to total postzygotic isolation. If epistatic gene interactions arelikely to contribute to the there areten pairs of loci, and the transition probability evolution of reproductive isolation. They may lead to a at each locus is of the order of the mutation rate, the Epistasis and Speciation 543 total probability to get any transition is approximately CROW,J. F., and M. KIMURA, 1970 An Introduction to Population Ge- netics Theory. Harper & Row, New York. ten times as large. Hence, due to thefact that a peak shift DOBZHANSKY,T., 1937 Genetics of the Evolutionary Process. Columbia model has to assume the existence of many genes con- University Press, New York. tributing to postzygotic isolation, the probability of any ENDLER,J. A,, 1986 Natural Selection in the Wild. Princeton University Press, Princeton, N.J. transition taking place is higher than theprobability for EWENS,W. J., 1979 MathematicalPopulation Genetics. Springer, each locus. Berlin. The scenario analyzed above is one where two stable, FOLEY,P., 1987 Molecular clock rates at loci under stabilizing selec- tion. Proc. Natl. Acad. Sci. USA 84 7996-8000. moderately large populations exist for a long time in GIMELFARB,A., 1989 Genotypic variation for a quantitative character allopatry. The present model doesnot predict a higher maintained under stabilizing selection without mutation: epista- rate of speciation in the case of bottle necks or founder sis. Genetics 123 217-227. GORDON,M., 1927 The genetics of a viviparous topminnow Platy- events. The reason is that the amountof polymorphisms poecilus; the inheritance of two kinds of melanophores. Genetics existing in mutation selection equilibrium is not much 12: 253-283. higher in our epistatic model thanin the additive model ~USSLER,G., 1928 fiber Melanombildingen beiBastarden von Xiphophoms helleri und Platypoecilusmaculatus var. rubra. Klin. (A. WAGNER,unpublished results). Hence, the likeli- Wochenschrift 7: 1561-1562. hood thata founder populationis shifting to a different HERMSEN,J. G. T., 1963 The genetic basis of hybrid necrosisin wheat. peak is roughly the same in the additive and theepistatic Genetica 33: 245-287. HOFBAUER,J., and K SIGMUND,1988 TheTheory of Evolutionand model. Dynamical Systems. Cambridge University Press, Cambridge. KARLIN, S., and M. W. FELDMAN,1970 Linkage and selection: two locus A.W. and G.P.W. are indebted to REINHARD BURGER,JIM CHEWRUD symmetric viabi!ity model. Theor. Pop. Biol. 1: 39-71. and ALAN TEMPLETONfor numeroushelpful suggestions to improve the KOSSWIC,C., 1928 Uber Kreuzungen zwischen den Teleosteern Xi- manuscript and to NICK BARTONfor the generalization of the original phophorus helleri und Platypoecilusmaculatus. Z. Indukt. model. This article is communication no. 4 from the Center for Com- Abstammungs Vererbungsl. 47: 150-158. putational Ecology of the Yale Institute for Biospheric Studies. LANDE,R., 1985 Expected time for random genetic drift of a popu- lation between stable phenotypic states. Proc. Natl. Acad. Sci.USA LITERATURE CITED 82: 7641-7645. MACNAIR,M.R., and P. CHRISTIE,1983 Reproductive isolation as a ANDERS,F., M. SCHARTL,A. BARNEKOW and A. ANDERS,1984 Xiphopho- pleiotropic effect of copper tolerance in Mimulusguttatus. rus as an in uiuo model for studies on normal and defective con- Heredity 50: 295-302. trol of oncogenes. Adv. Cancer Res. 42 191-275. MAW, E., 1942 Systematicsand the Origin of Species. Columbia BARTON,N. H., 1986 The effects of linkage and densitydependent University Press, New York. regulation on gene flow. Heredity 57: 415-426. MULLER,H. J., 1942 Isolating mechanisms, evolution and tempera- BARTON, N.H., 1989a The divergence of a polygenic system subject ture. Biol. Symp. 6: 71-125. to stabilizing selection, mutation and drift. Genet. Res. 54: 59-77. NEI,M., 1987 MolecularEuolutionary Genetics. Columbia University BARTON,N. H., 1989b Founder effect speciation, pp. 229-256 in Spe- Press, New York. ciation and Its Consequences,edited by D. Otte andJ. A. Endler. Om, H.-I., 1974 Analysis of genes controlling F1 sterility in rice by Sinauer Associates, Sunderland, Mass. the use of isogenic lines. Genetics 77: 521-534. BARTON,N. H., and B. CHARLESWORTH,1984 Genetic revolutions, Om, H. A., 1987 Genetics of male and female sterility in hybrids of founder effects, and speciation. Annu. Rev.Ecol. Syst. 15: Drosophilapseudoobscura and D. persimilis. Genetics 116: 133-164. 555-563. BARTON,N. H., and S. ROUHANI,1987 The frequency of shifts between Om, A., 1989 Genetics of sterility in hybrids betweentwo Subspecies alternative equilibria. J. Theor. Biol. 125: 397-418. of Drosophila. Evolution 43 180-189. BODMER,W. F., and J. FELSENSTEIN,1967 Linkage and selection: theo- Om,H. A., andJ. A. COWE,1989 The genetics ofpostzygotic isolation retical analysisof the deterministic two locus random mating in the Drosophila uirilis group. Genetics 121: 527-537. model. Genetics 57: 237-265. POWELL,J. R., and R. DESALLE,1994 Drosophila molecular phylog- BURGER,R., 1983a On the evolution of dominance modifiers. I. A enies and their uses. Evol. Biol. 28 (in press). nonlinear analysis. J. Theor. Biol. 101: 585-598. SAUNDERS,A.R., 1952 Complementary lethal genes in the cowpea. BURGER, R.,198313 Nonlinear analysis of some models for the evo- S. Afr. J. Sci. 48: 195-197. lution of dominance. J. Math. Biol. 16: 269-280. TEMPLETON,A.R., 1982 Genetic architectures of speciation, pp. BURGER, R.,1983c Dynamics of the classical genetic model for the 105-121 in Mechanisms of Speciation, edited by C. BARAGOZZI. evolution of dominance. Math. Biosci. 67: 125-143. Alan R. Liss, New York. BURGER,R., 1988 Mutation-selectionbalance and continuum-of- TEMPLETON,A. R., T. J. CREASEand F. SW, 1985 The molecular alleles models. Math. Biosci. 91: 67-83. through ecological genetics of abnormal abdomen in Drosophila BURGER, R.,1989 Linkage and themaintenance of heritable variation mercatorum. I. Basic genetics. Genetics 111: 805-818. by mutation-selection balance. Genetics 121: 175-184. WIEBE,G. A., 1934 Complementary factors in barley giving a lethal CARSON, H. L., and A. R. TEMPLETON,1984 Genetic revolutions in progeny. J. Hered. 25: 273-274. relation to speciation phenomena: The founding of new popu- WI~BRODT,J., D. ADAM, B. MALITSCHEK,W. ~UELER, F. RAULF et al., lations. Annu. Rev. Ecol. Syst. 15 97-131. 1989 Novel putative receptor tyrosinekinase encoded by the CHARLESWORTH,B., 1970 Selection in populations with overlapping melanoma-inducing Tu locus in Xiphophorus. Nature 341: generations. I. The use of Malthusian parameters in population 415-421. genetics. Theor. Popul. Biol. 1: 352-370. WRIGHT,S., 1952 The genetics of quantitative variability, pp. 5-41 in CHARLESWORTH,B., and S. ROUHANI,1988 The probability of peak Quantitative Inheritance,edited by K. MATHER. Her Majesty’s Sa- shifts in a founder population. 11. An additive polygenic trait. tionary Office, London. Evolution 42: 1192-1 145. Communicating editor: M. SLATKIN CHRISTIE,P., and M. R. MACNAIR,1984 Complementary lethal factors in two North American populations of &e yellow monkey flower. J. Hered. 75: 510-511. APPENDIX CO~E,J. A. 1992 Genetics and speciation. Nature 355 511-515. COYNE,J. A., and H. A. Om, 1989 Patterns of speciation in Drosoph- In thesequel, we use an approach based on Ljapunov ila. Evolution 43 362-381. functions (cJ HOFBAUERand SIGMUND1988) in order to 544 A. Wagner, G. P.and Wagner P. Similion

qualitatively determine theglobal dynamical behavior of p2 = p, represents an invariant manifold that divides the (5) in the epistatic case (p = -34) , using the fitness simplex into two disjoint domains of attraction. We focus matrix M,. The specific Ljapunov functions used our attention now on orbits starting in the interiorof p, throughout have been first introduced by B~JRGER > p, and show that every orbit converges to p, = 1. Sym- (1983a-c) and used by BURGER(1989) to carry out the metric arguments hold for p, < p,. Orbits starting at a global analysis in the linear case. point ', where p; > p: holds, cannot reach the domain Using the matrix of genotypic values M, shown in Fig- of S where p2 < p, and vice versa. ure Id, we rescale, purely for technical convenience, the Let us assume that there exists a stationary equilib selection parameter s such that S' := 9s/16. The quali- rium ?, E intS forsome initial condition ?, ' E intS. First tative results presented below do not depend on the we consider the function 2 = p2p3/p1p4,observing that magnitude of s. The rescaling leads to a convenient al- D = plp4(1 - 2).Given 2, it is easy to show that gebraic form for the marginal mean fitnesses mi of gamete p,, in which the terms mZ3,m3,, m14and m,, are equal to 1 - sf. The marginal mean fitness values be- come

89 33 4z'=1-s'(16Pl+p3+zp4)81 1

and

T(P) = Pl P2P3 + Pl P2P4 + Pl P3P4 + P2P3P4 > 0. By the above relations we conclude that,starting in 2 5 1, i. e., D > 0, Z > 0 holds and for orbits starting anywhere in intS 2 > 1 will eventually hold, so that D < 0 holds if and the mean fitness of the population the trajectory remains in intS. Next we show that if D < 0, then d/dt&/p, - 1)2> 0 holds. Indeed, if D < 0 81 1 1 d2) = 1 - sf7 p: + $4 + (; p, + ,p,)(p, + p,) "("dt P3 - 1)2 = 2k- 1) !p) 1 = 2 - 1) [la," ,""I (A4) In thefirst place, we observe by inspection of (Al), that no interior equilibrium point $ can occur if r = 0 and = 2(: - 1)2 (dp2 - >o p = 0, since there is no 6E intS, such that the necessary ;) and sufficient condition m(,2)(5) = mi')($), Vk, 1 E (1, 2, 3, 4} for the existence of an interior equilibrium holds. Thus, provided there is negative linkage disequi- point is met. An analogous argument applies to the librium, the distance of each orbit to the plane 9, = p, boundary faces of S. increases. But this is in contradiction to our assumption The flow along the edges of S behaves as follows: of the existence of an internal equilibrium point, since Along the edges p, + p2 = 1 and p, + p, = 1 it points at any such point 8, D < 0 must hold, which in turn towards the vertex p2 = 1, along p1 + p3 = 1 and p, + implies that d/dt(p2/p3 - 1)'1;> 0. Therefore jcan not p, = 1 it points towards p, = 1 and along p, + p4 = 1 be anequilibrium point andany orbit starting in p, > p, it points towards p, = 1. An unstable equilibrium exists approaches one of the vertex equilibria. Inspection of at p3 + p2 = 1 with p2 = p3 = %. At this edge the flow the Jacobian matrix A = (akl) = dP;,/dp, of (5) shows points towards p3 = 1 for p3 > % and into the opposite that its eigenvalues evaluated at the vertex equilibrium direction for p, < %. This equilibrium vanishes as soon p, = 1 are as r > 0. Furthermore, one can exclude the occurrence 3's of a symmetric interior equilibrium point even if r # 0, A r< 0 4 i. e., an equilibrium where 6, p4 and p2 = p, holds. This is due to the fact that for all fi€ intS, m\,) # miz) holds, 3's A2=A3=->0 so that there is no ?, E intS with e, = p,(m, - KG) - rD 16 = p,( m4 - m) - rD = p4, this also representing a fun- damental difference to the linear case (p = 0) analyzed in the main text. Since mi2) = mi2)iffp, = p,, the plane Epistasis and Speciation 545

A, is irrelevant here, since it corresponds to trajectories p, = 1 or p, = 1. Within this plane, however, there exists outside the simplex S. The signs of A,, A, and A, imply a further equilibrium point, which can be explicitly cal- that thereexists onlya onedimensional stable manifold culated for r = 0 and is given by b1 = 0, = p, = 3/20, for the vertex equilibrium p, = 1. It is therefore not and j4= 7/10. For r > 0 no analytical solution for the locally stable. That p, = 1 can also not be a locally stable equilibrium point within p, = p, was obtained. However, equilibrium pointcan be seen by analyzing the behavior the existence of an internal fixed point for r > 0 follows of (5a) with m, specified by (Ala) and m specified by from a stability analysis of p, = 1. For examining the (A2)in a sufficiently small neighborhood of p, = 1. In stability ofp, = 1 within the planep, = p, the differential such a neighborhood pf = 0, i E {2,3,4}and by using equation describing the dynamics of p, is substituted by p, = 1 - p, - p, - p, one observes that p: - 1 -2p, - p := p, = p, and in an approach similar to (A6) one 2p3 - 2p, holds. Thus, (5a) becomes arrives at

p4-pl(y-r) -pT.3s’ which shows that in this neighborhood p, is decreas- ing. Therefore we canconclude that starting in If r > 3s‘/4, p, will be negative and p, will therefore be {?, E SI p20 > p30) trajectories will converge to p, = 1. unstable. Since there is no other equilibrium point on Completely analogous arguments show that for initial the boundary of the plane, there has to be an equilib conditions in {?, E SI $20 < p30) all orbits will converge rium point in the interioroft, = p,. No analytical results top, = 1. regarding stability of thisequilibrium point within p, = The case of p, = p, is of lessimportance for the equi- p, have been obtained, although we have calculated its librium behavior of the system, since it is only an un- approximate coordinates numerically for specific values stable set of measure zero within the simplex and small of r and s, and numerical integration of the system (5) stochastic perturbations would immediately lead the dy- within this manifold indicated its stability (results not namics away from it andto one of the equilibrium points shown).