Copyright 0 1989 by the Genetics Society of America Effects of Mutation on Selection Limits in Finite Populations With Multiple Alleles Zhao-Bang Zeng, Hidenori Tachida andC. Clark Cockerham Department of Statistics, North Carolina State University, Raleigh, North Carolina 27695-8203 Manuscript received March 10, 1989 Accepted for publication May, 3, 1989 ABSTRACT The ultimate response to directional selection (ie.,the selection limit) under recurrent mutation is analyzed by a diffusion approximation for a population in which there are k possible allelesat a locus. The limit mainly depends on two scaled parameters S (= 4Nsa,) and 0 (= 4Nu) and k, the number of alleles, where N is the effective population size, u is the mutation rate, s is the selection coefficient, and a: is the variance of allelic effects. When the selection pressure is weak (S 5 0.5), the limit is given approximately by 2SaJ 1 - (1 + c‘)/k]/(e + 1) for additive effects of alleles, where c is the coefficient of variation of the mutation rates among alleles. For strong selection, other approximations are devised to analyze the limit in different parameter regions. The effect of mutation on selection limits largely relies on the potential of mutation to introduce new and better alleles into the population. This effect is, however, bounded under the present model. Unequal mutation rates among alleles tend to reduce the selection limit, and can have a substantial effect only for small numbers of alleles and weak selection. The selection limit decreases as the mutation rate increases. HE cause of responses to directional selection ROBERTSON(1 955, 1964) for estimating the amount T with recurrent mutation can be generally di- of new genetic variance.This mutationmodel assumes vided into two phases. In the first phase, the response that the genetic variance produced by mutation per to selection is rapid andmostly due togenetic variance generation is constant over time independent of the existing in the base population. As the genes in the background level ofpopulation mean andgenetic initial population approach fixation, gradually more variance. Many recent theoretical analyses used this response to selection comes from new genetic varia- mutationmodel, which includeKIMURA (1965), tion introduced by mutation and the process of re- LANDE(1975) and TURELLI(1 984) on the mainte- sponse is transformed into the second phase. Com- nance of genetic variance with mutation and stabiliz- pared with the first phase, the second phase of selec- ing selection; LANDE(1 976, 1977) and TURELLI,GIL- tionresponse, largely relying on new genetic LESPIE and LANDE(1 988) on the prediction andesti- variation,covers a much longer time scale with a mationof therate ofevolution of quantitative relatively small response per generation. But, sooner characters from genetic drift; CHAKRABORTYNEI and or later, the response will reach a plateau balancedby (1982) and LYNCH and HILL (1986) on the genetic mutation and directional selection because the poten- variation within and between populations; and HILL tial genetic variance accessible by mutation in a locus (1 982a,b) and HILL RASBASHand (1986) on long-term must be finite. In this paper, we formulate this equi- response to artificial selection. This model has the librium and analyze in particular thepopulation mean appeal of relating various evolutionary quantities to at this mutation-selectionlimit. The analysis is directly the parameter Vm, the rate of input of new genetic related to the evaluation of mutation effects on the variance by polygenic mutation, which is estimable long-term response of populations to artificial selec- from divergence andselection experiments under cer- tion. It has also some relevance to evolutionary prob- tain assumptions (LANDE 1975; HILL 1982b; LYNCH lems, such as selection for fitness in natural popula- 1988). But it also has some limitations. Since the tions. However, due to the nature of the problem, model does not produce a bound of genetic variance this kind of analysis depends very much on the muta- in an infinite population in the absence of selection,it tion model. is subject to the restrictions that the genetic variance Two extremepolygenic mutation models have been in a population is not near saturation (LANDE 1975) used in recent theoretical analyses. One is the constant and the populationmean is not very far from the variance model, originally proposed by CLAYTONand original value, because in reality there must be a limit to the range of allelic effects and hence the genetic The publication costs of this article were partly defrayed by the payment of page charges. This article must therefore be hereby marked“advertisement” variance at a locus. The observations that thebetween- in accordance with 18 U.S.C. $1734 solelyto indicate this fact. population genetic variance (CHAKRABORTYand NEI Genetics 122: 977-984 (August, 1989) 978 Z.-B. C.Zeng, C. H. Tachidaand Cockerham 1982; LYNCHand HILL 1986) andthe cumulative ing discussion we analyze only the response from one selection response frommutation (HILL 1982b) locus with the understanding that a summation over asymptotically increase linearly with timewithout all lociis implied provided thatwe are willing to ignore bound by using this mutation model reflectthese the effects of linkage disequilibrium. While linkage limitations. disequilibrium builds up in transient states with direc- The other model is the k-altele model in which tional selection, the influence may be minor at equi- mutation is assumed to occur amongK alleles. In many librium. papers k is restricted to two or three or five and in others k is set to be arbitrary, even to be infinite. MODEL AND SELECTIONLIMITS These papers includeLATTER (1 960), BULMER(1 972), TURELLI(1 984), BARTON(1 986) and SLATKIN(1 987) Consider the conventional diffusion model of mul- on themaintenance of genetic variance with mutation tiple-allele selection with mutation in a finite popula- and stabilizing selection; COCKERHAM(1 984) and tion of effective size N. Assume thatthere are k COCKERHAMand TACHIDA(1 987) on the genetic var- possible alleles at a locus available for mutation in a iation within and between populations; KIMURA population. Let xi and ai be the frequency and phe- (198 1) on the molecular evolution under stabilizing notypic effect respectively of the ith allele, Ai, and let selection; and LI (1977) and WATTERSON(1977) on w = m(xl, . , xk-1) denote the mean fitness of the the heterozygosity under mutation and selection. In population, assumed to be approximatelyconstant, contrast with the constant variance model, the k-allele but varying by amounts of order O(l/N), due to the model behaves properly in differenttime horizons effect of small selective forces, as allele frequencies (COCKERHAM and TACHiDA 1987) and depends much vary. Foradditive effects of genes and directional more on the parameter 4Nu than V,, where N is the selection, iV = I + 2slxl + . + 2skXk where st is the effective population size and u is the mutation rate at selection coefficient of the ith allele. (For example, in a locus. In some problems, the results do not critically the case of phenotypic truncation selection si i(ut - depend on which mutation model is used (e.g., TUR- ii)/up where i is the intensity of selection, up is the phenotypic standard deviation of the character, and ELLI 1984). But in some other problems (the problems ii involving long-term response to drift and selection), is the mean effect of the alleles.) Suppose that in each generation A, mutates to A, the results differ markedly for different models. By with probability # j). Then, subject to the usual using the k-allele model, COCKERHAMand TACHIDA uji(i multinomial-type sampling variation from one gener- (1987) observed that without selection the between- ation to the next, the change, 6x,, in the ith allele’s population genetic variance does not increase indefi- relative frequency in one generation has a mean nitely, but to a finite equilibrium value, which is in sharp contrastwith the analyses of CHAKRABORTYand Mi 2 xpji - xi 2 uq NEI (1982) and LYNCH and HILL(1986). Despite this j .i difference, however, the two models give comparable results as to the predictions of the short-term change of genetic variances within and between very small populations starting from afixed population. a variance V,i = x,( 1 - xi)/2N, and 6x;, 6xj have a It is not possible to distinguish between the two covariance Vq = - xixj/2N. If we denote polygenic mutation models from experimental results, @(x*,. , xk-1; t) as the joint probability density of because the two models do not differ markedly on the the first k - 1 allele frequencies at generation t, the short-term accumulation of genetic variance from mu- Kolmogorov forward equation is tation and it is also very difficult to experimentally k-I k-1 k- 1 eliminate the confounding effects from selection and d9/dt = (G) 2 2 d2( Vij@)/dXIdX, - 2 d(Mi@)/dX,. drift. Most mutation accumulation and selection ex- :=I ;=I i= 1 periments are carried out on unselected base popula- Unfortunately no equilibrium solution seems to have tions for a relatively short time, yet the results are been obtained underthis general condition. However, usually ambiguous as to thepattern of effects of for the special case where ujz = ui for all j # i, the polygenic mutation (see CLAYTONand ROBERTSON above equation has a stationary solution as was first 1964; HOLLINGDALEand BARKER 197 1 ; MUKAI1979). shown by WRIGHT(1949) In this paper, we determine thepopulation mean at the equilibrium between mutation and directional se- lection by using the k-allele model. This equilibrium mean is the limit to selection under recurrent muta- tion. (Note that no limit is expected under the con- where ti = 4Nui, xk = 1 - x1 - . - &-I, and c is a stant variance model, see HILL 1982b).