J. Genet., Vol. 75, Number 1, April I996, pp. 33-48. (() Indian Academy of Sciences Effects of the shape of distribution of mutant effect in nearly neutral mutation models HIDENORI TACHIDA Department of Biology, Faculty of Science, Kyushu University 33, Fukuoka 812, Japan Abstract. Models of the theory of nearly neutral mutation incorporate a continuous distribu- tion of mutation effects in contrast to the theory of purely neutral mutation which allows no mutations with intermediate effects. Previous studies of one such model, namely the house-of- cards mutation model, assumed normal distribution of mutation effect. Here I study the house-of-cards mutation model in random-mating finite populations using the weak-mutation approximation, paying attention to the effects of the distribution of mutant effects. The average selection coefficient, substitution rate and average heterozygosity in the equilibrium and transient states were studied mainly by computer simulation. The main findings are: (i) Very rapid decrease of the substitution rate and very slow approach to equilibrium as selection becomes stronger are characteristics of assuming normal distribution of mutant effect. If the right tail of the mutation distribution decays more rapidly than that of the normal distribution, the decrease of substitution rate becomes slower and equilibrium is achieved more quickly. (ii) The dispersion index becomes smaller or larger than 1 depending on the time and the intensity of selection. (iii) Let N be the population size. When selection is strong the ratio of 4N times the substitution rate to the average heterozygosity, which is expected to be 1 under neutrality, is larger than 1 in earlier generations but becomes less than 1 in later generations. These findings show the importance of the distribution of mutant effect and time in determina- tion of the behaviour of various statistics frequently used in the study of molecular evolution. Keywords. Protein evolution; substitution rate; average heterozygosity; population genetics; dispersion index. 1. Introduction Understanding the mechanism of molecular evolution is one of the most important problems in biology. Kimura (1968) proposed the neutral theory, which states that the main cause of evolutionary change at the molecular level is random fixation of selectively neutral or very nearly neutral mutations rather than Darwinian selection. In its pure form (Kimura 1987 and later), the neutral theory assumes that there are two discrete classes of mutations depending on their selection coefficient, s. One class is neutral mutations whose selection coefficients satisfy N~ rsl << 1, where N~ is the effective population size. The other class is deleterious mutations whose selection coefficients satisfy -Nes >> 1, and these do not contribute to molecular evolution. The theory allows no mutations with intermediate effects. The proposal aroused a lot of contro- versy (see Lewontin 1974 and Kimura 1983 for review) but now the theory is widely accepted as an explanation for most of DNA changes in noncoding regions and synonymous sites. However, the mechanism for amino acid substitutions (replacement substitutions) is still controversial (Gillespie 1991). Ohta (1972) proposed the nearly neutral mutation theory to explain replacement substitutions. In this theory mutations with intermediate effects, i.e. Nes ~- O(1), are permitted and the distribution of mutation effect is considered somewhat continuous 33 34 Hidenori Tachida (see Ohta 1992a for review). The first model that explicitly incorporated a continuous distribution of mutation effects was that by Ohta (1977). In this model an exponential distribution was employed as a distribution of mutation effects. More precisely, the difference between the fitnesses of the mutated and original alMes was assumed to be distributed exponentially. Such models with fixed distributions for differences between the mutated and the original alleles are in a class called the shift model (Ohta and Tachida 1990). In the shift model the existence of advantageous mutations was corlsidered unlikely because, if they exist, the substitution rate becomes extremely large in large populations, which is not observed in molecular data (Kimura 1979). Ohta and Tachida (1990) introduced another type of nearly neutral mutation model, called the fixed model, in which the fitness of the mutant allele, not the difference between the fitnesses of the mutated and the original alleles as in the shift model, has a fixed distribution. This mutation model is the same as the house-of-cards model of Kingman (1978), and we call the model by this name from now on. The most distinguishing feature of this model from the shift model is that the distribution of the fitness difference between the mutated and original alMes changes as the population evolves. Thus, when the average fitness of the population is low, most mutations are advantageous; when the average fitness of the population is high most mutations become deleterious. Even though advantageous mutations are allowed in this model, the substitution rate does not become large in large populations because such populations have high average fitnesses and most mutations are deleterious. This is in accord with the traditional view of mutation, namely that mutations are mostly deleterious after evolution. Ohta and Tachida (1990) and Ohta (1992b) considered the behaviour of this mutation model in geographically structured populations while Tachida (1991) considered a simpler case of random-mating populations. Tachida's (1991) conclusions are, in summary: (i) The behaviour is mainly determined by the product c~ = 4Neo', where o" is the standard deviation of the mutant distribution. (ii) If 0~ is less than 0.2, the behaviour is almost the same as that of the neutral model. (iii) If c~ is between 0.2 and 3-5, both selection and random genetic drift contribute to the gene evolution. Although the long-term substitution rate is reduced as c~ increases, it is difficult to distinguish its behaviour from that of the neutral model using conventional neutrality tests such as that using the average and the variance of heterozygosity. (iv) If ~ is larger than 5, selection dominates the process. The average fitness quickly goes up and after that very few substitutions occur. However, it takes a long time to reach equilibrium. In all these studies, the distribution of mutant fitness is assumed to be normal. However, there is no biological reason for assuming normality of mutant fitness distribution. Thus it is necessary to investigate the effects of shape of distribution on the evolution. In the present paper I investigate the house-of-cards model in finite panmictic populations employing various mutant distributions. Specifically, I investigate the effects of the distribution shape on average fitness, substitution rate, heterozygosity and dispersion index. To save computer time, i use a jump-process approximation that can be used when the mutation rate is small. I first describe the model and explain this approximation, and then summarize results obtained by this approximation. House-of-cards mutation model 35 Appropriateness of the approximation was checked by comparison with the result obtained by the simulation study of Tachida (1991) on the Wright-Fisher model. 2. Analysis We assume the diploid Wright-Fisher model with fixed population size N (see Crow and Kimura 1970). Consider a locus with an infinite number of alleles (the infinite-allele model of Kimura and Crow 1964). Mutation always results in a new allele, labelled by a random variable S. We call S the selection coefficient of the allele. The fitness of an individual that has two alleles labelled by S x and S 2 is 1 + S 1 + S 2. For different alleles, S is independently distributed with the same density, fo(s). We designate the variance of it by o-z. This mutation model is the house-of-cards model of Kingman (1978). In a finite population the frequency changes become a multidimensional Markov process and it is difficult to deal directly with this process (but see Ethier and Kurtz 1994). So we use an approximation that is valid for small mutation rates. 2.1 Weak-mutation approximation Let u be the mutation rate per generation. IfNu is much smaller than 1 the population is expected to be mostly monomorphic, and occasionally fixations of new alleles occur very quickly. This can be approximated by a one-dimensional Markovian jump process (Zeng et al. 1989; Tachida 1991). We briefly explain this jump process. In this process the population is characterized by the selection coeff• of the allele currently fixed. Let Po (s; t) be the density of the population currently fixed by an allele with the selection coefficient s at time t. A transition from a state s to another state with a selection coefficient between r and r + dr in a very short time At occurs with a probability 2(r - s) 2NuAt ~e (r) dr, (1) 1 - exp [- 4N(r - s)] a~ because 2Nufo (r)drAt mutations within this range occur and the probability of fixation for such mutations is 2(r-s)/(1-exp[-4N(r--s)~) (Kimura 1962). Thus, if we measure time in units of 1/u generations, the transition equation for Po(S; t) is written as (Tachida 1991) dPo(S,t) = 2N { I ~ 2(s-r) d----7-- ~ 1 - exp [- 4N(s -r)j f~176 t)dr 2(r - s) t) dr }. -fT~l_exp[_4N(r_s)]fo(r)p(s , _ (2) If we scale s and r by the standard deviation o" offo (s), the equation can be written as dp(s, t) { f ~ (s-r) d~ - ~ ~o 1 - exp [- ~(s -r)] f(s)p(r' t)dr - f7 oo 1 - exp[--~(r(r-s) _s)]f(r)p(s,t)dr } ' (3) 36 Hidenori Tachida where a = 4No- and p(s, t) andf(s) are densities of the population and mutation effect respectively for scaled variables. Thus the process is characterized by a and the shape of the standardized mutation density f(s).