Selection, Generalized Transmission and the Evolution of Modifier Genes

Selection, Generalized Transmission and the Evolution of Modifier Genes. I. The Reduction Principle

Lee Altenberg* and Marcus W. Feldmant

*Department of Statistics, North Carolina State University, Raleigh, North Carolina 27695-8203 and ?Department of Biological Sciences, Stanford University, Stanford, CaEi,fornia 94305-5020 Manuscript received March 19, 1987 Revised copy acceptedJuly 16, 1987

ABSTRACT Modifier gene models are used to explore the evolution of features of organisms, such as the genetic system, that are not directly involved in the determination of fitness. Recent work has shown that a general “reduction principle”holds in models of selectively neutral modifiers of recombination, mutation, and migration. Here we present a framework for models of modifier genes that shows these reduction results to be part of a more general theory, for which recombination and mutation are special cases.-The deterministic forces that affect the genetic composition of a population can be partitioned into two categories: selection and transmission. Selection includes differential viabilities, fertilities, and mating success. Imperfect transmission occurs as a result of such phenomena as recombination, mutation and migration, meiosis, gene conversion, and meiotic drive. Selectively neutral modifier genes affect transmission, and a neutral modifier gene can evolve only by generating association with selected genes whose transmission it affects.-We show that, in randomly mating populations at equilibrium,imperfect transmission of selected genes allows a variance in their marginal fitnesses to be maintained. This variance in the marginal fitnesses of selected genes is what drives the evolution of neutral modifier genes. Populations with a variance in marginal fitnesses at equilibrium are always subject to invasion by modifier genes that bring about perfect transmission of the selected genes. It is also found, within certain constraints, that for modifier genes producing what we call “linear variation” in the transmission processes, a new modifier allele can invade a population at equilibrium if it reduces the level of imperfect transmission acting on the selected genes, and will be expelled if it increases the level of imperfect transmission. Moreover, the strength of the induced selection on the modifier gene is shown to range up to the order of the departure of the genetic system from perfect transmission.

ODELS of modifier genes have been developed (ESHELand FELDMAN1982). The possibility that these M over the last 20 yr to extend the synthetic parallel results reflect an underlying unity among theory of evolution to features of organisms which fall modifier gene models is the motivation for the current outside of the classical purview of heritable variation work. for fitness. The biological phenomena encompassed We introduce here a theoretical approach to the in modifier gene models are diverse, and include evolution of modifier genes that partitions the forces features of the genetic system such as recombination of evolution, other than drift, into two classes, selec- and mutation, features of the mating system such as tion and transmission. Transmission is the determi- selfing, and ecological features such as migration nation of offspring types by parental types. Imperfect rates. transmission occurs whenever the gametes in off- A principle for the evolution of neutral modifier spring are different from those in their parents. Selec- genes, the “reduction principle,” has been developed tion occurs when different types in the population by FELDMAN,CHRISTIANSEN and BROOKS(1980), LI- differ in their relative contributions to the next gen- BERMAN and FELDMAN(1 986a,b; 1987) and FELDMAN eration. A large class of modifier gene models can be and LIBERMAN(1 986). In models of recombination, unified once they are seen to involve genetic variation mutation and migration modification in which the for transmission processes. In contrast to genes that modifying gene is multiallelic, LIBERMANand FELD- produce differences in fitness, neutral modifier genes MAN have shown that there is a reduction principle produce differences in transmission processes. Modi- that holds under a wide variety of conditions, and fiers of recombination, mutation, segregation distor- they suggest that complete reduction of each of these tion, and migration are in this class. has the property of “evolutionary genetic stability” We investigate the initial increase properties of modifier alleles introduced into randomly mating dip- Dedicated to Luca Cavalli-Sforza on his 65th birthday. loid populations near genetic equilibrium for a set of

Genetics 117: 559-572 (November, 1987) 560 L. Altenberg and M. W. Feldman polymorphic loci undergoing viability selection and Varieties of transmission processes: Models of one generalized transmission. The main results can be locus without mutation or segregation distortion are summarized as follows. First, selection for or against examples of perfect transmission. The transmission the modifier requires that there be a variance in the probabilities with perfect transmission are equilibrium marginal fitnesses of the selected loci. This in turn requires that the selected loci be imper- Tid (i tj,k) = Y2(6ij + a&), fectly transmitted. We obtain two results that gener- where 6, is the Kronecker delta (6) function, alize the reduction principle. First, we show that a population that maintains an equilibrium variance in the marginal fitnesses, due to imperfect transmission of loci under selection, can always be invaded by a Different genetic systems may produce different new modifier allele that produces perfect transmis- levels of imperfect transmission during reproduction. sion. Second, we show that when the modifier and A quantity that plays a central role in our analysis is major loci are tightly linked, a new modifier allele the level of perfect transmission that is guaranteed by that produces “linear” variation in the transmission the genetic system for all possible parental genotypes. process can invade if and only if it reduces the level This upper bound (over all genotypes) on the proba- of imperfect transmission. We also obtain results on bility of imperfect transmission we refer to as a. Any the possible strength of selection induced on a tightly transmission matrix can be written in the form, linked neutral modifier, and show that it is on the order of the variation in equilibrium marginal fitnesses, which is bounded by the extent of imperfect which involves a mixture of the perfect transmission transmission characteristic of the genetic system. matrix, Tid and a transmission matrix, P. The value a is the minimum value for which the elements of the GENERALIZED TRANSMISSION matrix The term haplotype will be used to mean a gamete’s T - (1 - c~)T;,jL 0 genotype, or a gamete’s contribution to a diploid (4) genotype. Perfect transmission occurs when a diploid are non-negative, so that 1 - a is a measure of the individual’s two haplotypes, as originally derived from degree of perfect transmission present in T. its parents, are transmitted unchanged to its gametes, Tables 1, 2 and 3 show the transition probabilities and in equal proportions. For a multilocus parental for several standard models of mutation, segregation haplotype to be perfectly transmitted, the whole hap- distortion, and recombination, and an example of a lotype must behave as a Mendelian unit. Imperfect model of gene conversion at two loci. The value of a transmission occurs when gamete haplotypes differ for each of these models is given. from the parental haplotypes as a consequence of mutation, recombination, gene conversion, translo- cation, etc. EVOLUTION OF HAPLOTYPES UNDER SELECTION AND GENERALIZED TRANSMISSION To represent the transmission relations between parental and gamete haplotypes, we first enumerate Here we formulate a general model of evolution all of the possible haplotypes, {i,j, k, . ). The trans- for a diploid population undergoing random mating, mission can be represented by the set of probabilities viability selection, and transmission of haplotypes T(i cj,k) that a gamete produced by a parent with from zygote to gamete. Let z be the vector represent- haplotypes j and k has haplotype i; ing the frequencies of all gametic haplotypes in the population. Let T(i c j, k) 2: 0, 1 T(i c j, k) = 1 i (1) w = llwyll&, for all j, k. where w,, is the viability of a genotype composed of For simplicity we restrict our analysis here to haplotypes i and j. We assume sex symmetry so that autosomal systems so that there is symmetry in T: WI,12 = W,p,. T(i + j, k) = T(i k, j). If there are n possible With a life cycle consisting of random union of haplotypes, then the genetic transmission system can gametes, viability selection, and generalized transmis- be represented as an n by n2 matrix: sion, the recursion on the frequencies of gamete haplotypes is T = IIT(i +ji,j2) ll&,j2=i- (2) nn This matrix is just the “segregation table” for all of zik: = 1 ZJZJ(i cj,K)W$ 1=1 k=l the genotypes in the population. Matrices satisfying (54 ( 1) are called biparental transmission matrices. (i = 1, 2, --.,n) Evolution of Modifier Genes 56 1

TABLE 1 of haplotype a is w, = ZJ wyzy It represents the net Transmission table for mutation and segregation distortion effect of selection on a haplotype averaged over all the genotypes in which it appears. With perfect trans- Frequencies of gamete haplotypes mission, variation in marginal fitnesses is the sole force

Parental genotypes Ai acting to change haplotype frequencies, in which case (5) becomes

Gz: = L, w,,zJ, or, Gz’ = diag(Wz)z (7) J With ml= mp = k = 0 we have perfect transmission. (see EWENS1979, pp. 40-4 1). The relative frequency With k = 0, (Y = max(m,, [see (3)]. mz) of haplotype changes in one generation by an amount With ml = m2 = 0, (Y = I k 1 [see (3)]. i proportional to ZJ wyz, - i. TABLE 2 Employing the marginal fitnesses, the general recursion (5) can be written in the form Transmission table for recombination z’ = YDz, (8) Frequencies of gamete haplotypes where Y is a column stochastic matrix, and D is a Parental genotypes AIBl Ai& AnBl AS& positive diagonal matrix, as follows: AiBiIAIBI 1 0 0 0 A iBi/AiBz %J ‘/s 0 0 AiBiIAzBi YZ 0 % 0 AIBI/A~B~ %(1 -R) %R %R %(1 -R) AiBzIAiBz 0 I 0 0 AiBzIAnBi %R %(1 -R) %(I - R) %R and AiBz/AzBz 0 % 0 Y2 AzBIIAdi 0 0 1 0 AzBiIAzBz 0 0 L/n % AzBzIAzBz 0 0 0 1

Here (Y = R. or in matrix form Y is column stochastic since

1 wjk rkwjk Z’ = 1 111 LkwjhT(i t j, k) 11tj-1 2, (5b) zk - T(i t j, k) = -= 1. wk ik wj k Wj where EQUILIBRIUM PROPERTIES OF POPULATIONS G = w+zjzj = zTwz (6) UNDER SELECTION AND IMPERFECT ij TRANSMISSION is the mean fitness of the population. In this treatment we are concerned with the evolu- A central quantity involved in our analysis is the tion of modifier genes introduced into populations marginal jtness of a haplotype. The marginal fitness near equilibrium. Here we derive the most salient

TABLE 3 Transmission table for unbiased gene conversion at two loci

Frequencies of gamete haplotypes Parental genotypes

~~ ~ a = rate of gene conversion at locus A, b = rate of gene conversion at locus B. Here a = a + b - 2ab. 562 L. Altenberg and M. W. Feldman property of populations at equilibrium, which is the is partitioned so that one part, the set of selected loci, relation between imperfect transmission and variation A, undergoes selection and generalized transmission, in marginal fitnesses. while the other part, the modifier locus M, controls Consider a population that is at an equilibrium for the transmission of the first part. Thus haplotypes are the frequencies of the haplotypes. Recursion (8) pro- indexed with two subscripts, so that duces the equilibrium identity MaAz = ?Di. (10) i represents the haplotype with modifier allele a and With perfect transmission 2 = Di, so that multilocus haplotype i for the set of loci under selec- .. tion. Its frequency is zut. We assume that the modifier alleles have no effect on the fitness of their carriers. Thus, the fitness of genotype MaAJ/MJk will be WJk * and z& = ZZI for all haplotypes i such that zi> 0. This is for all a, b. We use the notation A, for haplotypes that true regardless of any frequency dependence of the consist of alleles Gl,G21G3t 1 . with the subscripts 1, 2, w’s. Hence, we have the well known 3, etc., indicating the selected gene locus. It will be Result 1: With perfect transmission, all the haplotypes assumed throughout that the location of modifier present in the population have equal marginal jtnesses at locus is external to the A haplotype. If GI is the equilibrium. selected locus nearest the modifier locus, we denote by r the probability of recombination between M and In the presence of imperfect transmission, there can GI. be differences at equilibrium in the marginal fitnesses We assume that the modifier alleles are perfectly of the haplotypes present. Moreover, the potential transmitted regardless of the processes acting on the magnitude of the variation in the equilibrium mar- selected haplotypes, so that the only force acting on ginal fitnesses can be shown to depend on the maxi- the modifier locus is the induced selection resulting mum level of imperfect transmission, a, characteristic from its effect on the transmission of the other loci. of the genetic system. On substituting the expressions To take account of recombination between M and GI (3) into the equilibrium equations (lo), since P 2 0 we introduce two new transmission matrices, T* and we easily deduce that z&/k 5 (1 - a)-’ for all i. ?*, which describe the transmission of the complete Later we show that the strength of selection on the system of modifier and selected genes and also keep modifier gene depends quantitatively on the magni- track of each modifier allele during transmission. tude of the difference between the maximal marginal T*(ait aj I bk) is the probability that haplotype M,A, fitness in the population and the mean fitness. This is produced from haplotype M A in genotype M,AJ/ quantity is a variant on the genetic load defined by MJk, given that no recombinationa. J has occurred be- CROW(1 958), and we call it the “selection potential”: tween M and GI. F*(ai + ak I bj) is the probability that Definition 1: The selection potential, V, present in a haplotype M,A, is produced from haplotype MaAk in population is defined as genotype M,A,/Mt.& given that recombination has max(wJ occurred between M and GI. Conditioning in this i L V=- - - 1 =- (1 1) way on recombination between M and GI takes ac- W 1 -L’ count of any possible interference between crossovers in the region between M GI where L is the genetic load, 1 - (z3/maxi(wi)), as and and the transmission defined by CROW(1958). of the A haplotype. In the absence of interference, T*(ai t uj bk) = f*(ai t bk) for all a, b, i, j, K. The upper bound to the equilibrium selection po- I aj I The recursion (8) for the general modifier model tential is with random mating can then be rewritten as a V5- 1 - a’ (12) W cjk which increases without limit as a increases toward 1, (13) and approaches 0 as a tends to 0. This definition of + rF*(bi t bk 1 cj)~. selection potential corresponds to the less common definition of genetic load used by EWENS(1979, p. In terms of T* and ?* the original T defined in (1) 66). can be written 1 GENERAL MODIFIER GENE MODEL T(ai t uj, bk) = - (1 - r)T*(ai t aj 1 bk) 2 The general modifier gene model is a special case of formulation (5) with haplotypes undergoing both +-r selection and imperfect transmission. The haplotype 2 Evolution of Modifier Genes 563

T(ai t uj, ah) ing notation: 1 = - (1 - r)[T*(aic uj I ak) + T*(ai t ak I aj)] 2 r + - [F*(ai t aklaj) + T*(ai + ajlak)], 2 and T(ai t bj, ck) = 0 if b # a and c # a, where Z, T(ai t ajlbk) = 1 for all a, b, j, K, and T(ci t ajl bk) = Qci e- aj I bk) = o if c z a. The original T‘s are not used after this point, so for convenience the asterisks from the new sets of T’s are suppressed. The new T’s can be represented in matrix and form as D = diage), Tab = IIT(ai ajl I bjZ)ll!~~~2=1 and as in (9). Qb are column stochastic matrices. At any equilibrium for (13), the following identity +ab = IIF(ai aj1 I bj2)11:J]J2=1* must be satisfied: In the absence of interference Tab = ‘kab.It will be assumed that there are no position effects between M -1 fbi = ibji&wjk[(1 - r)T(bi t bj I ck) and A. That is, the linkage phase between M and A w cjk (16) within the diploid genotype does not affect the trans- + @(bi t bklcj)]. mission of A. This is expressed as: This can be written in a vector form analogous to (8): T(ai t aj I bk) = T(bi t bj I ah) and ib = QbDib. (17) qui t aj 1 bk) = qbi c bj I ak), * * The “external stability” of equilibria: To investi- or in matrix form, Tab = Tbaand Tab = Tbo.This gate the long-term evolutionary fate of the modifier excludes the possibility that the modifier controls locus, we need to know those characteristics of new transmission processes that occur during the haploid modifier alleles that determine whether they will in- phase of the life cycle. crease in the population when introduced near an Segregation distortion acting on the modifier locus equilibrium. Let a new modifier allele, Ma, be intro- is excluded here because this would constitute imper- duced into the population near i, with fa, the fre- fect transmission of the modifier. This entails Z, quency of haplotype MaAi. From (13) the vector form ~(aicaj~bk) = 1, Z, Quitajlbk) = 1 for all a, b,j, of the linearized recursion for the cui’s,ignoring terms This restriction is actually necessary to ensure that k. of order t:i, is T and ?’ satisfy condition (1) in the original definition .. of transmission matrices. e’ = Q,De. (18) If the selected haplotypes are perfectly transmitted Assuming that the polymorphic equilibrium i is locally in an organism of modifier genotype Ma/Mb,then stable to perturbations in the frequencies of the hap- T&i t aj I bk) = F,&i t aj I bk) = 6, lotypes already present, the initial increase of the new (14) modifier allele is determined by the spectral radius of 0 if i#j the stability matrix, =i1 if i=j7 p(Q,@. (1 9) Only if r = 0, however, will this constitute perfect If p(QaD)> 1, the new modifying {llele increases. If transmission of the entire haplotype of both modifier p(QaD) < 1, it decreases and if p(QaD) = 1, the modi- and selected loci. fying allele cannot change at a geometric rate. We call these the “external stability” properties of the equilibrium. ANALYSIS OF THE RECURSIONS

REDUCTION PRINCIPLE In this section we investigate the fate of new modifier alleles introduced into the population in the Since Qa a column stochastic matrix’, the sp_ectral neighborhood of a stable equilibrium. radius p(QaD)can be different from 1 only if D # I; The equilibrium identity: We introduce the follow- i.e., only if there is a nonzero variance in the marginal 564 L. Altenberg and M. W. Feldman fitnesses of the selected haplotypes at equilibrium. (ii) In principle, because the frequency vector v and Thus, when a population is near an equilibrium, se- the fitness matrix W are the only facts about the lection cannot act on a newly introduced modifier polymorphic equilibria that are relevant to the fate of allele to change its frequency at a geometric rate the new modifier allele, Result 2 can be shown to hold unless there is an equilibrium selection potential. no matter what linkage arrangements obtain for those In this section we present the main results about modifier alleles and selected loci segregating at the the initial increase properties of a new modifier allele polymorphism. Moreover, the modifier alleles present introduced to the randomly mating population at a at the polymorphism may even control their own polymorphic equilibrium. These results concern exter- linkage to the selected loci without changing this nal stability of the equilibria with respect to changes result. The new modifier allele, however, must fit the in the direction of the modifier frequencies. Although assumptions given earlier (that it be external to the the assumption of the existence of the equilibria may selected haplotype and have a single recombination entail constraints on the selection regimes and trans- rate r with the nearest selected locus) for the proof of mission probabilities, in none of the results do these Result 2 to hold. constraints or closed form solutions of the equilibrium frequencies appear. The only generic assumptions Case 2. Linear variation in the transmission made are as follows: (1) There is polymorphism in the So far we have considered only modifier alleles that selected haplotypes, and (2) new modifier alleles do produce large reductions in the imperfect transmis- not cause the selected haplotypes present at equilib- sion acting on a set of selected loci. Now we consider rium to be transformed into haplotypes not present. modifiers that can change the level of imperfect trans- Our results regarding the evolution of reduced mission over a continuous range of values. Here, the levels of imperfect transmission fall into two cases way changes in T reflect variation at the modifier according to the effect of the new modifier allele on locus is critical in the evolution of modification. transmission: The simplest kind of variation in transmission con- stitutes a basic “building block” of variation for more Case 1. Imperfect transmission eliminated at the complex systems. Suppose that each selected haplo- selected loci type, A,, has a certain probability, m,of being “hit” by Result 2: a) A population at equilibrium with a nonzero some transforming process, and given that it is hit, selection potential can always be invaded by a new modijer it is transformed into various other selected haplo- allele that eliminates imperfect transmission at the selected types with different probabilities. These probabilities, loci, for any amount of linkage of the modijier to these loci. Tl(i c j Ik), include possible dependence on both b) In particular, for a modijier locus that is absolutely haplotypes A, and Ak in the diploid genotype. If the linked to the selected loci (which are absolutely linked in effect of the modifier gene is to scale the “hit” rate, the presence of the new modajier allele), the spectral radius m, up or down equally for all haplotypes, it will be of the external local stability matrix equals one plus the said to produce “linear” variation in the transmission. equilibrium selection potential, i.e., Definition 2: For linear variation, the transmission matrix for modifier genotype MJMb can be represented as a weighted average,

T(mab) = (1 - mab)Ttd + mabT1, c) Moreover, for all r 5 1/2, the spectral radius of the external local stability matrix is always greater than one where TIis a transmission matrix that is independent by at least the equilibrium marginaljtness variance, i.e., of the allelic configuration at the M locus, and the modifier’s sole effect to determine the parameters mab. Examples are discussed after Result 3. We consider here the initial increase properties of d) For a given set of equilibriumjtnesses and haplotype a new modifier allele that produces linear variation in frequencies, the selective advantage of a new mod$er allele the transmission. Attention is restricted to the intro- that eliminates imperfect transmission decreases with duction of new modifier alleles into populations that looser linkage to the selected haplotypes. are fixed at the modifier locus, but all of the initial e) The selection potential is always greater than the increase results extend to a class of modifier poly- population variance in the marginal jtnesses. morphisms discussed in LIBERMANand FELDMAN (1986a,b; 1987). The proofs of this and subsequent results are given in Consider a population at a stable equilibrium satis- the APPENDIX. fying (16) where the modifier locus is fixed for one Remarks: (i) Result e) does not require that the allele, MI, which produces transmission matrices population be at equilibrium, but holds for populations in the transient phase of their dynamics as well. Evolution of Modifier Genes 565 and and

T1 = Ili;(li c lj21 1jl)llyj,j2=l. Denote the total frequency of each selected haplotype by Therefore,

vi = %hi. b

In this case ilk = 6k > 0 for all k. At equilibrium, from (16) the selected haplotype frequencies must satisfy where the identity

i = ItlD)f = [(l - r)*l + T$~]D;, (20) where From (18), the recursion on the frequencies of haplotypes containing the new modifier allele is

E’ = Q2De = [(l - r)?, + r$p]De. (26)

M2 will increase when rare if and only if

p(~2D)= p([(l - r)+2 + r921D)> 1. and D is the diagonal matrix of relative marginal fitnesses defined in (15). 4, and Y are both column Result 3: A new allele at a modijier locus that is stochastic. absolutely linked to a set of selected loci is introduced into a population near an equilibrium in which the modijier is fied, with a nonzero selection potential for the selected Reduction principle for linear variation in haplotypes. If the new modijier allele produces linear transmission variation in the transmission of the selected loci, then if Q2 is irreducible (reducible) A new modifier allele Mz,yielding transmission a) the new modijier allele will increase (not decrease) in matrices T2 and ?‘z as a heterozygote with M1 is introduced into the population. A general way to frequency at a geometric rate if it brings the transmission closer to perfect transmission, i.e.,mnI/mll< 1 and represent linear variation between TI and T2, and b) it will decrease (not increase) in frequency at a between Tl and T2 uses (14), and Definition 2, as follows: geometric rate if it takes the transmission further away from perfect transmission, i.e., m21/m11 > 1. T(2i t 2jl lk) = (1 - m21)c5ji Moreover, (234 + mnlT(1i t ljl lk) c) the asymptotic rate of change in the frequency of the is the relation between T1and T2,and modijier allele is an increasing function of the change it makes in the transmission probabilities. f(2i c 2kl lj) = (1 - mZ1)& (23b) Remarks: (i) The result is proven for r = 0 but will + m21f(li c 1kJlj) hold also for sufficiently small positive values of r (KARLINand MCGRECOR1972) when Y2 is irreduci- is the relation between ?I and ‘fz. Here mZ1 represents the parameter for MzIMl. ble. Remark: Recall from the characterization of linear (ii) The proof of this result employs Theorem 5.2 variation that ml1and m21 are interpreted as being the of KARLIN (1982) and is included in the APPENDIX. overall probability that the selected haplotypes are The variation in the transmission for which this Theo- “hit” by a transforming process. The above definitions rem applies must be linear and this is the condition in the analysis of mutation modification by LIBERMAN of T2 and f‘2 in terms of T1and ‘k, require that mll be scaled to 1. Thus mZ1is the “hit” rate for modi- and FELDMAN(1 986b) that yields reduction. In their fier genotype MJM1 relative to modifier genotype model, the equilibrium relation with the modifier MI/MI. We will refer to mzl as the value of the fixed on M1 is “modified parameter” for modifier genotype M2/Ml. Define 566 L. Altenberg and M. W. Feldman

and, setting r = 0, the recursion for the new modifier Strength of selection on modifiers allele M2 is Previous treatments of modifier gene evolution have generally not examined the magnitude of the selective forces that act on modifier genes, one excep- tion being the work on inversions by CHARLESWORTH and CHARLESWORTH(1973). WRIGHT(1964) stressed that pleiotropic effects of modifier genes could cause them to have intrinsic fitness differences. It has gen- Set positive constants mll,fil,f12, f21, f22 such that erally been felt that such selection on the modifier locus would overwhelm any selection due to its effects on transmission (e.g., KARLINand MCGREGOR1974). Here we show that the selection on modifier genes due to their effects on transmission can be quite wherefll +f21 =fi2 + f22 = 1; then linear variation strong. requires that p12and v12 satisfy From Result 2b, in the case of an absolutely linked modifier allele that eliminates all imperfect transmission, its induced marginal selective advantage is equal to the equilibrium selection potential. This advantage which entails vIl/pl 1 = v12/p12, i.e., there exists b such decreases with looser linkage of the modifier but is that uy = bp, for all i, j. This is the assumption under always greater than the population variance in the which Liberman and Feldman obtain the reduction equilibrium marginal fitnesses (Results 2c and 2d). result, namely N = bM. The definition of linear vari- For one special case of transmission, we can derive ation here includes the previous models of recombi- an estimate of the amount of selection on a new nation modification by NEI (1967), FELDMAN(1972), modifying allele that yields linear variation in the FELDMANand BALKAU(1 973) as well as the migration transmission. It will be a function of how far the new and mutation modification models of BALKAUand value of the modified parameter deviates from the FELDMAN(1 973), KARLINand MCGREGOR(1 974) and parameter value of the population at the original TEAGUE(1977). equilibrium. The transmission for which this estimate can be obtained is the “house of cards” mutation Linear variation embodies the fourth condition distribution defined by KINGMAN(1980). This is sim- given in LIBERMANand FELDMAN(1986a) for the ply a mutation process which has no memory; all reduction principle to hold for modifiers of recombi- haplotypes mutate at the same frequency, and the nation, mutation, and migration, namely, that besides probability that the mutant is a given haplotype does viability selection, the only evolutionary force in the not depend on what the original haplotype was. There system is that feature of the genetic system subject to are several familiar analogs to “house of cards” (i.e., genetic modification. This requirement is illustrated memoryless) distributions for models of migration, in the models of recombination modification in FELD- including the Wright island model, the Levene model, MAN, CHRISTIANSENand BROOKS(1980). In their and the Deakin model (see KARLIN,1982). Model I, which produces the reduction result, with With linear variation in the transmission, and no r = 0 the stability recursion on the new modifier allele interference between the mutation process and re- is combination between M and A, the general form for * the transmission matrix with a memoryless distribu- c‘ = (1 - r2)De tion is 31Wll + 33w13 31Wl2 33w23 I + T(ai t uj I bk) = (1 - mab)c$, + ma&. (27) where m,b is the overall mutation rate and p, is the I 0 0 probability of producing selected haplotype i given that there is a mutation, and T = T. The modifier, 0 0 therefore, may change the overall rate but not the 0 0 relative distribution of mutations. c 3iw13 + 33w3, 6IW14 + 33w34 Suppose that the population is at an equilibrium, 32w23 + 34w34 32w24 + 34w44 with a modified parameter of ml Substitution of (27) into (21) yields where r2 is the recombination rate produced by the new modifier heterozygote. This again fits the form ?I = (1 - mll)I + mllP, and = YIQ, (28) for linear variation, and Result 3 can be seen to apply. where P = diag(p)U, pT= (PI,p2, . . , pn), and U is Evolution of Modifier Genes 567 the n by n matrix of ones. The equilibrium value G (30) becomes solves the identity i2m:l(1 - mil)-' - pi)’. (31) i = [(l - m11)I + mllP] DG i (29) If no selection were acting then under mutation) = (1 - ml1)DC mllp + alone the equilibrium haplotype frequencies would all for any r, which emerges from (20) using the identity be QDG = DG. 4.= p. A new modifier allele, M2, is now introduced into 2 1. the population with mZ1 as the value of the modified Thus the squared term is the deviation of the selected parameter in the M2/M1heterozygotes. From (26) the haplotype frequencies from what they would be under linearized recursion on the frequencies of the haplo- pure mutation. The effect of adding selection to a types bearing M2 is system of pure mutation is to create selection for decreased mutation. t’ = +2[(1 - r)I + r~lDt Selection inversions where mZ1 has been substituted for m11 in (28) to on produce 9,. Since there is assumed to be no interfer- A new chromosomal inversion is formally the same as a new modifier allele that eliminates recombination ence, Gz = ?2Q. Then we have the following between the loci flanking the inversion, and is tightly Result 4: Consider the case of linear variation in linked to them. If we consider the two flanking loci in transmission with a memoryless distribution specijied by isolation, a rate f of recombination between them, (ZS), with p, > 0 for all i. For a tightly linked modafaer, prior to the introduction of the inversion, permits a the spectral radius of the stability matrix for a new modi- maximum selection potential of f/(1 - f)when there jjing allele with parameter m21as a heterozygote, with mZ1 is a two-locus polymorphic equilibrium [see (12)]. Re- close to ml > 0, is approximately: sult 2b can therefore be used to give an upper bound p(e2D) r 1 + (mil - m21) for the selective force on an inversion in terms of the * map length alone. The expression for the selective .(Dm:] P;’$GJ-’ P;~$(G, - $1‘. force given by CHARLESWORTHand CHARLESWORTH I : (1973) requires in addition an estimate of the linkage Remarks: The spectral radius is the actual value for disequilibrium. the asymptotic relative marginal fitness of the new Result 5: The relative amount of induced selective modifier allele, i.e., advantage on a chromosomal inversion can range up to

WM, -- - P(92D). W 1 -f’ Consistent with Result 3, a new modifier allele can where f is the map length of the inversion, in units of increase if and only if it reduces the amount of mu- crossover frequency. tation. The term Remark: In the examples of CHARLESWORTHand p;1qZii - i)2 (30) CHARLESWORTH(1973), the strength of selection on i the inversion is seen to approach f/(1 - f)as epistasis is of the same order as the marginal fitness variance and f become smaller. From the case in Table 1 of and is zero if and only if the variance of the marginal KARLINand CARMELLI(1975), where the initial in- fitnesses is zero. Therefore, for small modifications of crease of an appropriate new inversion violates the the transmission process the selection for or against “mean fitness principle” (KARLIN and MCGREGOR the new modifier allele will be on the order of the 1974), the strength of selection on inversions bor- equilibrium fitness variance in the population times dered by two loci that are 2.5 to 3 centimorgans apart the deviation of the value of its modified parameter can be seen to be more than 98.8% of f/(1 - f). from the equilibrium value before its introduction. DISCUSSION The selective force on the modifier may be interpreted as a consequence of the selection on the major General considerations: We have shown in this genes rather than imperfect transmission. By substi- paper that many genetic models in the literature that tuting from an alternative form of the equilibrium incorporate complex transmission, selection, recom- identity (29), namely bination, mutation, and modification, are special cases of recursion (5). A basic finding from this general framework is the intimate connection between imperfect transmission, equilibrium selection potential (i.e., 568 L. Altenberg and M. W. Feldman variation in marginal fitnesses), and the evolution of hiking effects at all. But to the contrary, Result 2a transmission under the control of modifier genes. The demonstrates that in fact hitchhiking can occur in the “selection potential” we have defined enters quanti- absence of linkage between the modifier locus and the tatively into the strength of selection acting on modi- selected loci, and when the alleles of the selected loci fier genes by providing an upper bound for the poten- are very close to equilibrium. tial relative rate of increase in frequency of any hap- This paradox is resolved by noting that when im- lotype in the population due to selection. With perfect perfect transmission permits a positive selective poten- transmission, this value vanishes at any equilibrium. tial at equilibrium, there is a constant net “flow,” by In the presence of imperfect transmission, equilibria imperfect transmission, from the fitter haplotypes to may be reached where the selection potential is kept the less fit. By altering this “flow,” a modifier allele above zero by an amount dependent on the level of can come to be in disequilibrium with the selected imperfect transmission. This balance occurs because haplotypes, and may therefore acquire a marginal haplotypes that are depleted by selection are replen- fitness different from the mean of the population. For ished from other haplotypes by imperfect transmis- Result 2a, the linkage disequilibrium between the sion, and conversely, haplotypes whose numbers are modifier and the selected loci that causes hitchhiking augmented by selection are depleted by imperfect is generated by selection every generation, and even transmission. free recombination can no more than halve this dis- Selective forces on modifier genes: When the mod- equilibrium by the next phase of selection. It should ifier locus is perfectly transmitted, as we have assumed be noted that any continuous dependence of T p- W throughout, the only force changing the frequencies on allele frequencies does not enter into Q,D, so of its alleles is selection, manifested as differences in provided that p(Q,D) # 1 neither the magnitude nor the alleles’ marginal fitnesses. When the modifier direction of selection on a new modifier allele are locus is selectively neutral, the marginal fitness of each affected by frequency dependence. modifier allele is simply the average of the fitnesses of What do the estimates from Results 2 and 4 for the the selected haplotypes with which it is associated. strength of selection on a new modifier allele suggest Summing over i in (1 3) shows that for the order of magnitude of selection on modifiers in nature? The selection potential is the maximum possible rate of increase for modifier alleles, as well as selected alleles, and is exploited to its fullest by a modifier that stops all imperfect transmission and is where xb is the total frequency of modifier allele Mb, tightly linked to the selected haplotype. and As the guaranteed level of perfect transmission of the haplotypes decreases, i.e., as the value of a in- wMb = zbjwjlxb creases, the upper bound on the equilibrium selection potential, a/(l - a), increases without limit (as a goes is its marginal fitness. to one). The typical values of a in nature depend on These marginal fitnesses would clearly be the same the particular imperfect transmission process. In the for all of the modifier alleles if they were randomly case of mutation, even though per-locus mutation associated with the selected haplotypes. Evolution of rates are quite small, per-haplotype rates can be quite modifier genes occurs because, through their effects high, of order 1. Similarly, per-chromosome recom- on transmission, they may be able to create linkage bination rates can be on this order (VONWETTSTEIN, disequilibrium between themselves and the selected RASMUSSENand HOLM1984, p. 399). Therefore, the haplotypes. Those modifier alleles that occur more selection potentials that may be typical of natural frequently with the fitter types in the population in- populations at equilibrium, which are the potential crease in freqyency. This is known as “hitchhiking,” strengths of selection that can be induced on neutral and is the fundamental mechanism by which neutral, modifier genes, may be quite strong. So we may perfectly transmitted modifier genes evolve. consider what happens when selection acts directly on The dynamics of hitchhiking in populations that are the modifier locus, which may be a biologically more in a transient phase of their evolution differ greatly reasonable assumption given the ubiquity of pleio- from the dynamics in populations that are near equi- tropy (WRIGHT1964). We see that the induced selec- librium (CHARLESWORTH,1976; STROBECK,MAY- tion on a modifier due to its effect on transmission NARD SMITHand CHARLESWORTH,1976). Hitchhiking can overwhelm considerable direct selection acting on is usually thought of as a process whereby an allele at the modifier. If direct selection gives modifier geno- one locus increases in frequency by being linked to an type M1/M2 a selective valuefi2, and interacts multiallele at another locus that is increasing in frequency plicatively with selection on the major loci, then the due to selection (EWENS1979, p. 205). It might be fitness of genotype MIA,/M2A, will befi2w8, and the expected that at an equilibrium, because there are no rate of change in the frequency of modifier allele M2 changes in the frequencies, there could be no hitch- when rare will befinp(Q2D)- 1. Evolution of Modifier Genes 569 Reduction principle: In Result 2, the condition for transmission event, but may change the distribution of the invasion of a new modifier allele was that it elim- gametic haplotypes produced by a given parental gen- inate imperfect transmission acting on the selected otype, and this also produces deviations from linear loci provided a nonzero selection potential is main- variation. tained at equilibrium by imperfect transmission. This Examples of nonlinear variation include models of is seen to be true for all viability regimes, and all recombination modification in the presence of muta- polymorphisms at the modifier and selected loci. Re- tion (FELDMAN,CHRISTIANSEN and BROOKS 1980), sult 3, for the case of linear variation, is more re- models such as recombination modification in the stricted in scope, applying only for tightly linked mod- presence of migration (CHARLESWORTHand CHARLES- ifier genes that produce linear variation in the trans- WORTH 1979b), the model of CHRISTIANSENand mission, with the population fixed on the modifier FELDMAN(1975) where a modifier controls either gene. In this case, a new modifier allele introduced recombination or migration in a population undergo- into a population bearing a selection potential induced ing recombination and migration, and the model of a by imperfect transmission can increase if and only if modifier acting on recombination between multiple it takes the transmission closer to perfect transmission, loci by CHARLESWORTHand CHARLESWORTH(1979a). and is excluded if it takes the transmission further In the next section we discuss some examples of non- away. We conjecture that our results for linear varia- linear variation that lead to increases in the modified tion will hold for arbitrary levels of linkage between parameter. the modifier and selected genes. Evolutionary genetic stability of perfect transmis- Result 3 has several implications. In previous treat- sion: ESHELand FELDMAN(1 982) developed “evolu- ments of recombination modification, only two loci tionary genetic stability” of a phenotype as a first order under selection with two alleles each were allowed, criterion for establishing that the long-term tendency and it was assumed that both double heterozygotes at of evolution is toward this phenotype. In their papers the selected loci had the same fitness, and had the on the*general reduction principle in the cases of same crossover frequencies for any given modifier mutation, recombination, and migration modifica- genotype. None of these assumptions enters Result 3: tion, LIBERMANand FELDMAN(1986a,b; 1988) con- there may be multiple alleles at each selected locus jecture that for the models they consider, the pheno- and there may be multiple loci under selection. If types of zero mutation, zero recombination, and zero there are more than two selected loci, however, for migration have evolutionary genetic stability given the variation of the transmission to be linear, there four conditions: must be complete interference between crossover events among the loci, and the modifier must simul- 1. The selection of the major locus (or loci) is at taneously control every crossover event for the set of the level of differential genotypic viabilities. loci. Different fitnesses are allowed for the different 2. These viabilities do not change over time. linkage phases in the multiple heterozygotes at the 3. There is random mating. selected loci (i.e., position effects on fitness are al- 4, Besides viability selection, the only evolutionary lowed). For any given modifier genotype, different force operating is that feature of the genetic recombination rates for the different double hetero- system subject to genetic modification. zygotes at the selected loci are also allowed. For multiple loci under selection, the crossover frequency In the cases analyzed here, we would expect that distribution among the loci need not be equal for the after its initial increase a new modifying allele would different heterozygotes. All that is required is that the change in fTequency until a new equilibrium is ap- modifier gene affect the crossover rates linearly. proached at which the population average parameter Most of the basic models of modification in the value is less than that at the initial equilibrium. In literature treat linear variation, including models of other words, perfect transmission should be the state mutation modification (LIBERMANand FELDMAN towards which evolution proceeds in these cases. The 1986b; HOLSINCERand FELDMAN1983b; KARLIN proof of this result, analogous to that of KARLINand and MCGREGOR1974), recombination modification LESSARD(1984), appears to be significantly more dif- (FELDMAN,CHRISTIANSEN and BROOKS1980; TEACUE ficult than that of our results here. 1976; CHARLESWORTH1976) and migration modifi- Because perfect transmission is not the norm in cation (BALKAUand FELDMAN1973; KARLIN and nature, with the ubiquity of mutation and the preva- MCGRECOR1974; TEACUE1977; LIBERMANand lence of different kinds of recombination, it is impor- FELDMAN1988). Several factors cause a model to tant to ask whether there are ever conditions under deviate from linear variation, including nonrandom which a new modifier allele that increases the level of mating, and the involvement of more than one imper- imperfect transmission in the population can invade. fect transmission process during the life cycle. Also, In fact, there are cases where it can. When selection the modifier may not simply vary the overall chance fluctuates, recombination rates (CHARLESWORTH that a selected haplotype is “hit” by an imperfect 1976) or mutation rates (GILLESPIE1981a) may in- 570 L. Altenberg and M. W. Feldman

crease. When the population is of finite size and CHARLESWORTH,B., and D. CHARLESWORTH,1973 Selection of random genetic drift is included in the model, GIL- new inversions in multi-locus genetic systems. Genet. Res., 21: 167-1 83. LESPIE (198 1b) has shown that mutation rates may CHARLESWORTH,D., and B. CHARLESWORTH,1979a Selection on increase and the same has been shown for recombi- recombination in a multi-locus system. Genetics 91: 575-580. nation by FELSENSTEIN(1 974) and FELSENSTEINand CHARLESWORTH,D., and B. CHARLESWORTH,1979b Selection on YOKOYAMA(1976). When one gene controls the seg- recombination in clines. Genetics 91: 58 1-589. regation pattern at another locus, which is also subject CHARLESWORTH,D., B. CHARLESWORTHand C. STROBECK, to viability selection, the recombination between the 1979 Selection for recombination in partially self-fertilizing populations. Genetics 93: 237-244. pair is not necessarily reduced (PROUT, BUNGAARD CHRISTIANSEN,F. B., and M. W. FELDMAN,1975 Subdivided and BRYANT1973; THOMSONand FELDMAN1974). populations: a review of one- and two-locus deterministic the- Another example is provided by the case of recombi- ory. Theor. Popul. Biol. 7: 13-38. nation modification analyzed by FELDMAN, CHRIS- CROW,J. F., 1958 Some possibilities for measuring selection in- tensities in man. Hum. Biol. 30 1-13. TIANSEN and BROOKS(1980), where the loci whose ESHEL,I., and M. W. FELDMAN,1982 On evolutionary genetic linkage is controlled by the modifier also undergo stability of the sex ratio. Theor. Popul. Biol. 21: 430-439. mutation. In this case there are conditions where a EWENS,W. J., 1979 Mathematical Population Genetics. Springer- new modifier allele can invade the population if it Verlag, Berlin. increases the frequency of recombination between the FELDMAN,M. W., 1972 Selection for linkage modification. I. Random mating populations. Theor. Popul. Biol. 3: 324-346. loci. FELDMAN,M. W., and B. BALKAU,1973 Selection for linkage How is this latter result possible in light of Result 3 modification. 11. A recombination balance for neutral modi- on modifiers producing linear variation in transmis- fiers. Genetics 74: 71 3-726. sion? The answer is that in the presence of mutation, FELDMAN,M. W., and U. LIBERMAN,1986 An evolutionary re- the variation produced by a modifier of recombina- duction principle for genetic modifiers. Proc. Natl. Acad. Sci. USA 83: 4824-4827. tion is not linear. In this example, condition 4 in the FELDMAN, M. W., F. B. CHRISTIANSENand L. D. BROOKS, conjecture of LIBERMANand FELDMAN(1 986a,b), 1980 Evolution of recombination in a constant environment. above, is violated. In other cases where increases in Proc. Natl. Acad. Sci. USA 77: 4838-4841. imperfect transmission can evolve in populations near FELSENSTEIN,J., 1974 The evolutionary advantage of recombi- equilibrium, the variation in the transmission again nation. Genetics 78: 737-756. does not fit the form for linear variation. This is the FELSENSTEIN,J., and S. YOKOYAMA,1976 The evolutionary advantage of recombination. 11. Individual selection for recom- case, for example, when recombination can evolve to bination. Genetics 83: 845-859. increase in the presence of migration (CHARLES- GILLESPIE,J. H., 1981a Mutation modification in a random envi- WORTH and CHARLESWORTH, 1979b), or selfing ronment. Evolution 35: 468-476. (CHARLESWORTH, CHARLESWORTHand STROBECK GILLESPIE,J. H., 1981b Evolution of the mutation rate at a 1979; HOLSINCERand 1983a), or mutation heterotic locus. Proc. Natl. Acad. Sci. USA 78: 2452-2454. FELDMAN HOLSINGER,K. E., and M. W. FELDMAN,1983a Linkage modifi- can increase in the presence of fertility selection (HOL- cation with mixed random mating and selfing: a numerical SINGER, FELDMANand ALTENBERC1986) or selfing study. Genetics 103: 323-333. (HOLSINGERand FELDMAN1983b). The generalized HOLSINGER,K. E., and M. W. FELDMAN,1983b Modifiers of reduction results we obtain here show that it is the mutation rate: Evolutionary optimum with complete selfing. nature of the variation in the transmission process that Proc. Natl. Acad. Sci. USA 80: 6732-6734. HOLSINGER,K., M. W. FELDMAN and L. ALTENBERG, determines the direction of its evolution. Factors in- 1986 Selection for increased mutation rates with fertility fluencing the functional form of this variation should differences between matings. Genetics 112: 909-922. therefore be the focus for developing further under- KARLIN, S., 1982 Classifications of selection-migration structures standing of the evolution of genetic transmission. and conditions for a protected polymorphism. Evol. Biol. 14: 61-204. KARLIN,S., and D. CARMELLI,1975. Numerical studies on two loci We wish to thank FREDDYCHRISTIANSEN, URI LIBERMAN, LUIGI selection models with general viab es. Theor. Popul. Biol. 7: CAVALLI-SFORZA,JOSEPH OLICER, SHR~PAD TULJAPURKAR, KENT 399-421. HOLSINGERand JOELPECK for their helpful comments, and RICH- KARLIN, S., and S. LFSSARD, 1984 On the optimal sex ratio: a ARD LEWONTINfor the idea (in the context of cultural evolution) stability analysis based on a characterization for one locus that selection and transmission form a useful dichotomy. multiallele viability models. J. Math. Biol. 20 15-38. This research was supported in part by research grants GM KARLIN, S., and J. MCGREGOR,1972 Polymorphisms for genetic 28016, GM 10452, GM 07181 and FD 2589-9414L from the and ecological systems with weak coupling. Theor. Popul. Biol. National Institutes of Health. 3: 210-238. KARLIN, S., and J. MCGRECOR,1974 Towards a theory of the evolution of modifier genes. Theor. Popul. Biol. 5: 59-103. LITERATURE CITED KINGMAN,J. F. C., 1980 Mathematics of Genetic Diversity. Society for Industrial and Applied Mathematics, Philadelphia. BALKAU,B. J., and M. W. FELDMAN,1973 Selection for migration LIBERMAN,U., and M. W. FELDMAN,1986a A general reduction modification. Genetics 74 171-174. principle for genetic modifiers of recombination. Theor. CHARLESWORTH,B, 1976 Recombination modification in a fluc- Popul. Biol. 30: 341-371. tuating environment. Genetics 83: 181-195. LIBERMAN,U., and M. W. FELDMAN,1986b Modifiersof mutation Evolution of Modifier Genes 57 1

rate: A general reduction principle. Theor. Popul. Biol. 30 M( = l,e‘M = eT,and e‘[ = 1, we have 125-1 42. LIBERMAN,U., and M. W. FELDMAN,1988 The reduction princi- P(MD) 2 E E$,. ple for genetic modifiers of the migration rate. In: Mathematical Evolutionary Theory, Edited by M. W. FELDMAN.Princeton In the present case [ is DG, since University Press, Princeton, N.J. (to appear) NEI, M., 1967 Modification of linkage intensity by natural selection. Genetics 57: 625-641. PROUT,T., J. BUNDGAARDand S. BRYANT,1973 Population genetics of modifiers of meiotic drive. I. The solution of a special Thus case and some general implications. Theor. Popul. Biol. 4 [(l - r)I + rQ]D; = DG, 446-465. STROBECK,C., J. MAYNARD SMITH and B. CHARLESWORTH, and therefore

1976 The effects of hitchhiking on a gene for recombination. *A wi w, - Genetics 82: 547-558. p(Q,D) = p([(l - r)I + rQ]D) 2 3,~T - TEAGUE,R., 1976 A result on the selection of recombination I ww 133) altering mechanisms. J. Theor. Biol. 59 25-32. TEAGUE,R., 1977 A model of migration modification. Theor. Popul. Biol. 12: 86-94. THOMSON,G. J., and M. W. FELDMAN,1974 population genetic Thus, when not all 6,are equal, of modifiers of drive. 11. Linkage modification in the segregation distortion system. Theor. Popul. Biol. 5: 155-162. p(0.D) = p([(l - r)I + rQ]D) > 1, VON WETTSTEIN,D., S. W. RASMUSSENand P. B. HOLM,1984 The and the new modifier increases when rare. synaptonemal complex in genetic segregation. Annu. Rev. Ge- Result 2b: if r = 0 then Q, = I, and so p(Q,D) = p(D) = maxi net. 18: 331-413. (WJW). WRIGHT,S., 1964 Pleiotropy in the evolution of structural reduc- Result 2c: See (33). tion and of dominance. Am. Nat. 98: 65-69. Result 2d This is a direct result from Theorem 5.2 of KARLIN Communicating editor: M. TURELLI (1982), which states that ~([(l- a)I + aM]D) is a decreasing function of a, where M is a stochastic matrix and D a positive APPENDIX: PROOFS OF THE RESULTS diagonal matrix. Result 2e: This is an incidental implication of Results 2b and Proof of Result 2 2d and Definition 1, which give Here the new modifier allele Mayields perfect transmission, hence

T(ai + aj I bk) = f(ui t uj I bk) = 6, for all i, j, b, k. Then (1 8) reduces to Proof of Result 3

1 The recursion on the frequency of M2 is, from (1 5) and (1 8) e:, = y [(l - r)eoaG, + rt, c ealw,], W I e’ = [(l - m2,)((1 - r)I + rQ) (34) or, in vector form, + mnl((1 - r)irl + r91)lDe, e’ = Q.De = [(l - r)I + rQ]De, (32) where Q, PI,GI and D are defined as in (25), (21), (22) and (15). where D is defined as before in (10) and Q = 113, $llrd=l = We know by the Perron-Frobenius theorem that since 7% the strictly positive eigenvector i in (20) has eigenvalue 1, DIWD~,with D1 = diag(+), and D2 = diag(G;’). Result 2a: We must show that the spectral radius ~([(l- r)*’ + rGl]D) = 1. At r = 0, equation (34) reduces to p(Q.D) = p([(l - r)I + rQ]Di > 1, for all 0 5 r I!h. e’ = [(I - m2l)I + mzIirl]De= ~2Dc. Observe first that all eigenvalues of Q are real since both We now use Theorem 5.2 of KARLIN(1982), which states: matrices D1WD2 and (D2DI)’/’W(D1D2)’/2 have the same eigen- Let M be an irreducible stochastic matrix. Consider the values and the latter, being symmetric, has all real eigenvalues. family of stochastic matrices Since Q is stochastic its spectral radius is 1 (all eigenvalues of Q lie between -1 and 1) and therefore the eigenvalues of (1 - r)I Ma = (1 - a)I + aM. rQ are confined to the positive interval 0 5 X I1 provided + Then for any positive diagonal matrix D (D # cI, c > 0), 0 s r I%. The symmetric matrix p(a) = p(MnD) is strictly decreasing as a increases. (DID~)I/~[(l r)(D1D2)-I ~W](DID~)”~ - + Therefore, p(Q,D) is non-increasing in mZ1. If mZ1 = 1, :hen is non-negative and symmetric and it has the same non-negative p(Q2D)= 1, so when r = 0, p(Q2D) 2 1 if m21 < 1 and p(Q2D) 5 eigenvalues as the matrix (1 - r)I + rQ. Hence the former 1 if mz1 > 1. matrix is positive semidefinite. Implicit in these inequalities on mzl is the scaling mll = 1. This enables us to use Theorem 5.1 Corollary F.2 of KARLIN With respect to the general situation, the previous inequalities (1982), which states that for a positive diagonal matrix D = on mZ1are replaced by inequalities in m2JmI1. From the theofy diag(D,) and a matrix M that is symmetrizable to a positive of small parameters (KARLINand MCGRECOR1972), when YI semidefinite matrix (i.e., M = ElPE2 where El and E2 are is irreducible, Result 3 holds also for some range of r > 0. The diagonal positive matrices and P is positive semidefinite), with general case 0 < r < Yz is conjectured to hold also. 57 2 L. Altenberg and M. W. Feldman

Proof of Result 4 We then write

Assume first that pi > 0 for all i, and that r = 0. We make use of a result due to KARLIN(1982, p. 173) to the effect that if the matrix S has the representation

S=E+R, (35) where E = diag(el, e2, . . . , e,) and R = 11 uivl 11 with u,vj 2 0 for by (29). We may then expand (39) at A = 1 and neglect terms all p(S) i andjand e, < 1 for all i then the spectral radius of S is O(mll - mZ1)’Lto obtain that value of A for which

I + uiv,/(e,- A) = 0. (36) i= I In the present _case the matrix of external stability is In the same way, [( 1 - mnl)I + mz1P]D,and comparing this to (35) we may use E = (1 - m21)diag(&) (37) and On substituting (42) and (43) into (40) we conclude that, except R = m21 IIpici,It (38) for terms O(ml I - mZ1)‘, where = &,/$. Note that from (29)

qi- (1 - ~~JcJ,/$J= mllp, so that 1 - (1 - ~II)&/$ > 0. For mZl sufficiently close to mll obviously (1 - m21)3,/$ < 1 also. Hence by KARLIN’S result quoted above the required spectral radius A. is the unique solution of (45)

;i U(A)= 1 + mZ1 p,&/[(l - mZ1) - AI = 0. (39) which upon substitution into (44) produces the conclusion of the stated Result 4. Clearly when mZI = mll, A, = 1 and for m21 close to mll, A, is It should be noted that if ph = 0 for some h then with r = 0, close to 1. For A, near 1 we have, by Taylor’s theorem, for any m2, with 0 5 mZ1C 1 it can be shown that

= &)2 U(1) U(A,) + U‘(l)(l - A,) + O(1 - mll - m21 - 1 - mZ1 p(Y*D) = 1 + - 1 +(I -%)I? with U(A,) = 0. Hence, to this order of approximation, 1 -mll 1 -mll A” = I - U(l)/U’(l). (40) where fi is the selection potential at this equilibrium.