Copyright  1999 by the Genetics Society of America

The Reinforcement of Mating Preferences on an Island

Mark Kirkpatrick*,† and Maria R. Servedio*,1 *Department of Zoology, University of Texas, Austin, Texas 78712 and †Ge´ne´tique and Environnement, Institute de l’Evolution, Universite´ Montpellier 2, 34095 Montpellier, France Manuscript received April 10, 1998 Accepted for publication September 15, 1998

ABSTRACT We develop a haploid model for the reinforcement of female mating preferences on an island that receives migrants from a continent. We find that preferences will evolve to favor island males under a broad range of conditions: when the average male display trait on the island and continent differ, when the preference acts on that difference, and when there is standing genetic variance for the preference. A difference between the mean display trait on the continent and on the island is sufficient to drive reinforcement of preferences. Additional postzygotic isolation, caused, for example, by either epistatic incompatibility or ecological selection against hybrids, will amplify reinforcement but is not necessary. Under some conditions, the degree of preference reinforcement is a simple function of quantities that can be estimated entirely from phenotypic data. We go on to study how postzygotic isolation caused by epistatic incompatibilities affects reinforcement of the preference. With only one pair of epistatic loci, reinforcement is enhanced by tighter linkage between the preference genes and the genes causing hybrid incompatibility. Reinforcement of the preference is also affected by the number of epistatically interacting genes involved in incompatibility, independent of the overall intensity of selection against hybrids.

HILE mating preferences are clearly fundamen- ous. Evidence of reinforcement has, however, been W tal as isolating mechanisms in many groups of found in several detailed studies of geographical varia- animals, it is unclear whether they are the cause or tion in mating behavior within species, for example, in the effect of speciation. A common view is that mating frogs (Gerhardt 1994), flies (Noor 1995), and birds preferences are an important cause of speciation. The (Saetre et al. 1997). Strong support for the widespread hypothesis here is that mating preferences diverge in operation of reinforcement comes from the work of isolated populations as a by-product of genetic change Coyne and Orr (1989b, 1997; see also Noor 1997). caused by either adaptation or genetic drift. This diver- Their review of studies on 171 species of Drosophila gence causes prezygotic isolation that results in spe- shows that sympatric pairs of species have far stronger ciation (Mayr 1963; Rice and Hostert 1993). The behavioral isolation than allopatric pairs that have been converse possibility, that speciation causes preference isolated for comparable amounts of time. evolution, was suggested by Dobzhansky (1940). Under Theoretical objections to the reinforcement hypothe- his idea of reinforcement, an important side-effect of sis have also been raised. Reinforcement involves two divergence in allopatry is partial postzygotic isolation. competing forces that act on a mating preference. The Crosses between individuals from different populations first is indirect selection. Even if preferences do not produce offspring with reduced fitness. When two popu- directly affect survival or the number of gametes pro- lations come into contact, there is a selective premium duced, preference alleles favoring heterotypic matings for individuals from each population to avoid mating are indirectly selected against because they are as- with individuals from the other. Consequently, mating sociated with low-fitness hybrid genotypes. This is the preferences evolve to reinforce the postzygotic isolation, engine that drives reinforcement. The countervailing accelerating the process of speciation. force is gene flow: hybridization causes the preference Although intuitively appealing, the reinforcement hy- genes in the two populations to become homogenized. pothesis has a checkered history of support from evo- The most obvious theoretical problem for reinforce- lutionists (reviewed by Butlin 1987, 1989; Howard ment is that the force of gene flow might overwhelm 1993). On the empirical side, weaknesses have been indirect selection and prevent reinforcement from suc- identified in many early studies that were offered in ceeding (Moore 1957; Mayr 1963). Recombination support of the idea, making their interpretation ambigu- breaks down the genetic associations between prefer- ence genes and the loci selected against in hybrids (Felsenstein 1981; Barton and Hewitt 1985), which Corresponding author: Mark Kirkpatrick, Department of Zoology, Uni- versity of Texas, Austin, TX 78712. E-mail: [email protected] weakens the force of indirect selection favoring rein- 1 Present address: Section of Ecology and Systematics, Corson Hall, forcement. Cornell University, Ithaca, NY 14853. Several models have established that reinforcement

Genetics 151: 865–884 (February 1999) 866 M. Kirkpatrick and M. R. Servedio of mating preferences can succeed, however, at least in specific model in which hybrid unfitness is caused by some circumstances. Sved (1981) considered the situa- epistatic interactions. Results there show how the ge- tion of complete hybrid inviability or sterility and de- netic basis of hybrid unfitness can affect the outcome rived analytic expressions for the reinforcement of a of reinforcement. specific type of mating preference. Three simulation There are four features of the model we emphasize studies since then have found that reinforcement of at the start. First, we assume that mating preference mating preferences succeeds in some but not all cases. genes are free of direct selection; that is, preference Spencer et al. (1986) developed a model that includes genes do not affect the survival or the number of ga- demography as well as evolution and tracked the fate metes produced by an individual. This allows us to look of a single population when hybrids are completely in- at the effects of reinforcement on preference evolution fertile or inviable. Liou and Price (1994) extended that in isolation from other evolutionary forces. In our model by allowing reciprocal introgression between a model, reinforcement results from genetic associations pair of species that have only partial postzygotic isola- (linkage disequilibrium) between preference alleles tion. Servedio and Kirkpatrick (1997) compared situ- and other genes that are directly selected, for example ations in which there is reciprocal introgression with those that cause hybrids to have low fitness, as envi- situations involving one-way gene flow. Because these sioned by Dobzhansky. Second, the model is quite gen- simulation models make quite restrictive assumptions eral in several respects regarding genetics (e.g., there about genetics and behavior, it is not known how gen- can be any number of genes and any linkage relations eral their qualitative conclusions might be or how the between the genes that affect hybrid fitness, the prefer- results can be applied quantitatively to natural popula- ence, and the display) and behavior (any form of mating tions. preference is allowed). The model is therefore free of This article develops an analytic model for investigat- several assumptions that restrict the generality of earlier ing the reinforcement of mating preferences on an is- models. Third, the model treats the average value of land that receives immigrants from a continent. It con- the display trait as a known quantity, rather than ac- siders what happens when postzygotic isolation is weak counting for its evolution explicitly. This approach and so applies to the early phases of divergence. We makes the results general to all forms of natural and choose to focus on this situation for two reasons. First, sexual selection that might act on a display. Fourth, the islands are sites of rapid and profuse speciation (Mayr migration rate of continental individuals onto the island 1963), and it is of interest to see how reinforcement is also viewed as a known quantity. Again, this buys might play a role in these radiations. Second, it is simpler generality: no restrictive assumption is made about how to study a single island population than two or more the values of the mating preference and display trait populations in which the picture is complicated by the affect rates of hybridization. But another consequence effects of reciprocal gene flow and spatial structure. is that our model does not describe how evolution might Our main aim is to determine if and how mating shut off introgression and lead to complete reproduc- preferences that discriminate against the continental tive isolation, as Dobzhansky suggested. Thus ours is a immigrants will evolve on an island. Major questions model for the reinforcement of mating preferences, but are: Can reinforcement of mating preferences occur, not for the reinforcement of prezygotic isolation. and if so how big of an impact will it have? Does the way in which hybrids are selected against determine if THE GENERAL MODEL reinforcement succeeds? How is reinforcement affected by genetic details such as the number of loci, their Our model considers the evolution of three kinds of linkage, and the distribution of their effects? When can characters: a female mating preference, a male display observable phenotypic characters be used to make pre- trait that the preference acts on, and incompatibility dictions about the outcome of reinforcement? traits that reduce the fitness of hybrids. To make the The article begins by describing a general haploid analysis tractable, we assume that the genome is haploid. model. Initially, we make no specific assumptions about There can be any number of autosomal loci underlying what causes hybrids to have reduced fertility and/or the traits. The model therefore covers the special cases viability. The results therefore apply to cases where eco- where only a single locus affects the female preference logical factors select against hybrids that have pheno- or male display trait and situations where these charac- types intermediate between the parental forms (e.g., ters vary quantitatively as the result of contributions Grant and Grant 1993; Schluter 1995, 1996), for from many loci. Two alleles, denoted 0 and 1, segregate example, and to those where epistatic interactions cause at each locus. Any linkage map is allowed; no assump- developmental problems in hybrids (reviewed by Coyne tion is made about recombination rates. The population 1992; Wu and Palopoli 1994; Turelli and Orr 1995). size on the island is large enough that the effects of We find that evolution will typically reinforce prefer- drift can be ignored. ences on the island so that females there discriminate To analyze the model, we use notation and results against males from the continent. We then develop a developed by Barton and Turelli (1991), who intro- Reinforcement of Mating Preferences 867 duced a general framework for studying evolution in ten in the same form as Equation 1, but with T replacing T P multilocus systems. Their approach is extended here to P, T replacing P, ␥i replacing ␥i , and eT replacing eP on allow for migration and a series of selection events that the right-hand side. It is possible to generalize the model occur over the course of each generation. to allow for epistatic interactions among the display trait In our model, the mating preference can be any trait genes (see Kirkpatrick and Barton 1997). that affects the probabilities that a female will mate When a preference and a display trait are framed with different kinds of males (Kirkpatrick and Barton in this general way, Kirkpatrick and Barton (1997) 1997). This general view of a preference subsumes many found that a key parameter governing evolution of the earlier models of mating preferences as special cases. preference is the phenotypic correlation between the For example, a preference might control a mating pref- preference in females and the display trait in males erence function (as in Lande 1981) or the tendency of among mated pairs. We denote this correlation by ␳.Itis a female to mate with one of two types of males (as in a phenotypic measure and so is free of any assumptions O’Donald 1980). But the present model is more gen- about the genetics underlying the preference and dis- eral and allows a female’s preference, P, to be any mea- play trait. This correlation is, however, responsible for surable aspect of her phenotype: the period of day when creating the genetic associations between preference she searches for mates, for example, or the average alleles and the genes that affect the display trait and number of males she inspects before making a decision. hybrid fitness. The value for ␳ can be calculated given We assume that genetic variation in the preference a specific set of assumptions about how mating occurs. is contributed by one or more loci with additive effects, Alternatively, it could also be measured directly in na- and we denote this set of loci as P. These loci do not ture. Consider a species of frog where pairs of individu- affect survival or immediate fecundity (that is, the num- als can be caught in copula. For each pair, one would ber of eggs a female produces). The preference pheno- measure the male’s display trait (e.g., the fundamental type of a particular female can always be written in the frequency of the call) and also the female’s preference form (e.g., the percentage of times she approaches a speaker

P playing a low-pitched rather than a high-pitched call). P ϭ P ϩ ͚ ␥i (Xi Ϫ pi) ϩ eP. (1) i෈P The correlation between these two numbers gives an estimate of ␳. The value of ␳ may change as a population The summation is over each of the loci that contribute evolves; in that case, the value relevant to the equations to variation in the preference (the symbol ෈ indicates below is the one that prevails in the current generation. there is one term in the sum for each locus in set P), The last factor relevant to the model is hybrid incom- P is the mean of the preference on the island at the patibility. Because many things reduce the viability and/ P start of a generation, ␥i is the difference in the effects or fertility of hybrids in nature, for now we do not on the preference of the two alleles at locus i, and eP is commit to any specific assumption about the causes of a random environmental effect. The variable Xi takes incompatibility. The model therefore allows for many on value 1 if the individual carries allele 1 at locus i and possible modes of selection against hybrids. (Later in value 0 if it carries allele 0. The frequency of allele 1 the article we focus on hybrid incompatibility caused by among zygotes on the island is pi, and the frequency of epistasis.) The set of all the loci involved in determining P the allele 0 is qi ϭ 1 Ϫ pi. The quantity ␥i (Xi Ϫ pi)in hybrid incompatibility is denoted H. Equation 1 measures how much locus i causes that indi- Regarding migration, a proportion m of the individu- vidual’s preference to deviate from the average prefer- als on the island arrives from the continent in each ence. (Two quantities on the right side of Equation 1, generation. No assumption about the sex ratio of the P and pi, change across generations but those effects migrants is made. Without migration, the model re- cancel in such a way that genotypic values remain con- duces to a “null model” of sexual selection: because stant.) The notation used throughout the article is sum- there is no other force acting on the preference, it is marized in appendix a. free to equilibrate at any value (Kirkpatrick and Ryan Females choose mates on the basis of a trait that 1991). With gene flow from the continent, however, males display. The model allows this trait to be expressed migration and indirect selection produce forces that act either in males alone or in both sexes. Natural selection on the preference, and our goal is to find out how these acts on the trait, but no specific assumption is made forces interact. The model does not make assumptions about the form of the fitness function. Likewise, the about the history of the two populations. It therefore model allows for any form of sexual selection that might applies to cases of secondary contact (in which the island be caused by the female preference. The value for a and continent populations initially diverge in isolation) male’s display trait phenotype, T, is affected by a set of and also to cases where an island is colonized from a loci denoted T. We assume these loci have additive continent and continuously receives gene flow there- effects on T and that the genes are expressed the same after. way in both sexes if the trait is present in females. The The order in which the events of selection and migra- value of a display trait phenotype can therefore be writ- tion occur in the life cycle varies according to the ecol- 868 M. Kirkpatrick and M. R. Servedio ogy of the species. Dispersal in some species occurs any number of loci, the disequilibria are defined by the before natural selection acts on the male display trait, relation for example, but afterward in other species. Accord- C ϭ E[(X Ϫ p )(X Ϫ p )(X Ϫ p )...], (4) ingly, we make no restrictions about the ordering of ijk... i i j j k k migration and natural selection in the life cycle and where E[·] stands for the expectation taken over all let the model determine when and why the ordering genotypes in the population. The value of C in the matters. population changes both within and between genera- Evolution of a preference gene: This section intro- tions as the result of selection, migration, and recombi- duces the model for the evolution of a preference locus, nation. Calculations for the C that appear in this model explaining the major components and notation. Each are given in appendix c. term is then considered in detail in the subsequent The superscript n in the a˜ n and C n of Equations 2 section. and 3 indicates that those quantities are evaluated at Evolution of the preference can be described exactly the point in the life cycle when natural selection acts using the notation of Barton and Turelli (1991). The on the male display trait. Likewise, terms carrying an s change in the frequency of an allele at preference locus superscript are evaluated at the time that sexual selec- i in a single generation is tion (mate choice) happens, while the superscript h in-

n n s s h h dicates quantities evaluated when selection acts against ⌬pi ϭ Mi ϩ ͚ a˜ UC iU ϩ ͚ a˜ UC iU ϩ ͚ a˜ UC iU. (2) UʕT UʕT UʕH hybrids. An important implication of Equation 2 is that the The right-hand side is a sum of four terms, correspond- associations between alleles at two, three, and larger ing to the four events that change preference allele numbers of loci can contribute to the evolution of the frequencies over the course of a generation. The first preference. The Barton-Turelli framework, in contrast term represents the effect of migration, which causes to many other theoretical approaches, can account for evolution whenever there are differences in the allele these associations exactly. In practice, however, expres- frequencies of individuals on the island and the conti- sions for the C can quickly become unmanageable. The nent. The next three terms on the right of Equation 2 approach is made practical by a clever method of ap- are summations that reflect the impact on the prefer- proximation that they devised. Provided the forces of ence of natural selection acting on the male display selection and migration acting on genotype frequencies trait, of sexual selection acting on that trait, and of are weak and recombination rates are not too small, selection against hybrid incompatibility. The symbol ʕ then a population will rapidly evolve to a state called indicates that the summations are taken over all subsets “quasi-linkage equilibrium” (QLE; Kimura 1965; Nagy- of the set listed to the right of the symbol (including laki 1976). At this point, allele frequencies will change that set itself). relatively slowly and the disequilibria between sets of To better understand this notation, consider the first loci (the C) will be small compared to their maximum summation. It is taken over all sets of male display trait possible values. These facts allow one to find simple ap- loci in the genome, including the full set T. For exam- proximations for the a˜ and C that appear in Equation 2. ple, if genetic variation in the male trait is attributable To make the model tractable, we analyze it assuming entirely to the two loci j and k, then that summation is the population is in QLE. Biologically, this means that n n n n n n n n ͚ a˜ UC iU ϭ a˜ j C ij ϩ a˜ k C ik ϩ a˜ jkC ijk. (3) migration is weak, the forces of selection on alleles and UʕT sets of alleles are small, differences between females in The sums in Equations 2 and 3 involve the a˜ and C, their mating preferences are not large, and recombina- which are, respectively, the selection coefficients and genetic tion rates are not very small (see appendices b and c). disequilibria defined by Barton and Turelli (1991). The results will be most accurate for the early stages of The a˜ are selection coefficients that represent the force divergence, when postzygotic isolation is still weak. of selection in favor of allele 1 at the combination of Forces acting on the preference: With an overview of loci listed in the subscripts, averaged over males and the model’s notation in hand, we now consider each of females. Thus a˜j is strength of selection favoring allele the components of Equation 2 in detail. The subsequent 1 at locus j, while a˜jk measures selection that brings section then combines the results to find the overall together allele 1 at locus j with allele 1 at locus k. Selec- dynamics of the preference. tion coefficients can be calculated for any given assump- Migration: The first term on the right side of Equation 2 tions about how selection acts; details are given in ap- represents the effect of migration on preference locus i: pendix b and in Barton and Turelli (1991). M ϭ md m. (5) The C measure the degree of disequilibrium or asso- i i ciation in the population between alleles 1 at the loci The migration rate m is defined as the proportion of listed in the subscript. In the case of two loci, Cij is individuals on the island that have immigrated that gen- equal to the standard measure of linkage disequilibrium eration from the continent, averaged over the two sexes, m (often denoted D). Generalizing to situations involving and d i is the difference between the frequencies of Reinforcement of Mating Preferences 869

P T allele 1 on the continent and on the island: n ␳ ␥i ␥j C ij ϭ mdij (Rij ϩ␶n) ϩ pqij , (9) m m m 2 ␴P␴T d i ϭ p i Ј Ϫ p i . (6) The prime (Ј) indicates values on the continent, and while the disequilibrium between a preference locus i and two male trait loci j and k is so piЈ is the frequency of the preference allele there. A superscript m denotes quantities that are evaluated at n C ijk ϭ mdijk(Rijk ϩ␶n). (10) that point in the life cycle when migration occurs. Migra- tion is a direct evolutionary force, as it causes an allele’s In these expressions, frequency to change independent of that gene’s statisti- dU ϵ ͟di, pqU ϵ ͟piqi, (11) cal associations with other genes. In contrast, all the i෈U i෈U other forces acting on the preference in this model are indirect: the preference evolves as a result of its genetic ␴P and ␴T are, respectively, the phenotypic standard associations with other traits. This fact is evident from deviations for the preference and trait on the island, the three summations that appear in Equation 2, all of and ␳ is the correlation between the preference pheno- which involve the C. type of females and the display trait phenotype of males Natural selection on the male display trait: The second in mated pairs. The indicator variable ␶n takes the value factor affecting evolution of the preference is natural 0 if natural selection on the display trait occurs before selection acting on the male trait. Equation 2 expresses migration and the value 1 if not. RU is a function of the effects of natural selection in terms of selection co- the recombination rates among the loci listed in its efficients (the a˜) and the genetic disequilibria (the C). subscript: The selection coefficients are calculated in appendix 1 Ϫ rU b assuming that the fitness function is relatively smooth RU ϵ . (12) over the trait’s range of phenotypic variation (specifi- rU cally, that its third and higher derivatives are small near Here rU is the multilocus recombination rate among the the trait mean). In that case, selection coefficients in- loci in set U, that is, the probability that there is at least volving sets of three or more male trait loci are negligi- one recombination event among the loci in that set. ble, and we need to consider only the effects of selection (For example, rU is the standard recombination rate on single loci and on pairs of loci. The selection coeffi- between loci i and j.) For our QLE approximations to cient for a single male trait locus j is hold, rU must be substantially larger than 0 (see Barton T 1 T and Turelli 1991). The function R takes on larger a˜j ϭ␥j [␤n ϩ ( ⁄2 Ϫ pj)␥j ⌫n], (7) U values as linkage between the loci in set U becomes while the selection coefficient for a pair of male trait tighter. loci j and k is Equation 9 shows that two factors contribute to the

T T covariance between a preference gene and a display trait a˜jk ϭ␥j ␥k ⌫n. (8) gene: migration, which builds up genetic correlations

Here ␤n and ⌫n are, respectively, the directional and across the whole genome, and mate choice, which gen- stabilizing selection gradients acting on the display trait erates associations specifically between preference and phenotype during natural selection, averaged over display trait loci. The magnitude of the covariance at males and females. If the display trait is expressed only the time that natural selection acts is affected by the in males, then these gradients are simply half of the order of selection and migration, because migration values measured for the males (because the values for generates linkage disequilibrium within the generation. females are 0). Methods with which to estimate these Recombination rates affect the contribution from mi- selection gradients in natural populations are discussed gration, but not from mate choice: the linkage dis- by Lande and Arnold (1983). The values of ␤n and ⌫n equilibrium at QLE caused by sexual selection is in- will change as the display evolves; the relevant values dependent of the recombination rates (Lande 1981; are those that obtain in the current generation. Kirkpatrick 1982; Barton and Turelli 1991). Equa- These and following expressions for the selection co- tion 10 implies that only migration generates genetic efficients and genetic disequilibria have been greatly associations between sets of three loci: under our as- simplified by dropping terms of order a˜ 2 or smaller. sumptions of weak additive effects for the display trait Thus these approximations are valid when the selection and preference loci, mate choice does not contribute coefficients and migration rate are much smaller than to these three-way disequilibria. 1. Explicit bookkeeping of the terms that have been The overall impact of natural selection on the display dropped is done in appendices b and c. trait is found by summing over sets of trait loci (the first The genetic disequilibria (the C) are calculated in summation on the right side of Equation 2), using the appendix c, which shows that at QLE the disequilibrium results from Equations 7 through 10. The change at between a preference locus i and a trait locus j is preference locus i is 870 M. Kirkpatrick and M. R. Servedio

n n is a measure of the force of selection on all the incompat- ͚ a˜ UC iU ϭ␤n Ϫmdi␶n(T Ϫ TЈ) UʕT ΄ ibility loci with respect to preference locus i. As with ␳ ␥P Equation 13, terms of order a˜ 3 and smaller have been ϩ ␥T ϩ ␴ 2 i mdi͚ j dj R ij Th T pqi dropped from Equation 16. (Despite the minus sign, j෈T 2 ␴P ΅ we will see that Ii is typically positive and that selection P ϩ⌫n[␶nmdiS1 Ϫ␥i pqiS 2 ϩ mdiS 3i], (13) against hybrids does indeed favor reinforcement.) where This is the last of the terms that enters into Equation 2 for the evolution of a preference gene. Next we will T T 1 T assemble these results to find the overall dynamics of S 1 ϭ ͚␥j dj ␥j Ϫ pj ϩ ͚␥k dk , (14a) j෈T ΄ ΂2 ΃ k෈T ΅ k϶j the preference. Evolution of the mean preference: A general expres- ␳ T 3 1 sion for the rate of evolution of a single preference S 2 ϭ ͚(␥j ) pj Ϫ pqj, (14b) 2␴P␴T j෈T ΂ 2΃ allele is obtained by substituting the results from the 1 last section into Equation 2. The result can be simplified ϭ ␥T ␥T Ϫ ϩ ␥T m S 3i ͚ j dj j pj R ij ͚ k dkR ijk . (14c) by substituting di for di (which appears in Equation 5), j෈T ΄ ΂2 ΃ k෈T ΅ k϶j a step that is justified because the change in preference Equation 13 has been simplified by dropping terms of allele frequencies within a generation caused by selec- 2 order a˜ 3 and smaller. tion is small enough (of order a˜ ) that it can be ne- Sexual selection on the male display trait: The third factor glected in our approximations. influencing evolution of the preference is sexual selec- To find the rate of change in the mean preference tion on the male display trait. The calculation for its at QLE, we sum across all the preference loci, effect follows that for natural selection. A difference P ≈ ⌬P ϭ ͚␥i ⌬pi Ϫm(P Ϫ PЈ) is that migration always comes before mate choice, as i෈P mating is the last event in each generation before the ϩ (␤ ϩ␤) m(P Ϫ PЈ)(T Ϫ TЈ) start of the next. Thus the summation in Equation 2 n s ΄ corresponding to sexual selection is given by Equation P T ␳ 2 2 13 with three changes: the directional sexual selection ϩ m͚␥i di͚␥j dj R ij ϩ h Ph T␴P␴T i෈P j෈T 2 ΅ gradient ␤s replaces its natural selection counterpart ␤n,

the stabilizing sexual selection gradient ⌫s replaces ⌫n, Ϫ␤n(1 Ϫ␶n)m(P Ϫ PЈ)(T Ϫ TЈ) and ␶n is set equal to 1. Selection against hybrid incompatibility: The final factor Ϫ (␶n⌫n ϩ⌫s)m(P Ϫ PЈ)S1 causing evolution of a preference gene is selection on 2 2 P P Ϫ (⌫n ϩ⌫s)(␴PhPS 2 ϩ m͚␥i diS 3i) Ϫ m͚␥i diIi, the loci that cause hybrid incompatibility. The last sum- i෈P i෈P mation on the right side of Equation 2 accounts for all (18)

forms of selection acting on F1 hybrid offspring, back- 2 2 where h P is the heritability of the preference and h T the crosses, and all other types of recombinants. heritability of the male trait. The direct effect of migra- Because for now we are not committing to the details tion appears as the first term of Equation 18. It reduces of how selection acts against hybrids, the selection coef- h the difference between the average preferences on the ficients a˜ U remain undetermined. The disequilibria Ϫ Ј h island and continent, which is (P P ). The next four C iU can, however, be calculated. Using the results from terms, involving the selection gradients ␤ and ⌫, reflect appendix c,wefind the impact of natural and sexual selection on the display h C iU ϭ mdiU(R iU ϩ␶h), (15) trait. This is our most general result for the evolutionary dynamics of the preference. where ␶ ϭ 0 if selection on hybrid incompatibility oc- h Reinforcement at equilibrium: The outcome of reinforce- curs before migration and 1 otherwise. This shows that ment depends on a balance struck between migration, the associations between preference loci and incompati- which makes preferences on the island similar to those bility loci are proportional to the migration rate, m, and on the continent, and indirect selection on the prefer- the differences in allele frequencies on the island and ence genes, caused by selection on hybrid incompatibil- continent at these loci, d . iU ity and the display trait. How large is the difference The effect on preference locus i of selection against between preferences on the island and continent when hybrid incompatibility is therefore an equilibrium is reached? To answer that question, we h ͚ a˜ UCU ϭϪmdiIi, (16) will introduce an additional assumption, define a new UʕH way to measure the preference, and calculate the equi- where librium value of a term in Equation 18.

h The new assumption is that the preference loci are Ii ϭϪ͚ a˜ U dU(R iU ϩ␶h) (17) UʕH unlinked to the loci affecting the display trait and hybrid Reinforcement of Mating Preferences 871 incompatibility. The reason for this assumption is that exactly equal to zero. Complete postzygotic isolation, it greatly simplifies Equation 18. The terms S 3i and Ii no which has sometimes been thought to be necessary for longer depend on i and can be written simply as S 3 reinforcement to evolve (e.g., Butlin 1989; Rice and and I. Further, the recombination function R ij becomes Hostert 1993), is not required. The outcome depends equal to 1. in part on quantities that one might anticipate, such as 2 To describe the preference equilibrium, it is helpful the heritability of the preference (h P) and the amount to focus on the difference between the mean prefer- of hybrid incompatibility (I). The equilibrium also de- ences on the island and the continent. In the absence pends, however, on the intensity of stabilizing selection of reinforcement of the mating preference, migration on the male trait (the ⌫’s) and on whether migration will homogenize the preference genes in the two popu- in each generation happens before or after natural se- lations and the difference will vanish. Thus any differ- lection on the display (␶n). ence between the preferences on the island and conti- Several of the terms in Equation 21 can be estimated nent at equilibrium is the result of reinforcement. from phenotypic data. One is the difference in the aver- Further, if we divide this difference by the phenotypic age value of the display trait on the island and continent, standard deviation of the preference on the island, we D T. The result also involves the S terms, however, which have a dimensionless index of reinforcement. This depend on the genetic details of the display trait (that allows one to make comparisons across different species is, recombination rates, allele frequencies, and allelic and different kinds of preferences. Doing the same for effects). Unfortunately, those quantities cannot be esti- the male display trait gives us these two new measures: mated for any species with the data now available, and there do not seem to be any obvious generalizations P Ϫ PЈ T Ϫ TЈ DP ϵ , D T ϵ . (19) that can be made about their magnitude or even their ␴P ␴T sign. The last ingredient needed is an expression for the But things simplify a lot under some circumstances. overall selection gradients acting on the male display When migration and stabilizing selection are sufficiently trait at equilibrium. When the display trait is at equilib- weak, and when migration occurs before natural selec- rium, the force of directional selection is offset by gene tion acts on the display trait, then flow. When these two forces are in balance, it is straight- 1 2 Dˆ P Ϸ ⁄2␳h P(1 ϩ I)Dˆ T. (22) forward to show that This simple result tells us how much reinforcement mDˆ ␤ ϩ␤ ≈ T causes preferences on the island to diverge from those s n 2 , (20) ␴Th T on the continent. That difference is proportional to Dˆ T, 2 the difference at equilibrium in the display trait means where h T is the heritability of the display on the island, that is, its additive genetic variance divided by its pheno- on the island and continent. The constant of propor- typic variance. (This result is only an approximation tionality depends on only three quantities: the pheno- because it neglects the force of indirect selection on typic correlation between the male display and the fe- 2 male preference among mated pairs ␳, the heritability T. But that force is O(a˜ ), however, and therefore can 2 of the preference h P, and the intensity of hybrid incom- be neglected for our present needs.) 1 We finally arrive at an expression for the degree of patibility I. The factor of ⁄2 results from our assumption reinforcement in the mating preference at equilibrium. that only one sex (females) choose their mates. Many Set the left side of Equation 18 to 0, use Equations 19 other variables, such as the type of selection acting and 20, and assume that the migration rate is not zero. against hybrids, number of genes affecting the display Then we find trait and preference, and the migration rate, apparently do not affect the outcome. 1 2 Dˆ p ϭ [ ⁄2␳h PDˆ T Ϫ (⌫n ϩ⌫s) Figure 1 illustrates how divergence of the island pref- erence changes with Dˆ . Although reinforcement causes ϫ␴h 2S /m]/[1 Ϫ 2mDˆ 2 /h 2 ϩ (1 Ϫ␶)␤ ␴ Dˆ T P P 2 T T n n T T the island preferences to differ from those on the conti-

ϩ (␶n⌫n ϩ⌫s)S1 ϩ (⌫n ϩ⌫s)S 3 Ϫ I]. nent at equilibrium, the difference is not necessarily large. With ␳ϭ0.25, h 2 ϭ 0.4, and I ϭ 0.2, for example, (21) P a difference of three phenotypic standard deviations This is our most general result for the divergence of the between the display trait means on the island and conti- island and continent mating preferences at equilibrium. nent will cause the mean preferences to differ by only An immediate message that Equation 21 conveys is 18% of a phenotypic standard deviation. that divergence of the island preference happens under Equations 21 and 22 are our main results. Before a broad range of conditions, despite the presence of going further, we recapitulate the major simplifying as- gene flow and regardless of the mechanism of postzy- sumptions and then draw several implications. The fun- gotic isolation. This conclusion follows because only a damental approximation that makes the analysis possi- very unusual combination of parameters will make Dˆ P ble is that selection coefficients (the a˜) are small. This 872 M. Kirkpatrick and M. R. Servedio

bilizing selection on the display trait are weak. Then the amount of reinforcement in the preference is directly proportional to the difference between the average male trait on the island and continent (Equation 22). Further, when postzygotic isolation is negligible (I ≈ 0), the constant of proportionality depends only on things that can in principle be estimated from phenotypic data: the phenotypic correlation between the display in males and the preference in females among mated pairs, ␳, 2 and the heritability of the preference, hP. Up to now, no assumption has been made about what mechanism causes hybrid incompatibility, that is, post- zygotic isolation. But several interesting questions are still unanswered, such as: How much is reinforcement Figure 1.—The difference between the average mating preference on the island and continent as a function of the strengthened by selection against hybrids? How does difference between the average display traits in those two the genetic architecture of postzygotic isolation affect populations, from Equation 22. Both differences are measured reinforcement? We now introduce a specific model for in units of phenotypic standard deviations. I is the intensity postzygotic isolation to address those issues. of selection against hybrid incompatibility, as defined by Equa- 2 tion 17. Other parameter values are h P ϭ 0.4 and ␳ϭ0.25. HYBRID INCOMPATIBILITY FROM EPISTASIS situation holds when fitness effects of individual loci Dobzhansky (1937) and Muller (1942) hypothe- and sets of loci are small. It also implies that the impact sized that postzygotic isolation between species could of migration is weak, meaning that the migration rate be caused by epistatic interactions between sets of loci. and/or genetic differences between the island and con- They pointed out that populations isolated from the tinent are small. Quantitatively, each of the terms follow- same ancestral stock could undergo substitutions at dif- ing the 1 in the denominator of Equation 21 must be ferent loci. If the new alleles at these loci cause problems small. In biological terms, the upshot is that our results in development or gametogenesis when they are com- are most accurate when prezygotic and postzygotic isola- bined in the same individual, these genes will contribute tion between the island and continent are weak. Regard- to postzygotic isolation when the populations come back ing recombination, the basic model allows any pattern into contact. There is now abundant evidence that hy- of linkage between the loci (as long as the recombina- brid incompatibility in some groups of animals is indeed tion rates are not very small). But to find an expression caused by epistatic interactions (Coyne 1992; Wu and for the equilibrium of the average preference, we as- Palopoli 1994; Turelli and Orr 1995). Further, some sumed that preference loci are unlinked to other genes cases involve interactions between sets of more than two in the model. If there is linkage, one can solve for the loci (Muller 1942; Wu and Palopoli 1994). Motivated equilibrium if values for the recombination rates and by those data, we now look at a specific model in which the allelic effects are given. hybrid incompatibility is caused by epistasis acting on Two interesting conclusions flow from these results. sets of interacting loci. We commit to specific assump- First, a difference between the average display on the tions about how epistasis operates, and so the results of continent and island, no matter what causes that differ- this section are not general to all forms of epistatic ence, is sufficient to drive reinforcement of the prefer- incompatibility. ence. But if the island and continent displays are the We allow there to be any number of epistatically inter- same (Dˆ T ϭ 0), then females cannot distinguish the two acting sets of loci, and within each set there may be any types of males, and reinforcement is doomed. Addi- number of loci. For convenience, we designate the allele tional postzygotic incompatibility is not needed: Equa- that is most common on the island as allele 1. When tions 21 and 22 show that the average preference on all of the alleles carried by an individual in one of these the continent and island can differ (that is, Dˆ P ϶ 0) epistatic sets are of the same type, either 0 or 1, then even when there is no hybrid incompatibility (that is, development and meiosis proceed normally. Within a I ϭ 0). There is an intuitive explanation for this result. set consisting of four loci, for example, these high fitness If the combined forces of natural and sexual selection genotypes are {0, 0, 0, 0} and {1, 1, 1, 1}. But if an keep males on the island distinct from those on the individual carries a mixture of alleles of both kinds, then continent, then females that mate with immigrant males fitness is reduced. We assume that fitness is reduced by will have offspring that are poorly adapted to the local the same amount for all mixtures of alleles in a given ecological conditions, that will have difficulty finding a set (e.g., genotypes {0, 0, 1, 1} and {0, 1, 0, 0} have the mate, or both. same fitness), but different sets can have different fitness A second conclusion applies when migration and sta- effects. We number the epistatic sets to refer to them. Reinforcement of Mating Preferences 873

The number of loci in set Hi is ni, and the loss of fitness reinforcement is always strengthened whenever linkage caused by a mixture of alleles in that set is si. As in the between the preference locus and the epistatic loci is rest of the article, the set of all loci in the genome tightened. Intuitively, the reason is that tighter linkage involved in hybrid incompatibility is referred to as H. keeps the island-specific preference allele more highly Having laid out assumptions for how incompatibility correlated with high fitness genotypes (Felsenstein works, we can now calculate its impact on the mating 1981). The effect of the recombination rate between preference. We saw earlier that the rate of evolution of the epistatic loci depends on the linkage map. If the the preference depends on Ii (Equations 17 and 18). preference locus does not lie between the two epistatic That quantity measures the impact on preference locus loci, one can show that decreased recombination be- i of selection on all the loci involved in hybrid incompati- tween the epistatic loci decreases reinforcement. Intu- bility. The selection coefficients that appear in Ii are itively, lower recombination means less opportunity for calculated in appendix b for our model of epistasis. For selection against recombinants. On the other hand, if any subset U of loci in epistatic set Hj, the selection the preference locus lies between a pair of epistatic loci coefficient is on a chromosome, decreasing any of the recombination rates increases reinforcement. a h ϭ s k(j,U), (23) ˜ U j How does the number of loci in each epistatic set where affect reinforcement of the preference? That question can be answered when the loci within each epistatic set U k(j,U) ϭ ͟ pk ϩ (Ϫ1)| | ͟ qk (24) are unlinked to each other and to the preference loci. k෈Hj k෈Hj ΄ kÓU kÓU ΅ For any set U of unlinked loci, the recombination func- UϪ1 and U denotes the number of loci in set U. (The tion RU then equals 1/(2| | Ϫ 1). Using that fact and | | Equation 26 gives the overall impact of epistatic incom- products, which are over all loci in Hj that are not also patibility, in U, are defined to equal 1 when Hj ϭ U.) Substituting into Equation 17, the combined effect of selection on I ϭ ͚sj f(nj ), (28) all the epistatic loci is j ≈ where Ii Ϫ͚sj ͚ k(j,U)dU(R iU ϩ␶h). (25) j UʕHj n n (Ϫ1)(iϩ1) 1 f(n) ϭ Ϫ . (29) The effects of the incompatibility genes simplify when ͚ i n ΄ ϭ ΂ i ΃ 2 Ϫ 1 ΅ 2 Ϫ 1 the continent and island are near fixation for opposite i 1 alleles at these loci, which requires that migration is This expression for I can be entered directly into the weak relative to epistatic selection (m Ӷ si). In that case, earlier results for the equilibrium of the mean prefer- U Ϫ1)| |, and k(j,U) is approximately equal to 1 ence (Equations 21 and 22), which also assume the)ف dU is if U is not equal to H ,to2ifU equals H and U is preference loci are unlinked to the incompatibility loci. j j | | even, and to 0 otherwise. Then The results show how the degree of preference rein- forcement is amplified by selection against hybrid re- U I ≈ Ϫ s R ϩ (Ϫ1)| |R . (26) i ͚ j ΄ i Hj ͚ iU΅ combinants. Equation 28 shows that the overall effect is j UʕHj determined by a simple sum of the selection coefficients Two useful observations can be made here. The tim- acting on each epistatic set, weighted by the factor f(), ing of migration relative to epistatic selection does not which is determined by the number of loci in each set. affect reinforcement: ␶h has disappeared. This is because 4 This factor increases from a value of f ϭ ⁄3, when the when migration introduces only pure continental geno- set consists of a pair of loci, to a value of f ϭ 2, when types to the island, maladapted recombinant genotypes there are three loci, to a value of 2.8 with five loci. are produced and selected against only after recombina- The rise is slow, approximately logarithmic, as shown tion. Second, we can assess the effects of recombination in Figure 2. This trend reflects the outcome of two on reinforcement. Consider what happens when hybrid conflicting factors. As the number of loci increases, incompatibility depends on pairs of epistatically inter- there is a larger number of recombinant genotypes for acting loci. Focusing on a single preference locus i and selection to act against, which enhances reinforcement. a single epistatic pair j and k, with the help of Equation On the other hand, the high fitness genotypes are bro- 12, Equation 26 becomes ken apart rapidly when there are many freely recombin- ing loci. That makes the high fitness genotypes more ≈ 1 Ϫ rij 1 Ϫ rik 2(1 Ϫ rijk) Ii s ϩ Ϫ . (27) rare, decreasing the effectiveness of the indirect selec- Ά r r r · ij ik ijk tion that favors reinforcement. The net effect is that This quantity is always positive and is proportional to although reinforcement is enhanced when the number the selection coefficient s against epistatic recombi- of epistatically interacting factors increases, the increase nants. Thus stronger selection against hybrids increases is not dramatic. In sum, preference reinforcement is reinforcement, as expected. Further, one can show that affected by the number of loci that interact to cause 874 M. Kirkpatrick and M. R. Servedio

the frequency of allele P1 on the continent is denoted pЈP. On the island, we assume that the overall impact of

natural and sexual selection favors the T1 allele and is sufficiently strong to keep the display trait locus poly- morphic in the face of migration. At the epistatic loci, we imagine that selection is sufficiently strong that the

alleles A1 and B1 are near fixation on the island (m Ӷ

sE), but migration from the continent continually intro- duces the alternative alleles. Last, for consistency with the analytic results we assume that all the parameters

(m, sT, sE, ␣0, and ␣1) are much smaller than 1. Given those assumptions, the first job is to relate the parameters just described to those of the general model. Figure 2.—The relative impact on reinforcement of the The simple expression for the preference equilibrium number of epistatically interacting incompatibility loci under given by Equation 22 applies here. Without any loss of free recombination, from Equation 29. generality, we can measure the means of the preference and display trait in terms of allele frequencies, and so

we have P Ϫ PЈϭp P Ϫ pЈP and T Ϫ TЈϭpT. Because hybrid incompatibility, as well as the strength of selec- allele frequencies are free of nonheritable environmen- tion against hybrid recombinants. 2 tal effects, the preference heritability is hP ϭ 1 and the phenotypic variance is equal to the genetic variance for 2 2 A FOUR-LOCUS MODEL both characters: ␴P ϭ pqP and ␴T ϭ pqT. A calculation shows that for weak preferences the correlation among The genetically simplest case to which these results ≈ √ mated pairs is ␳ (␣0 ϩ␣1) pqp pqT. Last, epistatic selec- apply is one where the preference and trait are each tion involves only a pair of loci, and from Equation 27 controlled by one locus and a single pair of loci interact we find that the intensity of hybrid incompatibility is to cause epistatic incompatibility. This section uses the 4 I ϭ ⁄3. results of the last to study evolution of the preference Reinforcement at equilibrium: How much will the in this simple situation. We have four motivations. First, preference on the island diverge from the continent? we show how the results developed above apply to small The answer, found using Equation 22 and the results numbers of loci as well as to more genetically complex above, is situations. Second, we develop some new biological re- sults. For example, we will see that there are cases in pˆ P Ϫ pЈP 1 4 ϭ ␣ ϩ␣ √ˆ Ϫ ˆ √ ( 0 1) pqˆP΂1 sE΃p T. (30) which reinforcement is doomed to failure. Third, we pˆqˆP 2 3 check the accuracy of the analytic results using exact simulations. Fourth, simulations are used to see if the To complete the picture, we need an expression for analytic results for the haploid model generalize to dip- the display trait equilibrium, pˆT. One can show at the loidy. migration-selection equilibrium that its value is 1 Ϫ 2m/

Suppose that females carrying preference allele P0 (sT ϩ␣1pP Ϫ␣0qP) (to first-order accuracy in the a˜). prefer to mate with males carrying display trait allele Substituting that into Equation 30 leads to a cubic equa- ˆ T0, while P1 females prefer T1 males. Given a choice tion in pP that can be easily solved using, for example, between one of each, a Pi female will mate with her Mathematica (Wolfram 1996). The solution is long, preferred male 1 ϩ␣i times more often than the other so we do not show it here, but we illustrate its most male. Thus ␣i ϭ 0 implies random choice, while a large interesting features with examples shown in Figure 3 value of ␣i implies a strong preference. (More details and Table 1. about this mate choice model can be found in Kirkpat- As seen in Figure 3, there can be no divergence of rick 1982, in which the preference strength is parame- the island preference (pˆ P Ϫ pЈP ϭ 0) if there is no ge- terized slightly differently.) On the island, T1 males have netic variation for the preference among migrants from a viability advantage of sT over the T0 males, and we the continent (pЈP ϭ 0 or 1). When the migrants are assume the display trait is not expressed in females. A polymorphic for the preference, however, genetic varia- pair of loci denoted A and B cause epistatic incompati- tion for the preference is also maintained on the island. bility; the recombinant genotypes A0 B1 and A1 B0 have That allows disequilibrium to build up between the pref- fitness (1 Ϫ sE) relative to A0 B0 and A1 B1 individuals. erence locus and the other loci, causing reinforcement. For simplicity, we assume all four loci are unlinked. (One can show that reinforcement also occurs if genetic The continental population is fixed at the display trait variation for the preference is maintained on the island and incompatibility loci (that is, for alleles A0, B0, and by mutation.) In short, we conclude that for reinforce-

T0). The preference can be polymorphic, however, and ment to succeed in the four-locus model with weak Reinforcement of Mating Preferences 875

smaller. Table 1 shows that the relative error is propor- tional to the size of the parameters and becomes very small (at most a few percent) when all parameter values are much smaller than 1. These two properties strongly support our analytic results for the amount of reinforce- ment in the preference at equilibrium. The simulations suggest that the equilibria for the four-locus model are globally stable. For any given set of parameters, populations evolve to the same equilibrium regardless of the initial conditions. (That conclusion can be supported analytically: we derived an approxima- tion for the leading eigenvalue using Equation 18 and found that the equilibrium is always locally stable.) Thus it seems that history does not affect the outcome: it does Figure 3.—Difference between the frequencies of prefer- not matter whether the island is isolated for a period ence allele 1 on the island and continent as a function of its frequency on the continent for the four-locus model. ᭹ and before it starts to receive migrants from the continent. ᭡, results from the analytic approximation based on Equation Last, we built a diploid simulation model to see if the 30; ᭺ and ᭝, exact results from simulations. ᭺ and ᭹, sT ϭ analytic results, which assume haploidy, might apply 0.15, sE ϭ 0.1, ␣0 ϭ␣1 ϭ 0.1, m ϭ 0.01. ᭝ and ᭡, sT ϭ 0.07, there. Diploidy raises the issue of dominance. Rather ϭ ␣ ϭ␣ ϭ ϭ sE 0.05, 0 1 0.05, m 0.005. than explore all the possible effects of dominance, we made assumptions intended to make the diploid simula- selection and migration, there must be standing genetic tions as comparable to the haploid model as possible variation in the preference. (and that avoid introducing additional parameters). Ap- Simulations: We ran exact simulations of the four- propriate parameterizations for selection on the male locus model for three reasons: to check the accuracy of display locus are provided by Gomulkiewicz and Has- the analytic approximations, to determine the stability tings (1990). We took the relative viabilities of the T0 of the equilibria, and to see how results from the haploid T0, T0 T1 and T1 T1 genotypes, respectively, as 1/(1 ϩ model might apply to diploids. sT), 1, and 1 ϩ sT. Matings occurred at the frequencies The simulations show that the analytic approxima- shown in Table 2. Regarding dominance and epistatic tions are quite accurate. We measured the relative error incompatibility, several studies of hybrid fitness imply of the approximations as (pˆP Ϫ p˜P)/(p˜P Ϫ pЈP), where pˆP that hybrid genotypes that are further from either of is the analytic approximation based on Equation 30 and the pure parental genotypes have lower fitness (Coyne p˜P is the exact result from simulation. This is a sensitive 1985; Wu and Davis 1993; Davis et al. 1994; Hollocher test of our analytic results for the effects of reinforce- and Wu 1996). We therefore assigned nonrecombinants ment: it measures the error in the approximation for (e.g., A0 A0 B0 B0) fitness of 1 ϩ sE, genotypes with one the equilibrium (which is pˆP Ϫ p˜P) relative to the amount incompatible allele (e.g., A0 A0 B0 B1) fitness of 1, and of reinforcement in the preference (which is p˜P Ϫ pЈP). genotypes with two pairs of incompatible alleles (e.g., For the examples shown in Figure 3, the error is always A0 A1 B0 B1 and A0 A0 B1 B1) fitness of 1/(1 ϩ sE). Ͻ8.5%. Further, it decreases as the parameters become We quantified the discrepancy between the analytic

TABLE 1 Accuracy of the analytic approximation for the preference equilibrium in the four-locus model

p˜P, Error, p˜P, Error, sT sE ␣0 ϭ␣1 mpPЈ pˆP haploid haploid (%) diploid diploid (%) 0.1 0.1 0.05 0.01 0.25 0.25799 0.25785 1.7 0.26212 Ϫ34 0.05 0.05 0.025 0.005 0.25 0.25371 0.25369 0.64 0.25471 Ϫ21 0.01 0.01 0.005 0.001 0.25 0.25070 0.25070 0.10 0.25073 Ϫ4.4 0.15 0.1 0.08 0.01 0.5 0.51971 0.51878 4.9 0.52904 Ϫ32 0.075 0.05 0.04 0.005 0.5 0.50926 0.50904 2.5 0.51164 Ϫ20 0.015 0.01 0.008 0.001 0.5 0.50176 0.50175 0.5 0.50185 Ϫ5.0

The analytic approximation for the equilibrium of preference allele 1 (ϭ pˆP) was calculated from Equation 30 as explained in the text, and its exact equilibrium (ϭ p˜P) was found by computer iteration of the exact equations for both the haploid and diploid models. The relative error is calculated as (pˆP Ϫ p˜P)/(p˜P Ϫ p PЈ). Two sets of parameter values are shown; the relative error within each set decreases as the parameters become smaller. 876 M. Kirkpatrick and M. R. Servedio

TABLE 2 Relative mating preferences in the diploid four-locus model

Male genotype Female genotype T0 T0 T0 T1 T1 T1

P0 P0 1 ϩ␣0 1 1

z 0 z 0 z 0(1 ϩ␣0)

P0 P1 1 1 1

(z1/2) (1/(1 ϩ␣0) ϩ (1 ϩ␣1)) z 1 (z 1/2) ((1 ϩ␣0) ϩ 1/(1 ϩ␣1))

P1 P1 1 1 1 ϩ␣1

z 2(1 ϩ␣1) z 2 z 2

The frequency for each type of mating is found by multiplying the frequencies of the male and female genotypes (after natural selection) by the corresponding element in this table. The factors z 0, z 1, and z 2 are defined so that the frequencies of all matings involving a given female genotype sum to that genotype’s frequency among mating females. haploid approximations and the exact result from simu- process could develop in multiple preference-trait pairs, lations of the diploid model with the same measure that thus amplifying the overall contribution of reinforce- we used for the haploid simulations. Table 1 shows that ment to prezygotic isolation. The strong postzygotic iso- the discrepancy for diploids is roughly an order of mag- lation emphasized as necessary to reinforcement in ear- nitude larger than it is for haploids. That is perhaps not lier theoretical studies (e.g., Sved 1981; Spencer et al. surprising because the analytic results are an approxima- 1986; Liou and Price 1994) is apparently not necessary tion for a haploid system and not a diploid one. Never- for the process to begin. Our results do not, however, theless, the discrepancy with the diploid simulations show if or how reinforcement of preferences might fin- declines rapidly as the parameters become smaller. We ish the process of speciation by reducing genetic intro- conclude that our analytic results may give a reasonable gression to nil. quantitative guide for diploids when there is no domi- Hybrid incompatibility in some groups of animals is nance and when parameter values are very small. caused by epistatic interactions between loci that have diverged in the two populations (Dobzhansky 1937; Muller 1942; Coyne 1992). When postzygotic isolation DISCUSSION is caused by a pair of epistatically interacting loci, the These results show that reinforcement of a mating model shows that reinforcement is enhanced by tighter preference is expected quite generally when one popu- linkage between those loci and the preference genes. lation receives migrants from another. We expect to see Reinforcement is also affected by the number of epistat- females prefer males from their own population when ically interacting loci. When several sets of epistatic loci only three conditions are met: there is standing genetic contribute to postzygotic isolation, their effects on rein- variation for a female mating preference, mate choice forcement are additive. But because hybrid incompati- produces a phenotypic correlation in mated pairs be- bility of this sort is not necessary for reinforcement, tween the preference in females and some trait in males, there is apparently no paradox in the experimental re- and the average value of that trait differs between the sults that suggest reinforcement that has occurred be- populations. These criteria make sense: reinforcement tween some species pairs that show no postzygotic isola- requires that the preference can evolve, that the prefer- tion in the laboratory (Coyne and Orr 1989b). ence affects mate choice, and that males from different Earlier models of reinforcement differ in one or more populations can be distinguished. Hybrid incompatibil- critical ways from ours. First, rather than studying a ity will amplify reinforcement but, perhaps surprisingly, mating preference acting on a display trait, most models is not necessary. focus on reinforcement via assortative mating. For ex- Postzygotic isolation is weak in some groups of ani- ample, reinforcement has been modeled via the spread mals, such as birds (Grant and Grant 1997). Our re- of genes that decrease migration between populations sults show that reinforcement of preferences can con- (Maynard Smith 1966; Balkau and Feldman 1973), tribute to speciation in these cases, before strong hybrid that cause a change in flowering time (Crosby 1970; unfitness develops. While the model here considers only Dickinson and Antonovics 1973; Caisse and Anto- a single preference and display trait, clearly the same novics 1978), and that intensify mating discrimination Reinforcement of Mating Preferences 877 against individuals carrying a different genotype (San- force of this selection is transmitted to the preference via derson 1989; Kelly and Noor 1996). While all of these the genetic correlations produced by migration between mechanisms are biologically plausible, we do not yet pairs of display trait loci and individual preference loci. have a unified theoretical picture of which lead most The size and direction of this effect depend on genetic easily to reinforcement. Felsenstein (1981), however, details, such as recombination rates, allele frequencies, argues persuasively that certain forms of assortative mat- and the size of allelic effects (seen in Equations 14). ing are much more favorable than others. Second, most One finding of interest is that the maintenance of earlier models assume there is two-way introgression genetic variation for the preference may influence the between the incipient species rather than the one-way outcome of reinforcement. Standing genetic variation gene flow studied here. Examples of one-way introgres- for mating preferences has been found in several species sion and two-way introgression have both been found in (Bakker and Pomiankowski 1995). Results from our nature. Situations involving two-way introgression may general model show that preferences on an island will often make reinforcement easier, because preference typically diverge from the continent in these cases. But alleles favored in one population can migrate out of in other species, preference evolution may have to wait it and return several generations later (Servedio and for the appearance of rare mutations. The four-locus Kirkpatrick 1997). In this regard, the results from the model shows there are some cases where the forces present model may give a conservative picture of the favoring reinforcement are too weak to cause that to power of reinforcement. Third, many earlier models happen. Are there other cases where reinforcement can study reinforcement in a clinal hybrid zone, rather than cause a new preference mutant to spread? There seem in an island population (e.g., Caisse and Antonovics to be at least two possibilities. One is if there are multiple 1978; Sanderson 1989; Liou and Price 1994). A ques- loci affecting the male trait and/or hybrid incompatibil- tion that our results leave unanswered is whether a pref- ity. These additional loci would favor the new prefer- erence that has been reinforced can spread out beyond ence by producing additional indirect selection on it. the area of hybridization and establish a new prezygotic Second, genes with large fitness effects could produce isolating mechanism throughout a species’ range (see outcomes quite different than our analytic results, which Barton and Hewitt 1985; Sanderson 1989). assume weak effects. In fact, we have found such exam- Implications: How can reinforcement of the island ples in simulations of the four-locus model, where a new preference happen without additional postzygotic isola- preference will spread (for example with ␣0 ϩ␣1 ϭ 8 tion caused, for example, by epistatic selection against and sE ϭ 0.5). To sum up, we generally expect to see hybrids? That conclusion seems at odds with the com- preference reinforcement if some force maintains mon intuition that hybrid incompatibility is the motor standing genetic variation for the preference. If prefer- that drives reinforcement. In fact, a type of “good genes” ence evolution depends on the spread of occasional new effect caused by the display trait alone is sufficient. mutations, however, then reinforcement will succeed When the display trait on the island and continent differ under some situations but not others. despite migration, then either natural selection or sex- Connections between theory and data: Results from ual selection, or both, give island-like males an overall these models make the qualitative point that reinforce- fitness advantage. Females that mate continent-like ment of mating preferences can occur under plausible males have offspring that will suffer from lower survival conditions. But they can also be used to forge two kinds (if natural selection causes the difference) or lower mat- of quantitative links to empirical studies. ing success (if sexual selection). On the other hand, First, under some conditions the amount of reinforce- preference genes that associate themselves with island- ment can be predicted entirely in terms of phenotypi- adapted genotypes spread, causing reinforcement. An cally measurable quantities (see Equation 22). The main important difference with other good genes scenarios provisos are that the preference is not under direct is that migration contributes to both the genetic vari- selection, that genes affecting the preference are not ance for total fitness and the genetic correlations be- linked to other genes involved in reinforcement, and tween the preference and high fitness genotypes, ampli- that both migration and stabilizing selection on the fying the power of indirect selection on the preference. display trait are weak. In that situation, the difference Additional incompatibility genes of the sort envisioned between the mean preference on the island and conti- by Dobzhansky (1940) are not strictly necessary for nent is directly proportional to the difference between preference reinforcement. Ecological or sexual selec- the mean display trait values in the two populations. tion acting against hybrids is sufficient, as suggested by This simple relation holds no matter how different the Coyne and Orr (1989b). environments are on the continent and the island. Fur- Why does stabilizing selection on the display trait ther, we expect to see this relation even among pairs affect the outcome of reinforcement (shown by Equa- of populations that show no effects of hybrid incompati- tion 21)? As with any trait that has additive genetic bility in the lab. variation, stabilizing selection on the display generates A similar relation between the display and preference epistatic fitness interactions between pairs of loci. The is also seen in several simple “null models” of sexual 878 M. Kirkpatrick and M. R. Servedio selection (Lande 1981; Kirkpatrick 1982). If isolated causing the population to evolve in a “runaway process” populations experience exactly the same natural selec- (Lande 1981; Iwasa and Pomiankowski 1995). By anal- tion pressures, then the models predict a linear relation ogy, we expect that the equilibria described here may between the mean display and mean preference in the sometimes be unstable. different populations. That relationship disappears, In our model of an island population the rate of however, if the different populations experience differ- introgression is controlled by the migration rate, which ent natural selection pressures. By contrast, the rein- is a fixed quantity. When sympatric species hybridize, forcement model predicts a linear relation between the however, the hybridization rate is influenced by the means even when natural selection pressures vary be- mating preference and display trait, as well as the ecol- tween populations that receive migrants from the same ogy of the species. This then generates a process in source. which hybridization is decreased as reinforcement de- These observations suggest that there may be situa- velops. Our model does not account for this feedback. tions in which the predictions of this model could be It is possible that our results for the preference equilib- tested. Consider a group of semi-isolated populations rium would not change much if it did, at least for low (“islands”) that receive migrants from the same source hybridization rates. The logic here is that our results population (“continent”) and that differ in the average show the amount of reinforcement is approximately value of a display trait. The average mating preference independent of the rate of migration, and hence the in each population could be quantified, for example, rate of hybridization, when migration is weak (Equation by using binary choice trials that measure how much 22). The explanation is that gene flow has two opposing more frequently island females respond to male displays effects on the preference that approximately cancel. Its from their own populations than to displays of continen- direct effect is to make the island preference more simi- tal males. The model predicts a linear relation between lar to that of the continent. But increasing migration this measure of preference and the divergence in the also strengthens the genetic correlations between genes male trait. Further, if postzygotic isolation is weak, then that are directly selected and the preference genes, fa- the slope of the relation is determined by phenotypically voring reinforcement. measurable quantities: the phenotypic correlation be- The haploid system studied here captures many but tween the preference and display trait in mated pairs, not all of the genetic effects found in a diploid popula- and the heritability of the preference. tion. Most obvious is that a haploid model cannot ac- A second use of the models is to identify genetic count for dominance, which is ubiquitous in hybrid details that affect reinforcement and might therefore crosses and figures prominently in Dobzhansky’s affect the course of speciation. We find there are a (1937) and Muller’s (1942) ideas about the evolution number of such features, including linkage relations of hybrid incompatibilities. Chromosomal divergence and the frequencies of male display alleles and their between populations often causes hybrids to be under- phenotypic effects. One intriguing result is that rein- dominant for fitness (White 1973). There is indirect forcement can be affected by the number of loci that experimental evidence that epistatic incompatibilities interact epistatically to cause hybrid incompatibility, in- in many groups of animals involve loci with strong domi- dependent of the absolute strength of hybrid break- nance effects (Turelli and Orr 1995), and dominance down. In fact, genetic analyses are just now achieving affects the probability that these loci will diverge in the enough resolution to determine the number of inter- first place (Wagner et al. 1994). Further, we cannot acting loci involved in hybrid breakdown (Wu and Palo- directly relate the selection coefficients for hybrid in- poli 1994). compatibility that appear in our model to experimental

Unanswered questions: Our results leave many ques- data on the viability and fertility of offspring from F1 and tions about reinforcement still unresolved. One strength backcross matings. A second limitation of our haploid of the model—its generality—is also a weakness. The model is that it cannot study the effects of sex linkage. equilibrium for the display trait is viewed, in effect, as X chromosomes often play an important role in postzy- a given quantity. But of course the display coevolves gotic isolation. They contribute strongly to inviability with the preference as a result of sexual selection. What and/or infertility of F1 hybrid offspring (Haldane 1922; difference might that make to our conclusions? Coyne and Orr 1989a; Coyne 1992; Turelli and Orr Adding assumptions about how females choose mates 1995), and sex linkage can affect the outcome of rein- would let us find the equilibrium for the male trait, forcement (Kelly and Noor 1996). which in turn could be inserted into our formulas for A key assumption made throughout this article is that the preference equilibrium. Further, we could then de- the preference genes are free of direct selection; that termine the stability of the equilibria, which cannot is, they do not affect the survival or immediate fecundity be decided until all the dynamic details of the model of their bearers. But what if the preference genes are (including how sexual selection acts on the display) directly selected? One expects that the preference equi- are specified. Models of sexual selection in a single librium would be changed to a point that balances the population show that the equilibria can be unstable, forces of direct selection on the preference with the Reinforcement of Mating Preferences 879 forces of migration and indirect selection that already plant populations. IX. Evolution of reproductive isolation in clinal populations. Heredity 40: 371–384. appear in the model. In short, a compromise would be Coyne, J. A., 1985 The genetic basis of Haldane’s rule. Nature 314: struck. Just where the compromise is reached depends 736–738. on what direct selection on the preference genes favors Coyne, J. A., 1992 Genetics and speciation. Nature 355: 511–515. Coyne, J. A., and H. A. Orr, 1989a Two rules of speciation, pp. and how strongly it does so. Qualitatively, if selection 180–207 in Speciation and Its Consequences, edited by D. Otte and favors the same preference on the island and the conti- J. A. Endler. Sinauer Associates, Sunderland, MA. nent, then it will be a conservative force that inhibits Coyne, J. A., and H. A. Orr, 1989b Patterns of speciation in Dro- sophila. Evolution 43: 362–381. divergence of the island population. On the other hand, Coyne, J. A., and H. A. Orr, 1997 “Patterns of speciation in Drosoph- if conditions on the island favor preferences different ila” revisited. Evolution 51: 295–303. from those on the continent, then prezygotic isolation Crosby, J. L., 1970 The evolution of genetic discontinuity: computer models of the selection of barriers to interbreeding between could be greatly accelerated. species. Heredity 25: 253–297. This returns us to the question posed at the opening Davis, A. W., E. G. Noonburg, and C.-I. Wu, 1994 Evidence for of the article. Direct selection on the pleiotropic effects complex genic interactions between conspecific chromosomes underlying hybrid female sterility in the Drosophila simulans clade. of preference genes is in fact a leading alternative hy- Genetics 137: 191–199. pothesis to reinforcement for how mating preferences Dickinson, H., and J. Antonovics, 1973 Theoretical considerations diverge during speciation. Theoretical arguments that of sympatric divergence. Am. Nat. 107: 256–274. Dobzhansky, T., 1937 Genetics of the Evolutionary Process. Columbia direct selection may often be stronger than the forces University Press, New York. of indirect selection acting on preferences within spe- Dobzhansky, T., 1940 Speciation as a stage in evolutionary diver- cies have been made (Kirkpatrick 1987, 1996; Kirk- gence. Am. Nat. 74: 312–321. Felsenstein, J., 1981 Skepticism towards Santa Rosalia, or Why are patrick and Barton 1997). Many artificial selection there so few kinds of animals? Evolution 35: 124–138. experiments have produced some prezygotic isolation Gerhardt, H. C., 1994 Reproductive character displacement on as a by-product (Rice and Hostert 1993), suggesting female mate choice in the gray tree frog Hyla chrysoscelis. Anim. Behav. 47: 959–969. that preference genes do indeed produce pleiotropic Gomulkiewicz, R., and A. Hastings, 1990 Ploidy and evolution by effects on which selection can act. A third line of evi- sexual selection: a comparison of haploid and diploid female dence for direct selection comes from a number of choice models near fixation equilibria. Evolution 45: 757–770. Grant, B. R., and P. R. Grant, 1993 Evolution of Darwin’s finches comparative and functional analyses of preferences that caused by a rare climatic event. Proc. R. Soc. London B Biol. Sci. suggest that natural selection has played a role in their 251: 111–117. origin (Ryan 1990). From a theoretical perspective, Grant, P. R., and B. R. Grant, 1997 Genetics and the origin of bird species. Proc. Natl. Acad. Sci. USA 94: 7768–7775. both direct selection on preference genes and indirect Haldane, J. B. S., 1922 Sex-ratio and unisexual sterility in hybrid selection of preferences via reinforcement are viable animals. J. Genet. 12: 101–109. hypotheses for how prezygotic isolation originates. What Hollocher, H., and C.-I. Wu, 1996 The genetics of reproductive isolation in the Drosophila simulans clade: X vs. autosomal effects is still unresolved is the relative contributions that they and male vs. female effects. Genetics 143: 1243–1255. each make to speciation. Howard, D. J., 1993 Reinforcement: Origin, dynamics, and fate of an evolutionary hypothesis, pp. 46–69 in Hybrid Zones and the This work benefited tremendously from discussions with M. Asmus- Evolutionary Process, edited by R. G. Harrison. Oxford University sen, T. Bataillon, N. Barton, J. Britton-Davidian, B. Charlesworth, J. Press, Oxford. Coyne, P. David, G. Ganem, T. Lenormand, A. P. Møller, I. Olivieri, Iwasa, Y., and A. Pomiankowski, 1995 Continual change in mating H. A. Orr, S. Otto, T. Price, and M. Ryan. M.K. is most grateful preferences. Nature 377: 420–422. for support from the Centre National de la Recherche Scientifique Kelly, J. K., and M. A. F. Noor, 1996 Speciation by reinforcement: (France), the Guggenheim Foundation, and the National Science a model derived from studies of Drosophila. Genetics 143: 1485– Foundation (grant DEB-9407969). M.R.S. is very grateful to the Na- 1497. Kimura, M., 1965 Attainment of quasilinkage equilibrium when tional Science Foundation for support through a Graduate Fellowship. gene frequencies are changing by natural selection. Genetics 52: This is contribution 98-087 from the Institut des Sciences de l’Evolu- 875–890. tion. Kirkpatrick, M., 1982 Sexual selection and the evolution of female choice. Evolution 36: 1–12. Kirkpatrick, M., 1987 Sexual selection by female choice in polygy- nous animals. Annu. Rev. Ecol. Syst. 18: 43–70. LITERATURE CITED Kirkpatrick, M., 1996 Good genes and direct selection in the evolu- tion of mating preferences. Evolution 50: 2125–2140. Bakker, T. C. M., and A. Pomiankowski, 1995 The genetic basis Kirkpatrick, M., and N. H. Barton, 1997 Evolution of mating of female mate preferences. J. Evol. Biol. 8: 129–171. preferences for male genetic quality. Proc. Natl. Acad. Sci. USA Balkau, B., and M. W. Feldman, 1973 Selection for migration modi- 94: 1282–1286. fication. Genetics 74: 171–174. Kirkpatrick, M., and M. J. Ryan, 1991 The evolution of mating Barton, N. H., and G. M. Hewitt, 1985 Analysis of hybrid zones. preferences and the paradox of the lek. Nature 350: 33–38. Annu. Rev. Ecol. Syst. 16: 113–148. Lande, R., 1981 Models of speciation by sexual selection on poly- Barton, N. H., and M. Turelli, 1991 Natural and sexual selection genic traits. Proc. Natl. Acad. Sci. USA 78: 3721–3725. on many loci. Genetics 127: 229–255. Lande, R., and S. J. Arnold, 1983 The measurement of natural and Butlin, R., 1987 Speciation by reinforcement. Trends Ecol. Evol. sexual selection on correlated characters. Evolution 37: 1210– 2: 8–13. 1226. Butlin, R., 1989 Reinforcement of premating isolation, pp. 158–179 Liou, L. W., and T. D. Price, 1994 Speciation by reinforcement of in Speciation and Its Consequences, edited by D. Otte and J. A. premating isolation. Evolution 48: 1451–1459. Endler. Sinauer Associates, Sunderland, MA. Maynard Smith, J., 1966 Sympatric speciation. Am. Nat. 100: 637– Caisse, M., and J. Antonovics, 1978 Evolution in closely adjacent 650. 880 M. Kirkpatrick and M. R. Servedio

Mayr, E., 1963 Animal Species and Evolution. Harvard University Press, and 0 at locus i on the preference and dis- Cambridge, MA. Moore, J. A., 1957 An embryologist’s view of the species concept, play trait. pp. 325–338 in The Species Problem, edited by E. Mayr. American eP, eT Random environmental component in the Association for the Advancement of Science, Washington, DC. phenotype of an individual for the prefer- Muller, H. J., 1942 Isolating mechanisms, evolution and tempera- ture. Biol. Symp. 6: 71–125. ence and display trait. Nagylaki, T., 1976 The evolution of one- and two-locus systems. Xi Indicator that equals 1 if the individual car- Genetics 83: 583–600. ries allele 1 at locus i, 0 otherwise. Noor, M. A., 1995 Speciation driven by natural selection in Drosoph- ila. Nature 375: 674–675. ␳ Correlation between the phenotypes of the Noor, M. A., 1997 How often does sympatry affect sexual isolation preference in females and the display trait in Drosophila? Am. Nat. 149: 1156–1163. in males among mated pairs of individuals. O’Donald, P., 1980 Genetic Models of Sexual Selection. Cambridge University Press, Cambridge, UK. m Migration rate, averaged over males and Rice, W. R., and E. E. Hostert, 1993 Laboratory experiments on females. speciation: What have we learned in 40 years? Evolution 47: 1637– a˜U Selection coefficient favoring the associa- 1653. tion between allele 1 at each of the loci in Ryan, M. J., 1990 Sensory systems, sexual selection, and sensory exploitation. Oxf. Surv. Evol. Biol. 7: 157–195. set U. Saetre, G.-P., T. Moum, S. Bures, M. Kra´l, M. Adamjan et al., 1997 A CU Genetic association (i.e., convariance or sexually selected character displacement in flycatchers reinforces linkage disequilibrium) between loci in set premating isolation. Nature 387: 589–592. Sanderson, N., 1989 Can gene flow prevent reinforcement? Evolu- U. tion 43: 1223–1235. n, s, h, Superscripts for quantities evaluated at the Schluter, D., 1995 Adaptive radiation in sticklebacks: trade-offs in m time of natural selection on the display feeding performance and growth. Ecology 76: 82–90. Schluter, D., 1996 Ecological causes of adaptive radiation. Am. trait, sexual selection on the display trait, Nat. 148: S40–S64. selection against hybrid incompatibility, Servedio, M. R., and M. Kirkpatrick, 1997 The effects of gene and migration. flow on reinforcement. Evolution 51: 1764–1772. Spencer, H. G., B. H. McArdle and D. M. Lambert, 1986 A theoreti- dij... (pЈi Ϫ pi)(pЈj Ϫ pj )... cal investigation of speciation by reinforcement. Am. Nat. 128: ␤n, ␤S Directional selection gradients on the dis- 241–262. play trait caused by natural and sexual selec- Sved, J., 1981 A two-sex polygenic model for the evolution of premat- ing isolation. I. Deterministic theory for natural populations. tion. Genetics 97: 197–215. ⌫n, ⌫s Stabilizing selection gradients on the dis- Turelli, M., and H. A. Orr, 1995 The dominance theory of Hal- play trait caused by natural and sexual selec- dane’s rule. Genetics 140: 389–402. Wagner, A., G. P. Wagner and P. Similion, 1994 Epistasis can tion. facilitate the evolution of reproductive isolation by peak shifts: pqij... pi qi pj qj ... 2 2 A two-locus two-allele model. Genetics 138: 533–545. ␴P, ␴T Phenotypic variances of the preference and White, M. J. D., 1973 Animal Cytology and Evolution, Ed. 3. Cam- bridge University Press, Cambridge, UK. display trait. Wolfram, S., 1996 The Mathematica Book, Ed. 3. Wolfram Media/ ␶n, ␶h Indicators that equal 1 if migration hap- Cambridge University Press, Cambridge, UK. pens before natural selection on the display Wu, C.-I., and A. W. Davis, 1993 Evolution of postmating reproduc- trait and hybrid incompatibility, 0 other- tive isolation: the composite nature of Haldane’s rule and its genetic bases. Am. Nat. 142: 187–212. wise. Wu, C.-I., and M. F. Palopoli, 1994 Genetics of postmating repro- rU Multilocus recombination rate: frequency ductive isolation in animals. Annu. Rev. Genet. 27: 283–308. with which recombination breaks up alleles Communicating editor: M. A. Asmussen in set U.

RU The recombination function (1 Ϫ rU)/rU. S1, S 2, S 3i Terms generated by stabilizing selection on the male display. I Measure of the force of selection against all APPENDIX A: NOTATION i forms of hybrid incompatibility with respect P, T Individual phenotype for the female mat- to preference locus i. 2 2 ing preference and male display trait. h P,hT Heritabilities of the preference and display P, T Mean phenotypes of the mating preference trait.

and display trait among zygotes on the is- DP, D T Difference between the mean phenotypes land. on the island and continent for the prefer- P, T, H Sets of loci affecting the preference, display ence and display trait, measured in pheno- trait, and hybrid incompatibility. typic standard deviations. p Frequency of allele 1 at locus i among zy- U Number of loci in set U. i | | gotes on island; qi ϭ 1 Ϫ pi. Hi Set of epistatically interacting loci that con- Ј (prime) Indicates a quantity evaluated on the conti- tribute to hybrid incompatibility (epistatic

nent (e.g., pЈP, TЈ). incompatibility model). P T ␥i , ␥i Difference in the effect between alleles 1 ni Number of loci in epistatic incompatibility Reinforcement of Mating Preferences 881

set Hi (epistatic incompatibility model). {j}, and {i,j}. The coefficients aU,л represent the force of si Selection coefficient acting on epistatic in- selection on females that favors combining alleles 1 at compatibility in set Hi (epistatic incompati- the loci in set U, while the aл,V are the corresponding bility model). force in males. The coefficients aU,V measure the force k(j,U) Function of allele frequencies in set Hj of selection produced by nonrandom mating that favors given by Equation 24 (epistatic incompati- alleles 1 at the U loci in females combining with alleles bility model). 1 at the V loci in males. The C are measures of genetic f(n) Effect on reinforcement of the number of association (linkage disequilibrium) defined by Equa- loci n in an epistatically interacting set (epi- tion 4 of the text.

static incompatibility model). If we let kU, k V*, and kU,V denote, respectively, the coeffi- sT Relative viability advantage of T1 males on cients of ␨U, ␨V*, and ␨U ␨V* in the fitness function, then island (four-locus model). from Equation B1 we have sE Epistatic selection coefficient against A1 B 2 kU ϭ aU,л Ϫ ͚ aU,V CV. and A2 B1 recombinants (four-locus model). VʕG ␣ Strength of preference of a female carrying i Doing the same for k* and k , then rearranging terms, allele P for a male carrying allele T (four- V U,V i i we find locus model). ␨ Ϫ i Xi pi. aU,л ϭ kU ϩ ͚ aU,V CV, (B3a) VʕG * aл,V ϭ kV ϩ ͚ aU,V CU, (B3b) APPENDIX B: SELECTION COEFFICIENTS UʕG

This appendix begins by giving an overview of the aU,V ϭ kU,V. (B3c) notation for selection coefficients and genetic disequi- The a˜ are then given by relations libria in multilocus systems developed by Barton and Turelli (1991), which for brevity we refer to as BT. 1 1 * a˜U ϵ (aU,л ϩ aл,U) ϭ (kU ϩ k U) ϩ ͚ a˜U,V CV, (B4a) Next, the selection coefficients (the a˜) produced by 2 2 VʕG natural and sexual selection on the male display are found. Last, we calculate the selection coefficients and caused by our detailed model of epistatic hybrid incom- 1 a˜U,V ϭ (kU,V ϩ kV,U). (B4b) patibility. 2 How to calculate BT selection coefficients: The selec- tion coefficients are calculated using a special form of The selection coefficients of the form a˜U measure the fitness function defined by BT. This fitness function average intensity of selection within each sex favoring depends on the genotypes of the female and the male allele 1 at the combination of loci in set U. Coefficients in a mated pair. (Defining fitness in terms of mated pairs of the form a˜U,V, on the other hand, reflect the force of rather than individuals makes it possible to account for nonrandom mating that combines allele 1 at the loci the genetic effects of any form of nonrandom mating.) in set U coming from one sex with allele 1 in the loci We denote the genotype of a female by the vector X in set V coming from the other sex. whose elements are 0 or 1 depending on which allele For our purposes, the a˜ only have to be calculated the female carries at the given locus; a male’s genotype to leading order in the parameters because our QLE is given by the vector X*. Then the fitness of a mated approximation for ⌬pi is accurate only up to terms of 2 pair can be written in general as order a˜ . (In Equation 2, the a˜ always appear in products with the C, which are themselves of order a˜ at QLE. W(X,X*) * Thus only leading-order approximations for the a˜ are ϭ 1 ϩ ͚ aU, л(␨U Ϫ CU) ϩ ͚ aл,V(␨V Ϫ CV) W UʕG VʕG needed.) The summation on the right side of Equation ϩ ␨ Ϫ ␨* Ϫ B4a therefore can be neglected because a˜U,V CV is of ͚ ͚ aU,V( U CU)( V CV), (B1) 2 UʕG VʕG order a˜ when the population is at QLE. Thus

1 2 where a˜U ϭ ⁄2(kU ϩ kU*) ϩ O(a ). (B4c)

␨i ϭ Xi Ϫ pi, ␨U ϭ ͟␨i , (B2) These results give us an algorithm for calculating the i෈U BT selection coefficients that appear in our model. The

␨* is defined similarly to ␨ but with X i* replacing Xi, and relative fitness of a mated pair is written in the form of G is the set of all the genes under selection. Equation B1, then Equations B4b and B4c are used The summations in Equation B1 are over all sets U and to calculate the a˜. In cases where the fitness function 2 V of loci in the genome. For example, with a genome includes terms such as ␨i , they can be reduced (see BT, 2 consisting only of the two loci i and j, the first summation p. 232) using the fact that ␨i ϭ piq i ϩ (1 Ϫ 2pi)␨i. would contain three terms corresponding to U ϭ {i}, That procedure gives exact results for the a˜ when the 882 M. Kirkpatrick and M. R. Servedio allele frequencies (the p) and the disequilibria (the C) We can now extract the BT selection coefficients. take the values that hold at the moment that selection Equation B7 is in the form of (B1), and so we can use is acting. Those quantities change during the course of Equations B4 to get a generation as the result of migration and selection. a ϭ␥[␤ϩ(1⁄ Ϫ p )␥ ⌫] ϩ O(a 2) (B8a) One can show, however, that at QLE the change within ˜i i 2 i i ˜ a generation in the value of an a˜ is of order a˜ 2. Again, and our final approximation for the change in a preference 2 allele frequency only requires the a˜ to first-order accu- a˜ij ϭ␥i␥j⌫ϩO(a˜ ) for i ϶ j, (B8b) racy. Thus we can simply calculate the a˜ for natural where selection, sexual selection, and migration as if each of these events was acting in isolation from the others. ␤ϭ(␤M ϩ␤F)/2 and ⌫ϭ(⌫M ϩ⌫F)/2, Selection on an additive trait: We use the method just (B9) described to find the selection coefficients that result and i and j represent any two loci contributing to the from selection acting on traits that have only additive trait. Selection coefficients for sets of three or more loci genetic variation. These results are needed to determine (for example, a˜ijk) and association coefficients (which the effects on the preference of natural and sexual selec- are of the form a˜U,V) can be neglected here because they tion on the male display. are of order a˜ 2 (that is, they include terms such as ␤M An individual’s phenotype for any additive trait can ␤F, ␤M ⌫F, etc.). Equations B8 are the basis for Equations always be written as in Equation 1, 7 and 8 in the text. Selection from epistatic incompatibility: Next we cal- z ϭ z ϩ ͚␥i(Xi Ϫ pi) ϩ e z , (B5) i culate the selection coefficients produced by our de- tailed model of epistatic incompatibility. Recall that we where the coefficient ␥i measures the difference be- tween the effect of allele 1 and allele 0 at locus i, and allow for the possibility of multiple sets of epistatically interacting loci, denoted H1, H2,....SetHi consists of ez is the environmental component of the phenotype. Denote the fitness of an individual with phenotype z as ni loci, and an individual’s fitness is reduced by si if it w(z). When the fitness function is fairly smooth, the carries a mixture of 0 and 1 alleles at those loci. We relative fitness of an individual can be approximated by therefore construct a BT fitness function in which fitness a Taylor series as is reduced whenever an individual does not carry all 0 or all 1 alleles within each set. The fitness of a mated w(z) 1 ≈ 1 ϩ␤(z Ϫ z) ϩ ⌫(z Ϫ z)2. (B6) pair is again the product of the fitnesses of the female w 2 and male: Here ␤ is the directional selection gradient, defined as W(X,X*) 1 ϭ {1 Ϫ ͚si [1 Ϫ ͟ Xj Ϫ ͟ (1 Ϫ Xj)]} the regression of an individual’s relative fitness onto W W i j෈Hi j෈Hi its phenotypic value, and ⌫ is the stabilizing selection * * ϫ {1 Ϫ ͚si[1 Ϫ ͟ X j Ϫ ͟ (1 Ϫ Xj )]} gradient, defined as the regression of relative fitness i j෈Hi j෈Hi onto the square of the deviation of the phenotype from ϭ 1 ϩ ͚[si ͟ (pj ϩ␨j)] ϩ ͚[si ͟ (qj Ϫ␨j)] its mean (see Lande and Arnold 1983). i j෈Hi i j෈Hi

This result is used to form the BT fitness function * * 2 ϩ ͚[si ͟ (pj ϩ␨j )] ϩ ͚[si ͟ (qj Ϫ␨j )] ϩ O(s ). for viability selection on an additive quantitative trait. i j෈Hi i j෈Hi Denote the phenotypes for the female and male in a (B10) mated pair as z and z*, respectively. Under simple viabil- Using Equations B4, we find that the selection coeffi- ity selection, the fitness of a mated pair is simply the cient for any set of epistatic incompatibility loci U that product of the viabilities of the male and the female. is a subset of H is Equations B2, B5, and B6 then give i U 2 a˜U ϭ si[ pj ϩ (Ϫ1)| | qj] ϩ O(a˜ ), (B11) W(z,z*) 1 ͟ ͟ ≈ ϩ ␥ ␨ ␤F ϩ ␥ ␨ 2⌫F j෈Hi j෈Hi 1 (͚ i i) (͚ i i) jÓU jÓU W ΄ i 2 i ΅ where U denotes the number of loci in set U. The * M 1 * 2 M | | ϫ 1 ϩ (͚␥i ␨i )␤ ϩ (͚␥i ␨i ) ⌫ . (B7) products, which are over all loci in Hi that are not also ΄ i 2 i ΅ in U, are defined to equal 1 when Hi ϭ U. This gives The superscripts F and M have been added to allow for Equations 23 and 24 of the text. the possibility that selection acts differently on females and males. Equation B7 is an approximation that as- APPENDIX C: GENETIC DISEQUILIBRIA sumes stabilizing selection is weak (specifically, that the environmental variance of the trait is much smaller than In this appendix we calculate the genetic disequilibria 1/⌫). (the C) at quasi-linkage equilibrium. The C that are Reinforcement of Mating Preferences 883 important to preference evolution in our model are are multiplied because migration does not produce any those that involve both a preference locus and other interactions between the sexes (see the discussion just loci that are under direct selection (see Equation 2). before Equation B7). The fitness for each genotype is These disequilibria result from just two factors, migra- that which changes its frequency by the same amount tion and sexual selection. Under the QLE approxima- that migration does. tion, one can show that the value of a C at the zygote If both the island and continent are at QLE, then the stage is the sum of the values that obtain from migration frequency of any genotype is to leading order equal to and sexual selection acting independently (see BT the product of the frequencies of the alleles it carries 2 Equation 25). For Equation 2, however, we need the [for example, P111 ϭ pi p j p k ϩ O(a˜ )]. Using this fact values of the C at the points in the life cycle when and Equations B2 and B4c, some algebra then shows natural and sexual selection act. These differ from the that the force of migration favoring association between values in zygotes because of changes within each genera- alleles 1 in any set of loci U is tion caused by selection and migration. The changes in qЈqЈqЈ ... pЈqЈqЈ ... the C caused by selection are of order a˜ 2, and so they ϭ Ϯ i j k Ϯ i j k a˜U m΂ can be neglected for our approximations. Migration qiqjqk ... piqjqk ... causes changes within a generation in the C that are Ј Ј Ј Ј Ј Ј qi pj qk ... pi pj qk ... 2 of order a˜, however, and so those changes must be Ϯ Ϯ ϩ O(a ), (C2) qipjqk ... pipjqk ...΃ accounted for. The effects of migration and sexual selection are cal- where m is the average of the male and female migration, culated in the next two sections. The overall disequilib- and Ϯ is read as a plus if there are an even number of ria are then presented in the final section. qЈ in the following term and a minus otherwise. Fac- Disequilibria caused by migration: We first calculate toring that expression gives us the effective selection the disequilibrium generated by migration. The ap- coefficient caused by migration, proach is to use the BT method by treating migration dU 2 as a form of selection. The idea is to write down a a˜U ϭ m ϩ O(a˜ ), (C3) “fitness” function that produces exactly the same change pqU in genotype frequencies as migration does. We can then where dU and pqU are defined by Equations 6 and 11 in determine the “effective selection coefficients” (the a˜) the text. This result implies that for the QLE approxima- that result from migration, and use these to find approx- tion to hold with migration, m must be sufficiently small imations for the disequilibria (the C) at QLE. that the magnitude of a˜U given by Equation C3 is much The BT fitness function reflecting the effects of migra- smaller than 1. tion is We can calculate the disequilibrium in zygotes at QLE among the preference locus i and a set of loci U that W(X,X*) PЈ˜i˜jk˜,... ϭ 1 Ϫ m f ϩ m f ͚ Yi,˜iYj,˜jYk,k˜ ... affect hybrid compatibility with the results above and W ΄ ˜i,˜j,k˜,... ΂ P˜˜˜ ΃΅ ijk,... Equation 25 of BT:

PЈ˜i˜jk˜,... ϫ Ϫ ϩ *˜ *˜ *˜ 1 mm mm ͚ Y i,iY j,jY k,k ... , 1 Ϫ riU 2 2 ΄ ˜i,˜j,k˜,... ΂ P˜i˜jk˜,... ΃΅ CiU ϭ aiUpqiU ϩ O(a˜ ) ϭ mdiU RiU ϩ O(a˜ ). ΂ riU ΃ (C1) (C4) where The disequilibria are different after migration than  ˜ 1 Ϫ Xi for i ϭ 0 they are in zygotes. Equations 5 and 12a of BT show Y ˜ ϭ . i,i  that the change within a generation in the disequilibria Xi for ˜i ϭ 1 among a set of loci A caused by selection is The migration rates mf and mm for males and females, 2 2 respectively, are the fractions of the island population ⌬sCA ϭ ͚aUCAU ϩ O(a˜ ) ϭ a˜ApqA ϩ O(a˜ ) (C5) U that consist of newly arrived migrants in each genera- tion. The summation indices ˜i, ˜j, k˜,...include all loci at QLE. Using our earlier result for the effective selec- in the genome, and they take on the values 0, 1. P˜i˜jk˜... is tion coefficient generated by migration (Equation C3), the frequency of genotype (˜i, ˜j, k˜,...)ontheisland we find the change within a generation in CiU caused at the time in the life cycle that migration occurs, and by migration is

PЈ˜i˜jk˜... is its frequency among migrants arriving from the 2 ⌬mCiU ϭ mdiU ϩ O(a˜ ). (C6) continent. The definition for Y i,i*˜ is analogous to that for Yi,i˜ , but with Xi* replacing Xi. Equations C4 and C6 give Equation 15 of the text, which The rationale behind the form of Equation C1 is as accounts for the timing of migration with the indicator follows. The right side is the product of the “fitnesses” variable ␶h. If migration happens before natural selec- of a male and a female in a mated pair; these terms tion on the display trait, then ␶h ϭ 1 and the additional 884 M. Kirkpatrick and M. R. Servedio disequilibrium generated within the generation by mi- equilibrium between a single preference locus i and a gration (Equation C6) is added to that in the zygotes single display trait locus j. Summing Equations C4, C6, (Equation C4). and C8 gives

Disequilibria caused by mating: The genetic conse- P T n ␳ ␥i ␥j 2 quences of sexual selection were studied by Kirkpat- C ij ϭ mdij(Rij ϩ␶n) ϩ pqij ϩ O(a˜ ) (C9) 2 ␴ ␴ rick and Barton (1997). A simple extension of their P T

Equation 6 shows that when the preference and display at the time that natural selection acts. The factor of ␶n traits are affected by genes with additive effects, the accounts for how migration changes disequilibria within strength of selection bringing together allele 1 at prefer- a generation (see Equation C6): it is defined as 0 if ence locus i from one sex with allele 1 at display trait natural selection occurs before migration, and as 1 oth- locus j in the other sex is given by the association coeffi- erwise. This is Equation 9 of the text. cient By the same logic, when sexual selection acts this disequilibrium is P T ≈ ␳ ␥i ␥j a˜i,j , (C7) ␳ ␥P␥T ␴ ␴ s i j 2 2 P T C ij ϭ mdij(Rij ϩ 1) ϩ pqij ϩ O(a˜ ). (C10) 2 ␴P␴T where ␳ is the phenotypic correlation between the pref- erence in a female and the display trait in a male among The factor of ␶n in Equation C9 has been replaced by mated pairs. This result holds for any form of mate 1 because migration must occur before sexual selection, choice behavior, regardless of how it is expressed. Equa- which is the last event of the generation. tion 25 of BT then shows that at QLE the disequilibrium Because sexual selection does not generate three-way between these loci is disequilibria between a preference locus i and display trait loci j and k, only migration needs to be considered. P T 2 ␳ ␥i ␥j 2 Thus at QLE, Equations C4 and C6 show Cij ϭ a˜i,j pqij ϩ O(a˜ ) ϭ pqij ϩ O(a˜ ). (C8) 2 ␴P␴T n 2 C ijk ϭ mdijk(Rijk ϩ␶n) ϩ O(a˜ ), (C11) Sexual selection with an additive preference and display and trait produces only these pairwise disequilibria; associa- s 2 tions between sets of three or more loci are not gener- C ijk ϭ mdijk(Rijk ϩ 1) ϩ O(a˜ ). (C12) ated when individual genes have small additive effects This gives Equation 10 of the text. on the preference and male traits. The recombination The disequilibria between a preference locus i and a rate between the loci affects how rapidly the QLE value set of loci U involved in hybrid incompatibility are is approached, but not its final value. caused entirely by migration. At QLE, Equations C4 and Overall disequilibria at QLE: Two kinds of genetic C6 give disequilibria appear in the reinforcement model: those h 2 C iU ϭ mdiU (RiU ϩ␶h) ϩ O(a˜ ). (C13) involving preference and display trait loci and those involving preference and hybrid incompatibility loci. This gives Equation 15 of the text and is the last of the Both migration and mate choice contribute to the dis- disequilibria needed in the model.