Theoretical Evolutionary Genetics of Flowering Plant Mating System And

Theoretical evolutionary genetics of ﬂowering plant mating system and self-incompatibility

A DISSERTATION SUBMITTED TO THE FACULTY OF THE UNIVERSITY OF MINNESOTA BY

Alexander Harkness IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY Advised by Yaniv Brandvain December 2020 ©2020 by Alexander Harkness Acknowledgments I would like to thank Yaniv Brandvain and Emma Goldberg for years of guidance, patience, and close attention. I would like to thank the Univer- sity of Minnesota for granting the Doctoral Dissertation Fellowship, which allowed me to complete and improve this dissertation, and the Department of Ecology, Evolution, and Behavior for the Graduate Excellence Fellow- ship, which allowed me to hit the ground running in my ﬁrst year. Chapters 1 and 2 were coauthored with Yaniv Brandvain and Emma Goldberg. Chapter 3 was coauthored with Yaniv Brandvain.

i Abstract

The mating system of a diploid eukaryote is an outcome of intragenomic coevolution. Close relatives are more likely to share recessive deleterious mutations at many locations, so an allele at another locus that reduces the probability of inbreeding will increase offspring’s expected fitness. Self-incompatibility in flowering plants, which acts through a polymorphic locus (an S-locus) that rejects pollen when pollen and pistil haplotypes match, is a particularly old and widespread inbreeding avoid- ance adaptation that has persisted through long-term balancing selection among different S-locus haplotypes (S-haplotypes). Intragenomic coevolution occurs between the individual elements of the S-locus: those expressed in pollen and those expressed in pistils. When intragenomic coevolution is dis- turbed, selection on mating system or on particular mating system adaptations is shifted and the population may adapt in new ways. In this thesis, the theoretical consequences of three dis- turbances to the intragenomic coevolution of mating system in flowering plants are determined. First, it is shown that isolation of the genetic load in separate inbreeding populations produces a transitory benefit upon secondary contact to a mutation promoting outcrossing, but that this benefit evaporates

ii too rapidly as the populations reassimilate to favor the evolution of greater outcrossing consistently. Second, it is shown that, under the taxonomically widespread ribonuclease-based self-incompatibility system, the evolution of a novel S-haplotype greatly disturbs inter-haplotype coevolution, and may either lead to coexistence of all haplotypes (diversiﬁcation) or extinction of multiple haplotypes (collapse) in a rescue-like process. Third, it is shown that biased patterns of pollen rejection form between non-coevolved S-haplotypes from isolated populations, which may favor the introgression of some haplotypes, prevent introgression of others, and cause some to be lost by swamping introgression.

iii Contents

List of Tables vi

List of Figures vii

1 The evolutionary response of mating system to heterosis 1 1.1 Introduction ...... 3 1.2 Model ...... 6 1.2.1 General model description ...... 6 1.2.2 Invasion condition ...... 10 1.2.3 Simulation model ...... 11 1.3 Results ...... 18 1.3.1 Outcrossing mutation never fixed in a three-locus deterministic model ...... 18 1.3.2 Outcrossing mutation could fix under restricted cir- cumstances in stochastic simulations ...... 18 1.3.3 Intermediate selection favored outcrossing . . . . . 19 1.3.4 No effect of loose linkage ...... 19 1.3.5 Outcrossing was sometimes lost after initially invading 20 1.3.6 Ultimate mean fitness and time to extinction were bimodal ...... 20

iv 1.3.7 Purging was incomplete when the outcrossing allele fixed ...... 21 1.3.8 Test 1: rare genetic background favored outcrossing mutation ...... 21 1.3.9 Test 2: unequal fitness reduced fixation proportion . 22 1.3.10 Test 3: continuous migration reduced fixation proportion ...... 22 1.3.11 Test 4: additive modifiers fixed more often ...... 22 1.3.12 Test 5: polygenicity was insufficient to favor outcrossing under reasonable parameters ...... 23 1.3.13 Test 6: unpurged inbreeding depression was more favorable to outcrossing than local drift load . . . . . 24 1.4 Discussion ...... 24 1.4.1 Implications for mating system evolution ...... 25 1.4.2 Robustness of model conclusions ...... 28 1.4.3 Conclusion ...... 33

2 Diversiﬁcation or collapse of self-incompatibility haplotypes as a rescue process 42 2.1 Introduction ...... 43 2.1.1 Conceptual challenges ...... 46 2.2 Model and Results ...... 50 2.2.1 Collaborative Nonself-Recognition ...... 50 2.2.2 Model outline ...... 53 2.2.3 Equilibrium frequency of SC intermediate ...... 58 2.2.4 Invasion of lock mutant ...... 60 2.2.5 Rescue of doomed haplotypes ...... 62 2.2.6 Long-term behavior ...... 68 2.3 Discussion ...... 71

v 3 Nonself-recognition-based self-incompatibility can alternatively promote or prevent introgression 92 3.1 Introduction ...... 93 3.2 Description ...... 99 3.3 Results ...... 108 Self-recognition-based SI ...... 108 Nonself-recognition-based SI ...... 108 3.4 Discussion ...... 112

Bibliography 127

A Appendix: condition for increase of the outcrossing modiﬁer allele with two viability loci 143

B Appendix: genotype frequency recursions for S-haplotype di- versiﬁcation 145

C Appendix: genotype frequency recursions for S-haplotype introgression 148 C.1 Nonself-recognition ...... 148 C.2 Self recognition ...... 150

Supplementary Figures 153

vi List of Tables

2.1 Notation ...... 81

vii List of Figures

1.1 Invasion condition for the outcrossing allele in the deterministic model ...... 34 1.2 Outcrossing allele fixation proportions ...... 35 1.3 Allele frequency trajectories of the allele, M ...... 36 1.4 Simulation outcomes in the final generation ...... 37 1.5 Fixation proportion with initially asymmetric viability genotype frequencies ...... 38 1.6 Effect of heterosis magnitude and architecture on fixation . 39 1.7 Effect of unpurged inbreeding depression ...... 40

2.1 Rejection of self pollen in the style ...... 80 2.2 Haplotype classes ...... 82 2.3 Model steps ...... 83 2.4 Conversion-selection balance of SC gene convertants . . . 84 2.5 Fitness of a pollen-limited SI mutation ...... 85 2.6 Doomed haplotypes ...... 86 2.7 Expansion probability ...... 87 2.8 Stable distribution of haplotype number ...... 88

2.9 Haplotype number transition matrix for Rconversion = 0.3 . . . 89 2.10 Long-term simulated haplotype number trajectories . . . . . 90

3.1 Pollen compatibility under self- and nonself-recognition . . . 120

viii 3.2 Evolutionary dynamics of shared foreign, shared local, unique foreign, and unique local S-haplotypes with self-recognition based SI ...... 121 3.3 Evolutionary dynamics of shared foreign, shared local, unique foreign, and unique local S-haplotypes with nonself-recognition based SI ...... 122 3.4 Effect of unique haplotypes on invasion threshold ...... 123 3.5 Bidirectional migration ...... 124 3.6 Equilibrium frequency of foreign haplotypes in each population with bidirectional migration ...... 125 3.7 Loss of diversity ...... 126

S1 Distribution of ﬁnal mean population ﬁtness at the end of the simulation ...... 154 S2 Distribution of simulation durations with outcrossing allele M dominant ...... 155 S3 Distribution of simulation durations with outcrossing allele M additive ...... 156 S4 Extent of adaptation ...... 157 S5 Purging dynamics ...... 158 S6 Unequal loads ...... 159 S7 Survival probability of a new gene convertant ...... 160 S8 Distribution of haplotype number after a single rescue/collapse event ...... 161 S9 Stable distribution of haplotype number with lower threshold 162

ix Chapter 1

The evolutionary response of mating system to heterosis

This chapter is a reprint of Harkness et al. [2019a].

Abstract

Isolation allows populations to diverge and to fix different alleles. Deleterious alleles that reach locally high frequencies contribute to genetic load, especially in inbred or selfing populations, in which selection is relaxed. In the event of secondary contact, the recessive portion of the genetic load is masked in the hybrid offspring, producing heterosis. This advantage, only attainable through outcrossing, should favor evolution of greater outcrossing even if inbreeding depression has been purged from the contributing populations. Why, then, are selfing-to-outcrossing transitions not more common? To evaluate the evolutionary response of mating system to heterosis, we model two monomorphic populations of entirely selfing individuals, introduce a modifier allele that increases the rate

1 of outcrossing, and investigate whether the heterosis among populations is sufficient for the modifier to invade and fix. We find that the outcrossing mutation invades for many parameter choices, but it rarely fixes unless populations harbor extremely large unique fixed genetic loads. Reversions to outcrossing become more likely as the load becomes more polygenic, or when the modifier appears on a rare background, such as by dispersal of an outcrossing genotype into a selfing population. More often, the outcrossing mutation instead rises to moderate frequency, which allows recombination in hybrids to produce superior haplotypes that can spread without the mutation’s further assistance. The transience of heterosis can therefore explain why secondary contact does not commonly yield selfing- to-outcrossing transitions.

2 1.1 Introduction

Secondary contact between previously isolated populations is a dramatic event with a variety of potential evolutionary repercussions, including genetic rescue [Richards, 2000], the exposure of epistatic incompatibilities evolved in isolation [Bateson, 1909, Dobzhansky, 1937, Muller, 1942], and the release of selfish genetic elements [Fishman and Willis, 2005]. A particularly common consequence of secondary contact is heterosis, an increase in fitness of the offspring of interpopulation crosses relative to within-population crosses. Heterosis is especially likely to accumulate between highly inbred or self-fertilizing (selfing) populations. Their small effective population sizes allow deleterious alleles to reach high frequencies locally, a phenomenon called local drift load [Whitlock et al., 2000]. The part of the local drift load attributable to recessive alleles is masked in hybrids, resulting in heterosis. Empirical heterosis estimates in populations of different census sizes [Heschel and Paige, 1995], levels of isolation [Richards, 2000], and breeding systems [Busch, 2006] generally support the proposition that isolated or predominantly selfing populations accumulate greater heterosis than effectively large, connected populations [but see Ouborg and van Treuren, 1994]. Here we explore the evolutionary consequences of the exposure of heterosis upon secondary contact between selfing populations that have each incurred their own genetic load. Specifically, we ask if such heterosis allows diverged populations to shed their genetic load, and if such heterosis favors the invasion and fixation of a mutation that increases the rate of cross-fertilization (outcrossing). Whereas the potential for heterosis increases as isolated populations inbreed, individuals can only exploit this advantage through outcrossing. Heterosis therefore provides an advantage of outcrossing in historically selfing populations, in which the advantage of outcrossing is usually low.

3 The reduced advantage of outcrossing within selfing populations stems from the purging of inbreeding depression: the removal or fixation of deleterious recessive alleles in highly homozygous selfing populations. Purg- ing may determine the long-term trajectory of mating system evolution: transitions to predominant selfing are traditionally expected to be irreversible because an increase in the selfing rate tends to purge deleterious alleles, reducing inbreeding depression, which then allows the evolution of even greater selfing in a positive feedback loop [Lande and Schemske, 1985]. But heterosis provides an alternative advantage that might be sufficient to favor the evolution of greater outcrossing even in predominantly selfing populations [Igic´ and Busch, 2013]. Could this heterosis favor the secondary evolution of outcrossing? While inferred patterns of trait evolution suggest selfing-to-outcrossing transitions are indeed rare [Stebbins, 1974, Barrett et al., 1996, Escobar et al., 2010], some reversions to outcrossing have been hypothesized [Barrett and Shore, 1987, Armbruster, 1993, Olmstead, 1990, Bena et al., 1998]. Indeed, it has been suggested that heterosis in the face of secondary contact has favored the secondary evolution of outcrossing [Igic´ and Busch, 2013]. Since purging only eliminates segregating variants within a population, heterosis caused by differentiation among populations is a logical route to the evolution of greater outcrossing. In contrast with the rich body of theory on the conditions for outcrossing-to-selfing transitions [Fisher, 1941, Kimura, 1959, Nagylaki, 1976, Lloyd, 1979, Lande and Schemske, 1985], there is a dearth of theory on selfing-to-outcrossing transitions. Their relative rarity is not a sufficient reason to neglect them: by exam- ining exceptions to typical patterns, theory can explain why the exceptions are rare. Theory predicts that heterosis will indeed be favorable to outcrossing [Theodorou and Couvet, 2002], but since previous models focused on het-

4 erosis maintained at equilibrium among connected populations, the effects of the potentially larger heterosis accumulated among long-isolated populations remain unknown. In one way, secondary contact among long- isolated populations should be more favorable to outcrossing because arbitrarily large levels of heterosis could have accumulated (limited only by the accumulation of incompatibilities or other isolating barriers in the in- terim). But this elevated heterosis is also transient: once secondary contact unites alleles into the same population, selection will reduce the frequencies of inferior alleles and will thus undermine the genetic differentiation underlying heterosis. There is thus a limited period in which outcrossing is highly favorable, after which heterosis declines. If high levels of outcrossing evolve during this window, the remaining deleterious alleles may be masked, ensuring a more lasting disadvantage of selfing. Alterna- tively, heterosis may be depleted before outcrossing mutations reach high frequencies, returning the populations to a highly selfing equilibrium. The final effects of secondary contact are thus unclear, and they require an explicit population genetic model. To investigate the conditions under which heterosis may allow greater outcrossing to evolve from predominant selfing, we simulated the invasion of a modifier mutation that increases the rate of outcrossing in a pair of completely selfing populations that have recently come into secondary contact. Our results verify that local drift load initially favors outcrossing, but they also show that in most cases this advantage is lost too quickly for an outcrossing allele to fix. The benefit of an outcrossing mutation depletes so quickly that only unrealistically great levels of initial heterosis are sufficient to propel it to fixation with any regularity. We find that even these demanding conditions were very generous compared to those in several more realistic variant scenarios, suggesting that secondary contact should rarely result in a reversion to outcrossing. Heterosis alone may never be

5 sufﬁcient. If heterosis contributes to reversions to outcrossing, it is likely only in conjunction with additional advantages of outcrossing such as unpurged inbreeding depression.

1.2 Model

We first outline the general structure of the model, then use a simple version of the model to derive the invasion condition for an outcrossing allele analytically, and finally describe in detail the specific simulation steps for the main study. All simulations and numerical iterations were performed using custom R scripts [R Core Team, 2020].

1.2.1 General model description

Population history

We began with two selfing populations generated perhaps by a vicariance event that split an ancestral population in half. The two populations’ highly selfing mating system could have been inherited from their common ancestor, or it could have evolved in both populations after they split. We assume that each of these parental populations had been selfing long enough to become monomorphic, with unique sets of fixed alleles at L total viability loci. Any differences in the frequencies of deleterious alleles can contribute to heterosis [Whitlock et al., 2000], but we focus on the con- ceptually simple extreme case of fixation versus absence. One population possessed the inferior allele at LA viability loci, while the other population possessed the inferior allele at the remaining LB loci (LA + LB = L). Such a pattern could be caused by unique beneficial mutations fixed by selection in each population (in which case the superior alleles are derived) or

6 by unique deleterious mutations fixed by chance in each population (in which case the inferior alleles are derived). The same pattern could also be caused by alternative fixation of segregating alleles in each population, either because the inferior allele fixed by chance in one population or, perhaps more realistically, the currently inferior allele was temporarily superior in one population’s local environment. Whatever the populations’ historical environments were, we assume that they currently occupy identical environments in which one allele is unambiguously superior. However realistic monomorphism may be, it is convenient for isolating the effects of heterosis because fixed differences only contribute to heterosis, whereas segregating deleterious recessives at different frequencies would contribute both to heterosis and to inbreeding depression. We consider segregating alleles separately in a variant model. The two parental populations then came into secondary contact, form- ing a daughter population composed of NA individuals from one of the contributing populations and NB individuals from the other (NA + NB = N). In the most general case, we assume that, each generation, a fraction ν of seeds produced by residents in the daughter population were replaced by seed migrants drawn equally from the two parental populations. However, we assume in our focal case (described later) that the daughter population is formed by a single event and receives no additional migrants thereafter

(ν = 0). Any F1 individuals produced by outcrossing within the daughter population would be heterozygous at all loci at which their parents were ﬁxed for divergent alleles. Our model describes evolution from the time of secondary contact on- wards. Selection acts directly on viability and indirectly on mating system, which determines the viability of offspring. Selﬁng confers a transmission advantage [Fisher, 1941], while outcrossing confers the potential advantage of producing heterozygous offspring. Viability selection was chosen

7 so that selection would occur after mating, and so the modifier’s effects on progeny genotypes would be exposed to selection in a single generation. This allows us to use the change in allele frequency between two consecutive generations of adult cohorts as an invasion criterion because it reflects the effects of both offspring fitness and transmission advantage. The inferior alleles must be at least partially recessive to cause heterosis, and we assumed them to be completely recessive for convenience. If the deleterious alleles were only partially recessive, heterosis would decrease because they would be only partially masked, and they would also be more rapidly removed by selection. Deleterious alleles, especially strongly deleterious ones, also tend to be more recessive in nature [Mackay et al., 1992, Caballero and Keightley, 1994]. Thus, in our model, each inferior allele was recessive and conferred a viability of 1 − s, where s is the selection coefficient against any inferior allele. Viability was multiplicative across loci, so an individual had fitness (1 − s)l, where l is the number of loci at which the individual was homozygous for the inferior allele. Fitness effects at multiple loci are multiplicative if the loci have independent effects on the probability of survival or reproduction [Charlesworth and Charlesworth, 2010, p. 166]. However, we assumed viability selection to be soft so that the population size remained constant whatever the average viability was. Strictly speak- ing, therefore, the viability calculated from the product of viability effects across loci was not itself the survival probability, which was determined by the viability relative to competing genotypes under soft selection. It can nevertheless be imagined as some trait which is the sole basis of viability selection.

8 Mating system

We considered a rare outcrossing allele, M, at a modifier locus unlinked to any of the L viability loci under direct selection. The outcrossing allele is initially rare because it could not have been maintained at high frequency in the purged ancestral populations, although recurrent mutation could have kept it at a low-frequency mutation-selection balance. The common selfing allele at this locus is denoted m. We assume throughout that mm homozygotes are completely selfing and that MM homozygotes are completely outcrossing, but we consider both additive and dominant versions of M. An outcrossing allele of large effect maximizes the probability that outcrossing will take place, whereas a more biologically realistic small-effect allele would require many additional simulations in which the outcrossing phenotype was never expressed. The outcrossing phenotype we modeled best corresponds to a morphological trait that prevents self- pollination, rather than to self-incompatibility, because it caused random rather than disassortative mating. In the terminology of Lloyd [1992], seed discounting was complete: every selfed seed came at an opportunity cost of not producing one outcrossed seed. Of the different modes of selfing Lloyd [1992] defined, ours was similar to prior selfing in that the fraction of ovules selfed was independent of the supplies of self and outcross pollen. However, each of Lloyd’s modes of selfing assumed a limited supply of ovules for each dam, while ours did not. The modifier genotype only controlled mating, and it had no direct effect on viability or fecundity. Therefore, selfing in our model lacked the advantage of reproductive assurance [Lloyd, 1992] and the disadvantage of pollen discounting [Nagylaki, 1976, Holsinger et al., 1984, Lloyd, 1992]. Our model does have Fisher’s [1941] automatic advantage of selfing: self- fertilization transmits more gametes than outcrossing because the addi-

9 tional siring success on selfed ovules does not detract from outcross siring success.

1.2.2 Invasion condition

We analytically obtained the condition under which an outcrossing allele was expected to increase in frequency initially, for a simple case of the model. The population size was infinite, and there was no ongoing migration (ν = 0). The genome consisted of three unlinked loci: one modifier locus controlling the outcrossing rate and two viability loci. The daughter population initially had equal frequencies of the two parental genotypes: homozygous for the superior allele at one viability locus and for the inferior allele at the other viability locus. Thus all individuals initially had fitness 1−s. The Appendix (Appendix A) contains the invasion condition (Inequal- ity A.1) and its derivation. Essentially, the selfing allele m always increases in frequency in the seed pool before selection because of its transmission advantage. For the outcrossing allele M to increase in frequency by the next generation, M-carrying offspring must have sufficiently greater average viability such that m decreases in frequency among adults despite having increased in frequency among seeds. However, numerical iteration showed that, at least in the deterministic case, two viability loci were too few to allow M to reach fixation even for extremely large s (Fig. 1.1). In contrast, recombination would take longer to break up the repulsion-phase linkage disequilibrium among greater numbers of loci, which could delay the purging of the load. We therefore proceeded to a model that could accommodate more viability loci.

10 1.2.3 Simulation model

We used a simulation model to investigate the dynamics and fixation probability of the outcrossing allele. In contrast to deterministic numerical iteration, stochastic simulations allowed us to identify trajectories that differed from the expected outcome. We were interested in possible transitions to greater outcrossing rates even if outcrossing was not a guaranteed consequence of initial heterosis. In particular, we investigated whether increased outcrossing was substantially more probable with more initial heterosis, using simulations to estimate the probability of fixation of a new mutation that increases outcrossing. Additionally, simulations allowed us to incorporate the disadvantages of reduced recombination (and thus of selfing) caused by stochasticity in the haplotypes that are produced and survive [Muller, 1964, Hill and Robertson, 1966]. These stochastic effects could play a major role in the evolution of outcrossing. We conducted these simulations under multiple models of the magnitude and genetic architecture of heterosis, and of population history. Our simulation model was identical to the analytic model used above to obtain the invasion condition, except that we simulated with finite population size and more than two viability loci, which were not necessarily unlinked.

Genetic architecture

We assumed that the viability loci were evenly spaced along nchrom chromosomes. Each chromosome except the ﬁrst carried the same number of loci. The ﬁrst chromosome took the remainder if the number of loci was not divisible by the number of chromosomes. In each region between viability loci, recombination occurred with probability r independently of recombination elsewhere in the genome. Viability loci on separate chromosomes assorted independently. The relative positions of the superior alleles allot-

11 ted to each genotype were randomized at the beginning of each simulation. We neglected ongoing mutation during the invasion process because it would be dwarfed by the heterosis built up before contact. Instead, the only mutation was the original one giving rise to the outcrossing allele M at frequency 1/2N in a population composed of N diploid individuals. The outcrossing modiﬁer locus was unlinked to any of the viability loci.

Mating

Each generation, we created a seed pool ten times as large as the number of adults in the daughter population (i.e., 10N). For each seed to be generated, a dam was chosen at random to outcross or self with a probability determined by its genotype at the modiﬁer locus. If the dam selfed, it was also the sire. If the dam outcrossed, a sire was chosen at random from the population. One offspring was then produced as the product of a recombinant gamete from each parent.

Selection

Soft viability selection among seeds generated the next adult generation. Such a scenario is similar to competition among seeds or seedlings in a small plot: although all individuals would be physically capable of survival on their own, competition eliminates the worse competitors. Ten percent of seeds were sampled without replacement to survive to adulthood, so that each generation had N adults. Seed i’s probability of being sampled was weighted by its viability, (1 − s)li , where s is the selection coefﬁcient at each viability locus, and li is the number of viability loci homozygous for the inferior allele in individual i. Because we assumed viability selection in the deterministic model, we also assumed viability selection in the simulation so that we could fairly compare their results.

12 Duration

Each simulation was terminated when the outcrossing allele reached fixation, when the selfing allele reached fixation, or when 1000 generations had passed, whichever came first. We found that 1000 generations was usually more than sufficient to allow the outcrossing allele to run its course to fixation or extinction in preliminary simulations. If a parameter combination resulted in one or more simulations that failed to reach fixation or loss of the outcrossing allele before 1000 generations had passed, all simulations for that parameter combination were discarded. Two parameter combinations (s = 0.1, r = 0.5, LA = LB = 50 and s = 0.7, r = 0.1,

LA = LB = 14) were discarded, and all simulations for these parameter combinations were rerun.

Constant parameters

We held some parameters constant for all simulations. We chose a small population size of N = 100 because the simulation scaled poorly with N, but we compensated by performing enough replicates to observe the diversity of outcomes. We chose 2000 trials per parameter combination so that a new neutral mutation would ﬁx 10 times on average, allowing us to observe deviations below this expectation. Fixations of M more or less frequent than the neutral expectation imply positive or negative selection on M, respectively.

Simulation scenarios

Our primary question is whether heterosis can allow outcrossing to evolve. Because the answer seemed to be “no” in most cases, as we detail below, we pushed our simulation efforts toward parameters chosen to be gener-

13 ally favorable to outcrossing even if they strained the boundaries of real- ism. We call the scenario using this set of parameters the focal scenario and devote most of our description to it. But in order to extend our conclusions to more realistic scenarios, we also conducted additional tests. Some were expected to be more favorable to outcrossing than the focal scenario, while others were expected to be less favorable. We assumed equal sizes of the parent populations in the focal scenario

(NA = NB = 50). This postponed the eventual loss of either genotype, an outcome that would eliminate the advantage of outcrossing. In follow-up Test 1, we relaxed this assumption and examined population size ratios of 1:9, 1:4, 3:7, and 2:3 (NA 6= NB), and 1:1 (NA = NB), all with total population size N = 100. We distinguished cases where M arose on the rarer or the more common background. For this test, we assumed M was dominant, nchrom = 2, r = 0.5, s = 0.1, and LA = LB = 5, 25, or 50. We assumed equal ﬁtnesses of the two parental genotypes in the focal scenario so that neither could simply outcompete the other before M had a chance to invade. We therefore set the number of loci with deleterious alleles to be the same in the two parental populations, LA = LB, and varied this number from 5 to 50 in increments of 1. In follow-up Test 2, we allowed the populations to have unequal numbers of ﬁxed deleterious alleles. We examined LA : LB ratios of 1:9, 1:4, 3:7, and 2:3 (LA 6= LB), and 1:1 (LA = LB). For this test, we assumed M was dominant, nchrom = 2, r = 0.5, s = 0.1, and NA = NB = 50. The focal scenario assumed that there was no ongoing migration after formation of the daughter population (ν = 0). We thought this could favor outcrossing because M would not be swamped out by migration. How- ever, it is also possible that migration of inferior parental genotypes could regenerate the segregating load and thus the advantage of outcrossing. In follow-up Test 3, we allowed the initial parental populations to continue

14 to contribute migrant seeds to the focal daughter population (ν > 0). We varied the migration rate ν from 0.1 to 0.5 in increments of 0.1. We further set LA = LB = 25 and s = 0.3 because these parameters resulted in an intermediate fixation probability in the focal case, and we assumed M was dominant, nchrom = 2, r = 0.5, and NA = NB = 50. At the modifier locus, we set M to be completely dominant in the focal scenario. This ensured that the outcrossing allele was expressed even when rare. In follow-up Test 4, we instead made the outcrossing allele additive. In this test, ovules from Mm dams were equally likely to be selfed or outcrossed. The other parameters in this test were nchrom = 2, r = 0.5, s = 0.3, NA = NB = 50, and LA = LB = 5, 25, or 50. The central conceit of the focal scenario was that the load was oligogenic, with few enough loci that all were either unlinked or only loosely linked. The parameters chosen to represent this scenario were a small (s = 10−5, 10−4, 10−3, and 10−2) or large (s = 0.1 to 0.9 in increments of 0.1) selection coefficient for all loci, LA = LB = 5 to 50 viability loci in increments of 1, and a recombination frequency of r = 0.5 or 0.1. We examined large selection coefficients because they produced large magnitudes of initial heterosis, favoring outcrossing. But in nature, such strongly deleterious alleles could not have fixed in the first place. Large levels of local drift load would only be plausible if composed of many alleles of small effect. In follow-up Test 5, we considered more polygenic heterosis. In this test, we assumed a large number of individually weakly deleterious alle- −5 −3 les, LA = LB = 2500 and s = 10 or 10 . With this many loci, many of them would certainly be tightly linked in nature, so we lowered the value of r. We set nchrom = 10 chromosomes and r = 0.002, which gave each chromosome of 500 loci a map length slightly less than 1 morgan (slightly less because there were only 499 inter-locus regions with a 1/500 recombination probability each). The other parameters were NA = NB = 50 and

15 dominant M. We assumed in the focal model that all deleterious alleles in the ancestral populations had either been purged or fixed. This idealized endpoint of the purging process generated substantial heterosis from the large differences in allele frequencies but also eliminated inbreeding depression in the ancestral populations. However, purging need not be complete in nature, and inbreeding depression may contribute more to the evolution of outcrossing than heterosis does. In follow-up Test 6, we allowed the ancestral populations to retain some unpurged inbreeding depression. In each ancestral population, 25 loci were segregating for a low-frequency inferior allele and a high-frequency superior allele. Each ancestral population was segregating for different loci: the 25 loci segregating in one population were fixed for the superior allele in the other. Each individual in the first generation was homozygous for the inferior allele at 5 loci randomly se- lected out of the 25 segregating loci in the ancestral population from which that individual originated. These segregating alleles only affected the initial genotype composition of the daughter population, and the ancestral populations were not tracked afterwards. The selection coefficient against each deleterious allele was 0.14. Collectively, these parameters result in an inbreeding depression of approximately 0.46 within each parental population, equal to the level of unpurged inbreeding depression that Willis [1999] found to remain after artificial inbreeding of Mimulus. For this test, we varied the total number of viability loci diverging between the ancestral populations (including the 25 segregating sites plus additional fixed sites) from 25 (all segregating) to 50 (half segregating, half fixed). The fixed sites functioned identically to those in the focal scenario. For each parameter combination in this test, we ran a control in which the 25 segregating sites were replaced with 25 fixed differences.

16 Model outputs

For each parameter combination, we recorded the proportion of simulations resulting in the fixation of the outcrossing allele. This was our estimate of the fixation probability. For the subset of parameter combinations in which s = 0.3 and LA = LB = 5, 25, or 50, we ran an additional 2000 trials per parameter combination to track the outcrossing allele frequency through time, as well as recording the population mean fitness and the duration of the simulation in generations. Previous simulations showed these parameter combinations to result in a wide range of fixation proportions. We also tracked the dynamics for Test 4, with an additive outcrossing allele. We did not track evolution in the ancestral populations, so heterosis between them could not evolve. However, inbreeding depression in the daughter population did evolve, and, much like heterosis, this metric captured the advantage of crossing with unlike genotypes. In a sense, the initial heterosis between parental populations was converted to inbreeding depression by introducing segregating deleterious recessives into the daughter population. We therefore tracked inbreeding depression in the daughter population through time, along with duration and mean fitness, for a new set of simulations with s = 0.3, LA = LB = 5, 25, or 50, and a dominant outcrossing allele. Inbreeding depression was estimated each generation by generating 5000 selfed and 5000 outcrossed offspring from random dams irrespective of modifier genotype, calculating the mean fitness of these two sets of offspring, and substituting the resulting fitnesses into the formula for inbreeding depression, δ = 1 − wi/wo, where wi and wo are the mean fitness of inbred (selfed) and outcrossed offspring.

17 1.3 Results

We first briefly report results for the deterministic model. Then we report the main results from our focal simulation model, including both fixation probabilities and transient dynamics. We conclude with results from our follow-up tests of more general model assumptions.

1.3.1 Outcrossing mutation never ﬁxed in a three-locus deterministic model

Our analysis of the invasion condition (Inequality A.1) showed that the outcrossing allele, M, was expected to increase in frequency when the selection coefficient, s, was slightly below 0.67 for an initial frequency of M of 0.005. The threshold for s approached 2/3 as the frequency of M approached zero. This result comports with the 1/2 inbreeding depression threshold calculated by Kimura [1959] because inbreeding depression is 1/2 when the two initial genotypes are at equal frequencies and s = 2/3. Numerical iteration for a single generation confirmed that M was lost in a single generation below the threshold and was lost after six or more generations above it (Fig. 1.1). Thus, the outcrossing allele never fixed in the deterministic model.

1.3.2 Outcrossing mutation could ﬁx under restricted cir- cumstances in stochastic simulations

A rare outcrossing mutation fixed in some of the simulations (Fig. 1.2). For many parameter combinations we considered, the mutation fixed more often than the expectation (0.005) for a neutral allele. However, M never fixed when L ≤ 100 for the small coefficients of s < 0.1. All further results

18 for the focal scenario are reported for s ≥ 0.1 (results for Test 5, in which s < 0.1 but L = 5000, are reported later). Across all of parameter space that we examined, the outcrossing allele only fixed more often than the neutral expectation when the F1s had a fitness of at least 29.12 times that of individuals in either parental population, which translates to an initial heterosis of 2812%. The outcrossing allele fixed when many loci were under selection and/or when there was strong selection at each locus.

1.3.3 Intermediate selection favored outcrossing

The fixation proportion was greater for larger values of the selection coefficient, s, than for the smallest values (Fig. 1.2). However, the relationship was not monotonic. Fixation proportion increased sharply going from s = 0.1 to s = 0.2. Fixation proportion reached a maximum somewhere in the range from s = 0.2 to 0.4, but the exact value depended on the number of viability loci. At LA = LB = 50, fixation was maximized at s = 0.3.

1.3.4 No effect of loose linkage

There was no qualitative difference between the results of simulations with free recombination between viability loci (r = 0.5) and those with reduced recombination (r = 0.1; Fig. 1.2). The outcrossing allele fixed more often than the neutral expectation in five more parameter combinations when r = 0.1 (303 parameter combinations) than when r = 0.5 (298 parameter combinations). The maximum difference in fixation proportion for any parameter combination was 0.06. The maximum fixation proportions were 0.6545 and 0.6455 for r = 0.1 and r = 0.5, respectively. Differences showed no consistent pattern with respect to s or L. All further results of the focal scenario are reported for r = 0.5.

19 1.3.5 Outcrossing was sometimes lost after initially invading

The outcrossing allele, M, often ultimately went extinct even when it initially increased in frequency (Fig. 1.3). There was a qualitative difference between parameter combinations in which M sometimes fixed and those in which it never fixed. In parameter combinations in which M never fixed, the initial increase in frequency was much smaller (Fig. 1.3, left column). Loss was rapid when it occurred, regardless of the frequency of M before the decrease in frequency began.

1.3.6 Ultimate mean ﬁtness and time to extinction were bimodal

All simulation durations and final mean fitnesses are summarized in Fig. S1, S2, S3, and Fig. 1.4. Simulations resulting in fixation of M most often ended with high but not maximal mean population fitness. Simulations resulting in loss of M showed a bimodal pattern, with the largest number eventually ending with very high fitness, and a smaller number rapidly ending with very low fitness (Fig. 1.4). The shortest and longest runs resulted in extinction of M, whereas runs resulting in fixation lasted an intermediate duration (Fig. 1.4). Purging was more complete in the longer runs than in the shorter ones (Fig. S4): the mean of the final mean fitnesses of all simulations with LA = LB = 50 was 0.81, while the mean for the subset with below-average duration was 0.69. The modal duration was one generation for all six of the parameter combinations that were tracked through time (s = 0.3, LA = LB = 5, 25, or 50, dominant or additive M). Two parameter combinations were exceptions to the general pattern. At

LA = LB = 5, no ﬁxations were observed and the ultimate ﬁtness had one

20 mode concentrated near zero (Fig. S1), and in the additive model with

LA = LB = 50, most extinctions of M were rapid (Fig. S3).

1.3.7 Purging was incomplete when the outcrossing allele ﬁxed

Inbreeding depression was gradually purged in all simulations but to a lesser degree in simulations in which the outcrossing allele reached fixation (Fig. 1.4). In contrast, purging was often nearly complete when the outcrossing allele was lost. There was no obvious difference in the rate of purging between runs in which M fixed and those in which it was lost, assuming the same parameter values (Fig. S5). However, runs in which M was destined to fix were shorter, and thus had less time to purge.

1.3.8 Test 1: rare genetic background favored outcrossing mutation

We initially assumed that the two parental populations contributed exactly the same number of individuals to the daughter population. The frequencies of the two initial genotypes determined the probability of an F1 cross because outcross matings were at random, and we varied those initial frequencies partly to determine the effect of this probability. Compared to when the initial genotypes were at equal frequencies, the outcrossing allele ﬁxed slightly more often when it arose on the rarer background, but much less often when it arose on a very common background (Fig. 1.5). However, this effect was only visible when there were many viability loci

(LA = LB = 50) because no ﬁxations occurred at LA = LB = 5 or 25 when s = 0.1.

21 1.3.9 Test 2: unequal ﬁtness reduced ﬁxation proportion

The assumption that individuals from the two parent populations were of equal fitness conveniently prevented one from simply outcompeting the other before mating system could change. We relaxed this assumption to test if it was essential for outcrossing to evolve. For a given total load, the fixation proportion was greater when the two initial genotypes carried equal numbers of deleterious alleles (LA = LB) than when one genotype carried more (LA 6= LB) (Fig. S6). In many cases, smaller but more equal numbers of deleterious alleles per population resulted in a greater fixation proportion: e.g., LA = LB = 35 (0.4555 fixation proportion) vs. LA = 30,

LB = 45 (0.4255 ﬁxation proportion).

1.3.10 Test 3: continuous migration reduced ﬁxation proportion

Continuous migration of less fit genotypes from the parent populations into the daughter population could swamp out M or slow the purging process but was not modeled in the focal case. We therefore tested its effects by increasing the migration rate, ν. When ν > 0, the fixation proportion was always lower than when ν = 0, and it generally decreased as ν increased. The fixation proportions were 0.1855 (ν = 0.0), 0.1655 (ν = 0.1), 0.1565 (ν = 0.2), 0.1510 (ν = 0.3), 0.1515 (ν = 0.4), and 0.1300 (ν = 0.5).

1.3.11 Test 4: additive modiﬁers ﬁxed more often

When dominant outcrossing alleles invaded, they often segregated at high frequencies and then declined to extinction instead of reaching ﬁxation (Fig. 1.3). We tested whether this pattern was due to dominance. Com- pared to dominant outcrossing alleles, alleles that additively increased

22 the outcrossing rate did not slow their invasion at high frequencies before reaching ﬁxation. Such additive alleles spent less time segregating, and their allele frequency trajectory less often reversed direction. This resulted in an increased ﬁxation probability for additive alleles. At LA = LB = 5,

M never ﬁxed. At LA = LB = 25, M ﬁxed in 343 (M dominant) or 598

(M additive) out of 2000 trials. At LA = LB = 50, M ﬁxed in 1292 (M dominant) or 1599 (M additive) out of 2000 trials.

1.3.12 Test 5: polygenicity was insufﬁcient to favor outcrossing under reasonable parameters

In the focal scenario, the outcrossing allele fixed more often when the ancestral populations differed by many fixed viability loci. Holding the selection coefficient constant, the fixation proportion generally increased with the number of viability loci (Fig. 1.2). Fixation was maximized at the greatest number of viability loci modeled (LA = LB = 50). This effect was not due simply to initial heterosis being stronger with more loci, though. The outcrossing allele fixed more often under a polygenic initial heterosis than an oligogenic initial heterosis of comparable magnitude (Fig. 1.6). Based on this observation, we tested whether more reasonable loads could favor outcrossing if they were even more polygenic. However, the effect of polygenicity was not sufficient to favor M unless the total magnitude of initial heterosis was also extremely large. The highly polygenic loads (L = 2500) resulted in either no fixations of M for a small selection coefficient (s = 10−5) or a fixation proportion of 0.006 for a larger selection coefficient (s = 10−3).

23 1.3.13 Test 6: unpurged inbreeding depression was more favorable to outcrossing than local drift load

We assumed in the focal scenario that all deleterious alleles had reached fixation or loss in the purged ancestral populations. In this test, we instead assumed that 25 loci in each ancestral population retained a low-frequency segregating deleterious allele unique to that population. The fixation probability was greater when there were both fixed and segregating differences than in controls in which all differences were fixed (Fig. 1.7). The difference in fixation proportion was greatest when there were few total diverging sites (0.6265 with 25 segregating sites per population; 0.0045 with 25 pairs of fixed differences per population) and decreased as the total number of diverging sites increased (0.6265 with 25 segregating sites and 25 fixed differences per population; 0.5935 with 50 fixed differences per population).

1.4 Discussion

When two previously selfing populations come into secondary contact, initial heterosis is immediately favorable to outcrossing. We find that an allele that confers complete outcrossing invades an otherwise completely selfing population and rapidly reaches a moderate frequency for many parameter combinations. We further find, however, that the initial allele-frequency differentiation underlying heterosis is eventually eliminated by selection among the now-segregating alleles. This loss of heterosis removes the advantage of outcrossing. The outcrossing mutation is then usually lost, and only for extremely large initial heterosis can it reach fixation before it becomes obsolete. Even when selfing ultimately prevails, the bout of recombination in the outcrossed hybrids and backcrosses sorts the superior

24 alleles from each population to produce fit new haplotypes. Secondary contact thus promotes a transient increase in outcrossing, followed by removal of inferior alleles originally fixed in the donor populations, an increase in population mean fitness, and finally a return to selfing. In this sense, adaptive introgression of superior alleles from each population is parallel to purging of inbreeding depression within a population with respect to its effects on the advantage of outcrossing. Although some parameter combinations allowed the outcrossing allele to fix more often than expected by chance, it was still often rapidly lost after reaching high frequencies (red lines in Fig. 1.3, middle and right columns). This is because recombination in outcrossed individuals generated lightly loaded haplotypes that suffered less from selfing. These haplotypes were introduced into the selfing subset of the population because individuals heterozygous for the outcrossing allele produced some seeds homozygous for the selfing allele when they received pollen carrying the selfing allele. Competition between the less and more heavily loaded selfing haplotypes resulted in purging and an increase in the mean fitness of selfed offspring. That is, selfing generated linkage disequilibrium between the selfing allele and high-fitness haplotypes, in a manner similar to the way in which the “reduction principle” [Feldman and Liberman, 1986] favors alleles that decrease the recombination rate because they generate LD with high-fitness haplotypes.

1.4.1 Implications for mating system evolution

The smallest initial heterosis that allowed the outcrossing allele to ﬁx more often than the neutral expectation was 2812%, a level almost fortyfold larger than the highest heterosis we found in the literature (73.6% heterosis in a self-compatible population of Leavenworthia alabamica; Busch

25 2006) and over sixtyfold larger than the 42.6% heterosis for total survival documented in Scabiosa columbaria [van Treuren et al., 1993]. The extent of initial heterosis required for the adaptive fixation of an outcrossing allele not only surpasses levels observed to date, but also likely exceeds any theoretically reasonable level. Surprisingly, we find more frequent fixation at intermediate selection coefficients than at larger ones (Fig. 1.2b, 1.2a). The transition from a positive to a negative effect of s on the fixation probability may be because purging was ineffective in the lower range of s. Glemin´ [2003] showed that purging by nonrandom mating is mostly ineffective below a population-size threshold on the order of N = 10/s. For the smallest selection coefficients in our focal model, the population size of N = 100 is right in the vicinity of this threshold. Below the threshold, more deleterious alleles should increase the magnitude of initial heterosis without being exposed to purging. Above the threshold, more deleterious alleles are more effectively purged, and the advantage of outcrossing is reduced. However, the coefficients at which the fixation probability is maximized (s ≈ 0.3) are unrealistically large. Even a disadvantage for a given deleterious allele of s = 0.1 is large enough that its fixation would be effectively impossible in nature. Polygenic loads did favor outcrossing more than oligogenic loads of equal magnitude (Fig. 1.6). A possible explanation is that, since it takes more generations for recombination to unite the best alleles at many loci, purging will be impeded and outcrossing will remain advantageous for longer. However, polygenicity does not appear to be a sufficient substitute for strong per-locus selection: 2500 unique fixed deleterious alleles per population with s = 10−3 only resulted in a fixation proportion of 0.006, and the same number of alleles with s = 10−5 resulted in no fixations of the outcrossing allele. The rarity of permanent selfing-to- outcrossing transitions, despite the possibility of secondary contact, can

26 thus be attributed partly to the transience of heterosis. The transience of heterosis is well known, though we did not initially ap- preciate what a strong barrier it would be to the evolution of outcrossing. Like inbreeding depression, heterosis becomes more depleted the more it is exposed. The equilibrium level of heterosis is lower when migration is greater because migration eliminates the allele-frequency differences underlying heterosis [Whitlock et al., 2000]. That is, gene flow depletes heterosis and with it the advantage of outcrossing. In fact, deleterious mutations accelerate genetic homogenization. This is because they provide a relative advantage to superior alleles from the other population, thereby increasing the frequency of migrant alleles through selection and thus increasing the effective migration rate through adaptive introgression [Bierne et al., 2002]. We only modeled gene flow from the ancestral populations into a focal population, not between the ancestral populations themselves. In nature, however, superior alleles could adaptively introgress between the ancestral populations, further reducing heterosis. But even transient increases in outcrossing rate could affect the distribution of mating systems in the biota. Despite the theoretical prediction that outcrossing rate should be bimodal [Lande and Schemske, 1985, Schemske and Lande, 1985], a substantial fraction of species examined are estimated to outcross at intermediate rates: so-called mixed-mating species [Goodwillie et al., 2005]. Mixed mating has been interpreted as either a stable equilibrium or as a transient step in the trajectory from predominant outcrossing to selfing [Igic´ and Busch, 2013]. Our model suggests a third possibility: mixed mating as a temporary hiatus from predominant selfing. It was not unusual for an outcrossing allele to segregate for over a hundred generations in our model. This is certainly short on a macroevolutionary timescale, but it could produce a constantly rotating class of mixed-mating populations. If so, populations in which mixed mat-

27 ing is observed should have a recent history of secondary contact. Fur- thermore, assuming secondary contact occurs on a per-population basis rather than species-wide, species with mixed-mating populations should also contain predominantly selﬁng populations that have not undergone secondary contact.

1.4.2 Robustness of model conclusions

Our focal scenario was designed to be very favorable to outcrossing. If outcrossing cannot evolve in this case, it cannot evolve under more stringent and realistic conditions. But since we had no reason to believe that the focal scenario was absolutely optimal for the evolution of outcrossing, we had to examine variants directly. We therefore relaxed the focal case’s assumptions (equal initial genotype frequencies, equal initial genotype fitnesses, no continuous migration, a dominant outcrossing mutation, an oligogenic load, and completely purged ancestral populations) one by one. First, the outcrossing allele fixed more often when it arose on an uncommon genetic background (Fig. 1.5), which could represent an outcrossing individual that arrived in a selfing population through a rare migration event. Second, the outcrossing allele fixed much less often when the two initial genotypes had unequal fitnesses, likely because the fitter haplotype was able to outcompete the less fit before the outcrossing allele could fix. Third, continuous migration from selfing donor populations slightly reduced the fixation probability for the outcrossing allele, possibly because migrant selfing alleles swamped out the mutation. Fourth, an additive outcrossing mutation was more likely to fix than a dominant one because its competitor, the selfing allele, was not masked when the additive outcrossing allele reached high frequencies. Fifth, polygenicity increased the fixation probability, but weakly deleterious fixed differences

28 did not greatly favor outcrossing even if there were many of them. Sixth, segregating deleterious alleles underlying unpurged inbreeding depression favored outcrossing much more than did an equal number of fixed differences between populations. A major point of departure between our model and others was that we assumed that heterosis had long accumulated in allopatry but was re- exposed en masse upon secondary contact. Other models of the evolution of heterosis [Whitlock et al., 2000, Bierne et al., 2002] or mating system [Theodorou and Couvet, 2002] assume some level of continuous migration. Sudden secondary contact is more favorable to outcrossing in some ways but less favorable in others. If migration is continuous, heterosis is limited by the amount of drift load that can stably persist in mutation-selection-migration-drift balance [Whitlock et al., 2000], whereas upon secondary contact after isolation, heterosis could start at a large but unstable level. Even our model of continuous migration assumed that heterosis began at an unstably high level, implying that migration only became continuous after a long initial period of isolation. Our model of unequal starting frequencies of the ancestral genotypes, however, showed that the outcrossing allele benefited slightly from being on a rare background. It seems that a small finite pulse of migration after long isolation combines the best of both worlds: the migrants can reap substantial heterosis from crossing with the residents, but since the deleterious alleles that the migrants bring are rare in the population, there is a more stable benefit for the migrants’ descendants to continue outcrossing. In the focal scenario, we assumed that the highly inbred mating systems of the parent populations would have eliminated the genetic variation underlying inbreeding depression. In nature, however, moderately or even highly selfing populations show inbreeding depression [Winn et al., 2011], which may sometimes be large [Herlihy and Eckert, 2002, 2004].

29 We found that the outcrossing mutation fixed far more often when there was unpurged inbreeding depression than when there was only local drift load composed of fixed differences (Fig. 1.7). This was likely because the main advantage of heterosis occurs in the first generation, when there is an opportunity to mask common recessive inferior alleles with dominant superior alleles from the other ancestral population. However, since these recessive deleterious alleles continue to be masked, they remain relatively common. Carriers for these alleles now receive relatively little advantage from outcrossing because a random mate is also likely to be a carrier. This contrasts with the architecture of inbreeding depression in which, since each deleterious allele is at low frequency, a random mate is unlikely to carry another copy. So long as it remains unpurged, inbreeding depression therefore provides a more lasting advantage to outcrossing. Heterosis may contribute to the advantage of outcrossing, but its effect seems to be marginal compared to that of inbreeding depression. Inbreeding depression can remain unpurged if individual deleterious alleles have weak enough effects to escape selection even after selfing exposes them [Charlesworth et al., 1990]. Also, inferior recessive alleles at loci in repulsion-phase linkage disequilibrium (pseudo-overdominance) resist purging because the superior double homozygote cannot be generated until the linkage disequilibrium is broken by recombination [Charlesworth and Willis, 2009]. An outcrossing population of Mimulus guttatus retained most of its inbreeding depression (0.46 remaining out of 0.57 initial) for lifetime fitness from germination to gamete production after five generations of artificially enforced selfing [Willis, 1999]. We used this magnitude estimate in our test of unpurged inbreeding depression. The oligogenic architecture we used (25 strongly deleterious loci, s = 0.14) likely could not have survived purging in the first place, but it does illustrate the essential difference between genetic loads composed of low- versus high-

30 frequency alleles. It should also be kept in mind that this large magnitude of inbreeding depression is close to the 0.5 threshold above which outcrossing is favored [Kimura, 1959] if inbreeding depression is static [but see Holsinger, 1988]. Some outcrossing populations like the one used for this example may retain so much unpurgeable inbreeding depression that, even if they underwent purging, they would not evolve toward predominant selfing. Comparisons among species, among populations, and among families within populations sometimes showed a negative relationship between selfing rate and inbreeding depression but showed no consistent overall pattern in one meta-analysis [Byers and Waller, 1999]. However, a subsequent meta-analysis focusing on experimental inbreeding studies did find reduction in inbreeding depression [Crnokrak and Barrett, 2002]. Overall, it appears that predominantly inbreeding populations have purged much of their inbreeding depression (which therefore cannot contribute to a reversion), but mixed-mating populations have retained it [Winn et al., 2011]. Although we considered some variant models, we have not been ex- haustive. The omission that most plausibly could favor a reversion is pollen discounting: a decrease in pollen success associated with a selfing phenotype. Greater pollen discounting reduces the transmission advantage, and complete pollen discounting totally eliminates it [Nagylaki, 1976]. Fur- thermore, since pollen discounting is simply a physical side-effect of a particular selfing phenotype, it is not dependent on genotype frequencies. Therefore, unlike heterosis, it will not be depleted by selection. We sus- pect our model without pollen discounting is nevertheless representative of at least some cases in nature. Zero or even negative pollen discounting has been observed in multiple studies of Ipomoea purpurea and Eichhor- nia paniculata, though other species (particularly those possessing multiple morphological correlates of selfing) showed positive pollen discounting

31 [reviewed in Busch and Delph, 2011]. A model incorporating pollen discounting would be similar to our own, except that the baseline advantage of outcrossing in the absence of heterosis would be greater. The same temporal pattern, a spike in the advantage of outcrossing followed by a return to the status quo, should occur regardless. We have also ignored outbreeding depression, the phenomenon in which inbred offspring (or within-population crosses) are fitter than outcrossed offspring (or among-population crosses) instead of the reverse. Outbreeding depression can be caused by local adaptation, negative epistasis, or underdominance. If outbreeding depression occurs, one or several of these factors has overwhelmed heterosis, and greater outcrossing is unlikely to evolve. Local adaptation contributes to outbreeding depression because, if each family or population is locally adapted to its own environment, crossing with distant genotypes maladapted to the local environment is a form of deleterious gene flow. Local adaptation occurs when the populations occupy different environments and is a strong predic- tor of outbreeding depression [Frankham et al., 2011]. Reversion should therefore be most likely when an initial vicariance event cuts off gene flow between the populations but does not otherwise alter their environments. Deleterious epistasis can also contribute to outbreeding depression. Each population can accumulate alleles that are neutral or beneficial on that population’s genetic background but deleterious on the other population’s background. An alternative phrasing is that each population accumulates coadapted gene complexes that break down in hybrids or admixed individuals: this is equivalent to negative epistasis because the non-coadapted alleles interact negatively relative to the coadapted alleles. It is easy to imagine deleterious epistasis accumulating with no limit short of complete hybrid inviability, and this process is hypothesized to beget speciation [Bateson, 1909, Dobzhansky, 1937, Muller, 1942]. Heterosis should

32 evolve as a function of the antagonistic effects of deleterious epistasis and local drift load, which should accumulate simultaneously. If deleterious epistasis gets too great relative to drift load, the populations will become permanently isolated. Since species do exist, reproductive isolation has apparently often overwhelmed heterosis in the long run, though heterosis may have won out in cases in which ephemeral species merge with the population from which they originated [Rosenblum et al., 2012].

1.4.3 Conclusion

Selfing-to-outcrossing transitions are expected to be rare because predominantly selfing populations lack much of the allelic diversity that normally favors outcrossing. Heterosis from local drift load circumvents this barrier because within-population variation is regenerated from among- population divergence. Outcrossing allows adaptive introgression, which increases offspring fitness but also eliminates the advantage of further outcrossing by homogenizing the originally differentiated allele frequencies. However, this advantage alone is ultimately too transient to allow reversions from predominant selfing to outcrossing in nature, consistent with the estimated rarity of such events. It may, however, marginally increase the probability of reversion if unpurged inbreeding depression is already great. Reversions, if possible, most likely occur in mixed-mating populations that have retained substantial unpurged inbreeding depression.

33 (a) (b) 0.08 8 0.06 6 p

∆ 0.04 4 0.02 Duration (generations) Duration 2 0.00 0.5 0.6 0.7 0.8 0.9 1.0 0.5 0.6 0.7 0.8 0.9 1.0

s s

Figure 1.1: Invasion condition for the outcrossing allele in the deterministic model. The population begins with equal frequencies of two viability genotypes, each fixed for a single inferior allele at a different locus. The outcrossing allele, M, initially exists only in Mm heterozygotes, at frequency 1/200. a M invades in the first generation (∆p > 0, dashed horizontal line) only when selection is sufficiently strong (large s for each viability locus). Despite this initial increase in frequency, M is lost in subsequent generations (not shown). b The transit time during which M is segregating before its loss increases with the strength of viability selection. The dashed vertical line shows the threshold s required for M to increase in frequency in the first generation (2/3), calculated from Inequality A.1. Extinction is defined as a frequency less than 1/200, and fixation as frequency greater than 199/200, to correspond to the loss of the final copy of an allele in a diploid population of 100 (like in the stochastic model, described later).

34 (a) (b)

free recombination, r = 0.5 reduced recombination, r = 0.1

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.6 0.8 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.5

0.6 0.4

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.3 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.4

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.2

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.1 0.2 Selection strength, s ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.0 Proportion with M fixed

10 20 30 40 50 10 20 30 40 50 Inferior alleles per population, LA Inferior alleles per population, LA

Figure 1.2: Outcrossing allele fixation proportions. Color indicates the proportion of simulations in which M fixed, for different combinations of number of viability loci (LA or LB) and selection strength at each of those loci (s). Dots mark parameter combinations for which fixation was more frequent than the neutral expectation. Panels a and b differ in the amount of recombination between the viability loci. Fixation proportion increases with the number of loci under selection, at a rate that depends on the selection coefficient. Fix- ation increases most steeply with number of loci for intermediate selection coefficients. High proportions of fixation can be achieved at lower levels of heterosis for more polygenic loads of loci of smaller effect. Loose linkage has little effect. For all parameter combinations, LA = LB.

35 LA = 5 loci LA = 25 loci LA = 50 loci

1.0 (a) (b) (c) 0.5 M dominant 0.0

1.0 (d) (e) (f) 0.5 M additive Outcrossing modifier frequency 0.0

−100 −75 −50 −25 0 −100 −75 −50 −25 0 −100 −75 −50 −25 0 Generations before fixation

Figure 1.3: Allele frequency trajectories of the allele, M. Each trajectory represents a single simulation run. To avoid clutter, only the first 10 runs longer than 10 generations resulting in fixation and the first 10 runs longer than 10 generations resulting in loss of M were plotted for each parameter combination. These trajectories therefore do not represent the fixation proportions (see text) or the frequency of very short simulations (see Fig. S2, S3). In each run, all individuals are initially homozygous for the selfing allele m, except that a single M mutation appears in a random individual in the first generation. Black trajectories resulted in fixation of M; red trajectories resulted in fixation of m. Time is measured relative to the final generation (the time of fixation or loss of M), called 0. Columns correspond to different numbers of viability loci. Rows correspond to different gene action of the modifier: dams that are heterozygous at the modifier locus outcross with probability 1 (top row) or 0.5 (bottom row). For all parameter combinations,

LA = LB and s = 0.3.

36 M extinct M extinct M extinct 600 600 600 400 400 400 Frequency Frequency Frequency 200 200 200 0 0 0 0 50 100 150 0.0 0.4 0.8 0.0 0.4 0.8 Duration Ultimate fitness Inbreeding depression

M fixed M fixed M fixed 600 600 600 400 400 400 Frequency Frequency Frequency 200 200 200 0 0 0 0 50 100 150 0.0 0.4 0.8 0.0 0.4 0.8 Duration Ultimate fitness Inbreeding depression

Figure 1.4: Simulation outcomes in the final generation. Duration, ultimate fitness, and final inbreeding depression were bimodal when M was lost and unimodal at intermediate values when M reached fixation. Parameter values were LA = LB = 50, and s = 0.3, and M was completely dominant.

37 ● ● ● ● ● ● ● 0.40

● 0.30 Proportion with M fixed ● 0.20 0.2 0.4 0.6 0.8 Ancestor frequency

Figure 1.5: Fixation proportion with initially asymmetric viability genotype frequencies. Simulations were run in which the outcrossing allele originated on a rarer or more common genetic background. These correspond to an ancestor frequency less than or greater than 0.5, respectively. The outcrossing allele is most likely to be lost when it appears on the background of a common genotype. Parameter values are s = 0.1, r = 0.5, and LA = LB = 50, and M is dominant. No ﬁxations occurred for LA = LB = 5 or 25 (not shown).

38 s = 0.1 s = 0.4 s = 0.7 1.0 s = 0.2 s = 0.5 s = 0.8 s = 0.3 s = 0.6 s = 0.9 0.8 0.6 0.4 0.2 Proportion with M fixed 0.0 0 20 40 60 80 100 120 ln(heterosis)

Figure 1.6: Effect of heterosis magnitude and architecture on fixation. Data are identical to those of Fig. 1.2a. Fixation generally increased with increasing heterosis. Comparing loads of equal magnitude, more polygenic loads (with smaller selection coefficients per locus) more often resulted in fixation than did more oligogenic loads.

39 ● 0 segregating sites per population 1.0 ● 25 segregating sites per population

● ●●●● ● ●● ●● 0.8 ●●● ●● ●●● ●●● ●● ●●● ●●● 0.6 ●● ● ●● ● ●

0.4 ● ●● ● ● ● ●

0.2 ●

Proportion with M fixed ● ●● ● ●●●● 0.0 25 30 35 40 45 50

Inferior alleles per population, LA

Figure 1.7: Effect of unpurged inbreeding depression. In this figure, LA refers to the total number of alleles diverging in the two ancestral populations, including both fixed and segregating sites. If there are segregating sites, each individual is homozygous for the inferior allele at a random 5 and homozygous for the superior allele at a random 20 out of the 25 total segregating sites unique to that ancestral population. If there are no segregating sites, those sites are replaced with fixed differences between the ancestral populations. For each parameter value, the fixation proportion is the average over 2000 replicate simulations. Parameter values are s = 0.14, r = 0.5, and M dominant for all simulations. The fixation proportion was greater when 25 sites were segregating and increased as the number of fixed differences increased. Replacing 25 fixed sites with segregating sites increased the fixation probability more than adding 25 fixed sites.

40 Predominant selfing and predominant outcrossing are alternative stable states. Traits that promote outcrossing are disadvantageous or less advantageous in predominantly selfing populations because of the low levels of inbreeding depression and the cost of forgoing the automatic transmission advantage of self-fertilization. High homozygosity exposes new deleterious recessive mutations and prevents further inbreeding depression from building up, but among-population divergence and heterosis remain possible. Heterosis provides an advantage to outcrossing in a selfing population, but as fitter alleles introgress, heterosis typically decreases too quickly for outcrossing traits to evolve in a predominantly selfing population. The evolution of outcrossing adaptations is therefore largely restricted to populations that have long remained predominantly outcrossing. These adaptations may become highly elaborate and may induce new selective pressures beyond the selection for outcrossing which favored them in the first place. In flowering plants, self-incompatibility induces negative frequency-dependent selection, which maintains great polymorphism at the locus controlling it. Negative frequency-dependence on novel incompatibility alleles could also explain how this locus originally became so diverse, but classic models of this diversification process are inconsistent with more recently discovered aspects of the molecular and functional genetics of self-incompatibility. In the following chapter, I develop a population genetic model showing that diversification of a ribonuclease- based self-incompatibility locus requires a rescue-like stochastic process that may alternatively lead to a substantial reduction in diversity.

41 Chapter 2

Diversiﬁcation or collapse of self-incompatibility haplotypes as a rescue process

Abstract

In angiosperm self-incompatibility systems, pollen with an allele matching the pollen recipient at the self-incompatibility locus is rejected. Extreme allelic polymorphism is maintained by frequency-dependent selection favoring rare alleles. However, two challenges result in a “chicken-egg” problem for the spread of a new allele (a tightly linked haplotype in this case) under the widespread “collaborative non-self recognition” mechanism. A

42 novel pollen-function mutation alone would merely grant compatibility with a nonexistent style-function allele: a neutral change at best. A novel pistil-function mutation alone could only be fertilized by pollen with a nonexistent pollen-function allele: a deleterious change that would eliminate all seed set. However, a pistil-function mutation complementary to a previously neutral pollen mutation may spread if it restores self-incompatibility to a self-compatible intermediate. We show that novel haplotypes can also drive elimination of existing ones with fewer siring opportunities. We calculate relative probabilities of increase and collapse in haplotype number given the initial collection of incompatibility haplotypes and the population gene conversion rate. Expansion in haplotype number is possible when population gene conversion rate is large, but large contractions are likely otherwise. A Markov chain model derived from these expansion and collapse probabilities generates a stable haplotype number distribution in the realistic range of 10–40 under plausible parameters. However, smaller populations might lose many haplotypes beyond those lost by chance during bottle- necks.

2.1 Introduction

Self-incompatibility (SI), a common strategy by which plants ensure outcrossing, is a classic example of extreme allelic polymorphism maintained by long-term balancing selection. An SI plant rejects self pollen, which is identiﬁed by a speciﬁcity phenotype encoded by a highly polymorphic self- incompatibility locus (S-locus). SI is widespread in plants: distinct non- homologous SI systems have been discovered in the Poaceae [Li et al.,

43 1997], Papaveraceae [Foote et al., 1994], Solanaceae [McClure et al., 1989], Brassicaceae [Stein et al., 1991], and Asteraceae [Hiscock et al., 2003], and some form of SI is thought to be present in nearly 40% of plant species across 100 families [Igic´ et al., 2008]. Rejection occurs when the pollen specificity matches the pistil specificity, which is likewise encoded by the S-locus. Pollen with a rare specificity has an advantage because it is less likely to encounter a pistil with a matching specificity and is thus less likely to be rejected. This advantage of rarity results in balancing selection, which maintains polymorphism at the S-locus by protecting S-locus alleles (S-alleles) from loss through drift [Wright, 1939]. While it is “fairly obvious that selection would tend to increase the frequency of any additional alleles that may appear” [Wright, 1939], it is not obvious how any additional alleles may appear. The origin of novel S-alleles is most mysterious in the “collaborative non-self recognition” SI system, whose molecular mechanism has been recently unraveled. Collaborative nonself-recognition, which is described below in the section of the same name, appears inhospitable to novel S-alleles for two reasons. First, because a style with a novel specificity suffers from pollen limitation, and because pollen capable of fertilizing non-existent stylar specificities experience no siring advantage, the spread of a novel allele presents a “chicken-or-egg” problem. Second, we show that if a new S-allele can invade, pollen types incapable of fertilizing it risk extinction because of their siring disadvantage, and therefore a new S-allele can lead to a collapse in the number of S-alleles, rather than an expansion. We develop a population genetic model of the expansion, collapse, and long-term evolution of S-allele number under this widespread SI system, and show when and how these challenges can be overcome, providing many testable hypotheses for the diversification of S-alleles in this system. While the SI system originally described in Nicotiana [East and Man-

44 gelsdorf, 1925] is particularly well-studied, a molecular genetic understanding of the “collaborative non-self recognition” in this system has only been elucidated recently. Counts of 10–28 alleles have been directly observed in several other species in Solanaceae, and Physalis crassifolia has been estimated to harbor as many as 44 [Lawrence, 2000]. In this SI system, each haploid pollen grain carries one allele at the S-locus, and the pollen is rejected if its allele matches either of those carried by the diploid pollen recipient [East and Mangelsdorf, 1925]. It is classiﬁed as a form of gametophytic SI (as opposed to sporophytic) because the pollen’s phenotype is determined by the genotype of the haploid male gametophyte (i.e., the pollen’s own genotype) as opposed to the genotype of the diploid sporophyte plant that produced it. The SI systems in the Plantaginaceae and the distantly related Rosaceae have been found to be homologous to that in the Solanaceae, which implies that this system was present in the common ancestor of the asterids and rosids [Igic´ and Kohn, 2001, Steinbachs and Holsinger, 2002], estimated to have lived about 120 mya [Tank et al., 2015]. We refer to it as the collaborative nonself-recognition system after its hypothesized mechanism [Kubo et al., 2010] to distinguish it from other, non-homologous gametophytic SI systems but note that this mechanism has only been deﬁnitively demonstrated in Petunia, and recognize that even homologous systems may make use of distinct molecular mechanisms. Nonetheless, S-alleles appear to be very long-lived, consistent with long-term balancing selection. A given allele is often more closely related to an allele in another species or even another genus than it is to other alleles in the same population, which is consistent with the alleles having persisted since the common ancestor of those species or genera [Igic´ and Kohn, 2001]. Understanding the forces shaping allelic diversity within and across species requires theory that incorporates our modern molecular genetic understanding of the structure and function of the S-

45 locus.

2.1.1 Conceptual challenges

A series of influential models investigated how a system of many incompatibility alleles could arise, but only very recently have theorists incorporated assumptions reflecting the contemporary understanding of the molecular genetics underlying SI. Under the simplest self-recognition system of SI, Wright [1939] showed that each fully functional S-allele is under negative frequency-dependent selection because pollen carrying rare alleles is compatible with a larger fraction of the population. As such, the number of S-alleles is limited only by their rate of input by mutation, and their stochastic loss from a population by drift. Charlesworth and Charlesworth [1979] showed that in SI system in which a single mutation can generate a new specificity, novel functional S-alleles can invade when inbreeding depression is high. However, single-mutation models of incompatibility are inconsistent with the separate pistil- [McClure et al., 1989] and pollen- expressed [Lai et al., 2002, Entani et al., 2003, Sijacic et al., 2004] products that were later discovered. Uyenoyama et al. [2001] and Gervais et al. [2011] showed that if a new specificity requires separate mutations in a pollen-function and a tightly linked pistil-function locus, new S-alleles can be generated through an self-compatible (SC) intermediate, but one or more S-alleles are also lost for many parameter combinations. Sakai [2016] showed that if each pollen-function mutation is more likely to be rejected by the pistil-function allele that was on its initial genetic background, then the number of S-alleles can increase from zero into the range observed in nature. A critical common element among all these models is the assumption that SI functions through self-recognition. In a self-recognition system,

46 such as the SCR/SRK system found in the Brassicaceae [Kachroo et al., 2002], pollen is accepted by default, but self pollen carries a product that induces rejection in the maternal plant. However, the SI system in question functions not through self-recognition, but rather through a “collaborative nonself-recognition mechanism” [Kubo et al., 2010]. In a nonself- recognition system, pollen is rejected by default, but cross pollen carries a product that allows it to bypass rejection. The crucial difference from a theoretical perspective is in the behavior of a mutant that expresses a new, unrecognized speciﬁcity. A new speciﬁcity would be compatible with existing phenotypes under self-recognition (it is new, but still not self), whereas it would be incompatible with existing phenotypes under nonself- recognition (it is no longer recognized as nonself). Results from models of self-recognition systems might therefore not be directly applica- ble to nonself-recognition. Despite the ubiquity of collaborative non-self recognition-based SI, few models of S-allele evolution explicitly consider this system. Fujii et al. [2016] proposed that a pollen-function mutation complementary to a novel pistil-function mutation could arise on a single background and then spread to other backgrounds through gene conversion. Bod’ova´ et al. [2018] enumerated several possible evolutionary pathways to new S-alleles and demonstrated that the pathway hypothesized by Fujii et al. [2016] could either expand or reduce S-allele number. An explanation of the long-term trajectory of S-allele number must therefore account not only for the steps needed to create new fully functional alleles but also for dynamics that can lead to their loss. The stochastic process involving the risk of collapse, which follows from the collaborative non-self recognition system, raises radically different questions than one where drift is the only force opposing continual growth. Are S-allele numbers repeatedly expanding and contracting, or do they reach equilibria where changes are rare? We might sometimes

47 observe changes in progress if they are frequent, but infrequent changes might be buried in the past. When a contraction occurs, is it usually small or catastrophic? Large contractions, followed by re-expansion, should result in more turnover in alleles and fewer shared alleles among species than would small contractions. Since contraction is possible whether SI is maintained throughout [Bod’ova´ et al., 2018] or is temporarily lost in SC intermediates [Uyenoyama et al., 2001, Gervais et al., 2011, Bod’ova´ et al., 2018], neither pathway is unidirectional. Is the pathway maintaining SI still sufficient for expansion, or are SC intermediates necessary? If new proto- S-alleles are segregating in natural populations, are they self-compatible, self-incompatible, or both? Answers to these questions will aid interpretation of S-allele genealogies, which document both similarities and differences between species in their complement of S-alleles. For example, the large historical contraction in S-allele number in the common ancestor of Physalis and Witheringia has been interpreted as a demographic bottleneck [Paape et al., 2008], but a firmer theoretical investigation could assess whether a collapse restricted to the S-locus is a plausible alternative explanation. We find that diversification and collapse of S-alleles result from a rescue process similar to evolutionary or genetic rescue of a population. To approximate the relative probabilities of collapse and expansion in S-allele number as well as the distribution of collapse magnitudes, we therefore expand the analogy of evolutionary rescue [Orr and Unckless, 2008] to model the extinction of haplotypes and to incorporate the frequency-dependent dynamics associated with SI. We find a negative relationship between initial S-allele number and the probability of further expansion. By construct- ing and iterating a Markov chain out of these contraction and expansion probabilities, we find stable long-term distributions of S-allele number. We show that for large but plausible values of population size and rate of gene

48 conversion, this process can generate numbers of S-allele numbers comparable to those found in nature (20–40). However, contractions can be very large when they occur, often eliminating the majority of S-alleles. These results suggest that an evolutionary opportunity to expand the number of S-alleles is intrinsically connected to the risk of a contraction in S- allele number. Our results yield numerous novel predictions. First we predict that novel stylar rejection alleles will arise on SC haplotypes which are maintained by mutation-selection balance. Second, we predict that, when many but not all S-alleles are fortuitously capable of fertilizing plants carrying the novel S-allele, the S-alleles lacking this capability are very likely to be lost. This is because the alleles possessing this fortuitous compatibility eliminate their competitors when a novel speciﬁcity arises, thereby reducing the total number of surviving alleles. However, the fortuitously compatible S-alleles themselves are never expected to be lost at equilibrium. The joint effect of increasing the number of fortuitously compatible S-alleles is to increase the probability that some S-alleles are lost but also to increase the minimum number of surviving S-alleles. Third, we predict that intraspeciﬁc variation in S-allele number can sometimes be explained by an invasion of a runaway SI allele rather than a bottleneck or a transition to SC. Coun- terintuitively, we therefore predict that when allopatric populations exhibit large disparities in S-allele number, the population with the smaller number of S-alleles will harbor a stylar allele absent from and incompatible with pollen from the population with more S-alleles, while all S-alleles from the population with more S-alleles can be fertilized by pollen in the population with fewer S-alleles.

49 2.2 Model and Results

2.2.1 Collaborative Nonself-Recognition

The risk of contraction in S-allele number in the “collaborative nonself- recognition” incompatibility system is rooted in the system’s biological details. This system involves two kinds of complementary products: pistil- expressed ribonucleases (RNases) [McClure et al., 1989] and pollen-expressed F-box proteins [Lai et al., 2002, Entani et al., 2003, Sijacic et al., 2004]. To- gether, these products achieve a form of collaborative nonself-recognition ﬁrst described by Kubo et al. [2010]. Under this system, each “allele” at the S-locus is actually a tightly linked haplotype containing one RNase gene and a collection of paralogous F-box genes. The RNase gene is highly polymorphic, and each functionally distinct haplotype possesses a different RNase allele. Nonself pollen is recognized through the complement of F-box proteins it expresses. Each F-box paralog produces a functionally distinct product, and each of these products is capable of detoxifying one or more forms of RNase. Pollen is only successful if it expresses both of the two F-box proteins to match the diploid pollen recipient’s two RNase alleles. Self-fertilization is prevented because each fully functional haplotype lacks a functional copy of the F-box gene that corresponds to the RNase on the same haplotype, and so each pollen grain necessarily lacks one of the two F-box proteins required to fertilize the plant that produced it. Rejection is not restricted to self-pollination: any two individuals that share one haplotype will reject half of each other’s pollen, while individuals that share both haplotypes are completely incompatible. Rejection is therefore more likely between closely related individuals. Tight genetic linkage across all components of the S-locus reduces the probability that recombination will cause a haplotype to lose a functional F-box paralog

50 (reducing its siring opportunities) or gain the F-box paralog that detoxifies its own RNase (inducing SC). This system presents two novel challenges that are absent in self- recognition systems. The first is a “chicken-egg problem.” A novel F-box specificity alone is at best neutral because it detoxifies an RNase that does not yet exist. A novel RNase alone is deleterious because it degrades all pollen and renders the plant ovule-sterile. Both of these mutations must invade in order to generate a new, fully functional S-haplotype. Second, for every novel RNase that arises, the corresponding F-box specificity must appear on every other haplotypic background in order to restore cross- compatibility among all haplotypes. Building on the hypothesis for expansion of haplotype number proposed by Fujii et al. [2016], Bod’ova´ et al. [2018] showed how cross-compatibility could be restored among all incompatibility classes after novel RNase and F-box mutations have invaded. If an initially neutral F-box mutation already exists when its complementary RNase mutation arises, the F-box mutation will then invade because it confers the advantage of compatibility with the new RNase. To restore full cross-compatibility among haplotypes, all haplotypes must acquire F-box paralogs complementary to all RNase alleles other than their own either through gene conversion [Fujii et al., 2016] or recurrent mutation [Bod’ova´ et al., 2018]. This means that the haplotype bearing the RNase mutation must also acquire the F-box complementary to its ancestor. Once this occurs, the resulting haplotype is compatible with pollen recipients carrying all other haplotypes, but haplotypes still lacking the new F-box are not compatible with pollen recipients carrying the new RNase. As the RNase mutation increases in frequency, siring opportunities decrease for haplotypes that still lack the new F-box, and they are gradually driven extinct. If all doomed haplotypes acquire their missing F-box before they are lost, expansion has occurred. But if some doomed haplotypes go extinct, their

51 RNase alleles are lost and contraction has occurred. Bod’ova´ et al. [2018] simulated the dynamics of this process along with several other expansion pathways and found that up to 14 haplotypes could be maintained. Based on this biological background, we introduce a set of metaphors to make the interactions among haplotypes more intuitive. Each form of pistil-function RNase is a lock, and each form of pollen-function F-box protein is a key. A diploid plant codominantly expresses two different locks in its pistils. Each pollen grain expresses every paralogous key in its haploid genome, and these keys collectively form that pollen grain’s key ring. The pollen must unlock both of the pollen recipient’s locks in order to fertilize it, which requires keys to both locks. Each fully functional haplotype is SI because its key ring lacks the key to the lock on the same haplotype (Fig. 2.1). In contrast to Bod’ova´ et al. [2018] our model addresses the realistic possibility that novel RNase alleles (locks) will have ovular ﬁtness below the population average because keys to this lock are initially rare. Thus, while Bod’ova´ et al. [2018] assumed the novel RNase was neutral, we model the case when this RNase suffers more intense pollen limitation than the rest of the population. How could such a stylar mutation, which increases pollen limitation, increase ovular ﬁtness (as is required for its adaptive spread)? We suggest that novel locks can be favored on self-compatible genetic backgrounds otherwise maintained by the balance of selection against self-compatibility and gene conversion of F-box alleles which restore self-compatibility. By restoring SI and preventing self- fertilization, a new lock can provide an ovular advantage and can spread by natural selection. Concurrently, the complementary key spreads within and among haplotypes (“key rings”), and the number of S-allele increases if all S-alleles are “rescued” before S-haplotypes lacking this novel key are driven to extinction due to their mating disadvantage.

52 2.2.2 Model outline

We identify six haplotype classes with regard to their phenotypes expressed in pollen and pistils (Fig. 2.2), and we evaluate a three-step evolutionary pathway to expansion or contraction of S-haplotype number (Fig. 2.3). We define “contraction” as any transition to a state with fewer lock alleles. We define “expansion” as the transition from an initial state in which all haplotypes are cross-compatible with all other haplotypes to a final state that is the same but with one additional cross-compatible SI haplotype. In Step 1, the “conversion” step, a haplotype acquires the key to its own lock through gene conversion, rendering it SC. This ensures that when a lock mutation occurs on this haplotype, the resulting mutant will already possess every key but its own and will be cross-compatible as a sire with all other haplotypes as soon as it arises. A supply of such SC haplotypes is maintained at the balance between gene conversion and strong selection against inbreeding. The invasion and fixation of an SC haplotype should function identically under self- and nonself-recognition, and this process has already been thoroughly covered by Uyenoyama et al. [2001] and Gervais et al. [2011]. We return to it briefly below. However, we focus on cases in which inbreeding depression is strong enough that SC is at a net disadvantage—ours is a model of diversification of S-haplotypes, not the transition to SC. For such cases, there is no risk that the population will become entirely SC: the only possible outcomes are increase, decrease, and stasis in the number of SI haplotypes. In Step 2, the “mutation” step, the lock allele on the SC intermediate haplotype mutates to a new specificity, one that can be unlocked by an existing, previously neutral key variant. The new lock is a necessary component of a new haplotype. The SC intermediate previously had the key to every lock, but now that its own lock has mutated, the resulting mutant has

53 the key to every lock but its own and is once again SI. The pre-existence of a complementary key on other haplotypes ensures there is at least some pollen compatible with the new lock. However, in the likely event that the complementary key is present on only a subset of other haplotypes, the new lock will reject more pollen than other locks. If the amount of compatible pollen received is a limiting factor on seed set, the lock mutation might reduce ovule success. But since the mutant is compatible as a sire with all other haplotypes (thanks to the “conversion” step), its increased pollen success can compensate for lost ovule success. This might seem to be asking a lot of nature in this step – the mutation to a new lock coinciden- tally occurs on a rare SC haplotype, and the lock mutation’s matching key already exists in the population despite being previously neutral. However, these events appear less contrived when we consider the alternatives and realize that there is always an opportunity for these rare events to occur. If the lock mutation occurs on an SI haplotype, the resulting mutant will derive no advantage because it was already SI, but it will still suffer additional pollen limitation. If the lock mutation lacks a complementary key, it will reject all pollen and suffer complete ovule failure. Such mutations should surely occur, but they should remain at low frequencies because they are strongly deleterious. Since there is a constant supply of SC haplotypes at gene conversion-selection balance, the population can wait indeﬁnitely until one such haplotype mutates to a lock with a pre-existing key. Finally, in Step 3, the “rescue/collapse” step, the remaining haplotypes lacking the key to the new lock are driven to extinction unless they can acquire the missing key through gene conversion in their remaining time. This restores mutual cross-compatibility among all surviving haplotypes, whether there are more (expansion) or fewer (contraction) than the initial number. The six relevant haplotype classes are described in Fig. 2.2. Table 2.1

54 gives notation for all parameters and variables. There are initially n lock alleles, but diversity arises from which keys each haplotype possesses.

The ﬁrst haplotype class consists of nL haplotypes that possess one of the initial n locks and a key ring that is “complete” [Bod’ova´ et al., 2018]. That is, they possess the keys to all extant locks but their own, including the mutant lock. This ﬁrst class is denoted L for “lucky” because its members fortuitously possess the key to the new lock. Each haplotype in this class initially occurs at frequency dL/n, where the parameter dL is an arbitrary initial proportion. In nature, the parameter nL would be determined by the extent of gene conversion that has already occurred: if the neutral novel key has been copied from its original background to many others, then nL is high. Second, for each of these nL complete haplotypes, there exists one haplotype that is identical except that it lacks the key to the new lock. These could represent ancestral versions of the complete haplotypes that have not yet gained the key to the new lock. This second class is denoted C for “chump” because members of this class are simply inferior versions of the corresponding “lucky” haplotypes. Each haplotype in this class occurs at frequency (1 − dL)/n. Third, there are nU haplotypes that have one of the initial n locks and an incomplete key ring that lacks only the key to the mutant lock. Unlike the previous class, there exists no complete version of these haplotypes initially. This third class is denoted U for “unlucky” because members of this class cannot unlock the new lock like the “lucky” haplotypes can, but neither must they compete with a superior version of themselves like the “chump” haplotypes must. Each haplotype in this class occurs at frequency 1/n. Fourth, there is the ancestral haplotype from which the SC intermediate arose. It is like any other incomplete haplotype (e.g., a U haplotype), except that it has the same lock as the SC intermediate. This fourth class is denoted A for “ancestral” because its lock is ancestral to that of the lock mutant. It occurs

55 (0) (0) (0) at frequency 1/n − pI − pM , where pI is the initial frequency of the SC (0) intermediate, and pM is the initial frequency of the lock mutant. Fifth, there is the SC intermediate. Pollen carrying this haplotype is compatible with all plants, and pistils carrying it reject only the “A” haplotype. This fifth class is denoted I for “intermediate” because it is the intermediate between the ancestral class and the new SI haplotype to be generated. Note that gene conversion should generate some SC haplotypes from all classes, not just Class A. We do not name these haplotypes because all SC haplotypes should remain at low frequency, and we only name Class I because it is the background on which the lock mutation arises. We lump all other SC haplotypes into the class from which they originated (L or U). Sixth, there is the haplotype bearing the lock mutation. It is identical to the intermediate haplotype, except that it has a novel lock and is thus SI. It is denoted M for “mutant” because of its novel lock mutation. This haplotype increases its own fitness over its SC predecessor by preventing selfing, but it also decreases the fitness of the “unlucky” and “chump” haplotypes by present- (0) ing a lock to which they lack the key. It occurs at initial frequency pM There are thus nL + nU + 1 = n pre-existing haplotypes (including the mutant’s ancestor) along with the mutant itself for a total of n + 1 haplotypes. Throughout out model, we assume that selection dominates random genetic drift, so we use deterministic equations to describe the change in haplotype frequency by selection. As such, the frequency of the SC intermediate after Step 1 (Fig. 2.3), “conversion,” is controlled by the deterministic equilibrium between selection and gene conversion, and the decay of doomed haplotypes after Step 2, “mutation,” occurs through deterministic genotype frequency trajectories (see Appendix B). However, our model includes numerous stochastic forces that play a critical role in mediating the diversification or collapse of S-allele diversity. The first event we model as a random variable is the distribution of compatible pollinations

56 on the lock mutant after Step 2. This distribution is used to demarcate parameter values in which compatible pollen supply does or does not limit ovule success. Most critically, we model the probability of survival of new gene convertants in Step 3, the “rescue/collapse” step, as a stochastic process. Specifically, we use the branching processes approximation of Fisher [1923] and Haldane [1927] to approximate this survival probability based on the fitness benefit of the convertant, though we check the accuracy of this approximation through stochastic simulation. The third and final random variable is the number of surviving haplotypes after rescue/collapse, which is itself calculated from the survival probability and the expected supply of gene convertants. We use this distribution to parame- terize a transition matrix, and iterate this as a Markov Chain to determine the long-run stable distribution of haplotype number numerically. Though we do not model long-term drift in allele frequencies (only the short-term stochastic loss of new alleles), we nevertheless refer to a population size parameter in order to convert the proportion of gene convertants into a whole number of copies (the relevant quantity for rescue probability). If the long-term number of haplotypes predicted by Bod’ova´ et al. [2018] is limited by the supply of gene convertants in the small populations that can feasibly be simulated, a predominantly deterministic approach may allow us to examine populations with greater supplies of gene convertants and possibly greater long-term haplotype numbers. All deterministic iterations and stochastic simulations were written in R [R Core Team, 2020] and additional packages [Wickham et al., 2019, Warnes et al., 2020, Neuwirth, 2014, Wickham, 2011, Douglas Nychka et al., 2017, Garnier, 2018, Xie, 2020]. Mathematica [Wolfram Research, Inc., 2020] was used to solve for equilibrium frequencies numerically (see below). Scripts and their data outputs are deposited on Dryad [Harkness et al., 2020a].

57 2.2.3 Equilibrium frequency of SC intermediate

Step 1, the “conversion” step, consists of the generation and equilibra- tion of the SC intermediate (Fig. 2.3). The SC intermediate provides the advantage of compatibility with other individuals carrying the same lock: this advantage can counteract pollen limitation when the SC intermediate later acquires a new lock by mutation. However, SC also carries the disadvantage of allowing some self-fertilization. We are not interested in the situation in which SC haplotypes invade and fix in the population. We therefore focus on cases in which SI is maintained by extreme inbreeding depression: specifically, all selfed offspring are eliminated before they can reproduce. In this case, for many values of S-haplotype number and primary selfing rates, the SC intermediate is maintained at low frequency by the balance between gene conversion and selection. What is this frequency? We determine conversion-selection balance of the SC intermediate using a simple model. A population contains n SI haplotypes that are all mutually cross-compatible. Since all haplotypes are cross-compatible, a diploid’s maternal and paternal haplotype carry the key to the other’s lock. This state does not normally result in SC pollen because, though the diploid possesses keys to both of its own locks, these keys are on separate haplotypes. The absence of recombination at the S-locus then ensures that none of the plant’s pollen carries both keys. However, gene conversion allows a lock to be copied from one haplotype to another, potentially generating an SC haplotype from any diploid carrying at least one SI haplotype. In this population, gene conversion in zygotes generates SC haplotypes at rate µ. Individuals carrying two SC haplotypes self at rate σ, and those carrying one self at rate σ/2. We model extreme inbreeding depression such that all selfed offspring are inviable, and so the selfing

58 rate is equivalent to a fecundity penalty. However, SC haplotypes gain an advantage in pollen. Assuming all individuals are heterozygous at the lock n−2 locus, SI haplotypes are rejected with probability n , whereas SC haplotypes are never rejected because they carry all keys. If SC haplotypes are rare, the expected change in frequency of all SC haplotypes, p, is

! 1 − σ 1 ∆p = p 2 + − 1 − µ + µ (2.1) 2 (1 − σp) 2 2 1 − n (1 − p) If n, µ, and σ are treated as constant parameters, ∆p is a cubic function of p. We determine the equilibrium frequency by setting ∆p = 0 and solving numerically using Mathematica’s NSolve function [Wolfram Research, Inc., 2020]. We solve for the equilibrium frequency of SC haplotypes, varying µ from 10−4 to 10−3 in steps of 10−4, σ from 0.1 to 1 in steps of 0.1, and n from 3 to 40 in steps of 1 (Fig. 2.4). The biologically meaningful domain of ∆p is the real numbers between zero and one, so we ignore solutions outside this interval. We ﬁnd two broad categories of outcomes depending on the parameter values. First are cases in which ∆p is positive from p = 0 to p = 1, and there are no internal equilibria. In such cases, SC haplotypes are expected to rise to ﬁxation despite intense inbreeding depression. This occurs when SC confers a large pollen advantage because n is small, or when it in- curs a small ovule disadvantage because σ is small [Charlesworth and Charlesworth, 1979, Uyenoyama et al., 2001]. Second are cases in which there are two solutions between p = 0 and p = 1. In these cases, the smaller solution is stable, and the larger is unstable. The unstable solution tends to be fairly large, usually between 0.1 and 0.5, so we warn that predicted frequencies of unstable equilibria are likely imprecise. Because this internally unstable equilibrium cannot be approached by any biological process, we report and investigate only the stable solution. Stable equilib-

59 ria for µ = 0.0003, with σ from 0.1–1.0, and n from 0–40 are plotted in (Fig. 2.4).

2.2.4 Invasion of lock mutant

With the existence of SC haplotypes at conversion-selection balance, we can generate a new SI haplotype through mutation. If an SC haplotype’s lock mutates to a novel specificity, it will no longer have the key to its own lock, making it SI. It will, however, possess the key to every other lock, making it universally cross-compatible as a sire. The SC haplotype therefore acts as an SC “intermediate” haplotype I to the new SI “mutant” haplotype M (Fig. 2.2). As Bod’ova´ et al. [2018] showed, competition between complete haplotypes possessing all others’ keys and incomplete haplotypes lacking some keys can result in extinction of the incomplete haplotypes. This occurs because the incomplete haplotypes have pollen success below the population average. In our model, two haplotypes possess the keys to all other locks: the intermediate I and mutant M, haplotypes. These haplotypes are compatible as sires with every other haplotype, but only a subset of haplotypes are compatible with them. In I, the siring advantage partially compensates for the loss of ovule success through selfing, resulting in a smaller but still deleterious net effect on fitness. The mutation from I to M restores SI while maintaining the siring advantage. However, this mutation also potentially increases pollen limitation because the new lock rejects the majority of pollen in the population. In order for M to invade and for the population to progress to the rescue/collapse step, M must have fitness above the population average despite the tradeoff of pollen limitation. We use a simple model of pollen limitation to determine if M will invade when rare. As SC haplotypes are rare and of fitness below the population

60 average, so the invasion of M depends only on whether M is superior to L and U haplotypes. We assume L, U, and M haplotypes are equally efficient at rejecting self pollen, and so the quantity of self pollen is irrelevant in determining these haplotypes’ fitnesses relative to each other, and since self pollen acts simply to ‘waste’ ovules, we do not consider self- pollen as a source that alleviates pollen limitation. For simplicity, we assume that each flower carries 10 ovules, and receives a number of pollen grains equal to the parameter G. The proportion of G that is compatible with the maternal flower is a binomial random variable, but the expectation of this variable is sufficient to determine fitness, so we treat ovule success as a deterministic function of the proportion of the compatible pollen. The proportion of pollen that is compatible with the maternal flower is nLdL/ (nL + nU ) for UM genotypes, dL (nL − 1) / (nL + nU ) for LM genotypes, and (nL + nU − 2) / (nL + nU ) for all other genotypes. Fertilization occurs through the following process: one ovule subtracts one compatible pollen grain from the flower’s stock of compatible pollen to produce a seed, then the next ovule does the same, and this process continues until either ovules run out (full seed set) or compatible pollen runs out (partial seed set). We therefore define a flower’s ovule success as min(ovules, compatible pollen)/(ovules). The ovule success of M is always less than or equal to the ovule success of L and U because a smaller proportion of pollen is compatible with it. The net fitness of M also depends on its pollen success. When M is rare, M pollen is almost never rejected because it is compatible with all non-M plants. We therefore define the pollen success of M as 1, the maximum value. If M is rare, all L and U haplotypes have a pollen success of approximately (n − 2) /n, the probability that the pollen matches neither of a random maternal plant’s haplotypes. We then divide the ovule and pollen success of M by the mean values of pollen and ovule success

61 (approximated by the success of L or U, which are equivalent when M is rare) to get a measure of its fitness relative to the mean. The M haplotype invades if half the sum of these quantities, a measure of relative fitness, is greater than one. We calculate this relative fitness measure for nL = 2,

G = 100–1000, dL = 0.01, 0.1, or 0.5, and nU = 3, 5, or 7 (Fig. 2.5). We find that the relative fitness of M increases as G or dL increases, and it decreases as nU increases. Depending on the choice of dL and nU, M might have fitness below the population average throughout G = 100– 1000, fitness above the population average throughout, or there might be a threshold value of G within 100-1000 above which M is favored. We note that if M is disfavored, this novel specificity goes extinct, and we await a mutation to another novel specificity (or a recurrent mutation to this specificity which could be favored if the number of “lucky haplotypes” is, by chance, higher). As such, with every novel specificity, this process repeats itself until one increases in frequency.

2.2.5 Rescue of doomed haplotypes

Numerical iteration of genotype frequencies when M is beneﬁcial shows that M will rise to high frequencies and eliminate every haplotype that lacks the key to the new lock (Fig. 2.6). These lost haplotypes include all of the U haplotypes along with the lock alleles they carry. Therefore, the overall process results in a collapse in haplotype number at equilibrium rather than expansion. This is consistent with the deterministic analytical results of Bod’ova´ et al. [2018] which showed that without mutation and with more than seven S-alleles, a novel female incompatibility would always decrease the number of S-alleles when not all haplotypes were initially complete. Multiple haplotypes are now doomed to extinction at equilibrium. How-

62 ever, if a doomed haplotype acquires the key to the new lock before equilibrium is reached, the resulting haplotype’s fitness increases with the frequency of the new lock. This should protect the resulting gene convertant from being driven from low frequencies to extinction. It might therefore be possible for doomed haplotypes to be rescued by gene conversion. If all haplotypes are rescued before any one of them is lost, then the number of lock alleles at equilibrium is one greater than the initial number, and expansion has occurred. Bod’ova´ et al. [2018] previously noted the role of a rescue-like process in this pathway, though they modeled it through recurrent mutation rather than gene conversion. The underlying mathematics of this rescue process are similar in form to the evolutionary rescue of a declining population by new mutation. The origin of a new lock is a kind of change in the (genetic) environment, the declining frequency of a doomed haplotype is analogous to a declining population, and the acquisition of the new key by the doomed haplotypes is analogous to the generation of new beneficial mutations. We therefore base our model on Orr and Unckless’ (2008) model of evolutionary rescue by new mutations. There are two components of this model: the supply of beneficial mutations and the probability that any one of them will survive. The probability that the population is rescued is the probability that at least one of these mutations survives. Similarly, the probability that a given haplotype is rescued is the probability that at least one copy adds the new key to its key ring and survives. Orr and Unckless [2008] used a model of survival probability in a population of decreasing size [Otto and Whitlock, 1997], which is itself a modification of the branching-process model used by Fisher [1923] and Haldane [1927]. However, we modify the original model in a different way because the declining frequency alters the frequency-dependent fitness of the potentially rescuing gene convertants and thus the survival probability of a new mutant increases over time.

63 Thus, rather than varying the population size over time, we determine the time-dependent selection coefficient and the input of new convertants as a function of genotypic frequencies. In the original model [Orr and Unckless, 2008], selection for the rescuing mutation is unconditionally positive. Survival thus implies eventual fixation, and the survival probability equals the fixation probability. In contrast, SI involves balancing selection among haplotypes, so survival does not imply fixation. But since the branching-process approximation is actually the probability of surviving early loss by drift, it can be used in either case. Haldane [1927] defined s as the number of additional offspring produced by the mutant haploid individual above the population average. We model survival of a haplotype and its descendants rather than a diploid individual, so we retain Haldane’s assumption of haploidy. In our model, s can be re-interpreted as the additional number of descendant copies of a gene convertant above the average for all haplotypes. This advantage s depends on the changing frequency of genotypes that accept the gene convertant but reject its ancestor. The survival probability for a single gene convertant is therefore also frequency-dependent, which complicates the calculation of the probability that at least one gene convertant survives. In the model of Orr and Unckless [2008], the population declines at a predictable rate determined by a standard model of negative population growth. This allowed them to express the number of mutations per generation as a function of time. We cannot do the same with the number of gene conversions per generation because we do not have an explicit formula for genotype frequencies as a function of time. We can, however, numerically iterate recursion equations (see Appendix B) to get a trajectory of genotype frequencies. We can then retroactively calculate the expected number of gene conversions each generation in this hypothetical history as a product of the per-individual gene conversion rate, the total number

64 of individuals, and the frequency of heterozygotes for the key to the new lock. The relevant gene conversion parameter is not the per-nucleotide probability of gene conversion. We are interested in a speciﬁc gene conversion event: the copying of a key to a haplotype that did not previously possess it, while maintaining SI. The relevant quantity is the rate of these gene conversions, which would be expressed as the number of such gene conversion events per individual per generation. Furthermore, we use this rate only to determine the expected number of haplotype copies that gain a key each generation, which is the product of the rate of these gene conversions and the population size. We therefore report this product as a single compound parameter Rconversion, the expected number of key-gain gene conversions per generation in the whole population. There are other gene conversion events in nature not captured by this Rconversion. First, gene conversion might replace a functional F-box paralog with a non-functional or deleted allele: i.e., it might remove a key. Such a conversion event should reduce the siring success of the haplotype and be unconditionally deleterious. We do not track these gene conversions in any step of the model. Second, gene conversion might induce SC by uniting a key with its complementary lock, as occurred in the previous step at rate µ. We do not track these gene convertants in this step because they should be maintained at low frequency at conversion-selection balance. We also do not track gene conversion events that do not affect pollen speciﬁcity at all.

The parameter Rconversion is therefore smaller than the raw rate of functional gene conversion, which also includes the production of neutral convertants and short-lived deleterious convertants. The probability of rescue depends on the probability of survival of a new gene convertant. We use deterministic expectations for the genotype frequency trajectory and the supply of potentially rescuing gene conver-

65 tants each generation. All stochasticity in our model therefore arises from whether each potentially rescuing convertant actually survives. We numerically approximated the survival probability of a new gene convertant using ∞ i−1 X P (survival) s = (2.2) i i from Haldane [1927]. The survival probability of a hypothetical gene convertant was approximated through the following procedure. Proposed survival probabilities were taken from the sequence from 0 to 1 in increments of 0.01. Each generation, we calculated the pollen success of every haplotype and defined s as the difference between the fitness of a rescuing gene convertant and the mean fitness. Variation in fitness was completely determined by pollen success because ovule success was equal for all SI haplotypes, save the novel S-allele modeled above. We then chose a survival probability that, when substituted into Eq. 2.2 (truncated to the first 100 terms of the sum), resulted in the value of s closest to the value calculated from the genotype frequencies. We checked this approximation against simulations for each generation of a common genotype frequency trajectory with nL = 2, nU = 2, and a population size of 10,000. The ac- tual survival probability was determined by simulating the survival of a new gene convertant conditional on arising in a given generation. The convertant was judged to have survived if it survived for at least ten generations. The approximation closely followed the simulated results (Fig. S7), so we used the approximation for all future calculations. Once the genotype frequencies, the expected number of gene convertants, and the survival probability of a new gene convertant were known for every generation, we calculated the probability that none of them survived as the product of the complements of the survival probabilities. The complement of this probability is the probability that one doomed haplotype is

66 rescued at least once, assuming the expected number of gene convertants under the expected genotype frequency trajectory. We approximated the probability that a given number of haplotypes were rescued as the probability that each of them was rescued independently. The rescue probabilities are not strictly independent because one rescue event alters the trajectory of genotype frequencies. Rescuing one haplotype essentially creates another lucky haplotype, which could outcompete the remaining doomed haplotypes. However, if the rescues happen in relatively rapid succession, each rescued haplotype will have less effect on subsequent rescues because it is still at low frequency. Orr and Unckless [2008] found that rescue is most likely to occur while the population is still relatively large and mutation supply is highest. That is, there is a short period when the probability of rescue is highest. If rescue in our model is also most likely in a short window (when the doomed haplotypes are still at high frequency), then it is plausible that the multiple rescues occur at similar times and are approximately independent. With this assumption of independence, we generated a probability distribution of the number of surviving haplotypes after rescue or collapse. A natural question is, are large collapses guaranteed, or is there a non-trivial probability of expansion? For all iterations, we assumed that at least two haplotypes had already acquired the key to the new lock. This is the minimum number for the lock- mutation not to confer ovule-sterility because, if only one haplotype could fertilize it, all ovules carrying the lock-mutation would be fertilized by the same pollen haplotype. The maternal haplotype of the resulting offspring would reject all pollen except the paternal haplotype, which would reject itself. Therefore, when there is exactly one haplotype of class L, all ovules carrying the lock mutation grow up to be ovule-sterile adults. In this case, the lock mutation is deleterious rather than neutral and should be rapidly lost. Though this scenario might often occur, it results in no change in the

67 number of S-haplotypes and can be safely ignored in this model. We calculated the probability that all haplotypes survived for all combinations of nL = 2, 5, 10, 20, nU = 2, 5, 10, 20, and Rconversion in the range of 0.1 – 1 in increments of 0.01 and the range of 1 – 10 in increments of 0.1. We found that the probability that all haplotypes were rescued (the expansion probability) decreased with the number of doomed haplotypes and increased with the population rate of gene conversion Rconversion (Fig.

2.7). For nL = 2, the expansion probability rapidly saturated near 1.0 as

Rconversion increased regardless of nU . The expansion probability greatly decreased for nL = 3 relative to nL = 2 and continued decreasing gradually for larger values of nL. Though increasing nL decreased the probability of expansion, it also increased the minimum number of haplotypes after collapse because lucky haplotypes could not be lost (see Discussion on complete, i.e., lucky, haplotypes). The effect of nU was also to decrease the expansion probability, though this effect was less than that of nL. The distribution of haplotype number after rescue/collapse responded similarly to Rconversion for different values of nL, gradually shifting rightward toward maintenance of the initial number of haplotypes, but never reaching high probabilities of expansion for Rconversion ≤ 0.1 (Fig. S8). For nL = 2 and nL = 5, the modal outcome was loss of all or almost all doomed haplotypes.

2.2.6 Long-term behavior

The population has now passed through the ﬁnal “rescue/collapse” step of the three-step pathway (Fig. 2.3). However, natural populations might traverse this pathway repeatedly, and they must have undergone successive rounds of expansion to produce their current haplotypic diversity. The distribution of haplotype number in nature should therefore be governed

68 by the long-term balance between collapse and expansion. We can make crude predictions for the long-term evolution of haplotype number by treat- ing the diversification process as a Markov chain. Each state of this system is a positive integer representing a number of fully functional haplotypes currently present in the population. The transition probabilities out of each state are given by the distribution of haplotype number after a rescue/collapse event (calculated above). The stationary distribution of this Markov chain represents the probability distribution of haplotype number at expansion-collapse equilibrium. The stable distribution can be obtained without reference to the waiting times between transitions, which we do not model explicitly. In order to create transition matrices of finite size, we enforce a lower reflecting boundary of three haplotypes, the biological minimum that allows any fertilization, and an upper reflecting boundary of 43 haplotypes, near the upper range observed for taxa with SI presumed to function by collaborative non-self recognition [Lawrence, 2000]. We then iterate each transition matrix for 1000 introductions of a new lock to estimate its stationary distribution. Our previous rescue probability calculations showed that expansion was likely even at large nU as long as nL = 2 and Rconversion ≥ 0.1 (Fig. 2.7). Such large expansion probabilities are necessary to produce the many haplotypes observed in nature. To cover a range of expansion probabilities, we generate transition matrices for Rconversion = 0.1–0.4 in intervals of 0.1, as well as Rconversion = 0.01 and 1. We again assume that the initial frequency of each L haplotype is low, dL = 0.01. We also assume that nL = 2 remains a constant parameter between states, and that all changes in haplotype number are represented by changes in nU . This value of nL allowed non-trivial expansion probabilities. In reality, the transition matrix should be partly determined by several random variables: the number of L haplotypes and the initial frequency of each. The number would be de-

69 termined by the extent to which gene conversion had already spread the new key before the corresponding lock originated. Likewise, the frequencies would be determined by the outcome of drift between corresponding L and C haplotypes. We sidestep this complication and assume constant parameters nL and dL. We found that the stationary distribution shifted toward greater numbers of haplotypes as Rconversion increased (Fig. 2.8). At Rconversion = 0.01, the modal haplotype number was eight, while at Rconversion = 1, it was 37. In the interval from Rconversion = 0.2–0.4, the stable distribution was centered around intermediate values near the 20–30 range. At Rconversion = 0.3, the transition matrix showed that diversification was almost guaranteed at low haplotype numbers, while partial collapses were likely at high haplotype numbers (Fig. 2.9). To visualize the typical history of a population, we also simulated five haplotype-number trajectories at Rconversion = 0.3 for 100 transitions starting at three initial haplotypes. We found that haplotype number increased rapidly for about the first 20 transitions in all simulations, after which the trajectories fluctuated in the range of 20–30 haplotypes (Fig. 2.10). The foregoing model only considers changes in the number of S-haplotypes, but another relevant process is the transition from SI to SC. A transition to SC is expected when SC alleles are beneficial and their frequency at conversion-selection balance is at fixation. We repeat the previous model of long-term haplotype number evolution, but additionally assume that any population below a threshold number of S-haplotypes will transition to SC, losing all other S-haplotypes as the SC haplotype rises to fixation. We assume transitions to SC are irreversible, as appears to be the case for the collaborative nonself-recognition system [Igic´ et al., 2004], and so SC populations leave the Markov process entirely. Thus, any probability mass below this threshold is eliminated every generation, and the remaining

70 probabilities are divided by their sum so they add to one. The resulting distribution describes the probability of each haplotype number conditional on remaining SI. We set this lower threshold at 11 haplotypes, the minimum number for which we found an internal equilibrium frequency of an SC intermediate given µ = 0.0003 and σ = 0.5 (Fig. 2.4). We also set the starting haplotype number at 11 haplotypes and note that, though these parameters would not allow diversification to 11 haplotypes in the first place, initial diversification could have occurred if parameters were originally even less favorable to SC (e.g., larger σ). All long-run distributions were unchanged after adding this threshold except for R = 0.01, for which the probability was strongly concentrated at the new minimum of 11 (Fig. S9).

2.3 Discussion

We developed a new model of “haplotypic rescue” to estimate the relative probabilities of expansion and contraction in S-haplotype number, calculate the distribution of contraction magnitudes, and predict the long-term evolution of haplotype number. We ﬁnd that expansion from low haplotype number is possible if gene conversion is frequent or the population is large. When collapses occur instead, they can be large, easily resulting in the loss of the majority of S-haplotypes. A unique prediction of this model is that recently bottlenecked populations should have a “debt” such that, once the appropriate gene conversion event occurs, they will experience a sudden reduction in haplotype number in addition to the reduction directly caused by the bottleneck. Despite the possibility of collapse, we found that stable distributions of haplotype number within the range of 20–40 were possible in the long term. These numbers are greater than the upper limit of 14 haplotypes found in the model of Bod’ova´ et al. [2018], and are within the range found

71 in nature. This discrepancy is most likely due to the large supplies of rescuing gene convertants we modeled, and it suggests that arbitrarily many haplotypes can be maintained for a sufficiently large population size or high rate of gene conversion. Population size and S-haplotype number have rarely been estimated for the same populations. However, complete sampling of three populations of Pyrus pyraster, a rare woody perennial, revealed 9–25 S-haplotypes per population despite small (8–88 individuals) population sizes [Hoebee et al., 2011]. Our model fails to predict the maintenance of so many haplotypes in such small populations at equilibrium, but it is possible that these populations are not at equilibrium. If the waiting times between successive RNase invasions are long, then it is possible that collapses in haplotype number lag far behind reductions in population size. In this case, we should predict that S-haplotype number will drop precipitously in P. pyraster in the (possibly distant) future. Alter- natively, some populations might be locked at a long-term static haplotype number. For example, we found that a new S-haplotype cannot invade if pollen limitation is too great, in which case neither expansion nor collapse is possible. A major barrier to the evolution of new S-haplotypes is that a new RNase specificity is expected to reject the majority of pollen in the population because compatibility with the RNase was previously neutral. In the complete absence of compatible pollen, RNase mutations cannot invade because mutant plants cannot set seed. Both Harkness et al. [2019b] and “pathway five” in Bod’ova´ et al. [2018] accounted for this complication by supposing that F-box variants complementary to the RNase preceded the RNase itself, thereby providing a supply of compatible pollen. However, both of these models, as well as all other pathways modeled in Bod’ova´ et al. [2018] assumed that pollen limitation was binary: plants that received any pollen had the maximum ovule success, and plants that received no

72 pollen had no ovule success. That is, RNase mutants have ovule success equal to that of non-mutants despite rejecting the majority of pollen. In- stead, we explore the gradation between total ovule-sterility when pollen is limiting and equal undiminished ovule success when pollen is abundant. We find that a novel RNase can invade despite causing increased pollen limitation as long as compatible pollen is sufficiently abundant. We also find that the relative fitness of an RNase mutation decreases as the number extant RNases increases: the frequency-dependent advantage of rarity is less when the competing RNases are numerous and rare than when they are few and common. This diminishing advantage of RNase nov- elty could act as a negative feedback that limits how many RNase alleles can be produced in a single population. We therefore predict a negative relationship between pollen limitation and RNase number in nature. Nev- ertheless, pollen limitation appears to be a surmountable barrier to RNase diversification. Once the novel RNase invades, S-haplotype number might still either contract or expand. The probability of expansion depends on the population size and gene conversion rate. However, the rate at which gene conversion copies a functional F-box from one haplotype to another is unknown. Per-nucleotide rates of gene conversion have been estimated to be 400 times the rates of crossovers in Drosophila [Gay et al., 2007]. As- suming a crossover rate of 10−8 per meiosis per nucleotide, this yields a gene conversion rate on the order of 10−6. A functional gene conversion rate of 10−6 would produce few haplotypes (≈ 8) for populations on the order of 10,000, ≈ 10 − 30 for populations on the order of 100,000, and larger numbers for larger populations. Such populations are large but not unrealistic. Note, however, that relevant quantity is not merely the per- nucleotide rate, but the rate at which a functional F-box is gained. This rate excludes gene conversion events that eliminate a pollen specificity as

73 well as events that fail to affect specificity at all. It is therefore smaller than the per-nucleotide rate of gene conversion, though how much smaller is unknown. The rate at which F-box paralogs are copied would constrain not only the probability of rescue (through Rconversion) but also the input of SC gene convertants (through µ). It might be possible to estimate the rate of functional gene conversion indirectly. To perform such an estimate, a study would need to measure both the frequency and seed set of wild SC individuals relative to SI individuals in a predominantly SI population. Since the equilibrium frequency of SC haplotypes is determined by their ovule fitness and the gene conversion rate, it should be possible to estimate the gene conversion rate given the observed frequency and fitness of SC. Other than large populations and rapid gene conversion, another possible explanation of high S-haplotype diversity is that gene flow buoys species-wide diversity despite local contractions. Uyenoyama et al. [2001] found that, similar to our results, S-allele number was unlikely to expand within a single population because coexistence was rarely possible between a mutant SC intermediate and its ancestral haplotype. They proposed that expansion might instead occur through local turnover in S- alleles followed by introduction of the novel alleles into the broader metapopulation. However, this model assumed that incompatibility operates through self-recognition, in which introgressed S-alleles gain an advantage because they are compatible with resident plants that do not recognize them as self. It is thus not obvious whether the conclusion translates to nonself- recognition. Castric et al. [2008] found less divergence between putative shared ancestral variants of S-locus genes in Arabidopsis lyrata and A. halleri than between non-S orthologs, consistent with elevated introgression at the S-locus in this self-recognition system. An obvious empirical question is whether this result could be replicated in a species with a

74 nonself-recognition system. What would our model predict? Under collaborative nonself-recognition, each novel RNase requires a corresponding F-box. A novel introgressed RNase allele will not necessarily be detoxified by local F-box proteins, and it might thus inflict ovule-sterility. If there is insufficient compatible pollen to support the migrant haplotype, SI might act as a barrier to introgression. This barrier could be overcome, but it would require either that the key corresponding to the foreign lock already exists in the population (e.g., as a dual-function key or as a segregating neutral variant), or that there is sufficient migration to supply the corresponding foreign keys. Given these complications, the impact of gene flow on the diversification / collapse of S-allele diversity and the impact of S allele diversity on hybridization and introgression in collaborative nonself recognition based SI require additional research. We found that the expansion probability greatly decreased as the number of complete haplotypes increased, a second form of negative feedback in addition to that imposed on the invasion of the RNase in the first place. Greater numbers of complete haplotypes reduce the expansion probability in at least two ways. First, by increasing the number and thus the frequency of fit haplotypes, they increase the mean siring success. When compatibility with the new RNase is more common, the competitive advantage of this compatibility is reduced. Second, with more complete haplotypes, the equilibrium frequency of the novel RNase is lesser. These two effects reduce the advantage of compatibility with the new RNase and thus the survival probability of the rescuing gene convertant. This essential parameter of the model, the initial number of haplotypes possessing the F-box complementary to the novel RNase specificity, has not been systematically estimated. The existence of the complementary F-box on multiple backgrounds is a prerequisite for both expansion and collapse. The prevalence of pre-existing complementary F-boxes in a population

75 could be determined by experimentally applying pollen to plants carrying a novel, functional RNase allele. Successful fertilization would be consistent with a pre-existing F-box complementary to the RNase. An alternative explanation for fertilization, the loss of RNase function in the dam, could be ruled out with the appropriate control: F-box knockouts should be compatible with RNase loss-of-function mutants but not with novel RNase alleles. Novel RNase alleles could be generated by mutagenesis, or they could be introduced from distantly related populations transgenically or through introgression. Besides bounding the parameters of the model, empirical research could also test for evidence of recent historical collapses. Consider a pair of allopatric populations that initially shared all S-haplotypes. However, a novel RNase has recently invaded Population A and eliminated several pre-existing haplotypes, resulting in a contraction in haplotype number. This RNase never arose in Population B, which retains the original complement. Every remaining haplotype in Population A other than the novel haplotype itself should carry the F-box complementary to the novel RNase. Of course, these vestigial F-boxes would eventually become pseudoge- nized, but they might still be functional shortly after the collapse. In Pop- ulation B, several haplotypes should either be polymorphic for the novel RNase’s complementary F-box or should lack it entirely. This situation could be tested by reciprocally crossing individuals from the two populations. A substantial proportion of B pollen should be rejected by recipients in Population A: a proportion in excess of that predicted by the number and frequency of shared S-haplotypes. A potentially dramatic test would be to introgress the novel RNase from Population A to Population B. In the short run, the siring success of all haplotypes not already compatible with the new RNase would decrease. In the long run, the frequency of these unlucky haplotypes would decrease to extinction. That is, we predict

76 that S-haplotype collapse should be contagious between closely related populations, even if both populations are SI. This work builds on that of Bod’ova´ et al. [2018], who model invasion and coexistence conditions for S-haplotypes and also directly simulate the dynamics of the diversification process. One of their key insights from their coexistence model is that, if some haplotypes are more complete than others (i.e., possess more functional pollen specificities), the less complete haplotypes can be driven to extinction at equilibrium because of their inferior pollen success. They show that a new S-haplotype cannot coexist indefinitely with all pre-existing haplotypes except in the limited cases that either all haplotypes are complete or exactly one haplotype is complete. They note, however, that even transient coexistence might provide a window in which further mutations might make all haplotypes complete and thus capable of permanent coexistence. This rescue process is implicitly incorporated in their stochastic simulations, and we model it analytically in this study. Their coexistence model also shows that SC haplotypes will reach fixation except for high values of inbreeding depression (≥ 0.75), which informs our focus on intense inbreeding depression. Bod’ova´ et al. [2018] consider two kinds of pathways: one that maintains SI throughout and several that go through an SC intermediate. There is good reason to believe that, although the pathway maintaining SI is traversed moderately often in the simulations of Bod’ova´ et al. [2018], this pathway should be rare in nature. This is because when an RNase mutation occurs on an SI background, it reduces the amount of pollen accepted but does not provide any benefit. Such a mutation is neutral if ovule success is not limited by the supply of compatible pollen [Bod’ova´ et al., 2018] but deleterious if it is. In contrast, we have shown that a pathway through an SC intermediate is possible even with the barrier of pollen limitation because the RNase mutation prevents self-fertilization. Finally, we have followed

77 up on the suggestion by Bod’ova´ et al. [2018] to incorporate gene conversion of F-box paralogs as hypothesized by Kubo et al. [2015] and Fujii et al. [2016]. Although recurrent gene conversion in our model functions similarly to recurrent mutation in their model [Bod’ova´ et al., 2018], an important difference is that functional gene conversion can only occur in heterozygotes for an F-box locus. We show that, despite this limitation on the supply of potentially rescuing variants, diversification is still possible. Even though we assume maximal inbreeding depression throughout, we find that populations are susceptible to invasion and fixation of SC haplotypes unless selfing rates are also high (Fig. 2.4). The possibility of transition to SC does not seem to affect the distribution of haplotype numbers of those populations that remain SI unless the stable distribution crosses the minimum for maintaining SI (Fig. S9), likely because there is a strong upward pull when haplotype numbers start below the stable range (Fig. 2.9, Fig. 2.10). However, populations can only diversify from low to moderate numbers of haplotypes in the first place if selfing rate is high. Thus, though diversification is possible through this pathway alone, transitions to SC might be very common at low haplotype numbers in nature. A trait that is frequently lost but rarely or never regained can still common among many species if species with the trait speciate more frequently or species without the trait go extinct more frequently. This macroevolutionary process, species-level selection, is hypothesized to maintain SI among species despite frequent and possibly irreversible transitions to SC [Igic´ et al., 2008, Goldberg et al., 2010]. But if diversification is precarious at low numbers, how have high numbers been reached? And if high inbreeding depression and selfing rate are required, why would a population not have either purged the inbreeding depression or evolved simpler mechanisms of reducing selfing? Future models of the origin of the polyallelic S-locus ought to answer these questions, but the present model is best

78 viewed as one of continued diversification rather than one of origin. SI has developed into a rich study system through the continued interaction between theory and empirical research. Experimental demon- stration of the genetic control of SI [East and Mangelsdorf, 1925] and field research showing the number of S-alleles in natural populations [Emer- son, 1938, 1939] inspired theoretical explanations of the balancing selection capable of maintaining this diversity [Wright, 1939, Charlesworth and Charlesworth, 1979]. The theoretical potential for balancing selection indi- cated the S-locus as a candidate for long-term polymorphism, and S-allele phylogenies confirmed this possibility [Igic´ and Kohn, 2001, Steinbachs and Holsinger, 2002]. The discovery of the fine-scale genetic basis of nonself-recognition [Kubo et al., 2010, 2015] necessitated new theoretical explanation of the expansion process, and this theory now points to the unanticipated possibility of S-allele collapse through runaway gene convertants.

79 Nonself Pollen Self Self grain

1 2 3 2 1 3 3 1 2 Stigma

2 1 1 Pollen 3 3 2 tube

1 2 1 2 1 2 = RNase Style = F-box

Figure 2.1: Rejection of self pollen in the style. Each haplotype contains a single lock and multiple paralogous keys. Each pistil expresses the two locks in its diploid genotype, and each pollen grain expresses all keys in its haplotype. A pollen tube is arrested in the style unless it contains keys to both of the pistil’s locks. Each haplotype lacks the key to its own lock, but contains the keys to all other locks. Self-fertilization is prevented because pollen necessarily lacks one of the keys expressed by the plant that produced it.

80 Table 2.1: Notation. Fixed parameters are set at the beginning of the diversiﬁca- tion/collapse process and remain constant throughout. Initial conditions are set at the beginning of the process, but their only effect is to determine the initial genotype and haplotype frequencies. Dynamic variables vary throughout the simulation and continue to be tracked. The dynamic variables are not independent: the collection of genotype frequencies is sufﬁcient to describe the current state of the population completely.

n Fixed parameter Number of locks, excluding the mutant

nL Fixed parameter Number of lucky haplotypes

nU Fixed parameter Number of unlucky haplotypes G Fixed parameter Number of pollen grains received on each stigma µ Fixed parameter Per-individual rate at which gene conversion produces SC haplotypes

Rconversion Fixed parameter Total rate at which gene conversion produces rescuing haplotypes

dL Initial condition Initial frequency of new key within a lucky-unlucky pair

pX Dynamic variable Frequency of all haplotypes of class X

PXY Dynamic variable Frequency of genotype with haplotypes of classes X and Y

FXY Dynamic variable Frequency of pollen compatible with genotype XY

DXY Dynamic variable PXY /FXY

wX Dynamic variable Fitness of any haplotype of class X w¯ Dynamic variable Mean ﬁtness of all haplotypes

wR Dynamic variable Fitness of a potentially rescuing gene convertant

s Dynamic variable Additional ﬁtness of a gene convertant, wR − w¯

81 Lucky (L) 1 2 3 4

Chump (C) 1 2 3

Unlucky (U) 2 1 3

Ancestral (A) 3 1 2

Intermediate (I) 3 1 2 3

= Pistil RNase

Mutant (M) 4 1 2 3 = Pollen F-box

Figure 2.2: Haplotype classes. In this example, the population begins with three complete haplotypes (those with the key to every lock but their own). Gene conversion generates an SC “intermediate” haplotype from the “ancestral” haplotype by introducing the key to the ancestral haplotype’s own lock. The SC intermediate acquires a novel lock by mutation, restoring SI and generating the “mutant” haplotype. All haplotypes lack the key to the new lock except for those “lucky” enough to carry it in advance, which have an advantage over the “chump” haplotypes that bear their corresponding locks. Haplotypes that are “unlucky” merely lack the new key but don’t compete against other haplotypes with their same locks.

82 Conversion Mutation Rescue/Collapse

1 2 3 4 1 2 3 4 1 2 3 4 C C C

1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 L L L L

2 1 3 4 2 1 3 4 2 1 3 4 C C C

2 1 3 4 5 2 1 3 4 5 2 1 3 4 5 2 1 3 4 5 L L L L ? ?

3 1 2 4 3 1 2 4 3 1 2 4 3 1 2 4 U U U U

3 1 2 4 5 U*

4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 A A A A

4 1 2 3 5 A*

4 1 2 3 4 4 1 2 3 4 I I

5 1 2 3 4 5 1 2 3 4 M M

Figure 2.3: Model steps. First, gene conversion (vertical arrow) generates the SC intermediate I. Second, a mutation in the intermediate’s lock (starburst) generates a mutant SI haplotype M. The intermediate is deleterious and transient, though recurrent mutation continually produces phenotypically identical SC haplotypes. Now that their key has a complementary lock, L haplotypes increase in ﬁtness and outcompete their corresponding C haplotypes. Third, the other haplotypes lacking the new key (U, A) are driven to extinction, but gene conversion might or might not “rescue” them (arrows with question marks) by producing convertant haplotypes carrying the new key (gray boxes).

83 Selfing rate 0.06

0.2 0.7

0.3 0.8

0.4 0.9

0.5 1 0.04

0.6 0.02 Equilibrium frequency

0.00 10 15 20 25 30 35 40 RNases

Figure 2.4: Conversion-selection balance of SC gene convertants. Equilibrium frequency was calculated assuming the gene conversion rate µ = 0.0003. Equilibrium frequency declined with increasing RNase number and selfing rate. The left-most point of each curve represents the threshold RNase number below which an SC haplotype will rise to fixation rather than equilibrate at low frequency. This threshold appears as an abrupt stop rather than an asymptote because RNase number can only take on integer values. No internal equilibrium was found for selfing rate σ = 0.1.

84 dL: 0.01 dL: 0.1 dL: 0.5

2.0

1.5

3 1.0 9

Fitness 27

0.5

0.0 25 50 75 100 25 50 75 100 25 50 75 100 Pollen/Ovules

Figure 2.5: Fitness of a pollen-limited SI mutation. The mutation from the SC intermediate to the new SI haplotype prevents selfing but induces pollen limitation, resulting in a tradeoff. Mutant fitness is calculated relative to that of non-mutant SI haplotypes, which we use as an approximation of population mean fitness. All flowers received equal quantities of pollen drawn proportionally from the pollen pool, and all differences in ovule success were determined by the proportion of that pollen that was compatible. Fitness is plotted against the ratio of pollen received to ovules in a flower: ovule number was held constant at 10, and the pollen supply G was varied. Fitness was calculated assuming nL = 2.

85 L 0.6 0.4

M 0.2 Frequency U A C 0.0 0 20 40 60 80 100 Generation

Figure 2.6: Doomed haplotypes. When the low-frequency self-compatible intermediate (not plotted) acquires a novel RNase by mutation, the resulting haplotype (M) invades. Those haplotypes possessing the key to the new lock (L) also beneﬁt and reach high frequencies, while haplotypes lacking the key (C, U, A) are driven extinct. All trajectories (0) 1 were generated by numerical iteration with parameters dL = 0.01, PM = 100n , and nL = nU = 2.

86 nL = 2 nL = 3 0 0 −20 −20

nU = 2 nU = 5 nU = 10 −40 −40 ) nU = 20 n o i s

n −1.0 0.0 1.0 −1.0 0.0 1.0 a p x e = =

P nL 4 nL 5 ( 0 0 10 g o l −20 −20 −40 −40 −1.0 0.0 1.0 −1.0 0.0 1.0

log10(Rconversion)

Figure 2.7: Expansion probability. Each curve represents the probability all doomed haplotypes are rescued for a given number of doomed haplotypes. The probability that all are rescued decreases as the number of doomed haplotypes (nU ) increases, and it also usually decreases as the number of “lucky” haplotypes increases. Rescue is very unlikely unless either the population rate of gene conversion (Rconversion) is very high or nL is small.

87 iue28 tbedsrbto fhpoyenme.Tesal ubro haplotypes of number stable The number. haplotype as of increased distribution Stable 2.8: Figure 00ses(xasoso olpe) and collapses), or (expansions steps 1000 near minimum around biological haplotypes the exceed to came ber indit h 3dcategory. 43rd the the counting into haplotypes, binned total 43 (i.e., haplotypes incomplete 40 Frequency 0.0 0.4 0.8 0.0 0.4 0.8 R R R c 10 conversion c o o n n v v e R e r s r conversion s i o i o n h upyo eecnetns nrae.Tehpoyenum- haplotype The increased. convertants, gene of supply the , n 30 = = 0.01 0.2 0 = Total haplotypes . 3 r04 o l aes h akvcanwsrnfor run was chain Markov the panels, all For 0.4. or R R 10 c c 88 n o o L n n v v 2 = e e r r s s i i o o l rbblt fpouigmr than more producing of probability All . R n n 30 conversion = = 0.1 0.3 0 = . L 1 n rdcd20–40 produced and , R and R 10 c o c n o M v n e v r e altps is haplotypes) s r i s o i n o 30 n = = 0.4 1 1.0 40

0.8 30

0.6

0.4 20

0.2 Transition probability Transition 10 Haplotypes after transition

0.0 10 20 30 40 Haplotypes before transition

Figure 2.9: Haplotype number transition matrix for Rconversion = 0.3. The transition probability is from an initial state (horizontal axis) to the next state (vertical axis), where each state is a number of S-haplotypes. For all states, nL = 2, and only nU changes between states. The diagonal from bottom-left to top-right (white line) represents no change in haplotype number but could result from either loss of a novel haplotype (i.e., stasis) or maintenance of a novel haplotype and loss of an old one (i.e., turnover). The off- diagonal immediately above that represents gain and maintenance of single novel haplotype. The probability of diversiﬁcation decreases with increasing haplotype number. It remains greater than 0.5 until nL + nU = 18 and remains the most likely outcome until nL + nU = 22.

89 25 20 15 10 Haplotype number 5 0 0 20 40 60 80 100 Time (transitions)

Figure 2.10: Long-term simulated haplotype number trajectories. Five trajectories were simulated for 100 transitions from the transition matrix for Rconversion = 0.3 and nL = 2.

The initial state was three haplotypes (nL = 2, nU = 1). Haplotype number increased rapidly at ﬁrst and continued to ﬂuctuate after entering the long-term stable range.

90 The evolution of new ribonuclease-based S-haplotypes is complicated by the requirement for intragenomic coevolution among S-haplotypes. Co- existence among S-haplotypes is normally enforced by negative frequency- dependent selection, but novel S-haplotypes to which the others are not yet coevolved disrupt this equilibrium and may drive extant haplotypes to extinction. Diversiﬁcation occurs only when all haplotypes are able to adapt to each other before any are driven extinct. This rescue-like process can produce long-term diversiﬁcation, but some populations, particularly small ones, may instead equilibrate with very few haplotypes because of large or frequent collapses in diversity. Another case in which S-haplotypes might not be mutually coevolved is when they exist in different populations. Haplotypes are selectively con- strained to maximize pollen success by being compatible with all other haplotypes which they regularly encounter. Selection for compatibility with rarely encountered haplotypes, such as those unique to other, relatively isolated populations, is weaker. The guarantee of cross-compatibility among S-haplotypes thus does not hold across populations. Cross-incompatibility should present a substantial barrier to hybridization and introgression among populations exchanging migrants rarely or long-isolated populations brought into secondary contact. In a population genetic model of introgression among self-incompatible populations, In the following chapter, I show that cross-incompatible S-haplotypes may substantially distort the direction of introgress. Depending on the number of cross-incompatible haplotypes in each population, introgression may be reduced relative to the migration rate or become strongly biased in one direction, potentially replacing some or all resident S-haplotypes with invading ones. These patterns should also affect genome-wide introgression, especially at sites linked to the S- locus.

91 Chapter 3

Nonself-recognition-based self-incompatibility can alternatively promote or prevent introgression

This chapter has not yet been published elsewhere. A preprint is available as Harkness and Brandvain [2020].

Summary

• Traditionally, we expect that self-incompatibility alleles (S-alleles), which prevent self-fertilization, should beneﬁt from negative-frequency dependent selection and rise to high frequency when introduced to a new population through gene ﬂow. However, the most taxonomically widespread form of self-incompatibility, the ribonuclease-based system ancestral to the core eudicots, functions through nonself-

92 recognition, which drastically alters the process of S-allele diversi- ﬁcation.

• We analyze a model of S-allele evolution in two populations connected by migration, focusing on comparisons among the fates of S- alleles originally unique to each population and those shared among populations.

• We ﬁnd that both shared and unique S-alleles originating from the population with more unique S-alleles were usually ﬁtter than S- alleles from the population with fewer. Resident S-alleles were often driven extinct and replaced by migrant S-alleles, though this outcome could be averted by pollen limitation or biased migration.

• Nonself-recognition-based self-incompatibility will usually either disfavor introgression of S-alleles or result in the whole-sale replacement of S-alleles from one population with those from another.

3.1 Introduction

In ﬂowering plants, a self-incompatibility locus (S-locus) is a highly polymorphic region of the genome responsible for rejecting self pollen and is maintained by negative frequency-dependent selection, a form of balancing selection. A given S-locus allele (S-allele) encodes paired pollen and pistil phenotypes, and the pistil phenotype rejects pollen expressing the matching pollen phenotype, which necessarily includes self pollen. With self-incompatibility (SI), the ﬁtness of an S-allele increases as it becomes rarer because rare alleles allow more mating opportunities. This process preserves polymorphism because alleles that drift to lower frequencies are pushed back to intermediate frequencies. It may also spur new poly-

93 morphism: new mutations are automatically rare, and their initial advantage under negative frequency-dependence favors their invasion [Wright, 1939, Charlesworth and Charlesworth, 1979]. On first reflection, this advantage of rarity would seem to apply equally to novel mutations and rare migrant alleles. For one major form of self-incompatibility, self-recognition, the effects of this advantage of rarity have been modeled for both novel mutations [Uyenoyama et al., 2001] and rare migrants [Schierup et al., 1998, Muirhead, 2001, Schierup and Vekemans, 2008]. But for nonself- recognition, the more taxonomically widespread form, only new mutations [Bod’ova´ et al., 2018] or, similarly, new products of gene conversion [Hark- ness et al., 2019b] have been modeled. Here we ask if and when migration will lead to invasion of a new S-allele under nonself-recognition self- incompatibility. The answer to this work can influence both the-long term diversification of S-alleles and the role of S-allele divergence in preventing or promoting gene flow between diverged populations or species. Invasion of a novel S-allele through rare advantage is simple if a single mutation can generate both a new pollen and pistil specificity [Charlesworth and Charlesworth, 1979]. However, most self-incompatibility systems operate through linked but distinct pollen- and pistil-specificity loci, and a mutation at one component cannot affect the specificity encoded by the other. In such systems, S-alleles are more precisely called S-haplotypes, and while their maintenance by long term balancing selection is straightforward, the origin and diversification of novel two-locus S-haplotypes is complex. Uyenoyama et al. [2001] found in a deterministic two-mutation model that balancing selection in a single population could only increase S-haplotype diversity under stringent parameters and would usually lead either to loss of self-incompatibility entirely or to mere replacement of one S-haplotype with a novel one. They proposed that, though a local one- for-one replacement would not affect net local S-haplotype diversity of a

94 subpopulation, all subpopulations collectively would retain the old haplotype while acquiring a new one. Subsequent gene flow could then spread the novel haplotype among subpopulations or reintroduce the old haplotype into the subpopulation in which it was lost. Using extensive stochastic simulations of the same basic model, Gervais et al. [2011] found more relaxed conditions for local diversification but still invoked population structure to explain why natural populations harbor more haplotypes than were observed in their simulations. But the foregoing are models of self-recognition based self-incompatibility, while the widespread S-locus ribonuclease (S-RNase) system ancestral to core eudicots functions through nonself-recognition [Kubo et al., 2010]. Under self-recognition, each haplotype encodes a pollen specificity that is uniquely rejected by a pistil specificity on the same haplotype. This guar- antees cross-compatibility of any two different S-haplotypes. The more complex nonself-recognition system can be envisioned as a lock-and-key system, in which each haplotype carries one pistil-specificity lock and many pollen-specificity keys [Harkness et al., 2019b]. Each lock can be unlocked by one key, and each haplotype carries a key ring that includes the key to every lock in the population but the haplotype’s own lock. Haploid pollen must carry keys to both the diploid seed parent’s locks in order to fertilize it. As such, nonself-recognition does not guarantee cross-compatibility with foreign specificities – as there is no reason to hold a key to a lock one never encounters. Therefore, just as Dobzhansky-Muller Incompatibilities (DMIs) [Dobzhansky, 1934, Muller, 1942] can accumulate between populations because the mutations with deleterious epistatic interactions are not normally exposed to one another, keys to locks present only in other populations are neutral and can easily be lost. Novel migrant haplotypes under nonself-recognition may therefore possess the advantage of rarity, just as under self-recognition, but they may also face disadvantages both

95 as pollen parents – because they are incompatible with unique local haplotypes – and as seed parents – because they face a drastic reduction in compatible pollen. We note that the dichotomy between self- and nonself-recognition is distinct from the better-known dichotomy between gametophytic (GSI) and sporophytic SI (SSI). In GSI, the phenotype of a pollen grain is completely determined by that pollen grain’s own haploid (gametophyte) genotype, while in SSI, the pollen phenotype is determined by that of the diploid (sporophyte) pollen parent. Different combinations of gametophytic or sporophytic and self- or nonself-recognition are possible: the S-locus receptor kinase system in Brassicaceae functions through sporophytic self- recognition, the programmed cell death system in poppy functions through gametophytic self-recognition, and the S-locus ribonuclease (S-RNase) system ancestral to the core eudicots functions through gametophytic nonself- recognition [Fujii et al., 2016]. We are not aware of any examples of sporophytic nonself-recognition, but such a system might occur in any of the many taxa that have not been mechanistically characterized. Recent models of the evolution of new S-haplotypes under non-self recognition, Bod’ova´ et al. [2018] and Harkness et al. [2019b] found novel and surprising dynamics absent in self-recognition. “Complete” haplotypes, those that hold the keys to all other locks in the population and are thus compatible as pollen with all other haplotypes, have greater pollen success than “incomplete” haplotypes, which are missing some keys. An incomplete haplotype therefore cannot survive indeﬁnitely in competition with more complete ones. Novel complete haplotypes may therefore eliminate existing incomplete haplotypes and actually reduce S-haplotype diversity unless subsequent mutation or gene conversion equalizes com- pleteness among haplotypes. Stochastic simulations show that moderate numbers of haplotypes can be maintained through a continuous cycle of

96 replacement in small or medium populations [Bod’ova´ et al., 2018], and deterministic approximations show that many haplotypes can be maintained in large populations or populations undergoing frequent gene conversion [Harkness et al., 2019b]. Nonself-recognition thus presents unexpected challenges for the evolution of novel S-haplotypes, but what about the arrival of S-haplotypes through migration? Although both processes begin with an initially rare variant, the history of that rare variant is very different for each process. A novel S-haplotype generated by mutation or gene conversion is only one step removed from the other haplotypes in the same population: either it has added or lost a key on its key ring, or its lock has mutated such that it is unlocked by a different key. Whichever change occurs first, it is only the first step in a multi-step process requiring a new lock on one haplotype and a new key to that lock on every other haplotype. In contrast, a never-before-seen migrant haplotype has already completed this process: it has the keys to every lock in its own population, and all other haplotypes in its population have the key to its lock. Furthermore, there may be a consistent stream of migrant haplotypes every generation, whereas mutation or gene conversion presumably produce functionally novel haplotypes at a much lesser rate. These differences, the pre-existence of fully formed haplotypes and their continued introduction through migration, suggest that self-recognition might have very different implications for migration than for mutation or gene conversion. We develop and compare population genetic models of gene flow under gametophytic self- and nonself-recognition to determine the effect of the widespread collaborative nonself-recognition on gene flow. We find that the general pattern under nonself-recognition is for gene flow to occur from the population with more unique haplotypes to the population with fewer, which may eliminate unique S-locus diversity in the recipient but

97 may also facilitate the spread of novel alleles to multiple populations. For the reader who would like more biological detail, we brieﬂy present the molecular mechanism of S-RNase nonself-recognition SI here. Read- ers less interested in these details can skip to the Description section, which abstractly describes the haplotype cross-compatibility relationships necessary to understand the theoretical consequences of this system.

The molecular mechanism of S-RNase nonself-recognition is an area of ongoing research, and we merely present the leading hypothesis [reviewed by Williams et al., 2015], known as collaborative nonself-recognition [Kubo et al., 2010]. Under this model, the S-locus consists of multiple tightly linked sites: a single highly polymorphic RNase gene and many biallelic (functional vs. nonfunctional/absent) S-locus F-box (SLF) genes around it [Kubo et al., 2010]. The different SLF genes are diverged paralogs, but the allelic diversity at any one paralog is minimized by recurrent gene conversion [Kubo et al., 2015] despite the diversifying effect of long- term balancing selection among haplotypes. An SLF protein is expressed only in pollen and forms a part of the multi-protein SCF (Skp1-Cullin1-F- box protein) complex, a ubiquitin ligase [Lai et al., 2002, Qiao et al., 2004, Hua and Kao, 2006]. Each diverged form of SLF protein targets this complex toward one or more allelic forms of RNase [Kubo et al., 2010]. There are thus multiple versions of this complex, each targeted toward different RNases as determined by which SLF the complex possesses. The complex’s ultimate effect is to interfere with the RNases with which it interacts, though it is unclear whether this occurs through ubiquitin-labeling of the RNase for degradation by a proteasome or an alternative means [Hua and Kao, 2006]. The RNase is expressed in the style and transported inside the growing pollen tube regardless of whether the pollen is compatible or incompatible [Luu et al., 2000, Goldraij et al., 2006]. If the pollen tube

98 does not produce the SLF that targets this RNase, the RNase will slow or arrest pollen tube growth by digesting RNA in the pollen tube [Huang et al., 1994]. Each maternal plant produces the two RNases in its diploid genotype (the metaphorical “locks”), so pollen is rejected by default unless it possesses the SLFs (the metaphorical “keys”) necessary to target both RNases. Self-incompatibility among copies of the same haplotype is achieved because each haplotype carries a nonfunctional or absent allele at the SLF paralog complementary to the haplotype’s own RNase, and cross-compatibility among different haplotypes is achieved because each haplotype carries a functional allele at the SLF paralogs complementary to all other RNases [Kubo et al., 2010].

3.2 Description

We modeled the evolution of the S-locus in two self-incompatible populations, the local population and the foreign population, of inﬁnite size connected by pollen migration. In the simplest versions of this model, migration was unidirectional. From the perspective of the population receiving immigrants, that population is “local” while the population sending emi- grants is “foreign.” For consistency, we retained this terminology throughout, but the local and foreign designations become arbitrary when migration is bidirectional. Each population harbored a number of S-haplotypes, some unique to that population and some shared between populations. We considered both self- and nonself-recognition models, but incompatibility was always gametophytic: the pollen phenotype was determined by its own haploid genotype, and the maternal phenotype was determined codominantly by its two haplotypes. Under GSI, the equilibrium condition in the absence of migration is equal frequency among S-haplotypes [Nagy- laki, 1975, Boucher, 1993, Steiner and Gregorius, 1994], so we set this as

99 the initial state for each population. Selection occurred only through pollen competition and maternal fecundity: there was no selection on viability. In pollen competition, all pollen compatible with a given maternal genotype competed equally to fertilize individuals of that maternal genotype. Pollen had zero success on maternal genotypes with which it was incompatible. We allowed for pollen limitation of maternal fecundity, in which maternal genotypes that accepted more pollen enjoyed greater seed success. Two considerations suggest that pollen limitation is an important process to model. First, since carriers of unique migrant haplotypes reject the majority of all pollen they receive (all resident haplotypes), they are likely to suffer severe pollen limitation in nature. Second, since pollen limitation is especially unfavorable to migrant haplotypes, we expect it to act as a barrier to gene ﬂow that might partially counteract the advantage of rarity. We use our lock and key metaphor [Harkness et al., 2019b] to model nonself recognition. That is, the style is a door locked by two codominant RNase “locks,” and a pollen grain must carry the SLF “keys” to both of a seed parent’s locks in order to unlock the door and proceed to fertilization. For simplicity, we imagine that each key unlocks one lock, but in reality some SLFs are complementary to two or more RNases [e.g. Sun and Kao, 2013]. Pollen is always incompatible with the plant that produced it because either possible key ring it could possess is missing one of the keys to one of the plant’s own two locks. For self-recognition, we use the simplest model of a two-gene S-locus we could conceive, though we could not invent an adequate metaphor for this system. The S-locus under self-recognition consists of one polymorphic pistil-expressed gene tightly linked to one polymorphic pollen- expressed gene. Each pollen allele corresponds to one pistil allele, and corresponding alleles always exist on the same haplotype. By default, all pollen is accepted, but pollen carrying the pistil allele corresponding

100 to either of the seed parent’s codominantly expressed pistil alleles is rejected. Since each haplotype carries a pollen allele that would be rejected by its own pistil allele, each haplotype is self-incompatible. This is similar to the programmed cell death mechanism in poppy, in which recognition between the pollen and pistil components triggers self-destruction in the pollen [Franklin-Tong et al., 1993], but differs from many better-known self-recognition based systems [Hiscock, 2002]. We chose this system to isolate the effects of self vs nonself based recognition away from complications of dominance hierarchies among alleles in sporophytic SI systems. Self- and nonself-recognition lead to different behavior in inter-population crosses. A self-recognition pistil allele only rejects one pollen allele: its complement. Non-matching pollen will always be accepted, regardless of its population of origin. In contrast, with nonself-recognition, a pistil locks out all pollen without a key. In a single population, this difference is immate- rial: selection for maximal pollen success will ensure every S-haplotype is “complete” [Bod’ova´ et al., 2018] with respect to its own population, mean- ing that it carries keys to all locks in that population except its own lock. However, there would be little selection for a key to a lock that is rarely encountered and no selection for a key to a lock that is never encountered. Given enough time and isolation, populations will come to diverge in their sets of locks (either through novel mutations or differential loss), and each population will only maintain keys to locks present within that population. This leads to very different outcomes for haplotypes that are shared among populations and haplotypes unique to a population. Pollen from one population will always be compatible with any other haplotype from the same population, whether shared or unique. But pollen from one population will only be compatible with a haplotype from another population if that haplotype is shared with the pollen’s population of origin. We only explicitly model this ideal high-isolation case, but consider other plausible biological

101 in the Discussion. We classify all S-haplotypes based on their uniqueness and population of origin. This is sufﬁcient for our model because the compatibility of pollen with locally unique S-haplotypes is determined by the pollen’s population of origin under nonself-recognition. S-haplotypes are either unique to the local population (UL), unique to the foreign population (UF ), shared and originating from the local population (SL), or shared and originating from the foreign population (SF ). Each class may contain one or more S-haplotypes, and we assume all haplotypes within a class always occur in equal proportions. We justify this assumption on the logic that, so long as it were true initially, it would remain true because haplotypes within the same class would have equal pollen and ovule success. Shared local and shared foreign haplotypes come in pairs, e.g., a local

S1 and a foreign S1. We assume under nonself-recognition that the members of a shared pair differ in their key rings: each has only the keys to locks originating from its own population. This distribution of keys is expected at equilibrium under complete isolation between two populations because keys to locks that are never encountered are under no selective constraint to remain functional. In nature, however, the populations may not be at equilibrium or may not have been completely isolated in the past. Under self-recognition, the members of a shared pair are functionally identical, though in nature they could be characterized by different neutral mutations. We use the same classiﬁcation for self- and nonself- recognition for ease of comparison, but note that population of origin only affects phenotype under nonself-recognition, in which it determines an S- haplotype’s key ring. Functionally, there is only a single difference between our self- and nonself-recognition models: S-haplotypes originating from one population are incompatible as pollen with all haplotypes unique to another population under nonself-recognition but compatible under self-

102 recognition (Fig. 3.1). This characterization allows us to compare the expected amount and pace of gene ﬂow at the S-haplotype across comparable parameters for self- and non-self based SI. We denote the frequency of each class in each population by pL and pF . E.g., in the local population, the frequency of all haplotypes initially L unique to the local population (UL) is pUL. The frequency of a haplotype class is the sum of the frequencies of all haplotypes in that class. Since all haplotypes in a population begin at equal frequencies, the initial frequencies of the haplotype classes are determined by the number of haplotypes in each class. These numbers are denoted nL (number unique to the local population) nF (number unique to the foreign population), and nS (number shared). In the local population, initially,

L pUL = nL/ (nL + nS) L pSL = nS/ (nL + nS) L L pUF = pSF = 0

In the foreign population, initially,

F pUF = nF / (nF + nS) F pSF = nS/ (nF + nS) F F pUL = pSL = 0

Gene ﬂow occurs through pollen migration such that a proportion mFL of pollen in the local pollen pool is migrant pollen from the foreign population, and mLF of pollen in the foreign pollen pool is migrant pollen from the local population. When migration is unidirectional from foreign to local, the foreign population remains at its initial equilibrium, so we only track the local population. When migration is bidirectional, we track both populations. We model pollen limitation through a root function: a given genotype’s b seed success is aX , where aX is the frequency of all pollen compatible with

103 genotype X in the available pollen pool and b is a shape parameter (with b = 0 corresponding to no pollen limitation). This allows seed success to saturate as compatible pollen received increases. Since the origin of a shared haplotype only affects its behavior in pollen, we drop the origin designation from shared haplotypes in aX , leaving the unique haplotypes to be distinguished solely by their origin. E.g., aUFSF = aUFSL, so we instead simply write aFS. We ﬁrst describe analytical results for the case of unidirectional migration, and then describe numerical results for bidirectional migration. For this case, we focus entirely on frequencies within the local population and drop the superscript denoting population: e.g., the frequency in the lo- L cal population of an allele initially unique to the foreign population pUF is simply labeled pUF . We use this case to lay out the three deterministic processes underlying haplotype frequency change. First, selection on ovules occurs through pollen limitation. Each diploid ∗ b genotype X contributes a number of successful ovules PX = PX aX . Sec- ond, assuming unidirectional migration, pollen migration generates the pollen pool. So

∗ nF pUF = (1 − m)pUF + m nF + nS ∗ pUL = (1 − m)pUL ∗ nF pSF = (1 − m)pSF + m nU + nS ∗ pSL = (1 − m)pUF where a ∗ denotes a frequency after migration. Third, pollen selection occurs through differential pollen success, resulting in new genotype frequencies. Pollen ﬁtness is a function of the proportion of stylar genotypes it can pollinate – which depends only on a pollen speciﬁcity’s frequency under self-recognition, and on the frequency

104 of styles to which it holds the key under nonself-recognition. These differ- ing rules are reﬂected in the genotype frequency after mating under self- and nonself-recognition incompatibility (Appendix C). We note that with nonself-recognition, no individual ever carries both a unique foreign and 0 unique local haplotype (PUFUL = 0) because neither haplotype holds the key to the other’s lock. We further note that these analytical results did not include pollen limitation, a biologically relevant process which we explore in our numeric iterations. We derive analytical results for a special case. Focusing on nonself- recognition and with no pollen limitation, the marginal ﬁtnesses of the shared haplotypes are −1 nS − 1 −1 −1 wSF = PUFUF aFF + PUFSF aFS + PUFSLaFS + nS nS − 2 −1 −1 −1 PSFSF aSS + PSFSLaSS + PSLSLaSS (3.1) nS

−1 nS − 1 −1 −1 wSL = PULULaLL + PULSLaLS + PULSF aLS + nS nS − 2 −1 −1 −1 PSFSF aSS + PSFSLaSS + PSLSLaSS (3.2) nS

Taking the difference, the third term in each cancels out, −1 nS − 1 −1 −1 wSF − wSL = PUFUF aFF + PUFSF aFS + PUFSLaFS − nS −1 nS − 1 −1 −1 PULULaLL − PULSLaLS + PULSF aLS (3.3) nS and all remaining terms depend on the frequency of carriers for the unique

105 haplotypes. The sign of this difference is obvious when one population harbors no unique haplotypes. For example, when the local population has no unique S-haplotypes (nL = 0) and the foreign population has at least one unique S-haplotype (nF > 0), the ﬁtness of shared haplotypes from the foreign population exceeds that of shared haplotypes from the local population. Since wSL < wSF at all genotype frequencies, there is no non-zero equilibrium frequency of pSL. So long as pSF > 0 and pUF > 0, the SL haplotype will be eliminated at equilibrium. Similarly, when nL > 0 and nF = 0, wSL > wSF at all genotype frequencies. The biological interpretation of this result is straightforward. The only difference between shared haplotypes originating from different populations is their collection of pollen-function keys: all L haplotypes (shared or unique) are compatible with S-haplotypes initially unique to the local population (UL), while SF haplotypes are compatible with UF haplotypes. If UF haplotypes exist but UL haplotypes do not, SF haplotypes have all the pollen success of SL haplotypes plus additional pollen success on UF haplotypes. In this case, each SF haplotype is always ﬁtter than its SL counterpart, and selection will drive all SL haplotypes extinct unless opposed by biased migration. That is, a population harboring no unique haplotypes will have its shared haplotypes replaced by shared haplotypes from the other population. No such process occurs under self-recognition. This analytical result suggests an important role for the relative numbers of unique haplotypes in each population. To investigate the effect of these and other parameters in more general cases, we implemented the full deterministic model in R. Using this model, we tracked evolution of haplotype frequencies under scenarios varying the numbers of shared (nS) and unique haplotypes (nL and nF ), the presence (b = 1/2) or absence of pollen limitation (b = 0), the duration of migration (continuous or a single pulse), and the form of self-incompatibility (self- or nonself-recognition).

106 For this part of the investigation, we kept migration unidirectional from foreign to local (mLF = 0). Migration persisted throughout the iterations if continuous or ceased after a single generation if pulsed, and it occurred at the same modest rate (mFL = 0.01) whether continuous or pulsed. After observing that nonself-recognition sometimes facilitated and sometimes prevented introgression of migrant haplotypes, we quantiﬁed the minimum value of mFL that resulted in invasion of SF for different values of nL and nF under nonself-recognition. We varied mFL from 0–0.1 in increments of 0.01 and varied nL and nF from 0–10. We considered values of nS = 5 or 20 and b = 0 or 1/2 but held mLF = 0 constant. Invasion was considered to have occurred if pUF + pSF > pUL + pSL in the local population at generation 1000. Note that this deﬁnition of invasion does not imply positive selection on the SF haplotypes: a neutral haplotype would also invade under this unidirectional migration scheme. Finally, we tracked haplotype frequencies for several bidirectional migration scenarios. We considered four scenarios in which migration and the number of shared haplotypes were held constant (mLF = mFL − 1, nS = 5), in which one (nL = 0 and nF = 1) or both populations possessed a unique allele (nL = nF = 1), and in which pollen limitation was either present (b = 1/2) or absent (b = 0). Additionally, we more closely investigated the haplotype freqeuncy trajectories in another four scenarios, which had resulted in coexistence of foreign and local haplotypes at equilibrium. In these scenarios, the numbers of shared and unique haplotypes and the strength of pollen limitation were constant (nS = 20, nL = nF = 1, b = 1/2), and migration was either low and symmetric (mLF = mFL = 0.02), high and symmetric (mLF = mFL = 0.09), asymmetric with more immigration to the local population (mLF = 0.05, mFL = 0.06), or asymmetric with more immigration to the foreign population (mLF = 0.05, mFL = 0.04).

107 3.3 Results

Self-recognition-based SI: The consistent outcome of self-recognition was for rare pistil speciﬁcities to rise to higher frequencies regardless of their origin (Fig. 3.2). All unique haplotypes always coexisted at equilibrium for both ongoing unidirectional migration (Fig. 3.2A,B) and a one time migration pulse (Fig. 3.2C,D). Under self-recognition, pollen limitation did not severely impact the spread of foreign S-haplotypes (cf., the right hand columns of Fig. 3.2A and Fig. 3.2C, to Fig. 3.2B and Fig. 3.2D) because styles of local and foreign shared and unique haplotypes can accept all nonself pollen. Rather, the equilibrium frequency of SL and SF haplotypes was determined entirely by migration pressure. Under continuous unidirectional migration, shared haplotypes originating in the donor population globally displaced shared haplotypes originating in the recipient population (Fig. 3.2). But under a single pulse of unidirectional migration, when only selection was ongoing, the initially low-frequency unique haplotypes migrating from the foreign population (UF ) rose to match the frequency of the haplotypes unique to the local population (UL), while shared haplotypes (SF ) remained at their initial low frequency (Fig. 3.2). We note that the equilibrium frequency of all shared haplotypes relative to all unique haplotypes increased as the number of shared haplotypes increased (compare dashed and full lines in Fig. 3.2), as would be expected from increasing the number of haplotypes in a category and counting them collectively. A similar result also occurs with nonself-recognition, as seen in Fig. 3.3. Because results from the case of self recognition are straightforward and conﬁrmed current understanding, we spend the remainder of our efforts on the results from nonself-recognition SI.

108 Nonself-recognition-based SI: Under nonself-recognition, the results are sensitive to parameter values, and many outcomes were possible. Without pollen limitation and with ongoing unidirectional gene flow, foreign unique S-haplotypes can often invade. For example, with one foreign and no local unique S-haplotypes, the unique foreign S-haplotype rapidly establishes and reaches its equilibrium frequency (Bottom row, first column of Fig. 3.3A). This process is somewhat slower when the local population also has a unique S-haplotype (Bottom row, second column of Fig. 3.3A). Unlike the case of self-recognition, with nonself-recognition, the foreign unique S-haplotype replaces the local unique S-haplotype (when green lines in e.g. the bottom panel of Fig. 3.3A are above zero, pink lines head towards zero). Additionally, because pollen of both local shared and local unique S-haplotypes is incompatible with styles with unique foreign locks, its fitness decreased as the unique foreign S-haplotype rose in frequency, consistent with our analytical predictions (Eq. 3.3). In contrast, the shared foreign S-haplotype enjoyed rapidly increasing pollen fitness as the unique foreign haplotype rose in frequency (compare the slower rise in frequency of shared foreign haplotypes in the bottom row of Fig. 3.3A to that in Fig. 3.2A). As a result, foreign shared haplotypes replaced local shared haplotypes, just as foreign unique haplotypes replaced local unique haplotypes. In the absence of pollen limitation, increasing the number of shared haplotypes from 5 to 20 did not qualitatively affect the outcome of migration but modified the equilibrium frequency of shared and unique haplotypes (compare dashed and full lines Fig. 3.3A), as observed in the case of self-recognition, above. By contrast, the number of shared S-haplotypes mediated the spread of foreign S-haplotypes when there was pollen limitation. For example, in Fig. 3.3B, foreign haplotypes cannot invade and replace local haplotypes when there are five shared haplotypes but can

109 when there are twenty. This difference reflects the extent of pollen limitation faced by unique foreign haplotypes – within twenty shared haplotypes, a plant with a unique foreign S-haplotype can be fertilized by 95% (20 of 21) of local pollen haplotypes, a large increase from the 83% (5 of 6) in the case of five shared S-haplotypes. Despite long-term balancing selection favoring the proliferation of S- haplotypes, the initial disadvantages of rejecting most local pollen as a maternal plant and being rejected as pollen by unique local haplotypes can disfavor rare foreign haplotypes. As such, although foreign S-haplotypes displace local ones with ongoing migration (Figures 3.3A,B), a single small pulse of migration does not result in the invasion of foreign haplotypes unless there are no unique local haplotypes (Figures 3.3C,D) under nonself- recognition. To quantify the effects of shared and unique haplotypes, as mediated by the extent of gene flow, we calculated the minimum value of mFL for which pUF + pSF > pUL + pSL at generation 1000 (Fig. 3.4). Without pollen limitation, invasion was possible for a broad range of nL and nF values, though it could occur even at low migration rates when nF ≥ nL. With pollen limitation and nS = 5 shared haplotypes, invasion was only possible for mFL ≤ 0.1 when nF ≥ nL. However, the exact effect of pollen limitation depended on nS. At nS = 5, pollen limitation consistently raised the threshold migration for invasion. At nS = 20, pollen limitation usually raised the threshold when nL < nF but lowered it when nL > nF . When pollen is limiting and there is some migration, the seed success of unique haplotypes is reduced. When there are many haplotypes, the pollen advantage of rarity is smaller. These disadvantages combined may be sufficient to eliminate rare unique haplotypes, thereby eliminating a barrier to the invasion of foreign haplotypes whether shared or unique. We demonstrate such a process for a case of bidirectional migration below.

110 When migration was equal in both directions, the equilibrium state depended mainly on the numbers of unique haplotypes. If only one population possessed a unique haplotype, the shared haplotype from the same population globally replaced the shared haplotype from the other population alongside the invading unique haplotype (Fig. 3.5). If both populations possessed a unique haplotype, both populations converged on the same equilibriumin the absence of pollen limitation or maintained their initial states in the presence of pollen limitation (Fig. 3.5). Under broader ranges of migration rates, including unequal ones, four results were possible: elimination of local haplotypes, elimination of foreign haplotypes, coexistence of foreign and local haplotypes within each population, or maintenance of each unique haplotype only in its population of origin (Fig. 3.6). The population sending more emigrant pollen tended to preserve its haplotypes. Without pollen limitation, coexistence occurred easily, but the exact equilibria of foreign and local haplotypes depended on migration rates. With pollen limitation, coexistence only occurred in a narrower band of intermediate migration rates. In most cases, both populations converged on the same equilibrium as long as migration was nonzero in either direction. But in the presence of both pollen limitation and few shared haplotypes, each population could maintain its own unique haplotype at high frequency while preventing much introgression of the other population’s unique haplotype. Sufﬁciently biased migration could overcome this maintenance of distinct haplotypes and allow one unique haplotype to replace the other globally. In some cases, pollen limitation could drive unique haplotypes from both populations extinct (Fig. 3.7). Since unique haplotypes could not accept immigrant pollen, they suffered reduced seed success compared to shared haplotypes. Once the unique haplotypes were gone, the distinction between shared haplotypes from different populations became irrelevant

111 to selection: each kind carried a key to a different extinct lock. The shared haplotypes could then coexist in the absence of unique haplotypes.

3.4 Discussion

Existing theory predicts that rare advantage at an S-locus will elevate gene flow by favoring rare immigrant S-haplotypes [e.g., Pickup et al., 2019]. We confirm this prediction for self-recognition systems, but find that nonself- recognition can either enhance or reduce gene flow depending on the S-haplotypes present in each population. The crucial parameter is the number of S-haplotypes unique to each population because pollen originating from one population is incompatible with S-haplotypes unique to the other. Unique S-haplotypes can act as a barrier to incoming gene flow by rejecting foreign pollen. At the same time, they can enhance outgoing gene flow because, once they enter the other population, they reject resident pollen and reduce its fitness relative to immigrant pollen. We find that introgression of S-haplotypes from the population with more S-haplotypes to the population with fewer is usually elevated, while introgression in the reverse direction is usually reduced. If this haplotype-based bias points in the same direction as the bias in migration, or if migration is unbiased, introgression is elevated. But the haplotype-based bias instead impedes introgression if it is opposite to the direction of migration bias. We should therefore expect very different patterns of gene flow at the S-locus in taxa with nonself-recognition, the ancestral state in core eudicots [Igic´ and Kohn, 2001], and taxa that have evolved some form of self-recognition, as in Papaveraceae and Brassicaceae. Importantly, biased introgression at the S-locus under non-self recognition applied to shared haplotypes as well as unique haplotypes. As a unique migrant haplotype invades, the advantage of compatibility with that

112 haplotype increases. Thus, a unique haplotype can invade by the advantage of rarity while the shared haplotypes from the same population ride its coattails. A common pattern we observed was that the population with more unique haplotypes or higher emigration rates essentially swamped the other population, replacing both unique and shared haplotypes. The role of introgressed locks in facilitating the invasion of keys originating from the same population is analogous to the surprising effect of introgressed female preferences on male traits in sexual selection predicted by Servedio and Burger¨ [2014]. Sexual selection has classically been viewed as a barrier to introgression because, if two populations have divergent female preferences, migrant males with out-of-place traits will suffer reduced reproductive fitness. However, female preferences may themselves introgress if they are not under direct selection. As a prefer- ence introgresses, the reproductive fitness of the corresponding male trait increases, allowing the trait to increase in frequency. This facilitated introgression can counteract the fitness advantage of so-called “magic traits” that are both adaptive in the local environment and initially preferred by local females. Pistil-expressed self-incompatibility locks are analogous to strong female preferences for a given pollen-expressed key, analogous to a male trait. The difference is that, instead of one male trait with different values in each population, pollen keys are a collection of many binary traits. Thus, pollen with more keys may be preferable to many seed parents in both populations, resulting in a directional asymmetry absent from pure Fisherian sexual selection. Pollen limitation and the number of shared haplotypes also affected introgression of foreign S-haplotypes. Pollen limitation typically reduced introgression and could result in the maintenance of two effectively isolated populations with their own sets of haplotypes. In contrast, introgression increased as the number of shared haplotypes increased. Shared hap-

113 lotypes, unlike unique haplotypes, could accept immigrant pollen. When there were many shared haplotypes, immigrant pollen was rarely rejected and had high fitness. Surprisingly, when migration was bidirectional and roughly symmetric, pollen limitation could instead eliminate unique haplotypes in both populations, which reduced the overall number of haplotypes but allowed shared haplotypes to introgress freely. Unique haplotypes were lost because they rejected migrant pollen and thus suffered greater pollen limitation than shared haplotypes, which did not. This is a consequence of the model’s square root function of pollen limitation, in which seed success monotoni- cally increases with compatible pollen and never fully flattens. Therefore, accepting more pollen always increases seed success somewhat, and unique haplotypes are disadvantaged as seed parents. If seed success instead plateaus, unique haplotypes should still be disadvantaged unless both shared and unique haplotypes accept enough pollen to reach the plateau. From a genomic perspective, each invasion of a unique migrant haplotype under nonself-recognition should reduce among-population neutral divergence for all haplotypes at the S-locus. In contrast, invasion of a migrant haplotype under self-recognition should only reduce divergence among copies of a single haplotype. We therefore predict greater among-population neutral divergence at the S-locus in species with self- recognition than in those with nonself-recognition. Surprisingly, a greater rate of S-haplotype diversification might therefore lead to reduced divergence between populations at the S-locus. Data on S-allele overlap and divergence across populations is available, at least for self-recognition. In Arabidopsis halleri and A. lyrata, which possess the self-recognition- based S-locus receptor kinase (SRK) incompatibility system, Castric et al. [2008] found 18 pairs of SRK alleles diverging by less than or equal to 12

114 substitutions. The remaining 12 alleles in A. halleri and 20 alleles in A. lyrata had no such counterparts in the other population. The low divergence within each pair could be explained by elevated gene ﬂow of these alleles but not by the mere reduced effective population size of a single S-allele. This pattern is consistent with our expectation for self-recognition that migrant S-alleles will rise to high frequency without greatly reducing diversity in the S-locus as a whole. However, we also predict that any pairs of functionally identical S-alleles that were initially shared by the two species would have persisted in each population. It is possible that these originally shared alleles have since been lost in one or both populations or that they have since accumulated enough neutral or functional divergence that they are no longer recognizably shared. We also note that the SRK system is sporophytic rather than the gametophytic self-recognition system we modeled. More comprehensive theory already exists for the interaction between haplotype number and migration in the case of self-recognition. In a model combining migration and balancing selection (either symmetrical overdominance or SI), Muirhead [2001] predicted for a given migration rate both the expected total number of alleles and the distribution of the proportion of alleles shared between two populations, three populations, etc. For the total number, she found a non-monotonic relationship in which allele number is minimized for intermediate migration rates. This non-monotonic relationship was previously observed in simulations by Schierup [1998]. For the proportions of shared alleles, she found that increasing the migration rate skewed the distribution towards alleles shared among many populations. In the SI version of this model, all unlike S-haplotypes were assumed to be cross-compatible. This is equivalent either to nonself-recognition in which all haplotypes are complete or to self-recognition. However, it is not equivalent to nonself-recognition in which some haplotypes are more complete

115 than others. Theory on both self- and nonself-recognition has revealed hurdles to S-haplotype diversification and suggested that gene flow can help populations overcome these hurdles [Uyenoyama et al., 2001, Gervais et al., 2011, Harkness et al., 2019b]. Our results constrain how S-haplotype diversification could occur in a subdivided population or metapopulation. Uyenoyama et al. [2001] found that local turnover, replacement of old haplotypes without increasing the total number, was possible under a much broader set of parameter values than local diversification. They therefore hypothesized that diversification occurs through local turnover followed by introgression of the new haplotype into the metapopulation and reintroduction of the lost haplotype from the metapopulation. We predict under this process that novel S-haplotypes from the same population can easily spread simultaneously, but novel haplotypes from different populations may interfere and eliminate each other. Reintroduction of a haplotype lost to turnover would require the reintroduced haplotype to become compatible with the novel haplotype. Until cross-compatibility is restored, the novel haplotype may continue to replace the locally lost haplotype in every subpopulation. Gene conversion could restore cross-compatibility [Kubo et al., 2015, Fujii et al., 2016], potentially halting the loss of a haplotype [Bod’ova´ et al., 2018, Harkness et al., 2019b]. Although we isolated effects of migration, in nature it should operate simultaneously with mutation and gene conversion. There may be a separa- tion of time scales such that migration resolves itself before new mutations or gene conversions can occur, but certain mutations or gene conversions might push the system toward a new equilibrium. Some of the most common events might be those that induce self-compatibility: either the lock suffers a loss-of-function mutation or a haplotype acquires the key to its own lock by gene conversion. Mutant haplotypes with a nonfunctional lock

116 would reject no pollen and would reduce pollen limitation, and these self- compatible mutants might introgress more easily than self-incompatible haplotypes. In contrast, a haplotype with the key to its own lock would only affect the pollen phenotype: self pollen would be compatible, but the maternal plant would still reject the same subset of nonself pollen. While acquiring a key (which offers new siring opportunities) would typically increase fitness more than losing a lock, the substantial pollen limitation suffered by migrant haplotypes might reverse this inequality: losing a lock would completely eliminate pollen limitation by accepting all pollen, while gaining a key would only mitigate pollen limitation by accepting self pollen. If accepting self pollen is insufficient to achieve full seed set, losing a lock should provide greater benefit to seed set than gaining a key. We assumed that all haplotypes were incompatible with haplotypes unique to other populations. But if the difference in S-haplotypes between populations was caused by recent differential loss, it is likely that each population would temporarily retain cross-compatibility with locally lost haplotypes. That is, the loss of a pistil-function lock does not imply immediate loss of the corresponding pollen-function key from all other haplotypes. However, mutations to this now-neutral key would eventually render it nonfunctional. Another possibility is that, if two pistil specificities correspond to the same dual-function pollen key, cross-compatibility with a foreign pistil lock might have been maintained as a byproduct of selection for compatibility with a local pistil lock. Furthermore, even if foreign haplotypes are initially rejected by local unique haplotypes, the foreign haplotypes could acquire the missing local key through gene conversion with local shared haplotypes. If a haplotype is being driven extinct, it may be rescued by acquiring a missing key and increasing its pollen fitness [Bod’ova´ et al., 2018, Harkness et al., 2019b]. We have only modeled the S-locus, but any S-haplotype in nature

117 comes as part of a whole parental genome. That genome might not be as well adapted to the local environment as resident genomes. In this case, linkage disequilibrium between the S-locus and loci involved in environmental adaptation might slow the introgression of migrant S-haplotypes. Recombination can break down this linkage disequilibrium in the long run, but there may be few opportunities to recombine off a locally maladapted background if hybrids are rare. Gene flow at the S-locus, like other loci, should be limited by the density of locally adapted loci around it. The discovery of collaborative nonself-recognition has unexpected theoretical implications for the behavior of the S-locus. Recent theory on S-haplotype diversification reveals complicated dynamics of collapse and rescue [Bod’ova´ et al., 2018, Harkness et al., 2019b]. Similarly, we show that gene flow is more complex under nonself- than under self-recognition. The S-locus, an ancient and widespread feature controlling the breeding system of many flowering plants, seems only to get stranger the more it is investigated.

118 Theory on the evolution of plant mating system and self-incompatibility reveals many counterintuitive expectations. For self-incompatibility, these expectations are logically necessary consequences of the leading hypothesis for the genetic control of the system. They lead to testable predictions of the collections of S-haplotypes within and among populations, as well as the compatibility relationships among haplotypes. There should be among-population variation in S-haplotype number partially caused by collapses in S-haplotype diversity, with small populations containing few functional haplotypes but also pseudogene remnants of pollen-expressed genes conferring compatibility with many locally lost haplotypes. Isolated conspeciﬁc populations should contain both cross-compatible and cross- incompatible haplotypes, observable through controlled crosses, with each population containing pollen pseudogenes corresponding to differentially lost haplotypes, but with no such pseudogenes for haplotypes truly novel to one population. Several of these tests are currently limited by the difﬁ- culty of sequence assembly for the highly repetitive S-locus, but ongoing advancements in long-read sequencing technology may greatly simplify this problem. The study of plant mating system continues to be a produc- tive exchange between empirical and theoretical research.

119 a Self-recognition Nonself-recognition Local Foreign Local Foreign Population Population Population Population

S1 S1 S1 S1 S2 S2 S2 S2 S3 S3 S3 S3 S4 S4

b Pollen grain S3 pollen genotype 3 3 3 1 2 Stigma Style Pollen tube phenotype 1 Style phenotype 3 2 = RNase S1S4 pistil 1 4 1 4 genotype = F-box

Figure 3.1: Pollen compatibility under self- and nonself-recognition. The focal S3 haplotype is shared and originates from the local population. a shows the consequences of self- and nonself-recognition for compatibility with other haplotypes. Under self- recognition, S3 pollen is only incompatible with S3-carrying plants. Under nonself- recognition, S3 pollen is still incompatible with S3-carrying plants, but it is also incompatible with carriers for haplotypes unique to another population (S4). b details the mech- anistic basis of rejection for a local S3 pollen grain on a foreign S1S4 plant for self- and nonself-recognition. Under self-recognition, neither S1 nor S4 matches S3, so the pollen is not rejected. Under nonself-recognition, the local S3 haplotype’s key ring contains keys to all locks present in the local population (allowing it to unlock S1) but not the keys to locks unique to the foreign population (S4), so it is rejected. We place both rejection mechanisms in the style to match the RNase-based system, but rejection could occur at the stigma (as it does in some taxa) without affecting the model.

120 ie hwtecsso wnyadﬁesae altps epciey Generations respectively. haplotypes, shared ﬁve full and and a twenty Dashed on of increase respectively. cases haplotypes, foreign the and show local lines unique of number the show ( 0.01 pulse and migration ( time 0.01/generation one of a rate for sults migration Re- (unidirectional) SI. steady based a self-recognition with for S-haplotypes sults (pink), unique local (blue), unique local shared and (orange), (green), foreign foreign shared of dynamics Evolutionary 3.2: Figure Self recognition 3.2C Migration Pulse Continuous Migration n c a s n ih( with and )

h Frequency Frequency a 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 r e d log 10 10 10 SL SL 0 0 cl ntex-axis. the on scale 5 3.2B 10 10 0 Local 0 Local 1 1 No Pollen Limitation and 10 10 20 2 2 Generation Generation 3.2D UF UF SF SF SL SL 10 10 Allele class 3 3 olnlmtto.Fct tpadbsd h plots the beside and atop Facets limitation. pollen ) 10 10 UL UL UL SL SL 0 0 3.2C 121 10 10 1 Local 1 Local 1 1 and 10 10 2 2 3.2D SF UF UF SF UL SF SL SL 10 10

.W hwrslswtot( without results show We ). 3 3

1 Foreign 1 Foreign 0 1 Foreign 1 Foreign 0 SL d b 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 Pollen Limitation 10 10 3.2A UL UL UL SL SL 0 0 UF Generation Generation 10 10 1 Local 1 Local and 1 1 10 10 UL 3.2B 2 2 UF UL SF SF SL SL 10 10

.Re- ).

3 3

3.2A 1 Foreign 1 Foreign 0 Foreign 1 Foreign 0 ullnsso h ae ftet n v hrdhpoye,rsetvl.Generations respectively. haplotypes, shared ﬁve a and on and twenty increase Dashed of respectively. cases haplotypes, the foreign show and lines local full unique of number the show plots eut o n iemgainple00 ( 0.01 pulse migration ( ( time 0.01/generation one of a rate for migration Results (unidirectional) SI. steady based nonself-recognition a with for S-haplotypes Results (pink), unique local (blue), unique local and shared (green), (orange), foreign foreign shared of dynamics Evolutionary 3.3: Figure 3.2A Non−self recognition

and Migration Pulse Continuous Migration n c a s h

3.2C Frequency Frequency a 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 r e d log n ih( with and ) 10 10 10 SL SL SL 0 0 cl ntexai.Fgr aoti dnia ota nFgr 3.2. Figure in that to identical is layout Figure x-axis. the on scale 5 10 10 0 Local 0 Local 1 1 No Pollen Limitation 3.2B 10 10 20 2 2 Generation Generation and UF UF SF SF SF SF SL 10 10 Allele class 3 3 10 10 3.2D UL UL SL SL 0 0 olnlmtto.Fct tpadbsd the beside and atop Facets limitation. pollen ) 122 10 10 1 Local 1 Local 1 1 3.2C 10 10 2 2 and SF UF SF UL UL SL SL 10 10

3.2D 3 3

1 Foreign 1 Foreign 0 1 Foreign 1 Foreign 0 SL .W hwrslswithout results show We ). d b 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 Pollen Limitation 10 10 UL UL SL SL 0 0 UF Generation Generation 10 10 3.2A 1 Local 1 Local 1 1 10 10 and UL 2 2 UL UL SF SF SL SL 10 10 3.2B

3 3

1 Foreign 1 Foreign 0 Foreign 1 Foreign 0 ). No pollen limitation Pollen limitation

10 x x x x x x x x x x x x x x x x x x x x Few shared alleles 9 x x x x x x x x x x x x x x x x x x x 8 x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x

) 7

L 6 x x x x x x x x x x x x x x x x x 5 x x x x x x x x x x x x x x x x 4 x x x x x x x x x x x x x x x 3 x x x x x x x x x 2 x x x x 1 x x 0

10 x x x x x x x x Many shared alleles 9 x x x x x x x 8 x x x x x x 7 x x x x x 6 x x x x 5 x x x 4 x x

# local unique alleles (n 3 x 2 x 1 0 0 1 2 3 4 5 6 7 8 910 0 1 2 3 4 5 6 7 8 910 # foreign unique alleles (nF) Minimum migration x for invasion 0.00 0.02 0.04 0.06 0.08 0.1 >0.1

Figure 3.4: Effect of unique haplotypes on invasion threshold. The minimum rate of unidirectional migration (mFL) needed for the frequency of all foreign haplotypes to exceed the frequency of all local haplotypes (pUF + pSF > pUL + pSL) in the local population at generation 1000. Invasion could occur at low migration rates when the number of foreign unique haplotypes (nF ) exceeded the number of local unique haplotypes (nL) but required high migration rates when nL > nF . Many shared haplotypes (nS = 20) lowered the invasion threshold compared to few shared haplotypes (nS = 5). Pollen limitation raised the invasion threshold when nS = 5, but when nS = 20, the effect of pollen limitation was to erase partly the effect of nF . Seed set for a genotype was a square root function of the proportion of pollen compatible with that genotype. The y = x line is shown in faint red.

123 1 Foreign, 0 Local 1 Foreign, 1 Local

a 1.00 Foreign pop b 1.00 Foreign pop 0.75 SF 0.75 0.50 0.50 SF 0.25 UF 0.25 UF SL UL UL 0.00 0.00 1.00 1.00 Local pop Local pop 0.75 SL SF 0.75 Frequency Frequency 0.50 0.50 SL 0.25 UF 0.25 UL SF UL UF

No Pollen Limitation No Pollen 0.00 0.00 1 10 100 1000 1 10 100 1000 Generation Generation

c 1.00 Foreign pop d 1.00 Foreign pop 0.75 SF 0.75 SF 0.50 0.50 0.25 UF 0.25 UF UL UL 0.00 0.00 1.00 1.00 Local pop Local pop 0.75 SL SF 0.75 SL Frequency Frequency 0.50 0.50 0.25 UF 0.25 UL

Pollen Limitation Pollen UL 0.00 0.00 1 10 100 1000 1 10 100 1000 Generation Generation

Allele class SF SL UF UL

Figure 3.5: Bidirectional migration. Migration rates are mFL = mLF = 0.01. There were nS = 5 shared haplotypes. All local haplotypes were lost when only the foreign population possessed a unique haplotypes, regardless of pollen limitation (A, C). When both population possessed one unique haplotype, local and foreign haplotypes coexisted either by converging toward the same intermediate equilibrium frequencies in both populations in the absence of pollen limitation (B) or by being preserved at high frequencies in their original populations in the presence of pollen limitation (D).

124 Local population Foreign population Many shared haplotypes

0.10 No pollen limitation 0.08 0.06 0.04 0.02 0.00 Few shared haplotypes

0.10 No pollen limitation ) L

F 0.08 M

( 0.06 0.04 0.02 0.00 0.10 Many shared haplotypes Pollen limitation 0.08 0.06 0.04 0.02 Foreign to local migration rate Foreign 0.00

0.10 Few shared haplotypes Pollen limitation 0.08 0.06 0.04 0.02 0.00 0.00 0.02 0.04 0.06 0.08 0.100.00 0.02 0.04 0.06 0.08 0.10 Local to foreign migration rate (MLF)

Frequency of foreign haplotypes 1.00 0.75 0.50 0.25 0.00

Figure 3.6: Equilibrium frequency of foreign haplotypes in each population with bidirectional migration. Self-incompatibility functions through nonself-recognition in all panels.

Each population harbors one unique haplotype (nL = nF = 1), and many (nS = 20) or few (nS = 5) shared haplotypes. In the absence of pollen limitation, foreign and local haplotypes usually coexist unless migration rates are extremely biased. With pollen limitation, moderately biased migration rates result in the loss of foreign or local haplotypes, and coexistence is only possible for a narrower band of more nearly equal migration rates.

125 mLF = mFL = 0.02 mLF = mFL = 0.09 a 1.00 Foreign pop b 1.00 Foreign pop 0.75 0.75 SF SF 0.50 0.50 SL SL 0.25 0.25 0.00 0.00

1.00 Local pop 1.00 Local pop 0.75 0.75 SL SL Frequency 0.50 Frequency 0.50 SF SF 0.25 0.25 0.00 0.00

Symmetric Migration 1 10 100 1 10 100 Generation Generation

mLF = 0.05, mFL = 0.06 mLF = 0.05, mFL = 0.04 c 1.00 Foreign pop d 1.00 Foreign pop 0.75 SF 0.75 SL 0.50 0.50 0.25 SL 0.25 SF 0.00 0.00

1.00 Local pop 1.00 Local pop 0.75 SF 0.75 SL Frequency 0.50 Frequency 0.50 0.25 SL 0.25 SF 0.00 0.00 1 10 100 1 10 100 Asymmetric Migration Generation Generation

Allele class SF SL UF UL

Figure 3.7: Loss of diversity. At some intermediate migration rates, shared haplotypes (SL and SF ) can coexist while all unique haplotypes (UF and UL) are lost. This outcome can occur for both symmetric (A, B) and asymmetric (C, D) migration rates, with asymmetric rates altering the equilibrium frequencies of the shared alleles. There were nS = 20 shared haplotypes, nL = 1 unique local haplotype, and nF = 1 unique foreign haplotype, and seed success was a square root function of compatible pollen.

126 Bibliography

127 Bibliography

W. S. Armbruster. Evolution of plant pollination systems: hypotheses and tests with the neotropical vine Dalechampia. Evolution, 47(5):1480– 1505, 1993.

S. C. Barrett and J. S. Shore. Variation and evolution of breeding systems in the Turnera ulmifolia L. complex (Turneraceae). Evolution, 41(2):340– 354, 1987.

S. C. H. Barrett, L. D. Harder, and A. C. Worley. The comparative biology of pollination and mating in ﬂowering plants. Phil. Trans. R. Soc. Lond. B, 351(1345):1271–1280, 1996.

W. Bateson. Heredity and variation in modern lights. In A. C. Seward, editor, Darwin and modern science, pages 85–101. Cambridge University Press, 1909.

G. Bena, B. Lejeune, J.-M. Prosperi, and I. Olivieri. Molecular phylogenetic approach for studying life-history evolution: the ambiguous example of the genus Medicago L. Proc. R. Soc. Lond., B, Biol. Sci., 265(1401): 1141–1151, 1998.

N. Bierne, T. Lenormand, F. Bonhomme, and P. David. Deleterious mutations in a hybrid zone: can mutational load decrease the barrier to gene ﬂow? Genet. Res., 80(3):197–204, 2002.

128 K. Bod’ova,´ T. Priklopil, D. L. Field, N. H. Barton, and M. Pickup. Evolution- ary pathways for the generation of new self-incompatibility haplotypes in a nonself-recognition system. Genetics, 209(3):861–883, 2018.

W. Boucher. A deterministic analysis of self-incompatibility alleles. Journal of Mathematical Biology, 31(2):149–155, 1993.

J. W. Busch. Heterosis in an isolated, effectively small, and self-fertilizing population of the ﬂowering plant Leavenworthia alabamica. Evolution, 60(1):184–191, 2006.

J. W. Busch and L. F. Delph. The relative importance of reproductive assurance and automatic selection as hypotheses for the evolution of self- fertilization. Annals of Botany, 109(3):553–562, 2011.

D. Byers and D. Waller. Do plant populations purge their genetic load? Effects of population size and mating history on inbreeding depression. Annual Review of Ecology and Systematics, 30(1):479–513, 1999.

A. Caballero and P. D. Keightley. A pleiotropic nonadditive model of variation in quantitative traits. Genetics, 138(3):883–900, 1994.

V. Castric, J. Bechsgaard, M. H. Schierup, and X. Vekemans. Repeated adaptive introgression at a gene under multiallelic balancing selection. PLoS Genetics, 4(8), 2008.

B. Charlesworth and D. Charlesworth. Elements of evolutionary genetics. Roberts and Company Publishers Greenwood Village, CO, 2010.

D. Charlesworth and B. Charlesworth. The evolution and breakdown of S-allele systems. Heredity, 43(1):41–55, 1979.

D. Charlesworth and J. H. Willis. The genetics of inbreeding depression. Nat. Rev. Genet., 10(11):783–796, 2009.

129 D. Charlesworth, M. Morgan, and B. Charlesworth. Inbreeding depression, genetic load, and the evolution of outcrossing rates in a multilocus system with no linkage. Evolution, 44(6):1469–1489, 1990.

P. Crnokrak and S. C. Barrett. Perspective: purging the genetic load: a review of the experimental evidence. Evolution, 56(12):2347–2358, 2002.

T. Dobzhansky. Studies on hybrid sterility. Zeitschrift fur¨ Zellforschung und mikroskopische Anatomie, 21(2):169–223, 1934.

T. Dobzhansky. Genetics and the origin of species. Columbia University Press, 1937.

Douglas Nychka, Reinhard Furrer, John Paige, and Stephan Sain. ﬁelds: Tools for spatial data, 2017. URL https://github.com/NCAR/Fields.R package version 10.3.

E. M. East and A. J. Mangelsdorf. A new interpretation of the hereditary behavior of self-sterile plants. Proceedings of the National Academy of Sciences of the United States of America, 11(2):166–171, 1925.

S. Emerson. The genetics of self-incompatibility in Oenothera organensis. Genetics, 23(2):190–202, 1938.

S. Emerson. A preliminary survey of the Oenothera organensis population. Genetics, 24(4):524–537, 1939.

T. Entani, M. Iwano, H. Shiba, F. Che, A. Isogai, and S. Takayama. Com- parative analysis of the self-incompatibility (S-) locus region of Prunus mume: identiﬁcation of a pollen-expressed F-box gene with allelic diversity. Genes to Cells, 8(3):203–213, 2003.

130 J. S. Escobar, A. Cenci, J. Bolognini, A. Haudry, S. Laurent, J. David, and S. Glemin.´ An integrative test of the dead-end hypothesis of selﬁng evolution in Triticeae (Poaceae). Evolution: International Journal of Organic Evolution, 64(10):2855–2872, 2010.

M. W. Feldman and U. Liberman. An evolutionary reduction principle for genetic modiﬁers. Proc. Natl. Acad. Sci. U.S.A., 83:4824–4827, 1986.

R. A. Fisher. XXI.—On the dominance ratio. Proceedings of the Royal Society of Edinburgh, 42:321–341, 1923.

R. A. Fisher. Average excess and average effect of a gene substitution. Ann. Eugen., 11(1):53–63, 1941.

L. Fishman and J. H. Willis. A novel meiotic drive locus almost completely distorts segregation in Mimulus (monkeyﬂower) hybrids. Genetics, 169 (1):347–353, 2005.

H. Foote, J. P. Ride, V. E. Franklin-Tong, E. A. Walker, M. J. Lawrence, and F. Franklin. Cloning and expression of a distinctive class of self- incompatibility (S) gene from Papaver rhoeas L. Proceedings of the National Academy of Sciences, 91(6):2265–2269, 1994.

R. Frankham, J. D. Ballou, M. D. Eldridge, R. C. Lacy, K. Ralls, M. R. Dudash, and C. B. Fenster. Predicting the probability of outbreeding depression. Conservation Biology, 25(3):465–475, 2011.

V. E. Franklin-Tong, J. P. Ride, N. D. Read, A. J. Trewavas, and F. C. H. Franklin. The self-incompatibility response in Papaver rhoeas is mediated by cytosolic free calcium. The Plant Journal, 4(1):163–177, 1993.

S. Fujii, K. Kubo, and S. Takayama. Non-self-and self-recognition models in plant self-incompatibility. Nature Plants, 2(9), 2016.

131 S. Garnier. viridis: Default Color Maps from ’matplotlib’, 2018. URL https://CRAN.R-project.org/package=viridis. R package version 0.5.1.

J. C. Gay, S. Myers, and G. McVean. Estimating meiotic gene conversion rates from population genetic data. Genetics, 177(2):881–894, 2007.

C. E. Gervais, V. Castric, A. Ressayre, and S. Billiard. Origin and diver- siﬁcation dynamics of self-incompatibility haplotypes. Genetics, 188(3): 625–636, 2011.

S. Glemin.´ How are deleterious mutations purged? Drift versus nonrandom mating. Evolution, 57(12):2678–2687, 2003.

E. E. Goldberg, J. R. Kohn, R. Lande, K. A. Robertson, S. A. Smith, and B. Igic.´ Species selection maintains self-incompatibility. Science, 330 (6003):493–495, 2010.

A. Goldraij, K. Kondo, C. B. Lee, C. N. Hancock, M. Sivaguru, S. Vazquez- Santana, S. Kim, T. E. Phillips, F. Cruz-Garcia, and B. McClure. Com- partmentalization of S-RNase and HT-B degradation in self-incompatible Nicotiana. Nature, 439(7078):805–810, 2006.

C. Goodwillie, S. Kalisz, and C. G. Eckert. The evolutionary enigma of mixed mating systems in plants: occurrence, theoretical explanations, and empirical evidence. Annual Review of Ecology, Evolution, and Sys- tematics, 36:47–79, 2005.

J. B. S. Haldane. A mathematical theory of natural and artiﬁcial selection, part V: selection and mutation. Mathematical Proceedings of the Cambridge Philosophical Society, 23(7):838–844, 1927.

132 A. Harkness and Y. Brandvain. Nonself-recognition-based self- incompatibility can alternatively promote or prevent introgression. bioRxiv, 2020.

A. Harkness, Y. Brandvain, and E. E. Goldberg. The evolutionary response of mating system to heterosis. Journal of evolutionary biology, 32(5): 476–490, 2019a.

A. Harkness, E. E. Goldberg, and Y. Brandvain. Diversiﬁcation or collapse of self-incompatibility haplotypes as outcomes of evolutionary rescue. bioRxiv, 2019b. DOI: 10.1101/641613.

A. Harkness, E. E. Goldberg, and Y. Brandvain. Diversiﬁcation or collapse of self-incompatibility haplotypes as a rescue process, 2020a. Dataset, https://doi.org/10.5061/dryad.08kprr50j.

A. Harkness, E. E. Goldberg, and Y. Brandvain. Diversiﬁcation or collapse of self-incompatibility haplotypes as a rescue process. The American Naturalist, Nov. 2020b. ISSN 0003-0147. doi: 10.1086/712424. Pub- lisher: The University of Chicago Press.

C. R. Herlihy and C. G. Eckert. Genetic cost of reproductive assurance in a self-fertilizing plant. Nature, 416(6878):320–323, 2002.

C. R. Herlihy and C. G. Eckert. Experimental dissection of inbreeding and its adaptive signiﬁcance in a ﬂowering plant, Aquilegia canadensis (Ranunculaceae). Evolution, 58(12):2693–2703, 2004.

M. S. Heschel and K. N. Paige. Inbreeding depression, environmental stress, and population size variation in scarlet gilia (Ipomopsis aggre- gata). Conserv. Biol., 9(1):126–133, 1995.

W. G. Hill and A. Robertson. The effect of linkage on limits to artiﬁcial selection. Genet. Res., 8(03):269–294, 1966.

133 S. J. Hiscock. Pollen recognition during the self-incompatibility response in plants. Genome Bilogy, 3(2):reviews1004.1, 2002.

S. J. Hiscock, S. M. McInnis, D. A. Tabah, C. A. Henderson, and A. C. Brennan. Sporophytic self-incompatibility in Senecio squalidus L. (Asteraceae)—the search for S. Journal of Experimental Botany, 54 (380):169–174, 2003.

S. Hoebee, S. Angelone, D. Csencsics, K. Ma¨att¨ anen,¨ and R. Holdereg- ger. Diversity of S-alleles and mate availability in 3 populations of self- incompatible wild pear (Pyrus pyraster). Journal of Heredity, 103(2): 260–267, 2011.

K. E. Holsinger. Inbreeding depression doesn’t matter: the genetic basis of mating-system evolution. Evolution, 42(6):1235–1244, 1988.

K. E. Holsinger, M. W. Feldman, and F. B. Christiansen. The evolution of self-fertilization in plants: a population genetic model. Am. Nat., 124(3): 446–453, 1984.

Z. Hua and T. Kao. Identiﬁcation and characterization of components of a putative Petunia S-locus F-box–containing E3 ligase complex involved in S-RNase–based self-incompatibility. The Plant Cell, 18(10):2531–2553, 2006.

S. Huang, H. S. Lee, B. Karunanandaa, and T. Kao. Ribonuclease activity of Petunia inﬂata S proteins is essential for rejection of self-pollen. The Plant Cell, 6(7):1021–1028, 1994.

B. Igic´ and J. W. Busch. Is self-fertilization an evolutionary dead end? New Phytologist, 198(2):386–397, 2013.

134 B. Igic´ and J. R. Kohn. Evolutionary relationships among self- incompatibility RNases. Proceedings of the National Academy of Sci- ences, 98(23):13167–13171, 2001.

B. Igic,´ L. Bohs, and J. R. Kohn. Historical inferences from the self- incompatibility locus. New Phytologist, 161(1):97–105, 2004.

B. Igic,´ R. Lande, and J. R. Kohn. Loss of self-incompatibility and its evolutionary consequences. International Journal of Plant Sciences, 169(1): 93–104, 2008.

A. Kachroo, M. E. Nasrallah, and J. B. Nasrallah. Self-incompatibility in the Brassicaceae: receptor-ligand signaling and cell-to-cell communication. Plant Cell, 14 Suppl:S227–S238, 2002. doi: 10.1105/tpc.010440.

M. Kimura. Conﬂict between self-fertilization and outbreeding in plants. Annu. Rep. Natl. Inst. Genet. Japan, 9:87–88, 1959.

K. Kubo, T. Entani, A. Takara, N. Wang, A. M. Fields, Z. Hua, M. Toyoda, S. Kawashima, T. Ando, A. Isogai, et al. Collaborative non-self recognition system in S-RNase–based self-incompatibility. Science, 330(6005): 796–799, 2010.

K. Kubo, T. Paape, M. Hatakeyama, T. Entani, A. Takara, K. Kajihara, M. Tsukahara, R. Shimizu-Inatsugi, K. K. Shimizu, and S. Takayama. Gene duplication and genetic exchange drive the evolution of S-RNase- based self-incompatibility in Petunia. Nature Plants, 1, 2015.

Z. Lai, W. Ma, B. Han, L. Liang, Y. Zhang, G. Hong, and Y. Xue. An F-box gene linked to the self-incompatibility (S) locus of Antirrhinum is expressed speciﬁcally in pollen and tapetum. Plant Molecular Biology, 50(1):29–41, 2002.

135 R. Lande and D. W. Schemske. The evolution of self-fertilization and inbreeding depression in plants. I. Genetic models. Evolution, 39(1):24– 40, 1985.

M. Lawrence. Population genetics of the homomorphic self-incompatibility polymorphisms in ﬂowering plants. Annals of Botany, 85(suppl 1):221– 226, 2000.

X. Li, N. Paech, J. Nield, D. Hayman, and P.Langridge. Self-incompatibility in the grasses: evolutionary relationship of the S gene from Phalaris coerulescens to homologous sequences in other grasses. Plant Molec- ular Biology, 34(2):223–232, 1997.

D. G. Lloyd. Some reproductive factors affecting the selection of self- fertilization in plants. Am. Nat., 113(1):67–79, 1979.

D. G. Lloyd. Self-and cross-fertilization in plants. II. The selection of self- fertilization. Int. J. Plant Sci., 153(3, Part 1):370–380, 1992.

D. Luu, X. Qin, D. Morse, and M. Cappadocia. S-RNase uptake by compatible pollen tubes in gametophytic self-incompatibility. Nature, 407(6804): 649–651, 2000.

T. Mackay, R. F. Lyman, and M. S. Jackson. Effects of P element insertions on quantitative traits in Drosophila melanogaster. Genetics, 130(2):315– 332, 1992.

B. A. McClure, V. Haring, P. R. Ebert, M. A. Anderson, R. J. Simpson, F. Sakiyama, and A. E. Clarke. Style self-incompatibility gene products of Nicotiana alata are ribonucleases. Nature, 342(6252):955–957, 1989.

C. A. Muirhead. Consequences of population structure on genes under balancing selection. Evolution, 55(8):1532–1541, 2001.

136 H. J. Muller. Isolating mechanisms, evolution, and temperature. In T. Dobzhansky, editor, Biological Symposia, volume 6, pages 71–125, 1942.

H. J. Muller. The relation of recombination to mutational advance. Mutat. Res., 1(1):2–9, 1964.

T. Nagylaki. The deterministic behavior of self-incompatibility alleles. Ge- netics, 79(3):545–550, 1975.

T. Nagylaki. A model for the evolution of self-fertilization and vegetative reproduction. J. Theor. Biol., 58(1):55–58, 1976.

E. Neuwirth. RColorBrewer: ColorBrewer Palettes, 2014. URL https://CRAN.R-project.org/package=RColorBrewer. R package version 1.1-2.

R. G. Olmstead. Biological and historical factors inﬂuencing genetic diversity in the Scutellaria angustifolia complex (Labiatae). Evolution, 44(1): 54–70, 1990.

H. A. Orr and R. L. Unckless. Population extinction and the genetics of adaptation. The American Naturalist, 172(2):160–169, 2008.

S. P. Otto and M. C. Whitlock. The probability of ﬁxation in populations of changing size. Genetics, 146(2):723–733, 1997.

N. Ouborg and R. van Treuren. The signiﬁcance of genetic erosion in the process of extinction. IV. Inbreeding load and heterosis in relation to population size in the mint Salvia pratensis. Evolution, 48(4):996–1008, 1994.

137 T. Paape, B. Igic,´ S. D. Smith, R. Olmstead, L. Bohs, and J. R. Kohn. A 15-myr-old genetic bottleneck. Molecular Biology and Evolution, 25(4): 655–663, 2008.

M. Pickup, Y. Brandvain, C. Fra¨ısse, S. Yakimowski, N. H. Barton, T. Dixit, C. Lexer, E. Cereghetti, and D. L. Field. Mating system variation in hybrid zones: facilitation, barriers and asymmetries to gene ﬂow. New Phytologist, 224(3):1035–1047, 2019.

H. Qiao, H. Wang, L. Zhao, J. Zhou, J. Huang, Y. Zhang, and Y. Xue. The F-box protein AhSLF-S2 physically interacts with S-RNases that may be inhibited by the ubiquitin/26S proteasome pathway of protein degradation during compatible pollination in Antirrhinum. The Plant Cell, 16(3):582–595, 2004.

R Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, 2020. URL https://www.R-project.org/.

C. M. Richards. Inbreeding depression and genetic rescue in a plant metapopulation. Am. Nat., 155(3):383–394, 2000.

E. B. Rosenblum, B. A. Sarver, J. W. Brown, S. Des Roches, K. M. Hard- wick, T. D. Hether, J. M. Eastman, M. W. Pennell, and L. J. Harmon. Goldilocks meets Santa Rosalia: an ephemeral speciation model ex- plains patterns of diversiﬁcation across time scales. Evolutionary Biol- ogy, 39(2):255–261, 2012.

S. Sakai. How have self-incompatibility haplotypes diversiﬁed? Generation of new haplotypes during the evolution of self-incompatibility from self- compatibility. The American Naturalist, 188(2):163–174, 2016.

138 D. W. Schemske and R. Lande. The evolution of self-fertilization and inbreeding depression in plants. II. Empirical observations. Evolution, 39 (1):41–52, 1985.

M. H. Schierup. The number of self-incompatibility alleles in a ﬁnite, subdivided population. Genetics, 149(2):1153–1162, 1998.

M. H. Schierup and X. Vekemans. Genomic consequences of selection on self-incompatibility genes. Current opinion in plant biology, 11(2): 116–122, 2008.

M. H. Schierup, X. Vekemans, and F. B. Christiansen. Allelic genealogies in sporophytic self-incompatibility systems in plants. Genetics, 150(3): 1187–1198, 1998.

M. R. Servedio and R. Burger.¨ The counterintuitive role of sexual selection in species maintenance and speciation. Proceedings of the National Academy of Sciences, 111(22):8113–8118, 2014.

P. Sijacic, X. Wang, A. L. Skirpan, Y. Wang, P. E. Dowd, A. G. McCub- bin, S. Huang, and T. Kao. Identiﬁcation of the pollen determinant of S-RNase-mediated self-incompatibility. Nature, 429(6989):302–305, 2004.

G. L. Stebbins. Flowering plants: evolution above the species level. The Belknap Press of Harvard University Press, Cambridge, Massachusetts, 1974.

J. C. Stein, B. Howlett, D. C. Boyes, M. E. Nasrallah, and J. B. Nasrallah. Molecular cloning of a putative receptor protein kinase gene encoded at the self-incompatibility locus of Brassica oleracea. Proceedings of the National Academy of Sciences, 88(19):8816–8820, 1991.

139 J. Steinbachs and K. Holsinger. S-RNase–mediated gametophytic self- incompatibility is ancestral in Eudicots. Molecular Biology and Evolution, 19(6):825–829, 2002.

W. Steiner and H.-R. Gregorius. Single-locus gametophytic incompatibility: the symmetric equilibrium is globally asymptotically stable. Journal of Mathematical Biology, 32(6):515–520, 1994.

P. Sun and T. Kao. Self-incompatibility in Petunia inﬂata: The relationship between a self-incompatibility locus F-box protein and its non-self S- RNases. Plant Cell, 25(2):470–485, 2013.

D. C. Tank, J. M. Eastman, M. W. Pennell, P. S. Soltis, D. E. Soltis, C. E. Hinchliff, J. W. Brown, E. B. Sessa, and L. J. Harmon. Nested radiations and the pulse of angiosperm diversiﬁcation: increased diversiﬁcation rates often follow whole genome duplications. New Phytologist, 207(2): 454–467, 2015.

K. Theodorou and D. Couvet. Inbreeding depression and heterosis in a subdivided population: inﬂuence of the mating system. Genet. Res., 80 (2):107–116, 2002.

M. K. Uyenoyama, Y. Zhang, and E. Newbigin. On the origin of self- incompatibility haplotypes: transition through self-compatible intermediates. Genetics, 157(4):1805–1817, 2001.

R. van Treuren, R. Bijlsma, N. Ouborg, and W. van Delden. The signif- icance of genetic erosion in the process of extinction. IV. Inbreeding depression and heterosis effects caused by selﬁng and outcrossing in Scabiosa columbaria. Evolution, 47(6):1669–1680, 1993.

G. R. Warnes, B. Bolker, and T. Lumley. gtools: Various R Programming

140 Tools, 2020. URL https://CRAN.R-project.org/package=gtools.R package version 3.8.2.

M. C. Whitlock, P. K. Ingvarsson, and T. Hatﬁeld. Local drift load and the heterosis of interconnected populations. Heredity, 84(4):452–457, 2000.

H. Wickham. The split-apply-combine strategy for data analysis. Journal of Statistical Software, 40(1):1–29, 2011. URL http://www.jstatsoft.org/v40/i01/.

H. Wickham, M. Averick, J. Bryan, W. Chang, L. D. McGowan, R. Franc¸ois, G. Grolemund, A. Hayes, L. Henry, J. Hester, M. Kuhn, T. L. Pedersen, E. Miller, S. M. Bache, K. Muller,¨ J. Ooms, D. Robinson, D. P. Seidel, V. Spinu, K. Takahashi, D. Vaughan, C. Wilke, K. Woo, and H. Yutani. Welcome to the tidyverse. Journal of Open Source Software, 4(43): 1686, 2019. doi: 10.21105/joss.01686.

J. S. Williams, L. Wu, S. Li, P. Sun, and T. Kao. Insight into S-RNase- based self-incompatibility in Petunia: recent ﬁndings and future directions. Frontiers in plant science, 6:41, 2015.

J. H. Willis. The role of genes of large effect on inbreeding depression in Mimulus guttatus. Evolution, 53:1678–1691, 1999.

A. A. Winn, E. Elle, S. Kalisz, P.-O. Cheptou, C. G. Eckert, C. Goodwillie, M. O. Johnston, D. A. Moeller, R. H. Ree, R. D. Sargent, and M. Vallejo- Mar´ın. Analysis of inbreeding depression in mixed-mating plants provides evidence for selective interference and stable mixed mating. Evo- lution, 65(12):3339–3359, 2011.

Wolfram Research, Inc. Mathematica, Version 12.1, 2020. URL https://www.wolfram.com/mathematica. Champaign, IL, 2020.

141 S. Wright. The distribution of self-sterility alleles in populations. Genetics, 24(4):538–552, 1939.

Y. Xie. knitr: A General-Purpose Package for Dynamic Report Generation in R, 2020. URL https://yihui.org/knitr/. R package version 1.28.

142 Appendix A

Appendix: condition for increase of the outcrossing modiﬁer allele with two viability loci

Here, we derive the values of the selection coefficient, s, for which the frequency of the outcrossing allele, M, initially increases in frequency. We assume here two viability loci, which, when combined with the three mating system genotypes, yields 27 genotypes to track. The frequency p of the outcrossing allele increases to p0 in the next generation when W p0 = p M > p, W¯ ¯ where WM is the marginal fitness of the outcrossing allele and W is the mean fitness in the population. Multiplying both sides by W¯ and expanding pWM into a weighted average of genotype frequencies, this condition becomes

3 12 12 2 X ∗ X ∗ X ∗ ¯ (1 − s) ciXi + (1 − s) ciYi + ciZi > pW, i=1 i=1 i=1

143 where s is the selection coefficient for each viability locus, the Xi are the frequencies of the three genotypes homozygous for the inferior allele at both viability loci, the Yi are the frequencies of the twelve genotypes homozygous for a single deleterious allele, the Zi are the frequencies of the twelve genotypes homozygous for the deleterious allele at neither viability locus, ci is the proportion of M alleles (0, 0.5, or 1) at the modifier locus in the ith genotype, and ∗ denotes a frequency after mating but before selection. Expanding 3 12 12 ¯ 2 X ∗ X ∗ X ∗ W = (1 − s) Xi + (1 − s) Yi + Zi i=1 i=1 i=1 3 ! 3 12 ! 2 X ∗ X ∗ X ∗ = 1 + s Xi − s 2 Xi + Yi i=1 i=1 i=1 and substituting 3 12 12 X ∗ X ∗ X ∗ ∗ ciXi + ciYi + ciZi = p , i=1 i=1 i=1 and collecting terms in powers of s, 3 3 ! 3 3 12 12 ! 2 X ∗ X ∗ X ∗ X ∗ X ∗ X ∗ ∗ s p Xi − ciXi +s 2 ciXi − 2p Xi + p ciYi − Yi < (p − p) . i=1 i=1 i=1 i=1 i=1 i=1 Given all genotype frequencies, this quadratic inequality can be solved to determine the values of s for which p0 > p. When there are no double homozygotes for the inferior alleles, as is the case before F2s and backcrosses are generated, the s2 term becomes zero, the inequality becomes linear, and the invasion condition is p − p∗ s > . (A.1) P12 ∗ P12 ∗ p i=1 Yi − i=1 ciYi The numerator in this expression is the decrease in the frequency of the outcrossing allele due to the transmission advantage of selfing. The de- nominator is a quantity that increases as the proportion of outcrossing individuals homozygous for neither deleterious allele increases.

144 Appendix B

Appendix: genotype frequency recursions for S-haplotype diversiﬁcation

The following genotype frequency recursions governed changes in the frequencies of the classes of SI haplotypes. They were used to produce the trajectories leading to extinction of doomed haplotypes, which were in turn used to calculate rescue probabilities. SC haplotypes were assumed to be rare and were neglected in these recursions. Each PXY represents the frequency of a diploid genotype consisting of one haplotype of class X and one of class Y . The genotype frequency recursions are

0 nL − 2 nL − 1 PLL = 2pL PLL + (PLC + PLU + PLA + PLM ) nL 2nL

145 0 nL − 2 nL − 2 nL − 1 PLC = pL PLC + PCC + (PCU + PCA + PCM ) 2nL nL 2nL nL − 2 nL − 2 nL − 1 + pC PLL + PLC + (PLU + PLA) nL 2nL 2nL

0 nL − 1 1 PLU = pL (PLU + PCU ) + PUU + (PUA + PUM ) 2nL 2 1 nU − 1 + pU PLL + (PLC + PLA) + PLU 2 2nU

0 nL − 1 1 PLA = pL (PLA + PCA) + PUA 2nL 2 1 + p P + (P + P ) A LL 2 LC LU

0 nL − 1 1 PLM = pL (PLM + PCM ) + PUM 2nL 2 1 + p P + (P + P + P ) M LL 2 LC LU LA

0 nL − 2 nL − 2 nL − 1 PCC = 2pC PLC + PCC + (PCU + PCA) 2nL nL 2nL

0 nL − 1 1 PCU = pC (PLU + PCU ) + PUU + PUA 2nL 2 1 + p (P + P + P ) + P U 2 LC CA CM CC

146 0 nL − 1 1 PCM = pC (PLM + PCM ) + PUM 2nL 2 1 + p (P + P + P ) + P M 2 LC CU CA CC

0 nU − 1 nU − 2 PUU = 2pU (PLU + PCU + PUA) + PUU 2nU nU

0 1 nU − 1 PUA = pU (PLA + PCA) + PUA 2 2nU 1 + p (P + P ) + P A 2 LU CU UU

1 P 0 = p (P + P + P ) + P UM M 2 LU CU UA UU

147 Appendix C

Appendix: genotype frequency recursions for S-haplotype introgression

C.1 Nonself-recognition

After pollen migration, the frequency of each haplotype in the pollen pool is changed from p to p∗. For shared haplotypes, population of origin only affects pollen behavior. Therefore, there are effectively only three kinds of haplotypes from the perspective of maternal plant phenotype: local unique haplotypes, foreign unique haplotypes, and all shared haplotypes collectively. We drop the population of origin from shared haplotypes when denoting the pollen accepted by a genotype: e.g., both UFSF and UFSL accept the same proportion aFS of pollen rather than distinct values of aUFSF and aUFSL. Assuming no pollen limitation, mating with nonself-recognition produces the genotype frequencies:

148 ∗ ∗ ∗ 0 pUF −1 ∗ −1 PUFSF + PUFSL PUFUF = aFF (nF − 2)PUFUF + aFS(nF − 1) nF 2

0 PUFUL = 0

0 ∗ ∗ −1 1 ∗ −1 nF − 1 ∗ −1 PUFSF = pUF PSFSFaSS + PUFSFaFS + PSFSLaSS + 2 nF ∗ ∗ −1 1 ∗ −1 nS − 1 ∗ −1 nS − 1 pUF PUFUFaFF + PUFSFaFS + PUFSLaFS 2 nS nS

0 ∗ ∗ −1 1 ∗ −1 nF − 1 ∗ −1 PUFSL = pUF PSLSLaSS + PUFSLaFS + PSFSLaSS + 2 nF ∗ ∗ −1 1 ∗ −1 nS − 1 ∗ −1 nS − 1 pSL PUFUFaFF + PUFSFaFS + PUFSLaFS 2 nS nS

0 ∗ ∗ −1 nL − 2 PULUL = pUL PULULaLL + nL 1 ∗ −1 nL − 1 ∗ −1 nL − 1 PULSFaLS + PULSLaLS 2 nL nL

0 ∗ ∗ −1 1 ∗ −1 nL − 1 ∗ −1 PULSF = pUL PSFSFaSS + PULSFaLS + PSFSLaSS + 2 nL ∗ ∗ −1 1 ∗ −1 nS − 1 ∗ −1 nS − 1 pUF PULULaLL + PULSFaLS + PULSLaLS 2 nS nS

149 0 ∗ ∗ −1 1 ∗ −1 nL − 1 ∗ −1 PULSL = pUL PSLSLaSS + PULSLaLS + PSFSLaSS + 2 nL ∗ ∗ −1 1 ∗ −1 nS − 1 ∗ −1 nS − 1 pSL PULULaLL + PULSFaLS + PULSLaLS 2 nS nS

0 ∗ ∗ −1 nS − 2 PSFSF = pUF PSFSFaSS + nS 1 ∗ −1 nS − 1 ∗ −1 nS − 2 PUFSFaFS + PSFSLaSS 2 nS nS

0 ∗ ∗ −1 nS − 2 1 ∗ −1 nS − 1 ∗ −1 nS − 2 PSFSL = pUF PSLSLaSS + PUFSLaFS + PSFSLaSS + nS 2 nS nS ∗ ∗ −1 nS − 2 1 ∗ −1 nS − 1 ∗ −1 nS − 2 pSL PSFSFaSS + PUFSFaFS + PSFSLaSS nS 2 nS nS

0 ∗ ∗ −1 nS − 2 PSLSL = pSL PSLSLaSS + nS 1 ∗ −1 nS − 1 ∗ −1 nS − 2 PULSFaLS + PSFSLaSS 2 nS nS

C.2 Self recognition

Assuming no pollen limitation, after migration and mating with self recognition, genotype frequencies are:

150 0 ∗ ∗ −1 nF − 2 PUFUF = pUF PUFUFaFF + nF 1 ∗ −1 nF − 1 ∗ −1 nF − 1 ∗ −1 nF − 1 PUFULaFL + PUFSFaFS + PUFSLaFS 2 nF nF nF

0 ∗ ∗ −1 1 ∗ −1 nF − 1 ∗ −1 ∗ −1 PUFUL = pUF PULULaLL + PUFULaFL + PULSFaLS + PULSLaLS + 2 nF ∗ ∗ −1 1 ∗ −1 nL − 1 ∗ −1 ∗ −1 pUL PUFUFaFF + PUFULaFL + PUFSFaFS + PUFSLaFS 2 nL

0 ∗ ∗ −1 1 ∗ −1 nF − 1 ∗ −1 PUFSF = pUF PSFSFaSS + PUFSFaFS + PSFSLaSS + 2 nF ∗ ∗ −1 1 ∗ −1 ∗ −1 nS − 1 ∗ −1 pUF PUFUFaFF + PUFULaFL + PUFSFaFS + PUFSLaFS 2 nS

0 ∗ ∗ −1 1 ∗ −1 nF − 1 ∗ −1 PUFSL = pUF PSLSLaSS + PUFSLaFS + PSFSLaSS + 2 nF ∗ ∗ −1 1 ∗ −1 ∗ −1 ∗ −1 nS − 1 pSL PUFUFaFF + PUFULaFL + PUFSFaFS + PUFSLaFS 2 nS

0 ∗ ∗ −1 nL − 2 PULUL = pUL PULULaLL + nL 1 ∗ −1 nL − 1 ∗ −1 nL − 1 ∗ −1 nL − 1 PUFULaFL + PULSFaLS + PULSLaLS 2 nF nL nL

151 0 ∗ ∗ −1 1 ∗ −1 nL − 1 ∗ −1 PULSF = pUL PSFSFaSS + PULSFaLS + PSFSLaSS + 2 nL ∗ ∗ −1 1 ∗ −1 ∗ −1 nS − 1 ∗ −1 pUF PULULaLL + PUFULaFL + PULSFaLS + PULSLaLS 2 nS

0 ∗ ∗ −1 1 ∗ −1 nL − 1 ∗ −1 PULSL = pUL PSLSLaSS + PULSLaLS + PSFSLaSS + 2 nL ∗ ∗ −1 1 ∗ −1 ∗ −1 ∗ −1 nS − 1 pSL PULULaLL + PUFULaFL + PULSFaLS + PULSLaLS 2 nS

0 ∗ ∗ −1 nS − 2 PSFSF = pUF PSFSFaSS + nS 1 ∗ −1 nS − 1 ∗ −1 nS − 1 PUFSFaFS + PSFSLaSS 2 nS nS

1 n − 1 P 0 = p∗ P ∗ a−1 + P ∗ a−1 + P ∗ a−1 S + SFSL UF SLSL SS 2 UFSL FS SFSL SS nS ∗ ∗ −1 1 ∗ −1 ∗ −1 nS − 1 pSL PSFSFaSS + PUFSFaFS + PSFSLaSS 2 nS

0 ∗ ∗ −1 nS − 2 PSLSL = pSL PSLSLaSS + nS 1 ∗ −1 ∗ −1 nS − 1 PULSFaLS + PSFSLaSS 2 nS

152 Supplementary Figures

153 M lost, LA = 5 1500 1000 500 Number of simulations 0 0.2 0.6 1.0 Ultimate mean fitness

M lost, LA = 25 M lost, LA = 50 1500 1500 1000 1000 500 500 Number of simulations Number of simulations 0 0 0.0 0.4 0.8 0.0 0.4 0.8 Ultimate mean fitness Ultimate mean fitness

M fixed, LA = 25 M fixed, LA = 50 1500 1500 1000 1000 500 500 Number of simulations Number of simulations 0 0 0.70 0.85 1.00 0.4 0.6 0.8 1.0 Ultimate mean fitness Ultimate mean fitness

Figure S1: Distribution of final mean population fitness at the end of the simulation. Ex- cept for LA = 5 (for which M never fixed), the distribution is bimodal at 0 and 1 when M is lost, and unimodal just below 1 when M reaches fixation. Most fixations of M occur before purging is complete. For all parameter combinations LA = LB, s = 0.3, and M is dominant.

154 Dominant, extinct, LA = 25 Dominant, extinct, LA = 50 600 600 200 200 Num simulations Num simulations 0 0 0 50 100 150 0 50 100 150 Duration Duration

Dominant, fixed, LA = 25 Dominant, fixed, LA = 50 600 600 200 200 Num simulations Num simulations 0 0 0 50 100 150 0 50 100 150 Duration Duration

Figure S2: Distribution of simulation durations with outcrossing allele M dominant. Ex- tinctions of M mostly occur after short or long durations, while ﬁxations mostly occur after intermediate durations. For all parameter combinations, LA = LB and s = 0.3. “Extinct” and “ﬁxed” in panel titles refer to the fate of M.

155 Additive, extinct, LA = 25 Additive, extinct, LA = 50 600 600 200 200 Num simulations Num simulations 0 0 0 50 100 150 0 50 100 150 Duration Duration

Additive, fixed, LA = 25 Additive, fixed, LA = 50 600 600 200 200 Num simulations Num simulations 0 0 0 50 100 150 0 50 100 150 Duration Duration

Figure S3: Distribution of simulation durations with outcrossing allele M additive. Extinc- tions of M mostly occur after short or long durations, while ﬁxations mostly occur after intermediate durations. For all parameter combinations, LA = LB and s = 0.3. “Extinct” and “ﬁxed” in panel titles refer to the fate of M.

156 LA = 5

● ●● ● ●● ● ● ●● ● ●●●●● ● ●●●●●● ● ●● 1.0 ●● ●● ●● ● ● ● ● ●● ●● ● ●● ● ● ●● ●●●●●● ● ● ●●● ●●●●● ● ● ●●●●●●●●● ●● ● ●●●● ●●●●● ●●●●●●●●● ● ●●●●●●●●●●● ● ● ● ●●● ●● ●●●●● ● ●●● ●●●●●● ● ●●●●●●● ● ● ●●●●●●●●●●●● ● ●●●● ●●●●● ● ● ●●●● ●● ● ● ●●●●●●●●●● ●●●●●●● ● ● ● ●● ●●●●●●● ●●●● ●●●●●●●●● ● ● ●●●●●●●●●●● ● ●●●●●● ● ● ●●●●●●●●● ●● ●●●●●●●●●● ● ●●●●●●●● ●● ● ●●●●●●●●●● ●●●●●●●●● ●● ●●●●●●●● ● ● ●●●●●●● ● 0.6 ● ● ● ●●●● ●●●●●● ●● ● ●●●●●●● ● ●●●●●●●●● ● ●●●●●●●● ●●●●●● ● ●●●●●●●● ●●●●●● ●●●●● ● ● ●●●● ●●●●●● ●●●●● ●●●●● ●●●●● ● ●●●●● ●●●●●●●● ●●●●●● ● ●●●●●●● ●●●●●● ●●●●●●● ●●●●●●● ●●●●●●● ●●●●●●● ●●●●●● 0.2 0 50 100 150

LA = 25

● ● ● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●● ● ●●● ● ● ● ● ● ● ● ●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●● ●● ●● ●● ● ● ● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●● ● ● ● ● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ● ● ● ●● ● ●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●● ●●●● ● ● ●● ● ● ● ● ● ●●●●●●●●●●●●●●●●●●●●●● ● ●●●● ●● ● ● ●●●●●●●●●●●●●●●●●●●●● ● ●● ●●● ● ● ● ● ●●●●●●●●●●●●●●● ● ●●●● ● ● ● ● ● ● ●●● ●● ● ● ● ●● ● ●●●●●● ● ● ●●● ● ● ●● ● ● ●● ● ● ●

0.8 ● ● 0.4 Final fitness ● ● ●● ●●● ● 0.0 0 50 100 150

LA = 50 ● ● ● ● ●● ● ●●● ●● ●● ●● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ●● ●● ● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ●●●●●●● ●● ●●●● ●●● ●●●● ● ● ● ● ● ● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ● ● ● ●● ● ● ●●●●●●●●●●●●●●●● ●●●●●●●●●●● ●●●●●●●●●●●●●● ●●● ●● ●● ● ● ● ● ● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●● ●● ●● ● ● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ● ● ● ●●● ● ● ● ● ●●●●●●●●●●●●●●●●●●●●● ●● ●● ●● ●● ● ● ● ● ● ● ● ●●●●●●●●●●●●●●●●●●●●●●●●●●● ● ● ● ● ● ● ● ● ●●●●●●●●●●●●●●●●●●●●●●● ● ● ● ●● ●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●● ● ● ●●●●●●●●●●●●●●●●● ● ●● ● ●● ●●●●●●●●●●●●●●●●● ●● ●● ● ●● ●●●●●●●●●●●●● ● ●●●●●●●●●●●●● ● ● ● ●●●●●●●●●●●●● ● ●●●●●●● ●●●●● ●●●●●●●●●● ● ● 0.8 ● ●● ● ● ●●●●●●●●●● ●● ●●●●● ●● ●●●●●●● ●● ●● ●●●●●●● ●●●●●● ● ●●●●● ●● ● ●●●● ●●●● ● ● ●● ●●●● ● ●●● ● ● ● ●● ● ● ● ● ● ●

0.4 ●

● ● ● ●● ●●● 0.0 0 50 100 150 Duration

Figure S4: Extent of adaptation. The mean fitness of the population at the end of the simulation is plotted against the duration of the simulation, in generations. Black circles represent simulations in which M fixed, and red circles represent simulations in which M was lost. Fitness increased more extensively in the simulations that lasted longer. Very little increase in fitness occurred in the shortest simulations. M tended to fix in the simulations of intermediate duration and incomplete fitness growth. For all parameter combinations, LA = LB, s = 0.3, and M is dominant. Parameter combinations with different LA are plotted separately. Data are identical to those of Fig. S1 and Fig. S2.

157 LA = 5 loci LA = 25 loci LA = 50 loci

1.0 (a) (b) (c) 0.0

1.0 (d) (e) (f) 0.0 Inbreeding depression −200 −100 0 −200 −100 0 −200 −100 0

Generations before fixation

Figure S5: Purging dynamics. Each trajectory tracks inbreeding depression through time for a single run in which the outcrossing allele M fixed (black) or went extinct (red). From the collections of all trajectories (a, b, c), one trajectory per possible outcome (fixation or extinction) was chosen haphazardly and plotted separately (d, e, f). Fixation was never observed for LA = 5 loci, so no black trajectory is plotted for this parameter combination. The M allele tended to reach fixation in shorter runs in which some inbreeding depression remained unpurged. Although losses of M often occurred after extensive purging, they also often occurred through early drift (the short red trajectories in the upper right corner).

158 ● ● ● ● ● ● ● 0.6

● ● ● ● ● ● ● 40 0.5 ● ● ● ● ● ● ●

● ● ● ● ● ● ● 0.4 30

B ● ● ● ● ● ● ●

L 0.3

● ● ● ● ● ● 20 0.2 ● ● ● ● ● 0.1 10 0.0 Proportion with M fixed

10 20 30 40 LA

Figure S6: Unequal loads. White dots represent parameter combinations in which M fixed more often than the expectation for a new neutral mutation (0.005). The diagonal from the lower left to the upper right represents equal loads in the two populations, while other cells represent unequal loads. On the diagonal from the top left to the lower right, the total number of viability loci is fixed, but the outcrossing allele is more likely to fix when the load is more equally distributed. The parameter values NA = NB = 50, s = 0.3, dominant M, r = 0.5, and nchrom = 2 were held constant.

159 0.8 0.4

Analytic Simulated 0.0

Survival probability 1.0 2.0 3.0 log10(Generation)

Figure S7: Survival probability of a new gene convertant. The history of a population of

10,000 individuals with nL = 2, nU = 2, during the rescue phase was ﬁrst determined by iterating the recursion equations. Survival probability was determined each generation using Haldane’s series expansion (analytic) or by simulating survival of a new gene convertant for 10 generations over 10,000 replicate simulations (simulated).

160 nL = 2 nL = 3 nL = 5 0.8 0.4 R = 0.01 0.0 0.8 0.4 R = 0.02 Probability 0.0 0.8 0.4 R = 0.1

0.0 2 6 10 3 7 11 5 9 13 Total haplotypes

Figure S8: Distribution of haplotype number after a single rescue/collapse event. This probability distribution of outcomes is conditional on the population rate of gene conversion Rconversion, and on the current numbers of lucky nL and unlucky nU haplotypes. For all panels, nU = 10. Expansion only occurs when the ﬁnal number of haplotypes is n + 1

(the rightmost bin in each histogram). For nL > 2 and small population gene conversion rates (Rconversion), the most likely outcomes is either the loss of all doomed haplotypes or the loss of all but one.

161 a oFg .,ecp htalpoaiiyms eo 1hpoye a eoe each removed sum. was their haplotypes by 11 divided below were mass probabilities probability remaining the all identi- and that is generation except Figure 2.8, threshold. lower Fig. with to number cal haplotype of distribution Stable S9: Figure Frequency 0.0 0.4 0.8 0.0 0.4 0.8 R R c 10 c o o n n v v e e r s r s i o i o n n 30 = = 0.01 0.2 Total haplotypes R R 10 162 c c o o n n v v e e r r s s i i o o n n 30 = = 0.1 0.3 R R 10 c o c n o v n e v r e s r i s o i n o 30 n = = 0.4 1