Theoretical evolutionary genetics of flowering plant mating system and self-incompatibility A DISSERTATION SUBMITTED TO THE FACULTY OF THE UNIVERSITY OF MINNESOTA BY Alexander Harkness IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY Advised by Yaniv Brandvain December 2020 ©2020 by Alexander Harkness Acknowledgments I would like to thank Yaniv Brandvain and Emma Goldberg for years of guidance, patience, and close attention. I would like to thank the Univer- sity of Minnesota for granting the Doctoral Dissertation Fellowship, which allowed me to complete and improve this dissertation, and the Department of Ecology, Evolution, and Behavior for the Graduate Excellence Fellow- ship, which allowed me to hit the ground running in my first year. Chapters 1 and 2 were coauthored with Yaniv Brandvain and Emma Goldberg. Chapter 3 was coauthored with Yaniv Brandvain. i Abstract The mating system of a diploid eukaryote is an outcome of in- tragenomic coevolution. Close relatives are more likely to share recessive deleterious mutations at many locations, so an al- lele at another locus that reduces the probability of inbreeding will increase offspring’s expected fitness. Self-incompatibility in flowering plants, which acts through a polymorphic locus (an S-locus) that rejects pollen when pollen and pistil haplotypes match, is a particularly old and widespread inbreeding avoid- ance adaptation that has persisted through long-term balancing selection among different S-locus haplotypes (S-haplotypes). Intragenomic coevolution occurs between the individual ele- ments of the S-locus: those expressed in pollen and those expressed in pistils. When intragenomic coevolution is dis- turbed, selection on mating system or on particular mating sys- tem adaptations is shifted and the population may adapt in new ways. In this thesis, the theoretical consequences of three dis- turbances to the intragenomic coevolution of mating system in flowering plants are determined. First, it is shown that iso- lation of the genetic load in separate inbreeding populations produces a transitory benefit upon secondary contact to a mu- tation promoting outcrossing, but that this benefit evaporates ii too rapidly as the populations reassimilate to favor the evolu- tion of greater outcrossing consistently. Second, it is shown that, under the taxonomically widespread ribonuclease-based self-incompatibility system, the evolution of a novel S-haplotype greatly disturbs inter-haplotype coevolution, and may either lead to coexistence of all haplotypes (diversification) or extinction of multiple haplotypes (collapse) in a rescue-like process. Third, it is shown that biased patterns of pollen rejection form between non-coevolved S-haplotypes from isolated populations, which may favor the introgression of some haplotypes, prevent intro- gression of others, and cause some to be lost by swamping introgression. iii Contents List of Tables vi List of Figures vii 1 The evolutionary response of mating system to heterosis 1 1.1 Introduction . 3 1.2 Model . 6 1.2.1 General model description . 6 1.2.2 Invasion condition . 10 1.2.3 Simulation model . 11 1.3 Results . 18 1.3.1 Outcrossing mutation never fixed in a three-locus de- terministic model . 18 1.3.2 Outcrossing mutation could fix under restricted cir- cumstances in stochastic simulations . 18 1.3.3 Intermediate selection favored outcrossing . 19 1.3.4 No effect of loose linkage . 19 1.3.5 Outcrossing was sometimes lost after initially invading 20 1.3.6 Ultimate mean fitness and time to extinction were bi- modal . 20 iv 1.3.7 Purging was incomplete when the outcrossing allele fixed . 21 1.3.8 Test 1: rare genetic background favored outcrossing mutation . 21 1.3.9 Test 2: unequal fitness reduced fixation proportion . 22 1.3.10 Test 3: continuous migration reduced fixation pro- portion . 22 1.3.11 Test 4: additive modifiers fixed more often . 22 1.3.12 Test 5: polygenicity was insufficient to favor outcross- ing under reasonable parameters . 23 1.3.13 Test 6: unpurged inbreeding depression was more favorable to outcrossing than local drift load . 24 1.4 Discussion . 24 1.4.1 Implications for mating system evolution . 25 1.4.2 Robustness of model conclusions . 28 1.4.3 Conclusion . 33 2 Diversification or collapse of self-incompatibility haplotypes as a rescue process 42 2.1 Introduction . 43 2.1.1 Conceptual challenges . 46 2.2 Model and Results . 50 2.2.1 Collaborative Nonself-Recognition . 50 2.2.2 Model outline . 53 2.2.3 Equilibrium frequency of SC intermediate . 58 2.2.4 Invasion of lock mutant . 60 2.2.5 Rescue of doomed haplotypes . 62 2.2.6 Long-term behavior . 68 2.3 Discussion . 71 v 3 Nonself-recognition-based self-incompatibility can alternatively promote or prevent introgression 92 3.1 Introduction . 93 3.2 Description . 99 3.3 Results . 108 Self-recognition-based SI . 108 Nonself-recognition-based SI . 108 3.4 Discussion . 112 Bibliography 127 A Appendix: condition for increase of the outcrossing modifier allele with two viability loci 143 B Appendix: genotype frequency recursions for S-haplotype di- versification 145 C Appendix: genotype frequency recursions for S-haplotype in- trogression 148 C.1 Nonself-recognition . 148 C.2 Self recognition . 150 Supplementary Figures 153 vi List of Tables 2.1 Notation . 81 vii List of Figures 1.1 Invasion condition for the outcrossing allele in the determin- istic model . 34 1.2 Outcrossing allele fixation proportions . 35 1.3 Allele frequency trajectories of the allele, M . 36 1.4 Simulation outcomes in the final generation . 37 1.5 Fixation proportion with initially asymmetric viability geno- type frequencies . 38 1.6 Effect of heterosis magnitude and architecture on fixation . 39 1.7 Effect of unpurged inbreeding depression . 40 2.1 Rejection of self pollen in the style . 80 2.2 Haplotype classes . 82 2.3 Model steps . 83 2.4 Conversion-selection balance of SC gene convertants . 84 2.5 Fitness of a pollen-limited SI mutation . 85 2.6 Doomed haplotypes . 86 2.7 Expansion probability . 87 2.8 Stable distribution of haplotype number . 88 2.9 Haplotype number transition matrix for Rconversion = 0:3 . 89 2.10 Long-term simulated haplotype number trajectories . 90 3.1 Pollen compatibility under self- and nonself-recognition . 120 viii 3.2 Evolutionary dynamics of shared foreign, shared local, unique foreign, and unique local S-haplotypes with self-recognition based SI . 121 3.3 Evolutionary dynamics of shared foreign, shared local, unique foreign, and unique local S-haplotypes with nonself-recognition based SI . 122 3.4 Effect of unique haplotypes on invasion threshold . 123 3.5 Bidirectional migration . 124 3.6 Equilibrium frequency of foreign haplotypes in each popula- tion with bidirectional migration . 125 3.7 Loss of diversity . 126 S1 Distribution of final mean population fitness at the end of the simulation . 154 S2 Distribution of simulation durations with outcrossing allele M dominant . 155 S3 Distribution of simulation durations with outcrossing allele M additive . 156 S4 Extent of adaptation . 157 S5 Purging dynamics . 158 S6 Unequal loads . 159 S7 Survival probability of a new gene convertant . 160 S8 Distribution of haplotype number after a single rescue/collapse event . 161 S9 Stable distribution of haplotype number with lower threshold 162 ix Chapter 1 The evolutionary response of mating system to heterosis This chapter is a reprint of Harkness et al. [2019a]. Abstract Isolation allows populations to diverge and to fix different al- leles. Deleterious alleles that reach locally high frequencies contribute to genetic load, especially in inbred or selfing pop- ulations, in which selection is relaxed. In the event of sec- ondary contact, the recessive portion of the genetic load is masked in the hybrid offspring, producing heterosis. This ad- vantage, only attainable through outcrossing, should favor evo- lution of greater outcrossing even if inbreeding depression has been purged from the contributing populations. Why, then, are selfing-to-outcrossing transitions not more common? To eval- uate the evolutionary response of mating system to heterosis, we model two monomorphic populations of entirely selfing in- dividuals, introduce a modifier allele that increases the rate 1 of outcrossing, and investigate whether the heterosis among populations is sufficient for the modifier to invade and fix. We find that the outcrossing mutation invades for many parameter choices, but it rarely fixes unless populations harbor extremely large unique fixed genetic loads. Reversions to outcrossing become more likely as the load becomes more polygenic, or when the modifier appears on a rare background, such as by dispersal of an outcrossing genotype into a selfing population. More often, the outcrossing mutation instead rises to moderate frequency, which allows recombination in hybrids to produce superior haplotypes that can spread without the mutation’s fur- ther assistance. The transience of heterosis can therefore ex- plain why secondary contact does not commonly yield selfing- to-outcrossing transitions. 2 1.1 Introduction Secondary contact between previously isolated populations is a dramatic event with a variety of potential evolutionary repercussions, including ge- netic rescue [Richards, 2000], the exposure of epistatic incompatibilities evolved in isolation [Bateson, 1909, Dobzhansky, 1937, Muller, 1942], and the release of selfish genetic elements [Fishman and Willis, 2005]. A particularly common consequence of secondary contact is heterosis, an increase in fitness of the offspring of