The Evolution of Sex Is Tempered by Costly Hybridization in 3 Boechera (Rock Cress) 4 5 Catherine A

bioRxiv preprint doi: https://doi.org/10.1101/2020.02.11.944256; this version posted February 12, 2020. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license.

1 2 The evolution of sex is tempered by costly hybridization in 3 Boechera (rock cress) 4 5 Catherine A. Rushworth1,2,3,4* and Tom Mitchell-Olds4 6 7 1. Department of Evolution and Ecology, Storer Hall, University of California Davis, One 8 Shields Avenue Davis, CA 95616 USA 9 10 2. Department of Plant and Microbial Biology, University of Minnesota, St. Paul, MN 11 55108 USA 12 13 3. University and Jepson Herbaria, University of California Berkeley, Berkeley, CA 14 94720 USA 15 16 4. Department of Biology and Center for Genomic and Computational Biology, Duke 17 University, Box 90338, Durham, NC 27708 USA ([email protected]) 18 19 * author for correspondence: [email protected] 20 21 Running title: Costly hybridization in sexual/asexual Boechera 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43

1 bioRxiv preprint doi: https://doi.org/10.1101/2020.02.11.944256; this version posted February 12, 2020. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license.

44 Abstract 45 46 Even after decades of research, the evolution of sex remains an enigma in evolutionary 47 biology. Typically, research addresses the costs of sex and asexuality to characterize 48 the circumstances in which one reproductive system is more favorable. Yet surprisingly 49 few studies address the influence of common traits that are obligately correlated with 50 asexuality, including hybridization and polyploidy; even though these traits have 51 substantial impacts on selective patterns. In particular, hybridization is well-known to 52 alter trait expression; these alterations may themselves represent a cost of sex. We 53 examined the role of reproductive isolation in the formation of de novo hybrid lineages 54 between two widespread species in the ecological model system Boechera. Of 664 55 crosses between Boechera stricta and Boechera retrofracta, 17% of crosses produced 56 F1 fruits. This suggests that postmating prezygotic barriers, i.e. pollen-pistil interactions, 57 form the major barrier to hybrid success in this system. These interactions are 58 asymmetrical, with 110 F1 fruits produced when B. stricta was the maternal parent. This 59 asymmetry was confirmed using a chloroplast phylogeny of wild-collected B. stricta, B. 60 retrofracta, and hybrids, which showed that most hybrids have a B. stricta chloroplast 61 haplotype. We next compared fitness of F2 hybrids and selfed parental B. stricta lines, 62 finding that F2 fitness was reduced by substantial hybrid sterility. Our results suggest 63 that multiple reproductively isolating barriers likely influence the formation and fitness of 64 hybrid lineages in the wild, and that these costs of hybridization likely have profound 65 impacts on the costs of sex in the natural environment. 66 67 Keywords 68 69 Reproductive isolation; postmating prezygotic barriers; crossing asymmetry; asexual 70 reproduction; apomixis; Brassicaceae 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86

2 bioRxiv preprint doi: https://doi.org/10.1101/2020.02.11.944256; this version posted February 12, 2020. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license.

87 Introduction 88 89 Among the greatest outstanding puzzles in evolutionary biology is understanding the 90 evolutionary processes that underlie the predominance of sex. A substantial swath of 91 research focuses on population genetic models balancing the costs of sex and 92 asexuality. Such costs are often framed as a comparison of sexual and asexual 93 reproduction in constant environments, where sexual organisms produce two 94 independent sexes. For this reason, these traditionally include the cost of producing 95 independent male organisms (the “twofold cost of sex”), the costs of finding and 96 attracting a mate, and the cost of recombination among co-adapted alleles (Maynard 97 Smith, 1978; Lehtonen et al., 2012). Simultaneously, costs of asexuality generally 98 include clonal interference (Gerrish and Lenski, 1998) and Muller’s Ratchet (Müller, 99 1964), which rely on mutation accumulation without recombination. 100 101 The tension among the various costs of sexual and asexual reproduction is complicated 102 by biological and ecological circumstances. Environmental variation has profound 103 consequences for key elements of reproductive system, including fitness (Orr, 2009), 104 gene flow (Richardson et al., 2014), and recombination rate (Bomblies et al., 2015; 105 Modliszewski and Copenhaver, 2017). Previous studies show that ecological variation, 106 in addition to population structure, contributes significantly to the maintenance of 107 multiple reproductive modes (Agrawal, 2009; Becks and Agrawal, 2010, 2012). Yet 108 despite the clear relevance of ecology for key elements of reproduction, few studies of 109 the evolution of sex are conducted in the field (Neiman et al., 2018). Additionally, 110 numerous asexual systems are hybrids (e.g., Vrijenhoek, 1978; Lutes et al., 2010; 111 Coughlan et al., 2017), suggesting that hybridization often causes, or at least co-occurs 112 with, asexuality. A substantial body of literature examining organisms across the tree of 113 life shows widespread effects of hybridization on fitness-related traits (Rieseberg et al., 114 1999; Abbott et al., 2013; Yakimowski and Rieseberg, 2014). The phenotypes altered 115 by hybridization will have profound downstream effects on sexual/asexual dynamics. 116 Yet disentangling the effects of asexuality from those of biological traits like 117 hybridization is largely intractable, as these factors are tightly associated. 118 119 An additional complication stems from the intrinsic biological characteristics of 120 sexual/asexual systems, and resulting variation in the applicability of the traditional 121 costs of sex to all study systems. In flowering plants, asexual reproduction is 122 widespread; apomixis (asexual reproduction via clonal seed) is found in nearly 20% of 123 angiosperm families (Hojsgaard et al., 2014). Most angiosperms produce flowers that 124 contain both male and female organs, termed “hermaphroditic” (Barrett, 2002; 125 Charlesworth, 2006). Thus, the framework for the costs of sex, namely the assumption 126 of separate male and female organisms and a lack of reproductive assurance in 127 sexuals, may be of limited relevance. 128 129 But additional costs of sex may exist, some of which offer promising avenues for 130 research in flowering plants (Meirmans et al., 2012). Chief among these is a fitness

3 bioRxiv preprint doi: https://doi.org/10.1101/2020.02.11.944256; this version posted February 12, 2020. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license.

131 reduction associated with mating system, i.e. inbreeding and/or outcrossing 132 (Charlesworth, 2006). Mating among related organisms, including self-fertilization, may 133 reduce fitness (inbreeding depression, Charlesworth and Willis, 2009), while 134 outcrossing between widely divergent individuals may likewise reduce fitness via 135 outbreeding depression (e.g., Waser and Price, 1994). Although Meirmans and 136 colleagues postulate that mating system negatively impacts sexuals, dissimilarity in the 137 genesis of sexual and asexual lineages (i.e. hybrid origins of asexuals) suggest that the 138 repercussions of mating system could also manifest in asexuals. 139 140 The influence of hybridization and mating system on the evolution of sex implicates the 141 presence and expression of reproductively isolating barriers between species. While 142 some barriers may prevent one species from encountering another (e.g., 143 ecological/geographic prezygotic barriers), others hinder interspecific fertilization 144 following mating (postmating prezygotic barriers). Postzygotic barriers render hybrids 145 inviable or sterile via chromosomal rearrangements, cytonuclear interactions or 146 Bateson-Dobzhansky-Muller incompatibilities (Coyne and Orr, 2004). Pollen-pistil 147 interactions are a common form of postmating prezygotic barrier in flowering plants. 148 These may be symmetrical, where each direction of a given cross is equally unable to 149 achieve fertilization; or asymmetrical, where one direction of a cross succeeds, but the 150 other fails (Tiffin et al., 2001). Importantly, multiple barriers often co-occur, bolstering 151 incomplete reproductive isolation caused by a single mechanism (Coyne and Orr, 152 2004). 153 154 The mustard genus Boechera is a widespread North American wildflower that engages 155 in both sexual and asexual reproduction (Böcher, 1951; Al-Shehbaz and Windham, 156 1993). Boechera is highly self-fertilizing when sexually reproducing, which enables 157 large-scale assessment of reproductive mode in interspecific hybrids via characteristic 158 levels of heterozygosity (Beck et al., 2012; Li et al., 2017). Apomixis is widespread in 159 Boechera, and co-occurs with hybridization or with outcrossing among divergent 160 intraspecific populations (Rushworth et al., 2018). The causative relationship between 161 hybridization and asexuality is unknown, although apomixis may be a result of metabolic 162 dysregulation caused by hybridization (Carman, 1997; Sharbel et al., 2010). In the field, 163 fitness of asexual lineages is higher than sexual lineages (Rushworth et al., 2019), 164 leading to two potential scenarios: first, uniformly high fitness of hybrid lineages that 165 frequently reproduce asexually; or rampant but maladaptive hybridization events with 166 rare transitions to asexuality “rescuing” hybrid genotypes. Previous studies of ecological 167 and genetic variation in Boechera stricta suggest that reproductive isolation between 168 subgroups is driven by ecological differentiation (Lee and Mitchell-Olds, 2011, 2013) 169 and a single chromosomal inversion underlies phenological divergence (Lee et al., 170 2017). But to date, little is known of postmating barriers in this group. 171 172 Here we explore three vignettes investigating the costs of hybridization in the evolution 173 of sex in Boechera. We found that the formation of hybrids between the two best- 174 characterized species, Boechera stricta and Boechera retrofracta, is hindered by the

4 bioRxiv preprint doi: https://doi.org/10.1101/2020.02.11.944256; this version posted February 12, 2020. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license.

175 expression of multiple reproductive isolating barriers. The main barrier to hybrid success 176 occurred before hybrids were formed, with most F1 crosses failing to set seed. This 177 barrier was asymmetrical, with substantially higher success for F1s with B. stricta as the 178 maternal parent than the reverse cross. Phylogenetic analysis of a chloroplast marker 179 confirmed that B. stricta most often acts as the maternal parent in wild-collected hybrids. 180 We next compared fitness of hybrid F2s with their selfed parental B. stricta lines, finding 181 that hybrid sterility substantially reduces F2 fitness, although fertile F2s produce more 182 fruits than selfed lines. These results have important implications for the speciation 183 process and for the evolution of sex in this ecological model system. 184 185 Materials and Methods 186 187 Crosses and plant growth 188 189 In the wild, B. stricta and B. retrofracta commonly co-occur, hybridize, and form asexual 190 lineages (Rushworth et al., 2018). In 2012, one line was selected from 11 populations of 191 each species for use as crossing parents (Figure 1, Table S1 in Supplementary 192 Material). In five populations, both species co-occurred in close proximity to one 193 another; these populations are thus considered sympatric. All lines were of known 194 genotype from previous studies (Song et al., 2006; Lee and Mitchell-Olds, 2011; 195 Rushworth et al., 2018). Each line had been previously grown in the greenhouse for 1-2 196 generations. Seeds were germinated on wet filter paper in petri dishes and grown in 197 greenhouses at Duke University, then vernalized for 6 weeks at 4°C with 12 hour 198 daylength. We used three staggered planting cohorts to allow for genotypic variation in 199 phenology. 200 201 Plants were subsequently moved to growth chambers in Duke University’s Phytotron. 202 Growth conditions were 22°C constant temperature with ambient relative humidity and 203 carbon dioxide, and 12 hour days at 350 micromoles of light. Reciprocal crosses were 204 performed on two to five flowers for each possible parental combination following 205 Schranz et al. (2005). Upon observation of cross failure, replicate crosses using 206 individuals from different cohorts were made. Concurrently, other flowers on the same 207 plants were permitted to self-fertilize. Additional flowers were self-fertilized by hand. 208 209 Fruiting success was calculated as the number of fruits divided by the number of flowers 210 for a given cross type (maternal B. stricta; paternal B. stricta; hand-selfed B. stricta; 211 hand-selfed B. retrofracta). To correct for differences in fruit set inherent to each 212 species, we also calculated crossing indices for each cross type (McDade and 213 Lundberg, 1982). Crossing indices were calculated by dividing fruiting success of an 214 interspecific cross by fruiting success of the hand-selfed maternal parent (i.e., maternal 215 B. stricta crosses divided by hand-selfed B. stricta; paternal B. stricta crosses divided by 216 hand-selfed B. retrofracta). 217

5 bioRxiv preprint doi: https://doi.org/10.1101/2020.02.11.944256; this version posted February 12, 2020. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license.

218 When F1s produced <15 seeds, we germinated all available F1 seeds; if a given 219 parental combination resulted in >15 seeds, we germinated 15 seeds. All F1 seeds 220 were germinated on filter paper in petri dishes and grown in greenhouses at Duke 221 University. All seeds available, regardless of germination status, were planted onto a 222 mix of MetroMix 200 and Fafard 4P Mix (Sun Gro Horticulture, Agawam, MA, USA) in 223 Ray Leach Cone-tainers (Steuwe and Sons, Tangent, OR, USA). 224 225 To verify that F1s were the product of successful crosses, viable F1s were genotyped at 226 three variable microsatellite loci (ICE3, c8, BF20; Clauss et al., 2002; Dobeš et al., 227 2004a; Song and Mitchell-Olds, 2007) following protocols from Beck et al. (2012). This 228 resulted in 50 viable unique F1 seed lines. Selfed parental lines were also genotyped to 229 verify self-fertilization. To verify reproductive mode of F1s, 3—12 (mean 9.1) F2 230 individuals were genotyped in the same manner as F1s. A given plant was determined 231 to be sterile if it failed to reproduce in three months, one month longer than the typical 232 growing season in the central Rocky Mountains. 233 234 Poor F2 seed set resulted in a final experimental total of seven unique F2 families, 235 derived from the same B. retrofracta parent and four B. stricta individuals (Table S1). 236 Parental lines were also permitted to self for a second generation. Segregation of alleles 237 in all lines was observed, indicating that F1s reproduced sexually. 238 239 Greenhouse experiment 240 241 Fitness of F2s and their selfed parental lines was assessed in the greenhouse in 2014. 242 Seeds were germinated on wet filter paper in petri dishes and transplanted as 243 seedlings. Low germination for some F2 families resulted in an unbalanced design. In 244 total, 573 individuals (300 parents, 273 F2s) were used, with 60 replicates from each F1 245 parental line and 19-59 progeny for each F2 family (mean 39 replicates). 52 individuals 246 (51 hybrid F2s and one selfed parental individual) died during the course of the 247 experiment. 248 249 Plants were grown in randomized blocks and racks were rotated to minimize 250 microhabitat variation. Traits measured included those related to fitness (probability of 251 survival and reproduction, flower number, fruit number, aborted fruit number) and 252 biomass (rosette width, plant height, leaf number). Aborted fruits were those that 253 contained only partially developed (i.e. inviable) seeds. Average seed set was 254 calculated from four replicates of each genotype; for genotypes EJ1 and EJ2, less than 255 four individuals reproduced, resulting in seed set estimation from one and two 256 individuals, respectively. Total fitness was calculated as the number of fruits per 257 individual multiplied by average seed set per genotype, with all zeros included. 258 Fecundity was calculated as the number of fruits per individual multiplied by average 259 seed set per genotype, conditional on reproduction. 260 261 Statistical analyses 262

6 bioRxiv preprint doi: https://doi.org/10.1101/2020.02.11.944256; this version posted February 12, 2020. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license.

263 To understand how cross type influenced multiple fitness components, we used 264 generalized linear mixed models (GLMMs). We estimated each fitness component 265 (probability of reproduction, fruit number, aborted fruit number, seed set, and total 266 fitness) as a function of cross type (hybrid or selfed), with a scaled covariate of rosette 267 width in each model to account for the impact of plant size on reproductive output. 268 Because all F2s had the same paternal B. retrofracta genotype, this genotype was 269 constant; we thus compared four B. stricta lines with their hybrids. All models included 270 random effects for experimental block and parent nested in cross type. In four of five 271 models, a random effect of B. stricta parent line was also included; the model for 272 probability of reproduction estimated the variance among parents to be zero, and thus 273 this term was eliminated from the model. An additional random effect of genotype (F2 274 family or parental genotype) was incorporated in the total fitness model, as it had a 275 substantial effect on model fit. 276 277 Models for fruit number, aborted fruit number, and fecundity used a negative binomial 278 error distribution, while a binomial error distribution was used for the probability of 279 reproduction. A single outlier that produced 308 fruits was removed from the fruit 280 number, aborted fruit number, and total fitness models. Lifetime fitness was estimated 281 with a zero-inflated negative binomial distribution using the canonical link functions. In 282 most fitness models, structural zeros account for plants that failed to survive. Because 283 survival is not a factor in controlled conditions, structural zeros in the model indicate 284 plants that failed to reproduce, while the conditional portion of the model represents the 285 fecundity of the plants that did reproduce. Zero-inflation was modeled across both main 286 effects. 287 288 All analyses were run in R version 3.5.2. Directionality of cross success via F1 fruit 289 production was analyzed using Fisher’s exact tests in the base R stats package. Due to 290 very small sample sizes, statistical comparison of success was not possible for later 291 stages of hybrid development. GLMMs were run using the packages lme4 (probability of 292 reproduction model; Bates et al., 2015) and glmmTMB (all other models; Brooks et al., 293 2017). Significance was calculated via likelihood ratio tests. P-values were adjusted 294 using the Bonferroni-Holm method. Estimated marginal means were calculated with the 295 package ggeffects (Lüdecke, 2018). 296 297 Molecular phylogenetics 298 299 We used 112 samples of B. stricta, B. retrofracta, and B. retrofracta ´ B. stricta in a 300 molecular phylogenetic analysis. Samples from B. retrofracta (N=32) and B. retrofracta 301 ´ B. stricta (N=31) were selected from across central Idaho and western Montana, while 302 B. stricta samples (N=49) represent the full species range (Table S2). DNA was 303 extracted from each sample using either Qiagen DNeasy Plant Mini Kits (Qiagen, 304 Hilden, Germany), a modified CTAB protocol (Beck et al., 2012), or following Lee et al. 305 (2011). 306

7 bioRxiv preprint doi: https://doi.org/10.1101/2020.02.11.944256; this version posted February 12, 2020. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license.

307 To identify wild-collected hybrid parentage, we amplified the intron and second exon of 308 the chloroplast gene trnL, using primers c and d and thermal cycling protocols from 309 Dobes et al. (2004b). PCR was performed with 20 mL reactions consisting of 10uM of 310 each primer, 0.2 uL of 10 mg/mL BSA (Millipore/Sigma), and 2mM dNTPs, 10X buffer 311 and Choice-Taq DNA polymerase (all Denville Scientific, Metuchen, NJ, USA). PCR 312 was run on MJ Research PTC-200 thermal cyclers and Sanger sequencing was 313 performed at the UC Berkeley DNA Sequencing Facility on an ABI 3730xl analyzer 314 (Applied Biosystems). B. stricta sequences were provided by Baosheng Wang via 315 sequencing data in Wang et al. (2019). 316 317 A maximum parsimony phylogeny was inferred using Paup version 4.0a166 (Swofford, 318 2003). We performed a search from ten different random addition sequence starting 319 trees, using TBR branch swapping and a reconnection limit of 8. 320 321 Results 322 323 Hybrid crosses rarely succeed 324 325 The success of hybrid crosses was limited at multiple developmental stages. We 326 conducted 203 reciprocal crosses using unique parental combinations, with an average 327 of 3.07 flowers (±0.04 standard error, se) for each cross, resulting in 664 total crossed 328 flowers. Crosses had a 17.3% chance of successfully producing fruits. This resulted in 329 115 total F1 fruits from all maternal plants, an average of 0.53 fruits per flower (±0.06 330 se). Although Boechera fruits from autonomous self-fertilization may produce upwards 331 of 100 seeds (Rushworth et al., 2011), these F1 fruits produced an average of 12.98 332 seeds per fruit (±2.32 se), totaling 2804 F1 seeds from all crosses. Despite low average 333 seed set, 59 of 115 F1 fruits produced >15 F2 seeds. 334 335 To assess seed viability, we planted 1190 F1 seeds, resulting in 829 germinants from 336 98 F1 fruits (an average of 1.25 seeds ±0.14 se per cross). Seventeen fruits produced 337 no viable seed. This represented a 70% rate of viability. 705 seedlings from these 98 F1 338 lines were planted onto soil, but in 77 lines, all germinants perished. 88 plants from 21 339 F2 lines survived. Of these 88 plants, 59 plants from 13 lines reproduced. The 340 remaining plants were sterile. Genotyping of F2 seeds showed that six of these 13 lines 341 were not successful crosses (i.e., they were identical to the maternal genotype), leaving 342 seven total F2 crossing families for the greenhouse experiment. 343 344 Directionality of cross affects success 345 346 B. stricta was the maternal parent for all seven successful F2 lines used in our 347 experiment. The influence of cross directionality on success became apparent early on. 348 Despite conducting a roughly equivalent number of crosses in each direction (317 349 flowers with B. stricta as maternal parent, 347 paternal B. stricta crosses), only five of 350 115 F1 fruits, representing replicates of 62 unique parental combinations, were

8 bioRxiv preprint doi: https://doi.org/10.1101/2020.02.11.944256; this version posted February 12, 2020. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license.

351 produced by crosses with B. stricta as the paternal parent. We next compared the 352 proportion of unique parental combinations that resulted in fruits with a Fisher’s exact 353 test. 57 of 102 maternal B. stricta parental combinations resulted in fruits, compared to 354 5 of 101 paternal B. stricta combinations. These proportions are significantly different, 355 with maternal B. stricta crosses 23.9 times as likely to result in fruits as paternal B. 356 stricta crosses (Fisher’s exact test, 95% CI 8.84—81.53, P=2.199e-16; Figure 2). 357 358 We note that fruiting success was very low for hand-selfed B. stricta (8 fruits out of 37 359 crossed flowers, or 0.216 fruits/cross) and B. retrofracta (3 fruits out of 44 crossed 360 flowers or 0.068 fruits/cross), suggesting mechanistic issues with crossing. By 361 comparison, paternal B. stricta crosses produced 5 fruits out of 347 crossed flowers 362 (0.014 fruits/cross) and maternal B. stricta crosses produced 110 fruits from 317 363 crossed flowers (0.347 fruits/cross). The crossing index for maternal B. stricta was thus 364 1.605 (0.347/0.216), while the crossing index for paternal B. stricta was 0.211 365 (0.014/0.068), consistent with higher fruiting success in maternal B. stricta crosses 366 despite poor responses to hand-fertilization in both species. 367 368 Of five potential paternal B. stricta F1 lines, all reproduced. Four produced only one F2 369 seed, although one of these failed to germinate, while one produced 13 F2 seeds. 370 Genotyping confirmed that two of these crosses, including the high seed set genotype, 371 were not successful (i.e. the putative F1 was identical to the maternal parent). Of the 372 two remaining crosses, one germinant died early, and the last resulted in a sterile plant. 373 Thus, no paternal B. stricta crosses were ultimately successful in reproducing, 374 suggesting B. stricta is only suitable as a maternal parent. 375 376 Because hybridization dynamics may strongly differ in the field, we used a phylogenetic 377 analysis of the maternally-inherited chloroplast to assess cross directionality of wild- 378 collected B. stricta ´ B. retrofracta hybrids. Our plastid trnL alignment of 112 accessions 379 (49 B. stricta, 32 B. retrofracta, 31 hybrids) had 10 variable characters, five of which 380 were parsimony-informative. The most parsimonious trees had 11 changes, and 381 resolves two major groups, one containing 48 of 49 B. stricta accessions, and the other 382 containing 28 of 32 B. retrofracta (Figure 3). 30 of the 31 hybrid accessions fell in the B. 383 stricta group, with 28 of them sharing a haplotype with a sampled B. stricta accession. 384 Although previous studies show chloroplast haplotypes are shared between the two 385 species (Sharbel and Mitchell-Olds, 2001; Dobeš et al., 2004b), our results indicate that 386 B. stricta is usually the maternal parent in the wild. 387 388 Hybrid fitness is lower than selfed lineages 389 390 Total fitness (total maximum seed set, a product of the probability of reproducing and 391 fecundity) was lower in F2 hybrids than in selfed lineages (overall significance, Table 1). 392 On average, hybrids produced 510 (±43.3 se) seeds per plant, while selfed genotypes 393 produced 784 (±30.6 se) seeds per plant. 67.8% of hybrids (±0.03 se) reproduced vs. 394 96.2% of selfed lines (±0.01 se). Hybrids were far less likely to reproduce than selfed

9 bioRxiv preprint doi: https://doi.org/10.1101/2020.02.11.944256; this version posted February 12, 2020. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license.

395 lines, which drove this difference in fitness (zero-inflated model, Table 1, Figure 4A), 396 although this difference was not significant following correction for multiple comparisons 397 in the reproduction-only model (Table S3). 398 399 However, F2 hybrids produced more fruits than selfed lines, averaging 22.5 ±1.62 se 400 compared to 10.6 ±0.4 se for selfed lines (Figure 4B, Table 2). Hybrids had notably 401 higher variance in fruit number, with a standard deviation of 24.1 fruits compared to 6.17 402 for selfed lines. Indeed, one hybrid individual produced 308 fruits, while the maximum 403 number of fruits produced by a selfed line was 48. F2 hybrids also produced more 404 aborted fruits than selfed lineages; this difference was significant prior to correction for 405 multiple comparisons (P=0.02), but not after (Table S4). Fecundity, or seed set 406 conditional upon reproduction, did not differ between hybrids and selfed lines, with 407 hybrids producing an average of 709 seeds ±52.3 se compared to 814 ±30.0 se for 408 selfed lines (conditional model, Table 1, Figure S1, Table S5). Estimated marginal 409 means for all individual models are reported in Table S6. 410 411 Selfed lines were not significantly larger than hybrid lines (hybrid mean 90.2mm ±1.95se 412 vs. selfed mean 127mm ±1.02 se), although hybrid size was more variable (hybrid 413 standard deviation, sd=29 vs. selfed sd=15.8; Figure S2). Size had a significant effect 414 on total fitness (overall model, Table 1), probability of reproduction (zero-inflated 415 portion, Table 1) and fruit number (Table 2). Notably, a random effect of genotype 416 accounted for 14% of the variance in the total fitness model, far more than any other 417 random term in this model or in any other (Table S7). This suggests that genotype plays 418 a large role in the ultimate trajectory of hybrid lineages. 419 420 Discussion 421 422 Hybridization is known to co-occur with transitions to asexuality in numerous plant 423 systems (summarized in Asker and Jerling, 1992). Although a substantial body of 424 research explores the evolution of sex via a balance between its costs and those of 425 asexuality, few studies account for the effects of correlated traits. The strong link 426 between asexuality and hybridization across the tree of life is likely to influence the 427 evolution of sex in several unique ways. First, hybridization that occurs among sexual 428 lineages and results in outbreeding depression may represent a cost of sexual 429 reproduction. Alternatively, if hybridization is frequently associated with transitions to 430 asexuality, but most asexual hybrids exhibit lower fitness, hybridization may be 431 considered a cost of asexuality. Second, asexuality may act as a “rescue” of low fitness 432 hybrids, enabling reproductive assurance for lineages that might otherwise go extinct. 433 Third, hybridization alters phenotype expression in myriad ways, resulting in traits that 434 are transgressive, intermediate, or novel. These alterations, in turn, will have strong 435 ecological impacts on the evolutionary trajectories of hybrid lineages. 436 437 Here we explored the fitness consequences of de novo hybrids in controlled conditions. 438 We found that the costs of hybrid formation may far outweigh any benefits that

10 bioRxiv preprint doi: https://doi.org/10.1101/2020.02.11.944256; this version posted February 12, 2020. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license.

439 successful hybrid asexuals experience in the natural environment. F2 hybrids are less 440 likely to reproduce, which results in reduced fitness (Table 1, Figure 4), despite 441 producing more fruits than their selfed counterparts (Table 2). Hybrids are also difficult 442 to form; hybridization events led to F1 fruit formation in less than 20% of crosses. 443 Additionally, cross directionality is of profound importance. In our experiment, only 444 hybrids with B. stricta as the maternal parent were ultimately successful (Figure 2). A 445 chloroplast phylogeny corroborates this result, implicating asymmetrical reproductively 446 isolating barriers in the formation of hybrids (Figure 3). Collectively, multiple 447 reproductively isolating barriers reduce hybrid fitness, which has important 448 consequences for hybrid lineages in the wild. 449 450 Numerous traditional costs of sex, such as the twofold cost of males and the metabolic 451 costs of attracting a mate, have been discussed in the literature (Lehtonen et al., 2012). 452 Recently, Meirmans et al. (2012) proposed hybridization, inbreeding, and outcrossing, 453 which we collectively will refer to as mating system, as an additional cost of sex. As 454 conceived by the authors, outcrossing and hybridization will negatively impact the 455 fitness of sexual lineages, via reproductive isolation or outbreeding depression. 456 Similarly, inbreeding within small populations may reduce sexual fitness through 457 inbreeding depression. Through this lens, the fitness consequences of mating system 458 fall on sexual populations, and asexuals, once formed, are able to avoid fitness 459 reductions by avoiding mating altogether. 460 461 However, mating system may instead pose a cost to asexuals. Additionally, whether 462 mating system costs impact sexuals or asexuals may depend on other factors. For 463 example, the nature of hybridization costs will depend on the frequency of transition to 464 asexual reproduction following hybridization. If all hybrids transition to asexuality, the 465 developmental issues caused by reproductively isolating barriers may hinder asexual 466 formation and success. If the majority of hybrids are sexual, with rare transitions to 467 asexuality, the fitness detriments of hybridization will pose a cost to sexual populations. 468 469 Knowledge of a system’s mating system is critical to attributing this cost. For example, 470 frequent self-fertilization among sexual lineages will largely enable avoidance of the 471 fitness costs of hybridization. While asexual Boechera result from outcrossing events 472 between either populations or species, sexual lineages are highly self-fertilizing (Beck et 473 al., 2012; Li et al., 2017; Rushworth et al., 2018). Nevertheless, hybridization is clearly 474 common. Large-scale collections-based efforts have identified 400 unique asexual 475 hybrid combinations of Boechera in North American herbaria; the vast majority of these 476 are asexual (Li et al., 2017). This observation suggests two possible scenarios: either 477 rare hybridization events always co-occur with asexuality; or rampant hybridization 478 produces numerous short-lived sexual lineages, some of which transition to asexuality. 479 With more knowledge of both the mechanisms of asexual formation, and the 480 outcrossing and hybridization rate in natural populations, distinguishing between these 481 possibilities offers a promising area for future research. 482

11 bioRxiv preprint doi: https://doi.org/10.1101/2020.02.11.944256; this version posted February 12, 2020. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license.

483 Hybridization may represent a cost of both sex and asexuality, through its substantial 484 and varying influence on the expression of numerous phenotypic traits. Importantly, the 485 mode of expression will depend on the genetic basis of the trait under consideration. 486 Hybrids may produce traits that are intermediate to their parents, or the phenotype of 487 one parent, or even transgressive or novel (Rieseberg et al., 1999; Abbott et al., 2013; 488 Yakimowski and Rieseberg, 2014). Similarly, hybridization is likely to influence different 489 components of fitness. Fitness is an inherently complex trait, putatively underlain by 490 thousands of loci across the genome. Indeed, heterosis in maize is well-known to 491 increase fitness, likely through the masking of many deleterious recessive alleles 492 (Springer and Stupar, 2007). Simultaneously, certain fitness components may be 493 simple. For example, overdominance at a single locus SFT in tomato causes production 494 of indeterminate infructescences, vastly increasing the number of fruits (Krieger et al., 495 2010). Importantly, the genotype of each parent will have a substantial impact on the 496 nature of hybrid trait expression, as seen in our total fitness model. Further 497 disentangling of the unique fitness impacts of hybridity and its ramifications for the 498 evolution of sex is an open prospect in theory and experimental research, and Boechera 499 is well-situated for this work. 500 501 Impediments to Boechera hybrid formation at multiple stages of development suggest 502 the action of multiple reproductive barriers. Hybrid inviability (indicated by relatively low 503 germination of hybrids) and sterility (indicated by both a failure to reproduce and by the 504 production of more aborted, or sterile, fruits) are both factors in F2s, suggesting the 505 presence of Bateson-Dobzhansky-Muller incompatibilities. However, the main barrier to 506 hybrid formation in our crosses occurred prior to fertilization. The failure of most F1 507 crosses to set seed is consistent with postmating prezygotic barriers. Pollen-pistil 508 barriers, when incompatible interactions between pollen and pistil results in failed 509 hybridization, are known from a range of plant taxa including maize (Mangelsdorf and 510 Jones, 1926; Kermicle, 2006), gingers (Kay, 2006; Kay and Schemske, 2008; Yost and 511 Kay, 2009), monkeyflowers (Searcy and Macnair, 1990), and tomatoes (Baek et al., 512 2015). These interactions often act asymmetrically (Tiffin et al., 2001), as we see in this 513 study. Overall, these results indicate that the inability of hybrid formation may be 514 impacted by multiple mechanisms acting in tandem each with variability in symmetry 515 and their overall impact on the likelihood of hybrid formation in nature 516 517 A variety of mechanisms, including style length disparity (e.g., Kay, 2006; Baek et al., 518 2015) and the arrest of pollen tube growth (e.g., Kermicle, 2006), are implicated in these 519 pollen-pistil barriers. Mating system may also play a role; when one hybridizing species 520 is self-incompatible (SI) and the other is self-compatible (SC), SI pollen may pollinate 521 SC ovules, but the reverse cross fails through the production of improperly developed 522 seeds (Brandvain and Haig, 2005; Bedinger et al., 2017). Although both B. stricta and 523 B. retrofracta are SC, two lines of evidence suggest that this mechanism may play a 524 role. First, average microsatellite-based FIS in B. stricta is 0.89 (Song et al., 2006), while 525 average microsatellite-based GIS in B. retrofracta is 0.71 (Rushworth et al., 2018). 526 These population statistics are dependent on the level of heterozygosity estimated in 527 each given study, and care should be taken when comparing them. Nonetheless, the

12 bioRxiv preprint doi: https://doi.org/10.1101/2020.02.11.944256; this version posted February 12, 2020. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license.

528 large disparity may suggest larger effective population size in B. retrofracta, and 529 perhaps higher levels of outcrossing. If this is the case, the SI ´ SC mechanism predicts 530 that B. stricta will succeed only as a maternal parent when paired with B. retrofracta, 531 consistent with our results (Figures 2 and 3). Additional support for an SI ´ SC 532 mechanism would be found in the characterization of aberrant seeds formed by the 533 incompatible cross direction, which is a potential area for further research. Substantial 534 future work is needed to understand the genetic mechanisms underlying hybrid 535 incompatibility in Boechera. 536 537 Critical to the evolution of hybrid incompatibilities is the environments in which they 538 evolve. Geographic isolation, for example, may strongly impact the evolution of 539 reproductive barriers, leading to expression of barriers in sympatric but not allopatric 540 populations (Coyne and Orr, 2004). This pattern is often seen in the process of 541 reinforcement, where secondary contact leads to increased reproductive isolation 542 between populations (Hopkins, 2013). Although this study is underpowered to detect 543 differences between allopatric and sympatric populations, it is worth noting that all four 544 B. stricta lineages that successfully produced hybrids are from populations allopatric 545 with B. retrofracta. Additionally, ecological variation will determine patterns of natural 546 selection on hybrids. The suitability of B. stricta as a maternal, but not paternal, parent 547 suggests that hybrids are more likely to arise in B. stricta habitat. Research focusing on 548 natural selection on B. stricta ´ B. retrofracta hybrids should be undertaken in the 549 habitats in which these lineages naturally arise and thrive. 550 551 Studies of sexual/asexual dynamics are rarely undertaken in the field, which 552 substantially influences our understanding of the evolution of sex (Neiman et al., 2018). 553 Comparison with studies conducted in the natural environment are thus of vital 554 importance to understanding the real-life costs of sex. Additionally, pairing controlled 555 experiments with those in the field can offer new insight into evolutionary processes 556 (e.g., Anderson et al., 2011). In the field, asexual lineages have higher fitness than 557 sexuals, driven by substantially higher over-winter survival of asexuals, with no 558 evidence for differences in probability of reproduction, fruit number, or fecundity 559 (Rushworth et al., 2019). In contrast, the work presented here showed that de novo 560 hybrids were less likely to reproduce than selfed lineages, resulting in lower overall 561 fitness (Table 1). However, hybrids produced more fruits than sexuals, and had higher 562 variance in fruit number (Figure 4), suggesting that some hybrid lineages may enjoy 563 extremely high fitness. The underlying reasons for this disparity in results are unclear, 564 but if hybrids that form in the wild reproduce sexually and asexually, it is likely that 565 selective patterns between them will differ. For example, if the perceived rarity of wild 566 sexual hybrids is due to their rapid extinction, this may be due to the reproductive 567 incompatibilities reported here. Alternatively, the asexuals in Rushworth et al. (2019) 568 may have been recently formed, and thus lacked the accumulated mutations reported in 569 other asexual Boechera lineages (Lovell et al., 2017). Further work ascertaining the 570 range of ages for asexual lineages, and the stage at which asexuality occurs, would

13 bioRxiv preprint doi: https://doi.org/10.1101/2020.02.11.944256; this version posted February 12, 2020. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license.

571 inform our understanding of selective mechanisms interacting with hybridization and 572 asexuality. 573 574 The evolution of sex is strongly influenced by correlated traits that interact uniquely with 575 selective agents in the natural environment. Given the hybrid origins of many asexual 576 lineages, mating system—particularly outcrossing between species—may be 577 particularly important to sexual/asexual dynamics in flowering plants. We found that the 578 expression of multiple reproductively isolating barriers manifest at several life history 579 stages following hybridization, likely impacting the frequency of sexual hybrids in natural 580 populations. Our results contribute to a growing body of literature showing that the origin 581 and fitness of hybrids is interwoven with the costs of sex in the wild. 582 583 Funding 584 585 This work was supported by funding from Duke University Biology Department, the 586 National Institutes of Health (R01 GM086496 to T.M.-O), and the National Sciences 587 Foundation in the form of a Graduate Research Fellowship to C.A.R. and a Doctoral 588 Dissertation Improvement Grant to C.A.R. and T.M.-O.(DEB-1311269). 589 590 Acknowledgements 591 592 The authors wish to thank the Mitchell-Olds lab group, especially Baosheng Wang, for 593 access to B. stricta chloroplast sequences and germplasm. We also wish to 594 acknowledge Duke University Greenhouses and Phytotron, especially Greg Piotrowski, 595 for their assistance and advice. Dan Runcie and Yaniv Brandvain offered guidance with 596 statistical analyses. 597 598 Data availability 599 600 Upon acceptance, primary data on cross success and greenhouse study will be 601 deposited in Dryad. Original sequencing data will be deposited in Genbank and updated 602 in Supplementary Material. Sequencing data from B. stricta genotypes is found in Wang 603 et al. (2019). 604 605 606 607 608 609 610 611 612 613 614

14 bioRxiv preprint doi: https://doi.org/10.1101/2020.02.11.944256; this version posted February 12, 2020. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license.

615 References 616 617 Abbott R, Albach D, Ansell S, Arntzen JW, Baird SJE, Bierne N, et al. (2013). 618 Hybridization and speciation. Journal Of Evolutionary Biology 26: 229–246.

619 Agrawal AF (2009). Spatial heterogeneity and the evolution of sex in diploids. American 620 Naturalist 174 Suppl 1: S54-70.

621 Al-Shehbaz IA, Windham MD (1993). Boechera. In: Flora of North America Editorial 622 Committee (ed) Flora of North America North of Mexico, New York and Oxford 623 Vol 7, pp 348–412.

624 Anderson JT, Lee C-R, Mitchell-Olds T (2011). Life-history QTLs and natural selection 625 on flowering time in Boechera stricta, a perennial relative of Arabidopsis. 626 Evolution 65: 771–787.

627 Asker SE, Jerling L (1992). Apomixis in Plants. CRC Press, Inc.: Boca Raton, Florida, 628 USA.

629 Baek YS, Covey PA, Petersen JJ, Chetelat RT, McClure B, Bedinger PA (2015). 630 Testing the SI × SC rule: Pollen-pistil interactions in interspecific crosses 631 between members of the tomato clade (Solanum section Lycopersicon, 632 Solanaceae). American Journal of Botany 102: 302–311.

633 Barrett SCH (2002). The evolution of plant sexual diversity. Nat Rev Genet 3: 274–284.

634 Bates D, Maechler M, Bolker B, Walker S (2015). lme4: Linear mixed-effects models 635 using Eigen and S4. Journal of Statistical Software 67.

636 Beck JB, Alexander PJ, Allphin L, Al-Shehbaz IA, Rushworth C, Bailey CD, et al. 637 (2012). Does hybridization drive the transition to asexuality in diploid Boechera? 638 Evolution 66: 985–995.

639 Becks L, Agrawal AF (2010). Higher rates of sex evolve in spatially heterogeneous 640 environments. Nature 468: 89–92.

641 Becks L, Agrawal AF (2012). The evolution of sex is favoured during adaptation to new 642 environments. PLOS Biology 10: e1001317.

643 Bedinger PA, Broz AK, Tovar-Mendez A, McClure B (2017). Pollen-Pistil Interactions 644 and Their Role in Mate Selection. Plant Physiol 173: 79–90.

645 Böcher TW (1951). Cytological and embryological studies in the amphi-apomictic Arabis 646 holboellii complex. Kongel Danske Vidensk-Selskab Biol Skr 6: 1–59.

647 Bomblies K, Higgins JD, Yant L (2015). Meiosis evolves: adaptation to external and 648 internal environments. New Phytol 208: 306–323.

15 bioRxiv preprint doi: https://doi.org/10.1101/2020.02.11.944256; this version posted February 12, 2020. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license.

649 Brandvain Y, Haig D (2005). Divergent mating systems and parental conflict as a barrier 650 to hybridization in flowering plants. American Naturalist 166: 330–338.

651 Brooks ME, Kristensen K, van Benthem KJ, Magnusson A, Berg CW, Nielsen A, et al. 652 (2017). glmmTMB Balances Speed and Flexibility Among Packages for Zero- 653 Inflated Generalized Linear Mixed Modeling. The R Journal 9: 378–400.

654 Carman JG (1997). Asynchronous expression of duplicate genes in angiosperms may 655 cause apomixis, bispory, tetraspory, and polyembryony. Biological Journal Of 656 The Linnean Society 61: 51–94.

657 Charlesworth D (2006). Evolution of Plant Breeding Systems. Current Biology 16: 658 R726–R735.

659 Charlesworth D, Willis JH (2009). The genetics of inbreeding depression. Nature 660 Reviews Genetics 10: 783–796.

661 Clauss MJ, Cobban H, Mitchell-Olds T (2002). Cross-species microsatellite markers for 662 elucidating population genetic structure in Arabidopsis and Arabis (Brassicaeae). 663 Molecular Ecology 11: 591–601.

664 Coughlan JM, Han S, Stefanović S, Dickinson TA (2017). Widespread generalist clones 665 are associated with range and niche expansion in allopolyploids of Pacific 666 Northwest Hawthorns (Crataegus L.). Mol Ecol 26: 5484–5499.

667 Coyne JA, Orr H (2004). Speciation. Sinauer Associates: Sunderland, MA, USA.

668 Dobeš CH, Mitchell-Olds T, Koch MA (2004a). Intraspecific diversification in North 669 American Boechera stricta (= Arabis drummondii), Boechera × divaricarpa, and 670 Boechera holboellii (Brassicaceae) inferred from nuclear and chloroplast 671 molecular markers—an integrative approach. American Journal Of Botany 91: 672 2087–2101.

673 Dobeš CH, Mitchell-Olds T, Koch MA (2004b). Extensive chloroplast haplotype variation 674 indicates Pleistocene hybridization and radiation of North American Arabis 675 drummondii, A. x divaricarpa, and A. holboellii (Brassicaceae). Molecular Ecology 676 13: 349–370.

677 Gerrish PJ, Lenski RE (1998). The fate of competing beneficial mutations in an asexual 678 population. Genetica 102: 127–144.

679 Hojsgaard D, Klatt S, Baier R, Carman J, Hörandl E (2014). Taxonomy and 680 biogeography of apomixis in Angiosperms and associated biodiversity 681 characteristics. Critical Reviews In Plant Sciences 33: 414–427.

682 Hopkins R (2013). Reinforcement in plants. New Phytologist 197: 1095–1103.

16 bioRxiv preprint doi: https://doi.org/10.1101/2020.02.11.944256; this version posted February 12, 2020. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license.

683 Kay KM (2006). Reproductive isolation between two closely related hummingbird- 684 pollinated neotropical gingers. Evolution 60: 538–552.

685 Kay KM, Schemske DW (2008). Natural selection reinforces speciation in a radiation of 686 neotropical rainforest plants. Evolution 62: 2628–2642.

687 Kermicle JL (2006). A Selfish Gene Governing Pollen-Pistil Compatibility Confers 688 Reproductive Isolation Between Maize Relatives. Genetics 172: 499–506.

689 Krieger U, Lippman ZB, Zamir D (2010). The flowering gene SINGLE FLOWER TRUSS 690 drives heterosis for yield in tomato. Nature Genetics 42: 459–463.

691 Lee C-R, Mitchell-Olds T (2011). Quantifying effects of environmental and geographical 692 factors on patterns of genetic differentiation. Molecular Ecology 20: 4631–4642.

693 Lee C-R, Mitchell-Olds T (2013). Complex trait divergence contributes to environmental 694 niche differentiation in ecological speciation of Boechera stricta. Molecular 695 Ecology 22: 2204–2217.

696 Lee C-R, Wang B, Mojica JP, Mandáková T, Prasad KVSK, Goicoechea JL, et al. 697 (2017). Young inversion with multiple linked QTLs under selection in a hybrid 698 zone. Nature Ecology & Evolution 1: 0119–11.

699 Lehtonen J, Jennions MD, Kokko H (2012). The many costs of sex. Trends In Ecology & 700 Evolution 27: 172–178.

701 Li F-W, Rushworth CA, Beck JB, Windham MD (2017). Boechera microsatellite website: 702 an online portal for species identification and determination of hybrid parentage. 703 Database 2016: 1–11.

704 Lovell JT, Williamson RJ, Wright SI, McKay JK, Sharbel TF (2017). Mutation 705 accumulation in an asexual relative of Arabidopsis. PLOS Genetics 13: 706 e1006550.

707 Lüdecke D (2018). ggeffects: Tidy Data Frames of Marginal Effects from Regression 708 Models. Journal of Open Source Software 3: 772.

709 Lutes AA, Neaves WB, Baumann DP, Wiegraebe W, Baumann P (2010). Sister 710 chromosome pairing maintains heterozygosity in parthenogenetic lizards. Nature 711 464: 283–286.

712 Mangelsdorf PC, Jones DF (1926). The Expression of Mendelian Factors in the 713 Gametophyte of Maize. Genetics 11: 423–455.

714 Maynard Smith J (1978). The Evolution of Sex. Cambridge University Press: 715 Cambridge, UK.

17 bioRxiv preprint doi: https://doi.org/10.1101/2020.02.11.944256; this version posted February 12, 2020. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license.

716 McDade LA, Lundberg JG (1982). A New Tabular and Diagrammatic Method for 717 Displaying Artificial Hybridization Data, with an Example from Aphelandra 718 (Acanthaceae). Systematic Botany 7: 13.

719 Meirmans S, Meirmans PG, Kirkendall LR (2012). The Costs of Sex: Facing Real-world 720 Complexities. The Quarterly Review of Biology 87: 19–40.

721 Modliszewski JL, Copenhaver GP (2017). Meiotic recombination gets stressed out: CO 722 frequency is plastic under pressure. Current Opinion in Plant Biology 36: 95–102.

723 Müller HJ (1964). The relation of recombination to mutational advance. Mutation 724 Research 1: 2–9.

725 Neiman M, Meirmans PG, Schwander T, Meirmans S (2018). Sex in the wild: How and 726 why field-based studies contribute to solving the problem of sex. Evolution 72: 727 1194–1203.

728 Orr HA (2009). Fitness and its role in evolutionary genetics. Nat Rev Genet 10: 531– 729 539.

730 Richardson JL, Urban MC, Bolnick DI, Skelly DK (2014). Microgeographic adaptation 731 and the spatial scale of evolution. Trends in Ecology & Evolution 29: 165–176.

732 Rieseberg LH, Archer MA, Wayne RK (1999). Transgressive segregation, adaptation 733 and speciation. Heredity 83: 363–372.

734 Rushworth CA, Brandvain Y, Mitchell-Olds T (2019). Identifying the fitness 735 consequences of sex in complex natural environments. bioRxiv. doi: 736 10.1101/719252.

737 Rushworth CA, Song B-H, Lee C-R, Mitchell-Olds T (2011). Boechera, a model system 738 for ecological genomics. Molecular Ecology 20: 4843–4857.

739 Rushworth CA, Windham MD, Keith RA, Mitchell-Olds T (2018). Ecological 740 differentiation facilitates fine-scale coexistence of sexual and asexual Boechera. 741 American Journal Of Botany 105: 1–14.

742 Schranz ME, Dobes C, Koch MA, Mitchell-Olds T (2005). Sexual reproduction, 743 hybridization, apomixis, and polyploidization in the genus Boechera 744 (Brassicaceae). American Journal Of Botany 92: 1797–1810.

745 Searcy KB, Macnair MR (1990). Differential seed production in Mimulus guttatus in 746 response to increasing concentrations of copper in the pistil by pollen from 747 copper tolerant and sensitive sources. Evolution 44: 1424–1435.

748 Sharbel TF, Mitchell-Olds T (2001). Recurrent polyploid origins and chloroplast 749 phylogeography in the Arabis holboellii complex (Brassicaceae). Heredity 87: 59– 750 68.

18 bioRxiv preprint doi: https://doi.org/10.1101/2020.02.11.944256; this version posted February 12, 2020. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license.

751 Sharbel TF, Voigt M-L, Corral JM, Galla G, Kumlehn J, Klukas C, et al. (2010). 752 Apomictic and Sexual Ovules of Boechera Display Heterochronic Global Gene 753 Expression Patterns. The Plant Cell 22: 655–671.

754 Song BH, Clauss MJ, Pepper A, Mitchell-Olds T (2006). Geographic patterns of 755 microsatellite variation in Boechera stricta, a close relative of Arabidopsis. 756 Molecular Ecology 15: 357–369.

757 Song B-H, Mitchell-Olds T (2007). High genetic diversity and population differentiation in 758 Boechera fecunda, a rare relative of Arabidopsis. Molecular Ecology 16: 4079– 759 4088.

760 Springer NM, Stupar RM (2007). Allelic variation and heterosis in maize: How do two 761 halves make more than a whole? Genome Research 17: 264–275.

762 Swofford DL (2003). PAUP*. Phylogenetic Analysis Using Parsimony (*and Other 763 Methods), Version 4. Sinauer Associates: Sunderland, MA USA.

764 Tiffin P, Olson S, Moyle LC (2001). Asymmetrical crossing barriers in angiosperms. 765 Proc R Soc Lond B 268: 861–867.

766 Vrijenhoek RC (1978). Coexistence of clones in a heterogeneous environment. Science 767 199: 549–552.

768 Wang B, Mojica JP, Perera N, Lee C-R, Lovell JT, Sharma A, et al. (2019). Ancient 769 polymorphisms contribute to genome-wide variation by long-term balancing 770 selection and divergent sorting in Boechera stricta. Genome Biol 20: 126.

771 Waser NM, Price MV (1994). Crossing-distance effects in Delphinium nelsonii: 772 Outbreeding and inbreeding depression in progeny fitness. Evolution 48: 842– 773 852.

774 Yakimowski SB, Rieseberg LH (2014). The role of homoploid hybridization in evolution: 775 a century of studies synthesizing genetics and ecology. American Journal Of 776 Botany 101: 1247–1258.

777 Yost JM, Kay KM (2009). The evolution of postpollination reproductive isolation in 778 Costus. Sex Plant Reprod 22: 247–255.

779

19 bioRxiv preprint doi: https://doi.org/10.1101/2020.02.11.944256; this version posted February 12, 2020. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license.

780 Table 1. Hybrids have lower total fitness due to reduced probability of 781 reproduction. Results for the fixed effects of cross type (selfed vs. hybrid) and plant 782 size from a zero-inflated negative binomial GLMM. Left, the zero-inflation portion 783 models structural zeros, or plants that did not reproduce. Fewer hybrids reproduced 784 than selfed parents. Center, the conditional portion of the model shows that cross type 785 has no effect on seed set. Right, significance estimates for the overall model, 786 incorporating both portions. Estimates (coef) and standard errors (se) come from 787 conditional models, while test statistics (χ2 deviance, degrees of freedom, and P values) 788 come from likelihood ratio tests for each overall effect. Results from independent 789 models of reproduction probability, fecundity, and aborted fruit number are in 790 Supplementary Information. Significant P-values are shown in bold. 791

Zero-inflation Conditional Overall

term condition df coef se χ2 P coef se χ2 P df χ2 P

selfed -2.74 0.419 6.63 0.252 Cross 1 9.02 0.019 0.32 0.65 2 9.34 0.047 type hybrid -1.55 0.205 6.48 0.177 4.47e- 9.60e Size - 1 -2.65 0.182 46.91 6.57 0.053 2.45 0.24 2 49.35 11 -11 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812

20 bioRxiv preprint doi: https://doi.org/10.1101/2020.02.11.944256; this version posted February 12, 2020. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license.

813 Table 2. Hybrids produce more fruits than sexuals. Results for the fixed effects of 814 cross type (selfed vs. hybrid) and plant size from a negative binomial GLMM. Estimates 815 (coef) and standard errors (se) come from conditional models, while test statistics (χ2 816 deviance, degrees of freedom df, and P values) come from likelihood ratio tests for each 817 overall effect. Significant P-values are shown in bold. 818

term condition df coef se χ2 P

selfed 2.09 0.221 Cross type 1 8.98 0.019 hybrid 3.05 0.164 Size - 1 3.48 0.077 28.86 3.12e-07 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852

21 bioRxiv preprint doi: https://doi.org/10.1101/2020.02.11.944256; this version posted February 12, 2020. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license.

853 Figure 1. Map of study populations used as cross parents. Each circle represents a 854 population. B. stricta populations are represented by filled circles, and B. retrofracta 855 populations are represented by unfilled circles. Populations where B. stricta and B. 856 retrofracta co-occur are indicated by gray fill. Plants were collected in central Idaho and 857 western Montana (inset). One population, HES, was collected in Colorado and is not 858 shown. 859

46.0

0 25 50km

● ● 45.5 ● ● ●

45.0 Latitude ● ● ● ● ● ● ● ●

● 44.5 ● stricta retrofracta ● both

−115.0 −114.5 −114.0 −113.5 Longitude 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875

22 bioRxiv preprint doi: https://doi.org/10.1101/2020.02.11.944256; this version posted February 12, 2020. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license.

876 Figure 2. Pollen-pistil barriers are asymmetrical. Interspecific crosses with B. stricta 877 as the maternal parent were more likely to set seed than those with B. stricta as the 878 paternal parent. Proportion of unique parental combinations that resulted in F1 fruit are 879 shown. Error bars indicate 95% CIs. 880

1.00

0.75

0.50

Percent successful Percent 0.25

0.00

881 maternal paternal 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896

23 bioRxiv preprint doi: https://doi.org/10.1101/2020.02.11.944256; this version posted February 12, 2020. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license.

897 Figure 3. Phylogeny of chloroplast marker trnL. Sequenced taxa show a clear split 898 between two groups, one of which is predominantly B. stricta (filled circle), and the other 899 predominantly B. retrofracta (empty circle). 30 of 31 wild-collected B. stricta ´ B. 900 retrofracta hybrids group with B. stricta, consistent with B. stricta acting as maternal 901 parent in the majority of naturally-formed hybrids. 902

903 904 905 906 907 908 909 910 911 912 913 914 915 916 917

24 bioRxiv preprint doi: https://doi.org/10.1101/2020.02.11.944256; this version posted February 12, 2020. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license.

918 Figure 4. Hybrids are less likely to reproduce, but produce more fruits than sexual 919 lineages. Estimated marginal means from GLMMs show the main effect of cross type 920 on A) the probability of reproducing; B) fruit number. Bars show 95% confidence 921 intervals. 922 A B 1.0

30 ion 0.9 ber num

oduct 20

0.8 ruit f repr ed ed 10 ict ict 0.7 red red P P 0.6 0 923 Hybrid Selfed Hybrid Selfed 924