In The Princeton Guide to Evolution ed. J. Losos. Princeton University Press, Princeton, New Jersey, pp. 356-362. IV.8 Evolution of Mating Systems: Outcrossing versus Selfing Spencer C. H. Barrett OUTLINE that transitions to predominant selfing have occurred on numerous occasions, although selfing lineages are often 1. Definitions and measurement short-lived because of the negative effects of selfing on 2. Variation in mating patterns the genome. 3. Evolution of self-fertilization 4. Mechanisms of selection 5. The problem of mixed mating 6. Evolutionary history GLOSSARY Floral Design and Display. The morphological features of Mating systems vary enormously among groups of or- flowers and inflorescences that influence pollen dis- ganisms. This has led to diverse definitions and ap- persal and mating patterns in flowering plants; floral proaches for investigating their evolution and main- design involves characteristics of individual flowers, tenance. Animal mating systems are characterized by including their size, structure, color, and the spatial different patterns of parental investment in offspring and temporal presentation of female (pistil) and male and variation in the extent to which sexual selection (stamens) sex organs, and floral display concerns shapes male and female traits (see chapter VII.4). A the number of open flowers on a plant and their ar- primary focus of most studies is determining the causes rangement within and among inflorescences. and consequences of variation in mate number for fe- Inbreeding Depression. The reduction in viability and/ males and males. In contrast, in hermaphrodite organ- or fertility of inbred offspring in comparison with isms, particularly plants, the emphasis is largely on de- outbred offspring. It results primarily from the ex- termining the incidence of cross- and self-fertilization pression of deleterious recessive alleles in homo- and its fitness consequences. Most studies of mating- zygous genotypes and is expressed most strongly system evolution have been conducted on plants and when inbreeding occurs in primarily outcrossing largely concern the ecological and genetic mechanisms populations; a key parameter (d) in models of mat- responsible for evolutionary transitions from outcrossing ing-system evolution. to predominant self-fertilization, and the extent to which Mating System. Broadly defined, the mode of trans- mixed mating can be maintained as a stable strategy. mission of genes from one generation to the next Models of mating-system evolution involve the balance through sexual reproduction; however, in species between the transmission advantage of alleles affecting with separate sexes, it largely concerns the quantity the selfing rate and the reduced fitness of self-fertilized and quality of mates obtained by males and females. offspring because of inbreeding depression. Selfing pro- In contrast, in hermaphrodites, it is usually defined vides reproductive assurance whenever outcrossing is as the relative frequency of cross-fertilization (out- limited by the availability of pollinators or mates and crossing) and self-fertilization (selfing) in a popu- thus low density often plays an important role in mating- lation. In this article the last definition is used. system transitions. Reconstruction of the evolutionary Modes of Self-pollination. The various ways in which self- history of mating systems using phylogenies indicates pollination occurs in flowering plants, distinguished Evolution of Mating Systems 357 primarily by reproductive expenditure, whether a influences how and with whom a plant or animal can pollen vector is involved, and timing, relative to mate. But although the types of mating systems are cross-pollination. The evolution of selfing depends closely linked to the distinctive features of particular critically on the particular mode of self-pollination. taxa, it is not uncommon to find quite different mating Pollen and Seed Discounting. Important reproductive strategies among closely related species, especially in parameters for determining whether selfing will flowering plants. This indicates that mating systems are evolve in flowering plants; pollen discounting is evolutionarily labile and can respond to natural selec- the loss in outcrossed siring success caused by self- tion; therefore, understanding mating-system diversity pollination, whereas seed discounting is the reduc- has been a major theme in evolutionary biology since tion in outcrossed seed production caused by selfing, Charles Darwin’s influential work on the topic. either because selfing preempts ovules that could Reproductive biology is replete with terms for what have been outcrossed, or because self-fertilized are seemingly similar phenomena. Before we begin, it is seeds consume resources that could have been al- important to clarify what is meant by “mating system” located to outcrossed seeds. and how it relates to other terms associated with sexual Pollen Limitation. The reduction in potential seed pro- reproduction. This is important because the term has duction that occurs when ovules remain unfertilized various definitions and different usages, particularly in and too few embryos survive genetic death to com- the botanical versus zoological literature. The distinc- pete for maternal resources; results from insufficient tions depend largely on whether biologists are working delivery of pollen quantity and quality to flowers. on species with separate sexes (dioecy) versus those Persistent pollen limitation can result in the evolu- studying hermaphrodite (cosexual) organisms. Terms tion of selfing. that are commonly used in reproductive biology involve Reproductive Assurance. An increase in seed production the relative importance of sexual versus asexual re- caused by selfing when conditions for outcrossing production (reproductive system) (see chapter III.9), the are unfavorable because of an absence of pollinators occurrence of separate sexes versus hermaphroditism or mates; requires plants to be self-compatible and (sexual system), and within a population, who mates generally capable of autonomous self-pollination. with whom and how often (mating system) (see chapter Self-compatible and Self-incompatible. The two condi- VII.4); however, workers studying hermaphroditic or- tions that determine whether fertile hermaphrodites ganisms usually define mating system more specifically have the potential to produce offspring from self- as the relative frequency of cross-fertilization and self- fertilization. In plants, self-compatibility is the ability fertilization, and this is the definition used here, because to produce abundant seed following self-pollination, the main focus of this article is mating patterns in her- whereas in a self-incompatible plant, few, if any, seeds maphrodites. It is also important to note that the term are produced by self-pollination. Self-incompatibility breeding system is often used synonymously with mating is the most common antiselfing mechanism in flow- system in the animal and plant literature but usually ering plants. more broadly, to include mating behavior and parental care in animals, and the diverse reproductive traits that promote particular mating patterns in plants. Animal biologists in the field of behavioral ecology 1. DEFINITIONS AND MEASUREMENT classify mating systems largely on the basis of mate The mating system is a key life history trait because it number per male and female and are particularly inter- determines the quantity and genetic quality of offspring ested in determining the variance in reproductive success produced by an individual and thus the individual’s of the sexes and thus the scope of sexual selection. Here, reproductive fitness. One of the most striking features the mating system is viewed as a behavioral strategy for of organismal diversity is the existence of numerous obtaining mates, which also encompasses the nature of mating strategies, even though all serve the same basic parental care. Typical classes of animal mating systems function: to promote parental reproductive success. include monogamy, in which each sex mates with only Why do such diverse adaptations exist for an activity one partner during its lifetime, polygyny, where females crucial for persistence of species of sexual organisms? mate with a single partner but males are variable in mate Organismal variation in mating systems is determined, number, and polyandry, in which the reverse pattern in part, by the biological features that characterize a occurs. Information on the types of animal mating sys- particular taxon. For example, whether a group is sed- tems provides a powerful tool for predicting the degree entary or mobile and what type of sexual system it and direction of sexual dimorphism, as Darwin origi- possesses (e.g., hermaphroditism versus separate sexes) nally pointed out. These topics are discussed in more 358 Evolutionary Processes detail in Section VII: Evolution of Behavior, Society, 2. VARIATION IN MATING PATTERNS and Humans. Darwin wrote extensively about animal mating sys- The use of genetic markers for measuring mating pat- tems and sexual selection, but he also published three terns in populations has uncovered considerable varia- books on the reproductive biology of plants. This body of tion among plant species, and this diversity often pro- work initiated contemporary studies of mating-system vides the template for evolutionary transitions. Most evolution in plants. Because of the predominantly her- observed variation in outcrossing rate occurs in self- maphroditic sexual