bioRxiv preprint doi: https://doi.org/10.1101/030387; this version posted March 29, 2016. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license. 1 The genetic cost of Neanderthal introgression Kelley Harris1;2 and Rasmus Nielsen3;4;5 Stanford University Department of Genetics;1 Stanford, CA USA UC Berkeley Departments of Mathematics,2 Integrative Biology,3 and Statistics;4 Berkeley, CA USA 5 2 The Natural History Museum of Denmark; University of Copenhagen, Denmark 1 bioRxiv preprint doi: https://doi.org/10.1101/030387; this version posted March 29, 2016. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license. 3 Running title: Genetic cost of Neanderthal gene flow 4 Keywords: 5 gene flow, archaic hominins, nearly neutral theory, deleterious mutation load, heterosis 6 Corresponding author: 7 Kelley Harris 8 Stanford University Department of Genetics 9 Clark Center S240 10 Stanford, CA 94305 11 (916) 205-2904 12 [email protected] 2 bioRxiv preprint doi: https://doi.org/10.1101/030387; this version posted March 29, 2016. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license. 13 Abstract 14 Approximately 2-4% of genetic material in human populations outside Africa is derived from 15 Neanderthals who interbred with anatomically modern humans. Recent studies have shown that 16 this Neanderthal DNA is depleted around functional genomic regions; this has been suggested 17 to be a consequence of harmful epistatic interactions between human and Neanderthal alleles. 18 However, using published estimates of Neanderthal inbreeding and the distribution of mutational 19 fitness effects, we infer that Neanderthals had at least 40% lower fitness than humans on average; 20 this increased load predicts the reduction in Neanderthal introgression around genes without the 21 need to invoke epistasis. We also predict a residual Neanderthal mutational load in non-Africans, 22 leading to a fitness reduction of at least 0.5%. This effect of Neanderthal admixture has been 23 left out of previous debate on mutation load differences between Africans and non-Africans. 24 We also show that if many deleterious mutations are recessive, the Neanderthal admixture 25 fraction could increase over time due to the protective effect of Neanderthal haplotypes against 26 deleterious alleles that arose recently in the human population. This might partially explain why 27 so many organisms retain gene flow from other species and appear to derive adaptive benefits 28 from introgression. 29 1 Introduction 30 In recent years, prodigious technological advances have enabled extraction of DNA from the remains 31 of our extinct Neanderthal relatives [1]. Analysis of this ancient DNA revealed that Neanderthals 32 had lower genetic diversity than any living human population [2, 3]. By analyzing patterns of di- 33 vergence between distinct Neanderthal haplotypes, Pr¨uferet al. inferred that Neanderthals experi- 34 enced a strong population bottleneck, lasting approximately ten times longer than the out-of-Africa 35 bottleneck [2, 4, 5, 6]. 36 A classical consequence of population bottlenecks is that they interfere with natural selection 3 bioRxiv preprint doi: https://doi.org/10.1101/030387; this version posted March 29, 2016. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license. 37 by increasing evolutionary stochasticity [7, 8]. When effective population size is small and genetic 38 drift is therefore strong, weakly deleterious alleles have a tendency to persist in the population as if 39 they were neutral. Neanderthal exome sequencing has confirmed this prediction, providing direct 40 evidence that purifying selection was weaker in Neanderthals than in humans [3, 9]. Compared 41 to humans, Neanderthals have a relatively high ratio of nonsynonymous to synonymous variation 42 within proteins, indicating that they probably accumulated deleterious nonsynonymous variation 43 at a faster rate than modern humans do. 44 It is an open question whether archaic hominins' deleterious mutation load contributed to their 45 decline and extinction. However, there is clear evidence that Neanderthals escaped total genetic 46 extinction by interbreeding with the anatomically modern humans who left Africa between 50 and 47 100 thousand years ago [1]. In Europeans and Asians, haplotypes of Neanderthal origin have been 48 inferred to comprise 2{4% of each individual's genome. When pooled together, these Neanderthal 49 haplotypes collectively span about 30% of the human reference sequence [10, 11]. 50 The introgression of Neanderthal alleles related to hair, skin pigmentation, and immunity appear 51 to have provided non-Africans with adaptive benefits, perhaps because Neanderthals had pre- 52 adapted to life in Europe for thousands of years before humans left Africa [10, 11, 12, 13, 14]. 53 However, these positively selected genes represent a tiny fraction of Neanderthal introgression's 54 genetic legacy. A larger number of Neanderthal alleles appear to have deleterious fitness effects, 55 with putative links to various diseases as measured by genome-wide association studies [10, 15]. 56 The distribution of deleterious mutations in humans has been the subject of much recent re- 57 search. A controversial question is whether the out-of-Africa bottleneck created differences in 58 genetic load between modern human populations [16, 17]. Some previous studies concluded that 59 this bottleneck saddled non-Africans with potentially damaging genetic variants that could affect 60 disease incidence across the globe today [18, 19, 20], while other studies have concluded that there 4 bioRxiv preprint doi: https://doi.org/10.1101/030387; this version posted March 29, 2016. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license. 61 is little difference in genetic load between Africans and non-Africans [21, 9]. Although previous 62 studies have devoted considerable attention to simulating the accumulation of deleterious mutations 63 during the out-of-Africa bottleneck, none to our knowledge have incorporated the fitness effects of 64 introgression from Neanderthals into non-Africans. 65 In this paper, we quantify the deleterious effects on humans of introgression with Neanderthals 66 with a high mutational load. We present simulations showing that archaic introgression may have 67 had fitness effects comparable to the out-of-Africa bottleneck, saddling non-Africans with weakly 68 deleterious alleles that accumulated as nearly neutral variants in Neanderthals. 69 2 Results 70 To assess the fitness effects of Neanderthal introgression on a genome-wide scale, we used forward- 71 time simulations incorporating linkage, exome architecture, and population size changes to model 72 the flux of deleterious mutations across hominin species boundaries. We describe three main con- 73 sequences of this flux, which are not mutually exclusive and whose relative magnitudes depend 74 on evolutionary parameters such as the distribution of dominance coefficients and fitness effects 75 of new mutations. One consequence is strong selection against early human/Neanderthal hybrids, 76 implying that the initial contribution of Neanderthals to the human gene pool may have been 77 much higher than the contribution that persists today. A second consequence is depletion of Nean- 78 derthal ancestry from conserved regions of the genome, a pattern that has been previously inferred 79 from genetic data [10, 11] and interpreted as evidence for partial reproductive incompatibilities 80 between humans and Neanderthals. A third consequence is the persistence of deleterious alleles in 81 present-day humans, creating a difference in mutation load between non-Africans (who experienced 82 Neanderthal admixture) and Africans who did not. 5 bioRxiv preprint doi: https://doi.org/10.1101/030387; this version posted March 29, 2016. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license. 83 2.1 The Reduced Fitness of Neanderthals 84 Our first step toward quantifying these three consequences of introgression was to estimate pre- 85 admixture mutation loads in humans and Neanderthals. We accomplished this using simulations 86 where all humans and Neanderthals experience deleterious mutations drawn from the same distri- 87 bution of fitness effects (DFE), such that any differences in mutation load are driven by differences 88 in demographic history. Because the fitness effects of noncoding mutations are difficult to measure, 89 we restricted our attention to deleterious mutations that alter protein coding sequences (nonsyn- 90 onymous or NS mutations). There have been several estimates of the distribution of selection 91 coefficients in human protein coding genes [22, 23, 24, 25]. We here use the estimates of Eyre- 92 Walker, et al. who found that the DFE of human NS mutations is gamma-distributed with shape 93 parameter 0.23 and mean selection coefficient -0.043 [26]. Although it is probably unrealistic to 94 neglect the fitness effects of synonymous and non-exonic mutations, it is also conservative in that 95 additional deleterious mutations would only increase the human/Neanderthal load difference be- 96 yond the levels estimated here.