| INVESTIGATION Testing for Ancient Selection Using Cross-population Allele Frequency Differentiation Fernando Racimo1 Department of Integrative Biology, University of California, Berkeley, California 94720 ABSTRACT A powerful way to detect selection in a population is by modeling local allele frequency changes in a particular region of the genome under scenarios of selection and neutrality and finding which model is most compatible with the data. A previous method based on a cross-population composite likelihood ratio (XP-CLR) uses an outgroup population to detect departures from neutrality that could be compatible with hard or soft sweeps, at linked sites near a beneficial allele. However, this method is most sensitive to recent selection and may miss selective events that happened a long time ago. To overcome this, we developed an extension of XP-CLR that jointly models the behavior of a selected allele in a three-population tree. Our method - called “3-population composite likelihood ratio” (3P-CLR) - outperforms XP-CLR when testing for selection that occurred before two populations split from each other and can distinguish between those events and events that occurred specifically in each of the populations after the split. We applied our new test to population genomic data from the 1000 Genomes Project, to search for selective sweeps that occurred before the split of Yoruba and Eurasians, but after their split from Neanderthals, and that could have led to the spread of modern-human-specific phenotypes. We also searched for sweep events that occurred in East Asians, Europeans, and the ancestors of both populations, after their split from Yoruba. In both cases, we are able to confirm a number of regions identified by previous methods and find several new candidates for selection in recent and ancient times. For some of these, we also find suggestive functional mutations that may have driven the selective events. KEYWORDS composite likelihood; Denisova; Neanderthal; population differentiation; positive selection ENETIC hitchhiking will distort allele frequency patterns to search for selective sweeps that occurred in the ancestral Gat regions of the genome linked to a beneficial allele that population of all present-day humans. For example, Green is rising in frequency (Smith and Haigh 1974). This is known et al. (2010) searched for genomic regions with a depletion of as a selective sweep. If the sweep is restricted to a particular derived alleles in a low-coverage Neanderthal genome, rela- population and does not affect other closely related popula- tive to what would be expected given the derived allele fre- tions, one can detect such an event by looking for extreme quency in present-day humans. This is a pattern that would patterns of localized population differentiation, like high val- be consistent with a sweep in present-day humans. Later on, ues of Fst at a specific locus (Lewontin and Krakauer 1973). Prüfer et al. (2014) developed a hidden Markov model This and other related statistics have been used to scan the (HMM) that could identify regions where Neanderthals fall genomes of present-day humans from different populations, outside of all present-day human variation (also called “ex- to detect signals of recent positive selection (Akey et al. 2002; ternal regions”) and are therefore likely to have been affected Weir et al. 2005; Oleksyk et al. 2008; Yi et al. 2010). by ancient sweeps in early modern humans. They applied Once it became possible to sequence entire genomes of their method to a high-coverage Neanderthal genome. Then, archaic humans (like Neanderthals) (Green et al. 2010; they ranked these regions by their genetic length, to find Meyer et al. 2012; Prüfer et al. 2014), researchers also began segments that were extremely long and therefore highly com- patible with a selective sweep. Finally, Racimo et al. (2014) Copyright © 2016 by the Genetics Society of America used summary statistics calculated in the neighborhood of doi: 10.1534/genetics.115.178095 sites that were ancestral in archaic humans but fixed derived Manuscript received May 11, 2015; accepted for publication November 18, 2015; published Early Online November 20, 2015. in all or almost all present-day humans, to test whether any of Supporting information is available online at www.genetics.org/lookup/suppl/ these sites could be compatible with a selective sweep model. doi:10.1534/genetics.115.178095/-/DC1. 1Address for correspondence: 1506 Oxford St., Berkeley, CA 94709. While these methods harnessed different summaries of E-mail: [email protected] the patterns of differentiation left by sweeps, they did not Genetics, Vol. 202, 733–750 February 2016 733 attempt to explicitly model the process by which these pat- the ancestral population (b) and variance proportional to the terns are generated over time. drift time v from the ancestral to the present population, Chen et al. (2010) developed a test called “cross-population jb ðb; vbð 2 bÞÞ; composite likelihood ratio” (XP-CLR), which is designed pA N 1 (1) to test for selection in one population after its split from a where v ¼ t =ð2N Þ and N is the effective size of popu- second, outgroup, population t generations ago. It does so AB e e AB lation A. by modeling the evolutionary trajectory of an allele under This is a Brownian motion approximation to the Wright– linked selection and under neutrality and then comparing Fisher model, as the drift increment to variance is constant the likelihood of the data for each of the two models. The across generations. If a SNP is segregating in both populations— method detects local allele frequency differences that are i.e., has not hit the boundaries of fixation or extinction—this compatible with the linked selection model (Smith and Haigh process is time reversible. Thus, one can model the frequency 1974), along windows of the genome. of the SNP in population a with a normal distribution having XP-CLR is a powerful test for detecting selective events mean equal to the frequency in population b and variance restricted to one population. However, it provides little infor- proportional to the sum of the drift time (v) between a and mation about when these events happened, as it models all the ancestral population and the drift time between b and the sweeps as if they had immediately occurred in the present ancestral population (c): generation. Additionally,if one is interested in selective sweeps that took place before two populations a and b split from each pAjpB NðpB; ðv þ cÞpBð1 2 pBÞÞ: (2) other, one would have to run XP-CLR separately on each pop- ulation, with a third outgroup population c that split from the For SNPs that are linked to a beneficial allele that has pro- ancestor of a and btABC generations ago (with tABC . tAB). duced a sweep in population a only, Chen et al. (2010) model Then, one would need to check that the signal of selection the allele as evolving neutrally until the present and then appears in both tests. This may miss important information apply a transformation to the normal distribution that de- about correlated allele frequency changes shared by a and b, pends on the distance to the selected allele r and the strength but not by c, limiting the power to detect ancient events. of selection s (Fay and Wu 2000; Durrett and Schweinsberg ¼ 2 r=s; To overcome this, we developed an extension of XP-CLR 2004). Let c 1 q0 where q0 is the frequency of the ben- that jointly models the behavior of an allele in all three eficial allele in population a before the sweep begins. The populations, to detect selective events that occurred before frequency of a neutral allele is expected to increase from p or after the closest two populations split from each other. to 1 2 c þ cp if the allele is linked to the beneficial allele, and Below we briefly review the modeling framework of XP-CLR this occurs with probability equal to the frequency of the and describe our new test, which we call the “3-population neutral allele (p) before the sweep begins. Otherwise, the composite likelihood ratio” (3P-CLR). In Results, we show frequency of the neutral allele is expected to decrease from this method outperforms XP-CLR, when testing for selection p to cp: This leads to the following transformation of the that occurred before the split of two populations, and can normal distribution, distinguish between those events and events that occurred ð j ; ; ; v; cÞ after the split, unlike XP-CLR. We then apply the method to f pA pB r s population genomic data from the 1000 Genomes Project 1 pA þ c 2 1 2ðð þ 2 2 Þ2= 2s2Þ ¼ pffiffiffiffiffiffi pA c 1 cpB 2c ð Þ e I½12c;1 pA (Abecasis et al. 2012), to search for selective sweep events 2ps c2 that occurred before the split of Yoruba and Eurasians, but 1 c 2 pA 2ðð 2 Þ2= 2s2Þ þ pffiffiffiffiffiffi pA cpB 2c ð Þ; e I½0;c pA after their split from Neanderthals. We also use it to search 2ps c2 for selective sweeps that occurred in the Eurasian ancestral (3) population and to distinguish those from events that oc- 2 curred specifically in East Asians or specifically in Europeans. where s ¼ðv þ cÞpBð1 2 pBÞ and I½x;yðzÞ = 1 on the interval ½x; y and 0 otherwise. For s/0orr s; this distribution converges to the neu- Materials and Methods tral case. Let v be the vector of all drift times that are relevant XP-CLR to the scenario we are studying. In this case, it will be equal to ðv; cÞ but in more complex cases below, it may include addi- First, we review the procedure used by XP-CLR to model the tional drift times.