UNIVERSITY OF CALIFORNIA Los Angeles Demography-aware inference of the strength of natural selection A dissertation submitted in partial satisfaction of the requirements for the degree Doctor of Philosophy in Bioinformatics by Vicente Diego Ortega Del Vecchyo 2016 © Copyright by Vicente Diego Ortega Del Vecchyo 2016 ABSTRACT OF THE DISSERTATION Demography-aware inference of the strength of natural selection By Vicente Diego Ortega Del Vecchyo Doctor of Philosophy in Bioinformatics University of California, Los Angeles, 2016 Professor John P. Novembre, Co-Chair Professor Kirk Edward Lohmueller, Co-Chair Levels of genetic and possibly phenotypic variation are influenced by how natural selection acts to change the frequency of deleterious and advantageous mutations. However, the demographic history of a population influences the efficacy of natural selection to keep deleterious variants at low frequencies and to raise the frequency of advantageous mutations. I present three projects where I study how natural selection works in the context of different demographic histories. On the first project, I study the early demographic history of dogs and wolves since their divergence using genomic data. I inferred population bottlenecks in dogs and wolves and I found evidence for gene flow between dogs and wolves after their divergence. I develop a summary statistic to find the most plausible demographic model for dogs and wolves, where I found ii evidence for a demographic model stating that dogs evolved from one single location. This project laid the foundation to study how advantageous and deleterious variants behave in the context of the bottlenecks found in dogs and wolves. On the second chapter, I leverage the demographic models I inferred to study how demographic processes have influenced levels of deleterious genetic variation in dogs using 90 whole-genome sequences from breed dogs, village dogs and gray wolves. I used the ratio of heterozygosity at amino-acid changing variants over silent variants to show how bottlenecks associated with domestication and breed formation in dogs have affected the efficacy of negative selection. I show multiple lines of evidence indicating that bottlenecks, and not inbreeding, are driving the patterns of deleterious genetic variation we observed in dogs. In the third project, I develop a novel likelihood-based method that uses the lengths of pairwise haplotype identity by state among haplotypes carrying rare variants. The method conditions on the present-day frequency of the allele and is based on theory predicting that, under constant population sizes, the alleles under negative selection are on average younger than neutral alleles and should have higher average levels of haplotype identity among variant carriers. I developed a computational framework to obtain the probability distribution of the lengths of pairwise haplotype identity given a certain selection coefficient, demographic scenario and present-day allele frequency. Simulations indicate that our method provides unbiased estimates of selection under constant population sizes and realistic demographic scenarios. I show how the method can also be used to estimate the parameters that define the distribution of selective coefficients of a set of rare variants. I provide an example of how to apply iii this method to estimate the distribution of selective coefficients of a set of amino-acid changing variants in the UK10K, a large genomic dataset of British individuals. iv The dissertation of Vicente Diego Ortega Del Vecchyo is approved. Kenneth L. Lange Janet S. Sinsheimer Robert Wayne Kirk Edward Lohmueller, Committee Co-Chair John P. Novembre, Committee Co-Chair University of California, Los Angeles 2016 v To mom, dad, and Roxana vi Table of Contents Introduction ..................................................................................................................... 1 Neutral theory and nearly neutral theory of molecular evolution .......................... 2 Interaction between selection and demography...................................................... 8 Inference of demography using genome-wide scale data .................................... 12 The interaction between demography and the distribution of fitness effects .... 16 Outline of Dissertation ............................................................................................. 19 Inferring the dynamic early history of dogs and wolves .......................................... 21 Introduction ............................................................................................................... 21 Results ....................................................................................................................... 23 Genetic Distance Metrics ........................................................................................ 23 Population Size Change Inference From Single Genome Sequences ................... 27 Quality control of population size change inferences .............................................. 29 Post-Divergence Gene Flow ................................................................................... 36 Model fit using the ABBA/BABA/BBAA configurations statistics ............................. 42 Discussion ................................................................................................................. 49 Bottlenecks and selective sweeps during domestication have increased deleterious genetic variation in dogs ......................................................................... 53 Introduction ............................................................................................................... 53 Results ....................................................................................................................... 54 Description of the data ............................................................................................ 54 Genome-wide patterns of deleterious variation ....................................................... 57 vii Forward-in-time simulations using PReFerSim ....................................................... 61 The role of recent inbreeding .................................................................................. 74 Discussion ................................................................................................................. 78 Inference of the distribution of fitness effects of segregating variants using haplotypic information ................................................................................................. 79 Introduction ............................................................................................................... 79 Results ....................................................................................................................... 81 Inference of selection .............................................................................................. 81 Importance sampling ............................................................................................... 84 Integration over the space of allele frequency trajectories using importance sampling .................................................................................................................. 86 Estimation of selection in constant population sizes ............................................... 90 Estimation of selection in non-equilibrium demographic scenarios ......................... 96 Inference of the distribution of fitness effects of variants at a particular frequency .............................................................................................................................. 102 Connecting the distribution of fitness effects of variants at a particular frequency with the distribution of fitness effects of new variants ........................................... 106 Inference of the distribution of fitness effects of 1% frequency variants in the UK10K dataset ...................................................................................................... 108 Discussion ............................................................................................................... 112 Appendix - Simulation Command Lines ................................................................... 117 Bibliography ................................................................................................................ 124 viii List of Figures Figure 2-1.- Geographic distribution of sampled lineages. ............................................. 22 Figure 2-2.- Neighbor-joining tree of canid samples plus the Boxer reference (CanFam3.0) for all positions passing our quality filters and for which there was no missing data for any sample. ........................................................................... 26 Figure 2-3.- Ne trajectories of 6 canid lineages reconstructed using the PSMC method. Dark and light lines indicate whole genome based estimates and bootstrap estimates, respectively. ........................................................................ 29 Figure 2-4.- Ne trajectories of 6 canid lineages reconstructed with the PSMC method using all the genomic information that passed our quality filters (dashed lines) and excluding 43 regions with runs of homozygosity (solid lines). ..................... 32 Figure 2-5.- Ne trajectories of 6 canid lineages reconstructed with the PSMC method using the sites that had a GQ ≥ 10. .....................................................................