Fossils From Vindija , (38–44 kya)

Admixture between Archaic and Modern Humans

Alan R. Rogers

September 23, 2021

1 / 64 2 / 64 Hominin tooth from , Altai Mtns, southern Hominin finger from Denisova Cave Siberia (41 kya)

3 / 64 4 / 64 Sites yielding Archaic hominin DNA Outline

I Estimating admixture from shared derived alleles I Deep separation plus extensive LD. I Selection against archaic DNA I Excess in Asia I Consequences of small population size I Multiple populations

5 / 64 6 / 64 Nucleotide site patterns Population tree

. . Ancestral allele (0) is shared with . . . . chimp...... Nucleotide Derived allele (1) shared by two ...... Site Pattern ...... human populations...... ea en an ...... Eur 1 1 0 ...... Pattern ea: most common; ...... Afr 1 0 1 ...... reflects history of population ...... Nea 0 1 1 splits...... Chmp 0 0 0 ...... # 303,340 103,612 95,347 Patterns en & an: how do they ...... arise? AENC A, Africa; E, Europe; N, Neanderthal; C, chimpanzee Why does en exceed an?

7 / 64 8 / 64 Embedded gene genealogy with mutation Incomplete lineage sorting

Pattern an Pattern en ...... I Genealogy of 4 genes shown ...... in color...... Bullet ( ) marks mutation ...... I ...... • ...... from allele 0 to allele 1...... •...... •...... I Descendants of mutant have ...... •...... allele 1; others have 0...... I Gene genealogy matches ...... phylogeny ...... I Mutant allele shared by ...... AENC ...... closest relatives, A and E. AENC AENC 1 1 0 0 1 0 1 0 0 1 1 0 These two should be equally common

9 / 64 10 / 64 Nucleotide site patterns again Neanderthal admixture inflates en site pattern.

Admixed locus Not admixed Nucleotide Site Pattern ...... ea en an ...... European 1 1 0 ...... African 1 0 1 ...... Neanderthal 0 1 1 ...... •...... Chimpanzee 0 0 0 ...... # sites 303,340 103,612 95,347 ...... •...... Common pattern (ea) reflects history of population splits...... m...... m...... Absent admixture, the other two should be equally common AENC AENC 0 1 1 0 0 1 1 0 Why does en exceed an? Key: A, Africa; E, Europe; N, Neanderthal; C, chimpanzee.

11 / 64 12 / 64 Estimate from Neandertal DNA Eurasians share some derived alleles with archaics

DNA of modern Eurasians is 1.5–2.1% Neandertal I Neanderthal matches French 4.6% more often than Yoruban (Pr¨uferet al 2014). (African). I Same is true for modern people of east Asia and , but not Africa. Green et al (2010) Denisova matches French 1.8% more often than Yoruban (African). I Admixture must have occurred after moderns left Africa but before they expanded throughout the world. Archaic component of Eurasian genome more Neanderthal than Denisovan. Eurasian introgressed segments most similar to Neanderthal I Green et al 2010; Reich et al. 2010. from (Mezmaiskaya). (Pr¨uferet al 2014)

13 / 64 14 / 64 So do Asians and Papuans Apparent gradual decline in Neanderthal admixture

Asians and Papuans carry as many Neanderthal alleles as Europeans do: 1.5–2.1%.

15 / 64 16 / 64 Apparent gradual decline in Neanderthal admixture Denisovan DNA most common in , NG, and Oceania

This turned out to be an artifact.

The estimators of admixture were biased.

Magnitude of bias differed in samples of different age.

17 / 64 18 / 64 Outline Another approach: look for deep separation plus extensive LD

Estimating admixture from shared derived alleles ◦ Examine variation in modern human DNA. I Deep separation plus extensive LD. I Look for long segments of chromosome with many nucleotide I Selection against archaic DNA I differences. I Excess Neanderthal in Asia I Many nucleotide differences deep separation. I Consequences of small population size ⇒ I Long segment extensive LD I Multiple Denisovan populations ⇔ I Extensive LD short residence in modern population. ⇒

19 / 64 20 / 64 OAS1 innate immunity locus Worldwide frequency of Melanesian OAS1 allele

I two forms of gene in Melanesia: I one shared with rest of world I one only in Melanesia

21 / 64 22 / 64 Melanesian OAS1 allele w/i Melanesia Melanesian OAS1 allele is old yet young

I The 2 alleles differ at many nucleotide sites separation ⇒ time 3.4 my. ∼ I Long (90 kb) LD block they’ve been together only 25 ky ⇒ ∼ I Melanesian allele matches that in Denisovan hominin skeleton. archaic admixture into Melanesia ⇒

23 / 64 24 / 64 Outline

Introgressed segments in Eurasian genomes Estimating admixture from shared derived alleles ◦ Deep separation plus extensive LD. Blue, Europe; red, Asia. ◦ I Selection against archaic DNA Inserts are dense in some I Excess Neanderthal in Asia regions, scarce in others. I Consequences of small population size

Vernot and Akey (2014) I Multiple Denisovan populations

25 / 64 26 / 64 Selection against hybrids

Neanderthal inserts rare where selection is strong

Sankararaman et al (2014)

Vernot and Akey (2014) Decreasing selection → Introgressed segments rare where modern-Neandertal difference large.

27 / 64 28 / 64 Outline Asians have more Neandertal than Europeans

Estimating admixture from shared derived alleles ◦ Deep separation plus extensive LD. ◦ Selection against archaic DNA ◦ I Excess Neanderthal in Asia I Consequences of small population size I Multiple Denisovan populations

Vernot and Akey (2014)

29 / 64 30 / 64 Did Asians receive a 2nd dose on Neanderthal DNA?

2-pulse model fits; 1-pulse model doesn’t

Vernot and Akey (2014)

Vernot and Akey (2014)

31 / 64 32 / 64 Why do Asians have more Neanderthal DNA than Has selection removed Neanderthal DNA in Europe Europeans?

I Selection removes variation. Hypotheses I Other things equal, variation should be low where selection 1. 2nd pulse of Neanderthal admixture into Asia has been strong. 2. European dilution. 2nd pulse of non-Neanderthal admixture I Hypothesis: such regions should contain fewer Neanderthal into Europe. inserts in Europe than in Asia. 3. Purifying selection. Selection against Neanderthal alleles more Vernot and Akey (2015) effective in Europe, because of larger population size. Two recent papers have tested hypothesis 3.

33 / 64 34 / 64

Measuring these effects Illustration of Rind and Rpop (not real data)

B measures the strength of purifying selection. B = 0 means strong selection; B = 1 means no selection.

Rind > 1 means the average Asian has more Neanderthal DNA than the average European.

Rpop > 1 means the Asian population includes more of the Neanderthal genome than the European one. R = 1.2 Average Asian has 20% more Neanderthal DNA ind ⇒ than average European. R = 1 Asia and Europe include same fraction of Neanderthal pop ⇒ genome.

35 / 64 36 / 64 Selection doesn’t explain Asian excess

Selective constraint decreases from left Selective constraint to right. decreases from left to right. Rpop declines where selection is weak.

No effect on Rind Vernot and Akey Selection not attribute this to responsible for greater drift in Asia. (But why Asian excess. Decreasing selection Decreasing selection (Vernot et al 2015) → does this affect only → neutral loci?)

37 / 64 38 / 64 2nd source of admixture Primary archaic admixture ...... • ...... λ ...... λ ...... Red: derived ...... κ ...... κ Blue: ancestral ...... • ...... ζ ...... m ...... ζ ...... N ...... mN ...... δ ...... δ ...... α ...... α ...... mD ...... m ...... D .. .. XYND XYND X , Africa; Y , Europe; N, Neanderthal; D, Denisova 0 1 1

39 / 64 40 / 64 Secondary (ghost) archaic admixture ...... 0.3 . . . . ◦...... Ghost admixture causes bias ...... •...... Admixture: 5% Neanderthal ...... 0.2 ... λ ...... ◦ & 2.5% Denisovan...... E[RN ] ...... ◦ ...... ◦◦ ...... •...... ◦ ...... 0.1 ...... ◦◦◦ Red line and circles: Expected ...... ◦◦◦◦◦ bias ...... ◦◦◦◦◦◦◦◦◦◦◦ ◦ ...... mN ◦◦...... ◦◦ ...... ◦◦◦◦◦ ...... ◦ ...... ← ...... value of estimator of mN ...... mD ...... 0.0 ...... Horizontal dashed line: true ...... κ 0.1 0.3 0.5 ...... parameter value ...... Neanderthal-Denisovan ...... separation time ζ ...... The difference is bias...... mN ...... (units of NXYND generations) δ ...... α ...... m ...... D .. .. XYND 0 1 1

41 / 64 42 / 64 Ghost admixture causes bias Outline

Estimating admixture from shared derived alleles ◦ If East Asians received genes from as well as Deep separation plus extensive LD. , our estimates of Neanderthal admixture would be ◦ Selection against archaic DNA inflated. ◦ Excess Neanderthal in Asia ◦ May explain Asian excess of Neanderthal admixture. I Consequences of small population size I Multiple Denisovan populations

43 / 64 44 / 64 Consequences of small population size Estimated times to most recent common ancestor (TMRCA)

I Low heterozygosity I Drift strong, so deleterious mutations accumulate. I Extinction? Long stretches of low TMRCA recent close inbreeding. ⇒ Pr¨uferet al (2014) (Pr¨uferet al 2014)

45 / 64 46 / 64 Altai Neanderthal was very closely inbred Long history of small population size

All of these pedigrees are plausible. (Pr¨uferet al 2014)

Even after removing the effects of inbreeding during the last few generations, the Altai Neandertal is still highly inbred.

47 / 64 48 / 64 Other archaics were less inbred Coding sequence variation in archaics

Altai Neanderthal had many runs of homozygosite longer than 10 cM.

A centimorgan (cM) is about a Castellano et al (2014) study 17,367 protein-coding genes in million nucleotides. several Neanderthals, the Denisovan, and several moderns.

Vindija Neanderthal had fewer such runs.

Denisovan had none.

All had runs in range 2.5–10 cM.

49 / 64 50 / 64 Neanderthals had low heterozygosity Archaics had low heterozygosity

Heterozygosity Species Population 1000 × Neanderthal El Sidr´on 0.143 Vindija 0.127 Altai 0.113 Modern African 0.507 European 0.387 Asian 0.358

Low heterozygosity small population. ⇒ Long runs of homozygosity recent close inbreeding. ⇒ Castellano et al (2014)

51 / 64 52 / 64 Tree based on mutations (left) and drift (right) Selection less effective in Neanderthals

Neighbor-joining tree FST tree

Long archaic branches on F tree small population size. ST ⇒ Castellano et al (2014)

53 / 64 54 / 64 How many deleterious mutations would you Selection less effective in expect? Neanderthals How large would these Many Neanderthal alleles have deterious effects tend large effect on protein structure to be? and are probably deleterious. (Castellano et al 2014) Simulation designed to answer these questions. (Harris and Nielsen 2015)

55 / 64 56 / 64 Simulated fitnesses of Neanderthals and moderns A few alleles of large effect, or many small effects?

57 / 64 58 / 64 Most of the action is where N s 1 Neanderthal effects strongest on brain A ≈

Largest effect on fitness: 0.0001 < s < 0.001

Or: 0.1 < NAs < 1, where NA is size of archaic population.

Genetic disease dominated by alleles with small effect.

(Simonti et al 2016)

59 / 64 60 / 64 Outline Papuans got DNA from 2 Denisovan pops

Vertical axis: length of introgressed segment Estimating admixture from shared derived alleles ◦ Deep separation plus extensive LD. Horizontal: diff btw segment and ◦ Denisovan genome as fraction of Selection against archaic DNA ◦ Denisovan-modern diff. Excess Neanderthal in Asia ◦ Consequences of small population size Color: number of fragments in ◦ this bin of length dissimilarity I Multiple Denisovan populations × Two Denisovan populations: one Jacobs et al (2019) 0.15 and one at 0.2.

61 / 64 62 / 64 E Asians got DNA from 1 Denisovan pop Summary I Two ways to detect archaic admixture: 1. Admixture distorts the pattern by which archaic and modern populations share derived alleles. 2. Deep separation plus extensive LD. I Neanderthals 1.5–2.1% of DNA in Eurasia; not Africa. → I Denisovan DNA most common in Melanesia and Oceania. I Archaic DNA uneven along chromosome: rare where selection is strong or modern-archaic difference large. I Neanderthal in Asia than Europe, probably because Asians received a 2nd pulse of admixture. I Small archaic populations many deleterious alleles. Only one Denisovan population at 0.07. → I Fitness reduction probably reflects many weakly deleterious mutations. I There were multiple Denisovan subpopulations, which made unequal contributions of moderns in different regions.

63 / 64 64 / 64