DISSERTATION / DOCTORAL THESIS Titel der Dissertation /Title of the Doctoral Thesis „The use of phylogenomic tools to investigate diploid and polyploid evolution in Dactylorhiza and other orchids“ verfasst von / submitted by Marie Kristine Brandrud angestrebter akademischer Grad / in partial fulfilment of the requirements for the degree of Doctor of Philosophy (PhD) Wien, 2019 / Vienna 2019 Studienkennzahl lt. Studienblatt / A 794 685 437 degree programme code as it appears on the student record sheet: Dissertationsgebiet lt. Studienblatt / Biologie/Biology field of study as it appears on the student record sheet: Betreut von / Supervisor: Ass.-Prof. Dipl.-Ing. Dr. Ovidiu Paun 2 ACKNOWLEDGMENTS I would like to thank Ass. Prof. Ovidiu Paun for allowing me to do this PhD here in Vienna, including me in his research group and being my supervisor these past 3 and a half years. I would also like to thank all our collaborators, and especially Prof. Mikael Hedrén, for sharing all his knowledge and time. Colleagues has come and gone during my time here, but I have been very lucky to always have a good group of PhD students around. I could never have imagined going through this PhD without my steady and silly office colleague Thomas Wolfe, that started his PhD at the same time as me and has been going through the same phases in parallel. Our project working on “The time capsule analysis of ancient variation stuck in polyploids” is now coming to an end. Our Post-Doc, when I started, Emiliano Trucchi as well as Lucia Castello and Sonia Mentrida was good support in the beginning to settle in in the lab in Vienna and I am very thankful for the support and coffee that always would be found in Jacky Heckenhauer’s of- fice. Jacky is also especially acknowledged for helping to translate the abstract for this thesis. Furthermore, it has been wonderful to share experiences, valuable discussions, lunch and ex- cellent coffee with Mimmi Eriksson, Aglaia Szukala, Gil Yardeni, Marylaure de La Harpe, Huijing Shang, David Field, Juliane Baar, Jamie McCann, Camille Criste, Rita Tonin, Nico Criscuolo and Jacky Hess. Ásgeir Trausti Einarsson, Mammút and Ólafur Arnalds are acknowledged for providing inspi- ration throughout this PhD. From the outside world it has been wonderful to live in a really great flat share and having flatmates accepting an at times messy PhD student. Chiara Rudel is also acknowledged for looking over the translation of the abstract and always being there to listen. I would like to thank my family and especially my parents for always supporting, being enthusiastic and believing in me. My dad especially for nice discussion and comments on the project and my mum for amazing photographs in the field and for always believing there is a solution. I also do not know how I would have managed without my Austrian family - I am very grateful to have Hannah and Clemens Tussetschläger in my life and I always enjoy every weekend trip in Innsbruck. It has also been nice to stay in touch with fellow students from my home university – Oslo, José Cerca de Oliveira and Siri Birkeland to mention a few. Moral support has also always been there from my friends back home. Bjørn Midthun is acknowl- edged for IT help (e.g. computer’s refusal of the new battery while in the field and restoring, what should have been, eliminated files) and I am indebted to almost the whole Grech family 3 for technical support on the crucial printing and scanning of a form that refused to print cor- rectly. And lastly, David Grech. As much for scientific discussions, stupid laughs and distractions, pushing me forward and being especially caring, supportive and understanding in this last phase. David is also acknowledged for helping with the proofreading of the thesis. I would also like to thank the reviewers for accepting to review the thesis. This project was funded by the Austrian Science Fund (FWF, Project Y661-B16, a grant giv- en to Ass. Prof. Ovidiu Paun). “tiny changes to earth” – Scott Hutchison 4 TABLE OF CONTENTS ABSTRACT……………………………………………………………………………………6 ZUSAMMENFASSUNG………………………………………………………………………7 INTRODUCTION………………………………………………………………...…………...8 THESIS QUESTIONS……………………………………………………………...………...17 LITERATURE CITED…………………………………………………...……………...…...18 CHAPTER 1 – Phylogenomic Relationships of Diploids and the Origins of Allotetraploids in Dactylorhiza (Orchidaceae): RADseq Data Tracks Reticulate Evolution……………..…….23 CHAPTER 2 – The impact of recurrent origins, genetic drift and gene flow on the genetic structure of allopolyploid marsh orchids………………………………………………….….70 CHAPTER 3 – Restriction site-associated DNA sequencing supports a sister group relation- ship of Nigritella and Gymnadenia (Orchidaceae)………………………………………….104 CHAPTER 4 – Iterative allogamy-autogamy transitions drive actual and incipient speciation during the ongoing radiation within the orchid genus Epipactis (Orchidaceae)……...……..128 THESIS CONCLUSION....….……………………………………………………………...162 APPENDIX – Conference Contributions…………………………………………………...165 5 ABSTRACT Disentangling phylogenetic relationships is challenging in recently evolved groups, especially in the presence of ongoing hybridization. Allopolyploids originate from the merging of two or more entire genomes and are in most cases immediately isolated from their parents. The com- plexity of the genome ancestry is increased by hybridization with other allopolyploids as well and backcrossing with their parental species. Dactylorhiza (Orchidaceae: Orchidinae) is a genus much affected by allopolyploid speciation and reticulate phylogenetic relationships. Here we use the genetic variation at tens of thousands of genomic positions to trace evolu- tionary history in Dactylorhiza by first delimiting the diploid taxa and establishing their rela- tionship and then grouping 16 allopolyploids by maximum affiliation to their putative parents. The two sibling allopolyploids, D. majalis and D. traunsteineri were studied in more depth to investigate genetic structure and demography. The diploid parents throughout their European distribution were used to create two artificial genomic references for which two pipelines were used to align the polyploid reads and call genotypes/infer genotype likelihood. Two in- dependent origins are suggested for D. traunsteineri and D. majalis, though gene flow does occur. Two other boreal sister genera Nigritella and Gymnadenia are morphologically dis- tinct, though single-marker studies have previously given contradictory results on the inclu- sion/exclusion of Nigritella in Gymnadenia. Thousands of genetic markers were used to un- ravel their phylogenetic status. Lastly, evolutionary and ecological questions were addressed within the genus Epipactis on speciation via transitions from allogamy to autogamy. Genomic inferences based on RADseq (i.e. restriction-site associated DNA sequencing) data have be- come widely applied to answer evolutionary questions in diploid organisms and is included in all four chapters of this PhD, also showing its application to polyploid organisms. We obtain highly congruent evolutionary relationships inferred both with genotype-based and genotype- free analytical methods and develop bioinformatic approaches that can be applied to other naturally occurring, non-model, polyploid complexes. 6 ZUSAMMENFASSUNG Die Untersuchung phylogenetischer Beziehungen von sich kürzlich entwickelten Gruppen stellt eine große Herausforderung dar, insbesondere bei fortlaufender Hybridisierung. Allopo- lyploide Organismen stammen aus der Verschmelzung von zwei oder mehreren Genomen und werden in den meisten Fällen sofort von ihren Eltern isoliert. Die Komplexität der genomi- schen Abstammung wird durch die Hybridisierung mit anderen Allopolyploiden sowie durch Rückkreuzung mit ihren möglicherweise vorkommenden Elternarten erhöht. Die Gattung Dactylorhiza (Orchidaceae: Orchidinae) ist stark von allopolyploider Artbildung und retikulä- ren phylogenetischen Beziehungen betroffen. In dieser Arbeit verwenden wir genetische Va- riation, um die Evolutionsgeschichte in Dactylorhiza zu rekonstruieren, indem wir zunächst die diploiden Taxa abgrenzen, ihre Verwandtschaftsbeziehung festlegen und dann 16 allopo- lyploide Taxa nach maximaler Zugehörigkeit zu ihren mutmaßlichen Eltern gruppieren. Die beiden Geschwister Allopolyploide, D. majalis und D. traunsteineri, wurden untersucht, um die genetische Struktur und Demografie zu untersuchen. Die diploiden Eltern wurden in ihrer gesamten europäischen Verbreitung dazu verwendet, zwei künstliche genomische Referenzen zu erstellen um die polyploiden Werte abzugleichen und Genotypen zu ermitteln bzw. Geno- typ-Wahrscheinlichkeit zu nennen. Für D. traunsteineri und D. majalis wurden zwei unab- hängige Ursprünge ermittelt, obwohl ein Genfluss zwischen beiden Arten herrscht. Zwei wei- tere boreale Schwestergattungen Nigritella und Gymnadenia sind morphologisch unterschied- lich, obwohl Einzelmarkerstudien zuvor widersprüchliche Ergebnisse zum Einschluss / Aus- schluss von Nigritella in Gymnadenia erbracht haben. Tausende genetischer Marker wurden verwendet, um ihren phylogenetischen Status aufzuklären. Schließlich wurden evolutionäre und ökologische Fragen in der Gattung Epipactis zur Artbildung durch einen Übergang von Allogamie zur Autogamie angesprochen. RAD-Daten (Restriktionsstellen Assoziierte DNA) werden in weitem Umfang zur Beantwortung von evolutionären Fragestellungen diploider Organismen angewendet. Die vier Kapitel dieser Doktorarbeit zeigen den Nutzen von RAD- Daten für die Arbeit