Algebraic Statistics of Gaussian Mixtures vorgelegt von M.Sc. Mathematik Carlos Enrique Am´endolaCer´on geb. in Mexiko-Stadt von der Fakult¨atII - Mathematik und Naturwissenschaften der Technischen Universit¨atBerlin zur Erlangung des akademischen Grades Doktor der Naturwissenschaften - Dr.rer.nat. - genehmigte Dissertation Promotionsausschuss: Vorsitzender: Prof. Dr. Peter Friz Gutachter: Prof. Dr. Bernd Sturmfels Gutachter: Prof. Dr. Christian Haase Gutachter: Prof. Dr. Reinhold Schneider Tag der wissenschaftlichen Aussprache: 15. November 2017 Berlin 2017 Acknowledgements First of all, there are no words to appropriately thank my advisor, Bernd Sturm- fels, for having given me this once-in-a-lifetime opportunity to be his student and having guided me with the best wisdom through this wonderful journey. It has been literally a dream come true to be able to learn from and to work with him. His passion for Math is unlimited, and an inexhaustible source of inspiration and admiration. I am also infinitely grateful to my co-advisor Christian Haase, for his continuous presence and consistent support, for welcoming me to and making me feel at home with Team Haase, and for being such an awesome and fun person. I would like to thank my brilliant collaborators: Mathias Drton, Jean-Charles Faug`ere,Kristian Ranestad, Alexander Engstr¨om,Jose Rodriguez and the whole Mathematics Research Communities Likelihood group, especially Serkan Ho¸sten. They are just the tip of a whole iceberg of wonderful mathematicians I have had the chance to meet across the world in many exciting workshops and conferences: Caroline Uhler, Piotr Zwiernik, Shaowei Lin, Seth Sullivant, Timo de Wolff, Peter B¨urgisser,Jonathan Hauenstein, Anton Leykin, Josephine Yu, Elizabeth Gross, Ang´elicaCueto, Diane Maclagan, Liam Solus, Jan Draisma, Emil Horobet, Rob Eggermont, Ralph Morrison, Bo Lin, Joe Kileel, Christiane G¨orgen,Eliana Duarte, Yue Ren, Jacinta Torres, Mario Kummer, Paul Breiding,... Thanks to my Einstein family: to the highly talented and friendly postdocs Laura Escobar, Fatemeh Mohammadi and Marta Panizzut; and to my academic sister Kathl´enKohn who has been there from the beginning. It is always a plea- sure to come to the office and find you there. I am grateful for the generous funding of the Einstein Foundation for making the whole project possible. In the same way, I am very grateful to Michael Joswig for accommodating us in the Discrete Mathematics/Geometry research group at TU Berlin. I thank each one of these colleagues and send `comrade wishes' to Georg Loho, Benjamin Schr¨oter and Andr´eWagner who have also their corresponding dissertation defense in this period. Special thanks to Antje Schulz; without her help, patience and always friendly smile, it would have been impossible to sort through all the necessary paperwork/regulations and able to focus on research. I would like to thank the Villa community at FU Berlin for the fun and interest- ing events (including DiscreTea), and specially my academic siblings Florian Kohl, Lena Walter and Jan Hofmann for uncountably many good times. I am grateful to the Berlin Mathematical School for being a very good promoter of Mathematics in Berlin and for the chance of being in such a friendly and international community. Many special thanks to Krishnan Mody, Andra Constantinescu, Arya Tafvizi, Josu´eTonelli Cueto, Anna Seigal, Elina Robeva, Kaie Kubjas, Corey and Debby Harris, and other amazing friends for all that uplifting encouragement in the end. Very heartfelt thanks to my family: Mom, Dad, Andrea (+ Celine) and my abuelitas, without you I would not have even got to the first step of this voyage. Your love made missing you a little more bearable. Last but not least, I am very grateful to Prof. Dr. Reinhold Schneider for agreeing to take the time to review this work and to Prof. Dr. Peter Friz for being the chair of the dissertation committee. Thank you all, and may the Force be with you. iv Abstract In this work we study the statistical models known as Gaussian mixtures from an algebraic point of view. First, we illustrate how algebraic techniques can be useful to address funda- mental questions on the shape of Gaussian mixture densities, namely the problem of determining the maximum number of modes a mixture of Gaussians can have, depending on the number of components and the dimension. We proceed to look at the statistical problem of estimation of the parameters of a Gaussian mixture. We present and compare the prominent methods of maximum likelihood estimation and moment matching, and from this study a fundamental difference in algebraic complexity is revealed between the two approaches. With the above statistical motivations, we introduce the algebraic objects that will permit us to obtain statistical inference results, mainly on the identifiability problem. These objects are Gaussian moment varieties and their corresponding secant varieties. We study them by asking for their dimension, for their degree and for the equations that define them. We provide many answers, conjectures and open questions in this direction. Finally, we explore further connections and analogues to algebraic geometry from commonly used submodels of statistical interest. We compare what we learn from the algebraic perspective to recent tensor decomposition methods in the ma- chine learning community. Throughout, we mention current research directions that continue the effort of including Gaussian densities and their mixtures in algebraic statistics. Zusammenfassung In dieser Arbeit studieren wir die statistischen Modelle, die als zusammengeset- zte Normalverteilungen oder auch als Gaußsche Mischverteilungen bekannt sind, vom algebraischen Standpunkt aus. Zun¨achst betrachten wir, wie algebraische Methoden dabei helfen k¨onnen,grund- legende Fragen ¨uber die Form der Dichte einer solchen Verteilung zu kl¨aren: Zum Beispiel wollen wir die Anzahl lokaler Maxima bestimmen, die eine Mischung von Gaußverteilungen, abh¨angigvon der Anzahl der Komponenten sowie der Dimen- sion, h¨ochstens haben kann. Anschließend besch¨aftigenwir uns mit der statistischen Fragestellung, wie man die Modellparameter (Erwartungswerte und Varianzen der Komponenten sowie deren Gewichte) bestimmen kann. Wir vergleichen die g¨angigenMethoden: die Maximum-Likelihood-Methode und die Momentenmethode. Dieser Vergleich of- fenbart einen grundlegenden Unterschied in der algebraischen Komplexit¨atzwis- chen den beiden Ans¨atzen. Motiviert durch diese statistischen Erkenntnisse f¨uhrenwir diejenigen algebrais- chen Objekte ein, die uns statistische Ergebnisse { haupts¨achlich zum Identifizier- barkeitsproblem { liefern: die Gaußschen Momentenvariet¨atenund deren Sekan- tenvariet¨aten. Wir untersuchen ihre Dimension sowie ihren Grad und die alge- braischen Gleichungen, die sie definieren. Wir beantworten viele dieser Fragen und formulieren weitere offene Fragen und Vermutungen. Schließlich erforschen wir Verbindungen und Analogien zur algebraischen Ge- ometrie aus g¨angigenUntermodellen, die von statistischer Relevanz sind. Wir vergleichen unsere Erkenntnisse aus der Perspektive der algebraischen Geometrie mit modernen Tensor-Zerlegungsmethoden aus dem Bereich des maschinellen Ler- nens. Wir verweisen durchgehend auf vielversprechende Forschungsfragen, die Nor- malverteilungen und deren Mischverteilungen in die Algebraische Statistik mitein- beziehen. Declaration of Authorship I, CARLOS AMENDOLA,´ declare that this thesis titled \Algebraic Statistics of Gaussian Mixtures" and the work presented in it are my own or based on joint work of my own with co-authors. Parts of this thesis are prepublished. • Chapter 2 is based on the joint paper Maximum Number of Modes of Gaus- sian Mixtures [AEH17] with Alexander Engstr¨om(Aalto University) and Christian Haase (FU Berlin). A preprint is available at arXiv:1702.05066. • Chapter 3 is based on the joint paper Maximum Likelihood Estimates for Gaussian Mixtures are Transcendental [ADS15] with Mathias Drton (UW Seattle) and Bernd Sturmfels (MPI Leipzig and UC Berkeley). It was pre- sented in Berlin at MACIS 2015 International Conference on Mathematical Aspects of Computer and Information Sciences and published as one of the revised selected papers. The final publication is available at Springer via http://dx.doi.org/10.1007/978-3-319-32859-1 • Chapters 4 and 5 are based on two joint papers: the article Moment Vari- eties of Gaussian Mixtures [AFS16] with Jean-Charles Faug`ere(INRIA) and Bernd Sturmfels published in the Journal of Algebraic Statistics (available online at http://dx.doi.org/10.18409/jas.v7i1.42) and the article Algebraic Identifiability of Gaussian Mixtures [ARS17] with Kristian Ranestad (Uni- versity of Oslo) and Bernd Sturmfels published in International Mathematics Research Notices (available online at: https://doi.org/10.1093/imrn/rnx090). Contents 1. Introduction 1 1.1. Pearson's Crabs: Algebraic Statistics in 1894 . 2 1.2. Fisher's Approach and Hill-Climbing . 4 1.3. Motivating Questions . 6 2. The Peaks of Mixture Densities 9 2.1. Background . 9 2.2. Examples and Conjecture . 12 2.3. Many Modes . 16 2.4. Not Too Many Modes . 17 2.5. Future Work . 21 3. Maximum Likelihood Estimation and Transcendence 23 3.1. In Search of the ML Degree . 23 3.2. Reaching Transcendence . 25 3.3. Many Critical Points . 30 3.4. Further Discussion . 34 4. Method of Moments and Moment Varieties 37 4.1. Moments and Cumulants . 37 4.2. The Pearson Polynomial . 40 4.3. Comparison to Maximum Likelihood . 44 4.4. Varieties of