Integral Geometry, Hamiltonian Dynamics, and Markov Chain Monte Carlo by MASS ACHUSES INS ITUTE OF TECHNOLOGY Oren Mangoubi JUN 16 2016 B.S., Yale University (2011) LIBRARIES Submitted to the Department of Mathematics MCHVES in partial fulfillment of the requirements for the degree of Doctor of Philosophy at the MASSACHUSETTS INSTITUTE OF TECHNOLOGY June 2016 @ Oren Mangoubi, MMXVI. All rights reserved. The author hereby grants to MIT permission to reproduce and to distribute publicly paper and electronic copies of this thesis document in whole or in part in any medium now known or hereafter created. AuthorSignature redacted .................. C/ Department of Mathematics April 28, 2016 Certified by. Signature redacted Alan Edelman Professor Thesis Supervisor Accepted bySignature redacted Jonathan Kelner Chairman, Applied Mathematics Committee 2 Integral Geometry, Hamiltonian Dynamics, and Markov Chain Monte Carlo by Oren Mangoubi Submitted to the Department of Mathematics on April 28, 2016, in partial fulfillment of the requirements for the degree of Doctor of Philosophy Abstract This thesis presents applications of differential geometry and graph theory to the design and analysis of Markov chain Monte Carlo (MCMC) algorithms. MCMC al- gorithms are used to generate samples from an arbitrary probability density ir in computationally demanding situations, since their mixing times need not grow expo- nentially with the dimension of w. However, if w has many modes, MCMC algorithms may still have very long mixing times. It is therefore crucial to understand and reduce MCMC mixing times, and there is currently a need for global mixing time bounds as well as algorithms that mix quickly for multi-modal densities. In the Gibbs sampling MCMC algorithm, the variance in the size of modes inter- sected by the algorithm's search-subspaces can grow exponentially in the dimension, greatly increasing the mixing time. We use integral geometry, together with the Hes- sian of r and the Chern-Gauss-Bonnet theorem, to correct these distortions and avoid this exponential increase in the mixing time. Towards this end, we prove a general- ization of the classical Crofton's formula in integral geometry that can allow one to greatly reduce the variance of Crofton's formula without introducing a bias. Hamiltonian Monte Carlo (HMC) algorithms are some the most widely-used MCMC algorithms. We use the symplectic properties of Hamiltonians to prove global Cheeger- type lower bounds for the mixing times of HMC algorithms, including Riemannian Manifold HMC as well as No-U-Turn HMC, the workhorse of the popular Bayesian software package Stan. One consequence of our work is the impossibility of energy- conserving Hamiltonian Markov chains to search for far-apart sub-Gaussian modes in polynomial time. We then prove another generalization of Crofton's formula that ap- plies to Hamiltonian trajectories, and use our generalized Crofton formula to improve the convergence speed of HMC-based integration on manifolds. We also present a generalization of the Hopf fibration acting on arbitrary- ghost- valued random variables. For # = 4, the geometry of the Hopf fibration is encoded by the quaternions; we investigate the extent to which the elegant properties of this encoding are preserved when one replaces quaternions with general 0 > 0 ghosts. 3 Thesis Supervisor: Alan Edelman Title: Professor 4 Acknowledgments I am very grateful to my advisor and coauthor Alan Edelman' for his guidance and collaboration on this thesis. I am also deeply grateful to my coauthor Natesh Pillai2 for his collaboration and advice on the Hamiltonian mixing times chapter of this thesis. I could not have finished this thesis without their insights. I am deeply thankful as well for indispensable advice and insights from Aaron Smith3 , Youssef Marzouk4 , Michael Betancourt5 , Jonathan Kelner1 , Michael La Croixi, Jiahao Chen', Laurent Demanet', Dennis Amelunxen', Ofer Zeitouni' 8 , Neil Shephard2 , and Nawaf Bou-Rabee'. I would also like to thank my mentors and previous coauthors Stephen Morse'o, Yakar Kannai7 , Edwin Marengo", and Lucio Frydman"2 . I am very grateful to my other mentors and professors at MIT and Yale, especially Roger Howe1", Gregory Margulis13 , Victor Chernozhukov1 4 , Kumpati Narendra'0 , Andrew Barron 15, Ivan Marcus16 , Paulo Lozano 4, and Manuel Martinez-Sanchez'. For valuable opportuni- ties to learn, teach and conduct research, I would like to thank the MIT Mathematics department and the Theory of Computation group at the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL), as well as the Yale Mathematics and Elec- trical Engineering departments, the Weizmann Institute Mathematics and Chemical Physics departments, and the Northeastern Electrical Engineering department. I am very thankful to have been blessed with a kind and loving family for who's 'MIT Mathematics Department 2 Harvard Statistics Department 3University of Ottawa Mathematics and Statistics Department 4 MIT Department of Aeronautics and Astronautics 5University of Warwick Statistics Department 6 City University of Hong Kong Mathematics Department 7Weizmann Institute of Science Mathematics Department 8 Courant Institute of Mathematical Sciences at NYU 9Rutgers Mathematical Sciences Department 0Yale Electrical Engineering Department "Northeastern University Electrical Engineering Department 12Weizmann Institute of Science Chemical Physics Department 13Yale Mathematics Department 1 4MIT Economics Department "5 Yale Statistics Department ' 6 Yale History Department 5 encouragement and support I am forever grateful: My mother and father, my brothers Tomer and Daniel, and, most importantly, my grandparents M6m6, Oma and Opa, as well as P6p6 (of blessed memory). I am also very thankful to my friends for their kindness and companionship. My schoolteachers at Schechter and Gann, especially my Mathematics teacher Mrs. Voolich and my Science teacher Mrs. Schreiber, have been an inspiration to me as well. I also thank the MITxplore program for giving me the opportunity to design and teach weekly Mathematics enrichment classes for children in Cambridge and Boston public schools. I deeply appreciate the generous support of a National Defense Science and Engi- neering Graduate (NDSEG) Fellowship, as well as support from the National Science Foundation (NSF DMS-1312831) and the MIT Mathematics Department. Thesis Committee: " Professor Alan Edelman Thesis Committee Chairman and Thesis Advisor Professor of Applied Mathematics, MIT " Professor Natesh Pillai Associate Professor of Statistics, Harvard * Professor Youssef Marzouk Associate Professor of Aeronautics and Astronautics, MIT " Professor Jonathan Kelner Associate Professor of Applied Mathematics, MIT 6 Contents I Introductih 9 1.1 Somic wid liy-u -&( 1\CL C ig1(1 uI .. .. .. .. ..... ... 10 L 1. 1 RandomN Walk Metropoi .......... .......... 10 1.1.2 Gibbs sampling algoritin . ........ ........... 10 1.1.3 Hamiltonian Monte Carlo ....... ....... 12 1.2 Iiitegral & differential geometrY prelimninari . ....... ...... 14 neas... .. .. .. .. .. .. .. ...15 1.2. Kine a Wi m 1.2.2 The Crofton for . ......... 17 1.2.3 Concentration. ....... ............. 17 1.2.4 The Chern-GaussR nnm. .... ....... ... 18 1.3 Conitribution-, of this the>.......................1 1.3 ribCon ti ns f t is h ............ ............. 19 2 Integral Geometry for Gibw Sa-mmers 21 .21 2.1 Illt roductio .. ...... ...... ...... ...... ...... 2.2 A first-ordei ewtn (iiin7 ie I eiuion 1>...... ...... ..... 28 2.2.1 The Crofton formula Gibbs sample ..... ..... .... 29 2.2.2 Traditional weights vs. integral geometry weigmt ... .... 30 2.3 A generalized Crofton forini ............ ........... 31 2.3.1 The generalized Crofton formula Gibbs saimp ... ...... 44 ~(K~'~; KUi 2.3.2 The pe-- j densite ................. ............ 46 2.3.3 An NCIm n aie( awo t e e ...... ..................... ... 4 7 7 -1 2.3.5 Higliei-or( hi' Cv~veril-Lill .. -u e .(W.1.j .t.1. 50 2. 3. 6 Colle ctioni-of-sphieres xaiilIe aii( oietatidm-)-miii 51 2.3.7 Vairianiicc due to )1r(l-efltYklma(nisue(( 2..> lieoiet a; )t )oul1(s (Ieive( usin- 1C C .i en 31 i t-O)raj( oiitY....... .. ...... .. ...... .. ...... .. ..... .. .. .... 60 2.4 Ranidoiii iilatrix a-pplicattionl: Sa111pf)lig 0t lestocia'st a 62 2.4.1 Approxiinate sanljplilig alo-ov)Y1+2 1!?~Cr 2...........63 2.5 Conditioinlg oni iiiltiple eigenivall( .. ...... ..... ....... 65 2.(6 Coniditioilling on- ai singp.le-eigeinvalti ue raI' er.. ...... ....... 66 3 Mixing Times of H-amiitouia-i 1\Iloit~e Carlc 71i 3.1 Itroducti . .... ..... ...... ..... ..... ........ 71 3.2 Haiifltoia 1, AUI I,- I .................................. .. .. .. ....72 3.3 Clieger- bo)01 s111 in X!ItLCJC > ......................... 78 :3.4 Daindoiii \\itlIK iiiip1 rnn u&'..... ..... .. .. .. .. .. .... 80 4 A Generalization of Crofton's Formula to H ~r'i with Applictn~ii ~ atho IV~Ionte c>'a,2. 85 4.2 Cr-oftoni foiiuime 101- liiniironin uIvnanmII-. ................ 85 4.3 Mnhuifoli integrationi usinig HNWC anil the fllii 1111 ( irotti Oi bul 88 5 A Hopf Fibration for ' -Ghost Gaiissiau- 91 5.1 hit-odlUCti(o .. ... ..... ...... ..... ..... ........ 91 5.2 Defiingi th .......... .......................... 92 5.3 Hopf Fibrationi oni'.. .... ..... ..... ...... ..... ... 94 8 Chapter 1 Introduction Applications of sampling on probability distributions, defined on Euclidean space or on other manifolds, arise in many fields, such as Statistics [18, 6, 24], Machine Learning [4], Statistical Mechanics