Probabilistic Machine Learning for Circular Statistics Models and inference using the Multivariate Generalised von Mises distribution Alexandre Khae Wu Navarro Department of Engineering University of Cambridge This thesis is submitted for the degree of Doctor of Philosophy Magdalene College 2017 I would like to dedicate this work to all those that gave me support to overcome difficult moments. Declaration I hereby declare that except where specific reference is made to the work of others, the contents of this dissertation are original and have not been submitted in whole or in part for consideration for any other degree or qualification in this, or any other university. This dissertation is my own work and contains nothing which is the outcome of work done in collaboration with others, except as specified in the text and Acknowledgements. This dissertation contains fewer than 65,000 words including appendices, bibliography, footnotes, tables and equations and has fewer than 150 figures. Alexandre Khae Wu Navarro 2017 Acknowledgements This thesis would not be possible without the support of several people and organisations. Here I briefly note down some of the remarkable people whom I am utterly grateful for their role in this project. I would like to thank the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (Capes), the Cambridge Trusts and Sir Colin Corness Student Bursary, without which the research described in this thesis would have not been possible. I have also to thank Dr. Richard Turner for his instrumental role in the completion of this project through his insightful remarks, support and guidance. Another major contributor was Dr. Jes Frellsen, whom I had many instigating discussions that spurred our collaboration. Magdalene College and the Engineering Department also provided support for this thesis on multiple levels. In particular, my tutors Prof. Paul Dupree and Dr. Luke Skinner have been extremely helpful in times of unexpected hardships which I faced during the course of my studies. Cris Percival’s efficiency was also paramount for surpassing these obstacles life hurdled at me. I have also to thank Dr. Sae Franklin and prof. David Franklin for their invaluable advice, as well as Louise Segar and Diane Hazell for interesting conversations. Last, but definitely not least, I wish to thank my wife Manoela Carpenedo, my family and my friends for who stood by my side in the moments of joy and tears prompted by this project. Abstract Probabilistic machine learning and circular statistics—the branch of statistics concerned with data as angles and directions—are two research communities that have grown mostly in isolation from one another. On the one hand, probabilistic machine learning community has developed powerful frameworks for problems whose data lives on Euclidean spaces, such as Gaussian Processes, but have generally neglected other topologies studied by circular statistics. On the other hand, the approximate inference frameworks from probabilistic machine learning have only recently started to the circular statistics landscape. This thesis intends to redress the gap between these two fields by contributing to both fields with models and approximate inference algorithms. In particular, we introduce the multivariate Generalised von Mises distribution (mGvM), which allows the use of kernels in circular statistics akin to Gaussian Processes, and an augmented representation. These models account for a vast number of applications comprising both latent variable modelling and regression of circular data. Then, we propose methods to conduct approximate inference on these models. In particular, we investigate the use of Variational Inference, Expectation Propagation and Markov chain Monte Carlo methods. The variational inference route taken was a mean field approach to efficiently leverage the mGvM tractable conditionals and create a baseline for comparison with other methods. Then, an Expectation Propagation approach is presented drawing on the Expectation Consistent Framework for Ising models and connecting the approximations used to the augmented model presented. In the final MCMC chapter, efficient Gibbs and Hamiltonian Monte Carlo samplers are derived for the mGvM and the augmented model. Table of contents List of figures xv List of tables xxi I Introduction1 1 A primer on circular data and associated distributions3 1.1 Distributions over the unit circle . .4 1.1.1 Constructions based on the wrapping transformation . .5 1.1.2 Constructions based on polar transformations . .6 1.1.3 Other constructions in the literature . 11 1.2 Multivariate extensions . 12 1.2.1 Distributions over the surface of (hyper-)spheres . 12 1.2.2 Distributions over the surface of (hyper-)toruses . 15 1.2.3 A short note on other multivariate extensions of circular distri- butions . 17 1.2.4 Comment on the relationships between distributions on spheres, toruses and other manifolds . 17 1.3 Aims and contributions of the thesis . 19 II Models 23 2 The multivariate Generalised von Mises distribution 25 2.1 The mGvM model . 25 2.1.1 Generative view of the mGvM .................. 29 2.1.2 Related models . 30 2.1.3 Higher order extensions . 32 xii Table of contents 2.2 Relevant properties of the mGvM and TN for inference and modelling . 33 2.2.1 The mGvM and TN are maximum entropy distributions . 33 2.2.2 Conditional and marginal distributions . 35 2.2.3 Modes . 38 2.3 Modelling with the multivariate Generalised von Mises . 40 2.3.1 Relationship to other distributions: Posterior and approximations 41 2.3.2 Circular regression . 43 2.3.3 Latent variable modelling . 50 2.4 Related work . 55 2.5 Summary . 56 3 Augmented representations of the mGvM 59 3.1 Derivations for the mGvM ......................... 60 3.1.1 Exchangeablility and modelling . 60 3.2 Related work . 67 3.3 Augmented distributions for more general distributions on Stiefel Manifolds 68 3.4 Summary . 69 III Inference and learning 71 4 Variational free energy methods 73 4.1 Inference . 73 4.1.1 Variational Free Energy . 74 4.1.2 Mean field variational inference for the mGvM .......... 78 4.2 Experimental Results . 81 4.2.1 Circular regression . 81 4.2.2 Latent variable modelling . 83 4.3 Summary . 87 5 Expectation Propagation inference for the mGvM 89 5.1 Inference . 89 5.1.1 Introduction to EP and its connection to variational inference . 90 5.1.2 Lifted Expectation Propagation approximation for the mGvM . 92 5.2 Experimental results . 98 5.3 Summary . 101 Table of contents xiii 6 Approaches based on Markov chain Monte Carlo 103 6.1 Simulation and inference . 103 6.1.1 Gibbs sampling . 104 6.1.2 Hamiltonian Monte Carlo . 105 6.1.3 MCMC methods for the augmented representation . 107 6.1.4 Contrastive divergence learning . 108 6.2 Experiments . 109 6.2.1 Analysis of sampler behaviour . 110 6.2.2 Contrastive divergence learning for the mGvM .......... 112 6.3 Conclusions . 113 IV Conclusions 123 7 Conclusions and further directions 125 7.1 Conclusions arising from the thesis contributions . 125 7.2 Further work . 127 References 129 Appendix A Sparse models using the augmented representations 141 Appendix B Variational inference for the augmented mGvM represen- tation 143 Appendix C Improving numerical stability of the Generalised von Mises moment calculations 147 List of figures 1.1 Graphical summary of construction procedure for circular distributions from Euclidean distributions exemplified through the Gaussian distribu- tion circular counterparts. .5 1.2 Illustration of the construction of a wrapped distribution: the shaded regions in the distribution to the left represent 2π intervals with 2π multiples of an angle ϕ ∈ [0, 2π) and their respective probability (dashed coloured lines). A wrapped distribution, shown in the right figure, is constructed by condensing all the probability mass of the 2π multiples of ϕ from the distribution in the left figure to the point ϕ........6 1.3 Illustration of the genesis of projected distributions as marginalisation over a radial component. Here, the probability of ϕ is obtained by inte- grating over the ray forming an angle ϕ with the x1 axis and renormalising. 7 1.4 Graphical interpretation of the construction of an intrinsic distribution: the Generalised von Mises distribution is constructed by conditioning a two-dimensional distribution to the unit circle, and then expressing it through an angle. .9 1.5 Relationship between directional distributions based on conditioning a multivariate Gaussian to the unit circle: arrows show additional assumptions over the covariance matrix of the base Gaussian, extended from (Mardia, 1999)............................. 15 1.6 A diagram outlining the relationship between circular and directional distributions, their multivariate extensions and the underlying Gaussian’s covariance structure. A new distribution introduced in this thesis is highlighted in red.............................. 18 xvi List of figures 1.7 A diagram indicating the tractable and intractable distributions on the (hyper-)torus, (hyper-)spheres and Stiefel manifolds. By tractable we imply that numerical stable and efficient approximations exist for calculating the moments of the underlying distribution (this criterion excludes the infinite series approximation for the GvM, which are only accurate for low concentration values). The methods presented in this thesis provide ways to perform inference and learning in the intractable distributions by using the tractable ones. 19 2.1 Example of a two-dimensional mGvM distribution with four modes plotted in the [0, 2π) × [0, 2π) plane and as a torus. Darker tones denote high probability zones, while lighter tones indicate low-probability regions. 27 2.2 Example of a two-dimensional TN plotted in the [0, 2π) × [0, 2π) plane and a torus, a unimodal distribution which is a special case of the mGvM. Darker tones denote high probability zones, while lighter tones indicate low-probability regions. 31 2.3 Diagrams outlining how circular prior distributions be combined with different likelihoods yield a multivariate Generalised von Mises posterior.