Towards Learning Powerful Deep Graph Neural Networks and Embeddings A DISSERTATION SUBMITTED TO THE FACULTY OF THE GRADUATE SCHOOL OF THE UNIVERSITY OF MINNESOTA BY Saurabh Verma IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF Doctor of Philosophy Professor Zhi-Li Zhang, Advisor June, 2020 © Saurabh Verma 2020 ALL RIGHTS RESERVED Acknowledgements Started in the year 2008, it has been such a fun and exciting journey. Looking back now, I see myself have grown so much and have learned so many enjoyable life lessons. This happy journey have not been possible without the great mentors and good friends of my life. I am one of the lucky person who is blessed with great mentors. Many of my life’s prestigious moments and this thesis have not been possible without my advisor, Zhi-Li Zhang. Undoubtedly, he is one of the great advisor who not only uplifted my research skills but always inspired me to do great research work. He is also one of the most enthusiastic people, I ever met and can be seen from his passion for reading all kind of mathematic books and that too just for fun. Freedom to purse your wild ideas and truly caring about his student growth are his most admiring qualities. On several occasions, he went above and beyond to help me both academically and personally for which I’ll always be grateful to him. Another great mentor and inspiration of my life is Estevam Hruschka who first intro- duced me to world of research. He is the first person who showed me that the research is fun and worked with me to publish my first paper. Without him, I wouldn’t be here. Some mentors who will always have a special place in my life: Prabhat, Negi and Rakesh Sir; Hrishikesh Sharma; Jun and Raj; Aude, Nima and Shaili; Saurabh, Subhojit and Xia. A big thanks goes to my thesis committee members who not only helped me in polishing i this thesis but also played a major role in boosting my research career in lot of ways. In particular, I was lucky to meet Prof. Jaideep Srivastava who provided his early guidance and helped me to lay a strong foundation of my research career from the very beginning. Prof. Georgios Giannakis’s deep knowledge of the field inspired me to work hard and keep exploring the field. Prof. Abhishek Chandra’s thought provoking questions helped me realized the big picture. The other half part of this journey is made memorable by my dear friends. Old friends who are there from the very beginning: Saurabh, Rohit, Arpan, Baba, JP, Prolok, Prithvi, Harish, Chacha, Ankit, Tatu, Kanjad, Jade, Bhaiaya, PCC, Soni, Deval, GG, Captain, kunu, Ravi, Hema, Junglee, plus lot many Tronix friends. New friends who cheered me and kept the fun going: Guru, Anurag, Shalini, Taluk, Arvind, Pariya, Cheng, Braulio, Yang, Golshan, Hesham, Xinyue, Avinash and Malina. Lastly, I would like to acknowledge the grants that supported my research: NSF grants CNS 1618339, CNS 1617729, CNS1814322, CNS183677 and US DoD DTRA DTRA grant HDTRA1-09-1-0050 and HDTRA1-14-1-0040 and ARO MURI Award W911NF- 12-1-0385. ii Dedication To my loving parents, brother & sister for always believing and supporting me. iii Abstract Learning powerful data embeddings has recently become the core of machine learning algorithms especially in natural language processing and computer vision domains. In the graph domain, the applications of learning graph embeddings are vast and have distinguished use-cases across multi-cross domains such as bioinformatics, chemoinfor- matics, social networks and recommendation systems. To date, graph remains the most fundamental data structure that can represent many forms of real-world datasets. How- ever, due to its rich but complex data structure, graph presents a significant challenge in forging powerful graph embeddings. Even standard deep learning techniques such as Recurrent Neural Networks (RNNs) or Convolutional Neural Networks (CNNs) are not capable enough to operate on the data lying beyond 1D sequence of say words or 2D pixel-grid of images and therefore, cannot generalize to arbitrary graph structure. Re- cently, Graph Neural Networks (GNNs) have been proposed to alleviate such limitations but the current state is far from being mature in both theory and applications. To that end, this thesis aims at developing powerful graph embedding models for solv- ing wide-variety of real-world problems on the graph. We study some of the major ap- proaches for devising graph embedding namely Graph Kernel Or Spectrum and GNN. We expose and tackle some of their fundamental weakness and contribute several novel state-of-the-art graph embedding models. These models can achieve superior perfor- mance in solving many real-world problems on graphs such as node classification, graph classification or link prediction over existing methods and that too comes with desirable theoretical guarantees. We first study the capabilities of Graph Kernel or Spectrum ap- proaches toward yielding powerful graph embeddings in terms of uniqueness, stability, sparsity and computationally efficiency. Second, we propose Graph Capsule Neural Net- work that can yield powerful graph embeddings by capturing much more information encoded in the graph structure in comparison with existing GNNs. Third, we devise a first ever universal and transferable GNN and thus, makes transfer learning possi- ble in graph domain. Specifically with this particular GNN, graph embeddings canbe iv shared and transfered across different models and domains, reaping the huge benefits of transfer learning. Lastly, there is a dearth of theoretical explorations of GNN models such as their generalization properties. We take the first step towards developing a deeper theoretical understanding of GNN models by analyzing their stability and deriv- ing their generalization guarantees. To the best of our knowledge, we are the first to study stability bounds on graph learning in a semi-supervised setting and derive related generalization bounds for GNN models. In summary, this thesis contributes several state-of-the-art graph embeddings and novel graph theory, specifically (i) Powerful Graph Embedding called Family of Graph Spec- tral Distances (Fgsd) (ii) Highly Informative GNN Called Graph Capsule Neural Net- work (GCAPS) (iii) Universal and Transferable GNN called Deep Universal and Trans- ferable Graph Neural Network (Dugnn) (iv) Stability Theory and Generalization Guar- antees of GNN. v Contents Acknowledgements i Dedication iii Abstract iv List of Tables xi List of Figures xiv 1 Introduction 1 1.1 Core of Machine Learning: Data Embeddings ............... 1 1.2 Thesis Statement ............................... 2 1.3 Thesis Outline and Original Contributions ................. 2 1.4 Bibliographic Notes .............................. 4 2 Background and Motivation 6 vi 2.1 Background .................................. 6 2.1.1 Learning Powerful Data Embeddings ................ 6 2.1.2 Standard Deep Learning Techniques ................ 8 2.1.3 Graph Neural Networks ....................... 9 2.1.4 Graph Kernels ............................ 11 2.2 Motivation .................................. 11 2.2.1 Limitation of Existing Graph Embedding Models ......... 12 3 Learning Unique, Stable, Sparse and Computationally Fast Graph Em- beddings 14 3.1 Introduction .................................. 14 3.2 Our Graph Spectrum Approach ....................... 15 3.3 Related Work ................................. 17 3.4 Family of Graph Spectral Distances and Graph Spectrum ........ 19 3.5 Uniqueness of Family of Graph Spectral Distances and Embeddings .. 21 3.6 Unifying Relationship Between FGSD and Graph Embedding and Dimen- sion Reduction ................................ 23 3.7 Stability of Family of Graph Spectral Distances and Embeddings .... 24 3.8 Sparsity of Family of Graph Spectral Distances and Embeddings .... 26 3.9 Fast Computation of Family of Graph Spectral Distances and Embed- dings ..................................... 28 3.10 Experiments and Results ........................... 30 vii 3.11 Conclusion .................................. 34 4 Learning Highly Informative Graph Embeddings With Graph Capsule Neural Networks 35 4.1 Introduction .................................. 35 4.2 Related Work ................................. 39 4.3 Graph Capsule CNN Model ......................... 40 4.4 Graph Capsule Networks ........................... 42 4.5 Designing Graph Permutation Invariant Layer ............... 46 4.5.1 Problems with Max-Sort Pooling Layer .............. 46 4.5.2 Covariance as Permutation Invariant Layer ............ 47 4.6 Designing GCAP-CNN with Global Features ............... 48 4.7 Experiment and Results ........................... 51 4.8 Conclusion .................................. 55 5 Learning Universal and Transferable Graph Neural Network Embed- dings 57 5.1 Introduction .................................. 57 5.2 Related Work ................................. 60 5.2.1 Input Layer .............................. 63 5.2.2 Universal Graph Encoder ...................... 65 5.2.3 Multi-Task Graph Decoder ..................... 69 viii 5.3 Experiment and Results ........................... 71 5.4 Ablation Studies and Discussion ...................... 74 5.5 Conclusions .................................. 78 6 Stability and Generalization Guarantees of Graph Neural Networks 79 6.1 Introduction .................................. 79 6.2 Related Work ................................. 82 6.3 Graph Capsule & Graph Convolution Neural Networks .......... 84 6.4 Main Result .................................