DOCSLIB.ORG

Graph Convolutional Networks Yunsheng Bai Overview

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Graph Convolutional Networks Yunsheng Bai Overview Graph Convolutional Networks Yunsheng Bai Overview 1. Improve GCN itself a. Convolution i. Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering (NIPS 2016) ii. Semi-Supervised Classification with Graph Convolutional Networks (ICLR 2017) iii. Dynamic Filters in Graph Convolutional Networks (2017) b. Pooling (Unpooling) i. Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering (NIPS 2016) 2. Apply to new/larger datasets/graphs a. Use GCN as an auxiliary module i. Structured Sequence Modeling with Graph Convolutional Recurrent Networks (2017) b. Use GCN only i. Node/Link classification/prediction: http://tkipf.github.io/misc/GCNSlides.pdf 1. Directed graph a. Modeling Relational Data with Graph Convolutional Networks (2017) ii. Graph classification, e.g. MNIST (with or without pooling) Roadmap 1. Define Convolution for Graph 2. Architecture of Graph Convolutional Networks 3. Improvements: Generalizable Graph Convolutional Networks with Deconvolutional Layers a. Improvement 1: Dynamic Filters -> Generalizable b. Improvement 2: Deconvolution Define Convolution for Graph Laplace http://www.norbertwiener.umd.edu/Research/lectures/2014/MBegue_Prelim.pdf Graph Laplacian http://www.norbertwiener.umd.edu/Research/lectures/2014/MBegue_Prelim.pdf Graph Laplacian Labeled graph Degree matrix Adjacency matrix Laplacian matrix https://en.wikipedia.org/wiki/Laplacian_matrix Graph Fourier Transform L: (Normalized) Graph Laplacian D: Degree Matrix W: Adjacency Matrix U: Eigenvectors of L (Orthonormal b.c. L is symmetric PSD) Λ: Eigenvalues of L : Fourier Transform of x Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering (NIPS 2016) 1-D Convolution Ex. x = [0 1 2 3 4 5 6 7 8] f = [4 -1 0] y = [0*4 1*4+0*(-1) 2*4+1*(-1)+0*0 3*4+2*(-1)+1*0 ...] = [0 4 7 10 ...] I made this based on my EECS 351 lecture notes. Convolution <--> Multiplication in Fourier Domain View X and F as vectors I made this based on my EECS 351 lecture notes. Spectral Filtering ~ to the Convolution: previous slide Filter a signal: “As we cannot express a meaningful translation x(1) y(1) e1 operator in the vertex 1 0 0 (2) (2) e x domain, the convolution e1 e2 e3 0 2 0 2 = y operator on graph G is 0 0 3 e defined in the Fourier 3 x(3) y(3) domain” Inverse Fourier Non-parametric Fourier Transform Transform Filter of x Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering (NIPS 2016) Spectral Filtering x(1) e1 1 0 0 e x(2) e1 e2 e3 0 2 0 2 0 0 3 e 3 x(3) Inverse Fourier Non-parametric Fourier Transform Transform Filter of x e e *<e ,x> e1 + *<e ,x> 2 + *<e ,x> 3 = 1 1 2 2 3 3 Fourier Basis Spectral Filtering x(1) e1 1 0 0 e x(2) e1 e2 e3 0 2 0 2 0 0 3 e 3 x(3) Inverse Fourier Non-parametric Fourier Transform Transform Filter of x = e e e 1*<e1,x> 1 + 2*<e2,x> 2 + 3*<e3,x> 3 The result of the convolution is the original signal: (1) first Fourier Transformed (2) then multiplied by a filter (3) finally inverse Fourier Transformed Spectral Filtering Convolution: Filter a signal: x(1) y(1) e1 1 0 0 (2) (2) e x e1 e2 e3 0 2 0 2 = y 0 0 3 e 3 x(3) y(3) Inverse Fourier Non-parametric Fourier Transform Transform Filter of x Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering (NIPS 2016) Better Filters Convolution: Filter a signal: x(1) y(1) e1 1 0 0 (2) (2) e x e1 e2 e3 0 2 0 2 = y 0 0 3 e 3 x(3) y(3) Inverse Fourier Localized & Fourier Transform Transform Polynomial of x Filter Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering (NIPS 2016) Better Filters: Localized Labeled graph Degree matrix Adjacency matrix Laplacian matrix L2 Wavelets on Graphs via Spectral Graph Theory (2011) Better Filters: Localized Filter: Filter a signal: x(1) x(1) x(1) 1-step neighbors 2-step neighbors x(2) x(2) x(2) K=3 x(3) x(3) x(3) = Θ * + Θ * * + Θ *( )* 0 x(4) 1 x(4) 2 x(4) x(5) x(5) x(5) x(6) x(6) x(6) Fixed Θ for every neighbor :( (Dynamic Filters in Graph Convolutional Networks (2017)) Better Filters, but O(n2) Convolution: Filter a signal: Filter: s(1) Computing Eigenvectors s(2) e1 e2 e3 O(n3) :( s(3) I am actually confused. They used Chebyshev polynomials to approximate the filter, but at the end of the day, the filtered signal is the same as the previous slide. O(n2) :( In fact, authors of Semi-Supervised Classification with Graph Convolutional Networks (ICLR 2017) set K=1, so no Chebyshev at all. Approximations If K=1, filtering becomes: If set (further approximate): Then, filtering becomes: If input is a matrix: Then, filtering becomes: Filter parameters: Convolved signal matrix: Filtering complexity: Semi-Supervised Classification with Graph Convolutional Networks (ICLR 2017) Illustration of -1/2 -1/2 D * A * D * X * Feature 2 of Word Embedding of Node 1 F F Node 1 i i l l t t * * = L X e e Z r r Feature 2 of 1 2 Word Embedding of Node 6 Node 6 Feature 1 of Node 6 Semi-Supervised Classification with Graph Convolutional Networks (ICLR 2017) Roadmap 1. Define Convolution for Graph 2. Architecture of Graph Convolutional Networks 3. Improvements: Generalizable Graph Convolutional Networks with Deconvolutional Layers a. Improvement 1: Dynamic Filters -> Generalizable b. Improvement 2: Deconvolution Architecture of Graph Convolutional Networks Schematic Depiction Semi-Supervised Classification with Graph Convolutional Networks (ICLR 2017) Roadmap 1. Define Convolution for Graph 2. Architecture of Graph Convolutional Networks 3. Improvements: Generalizable Graph Convolutional Networks with Deconvolutional Layers a. Improvement 1: Dynamic Filters -> Generalizable b. Improvement 2: Deconvolution Improvements: Generalizable Graph Convolutional Networks with Deconvolutional Layers Roadmap 1. Define Convolution for Graph 2. Architecture of Graph Convolutional Networks 3. Improvements: Generalizable Graph Convolutional Networks with Deconvolutional Layers a. Improvement 1: Dynamic Filters -> Generalizable b. Improvement 2: Deconvolution Improvement 1 Dynamic Filters -> Generalizable Roadmap 1. Define Convolution for Graph 2. Architecture of Graph Convolutional Networks 3. Improvements: Generalizable Graph Convolutional Networks with Deconvolutional Layers a. Improvement 1: Dynamic Filters -> Generalizable i. Basics ii. Ideas iii. Ordering iv. Example b. Improvement 2: Deconvolution Baseline Filter: Semi-Supervised Classification with Graph Convolutional Networks (ICLR 2017) 0 12 0 0 0 12 ... 0 0 0 0 11 11 3 2 1 11 0 11 0 0 0 Filter 4 0 0 0 0 1 All share 11 11 the same 0 0 0 0 5 parameter. 11 11 6 11 11 11 11 0 0 0 11 11 11 11 0 Poor Filter: Parameters All Different (No Sharing) 0 1212 0 0 0 1216 ... 0 0 0 0 1112 1116 3 2 0 0 0 0 1 What if they 1121 1123 Filter 4 share? 1 1131 0 0 0 0 1136 0 0 0 0 5 1145 1145 6 1151 1152 1153 1154 0 0 What if they share? 0 1162 1163 1164 1165 0 Proposed Filter: Parameters Shared across Nodes with Same # of Neighbors 0 ’1221 0 0 0 ’1222 ... 0 ’ 0 0 0 ’ 1121 1122 3 2 1 ’1121 0 ’1122 0 0 0 Share the Filter same ’ 4 112 1 ’1121 0 0 0 0 ’1122 0 0 ’ 0 ’ 0 5 1121 1122 6 ’1141 ’1142 ’1143 ’1144 0 0 Share the same ’114 0 ’1141 ’1142 ’1143 ’1144 0 Proposed Filter: Total Size O(N2*F*C) ’1211 0 0 0 0 0 ... ’ 0 0 0 0 0 1111 3 2 1 ’1121 ’1122 0 0 0 0 Filter 1 4 ’ ’ ’ 0 0 0 (view N=6, 1131 1132 1133 without F=1, adjacency ’ ’ ’ ’ 0 0 5 C=2 1141 1142 1143 1143 info) 6 ’1151 ’1152 ’1153 ’1154 ’1155 0 ’1161 ’1162 ’1163 ’1164 ’1165 ’1166 2 2 Proposed Filter: Total Size O(nmax *F*C) <= O(N *F*C) 0 0 0 0 0 0 ... 0 0 0 0 0 0 3 2 1 ’1121 ’1122 0 0 0 0 Filter 1 4 N=6, 0 0 0 0 0 0 (view F=1, without adjacency C=2, ’ ’ ’ ’ 0 0 5 1141 1142 1143 1144 info) 6 nmax=4 0 0 0 0 0 0 0 0 0 0 0 0 Proposed Filter: Generalizable to Regular CNN 0 0 0 0 0 0 Moving Filter ... Stride == 1 0 0 0 0 0 0 0 0 0 0 0 ’ ’ 0 0 0 0 Filter 1121 1122 0 1 2 3 0 1 0 0 0 0 0 0 (view without 0 4 5 6 0 adjacency ’ ’ ’ ’ 0 0 1141 1142 1143 1144 info) 0 0 0 0 0 0 0 0 0 0 0 Regular 2-D image, Regular CNN 0 0 0 0 0 0 Proposed Filter: More Sharing of Weights 0 0 0 0 0 0 ... 0 0 0 0 0 0 3 2 1 ’1121 ’1122 0 0 0 0 Filter 1 4 Weights from 0 0 0 0 0 0 (view previous rows without adjacency are related to ’ ’ ’ ’ 0 0 5 later rows. 1141 1142 1143 1144 info) 6 0 0 0 0 0 0 0 0 0 0 0 0 Proposed Filter: More Sharing of Weights 0 0 0 0 0 0 ... 0 0 0 0 0 0 3 2 1 ’1121 ’1122 0 0 0 0 Filter 1 4 But copy or 0 0 0 0 0 0 (view other relations? without adjacency If copy, ’ ’ ’ ’ 0 0 5 randomly copy? 1141 1142 1143 1144 info) 6 0 0 0 0 0 0 0 0 0 0 0 0 Proposed Filter: Soft Assignment (Dynamic Filters in Graph Convolutional Networks (2017)) 0 ’1221 0 0 0 ’1222 ..
Load more

Recommended publications
  • Fast Spectral Approximation of Structured Graphs with Applications to Graph Filtering
    algorithms Article Fast Spectral Approximation of Structured Graphs with Applications to Graph Filtering Mario Coutino 1,* , Sundeep Prabhakar Chepuri 2, Takanori Maehara 3 and Geert Leus 1 1 Department of Microelectronics, Faculty of Electrical Engineering, Mathematics and Computer Science, Delft University of Technology, 2628 CD Delft , The Netherlands; [email protected] 2 Department of Electrical and Communication Engineering, Indian Institute of Science, Bangalore 560 012, India; [email protected] 3 RIKEN Center for Advanced Intelligence Project, Tokyo 103-0027, Japan; [email protected] * Correspondence: [email protected] Received: 5 July 2020; Accepted: 27 August 2020; Published: 31 August 2020 Abstract: To analyze and synthesize signals on networks or graphs, Fourier theory has been extended to irregular domains, leading to a so-called graph Fourier transform. Unfortunately, different from the traditional Fourier transform, each graph exhibits a different graph Fourier transform. Therefore to analyze the graph-frequency domain properties of a graph signal, the graph Fourier modes and graph frequencies must be computed for the graph under study. Although to find these graph frequencies and modes, a computationally expensive, or even prohibitive, eigendecomposition of the graph is required, there exist families of graphs that have properties that could be exploited for an approximate fast graph spectrum computation. In this work, we aim to identify these families and to provide a divide-and-conquer approach for computing an approximate spectral decomposition of the graph. Using the same decomposition, results on reducing the complexity of graph filtering are derived. These results provide an attempt to leverage the underlying topological properties of graphs in order to devise general computational models for graph signal processing.
    [Show full text]
  • Adaptive Graph Convolutional Neural Networks
    The Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18) Adaptive Graph Convolutional Neural Networks Ruoyu Li, Sheng Wang, Feiyun Zhu, Junzhou Huang∗ The University of Texas at Arlington, Arlington, TX 76019, USA Tencent AI Lab, Shenzhen, 518057, China Abstract 1 Graph Convolutional Neural Networks (Graph CNNs) are generalizations of classical CNNs to handle graph data such as molecular data, point could and social networks. Current filters in graph CNNs are built for fixed and shared graph structure. However, for most real data, the graph structures varies in both size and connectivity. The paper proposes a generalized and flexible graph CNN taking data of arbitrary graph structure as Figure 1: Example of graph-structured data: organic com- input. In that way a task-driven adaptive graph is learned for pound 3,4-Lutidine (C7H9N) and its graph structure. each graph data while training. To efficiently learn the graph, a distance metric learning is proposed. Extensive experiments on nine graph-structured datasets have demonstrated the superior performance improvement on both convergence speed and In some cases, e.g classification of point cloud, the topo- predictive accuracy. logical structure of graph is more informative than vertex feature. Unfortunately, the existing graph convolution can not thoroughly exploit the geometric property on graph due Introduction to the difficulty of designing a parameterized spatial kernel Although the Convolutional Neural Networks (CNNs) have matches a varying number of neighbors (Shuman et al. 2013). been proven supremely successful on a wide range of ma- Besides, considering the flexibility of graph and the scale of chine learning problems (Hinton et al.
    [Show full text]
  • Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering
    Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering Michaël Defferrard Xavier Bresson Pierre Vandergheynst EPFL, Lausanne, Switzerland {michael.defferrard,xavier.bresson,pierre.vandergheynst}@epfl.ch Abstract In this work, we are interested in generalizing convolutional neural networks (CNNs) from low-dimensional regular grids, where image, video and speech are represented, to high-dimensional irregular domains, such as social networks, brain connectomes or words’ embedding, represented by graphs. We present a formu- lation of CNNs in the context of spectral graph theory, which provides the nec- essary mathematical background and efficient numerical schemes to design fast localized convolutional filters on graphs. Importantly, the proposed technique of- fers the same linear computational complexity and constant learning complexity as classical CNNs, while being universal to any graph structure. Experiments on MNIST and 20NEWS demonstrate the ability of this novel deep learning system to learn local, stationary, and compositional features on graphs. 1 Introduction Convolutional neural networks [19] offer an efficient architecture to extract highly meaningful sta- tistical patterns in large-scale and high-dimensional datasets. The ability of CNNs to learn local stationary structures and compose them to form multi-scale hierarchical patterns has led to break- throughs in image, video, and sound recognition tasks [18]. Precisely, CNNs extract the local sta- tionarity property of the input data or signals by revealing local features that are shared across the data domain. These similar features are identified with localized convolutional filters or kernels, which are learned from the data. Convolutional filters are shift- or translation-invariant filters, mean- ing they are able to recognize identical features independently of their spatial locations.
    [Show full text]
  • Graph Convolutional Networks Using Heat Kernel for Semi-Supervised
    Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19) Graph Convolutional Networks using Heat Kernel for Semi-supervised Learning Bingbing Xu , Huawei Shen , Qi Cao , Keting Cen and Xueqi Cheng CAS Key Laboratory of Network Data Science and Technology, Institute of Computing Technology, Chinese Academy of Sciences School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, China fxubingbing, shenhuawei, caoqi, cenketing, [email protected] Abstract get node by its neighboring nodes. GraphSAGE [Hamilton et al., 2017] defines the weighting function as various aggrega- Graph convolutional networks gain remarkable tors over neighboring nodes. Graph attention network (GAT) success in semi-supervised learning on graph- proposes learning the weighting function via self-attention structured data. The key to graph-based semi- mechanism [Velickovic et al., 2017]. MoNet [Monti et al., supervised learning is capturing the smoothness 2017] offers us a general framework for designing spatial of labels or features over nodes exerted by graph methods. It takes convolution as a weighted average of multi- structure. Previous methods, spectral methods and ple weighting functions defined over neighboring nodes. One spatial methods, devote to defining graph con- open challenge for spatial methods is how to determine ap- volution as a weighted average over neighboring propriate neighborhood for target node when defining graph nodes, and then learn graph convolution kernels convolution. to leverage the smoothness to improve the per- formance of graph-based semi-supervised learning. Spectral methods define graph convolution via convolution One open challenge is how to determine appro- theorem. As the pioneering work of spectral methods, Spec- priate neighborhood that reflects relevant informa- tral CNN [Bruna et al., 2014] leverages graph Fourier trans- tion of smoothness manifested in graph structure.
    [Show full text]
  • GSPBOX: a Toolbox for Signal Processing on Graphs
    GSPBOX: A toolbox for signal processing on graphs Nathanael Perraudin, Johan Paratte, David Shuman, Lionel Martin Vassilis Kalofolias, Pierre Vandergheynst and David K. Hammond March 16, 2016 Abstract This document introduces the Graph Signal Processing Toolbox (GSPBox) a framework that can be used to tackle graph related problems with a signal processing approach. It explains the structure and the organization of this software. It also contains a general description of the important modules. 1 Toolbox organization In this document, we briefly describe the different modules available in the toolbox. For each of them, the main functions are briefly described. This chapter should help making the connection between the theoretical concepts introduced in [7, 9, 6] and the technical documentation provided with the toolbox. We highly recommend to read this document and the tutorial before using the toolbox. The documentation, the tutorials and other resources are available on-line1. The toolbox has first been implemented in MATLAB but a port to Python, called the PyGSP, has been made recently. As of the time of writing of this document, not all the functionalities have been ported to Python, but the main modules are already available. In the following, functions prefixed by [M]: refer to the MATLAB implementation and the ones prefixed with [P]: refer to the Python implementation. 1.1 General structure of the toolbox (MATLAB) The general design of the GSPBox focuses around the graph object [7], a MATLAB structure containing the necessary infor- mations to use most of the algorithms. By default, only a few attributes are available (see section 2), allowing only the use of a subset of functions.
    [Show full text]
  • Multi-Dimensional Graph Fourier Transform Takashi Kurokawa, Non-Member, IEEE, Taihei Oki, Non-Member, IEEE, and Hiromichi Nagao, Non-Member, IEEE
    1 Multi-dimensional Graph Fourier Transform Takashi Kurokawa, Non-Member, IEEE, Taihei Oki, Non-Member, IEEE, and Hiromichi Nagao, Non-Member, IEEE Abstract—Many signals on Cartesian product graphs appear in the real world, such as digital images, sensor observation time series, and movie ratings on Netflix. These signals are “multi- 0.3 dimensional” and have directional characteristics along each factor graph. However, the existing graph Fourier transform 0.2 does not distinguish these directions, and assigns 1-D spectra to signals on product graphs. Further, these spectra are often 0.1 multi-valued at some frequencies. Our main result is a multi- dimensional graph Fourier transform that solves such problems 0.0 associated with the conventional GFT. Using algebraic properties of Cartesian products, the proposed transform rearranges 1- 0.1 D spectra obtained by the conventional GFT into the multi- dimensional frequency domain, of which each dimension repre- 0.2 sents a directional frequency along each factor graph. Thus, the multi-dimensional graph Fourier transform enables directional 0.3 frequency analysis, in addition to frequency analysis with the conventional GFT. Moreover, this rearrangement resolves the (a) multi-valuedness of spectra in some cases. The multi-dimensional graph Fourier transform is a foundation of novel filterings and stationarities that utilize dimensional information of graph signals, which are also discussed in this study. The proposed 1 methods are applicable to a wide variety of data that can be 10 regarded as signals on Cartesian product graphs. This study 3 also notes that multivariate graph signals can be regarded as 2- 10 D univariate graph signals.
    [Show full text]
  • Graph Wavelet Neural Network
    Published as a conference paper at ICLR 2019 GRAPH WAVELET NEURAL NETWORK Bingbing Xu1,2 , Huawei Shen1,2, Qi Cao1,2, Yunqi Qiu1,2 & Xueqi Cheng1,2 1CAS Key Laboratory of Network Data Science and Technology, Institute of Computing Technology, Chinese Academy of Sciences; 2School of Computer and Control Engineering, University of Chinese Academy of Sciences Beijing, China fxubingbing,shenhuawei,caoqi,qiuyunqi,[email protected] ABSTRACT We present graph wavelet neural network (GWNN), a novel graph convolutional neural network (CNN), leveraging graph wavelet transform to address the short- comings of previous spectral graph CNN methods that depend on graph Fourier transform. Different from graph Fourier transform, graph wavelet transform can be obtained via a fast algorithm without requiring matrix eigendecomposition with high computational cost. Moreover, graph wavelets are sparse and localized in vertex domain, offering high efficiency and good interpretability for graph con- volution. The proposed GWNN significantly outperforms previous spectral graph CNNs in the task of graph-based semi-supervised classification on three bench- mark datasets: Cora, Citeseer and Pubmed. 1 INTRODUCTION Convolutional neural networks (CNNs) (LeCun et al., 1998) have been successfully used in many machine learning problems, such as image classification (He et al., 2016) and speech recogni- tion (Hinton et al., 2012), where there is an underlying Euclidean structure. The success of CNNs lies in their ability to leverage the statistical properties of Euclidean data, e.g., translation invariance. However, in many research areas, data are naturally located in a non-Euclidean space, with graph or network being one typical case. The non-Euclidean nature of graph is the main obstacle or chal- lenge when we attempt to generalize CNNs to graph.
    [Show full text]
  • Bridging Graph Signal Processing and Graph Neural Networks
    Bridging graph signal processing and graph neural networks Siheng Chen Mitsubishi Electric Research Laboratories Graph-structured data Data with explicit graph Social Internet of networks things Traffic Human brain networks networks 1 Graph-structured data graph-structured data graph structure Data with explicit graph Social Internet of networks things Traffic Human brain networks networks 1 Graph-structured data graph-structured data graph structure Data with explicit graph Data with implicit graph Social Internet of networks things 3D point cloud Action recognition Traffic Human brain Semi-supervised Recommender networks networks learning systems 1 Graph-structured data Data science toolbox Convolution data associated with data associated with regular graphs irregular graphs 2 Graph-structured data science A data-science toolbox to process and learn from data associated with large-scale, complex, irregular graphs 3 Graph-structured data science A data-science toolbox to process and learn from data associated with large-scale, complex, irregular graphs Graph signal processing (GSP) Graph neural network (GNN) • Extend classical signal processing • Extend deep learning techniques • Analytical model • Data-driven model • Theoretical guarantee • Empirical performance Graph filter bank Graph neural network 3 Sampling & recovery of graph-structured data GSP: Sampling & recovery Task: Approximate the original graph-structured data by exploiting information from its subset Look for data at each node 4 GSP: Sampling & recovery Task: Approximate
    [Show full text]
  • An Exposition of Spectral Graph Theory
    An Exposition of Spectral Graph Theory Mathematics 634: Harmonic Analysis David Nahmias [email protected] Abstract Recent developments have led to an interest to characterize signals on connected graphs to better understand their harmonic and geometric properties. This paper explores spectral graph theory and some of its applications. We show how the 1-D Graph Laplacian, DD, can be related to the Fourier Transform by its eigenfunctions which motivates a natural way of analyzing the spectrum of a graph. We begin by stating the Laplace-Beltrami Operator, D, in a general case on a Riemann manifold. The setting is then discretized to allow for formulation of the graph setting and the corresponding Graph Laplacian, DG. Further, an exposition on the Graph Fourier Transform is presented. Operations on the Graph Fourier Transform, including convolution, modulation, and translation are presented. Special attention is given to the equivalences and differences of the classical translation operation and the translation operation on graphs. Finally, an alternate formulation of the Graph Laplacian on meshes, DM, is presented, with application to multi-sensor signal analysis. December, 2017 Department of Mathematics, University of Maryland-College Park Spectral Graph Theory • David Nahmias 1. Background and Motivation lassical Fourier analysis has found many important results and applications in signal analysis. Traditionally, Fourier analysis is defined on continuous, R, or discrete, Z, domains. CRecent developments have led to an interest in analyzing signals which originate from sources that can be modeled as connected graphs. In this setting the domain also contains geometric information that may be useful in analysis. In this paper we analyze methods of Fourier analysis on graphs and results in this setting motivated by classical results.
    [Show full text]
  • Spectral Temporal Graph Neural Network for Multivariate Time-Series Forecasting
    Spectral Temporal Graph Neural Network for Multivariate Time-series Forecasting ∗y y Defu Cao1, , Yujing Wang1,2,, Juanyong Duan2, Ce Zhang3, Xia Zhu2 Conguri Huang2, Yunhai Tong1, Bixiong Xu2, Jing Bai2, Jie Tong2, Qi Zhang2 1Peking University 2Microsoft 3ETH Zürich {cdf, yujwang, yhtong}@pku.edu.cn [email protected] {juaduan, zhuxia, conhua, bix, jbai, jietong, qizhang}@microsoft.com Abstract Multivariate time-series forecasting plays a crucial role in many real-world ap- plications. It is a challenging problem as one needs to consider both intra-series temporal correlations and inter-series correlations simultaneously. Recently, there have been multiple works trying to capture both correlations, but most, if not all of them only capture temporal correlations in the time domain and resort to pre-defined priors as inter-series relationships. In this paper, we propose Spectral Temporal Graph Neural Network (StemGNN) to further improve the accuracy of multivariate time-series forecasting. StemGNN captures inter-series correlations and temporal dependencies jointly in the spectral domain. It combines Graph Fourier Transform (GFT) which models inter-series correlations and Discrete Fourier Transform (DFT) which models temporal de- pendencies in an end-to-end framework. After passing through GFT and DFT, the spectral representations hold clear patterns and can be predicted effectively by convolution and sequential learning modules. Moreover, StemGNN learns inter- series correlations automatically from the data without using pre-defined priors. We conduct extensive experiments on ten real-world datasets to demonstrate the effectiveness of StemGNN. 1 Introduction Time-series forecasting plays a crucial role in various real-world scenarios, such as traffic forecasting, supply chain management and financial investment.
    [Show full text]
  • Filtering and Sampling Graph Signals, and Its Application to Compressive Spectral Clustering
    Filtering and Sampling Graph Signals, and its Application to Compressive Spectral Clustering Nicolas Tremblay(1;2); Gilles Puy(1); R´emiGribonval(1); Pierre Vandergheynst(1;2) (1) PANAMA Team, INRIA Rennes, France (2) Signal Processing Laboratory 2, EPFL, Switzerland Introduction to GSP Graph sampling Application to clustering Conclusion Why graph signal processing ? N. Tremblay Compressive Spectral Clustering Gdr ISIS, 17th of June 2016 1 / 47 Introduction to GSP Graph sampling Application to clustering Conclusion Introduction to GSP Graph Fourier Transform Graph filtering Graph sampling Application to clustering What is Spectral Clustering ? Compressive Spectral Clustering A toy experiment Experiments on the SBM Conclusion N. Tremblay Compressive Spectral Clustering Gdr ISIS, 17th of June 2016 2 / 47 Introduction to GSP Graph sampling Application to clustering Conclusion Introduction to graph signal processing : graph Fourier transform N. Tremblay Compressive Spectral Clustering Gdr ISIS, 17th of June 2016 3 / 47 Introduction to GSP Graph sampling Application to clustering Conclusion What's a graph signal ? N. Tremblay Compressive Spectral Clustering Gdr ISIS, 17th of June 2016 4 / 47 Introduction to GSP Graph sampling Application to clustering Conclusion Three useful matrices The adjacency matrix : The degree matrix : 20 1 1 03 22 0 0 03 61 0 1 17 60 3 0 07 W = 6 7 S = 6 7 41 1 0 05 40 0 2 05 0 1 0 0 0 0 0 1 The Laplacian matrix : 2 2 −1 −1 0 3 6−1 3 −1 −17 L = S − W = 6 7 4−1 −1 2 0 5 0 −1 0 1 N. Tremblay Compressive Spectral Clustering Gdr ISIS, 17th of June 2016 5 / 47 Introduction to GSP Graph sampling Application to clustering Conclusion Three useful matrices The adjacency matrix : The degree matrix : 2 0 :5 :5 03 21 0 0 03 6:5 0 :5 47 60 5 0 07 W = 6 7 S = 6 7 4:5 :5 0 05 40 0 1 05 0 4 0 0 0 0 0 4 The Laplacian matrix : 2 1 −:5 −:5 0 3 6−:5 5 −:5 −47 L = S − W = 6 7 4−:5 −:5 1 0 5 0 −4 0 4 N.
    [Show full text]
  • Graph Based Convolutional Neural Network
    EDWARDS, XIE: GRAPH CONVOLUTIONAL NEURAL NETWORK 1 Graph Based Convolutional Neural Network Michael Edwards Swansea University Swansea, UK Xianghua Xie http://www.csvision.swan.ac.uk Abstract The benefit of localized features within the regular domain has given rise to the use of Convolutional Neural Networks (CNNs) in machine learning, with great proficiency in the image classification. The use of CNNs becomes problematic within the irregular spatial domain due to design and convolution of a kernel filter being non-trivial. One so- lution to this problem is to utilize graph signal processing techniques and the convolution theorem to perform convolutions on the graph of the irregular domain to obtain feature map responses to learnt filters. We propose graph convolution and pooling operators analogous to those in the regular domain. We also provide gradient calculations on the input data and spectral filters, which allow for the deep learning of an irregular spatial do- main problem. Signal filters take the form of spectral multipliers, applying convolution in the graph spectral domain. Applying smooth multipliers results in localized convo- lutions in the spatial domain, with smoother multipliers providing sharper feature maps. Algebraic Multigrid is presented as a graph pooling method, reducing the resolution of the graph through agglomeration of nodes between layers of the network. Evaluation of performance on the MNIST digit classification problem in both the regular and irregu- lar domain is presented, with comparison drawn to standard CNN. The proposed graph CNN provides a deep learning method for the irregular domains present in the machine learning community, obtaining 94.23% on the regular grid, and 94.96% on a spatially irregular subsampled MNIST.
    [Show full text]