Graph Structure of Neural Networks Jiaxuan You1, Jure Leskovec1, Kaiming He2, Saining Xie2 Stanford University1 Facebook AI Research2 1 Neural networks consist of neurons We know connections between neurons affect NN performance But how? J. You, J. Leskovec, K. He, S. Xie, Graph Structure of Neural Networks, ICML 2020 2 ? ⟺ Underneath a NN, there is a graph We want to find a proper graph representation of NN to answer: Is there a link between the graph structure and NN performance? If so, what are structural signatures of well-performing NNs? Can these signatures generalize across tasks and datasets? … J. You, J. Leskovec, K. He, S. Xie, Graph Structure of Neural Networks, ICML 2020 3 Overview: Methodology § A novel representation of neural networks: relational graphs 1 2 ⟺ 4 3 § Relational graphs can represent diverse neural architectures 1 2 MLP layer 1 Translate Translate 3x3 conv to MLP to ResNet MLP layer 2 3x3 conv 4 3 … … § Tools from network science à Graph structure vs NN performance WS-flex graph WS-flex graph Translate to 5-layer MLP Complete graph J. You, J. Leskovec, K. He, S. Xie, Graph Structure of Neural Networks, ICML 2020 4 Overview: Key Findings § Consistent “Sweet Spot” for top NNs across architectures 5-layer MLP on CIFAR-10 8-layer CNN on ImageNet ResNet-34 on ImageNet Complete graph: Complete graph: Complete graph: 33.34 ± 0.36 48.27 ± 0.14 25.73 ± 0.07 Best graph: Best graph: Best graph: • Graphs with certain structure 32.05 ± 0.14 46.73 ± 0.16 24.57 ± 0.10 measures consistently performs well (controlling computational budgets) § Top artificial NNs are similar to real biological NNs • Graph structure of the best 5-layer MLP we found, is similar to the macaque whole cortex network J. You, J. Leskovec, K. He, S. Xie, Graph Structure of Neural Networks, ICML 2020 5 Overview: Methodology § A novel representation of neural networks: relational graphs ß § Computational vs. Relational graphs § Neural computation as message exchange on relational graphs 1 2 ⟺ 4 3 § Relational graphs can represent diverse neural architectures § Tools from network science à Graph structure vs NN performance J. You, J. Leskovec, K. He, S. Xie, Graph Structure of Neural Networks, ICML 2020 6 Background: NNs as Computational Graphs MLP model Computational graphs MLP layer 1 layer Computational 1 2 3 4 MLP layer graphs MLP layer MLP layer 1 2 3 4 … J. You, J. Leskovec, K. He, S. Xie, Graph Structure of Neural Networks, ICML 2020 7 Background: Related Work Existing graph-based architecture design approaches focus on computational graphs Deep Expander Networks, Prabhu et al., 2018 RandWire, Xie et al., 2019 NAS-Bench-101, Ying et al., 2019 Generate bipartite Generate directed graphs over neurons acyclic graphs over NN layers J. You, J. Leskovec, K. He, S. Xie, Graph Structure of Neural Networks, ICML 2020 8 Limitations: NNs as Computational Graphs Limitations: • Lack of flexibility MLP model • Directed acyclic graphs Computational graphs • Disconnection with neuroscience MLP layer 1 layer • Brain networks have flexible structure Computational 1 2 3 4 • Bi-directional information exchange MLP layer graphs MLP layer MLP layer 1 2 3 4 … J. You, J. Leskovec, K. He, S. Xie, Graph Structure of Neural Networks, ICML 2020 9 Our Approach: Relational Graphs Relational Graphs 1 round of message exchange Relational graph definition: • Nodes are neurons 1 2 • Edges specify (undirected) connectivity between neurons; • Computation is conducted by message exchange over the graph structure, where a 4 3 node exchange messages with its neighbors J. You, J. Leskovec, K. He, S. Xie, Graph Structure of Neural Networks, ICML 2020 10 Our approach: Relational Graphs Computational graphs Relational Graphs 1 layer 1 round of message exchange Relational graph definition: • Nodes are neurons 1 2 3 4 1 2 • Edges specify (undirected) connectivity between neurons; ⟺ • Computation is conducted by message exchange over the graph structure, where a 4 3 node exchange messages with its neighbors 1 2 3 4 J. You, J. Leskovec, K. He, S. Xie, Graph Structure of Neural Networks, ICML 2020 11 Our approach: Relational Graphs Computational graphs Relational Graphs 1 layer 1 round of message exchange 1 2 3 4 1 2 ⟺ 4 3 1 2 3 4 J. You, J. Leskovec, K. He, S. Xie, Graph Structure of Neural Networks, ICML 2020 12 Our Approach: Relational Graphs Computational graphs Relational Graphs 1 layer 1 round of message exchange 1 2 3 4 1 2 ⟺ 4 3 1 2 3 4 J. You, J. Leskovec, K. He, S. Xie, Graph Structure of Neural Networks, ICML 2020 13 Benefits of Relational Graphs Benefits: • Flexibility Computational graphs Relational Graphs • No restrictions on graph structure 1 layer 1 round of message exchange (we focus on undirected graph) 1 2 3 4 • Connections with neuroscience 1 2 • Bi-directional information exchange ⟺ Biological neural network: Artificial neural network: 4 3 Macaque whole cortex Best 5-layer MLP 1 2 3 4 J. You, J. Leskovec, K. He, S. Xie, Graph Structure of Neural Networks, ICML 2020 14 Diverse Relational Graphs Computational graphs Relational Graphs Computational graphs Relational Graphs 1 layer 1 round of message exchange 1 layer 1 round of message exchange 1 2 3 4 1 2 3 4 1 2 1 2 ⟺ ⟺ 4 3 4 3 1 2 3 4 1 2 3 4 J. You, J. Leskovec, K. He, S. Xie, Graph Structure of Neural Networks, ICML 2020 15 Neural Computation as Message Exchange Computational graphs Relational graphs Directed message flow Bi-Directed message exchange 1 2 3 4 ���(⋅) ���(⋅) 1 2 ⟺ 4 3 1 2 3 4 ���(⋅) ���(⋅) ���(⋅) ���(⋅) ���(⋅) ���(⋅) J. You, J. Leskovec, K. He, S. Xie, Graph Structure of Neural Networks, ICML 2020 16 Neural Network as Rounds of Message Exchange A relational graph with A 5-layer 5 rounds of message exchange Neural network ���(⋅) ���(⋅) 1 2 ⟺ 4 3 ���(⋅) ���(⋅) This is how Graph Neural Networks compute embeddings! J. You, J. Leskovec, K. He, S. Xie, Graph Structure of Neural Networks, ICML 2020 17 Side Note: Connections with GNNs ���(⋅) ���(⋅) This is how Graph Neural Networks 1 2 compute embeddings! Specialty of GNNs: (1) Graph structure is regarded as the input instead of neural architecture; (2) Message functions are shared across 4 3 all the edges to respect input graph’s invariance properties. ���(⋅) ���(⋅) J. You, J. Leskovec, K. He, S. Xie, Graph Structure of Neural Networks, ICML 2020 18 Overview: Methodology § A novel representation of neural networks: relational graphs § Relational graphs can represent diverse neural architectures ß § Can represent architectures from MLP to ResNet 1 2 MLP layer 1 Translate Translate 3x3 conv to MLP to ResNet MLP layer 2 3x3 conv 4 3 … … § Tools from network science à Graph structure vs NN performance J. You, J. Leskovec, K. He, S. Xie, Graph Structure of Neural Networks, ICML 2020 19 Relational Graphs à Diverse Architectures Relational Graph �!(�!) �!(�") �! �" ���(⋅) ) # � ( ! � �# �$ The same relational graph à diverse architectures 4 Key components J. You, J. Leskovec, K. He, S. Xie, Graph Structure of Neural Networks, ICML 2020 20 Relational Graphs à Diverse Architectures Relational Graph 4-layer 4-dim MLP scalar �!(�!) MLP layer 1 � (� ) Node feature: �% ∈ ℝ � ! " � 4-dim 4-dim ! " Message: � � = � � ���(⋅) % & %& & MLP layer 2 Aggregation: ��� ⋅ = �∑ ⋅ 1 1 ) # � Rounds: � = 4 2 2 ( ! MLP layer 3 � translate 3 3 �# �$ MLP layer 4 4 4 MLP as relational graph J. You, J. Leskovec, K. He, S. Xie, Graph Structure of Neural Networks, ICML 2020 21 Relational Graphs à Diverse Architectures Relational Graph 4-layer 64-dim MLP vector �!(�!) MLP layer 1 !' � (� ) Node feature: �% ∈ ℝ � ! " � 64-dim 64-dim ! " Message: � � = � � ���(⋅) % & %& & MLP layer 2 Aggregation: ��� ⋅ = �∑ ⋅ 1 1 ) # � Rounds: � = 4 2 2 ( ! MLP layer 3 � translate 3 3 �# �$ MLP layer 4 4 4 MLP as relational graph J. You, J. Leskovec, K. He, S. Xie, Graph Structure of Neural Networks, ICML 2020 22 Relational Graphs à Diverse Architectures Relational Graph 4-layer 65-dim MLP vector �!(�!) MLP layer 1 � (� ) Node feature: � ∈ ℝ!', � ∈ ℝ!) � ! " � !,",$ # 65-dim 65-dim ! " Message: � � = � � ���(⋅) % & %& & MLP layer 2 1 1 ) Aggregation: ��� ⋅ = �∑ ⋅ # � Rounds: � = 4 2 2 ( ! MLP layer 3 � translate 3 3 �# �$ MLP layer 4 4 4 MLP as relational graph J. You, J. Leskovec, K. He, S. Xie, Graph Structure of Neural Networks, ICML 2020 23 Relational Graphs à Diverse Architectures Relational Graph ResNet-34 64-channels � (� ) tensor ! ! 3x3 conv !'×+×, �!(�") Node feature: �% ∈ ℝ �! �" ���(⋅) Message: �% �& = �%& ∗ �& 3x3 conv 64 channels 64 channels ) �&' is 3x3 conv kernel 1 1 # 2 2 � ( Aggregation: ��� ⋅ = �∑ ⋅ 3 3 ! 3x3 conv � Rounds: � = 34 4 4 � � # $ translate 3x3 conv … ResNet-34 as relational graph J. You, J. Leskovec, K. He, S. Xie, Graph Structure of Neural Networks, ICML 2020 25 Overview § A novel representation of neural networks: relational graphs § Relational graphs can represent diverse neural architectures § Network science à Graph structure vs NN performance ß WS-flex graph § Graph measures that characterize graph properties § Graph generators that generate diverse graphs WS-flex graph § Control computational budget Complete graph J. You, J. Leskovec, K. He, S. Xie, Graph Structure of Neural Networks, ICML 2020 26 Measuring Graph Structure § Graph measures: § Global: average path length (�) The average shortest path distance between any pair of nodes § Local: clustering coefficient (�) A measure of the degree to which nodes in a graph tend to cluster together � = 2.3 � = 2.0 � = 1.4 � = 0 � = 0.1 � = 0.5 J. You, J. Leskovec, K. He, S. Xie, Graph Structure of Neural Networks, ICML 2020 27 Generating Diverse Graphs § WS-flex graphs have a much better coverage Classic families of graphs Proposed WS-flex graphs J. You, J. Leskovec, K. He, S. Xie, Graph Structure of Neural Networks, ICML 2020 30 Generating Diverse Graphs § Visualization of WS-flex graphs WS-flex graph WS-flex graph Complete graph J.