Filtering and Sampling Graph Signals, and its Application to Compressive Spectral Clustering Nicolas Tremblay(1;2); Gilles Puy(1); R´emiGribonval(1); Pierre Vandergheynst(1;2) (1) PANAMA Team, INRIA Rennes, France (2) Signal Processing Laboratory 2, EPFL, Switzerland Introduction to GSP Graph sampling Application to clustering Conclusion Why graph signal processing ? N. Tremblay Compressive Spectral Clustering Gdr ISIS, 17th of June 2016 1 / 47 Introduction to GSP Graph sampling Application to clustering Conclusion Introduction to GSP Graph Fourier Transform Graph filtering Graph sampling Application to clustering What is Spectral Clustering ? Compressive Spectral Clustering A toy experiment Experiments on the SBM Conclusion N. Tremblay Compressive Spectral Clustering Gdr ISIS, 17th of June 2016 2 / 47 Introduction to GSP Graph sampling Application to clustering Conclusion Introduction to graph signal processing : graph Fourier transform N. Tremblay Compressive Spectral Clustering Gdr ISIS, 17th of June 2016 3 / 47 Introduction to GSP Graph sampling Application to clustering Conclusion What's a graph signal ? N. Tremblay Compressive Spectral Clustering Gdr ISIS, 17th of June 2016 4 / 47 Introduction to GSP Graph sampling Application to clustering Conclusion Three useful matrices The adjacency matrix : The degree matrix : 20 1 1 03 22 0 0 03 61 0 1 17 60 3 0 07 W = 6 7 S = 6 7 41 1 0 05 40 0 2 05 0 1 0 0 0 0 0 1 The Laplacian matrix : 2 2 −1 −1 0 3 6−1 3 −1 −17 L = S − W = 6 7 4−1 −1 2 0 5 0 −1 0 1 N. Tremblay Compressive Spectral Clustering Gdr ISIS, 17th of June 2016 5 / 47 Introduction to GSP Graph sampling Application to clustering Conclusion Three useful matrices The adjacency matrix : The degree matrix : 2 0 :5 :5 03 21 0 0 03 6:5 0 :5 47 60 5 0 07 W = 6 7 S = 6 7 4:5 :5 0 05 40 0 1 05 0 4 0 0 0 0 0 4 The Laplacian matrix : 2 1 −:5 −:5 0 3 6−:5 5 −:5 −47 L = S − W = 6 7 4−:5 −:5 1 0 5 0 −4 0 4 N. Tremblay Compressive Spectral Clustering Gdr ISIS, 17th of June 2016 5 / 47 Introduction to GSP Graph sampling Application to clustering Conclusion What's a graph Fourier transform ? [Hammond '11] • U is the Fourier basis of the graph • the Fourier transform of a signal x reads : > L = S − W =UΛU x^ =U >x • Λ = Diag(λ1; λ2; ··· ; λN ) the spectrum A low frequency Fourier mode A high frequency Fourier mode N. Tremblay Compressive Spectral Clustering Gdr ISIS, 17th of June 2016 6 / 47 Introduction to GSP Graph sampling Application to clustering Conclusion The graph Fourier transform encodes the structure of the graph Slide courtesy of D. Shuman N. Tremblay Compressive Spectral Clustering Gdr ISIS, 17th of June 2016 7 / 47 Introduction to GSP Graph sampling Application to clustering Conclusion Introduction to graph signal processing : filtering graph signals N. Tremblay Compressive Spectral Clustering Gdr ISIS, 17th of June 2016 8 / 47 Problem : this costs L's diagonalisation [O(N3)]. p Solution : we use a poly approx of order p of h : X l h~(λ) = αl λ ' h(λ): l=1 Indeed, in this case : p p > X l > X l H~x =U h~(Λ)U x =U αl Λ U x = αl L x ' Hx l=1 l=1 ) Only involves matrix-vector multiplications [costs O(pN)]. Introduction to GSP Graph sampling Application to clustering Conclusion Graph filtering 1 0.8 ) 0.6 Given a filter function h defined in the λ g( Fourier space. h 0.4 0.2 0 1 2 λ In the node space, the signal x filtered by h reads : x h =U h(Λ)U > x = Hx N. Tremblay Compressive Spectral Clustering Gdr ISIS, 17th of June 2016 9 / 47 p Solution : we use a poly approx of order p of h : X l h~(λ) = αl λ ' h(λ): l=1 Indeed, in this case : p p > X l > X l H~x =U h~(Λ)U x =U αl Λ U x = αl L x ' Hx l=1 l=1 ) Only involves matrix-vector multiplications [costs O(pN)]. Introduction to GSP Graph sampling Application to clustering Conclusion Graph filtering 1 0.8 ) 0.6 Given a filter function h defined in the λ g( Fourier space. h 0.4 0.2 0 1 2 λ In the node space, the signal x filtered by h reads : x h =U h(Λ)U > x = Hx Problem : this costs L's diagonalisation [O(N3)]. N. Tremblay Compressive Spectral Clustering Gdr ISIS, 17th of June 2016 9 / 47 Introduction to GSP Graph sampling Application to clustering Conclusion Graph filtering 1 0.8 ) 0.6 Given a filter function h defined in the λ g( Fourier space. h 0.4 0.2 0 1 2 λ In the node space, the signal x filtered by h reads : x h =U h(Λ)U > x = Hx Problem : this costs L's diagonalisation [O(N3)]. p Solution : we use a poly approx of order p of h : X l h~(λ) = αl λ ' h(λ): l=1 Indeed, in this case : p p > X l > X l H~x =U h~(Λ)U x =U αl Λ U x = αl L x ' Hx l=1 l=1 ) Only involves matrix-vector multiplications [costs O(pN)]. N. Tremblay Compressive Spectral Clustering Gdr ISIS, 17th of June 2016 9 / 47 Introduction to GSP Graph sampling Application to clustering Conclusion A few applications 2 > • Tikhonov regularization for denoising : argminf fkf − yk2 + γf Lf g ∗ 2 • Wavelet denoising : argminafkf − W ak2 + γ kak1,µg Slide courtesy of D. Shuman • Compression via filterbanks, etc. N. Tremblay Compressive Spectral Clustering Gdr ISIS, 17th of June 2016 10 / 47 Introduction to GSP Graph sampling Application to clustering Conclusion Introduction to GSP Graph Fourier Transform Graph filtering Graph sampling Application to clustering What is Spectral Clustering ? Compressive Spectral Clustering A toy experiment Experiments on the SBM Conclusion N. Tremblay Compressive Spectral Clustering Gdr ISIS, 17th of June 2016 11 / 47 How to reconstruct the original signal ? Basically, we need : 1. a (low-dimensional) model for the signal to sample 2. a method to choose the nodes to sample 3. a \decoder" that exactly recovers the signal given its samples Introduction to GSP Graph sampling Application to clustering Conclusion Sampling a graph signal consists in : 1. choosing a subset of nodes 2. measuring the signal on these nodes only N. Tremblay Compressive Spectral Clustering Gdr ISIS, 17th of June 2016 12 / 47 How to reconstruct the original signal ? Basically, we need : 1. a (low-dimensional) model for the signal to sample 2. a method to choose the nodes to sample 3. a \decoder" that exactly recovers the signal given its samples Introduction to GSP Graph sampling Application to clustering Conclusion Sampling a graph signal consists in : 1. choosing a subset of nodes 2. measuring the signal on these nodes only N. Tremblay Compressive Spectral Clustering Gdr ISIS, 17th of June 2016 12 / 47 Introduction to GSP Graph sampling Application to clustering Conclusion Sampling a graph signal consists in : 1. choosing a subset of nodes 2. measuring the signal on these nodes only How to reconstruct the original signal ? Basically, we need : 1. a (low-dimensional) model for the signal to sample 2. a method to choose the nodes to sample 3. a \decoder" that exactly recovers the signal given its samples N. Tremblay Compressive Spectral Clustering Gdr ISIS, 17th of June 2016 12 / 47 Definition (Bandlimited graph signal [Puy '15, Chen '15, Anis '16, Segarra '15] ) N k A k-bandlimited signal x 2 R on G is a signal that satisfies, for someα ^ 2 R x = Uk α^; Introduction to GSP Graph sampling Application to clustering Conclusion Smoothness assumption In 1D signal processing, a smooth signal has most of its energy at low frequencies. 1 3 0.9 2.5 0.8 0.7 2 0.6 0.5 1.5 0.4 1 0.3 0.2 0.5 0.1 0 0 −1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 −1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 Smooth signal in time Fourier transform N. Tremblay Compressive Spectral Clustering Gdr ISIS, 17th of June 2016 13 / 47 Introduction to GSP Graph sampling Application to clustering Conclusion Smoothness assumption In 1D signal processing, a smooth signal has most of its energy at low frequencies. 1 3 0.9 2.5 0.8 0.7 2 0.6 0.5 1.5 0.4 1 0.3 0.2 0.5 0.1 0 0 −1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 −1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 Smooth signal in time Fourier transform Definition (Bandlimited graph signal [Puy '15, Chen '15, Anis '16, Segarra '15] ) N k A k-bandlimited signal x 2 R on G is a signal that satisfies, for someα ^ 2 R x = Uk α^; N. Tremblay Compressive Spectral Clustering Gdr ISIS, 17th of June 2016 13 / 47 Sampling procedure : draw n nodes according to p : f!i gi2[1;n]. We create a matrix M that measures the signal x only on the selected nodes : 1 if j = ! M := i ij 0 otherwise; N For any signal x 2 R on G, its sampled version is y = Mx (it has size n < N).