Structure Preserving Embedding Blake Shaw [email protected] Tony Jebara [email protected] Department of Computer Science, Columbia University, 1214 Amsterdam Ave, New York, NY 10027 Abstract cover a low-dimensional set of coordinates for each ver- tex that implicitly encodes the graph's binary connec- Structure Preserving Embedding (SPE) is tivity. an algorithm for embedding graphs in Eu- clidean space such that the embedding is low- Graph embedding algorithms place nodes at points dimensional and preserves the global topolog- on some surface (e.g. Euclidean space) and connect ical properties of the input graph. Topology points with an arc if the nodes have an edge between is preserved if a connectivity algorithm, such them. One traditional objective of graph embedding is as k-nearest neighbors, can easily recover the to place points on a surface such that arcs never cross. edges of the input graph from only the coor- In this setting, it is well known that any graph can dinates of the nodes after embedding. SPE be embedded in 3-dimensional Euclidean space and is formulated as a semidefinite program that planar graphs can be embedded in 2-dimensional Eu- learns a low-rank kernel matrix constrained clidean space. However, there are many uses for graph by a set of linear inequalities which captures embedding that do not relate to arc crossing and thus the connectivity structure of the input graph. there exists a suite of embedding algorithms with dif- Traditional graph embedding algorithms do ferent goals (Chung, 1997; Battista et al., 1999). One not preserve structure according to our def- motivation for embedding graphs is to solve compu- inition, and thus the resulting visualizations tationally hard problems geometrically. For example, can be misleading or less informative. SPE using hyperplanes to separate points after graph em- provides significant improvements in terms bedding is useful for efficiently approximating the NP- of visualization and lossless compression of hard sparsest cut problem (Arora et al., 2004). This graphs, outperforming popular methods such article will focus on graph embedding for visualiza- as spectral embedding and Laplacian eigen- tion and compression. Given only the connectivity maps. We find that many classical graphs of a graph, can we efficiently recover low-dimensional and networks can be properly embedded us- point coordinates for each node such that these points ing only a few dimensions. Furthermore, can easily be used to reconstruct the original struc- introducing structure preserving constraints ture of the network? We are interested in reversible into dimensionality reduction algorithms pro- graph embeddings (from a graph to points back to a duces more accurate representations of high- graph). If the graph can be easily reconstructed from dimensional data. the points and these require low-dimensionality (and storage), the method can be useful for both visualiza- tion and compression. 1. Introduction Many embedding algorithms find compact coordinates Graphs are essential for encoding information, and for nodes in a graph. Given an unweighted graph graph and network data is increasingly abundant in consisting of N nodes and jEj edges represented as fields ranging from computational biology to computer a symmetric adjacency matrix A 2 f0; 1gN×N spec- vision. Given a graph where vertices and edges rep- ifying which pairs of nodes are connected, spectral resent pairwise interactions between entities (such as embedding finds a set of coordinates for each node d links between websites, friendships in social networks, ~yi 2 R for i = 1;:::;N by applying singular value or bonds between atoms in molecules), we hope to re- decomposition (SVD) or principal component analy- sis (PCA) to the adjacency matrix A and using the d Appearing in Proceedings of the 26 th International Confer- eigenvectors of A with the largest eigenvalues as the ence on Machine Learning, Montreal, Canada, 2009. Copy- coordinates. Similarly, Laplacian eigenmaps (Belkin right 2009 by the author(s)/owner(s). Structure Preserving Embedding A Möbius Ladder Spectral Embedding Two Spring Embeddings SPE Embedding Möbius Band Graph Figure 1. Embedding the classical Mobius Ladder Graph. Given the adjacency matrix (left), the visualizations produced by spectral embedding and spring embedding (middle) do not accurately capture the graph topology. The SPE embedding is compact and topologically correct. & Niyogi, 2002) employ a spectral decomposition of the right shows a typical result when a poor random the graph Laplacian L = D − A, or normalized graph initialization is used. Due to local minima in spring − 1 − 1 Laplacian L = I − D 2 AD 2 , where D = diag(A1) embedding algorithms, the graph is no longer visually and use the d eigenvectors of L with the smallest non- recognizable or accurate. We are motivated to find zero eigenvalues. Unfortunately, in practice there are a simple tool for properly visualizing graphs such as often many eligible eigenvectors of A or L, and fur- the M¨obiusladder, as well as large complex network thermore the resulting coordinates do not preserve the datasets. The tool should be accurate and should cir- topology of the input graph exactly. We propose learn- cumvent local minima issues. ing a positive semi-definite kernel matrix K 2 N×N R Structure preserving constraints can also benefit di- whose spectral decomposition yields a small set of mensionality reduction algorithms. These methods eigenvectors which preserve the topology of the input similarly find compact coordinates that preserve cer- graph. Specifically, given a connectivity algorithm G tain properties of the input data. Multidimensional (such as k-nearest neighbors, b-matching, or maximum scaling preserves distances between data points (Cox weight spanning tree) which accepts as input a kernel & M.Cox, 1994). Nonlinear manifold learning algo- K specifying an embedding and returns an adjacency rithms preserve local distances described by a graph matrix, we call an embedding structure preserving if on the data (Tenenbaum et al., 2000; Roweis & Saul, the application of G to K exactly reproduces the in- 2000; Weinberger et al., 2005). For these algorithms put graph: G(K) = A. This article proposes SPE, the input consists of high-dimensional points as well an efficient convex optimization based on semidefinite as binary connectivity. Many of these manifold learn- programming for finding an embedding K such that ing techniques preserve local distances but not graph K is both low-rank and structure preserving. topology. We show that adding explicit topological Traditional graph embedding algorithms such as spec- constraints to these existing algorithms is crucial for tral embedding and spring embedding do not explicitly preventing folding and collapsing problems that occur preserve structure according to our definition and thus in dimensionality reduction. in practice produce poor visualizations of many sim- The rest of the article is organized as follows. In Sec- ple classical graphs. In Figure 1 we see the classical tion 2, we introduce the concept of structure preserv- M¨obiusladder graph and the resulting visualizations ing constraints and formulate these constraints as a set from the two methods. The spectral embedding looks of linear inequalities for a variety of different connec- degenerate and does not resemble the M¨obiusband in tivity algorithms. We then derive an objective func- any regard. The eigenspectrum indicates that the em- tion in Section 3 which favors low-dimensional embed- bedding is 6-dimensional when we expect to be able to dings close to the spectral embedding solution. In Sec- embed this graph using fewer dimensions. Two spring tion 4, we combine the structure preserving constraints embeddings are shown. The left spring embedding is and the objective function into a convex optimization a good diagram of what the graph should look like; and propose a semidefinite program for solving it ef- however, we see that the twist of the M¨obiusstrip ficiently. We present a variety of experiments on real is not accurately captured. Given the coordinates in and synthetic graphs in Section 5 and show improve- Euclidean space produced by this method, any sim- ments in terms of visualization quality and level of ple neighbor-finding algorithm G would connect nodes compression. We then briefly explore using SPE to im- along the red dotted lines, not the blue ones specified prove dimensionality reduction algorithms before con- by the connectivity matrix. Thus, the inherent con- cluding in Section 7. nectivity of the embedding disagrees with the actual connectivity of the graph. The spring embedding on Structure Preserving Embedding 2. Preserving Graph Structure has maximal weight (Fremuth-Paeger & Jungnickel, 1999). We assume we are given an input graph defined by both a connectivity matrix A as well as an algorithm Definition 2. Given a kernel matrix K, define the G which accepts as input a kernel matrix K, and out- weight between two points (i; j) as the negated pairwise puts a connectivity matrix A~ = G(K), such that A~ is distance between them: Wij = −Dij = −Kii − Kjj + close to the original graph A. In this article, we con- 2Kij. sider several choices for G including k-nearest neigh- Once again, the matrix W composed of elements W bors, epsilon-balls, maximum weight spanning trees, ij is simply a linear function of K. Given W , a maximum maximum weight generalized matching and other max- weight subgraph produces a graph with binary adja- imum weight subgraphs. We can evaluate how well the cency matrix A~ which maximizes P A~ W . For ex- embedding produced by K preserves graph structure ij ij ij ample when G is the generalized b-matching algorithm, by determining how much the input connectivity dif- G finds the connectivity matrix which has maximum fers from the connectivity computed directly from the weight while enforcing a set of degree constraints b for learned embedding. A simple metric to capture this i i = 1:::N as follows: G(K) = arg max P W A~ difference is the normalized number of pairwise errors A~ ij ij ij 1 P ~ ~ ~ ~ ~ ~ P ~ s.t.