Incorporating Network Structure with Node Contents for Community Detection on Large Networks Using Deep Learning

Neurocomputing 297 (2018) 71–81

Contents lists available at ScienceDirect

Neurocomputing

journal homepage: www.elsevier.com/locate/neucom

Incorporating network structure with node contents for community detection on large networks using deep learning

∗ Jinxin Cao a, Di Jin a, , Liang Yang c, Jianwu Dang a,b a School of Computer Science and Technology, Tianjin University, Tianjin 300350, China b School of Information Science, Japan Advanced Institute of Science and Technology, Ishikawa 923-1292, Japan c School of Computer Science and Engineering, Hebei University of Technology, Tianjin 300401, China

a r t i c l e i n f o a b s t r a c t

Article history: Community detection is an important task in social network analysis. In community detection, in gen- Received 3 June 2017 eral, there exist two types of the models that utilize either network topology or node contents. Some

Revised 19 December 2017 studies endeavor to incorporate these two types of models under the framework of spectral clustering Accepted 28 January 2018 for a better community detection. However, it was not successful to obtain a big achievement since they Available online 2 February 2018 used a simple way for the combination. To reach a better community detection, it requires to realize a Communicated by Prof. FangXiang Wu seamless combination of these two methods. For this purpose, we re-examine the properties of the modularity maximization and normalized-cut models and fund out a certain approach to realize a seamless Keywords:

Community detection combination of these two models. These two models seek for a low-rank embedding to represent of the

Deep learning community structure and reconstruct the network topology and node contents, respectively. Meanwhile, Spectral clustering we found that autoencoder and spectral clustering have a similar framework in their low- rank matrix re- Modularity maximization construction. Based on this property, we proposed a new approach to seamlessly combine the models of Normalized-cut modularity and normalized-cut via the autoencoder. The proposed method also utilized the advantages of Node contents the deep structure by means of deep learning. The experiment demonstrated that the proposed method can provide a nonlinearly deep representation for a large-scale network and reached an eﬃcient community detection. The evaluation results showed that our proposed method outperformed the existing leading methods on nine real-world networks. ©2018 Elsevier B.V. All rights reserved.

1. Introduction in biology, some different units belonging to an organization have some related functions that are interconnected with special The development of Internet has led to producing more and structures to characterize the whole effect of the organization. more variety of data, such as online comments, product reviews The interaction among a set of proteins in a cell can form an RNA and co-author networks, which have affected all aspects of peo- polymerase for transcription of genes. In computer science, finding ple’s lives, and thus the analysis of those data has attracted more the salient communities from an organization of people can create and more attention of researchers in various fields. One hot topic a guide which helps to web marketing, behavior prediction of in the studies of such social media or online data is to discovery users belonging to a community and understanding the functions the underlying structure with group effect, which is the so-called of a complex system [1] . community structure. The vertices (users) related to the commu- For community detection, some methods have put forward, nities in the network can be divided into groups, in which vertices which can be cast as graph clustering. In normalized cut (n-cut) have more multiple connection but the connections are relatively [4] , the Laplacian matrix is the main objective to be processed. sparse in the whole network. Those individuals or users belonging The eigenvectors with a non-zero eigenvalue, which are obtained to the same community share common profiles or have common by the eigenvalue decomposition (EVD) of graph Laplacian ma- interests. The identification of communities consisting of users trix, are treated as graph representation. Some other works can with similarity is very important, and has been applied in many be also transformed to spectral clustering. For example, the mod- areas, e.g. , sociology, biology and computer science. For example, ularity maximization model [10] first constructs a graph that is based on feature vectors, and then solves the top k eigenvectors as network representation for clustering. Here, we can deem mod- ∗ Corresponding author. ularity matrix as graph Laplacian matrix. We realize that, those E-mail addresses: [email protected] (J. Cao), [email protected] (D. Jin), [email protected] (L. Yang), [email protected] (J. Dang). two methods (i.e. modularity optimization and n-cut) can easily https://doi.org/10.1016/j.neucom.2018.01.065 0925-2312/© 2018 Elsevier B.V. All rights reserved. 72 J. Cao et al. / Neurocomputing 297 (2018) 71–81

Modularity maximization

Modularity matrix B Link information Deep structure

Output

The attributed Clustering network Incorporation Content of links and information Normalized cut Joint node content optimization

Markov matrix M

Fig. 1. The proposed framework. For community discovery on the attributed network, our input data consists of two parts which are modularity matrix and Markov matrix for the topological structure and node contents, respectively. Because of the similarity between spectral methods and autoencoder in terms of an approximation of the matrix reconstruction, the network representation is denoted by the low-dimensional encoding in the hidden layer of an autoencoder block. Finally, we achieve the fusion of network topology and node contents for community detection. Furthermore, we can append many autoencoder blocks and build a multiple layer deep neural network framework. capture topology-based and content-based features by EVD of the idea is shown in Fig. 1 . We realized that autoencoder is a type corresponding spectral matrices separately, as shown in Fig. 1 . of unsupervised learning methods, and thus only treat the low- However, those methods for community detection are often lim- dimensional encoding in the hidden layer as the network represen- ited to obtain the important information about the structure of the tation. In our method, we adopt modularity maximization model communities in networks. It demonstrates that one of the tech- and normalized-cut to portray linkage and content information, niques can overcome the problem, which only consider topologi- separately, and construct the spectral matrices (i.e. modularity ma- cal structure (or node contents), by fusing the vertex information trix and Markov matrix) as the input of the autoencoder. We de- (or say node contents) with linkage information for community sign a unified objective function to get the best reconstruction of detection [2] . the combination matrix that consists of modularity matrix and When considering both topological and content information for Markov matrix, while make use of autoencoder to get a best en- community discovery, we can combine these two objective func- coding in the hidden layer as the network representation which tions into one in the form of linearity directly. However, some clas- is used to finding communities nicely. Furthermore, by building a sical graph embedding methods, such as locally linear embedding multi-layers autoencoder, we adopt deep autoencoder to obtain a (LLE) [11] , show that the relations among vertices in the real-world powerful representation by means of the deep structure, and com- networks are not certainly linear. So, the model based on this lin- bine with the intrinsic information of the original data to achieve early combination strategy is still limited on real-world networks. an improvement for discovering communities. In total, our frame- Moreover, although we could get a network representation by fus- work has three main contributions as follows: ing those two types of information, the problem in the optimization of the combination model is, the low efficiency for deciding • First, in theory both the autoencoder and spectral methods are an appropriate ratio of two kinds of information due to manual related to the low-dimensional approximation of the specified tuning such ratios. corresponding matrix, i.e. the modularity matrix and Markov In the recent years, deep learning is used in many areas, such matrix. This study utilizes the autoencoder to obtain a low- as speech recognition [6] , image classification [7] and so on. As we dimensional encoding which can best reconstruct the joint known, neural network is a good framework for nonlinear compu- matrix consisting of the modularity matrix and the Markov tation with the elements that simulate the structure and properties matrix, and treats this encoding as the graph representations of neurons [8] . Among them, autoencoder is proposed by Ng [5] , for community detection. The important point is to propose an which aims to obtain features from the input data. We found that autoencoder-based method that can achieve the joint optimiza- autoencoder and spectral methods all intent to obtain the low- tion of modularity model and normalized-cut without a seam. dimensional approximation of the corresponding matrix. Based on • Second, this encoding supplies a nonlinear way to integrate this similarity, we adopt autoencoder as a breakpoint method to the linkage and content information. This helps to further solve the disadvantages of linear optimization, and to achieve the improve the performance of community detection, when using incorporation of these two different spectral methods. both those two types of information. The autoencoder not only In order to not only take the advantage of spectral methods encodes the important factors of the data in the process of re- but also achieve the incorporation of linkage and node content ducing the dimension, but also automatically learns the weight information, we propose an autoencoder-based method for com- of the relationship among the various factors to obtain the munity detection using the normalized-cut and modularity maxi- minimum of reconstruction error. In this framework, therefore, mization. Our work is inspired by the similarity in theory between the performance improvement of our method is realized by its autoencoder and spectral methods in terms of getting an intrin- self-tuning characteristic, but not depend on adjusting balance sic structure of the spectral matrix. The framework of our main factor. J. Cao et al. / Neurocomputing 297 (2018) 71–81 73

• Furthermore, by stacking a series of autoencoders, we built a the network topology and node contents respectively. In the DNN multi-layer autoencoder in favor of enhancing better general- framework, we found that autoencoder has similar properties to ization ability of the encoding in the hidden layer. Beneﬁtting spectral clustering in low-rank matrix reconstruction. Based on from the deep structure, we get a powerful encoding with both this property, we proposed a new approach to seamlessly combine topological and content information, which can effectively aid the models of modularity and normalized-cut via the autoencoder. network community detection. The autoencoder-based method is able to take the advantages of spectral methods and deep representation of the DNN, so that The rest of the paper is organized as follows. In Section 2 , achieves the joint optimization of modularity and normalized-cut we give a brief review of the related work. The proposed frame- for community detection based on both links and node contents. work and the relevant algorithms are introduced in Section 3 . Next, datasets and experimental setting are described in Section 4 , and followed experimental evaluation and the analysis of the balance 3.1. Spectral methods for community detection factor in this Section demonstrate the effectiveness of the proposed new method. The paper is then concluded in Section 5 . Many community detection methods have been proposed. Among them, one type of the most popular methods are the spec- 2. Related work tral methods. Here, we explain two spectral approaches: modularity maximization and normalized cut. There exist three aspects of relevant works regarding the topic here, which are community detection with topological structure or content information alone, and the combination of links and 3.1.1. Modularity maximization model node contents. As described above, it is not appropriate that node The topological structure of a network can be treated as a graph community memberships are denoted by using the network topol- G = ( V, E ), where V is the collection of vertices containing n nodes ogy or content information alone. Combining topology and con- {v1 ,…,vN }, and E the set of the edges in which each edge connects tent achieves an improvement for community detection, as showed two vertices in V . We treat a nonnegative symmetric binary ma-

= { } ∈ R N×N = in studies [1,2,3,17,19,20] . However, they utilized those two types trix A aij + as the adjacency matrix of G, and set aij 1 if = of information while did not take into account of their nonlin- there is an edge between vertex vi and vj , and aij 0 otherwise. ear combination. Besides, although those studies considered both Here, we make use of the modularity model to deal with com- linkage and content information, they usually modeled two differ- munity detection with network topology alone. This model is op- ent types of information separately, and then balanced these types timized through maximizing modularity, which means the division of information by using a manually adjusted hyper-parameter. This between the number of edges in the original graph and the ex- may keep them from the optimal improvement. pected number of edges in a random graph without community × As well-known, the conventional paradigm for generating fea- structure. We can deﬁne modularity matrix as B ∈ R N NR with el- = tures for vertices is based on feature extraction techniques which ements bij aij – ki kj /2 m, in which ki is the degree of vertex vi and typically involve some features based on network properties. m the number of edges. Newman [10] also relaxed this problem Among them, spectral approaches such as normalized cut [4] and when considering a network with C communities, and then modu- modularity maximization model [10] exploit the spectral prop- larity maximization can be transformed to the follow optimization erties of various matrix representations of graphs, especially the problem: Laplacian and adjacency matrices. These methods can be taken as T min tr X BX the dimensionality reduction techniques under linear perspective, X ∈ R N×C (1) while they suffer from both nonlinear computation and statisti- s.t. tr X T X = N cal performance drawbacks. In terms of the collaborative optimiza- = { } ∈ R N×C tion, spectral methods are deemed to as eigenvalue decomposition where tr( ) denotes the trace of a matrix, and X xij is (EVD) of a data matrix, however, it is hard to design a uniﬁed an indicator matrix. Based on Rayleigh Quotient [34] , the above framework or a general model fusing the different objective func- problem can be solved by extracting the largest C eigenvectors of tions of the spectral methods without a seam. the modularity matrix B . In this way, the optimization of modular- As for deep learning, it has achieved great success in many ity is taken as the eigenvalue decomposition of modularity matrix applications. This may because deep neural network (DNN) can in terms of the spectral methods. describe a complicated object using a huge nonlinear space and also have better generalization. Recently, there are several stud- 3.1.2. Normalized-cut ies on community detection using deep learning methods, such As to the content information of the network, we portray the as research works described in [21,34] . Although they improve content on vertex vi by using a multi-dimensional word vector. We the performance via DNN, only the structural or node content = { } ∈ R N×N use S sij to represent the similarity matrix of the graph information was taken into account in their studies. They did G, and sij is measured by using the cosine similarity between the not utilize the most important advantage of the DNN, which can word vector of the vertex vi and that of the vertex vj . Also, we have combine the different modalities altogether. For this reason, we taken into account of the manifold structure of node content space, focus on incorporating those two kinds of information together for and dealt with the original similarity matrix by using k - near neigh- community detection via the DNN framework. bor consistency method in [11]. It preserves the similarity value sij

3. Framework of community detection using deep learning between vertex vi and each of its k-near neighbor vertices and 0 otherwise. To fully utilize the advantages of deep neural network (DNN) By now, we got a similarity graph G in the form of similar- for combining network topology and content information, we ity matrix S in which sij denotes the weight of the edge between re-examine the properties of the modularity maximization model vertices vi and vj . So the normalized-cut is defined so that, we and normalized cut, which are the leading models for community want to cut the least edges to achieve that vertices belonging to detection, and re-search the DNN framework to find out a certain the same partition have a high similarity while the vertices be- approach appropriate to realize a seamless combination of the tween different partitions have a low similarity. In this process, we different modalities. These two models seek for a low-rank em- measure the degree of vertices in a subset g of graph G by using v ol(g) = ∈ s . When considering to divide the graph G into C bedding to represent of the community structure and reconstruct vi c ij 74 J. Cao et al. / Neurocomputing 297 (2018) 71–81 groups, the normalized-cut (n-cut) is treated as the optimization where r is the number of nodes in the hidden layer and N the problemin the following: number of nodes in the first or the third layer. The autoencoder can find the low-dimensional representation of a matrix by mini- T min tr Y LY Y ∈ R N×C mizing the reconstruction error between the data in the first layer

s.t. Y T DY = I ⎧ and that in the third layer. 1 ⎨ , ∈ (2) if vi g j N ∗ = δ = ( , ) = ( , ϕ ( φ( ) ) ) yij vol g B arg min O B B arg min O bi bi (6) ⎩ j δ δ = 0 , ot herwise B B i 1 ∗ ∗ = where δ = { W , d , W , d } is the parameter of the autoencoder- Here, let si j sij be the diagonal elements of the diagonal B matrix D , and the graph Laplacian matrix is set to D −1 L . Like the based model, and N the number of nodes in the ﬁrst layer. After optimization of modularity model, by relaxing the optimization training the autoencoder, one can obtain the parameters W and d , = { } ∈ R N×C and then get a low-dimensional encoding based on Eq. (4) which problem, the indication matrix Y yij can be produced by eigenvalue decomposition of the Laplacian matrix D −1 L . Notice is regarded as the graph representation of this network. that D −1 L = D −1 ( D – S ) = I – D −1 S . According to eigenvalue decom- On the other hand, we can also utilize Markov matrix M as the position of the matrix D −1 S , the matrix Y is represented by the input of autoencoder that achieves community detection with con- eigenvectors corresponding to the top C eigenvalues of the matrix tent information. In this autoencoder, the corresponding loss func- D −1 S . Here, we set M = D −1 S which is the so-called Markov matrix. tion of this reconstruction can be written as

N ∗ δ = ( , ) = ( , ϕ ( φ( ) ) ) 3.2. Community detection via autoencoder reconstruction M arg min O M M arg min O mi mi (7) δ δ M M i =1 As described in Section 3.1 , both modularity maximization δ = { , , ∗, ∗} model and normalized cut can be converted into eigenvalue where M W d W d denotes the parameter of the decomposition of their corresponding matrices. The Eckart-Young- autoencoder-based model to be learned. Mirsky Theorem [9] illustrates this point that the eigenvalue decomposition of spectral matrix is referred to the low-rank 3.3. Combination of links and node contents for community detection approximation of a matrix. via autoencoder reconstruction

∈ R m ×n Theorem 1. (Eckart-Young-Mirsky) Given a matrix with a As described above, either modularity maximization or normal- κ = T rank- , let the singular value decomposition (SVD) of be P Q ized cut aims to ﬁnd the low-dimensional representation as the T = T = where P P I and Q Q I. We can optimize this problem graph representation in the latent space. Their optimization prob-

− ∗ 2 lems are as follows: arg min F ∈ R m ×n (3) ∗ 2 . . ∗ = κ − s t rank( ) arg min B B F (8) B ∗∈ R N×N by using ∗ = P ∗Q T . Here, the singular value matrix only con- ∗ 2 κ − tains the largest singular values and 0 otherwise. arg min M M F (9) M ∗∈ R N×N

It explains that the SVD of a matrix aims to obtain the best ∗ T So we can adopt the orthogonal decomposition B = X B X and low-rank approximation of the original matrix. In special, the mod- ∗ M = Y Y T to solve ( 8 ) and ( 9 ), respectively. ularity matrix B is symmetric. The SVD of B is accomplished by M For community detection with both network topology and node B = P P T , regarded as the orthogonal decomposition of B where contents, we design a combination matrix Z = [ B, M ] T which con- P T P = I and the diagonal elements of the diagonal matrix are sists of the modularity matrix B for linkage information as well corresponding to the eigenvalues of B though the eigenvalue de- as Markov matrix M for content information, and then get a low- composition (EVD) of B . Thus, the indicator matrix X can be ob- dimensional encoding that can best lead to approximation of the tained by constructing the best rank- k approximation of the matrix combination matrix for the network representation. B under the Frobenuis norm. As mentioned above, the eigenvalue decomposition is highly re- ∗ 2 T ∗ ∗ T 2 arg min Z − Z = arg min [ B , M ] − [ B , M ] (10) ∗ 2 N×N F T F lated to autoencoder in terms of matrix reconstruction. And thus, Z ∈ R [ B ∗, M ∗] here we introduce an autoencoder with three layers to obtain a According to the explanation in [36] , we can construct approxi- low-dimensional encoding that often leads to the best reconstruc- mation in terms of matrix factorization Z ∗ = PH where all columns tion of the modularity matrix B . In this autoencoder, from the ﬁrst of matrix P are denoted as the basis features and all columns of H layer to the hidden layer, the encoder is responsible to map the in- mean the low-dimensional representation for data Z . And thus, we put B into a low-dimensional encoding H l = { h l } ∈ R r×N in which ij convert the optimization problem in ( 10 ) into the i th column h l denotes a representation corresponding to vertex i 2 arg min Z − PH vi in the latent space. r×N F H ∈ R + l h = φ( b ) = sig ( W · b + d ) (4) i i i By now, we got a new matrix reconstruction problem for com- From the hidden layer to the third layer, the decoder maps the munity detection using the rich data consisting of both network

l latent representation H to thereconstruction of the input data, topology and node contents. ∗ l ∗ l ∗ Based on the above discussions, we ﬁrst adopt an autoencoder b = ϕ h = sig W · h + d (5) i i i to obtain a low-dimensional encoding of the combination matrix Z ∗ where bi and bi means the ith column of the modularity matrix B while we achieve collective optimization of those two different ob- ∗ and that of the reconstruction matrix B respectively, and sig ( ) is jectives. We then deal with each column of the combination matrix 1 ( ) = an nonlinear mapping function (we adopt sig x 1+ e −x in this ar- Z by fusing the topology-based and the content-based “features”, ticle). The weight matrices W ∈ R r×N , W ∗ ∈ R N×r and the bias vec- and thus the obtained representation H a provides a good way of tors d ∈ R r×1 , d ∗ ∈ R N×1 are all to be learned in the autoencoder, combination of those two types of information. J. Cao et al. / Neurocomputing 297 (2018) 71–81 75

Network Modularity B Output Kmeans . matrix .

Hidden representation Clustering matrix H Content Markov Encoder 1 Encoder Decoder Decoder 1 on nodes matrix M

Deep autoencoder

Fig. 2. Construction of a multiple layer deep autoencoder framework for obtaining the deep representation of the graph. The input matrix Z consists of two parts, modularity matrix B and Markov matrix M , where the low-dimensional encoding in the hidden layer of an autoencoder block is treated as a community-oriented graph representation. The deep autoencoder structure used here does not necessarily consist of only two stacked autoencoders. It can adopt three or more stacked autoencoders. In the experiment, the detailed information of the structure of the deep autoencoders is shown in Section 4.5 .

So we use the combination matrix Z as the input of our autoencoder. The encoder maps the input data to a low-dimensional N ∂ ∂ a = { a } ∈ R r×N a ∗ encoding H h in which its i-th column h represents O ( Z , Z ) = O ( z , ϕ ( φ( z ) ) ) i i ∂ ∂ i i di di a representation related to vertex vi . In particular, the ith column i =1

= T , T T of the combination matrix is denoted as zi [bi mi ] . And the N N ∂ ∂ encoding can be written as = ( , ϕ ( φ( ) ) ) a = ρ , O zi zi hi i ∂ a ∂ = hi di = a = φ( ) = ( · + ) i 1 i 1 hi zi sig W zi d (11) ∂ × × a = + ρ = ( , ϕ ( φ( ) ) ) ∈ R r 2N ∈ R r 1 where h W zi d, and i ∂h a O zi zi denotes each where W is the weight matrix, d the bias vector, i 1 ( ) = node “responsible” for the whole error. To measure the difference and sig( ) nonlinear mapping function sig x 1+ e −x . Otherwise, a ∗ ρ the decoder maps a representation H to the reconstruction of the between the input data Z and the reconstruction Z , i in the en- input matrix Z , as follows: coder is written as ∗ ∗ ∗ N = ϕ a = ( · + ) ∂ ∂ zi hi sig W h d (12) ∗ a ρ = O ( z , ϕ ( φ( z ) ) ) = − O Z − Z · sig h , i ∂ a i i ∂ a ij ij i ∗ × ∗ × h h And the parameters W ∈ R 2N r and d ∈ R 2N 1 need to be i i =1 i learned in the decoder. So in the autoencoder, we can obtain the ω = ∗ + While, in the decoder, we have i W h d and the “respon- best low-dimensional representation by optimizing the following ρ∗ sibility” i is problem r N ∗ ∗ ρ = · ρ · ( ) , ∗ i Wij i sig zi δ = arg min O ( Z , Z ) = arg min O ( z i , ϕ ( φ( z i ) ) ) (13) δ δ i =1 i =1 where sig’ ( ) is obtained by taking the derivative of the function where δ = { W , d , W ∗, d ∗} is the set of the parameters to be learned sig ( ). in the optimization of this proposed model. The workflow of the proposed framework is as follows. 3.5. Construction of deep autoencoder 3.4. Optimization of the autoencoder-based model Deep learning is well known for its powerful data representation capability. By training the neural network layer by layer, the We apply stochastic gradient descent [35] to solve the problem representation produced by each layer will better act on the next in ( 13 ). To be specific, the parameters { W, d, W ∗, d ∗} of the model layer. To benefit from the deep structure, we build a deep autoen- are updated in each iteration according to: coder with a series of autoencoders, and train the model in the ∂ ∗ ∂ ∗ first autoencoder by reconstructing the input matrix Z , and then W = W − η · ( Z , Z ), d = d − η · ( Z , Z ), 1 ij ij O i i O 1 ∈ R r ×N ∂ W ij ∂ d i get a best latent representation H . And finally, we train 1 ∗ ∗ ∂ ∗ ∗ ∗ ∂ ∗ the next autoencoder by reconstructing the H from the above au- W = W − η · O ( Z , Z ) , d = d − η · O ( Z , Z ), 2 × ij ij ∂ ∗ i i ∂ ∗ toencoder, and get a new latent representation H 2 ∈ R r N. In this Wij di deep architecture, the number of nodes in the next autoencoder is

Because of the similarity in the inference processing of W, d less than that in the above autoencoder, i.e. r 2 > r 1 . The workﬂow ∗ ∗ and W , d we only show the inference of the update rules for the of this deep framework is shown as Table 1 . Besides, we show the parameters W, d as follows: conﬁguration of the number of nodes in the deep autoencoder in N Section 4.5 . ∂ ∗ ∂ O ( Z , Z ) = O ( z i , ϕ ( φ( z i ) ) ) In the following, we analyze the computational complexity of ∂ W ij ∂ W ij i =1 the proposed new algorithm. For a standard multilayer perceptron, N N which is the key component of the autoencoder, matrix multiplica- ∂ ∂ = ( , ϕ ( φ( ) ) ) a = ρ T , a O zi zi hi i zi tions occupy most of the computational time. In a single layer with ∂h ∂ W ij i =1 i i =1 76 J. Cao et al. / Neurocomputing 297 (2018) 71–81

Table 1 Our algorithm framework.

Algorithm

Input

B = { b } ∈ R N×N : Modularity matrix. ij M = D −1S : Markov matrix. Output × H ∈ R r N : The network representation. σ σ σ × 1. Set : DNN layers number, r 1 = 2 N, r : The number of nodes in layer σ , H 1 = Z = [ B , M ] T, H ∈ R r N is the input to layer σ .

2. Put the data matrix H 1 into the deep autoencoder. For σ = 1 to Build an autoencoder block with three layers, and put the data H σ into it. Train the autoencoder and obtain the hidden layer u σ using the function ( 13 ). Set H σ = u σ . End

Table 2 = vector f(vi ) of node vi , f(vi ) 1 means the jth word in the content Real-world datasets. “undirected” means that this is an undirected graph, and “0/1 on vertex v occurred, and f ( v ) = 1 otherwise. denotes that the node content information is described by the d -dimensions binary i i vector. V denotes the number of nodes, E the number of edges, and C the number Cora [22]: It is also a citation type dataset which includes sci- of communities. Here we applied all datasets uniformly in form of the undirected ence publications from 7 subﬁelds of machine learning, so the and unweighted graph. number of clusters is set to 7 in the experiments.

Dataset V E Type of G F Type of F C PubMed [22]: The PubMed Diabetes dataset consists of 19,729 scientiﬁc publications from three classes. Each publication in this Twitter629863 171 796 directed 578 0/1 9 dataset is described by a tf/idf word vector with 500 unique words. Texas 187 328 undirected 1,703 0/1 5

Washington 230 446 undirected 1,703 0/1 5 In this paper, for the uniﬁcation of dataset format, we adopt 500 Wisconsin 265 530 undirected 1,703 0/1 5 dimensions’ binary content feature vectors instead of tf/idf word Facebook107 1045 26,749 directed 576 0/1 7 vector, and set an element to 1 in the content feature vector cor-

Cora 2708 5,429 undirected 1433 0/1 7 responding to non-zero value of the tf/idf word vector. Citeseer 3312 4,732 undirected 3703 0/1 6

Uai2010 3363 45,006 directed 4972 0/1 19 Texas, Washington, Wisconsin [22]: These three subsets are all PubMed 19,729 44,338 directed 500 0/1 3 described by a lot of webpages and links are “generated” by some college students from those 3 Universities. These webpages are represented by vertices while the links among the webpages represent the edges. Vertices are labeled by webpages belonging to N input dimensions and N /2 output dimensions, back-propagation user’s class. Here, each webpage in those three datasets is de- algorithm requires O( N 2d /2) floating-point calculations, where d scribed by a 1703 dimensions’ binary vector respectively. ( d < < N ) is the number of mini-batchs. Besides, when the nodes Uai2010 [22] : This is a Wikipedia dataset of Wikipedia articles are sparsely connected, the time complexity of back-propagation is that appeared in the featured list. This dataset includes 3363 docu- reduced to be O( NMd /2), in which M ( M < < N ) is the number of ments and 45,006 links. The documents are from 19 distinct cate- links. And thus this step needs O( N ) time. Since the output func- gories and are represented by the tf/idf weighted vectors. Here, we tions of neurons are in general very simple to compute, one can make use of 4972 dimensions’ binary feature vectors instead of the assume that those costs are constant per neuron. Also, because tf/idf vectors. the given neural network has N neurons, this step needs O( N ) Facebook107, Twitter629863 [23] : Those two datasets are ex- time. Finally, the proposed new algorithm has the computational tracted to form ego-networks. The ground-truth of communities complexity O( N ) which is linear to the number of nodes in large is set up in accordance with the protocol as in [23] . Circles in graphs. those two datasets are defined as ground-truth. Thus, the number of groups on those two datasets is set to 7 and 9, respectively. 4. Experiments 4.2. Baselines for comparison Here we give the comparisons between our algorithm and some state-of-the-art community detection algorithms on a wealth of The baselines can be divided into three types: (1) make use of real-world networks. There are also some detailed descriptions network structural information alone, (2) consider node contents on the baseline methods, networked datasets and experimental alone, and (3) utilize both those two types of information. setups. In the first class, CNM [24] and Louvian [25] , which adopt the same greedy optimization as did by modularity maximization 4.1. Datasets description model, are used to partition network structure. Besides, Infomap [26] , Linkcommn [27] , Clique [13] and BigCLAM [28] are popu- We test the performance of our method on nine publicly- lar overlapping community detection methods also using network available datasets in which each dataset is described as fol- structure. lows, and the key information of those datasets are listed in The second type focuses on modeling the contents on nodes, ig- Table 2 . noring the network structure. In this class, we chose four methods Citeseer [22] : This is a citation network of computer science which have their own characteristics, i.e., Affinity Propagation (AP) publications that are labeled as six sub-fields, and thus we set the [29] and DP [30] are state-of-the-art non-parameter clustering al- number of communities 6. In our predefined graph, nodes stand gorithms, LDA [31] is a well-known topic model for data clustering for publications and undirected edges indicate the citation rela- algorithm, and MAC [14] is an overlapping community detection tionships. Content information on each node is stemmed words approach. from the research paper, which is represented as a 3703 dimen- The third class of methods uses both network structure and sions’ binary vector for a document. To be specific, in the word node contents. They consist of two different types, this is the J. Cao et al. / Neurocomputing 297 (2018) 71–81 77 overlapping and non-overlapping methods respectively. There are of sets C ∗ and C¯ can be obtained by two overlapping algorithms which are CESNA [19] and DCM [32] . τ We also consider PCL-DC [33] and Block-LDA [12] , which are non- ∗ 1 ∗ Jaccard C , C¯ = I min ( πi (C ) ) = min πi C¯ (17) overlapping approaches. τ i =1 Finally, we also consider some autoencoder-based methods for π π π τ comparison. We chose AE-link approach which is our method but where 1 , 2 ,…, τ are permutations drawn randomly from a using links alone, and AE-content approach which is also our ap- family of min-wise independent permutations defined on the uni- ∗ ¯ proach but using node contents alone. To be specific, for AE-link verse C and C belong to, and I is the identity function. we only set the modularity matrix as the input of the autoencoder, while AE-content is a normalized cut-based method optimized via 4.4. Parameters setting autoencoder. Many proposed methods considering both the topological and

4.3. Description for metrics content information often rely on a balance factor to control the effectiveness of incorporating those two kinds of information. T λ In this paper, we focus on community detection with the pre- In our input matrix [M, B] , we have a parameter , which is < λ < deﬁned number of communities. We assume that there are C com- set to 0 1, to balance the proportion of those two types of λ λ T λ munities. So the community detection problem is simpliﬁed to di- information, i.e. [(1- ) M, B] . With different values of , we vide the vertices into C different groups based on the network run 10 groups of the experiments on three datasets mentioned topology as well as content information. And thus, the community above, and get the accuracy curve with error bar. The results are detection results are evaluated by measuring the degree of how the shown in Fig. 3. λ detected labels are consistent with the ground-truth communities. As we can see, when is set up to 0 or 1, the performance of

In the experiments, two sets of labels which are the ground-truth our method gets a much lower accuracy than that of our method λ ∈ partitions C ∗ and the detected partitions C¯ are considered. Because with (0, 1). This conﬁrms that, considering both topological both overlapping and non-overlapping approaches exist in the and content information we can give a clearer view of commu- baselines, we quantify the performance of the algorithms by us- nity structure and help to achieve a big improvement for commu- λ ing two types of performance metrics. One type includes normal- nity detection. Furthermore, with the variation of in the area of ized mutual information ( NMI ) and the accuracy ( AC ) for evaluating [0.1, 0.9], the ratio between topological and content information is disjoint communities, and another type contains Jaccard similarity always variable, but the accuracy curve shows very small ﬂuctu-

( Jaccard ) and F-scores for evaluating overlapping communities. ations. This also suggests that, the deep autoencoder provides a powerful capability for automatically tuning the balance factor. It 1) NMI [15] : there are two sets of labels which are the ground- may further mean that the encoding of the hidden layer in an au- ∗ truth labels C and the detected labels C¯ respectively. The enti- toencoder block can capture the most important information from

ties cij of the matrix C denote the number of nodes belonging the topological and content information, instead of collecting all ∗ to group i of set C , which are also treated as the size of group information on the whole data. Thus, our method, different from ∗ j of set C¯ . So, the measure NMI(C , C¯ ) can be written as others also using those two types of information, obtains better performance without adjusting the balance factor, and realizes the ∗ c c − · cC C¯ · ij 2 = = cij log balance factor self-tuning. For this reason, we set λ = 0.5 in our ex- ∗ i 1 j 1 ci ·c· j NMI C , C¯ = (14) c ∗ c · c ¯ c· j periments in general. C c · · log i + C c · · log i =1 i N j=1 j N As shown in Fig. 2 , we run Kmeans algorithm on the low- ∗ ∗ dimensional encoding H to perform community detection. In gen- where the set C (C¯) includes cC (c ¯ ) communities. The value C eral, many existing community detection approaches (especially of NMI(C ∗, C¯ ) gets 1, when the detected labels C¯ are those in the machine learning area) make the automatic determi- agreement with the ground-truth labels C ∗. Conversely, we ob- nation of the number of communities as a separately solved prob- tain NMI(C ∗, C¯ ) = 0 . lem, i.e. model selection, which has been discussed in many re- 2) AC [15] : we also introduce a simple measure for evaluate the searches [37,38,39] . And also they require a given number ( C ) for performance of algorithms: the communities, so did the proposed new method in this work. | ∗| − ¯ ∗ C C AC C , C¯ = 1 − (15) |C ∗| 4.5. Setups in DNN structure which is the relative error in predicting the number of commu- We implement the deep autoencoder structure with 2, 3, 4 and nities. 5 layers on all datasets for detecting communities. The perfor- 3) F-scores [16] : we quantify the F-scores between C ∗ and C¯ in term mance is shown in Fig. 4 . In generally, the encoding of the deep of measuring the error produced by using the partition C¯ to autoencoder structure with 3 layers obtains a better performance predict the partition C ∗. So the F-scores is: of community detection than that of the deep autoencoder struc- ∗, ¯ · ∗, ¯ ture with 2 layers. This means that the deep structure plays an im- ∗ precesion C C recall C C F − scores C , C¯ = 2 · (16) portant role in yielding a good encoding for clustering. However, ∗, ¯ + ∗, ¯ precesion C C recall C C we also found that, in the Washington dataset, the encoding of

Like the text classiﬁcation tasks, here we treat C ∗ as a set of the deep autoencoder structure with more layers got a lower NMI

“relevant” partitions, and C¯ as a set of “retrieved” partitions, so the value. This may because, when the dimension of data decreases precision and recall are deﬁned as: with the increase of the deep level of this structure, some important information hold in the data will be lost, causing the perfor- ∗ ∗ ∩ ¯ ∩ ¯ ∗ C C ∗ C C mance decrease. And thus, the deep autoencoder structure in this , ¯ = , , ¯ = . precision C C ∗ recall C C |C | C¯ work utilize different number of stacked autoencoders for different real-world networks. Jaccard [18] : considering the expensive computation, we esti- Table 3 demonstrates, on all datasets, the node number in each mate it as did in [19] . An unbiased estimator of Jaccard similarity conﬁguration in the deep autoencoders. There are 2 ∼ 4 layers 78 J. Cao et al. / Neurocomputing 297 (2018) 71–81

Fig. 3. The performance of community detection with variation of the parameter λ for our algorithms on three real-world networks (averaged over 10 repeated experiments).

Fig. 4. Community detection performance in the deep autoencoder structure with 2, 3, 4 and 5 layers on the nine real-world networks.

Table 3 treated the low-dimensional representation in the hidden layer of

Neural network structures. a deep autoencoder as the output. Datasets Layer conﬁguration 4.6. Experimental results Texas 374–256 Wisconsin 530–256 Twitter629863 342–256–128 On the datasets, the results measured by four types of eval- Facebook107 2090–1024–512 uation metrics are listed in Tables 4–9 . The overlapping results Cora 5416–4096–2,048 in Tables 4 and 5 show the comparison of our method and oth- Uai2010 6726–4096–2048 ers in terms of the overlapping communities, and comparison in Washington 460–256–128–64

Citeseer 6624–4096–2048–1024 terms of non-overlapping results are listed in Tables 6–9. In addi- PubMed 39458–8192–8192–4096 tion, the average performance of methods is also shown in these tables. As we all know, neural networks with the deep structure have a powerful learning ability. In Tables 5 and 6 , MAC method has stacked autoencoders in the build-in deep framework, and the better results in most cases than those obtained by AE-link or number of nodes in each hidden layer is set to half of that of its in- AE-content methods. This may because, the clustering results, ob- put or output layer. Besides, we set sigmoid function as activations tained by the non-overlapping community detection methods (e.g., of each layer in the deep autoencoders. For training the model, we AE-link or AE-content), got bad accuracy when using the overlap- adopted 10 random initializations in the deep autoencoders, and ping evaluation metrics. Our methods did not outperform all other J. Cao et al. / Neurocomputing 297 (2018) 71–81 79

Table 4 Comparison of our method and some overlapping methods in terms of F-scores on the nine datasets. Here, bold means the best result, and asterisk corresponds to the second-best result.