DOCSLIB.ORG

Detecting Statistically Significant Communities

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Detecting Statistically Significant Communities 1 Detecting Statistically Significant Communities Zengyou He, Hao Liang, Zheng Chen, Can Zhao, Yan Liu Abstract—Community detection is a key data analysis problem across different fields. During the past decades, numerous algorithms have been proposed to address this issue. However, most work on community detection does not address the issue of statistical significance. Although some research efforts have been made towards mining statistically significant communities, deriving an analytical solution of p-value for one community under the configuration model is still a challenging mission that remains unsolved. The configuration model is a widely used random graph model in community detection, in which the degree of each node is preserved in the generated random networks. To partially fulfill this void, we present a tight upper bound on the p-value of a single community under the configuration model, which can be used for quantifying the statistical significance of each community analytically. Meanwhile, we present a local search method to detect statistically significant communities in an iterative manner. Experimental results demonstrate that our method is comparable with the competing methods on detecting statistically significant communities. Index Terms—Community Detection, Random Graphs, Configuration Model, Statistical Significance. F 1 INTRODUCTION ETWORKS are widely used for modeling the structure function that is able to always achieve the best performance N of complex systems in many fields, such as biology, in all possible scenarios. engineering, and social science. Within the networks, some However, most of these objective functions (metrics) do vertices with similar properties will form communities that not address the issue of statistical significance of commu- have more internal connections and less external links. nities. In other words, how to judge whether one com- Community detection is one of the most important tasks munity or a network partition is real or not based on in network science and data mining, which can reveal some rigorous statistical significance testing procedures. the hierarchy and organization of network structures. The Such testing-based approaches provide many advantages detection of meaningful communities is of great importance over other metrics. First of all, the statistical significance in many real applications [1]. In these diverse domains, each of communities can be quantified in terms of the p-value, detected community may provide us with some interesting which is a universally understood measure between 0 and 1 yet unknown information. in different fields. As a result, the threshold for the p-value Due to the importance of the community detection prob- is easy to specify since it corresponds to the significance lem, numerous algorithms from different perspectives have level. In contrast, numerical values generated from other been proposed during the past decades [1], [2]. Although objective functions are generally data-dependent, which is existing community detection methods have very different hard for people to interpret and determine a universal procedures to detect community structures, these methods threshold across all data sets. Furthermore, the p-value is can be roughly classified into different categories accord- typically derived under a certain random graph model in ing to the objective function and the corresponding search a mathematically sound manner, while many other metrics procedure [3]. The objective function is used for evaluating may be just defined in an ad-hoc manner. the quality of candidate communities, which serves as a In many real applications, it is critical to find network guideline for the search procedure and is critical to the suc- communities that are statistically significant. For instance, arXiv:1806.05602v2 [cs.SI] 11 Aug 2019 cess of the entire community detection algorithm. Probably the detection of protein complexes from protein-protein the modularity proposed by Newman and Girvan [4] is the interaction (PPI) networks corresponds to a special com- most popular objective function in the field of community munity detection problem [8]. However, the PPI networks detection. Based on modularity, some popular community derived from current high-throughput experimental tech- detection methods, such as Louvain [5] and FastGreedy [6], niques are very noisy, in which many spurious edges are have been proposed to search the communities in networks. present while some true edges are missing [9], [10]. More im- In addition, many other objective functions have been pro- portantly, some reported protein complexes (communities in posed to evaluate the goodness of communities as well, as PPI networks) need to be further validated by wet-lab exper- summarized in [7]. Overall, there is still no such an objective iments that are costly and time-consuming. Thus, in these types of applications, the quality of identified communities must be strictly controlled through rigourous hypothesis • Z. He is with School of Software, Dalian University of Technology, testing procedures in order to obtain really meaningful out- Dalian,China, and Key Laboratory for Ubiquitous Network and Service put (e.g. [11], [12], [13]). In fact, the issue of errors and noises Software of Liaoning Province, Dalian, China. E-mail: [email protected] in the network data has also be recognized in many other • H. Liang, Z. Chen and Y. Liu are with School of Software, Dalian fields such as social science [10]. Therefore, the detection of University of Technology, Dalian, China. statistically significant communities will be desirable as well • C. Zhao is with Institute of Information Engineering, CAS. in these fields. Furthermore, the p-value of each community 2 can be used as the metric in emerging applications such as used analytically for assessing the statistical significance network community prioritization [14]. of a single community. Then, we present a local search To address the problem of discovering statistically signif- method to detect statistically significant communities in an icant communities, some research efforts have been made iterative manner. Experimental results on both real data sets towards this direction (e.g. [15], [16], [17], [18], [19], [20], and the LFR benchmark data sets show that our method [21], [22]). In these methods, the first critical issue that is comparable with the competing methods in terms of must be solved is to define a metric to effectively assess different evaluation metrics. the statistical significance of communities. This is because The main contributions of this paper can be summarized the significance-based metric will be used as the objective as follows: function for finding statistically significant communities. Therefore, different measures have been presented for as- • We propose a new method for assessing the statis- sessing the statistical significance of communities, which can tical significance of one single community. To our be classified into two categories: the statistical significance knowledge, this is the first piece of work that de- of one network partition (e.g. [15], [16], [17], [18]) and the livers an analytical p-value (or its upper bound) for statistical significance of a single community (e.g. [19], [20], quantifying a community under the configuration [21], [22]). model. Unlike existing methods under the config- Although the metrics that evaluate a network partition uration model that quantify the statistical signifi- can provide a global view on the set of all generated cance through the membership probabilities of single communities, they generally cannot guarantee that every nodes, our method directly evaluates each candidate single community is statistically significant as well. In ad- community. dition, many different partitions of the same network may • We provide a systematic summarization and analysis lead to quite similar significance values, making it difficult on the existing methods for detecting statistically to determine which partition should be reported as the significant communities, which may serve as the final result. Furthermore, such partition-based significance foundation for the further investigation towards this metrics typically focus on the assessment of a set of non- direction. overlapping communities. Therefore, evaluating the statis- • We present a local search method to conduct the tical significance of each single community is more mean- community detection based on the proposed upper ingful in community detection. Meanwhile, these methods bound of p-value. Extensive empirical studies vali- can be also classified by the techniques for deriving the p- date the effectiveness of our method on community values: analytical methods (e.g. [15], [16], [19], [20], [21]) evaluation and detection. or sampling methods (e.g. [17], [18], [22]). The sampling The remaining parts of this paper are structured as method calculates an empirical p-value through generat- follows: Section 2 presents and discusses related work in ing a number of random graphs. Although the sampling a systematic manner. Section 3 introduces our definition method is easy to understand and implement, it is very on the statistical significance of a single community and time consuming since many random networks have to be the corresponding community detection method. Section 4 generated and analyzed. More importantly, the p-values of shows the experimental results and Section 5 concludes this the same community may be inconsistent in different runs paper. and the precision of the p-value
Load more

Recommended publications
  • Optimal Subgraph Structures in Scale-Free Configuration
    Submitted to the Annals of Applied Probability OPTIMAL SUBGRAPH STRUCTURES IN SCALE-FREE CONFIGURATION MODELS , By Remco van der Hofstad∗ §, Johan S. H. van , , Leeuwaarden∗ ¶ and Clara Stegehuis∗ k Eindhoven University of Technology§, Tilburg University¶ and Twente Universityk Subgraphs reveal information about the geometry and function- alities of complex networks. For scale-free networks with unbounded degree fluctuations, we obtain the asymptotics of the number of times a small connected graph occurs as a subgraph or as an induced sub- graph. We obtain these results by analyzing the configuration model with degree exponent τ (2, 3) and introducing a novel class of op- ∈ timization problems. For any given subgraph, the unique optimizer describes the degrees of the vertices that together span the subgraph. We find that subgraphs typically occur between vertices with specific degree ranges. In this way, we can count and characterize all sub- graphs. We refrain from double counting in the case of multi-edges, essentially counting the subgraphs in the erased configuration model. 1. Introduction Scale-free networks often have degree distributions that follow power laws with exponent τ (2, 3) [1, 11, 22, 34]. Many net- ∈ works have been reported to satisfy these conditions, including metabolic networks, the internet and social networks. Scale-free networks come with the presence of hubs, i.e., vertices of extremely high degrees. Another property of real-world scale-free networks is that the clustering coefficient (the probability that two uniformly chosen neighbors of a vertex are neighbors themselves) decreases with the vertex degree [4, 10, 24, 32, 34], again following a power law.
    [Show full text]
  • Processes on Complex Networks. Percolation
    Chapter 5 Processes on complex networks. Percolation 77 Up till now we discussed the structure of the complex networks. The actual reason to study this structure is to understand how this structure influences the behavior of random processes on networks. I will talk about two such processes. The first one is the percolation process. The second one is the spread of epidemics. There are a lot of open problems in this area, the main of which can be innocently formulated as: How the network topology influences the dynamics of random processes on this network. We are still quite far from a definite answer to this question. 5.1 Percolation 5.1.1 Introduction to percolation Percolation is one of the simplest processes that exhibit the critical phenomena or phase transition. This means that there is a parameter in the system, whose small change yields a large change in the system behavior. To define the percolation process, consider a graph, that has a large connected component. In the classical settings, percolation was actually studied on infinite graphs, whose vertices constitute the set Zd, and edges connect each vertex with nearest neighbors, but we consider general random graphs. We have parameter ϕ, which is the probability that any edge present in the underlying graph is open or closed (an event with probability 1 − ϕ) independently of the other edges. Actually, if we talk about edges being open or closed, this means that we discuss bond percolation. It is also possible to talk about the vertices being open or closed, and this is called site percolation.
    [Show full text]
  • Centrality in Modular Networks
    Ghalmane et al. EPJ Data Science (2019)8:15 https://doi.org/10.1140/epjds/s13688-019-0195-7 REGULAR ARTICLE OpenAccess Centrality in modular networks Zakariya Ghalmane1,3, Mohammed El Hassouni1, Chantal Cherifi2 and Hocine Cherifi3* *Correspondence: hocine.cherifi@u-bourgogne.fr Abstract 3LE2I, UMR6306 CNRS, University of Burgundy, Dijon, France Identifying influential nodes in a network is a fundamental issue due to its wide Full list of author information is applications, such as accelerating information diffusion or halting virus spreading. available at the end of the article Many measures based on the network topology have emerged over the years to identify influential nodes such as Betweenness, Closeness, and Eigenvalue centrality. However, although most real-world networks are made of groups of tightly connected nodes which are sparsely connected with the rest of the network in a so-called modular structure, few measures exploit this property. Recent works have shown that it has a significant effect on the dynamics of networks. In a modular network, a node has two types of influence: a local influence (on the nodes of its community) through its intra-community links and a global influence (on the nodes in other communities) through its inter-community links. Depending on the strength of the community structure, these two components are more or less influential. Based on this idea, we propose to extend all the standard centrality measures defined for networks with no community structure to modular networks. The so-called “Modular centrality” is a two-dimensional vector. Its first component quantifies the local influence of a node in its community while the second component quantifies its global influence on the other communities of the network.
    [Show full text]
  • Percolation Thresholds for Robust Network Connectivity
    Percolation Thresholds for Robust Network Connectivity Arman Mohseni-Kabir1, Mihir Pant2, Don Towsley3, Saikat Guha4, Ananthram Swami5 1. Physics Department, University of Massachusetts Amherst, [email protected] 2. Massachusetts Institute of Technology 3. College of Information and Computer Sciences, University of Massachusetts Amherst 4. College of Optical Sciences, University of Arizona 5. Army Research Laboratory Abstract Communication networks, power grids, and transportation networks are all examples of networks whose performance depends on reliable connectivity of their underlying network components even in the presence of usual network dynamics due to mobility, node or edge failures, and varying traffic loads. Percolation theory quantifies the threshold value of a local control parameter such as a node occupation (resp., deletion) probability or an edge activation (resp., removal) probability above (resp., below) which there exists a giant connected component (GCC), a connected component comprising of a number of occupied nodes and active edges whose size is proportional to the size of the network itself. Any pair of occupied nodes in the GCC is connected via at least one path comprised of active edges and occupied nodes. The mere existence of the GCC itself does not guarantee that the long-range connectivity would be robust, e.g., to random link or node failures due to network dynamics. In this paper, we explore new percolation thresholds that guarantee not only spanning network connectivity, but also robustness. We define and analyze four measures of robust network connectivity, explore their interrelationships, and numerically evaluate the respective robust percolation thresholds arXiv:2006.14496v1 [physics.soc-ph] 25 Jun 2020 for the 2D square lattice.
    [Show full text]
  • Incorporating Network Structure with Node Contents for Community Detection on Large Networks Using Deep Learning
    Neurocomputing 297 (2018) 71–81 Contents lists available at ScienceDirect Neurocomputing journal homepage: www.elsevier.com/locate/neucom Incorporating network structure with node contents for community detection on large networks using deep learning ∗ Jinxin Cao a, Di Jin a, , Liang Yang c, Jianwu Dang a,b a School of Computer Science and Technology, Tianjin University, Tianjin 300350, China b School of Information Science, Japan Advanced Institute of Science and Technology, Ishikawa 923-1292, Japan c School of Computer Science and Engineering, Hebei University of Technology, Tianjin 300401, China a r t i c l e i n f o a b s t r a c t Article history: Community detection is an important task in social network analysis. In community detection, in gen- Received 3 June 2017 eral, there exist two types of the models that utilize either network topology or node contents. Some Revised 19 December 2017 studies endeavor to incorporate these two types of models under the framework of spectral clustering Accepted 28 January 2018 for a better community detection. However, it was not successful to obtain a big achievement since they Available online 2 February 2018 used a simple way for the combination. To reach a better community detection, it requires to realize a Communicated by Prof. FangXiang Wu seamless combination of these two methods. For this purpose, we re-examine the properties of the mod- ularity maximization and normalized-cut models and fund out a certain approach to realize a seamless Keywords: Community detection combination of these two models. These two models seek for a low-rank embedding to represent of the Deep learning community structure and reconstruct the network topology and node contents, respectively.
    [Show full text]
  • Review of Community Detection Over Social Media: Graph Prospective
    (IJACSA) International Journal of Advanced Computer Science and Applications, Vol. 10, No. 2, 2019 Review of Community Detection over Social Media: Graph Prospective Pranita Jain1, Deepak Singh Tomar2 Department of Computer Science Maulana Azad National Institute of Technology Bhopal, India 462001 Abstract—Community over the social media is the group of globally distributed end users having similar attitude towards a particular topic or product. Community detection algorithm is used to identify the social atoms that are more densely interconnected relatively to the rest over the social media platform. Recently researchers focused on group-based algorithm and member-based algorithm for community detection over social media. This paper presents comprehensive overview of community detection technique based on recent research and subsequently explores graphical prospective of social media mining and social theory (Balance theory, status theory, correlation theory) over community detection. Along with that this paper presents a comparative analysis of three different state of art community detection algorithm available on I-Graph package on python i.e. walk trap, edge betweenness and fast greedy over six different social media data set. That yield intersecting facts about the capabilities and deficiency of community analysis methods. Fig 1. Social Media Network. Keywords—Community detection; social media; social media Aim of Community detection is to form group of mining; homophily; influence; confounding; social theory; homogenous nodes and figure out a strongly linked subgraphs community detection algorithm from heterogeneous network. In strongly linked sub- graphs (Community structure) nodes have more internal links than I. INTRODUCTION external. Detecting communities in heterogeneous networks is The Emergence of Social networking Site (SNS) like Face- same as, the graph partition problem in modern graph theory book, Twitter, LinkedIn, MySpace, etc.
    [Show full text]
  • Closeness Centrality for Networks with Overlapping Community Structure
    Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI-16) Closeness Centrality for Networks with Overlapping Community Structure Mateusz K. Tarkowski1 and Piotr Szczepanski´ 2 Talal Rahwan3 and Tomasz P. Michalak1,4 and Michael Wooldridge1 2Department of Computer Science, University of Oxford, United Kingdom 2 Warsaw University of Technology, Poland 3Masdar Institute of Science and Technology, United Arab Emirates 4Institute of Informatics, University of Warsaw, Poland Abstract One aspect of networks that has been largely ignored in the literature on centrality is the fact that certain real-life Certain real-life networks have a community structure in which communities overlap. For example, a typical bus net- networks have a predefined community structure. In public work includes bus stops (nodes), which belong to one or more transportation networks, for example, bus stops are typically bus lines (communities) that often overlap. Clearly, it is im- grouped by the bus lines (or routes) that visit them. In coau- portant to take this information into account when measur- thorship networks, the various venues where authors pub- ing the centrality of a bus stop—how important it is to the lish can be interpreted as communities (Szczepanski,´ Micha- functioning of the network. For example, if a certain stop be- lak, and Wooldridge 2014). In social networks, individuals comes inaccessible, the impact will depend in part on the bus grouped by similar interests can be thought of as members lines that visit it. However, existing centrality measures do not of a community. Clearly for such networks, it is desirable take such information into account. Our aim is to bridge this to have a centrality measure that accounts for the prede- gap.
    [Show full text]
  • A Centrality Measure for Networks with Community Structure Based on a Generalization of the Owen Value
    ECAI 2014 867 T. Schaub et al. (Eds.) © 2014 The Authors and IOS Press. This article is published online with Open Access by IOS Press and distributed under the terms of the Creative Commons Attribution Non-Commercial License. doi:10.3233/978-1-61499-419-0-867 A Centrality Measure for Networks With Community Structure Based on a Generalization of the Owen Value Piotr L. Szczepanski´ 1 and Tomasz P. Michalak2 ,3 and Michael Wooldridge2 Abstract. There is currently much interest in the problem of mea- cated — approaches have been considered in the literature. Recently, suring the centrality of nodes in networks/graphs; such measures various methods for the analysis of cooperative games have been ad- have a range of applications, from social network analysis, to chem- vocated as measures of centrality [16, 10]. The key idea behind this istry and biology. In this paper we propose the first measure of node approach is to consider groups (or coalitions) of nodes instead of only centrality that takes into account the community structure of the un- considering individual nodes. By doing so this approach accounts for derlying network. Our measure builds upon the recent literature on potential synergies between groups of nodes that become visible only game-theoretic centralities, where solution concepts from coopera- if the value of nodes are analysed jointly [18]. Next, given all poten- tive game theory are used to reason about importance of nodes in the tial groups of nodes, game-theoretic solution concepts can be used to network. To allow for flexible modelling of community structures, reason about players (i.e., nodes) in such a coalitional game.
    [Show full text]
  • 15 Netsci Configuration Model.Key
    Configuring random graph models with fixed degree sequences Daniel B. Larremore Santa Fe Institute June 21, 2017 NetSci [email protected] @danlarremore Brief note on references: This talk does not include references to literature, which are numerous and important. Most (but not all) references are included in the arXiv paper: arxiv.org/abs/1608.00607 Stochastic models, sets, and distributions • a generative model is just a recipe: choose parameters→make the network • a stochastic generative model is also just a recipe: choose parameters→draw a network • since a single stochastic generative model can generate many networks, the model itself corresponds to a set of networks. • and since the generative model itself is some combination or composition of random variables, a random graph model is a set of possible networks, each with an associated probability, i.e., a distribution. this talk: configuration models: uniform distributions over networks w/ fixed deg. seq. Why care about random graphs w/ fixed degree sequence? Since many networks have broad or peculiar degree sequences, these random graph distributions are commonly used for: Hypothesis testing: Can a particular network’s properties be explained by the degree sequence alone? Modeling: How does the degree distribution affect the epidemic threshold for disease transmission? Null model for Modularity, Stochastic Block Model: Compare an empirical graph with (possibly) community structure to the ensemble of random graphs with the same vertex degrees. e a loopy multigraphs c d simple simple loopy graphs multigraphs graphs Stub Matching tono multiedgesdraw from theno config. self-loops model ~k = 1, 2, 2, 1 b { } 3 3 4 3 4 3 4 4 multigraphs 2 5 2 5 2 5 2 5 1 1 6 6 1 6 1 6 3 3 the 4standard algorithm:4 3 4 4 loopy and/or 5 5 draw2 from5 the distribution2 5 by sequential2 “Stub Matching”2 1 1 6 6 1 6 1 6 1.
    [Show full text]
  • Structural Diversity and Homophily: a Study Across More Than One Hundred Big Networks
    KDD 2017 Research Paper KDD’17, August 13–17, 2017, Halifax, NS, Canada Structural Diversity and Homophily: A Study Across More Than One Hundred Big Networks Yuxiao Dong∗ Reid A. Johnson Microso Research University of Notre Dame Redmond, WA 98052 Notre Dame, IN 46556 [email protected] [email protected] Jian Xu Nitesh V. Chawla University of Notre Dame University of Notre Dame Notre Dame, IN 46556 Notre Dame, IN 46556 [email protected] [email protected] ABSTRACT ACM Reference format: A widely recognized organizing principle of networks is structural Yuxiao Dong, Reid A. Johnson, Jian Xu, and Nitesh V. Chawla. 2017. Struc- tural Diversity and Homophily: A Study Across More an One Hundred homophily, which suggests that people with more common neigh- Big Networks. In Proceedings of KDD ’17, August 13–17, 2017, Halifax, NS, bors are more likely to connect with each other. However, what in- Canada., , 10 pages. uence the diverse structures embedded in common neighbors (e.g., DOI: hp://dx.doi.org/10.1145/3097983.3098116 and ) have on link formation is much less well-understood. To explore this problem, we begin by characterizing the structural diversity of common neighborhoods. Using a collection of 120 large- 1 INTRODUCTION scale networks, we demonstrate that the impact of the common neighborhood diversity on link existence can vary substantially Since the time of Aristotle, it has been observed that people “love across networks. We nd that its positive eect on Facebook and those who are like themselves” [2]. We now know this as the negative eect on LinkedIn suggest dierent underlying network- principle of homophily, which asserts that people’s propensity to ing needs in these networks.
    [Show full text]
  • A Multigraph Approach to Social Network Analysis
    1 Introduction Network data involving relational structure representing interactions between actors are commonly represented by graphs where the actors are referred to as vertices or nodes, and the relations are referred to as edges or ties connecting pairs of actors. Research on social networks is a well established branch of study and many issues concerning social network analysis can be found in Wasserman and Faust (1994), Carrington et al. (2005), Butts (2008), Frank (2009), Kolaczyk (2009), Scott and Carrington (2011), Snijders (2011), and Robins (2013). A common approach to social network analysis is to only consider binary relations, i.e. edges between pairs of vertices are either present or not. These simple graphs only consider one type of relation and exclude the possibility for self relations where a vertex is both the sender and receiver of an edge (also called edge loops or just shortly loops). In contrast, a complex graph is defined according to Wasserman and Faust (1994): If a graph contains loops and/or any pairs of nodes is adjacent via more than one line the graph is complex. [p. 146] In practice, simple graphs can be derived from complex graphs by collapsing the multiple edges into single ones and removing the loops. However, this approach discards information inherent in the original network. In order to use all available network information, we must allow for multiple relations and the possibility for loops. This leads us to the study of multigraphs which has not been treated as extensively as simple graphs in the literature. As an example, consider a network with vertices representing different branches of an organ- isation.
    [Show full text]
  • Critical Window for Connectivity in the Configuration Model 3
    CRITICAL WINDOW FOR CONNECTIVITY IN THE CONFIGURATION MODEL LORENZO FEDERICO AND REMCO VAN DER HOFSTAD Abstract. We identify the asymptotic probability of a configuration model CMn(d) to produce a connected graph within its critical window for connectivity that is identified by the number of vertices of degree 1 and 2, as well as the expected degree. In this window, the probability that the graph is connected converges to a non-trivial value, and the size of the complement of the giant component weakly converges to a finite random variable. Under a finite second moment con- dition we also derive the asymptotics of the connectivity probability conditioned on simplicity, from which the asymptotic number of simple connected graphs with a prescribed degree sequence follows. Keywords: configuration model, connectivity threshold, degree sequence MSC 2010: 05A16, 05C40, 60C05 1. Introduction In this paper we consider the configuration model CMn(d) on n vertices with a prescribed degree sequence d = (d1,d2, ..., dn). We investigate the condition on d for CMn(d) to be with high probability connected or disconnected in the limit as n , and we analyse the behaviour of the model in the critical window for connectivity→ ∞ (i.e., when the asymptotic probability of producing a connected graph is in the interval (0, 1)). Given a vertex v [n]= 1, 2, ..., n , we call d its degree. ∈ { } v The configuration model is constructed by assigning dv half-edges to each vertex v, after which the half-edges are paired randomly: first we pick two half-edges at arXiv:1603.03254v1 [math.PR] 10 Mar 2016 random and create an edge out of them, then we pick two half-edges at random from the set of remaining half-edges and pair them into an edge, etc.
    [Show full text]