Bayesian Model Selection of Stochastic Block Models

Xiaoran Yan Indiana University Network Science Institute Bloomington, Indiana 47408 Email: [email protected]

Abstract—A central problem in analyzing networks is parti- the right model (model selection), and in particular picking the tioning them into modules or communities. One of the best tools right number of blocks (order selection) is crucial for success- for this is the stochastic block model, which clusters vertices into ful network modeling. Numerous statistical model selection blocks with statistically homogeneous pattern of links. Despite its flexibility and popularity, there has been a lack of principled techniques have been developed for classic independent data. statistical model selection criteria for the stochastic block model. Unfortunately, it has been a common mistake to use these Here we propose a Bayesian framework for choosing the number techniques in network models without rigorous examinations of blocks as well as comparing it to the more elaborate degree- of their compatibility, ignoring the fact that some of the corrected block models, ultimately leading to a universal model fundamental assumptions has been violated by moving into the selection framework capable of comparing multiple modeling combinations. We will also investigate its connection to the domain of relational data [12], [13], [14]. As a result, some minimum description length principle. employed these information criteria directly with out knowing the consequences [4], [15], while others remain skeptical and I.INTRODUCTION use it only when no alternatives is available [9], [16]. An important task in network analysis is community detec- Our main contribution in this paper is to develop a Bayesian tion, or finding groups of similar vertices which can then be model selection framework for comparing multiple SBM vari- analyzed separately [1]. Community structures offer clues to ants with different number of blocks1. In Section 2, we will the processes which generated the graph, on scales ranging first go over the Bayesian model selection approaches and the from face-to-face social interaction [2] through social-media minimum description length (MDL) principle [17], [18]. In communications [3] to the organization of food webs [4]. Section 3, we will propose Bayesian order selection for the However, previous work often defines a “community” as a SBM, leading to a Bayesian Information criterion (BIC) [19]. group of vertices with high density of connections within the We will also establish the equivalence between BIC and MDL. group and a low density of connections to the rest of the In section 4, we will generalize these results to the DC-SBM, network. While this type of assortative community structure leading to a universal model selection framework capable of is generally the case in social networks, we are interested in comparing multiple models that combines different model and a more general definition of functional community—a group order choices. We will compare its theoretic and empirical of vertices that connect to the rest of the network in similar results to previous MDL based approaches [18], [20] as well ways. A set of similar predators form a functional group in a as previous work on frequentist model selection [21]. food web, not because they eat each other, but because they feed on similar prey. In English, nouns often follow adjectives, II.BACKGROUNDANDRELATEDWORK but seldom follow other nouns. Model selection is about balancing the trade-off between The stochastic block model (SBM) is a popular network model complexity and fit to the data. Models with more model for such functional communities [5]. It splits vertices parameters have a natural advantage at fitting data. Simpler arXiv:1605.07057v1 [stat.ML] 23 May 2016 into latent blocks, within which all vertices are stochastically models have lower variability, and are less sensitive to noise equivalent in terms of how they connect to the rest of the in the data. A good model choice should avoid both over-fitting network [6]. As a generative model, it has a well-defined and under-fitting, and only include additional parameters when likelihood function with consistent parameter estimates. It they do capture meaningful structures in the data [19], [22]. also provides a flexible base for more general latent state The frequentist approach to model selection cast the prob- models. In particular, Karrer and Newman proposed a variant lem as a hypothesis testing. It focus on estimating the called the degree-corrected block models (DC-SBM) capable likelihood-ratio between a pair of candidate models under the of capturing specific degree distributions and Yaojia et al. null hypothesis. Such frequentist method has been used for generalized it to directed graphs [7], [8]. There are also other variants capable of modeling overlapping, hierarchical and 1We will focus on the model/order selection problems of the SBM and even “meta-data” dependent communities [9], [10], [11]. Degree-corrected block model. For mathematical convenience, we will define these models for undirected graphs. Directed versions require additional Performance of different latent state models vary under specifications but the general form remains the same [8]. Readers should different scenarios. For SBM and its generalizations, picking be able to generalize the result in this paper to these cases. both model selection and order selection [21], [23]. In this networks [15]. Recently, Comeˆ and Latouche derived a BIC paper, we shall follow the other major school of statistics and from the ICL for the vanilla SBMs [29]. We will redo the use Bayesian techniques for model selection. derivation in our notions and generalize it to degree-corrected block models in this paper. A. Bayesian model selection While point estimates like the maximum likelihood lead C. The minimum description length principle to over-fitting, Bayesian approaches take the whole posterior By compressing data with different coding schemes, infor- distribution into account, thus achieving the trade-off between mation theory has a long history dealing with the trade-off model complexity and its fit to data [19], [22]. These posteriors between complexity and fit. Searching for the model with best distributions can be formulated using Bayes’ rule, predictive performance is essentially finding a coding scheme P (M ) that lead to the minimum description length (MDL) [17], [30]. P (M | G) = i P (G | M ) i P (G) i Under the MDL principle, the trade-off takes the from of ZZZ 1 balancing between the description length of the coding scheme ∝ P (G | Mi, Πi)P (Πi | Mi)dΠi , (1) and that of the message body given the code [18], [31]. 0 MDL is closely related with Bayesian model selection, where we have assumed a uniform prior of models P (Mi), and particularly the BIC formulation [17]. In [20], [32], Peixoto the total evidence of data P (G) is constant for all models. demonstrated that for basic SBMs, the ICL or Bayesian The posterior P (Mi | G) has an intuitive interpretation for posteriors are mathematically equivalent to MDL criteria under model selection. It is proportional to P (G | Mi), which is the certain model constrains. This equivalence, as we will show integrated complete likelihood (as ICL in [24]) P (G | Mi, Πi) in this paper, underlies a fundamental connection in the form over the prior of parameters P (Πi | Mi). To compare models, of carefully designed Bayesian codes, which can be derived the standard approach is to divide one posterior with another from the ICL (1) with intuitions. in a likelihood-ratio style, leading to the Bayes factor [25]. Unlike the frequentist tests, Bayes factor uses the ICL without III.BAYESIANORDERSELECTIONOFTHE SBM any dependence on parameters. Therefore, it can be applied In this section, we will formally introduce the SBM, derive not only to nested model pairs, but two models of any form. the ICL for the SBM order selection problem, and design Ironically, instead of one preferred choice, a fully Bayesian an intuitive Bayesian code to demonstrate the fundamental approach would give a posterior distribution over all candidate connection between Bayesian order selection and the MDL. models. The maximum a posteriori (MAP) method presented Finally we will propose a Bayesian Information criterion (BIC) in this paper is technically a hybrid between Bayesian and [19] for order selection of the SBM. frequentist ideas. For comparison with frequentist and MDL based methods, we will still call it Bayesian. Nonetheless, A. The stochastic block model most results can be adapted for full Bayesian analysis. We represent our network as a simple undirected graph G = (V,E), without self-loops. G has n vertices in the set V , m B. Bayesian information criterion edges in the set E, and they can be specified by an adjacency While Bayesian model selection is compatible for models of matrix A where each entry Auv = 0 or 1 indicates if there any form, the posteriors can often be intractable. The exception is an edge in-between. We assume that there are k blocks is when the likelihood model is from a family with conjugate of vertices (choosing k is the order selection problem), so priors. The posteriors will then have closed form solutions. that each vertex u has a block label g(u) ∈ {1, . . . , k}. Here Fortunately for us, the block models fall under this category. ns = |{u ∈ V : g(u) = s}| is the number of vertices in block Bayes factor is also quite cumbersome when more than s, and mst = |{u < v, (u, v) ∈ E : g(u) = s, g(v) = t}| is the two candidate models are involved. Instead of the principled number of edges connecting between block s and block t. hypothesis testing framework, Bayesian information criterion We assume that G is generated by a SBM, or a “vanilla (BIC) [26] gives standard values for each candidate model: SBM” as we will call it throughout this paper for distinction. For each pair of vertices u, v, there is an edge between u BIC(Mi) = −2 ln P (Y | Mi, Πˆ i) + |Πi| ln n , (2) and v with the probability pg(u),g(v) specified by the k × k where |Πi| is the degree of freedom of the model Mi with a block affinity matrix p. Each vertex label g(u) is first inde- parameter set |Πi|, and n is number of i.i.d. samples in the pendently generated according to the prior probability qg(u) data. The above simple formulation consists of a maximized Pk with s=0 qs = 1. Given a block assignment, i.e., a function likelihood term and a penalty term for model complexity, in- g : V → {1, . . . , k} assigning a label to each vertex, the tuitively corresponding to the trade-off we are looking for. As probability of generating a given graph G in this model is we will later show, it is in fact a large sample approximation to twice the logarithm of the ICL (Equation (1)). P (G, g | q, p) BIC has been applied to different clustering models of i.i.d. k k Y ns Y mst nsnt−mst data [27], [24], [28]. Handcock et al. proposed a variants of = qi pst (1 − pst) , (3) BIC for order selection of a latent space clustering model on s=1 s≤t This likelihood factors into terms for vertices and edges, respective conjugate Dirichlet and Beta priors (δ and {α, β} conditioned on their parameters q, p respectively. respectively), we have Take the log of Equation (3), we have the log-likelihood ZZZ 1 P (M | G) ∝ d{p }d{q } P (G, g | q, p) k i st s X 0 log P (G, g | q, p) = ns log qs Z k ! s=1 ~ Y ns = dqDirichlet(~q|δ) qs k 4 X s=1 + (m log p + (n n − m ) log(1 − p )) . (4)   st st s t st st k Z 1 s≤t Y mst nsnt−mst  dpst Beta(pst|α, β) pst (1 − pst)  0 B. Bayesian posterior of the SBM s≤t ! Γ(Pk δ ) Qk Γ(n + δ ) A key design choice of MAP model selection is picking = s=1 s s=1 s s Qk Pk which parameters to integrate. By being partially frequentist, s=1 Γ(δs) Γ( s=1(ns + δs)) we have the freedom to decide how ”Bayesian” we wanted to  k  be. The decision can also be understood as balancing between Y Γ(α + β) Γ(mst + α) Γ(nsnt − mst + β)   Γ(α) Γ(β) Γ(nsnt + α + β) the bias in the learning task and the variance in application s≤t domains. For example, if the learning task is to find the latent =P (V, g | M ) × P (E, g | M ) , (7) state model with the most likely parametrization, regardless of i i specific latent block assignments, the integral over parameters where we have assumed the same beta prior {α, β} for all pst is not necessary, as in the paper [21]. entries, and applied the Euler integral of the first kind, and its Alternatively, if the learning task is to find the model with P multinomial generalization on the simplex s qs = 1. the most likely latent state, we do not need the sum over the Equation (7) shows, the ICL factors into terms for vertices latent state g, and benefit from the smaller variance because and edges. It holds for any latent state g, as long as ns of the bias we are willing to assume. However, if we plan and mst terms are consistent with the given g. For empirical to apply the learned latent state model to similar networks verifications, we will use a Monte Carlo sampling method to with different parameterizations, the integral over parameters find the most likely latent state gˆ. remains essential. This corresponds to the idea of Universal To get an idea of the posterior distribution P (Mi | G), we Coding in information coding theories [17], where a code has assume that the data G follows a uniform prior over random to achieve optimal average compression for all data generated graphs generated by a SBM with 5 prescribed blocks. For by the same code scheme without knowing its parametrization simplicity, we have also plugged in the uniform priors (i.e., a priori. This is the approach Handcock et al. adopted for δ∀s = 1, α = β = 1) for the parameters, as it is done in [33], order selection of their latent space model [15]. The same method was used in the active learning algorithm for vanilla P (Mi | G, g) ∝ P (G, g | Mi)   SBMs [33]. In this paper, we will follow this approach for its Qk ! k ns! Y mst!(nsnt − mst)! connection with MDL methods. We will use a Monte Carlo = (k − 1)! s=1   . (n + k − 1)! (nsnt + 1)! sampling method to find the most likely latent state gˆ. s≤t According to equation (1), we have the posterior of a SBM (8) M with the parameters {p, q}: i The distributions of the posterior with different number P (M ) of blocks are shown in Figure 1 (top). While the SBM P (M | G) = i P (G | M ) i P (G) i with correct number of blocks (red) does has slightly higher likelihood in average, it overlaps quite heavily with SBMs X ZZZ 1 ∝ d{pst}d{qi} P (G, g | p, q) , (5) with fewer blocks (green) or more blocks (blue). Further g 0 investigation reveals that most of the variance came from the randomness in the generated data. Once we fix the input graph where we assume the prior of models P (M ) is uniform, and i for all the candidate models, the SBM with correct number the total evidence of data P (G) is constant. Here the ICL of blocks always has a higher likelihood than the others, as P (G | M ) is integrated over both p and q entries, as well as i illustrated in Figure. 1 (bottom). summed over all latent states g. Since we are interested in the most likely latent state gˆ, we can forgo the sum, C. Bayesian code for the SBM ZZZ 1 Now we will design a Bayesian code based on the ICL. Ac- P (G, g | Mi) = d{pst}d{qs} P (G, g | q, p) . (6) cording to Grunwald¨ [17], the equivalence between Bayesian 0 and the MDL principle for model selection holds in general. In with P (G, gˆ | Mi) = maxg P (G, g | Mi). our case, if we choose the Jeffreys priors for pst and qs entries If we assume that the pst and qs entries are independent, (i.e., α = β = δ∀s = 1/2), the coding length according to P conditioned on the constrain s qs = 1, and they follows their the Bayesian model is asymptotically the same as the optimal Histogram 5) for each pair of blocks s, t, code for the edge allocations nsnt
given mst (log bits, uniformly random alloca- 0.0012 mst tions) 0.0010 According to [17], a mapping exists between probability 0.0008 distributions and prefix codes. In the Bayesian code, the distri- 0.0006 bution of possible realizations in part i (i > 1) conditioned on 0.0004 all previous code parts are all uniform, the optimal code length 0.0002 for part i thus can be quantified by the negative logarithm of Log-likelihood the corresponding combinatorial terms in Equation (9). -66 000 -65 500 -65 000 -64 500 -64 000 -63 500 -63 000 Log-likelihood The aforementioned Bayesian code is identical to the de- æ æ æ æ æ æ -64 000 æ æ æ æ æ æ scription length for the single level model discussed in [20]. In æ æ æ -64 500 fact, Peixoto mathematically arrived at the same equivalence in æ æ the appendix of [20]. Earlier formulations of MDLs, however, -65 000 æ æ are usually defined in terms of the entropy minimizers of -65 500 the likelihood functions, which is equivalent to the maximum -66 000 likelihood formulation in the BIC (Equation (2)).

-66 500 D. BIC for order selection of SBMs -67 000 æ æ ð of blocks æ 2 4 6 8 10 The key to transform Equation (7) to the BIC formulation (Equation (2)) is the Laplace’s approximation with uniform Fig. 1. Top: the histogram of log-integrated complete likelihoods (log-ICL) priors. Or equivalently, by using Stirling’s formula on the of the Bayesian model (Equation (8)). The distributions are gathered from factorials in Equation (9), randomly generated SBMs with 1000 vertices and 5 prescribed blocks. SBMs with different number of blocks k are fitted to the data. Specifically, the green P (G, g | Mi) = P (V, g | Mi) × P (E, g | Mi) distribution is from a SBM with k = 4, the red with k = 5 and the blue Qk √ k p with k = 9. The experiment is done using Monte Carlo sampling, with 500 2πns Y ng(u) Y 2π mst(nsnt − mst) ≈ s=1 √ total samples of G. Bottom: the change of log-ICL for sample graphs as the (n+k−1)
n p number of blocks grows. Three graphs are randomly generated according to n 2πn u s≤t 2π(nsnt)(nsnt + 1) the same prescribed blocks. All of them have highest log-likelihood at the Y mg(u)g(v) Y mg(u)g(v) correct number of blocks k = 5. (1 − ) ng(u)ng(v) ng(u)ng(v) u

1) number of blocks k (log k bits, implicit) where we made a mean-field assumption about both ns and n+k−1
2) code for the partition of n into k ns terms (log k−1 mst under constant number of blocks k. If the edge density bits, ), which will specify a block size sequence (ordered of graph scales as |E| = ρn2, with ρ being a constant such in terms of blocks) that 0 ≤ ρ  1, we have 3) code for assigning each vertex to blocks according to 2 6 n
k n the ns terms (log bits) − ln P (E, g | M ) ≈ − ln P (E, g | pˆ) + ln (n1,n2,...,nk) i 4 4) for each pair of blocks s, t, the number of undirected 2 4πρn (1 − ρ) m s t log m < k2 edges st going between block and block ( st ≈ − ln P (E, g | pˆ) + ln Θ(n2) . log(nsnt + 1) bits) 2 Putting it together with the term associated with V , in which ω, which replaces the p matrix in the vanilla SBM. The edge we again assumed mean-field ns terms, we get generating likelihood is now

− ln P (G, g | Mi) = − ln P (V, g | Mi) − ln P (E, g | Mi) Auv|g ∼ Poi(θuθvωgugv ) . k2 ≈ − ln P (V, g | qˆ) + Θ(k ln n) − ln P (E, g | pˆ) + ln Θ(n2) The parameter θu gives us control over the expected degree 2 of each vertex, which for instance, could be a measure of k2 = − ln P (G, g | p,ˆ qˆ) + ln Θ(n2) . (12) popularity in social networks. The likelihood stays the same 2 if we scale θu for all vertices in block s, provided we also Multiply by 2, we have the BIC for order selection in SBMs: divide ωst for all s by the same factor. Thus identification demands an additional constraint. Here we use a convenient ˆ 2 2 BICSBM (Mi) = −2 ln P (G, g|Mi, Πi) + k ln Θ(n ) , (13) one that forces θu to sum to the total number of vertices in P each block: θu = ns. The ICL of DC-SBM is then with k2 specifying the number of parameters in the block u:gu=s affinity matrix p and n2 represent the sample size as pairwise P (G, g | θ, ω, q) edge/non-edge interactions. Auv Y Y (θu θv ωg g ) (13) is simply the direct application of BIC to the SBM as = q u v exp(−θ θ ω ) gu A ! u v gugv it is defined in Equation (3). In [15], Handcock et al. arrived u udegree distribution is heavy tailed. posterior of a DC-SBM Mi with the parameters {θ, ω, q}: The DC-SBM addresses these problem by allowing degree P (M ) P (M | G) = i P (G | M ) heterogeneity within blocks. Each vertex gets an additional DC i P (G) DC i parameter θu, which scales the expected number of edges ZZZ 1 connecting to it [7]. DC-SBM also generalizes the edge gener- ∝ d{θu}d{ωst}d{qs} P (G, g | θ, ω, q) , (17) ating processs to Poisson, thus allowing multi-edges between 0 vertices. According to the block assignment g, the means of 2Notice that it is different from the notion of sparsity in Equation (15). Even these Poisson draws depends on the k×k block affinity matrix as ωst ≈ pst → 0, we can still have quadratic scaling of edge densities. Log-likelihood where we again assumed a uniform prior over P (Mi), and a -63 500 æ æ æ æ constant P (G). æ æ If θu, wst and qs entries are independent, with the constrains -64 000 P P æ æ u:g =s θu = ns and i qs = 1, we have the posterior: -64 500 æ æ æ æ æ æ æ u æ æ PDC (Mi | G, g) ∝ PDC (G, g | Mi) -65 000 æ æ æ æ æ ZZZ 1 -65 500 = d{θu}d{ωst}d{qs} P (G, g | θ, ω, q) - æ 0 66 000 ! Z ZZZ 1 -66 500 Y du = d{θu} θu d{ωst}d{qs} P (G, g | ω, q) æ ð of blocks 4 u 0 2 4 6 8 10 12 ≈P (Θ, g | Mi) × P (G, g | Mi) . (18) Fig. 2. The change of log-likelihood for given graphs as the number of The integrated DC-SBM has one additional factor, forming a blocks grows. The graph is a randomly generated DC-SBM with 1000 vertices pair of nested models with the integrated vanilla SBM. and 5 prescribed blocks. Within each block, the degrees follow a bimodal To prepare P (Θ, g | Mi) for Bayesian treatments, we first distribution comprised of two Poisson distributions with their means 3 times change the variables θ = n η in the first integral, making apart. Both the vanilla SBM (red) and DC-SBM (blue) with different number u g(u) u of blocks k are fitted to the data. The experiment is done using a Monte Carlo the integrand a proper multinomial distribution. Now if the sampling method. The log-likelihood values shown here has been normalized new parameters ηu follow their Dirichlet conjugate priors, for both models so that they are comparable across models. (The normalization   method will be formally introduced in the next section. k Z Y Y du Y du+1 P (Θ, g | Mi) ≈  d{ηu} ηu  × ng(u) s=1 4 g(u)=s u from 5 prescribed blocks and the same expected number of   k Z total edges. Degrees within each block now follows a bimodal Y Y du Y du+1 distribution. This degree heterogeneity forces the vanilla SBM =  dηDirichlet( ~ηs| ~γs) ηu  × ng(u) s=1 4 g(u)=s u to split vertices into separate high degree and low degree k Q ! blocks, while the DC-SBM can comfortably mix them together Y g(u)=s du! Y = (n − 1)! × ndu+1 , (19) in the same block. As a consequence, posterior of the DC- s (D + n − 1)! g(u) s=1 s s u SBM achieves much higher log-ICL with fewer blocks. It also where we applied the multinomial Euler integral on the correctly captures the correct number of blocks at k = 5, simplex P η = 1. At the last line of the derivation, unlike the monotonic increasing log-ICL of the vanilla SBM. u:gu=s u we again assume the priors are uniform (i.e., γ∀u = 1). Following the derivation of BIC for the vanilla SBM, we C. Towards a universal model selection framework apply the Stirlings formula to the factorials in (19), In theory, a Bayesian approach based on the full ICL of Equation (5) can be used for comparing multiple SBM P (Θ, g | Mi) √ variants with different number of blocks together. However, k Q ! k 2πdu Y g(u)=s Y Ds+ns our partially Frequentist approaches come with additional ≈P (Θ, g | ηˆ) √ × ns (Ds+ns−1)
complications. In the previous example (Figure. 2), we took s=1 D 2πDs s=1 s √ an extra step to normalize the log-ICL for the vanilla SBM k Q ! k g(u)=s 2πdu ˆ Y Y ns and DC-SBM so that they are comparable. The normalization =P (Θ, g | θ) √ × ns , (20) (Ds+ns−1)
step is required because our MAP approach leaves some s=1 D 2πDs s=1 s parameters out of integration or summation. As a result, the D = P d where s g(u)=s u is the total degree of vertices in block ICL is still dependent of these parameters, eventually leading θˆu du s. We have also plugged in the MLEs ηˆu = = . to divergence in maximum likelihoods for different models. ngu Dg(u) Putting back the factors from the vanilla SBM (Equation In our case, these are the block assignment variables g, which (12)), and take the logarithm of it, we have the log-ICL: can change drastically from the vanilla SBM to the DC-SBM given the same graph. ln P (G, g | M ) = ln P (Θ, g | M ) + ln P (G, g | M ) DC i i i To remedy the situation, we propose a normalization method k2 ≈ ln P (G, g | θ,ˆ q,ˆ pˆ) − ln Θ(n∗) that use a hybrid frequentist-Bayesian method by calculating 2 the expected difference between Equation (12)/(14) and (21): n n 2|E| + Θ( ln ) − Θ(n ln ) , (21) 0 0 2 k n ln PDC (G, gˆ | Mi) − ln PSBM (G, gˆ | Mi ) with again mean-field assumptions about ns and Ds terms. ≈ ln P (G, gˆ | θ,ˆ q,ˆ ωˆ) − ln P (G, gˆ0 | 1, qˆ0, pˆ) Notice that Θ(n∗) is a general form for the correct sample size n n 2|E| for graphs with different edge density scalings, corresponding + Θ( ln ) − Θ(n ln ) , 2 k n to both Equation (12) and (14). The blue curve in Figure 2 shows the empirical results. Here where gˆ and gˆ0 are the most likely block assignments for the data G is generated by a DC-SBM with n = 1000 vertices DC-SBM and vanilla SBM respectively. Equation (18) shows that the vanilla SBM and the DC- Normalization allows us to compare multiple models with SBM still forms a pair of nested models after the partial different k together. Now we can ask questions like which integrations. Therefore the analysis in [21] holds. If we assume model should we use conditioned on a given number of blocks. the underlying data is generated by the simpler vanilla SBM, If we compare the two models for the same k according to we have both models converge to the same values for shared Figure 2, the choice would be DC-SBM when k < 3, and parameters, i.e. gˆ =g ˆ0, qˆ =q ˆ0 and ωˆ =p ˆ. In [21], the authors the vanilla SBM when k > 6 (There is no clear winner when came into the conclusion that the difference between the 3 ≤ k ≤ 6). The best model overall is a vanilla SBM with maximum likelihood under the null model roughly follows a 10 blocks, which is consistent with our generative model (5 χ2 distribution with a degree of freedom n−k, but corrections blocks with bimodal degree distributions). are needed when the graph is sparse. To arrive at the BIC formulation, we discard the constants To verify that we have the same maximum log-likelihood in Equation (23) and multiply both sides by −2, ratio, we can rewrite (12) and (21) as: BICDC (Mi) = − 2 ln P (G, g|Mi, Πˆ i) k2 2 ∗ ln P (G, g | 1, q,ˆ pˆ) ≈ ln P (G, g | M ) + ln Θ(n∗) , + k ln Θ(n ) + 2 ln[Θ(n)] . (24) SBM i 2 2 Compared with the vanilla SBM (Equation (13)/(15)), ˆ k ∗ ln P (G, g | θ, q,ˆ pˆ) ≈ ln PDC (G, g | Mi) + ln Θ(n ) BIC has an additional penalty term which grows with 2 DC n n 2|E| the size of the network. Therefore, DC-SBM favors fewer − Θ( ln ) + Θ(n ln ) . blocks than the vanilla SBM, since the flexibility provided 2 k n by the additional parameters allow vertices with very different Therefore, we have the log-likelihood ratio, degrees to coexist in the same block. ˆ ΛDC (G, g) = ln P (G, g | θ, q,ˆ pˆ) − ln P (G, g | 1, q,ˆ pˆ) D. Results on real world networks n n 2|E| To illustrate how the universal framework might work on ≈ ln P (Θ, g | M ) − Θ( ln ) + Θ(n ln ) i 2 k n real world data sets, we investigate two simple social networks = ln P (Θ, g | θˆ) , using Equation (13) and (23). The first is a social network consisting of 34 members of a karate club, where undirected which is exactly the same as the log-likelihood ratio for edges represent friendships [2]. The network is made up of two hypothesis testing in paper [21]. The agreement between assortative blocks, each with one high-degree hub and many Bayesian and Frequentist methods is not a coincident, because low-degree peripheral vertices. In [21], the authors studied the we have used uniform priors in our derivation. model selection problem between the vanilla SBM and DC- Now we are ready to use the result in [21] for estimating SBM conditioned on k = 2. Using the frequentist likelihood the expected difference between the log-ICLs, ratio test, there were not enough evidence to reject the null

0 0 hypothesis that the network is generated by a vanilla SBM. E [ln PDC (G, gˆ | Mi) − ln PSBM (G, gˆ | Mi )] The result based on the log-ICL with normalization confirms n n 2|E| = [Λ (G, g)] + Θ( ln ) − Θ(n ln ) that the DC-SBM has a higher likelihood at k = 2 (see E DC 2 k n Figure 3, top). However, for DC-SBM, it maximizes at k = 1, 1 n n n 2|E| ≈ ln[( + )(n − k)] + Θ( ln ) − Θ(n ln ) , which means with degree correction, any blocking leads to 2 24|E| 2 k n over-fitting. Therefore, for any meaningful communities, the (22) better choice is the vanilla SBM with a bigger k, because the degree heterogeneity is not strong enough to justify the where d¯ is the average degree of vertices in block s and n s 24|E| DC-SBM. In fact, the best model according to the universal is the first order correction for sparse graphs [21]. Bayesian framework is the vanilla SBM with k = 4, which Equation (22) only holds when the vanilla SBM is the corresponds to the division with high/low degree blocks for generative model of the data. For datasets in general, however, each cluster. This is also reminiscent to the result using active we can guarantee this happen by making sure the number of learning [33], where the vanilla SBM labels most of the blocks k is so large that even the vanilla SBM over-fits. Going vertices correctly once the high degree vertices are known. back to Figure 2, the red curves peaks at k = 10. We can then The second example is a network of political blogs in the US normalized the log-likelihood of the DC-SBM by subtracting [3]. Here we focus on the largest component with 1222 blogs it with the expected difference (Equation (22)) at k = 10. and 19087 links between them. The blogs have known polit- Subtracting Equation (22) from (21), we have the nor- ical leanings, with either liberal or conservative labels. The malized posterior of the DC-SBM which is now directly network is assortative, with heavy tailed degree distributions comparable to Equation (12)/(14): within each block. As a consequence, the frequentist analysis ln P (G, g | M ) = ln P (G, g | θ,ˆ q,ˆ pˆ) in [21] supported the hypothesis that the network is generated DC i by a DC-SBM. The universal Bayesian framework confirms k2 1 n − ln Θ(n∗) − ln[( + )(n − k)] . (23) the previous conjectures based on frequentist arguments at 2 2 24|E| k = 2 (see Figure 3, bottom). In fact, the degree heterogeneity Log-likelihood [11] M. E. J. Newman and A. Clauset, “Structure and inference in annotated æ networks,” arXiv preprint arXiv:1507.04001, 2015. [Online]. Available: æ http://arxiv.org/abs/1507.04001 æ -200 æ æ [12] M. Qi and G. Zhang, “An investigation of model selection criteria for æ æ æ æ æ æ neural network time series forecasting,” European Journal of Opera- æ æ æ tional Research, vol. 132, no. 3, pp. 666–680, 2001. æ æ -210 æ æ æ æ [13] D. R. Anderson and K. P. Burnham, “Avoiding Pitfalls When Using æ æ æ æ æ æ Information-Theoretic Methods,” The Journal of Wildlife Management, æ æ æ æ æ æ vol. 66, no. 3, pp. pp. 912–918, 2002. [Online]. Available: -220 æ æ æ æ æ æ æ http://www.jstor.org/stable/3803155 æ æ æ æ æ æ [14] M. R. E. Symonds and A. Moussalli, “A brief guide to model æ æ æ æ æ æ æ æ æ selection, multimodel inference and model averaging in behavioural æ ð of blocks 5 10 15 20 25 ecology using Akaike’s information criterion,” Behavioral Ecology and Log-likelihood Sociobiology, vol. 65, pp. 13–21, Aug. 2010. [Online]. Available: -60 000 æ æ æ æ æ æ æ æ æ æ æ æ æ æ æ æ æ æ æ æ æ æ æ æ æ æ æ http://www.springerlink.com/index/10.1007/s00265-010-1037-6 æ æ æ æ æ æ æ æ [15] M. Handcock, A. Raftery, and J. Tantrum, “Model-based clustering æ -70 000 æ æ for social networks,” Journal of the Royal Statistical Society: Series æ æ æ A (Statistics in Society), vol. 170, no. 2, pp. 301–354, 2007. æ -80 000 [16] E. Airoldi, S. Fienberg, C. Joutard, and T. Love, “Discovery of latent æ patterns with hierarchical bayesian mixed-membership models and the issue of model choice,” Data mining patterns: new methods and appli- -90 000 cations, p. 240, 2008. [17] P. Grunwald,¨ The minimum description length principle. MIT press, -100 000 2007. æ ð of blocks [18] T. Peixoto, “Parsimonious module inference in large networks,” arXiv 5 10 15 20 25 preprint arXiv:1212.4794, 2012. [19] G. Claeskens and N. L. Hjort, Model Selection and Model Averaging. Fig. 3. The change of log-ICL for the karate club network (top) and the Cambridge, England: Cambridge University Press, 2008. political blogs network (bottom) as the number of blocks grows. Both the [20] T. P. Peixoto, “Hierarchical Block Structures and High-Resolution Model Physical Review X vanilla SBM (red) and DC-SBM (blue) with different k are fitted to the data. Selection in Large Networks,” , vol. 4, no. 1, p. The experiment is done using a Monte Carlo sampling method. The log- 011047, Jan. 2014. likelihood values shown here has been normalized for both models. [21] X. Yan, C. Shalizi, J. E. Jensen, F. Krzakala, C. Moore, L. Zdeborova,´ P. Zhang, and Y. Zhu, “Model selection for degree-corrected block models,” Journal of Statistical Mechanics: Theory and Experiment, vol. 2014, no. 5, p. P05007, 2014. here is so strong, that the vanilla SBM never overtake DC- [22] K. Burnham, Model selection and multi-model inference : a practical SBM even with very big k values. If you are interested in large information-theoretic approach, 2nd ed. New York NY: Springer, 2010. scale community structure of the political blogs network, such [23] Y. X. Wang and P. J. Bickel, “Likelihood-based model selection for stochastic block models,” arXiv preprint arXiv:1502.02069, 2015. as the political factions, it seems the DC-SBM with smaller k [Online]. Available: http://arxiv.org/abs/1502.02069 values is a more reasonable choice. [24] C. Biernacki, G. Celeux, and G. Govaert, “Assessing a Mixture Model for Clustering with the Integrated Completed Likelihood,” IEEE Trans. ACKNOWLEDGMENT Pattern Anal. Mach. Intell., vol. 22, no. 7, pp. 719–725, Jul. 2000. The authors would like to thank Cristopher Moore and Tiago [Online]. Available: http://dx.doi.org/10.1109/34.865189 [25] R. E. Kass and A. E. Raftery, “Bayes factors,” Journal of the american Peixoto for valuable inputs and discussions. statistical association, vol. 90, no. 430, pp. 773–795, 1995. [26] G. Schwarz, “Estimating the dimension of a model,” The annals of REFERENCES statistics, vol. 6, no. 2, pp. 461–464, 1978. [1] S. Fortunato, “Community detection in graphs,” Physics Reports, 2009. [27] C. Fraley and A. E. Raftery, “How many clusters? Which clustering [2] W. W. Zachary, “An information flow model for conflict and fission in method? Answers via model-based cluster analysis,” The computer small groups,” Journal of Anthropological Research, vol. 33, no. 4, pp. journal, vol. 41, no. 8, pp. 578–588, 1998. 452–473, 1977. [28] C. Biernacki, G. Celeux, and G. Govaert, “Exact and Monte Carlo [3] L. Adamic and N. Glance, “The political blogosphere and the 2004 calculations of integrated likelihoods for the latent class model,” US Election: Divided They Blog,” in Proc 3rd Intl Workshop on Link Journal of Statistical Planning and Inference, vol. 140, no. 11, pp. Discovery., 2005. 2991–3002, 2010. [Online]. Available: http://www.sciencedirect.com/ [4] S. Allesina and M. Pascual, “Food web models: a plea for groups,” science/article/pii/S0378375810001631 Ecology letters, vol. 12, no. 7, pp. 652–662, 2009. [29] E. Comeˆ and P. Latouche, “Model selection and clustering in stochastic [5] P. W. Holland, K. B. Laskey, and S. Leinhardt, “Stochastic blockmodels: block models with the exact integrated complete data likelihood,” ArXiv first steps,” Social networks, vol. 5, pp. 109–137, 1983. e-prints, Mar. 2013. [6] S. Wasserman and C. Anderson, “Stochastic a posteriori blockmodels: [30] M. Rosvall and C. T. Bergstrom, “An information-theoretic framework Construction and assessment,” Social Networks, vol. 9, pp. 1–36, 1987. for resolving community structure in complex networks,” Proceedings [7] B. Karrer and M. E. J. Newman, Stochastic blockmodels and community of the National Academy of Sciences, vol. 104, no. 18, p. 7327, 2007. structure in networks, Jan. 2011. [31] T. Peixoto, “Entropy of stochastic blockmodel ensembles,” PRE, vol. 85, [8] Y. Zhu, X. Yan, and C. Moore, “Oriented and degree-generated block no. 5, p. 056122, May 2012. models: generating and inferring communities with inhomogeneous [32] T. P. Peixoto, “Model selection and hypothesis testing for large-scale degree distributions,” Journal of Complex Networks, vol. 2, no. 1, pp. network models with overlapping groups,” ArXiv e-prints, Sep. 2014. 1–18, 2014. [33] C. Moore, X. Yan, Y. Zhu, J. Rouquier, and T. Lane, “Active learning [9] E. M. Airoldi, D. M. Blei, S. E. Fienberg, and E. P. Xing, “Mixed for node classification in assortative and disassortative networks,” membership stochastic blockmodels,” J. Machine Learning Research, in KDD ’11 Proceedings of the 17th ACM SIGKDD international vol. 9, pp. 1981–2014, 2008. conference on Knowledge discovery and data mining. ACM Press, [10] A. Clauset, C. Moore, and M. E. J. Newman, “Hierarchical 2011, p. 841. [Online]. Available: http://dl.acm.org/citation.cfm?doid= structure and the prediction of missing links in networks,” Nature, 2020408.2020552 vol. 453, no. 7191, pp. 98–101, 2008. [Online]. Available: http: //www.nature.com/doifinder/10.1038/nature06830