Dependent Relational Gamma Process Models for Longitudinal Networks
Sikun Yang 1 Heinz Koeppl 1
Abstract as friendships or interactions in a social network, is often dynamic since the relations among the entities within the A probabilistic framework based on the covariate- network may appear or disappear over time (Mucha et al., dependent relational gamma process is developed 2010). Accordingly, the latent groups composed of those to analyze relational data arising from longitu- temporally connected entities also form and decay over time. dinal networks. The proposed framework char- Hence, appropriate models are needed to enable a better acterizes networked nodes by nonnegative node- understanding of the formation and evolution of dynamic group memberships, which allow each node to networks (Phan & Airoldi, 2015). belong to multiple latent groups simultaneously, and encodes edge probabilities between each pair A probabilistic framework is proposed to model such dy- of nodes using a Bernoulli Poisson link to the namic networks by assuming the network of interest is com- embedded latent space. Within the latent space, posed of a set of latent groups. Each node of the observed our framework models the birth and death dynam- network is hence associated with a time-dependent member- ics of individual groups via a thinning function. ships vector that governs its involvement in multiple groups Our framework also captures the evolution of indi- and interactions with other nodes. The node-group member- vidual node-group memberships over time using ships are assumed to be gamma distributed, thus, naturally gamma Markov processes. Exploiting the recent nonnegative real-valued. Moreover, to capture time-evolving advances in data augmentation and marginaliza- interactions between groups of nodes, we model the birth tion techniques, a simple and efficient Gibbs sam- and death dynamics of individual groups explicitly via a pler is proposed for posterior computation. Experi- dependent relational gamma process (dRGaP). The ideal mental results on a simulation study and three real- number of latent groups can be adaptively learned from data world temporal network data sets demonstrate the via the shrinkage mechanism of the dRGaP. model’s capability, competitive performance and Explicitly modelling group birth/death dynamics can be use- scalability compared to state-of-the-art methods. ful in many applications. For instance, latent groups in a network of military disputes between countries could mean alliances such as NATO coordinating collective defence 1. Introduction to attacks by external forces (Schein et al., 2016a). These The study of relational data arising from various networks groups can be born and die afterwards. For example, the including social, biological and physical networks is becom- Warsaw Pact was established during the Cold War and dis- ing increasingly important due to the emergence of massive solved in later years. We demonstrate that our model can relational data collected from these domains. Many efforts discover interpretable latent structure on a real network of have been dedicated to develop statistical models in terms of military interstate disputes (Ghosn et al., 2004) that agrees community detection and missing link prediction for analyz- with our knowledge of international relations (Section 5). ing relational data arising from static networks, where either Furthermore, it is reasonable to model the time-evolving a single snapshot of the network of interest or an aggregated memberships of each individual node to interpret its joining network over time is presented; see (Goldenberg et al., 2010) and withdrawing behavior to these groups. Hence, we cap- for a review of the literature. However, network data, such ture the dynamics of individual node-group memberships evolving over time via gamma Markov processes. 1Department of Electrical Engineering and Information Tech- In contrast to dynamic network modelling using logistic nology, Technische Universitat¨ Darmstadt, Germany. Correspon- or probit mapping functions (Foulds et al., 2011; Heauku- dence to: Sikun Yang
3. Model Formulation the inter-group interaction weight λkk0 and intra-group in- teraction weight λkk can be generated as We represent a dynamic network by a sequence of adjacency matrices A(t) for each time t = 1, 2,...,T . For the sake of ( 0 Gamma(ξrk, β), if k = k clarity, we limit our focus to unweighted (binary) undirected λkk0 ∼ (2) Gamma(r r 0 , β), otherwise (symmetric) dynamic networks without self-links although k k the proposed model can straightforwardly be generalized where ξ ∼ Gamma(1, 1) and β ∈ R>0. to nonnegative real-weighted networks via the Poisson ran- domized gamma distribution (Zhou et al., 2016). We denote For dynamic relational data, we exploit the thinned CRMs each snapshot of a dynamic network by A(t) ∈ {0, 1}N×N framework to capture the birth/death dynamics of latent with N being the number of nodes. The binary symmetric groups assuming that the status of group k can be either matrix A(t) has entries A(t) = 1 if an edge between nodes i active or inactive at time t. More specifically, we use a ij (t) (t) (t) Bernoulli random variable bk = 1 to indicate the presence and j is present at time t, and Aij = 0 otherwise. Let Πij (t) be the link probability between nodes i and j. Our model of group k at time t, and bk = 0 otherwise. Accordingly, 0 specifies the interaction weight between group k and k is active at time t only if the two groups are both active at that time, i.e., (t) (t) (t) (t) (t) (t) λ 0 = λkk0 b b 0 . Aij | Πij ∼ Bernoulli(Πij ), t ∈ T kk k k Given the group interaction weight matrix Λ defined in independently for each i = 2,...,N and j = 1, . . . , i − 1, Eq. (2), we generate the time-dependent group interaction with (t) weights λkk0 using the thinning function introduced in Sec- K K ! tion 2.2 with the prior specification: h (t) (t)i X X (t) (t) (t) E Aij | Πij = 1 − exp − zik λkk0 zjk0 , ∗ ∗ k=1 k0=1 ωlk ∼ N IG(0, 1, 1), φk ∼ Cat(φ1, . . . , φD), ( T ) X (t) P (t) = σ ω + ω exp[−φ (t − l)2] , where zik ∈ R>0 characterizes the membership of node i xk 0k lk k to group k at time t ∈ T . In contrast to the latent feature l=1 relational models (Foulds et al., 2011; Heaukulani et al., where we fix the centres of the covariate-dependent kernel 2013; Kim et al., 2013) that assume binary node-group functions to the T discrete time points of the considered relationships, our nonnegative memberships capture how dynamic network. The probability of activity/inactivity of strongly each node associates with multiple groups. The group k at time t can be determined by the thinning function. (t) group interaction weight λkk0 modulates the probability A smooth thinning function can encourage the snapshots of that there exists a link between a node affiliated to group a dynamic network at nearby covariate values t to share a 0 k and a second node affiliated to group k at time t. Our similar set of groups. framework exploits the dynamics of the underlying relations between nodes on two levels: (1) The latent groups can be 3.2. Dynamics of Node-Group Memberships active and inactive; and (2) the memberships of each node to groups evolve over time. We now proceed to describe To capture the temporal dynamics of node-group member- each component of our framework in the following sections. ships, we represent the individual memberships as indepen- dently evolving gamma Markov chains. More specifically, (t) 3.1. Model of Active Groups the individual membership zik is assumed to be gamma distributed with shape parameter z(t−1), where z(t−1) corre- Many previous works (Kim et al., 2013; Xu, 2015) have ik ik sponds to the membership of the same node at the previous shown that explicitly modelling the dynamics of latent time, and rate parameter τ as groups using a distance-dependent Indian buffet process (dd-IBP) (Gershman et al., 2015) or a linear dynamical sys- z(t) ∼ Gamma(z(t−1), τ), t ∈ T (3) tem discovers interpretable latent structures and achieves ik ik (1) good predictive performance. Here, we build the group in- zik ∼ Gamma(θik, τ), teraction weight λkk0 on the relational gamma process con- struction (Zhou, 2015). For implementation convenience, where we assume each membership starts off in a dummy we use a truncated version of the infinite capacity model by state (θik for i = 1,...,N, and k = 1,...,K). We place fixing the maximum number of groups to K. That is, we first a gamma prior over the dummy node membership θik, i.e. generate a group weight r independently for each group k θik ∼ Gamma(1, 1). Given the group interaction weights k (t) (t) as rk ∼ Gamma(γ0/K, c), where γ0 denotes the concen- {λkk0 } and node-group memberships {zik } generated by (t) tration parameter and c denotes the rate parameters. Then, our model, we generate an edge Aij between two nodes i Dependent Relational Gamma Process Models for Longitudinal Networks and j at time t via the Bernoulli-Poisson link (BPL) function ables, parameters, and hyperparameters, is given by as rk ∼ Gamma(γ0/K, c), θik ∼ Gamma(1, 1), (5) K K ! (t) X X (t) (t) (t) ξ ∼ Gamma(1, 1), mij ∼ Poisson zik λkk0 zjk0 , (4) ( 0 k=1 k0=1 Gamma(ξrk, β), if k = k λ 0 ∼ (t) (t) kk Gamma(rkrk0 , β), otherwise Aij = 1(mij ≥ 1), t ∈ T ∗ ∗ ωlk ∼ N IG(0, 1, 1), φk ∼ Cat(φ1, . . . , φD), where mij is a latent Poisson count that measures how often T ! nodes i and j interact in the latent space representation. (t) n X o b ∼ Bernoulli σ ω + ω exp[−φ (t − l)2] , Marginalizing out m(t) from Eq. (4) yields k 0k lk k ij l=1 (t) (t−1) (1) " K K !# zik ∼ Gamma(zik , τ), zik ∼ Gamma(θik, τ), (t) X X (t) (t) (t) Aij ∼ Bernoulli 1 − exp − zik λkk0 zjk0 . K k=1 k0=1 (t) (t) (t) (t) X (t) (t) (t) λkk0 = bk λkk0 bk0 , mij ∼ Poisson zik λkk0 zjk0 , k,k0=1 The conditional distribution of the latent count m(t) can be ij (t) (t) expressed as Aij = 1(mij ≥ 1).
K K ! (t) (t) (t) X X (t) (t) (t) 3.5. Inference via Gibbs Sampling (mij | Aij , −) ∼ Aij Poisson+ zik λkk0 zjk0 , (1:t) (1) (t) k=1 k0=1 Let A denotes the sequence A ,...,A and similarly for Z(1:t) and Λ(1:t). The model parameters that need to be where x ∼ Poisson+(σ) is the zero-truncated Poisson dis- sampled include: latent node-group memberships {z(t)}, tribution with support only on the positive integers, and ik {θik}, individual group weights {rk}, scale parameter ξ, “–” denotes all other variables. We note that if A(t) = 0, ij groups interaction weights {λkk0 }, kernel weights {ωlk}, then m(t) = 0 almost surely (a.s.), and if A(t) = 1, then (t) ij ij kernel widths {φk}, thinning variables {bk }, and latent (t) PK PK (t) (t) (t) (t) mij ∼ Poisson+( k=1 k0=1 zik λkk0 zjk0 ). Thus, the counts {mij }. Exploiting the Polya-gamma´ data augmen- (t) (t) tation technique (Polson et al., 2013) and the data augmen- latent count mij only needs to be sampled for Aij = 1 with a rejection sampler (Zhou, 2015). This property makes tation and marginalization technique (Zhou et al., 2015), a the proposed model appealing for modelling large sparse simple and efficient Gibbs sampling algorithm is developed dynamic networks because the computational cost of our to perform the model inference. The details of our inference model scales linearly with the number of edges. algorithm are presented in the supplementary material.
3.3. Bayesian Nonparametric Interpretation 4. Related Work As K → ∞, the group weights and their corre- Prior works on dynamic networks modelling include the ex- sponding dummy node memberships vector constitute ponential random graph model (ERGM) (Guo et al., 2007), P∞ a draw from a gamma process as G = k=1 rkδθk , matrix and tensor factorization based methods (Dunlavy where θk = (θ1k, . . . , θNk) ∈ Θ is an atom sampled et al., 2011) and statistical models (Sarkar et al., 2007; 2014; from a N-dimensional base measure G0(dθk)/G0(Θ) = Ishiguro et al., 2010; Durante et al., 2014b; Schein et al., QN i=1 Gamma(1, 1). Accordingly, the intra- and inter- 2016a; Palla et al., 2016). Statistical dynamic network mod- group interaction weights and their corresponding pair of els received considerable attention because these models node memberships vector constitute a draw Λ | G = have favourable interpretability by providing uncertainty P∞ P∞ 0 λ 0 δ estimates for the uncovered latent representations (Hoff k=1 k =1 kk (θk,θk0 ) from a relational gamma pro- cess (Zhou, 2015). Via the thinned CRMs framework, Λ(t) | et al., 2001). Dynamic extensions of the mixed membership P∞ P∞ (t) (t) stochastic blockmodel (MMSB) (Airoldi et al., 2008) have Λ = 0 b b 0 λ 0 δ k=1 k =1 k k kk (θk,θk0 ) can be viewed as a draw from a covariate-dependent relational gamma process. been developed (Fu et al., 2009; Xing et al., 2010; Ho et al., For brevity, the definitions of these processes are presented 2011) using linear state space models to capture the evolu- in the supplementary material. tion of real-valued node-group memberships. Recently, an extended Kalman filter (EKF) based algorithm (Xu et al., 2014) was proposed to infer dynamic stochastic blockmod- 3.4. The Full Generative Model els (SBM) with competitive performance. Dynamic exten- The full generative model (truncated) for the observed dy- sions of the latent feature relational model (LFRM) (Miller (t) namic network data {A }t∈T along with the latent vari- et al., 2009) using an infinite factorial hidden Markov pro- Dependent Relational Gamma Process Models for Longitudinal Networks cess to capture the evolution of binary node-group member- the released code2 with the default settings. We also com- ships include the dynamic relational infinite feature model pare DRGPM with the hierarchical gamma process edge (DRIFT) (Foulds et al., 2011), the latent feature propagation partition model (HGPEPM) (Zhou, 2015)3 and the dynamic model (LFP) (Heaukulani et al., 2013), and the dynamic Poisson gamma model (DPGM) (Yang & Koeppl, 2018) multi-group membership graph model (DMMG) (Kim et al., that only models the evolving node memberships. More- 2013). over, we demonstrate that our model can discover highly interpretable latent structure on a military interstate disputes Our proposed model is a form of Poisson factorization dataset. In the experiments, we set the hyperparameters for model (Zhou et al., 2015; Gopalan et al., 2015), and can our model as γ = 1, β = 1, c = 1, τ = 1. Unless oth- be considered as the dynamic extension of the hierarchical 0 erwise stated, we use K = N/2 for initilization, where gamma process edge partition model (HGPEPM) (Zhou, N is the number of nodes. A sensitivity analysis revealed 2015). The dependent CRM framework (Foti et al., 2013) that we obtain similar results when instead setting γ = 0.1 has been exploited for dynamic topic models and dependent 0 or γ = 10. All the experiments were run on a standard latent feature models previously. To the best of our knowl- 0 desktop with 2.7 GHz CPU and 24 GB RAM. edge, this is the first attempt to model activity and inactivity of latent groups using a thinned CRMs framework in longi- 5.1. Simulation Study tudinal network modelling. Our Markov chain construction, used to capture the evolution of node-group memberships, Following the procedure suggested by Durante et al.(2016), is inspired by the data augmentation technique (Zhou et al., we generate synthetic data to evaluate our proposed model in 2015) that has been exploited for dynamic matrix factor- estimating the formation and evolution of the latent network ization (Acharya et al., 2015a; Schein et al., 2016b) and sturcture. We consider a dynamic network with N = 50 deep gamma belief networks (Zhou et al., 2016). We note nodes monitored for T = 70 equally spaced time snap- that the dynamic gamma process Poisson factorization (D- shots. To generate a time-varying network, we first gen- GPPF) (Acharya et al., 2015b) has been proposed using erate five regimes defining the true edge probabilities, as gamma Markov chains to model the evolution of latent shown in Figure2. We then simulate the network edges groups while the D-GPPF assumes node memberships are (t) (t) (t) Aij | Πij ∼ Bernoulli(Πij ) with each of the five regimes static over time. Yang & Koeppl(2018) directly generalized according to Figure1. To demonstrate that DRGPM can in- the relational gamma process model to dynamic networks fer interpretable latent structure while avoiding to overfit using gamma Markov chains to model node-group evolving the data, we compare DRGPM with D-GPPF and DPGM. behavior. For longitudinal networks, it is more reasonable We initialize all methods setting K = 30. to explicitly model the birth and death dynamics of latent groups by switching off redundant groups to avoid overfit- ting the data and to strengthen the interpretability of the latent network structure.
5. Experiments Figure 1. The graph showing which regime − i.e. true edge proba- bilities − for each snapshot is used to simulate the data. We demonstrate that the proposed Dependent Relational In Figure3, we depict the inferred link probabilities by Gamma Process Model (DRGPM) infers more interpretable DRGPM and D-GPPF in columns (b) and (d), respectively. latent structure compared with the related methods using We also depict the time-evolving node-group connections (t) (t) a synthetic example. Quantitive evaluations of our model, by computing the node-group association weights zik λkk compared with state-of-the-art methods as discussed in Sec- (t) for DRGPM, and rk zik for D-GPPF in columns (c) and (e), tion 4, are performed in terms of missing link prediction respectively. We note that D-GPPF needs to generate many and future network forecasting on three real-world data redundant groups to capture the time-evolving behavior of sets. The first baseline is the dynamic relational infinite fea- each node because of its unfavourable assumption that node ture model (DRIFT) for which we used the code provided (t) 1 memberships zik are static while group weights rk are by Foulds et al.(2011). Additionally, we implemented the time-dependent. Without this restriction, DRGPM charac- D-GPPF, where we set the hyperparameters and initialized terizes the evolving node-group associations by explicitly the model parameters with the values provided in (Acharya modelling time-dependent node-group memberships, and et al., 2015b). The third baseline is the dynamic stochastic hence generates an appropriate number of groups. In par- blockmodel (DSBM) based on an extended Kalman filter ticular, using the thinned CRM framework, DRGPM can (EKF) augmented with a local search, for which we use effectively activate newly-formed groups and switch off 1http://jfoulds.informationsystems.umbc. 2https://tinyurl.com/ydf29he9. edu/code/DRIFT.tar.gz. 3https://github.com/mingyuanzhou/EPM. Dependent Relational Gamma Process Models for Longitudinal Networks
Regime 1 Regime 2 Regime 3 Regime 4 Regime 5 Figure 2. True edge probabilities for the five regimes in the simulation study. 5 t 15 t 25 t 35 t 60 t
(a) (b) (c) (d) (e) Figure 3. We select five snapshots of the simulated network as shown in column (a). The link probabilities inferred by DRGPM and D-GPPF are shown in columns (b) and (d), respectively. The association weights of each node (row variable) to the groups (column (t) (t) (t) variable), as shown in columns (c) and (e), can be calculated as zik λkk for DRGPM and rk zik for D-GPPF, respectively. The pixel values are displayed on log10 scale.
DRGPM infers three groups (4, 11, 24) at t = 0, turns off Group 4 and turns on Groups 3 and 7 at t = 10. Group 28 is only active from t = 30 to 40. The comparison of DRGPM to DPGM is presented in the supplementary material.
Group 3 Group 4 Group 7 5.2. Quantitative Results For the quantitative evaluation, we consider the follow- ing data sets: (1) Face-to-face dynamic contacts network (FFDC): This dataset (Mastrandrea et al., 2015) records timestamped face-to-face contacts among 180 students for Group 11 Group 24 Group 28 (t) 7 school days. We generated a dynamic network consid- Figure 4. The thinning probabilities (mean of bk ) of six active groups inferred by DRGPM. ering each school day as a snapshot, and created an edge between each pair of students at time t if they have at least one contact recorded at that given time. (2) DBLP: The DBLP co-authorship network data (Asur et al., 2009) con- redundant groups over time, which strengthens the model in- tains the co-authorship information among 958 authors over terpretability for longitudinal network analysis. In Figure4, (t) ten years (1997-2006) in 28 conferences which spans three we depict the thinning probabilities (mean of bk ) over time related research topics-database, data mining, and artificial for six inferred active groups by DRGPM. We notice that Dependent Relational Gamma Process Models for Longitudinal Networks
Table 1. Quantitive evalution. We highlight the performance of the best scoring model in bold. Missing Link Prediction FFDC DBLP Enron Model AUROC PR AUROC PR AUROC PR HGPEPM 0.917 ± 0.006 0.354 ± 0.018 0.979 ± 0.004 0.791 ± 0.014 0.972 ± 0.001 0.443 ± 0.016 DSBM 0.878 ± 0.011 0.251 ± 0.017 0.913 ± 0.006 0.256 ± 0.009 0.916 ± 0.007 0.225 ± 0.023 D-GPPF 0.908 ± 0.005 0.313 ± 0.019 0.914 ± 0.005 0.308 ± 0.018 0.977 ± 0.002 0.499 ± 0.022 DRIFT 0.933 ± 0.006 0.416 ± 0.020 0.970 ± 0.019 0.491 ± 0.025 NA NA DPGM 0.921 ± 0.004 0.359 ± 0.020 0.960 ± 0.002 0.423 ± 0.032 0.979 ± 0.002 0.565 ± 0.014 DRGPM 0.924 ± 0.005 0.357 ± 0.018 0.963 ± 0.003 0.425 ± 0.023 0.983 ± 0.002 0.597 ± 0.017 Future Network Forecasting HGPEPM 0.733 ± 0.025 0.164 ± 0.022 0.714 ± 0.035 0.106 ± 0.027 0.828 ± 0.073 0.246 ± 0.140 DSBM 0.825 ± 0.085 0.181 ± 0.039 0.704 ± 0.030 0.091 ± 0.009 0.853 ± 0.059 0.325 ± 0.116 D-GPPF 0.842 ± 0.028 0.203 ± 0.046 0.734 ± 0.080 0.109 ± 0.046 0.878 ± 0.057 0.360 ± 0.121 DRIFT 0.848 ± 0.056 0.224 ± 0.025 0.745 ± 0.060 0.121 ± 0.054 NA NA DPGM 0.846 ± 0.017 0.221 ± 0.036 0.744 ± 0.053 0.123 ± 0.064 0.883 ± 0.051 0.361 ± 0.131 DRGPM 0.852 ± 0.033 0.226 ± 0.040 0.753 ± 0.057 0.127 ± 0.053 0.886 ± 0.067 0.363 ± 0.130 intelligence. We focus on a subset of 324 most connected DRGPM, although DRGPM has a significant advantage in authors over all time period. (3) Enron: The Enron data4 terms of computational cost due to the Bernoulli-Poisson contains 517,431 emails among 151 users over 38 months link (see Section 5.3). HGPEPM performs better than the dy- (from May 1999 to June 2002). We generated a dynamic namic models on the DBLP dataset because co-authorship network aggregating the data into monthly snapshots, and links change dramatically from one year to the next one, created an edge between each pair of users at time t if they and hence, the static model is better at fitting each snapshot have at least one email recorded at that given time. The independently. For the longitudinal Enron email network summary statistics are detailed in Table2. that is recorded monthly, DRGPM performs better than the Table 2. Details of the data sets used in our experiments. baseline methods. Dataset FFDC DBLP Enron # Nodes 180 324 151 Task 2: Forecasting future networks Next, we consider # Slices 7 10 38 the task of forecasting an unseen network snapshot A(t) # Edges 8,332 11,154 11,414 given observed snapshots A(1:t−1). Following previous Task 1: Predicting missing links First, we perform miss- works (Foulds et al., 2011; Heaukulani et al., 2013; Kim ing link prediction on the real-world data sets, and show the et al., 2013), we train the models on the first (t−1) snapshots proposed model’s predictive performance compared to the of the considered network, and then estimate the predictive (t) baseline models. We randomly hold out 20% of the network distribution of the unseen snapshot A by running MCMC entries (either links or non-links) for each snapshot as test sampling one time step into the future. We apply HGPEPM (t−1) data, and use the remaining 80% to predict the held-out to the most recent snapshot A , and then to perform pre- (t) entries. DRIFT was infeasible to run on the Enron dataset diction on the unseen snapshot A . For DRGPM, we set (t) (t−1) in a reasonable amount of time given our computational Λ = Λ , assuming the snapshots at nearby time points resource. For DSBM, we either set K to the true number share a similar set of groups. We generated 10 samples of of classes provided by the data set or initialize it by exam- Z (t) for each of the 1000 samples collected for Z (t−1). For ining the singular values of the first snapshot (Xu et al., DSBM, we use the method detailed in (Xu et al., 2014) to 2014). We apply HGPEPM to each snapshot of dynamic perform future network forecasting. Table1 shows the av- networks independently. For all probabilistic methods, we eraged performance for each model over different network use 2000 burn-in iterations, and collect 1000 samples from snapshots from 3 to T . Overall, DRGPM shows competi- the model posterior distribution. We estimate the posterior tive performance on all three datasets. This confirms that mean of the edge probability for each held-out edge in the DRGPM can flexibly characterize temporally local links via test data by averaging over the collected Gibbs samples. We time-evolving node memberships and switch off redundant then use these edge probabilities to evaluate the predictive groups to avoid overfitting the data. performance of each model by calculating the area under the curve of the receiver operating characteristic (AUROC) 5.3. Running Time and of the precision-recall (PR). In Table1, we report the The probilistic models achieve higher accuracy although average evaluation metrics for each model over 10 runs. these methods require more computation time to collect Overall, we found that DRIFT performs slightly better than MCMC samples. DSBM is much faster than the proba- 4https://www.cs.cmu.edu/˜enron/. bilistc models because its inference is performed using the Dependent Relational Gamma Process Models for Longitudinal Networks
(t) Figure 5. The activity (mean of bk ) of the four selected groups inferred from the MID network. extended Kalman filter. Table3 compares the per-iteration of four groups with positive memberships. For instance, we computation time of the sampling-based models (all mod- found that Group 1 corresponds to the second Congo war els are implemented in Matlab). The computational cost of (1998-2000). The first six nodes of the group are indeed the DRIFT scales in O(K¯ 2N 2T ), where K¯ is the expected num- belligerents of this war. Group 2 corresponds to the Bosnian ber of groups. The Bernoulli-Poisson link based models (D- War (1992-1995), and its associated nodes are Yugoslavia GPPF, DPGM, DRGPM) are much faster than the logistic and some NATO members that are indeed the belligerents of link based method (DRIFT) because the former models scale this war. Groups 3 and 4 are related to the regional disputes linearly with the number of non-zero entries in network data. between some African countries. Additional graphs of the (t) (t) 0 inferred groups, and tables showing the associated nodes For DRGPM, sampling {mij }i,j,t and {mikk0j}i,j,k,k ,t takes O(N eK¯ 2) with N e being the number of non-zero en- are presented in the supplementary material. (t) ¯ Table 4. The top 20 nodes associated with each of the four selected tries. Sampling {zik }i,k,t takes O(NKT ) and sampling (t) ¯ 2 groups as shown in Fig.5 from the MID network. The highest node {λ 0 } 0 takes O(K T ). Overall, the computational kk k,k ,t memberships to the corresponding selected groups throughout the complexity of DRGPM is O(N eK¯ 2 + NKT¯ + K¯ 2T ). whole period are reported for each node in parentheses. The computational complexity of D-GPPF and DPGM is Group Country O(N eK¯ + NK¯ + KT¯ ) and O(N eK¯ 2 + NKT¯ + K¯ 2), re- 1 Namibia (0.22), Chad (0.21), Zimbabwe (0.21), spectively. DRGPM is slightly faster than DPGM because Angola (0.20), Dem. Rep. Congo (0.10), DRGPM can effectively turn off redundant groups and hence Sudan (0.05), Zambia (0.01) achieves a lower computational cost. 2 Yugoslavia (0.60), Greece (0.13), Italy (0.04), UK (0.04), France (0.04), Belgium (0.03), Table 3. Comparison of computation time (seconds per iteration). Albania (0.03), Turkey (0.03), USA (0.02), FFDC DBLP Enron Spain (0.02), Netherlands (0.01), Germany (0.01) DRIFT 164.342 382.119 - 3 Nigeria (0.45), Ghana (0.31), Guinea (0.24) D-GPPF 0.145 0.242 0.292 4 Liberia (0.98), Sierra Leone (0.02) DPGM 0.748 1.676 1.705 DRGPM 0.623 1.217 1.234 6. Conclusion 5.4. Case Study: Military Interstate Disputes Dataset We proposed a probabilistic framework for longitudinal net- We investigate the military interstate disputes (MID) dataset work modelling based on the covariate-dependent relational that contains disputes events between 138 countries from gamma process. Our framework can characterize the group 1992 to 2001 (Ghosn et al., 2004) to explore the latent struc- birth/death dynamics using the thinned CRM, which en- ture discovered by DRGPM. A dynamic network was gener- ables us to investigate the evolution of the inferred latent ated by aggregating the data into monthly snapshots and a structure. The inferred latent dynamic structure can be use- link was created between each pair of two countries if either ful for various qualitative analyses in practical applications. country has disputes with the other one at that given time. We experimentally demonstrated the competitive predic- We applied DRGPM to this dynamic network initializing tive performance and scalability of our framework on three K = 30 groups. Most of the identified groups correspond to real-world datasets. Our generative model can be easily ex- some regional relations or conflicts. In Figure5, we depict tended in many interesting ways. For instance, it can be four interesting groups inferred by DRGPM and show the extended to dynamic multilayer networks (Durante et al., group activity by plotting the mean of the thinning function 2017) with relational data arising from multiple-related con- (t) bk . We normalized the node memberships to [0, 1] by divid- texts via multi-level hierarchical gamma processes (Zhou ing them by the sum of memberships within the same group. et al., 2015). Additionally, network side information can be In Table4, we report the top 20 nodes associated to each flexibly leveraged via a regression model (Rai et al., 2015). Dependent Relational Gamma Process Models for Longitudinal Networks
Acknowledgements Gershman, Samuel et al. Distance dependent infinite latent feature models. IEEE Trans. Pattern Anal. Mach. Intell., The authors thank the anonymous reviewers and Adrian 37(2):334–345, 2015. Soˇ siˇ c´ for their useful comments and suggestions. This re- search is funded by the European Union’s Horizon 2020 Ghosn, Faten et al. The mid3 data set, 1993-2001: Proce- research and innovation programme under grant agreement dures, coding rules, and description. Conflict Manage- 668858. ment and Peace Science, 21(2):133–154, 2004.
Goldenberg, Anna et al. A survey of statistical network mod- References els. Found. Trends Mach. Learn., 2(2):129–233, February Acharya, Ayan et al. Nonparametric Bayesian factor anal- 2010. ysis for dynamic count matrices. In AISTATS, pp. 1–9, 2015a. Gopalan, Prem et al. Scalable recommendation with hi- erarchical poisson factorization. In UAI, pp. 326–335, Acharya, Ayan et al. Nonparametric dynamic network mod- 2015. eling. In KDD Workshop on MLT, pp. 104–113, 2015b. Guo, Fan et al. Recovering temporally rewiring networks: a Airoldi, Edoardo M. et al. Mixed membership stochastic model-based approach. In ICML, pp. 321–328, 2007. blockmodels. J. Mach. Learn. Res., 9:1981–2014, June 2008. Heaukulani, Creighton et al. Dynamic probabilistic models for latent feature propagation in social networks. In ICML, Asur, Sitaram et al. An event-based framework for charac- pp. 275–283, 2013. terizing the evolutionary behavior of interaction graphs. ACM Trans. Knowl. Discov. Data, 3(4):16:1–16:36, 2009. Ho, Qirong et al. Evolving cluster mixed-membership block- model for time-evolving networks. In AISTATS, pp. 342– Dunlavy, Daniel M. et al. Temporal link prediction using 350, 2011. matrix and tensor factorizations. ACM Trans. Knowl. Discov. Data, 5(2):10:1–10:27, 2011. Hoff, Peter D. et al. Latent space approaches to social network analysis. JASA, 97:1090–1098, 2001. Dunson, David B. and Herring, Amy H. Bayesian latent vari- able models for mixed discrete outcomes. Biostatistics, 6 Hu, Changwei et al. Zero-truncated Poisson tensor factor- (1):11–25, 2005. ization for massive binary tensors. In UAI, pp. 375–384, 2015. Durante, Daniele et al. Bayesian logistic Gaussian process models for dynamic networks. In AISTATS, pp. 194–201, Ishiguro, Katsuhiko et al. Dynamic infinite relational model 2014a. for time-varying relational data analysis. In NIPS, pp. 919–927, 2010. Durante, Daniele et al. Nonparametric Bayes dynamic mod- elling of relational data. Biometrika, 101(4):883–898, Kim, Myunghwan et al. Nonparametric multi-group mem- 2014b. bership model for dynamic networks. In NIPS, pp. 1385– 1393, 2013. Durante, Daniele et al. Locally adaptive dynamic networks. Ann. Appl. Stat., 10(4):2203–2232, 2016. Kingman, J. F. C. Completely random measures. Pacific J. Durante, Daniele et al. Bayesian learning of dynamic multi- Math., 21(1):59–78, 1967. layer networks. JMLR, 18(43):1–29, 2017. Kingman, J. F. C. Poisson processes. OUP, New York, Ferguson, Thomas S. A Bayesian analysis of some nonpara- 1993. metric problems. Ann. Statist., 1(2):209–230, 1973. Mastrandrea, Rossana et al. Contact patterns in a high Foti, Nicholas et al. A unifying representation for a class school: A comparison between data collected using wear- of dependent random measures. In AISTATS, pp. 20–28, able sensors, contact diaries and friendship surveys. PLoS 2013. ONE, 10(9):1–26, 2015. Foulds, James R. et al. A dynamic relational infinite feature Miller, Kurt et al. Nonparametric latent feature models for model for longitudinal social networks. In AISTATS, pp. link prediction. In NIPS, pp. 1276–1284, 2009. 287–295, 2011. Mucha, Peter J et al. Community structure in time- Fu, Wenjie et al. Dynamic mixed membership blockmodel dependent, multiscale, and multiplex networks. Science, for evolving networks. In ICML, pp. 329–336, 2009. 328(5980):876–878, 2010. Dependent Relational Gamma Process Models for Longitudinal Networks
Palla, K. et al. Bayesian nonparametrics for sparse dynamic Zhou, Mingyuan et al. Negative binomial process count networks. ArXiv e-prints, 2016. and mixture modeling. IEEE Trans. Pattern Anal. Mach. Intell., 37(2):307–320, 2015. Phan, Tuan Q. and Airoldi, Edoardo M. A natural experi- ment of social network formation and dynamics. PNAS, Zhou, Mingyuan et al. Augmentable gamma belief networks. 112(21):6595–6600, 2015. Journal of Machine Learning Research, 17:1–44, 2016. Polson, Nicholas G. et al. Bayesian inference for logistic models using Polya–Gamma´ latent variables. JASA, 108 (504):1339–1349, 2013. Rai, Piyush et al. Large scale Bayesian multi-label learning via topic-based label embeddings. In NIPS, pp. 3222– 3230, 2015. Sarkar, Purnamrita et al. A latent space approach to dynamic embedding of co-occurrence data. In AISTATS, pp. 420– 427, 2007. Sarkar, Purnamrita et al. Nonparametric link prediction in large scale dynamic networks. Electron. J. Statist., 8(2): 2022–2065, 2014. Schein, Aaron et al. Bayesian Poisson Tucker decomposi- tion for learning the structure of international relations. In ICML, pp. 2810–2819, 2016a. Schein, Aaron et al. Poisson-gamma dynamical systems. In NIPS, pp. 5006–5014, 2016b. Wolpert, Robert L. et al. Poisson/gamma random field models for spatial statistics. Biometrika, 85(2):251–267, 1998. Wolpert, Robert L. et al. Stochastic expansions using contin- uous dictionaries: Levy´ adaptive regression kernels. Ann. Statist., 39(4):1916–1962, 08 2011. Xing, Eric P. et al. A state-space mixed membership block- model for dynamic network tomography. Ann. Appl. Stat., 4(2):535–566, 06 2010. Xu, Kevin. Stochastic block transition models for dynamic networks. In AISTATS, pp. 1079–1087, 2015. Xu, Kevin S. et al. Dynamic stochastic blockmodels for time-evolving social networks. J. Sel. Topics Signal Pro- cessing, 8(4):552–562, 2014. Yang, Sikun and Koeppl, Heinz. A Poisson gamma prob- abilistic model for latent node-group memberships in dynamic networks. In AAAI, 2018. Zhou, Mingyuan. Infinite edge partition models for over- lapping community detection and link prediction. In AISTATS, pp. 1135–1143, 2015. Zhou, Mingyuan. Discussion on “Sparse graphs using ex- changeable random measures” by F. Caron and E. B. Fox. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 79(5):1359–1360, 2017.