Variability and Reproducibility of Directed and Undirected Functional MRI Connectomes in the Human Brain

entropy

Article Variability and Reproducibility of Directed and Undirected Functional MRI Connectomes in the Human Brain

Allegra Conti 1,* , Andrea Duggento 2 , Maria Guerrisi 2, Luca Passamonti 3,4,*, Iole Indovina 1,5 and Nicola Toschi 2,6,*

1 Laboratory of Neuromotor Physiology, IRCCS Santa Lucia Foundation, 00179 Rome, Italy 2 Department of Biomedicine and Prevention, University of Rome Tor Vergata, 00133 Rome, Italy 3 Department of Clinical Neurosciences, University of Cambridge, Cambridge CB2 0QQ, UK 4 Institute of Bioimaging and Molecular Physiology, National Research Council, 20090 Milano, Italy 5 Saint Camillus International University of Health and Medical Sciences, 00131 Rome, Italy 6 Department of Radiology, Athinoula A. Martinos Center for Biomedical Imaging, Boston, MA 02129, USA * Correspondence: [email protected] (A.C.); [email protected] (L.P); [email protected] (N.T.) Received: 1 May 2019; Accepted: 4 July 2019; Published: 6 July 2019

Abstract: A growing number of studies are focusing on methods to estimate and analyze the functional connectome of the human brain. Graph theoretical measures are commonly employed to interpret and synthesize complex network-related information. While resting state functional MRI (rsfMRI) is often employed in this context, it is known to exhibit poor reproducibility, a key factor which is commonly neglected in typical cohort studies using connectomics-related measures as biomarkers. We aimed to ﬁll this gap by analyzing and comparing the inter- and intra-subject variability of connectivity matrices, as well as graph-theoretical measures, in a large (n = 1003) database of young healthy subjects which underwent four consecutive rsfMRI sessions. We analyzed both directed (Granger Causality and Transfer Entropy) and undirected (Pearson Correlation and Partial Correlation) time-series association measures and related global and local graph-theoretical measures. While matrix weights exhibit a higher reproducibility in undirected, as opposed to directed, methods, this diﬀerence disappears when looking at global graph metrics and, in turn, exhibits strong regional dependence in local graphs metrics. Our results warrant caution in the interpretation of connectivity studies, and serve as a benchmark for future investigations by providing quantitative estimates for the inter- and intra-subject variabilities in both directed and undirected connectomic measures.

Keywords: functional networks; functional magnetic resonance imaging; connectome; connectivity matrices; graphs; reproducibility; granger causality; transfer entropy

1. Introduction The interest in studying directed and undirected interactions between different regions in the human brain (i.e., the functional ‘connectome’) is growing exponentially [1–7], and the advent of graph-theoretical applications to neuroscience has provided additional avenues to represent, analyze and interpret information contained in complex, possibly dynamic networks like the human connectome [8–12]. Functional connectome estimates are typically derived from neuromonitoring data (e.g., resting state functional MRI—rsfMRI), and established methods for computing whole-brain functional connectivity matrices include both undirected and directed estimators [13–15]. As is well known, the reproducibility of MRI data varies widely across modalities, and rsfMRI data are known to exhibit significant inter- and intra-subject fluctuations [16–18]. While this can significantly bias and

Entropy 2019, 21, 661; doi:10.3390/e21070661 www.mdpi.com/journal/entropy Entropy 2019, 21, 661 2 of 16 hamper the interpretation of functional connectivity studies, which do not typically include targeted scan-rescan experiments to assess reproducibility, the impact of this variability on connectome matrices and related measures has not yet been systematically investigated. In this study, we explore, quantify and compare the intra- and inter-subject variability (and hence reproducibility) of both directed and undirected resting state functional connectivity measures in a large (1003 subjects) database of high-quality rsfMRI data consisting of 4 scan sessions per subject. In detail, we investigate the reproducibility of whole-brain adjacency matrices derived from Pearson Correlation (PearC), Partial Correlation (PartC) as well as multivariate Granger Causality (mGC) and multivariate Transfer Entropy (mTE). In addition, we investigate the reproducibility of both global (whole-brain) and local (node-wise) graphs metrics calculated for all four types of adjacency matrices. While PearC and PartC estimate zero-lag, non-directional associations between time-series, mGC and mTE inherently measure directed information transfer. Such directed measures have already been successfully employed in brain connectivity studies [5,19–23] and, while it is known that mTE and mGC are equivalent in the limit of Gaussian data [24], these two estimators may diﬀer when applied when the data distribution is skewed and/or the number of data points is extremely limited. mGC is based on choice of model parameters [25], while TE is model-free and possibly a superior tool for estimating nonlinear interactions [15] like the ones commonly found in biological signals.

2. Materials and Methods

2.1. rsfMRI Data We employed rsfMRI data from 1003 subjects, part of the Human Connectome Project (HCP) (S1200 PTN release [26]). Each subject underwent a total of 4 resting state scans (2 sessions on 2 different days, 2 scans per session, 1200 timepoints/scan, TR 720 ms, TE 33.1 ms, flip angle 52, FOV 208x180, thickness 2.0 mm; 72 slices; 2.0 mm isotropic voxel size, multiband factor 8, echo spacing 0.58 ms, BW 2290 Hz/Px, 14 minutes of acquisition time/scan). Subjects were scanned on a customized Siemens Skyra 3 T scanner [26] (Smith et al. 2013) with higher spatial and temporal resolutions as compared to common fMRI acquisition protocols. Data pre-processing was state-of-the-art and included gradient distortion and motion correction, field map pre-processing using spin echo field map (specific for each scanning day), intensity normalization and bias field removal. Residual artifacts were removed through an automatic classifier specifically trained to this particular dataset, which is able to remove measurement noise, additional motion or physiological artifacts with extremely high accuracy. Further details regarding data processing can be found in the HCP S1200 Release reference manual [https://www.humanconnectome.org/storage/app/media/documentation/s1200/HCP_S1200_ Release_Reference_Manual.pdf]. In this paper, we processed subject- and scan-wise fMRI timeseries (1200 points each) resulting from group independent component analysis (gICA) at dimensionality 15 (made available by the HCP consortium). Exemplary slices of the resulting node (i.e., component, also called “subnetworks” in this paper) maps, along with their physiological interpretations [27], are shown in Figure1. To cater for confounding elements introduced by locally varying hemodynamic response functions (HRFs) [28], fMRI timeseries were preprocessed using a the blind deconvolution approach [29] (maximum lag 10 s = 14 time points, threshold: 1 standard deviation) implemented in a publicly available toolbox (https://users.ugent.be/~{}dmarinaz/HRF_deconvolution.html). Stationarity of all signals was verified using the Augmented Dickey-Fuller test. All signals were successively standardized prior to additional processing. Entropy 2019, 21, 661 3 of 16 Entropy 2019, 21, x FOR PEER REVIEW 3 of 17

Figure 1.1.Brain Brain functional functional subnetworks subnetworks (i.e., independent(i.e., indepe components)ndent components) identiﬁed byidentified group independent by group independentcomponent analysis component (gICA) analysis in 1003 (gICA) subjects in 1003 drawn subjects from drawn the Human from the Connectome Human Connectome Project database. Project database.The subject-wise The subject-wise 15-node speciﬁc15-node time-seriesspecific time-ser relativeies relative to these to components these components were employed were employed for all foranalyses all analyses in this in paper. this paper.

2.2. Estimation Estimation of of Adjacency Adjacency Matrices Matrices Starting from the 15 subject-, scan- scan- and and node-specif node-speciﬁcic timeseries timeseries (see (see Figure Figure 22AA for an example), subject-wise undirected adjacencyadjacency matricesmatrices were were obtained obtained through through both both PearC PearC and and PartC PartC between between all allpairs pairs of timeseriesof timeseries through through in-house in-house code code written written in in in MATLAB in MATLAB (v. 2018a). (v. 2018a). Negative Negative correlations correlations were wereset to set zero. to Directedzero. Directed adjacency adjacency matrices matrices where wh obtainedere obtained through through mGC estimates,mGC estimates, in its mostin its recent most recentstate-space state-space formulation formulation [30,31], [30,31], and through and through mTE. mTE. mGC and and mTE mTE were were calculated calculated through through in-house in-house modified modiﬁed versions versions of the ofMatlab the MatlabTools for Tools the computationfor the computation of multiscale of multiscale Granger GrangerCausality Causality (http://www.lucafaes (http://www.lucafaes.net.net/msGC.html)/msGC.html [31] and)[ 31of] multivariateand of multivariate Transfer Transfer Entropy Entropy (http://www.lucafa (http://www.lucafaes.netes.net/cTE.html)/cTE.html [32]. )[In32 the]. Incase the of case GC, of GC,the autoregressivethe autoregressive order order was was chosen chosen my my minimizing minimizing th thee median median Schwartz Schwartz criterion criterion across across all all subjects subjects (order = 5).5). In In mTE, mTE, conditioning conditioning vectors vectors are are fo formedrmed according according to a non-uniform building scheme selecting pastpast terms terms up up to ato maximum a maximum lag of lag 5 time of points5 time while points minimizing while minimizing the conditional the entropyconditional [32]. entropyExamples [32]. of adjacencyExamples matricesof adjacency (which matrices are symmetrical (which are forsymmetrical undirected for estimators, undirected and estimators, asymmetrical and asymmetricalfor directed ones) for directed obtained ones) through obtained all four through methods all arefour shown methods in Figureare shown2B. in Figure 2B.

Entropy 2019, 21, 661 4 of 16 Entropy 2019, 21, x FOR PEER REVIEW 4 of 17

Figure 2. (A) Example node-wise timeseries from a single scan, used to compute subject-, session- and Figure 2. (A) Example node-wise timeseries from a single scan, used to compute subject-, session- and scan-speciﬁc adjacency matrices; (B) Examples of adjacency matrices obtained from both undirected scan-specific adjacency matrices; (B) Examples of adjacency matrices obtained from both undirected (ﬁrst row) and directed (second row) connectivity estimators for one subject. For each method, (first row) and directed (second row) connectivity estimators for one subject. For each method, the the median across 4 rsfMRI sessions from a single subject is shown. Diagonal elements are set to zero. median across 4 rsfMRI sessions from a single subject is shown. Diagonal elements are set to zero. 2.3. Global and Local Graph Metric Estimation 2.3. Global and Local Graph Metric Estimation Starting from the adjacency matrices, we computed both global and local graph-theoretical indices of functionalStarting connectivityfrom the adjacency for all three matrices, methods we/estimators. computed All both graph global measures and local were graph-theoretical computed via the indicesBrain Connectivity of functional Toolbox connectivity [33](https: for// sites.google.comall three methods/estimators./site/bctnet/), by usingAll graph functions measures available were for computed via the Brain Connectivity Toolbox [33] (https://sites.google.com/site/bctnet/), by using

Entropy 2019, 21, 661 5 of 16 Entropy 2019, 21, x FOR PEER REVIEW 5 of 17 weightedfunctions available adjacency for matrices weighted for ad bothjacency directed matrices (in case forof both mGC directed and mTE) (in case and of undirected mGC and (formTE) pearC and undirectedand partC) graph(for pearC metrics. and partC) graph metrics. For each subject and for each scan, we calculated graph metrics quantifying the centrality of a node within a network (local strength and betweenness centrality), its ability to transmit information at local level (local eefficiency)fficiency) and its integrationintegration propertiesproperties (clustering(clustering coecoefficient)fficient) [[33].33]. Also, since global graph metrics characterize the overall organization of a network, we computed the global strength, thethe globalglobal e efficiencyfficiency and and the the global global clustering clustering coe coefficientfficient as theas the average average of the of respectivethe respective local localmetrics metrics of all nodes.of all nodes. In addition, In addition, we calculated we calculat graphed transitivity graph transitivity (a property (a related property to therelated existence to the of existencetightly connected of tightly communities connected ofcommunities nodes). Figure of3 nodes).exemplifies Figure the meaning3 exemplifies and calculation the meaning of these and calculationgraph-theoretical of these metrics. graph-theoretical metrics.

Figure 3. ExemplificationExemplification of of local local (left) (left) and and global global (right) (right) graph graph metrics metrics as as a function of node neighborhood connectivity. Local metrics (left): taki takingng the the yellow/red yellow/red center center node node as a reference, its betweenness centrality and local efficiency efficiency increase as the average shortest path between the center node and all its neighbors decreases. Strength/degr Strength/degreeee is is proportional to to the number of connections, the clusteringclustering coe coefficientfficient increases increases with with the the ratio rati betweeno between the number the number of links of actually links actually present betweenpresent betweenthe vertices the withinvertices node within neighborhood node neighborhood and the numberand the ofnumber links thatof links could that possibly could existpossibly between exist betweenthem. Global them. metrics Global (right): metrics transitivity (right): transitivity and clustering and clustering coefficient coeffi bothcient reflect both prevalence reflect prevalence of clustered of clusteredconnectivity connectivity around nodes, around while nodes, efficiency while and efficiency strength and increase strength with increase the increase with of the the increase local effi ciencyof the localand strength efficiency of and each strength node in of the each graph node [33 in]. the graph [33]. 2.4. Inter- and Intra-Subject Variability Distributions 2.4. Inter- and Intra-Subject Variability Distributions Adjacency matrix weights (both directed and undirected) and both local and global Adjacency matrix weights (both directed and undirected) and both local and global graphgraph-theoretical measures, have their own native scales and dimensionalities and, in the case theoretical measures, have their own native scales and dimensionalities and, in the case of local graph of local graph metrics, this scale also depends on the specific brain region. The same issue, which metrics, this scale also depends on the specific brain region. The same issue, which results in results in dimensional and extremely different effect sizes across estimators, also affects graph-theoretical dimensional and extremely different effect sizes across estimators, also affects graph-theoretical metrics calculated from different estimators. Therefore, in order to compare inter- and intra-subject metrics calculated from different estimators. Therefore, in order to compare inter- and intra-subject fluctuations in adjacency matrices, as well as graph-theoretical metrics, we employed dimensionless, fluctuations in adjacency matrices, as well as graph-theoretical metrics, we employed dimensionless, normalized pairwise quantifiers of asymmetry, along with their estimated distributions. normalized pairwise quantifiers of asymmetry, along with their estimated distributions. In detail, we defined the pairwise normalized difference/asymmetry (ND) as: In detail, we defined the pairwise normalized difference/asymmetry (ND) as: a b NDND== − (1)(1) a+ b where a and b refer to two distinct scans (either intra- or inter-subject). ND is both normalized (i.e., a- where a and b refer to two distinct scans (either intra- or inter-subject). ND is both normalized dimensional) and bounded (i.e., between −1 and 1). (i.e., a-dimensional) and bounded (i.e., between 1 and 1). In the case of scalar quantities like global or− local graph-theoretical metrics, Equation (1) can be applied directly. When comparing adjacency matrices, Equation 1 was modified by using Frobenius distances between (in this case) matrices a and b as follows:

∙ ND= (2) ∙

Entropy 2019, 21, 661 6 of 16

In the case of scalar quantities like global or local graph-theoretical metrics, Equation (1) can be applied directly. When comparing adjacency matrices, Equation (1) was modified by using Frobenius distances between (in this case) matrices a and b as follows: s trace((a b) (a b)0) ND = − · − (2) trace((a + b) (a + b)0) Entropy 2019, 21, x FOR PEER REVIEW · 6 of 17 To empirically estimate (and successively compare) the intra- and inter-subject distributions in ND,To we empirically proceeded estimate as follows. (and Fromsuccessively each set compare) of 4 subject-wise the intra- scans, and inter-subject we sampled distributions all 6 possible in pairs ND, of scans,we proceeded computed as the follows. ND for From each each pair, set and of repeated4 subject-wise this step scans, for we all sampled 1003 subjects. all 6 possible For each pairs method, of thisscans, resulted computed in 1003 the ND6 = 6018for each distinct pair, NDand valuesrepeated referring this step to for intra-subject all 1003 subjects. variability. For each Subsequently, method, × inthis order resulted to build in 1003 a comparable × 6 = 6018 distinct distribution ND values related referring to inter-subject to intra-subject variability, variability. we Subsequently, randomly and in order to build a comparable distribution related to inter-subject variability, we randomly and repeatedly sampled (without replacement) four scans from four distinct subjects, hence constructing repeatedly sampled (without replacement) four scans from four distinct subjects, hence constructing set of 4 “inter-subject” scans from which 6 distinct pairs could be constructed and 6 distinct ND values set of 4 “inter-subject” scans from which 6 distinct pairs could be constructed and 6 distinct ND values could be calculated as above. To control for possible differences arising from intra-subject habituation, could be calculated as above. To control for possible differences arising from intra-subject the random sampling was performed so that each “inter-subject” set of four scans contained exactly 1 habituation, the random sampling was performed so that each “inter-subject” set of four scans sample from the 1003-sized set of first, second, third and fourth scans. This procedure was repeated contained exactly 1 sample from the 1003-sized set of first, second, third and fourth scans. This 1003 times, resulting in 1003 6 = 6018 distinct ND values referring to inter-subject variability. Figure4 procedure was repeated 1003× times, resulting in 1003 × 6 = 6018 distinct ND values referring to inter- summarizessubject variability. the sampling Figure process. 4 summarizes the sampling process.

FigureFigure 4. Sampling4. Sampling strategy strategy used used to construct to construct intra- andintra- inter-subject and inter-subject pairs and pairs subsequent and subsequent distributions ofdistributions ND (Equations of ND (1) and(Equations (2)). The 1 constructedand 2). The constructed samples for samples the basis for for the computing basis for computing the distributions the aredistributions shown, e.g., are in Figuresshown, 5e. andg., in6 .Figures 5 and 6. 2.5. Statistical Analysis 2.5. Statistical Analysis AfterAfter sampling sampling and and computation computation of of all all ND ND values values for for each each graph-theoretical graph-theoretical metric metric and and for for all all four typesfour of types adjacency of adjacency matrix, matrix, the resulting the resulting “inter” and“inter” “intra” and distribution“intra” distribution medians medians of ND were of ND compared were throughcompared nonparametric through nonparametric Mann-Whitney-U Mann-Whitney-U tests. Eﬀect test Sizes. Effect (ES) Size were (ES) evaluated were evaluated as the di asﬀerence the between the medians of the “inter” and “intra” distributions in ND (i.e., ES = “inter” “intra”). difference between the medians of the “inter” and “intra” distributions in ND (i.e., ES = “inter”− - “intra”). The result of each comparison is therefore a p-value with an associated ES. Additionally, in order to explore possible differences in variability across nodes (i.e., brain regions), we pooled ND values for all local graph-theoretical metrics across all four methods (4 metrics x four methods = 16 values per node), and tested the effect of anatomical localization using a nonparametric Kruskal- Wallis test. Whenever a significant effect was found, pairwise comparison between nodes was performed in order to further examine which pairwise differences were the main drivers of the overall effect. This procedure was repeated separately for inter- and intra-subject variability.

Entropy 2019, 21, 661 7 of 16

The result of each comparison is therefore a p-value with an associated ES. Additionally, in order to explore possible differences in variability across nodes (i.e., brain regions), we pooled ND values for all local graph-theoretical metrics across all four methods (4 metrics four methods = 16 values per node), × and tested the effect of anatomical localization using a nonparametric Kruskal-Wallis test. Whenever a significant effect was found, pairwise comparison between nodes was performed in order to further examine which pairwise differences were the main drivers of the overall effect. This procedure was repeatedEntropy 2019 separately, 21, x FOR for PEER inter- REVIEW and intra-subject variability. 7 of 17 3. Results 3. Results FigureFigure5 shows 5 shows the the “inter” “inter” and and “intra” “intra” distribution distribution forfor NDND obtained when when comparing comparing pairs pairs of of adjacencyadjacency matrices matrices (Equation (Equation (2))(2)) forfor allall fourfour methods (PearC, (PearC, PartC, PartC, mGC mGC and and mTE), mTE), along along with with effecteffect sizes. sizes. For For all all methods, methods, as as hypothesized, hypothesized, medianmedian intra-subject variability variability is issignificantly significantly lower lower 20 thanthan median median inter inter subject subject variability variability ( p(p< <10 10−−20 forfor all all methods). methods). The The largest largest ES ES was was associated associated with with mGCmGC and and PearC PearC (ES (ES= =0.03 0.03 for for both),both), followedfollowed by PartC and and mTE mTE (ES (ES == 0.020.02 for for both). both). Qualitatively, Qualitatively, mTEmTE showed showed the the largest largest median median variabilities variabilities (both intra- and and inter-subject), inter-subject), whereas whereas PearC PearC and and PartC PartC showedshowed similar similar median median variabilities. variabilities.

FigureFigure 5. 5.Intra- Intra- (blue) (blue) and and inter- inter- (orange) (orange) subject subject distributions distributions of of the the Normalized Normalized Di Differencesfferences (ND), (ND), for allfor four all connectivity four connectivity estimation estimation methods. methods. Effect Effect size (ES)size (i.e.,(ES) the(i.e., di thefference difference between between the median the median values ofvalues the inter- of the and inter- intra-subject and intra-subject ND distributions) ND distributions) are shown are shown in the in insets.the insets. Mann-Whitney Mann-Whitney U-tests U-tests for 20 −20 comparingfor comparing medians medians returned returnedp < 10 p −< 10forfor all all four four methods. methods.

FigureFigure6 shows6 shows the the “inter” “inter” and and “intra” “intra” distribution distribution forfor NDND (Equation (1)) (1)) obtained obtained when when comparingcomparing pairs pairs of of global global graph-theoretical graph-theoretical metrics, along along with with the the results results of of statistical statistical testing testing in in termsterms of ofp-values p-values and and ES. ES. Interestingly, Interestingly, intra- intra- andand inter-subjectinter-subject fluctuations fluctuations are are distinguishable distinguishable only only inin some some cases. cases. In In particular, particular, wewe foundfound statisticallystatistically significant significant differences differences (p ( p< <0.05)0.05) for for all all metrics metrics estimatedestimated through through PearC PearC (named (named “Bivariate “Bivariate Correlation” Correlation” in figure)in figure) and and PartC. PartC. However, However, in thesein these cases, intra-subjectcases, intra-subject fluctuations fluctuations were larger were thanlarger inter-subject than inter-subject fluctuations fluctuations (i.e., (i.e., negative negative ES). ES). On theOn the other hand,other when hand, looking when looking global metricsglobal metrics calculated calculated through through mGC, mGC, significant significant differences differences between between intra- vs.intra- inter-subject vs. inter-subject fluctuations fluctuations were mainly were mainly associated associated with positivewith positive ES. No ES. di Nofferences differences were were found found between ND distributions for global-graphs metrics evaluated through mTE. Notably, in this between ND distributions for global-graphs metrics evaluated through mTE. Notably, in this analysis, analysis, all significant differences were associated with minimal (compared to unity) effect sizes (of all significant differences were associated with minimal (compared to unity) effect sizes (of the order of the order of between 10−2 and 10−3). In addition, the widths of intra- and inter-subject variability between 10 2 and 10 3). In addition, the widths of intra- and inter-subject variability distributions distributions− were qualitatively− similar for all methods. The main analyses presented in Figures 5 and were qualitatively similar for all methods. The main analyses presented in Figures5 and6 were also 6 were also repeated using a finer parcellation (50 subnetworks—see supplementary materials (See repeatedFigures usingA1 and a finerA2)). parcellationThe results were (50 subnetworks—seefully compatible (both supplementary in effect sizes materialsand directions) (See Figureswith the A1 andmain A2)). results The resultsobtained were using fully 15 subnetworks. compatible (both in effect sizes and directions) with the main results obtained using 15 subnetworks.

Entropy 2019, 21, 661 8 of 16 Entropy 2019, 21, x FOR PEER REVIEW 8 of 17

FigureFigure 6.6. Intra-Intra- (gray) and and inter- inter- (orang (orange)e) subject subject distributions distributions of the of the Normalized Normalized Differences Differences (ND), (ND), forfor allall fourfour connectivity estimation estimation methods, methods, alon alongg with with the the p-value p-value resultin resultingg from from the thecorresponding corresponding Mann-Whitney-UMann-Whitney-U test. Effect Effect size size (ES) (ES) (i.e., (i.e., the the diff dierencefference between between the the median median values values of the of inter- the inter- and and intra-subjectintra-subject NDND distributions)distributions) are are shown shown in in the the insets. insets.

FigureFigure7 7 summarizessummarizes the the results results of comparin of comparingg ND distributions ND distributions computed computed from local from graph- local graph-theoreticaltheoretical metrics, metrics, along with along the results with the of statistica results ofl testing statistical as well testing as ES. asIn wellthis case, as ES. an intricate, In this case, ananatomically intricate, anatomically dependent pattern dependent emerged, pattern where emerged, the statistical where thedifference statistical between difference median between intra- and median intra-inter-subject and inter-subject fluctuations fluctuations depends on depends the specific on the node, specific with node, some with nodes some exhibiting nodes exhibitinghigher intra- higher intra-subjectsubject (as compared (as compared to inter-subject) to inter-subject) fluctuations. fluctuations. Asterisks Asterisks indicate indicate nodes nodes in which in which differences differences between inter- and intra-subject variabilities resulted in be different from 0 (p < 0.05, Mann-Whitney between inter- and intra-subject variabilities resulted in be different from 0 (p < 0.05, Mann-Whitney U test). These results highlight the strong spatial dependence across the brain in reproducibility of U test). These results highlight the strong spatial dependence across the brain in reproducibility of all local graph-theoretical indices. Only the betweenness centrality of the Hippocampal-Posterior all local graph-theoretical indices. Only the betweenness centrality of the Hippocampal-Posterior Cingulate subnetwork show statistically higher intra-subject (as opposed to inter-subject) variability Cingulate subnetwork show statistically higher intra-subject (as opposed to inter-subject) variability across all methods. Still, ES in local efficiency and strength estimated in the same subnetwork tend to acrossthe same all methods.direction in Still, all methods. ES in local Higher efficiency intra-subject and strength (as opposed estimated to inter-subject) in the same variability subnetwork of the tend tobetweenness the same direction centrality in and all methods.local efficiency Higher and intra-subject strength was (as also opposed found for to inter-subject)the salience and variability the ofsensory the betweenness motor subnetworks. centrality On and the localother ehand,fficiency the default and strength mode subnetwork was also found displayed for thehigher salience intra- and thesubject sensory (as motoropposed subnetworks. to inter-subject) On thestability other in hand, both the betweenness default mode centrality subnetwork and local displayed clustering higher intra-subjectcoefficient across (as opposed methods. to inter-subject)The same hold stability for the in betweenness both betweenness centrality centrality of the andfronto-temporal local clustering coesubnetwork.fficient across The results methods. shown The in sameFigure hold 7 allow for also the to betweenness qualitatively centrality compare the of thevariability fronto-temporal of the subnetwork.local metrics Theobtained results though shown undirected in Figure (PearC7 allow an alsod PartC) to qualitatively and directed compare (mGC and the mTE) variability methods. of the localIn particular, metrics obtained local efficiency, though strength undirected and (PearCclustering and coefficient PartC) and in the directed visuo-prefrontal (mGC and subnetwork mTE) methods. Inare particular, highly reproducible local efficiency, within strength the same and subject clustering when coe estimatedfficient inusing the mGC visuo-prefrontal and mTE. subnetwork are highly reproducible within the same subject when estimated using mGC and mTE.

Entropy 2019, 21, 661 9 of 16 Entropy 2019, 21, x FOR PEER REVIEW 9 of 17

FigureFigure 7. (A 7.)E (Aff)ect Effect size, size, i.e., i.e., di differencesfferences between between median intra- intra- and and inter-subject inter-subject distributions distributions in local in local graphgraph metrics metrics for for each each node node and and for for each each connectivity connectivity estimation estimation method method (PearC, (PearC, PartC, PartC, mGC mGC and and mTE).mTE). For eachFor metric,each metric, darker darker red and red blue and cells blue correspond cells correspond to higher to andhigher lower and ESlower values, ES respectively.values, The nodesrespectively. in which The di nodesfferences in which between differences inter- and be intra-subjecttween inter- and variabilities intra-subject are statisticallyvariabilities diarefferent fromstatistically 0 (p < 0.05) different are indicated from 0 with(p < 0.05) an asterisk; are indicated (B) Subnetworks with an asterisk; (i.e., ( components)B) Subnetworks with (i.e., larger intra-subjectcomponents) than with inter-subject larger intra-subject variability than in inter- varioussubject metrics variability are shown in various in blue, metrics while are shown subnetworks in blue, while subnetworks showing a larger intra-subject reproducibility are shown in red. showing a larger intra-subject reproducibility are shown in red. With reference to Figure 7, Table 1 summarizes the number of nodes where statistically With reference to Figure7, Table1 summarizes the number of nodes where statistically significant significant intra- vs. inter-subject differences were found. Overall, across all four methods (PearC, intra-PartC, vs. inter-subjectmGC and mTE) di thefferences percentage were of found.nodes (out Overall, of 15) where across inter- all and four intra-subject methods (PearC,variability PartC, mGCwere and statistically mTE) the different percentage varied of between nodes (out 28% of and 15) 38% where (bottom inter- row and of Table intra-subject 1), and in 3–35% variability of the were statistically15 nodes, di fftheerent inter-subject varied between variability 28% is andsignificantl 38% (bottomy higher row than of the Table intra-subject1), and in variability. 3–35% of the mGC 15 nodes,is the inter-subjectthe method with variability the lowest is significantly average number higher of thannodes the (across intra-subject metrics) where variability. the above-mentioned mGC is the method withdifference the lowest is averagesignificantly number different of nodesfrom zero. (across Also, metrics) node strength where is the the above-mentioned metric where the above- difference is significantlymentioned difference different fromis significantly zero. Also, different node from strength zero isin thethe highest metric whereaveragethe number above-mentioned of nodes difference(across is methods). significantly Importantly, different in from the absence zero in of the a gr highestound-truth, average it is numbernot possible of nodesto determine (across which methods). Importantly,source of in variability the absence dominates of a ground-truth, the overall effects. it is not This possible could only to determine be ascertained which with source synthetic of variability data which accurately models neuronal spiking and cardiovascular coupling [9]. dominates the overall effects. This could only be ascertained with synthetic data which accurately Figure 8 shows the results the Kruskal-Wallis test performed separately on the “intra” and on modelsthe neuronal“inter” ND spiking values. andIn both cardiovascular intra- and inter-subject coupling variability [9]. we found a significant effect of node Figure(p = 1.1e-4).8 shows In post-hoc the results comparisons the Kruskal-Wallis (Figure 8B) we test found performed that overall separately intra-subject on thevariability “intra” in and on thelocal “inter” graph-theoretical ND values. metrics In both of intra- the fronto-tem and inter-subjectporal subnetwork variability is lower we found as compared a significant to the e ffect of nodehippocampal-posterior (p = 1.1 10 4). Incingulate, post-hoc the comparisons cingulate cortex (Figure and8 B)the we fronto-polar found that subnetworks. overall intra-subject Also, × − variabilityoverall in inter-subject local graph-theoretical variability in local metrics graph-theore of the fronto-temporaltical metric is lower subnetwork in the salience is lower subnetwork as compared to theas hippocampal-posterior compared to the hippocampal-cerebellar, cingulate, the cingulate fronto-temporal cortex and and sensory/motor-limbic the fronto-polar subnetworks. subnetworks. Also, overall inter-subject variability in local graph-theoretical metric is lower in the salience subnetwork as compared to the hippocampal-cerebellar, fronto-temporal and sensory/motor-limbic subnetworks. Entropy 2019, 21, x FOR PEER REVIEW 10 of 17

Table 1. Number of nodes showing statistically different intra- and inter-subject variabilities. In Entropy 2019, 21, 661 10 of 16 brackets: (number of nodes showing higher inter- vs intra-subject variability/number of nodes showing higher intra- vs inter-subject variability). The right column shows the average percentage of Table1.nodes (standardNumber of deviation nodes showing in brackets), statistically across different metrics, intra- andwhere inter-subject the inter-subject variabilities. variability In brackets: is (numbersignificantly of nodes higher showing than the higher intra-subject inter- vs. one. intra-subject The bottom variability row shows/number the percentage of nodes count showing of the higher first intra-number vs. in inter-subject each cell. variability). The right column shows the average percentage of nodes (standard deviation in brackets),Betweenness across metrics, whereLocal the inter-subject variabilityClustering is significantly higherMean(sd) than the% out intra-subjectMethod one. The bottom row shows the percentageStrength count of the first number in each cell. Centrality Efficiency Coefficient of 15 PearsC 3(2/1)Betweenness 4(0/4)Local 5(1/4) Clustering3(0/3) Mean(sd)5%(6%) % Method Strength PartC 7(3/4)Centrality 5(0/5)Efficiency 7(2/5) Coe2(0/2)fficient out8%(9%) of 15 mGCPearsC 4(2/2) 3(2 /1)5(2/3) 4(0 /4)5(2/3) 5(1 /4)15(15/0) 3(0/3) 5%(6%)35%(38%) mTE PartC3(1/2) 7(3 /4)3(0/3) 5(0 /5)4(1/3) 7(2 /5)3(0/3) 2(0/2) 8%(9%)3%(3%) Mean(sd) mGC% 4(2/2) 5(2/3) 5(2/3) 15(15/0) 35%(38%) mTE28%(13%) 3(1 /2)28%(6%) 3(0/3) 35%(8%) 4(1/ 3)38%(41%) 3(0/3) 3%(3%) out of 15 Mean(sd) % out of 15 28%(13%) 28%(6%) 35%(8%) 38%(41%)

Figure 8. (A) Intra and inter ND values (across 1003 subjects; left: intra-subject variability, right: Figure 8. (A) Intra and inter ND values (across 1003 subjects; left: intra-subject variability, right: inter- inter-subject variability) for all local metrics (B) p-values obtained from pairwise comparison of subject variability) for all local metrics (B) p-values obtained from pairwise comparison of node- node-speciﬁc, intra- and inter-subject variability. specific, intra- and inter-subject variability.

Entropy 2019, 21, 661 11 of 16

4. Discussion In this paper, we have studied the inter- and intra-subject variability of directed and undirected connectivity matrices as well as of global as well as local graph-theoretical metrics derived from resting state functional MRI data in a large sample (n = 1003) of resting state fMRI data, which also has the unique property of including scan-rescan sessions. While previous reproducibility studies exist, they are commonly focused on specific data processing [34] or statistical [35] aspects, and are usually based on limited sample sizes. We employed a fully data-driven parcellation, derived from group independent component analysis performed by the HCP consortium and which is hence independent from any anatomical priors commonly present in classical parcellations. To further eliminate any unwanted sources of variability and improve the comparability between distributions, we adopted a bootstrap approach followed by nonparametric statistics on a normalized difference metric, which also allowed us to pool across methods and/or nodes and make qualitative inter-method comparisons. In addition, in order to investigate the dependence of our results on ICA dimensionality (and hence coarseness/fineness of the resulting parcellation), we repeated the main analyses using data based on a dimensionality of 50 instead of 15. Interestingly, all effect direction and sizes are highly comparable to the main results ad dimensionality 15. Our results show that while adjacency matrix weights generated using conventional methods like PearC or PartC exhibit 1) a qualitatively lower variability as compared to directed metrics and 2) a lower intra-subject vs. inter-subject variability; these differences virtually disappear when looking at global and local graphs-metrics values, where re-scanning the same subject results in the same statistical fluctuation as scanning a different subject. The overall higher variability in adjacency matrices obtained from directed metrics may be partially due to the higher complexity of both estimators and the multiple degrees of freedom (e.g., model order, embedding dimensions) in their computation, whose exploration was beyond the scope of this study. Also, the appropriateness of using of mGC in fMRI has been the object of discussion [36–42], mainly because of the confounding elements introduced by locally varying neurovascular coupling and the interaction of these confounds with, e.g., parameter choices, and hence the overall validity of the estimators. To this end, we applied a blind deconvolution approach that estimates a local HRF for each signal and returns an estimate of neural activity. Still, the application of this type of pipeline is not yet widespread in fMRI studies using causality methods, and that it also involves parameter choices whose effect on reproducibility merit investigation in a separate paper. Similarly, while partial and Pearson correlation do not involve parameter optimization, both mGC and mTE estimations are based on parameter choices inherent to each method. In this sense, the main aim of our work was to not to directly compare across methods, but rather to provide an estimation, in a high-quality and unique data sample, of overall the intra- and inter-subject reproducibility of a variety of connectomic metrics which are commonly used in the fMRI community. Interestingly, we found an articulate pattern of “intra” vs. “inter” variability differences as a function of metrics, as well as node (i.e., anatomical localization), which points towards a certain degree of unpredictability in the reproducibility of results obtained using each possible composite pipeline. Interestingly, in several nodes and metrics, median intra-subject variability was found to be larger than median inter-subject variability. In this context, previous studies have highlighted the higher intra-subject (as compared to inter-subject) variability of functional connectivity in specific regions of the brain [43–46]. Indeed, intra-subject variability depends on several factors such as diurnal rhythms [47,48], cognitive/behavioral contexts [49–51], caffeine intake [43], sleepiness [52–54] and attention [46,55], which may explain the high inter-subject fluctuations we found in this paper.

Limitations and Future Perspectives We employed public HCP time-series data stemming from a group ICA at dimensionality 15. Several previous investigations have extracted useful information employing “small” networks and graph metrics [10,20,56,57]. Also, additional analyses performed at higher dimensionality confirm Entropy 2019, 21, 661 12 of 16 the main results of this paper. However, we cannot exclude that the fluctuations quantified in this study may change as a function of parcellation, and should therefore be investigated ad hoc on the basis of data specifications and parcellation details in any specific study where a reproducibility assessment is warranted, e.g., in longitudinal investigations. Also, in view of the advent of novel connectivity estimators, like, e.g., Generalized Recurrent Neural Network-based Dynamic Causal Modeling [58] or model-based whole-brain effective connectivity (MOU-EC) [59], future investigations are needed to characterize the repeatability and robustness of the connectivity matrices generated by such strategies. In addition, since dynamical connectivity measures and connectivity density are also likely to be impacted by reproducibility issues [60], a study of the robustness of dynamic functional connectomes, based on nonparametric distribution-building approaches like the one we employed in this paper, appears warranted. Finally, future investigation of the reproducibility of the regional dependence of functional connectivity by using large-scale brain network models would also be useful for better understanding the causes of intra- and inter-subject fluctuations in conventional connectivity estimators we observed in this study.

5. Conclusions In this paper, we have studied inter- and intra-subject variability of directed and undirected connectivity matrices, as well as of global and local graph-theoretical metrics derived from resting state functional MRI data in a large sample (n = 1003) of resting state fMRI data released by the HCP. We analyzed both directed (Granger Causality and Transfer Entropy) and undirected (Pearson Correlation and Partial Correlation) association measures and derived graph-theoretical measures. Our results show that adjacency matrices exhibit a higher intra-subject (as compared to inter-subject) reproducibility. However, intra- and inter-subject dispersion of both global and local graphs-metrics resulted as being comparable. We therefore demonstrate that, even with large volumes of densely sampled rsfMRI data, reproducibility is an issue that should be constantly kept in mind and that connectomics studies should, whenever possible, include a scan-rescan validation arm, at least in a subset of subjects. Our results also serve as benchmarks for future investigations aiming to estimate the variability in other directed and undirected connectomic databases, and can also be employed for prospective power calculations in planning functional connectomics experiments.

Author Contributions: Conceptualization, N.T.; methodology, N.T., A.C. and A.D.; analysis, A.C.; investigation, N.T., A.C., A.D, L.P. and M.G.; writing—original draft preparation, N.T. and A.C.; supervision, N.T.; funding acquisition N.T., L.P. and I.I. All authors contributed to manuscript revision and editing. All authors contributed to paper validation and approved the final draft for submission. Funding: Research supported by the Italian Ministry of Health (PE- 2013-02355372) and by the Medical Research Council (MRC) (MR/P01271X/1) at the University of Cambridge. Acknowledgments: Data were provided by the Human Connectome Project, WU-Minn Consortium (Principal Investigators: David Van Essen and Kamil Ugurbil; 1U54MH091657) funded by the 16 NIH Institutes and Centers that support the NIH Blueprint for Neuroscience Research; and by the McDonnell Center for Systems Neuroscience at Washington University. This work was supported by the Italian Ministry of Health (PE-2013-02355372 and IRCCS Fondazione Santa Lucia Ricerca corrente). Luca Passamonti is funded by the Medical Research Council (MRC) (MR/P01271X/1) at the University of Cambridge Conflicts of Interest: The authors declare no conflict of interest. Entropy 2019, 21, 661 13 of 16 Entropy 2019, 21, x FOR PEER REVIEW 13 of 17

Appendix A. Variability and Reproducibility of Directed and Undirected Functional MRI AppendixConnectomes A Variability in the Human and Reproducibility Brain of Directed and Undirected Functional MRI Connectomes in the Human Brain

Figure A1. Intra- (blue) and inter- (orange) subject distributions of the Normalized Differences (ND), Figure 1. Intra- (blue) and inter- (orange) subject distributions of the Normalized Differences (ND), for adjacency matrices obtained from the ICA decomposition with dimensionality of 50, for all four for adjacency matrices obtained from the ICA decomposition with dimensionality of 50, for all four connectivityconnectivity estimation estimation methods. methods. Effect Effect size size (ES) (ES) (i.e., (i.e.,the difference the diff betweenerence the between median values the median of the values of the inter-inter- and and intra- intra- subject subject NDND di distributions)stributions) are areshown shown in the in insets. the insets.Mann-Whitney Mann-Whitney U-tests for U-tests for Entropycomparing 2019, 21 ,medians x FOR PEER returned REVIEW p < 10−20 for all four methods. 14 of 17 comparing medians returned p < 10− for all four methods.

Figure A2.FigureIntra- A2. (black) Intra- (black) and inter-and inter- (yellow) (yellow) subjectsubject distributions distributions of the of Normalized the Normalized Differences Di (ND),fferences (ND), for all fourfor connectivity all four connectivity estimation estimation methods, methods, using using ICA-derived ICA-derived time-seriestime-series at at dimensionality dimensionality 50. 50. Effect size (ES) (i.e.,Effect the size di (ES)fference (i.e., the between difference the between median the values median of values the inter- of the and inter- intra- and intra- subject subject ND ND distributions) together withdistributions) the p-values together resulting with the p from-values the resulting Mann-Whitney-U from the Mann-Whitney-U tests, are tests, shown are shown in the in insets. the insets.

References

Entropy 2019, 21, 661 14 of 16

References

1. Wang, H.E.; Bénar, C.G.; Quilichini, P.P.; Friston, K.J.; Jirsa, V.K.; Bernard, C. A systematic framework for functional connectivity measures. Front. Mol. Neurosci. 2014, 8, 8. [CrossRef][PubMed] 2. Blinowska, K.J. Review of the methods of determination of directed connectivity from multichannel data. Med. Boil. Eng. 2011, 49, 521–529. [CrossRef][PubMed] 3. Bullmore, E.; Sporns, O.; Bullmore, E. The economy of brain network organization. Nat. Rev. Neurosci. 2012, 13, 336–349. [CrossRef][PubMed] 4. Fornito, A.; Zalesky, A.; Bullmore, E.T. (Eds.) Chapter 3—Connectivity Matrices and Brain Graphs. In Fundamentals of Brain Network Analysis; Academic Press: San Diego, CA, USA, 2016; pp. 89–113. 5. Schmidt, C.; Pester, B.; Schmid-Hertel, N.; Witte, H.; Wismüller, A.; Leistritz, L. A Multivariate Granger Causality Concept towards Full Brain Functional Connectivity. PLoS ONE 2016, 11, 0153105. [CrossRef] [PubMed] 6. Schoffelen, J.-M.; Hultén, A.; Lam, N.; Marquand, A.F.; Uddén, J.; Hagoort, P. Frequency-specific directed interactions in the human brain network for language. Proc. Natl. Acad. Sci. USA 2017, 114, 8083–8088. [CrossRef][PubMed] 7. Salvador, R.; Suckling, J.; Schwarzbauer, C.; Bullmore, E.; Bullmore, E. Undirected graphs of frequency-dependent functional connectivity in whole brain networks. Philos. Trans. R. Soc. B Boil. Sci. 2005, 360, 937–946. [CrossRef] 8. Sporns, O. Graph theory methods: Applications in brain networks. Dialogues Clin. Neurosci. 2018, 20, 111–121. 9. Duggento, A.; Passamonti, L.; Valenza, G.; Barbieri, R.; Guerrisi, M.; Toschi, N. Multivariate Granger causality unveils directed parietal to prefrontal cortex connectivity during task-free MRI. Sci. Rep. 2018, 8, 5571. [CrossRef] 10. Toschi, N.; Riccelli, R.; Indovina, I.; Terracciano, A.; Passamonti, L. Functional connectome of the five-factor model of personality. Pers. Neurosci. 2018, 1, 1. [CrossRef] 11. Vecchio, F.; Miraglia, F.; Rossini, P.M. Connectome: Graph theory application in functional brain network architecture. Clin. Neurophysiol. Pract. 2017, 2, 206–213. [CrossRef] 12. Fornito, A.; Zalesky, A.; Bullmore, E.T. (Eds.) Chapter 1—An Introduction to Brain Networks. In Fundamentals of Brain Network Analysis; Academic Press: San Diego, CA, USA, 2016; pp. 1–35. 13. Deshpande, G.; LaConte, S.; James, G.A.; Peltier, S.; Hu, X. Multivariate Granger causality analysis of fMRI data. Hum. Brain Mapp. 2009, 30, 1361–1373. [CrossRef][PubMed] 14. Baccala, L.A.; Sameshima, K. Partial directed coherence: A new concept in neural structure determination. Boil. Cybern. 2001, 84, 463–474. [CrossRef][PubMed] 15. Vicente, R.; Wibral, M.; Lindner, M.; Pipa, G. Transfer entropy—A model-free measure of effective connectivity for the neurosciences. J. Comput. Neurosci. 2011, 30, 45–67. [CrossRef][PubMed] 16. Zhang, N.; Wang, F.; Turner, G.H.; Gore, J.C.; Avison, M.J.; Chen, L.M. Intra- and inter-subject variability of high field fMRI digit maps in somatosensory area 3b of new world monkeys. Neuroscience 2010, 165, 252–264. [CrossRef][PubMed] 17. Smith, S.M.; Beckmann, C.F.; Ramnani, N.; Woolrich, M.W.; Bannister, P.R.; Jenkinson, M.; Matthews, P.M.; Mcgonigle, D.J. Variability in fMRI: A re-examination of inter-session differences. Hum. Brain Mapp. 2005, 24, 248–257. [CrossRef][PubMed] 18. Seghier, M.L.; Price, C.J. Interpreting and Utilising Intersubject Variability in Brain Function. Trends Cogn. Sci. 2018, 22, 517–530. [CrossRef] 19. Vakorin, V.A.; Ross, B.; Krakovska, O.; Bardouille, T.; Cheyne, D.; McIntosh, A.R. Complexity analysis of source activity underlying the neuromagnetic somatosensory steady-state response. NeuroImage 2010, 51, 83–90. [CrossRef] 20. Baig, M.Z.; Kavakli, M. Connectivity Analysis Using Functional Brain Networks to Evaluate Cognitive Activity during 3D Modelling. Brain Sci. 2019, 9, 24. [CrossRef] 21. Shovon, M.H.; Nandagopal, N.; Vijayalakshmi, R.; Du, J.T.; Cocks, B. Directed Connectivity Analysis of Functional Brain Networks During Cognitive Activity Using Transfer Entropy. Neural Process. Lett. 2017, 45, 807–824. [CrossRef] Entropy 2019, 21, 661 15 of 16

22. Duggento, A.; Bianciardi, M.; Passamonti, L.; Wald, L.L.; Guerrisi, M.; Barbieri, R.; Toschi, N. Globally conditioned Granger causality in brain–brain and brain–heart interactions: A combined heart rate variability/ultra-high-field (7 T) functional magnetic resonance imaging study. Philos. Trans. R. Soc. A Math. Phys. Eng. Sci. 2016, 374, 20150185. [CrossRef] 23. Marinazzo, D.; Liao, W.; Chen, H.; Stramaglia, S. Nonlinear connectivity by Granger causality. NeuroImage 2011, 58, 330–338. [CrossRef][PubMed] 24. Barnett, L.; Barrett, A.B.; Seth, A.K. Granger Causality and Transfer Entropy Are Equivalent for Gaussian Variables. Phys. Rev. Lett. 2009, 103, 238701. [CrossRef][PubMed] 25. Seth, A.K.; Barrett, A.B.; Barnett, L. Granger Causality Analysis in Neuroscience and Neuroimaging. J. Neurosci. 2015, 35, 3293–3297. [CrossRef][PubMed] 26. Van Essen, D.C.; Smith, S.M.; Barch, D.M.; Behrens, T.E.; Yacoub, E.; Ugurbil, K. WU-Minn HCP Consortium The WU-Minn Human Connectome Project: An overview. NeuroImage 2013, 80, 62–79. [CrossRef][PubMed] 27. Toschi, N.; Duggento, A.; Passamonti, L. Functional Connectivity in Amygdalar-Sensory/(Pre)Motor networks at rest: New evidence from the Human Connectome Project. Eur. J. Neurosci. 2017, 45, 1224–1229. [CrossRef] [PubMed] 28. Deshpande, G.; Sathian, K.; Hu, X. Effect of hemodynamic variability on Granger causality analysis of fMRI. Neuroimage 2010, 52, 884–896. [CrossRef][PubMed] 29. Wu, G.-R.; Liao, W.; Stramaglia, S.; Ding, J.-R.; Chen, H.; Marinazzo, D. A blind deconvolution approach to recover effective connectivity brain networks from resting state fMRI data. Med. Image Anal. 2013, 17, 365–374. [CrossRef][PubMed] 30. Barnett, L.; Seth, A.K. Granger causality for state-space models. Phys. Rev. E 2015, 91, 040101. [CrossRef] 31. Faes, L.; Nollo, G.; Stramaglia, S.; Marinazzo, D. Multiscale Granger causality. Phys. Rev. E 2017, 96, 042150. [CrossRef] 32. Faes, L.; Nollo, G.; Porta, A. Information Domain Approach to the Investigation of Cardio-Vascular, Cardio-Pulmonary, and Vasculo-Pulmonary Causal Couplings. Front. Physiol. 2011, 2, 2. [CrossRef] 33. Rubinov, M.; Sporns, O. Complex network measures of brain connectivity: Uses and interpretations. NeuroImage 2010, 52, 1059–1069. [CrossRef][PubMed] 34. Song, X.; Panych, L.P.; Chou, Y.-H.; Chen, N.-K. A Study of Long-Term fMRI Reproducibility Using Data-Driven Analysis Methods. Int. J. Imaging Syst. Technol. 2014, 24, 339–349. [CrossRef][PubMed] 35. Chen, X.; Lu, B.; Yan, C.-G. Reproducibility of R-fMRI metrics on the impact of different strategies for multiple comparison correction and sample sizes. Hum. Brain Mapp. 2018, 39, 300–318. [CrossRef][PubMed] 36. Chen, Y.; Rangarajan, G.; Feng, J.; Ding, M. Analyzing multiple nonlinear time series with extended Granger causality. Phys. Lett. A 2004, 324, 26–35. [CrossRef] 37. Benhmad, F. Modeling nonlinear Granger causality between the oil price and U.S. dollar: A wavelet based approach. Econ. Model. 2012, 29, 1505–1514. [CrossRef] 38. Montalto, A.; Stramaglia, S.; Faes, L.; Tessitore, G.; Prevete, R.; Marinazzo, D. Neural networks with non-uniform embedding and explicit validation phase to assess Granger causality. Neural Netw. 2015, 71, 159–171. [CrossRef][PubMed] 39. Schreiber, T. Measuring Information Transfer. Phys. Rev. Lett. 2000, 85, 461–464. [CrossRef][PubMed] 40. Wibral, M.; Vicente, R.; Lizier, J.T. (Eds.) Directed Information Measures in Neuroscience; Understanding Complex Systems; Springer: Berlin/Heidelberg, Germany, 2014. 41. Wen, X.; Rangarajan, G.; Ding, M. Is Granger Causality a Viable Technique for Analyzing fMRI Data? PLoS ONE 2013, 8, e67428. [CrossRef][PubMed] 42. Ramsey, J.; Hanson, S.; Hanson, C.; Halchenko, Y.; Poldrack, R.; Glymour, C.; Halchenko, Y. Six problems for causal inference from fMRI. NeuroImage 2010, 49, 1545–1558. [CrossRef] 43. Laumann, T.O.; Gordon, E.M.; Adeyemo, B.; Snyder, A.Z.; Joo, S.J.; Chen, M.-Y.; Gilmore, A.W.; McDermott, K.B.; Nelson, S.M.; Dosenbach, N.U.; et al. Functional system and areal organization of a highly sampled individual human brain. Neuron 2015, 87, 657–670. [CrossRef] 44. Mueller, S.; Wang, D.; Fox, M.D.; Yeo, B.T.; Sepulcre, J.; Sabuncu, M.R.; Shafee, R.; Lu, J.; Liu, H. Individual Variability in Functional Connectivity Architecture of the Human Brain. Neuron 2013, 77, 586–595. [CrossRef] [PubMed] Entropy 2019, 21, 661 16 of 16

45. Chen, B.; Xu, T.; Zhou, C.; Wang, L.; Yang, N.; Wang, Z.; Dong, H.-M.; Yang, Z.; Zang, Y.-F.; Zuo, X.-N.; et al. Individual Variability and Test-Retest Reliability Revealed by Ten Repeated Resting-State Brain Scans over One Month. PLoS ONE 2015, 10, e0144963. [CrossRef][PubMed] 46. Kong, R.; Li, J.; Sun, N.; Sabuncu, M.R.; Schaefer, A.; Zuo, X.-N.; Holmes, A.J.; Eickhoff, S.; Yeo, B.T.T. Controlling for Intra-Subject and Inter-Subject Variability in Individual-Specific Cortical Network Parcellations. bioRxiv 2017, 213041. [CrossRef] 47. Hodkinson, D.J.; O’Daly, O.; A Zunszain, P.; Pariante, C.M.; Lazurenko, V.; O Zelaya, F.; A Howard, M.; Williams, S.C.R. Circadian and homeostatic modulation of functional connectivity and regional cerebral blood flow in humans under normal entrained conditions. Br. J. Pharmacol. 2014, 34, 1493–1499. [CrossRef] [PubMed] 48. Shannon, B.J.; Dosenbach, R.A.; Su, Y.; Vlassenko, A.G.; Larson-Prior, L.J.; Nolan, T.S.; Snyder, A.Z.; Raichle, M.E. Morning-evening variation in human brain metabolism and memory circuits. J. Neurophysiol. 2013, 109, 1444–1456. [CrossRef][PubMed] 49. Gordon, E.M.; Breeden, A.L.; Bean, S.E.; Vaidya, C.J. Working memory-related changes in functional connectivity persist beyond task disengagement. Hum. Brain Mapp. 2014, 35, 1004–1017. [CrossRef] [PubMed] 50. Lewis, C.M.; Baldassarre, A.; Committeri, G.; Romani, G.L.; Corbetta, M. Learning sculpts the spontaneous activity of the resting human brain. Proc. Natl. Acad. Sci. USA 2009, 106, 17558–17563. [CrossRef] 51. Tambini, A.; Ketz, N.; Davachi, L. Enhanced Brain Correlations during Rest Are Related to Memory for Recent Experiences. Neuron 2010, 65, 280–290. [CrossRef] 52. Yeo, B.T.; Tandi, J.; Chee, M.W. Functional connectivity during rested wakefulness predicts vulnerability to sleep deprivation. NeuroImage 2015, 111, 147–158. [CrossRef] 53. Laumann, T.O.; Snyder, A.Z.; Mitra, A.; Gordon, E.M.; Gratton, C.; Adeyemo, B.; Gilmore, A.W.; Nelson, S.M.; Berg, J.J.; Greene, D.J.; et al. On the Stability of BOLD fMRI Correlations. Cereb. Cortex 2017, 27, 4719–4732. [CrossRef] 54. Wang, C.; Ong, J.L.; Patanaik, A.; Zhou, J.; Chee, M.W.L. Spontaneous eyelid closures link vigilance fluctuation with fMRI dynamic connectivity states. Proc. Natl. Acad. Sci. USA 2016, 113, 9653–9658. [CrossRef][PubMed] 55. Shine, J.M.; Koyejo, O.; Poldrack, R.A. Temporal metastates are associated with differential patterns of time-resolved connectivity, network topology, and attention. Proc. Natl. Acad. Sci. USA 2016, 113, 9888–9891. [CrossRef][PubMed] 56. Vecchio, F.; Miraglia, F.; Marra, C.; Quaranta, D.; Vita, M.G.; Bramanti, P.; Rossini, P.M. Human brain networks in cognitive decline: A graph theoretical analysis of cortical connectivity from EEG data. J. Alzheimer’s Dis. 2014, 41, 113–127. [CrossRef][PubMed] 57. Anastasiadou, M.N.; Christodoulakis, M.; Papathanasiou, E.S.; Papacostas, S.S.; Hadjipapas, A.; Mitsis, G.D. Graph Theoretical Characteristics of EEG-Based Functional Brain Networks in Patients with Epilepsy: The Effect of Reference Choice and Volume Conduction. Front. Mol. Neurosci. 2019, 13, 221. [CrossRef] [PubMed] 58. Wang, Y.; Wang, Y.; Lui, Y.W. Generalized Recurrent Neural Network accommodating Dynamic Causal Modeling for functional MRI analysis. NeuroImage 2018, 178, 385–402. [CrossRef][PubMed] 59. Gilson, M.; Zamora-Lopez, G.; Pallares, V.;Adhikari, M.H.; Senden, M.; Campo, A.T.; Mantini, D.; Corbetta, M.; Deco, G.; Insabato, A. MOU-EC: Model-based whole-brain effective connectivity to extract biomarkers for brain dynamics from fMRI data and study distributed cognition. bioRxiv 2019, 531830. [CrossRef] 60. Preti, M.G.; Bolton, T.A.; Van De Ville, D. The dynamic functional connectome: State-of-the-art and perspectives. NeuroImage 2017, 160, 41–54. [CrossRef]

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).