PHYSICAL REVIEW RESEARCH 3, 013145 (2021)
Assessing the significance of directed and multivariate measures of linear dependence between time series
Oliver M. Cliff ,1,2,3,* Leonardo Novelli ,1,3 Ben D. Fulcher ,1,2 James M. Shine ,1,4 and Joseph T. Lizier 1,3 1Centre for Complex Systems, The University of Sydney, Sydney NSW 2006, Australia 2School of Physics, The University of Sydney, Sydney NSW 2006, Australia 3Faculty of Engineering, The University of Sydney, Sydney NSW 2006, Australia 4Brain and Mind Centre, School of Medical Sciences, The University of Sydney, Sydney NSW 2006, Australia
(Received 20 March 2020; accepted 21 January 2021; published 12 February 2021)
Inferring linear dependence between time series is central to our understanding of natural and artificial systems. Unfortunately, the hypothesis tests that are used to determine statistically significant directed or multivariate relationships from time-series data often yield spurious associations (Type I errors) or omit causal relationships (Type II errors). This is due to the autocorrelation present in the analyzed time series—a property that is ubiquitous across diverse applications, from brain dynamics to climate change. Here we show that, for limited data, this issue cannot be mediated by fitting a time-series model alone (e.g., in Granger causality or prewhitening approaches), and instead that the degrees of freedom in statistical tests should be altered to account for the effective sample size induced by cross-correlations in the observations. This insight enabled us to derive modified hypothesis tests for any multivariate correlation-based measures of linear dependence between covariance-stationary time series, including Granger causality and mutual information with Gaussian marginals. We use both numerical simulations (generated by autoregressive models and digital filtering) as well as recorded fMRI-neuroimaging data to show that our tests are unbiased for a variety of stationary time series. Our experiments demonstrate that the commonly used F-andχ 2-tests can induce significant false-positive rates of up to 100% for both measures, with and without prewhitening of the signals. These findings suggest that many dependencies reported in the scientific literature may have been, and may continue to be, spuriously reported or missed if modified hypothesis tests are not used when analyzing time series.
DOI: 10.1103/PhysRevResearch.3.013145
I. INTRODUCTION autocorrelation, the accuracy of many reported results across the empirical sciences may be called into question. Linear dependence measures such as Pearson correlation, The notion that autocorrelation affects the sampling distri- canonical correlation analysis, and Granger causality are used bution of time-series properties has a long history in statistics, in a broad range of scientific domains to investigate the with research often focusing on the relationship between two complex relationships in both natural and artificial processes. univariate processes. Seminal work by Bartlett [10,11]re- Despite their widespread use, concerns have been raised about vealed that autocorrelation can distort the degrees of freedom the hypothesis tests typically used to assess the statistical available to compute statistics such as Pearson correlation significance of such measures from time series [1–6]. Specif- coefficients. In practical terms, this induces an “effective sam- ically, the presence of autocorrelation in a signal—one of two ple size,” where the effective number of independent samples defining properties of a stationary time series [7,8]—has been used in computing an estimate is different to the actual length known to bias statistics since the beginning of time-series of the dataset. Two opposing strategies have been proposed analysis [9]. If left unaccounted, this bias yields a greater for handling autocorrelation: remove the autoregressive (AR) number of both spurious correlations and missed causalities components of the time series before computing statistics, or (Type I and Type II errors) due to size and power distor- modify the hypothesis tests that assess the distorted measure- tions of the hypothesis tests. With the recent findings [1–6] ments. The former approach, known as prewhitening, involves suggesting that existing techniques do not adequately address filtering the time series in order to render the residuals seri- ally independent [12]. Prewhitening is known to have many issues, such as reducing the size and power properties of hypothesis tests both in theory [13] and in practice, with *[email protected] simulated [14] and recorded [15,16] time-series data in a variety of domains. In contrast, the notion of modifying hy- Published by the American Physical Society under the terms of the pothesis tests remains relatively underused in practice, more Creative Commons Attribution 4.0 International license. Further often found in applications involving short time series and distribution of this work must maintain attribution to the author(s) high autocorrelation, where the statistical bias of measures and the published article’s title, journal citation, and DOI. is most pronouced (with or without prewhitening), e.g., in
2643-1564/2021/3(1)/013145(24) 013145-1 Published by the American Physical Society OLIVER M. CLIFF et al. PHYSICAL REVIEW RESEARCH 3, 013145 (2021) fMRI-based neuroimaging [1,2,17], as well as environmen- is now understood as a special case of conditional mutual tal and ecological studies [18,19]. Indeed, it was not until information, known as transfer entropy [35–37]. While this recently that the efficacy of a modified z-test for correlation unification provides an elegant perspective, there remains a analysis was demonstrated successfully on fMRI signals [2], clear divide between the theoretical foundations of Bartlett which have been widely characterized using correlation coef- (and others [38–40]) and the large family of multivariate linear ficients [20]. Nevertheless, the theory of autocorrelation on dependence measures that information theory provides. the undirected relationship between bivariate time series is In this work we bridge this gap by leveraging the con- now well developed. However, the extension of this theory cept of the effective sample size to derive hypothesis tests to multiple time series, and to directed relationships, remains for any correlation-based measure of linear dependence be- incomplete. tween covariance-stationary time series. This comprises a Motivated to study directed dependencies in economics, large family of well-known statistics based on ratios of gen- Granger [21] introduced a measure of causal influence be- eralized variance—such as Granger causality and mutual tween AR models nearly 60 years ago. Since then, it information—that we introduce in Sec. II. To achieve this, we has become exceedingly popular, exemplified by more than first provide the one-tailed and two-tailed tests for the sample 100 000 works indexed by Google Scholar that contain the partial correlations between two univariate processes under phrase “Granger causality” (as of June 2020). This impact autocorrelation in Sec. III. Although this result is important is reflected in the measure’s ubiquity in the scientific com- in its own right, in this work we primarily leverage it to munity beyond its origins in econometrics, generating highly construct the tests for more advanced inference procedures influential results on phenomena ranging from brain dynamics with multivariate and directed models of observed dynamics. [22–24] to climate change [25,26] and political relationships Following this, we introduce the modified -test (in Sec. IV), [27,28]. Granger causality controls for the confounding past which we show is suitable for assessing the significance of of a process through linear regression, building statistics and any linear dependence measure that can be expressed as a hypothesis tests via residuals rather than the original pro- ratio of generalized variances. Specifically, in Sec. V we use cess. However, researchers are becoming aware that certain the two-tailed test to derive hypothesis tests for conditional preprocessing techniques that increase autocorrelation, such mutual information estimates between bivariate time-series as filtering, raise the false-positive rate (FPR) of Granger data. We then use the chain rule for mutual information to ex- causality tests when using the well-established χ 2- and F- tend this result to multivariate time-series data. Finally, since distributions [3–5]. Even though these empirical studies have Granger causality can be expressed as a conditional mutual demonstrated that established Granger causality tests have information, in Sec. VI we extend our results further to derive distorted size and power properties (exhibiting Type I and II Granger causality tests for both bivariate and multivariate errors), it has remained unclear as to why and how to correct time-series datasets. More broadly, the modified -test can them. In this paper, we illustrate that these errors are due to be used for any measure that can be expressed in terms of an inflated variance of the null distribution as a function of conditional mutual information (or, equivalently, Wilks’ cri- autocorrelation remaining in the residuals, in the same way terion or partial correlation), e.g., canonical correlations and that bivariate correlation is affected. partial autocorrelation [7,8] or information-theoretic measures In order to unify Bartlett’s earlier investigations on corre- (for linear-Gaussian processes) such as predictive information lation coefficients (under autocorrelation) with more complex [41,42] and active information storage [43]. measures such as Granger causality, we must expand the for- Using numerical simulations throughout Sec. VII,we mer body of work to account for multivariate relationships. validate the modified -test and characterize the effect of One such multivariate generalization of Pearson correlation autocorrelation on both the χ 2- and F-test. Our experiments is referred to as Wilks’ criterion [29], which quantifies the involve generating samples from two first-order independent relationship between multiple sets of variables, and is - AR models and iteratively filtering the output signal such distributed for independent observations [30]. In particular, that the autocorrelation is increased for both time series; this we are interested in a special case of Wilks’ criterion popu- simulates empirical analysis in practice, and allows for the larized by Hotelling [31] that focuses on two sets of variables, process parameters to be modified while ensuring that the referred to as canonical correlation analysis. It was later es- null hypothesis (of no interprocess dependence) is not vio- tablished that, like Pearson correlation, estimates of canonical lated. We perform these experiments for mutual information correlations are inefficient under autocorrelation [6], introduc- and Granger causality in their unconditional, conditional, and ing Type I and II errors under hypothesis tests that assume multivariate forms. Our results generally agree with the hy- independence (such as the -distribution). Instead of deriv- potheses that the FPR of F- and χ 2-tests can be inflated by ing hypothesis tests directly for Wilks’ criterion or canonical either increasing the autocorrelation (through filtering) or, for correlations, here we use the equivalent information-theoretic the χ 2-test, the number of conditionals (through increasing formulation. the dimension of mutual information or the history length of Information theory’s general applicability arises in sim- Granger causality). These experiments mirror empirical appli- ply requiring a probability distribution that can be either cations where digital filtering is often used in preprocessing parametric or nonparametric [32,33]. When this probability for many purposes, such as handling nonstationary effects, distribution is modeled as a multivariate Gaussian, canonical which inadvertently increases autocorrelation and therefore correlation analysis and information theory overlap because the FPRs of unmodified tests. Given minimally sufficient mutual information can be decomposed into sums involv- effective samples, however, we confirm that the modified - ing canonical variables [34]. Moreover, Granger causality tests remain unbiased for all scenarios. We thus show that, in
013145-2 ASSESSING THE SIGNIFICANCE OF DIRECTED AND … PHYSICAL REVIEW RESEARCH 3, 013145 (2021) contrast, the size (Type I errors) and power (Type II errors) of tice, however, we have access only to a finite-length dataset F-orχ 2-tests are arbitrarily low for a large class of multivari- with T observations over which to compute the measures, ate linear dependence measures (approximately zero in certain introducing a variation in statistical estimates and manifest- instances) and overwhelmingly depends on the parameters of ing as nonzero values in the case of no relationship. Here the underlying independent processes. We further demonstrate we present this dataset as an m × T matrix z of consecutive that the common approach of prewhitening a signal (in order real-valued samples z(t ) ∈ Rm of the process Z (again, this is to remove the effect of autocorrelation) does not suffice to partitioned into submatrices x and y). To this end, the aim of control the FPR in almost all cases. Finally, by using a well- linear-dependence tests is to infer whether there is a statistical known brain-imaging dataset from the Human Connectome dependence between X and Y based on the sample paths x and Project [44], we verify that our previous numerical simula- y alone. tions yield comparable results to experiments on commonly We make the typical assumption that the underlying sys- used datasets. For these experiments, the χ 2-tests of mutual tem, Z, is a second-order stationary, purely nondeterministic information and Granger causality yield concerningly high process [7,8,47]. An important consequence of covariance- FPRs of over 80% and 65% for a nominal significance of stationarity is that the time series may be represented, after 5%—a 16- and a 13-fold increase—whereas our exact tests appropriate mean removal and differencing [48], by the maintain the ideal FPR for all experiments. Open-source ARMA model: MATLAB code is made available to allow users to perform p q correct hypothesis testing for all dependence measures, as Z(t ) = a(t ) + (u)Z(t − u) + (u)a(t − u), (4) well as the above experiments, at Ref. [45]. u=1 u=1 Our theoretical and empirical findings suggest that this work presents the first statistically sound approach for test- where and are vectors of autoregressive (AR) and a t ing the linear dependence between multivariate time-series moving-average (MA) parameters, and ( ) is uncorrelated data. Given that approaches such as prewhitening and Granger noise (the innovation process). We further assume that the a t ∼ N 0, causality are specifically designed to account for autocorre- noise is Gaussian, ( ) ( ) for some arbitrary noise Z lation, we conjecture that autocorrelation-induced statistical covariance , meaning that is a linear-Gaussian process. errors caused by F-orχ 2-tests (and others) may be even more prevalent in prior publications than previously suggested A. Cross-correlation and autocorrelation by several authors [1,3,4]. In particular, our case study of For covariance-stationary time series, the relationship be- brain-imaging data is concerning, because the neuroscience tween Zi(t ) and Z j (t + u) depends only on the difference in community employs techniques such as correlation, mutual times t and t + u of the observation but not on t itself. Once information, and Granger causality in order to infer pairwise the mean has been removed, such processes are fully defined dependence (known as “functional connectivity”). Implemen- by their cross-correlation, tation of our approach will enable correct inference of linear γ relationships within complex systems across myriad scientific ij(u) ρij(u) = , (5) applications. γii(0)γ jj(0)
with γij(u) = cov(Zi(t ), Z j (t + u)) the cross-covariance be- II. MEASURES OF LINEAR DEPENDENCE tween Zi(t ) and Z j (t + u). If ρii(u) = 0 for any u > 0, then In this work, we focus on multivariate signals, the univariate process Zi exhibits autocorrelation, and the col- lection of ρii(u)foru = 0, ±1, ±2,...is generally called the { ,..., }, = , ± , ± ,..., Z1(t ) Zm(t ) t 0 1 2 (1) autocorrelation function of Zi. The sample cross-correlation that is, a collection of m series sampled at equally spaced time coefficients are computed from time-series data z as intervals. Writing cij(u) rij(u) = , (6) c (0)c (0) Z(t ) = (Z1(t ),...,Zm(t )) , (2) ii jj = −1 T + = − we shall refer to the m series as an m-dimensional vector of with cij(u) N t=1 zi(t )z j (t u) where N T 1as multiple time series such that Z(t ) ∈ Rm. an unbiased estimate of the sample cross-covariance. For the purposes of inferring linear dependence, Z is par- The first linear dependence measure we discuss is Pear- titioned into one k-variate and one l-variate subprocess [46]: son’s product-moment correlation coefficient. For bivariate (m = 2) processes, we shall write X = X and Y = Y and X denote the cross-correlation between these variables [Eq. (5)] Z = , (3) ρ Y as XY (u). Pearson’s correlation coefficient is the lag-zero cross-correlation ρXY = ρXY (0), and quantifies the (symmet- reflecting an interest in the relationship between X and Y.The ric) association between paired observations of X and Y .The linear dependence of X on Y (or vice versa) is measured by sample correlation coefficient is then given by a scalar value that quantifies how much the outcomes of Y cxy reduce uncertainty over outcomes of X. Theoretically, in the rxy = , (7) c c absence of a linear relationship between X and Y (the null xx yy hypothesis, H0) the reduction of uncertainty is exactly zero, where cxy = cxy(0) is the sample covariance, and cxx = cxx(0) meaning that Y does not linearly predict X at all. In prac- and cyy = cyy(0) are sample variances [39].
013145-3 OLIVER M. CLIFF et al. PHYSICAL REVIEW RESEARCH 3, 013145 (2021)
In order to assess the statistical significance of linear de- another c-variate process W [50,51]. Partial correlation is pendence measures, such as the sample correlation coefficient, estimated by, first, computing the residual processes: σ 2 , = we must be able to compute their variance, i.e., r (x y) e |w = x − xˆ(w), (11) var(rxy). For independent but autocorrelated stationary pro- x cesses, the variance of the sample correlation coefficient can = − w , be estimated (to the first order) as [2,38,39] ey|w y yˆ( ) (12) − wherex ˆ(w) denotes the linear prediction of x from w via T 1 T − u σ 2 , ≈ −1 + , ordinary least squares. Then an appropriate test statistic for ˆr (x y) T 1 2 rxx(u)ryy(u) (8) T H ρ · = u=1 the null hypothesis 0: XY W 0 of no relation between X and Y , above any relationship with W , is the sample partial where rxx(u) and ryy(u)arethelagu sample autocorrelations correlation: [49]. Although we refer to Eq. (8) as Bartlett’s formula, this t ex|w (t ) ey|w (t ) first-order approximation is due to Clifford et al. [39], who rxy·w = . (13) 2 2 presented the variance estimator for spatial autocorrelation t ex|w (t ) t ey|w (t ) and an estimate of the effective number of independent sam- ples for correlation coefficients: By contrasting the formulas for sample partial correlation [Eq. (13)] with sample cross-correlation [Eq. (6)], it is evi- η , = + σ −2 , . ˆ (x y) 1 ˆr (x y) (9) dent that the former is equivalent to the bivariate correlation = between the residuals, i.e., rxy·w rex|w ey|w . An important consequence of Eqs. (8) and (9) is that hy- Unlike Pearson correlation, there is a dearth of research pothesis tests, such as Student’s t-test, should have degrees into the null distribution of partial correlation coefficients for of freedom corresponding to the effective sample size of the autocorrelated time-series data. As such, our first theoretical analyzed time series (i.e., the effective degree of freedom contribution (in Sec. III) is a derivation for the null distribution [2,11]), rather than the original sample size [39]. That is, if of sample partial correlations (13) under autocorrelation, i.e., both X and Y are autocorrelated, then the null distribution for extending the modified t-test [for bivariate correlation (10)] to rxy follows a modified Student’s t-test: facilitate residual processes. ηˆ (x, y) − 2 r ∼ t[ˆη(x, y) − 2], (10) C. Wilks’ criterion and canonical correlations xy 1 − r2 xy Relating two or more sets of variables is achieved similarly whereη ˆ (x, y) − 2 is the (estimated) effective degrees of free- to partial correlation (13), with the exception that the gener- dom. An examination of this formula reveals that, when both x alized variance is used, rather than the conditional variance and y are positively autocorrelated, then there are, effectively, [29,31]. Consider the relationship between the k-variate pro- fewer independent observations than in the original dataset cess X and the l-variate process Y, in the context of a c-variate (ˆη
013145-4 ASSESSING THE SIGNIFICANCE OF DIRECTED AND … PHYSICAL REVIEW RESEARCH 3, 013145 (2021)
[6]. Consequently, neither approach is suitable for inferring By deriving the exact hypothesis tests for mutual in- linear dependence between the majority of time-series data formation in Sec. V, we provide a critical component of due to the ubiquity of autocorrelation. In Sec. IV we address general-purpose techniques for measuring undirected rela- this issue by providing the null distribution to be used in tionships between sets of variables. However, mutual infor- the presence of autocorrelation [and of course when cross- mation was not originally intended to measure autocorrelated correlations are present amongst any two variables, i.e., Xi(t ) time-series dependencies, nor does it naturally model di- and Xj (t − u) may covary for any i, j, t,oru]. An application rected dependencies—this is the intended purpose of Granger that is of particular interest is mutual information, which is causality. equivalent to Wilks’ criterion (14) for Gaussian marginals. E. Granger causality D. Mutual information Granger causality was explicitly designed to capture one- Mutual information IX;Y|W is a fundamental concept in way dependence between stochastic processes by taking into information theory—and a building block of many other account the confounding influence of their past (i.e., the auto- measures—that quantifies the amount of information about correlation). By considering X as a target (predictee) process a process X obtained by observing another process Y (po- and Y as a source (predictor) process, Granger causality tentially in the context of a third process W , making it a FY→X|W explicitly aims to measure the causality (predictabil- conditional mutual information) [33]. In general, information ity) in Y about X in context of the relevant history of X theory facilitates multivariate analysis by simply requiring (and, potentially, a concomitant process W ). Of course, the well-defined probability distributions that can be either para- use of the term “causality” here refers to Wiener’s definition metric or nonparametric. (as a model of dependence based on prediction) rather than When these are normally distributed, mutual information Pearl’s (a mechanistic causal effect that can only be inferred takes a form that is equivalent to Wilks’ criterion [33,53]: using interventions); see Ref. [57] for a differentiation of these concepts. 1 |sxy|w| Iˆ |w =− . The main assumption underlying Granger causality is that x;y log | || | (15) 2 sx|w sy|w both X and Y are (vector) AR processes [21,46]. That is, This formula is asymptotically equivalent to the nested log- we assume that X (t ) and Y (t ) are causally dependent on the likelihood ratio (LR) of two models [54], and thus we can following states: ⎡ ⎤ ⎡ ⎤ use a null distribution also provided by Wilks [55]. Following X (t − 1) Y (t − 1) Wilks’ theorem [55], under the null hypothesis H : I | = . . 0 X;Y W (p) = ⎣ . ⎦, (q) = ⎣ . ⎦. 0 and with normally distributed marginals, mutual infor- X (t ) Y (t ) (19) − − mation estimates are asymptotically chi-square distributed X (t p) Y (t q) [35,53,54], Under this assumption, the (directed) influence from Y to X is 2 quantified by conditional mutual information [35]: 2T Iˆx;y|w ∼ χ (kl). (16) F , = I . For limited data, a more precise null distribution can be de- Y→X|W (p q) 2 X;Y (q)|X (p)W (20) rived from the standard F-test, albeit for the more specific Following Eq. (15), this measure can be estimated as a log- case of no autocorrelation and with one of the processes ratio of generalized variances [46]: being univariate [see Eq. (A4) and surrounding discussion in Appendix A2]. That is, the mutual information between |sx|x(p)w| Fˆy→x|w (p, q) = log . (21) an i.i.d. variable X and an l-variate Y, in the context of the |sx|x(p)y(q)w| concomitant W , is, under the null hypothesis H0 : IX;Y|W = 0, Note that, except for the rather narrow case of k = q = 1, T − (l + c + 1) [exp (2Iˆx;y|w ) − 1] ∼ F (l, T − (l + c + 1)). Granger causality is a multivariate measure (using generalized l variances rather than conditional variances). (17) The AR orders, p and q, of each process are typically The same arguments in Eqs. (15)–(17) hold for unconditional determined by statistical tests such as partial autocorrelation mutual information Iˆ by setting w =∅(and c = 0). x;y [8], Burg’s method [58], the Akaike or Bayesian information Although we found no discussion on autocorrelation- criterion (AIC or BIC), cross-validation [4], or active informa- induced biases of mutual information in literature, statistical tion storage [43,59,60]. In this paper, we use Burg’s method tests will clearly be incorrect if autocorrelation is not taken to infer the model order due to its efficiency and stability over into account. This is evident from the well-known result that the Yule-Walker equations [58]. Further, using results from mutual information reduces to a function of sample correla- this main text, we discuss the relationship between partial tion coefficients when the dependent processes are univariate autocorrelation and active information storage in Appendix E. [56]: In general, hypothesis tests for Granger causality can be Iˆ =− − 2 . 2 x;y|w log 1 rxy·w (18) derived from Wilks’ theorem [55]. That is, under the null hypothesis H :F → | (p, q) = 0, estimates of the Granger Moreover, it is clear by observing the equivalence between 0 Y X W causality from Y to X [Eq. (21)] are asymptotically chi-square Eqs. (14) and (15), noting that mutual information estimates distributed [46]: can be decomposed into sums involving canonical variables 2 [53]. T Fˆy→x|w (p, q) ∼ χ (klq). (22)
013145-5 OLIVER M. CLIFF et al. PHYSICAL REVIEW RESEARCH 3, 013145 (2021)
Alternatively, a finite-sample null distribution can be used B. Modified F-test for partial correlation = if the predictee process is univariate (k 1). Referring to The Student’s t-distribution in Eq. (25) allows for one- + + Appendix A, the restricted model has p c 1 parameters, tailed (upper or lower) tests for the partial correlation by using + + + and the unrestricted model has p lq c 1 parameters. the statistic (the LHS) as an input to the quantile function Accordingly, we can build the statistic: of the t-distribution. For two-tailed tests, another common T − (p + lq + c + 1) approach is to square the statistic and, subsequently, the null {exp [Fˆ → |w (p, q)] − 1}, (23) ∼ ν lq y x distribution. The square of a random variable Z t( ) that follows Student’s t-distribution (with parameter ν) follows an which, according to the standard F-test [Eq. (A4)], is dis- F-distribution with parameters 1 and ν, i.e., Z2 ∼ F (1,ν). tributed as Thus, under the null hypothesis H0:ρXY·W = 0, the square of the statistic in Eq. (25) (the LHS) follows an F-distribution: F (lq, T − (p + lq + c + 1)). (24) 2 rxy·w It should be emphasized, however, that the F-test is suitable n |w ∼ F (1, n |w ), (26) xy − 2 xy only for serially independent observations. Thus, although the 1 rxy·w Granger causality measure explicitly accounts for autocorre- with an effective degree of freedom, lation in vector AR processes, the established hypothesis tests = η , − − , assume either a sequence of completely independent residuals nxy|w ˆ (ex|w ey|w ) c 2 (27) (the F-test) or infinite data (the χ 2-test). The null distributions obtained from Eq. (9). We refer to the significance test that that we provide in Sec. VI overcome both of these issues, pro- uses this distribution as the modified F-test. Note that a form viding the first valid finite-sample tests for Granger causality. of Eq. (26) without modifying the degree of freedom is com- monly used for testing the coefficient of determination; thus III. MODIFIED TESTS FOR PARTIAL CORRELATION this approach could also be used for constructing a finite- sample test of the coefficient of multiple correlation under In this section, we derive one-tailed and two-tailed tests for autocorrelation. the null hypothesis, H0 : ρXY·W = 0, of no partial correlation between two univariate autocorrelated time series x and y, IV. MODIFIED -TESTS given a third (potentially multivariate) process w. These tests are valid for any covariance-stationary time series X, Y , and Although the modified t- and F-tests introduced above W and sample size T . are suitable for bivariate correlation-based measures, they are not appropriate for multivariate (and thus directed) null tests. ∗ A. Modified Student’s t-test for partial correlation Here we introduce the -distribution, which can be used for hypothesis testing all linear dependence measures throughout Recall from Eq. (13) that the sample partial correlation is this paper. equivalent to the sample correlation between ex|w and ey|w, Recall that, for independent X and Y and W , Wilks’ i.e., r ·w = r . Obtaining the null distribution for sample xy ex|w ey|w criterion [Eq. (14)] is -distributed. The exact form of the partial correlations between autocorrelated time series can -distribution has been extensively studied, with known re- thus be treated similarly to that of the correlation coefficients lationships to the F- and beta-distributions [30,52]. The main [see Eq. (10)]. purpose of this paper is to derive the finite sample distribution A well-known result is that the sample partial correlation ⊥⊥ ν of such statistics under autocorrelation, i.e., where X (t ) between independent observations is t-distributed t( ), under X (t − u) and Y (t )⊥⊥ Y (t − v)forsomeu, v > 0. the null hypothesis H0:ρXY·W = 0, with degrees of freedom ν = − − = w As we will show throughout this work, the distribution T c 2 and c dim( (t )) [61]. As such, the modi- for Wilks’ criterion with two independent but serially corre- fied statistic and null distribution is lated processes can be described by products of -distributed variables with different effective degrees of freedom. We ηˆ (ex|w, ey|w ) − c − 2 r ·w ∼ t[ˆη(e |w, e |w ) − c − 2]. ∗ = xy − 2 x y denote this distribution as (n), with the parameter n 1 rxy·w (n1,...,nb) comprising the degrees of freedom of each in- (25) dependent -distribution. That is, Wilks’ criterion is, under η , Here the (estimated) effective sample size ˆ (ex|w ey|w ) is still the null hypothesis, ∗-distributed: computed from Eq. (9) but with the autocorrelation functions |sxy|w| ∗ of the residual vectors ex|w and ey|w, rather than the original ∼ n , | || | ( ) (28) sample paths x and y. Intuitively, this is because rxy·w is sx|w sy|w itself a sample correlation of these residuals, so it is their where the ∗(n) distribution itself can be described by a autocorrelation—not that of the original time series—that di- product of independent -distributed variables: rectly determines the effective sample size. Another crucial addition is that the dimension of the conditional process c = b , ∼ , , . dim(w(t )) further reduces the number of degrees of freedom Li with Li (ni 1 1) (29) [61], for the same reason as in standard F-tests [4]. When i=1 the residual vectors, ex|w and ey|w, are (serially) independent, Notice that this reduces to the -distribution for two sets of then Eq. (25) becomes equivalent to the standard Student’s independent variables [30]; however, with the ∗-distribution t-distribution for partial correlation. we are able to include the effective sample sizes.
013145-6 ASSESSING THE SIGNIFICANCE OF DIRECTED AND … PHYSICAL REVIEW RESEARCH 3, 013145 (2021)
∗ Although the null distribution for the product of two in- the null hypothesis, H0 : IX;Y |W = 0, are -distributed: dependent -distributed variates is known [30], deriving the − Iˆ = − 2 exact distribution for the product of an arbitrary number of exp 2 x;y|w 1 rxy·w ∗ -distributed variates is nontrivial. Fortunately, a relationship ∼ (nxy|w ). (33) between the beta-, F-, and -distributions [30,52] allows for simple numerical methods. To generate the distribution Of course, due to the relationship between the F- and - ∗(n), we could sample beta-distributed variables: distribution [noted in Eq. (32)], we can construct an equivalent modified F-test for conditional mutual information: b b ni 1 n |w [exp (2 Iˆ |w ) − 1] ∼ F (1, n |w ). (34) L = V , with V ∼ B , , (30) xy x;y xy i i i 2 2 i=1 i=1 The null distributions we provide above explicitly account α,β for autocorrelation via the effective degrees of freedom nxy|w where B( ) is the beta distribution. Equivalently, one could and also reduce to the F-distribution for information-theoretic sample independent F-distributed variables: quantities when observations of the analyzed time series are η , = b b independent [cf. Eq. (17) by letting ˆ (x y) T ]. Further, ni → L = , with U ∼ F (1, n ). (31) when x and y are serially uncorrelated, and in the limit T i U + n i i ∞ χ 2 i=1 i=1 i i ,Eq.(34) becomes equivalent to the null distribution for mutual information (see the discussion in Appendix A2). In our experiments (and open-source code), we opt to sample Thus, the null distribution we present in Eq. (34) is a gen- ∗ independent beta-distributed variables and construct the - eralization of both the standard F-test (which is applicable distribution from their product as per Eq. (30). Throughout only for i.i.d. variables) as well as the asymptotic distribution ∗ this work, we refer to hypothesis tests that use the - (which is applicable only for infinite data). distribution and modify the degree of freedom to account for Although they are special cases of the modified -test, autocorrelation as “modified -tests.” there are important distinctions here from the χ 2-tests (16) From the relationship between the beta-, F-, and - and the standard F-tests (17) for conditional mutual informa- distributions [see Eqs. (29)–(31)], it is clear that the modified tion. The first is that we now have an effective sample size -test is a generalization of the modified F-test, becoming ηˆ that changes depending on the autocorrelation function of equivalent for univariate statistics. For instance, returning to the residuals ex|w and ey|w. The second is that the degrees partial correlation, we have that of freedom nxy|w is further reduced by c = dim (w(t )), the w | | dimension of the conditional time series , which appears in sxy|w 2 ∗ 2 = − r ∼ n |w , the finite-sample F-tests but not the asymptotic χ -tests. Both | || | 1 xy·w ( xy ) (32) sx|w sy|w of these differences introduce a significant bias in the estima- tion of linear dependence for many real-world applications, where n |w is the effective degree of freedom. Thus, either xy exemplified by the numerical simulations in Sec. VII A. the modified F-test or the modified -test could be used for univariate statistics (i.e., ratios of conditional variances). We can now derive explicit hypothesis tests for common B. Multiple time series directed and multivariate linear dependence measures using a Mutual information IX;Y|W can also be used to measure the similar approach. Note that, although the purpose of this work dependence between multivariate processes X and Y.Here is explicitly for linear dependence measures between time- we apply the chain rule and the results from the previous series data, the modified -test can be easily extended to more section to obtain a partial correlation decomposition that can general likelihood tests for ratios of generalized variances un- be used for constructing a null distribution in the presence of der spatial autocorrelation [39], which is also known to affect autocorrelation. the sampling properties of statistics. We begin by deriving The chain rule provides a decomposition of mutual infor- tests for the (conditional) mutual information between both mation as a sum of conditional mutual information terms: univariate and multivariate linear-Gaussian processes. k l Iˆ = Iˆ {gh} . x;y|w xy|w (35) V. MODIFIED TESTS FOR MUTUAL INFORMATION g=1 h=1
In this section, we obtain hypothesis tests for the mutual That is, mutual information estimates Iˆx;y|w between a k- information between multiple time series. We first present the variate process x and an l-variate process y, in the context of hypothesis tests explicitly for conditional mutual information the c-variate concomitant w, can be computed by summing for bivariate time series and, by using the chain rule, obtain over conditional mutual information terms [33]. Each condi- the null distribution for the mutual information between mul- tional mutual information term may be expressed as tivariate time series. ˆ {gh} ˆ I |w = I |v{gh} , (36) xy xg;yh xy|w where the conditional for the (g, h)-term is given by A. Two time series ⎡ ⎤ The conditional mutual information for linear-Gaussian x1:g−1 v{gh} = ⎣ ⎦, processes [Eq. (18)] is equivalent to the statistic in Eq. (32). xy|w y1:h−1 (37) Therefore, estimates of conditional mutual information under w
013145-7 OLIVER M. CLIFF et al. PHYSICAL REVIEW RESEARCH 3, 013145 (2021)
= ,..., × < Wilks’ criterion or conditional mutual information) in order with x1:g [x1 xg] a g T matrix when 0 g k, and the empty set, x1:g =∅, when g = 0. Using this notation, we to derive their finite-sample hypothesis tests. have an equivalent expression for mutual information as A. Two time series |sxy|w| exp (−2 Iˆx;y|w ) = |sx|w||sy|w| We shall first express the Granger causality for bivariate processes as a sum of conditional mutual information terms = − Iˆ {gh} exp 2 xy|w via the chain rule. Let upper indices (without parentheses) j g,h denote a backshifted variable, e.g., X (t ) = X (t − j) denotes the variable X (t ) lagged by j time indices. Then, by applying = − 2 . 1 r ·v{gh} (38) xgyh xy|w the chain rule (35) to Granger causality (20), we can compute g,h it as a sum of conditional mutual information estimates: Although this equation has a similar form to the well-known q Fˆ , = Iˆ { } , canonical correlation decomposition [30,53], the correla- y→x|w (p q) 2 j |v j (44) x;y y→x|w tions are over different variables. More importantly for our j=1 purposes, and assuming independence of these partial corre- with the jth conditional as the matrix lations (see below), its null distribution can thus be obtained ∗ ⎡ ⎤ from the -distribution: x(p) { } − ∗ v j = ⎣y( j 1)⎦, (45) exp (−2 Iˆ |w ) ∼ (n |w ), (39) y→x x;y xy w with parameter vector (0) =∅ and the limiting case giving y , i.e., the empty set. = {11} ,..., {kl} . nxy|w nxy|w nxy|w (40) Again, following Eq. (39) we conclude that under the null hypothesis H : F → (p, q) = 0, Granger causality estimates The remaining challenge is to compute the effective degree 0 Y X {gh} are distributed as follows: of freedom, n |w, for each independent partial correlation in xy ∗ Eq. (38). Recall that partial correlation can be computed from exp [−Fˆy→x|w (p, q)] ∼ (ny→x|w ), (46) ordinary least squares. The residual vector for the (g, h)-term in Eq. (38)is where {1} {q} { } { } = ,..., . gh = − v gh , ny→x|w ny→x|w ny→x|w (47) ex|v xg xˆg xy|w (41) Now, we can use the same approach from Sec. V to obtain {gh} = − v{gh} , ey|v yh yˆh xy|w (42) the effective degrees of freedom used in computing Granger causality estimates. First, the residuals for the jth partial cor- · where the hat inx ˆg( ) again denotes the linear prediction of xg relation in Eq. (44)are from the input argument. Then the degrees of freedom used in , { j} { j} computing the (g h) sample partial correlation are e |v = x − xˆ v → |w , (48) x y→x|w y x {gh} {gh} {gh} {gh} = η , − v − . { j} { j} nxy|w ˆ ex|v ey|v dim xy|w (t ) 2 (43) e = y j − yˆ j v . (49) y|vy→x|w y→x|w Throughout this analysis, we ordered the summations first over the dimensions of y, and then over the dimensions of x. Thus the number of degrees of freedom is different for each In practice, the order of these operations are arbitrary and was term, with the jth number computed as initially imposed solely for clarity in the chain-rule formula. { j} { j} { j} { j} n → |w = ηˆ e |v , e |v − dim v → |w (t ) − 2. (50) It should be noted that the modified -test assumes both y x x y→x|w y y→x|w y x that the residuals are completely independent and that the Unlike the standard F-test for Granger causality [Eq. (24)], estimated degrees of freedom is approximately correct. If, the modified -test takes into account the effective number instead, the residuals become slightly correlated due to sta- of degrees of freedom induced by autocorrelation in both the ∗ tistical errors in the regression, the -distribution should be predictee and predictor processes, with the two approaches generated by sampling dependent beta- or F-distributed vari- overlapping only when there is no autocorrelation in the resid- ables. Further, the effective degree of freedom is a first-order uals. This indicates that, with limited data, the F-test can be approximation, which may introduce biases in the hypothesis used only for assessing the significance of Granger causality tests. Although our numerical simulations in Sec. VII show no estimates from independent observations (y) to univariate au- such biases, we discuss the potential solutions in Appendix D. tocorrelated time series (x).
VI. MODIFIED TESTS FOR GRANGER CAUSALITY B. Multiple time series Recall that Granger causality can be expressed as a con- Finally, we present hypothesis tests for the most complex ditional mutual information [see Eq. (20)]. As such, we can linear dependence measure in the paper: the Granger causality leverage results from the previous section to introduce its null from an l-variate predictor process Y to a k-variate predictee distribution. Many other information-theoretic and likelihood process X, in the context of the c-variate concomitant process ratio-based measures could be similarly decomposed (from W . By virtue of the chain rule (35), this general expression of
013145-8 ASSESSING THE SIGNIFICANCE OF DIRECTED AND … PHYSICAL REVIEW RESEARCH 3, 013145 (2021)
Granger causality (20) decomposes into three nested sums: the time dimension with two types of low-pass causal filters: a finite-impulse response (FIR) linear-phase least-squares fil- k l q ter and an infinite-impulse response (IIR) Butterworth filter. Fˆ , = Iˆ { } , x→y|w (p q) 2 x y j |v gh j (51) g h y→x|w The filter order is variable, with higher filter orders generally g= = j= 1 h 1 1 increasing the autocorrelation of the time series. For each where the conditional for the (g, h, j) mutual information term experiment, we perform 1000 trials and, using the statistical is hypothesis-testing procedure described in Appendix A4,we ⎡ ⎤ consider the FPR to be the proportion of p-values that are x(p) ⎢ 1:g ⎥ significant at the nominal level (typically 5% in this paper) ⎢ (q) ⎥ {gh j} ⎢y1:h−1⎥ in comparison to the relevant null distribution. v → |w = ⎢ ⎥. (52) y x ⎣ ( j−1)⎦ These experiments allow us to study how each test behaves yh w under increasing levels of autocorrelation of both x and y, whilst ensuring that the null hypothesis (of no dependence) That is, the (g, h, j) term is the conditional Granger causality is not violated. Rather than using a filter, it would of course for dimension g of X and the predictor observation j steps be possible to increase the autocorrelation by selecting the back of the hth subprocess of Y. This is conditioned on ARMA parameters, (u) and (u), for each lag u. However, all dimensions and predictor observations below g, h, and the formulations are equivalent: AR processes are all-pole IIR j,aswellasonW . Again, following Eq. (39) the distribu- filters; MA processes are FIR filters; and ARMA processes are tion of Granger causality estimates, under the null hypothesis IIR filters with both poles and zeros. Thus, although these are H0:FX→Y (p, q) = 0, is given as identical formulations, we opt for digitally filtering processes to increase their autocorrelation as this is a common prepro- Fˆ , ∼ ∗ , exp [ y→x|w (p q)] (ny→x|w ) (53) cessing step performed by practitioners to remove artifacts where from time series (even differencing the signal is a type of filter). Moreover, as previously discussed, filtering the signals = {111} ,..., {klq} . ny→x|w ny→x|w ny→x|w has been shown in the past to bias various dependence mea- The effective degrees of freedom are, again, computed from sures such as Granger causality [4]. Until this work, however, the residuals, where the residual processes used in computing it has not been suggested that this bias is a function of autocor- the (g, h, j) partial correlation are relation nor has a valid hypothesis test been proposed based on the autocorrelation function. { } { } e gh j = x − xˆ v gh j , (54) x|vy→x|w g g y→x|w {gh j} = j − j v{gh j} . A. Mutual information tests for bivariate time series ey|v → |w yh yˆh y→x|w (55) y x First, we use this approach to evaluate the performance of The number of degrees of freedom for each term in the the hypothesis tests on assessing the significance of mutual chained sum can then be computed from these residuals: information estimates between two independent (but seri- {gh j} {gh j} {gh j} {gh j} ally correlated) time series. Our results are shown in Fig. 1, n → |w = ηˆ e |v , e |v − dim v → |w (t ) − 2. (56) y x x y→x|w y y→x|w y x where the “F-tests” are from the finite-sample distribution 2 We note that, although the same p and q are used for each of [Eq. (17)], “χ -tests” refer to the asymptotic LR distributions the subprocesses of x and y in our presentation, our decompo- [Eq. (16)], and the “Modified -tests” refer to our hypothesis sition facilitates setting an individual history length for each tests that account for autocorrelation [Eq. (33)]. As the plots 2 term in the chained sum. The only difference would be to infer illustrate, both the F-tests and the χ -tests overestimate the the optimal history length (p and q) for each residual vector in measures for higher filter orders (and therefore higher AR the chain. orders), yielding over 15% of false positives at the nominal significance of α = 0.05—approximately three times the FPR expected from the test. The figures on the right illustrate the VII. NUMERICAL VALIDATION significance level α against the FPR for an eighth-order filter. We perform numerical simulations in order to validate the From these figures, we can see that the FPR for the χ 2-tests modified -test and characterize the effect of autocorrelation is higher than nominal for all significance levels α ∈ (0, 1). on the (unmodified) F- and χ 2-tests. The simulations, detailed In comparison, the modified -test procedure yields the ex- in Appendix B1, involve generating observations from two pected FPR for all filter orders. first-order independent AR processes and iteratively filtering A filter order of zero in Fig. 1 refers to generating the time- the output signal such that the autocorrelation is increased for series data with the first-order AR model (B2) without any both time series. digital filtering. In this case, the χ 2- and F-tests yielded less Following Barnett and Seth [4], we illustrate the finite- than the nominal 5% FPR. This occurs when the number of sample effects by generating relatively short stationary time effective samples becomes greater than the original sample series with T = 29 = 512 observations from the stochastic size η(x, y) > T . Referring to Bartlett’s formula (9), this is processes to obtain our dataset. We begin by sampling first- due to the product of negative autocorrelation exhibited by order AR processes to obtain our time-series data, x, y, and w. the Y process (induced by Y =−0.8, indicating an effect of Then, to illustrate the effect of higher autocorrelation on the undersampling) and the positive autocorrelation exhibited by FPR of both methods, we digitally filter each time series along the X process (induced by X = 0.3). Counterintuitively, this
013145-9 OLIVER M. CLIFF et al. PHYSICAL REVIEW RESEARCH 3, 013145 (2021)
0.2 0.2
0.15 Modified Λ-test 0.15 Modified Λ-test χ2-test χ2-test F -test F -test 0.1 0.1 1 1
0.05 0.05 False positive rate (FPR) False positive rate (FPR) Actual FPR 0 Actual FPR 0 0 0 0 102030 01 103 104 01 Filter order Expected FPR Sample size Expected FPR (a) (a)
0.2 0.2
0.15 Modified Λ-test 0.15 Modified Λ-test χ2-test χ2-test F -test F -test 0.1 0.1 1 1
0.05 0.05 False positive rate (FPR) False positive rate (FPR) Actual FPR 0 Actual FPR 0 0 0 0 102030 01 103 104 01 Filter order Expected FPR Sample size Expected FPR (b) (b)
FIG. 1. Modified tests correctly assess the significance of the FIG. 2. Increasing the sample size does not mediate the effect mutual information estimated between two univariate time series of autocorrelation on the χ 2-andF-tests for mutual information. under FIR (a) and IIR (b) filtering. The mutual information was mea- We perform the same tests as Fig. 1, except with an exponentially sured[usingEq.(18)] between independent univariate time series increasing sample size and a fixed eighth-order FIR (a) and IIR [generated by Eq. (B2)] after filtering, and tested using the χ 2-test (b) filter. The subplots on the right show the FPR for T = 211 [Eq. (16)], F-test [Eq. (17)], and the modified -test [Eq. (33)]. The samples. plots show the effect of increasing the filter order on the FPR for both an (a) FIR filter and (b) IIR filter. The shaded regions on the right indicate α = 0.05, whilst the shaded regions on the left show the 95% are invalid, unless modified to account for effective sample confidence interval for the FPR (as defined in Appendix A4). The size. subplots on the right capture the FPR for all potential significance levels α (“Expected FPR”) with an eighth-order filtered signal. An ideal distribution is where the FPR equals α and thus sits perfectly B. Conditional mutual information tests for bivariate time series on the diagonal, as per our tests. We now extend the previous results by evaluating the effect of conditioning mutual information between x and y on an independent, tertiary process w. The FPRs for the χ 2-tests, would imply that the observations are anticorrelated in time. F-tests, and the modified -tests from these experiments are Such conservative results for the χ 2- and F-tests are likely to presented in Fig. 3, and exhibit similar characteristics to those induce lower statistical power [i.e., a lower true-positive rate of the mutual information tests in Fig. 1. Increasing the filter (TPR)] in scenarios when the effective sample size is greater order generally increases the FPR for both unmodified tests, than the original sample size. To verify this, we performed yet the modified -test remains unbiased, maintaining the 1000 trials where there was a small dependence of X on Y expected FPR of 5%. (see Appendix B1). The TPR was 0.049 (SE of 0.0068) for As discussed in Sec. V, an important distinction between the χ 2-test and 0.1570 (SE of 0.0115) for the modified -test. the null distributions for mutual information [Eq. (33)] is that Thus, the power of our hypothesis test is three times greater the effective degree of freedom in the ∗-distribution not than the χ 2-test in this scenario. only includes the effective sample size but also the dimension In Fig. 2 we show that increasing the sample size does not of the conditional c. To show the severity of the asymptotic mediate the effect of autocorrelation on the χ 2- and F-tests. approximation, we generate the x and y processes and filter This is due to the fact that the effective degree of freedom is the signal with an eighth-order FIR and IIR filter the same always a fraction of the degree of freedom. Thus, regardless of as before; however, we increase the number of independent the sample size, both the asymptotic (χ 2) and finite (F)tests processes c = dim (W (t )) in the multivariate conditional. The
013145-10 ASSESSING THE SIGNIFICANCE OF DIRECTED AND … PHYSICAL REVIEW RESEARCH 3, 013145 (2021)
0.2 0.4
0.15 Modified Λ-test 0.3 Modified Λ-test χ2-test χ2-test F -test F -test 0.1 0.2 1 1
0.05 0.1 False positive rate (FPR) False positive rate (FPR) Actual FPR 0 Actual FPR 0 0 0 0 102030 01 0 50 100 150 200 01 Filter order Expected FPR Dimension of conditional c Expected FPR (a) (a) FIR filter with c-variate conditional.
0.2 . 0 4 Modified Λ-test 0.15 Modified Λ-test χ2 χ2 -test -test F -test F -test 0.1 1 1 0.2
0.05 False positive rate (FPR) Actual FPR False positive rate (FPR) 0 0 0 Actual FPR 0 0 50 100 150 200 01 0 102030 01 Dimension of conditional c Expected FPR Filter order Expected FPR (b) IIR filter with c-variate conditional. (b) FIG. 4. The FPR of asymptotic tests increase with the dimension FIG. 3. Modified tests correctly assess the significance of the of the c-variate conditional process. The plots are the same as per conditional mutual information estimated between two univariate Fig. 3, except as a function of c with a fixed eighth-order FIR (a) and time series, conditioned on a third univariate time series; each time IIR filter (b). The subplots on the right show the FPR for each series underwent FIR (a) or IIR (b) filtering. Conditional mutual significance level α when c = 100. information was measured [using Eq. (18)] between two univariate time series, given a third univariate process [generated by Eq. (B2) = = = with k l c 1], after filtering, and tested for significance using subprocess, i.e., ρij = 0 for all i, j ∈ [1, m]. The results are the χ 2-, F-, and modified -tests. The subplots on the right show the shown in Fig. 5, where increasing the dimension approxi- FPR for each significance level α when the signal is filtered with an mately linearly increases the FPR of the original LR test to eighth-order filter. over 70% for both filters (continuing to increase for larger k and l), yet the modified -test remains unbiased. result is shown in Fig. 4, where the FPR of the F- and χ 2-tests In the tests above, no correlations between subprocesses − = increases somewhat linearly with the dimension; however, were included [e.g., Xi(t ) with Xj (t u)for j i]; however, the modified -test remains unbiased. As evidenced by the an internal cross-correlation between any of these subpro- improvement of the unmodified F-test over the χ 2-test, this cesses may further decrease the size and power of unmodified experiment demonstrates that even when the autocorrelation tests. Our experiments of Granger causality in the following function is the same, the dimension of the conditional must sections naturally incorporate examples with correlated sub- also be included in the hypothesis tests. processes in the mutual information calculation.
C. Mutual information tests for multivariate time series D. Granger causality tests for bivariate time series In this section, we present results for the hypothesis tests of This section examines the performance of each hypothesis mutual information between multivariate (m > 2) time series. test on estimates of Granger causality using the same (uni- The multivariate time series are partitioned into two indepen- variate) simulations from Sec. VII A. That is, the bivariate AR dent sets of processes, X and Y, one with dimension k and one model [Eq. (B2) with k = l = 1 and no conditional c = 0] is with dimension l. For each experiment, we let k = l and use simulated to generate T = 512 observations of the X and Y the state equations in Eq. (B2) to simulate m = k + l indepen- processes, which are then passed through FIR and IIR filters. dent AR processes for m ={2, 4,...,10}. These signals are Referring to Fig. 6, we perform this with each filter order and then filtered along the temporal dimension using eighth-order each filter type (FIR and IIR). After generating these sample FIR and IIR filters. This signal generation process ensures paths, the AR order of the predictee, p, and the predictor, that there is no correlation between signals within the same q, were inferred from Burg’s method [58]. We then compute
013145-11 OLIVER M. CLIFF et al. PHYSICAL REVIEW RESEARCH 3, 013145 (2021)
0.8 0.2
0.6 0.15 Modified Λ-test Modified Λ-test χ2-test χ2-test F -test 0.4 0.1 1 1
0.2 0.05 False positive rate (FPR) False positive rate (FPR) Actual FPR 0 Actual FPR 0 0 0 12345 01 0 102030 01 Dimension k, l Expected FPR Filter order Expected FPR (a) (a)
0.8 1 . 0 8 Modified Λ-test 0.6 2 χ -test Modified Λ-test . 2 0 6 F -test χ -test 0.4 1 1 0.4
0.2 0.2 False positive rate (FPR) Actual FPR False positive rate (FPR)
Actual FPR 0 0 0 0 0 102030 01 12345 01 Filter order Expected FPR Dimension k, l Expected FPR (b) (b) FIG. 6. Modified -tests correctly assess the significance of the FIG. 5. Modified -tests correctly assess the significance of the Granger causality estimated from one univariate time series to an- mutual information estimated between two multivariate time series. other. Granger causality, with history lengths p and q chosen via The mutual information was measured [using Eq. (38)] between the Burg’s method, is estimated [using Eq. (44)] between univariate time multivariate time series [generated by Eq. (B2) with an increasing series [generated by Eq. (B2)] after smoothing with an FIR (a) and dimension k and l of X and Y], passed through eighth-order FIR an IIR (b) filter. Estimates are then tested using the χ 2-test [Eq. (22)], (a) and IIR (b) filters, and tested for significance using the χ 2-test the F-test [Eq. (24)], and the modified -test [Eq. (46)]. The subplots [Eq. (16)] and the modified -test [Eq. (39)]. The subplots on the on the right show the FPR for each significance level α with an right show the FPR for each significance level α when k = l = 3. eighth-order filter.
Granger causality [via Eq. (44)] and use this estimate to obtain introduction, all of which can result in vastly different model p-values from the CDFs of the F-distribution [Eq. (24)], the orders. To ensure our results are not a consequence of poor χ 2-distribution [Eq. (22)], and the ∗-distribution [Eq. (46)]. model identification, in Fig. 7 we illustrate the FPRs for in- This is performed 1000 times in order to obtain a FPR of each creasing the predictor history length q from one to 200 (in approach. Whilst our experiment here is equivalent to a condi- increments of 20), whilst holding the autocorrelation length tional mutual information, unlike the experiments in previous constant with an eighth-order filter. This effectively introduces sections the autocorrelation within the time series naturally more terms in Eq. (44), causing a larger divergence between induces a cross-correlation amongst the variables within the the F-, χ 2- and modified -tests. As expected, the FPR of the predictor Y (q)(t ) and conditional X (p)(t ) processes. The re- χ 2- and F-tests linearly increases in this range, whereas the sults shown here illustrate that increasing the autocorrelation modified -test remains consistent with the 5% FPR. This length via filtering increases the FPR of Granger causality linear increase of the FPR in χ 2- and F-tests is somewhat under the F- and χ 2-tests, particularly when using an IIR counterintuitive to the notion of Granger causality, where one filter. In contrast, the FPR of Granger causality using the may expect that accounting for more history would reduce modified -tests remains mostly unbiased. It should be noted spurious correlations. However, the opposite is true, simply that, although within the confidence bounds, the FPR of the due to a lack of correct finite-sample distributions (in the case modified -tests appear to be not exact for high-order filters; of the unmodified tests). sources of error regarding this potential bias are discussed in In Fig. 8 we show the effect of increasing the sample size Appendix D. for tests on Granger causality estimates. Here we can see The model order for our experiments above was chosen the χ 2- and F-tests converging for sufficient sample sizes. using Burg’s method; however, the are numerous approaches Unlike mutual information (from Fig. 2), estimating Granger to inferring the “optimal” AR order as outlined in the causality involves regressing the autocorrelation of the
013145-12 ASSESSING THE SIGNIFICANCE OF DIRECTED AND … PHYSICAL REVIEW RESEARCH 3, 013145 (2021)
0.6 0.2
Modified Λ-test 0.15 Modified Λ-test 2 0.4 χ2-test χ -test F -test F -test 0.1 1 1 0.2 0.05 False positive rate (FPR) False positive rate (FPR) Actual FPR 0 Actual FPR 0 0 3 4 0 20 40 60 80 100 01 10 10 01 T History length q Expected FPR Sample size Expected FPR (a) (a) . 1 0 8
0.8 0.6 Modified Λ-test Modified Λ-test 2 χ2 χ -test -test F 0.6 F -test -test 0.4 1 1 0.4 0.2 0.2 False positive rate (FPR) Actual FPR False positive rate (FPR)
Actual FPR 0 0 0 0 3 4 01 0 20406080100 01 10 10 Sample size T Expected FPR History length q Expected FPR (b) (b) FIG. 8. The F-andχ 2-tests converge to the modified -test for FIG. 7. The FPR of the F-andχ 2-tests for Granger causality Granger causality with large sample sizes. This experiment is the increases with an increasing dependence on the past for time series same as Fig. 6, except with an exponentially increasing sample size generated by an eighth-order FIR (a) and IIR (b) filter. We illus- and a fixed eighth-order FIR (a) and IIR (b) filter. trate this by varying the history length of the predictor process, q, from one to 200. The subplots on the right show the FPR for each significance level α when q = 20 (a) and q = 40 (b) to approxi- The results are shown in Fig. 9, where the χ 2-tests inflate mately match the orders chosen via Burg’s method in Fig. 6. the FPR close to 100% for with higher dimensional processes. Although our corrected tests perform well for moderate di- predictee first, with the variance of these residuals reducing as mensionality, when k, l > 3 with the IIR filter, the FPR of the sample size grows. Thus, the effective sample size asymp- our modified -tests begin to have numerical issues. This is totically approaches the sample size, however, the precise caused by the regression matrix not being well conditioned, rate of this convergence depends the autocorrelation function i.e., the ratio of regressors to data points is too high. Nonethe- and may change for every pair of time series. Even for this less, we can see from the figure that our tests maintain a simple example, we see that on the order of 100 000 samples low FPR, becoming more conservative when the regression are required for convergence, which is not realistic in many is ill-posed. Moreover, a poorly conditioned regression can be empirical scenarios. easily tested for in practice. So we conclude that with min- imally sufficient observations, our tests maintain the desired FPR even for the most general case of multivariate Granger E. Granger causality tests for multivariate time series causality and, when the sample size is simply too small for Finally, we can evaluate the effect of increasing the dimen- reliable inference, our approach flags this as an issue. sionality of both processes on Granger causality inference. In these experiments, we vary the dimension of the processes X VIII. EFFECT OF PREWHITENING and Y from one to five. Recall from Eq. (51) that the number of terms involved in computing Granger causality (and its The rationale for applying prewhitening is to remove the null distribution) is the product klq, of the dimensionality autocorrelation in one time series such that the variance of (k, l) and the history length of the predictor (q). Due to the computed statistics becomes equivalent to serially indepen- relatively short time series length of T = 512 samples and dent observations [12]. That is, instead of modifying the high autocorrelation and dimensionality, allowing an arbitrary hypothesis tests, prewhitening modifies the input time series predictor history length of q results in the effective sample and thus the statistics themselves. Prewhitening is typically size approaching zero for the modified -tests. Thus, for these attempted by first inferring a model of one time series (x), and experiments we fix the history length of the predictor to q = 1. transforming the process x to a residual process x˜ through a
013145-13 OLIVER M. CLIFF et al. PHYSICAL REVIEW RESEARCH 3, 013145 (2021)
0.3 after mean removal and differencing, the most general model for covariance-stationary time series are ARMA models. However, inference procedures to learn both the order and 0.2 Modified Λ-test parameters of ARMA models are computationally expensive. χ2-test As such, many authors (and textbooks [12]) propose an AR(p) 1 model would suffice to render the residuals, x˜, independent, . presuming that autocorrelations decay rapidly for stationary 0 1 time series. Since this is the same assumption underlying Granger causality [regarding the residuals on the target after
False positive rate (FPR) fitting an AR(p) model], our results from Sec. VII D suggest 0 Actual FPR 0 12345 01that statistics computed from signals prewhitened in this way will remain biased. In fMRI research, the most popular pack- Dimension k, l Expected FPR ages that are used for preprocessing time-series data (AFNI, (a) SFL, and SPM) are similarly insufficient for handling auto- correlation due to their simplistic models [15]. Specifically, 1 the package AFNI uses an ARMA(1,1) model learned from each voxel, whereas FSL uses Tukey tapering to smooth the 0.8 data (see Appendix D), and SPM uses one global AR(1) model Modified Λ-test for all processes. Thus, each package assumes a fixed ARMA . 2 0 6 χ -test model can describe any arbitrary-order process. This is clearly . 1 insufficient, and if the wrong model is used, the residuals x˜ 0 4 remain dependent, resulting in an unknown variance of statis- tical estimates. This is evidenced by consistently high FPRs 0.2 in empirical studies [15]. False positive rate (FPR) 0 Actual FPR 0 For completeness of this paper, however, we implement 12345 01a number of prewhitening schemes in order to illustrate that Dimension k, l Expected FPR such an approach is insufficient for assessing linear depen- dence between typical time series. Our experiments—on the (b) same synthetic time series used in Sec. VII—show the effect FIG. 9. Modified -tests correctly assess the significance of of prewhitening univariate signals on the unmodified F-test the Granger causality estimated from one multivariate time series for a number of different models: the AR(1) and ARMA(1, 1) to another for increasing dimension k, l. Granger causality, with as well as the AR(p) and ARMA(p, q) with the optimal model each predictee history length chosen optimally, is estimated [using orders (p and q) learned from data. For the AR(p) model Eq. (51)] between multivariate time series [generated by Eq. (B2)] inference, we allow for p ∈ [1, 200] (where the sample size after filtering via eighth-order FIR (a) and IIR (b) filters. Estimates is T = 512), with the model order selected (and parameters are then tested using the χ 2-tests [Eq. (22)] and our modified - inferred) using Burg’s method [58]. Given the difficulty of test [Eq. (53)]. The subplots on the right show the FPR for each learning higher-order ARMA(p, q) models, however, we re- significance level α when k = l = 3. strict our search space, iterating through each potential p, q ∈ {1,...,5} and selecting the model with the lowest BIC score. filter constructed from its model. The same filter (with param- Our results for the performance of the F-test on mu- eters inferred from x) is then applied to the other time series y tual information estimates after prewhitening are shown in to create y˜. Specifically, assuming any arbitrary ARMA(p, q) Fig. 10. Contrasting these results to Fig. 1, the only bene- model for time series x, prewhitening involves learning the fit of prewhitening appears to be for the FIR-filtered time parameter vectors ˆ and ˆ from x, and then filtering the raw series when an ARMA(p, q) model is used. In almost all signals through the following equations: other scenarios, the FPRs are either equivalent to, or worse than, the original tests. Figure 11 illustrates the effect of p q prewhitening on Granger causality F-tests (compare to Fig. 6 x˜(t ) = x(t ) − ˆ (u)x(t − u) − ˆ (u)x˜(t − u), without prewhitening). Again, prewhitening appears to serve = = u 1 u 1 no benefit to tests for Granger causality, even for the relatively p q advanced ARMA(p, q) model. Concerningly, for IIR-filterd = − ˆ − − ˆ − . y˜(t ) y(t ) (u)y(t u) (u)y˜(t u) signals, the FPR increases to over 60% for all ARMA models u=1 u=1 with no scenario where the prewhitening approach results Using the same linear transformation (filter) for both time in an FPR within the expected range. In Appendix C we series renders their correlation theoretically invariant [12], demonstrate similar results when using BIC and AICc scoring however, the assumption is that x˜ is now serially independent, functions (as an alternative to Burg’s method) to infer AR(p) and so the variance of sample correlations (for instance) con- models for prewhitening. 2 verge toσ ˆr (x, y) = 1/T . It is possible that, with an ideal model, the F-orχ -tests A significant challenge in prewhitening signals is in se- used for a prewhitened time series may be comparable (in lecting an appropriate model of the autocorrelation function. size properties or FPR) to the modified -test. However, even Of course, for the same arguments as presented in Sec. II, restricting our search space to an ARMA(5, 5) model resulted
013145-14 ASSESSING THE SIGNIFICANCE OF DIRECTED AND … PHYSICAL REVIEW RESEARCH 3, 013145 (2021)
0.3 0.6 F -test, AR(1) F -test, AR(1) F -test, ARMA(1,1) F -test, ARMA(1,1) 0.2 F -test, AR(p) 0.4 F -test, AR(p) F -test, ARMA(p,q) F -test, ARMA(p,q) 1 1 0.1 0.2 False positive rate (FPR) False positive rate (FPR) 0 Actual FPR 0 0 Actual FPR 0 0102030 01 0102030 01 Filter order Expected FPR Filter order Expected FPR (a) (a)
0.3 1 F -test, AR(1) F -test, AR(1) . F -test, ARMA(1,1) 0 8 F -test, ARMA(1,1) 0.2 F -test, AR(p) F -test, AR(p) . F -test, ARMA(p,q) 0 6 F -test, ARMA(p,q) 1 1 0.4 0.1 0.2 False positive rate (FPR) False positive rate (FPR) 0 Actual FPR 0 0 Actual FPR 0 0102030 01 0102030 01 Filter order Expected FPR Filter order Expected FPR (b) (b)
FIG. 10. Prewhitening the time series does not mediate the FIG. 11. Prewhitening the time series does not mediate the bias bias in F-tests for mutual information estimates after FIR (a) or in F-tests for Granger causality estimates after FIR (a) or IIR (b) fil- IIR (b) filtering. The experiments are as per Fig. 1, with four tering. The experiments are as per Fig. 6, with four prewhitening prewhitening approaches used: the AR(1), ARMA(1, 1), AR(p), and approaches used (as per Fig. 10): the AR(1), ARMA(1, 1), AR(p), ARMA(p, q). For the AR(p)andARMA(p, q) models, the opti- and ARMA(p, q). mal order is inferred from Burg’s method and the BIC score. In many cases for the IIR-filtered time series, either the AR(p)orthe ARMA(p, q) failed to learn a stable model, and so these trials were In fMRI research, the blood-oxygen level-dependent removed, inducing nonuniform confidence intervals (reflected in the (BOLD) data are translated into a slowly varying (and thus shaded region). highly autocorrelated) multivariate time series that traces the haemodynamic response of different locales (voxels) in the brain. Digital filtering is then commonly used as a prepro- in an approximately 5000-fold increase in computational time cessing step to reduce line noise, nonstationarity and other over the modified -test [62] and, moreover, remained biased artefacts in neuroimaging data. This induces an (either finite in most scenarios. We conclude that, regardless of the model or infinite) impulse response that can increase autocorrelation, selected, the additional burden of prewhitening over using a even if the original signals were not serially correlated. To modified hypothesis test is unjustified and that the outcome characterize the FPR of linear dependence measures between of unmodified hypothesis tests (with or without prewhitening) empirical time series, we use completely independent time conveys inconsistent information about the underlying depen- series from a widely accessed brain-imaging dataset known dence structure between time series. as the the Human Connectome Project (HCP) resting state fMRI (rsfMRI) dataset [44] (see Appendix B2 for more de- tail; time series are selected from different random regions IX. CASE STUDY: HUMAN CONNECTOME of interest from different random subjects to ensure inde- PROJECT DATASET pendence, and then digitally filtered). This process is shown Studies in computational neuroscience often leverage sta- in Fig. 12, where a sample window of the raw and filtered tistical analysis in order to formulate and test biologically data from two independent time series appears on the left plausible models. An important application in this field is the panel. study of the human brain through fMRI, which is abundant First, we illustrate the effect of autocorrelation on mutual with short, autocorrelated time series that have been studied information by using a χ 2-test, with significance level 5%, using the measures of interest here [1–5]. As such, it is an computed with T = 800 samples of each independent time archetypal real-world application to illustrate the issues of series. We begin by estimating mutual information Iˆx;y be- autocorrelation for time-series analysis. tween two unrelated time series x and y, sampled from the
013145-15 OLIVER M. CLIFF et al. PHYSICAL REVIEW RESEARCH 3, 013145 (2021)
Time series data Mutual information Granger causality t − tt− t t − p t − t Raw I 1 1 1 Filtered X ;Y |W X Y
) W t
( t − q
x Time
Significance level α Significance level α Significance level α 010101 100% ) t ( y
Test at 5% False positive rate (FPR) 0%
Modified Λ-test (fixed): 5.5% Modified Λ-test: 4.7% Modified Λ-test: 5.9% Modified Λ-test (optimal): 6.5% Autocorrelation 2 2 χ2-test: 41.8% χ -test: 83.4% χ -test (optimal): 16.9% Lag χ2-test (fixed): 65.3%
FIG. 12. Application to brain-imaging data, demonstrating our correction for the otherwise dramatic inflation of false-positive rates (FPRs) of classical hypothesis tests of dependence estimates in the presence of autocorrelation. We perform 1000 experiments with both the χ 2-tests as well as the modified -tests for inferring the significance of two dependence measures: mutual information(between two univariate and two bivariate processes), and Granger causality. For each of these experiments, we randomly selected two uncorrelated fMRI time series, X and Y, from the Human Connectome Project [44] (see Appendix B2 for more details). A sample window of these univariate processes are shown in the left panel (gray lines), with the common preprocessing technique of band-pass digital filtering later applied to the signals (red lines). At the bottom of the left panel, we plot the sample autocorrelation function from both the raw and the filtered signal, illustrating the higher autocorrelation length (and lower effective sample size) induced by digital filtering. The panels on the right show the FPR of each test applied to the filtered signals as a function of the significance level α. For ideal hypothesis testing, we expect a line along the diagonal (i.e., the FPR equals the significance level). The χ 2-tests illustrate an increased FPR for all measures, as seen by both the plots and the FPR at 5% significance level. In contrast, our tests remain consistent with the expected FPR.
HCP dataset. We find that the test results are significantly X. DISCUSSION biased, yielding an FPR of 41.8% (a ninefold increase of the We have shown that the autocorrelation exhibited in expected rate). After prewhitening with an AR(p) filter, this covariance-stationary time-series data induces bias in the increased to over 88% (not shown in the figure). The modified hypothesis tests of a broad class of linear dependence mea- -test demonstrated an FPR of 4.7%, which is well within sures. By framing different dependence measures in unified the acceptable confidence interval. Next, we perform the same theoretical terms, we provide the first demonstration of how tests but with the mutual information between two sets of Bartlett’s formula can be applied to derive unbiased hypoth- bivariate time series x and y (k = l = 2); this yields an 83% esis tests, termed modified -tests, for mutual information FPR (a 16-fold increase). When we correctly test these same (and, consequently, Wilks’ criterion and Granger causality) measurements with the modified -test [Eq. (39)], we find a for both univariate and multivariate time-series data. These FPR of 5.2%, matching the desired level within the confidence measures are used in a wide range of disciplines, model- bounds. ing myriad important processes from anthropogenic climate For Granger causality, we perform two different tests: change [26] to the brain dynamics of dementia patients [63]. scenario (i) with the model orders, p and q, inferred for The continued use of flawed testing procedures in empirical each trial, and scenario (ii) with a fixed p = q = 100 for sciences is problematic, making it imperative that the correc- all trials. For unrelated signal pairs, the FPR of Granger tions reported here be incorporated into future studies. causality estimates using the χ 2-test is 16.9% (with opti- The effect of temporal autocorrelation on linear depen- mal history length around 18), increasing further to 90.5% dence has long been investigated in statistics; however, the after prewhitening with an AR(p) filter. If a longer em- majority of research has focused on simple linear correlations bedding length of 100 is chosen, the FPR is 66%—more [1,2,10,11,40], which are restricted to measuring symmetric than 13 times the expected value—increasing to 83.3% after bivariate dependence structures. These studies, while rep- prewhitening with an AR(p) model. When we test these same resenting important milestones in inferring the association measurements with the modified -test (46), we find a FPR between univariate time series, suffer from the inability to of 5.5% and 6.5% for scenarios (i) and (ii), respectively, capture both multivariate and directed dynamical depen- completely removing the false-positive bias exhibited by the dence. Even though measures such as mutual information χ 2-tests.
013145-16 ASSESSING THE SIGNIFICANCE OF DIRECTED AND … PHYSICAL REVIEW RESEARCH 3, 013145 (2021) and Granger causality can model a much richer set of de- directed inference of relationships in time-series analysis. pendence structures in temporal processes, the notion that Much like correlation coefficients, the issue was magnified their sampling properties could be altered in the presence in fields dealing with short, highly autocorrelated time series, of autocorrelation has thus far been largely overlooked. A as demonstrated in Fig. 12 for computational neuroscience major challenge of handling autocorrelation for more in- using fMRI recordings. Many extensions to Granger causality volved dependence measures was in extending Bartlett’s have been proposed to explicitly model the autocorrelation formula to multivariate relationships. Crucially, our approach function (such as ARMA [65] or state-space Granger causality facilitates not only independent multivariate relationships but [66]). However these approaches are also known to exhibit also correlated multivariate processes. Our results on the significant false-positive biases [67], aligning with our re- mutual information between multivariate time series used ex- sults in Sec. VIII on the ineffectiveness of prewhitening with amples where the subprocesses were all independent. When ARMA models. In this paper, we showed that modifying the extending this approach to Granger causality for arbitrarily effective degrees of freedom of the null distribution suffices to large history lengths and dimensionality, the subprocesses eliminate the bias across all examined time series, without the become inherently correlated (since one subprocess is a additional burden of inferring complex models or prewhiten- time-lagged version of another, and they involve significant ing. More advanced methods (such as state-space Granger autocorrelation). This provides strong evidence that our ap- causality) may retain other empirical advantages, but their proach is a generalization of Bartlett’s dependence studies hypothesis tests are likely to require incorporation of similar that is able to handle a much richer class of multivariate modifications based on effective sample sizes. More broadly, dependency structures beyond those already presented in this our results strongly suggest that any hypothesis tests dealing paper. with time-series analysis should be modified to account for Granger causality is the de facto measure of directed autocorrelation, regardless of regressing or conditioning on dependence between stationary time series. The typical AR components. The concerningly high FPRs exhibited in approach to assessing the significance of (potentially mul- our experiments suggest that relationships established using tivariate) estimates has been via the finite-sample F-tests previously tested Granger causality estimates should perhaps [64] or the asymptotic χ 2-test [46]. In this study, we have be revisited; particularly in fields that have high levels of shown that higher levels of autocorrelation (equivalently, a autocorrelation and limited data. higher-order autoregression) in the signal inflates the vari- Throughout this work, we have made the assumption that ance of these statistical estimates, inducing significant bias the time-series innovations are Gaussian and that all relation- for these traditional hypothesis tests. This means that using ships are linear. Thus, we have only discussed linear-Gaussian these tests induces errors when the predictor process is serially probability distributions for information-theoretic measures. correlated—precisely the situation that Granger causality was When instead applied to nonlinear time series, these prob- designed to address. One might logically surmise that this ability distributions are often inferred using nonparametric issue could be mediated by accounting for a longer history density estimation techniques such as nearest neighbor or ker- of the process, i.e., conditioning on additional AR variables. nel methods [68]. Spurious estimates of the nearest-neighbor Referring to Fig. 6, we have shown that this will result in the counts have previously been observed for autocorrelated sig- FPR being even further inflated. This is particularly concern- nals by Theiler [69], who provided a solution by excluding ing given that the autocorrelation function is one of the two observations that are close in time. This is now a popular properties that define a stationary process [7,8,47]—the other approach to effectively account for autocorrelation in den- is its mean, which has no effect on scale-invariant dependence sity estimation for nonlinear time-series analysis. In fact, in measures such as Granger causality, mutual information, and introducing transfer entropy—now understood as a model- Pearson correlation. Similar to the thinking behind Granger free extension of Granger causality [35]—Schreiber explicitly causality, another common approach to remove autocorre- recommended the use of a Theiler window (also known as lation is prewhitening, which intends to induce a serially serial- or dynamic-correlation exclusion) when kernel estima- independent process (of residuals) for hypothesis testing. Our tion methods are used [36]. The Theiler window approach has experiments in Fig. 10 and Fig. 11 (as well as Appendix C) been demonstrated to control the FPR for such estimators in illustrated that many common approaches to prewhitening fail practice [70], yet remains a heuristic with no theoretical guar- to control the FPR and, indeed, can further reduce the size antees and, similar to Pearson correlation, is often neglected in of the unmodified (F and χ 2) hypothesis tests. We conclude practical estimation of transfer entropy [71]. We hypothesize that the unmodified hypothesis tests cannot be used to reliably that the methods outlined in our work could be extended infer the significance of Granger causality or mutual infor- in future to provide a more rigorous approach to handling mation estimates when applied to covariance-stationary time autocorrelated nonlinear time series through, e.g., nonlinear series and should be replaced with modified tests, such as the versions of Bartlett’s formula [72], facilitating a broader class modified -tests, particularly for limited time-series data. of information-theoretic measures. Prior work [4] had already established that digital filtering Finally, the dependence structure discussed in this work is led to biased Granger causality estimates for shorter time assumed to be in the time domain, whereas many empirical series, yet this effect was not understood nor able to be cor- studies are concerned with other forms of dependence that rected until now. Due to the widespread use and influence could similarly bias hypothesis tests. Future work will be of Granger causality across fields including neuroscience, required to consider handling such correlation structures in ecology and economics, underlined by any examination of a similar fashion to that which we have presented, e.g., spec- the literature (see Sec. I), this was a serious deficiency for tral models [21] or spatial autocorrelation [39]. Indeed, the
013145-17 OLIVER M. CLIFF et al. PHYSICAL REVIEW RESEARCH 3, 013145 (2021) formula for the effective sample size [Eq. (9)] was developed However, as we show throughout the main text, the χ 2-test has for spatial autocorrelation and can thus be easily extended a significant bias when applied to limited and autocorrelated to handle spatiotemporal autocorrelation, allowing for even time-series data, which results in a large number of false broader class of null distributions that can be considered with positives. the modified -test. It is important to note that the LR test is but one of three classical procedures for hypothesis testing maximum ACKNOWLEDGMENTS likelihood estimates; the others are the Wald test and the Lagrange multiplier test [7,8,47]. The three tests overlap be- O.M.C. and B.D.F. were supported by NHMRC Ideas cause the null distribution of each asymptotically follows the Grant 1183280. J.T.L. was supported through the Australian χ 2-distribution. Thus, the same issues hold if one were to use Research Council DECRA grant DE160100630. J.M.S. was any test on linear dependence measures unless autocorrelation supported through a University of Sydney Robinson Fellow- is considered in the null distributions. ship and NHMRC Project Grant 1156536. J.M.S. and J.T.L. were supported through The University of Sydney Research Accelerator (SOAR) Fellowship program. O.M.C., J.T.L., and 2. Finite-sample F-test J.M.S. were supported through The Centre for Translational In regression analysis, the F-test is used to infer the sig- Data Science at The University of Sydney’s Research Incuba- nificance of nested models of independent observations with tor Funding Scheme. O.M.C., J.T.L., J.M.S., and B.D.F. were limited data, i.e., the finite-sample null distribution. Using supported through the Centre for Complex Systems at The the same notation as above, we obtain a distribution for the University of Sydney’s Emerging Aspirations scheme. High- comparing the nested models: performance computing facilities provided by The University T − p S − S of Sydney (Artemis) have contributed to the research results 1 0 1 ∼ F p − p , T − p . − ( 1 0 1) (A3) reported within this paper. p1 p0 S1 F-statistics can be reformulated as nested ratios of sample APPENDIX A: STATISTICAL HYPOTHESIS TESTS variances through simply rearranging the LHS of (A3): − The linear dependence measures discussed in this paper are T p1 [ˆ − 1] ∼ F (p1 − p0, T − p1). (A4) positive real-valued random variables ˆ ∈ R>0 that can be ex- p1 − p0 pressed as the ratio of the generalized variance of two models. Thus, the F-statistic is a function of the LR of two models and In general, we consider two nested models, the “restricted” we can show its asymptotic distribution is chi-square. First, a model with p0 parameters (under which the null hypothesis H Taylor expansion of the LHS of the F-statistic in Eq. (A4) 0 is true) and the “unrestricted” model with p1 parameters ˆ ≈ ˆ − ∼ H gives log ( ) 1. Moreover, for a random variable X (under which the alternate hypothesis 1 is true). These mod- ν ,ν = ν F ( 1 2 ), then Y limν2→∞ 1X has the chi-square distribu- els are referred to as nested since p0 < p1 and the restricted 2 tion χ (ν1). Thus, by this asymptotic relationship between the model parameter space is a subset of the unrestricted model F- and χ 2-distributions, we have space. The statistics are expressed in terms of the generalized sample variance of these models: d lim (T − p1)log(ˆ ) → (p1 − p0 ) F (p1 − p0, T ), (A5) T →∞ |s0| ˆ = , (A1) 2 T log (ˆ ) ∼ χ (p1 − p0 ). (A6) |s1| where si is the the residual sum-of-squares for model i and This result is discussed throughout the paper to explain the |si| is the generalized sample variance. The generalized sam- diverging behavior between the two tests. ple variance is, asymptotically, inversely proportional to the likelihood of each model, and so taking the log of the ratio 3. Surrogate-distribution tests of generalized sample variances (A1) is equivalent to the LR Another established approach to empirically generating a between two models (for a large enough number of samples) null distribution involves permuting, redrawing or rotating [8,65]. For this reason, statistics of the form in Eq. (A1) the observations of one variable x or y and computing the appear in a number of linear dependence measures, such as relevant statistic for each surrogate dataset [4,50,68]. Naive mutual information (with Gaussian marginals) and Geweke’s approaches to permuting or redrawing will completely de- definition of Granger causality [46]. stroy the autocorrelation profile of that variable, making this empirical distribution representative of serially independent 1. Asymptotic likelihood-ratio test observations and similar to the analytic F-distribution. Wilks’ theorem [55] is the basis of the χ 2-test, which states Indeed, such empirical generation of the CDF via permuta- that a test statistic constructed from the LR of nested models tion testing was attempted (for Granger causality) by Barnett will asymptotically follow a χ 2-distribution under the null hy- and Seth [4], and shown to incur the same inflated FPR issues pothesis. Since all statistics used in this work fit this definition, as the F- and χ 2-tests. Alternatively, constrained realization a χ 2-test can be used. That is, if the true model is the restricted approaches [73] can be used to generate surrogate time-series model, then as T →∞, the statistic is chi-square distributed: data that exhibit certain properties (such as the same power spectra) and have recently been shown effective for handling d 2 T log (ˆ ) → χ (p1 − p0 ). (A2) autocorrelation in EEG data [74]. Nonetheless, bootstrap tests
013145-18 ASSESSING THE SIGNIFICANCE OF DIRECTED AND … PHYSICAL REVIEW RESEARCH 3, 013145 (2021)
c are computationally inefficient and known to exhibit size and Eq. (B1), with w(t ) ∈ R , W (1) = 0.4Ic and unit variance power distortions, however typically less so than asymptotic innovation process. Each process, x, y, and w, are then in- tests [75]. As such, we consider a comparison to these empir- dependently filtered with either an FIR or an IIR filter. Both ical approaches outside the scope of our paper. filters were low-pass, with their cutoff set to a normalized frequency of π/2 radians and a variable filter order. 4. Drawing inferences In our study of the mutual information between bivariate processes, we inject a causal influence from the univariate ˆ Given an estimate and null distribution (e.g., the F-or process Y to X in order to test the TPR (i.e., statistical power χ 2 -distributions), we use the same general hypothesis testing of the test). In this scenario, we generate bivariate time-series χ 2 ˆ procedure for all measures. Here we use the -test for as data z from same state equations [Eq. (B1)] but with a small an example, however the same applies to all linear dependence ∗ causal influence from Y to X in the AR parameters: measures, including the -distributions, which are numeri- cally generated. X (1) XY (1) α (1) = , (B3) First, we set an arbitrary significance level , which is 0 Y (1) (ideally) the probability of rejecting the null hypothesis even = . = . =− . if it were true—this is set to 5% in this paper unless stated with XY (1) 0 03, whilst X (1) 0 3 and Y (1) 0 8, otherwise. Then the statistic T log (ˆ ) is input to the quan- as per Eq. (B2). 2 tile function of the χ -distribution with p1 − p0 degrees of freedom. The output (the complement of the p-value) is the 2. Human Connectome Project probability of measuring that value (or higher) under the null The HCP rsfMRI dataset [44] comprises 500 subjects im- hypothesis H : = 0. If the p-value is below the significance 0 aged at a 0.72 s sampling rate for 15 min in the (relatively level, then we deduce that the measured LR ˆ is significant. quiescent) resting state. This results in 1200 observations of A false positive occurs when the p-value is below the signif- spatially dense time-series data, which are then parcellated icance level α but the null hypothesis H is true, i.e., there 0 into 333 regions of interest in the brain. Thus, the dataset is no actual dependence between the variables = 0, yet ˆ contains 500 subjects with 333 brain regions each, and each is considered significant. Ideally, we expect the proportion of of these regions is associated with a stationary time series false positives (the FPR) to match the significance level α, of 1200 observations. The raw (BOLD) data of each region i.e., one would expect an FPR of 0.05 for a 5% significance was then preprocessed by removing the DC component, de- level α. This same procedure is used for all linear dependence trending, applying a third-order zero-phase (or forward and measures in this paper, with differing statistics and null distri- reverse) Butterworth bandpass filter (0.01–0.08 Hz). These butions. are common techniques used to remove potential artefacts. When the FPR is measured over R trials (usually R = 1 000 We also removed 200 observations from the start and the end in this paper), confidence intervals can be determined based of the time series, in order to minimize filter initialization on the binomial distribution of R draws of a random variable effects. This leaves T = 800 observations for the analysis. each with α = 0.05 chance of success. In order to build a scenario where the null hypothesis holds, we conduct experiments on 1000 time-series pairs, selecting APPENDIX B: EXPERIMENTAL SETUP different random regions of interest from different random 1. Numerical simulations subjects, making the corresponding time series completely independent of one another. The analysis was performed for For our experimental validation, we use an AR model sim- mutual information (between both univariate and bivariate ilar to the example proposed in [4], with two processes X and time series) and Granger causality in the same way as dis- Y that have no interdependence, digitally filtered to increase cussed for the simulated time-series experiments above. We their autocorrelation. We begin with the m-variate time-series use the same hypothesis testing procedure (discussed above) = ,..., × data z (z(1) z(T )), i.e., an m T matrix, where each with a 5% significance level for the χ 2-test and our newly realization is generated from a first-order vector AR model: proposed modified -test. z(t ) = (1)z(t − 1) + a(t ), (B1) APPENDIX C: ADDITIONAL PREWHITENING TESTS with x(t ) (1) 0 This Appendix provides further evidence that standard z(t ) = , (1) = X , (B2) y(t ) 0 (1) prewhitening techniques are insufficient for many covariance- Y stationary time series. In the main text, we show that and AR parameters X (1) = 0.3Ik and Y (1) =−0.8Il .The prewhitening time series is ineffective when the time-series innovations a(t ) = (a1(t ),...,am(t )) are uncorrelated with models are either AR(p) models that are inferred from Burg’s mean 0 and unit variance matrix var(a(t )) = Im. The matrix method or ARMA(p, q) models that are inferred from the z is then partitioned into k × T and l × T matrices denoted BIC score (up to a maximum order of p = q = 5). Here we x(t ) and y(t ). For each measure, we are interested in either the extend these results to show that AR(p) models inferred via mutual information between x and y, or the Granger causality the AICc (AIC with small-sample correction) and BIC scoring from y to x. If a third (conditional) process w is required functions are also insufficient. to contextualize these measures (in their conditional forms), In Fig. 13 we show the extended prewhitening results for we consider another first-order AR process w again using mutual information tests. The algorithm iterates through all
013145-19 OLIVER M. CLIFF et al. PHYSICAL REVIEW RESEARCH 3, 013145 (2021)
0.3 0.3
F -test, AR(1) F -test, AR(1) 0.2 F -test, AR(p) [AICc] 0.2 F -test, AR(p) [AICc] F -test, AR(p) [BIC] F -test, AR(p) [BIC] 1 1 0.1 0.1 False positive rate (FPR) False positive rate (FPR) 0 Actual FPR 0 0 Actual FPR 0 0 102030 01 0102030 01 Filter order Expected FPR Filter order Expected FPR (a) (a)
1 1
. . 0 8 F -test, AR(1) 0 8 F -test, AR(1) F -test, AR(p) [AICc] F -test, AR(p) [AICc] . . 0 6 F -test, AR(p) [BIC] 0 6 F -test, AR(p) [BIC] 1 1 0.4 0.4
0.2 0.2 False positive rate (FPR) False positive rate (FPR) Actual FPR 0 Actual FPR 0 0 0 0 102030 01 0102030 01 Filter order Expected FPR Filter order Expected FPR (b) (b)
FIG. 13. Inferring the AR models via AICc and BIC does not FIG. 14. Inferring the AR models via AICc and BIC does not consistently improve prewhitening results for mutual information improve prewhitening results for Granger causality tests with FIR tests with FIR (a) or IIR (a) filtering. The experiments are as per (a) or IIR (a) filtering. The experiments are as per Fig. 6, with two Fig. 1, with two additional prewhitening approaches used: the AR(p) additional prewhitening approaches used: the AR(p) model inferred model inferred from the AICc and BIC scores, respectively. from the AICc and BIC scores, respectively.
APPENDIX D: CONSIDERATIONS AND EXTENSIONS potential AR(p) models and selects the one that minimises OF BARTLETT’S FORMULA the AICc and BIC scores (independently). For the FIR-filtered data, this approach shows a slightly increased FPR above the In our derivations we use a first-order approximation of nominal value of 5%—this illustrates a marginal improvement Bartlett’s formula [Eq. (8)], that was originally described for over Burg’s method from Fig. 10. However, for the IIR-filtered spatially autocorrelated processes. However, since Bartlett’s data, the FPR approaches 100% and is significantly worse seminal work [10], there have been a number of other than even methods with no or minimal prewhitening [cf. extensions made to his formula as well as techniques intended the AR(1) models or the standard F-test in Fig. 1]. Simi- to overcome the issues of its empirical computation. larly, prewhitening is shown to be insufficient for Granger One of the more general cases of Bartlett’s formula is causality tests in Fig. 14, where equivalent experiments were due to Roy [40], who provided the large-sample distribution performed with no major qualitative differences (compared to between pairs of sample cross-correlations at differing lags. , , , Fig. 11). Consider the four processes Zi Z j Zk Zl .Let We do not show any further results for ARMA(p, q) mod- ∞ els, e.g., by increasing the maximum order or via a different v (i, j, k, l) = ρij(u)ρkl (u + v), (D1) criterion because the procedure to learn the parameters of u=−∞ higher-order ARMA models was too computationally ex- where ρ (u) are the cross-correlation as per Eq. (5), and pensive using the standard functions. Given this constraint, ij 1/2 all information criteria (AIC, AICc, or BIC) often chose sab(v) = T [rab(v) − ρab(v)], (D2) the maximum order in practice, and so using alternative approaches was redundant. We conjecture that prewhiten- as the standard error, with rab(v) the sample cross-correlation ing with ARMA(p, q) models may perform favorably to in Eq. (6). In general, the asymptotic distribution of the stan- AR(p) models given the ability to infer arbitrarily complex dard error, sab(v), is Gaussian with zero mean and covariance models. However, the constraints governed by the their in- lim cov(sab(v), sde(w)) ference procedures makes testing this currently intractable in T →∞ practice. ≈ w−v (a, d, b, e) + w+v (b, d, a, e)
013145-20 ASSESSING THE SIGNIFICANCE OF DIRECTED AND … PHYSICAL REVIEW RESEARCH 3, 013145 (2021)
− ρab(v)[ w (a, d, a, e) + w (b, d, b, e)] tionship between the large-sample approximations and the first-order approximations, we refer the reader to the dis- − ρ (w)[ (b, d, a, d) + (b, e, a, e)] de v v cussions in Refs. [11,38,78]. Bartlett did indeed present a + 1 ρ ρ , , , + , , , formula irrespective of sample size [11], which may yield an 2 ab(w) de(w)[ 0(a d a d) 0(a e a e) improvement for small sample distributions and give minor + , , , + , , , . 0(b d b d) 0(b e b e)] (D3) practical advantages, however, we did not find this necessary This derivation can be further generalized to the non-Gaussian for any experiments and instead follow the approximations in case, for instance, by allowing for skewed distributions [76]. Eq. (D5). More recently, a first-order approximation of this formula was Another potential source of error in the sampling distri- given by Afyouni et al. [2], which takes the same form as butions come from incorrectly estimating the autocorrelation Eq. (D3), with Eq. (D1) slightly modified. function raa(u). Tapering (also known as data windowing) is One consequence of knowing the full covariance structure commonly used in practice to regularize the autocorrelation [Eq. (D3)] is that such a distribution could further help in samples to better estimate their true value [2,51]. These ap- situations when the partial correlation terms that Wilks’ statis- proaches involve scaling the autocorrelation samples by some tic decomposes into are themselves correlated. That is, in the factor, with the maximum lag truncated below the dataset main text, we provided the variance for each ∗-distributed length. Using this method, we can appropriate Bartlett’s for- mula to variable Li (as a beta distribution by assuming independence). By assuming independence, we were able to obtain the sam- U T − u pling distribution of Wilks’ criterion as a product of these var(sab(0)) ≈ 1 + 2 λ(u)raa(u)rbb(u), (D6) ∗ T -distributions. However, if the random variables were cor- u=1 related, then we must use the multivariate form of Bartlett’s where λ(u) are a set of weights called the lag window and formula (D3), which provides the covariance of each variable U < T is the truncation point. The lag window comprises Li term with all other variables L j (i.e., we would have nij λ(u) values that decrease with increasing u; two common ap- for each i and j, rather than just ni). Accounting for this co- proaches are the Parzen and the Tukey windows (see Ref. [51] variance would require knowing the distribution of the general for details). Numerous truncation√ points√ U have also been product of correlated beta-distributions that, to the best of our proposed, e.g., T/4, T/5, T , and 2 T [2]. knowledge, is not an established result. In the above few sections we outlined a number of potential Another use case of Eq. (D3) is testing against an alterna- factors that could introduce small size or power distortions H ρ = tive hypothesis [ 1 : ab(0) 0], which is discussed at length in our hypothesis tests. To compare our approach with these by Afyouni et al. [2] for correlation coefficients. One could more complex extensions, we ran experiments with the effec- follow the same logic from this paper to provide the alterna- tive sample size computed from the full covariance matrix tive hypothesis test for linear dependence measures based on (D3), both with and without tapering. These were computed Wilks’ statistic. for the experiments from Fig. 1, however, as mentioned There are a number of special cases of Roy’s formula that above, the product of correlated beta- or F-distributed vari- are worth noting. In the event that we are interested in the ates is unknown and so our modified -test could not be , covariance cov(sab(v) sab(w)) between cross-correlation es- performed. Instead, we used the sums of z-transformed partial timates of two univariate processes Za and Zb at arbitrary lags correlations (each of which make up the conditional mutual = v and w, this is obtained from Eq. (D3) by setting d a and information term), rather than the products of squared partial = e b, reducing to the results reported in Refs. [77] and [7] correlations. That is, by transforming each partial correlation, (Theorem 11.2.3). Using this special case, the null distribu- we expect the sum of these correlation to be approximately tion of Pearson (zero-lag) correlation between two univariate Gaussian. In performing these tests, we found no notable = = processes var(sab(0)) can be obtained by setting v w difference in the FPRs for any of the validation experiments, ρ = 0. Finally, under the assumption that ab(0) 0, most of suggesting that these additions made no significant difference these terms disappear and we are left with Bartlett’s original towards reducing size or power distortions. formula [10]: lim var(s (0)) ≈ lim var(T 1/2 [r (0)]) APPENDIX E: PARTIAL AUTOCORRELATION →∞ ab →∞ ab T T AND ACTIVE INFORMATION STORAGE ∞ = ρii(u)ρ jj(u)(D4)The partial autocorrelation function conveys important u=−∞ information regarding the dependence structure of an AR process [7]. For a univariate stationary time series Z, the par- with a similar form given by a first-order approximation [38]: tial autocorrelation αZ (u)atlagu is the correlation between ∞ Z(t ) and Z(t − u), adjusted for the intervening observations −1 (u−1) = − ... − + u var(sab(0)) ≈ T (T −|u|)ρaa(u)ρbb(u). (D5) Z (t ) [Z(t 1); ; Z(t u 1)]. Denote Z as the (u) u=−∞ process of Z lagged by u time steps and Z as the history up until that lag (inclusive) Z(u) = [Z1 ; ... ; Zu]. Then, for a Due to symmetry of the autocorrelation function about lag- stationary time series, the partial autocorrelation function is zero for stationary processes, we can simply sum over the defined by [7] positive lags in Eq. (D5), u > 0, which was the form used throughout this paper [see Eq. (8)]. For the exact rela- αZ (1) = ρZZ1 (E1)
013145-21 OLIVER M. CLIFF et al. PHYSICAL REVIEW RESEARCH 3, 013145 (2021) and This same logic can be straightforwardly applied to other mea- sures such as excess entropy [41] and predictive information α (u) = ρ u (u−1) , u > 1. (E2) Z ZZ ·Z [42]. Although we use Burg’s method to identify the relevant his- In additional to quantifying the memory within a process, tory length p for an AR model of Z length for AR models in active information storage is often used for inferring the opti- this paper, it is common practice to use the partial autocorre- mal history length for both the Gaussian and non-Gaussian χ 2 lation function instead, since αZ (u) = 0foru > p [7,8,47]. cases [68]. This is typically achieved by using the -test Again, this is a statistical estimate and thus the order p is to infer the significance of increasing the embedding lengths inferred by testing each sample partial autocorrelationα ˆ Z (u) p. For AR processes with Gaussian innovations, we infer for significance against the null distribution. the embedding length p for X by first taking the difference Intriguingly, our work reveals a relationship between the δx (u) = Ax (u + 1) − AX (u) and then generating a p-value by 2 partial autocorrelation function and active information storage testing 2 δX (u)againstaχ (1) distribution, which represents [43]—a recently developed model-free measure for quan- the null hypothesis of no increase in information storage. If tifying memory in a process—under the linear Gaussian the p-value is below a threshold (say, 5%), then the test is = + assumption. The average active information storage AX quan- rejected and the lag is increased u u 1. This process is tifies the information storage in a process. For a p-order iterated until the null hypothesis is accepted, at which point we Markov process X, this is quantified by the mutual informa- surmise that the optimal lag p is the one at which δX (p + 1) is tion between the relevant history X (p)(t ) and variable X (t ): considered insignificant. This approach is similar to using the partial autocorrelation, as the difference δX (u) is equivalent to A = I . squared partial autocorrelation up to a factor of two. This can X (p) X;X (p) (E3) be seen from Eq. (E4): Since the average active information storage is a specific type δ (u) = A (u + 1) − A (u) of mutual information, we can use the chain rule to decompose X X X =−1 { − α + 2}. it into a sum of squared partial autocorrelations: 2 log 1 [ X (u 1)] (E5) p In contrast to measures of dependence between multiple pro- I =− / − ρ2 2 X;X (p) 1 2 log 1 XXu·X (u−1) cesses, the χ -test appears suitable here (without adjusting = u 1 for an effective sample size) for testing δX (u)foru > p.This p is because, after the full set of past variables is included in 2 u u (p) =−1/2 log 1 − [αX (u)] . (E4) the regression, any higher order residuals x − xˆ (ˆx ) with u=1 u > p have statistically zero autocorrelation for every lag.
[1] C. E. Davey, D. B. Grayden, G. F. Egan, and L. A. Johnston, [11] M. S. Bartlett, On the theoretical specification and sampling Filtering induces correlation in fMRI resting state data, properties of autocorrelated time-series, Suppl. J. R. Stat. Soc. NeuroImage 64, 728 (2013). 8, 27 (1946). [2] S. Afyouni, S. M. Smith, and T. E. Nichols, Effective degrees [12] J. D. Cryer and K.-S. Chan, Time Series Analysis: With Appli- of freedom of the Pearson’s correlation coefficient under auto- cations in R (Springer Science & Business Media, New York, correlation, NeuroImage 199, 609 (2019). 2008). [3] E. Florin, J. Gross, J. Pfeifer, G. R. Fink, and L. Timmermann, [13] D. Sul, P. C. Phillips, and C.-Y. Choi, Prewhitening bias in HAC The effect of filtering on Granger causality based multivariate estimation, Oxford Bull. Econ. Stat. 67, 517 (2005). causality measures, NeuroImage 50, 577 (2010). [14] M. Bayazit and B. Önöz, To prewhiten or not to prewhiten in [4] L. Barnett and A. K. Seth, Behaviour of Granger causality trend analysis? Hydrol. Sci. J. 52, 611 (2007). under filtering: Theoretical invariance and practical application, [15] W. Olszowy, J. Aston, C. Rua, and G. B. Williams, Accurate au- J. Neurosci. Methods 201, 404 (2011). tocorrelation modeling substantially improves fMRI reliability, [5] A. K. Seth, A MATLAB toolbox for Granger causal connectiv- Nat. Commun. 10, 1220 (2019). ity analysis, J. Neurosci. Methods 186, 262 (2010). [16] S. Yue and C. Y. Wang, Applicability of prewhitening to elim- [6] P. M. Robinson, Generalized canonical analysis for time series, inate the influence of serial correlation on the Mann-Kendall J. Multivariate Anal. 3, 141 (1973). test, Water Resour. Res. 38, 4 (2002). [7] P. J. Brockwell and R. A. Davis, Time Series: Theory and [17] M. R. Arbabshirani, E. Damaraju, R. Phlypo, S. Plis, E. Allen, Methods (Springer Science & Business Media, 1991). S. Ma, D. Mathalon, A. Preda, J. G. Vaidya, T. Adali, and V. D. [8] G. C. Reinsel, Elements of Multivariate Time Series Analysis Calhoun, Impact of autocorrelation on functional connectivity, (Springer Science & Business Media, New York, 2003). NeuroImage 102, 294 (2014). [9] G. U. Yule, Why do we sometimes get nonsense-correlations [18] J. R. Bence, Analysis of short time series: Correcting for auto- between time-series?—A study in sampling and the nature of correlation, Ecology 76, 628 (1995). time-series, J. R. Stat. Soc. 89, 1 (1926). [19] M. Macias-Fauria, A. Grinsted, S. Helama, and J. Holopainen, [10] M. S. Bartlett, Some aspects of the time-correlation problem Persistence matters: Estimation of the statistical significance of in regard to tests of significance, J. R. Stat. Soc. 98, 536 paleoclimatic reconstruction statistics from autocorrelated time (1935). series, Dendrochronologia 30, 179 (2012).
013145-22 ASSESSING THE SIGNIFICANCE OF DIRECTED AND … PHYSICAL REVIEW RESEARCH 3, 013145 (2021)
[20] K. J. Friston, Functional and effective connectivity: A review, [43] J. T. Lizier, M. Prokopenko, and A. Y. Zomaya, Local measures Brain Connect. 1, 13 (2011). of information storage in complex distributed computation, Inf. [21] C. W. J. Granger, Investigating causal relations by economet- Sci. 208, 39 (2012). ric models and cross-spectral methods, Econometrica 37, 424 [44] D. C. Van Essen, K. Ugurbil, E. Auerbach, D. Barch, T. (1969). Behrens, R. Bucholz, A. Chang, L. Chen, M. Corbetta, S. W. [22] A. Roebroeck, E. Formisano, and R. Goebel, Mapping directed Curtiss et al., The Human Connectome Project: A data acquisi- influence over the brain using Granger causality and fMRI, tion perspective, NeuroImage 62, 2222 (2012). NeuroImage 25, 230 (2005). [45] https://github.com/olivercliff/exact-linear-dependence. [23] W. Liao, J. Ding, D. Marinazzo, Q. Xu, Z. Wang, C. Yuan, [46] J. Geweke, Measurement of linear dependence and feedback Z. Zhang, G. Lu, and H. Chen, Small-world directed networks between multiple time series, J. Am. Stat. Assoc. 77, 304 in the human brain: Multivariate Granger causality analysis of (1982). resting-state fMRI, NeuroImage 54, 2683 (2011). [47] G. E. Box, G. M. Jenkins, G. C. Reinsel, and G. M. Ljung, Time [24] K. Friston, R. Moran, and A. K. Seth, Analysing connectivity Series Analysis: Forecasting and Control (John Wiley & Sons, with Granger causality and dynamic causal modelling, Curr. San Francisco, 2015). Opin. Neurobiol. 23, 172 (2013). [48] The most general form is the ARIMA model, where the integra- [25] R. K. Kaufmann and D. I. Stern, Evidence for human influ- tive (I) component accounts for nonstationary time series. Here ence on climate from hemispheric temperature relations, Nature we assume that appropriate differencing and mean removal have (London) 388, 39 (1997). already been performed in order to remove any integration of [26] D. D. Zhang, H. F. Lee, C. Wang, B. Li, Q. Pei, J. Zhang, and the time series. Y. An, The causality analysis of climate change and large-scale [49] The problem of recovering the autocorrelation functions from human crisis, Proc. Natl. Acad. Sci. USA 108, 17296 (2011). data, and thus the effective sample size, is nontrivial and has re- [27] J. R. Freeman, Granger causality and the times series analysis sulted in procedures such as tapering to handle noisy estimates of political relationships, Am. J. Polit. Sci. 27, 327 (1983). [2,51]; these notions are covered briefly in Appendix D. [28] R. Reuveny and H. Kang, International trade, political con- [50] M. J. Anderson and J. Robinson, Permutation tests for linear flict/cooperation, and Granger causality, Am. J. Polit. Sci. 40, models, Aust. NZ J. Stat. 43, 75 (2001). 943 (1996). [51] C. Chatfield and H. Xing, The Analysis of Time Series: An [29] S. S. Wilks, Certain generalizations in the analysis of variance, Introduction with R, 7th ed. (Chapman and Hall/CRC, Boca Biometrika 24, 471 (1932). Raton, 2019). [30] T. Pham-Gia, Exact distribution of the generalized Wilks’s [52] A. M. Mathai and P. N. Rathie, The exact distribution of Wilks’ statistic and applications, J. Multivariate Anal. 99, 1698 (2008). criterion, Ann. Math. Stat. 42, 1010 (1971). [31] H. Hotelling, Relations between two sets of variates, [53] D. R. Brillinger, Some data analyses using mutual information, Biometrika 28, 321 (1936). Braz. J. Probab. Stat. 18, 163 (2004). [32] D. J. MacKay, Information Theory, Inference and Learning Al- [54] L. Barnett and T. Bossomaier, Transfer Entropy as a Log- gorithms (Cambridge University Press, Cambridge, UK, 2003). Likelihood Ratio, Phys.Rev.Lett.109, 138105 (2012). [33] T. M. Cover and J. A. Thomas, Elements of Information Theory [55] S. S. Wilks, The large-sample distribution of the likelihood (John Wiley & Sons, New York, 2012). ratio for testing composite hypotheses, Ann. Math. Stat. 9,60 [34] J. Kay, Feature discovery under contextual supervision using (1938). mutual information, in Proceedings of IEEE IJCNN, (1992), [56] C. E. Davey, D. B. Grayden, M. Gavrilescu, G. F. Egan, and Vol. 4, pp. 7984. L. A. Johnston, The equivalence of linear Gaussian connectivity [35] L. Barnett, A. B. Barrett, and A. K. Seth, Granger Causality and techniques, Human Brain Mapping 34, 1999 (2013). Transfer Entropy are Equivalent for Gaussian Variables, Phys. [57] J. T. Lizier and M. Prokopenko, Differentiating information Rev. Lett. 103, 238701 (2009). transfer and causal effect, Eur.Phys.J.B73, 605 (2010). [36] T. Schreiber, Measuring Information Transfer, Phys. Rev. Lett. [58] M. De Hoon, T. Van der Hagen, H. Schoonewelle, and H. Van 85, 461 (2000). Dam, Why Yule-Walker should not be used for autoregressive [37] T. Bossomaier, L. Barnett, M. Harré, and J. T. Lizier, An In- modelling, Ann. Nucl. Energy 23, 1219 (1996). troduction to Transfer Entropy: Information Flow in Complex [59] J. Garland, R. G. James, and E. Bradley, Leveraging in- Systems (Springer, Cham, Switzerland, 2016). formation storage to select forecast-optimal parameters for [38] G. Bayley and J. Hammersley, The “effective” number of inde- delay-coordinate reconstructions, Phys. Rev. E 93, 022221 pendent observations in an autocorrelated time series, Suppl. J. (2016). R. Stat. Soc. 8, 184 (1946). [60] In general, the variables representing the past need not be a [39] P. Clifford, S. Richardson, and D. Hémon, Assessing the sig- sequence of consecutive temporal indices, nor have the same nificance of the correlation between two spatial processes, history length p for each dimension of X. For instance, one Biometrics 45, 123 (1989). could use a Takens embedding [79,80] or any other statistically [40] R. Roy, Asymptotic covariance structure of serial correlations significant set of variables [81]. However, for simplicity, in this in multivariate time series, Biometrika 76, 824 (1989). work we follow the vector AR model often used in Granger [41] W. Bialek, I. Nemenman, and N. Tishby, Complexity through causality analysis [Eq. (B1)] and thus use a consecutive se- nonextensivity, Phys. A 302, 89 (2001). quence of variables as the history of X (t ). [42] J. P. Crutchfield and D. P. Feldman, Regularities unseen, ran- [61] J. Wishart, The generalised product moment distribution in domness observed: Levels of entropy convergence, Chaos 13, samples from a normal multivariate population, Biometrika 25 (2003). 20A, 32 (1928).
013145-23 OLIVER M. CLIFF et al. PHYSICAL REVIEW RESEARCH 3, 013145 (2021)
[62] Using the MATLAB estimate function in order to infer the [72] C. Francq and J.-M. Zakoïan, Bartlett’s formula for a general parameters for each p and q; see our open-source toolkit [45] class of nonlinear processes, J. Time Ser. Anal. 30, 449 (2009). for details. [73] T. Schreiber and A. Schmitz, Improved Surrogate Data for [63] R. Franciotti, N. Falasca, L. Bonanni, F. Anzellotti, V. Maruotti, Nonlinearity Tests, Phys.Rev.Lett.77, 635 (1996). S. Comani, A. Thomas, A. Tartaro, J. Taylor, and M. Onofrj, [74] N. Schaworonkow, D. A. Blythe, J. Kegeles, G. Curio, and Default network is not hypoactive in dementia with fluctuating V. V. Nikulin, Power-law dynamics in neuronal and behavioral cognition: An Alzheimer disease-dementia with Lewy bodies data introduce spurious correlations, Human Brain Mapping 36, comparison., Neurobiol. Aging 34, 1148 (2013). 2901 (2015). [64] L. Barnett and A. K. Seth, The MVGC multivariate Granger [75] R. Davidson and J. G. MacKinnon, The size distortion of boot- causality toolbox: A new approach to Granger-causal inference, strap tests, Econ. Theory 15, 361 (1999). J. Neurosci. Methods 223, 50 (2014). [76] N. Su and R. Lund, Multivariate versions of Bartlett’s formula, [65] H. Boudjellaba, J.-M. Dufour, and R. Roy, Testing causal- J. Multivariate Anal. 105, 18 (2012). ity between two vectors in multivariate autoregressive moving [77] M. S. Bartlett, An Introduction to Stochastic Processes: With average models, J. Am. Stat. Assoc. 87, 1082 (1992). Special Reference to Methods and Applications,3rded. [66] L. Barnett and A. K. Seth, Granger causality for state-space (Cambridge University Press, Cambridge, 1978). models, Phys.Rev.E91, 040101(R) (2015). [78] M. H. Quenouille, Notes on the calculation of autocorrelations [67] A. Gutknecht and L. Barnett, Sampling distribution for single- of linear autoregressive schemes, Biometrika 34, 365 (1947). regression Granger causality estimators, arXiv:1911.09625 [79] F. Takens, Detecting strange attractors in turbulence, in Dynam- (2019). ical Systems and Turbulence, Lecture Notes in Mathematics [68] J. T. Lizier, JIDT: An information-theoretic toolkit for studying edited by D. Rand and L.-S. Young (Springer, Berlin, 1981), the dynamics of complex systems, Front. Robotics AI 1,11 Vol. 898, pp. 366–381. (2014). [80] O. M. Cliff, M. Prokopenko, and R. Fitch, Minimising the [69] J. Theiler, Spurious dimension from correlation algorithms Kullback–Leibler divergence for model selection in distributed applied to limited time-series data, Phys. Rev. A 34, 2427 nonlinear systems, Entropy 20, 51 (2018). (1986). [81] L. Novelli, P. Wollstadt, P. Mediano, M. Wibral, and J. T. Lizier, [70] See the Supporting Information of Novelli et al. [81]forasim- Large-scale directed network inference with multivariate trans- ilar experiment with transfer entropy on the same HCP rsfMRI fer entropy and hierarchical statistical testing, Netw. Neurosci. dataset used in our Fig. 12. 3, 827 (2019). [71] Weber et al. [82] report a contrasting finding of increased FPR [82] I. Weber, E. Florin, M. Von Papen, and L. Timmermann, The in transfer entropy inference due to filtering; however, the use influence of filtering and downsampling on the estimation of of a Theiler window is not specified. transfer entropy, PLoS ONE 12, e0188210 (2017).
013145-24