sign in sign up
<<
Home , Dynamical systems theory

Dynamical Theory for Causal Inference with Application to Synthetic Control Methods

Yi Ding Panos Toulis University of Chicago University of Chicago Department of Booth School of Business

Abstract

In this paper, we adopt results in nonlinear series analysis for causal inference in dynamical settings. Our motivation is policy analysis with panel data, particularly through the use of “synthetic control” methods. These methods regress pre-intervention outcomes of the treated unit to outcomes from a pool of control units, and then use the fitted regres- sion model to estimate causal effects post- intervention. In this setting, we propose to screen out control units that have a weak dy- Figure 1: A Lorenz attractor plotted in 3D. namical relationship to the treated unit. In , we show that this method can mitigate bias from “cherry-picking” of control However, in many real-world settings, different vari- units, which is usually an important concern. ables exhibit dynamic interdependence, sometimes We illustrate on real-world applications, in- showing positive correlation and sometimes negative. cluding the tobacco legislation example of Such ephemeral correlations can be illustrated with Abadie et al.(2010), and Brexit. a popular dynamical shown in Figure1, the (Lorenz, 1963). The trajectory resem- bles a butterfly shape indicating varying correlations 1 Introduction at different : in one wing of the shape, variables X and Y appear to be positively correlated, and in the In causal inference, we compare outcomes of units who other they are negatively correlated. Such dynamics received the treatment with outcomes from units who present new methodological challenges for causal in- did not. A key assumption, often made implicitly, is ference that have not been addressed. In relation to that the relationships of interest are static and invariant. the schooling example, our analysis of schooling effect For example, in studying the effects of schooling on later on earnings could occur on one wing of the system, earnings, we usually consider potential outcomes Yi(k), where the correlation is, say, positive. However, crucial for some student i had the student received k years policy decisions, such as college subsidies, could occur of schooling. Since only one potential outcome can be on the other wing where the relationship is reversed. observed for each student, causal inference needs to rely Such discord between data analysis and policy making on comparisons between students who received varying is clearly detrimental to policy effectiveness. years of schooling. The validity of the results there- Despite its longstanding importance in many scien- fore rests upon the assumption that the relationship tific fields, dynamical has not found between years of schooling and earnings is temporally a way into modern causal inference (Durlauf, 2005). static and unidirectional. The main goal of this paper is to leverage key results from dynamical systems to guide causal inference in Proceedings of the 23rdInternational Conference on Artificial the presence of dynamics. For concreteness, we focus Intelligence and (AISTATS) 2020, Palermo, Italy. on synthetic control methods (Abadie et al., 2010), PMLR: Volume 108. Copyright 2020 by the author(s). which are popular for policy analysis with panel data. Dynamical Systems Theory for Causal Inference with Application to Synthetic Control Methods

Our methods, however, can more generally be applied fields of policy analysis (Abadie et al., 2010; Kreif et al., when causal inference involves some form of matching 2016; Shaikh and Toulis, 2019), criminology (Saunders between treated and control units in the time domain. et al., 2015), politics (Abadie and Gardeazabal, 2003; Abadie et al., 2015), and (Billmeier and 2 Preliminaries Nannicini, 2013). Theoretically, the treatment effect estimator, τˆ1t, Here, we give a brief overview of comparative case is asymptotically unbiased as the number of pre- studies with panel data to fix concepts and notation. intervention periods grows when the outcome model is Later, in Section3, we describe our method. linear in (possibly unobserved) factors and the treated unit “lives” in the convex hull of the controls (Abadie Following standard notation, we consider J + 1 units, et al., 2010, Theorem 1). As such, a key assumption of with only one unit being treated. Let Y N be the it model continuity is implicitly made for identification, potential outcome for unit i at time t in a hypothetical where the weights w are assumed to be time-invariant. world where the intervention did not occur (denoted by j Furthermore, control selection in synthetic controls de- the exponent “N”), where i = 1, 2,...,J + 1, and t = pends only on the statistical fit between treated and 1, 2,...,T ; also let Y I be the corresponding potential it control outcomes in the pre-intervention period, which outcome assuming the intervention did occur. Let D it opens up the possibility of cherry-picking controls to be a binary indicator of whether unit i is treated at bias causal inference. In the following section, we illus- time t. By convention, and without loss of generality, trate these problems with an example. only unit 1 receives the treatment, and there exists T0 ∈ (0,T ) such that 2.1 Motivation: an Adversarial Attack to the Synthetic Control Method Dit = 1, if and only if i = 1 and t > T0.

The observed outcome for unit i at time t, denoted by As a motivation, we use the example of California’s tobacco control program in 1989, as described in the Yit, therefore satisfies: original paper of synthetic controls (Abadie et al., 2010). N I N The goal is to estimate the effect of Proposition 99, Yit = Yit + (Yit − Yit )Dit, (1) a large-scale tobacco control program passed by elec- I N torate majority in 1988 in California. The proposition where τit = Yit − Yit is the causal effect of interven- tion on unit i at time t. Suppose there exist weights took effect in 1989 through a sizeable tax hike per PJ+1 cigarette packet. The panel data include annual state w2, . . . , wJ+1 such that j=2 wj = 1 and wj ≥ 0, and N PJ+1 per-capita cigarette sales from 1970 to 2000 as outcome, Y1t = j=2 wjYjt, for t = 1,...,T0. Then, the causal effect of the intervention can be estimated through: along with related predictors, such as state median in- come and %youth population. We have a pool with 38 J+1 states as potential controls, after discarding states that X τˆ1t = Y1t − wjYjt, for every t > T0. (2) adopted similar programs during the 1980’s. j=2 The synthetic control methodology proceeds by cal- culating a weighted combination of control unit out- The time series defined with the term PJ+1 w Y in j=2 j jt comes to fit cigarette sales of California, using only Equation (2) is the synthetic control. This synthetic pre-1989 data. In this application, the weighted combi- control unit can be construed to be representative of nation is: Colorado (0.164), Connecticut (0.069), Mon- the treated unit (i = 1) had the treated unit not re- tana (0.199), Nevada (0.234), and Utah (0.334), where ceived treatment. Because of the constraints put on the numbers in the parentheses are the corresponding w , namely that they are nonnegative and sum to one, j weights. The implied model is the following: the fitted values of the weights reside on the edges of a polytope, and so many weights are to 0. Such YbCA,t = 0.164 × YCO,t + 0.069 × YCT,t (3) sparsity in the weights corresponds to control selection, + 0.199 × Y + 0.234 × Y + 0.334 × Y , and so only a few control units are used to model the MO,t NV,t UT,t outcomes of the treated unit. where Y denotes packet sales at a particular state and time (a state is denoted by a two-letter code; e.g., CA The synthetic control methodology is an important ex- stands for California). We note that time t in the model ample of comparative case studies (Angrist and Pischke, of Equation (3) is before intervention (t ≤ 1989), so 2008; Card and Krueger, 1994), and generalizes other that all states in the data, including California, are in well-known methods, such as “difference-in-differences”. control for the entire period considered in the model. As a methodology it is simple and transparent, and so synthetic controls have become widely popular in the The idea for causal inference through this approach is Yi Ding, Panos Toulis

that the same model in Equation (3) can be used to esti- idea is that state cigarette consumption data evolve on mate the counterfactual outcomes for California, YCA,t, the same attractor, whereas adversarial time series do for t > 1989, had California not been treated with the not. Thus, the latter should be less dynamically related tax hike in 1989. By comparing the post-intervention to the treated state, and so they should be removed data from actual California that was treated with the from the control pool. tobacco control program in 1989, and predictions for We note that our proposed method differs from recent synthetic California that hypothetically stayed in con- work in synthetic controls, which has mainly focused trol in 1989, we can estimate that per-capita cigarette on high-dimensional, matrix completion, or de-biasing sales reduced by 19 packs on average by Proposition 99, methods (Amjad et al., 2018; Ben-Michael et al., 2018; suggesting a positive causal effect. This is illustrated in Athey et al., 2018; Hazlett and Xu, 2018). These meth- the left figure of Figure2. As mentioned earlier, an im- ods take a regression model-based approach, whereas plicit assumption here is that of model continuity: we we treat panel data as a nonlinear . assume that the same model that fits pre-intervention More broadly, our approach shows that dynamical sys- California can be used to predict the counterfactual tems theory can be integrated into statistical frame- outcomes of a post-intervention, non-treated California. works of causal inference, a goal that so far has re- This model continuity assumption relies critically on mained elusive (Rosser, 1999; Durlauf, 2005). the choice of control units in the model of Equation (3). Currently, this choice relies mostly on the subject- 3 Methods matter expert, which leaves open opportunities for cherry-picking in constructing the control pool. To In this section, we describe our proposed method. In illustrate this problem we can perform the following Section 3.1, we present the method of convergent cross manipulation. First, we add 9 unemployment-related 1 0 0 mapping (CCM), which is the fundamental building time series , namely YAD1,t,...,YAD9,t, into the pool of block of our method. In Section 3.2, we motivate CCM potential controls, where “AD” stands for “adversarial”. through a theoretical analysis on a simple, non-trivial Second, before adding these units to the control pool we 0 time-series model. Finally, in Section 3.3, we give transform the time series as follows: YADi,t = YADi,t − details on our proposed method. 50 + 90 · I{t ≤ 1989}. This transformation ensures that the adversarial time series has similar scale to time 3.1 Convergent Cross Mapping (CCM) series on cigarette consumption before treatment. Since the synthetic control method only relies on statistical The basis of our approach is to consider the available fit, it may pick up the artificial time series from the panel data as a dynamical system. In particular, the new control pool. Indeed, the new synthetic California state of the system at time t is the collection of all time is now described by the following model: series, (Y1t,Y2t,...,Y(J+1)t), where J is the number of controls. Taken across all possible t, this implies a YbCA,t = 0.247 × YCO,t + 0.179 × YCT,t (4) , known as the , denoted by M = + 0.196 × YNV,t + 0.06 × YNH,t + 0.011 × YWY,t {(Yjt : j ∈ 1,...,J + 1) : t ∈ [0,T ]}, where T denotes + 0.3 × YAD∗,t, the length of time series, and is fixed. For example, when J = 1 there are two units in total and M is a where Y represents that the adversarial time series AD∗,t curve (possibly self-intersecting) on the plane. was selected — the specific index is irrelevant. This produces a new synthetic California that is drastically A seminal result in nonlinear dynamics is Takens’ theo- different than before. The weight on the artificial con- rem (Takens, 1981), which shows that the phase space trol unit is in fact the highest compared to the weights of a dynamical system can be reconstructed through on all other units, which is clearly undesirable. More time-delayed observations from the system. Specifically, importantly, with the new synthetic control Califor- let us define a delay-coordinate embedding of the form nia, we estimate a negative causal effect of 8 packs on average (see right sub-figure in Figure2). Yejt = [Yjt,Yj(t−τ),...,Yj(t−(d−1)τ)], (5) To address this problem, our paper leverages funda- where τ > 0 is the time delay. The key theoretical re- mental results in dynamical systems theory, such as sult of Takens(1981) is that the manifold, Mfj, defined time-delay embeddings. The goal is to pre-screen con- from outcomes {Yejt} is diffeomorphic (i.e., the mapping trol units based on how strongly related they are to is differentiable, invertible, and smooth) to the original treated units from a dynamical of view. The key manifold M, meaning that some important topological information is preserved, such as invariance to coordi- 1Data from the Local Area Unemployment Statistics (LAUS) program of the Bureau of Labor Statistics (Bu- nate changes. In other words, Mfj is a reconstruction reau of Labor Statistics, 2018). See Supplement for details. of M. It follows that different reconstructions Mfj, for Dynamical Systems Theory for Causal Inference with Application to Synthetic Control Methods

California California 160 synthetic California 160 synthetic California 140 140 120 120 100 100 80 80 60 60 per−capita cigarette sales (in packs) per−capita cigarette sales (in packs) 40 40

1970 1975 1980 1985 1990 1995 2000 1970 1975 1980 1985 1990 1995 2000

year year

Figure 2: Trends in per-capita cigarette sales. The solid line is actual California and the dashed line is synthetic California, while the vertical line indicates time of intervention. Left: original setting. Right: adversarial setting where the synthetic California is created according to Equation (4). various j, are diffeomorphic to each other, including the variable “drives” the other. original manifold M, which implies cross-predictability. Such prediction proceeds in discrete steps as follows. For two different reconstructions Mfj and Mfj0 , with First, we build a nonparametric model of Xt using their corresponding base time series Yjt and Yj0t, we the reconstruction manifold based on Yt. For a given could use Mfj to predict Yj0t and use Mfj0 to predict time point t we pick the (d + 1)-nearest neighbors from Yj. By measuring this cross-predictability, the relative t ∈ {1 + (d − 1)τ, 1 + dτ, . . . , L} in Yet, where L < T is strength of dynamical relationship between any two called the library size, and denote their time indices variables in the system can be quantified (Schiff et al., (from closest to farthest) as {t , ··· , t }. A linear 1996; Arnhold et al., 1999). 1 d+1 model for Xt is as follows: One recent method utilizing this idea is convergent cross Xd+1 mapping (Sugihara et al., 2012, CCM). In addition to Xbt = w(ti, t)Xti , (7) i=1 the idea of cross-predictability, CCM also relies on a smoothness implication of Takens’ theorem, whereby where w(ti, t) is the weight based on the Euclidean dis- neighboring points in the reconstructed manifold are tance between Yet and its i-th nearest neighbor on Yeti , P close to neighboring points in the original manifold. for example, w(ti, t) = exp(−di/d1)/ j exp(−dj/d1) This suggests that cross predictability will increase and and di = ||Yet − Yeti ||, with the usual L2 norm. The dif- stabilize as the number of data points grow. The cross ference defined by mean absolute error (MAE) between predictability is quantified by a CCM score, which we Xt and Xbt across t is defined as the CCM score of Yt will address later. on Xt (lower is better): Operationally, the generic CCM algorithm considers CCM(Xt | Yt) = |Xt − Xbt|, (8) two time series, say Xt and Yt, and their corresponding delay-coordinate embedding vectors at time t, namely CCM(Yt | Xt) = |Yt − Ybt|. X = [X ,X ,...,X ], (6) et t (t−τ) (t−dτ+τ) Intuitively, the CCM score captures how much informa- Yet = [Yt,Y(t−τ),...,Y(t−dτ+τ)], tion is in Yt about Xt. For instance, if Xt dynamically drives Yt we expect Xbt to be close to Xt. Similarly, where d is known as the embedding dimension and CCM(Yt | Xt) is obtained by repeating the above pro- t ∈ {1 + (d − 1)τ, 1 + dτ, . . . , T }. The manifold based cedure symmetrically, using MX of the values from the on the phase space of Yet is denoted by MY , and the delay embedding Xet of Xt. The value of CCM(Yt | Xt) manifold based on Xet is denoted by MX , where the quantifies the information in Xt about Yt. The two manifold definitions follow from Equation (5). The CCM scores jointly quantify the dynamic coupling be- idea is that these are diffeomorphic to the tween the two variables. As mentioned earlier, Takens’ original manifold of the dynamical system of Y and X. theorem implies that there exists a one-to-one mapping A manifold from the delay embedding of one variable such that the nearest neighbors of Yt identify the corre- can be used to predict the other variable, and the sponding time indices of nearest neighbors of Xt, if Xt quality of this prediction is an indication of which and Yt are dynamically related. As the library size, L, Yi Ding, Panos Toulis

increases, the reconstruction manifolds MX and MY Assumption 1. Fix t and let L → ∞. Suppose that: become denser and the distances between the nearest 0 (a) mini=1,...,d+1 min{ti, ti} → ∞; neighbors shrink, and so the CCM scores will converge; 1 (b) for both crossmaps, maxi |wY (ti, t) − | → 0, see Sugihara and May(1990); Casdagli et al.(1991) for d+1 and max |w (t0 , t) − 1 | → 0; more details. i X i d+1 0 0 (c) mini6=j |ti − tj| → ∞ and mini6=j |ti − tj| → ∞. From a statistical perspective, the CCM method is a form of nonparametric time-series estimation (Här- Remarks. Assumption 1(a) requires some form of dle et al., 1997). The unique feature of CCM, and smoothness for the delay-coordinate system, and is mild. more generally of delay embedding methods, is that Assumption 1(b) is similar to stationarity as it implies the nonparametric components are in fact the time exchangeability within the sets {ti : i = 1, . . . , d + 1} 0 indices t1, . . . , td+1 in Equation (7). This differs from, and {ti : i = 1, . . . , d + 1}. Assumption 1(c) may say, kernel smoothing (Hastie et al., 2001), where the be strict. It could fail, for instance, when the order statistics (e.g., {Y }) are periodic. target point Xt is fitted by neighboring observations eti to smooth estimation. Importantly, CCM is not in Theorem 1. Suppose that Assumptions1(a)-(c) hold competition with Granger causality (Granger, 1969), for the CCM scores of the autoregressive model in Equa- but rather complements it. The key problem with tion (9), then Granger causality is that it requires “separability” of  2t  the effects from different causal factors. This condition d µ t 2 − α 2 CCM(Xt | Yt) −→ FN (X0 − )α , σX , generally does not hold in real-world dynamical sys- 1 − α 1 − α2 tems that exhibit so called “weak coupling”. CCM is  µ CCM(Y | X ) −→d FN β(X − )αt−1, unique because it can work in such systems (Sugihara t t 0 1 − α et al., 2012). Finally, CCM is backed up by a growing 2 − α2t  literature in the physical as it is tailored to β2σ2 + 2σ2 , 1 − α2 X Y dynamical complex systems (Runge et al., 2019). where FN(µ, σ2) = |N(µ, σ2)| is the folded normal dis- 3.2 Theory of CCM on Autoregressive Model tribution with mean µ and variance σ2.

In this section, we illustrate CCM through an AR(1) Remarks. To unpack this theoretical result, we make autoregression model. Of course, AR(1) is a simple the following remarks: model that most certainly does not capture the details (a) When β = 0, the dependence of Y on X is entirely of real-world time series. However, its simplicity allows t t lost, and so X and Y evolve independently imply- us to do two things. First, we derive analytic formulas t t ing that there is no driving factor in the system. for the CCM scores in Equation (8). Due to CCM’s CCM captures this relationship, since CCM(X | nonlinear , such formulas are not easily attain- t Y ) = O (σ ) and CCM(Y | X ) = O (σ ), with able, in general. In fact, we are unaware of any other t P X t t P Y the two scores being independent (this is shown in analytic expressions for CCM in the literature, so our the proof of the theorem in the Supplement). work here makes a broader contribution. Second, we can compare the CCM formulas with the parameters of (b) When β is small or moderately large, Yt is weakly the AR(1) model to better understand CCM as a tool; dependent on Xt. On average, we expect to see that this is only possible because AR(1) is simple enough CCM(Yt | Xt) > CCM(Xt | Yt). CCM analysis that the strength of causal relationships between vari- indicates correctly that the driving factor in the ables is discernible from the model parameters. system is Xt and not Yt (recall that we are using the absolute error-CCM, and so smaller values are The outcome model we consider is as follows: better).

Xt = αXt−1 + µ + t,Yt = βXt−1 + µ + ζt, (9) (c) When β is very large we could sometimes have CCM(Y | X ) < CCM(X | Y ), which leads to the where α, β are fixed, with |α| < 1, β ≥ 0, µ is the drift, t t t t wrong “causal direction”. This shows some inherent  ∼ N(0, σ2 ) and ζ ∼ N(0, σ2 ) are zero-mean and t X t Y limitations of CCM, as it depends to some extent constant-variance normal errors, with σ > σ . In this Y X on predictive ability, and so it can fail in similar joint dynamical system of X and Y , it is evident that ways as Granger causality. X generally drives Y since X evolves independently of Y , whereas the evolution of Y depends on X. We In conclusion, Theorem1 is a new connection of statis- are interested in knowing how CCM quantifies this tics and nonlinear dynamics. As mentioned earlier, the asymmetric dynamic relationship between X and Y , goal is not to analyze AR(1) per se, which indeed is a and whether it captures the dependence on parameters simple model, but to understand CCM’s causal predic- 2 2 α, β, µ, σX , σY . tions by comparing to AR(1) coefficients. The theorem Dynamical Systems Theory for Causal Inference with Application to Synthetic Control Methods explains how and why CCM is capturing the directions Colorado Connecticut ● ● correctly, and justifies using CCM in synthetic controls. 0.5 0.4 ● ● ● ● ● ● 0.4 ● ● A similar analysis of CCM in more complex models ● 0.3 ● ● 0.3 ● ● ● would be desirable, but is generally hard since CCM is MAE ● MAE ● ● ● ● 0.2 ● 0.2 ● ● ● ● highly nonlinear. We leave this for future work. ● ● ● ● 4 8 12 16 4 8 12 16 3.3 CCM+SCM Method and Proposition 99 L L Montana Nevada Here, we present CCM scores in the Proposition 99 0.7 ● ● 0.6 ● 0.6 ● example introduced in Section 2.1. Specifically, Cali- ● 0.5 ● ● 0.4 ● 0.4 ● ● fornia (CA) is cross-mapped with five control states ● ● MAE ● ● MAE ● ● 0.3 ● ● ● ● ● 0.2 ● ● ● selected by the standard synthetic control method as 0.2 ● ● ● ● ● shown in Equation (3). We use per-capita cigarette 4 8 12 16 4 8 12 16 sales from the pre-intervention period as the outcome L L variable, which gives 19 data points for each unit’s time- Utah Smooth Connecticut series. The cross predictability measured by the CCM ● 0.4 ● scores for each pair is shown in Figure3. For example, 0.5 ● ● ● ● 0.3 ● 0.4 ● ● the California-Colorado pair includes two CCM curves, ● ● 0.2 ● ●

MAE ● MAE 0.3 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● namely, CCM(YCA,t | YCO,t) and CCM(YCO,t | YCA,t). 0.2 ● 0.1 ● ● 4 8 12 16 4 8 12 16 We see that cross predictability for all pairs roughly L L converge as the library size, L, grows. Furthermore, most pairs converge to the same low level of CCM CCM(~|CA) CCM(CA|~) score, indicating a strong and bidirectional dynamical relationship between the state pairs. The only obvious Figure 3: CCM scores based on MAE between Cali- exception is the California-Connecticut pair, where a fornia and control states with varying library size L. big gap occurs between the two curves exists, indicating CCM(CA |∼) means cross predicting California from a weak dynamical relationship between them. control, and CCM(∼| CA) means cross predicting con- trol from California. In particular, we see that Connecticut is better pre- dicted from California than the other way round. For sarial units in the control pool affected the performance this reason, we argue that Connecticut is not a suitable of synthetic controls. Figure4 shows that CCM+SCM control for California and should be removed from the is able to screen the adversarial units, and is able to donor pool. If we apply an averaging transformation produce a synthetic control that is indistinguishable to smooth out the 1970-1980 trend of Connecticut, the from the non-adversarial setting. In the following sec- CCM score changes and now shows a strong dynamic tion, we explore the performance of CCM+SCM further coupling between the two states (bottom-right plot in through simulated studies and real-world data. Figure3). If Connecticut is removed, SCM will pick Minnesota. However, CCM will screen Minnesota as well because the cigarette price trends are similar be- 4 Experiments and Applications tween Minnesota and Connecticut, but distinct from California. Here, we design adversarial settings where artificial units are added to the donor pool to bias the syn- Our proposed method is therefore to use CCM to filter thetic control method. Of particular interest is whether out controls that have a weak dynamical relationship CCM+SCM can help filter out the artificial units in with the treated unit, and then apply the standard all cases, without affecting the baseline performance SCM method as described in Section2. We refer to when artificial units are not present. We also consider this method as “CCM+SCM". In practice, we propose real-world applications. that CCM+SCM filters out a control unit if in the two CCM plots with the treated unit, either the minimum MAE or the MAE gap exceed some thresholds. To 4.1 Simulations with Artificial Units determine the cutoff values, we may use Monte-Carlo First, we expand the tobacco legislation example of simulations where we add noise to the original series, Section 2.1 by introducing a larger set of artificial units, and then estimate the null distribution of CCM values which are created adversarially. These artificial data under a hypothesis of weak dynamical relationship. were created based on real-world time series macroe- To illustrate the potential of CCM+SCM, we return to conomic data. The detailed generation process can be the example of Section 2.1, where we showed that adver- found in the Supplement. We run studies in Yi Ding, Panos Toulis

California California 160 synthetic California 160 synthetic California 140 140 120 120 100 100 80 80 60 60 per−capita cigarette sales (in packs) per−capita cigarette sales (in packs) 40 40

1970 1975 1980 1985 1990 1995 2000 1970 1975 1980 1985 1990 1995 2000

year year

Figure 4: Trends in per-capita cigarette sales after CCM pre-screening. The solid line is actual California and the dashed line is synthetic California, while the vertical line indicates time of intervention. Left: original setting. Right: adversarial setting where artificial units are added to the donor pool. which the true effect is known for the treated unit. We 4.2 Real-World Applications replace the true California with the following formula: Here, we consider how many artificial units CCM is able to filter out with real-world data. To that end, we work with two real applications: one is still the Califor- Y = 0.164 × Y + 0.069 × Y + 0.199 × Y eCA,t CO,t CT,t MO,t nia’s Tobacco Control Program studied in Abadie et al. + 0.234 × YNV,t + 0.334 × YUT,t + τI{t ≥ 1989}, (2010); and the other is on the economic costs of the Brexit referendum vote on UK’s GDP reported in Born et al.(2017). We only compare SCM and CCM+SCM. where τ is the true treatment effect and the other terms Our results are robust to the selection of the underlying construct the synthetic California from the original model. As before, artificial units are created from noisy data. This construction ensures that the ground truth copies of real-world time series. of synthetic California in the post-intervention period Figure6 shows the number of artificial units that are is known. To illustrate that CCM is a general frame- selected as controls, and the corresponding average work for pre-screening, we incorporate CCM to three treatment effect (ATE). Clearly, CCM+SCM selects different synthetic control methods: 1) SCM: vanilla much fewer artificial units than just SCM. In addition, synthetic control method by Abadie et al.(2010); 2) CCM+SCM generates more stable estimates of ATE. MC: matrix completion method for causal panel data Specifically, in the tobacco example, the ATE reported by Athey et al.(2017); 3) RSC: robust synthetic control by CCM+SCM remains stable around 20, which is method by Amjad et al.(2018). close to the original ATE (≈ 19 packets) estimated The comparisons between the ground truth and three by SCM without artificial controls. In contrast, the synthetic control methods with and without CCM are ATE estimate from SCM with artificial controls is more visualized in Figure5. The different methods lead varied in a range from 5 to 23. Another interesting to the same conclusion: with CCM pre-screening the phenomenon is that the ATE estimate from SCM can outcome estimates, in both pre- and post-intervention be negative under certain artificial controls in Brexit. periods, are closer to the ground truth than the origi- This means that the effects of the Brexit vote may have nal method alone. We note that MC works by matrix been overstated in ongoing econometric work that uses completion instead of selecting control units so CCM+MC synthetic control methods, as the estimates are likely behaves very similarly to MC. RSC alone does not work to be sensitive to control pool construction. well because it uses the artificial unit to construct the synthetic control, and the denoising via singular value 5 Discussion thresholding does not help here. The result suggests that CCM can help in synthetic control models, and In this section, we discuss some general aspects of our is robust to the selection of the underlying outcome work, particularly CCM as a causal inference method, imputation model. Intuitively, this is because CCM is and its underlying assumptions. able to capture nonlinear dynamical information that is not captured by standard statistical models. Our first point revolves around the use of CCM, and Dynamical Systems Theory for Causal Inference with Application to Synthetic Control Methods

120

120 120

100 100 100

CCM+SCM CCM+MC CCM+RSC Groundtruth Groundtruth Groundtruth 80 80 SCM MC 80 RSC per−capita cigarette sales (in packs)

1970 1980 1990 2000 1970 1980 1990 2000 1970 1980 1990 2000 year year year Figure 5: Comparisons of between ground truth and constructed synthetic California with and without CCM by three synthetic control methods. (In this simulation, τ = 4, but the results are robust to this value).

8 ● an attractor”. Stationarity in Theorem1 is thus not 22 6 directly applicable to Takens’ theorem, and is assumed

● 20 ●

4 only in our effort to understand CCM vis-à-vis AR(1). ATE nstates 18 In this work, we argue that CCM can strengthen the 2 16 model continuity assumption (pre- and post-treatment), 0 which is left implicit in synthetic controls. But there is SCM CCM+SCM SCM CCM+SCM Tobacco Tobacco much left to understand about the connection between 5 3 ● ● chaotic and stochastic systems (Casdagli, 1992). We 4 2 hope that this work provides a good motivation. 3 ATE 2 1 ncountries 1 6 Conclusion 0

0 SCM CCM+SCM SCM CCM+SCM Brexit Brexit In this paper, we leveraged results from dynamical SCM CCM+SCM systems theory to quantify the strength of dynamic Figure 6: Box plots of the number of artificial control relationship between treated and control units in causal units selected by each method (left), namely SCM and inference. We showed that this is useful in the context CCM+SCM, and average treatment effect (right) in of comparative cases studies to guard against cherry- the Tobacco and Brexit applications. picking of potential controls, which is an important concern in practice. More generally, our work opens up the potential for an related dynamical systems methods, for causal infer- interplay between dynamical systems theory and causal ence. In particular, due to the success of CCM in inference. In practice, interventions typically occur on quantifying dynamical relationships illustrated here, it complex dynamical systems, such as an auction or a may be tempting to consider CCM as a general method labor market, which always evolve, before and after for causal inference (Sugihara et al., 2012; Deyle et al., treatment. Future work could focus more on theoreti- 2013). However, we do not advocate doing that directly, cal connections between embedding methods, such as for two main reasons. First, the statistical properties of CCM, and standard treatment effects in econometrics, methods such as CCM are not well known. Theorem1 especially if we view the filtering process described in in this paper is a step to this direction, but more work is Section3 as a way to do treated-control matching. needed. Second, CCM does not account for the observa- tion model (e.g., the treatment assignment mechanism), which is crucial in causal inference (Imbens and Rubin, Acknowledgment 2015). We also provide counterexamples and additional discussion in the Supplement. We thank our anonymous reviewers for the Regarding the stationarity assumption in Theo- that improved this final version of the paper. We also rem1 (i.e., |α| < 1, and Assumption 1(b)), we note that thank Hao Ye, who provided assistance in understand- CCM assumes a deterministic (possibly chaotic) system, ing CCM. Y.D.’s work has been partially supported by and so stationarity is roughly mapped to “evolution on the NSF (CCF-1439156, CCF-1823032, CNS-1764039). Yi Ding, Panos Toulis

References Casdagli, M., Eubank, S., Farmer, J. D., and Gibson, J. (1991). reconstruction in the presence Abadie, A., Diamond, A., and Hainmueller, J. (2010). of noise. Phys. D, 51(1-3):52–98. Synthetic control methods for comparative case stud- ies: Estimating the effect of california’s tobacco con- Deyle, E. R., Fogarty, M., Hsieh, C.-h., Kaufman, L., trol program. Journal of the American Statistical MacCall, A. D., Munch, S. B., Perretti, C. T., Ye, Association, 105(490):493–505. H., and Sugihara, G. (2013). Predicting climate effects on pacific sardine. Proceedings of the National Abadie, A., Diamond, A., and Hainmueller, J. (2015). Academy of Sciences, 110(16):6430–6435. Comparative politics and the synthetic control method. American Journal of Political Science, Durlauf, S. N. (2005). and empirical 59(2):495–510. economics. The Economic Journal, 115(504):F225– F243. Abadie, A. and Gardeazabal, J. (2003). The economic costs of conflict: A case study of the basque country. Granger, C. W. J. (1969). Investigating causal relations American Economic Review, 93(1):113–132. by econometric models and cross-spectral methods. Econometrica, 37(3):424–438. Amjad, M., Shah, D., and Shen, D. (2018). Robust synthetic control. The Journal of Machine Learning Härdle, W., Lütkepohl, H., and Chen, R. (1997). A Research, 19(1):802–852. review of nonparametric time series analysis. Inter- national Statistical Review, 65(1):49–72. Angrist, J. D. and Pischke, J.-S. (2008). Mostly Hastie, T., Tibshirani, R., and Friedman, J. (2001). Harmless Econometrics: An Empiricist’s Compan- The Elements of Statistical Learning. Springer. ion. Princeton University Press. Hazlett, C. and Xu, Y. (2018). Trajectory balancing: Arnhold, J., Grassberger, P., Lehnertz, K., and Elger, A general reweighting approach to causal inference C. E. (1999). A robust method for detecting inter- with time-series cross-sectional data. dependences: Application to intracranially recorded eeg. Phys. D, 134(4):419–430. Imbens, G. W. and Rubin, D. B. (2015). Causal in- ference in statistics, social, and biomedical sciences. Athey, S., Bayati, M., Doudchenko, N., Imbens, G., Cambridge University Press. and Khosravi, K. (2017). Matrix completion methods for causal panel data models. Kreif, N., Grieve, R., Hangartner, D., Turner, A. J., Nikolova, S., and Sutton, M. (2016). Examination of Athey, S., Bayati, M., Doudchenko, N., Imbens, G., and the synthetic control method for evaluating health Khosravi, K. (2018). Matrix completion methods for policies with multiple treated units. Health Eco- causal panel data models. Technical report, National nomics, 25(12):1514–1528. Bureau of Economic Research. Lorenz, E. N. (1963). Deterministic nonperiodic flow. Ben-Michael, E., Feller, A., and Rothstein, J. (2018). Journal of the Atmospheric Sciences, 20(2):130–141. The augmented synthetic control method. arXiv Rosser, J. B. (1999). On the of complex preprint arXiv:1811.04170. economic dynamics. The Journal of Economic Per- Billmeier, A. and Nannicini, T. (2013). Assessing eco- spectives, 13(4):169–192. nomic liberalization episodes: A synthetic control Runge, J., Bathiany, S., Bollt, E., Camps-Valls, G., approach. The Review of Economics and Statistics, Coumou, D., Deyle, E., Glymour, C., Kretschmer, 95(3):983–1001. M., Mahecha, M. D., Muñoz-Marí, J., et al. (2019). Born, B., Müller, G. J., Schularick, M., and Sedlacek, Inferring causation from time series in earth system P. (2017). The Economic Consequences of the Brexit sciences. Nature communications, 10(1):1–13. Vote. Discussion Papers 1738, Centre for Macroeco- Saunders, J., Lundberg, R., Braga, A. A., Ridgeway, G., nomics (CFM). and Miles, J. (2015). A synthetic control approach to Bureau of Labor Statistics, U. D. o. L. (2018). Local evaluating place-based crime interventions. Journal Area Unemployment Statistics. of Quantitative Criminology, 31(3):413–434. Card, D. and Krueger, A. (1994). Minimum wages and Schiff, S. J., So, P., Chang, T., Burke, R. E., and Sauer, employment: A case study of the fast-food industry T. (1996). Detecting dynamical interdependence and in new jersey and pennsylvania. American Economic generalized synchrony through mutual prediction in Review, 84(4):772–93. a neural ensemble. Phys. Rev. E, 54:6708–6724. Casdagli, M. (1992). Chaos and deterministic ver- Shaikh, A. and Toulis, P. (2019). Randomization tests sus stochastic non-linear modelling. Journal of the in observational studies with staggered adoption of Royal Statistical Society: Series B (Methodological), treatment. University of Chicago, Becker Friedman 54(2):303–328. Institute for Economics Working Paper, (2019-144). Dynamical Systems Theory for Causal Inference with Application to Synthetic Control Methods

Sugihara, G., May, R., Ye, H., Hsieh, C.-h., Deyle, E., Fogarty, M., and Munch, S. (2012). Detecting causality in complex ecosystems. 338. Sugihara, G. and May, R. M. (1990). Nonlinear forecast- ing as a way of distinguishing chaos from measure- ment error in time series. Nature, 344(6268):734–741. Takens, F. (1981). Detecting strange attractors in turbulence. Lecture Notes in , Berlin Springer Verlag, 898:366.