IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 14, NO. 3, JUNE 2012 619 Causal Flow Yuya Yamashita, Tatsuya Harada, Member, IEEE, and Yasuo Kuniyoshi, Member, IEEE Abstract—Optical flow is a widely used technique for extracting delay correlation. In particular, Granger causality is widely used flow information from video images. While it is useful for esti- in economics and brain science, and it has been used in many mating temporary movement in video images, it only captures one research studies [9], [10]. aspect of extracting dominant flow information from a sequence of video images. In this paper, we propose a novel flow extraction Granger causality can quantify the intensity of a causal rela- approach called causal flow, which can estimate the dominant tionship from one time series to another. However, it cannot be causal relationships among nearby pixels. We assume flows in applied to time series that exhibit little variation (i.e., stay flat, video images as pixel-to-pixel information transfer,whereasthe or those with small sample size. In addition, this method is sus- optical flow measures the relative motion of pixels. Causal flow ceptible to noise and outliers. Videos recorded by fixed cameras is based on the Granger causality test, which measures causal influence based on prediction via vector autoregression, and is commonly include pixels which stay nearly flat or include noise. widely used in economics and brain science. The experimental In this paper, we propose causal flow, which can estimate the results demonstrate that causal flow can extract dominant flow dominant causal relationship among nearby pixels and, thus, information which cannot be obtained by current methods. is another way of the optical flow.WeimprovetheGranger Index Terms—Granger causality, optical flow, regularization, causality to realize a stable estimation of causal intensity, and video feature. propose the regularized Granger causality basedonaregu- larization technique and partial canonical correlation analysis (PCCA) [11]. This improvement in Granger causality is eval- I. INTRODUCTION uated by experiments with synthetic data. Finally, we estimate PTICAL FLOW IS the most popular approach for esti- causal flow from a video image dataset, and compare causal flow O mating flows in video images. Many methods for esti- with average optical flow to demonstrate the useful properties mating optical flow have been proposed, including the gradient- of causal flow. based method [1] and the block-based method [2]. The study of optical-flow estimation remains an important research topic II. RELATED WORKS [3]. In recent years, dominant flow information has been applied A. Granger Causality successfully in crowd flow segmentation [4] and anomaly detec- tion of traffic flow [5]. Granger causality is a time-series analysis method proposed The optical-flow methods calculate the motion between two by Granger [6]. Using this method, we can quantify causal in- frames which are taken at times and at every pixel po- tensity among time series. Granger causality has been applied to sition. Therefore, the optical flow captures the motion of every many kinds of data, such as economic indices [9], and has been pixel. However, the motion is only one aspect of flow informa- used in many research studies. tion from a sequence of video images. Here, we assume flows in First, we intend to explain the Granger causality for video images as pixel-to-pixel information transfer. Each pixel 1-D time series. Let and denote a 1-D time series is represented as a time series of image intensity (i.e., when . analyzing gray-scale video, pixels are described as a 1-D time Both time series data are normalized as series). Thus, we consider quantifying information transfer be- . In addition, all time series data that tween time series. appear in this paper are also normalized. When we intend to A relation among time series exists, such as stock prices or quantify causal intensity from time series to time series , electric potentials measured in regions of the brain, and many we compare the following two time series models: methods for estimating these relationships have been proposed. (1) For example, causality measures, such as Granger causality [6] and transfer entropy [7], can quantify causal relationships, and (2) cross-canonical correlation analysis (xCCA) [8] measures time where denotes the transpose. are delay embed- Manuscript received October 01, 2011; revised February 24, 2012; accepted ding vectors, and the parameter is the embedding dimension. March 05, 2012. Date of publication April 03, 2012; date of current version May 11, 2012. The associate editor coordinating the review of this manuscript and The delay embedding vectors are defined as approving it for publication was Dr. Mrinal Mandal. The authors are with the Department of Mechano-informatics, Graduate (3) School of Information Science and Technology, The University of Tokyo, Bunkyo-ku 113-8656, Tokyo Japan (e-mail: [email protected]; (4) [email protected]; [email protected]). Color versions of one or more of the figures in this paper are available online where are coefficients for the delay at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TMM.2012.2191396 embedding vectors, and are Gaussian 1520-9210/$31.00 © 2012 IEEE 620 IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 14, NO. 3, JUNE 2012 noise with zero mean, and variances, The partial covariance matrix is defined as respectively. A time series model, such as (1), is an autoregres- follows: sive model (AR model) of order . We can quantify causal in- tensity using the Granger causality. Shibuya et al. defined causal intensity as follows [12]: (12) where denotes the covariance matrix of and (5) . The other partial covariance matrices are also calculated in this manner. In this case, causal intensity is expressed by The approach of Granger causality is to compare the predic- the eigenvalues given by tion accuracy of an autoregressive model with that of a regres- solving (11). Following the relationship between CCA and sion model which incorporates the past information of another mutual information [15], causal intensity can be expressed as variable. If prediction accuracy is improved by incorporating in- information content: formation from another variable, a causal relationship exists. (13) Similarly, we can define the multidimensional version of Granger causality. The dimensions of two multidi- The eigenvalues corresponding to the data dimension are ob- mensional time series tained by solving this generalized eigenvalue problem. The ad- are and , respectively. vantage of the PCCA interpretation of the Granger causality is We compare the following two models in the multidimensional that we can determine relationships on canonical spaces that version of Granger causality: correspond to these eigenvalues. Causal relationships between two variables are strongly expressed on canonical spaces which (6) correspond to large eigenvalues, and canonical spaces, which (7) correspond to small eigenvalues, are insignificant to determine causal relationships. In this way, we can reduce the influence of outliers and noise on the canonical spaces corresponding to where and are delay embedding vectors of order , large eigenvalues. defined as III. REGULARIZED GRANGER CAUSALITY (8) Time series that are analyzed with Granger causality, such (9) as stock prices [9] and brain waves [10], usually show a con- stant change of their values. However, time series of brightness where are co- values in video images do not always show constant change. For efficient matrices. In the multidimensional case, if we define that example, some pixels of video images taken by a fixed camera the variance-covariance matrices of tend not to change at all. In the previous section, we introduced are , the causal intensity as defined PCCA interpretation of Granger causality, and explained that by Ladroue et al. [10] is this interpretation reduces to a generalized eigenvalue problem. This generalized eigenvalue problem requires the calculation of matrix inverse, and partial covariance matrices are closer to (10) nonsingular for the data which are static and are composed of small numbers of samples. Therefore, we can improve upon the where denotes the trace. PCCA’s treatment of such data by solving the generalized eigen- valueprobleminastableway. B. PCCA Interpretation of Granger Causality The partial covariance matrices introduced in (12) also re- We explained before that Granger causality compares two re- quires inverse matrix calculations, which cause instability. First, gression models. An alternative expression of the concept of we improve the calculation of the partial covariance matrices. Granger causality is provided by PCCA [13]. A popular mul- We can interpret the meaning of the partial covariance matrix tivariate analysis method, canonical correlation analysis (CCA) as the covariance matrix of residuals in the regression model, [14], determines the projections which maximize the correlation where and are dependent variables and is an in- between two multidimensional variants. Compared with CCA, dependent variable. Let denote ,theresidual PCCA employs a third variable and finds the projections, which which corresponds to the partial covariance matrix in (12) ap- maximize the correlation between two variables under the con- pears in the following regression model: dition that the information of the third variable is removed. The PCCA expression of Granger causality reduces to the following (14) generalized