Causal Flow Yuya Yamashita, Tatsuya Harada, Member, IEEE, and Yasuo Kuniyoshi, Member, IEEE

Home , Granger causality

IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 14, NO. 3, JUNE 2012 619 Causal Flow Yuya Yamashita, Tatsuya Harada, Member, IEEE, and Yasuo Kuniyoshi, Member, IEEE

Abstract—Optical flow is a widely used technique for extracting delay correlation. In particular, Granger causality is widely used flow information from video images. While it is useful for esti- in economics and brain science, and it has been used in many mating temporary movement in video images, it only captures one research studies [9], [10]. aspect of extracting dominant flow information from a sequence of video images. In this paper, we propose a novel flow extraction Granger causality can quantify the intensity of a causal rela- approach called causal flow, which can estimate the dominant tionship from one time series to another. However, it cannot be causal relationships among nearby pixels. We assume flows in applied to time series that exhibit little variation (i.e., stay flat, video images as pixel-to-pixel information transfer,whereasthe or those with small sample size. In addition, this method is sus- optical flow measures the relative motion of pixels. Causal flow ceptible to noise and outliers. Videos recorded by fixed cameras is based on the Granger causality test, which measures causal influence based on prediction via vector autoregression, and is commonly include pixels which stay nearly flat or include noise. widely used in economics and brain science. The experimental In this paper, we propose causal flow, which can estimate the results demonstrate that causal flow can extract dominant flow dominant causal relationship among nearby pixels and, thus, information which cannot be obtained by current methods. is another way of the optical flow.WeimprovetheGranger Index Terms—Granger causality, optical flow, regularization, causality to realize a stable estimation of causal intensity, and video feature. propose the regularized Granger causality basedonaregu- larization technique and partial canonical correlation analysis (PCCA) [11]. This improvement in Granger causality is eval- I. INTRODUCTION uated by experiments with synthetic data. Finally, we estimate PTICAL FLOW IS the most popular approach for esti- causal flow from a video image dataset, and compare causal flow O mating flows in video images. Many methods for esti- with average optical flow to demonstrate the useful properties mating optical flow have been proposed, including the gradient- of causal flow. based method [1] and the block-based method [2]. The study of optical-flow estimation remains an important research topic II. RELATED WORKS [3]. In recent years, dominant flow information has been applied A. Granger Causality successfully in crowd flow segmentation [4] and anomaly detec- tion of traffic flow [5]. Granger causality is a time-series analysis method proposed The optical-flow methods calculate the motion between two by Granger [6]. Using this method, we can quantify causal in- frames which are taken at times and at every pixel po- tensity among time series. Granger causality has been applied to sition. Therefore, the optical flow captures the motion of every many kinds of data, such as economic indices [9], and has been pixel. However, the motion is only one aspect of flow informa- used in many research studies. tion from a sequence of video images. Here, we assume flows in First, we intend to explain the Granger causality for video images as pixel-to-pixel information transfer. Each pixel 1-D time series. Let and denote a 1-D time series is represented as a time series of image intensity (i.e., when . analyzing gray-scale video, pixels are described as a 1-D time Both time series data are normalized as series). Thus, we consider quantifying information transfer be- . In addition, all time series data that tween time series. appear in this paper are also normalized. When we intend to A relation among time series exists, such as stock prices or quantify causal intensity from time series to time series , electric potentials measured in regions of the brain, and many we compare the following two time series models: methods for estimating these relationships have been proposed. (1) For example, causality measures, such as Granger causality [6] and transfer entropy [7], can quantify causal relationships, and (2) cross-canonical correlation analysis (xCCA) [8] measures time where denotes the transpose. are delay embed- Manuscript received October 01, 2011; revised February 24, 2012; accepted ding vectors, and the parameter is the embedding dimension. March 05, 2012. Date of publication April 03, 2012; date of current version May 11, 2012. The associate editor coordinating the review of this manuscript and The delay embedding vectors are defined as approving it for publication was Dr. Mrinal Mandal. The authors are with the Department of Mechano-informatics, Graduate (3) School of Information Science and Technology, The University of Tokyo, Bunkyo-ku 113-8656, Tokyo Japan (e-mail: [email protected]; (4) [email protected]; [email protected]). Color versions of one or more of the figures in this paper are available online where are coefficients for the delay at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TMM.2012.2191396 embedding vectors, and are Gaussian

1520-9210/$31.00 © 2012 IEEE 620 IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 14, NO. 3, JUNE 2012 noise with zero mean, and variances, The partial covariance matrix is deﬁned as respectively. A time series model, such as (1), is an autoregres- follows: sive model (AR model) of order . We can quantify causal intensity using the Granger causality. Shibuya et al. deﬁned causal intensity as follows [12]: (12)

where denotes the covariance matrix of and (5) . The other partial covariance matrices are also calculated in this manner. In this case, causal intensity is expressed by The approach of Granger causality is to compare the predic- the eigenvalues given by tion accuracy of an autoregressive model with that of a regres- solving (11). Following the relationship between CCA and sion model which incorporates the past information of another mutual information [15], causal intensity can be expressed as variable. If prediction accuracy is improved by incorporating in- information content: formation from another variable, a causal relationship exists. (13) Similarly, we can define the multidimensional version of Granger causality. The dimensions of two multidi- The eigenvalues corresponding to the data dimension are ob- mensional time series tained by solving this generalized eigenvalue problem. The ad- are and , respectively. vantage of the PCCA interpretation of the Granger causality is We compare the following two models in the multidimensional that we can determine relationships on canonical spaces that version of Granger causality: correspond to these eigenvalues. Causal relationships between two variables are strongly expressed on canonical spaces which (6) correspond to large eigenvalues, and canonical spaces, which (7) correspond to small eigenvalues, are insignificant to determine causal relationships. In this way, we can reduce the influence of outliers and noise on the canonical spaces corresponding to where and are delay embedding vectors of order , large eigenvalues. defined as III. REGULARIZED GRANGER CAUSALITY (8) Time series that are analyzed with Granger causality, such (9) as stock prices [9] and brain waves [10], usually show a constant change of their values. However, time series of brightness where are co- values in video images do not always show constant change. For efficient matrices. In the multidimensional case, if we define that example, some pixels of video images taken by a fixed camera the variance-covariance matrices of tend not to change at all. In the previous section, we introduced are , the causal intensity as defined PCCA interpretation of Granger causality, and explained that by Ladroue et al. [10] is this interpretation reduces to a generalized eigenvalue problem. This generalized eigenvalue problem requires the calculation of matrix inverse, and partial covariance matrices are closer to (10) nonsingular for the data which are static and are composed of small numbers of samples. Therefore, we can improve upon the where denotes the trace. PCCA’s treatment of such data by solving the generalized eigen- valueprobleminastableway. B. PCCA Interpretation of Granger Causality The partial covariance matrices introduced in (12) also re- We explained before that Granger causality compares two requires inverse matrix calculations, which cause instability. First, gression models. An alternative expression of the concept of we improve the calculation of the partial covariance matrices. Granger causality is provided by PCCA [13]. A popular mul- We can interpret the meaning of the partial covariance matrix tivariate analysis method, canonical correlation analysis (CCA) as the covariance matrix of residuals in the regression model, [14], determines the projections which maximize the correlation where and are dependent variables and is an in- between two multidimensional variants. Compared with CCA, dependent variable. Let denote ,theresidual PCCA employs a third variable and finds the projections, which which corresponds to the partial covariance matrix in (12) ap- maximize the correlation between two variables under the con- pears in the following regression model: dition that the information of the third variable is removed. The PCCA expression of Granger causality reduces to the following (14) generalized eigenvalue problem: The coefficient is given by solving the minimization problem

(11) (15) YAMASHITA et al.: CAUSAL FLOW 621

Regularization is a technique which stabilizes the calculation of regression model estimations such as (15). It is used here by applying a restriction to the coefﬁcient , and the restriction for the norm of is referred to as ridge estimation [16]. The minimization problem with ridge estimation is

(16) where is the regularization parameter which ﬁxes the intensity of restriction. Solving this minimization problem, is Fig. 1. (a) Positions of the eight pixels (block regions). (b) Causal intensities given as from the center pixel (block region) to surrounding pixels (block regions) and orientation of causal ﬂow. (17)

(18) from the center pixel to the surrounding eight pixels is calculated using regularized Granger causality, where the measures where is the unit matrix. The value of is substituted into are (14), and we then calculate the covariance matrix of the resid- and . The orientation of the causal ﬂow and the uals. The covariance matrix is divided into four block matrices intensity of the relationship are illustrated in Fig. 1(b) and deﬁned as (19)

(22) These block matrices are given as

(23)

(24)

The current method cannot be applied to data in which the in- (20) tensities of the pixels are static. In this case, regularized Granger causality can be applied to these data, and the causal flow can where denotes . be extracted. Subsequently, we explain the calculation of the generalized eigenvalue problem in (11). Regularization is applied to this problem in a similar way. The generalized eigenvalue problem B. Block-Based Causal Flow after applying regularization is In addition, we propose block-based causal flow, which is more robust to noise in video images. The way of thinking about block-based extension of causal (21) flow is very simple. We can replace the focused pixel and surrounding pixels of causal flow to block regions. This is because This type of regularization is called a Canonical Ridge [17] regularized Granger causality can be directly applied to mluti- in the CCA context, and the same approach can be applied to variate time series. PCCA. Block-to-block comparison is similar to block-based optical-flow estimation; however, causal flow estimation does not IV. CAUSAL FLOW need to search for a wide range, which is needed in block-based optical-flow estimation. The purpose of causal flow estimation In this section, we propose causal flow, which can extract is not to extract the moving speed of objects, so how far the stationary flow information from video images. object goes is not important. The calculation of causal intensity needs to only surround the block regions next to the focused A. Pixel-Wise Causal Flow region. The time series dimension extracted from each region First, we explain pixel-wise causal flow. Pixel-wise causal is determined by block size and color channel (RGB color: 3 flow is computed from the image intensity of one pixel and that or grayscale: 1). For example, if block regions are square of the surrounding eight pixels. To demonstrate, let the position 3 pixels on a side and the input movie is RGB color video, of the center pixel be (0,0). The positions of the surrounding (pixels in block region) (color channels) 27 dimensional eight pixels are illustrated in Fig. 1(a). The causal intensity time series are extracted from each region. Note that the focused 622 IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 14, NO. 3, JUNE 2012 region must not overlap with surrounding regions because the relationships intensity in (13) to eigenvalues of the generalize overlap of regions causes perfect correlation, and the causal eigenvalue problem of xCCA. intensity in (13) becomes infinity. Suppose that a 3-D time series and are definedbythe equations C. Comparison With Optical Flow 1) Gradient-Based Optical Flow: Let denote (28) image intensity as a function of position and time. The gradient-based method assumes that image intensity of position (29) which moves during does not change

(25) where are Gaussian noise with zero mean and variances. is fixed to the value of 0.1. In this experiment, we Taylor expansion is applied to the right side of (25) changed the variance of Gaussian noise from0to0.1andex- amined the intensity of the relationships with each method. In this synthetic dataset, past values became current values, (26) so that the causal relationship from to was strong, and a causal relationship in the inverse direction did not exist. The where denotes the high-order term. Each side of (26) divided parameters of both methods were that the embedding dimen- by and sion and the regularization parameters of the proposed method had the same value . (27) We did not search the optimal time delay parameters of both methods and we used 1. where and are the spatial and temporal derivatives of The results are shown in Fig. 2. xCCA was not able to calcu- the . is the optical flow. To compute the optical flow, addi- late the intensity of the relationship from to at 0, as tional constraint is needed. For example, Horn et al. [1] applied the representation of (13) diverged to infinity when the eigen- the global smoothness constraint and Lucas et al. [18] applied value 1 (i.e., perfect correlation). the local smoothness constraint. Despite the variance of noise, the causal intensity from The gradient-based optical-flow method estimates the to was near zero in the proposed method, but relationships moving speed of an object under the constraint of smoothness. existed in xCCA. This is because xCCA did not account for the Compared with optical flow, the causal flow estimate does influence of past values on current values. Considering not move speed but has the intensity of relationships between that the time series of image intensity shows gradual changes, pixels or block regions. Causal flow does not need smoothness except in the case of something passing through the pixel of constraint, so causal flow is more sensitive to the variation of a interest, this is an important property of the proposed method. small region. 2) Block-Based (Block Matching) Optical Flow: Another B. Small Sample Stability method of optical-flow estimation, such as Anandan’s method [2], computes direction-dependent confidence measures based In this experiment, we evaluated the performance of regular- on block matching. Block-based optical flow is more similar to ized Granger causality for small sample data. block-based causal flow. In block matching, the focused block Consider the following 3-D time series models: region is assumed as a template in the previous frame, and a (30) similar block region is searched for in the next frame. The similarity measure of block matching is calculated from each pixel (31) in the focused block region and the corresponding pixel of the block region in the next frame. One-to-one correspondence of where are Gaussian noise with zero mean and pixels in block matching exists. variances. is fixed to the value of 1. In this experiment, we As shown in (21), the regularized Granger causality used in changed from 0 to 0.1 and applied the proposed method to the causal flow estimates the projection for block regions. There- synthetic data. Causal intensity from to was strong, as in fore, block-based causal flow can treat illumination change, ob- the previous experiment, and the inverse direction displayed no ject rotation, and scale transition in principle. causal relationships. We experimented with a sample size 10. If we wished to extract short-term change of causal flow, a small sample size, such as ten frames, would not be improbable. V. S YNTHETIC DATA EXPERIMENTS The parameters of the method were that the time delay param- In this section, we describe the experimental evaluations for eter 1 and the embedding dimension 1. regularized Granger causality. Fig. 3 shows the results of experiments with different regularization parameters. In regularized Granger causality, we as- A. Comparison of Causality With Correlation sume that three regularization parameters have the same value We compare the proposed method with xCCA [8], which . When the parameter 0, this was equivalent to the PCCA measures time delay correlations among time series. In order interpretation of Granger causality in Section II-B. The calcula- to permit relative comparisons, we apply the representation of tion results were unstable at 0, and some values of intensity YAMASHITA et al.: CAUSAL FLOW 623

Fig. 2. Comparison of the proposed method (regularized Granger causality) with xCCA.

Fig. 3. Evaluation of small sample stability. (a) PCCA interpretation of Granger causality 0). (b) Proposed method . (c) Proposed method . (d) Proposed method . could not be calculated. In contrast, when was not zero, all C. Embedding Dimension and Regularization intensities of causal relationships could be calculated, and the direction of the relationship was also extracted at 1and In this experiment, we evaluate the relationship between the . It should be noted that the larger the regularization embedding dimension and regularization parameter. parameter, the smaller the causal intensity. Therefore, when we Suppose that a 3-D time series and are definedbythe apply regularized Granger causality to video images, the same equations regularization parameters must also be used in order to find the orientation of the causal flow by relative comparison. (32) 624 IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 14, NO. 3, JUNE 2012

Fig. 4. Effect of the embedding dimension and regularization parameter for causal intensity estimation. (a) .(b) .(c) 10. (d) 1000.

where (33) (36) where are Gaussian noise with zero mean and variances. and are fixed to the value of 0.1. In this syn- where denotes the coupling intensity, and .We thetic data, past values become current values, so set 40. The number of is decided not to have an effect that the causal relationship from to was strong and a causal in each direction and . relationship in the inverse direction did not exist. The coefficient We generated 2-D time series and from the data generated of is decided to generate the stationary time series. We using the ring-latticed Ulam map used time delay parameter 0, and regularization parameter . Fig. 4 shows the result of the experiments. The horizontal (37) axis indicates the embedding dimension, and the vertical axis indicates causal intensity. In this case, when the embedding dimension was equal to or more than 5, the effect of past (38) values could be estimated. Therefore, a certain size of has a good effect to improve the accuracy of causal intensity where denotes transpose, and denotes estimation. the Gaussian noise with zero mean and 0.01 variance. These data have a causal relationship from to . D. Nonlinear Reliability Fig. 5 shows the results of the estimated causal intensity with In this experiment, we used a synthetic multidimensional time different regularization parameters. The vertical axis indicates series generated by chaotic mapping. causal intensity, and the horizontal axis indicates coupling We used a ring-latticed unidirectionally coupled Ulam map intensity parameter . Compared with results of the normal [7]. The Ulam map is described as follows: PCCA version of Granger causality, causal intensity calculated from the second eigenvalue was inhibited in regularized (34) Granger causality. This is because the 2-D time series and (35) are not significantly different from 1-D time series. In addition, YAMASHITA et al.: CAUSAL FLOW 625

Fig. 5. Experimental results of nonlinear data. (a) No regularization (causal intensity calculated from the ﬁrst eigenvalue) 0. (b) No regularization (causal intensity calculated from the second eigenvalue) 0. (c) Regularization ( 1, ﬁrst eigenvalue). (d) Regularization ( 1, second eigenvalue).

Fig. 6. Causal flow of walking people. (a) One frame of original movies. Input Fig. 7. Causal flow of the traffic scene. (a) One frame of original movies. Input movie. (b) Average optical flow. (c)–(f) Causal flow with different parameters. movie. (b) Average optical flow. (c)–(f) Causal flow with different parameters. (c) Causal flow 1 1). (d) Causal flow 2 1). (e) Causal (c) Causal flow 1 .(d)Causalflow 2 1). (e) Causal flow flow 3 1). (f) Causal flow 1 5). 2 1). (f) Causal flow 1 5).

VI. REAL DATA EXPERIMENTS the causal intensity ratio of two directions was larger in the regularized Granger causality. The application A. Comparison Between Causal Flow and Optical Flow of regularization has the effect of improving estimation of not Finally, the extraction of causal ﬂow was tested on the crowd only the linear relationship, but also the nonlinear relationship. segmentation dataset [4], which consisted of crowd scene video 626 IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 14, NO. 3, JUNE 2012

Fig. 8. Similarity measure between the flow estimated from noisy video images and the ground truth flow extracted under the condition of no noise. In the average optical flow, HS means the Horn–Schunck method, and BM means the block-matching method.

Fig. 9. Evaluation of the relationship between movie length and computational time. In the average optical flow, HS means Horn–Schunck method, and BM means the block matching method. images taken from a set of websites. RGB 3-D time series were stairs is slower than that of people who walk through on the flat taken from each pixel, and we extracted the causal flow from floor. The result of the average optical flow in Fig. 6(b) shows these time series. We examined some different parameters for that high flow in the region of flat floor exists and relatively extracting causal flow, using a time delay parameter 1,2,3, low flow in the region of stairs. On the other hand, the causal embedding dimension 1,5 and regularized parameter flow captured high flow in not only the flat floor region but 1. The representation of causal also in the stairs region in Fig. 6(f). Constant flow exists in intensity introduced in (13) was used, employing only the max- the stairs region in this video, and the results show that the imum eigenvalue to acquire the causal intensity. The average causal flow can actually quantify the stationarity of flow. When optical-flow field was also calculated using the Horn–Schunck the embedding dimension was small, the estimated causal method [1]. flow changed according to the time delay parameter .If The results are shown in Figs. 6 and 7. In the video of was large, the entire flow in the time delay spanning from to walking people, the moving speed of people who climb up was extracted. YAMASHITA et al.: CAUSAL FLOW 627

Fig. 10. Flow estimation from various noisy video images. The block size of block-based causal ﬂow is 3, 5. In the average optical ﬂow, BM means the block matching method and the block size is 16.

B. Noise Robustness to the dataset. The Gaussian noise had zero mean and variances. Four types of flow estimation were applied to this noisy In this experiment, we evaluate the effect of a block-based dataset. First, the pixel-wise causal flow was applied. We ex- extension on the noisy dataset. We added the Gaussian noise amined the fixed parameter using embedding dimension 628 IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 14, NO. 3, JUNE 2012

5, time delay parameter 1, and regularization parameter and demonstrated the differences between causal flow and av- . Second, our proposed block- erage of optical flow in the experiments with real video images. based causal flow was applied. The embedding dimension and In the future, we intend to apply causal flow estimation to video delay parameters were the same as those of the first estimation data, which have already been analyzed by optical flow, to fur- method, and the block size was 3, 5, and 7. Third, the av- ther demonstrate the utility of causal flow. erage of the Horn–Schunck method optical flow was applied. Finally, the average of block-based optical flow was applied. REFERENCES We use two block sizes 5and 16. In each method, we compared the flow estimated from no noise video images with [1] B. K. P. Horn and B. G. Schunck, “Determining optical flow,” Artif. Intell., vol. 17, no. 1–3, pp. 185–203, 1981. those estimated from various noise. We used similarity measure [2] P. Anandan, A Computational Framework and an Algorithm for the between two flow fields, which was similar to angular error [19]. Measurement of Visual Motion. New York: Springer, 1989. The angular error is an angle in 3-D space between [3] T. Brox, C. Bregler, and J. Malik, “Large displacement optical flow,” and . However, norms of causal flow do not indi- presented at the IEEE Comput. Vis. Pattern Recogn. Conf., Miami, FL, Jun. 2009. cate the moving speed and scale of causal flow changes due to [4] S. Ali and M. Shah, “A Lagrangian particle dynamics approach for parameters, hence, we cannot use fixed value 1.0 to compare the crowd flow segmentation and stability analysis,” presented at the IEEE performance of flow estimation. To tackle theproblem,weused Comput. Vis. Pattern Recogn. Conf., Orlando, FL, 2007. medians of flow norms estimated from each of the video images [5] J.LeeandA.C.Bovik,“Estimationandanalysisofurbantraffic flow,” presented at the IEEE Int. Conf. Image Process., Cairo, Egypt, Nov. with no noise in the place of 1.0, and we assumed cosines of an- 2009. gles as similarity measures. [6] C. W. J. Granger, “Investigating causal relations by econometric Fig. 8 shows the result of the experiment. The similarity models and cross spectral methods,” Econometrica, vol. 37, no. 3, pp. 424–438, 1969. measure became rapidly smaller with the variance of noise [7] T. Schreiber, “Measuring information transfer,” Phys. Rev. Lett., vol. increasing in the pixel-wise causal flow. The changes of the sim- 85, no. 2, pp. 461–464, 2000. ilarity measures of the block-based causal flow were smaller [8] C. C. Loy, T. Xiang, and S. Gong, “Multi-camera activity correlation than pixel-wise causal flow and average optical flows. The analysis,” presented at the IEEE Comput. Vis. Pattern Recogn., Miami, FL, Jun. 2009. block-based causal flow with a large block region [9] C. Hiemstra and J. D. Jones, “Testing for linear and nonlinear granger was especially robust to noise. causality in the stock price-volume relation,” J. Finance, vol. 49, no. Fig. 10 shows the examples of estimated flows. 5, pp. 1639–1664, 1994. [10] C. Ladroue, S. Guo, K. Kendrick, and J. Feng, “Beyond element-wise interactions: Identifying complex interactions in biological processes,” C. Computational Complexity PLoS One, vol. 4, no. 9, p. e6899, 2009. [11] B. R. Rao, “Partial canonical correlations,” Trabajos de Estadistica y We provided the evaluation of computational complexity. As Invest. Operative, vol. 20, no. 2–3, pp. 211–219, 1969. with the previous experiment, we applied four methods to the [12] T. Shibuya, T. Harada, and Y. Kuniyoshi, “Causality quantification and its applications: Structuring and modeling of multivariate time series,” movie. We tested on a PC with an 8-core Intel Xeon 2.53-GHz presented at the ACM SIGKDD, Paris, France, Jun. 29–Jul. 1, 2009. processor, but we used only single core. The movie, which has [13] A. Fujita, J. R. Sato, K. Kojima, L. R. Gomes, M. Nagasaki, M. C. 50-pixels height and width, was used in this experiment. Movie Sogayar, and S. Miyano, “Identification and quantification of granger causality between gene sets,” J. Bioinform. Comput. Biol.,vol.8,no. length (number of frames) was changed from 100 to 5000. 4, pp. 679–701, 2010. The relationship between movie length and computational [14] H. Hotelling, “Relations between two sets of variates,” Biometrica, vol. time of each method is shown in Fig. 9. The computational time 28, no. 3, pp. 321–377, 1936. of the average optical-flow methods is proportional to movie [15] X. Yin, “Canonical correlation analysis based on information theory,” J. Multivariate Anal., vol. 91, no. 2, pp. 161–176, 2004. length. On the other hand, the effect of movie length is not so [16] A. E. Hoerl and R. W. Kennard, “Ridge regression: Biased estimation large in the proposed methods, but the size of the block region for nonorthogonal problems,” Technometrics, vol. 12, no. 1, pp. 55–67, is dominant. 1970. The parallelization approach can be easily applied to reduce [17] H. D. Vinod, “Canonical ridge and econometrics of joint production,” Econometrics, vol. 4, pp. 147–166, 1976. the computational time of the proposed methods. This is because [18] B. D. Lucas and T. Kanade, “An iterative image registration technique each orientation of causal flow is independently estimated from with an application to stereo vision,” in Proc. Imag. Understand. Work- the local region of the movie. In addition, estimating the flow shop, Apr. 1981, pp. 674–679. [19] J. L. Barron, D. J. Fleet, and S. S. Beauchemin, “Performance of optical from grid points with some grid spacing is also effective. flow techniques,” Int. J. Comput. Vis., vol. 12, no. 1, pp. 43–77, 1994.

VII. CONCLUSION

In this paper, we have proposed a novel video flow represen- Yuya Yamashita received the B.S. degree in engi- tation called causal flow. Causal flow is based on the time-series neering from the University of Tokyo, Tokyo, Japan, in 2010, where he is currently pursuing the M.Sc. de- analysis method, which can quantify causal relationships among gree in information science and technology at the In- time series of pixels. In addition, we proposed a regularized telligent Systems and Informatics Laboratory, Uni- Granger causality, which increases the reliability of causal flow versity of Tokyo. His research interests include data-mining estimation by using regularization and easily enables the methods in time-series data and computer vision. block-based extension of causal flow. We evaluated the property of regularized Granger causality in synthetic data experiments, YAMASHITA et al.: CAUSAL FLOW 629

Tatsuya Harada (M’02) received the M.S. degree Yasuo Kuniyoshi (M’03) received the M.Eng. and in mechanical engineering from the University of Ph.D. degrees in engineering from the University of Tokyo, Tokyo, Japan, in 1998, and the Ph.D. degree Tokyo, Tokyo, Japan, in 1988 and 1991, respectively. in mechanical engineering from the University of Currently, he is a Professor in the Department Tokyo, Tokyo, Japan, in 2001. of Mechano-Informatics, School of Information He was an Assistant Professor in the Intelligent Science and Technology, University of Tokyo, Cooperative Systems Laboratory, University of Tokyo, Japan. From 1991 to 2000, he was a Research Tokyo, from 2001 to 2006. Currently, he is an Scientist and then a Senior Research Scientist with Associate Professor at the University of Tokyo. the Electrotechnical Laboratory, AIST, MITI, Japan. His research interests include real-world intelligent From 1996 to 1997, he was a Visiting Scholar at systems, object and scene recognition, as well as MITAILab.In2001,hewasappointedAssociate data-mining methods in time-series data. Professor at the University of Tokyo, where he has been a Professor since 2005. He has published many technical papers. His research interests include the emergence and development of embodied cognition, humanoid robot intelligence, machine understanding of human actions, and intentions. Dr. Kuniyoshi received the IJCAI–’93 Outstanding Paper Award, Best Paper Awards from Robotics Society of Japan, Sato Memorial Award for Intelligent Robotics Research, Okawa Publications Prize, Gold Medal “Tokyo Techno- Forum21” Award, and other awards.