Canonical Correlation a Tutorial Magnus Borga January 12, 2001 Contents 1 About this tutorial 1 2 Introduction 2 3 Definition 2 4 Calculating canonical correlations 3 5 Relating topics 3 5.1 The difference between CCA and ordinary correlation analysis . 3 5.2Relationtomutualinformation................... 4 5.3 Relation to other linear subspace methods ............. 4 5.4RelationtoSNR........................... 5 5.4.1 Equalnoiseenergies.................... 5 5.4.2 Correlation between a signal and the corrupted signal . 6 A Explanations 6 A.1Anoteoncorrelationandcovariancematrices........... 6 A.2Affinetransformations....................... 6 A.3Apieceofinformationtheory.................... 7 A.4 Principal component analysis . ................... 9 A.5 Partial least squares ......................... 9 A.6 Multivariate linear regression . ................... 9 A.7Signaltonoiseratio......................... 10 1 About this tutorial This is a printable version of a tutorial in HTML format. The tutorial may be modified at any time as will this version. The latest version of this tutorial is available at http://people.imt.liu.se/˜magnus/cca/. 1 2 Introduction Canonical correlation analysis (CCA) is a way of measuring the linear relationship between two multidimensional variables. It finds two bases, one for each variable, that are optimal with respect to correlations and, at the same time, it finds the corresponding correlations. In other words, it finds the two bases in which the correlation matrix between the variables is diagonal and the correlations on the diagonal are maximized. The dimensionality of these new bases is equal to or less than the smallest dimensionality of the two variables. An important property of canonical correlations is that they are invariant with respect to affine transformations of the variables. This is the most important differ- ence between CCA and ordinary correlation analysis which highly depend on the basis in which the variables are described. CCA was developed by H. Hotelling [10]. Although being a standard tool in statistical analysis, where canonical correlation has been used for example in economics, medical studies, meteorology and even in classification of malt whisky, it is surprisingly unknown in the fields of learning and signal processing. Some exceptionsare[2,13,5,4,14], For further details and applications in signal processing, see my PhD thesis [3] and other publications. 3 Definition Canonical correlation analysis can be defined as the problem of finding two sets of Ý basis vectors, one for Ü and the other for , such that the correlations between the projections of the variables onto these basis vectors are mutually maximized. Let us look at the case where only one pair of basis vectors are sought, namely the ones corresponding to the largest canonical correlation: Consider the linear Ì Ì Ü = Ü Û^ Ý = Ý Û^ Ý combinations Ü and of the two variables respectively. This means that the function to be maximized is Ì Ì ÜÝ Û^ ] E [Û^ E [ÜÝ ] Ý Ü Ô Õ = = ¾ ¾ Ì Ì Ì Ì E [Ü ]E [Ý ] E [Û^ ÜÜ Û^ ]E [Û^ ÝÝ Û^ ] Ü Ý Ü Ý (1) Ì Û C Û ÜÝ Ý Ü Õ = : Ì Ì Û C Û Û C Û ÜÜ Ü ÝÝ Ý Ü Ý Û Û Ý The maximum of with respect to Ü and is the maximum canonical correlation. The subsequent canonical correlations are uncorrelated for different solutions, i.e. 8 Ì Ì Ì > E [Ü Ü ] = E [Û ÜÜ Û ]=Û C Û =¼ i j Üj ÜÜ Üj Üi Üi < Ì Ì Ì i 6= j: E [Ý Ý ] = E [Û ÝÝ Û ]=Û C Û =¼ j Ýj ÝÝ Ýj i for (2) Ýi Ýi > : Ì Ì Ì E [Ü Ý ] = E [Û ÜÝ Û ]=Û C Û =¼ i j Ýj ÜÝ Ýj Üi Üi 2 Û Û Ü Ý Ý The projections onto Ü and ,i.e. and , are called canonical variates. 4 Calculating canonical correlations Ý Consider two random variables Ü and with zero mean. The total covariance # matrix " Ì Ü Ü C C ÜÜ ÜÝ = E = C (3) Ý Ý C C ÝÜ ÝÝ C C Ü ÜÜ is a block matrix where ÜÜ and are the within-sets covariance matrices of Ì Ý C = C and respectively and ÜÝ is the between-sets covariance matrix. ÝÜ Ý The canonical correlations between Ü and can be found by solving the eigen- value equations ´ ½ ½ ¾ C C C C Û^ = Û^ ÜÝ ÝÜ Ü Ü ÝÝ ÜÜ (4) ½ ½ ¾ Û^ C C C C Û^ = Ý ÝÜ ÜÝ Ý ÝÝ ÜÜ ¾ where the eigenvalues are the squared canonical correlations and the eigen- Û^ Û^ Ý vectors Ü and are the normalized canonical correlation basis vectors.The number of non-zero solutions to these equations are limited to the smallest dimen- Ý Ü Ý sionality of Ü and . E.g. if the dimensionality of and is 8 and 5 respectively, the maximum number of canonical correlations is 5. Only one of the eigenvalue equations needs to be solved since the solutions are related by 8 < C Û^ = C Û^ ÜÝ Ý Ü ÜÜ Ü (5) : C Û^ = C Û^ ; ÝÜ Ü Ý ÝÝ Ý where × Ì C Û^ Û^ ÝÝ Ý Ý ½ = = : Ü (6) Ý Ì Û^ C Û^ ÜÜ Ü Ü 5 Relating topics 5.1 The difference between CCA and ordinary correlation analysis Ordinary correlation analysis is dependent on the coordinate system in which the variables are described. This means that even if there is a very strong linear rela- tionship between two multidimensional signals, this relationship may not be visible in a ordinary correlation analysis if one coordinate system is used, while in another coordinate system this linear relationship would give a very high correlation. CCA finds the coordinate system that is optimal for correlation analysis, and the eigenvectors of equation 4 defines this coordinate system. 3 Ý Example: Consider two normally distributed two-dimensional variables Ü and Ý · Ý = Ü · Ü ¾ ½ ¾ with unit variance. Let ½ . It is easy to confirm that the correlation Ý matrix between Ü and is ¼:5 ¼:5 : Ê = ÜÝ (7) ¼:5 ¼:5 This indicates a relatively weak correlation of 0.5 despite the fact that there is a Ý perfect linear relationship (in one dimension) between Ü and . A CCA on this data shows that the largest (and only) canonical correlation is Ì one and it also gives the direction [½½] in which this perfect linear relationship lies. If the variables are described in the bases given by the canonical correlation basis vectors (i.e. the eigenvectors of equation 4), the correlation matrix between the variables is ½ ¼ : Ê = ÜÝ (8) ¼ ½ 5.2 Relation to mutual information There is a relation between correlation and mutual information. Since informa- tion is additive for statistically independent variables and the canonical variates Ý are uncorrelated, the mutual information between Ü and is the sum of mutual Ü Ý i information between the variates i and if there are no higher order statistic de- pendencies than correlation (second-order statistics). For Gaussian variables this means X ½ ½ ½ ½ É = : Á ´Ü; Ý µ= ÐÓg ÐÓg (9) ¾ ¾ ¾ ¾ ´½ µ ´½ µ i i i i Kay [13] has shown that this relation plus a constant holds for all elliptically sym- metrical distributions of the form Ì ½ ´´Þ Þ µ C ´Þ Þµµ: cf (10) 5.3 Relation to other linear subspace methods Instead of the two eigenvalue equations in 4 we can formulate the problem in one single eigenvalue equation: ½ A^Û = Û^ B (11) where ¼ C Û^ C ¼ ÜÝ Ü Ü ÜÜ A = Û^ = : B = ; and (12) C ¼ Û^ ¼ C ÝÜ Ý Ý ÝÝ Solving the eigenproblem in equation 11 with slightly different matrices will give solutions to principal component analysis (PCA), partial least squares (PLS) and multivariate linear regression (MLR). The matrices are listed in table 1. 4 A B C Á PCA ÜÜ ¼ C Á ¼ PLS ÜÝ C ¼ ¼ Á ÝÜ ¼ C C ¼ ÜÜ CCA ÜÝ C ¼ ¼ C ÝÜ ÝÝ ¼ C C ¼ ÜÜ MLR ÜÝ C ¼ ¼ Á ÝÜ B Table 1: The matrices A and for PCA, PLS, CCA and MLR. 5.4 Relation to SNR Correlation is strongly related to signal to noise ratio (SNR), which is a more com- monly used measure in signal processing. Consider a signal Ü and two noise signals 1 ¾ ½ and all having zero mean and all being uncorrelated with each other. Let ¾ ¾ Ë = E [Ü ] Æ = E [ ] and i be the energy of the signal and the noise signals i a´Ü · µ b´Ü · µ ¾ respectively. Then the correlation between ½ and is E [a´Ü · µb´Ü · µ] ½ ¾ Ô = ¾ ¾ ¾ ¾ E [a ´Ü · µ ] E [ b ´Ü · µ ] ½ ¾ ¢ £ ¾ E Ü Õ = ¢ £¡ ¢ £¡ (13) ¾ ¾ ¾ ¾ E [Ü ]·E E [Ü ]·E ½ ¾ Ë Ô = : ´Ë · Æ µ´Ë · Æ µ ½ ¾ b Note that the amplification factors a and do not affect the correlation or the SNR. 5.4.1 Equal noise energies In the special case where the noise energies are equal, i.e. Æ = Æ = Æ ¾ ½ , equation 13 can be written as Ë = : (14) Ë · Æ This means that the SNR can be written as Ë : = (15) Æ ½ 1The assumption of zero mean is for convenience. A non-zero mean does not affect the SNR or the correlation. 5 Here, it should be noted that the noise affects the signal twice, so this relation between SNR and correlation is perhaps not so intuitive. This relation is illustrated in figure 1 (top). 5.4.2 Correlation between a signal and the corrupted signal Æ =¼ Æ = Æ ¾ Another special case is when ½ and . Then, the correlation between a signal and a noise-corrupted version of that signal is Ë Ô = : (16) Ë ´Ë · Æ µ In this case, the relation between SNR and correlation is ¾ Ë : = (17) ¾ Æ ½ This relation between correlation and SNR is illustrated in figure 1 (bottom).