DOCSLIB.ORG

Covariance Estimation in Two-Level Regression

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Covariance Estimation in Two-Level Regression 2nd International Conference on Control and Fault-Tolerant Systems, SysTol'13, October 9-11, 2013, Nice, France 1 Covariance Estimation in Two-Level Regression Nicholas Moehle and Dimitry Gorinevsky Abstract— This paper considers estimation of covariance ma- Using data from multiple units in a two-level data set can trices in multivariate linear regression models for two-level data improve the covariance estimation accuracy. produced by a population of similar units (individuals). The proposed Bayesian formulation assumes that the covariances Covariance matrix estimation for one-level datasets has at- for different units are sampled from a common distribution. tracted substantial attention earlier. Several approaches to the Assuming that this common distribution is Wishart, the optimal maximum likelihood estimation (MLE) with shrinkage (reg- Bayesian estimation problem is shown to be convex. This paper proposes a specialized scalable algorithm for solving this two- ularization) have been proposed, e.g., see [10], [11] where level optimal Bayesian estimation problem. The algorithm scales further references can be found. The approaches related to to datasets with thousands of units and trillions of data points this paper add regularization by using a Bayesian prior in a per unit, by solving the problem recursively, allowing new maximum a posteriori probability (MAP) estimation, such as data to be quickly incorporated into the estimates. An example an inverse Wishart prior for the covariance matrix, see [13]. problem is used to show that the proposed approach improves over existing approaches to estimating covariance matrices in The inverse Wishart is the conjugate prior for the covariance linear models for two-level data. matrix of a multivariate normal distribution. As such, the MAP solution is a generalization of the MLE solution and has the same attractive properties. In particular, the solution I. INTRODUCTION can be efficiently computed for large datasets using recursive Consider a two-level dataset least squares (RLS) or a related formulation. n oN Ti Two-level datasets are considered in multivariate analysis D = fxi(t); yi(t)gt=1 ; (1) i=1 of covariance (MANCOVA), e.g., see [17]. In MANCOVA, where index i is the dataset number, index t is the sample the null hypothesis is that all data follows the same distri- n number. Each xi(t) 2 R is a vector of independent bution. The goal is to decide if inter-unit covariance and m variables (regressors), and each yi(t) 2 R is a vector intra-unit covariance is compatible with this hypothesis. of dependent variables. There are N datasets at all; dataset Two-level regression is used in ‘soft’ applications such i includes a total of T samples. Each dataset describes a i as social sciences, biology, economics, and medicine (drug separate individual unit of the overall population. testing) [1], [2], [9], [18]. The most established solution This paper studies linear multivariate regression models of approaches are based on Gibbs sampling [8] and other the two-level data (1). For each unit, the model is described approximate methods. More scalable methods have been by the mean (regression parameters) and covariance of the developed to estimate covariance structure for two-level model residual. The means for different units are assumed to datasets using expectation-maximization (EM) methods as a be different; this is known as a model with fixed effects. The heuristic to find the maximum-likelihood estimates [3], [12]. covariances for different units are assumed to be different but related; this is a novel formulation. There appears to be little prior work on scalable two- In the motivating example considered below, the covari- level modeling applicable to machine data monitoring. One ance estimates are used in statistical monitoring to find out- exception is [6], discussing scalable algorithms for estimating liers, which are labeled as possibly anomalous data. Accurate a two-level regression model of aircraft fleet data. In [6], the covariance estimation is necessary for the monitoring to be same covariances are assumed for all units. accurate. The described two-level formulation can be useful in many other problems requiring accurate estimation of co- The contributions of this paper are as follows. First, variance matrices, such as classification, linear discriminant we propose a novel two-level linear regression formula- analysis, portfolio management, and others. tion with fixed effects in the regression parameters and in The special case N = 1 in (1) yields a standard multi- the covariances of the random effects. In this hierarchical variate regression. In that case, accurate estimation of the Bayesian formulation the unit covariances are realizations of 2 the generating Wishart distribution. Second, we show that for m elements of the covariance matrix requires that T1 m [7]. For large m this might not hold. For an ill-conditioned the proposed model, Bayesian optimal estimation for data (1) covariance matrix, even more data samples are required. yields a (nonlinear) convex optimization problem. Third, we formulate a specialized convex optimization algorithm that Nicholas Moehle is with the Department of Mechanical Engineering, solves the estimation problem using recursive updates and is Stanford University, [email protected] scalable to very large datasets. Finally, numerical examples Dimitry Gorinevsky is with Mitek Analytics LLC, Palo Alto, CA, dim- [email protected], and the Department of Electrical Engineering, Stanford demonstrate that the proposed formulation improves accu- University, [email protected] racy of the estimation compared to the known methods. II. BASELINE PROBLEM The resulting log posterior has the form This section considers a single-level dataset with N = 1, 1 1 `(B; ΣjD) = − (N + α ) log jΣj − α tr Ω Σ−1 (there is only one unit). In this section we drop the unit index 2 1 2 1 1 i = 1. The dataset (1) becomes (10) 1 fx(t); y(t)gT : (2) − tr (Y − BX)T Σ−1(Y − BX) : t=1 2 As a baseline for introducing the main contribution in the The Maximum A posteriori Probability (MAP) estimates next section, this section briefly recaps the known formu- of B and Σ minimize the negative log-posterior (10); this lation of multivariate Bayesian linear regression for dataset minimization problem is convex in the variables Σ−1 and (2). In what follows, we use p(·) for all probability density Σ−1B. The first-order optimality conditions can be solved functions and `(·) for all log likelihood functions. The analytically (e.g., see [17]), yielding meaning should be clear from the context. B = YXT (XXT )−1 (11) 1 A. Multivariate Regression Σ = (Y − BX)(Y − BX)T + α Ω : (12) N + α 1 1 Consider a linear regression model y(t) = Bx(t) + v(t), 1 where B 2 Rm×n is the regression parameter matrix and In what follows we assume that the scatter matrix XXT is v(t) 2 Rm is the residual. For data (2), the model can be invertible. In practice, selecting independent regressors leads compactly represented as to invertible XXT . Y = BX + V; (3) The estimates (11), (12) can be computed recursively. These estimates can be written in terms of the scatter matrices XXT , YXT , and YY T . For (11) this is obvious; Y = y(1) ··· y(T ) ; for (12) this requires expanding the matrix product in the X = x(1) ··· x(T ) ; (4) numerator. As additional data become available, the scatter V = v(1) ··· v(T ) : matrices can be updated using rank-one recursion. The Bayesian formulation assumes that the residuals vi(t) are independently generated by the normal distribution III. TWO-LEVEL MULTIPLE REGRESSION PROBLEM v(t) ∼ N (0; Σ); (5) The rest of the paper considers two-level dataset (1) and (3), (4) is replaced by Yi = BiXi + Vi, (i = 1;:::;N), with covariance matrix Σ 2 Sm, where Sm is the cone of where positive definite m × m matrices. The log likelihood function for Σ and B is Yi = yi(1) ··· yi(T ) ; Xi = xi(1) ··· xi(T ) ; (13) `(Σ;B j D) (6) 1 1 Vi = vi(1) ··· vi(T ) : = − T log jΣj − tr Σ−1(Y − BX)(Y − BX)T : 2 2 The parameter matrix B 2 Rm×n in (3) is considered a A. Baseline Approaches nuisance parameter; no prior for B is specified. As discussed The formulation of Section II can be applied to two-level in [11], [19], a prior for the covariance matrix Σ is needed covariance estimation in two ways described below. These if the number of samples T is insufficiently large. A well are are later used to benchmark the proposed approach. established approach, which we take as a baseline, is to use 1) Pooled Regression: The multivariate linear regression an inverse Wishart prior of Section II can be used for the pooled data Σ ∼ W−1(Ψ; ν); (7) X = X1 ··· XN Y = Y1 ··· YN : where Ψ 2 Sm is the scale matrix and the positive integer ν The obtained estimates of B and Σ are identical across the is the number of degrees of freedom. The inverse Wishart is population. The obvious deficiency of the pooled formulation the conjugate prior for the covariance matrix in a multivariate is that it ignores differences in the units. If the units are normal distribution. substantially different, the obtained model can be inaccurate. The log likelihood of the inverse Wishart prior is, up to This is illustrated in the example of Section V. additive constants, 2) Separate Regressions: The second approach is to ig- 1 1 `(ΣjΨ; ν) = − (ν + m + 1) log jΣj − tr ΨΣ−1 : (8) nore any similarity between the units. The approach of II 2 2 is then applied separately to each pair of data matrices Xi 1 and Y . For each unit i, separate and unrelated estimates of The change of variables Ω1 = ν+m+1 Ψ and α1 = ν+m+1, i transforms (8) into the log prior Bi and Σi are computed.
Load more

Recommended publications
  • New Insights Into the Statistical Properties of M-Estimators Gordana Draskovic, Frédéric Pascal
    New insights into the statistical properties of M-estimators Gordana Draskovic, Frédéric Pascal To cite this version: Gordana Draskovic, Frédéric Pascal. New insights into the statistical properties of M-estimators. IEEE Transactions on Signal Processing, Institute of Electrical and Electronics Engineers, 2018, 66 (16), pp.4253-4263. 10.1109/TSP.2018.2841892. hal-01816084 HAL Id: hal-01816084 https://hal.archives-ouvertes.fr/hal-01816084 Submitted on 26 Feb 2020 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. 1 New insights into the statistical properties of M-estimators Gordana Draskoviˇ c,´ Student Member, IEEE, Fred´ eric´ Pascal, Senior Member, IEEE Abstract—This paper proposes an original approach to bet- its explicit form, the SCM is easy to manipulate and therefore ter understanding the behavior of robust scatter matrix M- widely used in the signal processing community. estimators. Scatter matrices are of particular interest for many Nevertheless, the complex normality sometimes presents a signal processing applications since the resulting performance strongly relies on the quality of the matrix estimation. In this poor approximation of underlying physics. Noise and interfer- context, M-estimators appear as very interesting candidates, ence can be spiky and impulsive i.e., have heavier tails than mainly due to their flexibility to the statistical model and their the Gaussian distribution.
    [Show full text]
  • Robust Scatter Matrix Estimation for High Dimensional Distributions with Heavy Tail Junwei Lu, Fang Han, and Han Liu
    IEEE TRANSACTIONS ON INFORMATION THEORY 1 Robust Scatter Matrix Estimation for High Dimensional Distributions with Heavy Tail Junwei Lu, Fang Han, and Han Liu Abstract—This paper studies large scatter matrix estimation distribution family, the pair-elliptical. The pair-elliptical family for heavy tailed distributions. The contributions of this paper is strictly larger and requires less symmetry structure than the are twofold. First, we propose and advocate to use a new elliptical. We provide detailed studies on the relation between distribution family, the pair-elliptical, for modeling the high dimensional data. The pair-elliptical is more flexible and easier the pair-elliptical and several heavy tailed distribution families, to check the goodness of fit compared to the elliptical. Secondly, including the nonparanormal, elliptical, and transelliptical. built on the pair-elliptical family, we advocate using quantile- Moreover, it is easier to test the goodness of fit for the based statistics for estimating the scatter matrix. For this, we pair-elliptical. For conducting such a test, we combine the provide a family of quantile-based statistics. They outperform the existing results in low dimensions (20; 21; 22; 23; 24) with existing ones for better balancing the efficiency and robustness. In particular, we show that the propose estimators have compa- the familywise error rate controlling techinques including the rable performance to the moment-based counterparts under the Bonferonni’s correction, the Holm’s step-down procedure (25), Gaussian assumption. The method is also tuning-free compared and the higher criticism method (26; 27). to Catoni’s M-estimator for covariance matrix estimation. We Secondly, built on the pair-elliptical family, we propose further apply the method to conduct a variety of statistical a new set of quantile-based statistics for estimating scat- methods.
    [Show full text]
  • Multivariate Statistical Functions in R
    Multivariate statistical functions in R Michail T. Tsagris [email protected] College of engineering and technology, American university of the middle east, Egaila, Kuwait Version 6.1 Athens, Nottingham and Abu Halifa (Kuwait) 31 October 2014 Contents 1 Mean vectors 1 1.1 Hotelling’s one-sample T2 test ............................. 1 1.2 Hotelling’s two-sample T2 test ............................ 2 1.3 Two two-sample tests without assuming equality of the covariance matrices . 4 1.4 MANOVA without assuming equality of the covariance matrices . 6 2 Covariance matrices 9 2.1 One sample covariance test .............................. 9 2.2 Multi-sample covariance matrices .......................... 10 2.2.1 Log-likelihood ratio test ............................ 10 2.2.2 Box’s M test ................................... 11 3 Regression, correlation and discriminant analysis 13 3.1 Correlation ........................................ 13 3.1.1 Correlation coefficient confidence intervals and hypothesis testing us- ing Fisher’s transformation .......................... 13 3.1.2 Non-parametric bootstrap hypothesis testing for a zero correlation co- efficient ..................................... 14 3.1.3 Hypothesis testing for two correlation coefficients . 15 3.2 Regression ........................................ 15 3.2.1 Classical multivariate regression ....................... 15 3.2.2 k-NN regression ................................ 17 3.2.3 Kernel regression ................................ 20 3.2.4 Choosing the bandwidth in kernel regression in a very simple way . 23 3.2.5 Principal components regression ....................... 24 3.2.6 Choosing the number of components in principal component regression 26 3.2.7 The spatial median and spatial median regression . 27 3.2.8 Multivariate ridge regression ......................... 29 3.3 Discriminant analysis .................................. 31 3.3.1 Fisher’s linear discriminant function ....................
    [Show full text]
  • Multivariate L1 Statistical Methods: the Package
    JSS Journal of Statistical Software July 2011, Volume 43, Issue 5. http://www.jstatsoft.org/ Multivariate L1 Methods: The Package MNM Klaus Nordhausen Hannu Oja University of Tampere University of Tampere Abstract In the paper we present an R package MNM dedicated to multivariate data analysis based on the L1 norm. The analysis proceeds very much as does a traditional multivariate analysis. The regular L2 norm is just replaced by different L1 norms, observation vectors are replaced by their (standardized and centered) spatial signs, spatial ranks, and spatial signed-ranks, and so on. The procedures are fairly efficient and robust, and no moment assumptions are needed for asymptotic approximations. The background theory is briefly explained in the multivariate linear regression model case, and the use of the package is illustrated with several examples using the R package MNM. Keywords: least absolute deviation, mean deviation, mean difference, multivariate linear re- gression, R, shape matrix, spatial sign, spatial signed-rank, spatial rank, transformation- retransformation method. 1. Introduction Classical multivariate statistical inference methods (Hotelling's T 2, multivariate analysis of variance, multivariate regression, tests for independence, canonical correlation analysis, prin- cipal component analysis, and so on) are based on the use of the L2 norm. These standard moment-based multivariate techniques are optimal under the multivariate normality of the residuals but poor in their efficiency for heavy-tailed distributions. They are also highly sensitive to outlying observations. In this paper we present an R package MNM { available from the Comprehensive R Archive Network at http://CRAN.R-project.org/package=MNM { which uses different L1 norms and the corresponding scores (spatial signs, spatial signed- ranks, and spatial ranks) in the analysis of multivariate data.
    [Show full text]
  • Robust Estimation of Structured Scatter Matrices in (Mis)Matched Models Bruno Meriaux, Chengfang Ren, Mohammed Nabil El Korso, Arnaud Breloy, Philippe Forster
    Robust Estimation of Structured Scatter Matrices in (Mis)matched Models Bruno Meriaux, Chengfang Ren, Mohammed Nabil El Korso, Arnaud Breloy, Philippe Forster To cite this version: Bruno Meriaux, Chengfang Ren, Mohammed Nabil El Korso, Arnaud Breloy, Philippe Forster. Robust Estimation of Structured Scatter Matrices in (Mis)matched Models. Signal Processing, Elsevier, 2019, 165, pp.163-174. 10.1016/j.sigpro.2019.06.030. hal-02165848 HAL Id: hal-02165848 https://hal.archives-ouvertes.fr/hal-02165848 Submitted on 2 Jul 2019 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. Robust Estimation of Structured Scatter Matrices in (Mis)matched Models Bruno M´eriauxa,∗, Chengfang Rena, Mohammed Nabil El Korsob, Arnaud Breloyb, Philippe Forsterc aSONDRA, CentraleSup´elec, 91192 Gif-sur-Yvette, France bParis Nanterre University, LEME EA-4416, 92410 Ville d’Avray, France cParis Nanterre University, SATIE, 94230 Cachan, France Abstract Covariance matrix estimation is a ubiquitous problem in signal processing. In most modern signal processing applications, data are generally modeled by non-Gaussian distributions with covariance matrices exhibiting a particular structure. Taking into account this struc- ture and the non-Gaussian behavior improve drastically the estimation accuracy.
    [Show full text]
  • Minimum Covariance Determinant and Extensions
    Minimum Covariance Determinant and Extensions Mia Hubert,∗ Michiel Debruyne,y Peter J. Rousseeuw∗ September 22, 2017 Abstract The Minimum Covariance Determinant (MCD) method is a highly robust estimator of multivariate location and scatter, for which a fast algorithm is available. Since estimating the covariance matrix is the cornerstone of many multivariate statistical methods, the MCD is an important building block when developing robust multivariate techniques. It also serves as a convenient and efficient tool for outlier detection. The MCD estimator is reviewed, along with its main properties such as affine equivariance, breakdown value, and influence function. We discuss its computation, and list applications and extensions of the MCD in applied and methodological multivariate statistics. Two recent extensions of the MCD are described. The first one is a fast deterministic algorithm which inherits the robustness of the MCD while being almost affine equivariant. The second is tailored to high-dimensional data, possibly with more dimensions than cases, and incorporates regularization to prevent singular matrices. arXiv:1709.07045v1 [stat.ME] 20 Sep 2017 ∗Department of Mathematics, KU Leuven, Celestijnenlaan 200B, BE-3001 Leuven, Belgium yDexia Bank, Belgium INTRODUCTION The Minimum Covariance Determinant (MCD) estimator is one of the first affine equivari- ant and highly robust estimators of multivariate location and scatter1,2. Being resistant to outlying observations makes the MCD very useful for outlier detection. Although already in- troduced in 1984, its main use has only started since the construction of the computationally efficient FastMCD algorithm of3 in 1999. Since then, the MCD has been applied in numerous fields such as medicine, finance, image analysis and chemistry.
    [Show full text]
  • Estimation of the Inverse Scatter Matrix for a Scale Mixture of Wishart
    Estimation of the inverse scatter matrix for a scale mixture of Wishart matrices under Efron-Morris type losses Djamila Boukehil, Dominique Fourdrinier, Fatiha Mezoued, William Strawderman To cite this version: Djamila Boukehil, Dominique Fourdrinier, Fatiha Mezoued, William Strawderman. Estimation of the inverse scatter matrix for a scale mixture of Wishart matrices under Efron-Morris type losses. 2020. hal-02503972 HAL Id: hal-02503972 https://hal.archives-ouvertes.fr/hal-02503972 Preprint submitted on 10 Mar 2020 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. Estimation of the inverse scatter matrix for a scale mixture of Wishart matrices under Efron-Morris type losses Djamila Boukehil ∗ Dominique Fourdrinier y Fatiha Mezoued z William E. Strawderman x February 17, 2020 Abstract We consider estimation of the inverse scatter matrix Σ−1 for a scale mixture of Wishart matrices under various Efron-Morris type losses, tr[fΣ^ −1 − Σ−1g2Sk] for k = 0; 1; 2:::, where S is the sample covariance matrix. We improve on the standard estimators a S+, where S+ denotes the Moore-Penrose inverse of S and a is a positive constant, through an unbiased estimator of the risk difference between the new estimators and a S+.
    [Show full text]
  • Robust Estimation of Scatter Matrix, Random Matrix Theory and an Application to Spectrum Sensing
    Robust Estimation of Scatter Matrix, Random Matrix Theory and an Application to Spectrum Sensing Thesis by Zhedong Liu In Partial Fulfillment of the Requirements For the Degree of Master of Science King Abdullah University of Science and Technology Thuwal, Kingdom of Saudi Arabia April,2019 2 EXAMINATION COMMITTEE PAGE The thesis of Zhedong Liu is approved by the examination committee Committee Chairperson: Mohamed-Slim Alouini Committee Members: Mohamed-Slim Alouini, Abla Kammoun , H˚avard Rue 3 ©April,2019 Zhedong Liu All Rights Reserved 4 ABSTRACT Robust Estimation of Scatter Matrix, Random Matrix Theory and Applications to Spectrum Sensing Zhedong Liu The covariance estimation is one of the most critical tasks in multivariate statisti- cal analysis. In many applications, reliable estimation of the covariance matrix, or scatter matrix in general, is required. The performance of the classical maximum likelihood method relies a great deal on the validity of the model assumption. Since the assumptions are often approximately correct, many robust statistical methods have been proposed to be robust against the deviation from the model assumptions. M-estimator is an important class of robust estimator of the scatter matrix. The properties of these robust estimators under high dimensional setting, which means the number of dimensions has the same order of magnitude as the number of obser- vations, is desirable. To study these, random matrix theory is a very important tool. With high dimensional properties of robust estimators, we introduced a new method for blind spectrum sensing in cognitive radio networks. 5 ACKNOWLEDGEMENTS First of all, I would like to thank my supervisor Prof.
    [Show full text]
  • Asymptotic Properties of Robust Complex Covariance Matrix Estimates
    1 Asymptotic properties of robust complex covariance matrix estimates Mélanie Mahot Student, IEEE, Frédéric Pascal Member, IEEE, Philippe Forster Member, IEEE, Jean-Philippe Ovarlez Member, IEEE Abstract—In many statistical signal processing applications, [11], [12]. M-estimators of the covariance matrix are however the estimation of nuisance parameters and parameters of interest seldom used in the signal processing community. Only a is strongly linked to the resulting performance. Generally, these limited case, the Tyler’s estimator [13] also called the Fixed applications deal with complex data. This paper focuses on covariance matrix estimation problems in non-Gaussian envi- Point Estimator [14] has been widely used as an alternative to ronments and particularly, the M-estimators in the context of the SCM for radar applications. Concerning the M-estimators, elliptical distributions. Firstly, this paper extends to the complex notable exceptions are the recent papers by Ollila [15], [16], case the results of Tyler in [1]. More precisely, the asymptotic [17], [18], [19] who advocates their use in several applications distribution of these estimators as well as the asymptotic distribu- such as array processing. The M-estimators have also been tion of any homogeneous function of degree 0 of the M-estimates are derived. On the other hand, we show the improvement of studied in the case of large datasets, where the dimension of such results on two applications: DOA (directions of arrival) the data is of the same order as the dimension of the sample estimation using the MUSIC (MUltiple SIgnal Classification) [20]. algorithm and adaptive radar detection based on the ANMF One possible reason for this lack of interest is that their sta- (Adaptive Normalized Matched Filter) test.
    [Show full text]
  • Scatter Matrices and Independent Component Analysis 1 Introduction
    AUSTRIAN JOURNAL OF STATISTICS Volume 35 (2006), Number 2&3, 175–189 Scatter Matrices and Independent Component Analysis Hannu Oja1, Seija Sirkia¨2, and Jan Eriksson3 1University of Tampere, Finland 2University of Jyvaskyl¨ a,¨ Finland 3Helsinki University of Technology, Finland Abstract:In the independent component analysis (ICA) it is assumed that the components of the multivariate independent and identically distributed ob- servations are linear transformations of latent independent components. The problem then is to find the (linear) transformation which transforms the ob- servations back to independent components. In the paper the ICA is discussed and it is shown that, under some mild assumptions, two scatter matrices may be used together to find the independent components. The scatter matrices must then have the so called independence property. The theory is illustrated by examples. Keywords: Affine Equivariance, Elliptical Model, Independence, Indepen- dent Component Model, Kurtosis, Location, Principal Component Analysis (PCA), Skewness, Source Separation. 1 Introduction Let x1; x2;:::; xn denote a random sample from a p-variate distribution. We also write 0 X = (x1 x2 ¢ ¢ ¢ xn) for the corresponding n £ p data matrix. In statistical modelling of the observed data, one often assumes that the observations xi are independent p-vectors ”generated” by the model xi = Azi + b ; i = 1; : : : ; n ; where the zi’s are called standardized vectors, b is a location p-vector, A is a full-rank p £ p transformation matrix and V = AA0 is a positive definite p £ p (P DS(p)) scatter matrix. In most applications (two-samples, several-samples case, linear model, etc.), b = bi may be dependent on the design.
    [Show full text]
  • Tools for Exploring Multivariate Data: the Package ICS
    Tools for Exploring Multivariate Data: The Package ICS Klaus Nordhausen Hannu Oja David E. Tyler University of Tampere University of Tampere Rutgers, The State University of New Jersey Abstract This introduction to the R package ICS is a (slightly) modified version of Nordhausen, Oja, and Tyler(2008c), published in the Journal of Statistical Software. Invariant coordinate selection (ICS) has recently been introduced as a method for exploring multivariate data. It includes as a special case a method for recovering the unmixing matrix in independent components analysis (ICA). It also serves as a basis for classes of multivariate nonparametric tests, and as a tool in cluster analysis or blind discrimination. The aim of this paper is to briefly explain the (ICS) method and to illustrate how various applications can be implemented using the R package ICS. Several examples are used to show how the ICS method and ICS package can be used in analyzing a multivariate data set. Keywords: clustering, discriminant analysis, independent components analysis, invariant co- ordinate selection, R, transformation-retransformation method. 1. Introduction Multivariate data normally arise by collecting p measurements on n individuals or experimen- tal units. Such data can be displayed in tables with each row representing one individual and each column representing a particular measured variable. The resulting data matrix X is then p n × p, with the row vector xi 2 < denoting the measurements taken on the ith individual or > > > ith experimental unit. Hence X = [x1 ; : : : ; xn ]. To be consistent with the convention used in the programming language R, all vectors are understood to be row vectors.
    [Show full text]