DOCSLIB.ORG

Lecture 3. Transpositions, Kronecker Products, and Contractions

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Lecture 3. Transpositions, Kronecker Products, and Contractions From Matrix to Tensor: The Transition to Numerical Multilinear Algebra Lecture 3. Transpositions, Kronecker Products, and Contractions Charles F. Van Loan Cornell University The Gene Golub SIAM Summer School 2010 Selva di Fasano, Brindisi, Italy ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Where We Are Lecture 1. Introduction to Tensor Computations Lecture 2. Tensor Unfoldings Lecture 3. Transpositions, Kronecker Products, Contractions Lecture 4. Tensor-Related Singular Value Decompositions Lecture 5. The CP Representation and Rank Lecture 6. The Tucker Representation Lecture 7. Other Decompositions and Nearness Problems Lecture 8. Multilinear Rayleigh Quotients Lecture 9. The Curse of Dimensionality Lecture 10. Special Topics ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions What is this Lecture About? Preview Now we turn our attention to the engagement of these multidimensional data objects in two elementary operations. Transposition. While there is only one way transpose a matrix there are an exponential number of ways to transpose an order-d tensor. This will require (surprise surprise) more facility with multi-indexing. Contraction. The contraction of two tensors can be regarded as a generalization of matrix-matrix multiplication. Analogous to how the serious business of matrix decompositions builds on matrix multiplication, we will find that tensor contractions have a foundational role to play in tensor computations. ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions What is this Lecture About? More “Bridging the Gap” The Kronecker product helps bridge the gap between matrix computations and tensor computations. For example, the contraction between two tensors can sometimes be “reshaped” into a matrix computation that involves Kronecker products. So in advance of our introduction to tensor contractions, we will get familiar with this all-important matrix operation and some of its nearby “cousins.” ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Tensor Transposition Example: A(1:4, 1:5, 1:6) → B(1:5, 1:6, 1:4) A = randn([ 4 5 6]); B = zeros([5 6 4]); for i1=1:4 for i2 = 1:5 for i3 = 1:6 B(i2,i3,i1) = A(i1,i2,i3); end end end How to say this: B = A< [ 2 3 1 ] >. ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Tensor Transposition: The 3rd Order Case Six possibilities... If A ∈ IRn1×n2×n3 , then there are 6 = 3! possible transpositions identified by the notation A< [i j k] > where [i j k] is a permutation of [1 2 3]: A< [1 2 3] > b ijk < [1 3 2] > A bikj < [2 1 3] > A bjik B = =⇒ = aijk A< [2 3 1] > b jki A< [3 1 2] > b kij < [3 2 1] > A bkji for i = 1:n1, j = 1:n2, k = 1:n3. ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Tensor Transposition: General Case Definition n1×···×nd If A ∈ IR and p = [p1,..., pd ] is a permutation of the <p> n ×···×n index vector 1:d, then A ∈ IR p1 pd denotes the p-transpose of A and is defined by <p> A (jp1 ,..., jpd ) = A(j1,..., jd ) where 1 ≤ jk ≤ nk for k = 1:d. ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Matlab Tensor Toolbox: Transposition % If A is an order -d tensor and % p is a permutation of 1:d, then % the following assigns the p- transpose % of A to B ... B = permute(A,p) Problem 3.1. Any square matrix can be written as the sum of a symmetric matrix and a skew-symmetric matrix: „ A + AT « „ A − AT « A = + 2 2 Formulate a generalization of this result for order-3 tensors and write a Matlab function that embodies your idea. ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Tensor Notation: Subscript Vectors (Review) Reference n1×···×nd If A ∈ IR and i = (i1,..., id ) with 1 ≤ ik ≤ nk for k = 1:d, then A(i) ≡ A(i1,..., ik ) We say that i is a subscript vector. Bold font will be used designate subscript vectors. Bounds If L and R are subscript vectors having the same dimension, then L ≤ R means that Lk ≤ Rk for all k. Special Cases A subscript vector of all ones is denoted by 1. (Dimension clear from context.) If N is an integer, then N = N · 1. ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Tensor Transposition: General Case Definition n1×···×nd If A ∈ IR and p = [p1,..., pd ] is a permutation of the <p> n ×···×n index vector 1:d, then A ∈ IR p1 pd denotes the p-transpose of A and is defined by A<p>(j(p)) = A(j) 1 ≤ j ≤ n. ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Problem 3.2. Verify that if p and q are both permutations of 1:d, then `A<p>´<q> = A<p(q)> where A ∈ IRn1×···×nd . Problem 3.3. Suppose A ∈ IRn1×···×nd and that B = A[ p ] where p is a permutation of 1:d. Specify a permutation matrix P so that B(k) = A(p(k))P. ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions What is a Kronecker Product? Definition B ⊗ C is a block matrix whose ij-th block is bij C. An Example... " # " # b11 b12 b11C b12C ⊗ C = b21 b22 b21C b22C Replicated Block Structure ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions What Is a Kronecker Product? There are three ways to regard A = B ⊗ C... If b11 ··· b1n c11 ··· c1q . .. ⊗ . .. A = . . bm1 ··· bmn cp1 ··· cpq then (i). A is an m-by-n block matrix with p-by-q blocks. (ii). A is an mp-by-nq matrix of scalars. (iii). A is an unfolded 4-th order tensor A ∈ IRp×q×m×n: A(i1, i2, i3, i4) = B(i3, i4)C(i1, i2) ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions All Possible Entry-Entry Products 4 × 9 = 36 c11 c12 c13 b11 b12 ⊗ c21 c22 c23 b21 b22 c31 c32 c33 = b11c11 b11c12 b11c13 b12c11 b12c12 b12c13 b11c21 b11c22 b11c23 b12c21 b12c22 b12c23 b11c31 b11c32 b11c33 b12c31 b12c32 b12c33 b21c11 b21c12 b21c13 b22c11 b22c12 b22c13 b21c21 b21c22 b21c23 b22c21 b22c22 b22c23 b21c31 b21c32 b21c33 b22c31 b22c32 b22c33 ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Hierarchical Kronecker Products of Kronecker Products B ⊗ C ⊗ D = c11 c12 c13 c14 d11 d12 d13 b11 b12 c21 c22 c23 c24 ⊗ ⊗ d21 d22 d23 b21 b22 c31 c32 c33 c34 d31 d32 d33 c41 c42 c43 c44 A 2-by-2 block matrix whose entries are 4-by-4 block matrices whose entries are 3-by-3 matrices. ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Kronecker Product Properties Some Basic Facts... (B ⊗ C)T = BT ⊗ C T (B ⊗ C)−1 = B−1 ⊗ C −1 (B ⊗ C)(D ⊗ F ) = BD ⊗ CF B ⊗ (C ⊗ D) = (B ⊗ C) ⊗ D Note that B ⊗ C 6= C ⊗ B. However, it can be shown that C ⊗ B = PT (B ⊗ C)Q where P and Q are perfect shuffle permutations. More later. ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Kronecker Products and Structure Structured Factors Produce a Structured Product... B and C B ⊗ C Nonsingular Nonsingular Lower(Upper) triangular Lower(Upper) Triangular Banded Block Banded Symmetric Symmetric Positive Definite Positive Definite Stochastic Stochastic Toeplitz Block Toeplitz Permutation Permutation Orthogonal Orthogonal ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Kronecker Products and Data Sparsity Example 1. If B, C ∈ IRm×m and A ≈ B ⊗ C 2 then we are using O(m ) numbers ( the bij and cij ) to 4 represent/approximate O(m ) data (the aij .) Example 2. m×m If Bij ∈ IR and r X A ≈ Bk,1 ⊗ Bk,2 ⊗ ··· ⊗ Bk,d k=1 then we are using rdm2 numbers to represent/approximate a matrix with m2d entries. ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Kronecker Products and Fast Matrix-Vector Multiplication A Basic Reshaping Result... Y = CXBT ⇔ vec(Y ) = (B ⊗ C)vec(X ) In Practice... function y = KronTimesVector(B,C,y) % y = kron(B,C)*x [mb,nb] = size(B); [mc,nc] = size(C); Y = C*reshape(x,nc,nb)*B’; y = reshape(Y,mc*mb,1); If both matrices are n-by-n, then work is O(n3). Ordinarily, an n2-by-n2 matrix-vector product would be O(n4). ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Problem 3.4. How would you compute the matrix-vector product q×q pqr y = (Ip ⊗ A ⊗ Ir )x where A ∈ IR and x ∈ IR ? Assess the amount of work. Problem 3.5. How would you compute the matrix-vector product n×n n3 y = (A1 ⊗ A2 ⊗ A3)x where Ai ∈ IR and x ∈ IR ?. Hint: (A1 ⊗ A2 ⊗ A3) = (A1 ⊗ In2 )(In ⊗ A2 ⊗ In)(In2 ⊗ A3) Assess the amount of work. Problem 3.6. How would you compute the matrix-vector product n×n n3 y = (A1 ⊗ ··· ⊗ Ad )x where Ai ∈ IR and x ∈ IR ?. Assess the amount of work. ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Kronecker Products and Matrix Factorizations Just take the Kronecker Product of the Factors... LU: (PB ⊗ PC )(B ⊗ C) = (LB ⊗ LC )(UB ⊗ UC ) T Cholesky: B ⊗ C = (LB ⊗ LC )(LB ⊗ LC ) T Schur: (QB ⊗ QC ) (B ⊗ C)(QB ⊗ QC ) = TB ⊗ TC QR: B ⊗ C = (QB ⊗ QC )(RB ⊗ RC ) T SVD: B ⊗ C = (UB ⊗ UC )(ΣB ⊗ ΣC )(VB ⊗ VC ) There are some annoying details if the matrices are rectangular, e.g., (ΣB ⊗ ΣC ) needs to be permuted to obtain a true diagonal form.
Load more

Recommended publications
  • Multivector Differentiation and Linear Algebra 0.5Cm 17Th Santaló
    Multivector differentiation and Linear Algebra 17th Santalo´ Summer School 2016, Santander Joan Lasenby Signal Processing Group, Engineering Department, Cambridge, UK and Trinity College Cambridge [email protected], www-sigproc.eng.cam.ac.uk/ s jl 23 August 2016 1 / 78 Examples of differentiation wrt multivectors. Linear Algebra: matrices and tensors as linear functions mapping between elements of the algebra. Functional Differentiation: very briefly... Summary Overview The Multivector Derivative. 2 / 78 Linear Algebra: matrices and tensors as linear functions mapping between elements of the algebra. Functional Differentiation: very briefly... Summary Overview The Multivector Derivative. Examples of differentiation wrt multivectors. 3 / 78 Functional Differentiation: very briefly... Summary Overview The Multivector Derivative. Examples of differentiation wrt multivectors. Linear Algebra: matrices and tensors as linear functions mapping between elements of the algebra. 4 / 78 Summary Overview The Multivector Derivative. Examples of differentiation wrt multivectors. Linear Algebra: matrices and tensors as linear functions mapping between elements of the algebra. Functional Differentiation: very briefly... 5 / 78 Overview The Multivector Derivative. Examples of differentiation wrt multivectors. Linear Algebra: matrices and tensors as linear functions mapping between elements of the algebra. Functional Differentiation: very briefly... Summary 6 / 78 We now want to generalise this idea to enable us to find the derivative of F(X), in the A ‘direction’ – where X is a general mixed grade multivector (so F(X) is a general multivector valued function of X). Let us use ∗ to denote taking the scalar part, ie P ∗ Q ≡ hPQi. Then, provided A has same grades as X, it makes sense to define: F(X + tA) − F(X) A ∗ ¶XF(X) = lim t!0 t The Multivector Derivative Recall our definition of the directional derivative in the a direction F(x + ea) − F(x) a·r F(x) = lim e!0 e 7 / 78 Let us use ∗ to denote taking the scalar part, ie P ∗ Q ≡ hPQi.
    [Show full text]
  • Appendix a Spinors in Four Dimensions
    Appendix A Spinors in Four Dimensions In this appendix we collect the conventions used for spinors in both Minkowski and Euclidean spaces. In Minkowski space the flat metric has the 0 1 2 3 form ηµν = diag(−1, 1, 1, 1), and the coordinates are labelled (x ,x , x , x ). The analytic continuation into Euclidean space is madethrough the replace- ment x0 = ix4 (and in momentum space, p0 = −ip4) the coordinates in this case being labelled (x1,x2, x3, x4). The Lorentz group in four dimensions, SO(3, 1), is not simply connected and therefore, strictly speaking, has no spinorial representations. To deal with these types of representations one must consider its double covering, the spin group Spin(3, 1), which is isomorphic to SL(2, C). The group SL(2, C) pos- sesses a natural complex two-dimensional representation. Let us denote this representation by S andlet us consider an element ψ ∈ S with components ψα =(ψ1,ψ2) relative to some basis. The action of an element M ∈ SL(2, C) is β (Mψ)α = Mα ψβ. (A.1) This is not the only action of SL(2, C) which one could choose. Instead of M we could have used its complex conjugate M, its inverse transpose (M T)−1,or its inverse adjoint (M †)−1. All of them satisfy the same group multiplication law. These choices would correspond to the complex conjugate representation S, the dual representation S,and the dual complex conjugate representation S. We will use the following conventions for elements of these representations: α α˙ ψα ∈ S, ψα˙ ∈ S, ψ ∈ S, ψ ∈ S.
    [Show full text]
  • Block Kronecker Products and the Vecb Operator* CORE View
    View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by Elsevier - Publisher Connector Block Kronecker Products and the vecb Operator* Ruud H. Koning Department of Economics University of Groningen P.O. Box 800 9700 AV, Groningen, The Netherlands Heinz Neudecker+ Department of Actuarial Sciences and Econometrics University of Amsterdam Jodenbreestraat 23 1011 NH, Amsterdam, The Netherlands and Tom Wansbeek Department of Economics University of Groningen P.O. Box 800 9700 AV, Groningen, The Netherlands Submitted hv Richard Rrualdi ABSTRACT This paper is concerned with two generalizations of the Kronecker product and two related generalizations of the vet operator. It is demonstrated that they pairwise match two different kinds of matrix partition, viz. the balanced and unbalanced ones. Relevant properties are supplied and proved. A related concept, the so-called tilde transform of a balanced block matrix, is also studied. The results are illustrated with various statistical applications of the five concepts studied. *Comments of an anonymous referee are gratefully acknowledged. ‘This work was started while the second author was at the Indian Statistiral Institute, New Delhi. LINEAR ALGEBRA AND ITS APPLICATIONS 149:165-184 (1991) 165 0 Elsevier Science Publishing Co., Inc., 1991 655 Avenue of the Americas, New York, NY 10010 0024-3795/91/$3.50 166 R. H. KONING, H. NEUDECKER, AND T. WANSBEEK INTRODUCTION Almost twenty years ago Singh [7] and Tracy and Singh [9] introduced a generalization of the Kronecker product A@B. They used varying notation for this new product, viz. A @ B and A &3B. Recently, Hyland and Collins [l] studied the-same product under rather restrictive order conditions.
    [Show full text]
  • Tensor Manipulation in GPL Maxima
    Tensor Manipulation in GPL Maxima Viktor Toth http://www.vttoth.com/ February 1, 2008 Abstract GPL Maxima is an open-source computer algebra system based on DOE-MACSYMA. GPL Maxima included two tensor manipulation packages from DOE-MACSYMA, but these were in various states of disrepair. One of the two packages, CTENSOR, implemented component-based tensor manipulation; the other, ITENSOR, treated tensor symbols as opaque, manipulating them based on their index properties. The present paper describes the state in which these packages were found, the steps that were needed to make the packages fully functional again, and the new functionality that was implemented to make them more versatile. A third package, ATENSOR, was also implemented; fully compatible with the identically named package in the commercial version of MACSYMA, ATENSOR implements abstract tensor algebras. 1 Introduction GPL Maxima (GPL stands for the GNU Public License, the most widely used open source license construct) is the descendant of one of the world’s first comprehensive computer algebra systems (CAS), DOE-MACSYMA, developed by the United States Department of Energy in the 1960s and the 1970s. It is currently maintained by 18 volunteer developers, and can be obtained in source or object code form from http://maxima.sourceforge.net/. Like other computer algebra systems, Maxima has tensor manipulation capability. This capability was developed in the late 1970s. Documentation is scarce regarding these packages’ origins, but a select collection of e-mail messages by various authors survives, dating back to 1979-1982, when these packages were actively maintained at M.I.T. When this author first came across GPL Maxima, the tensor packages were effectively non-functional.
    [Show full text]
  • Handout 9 More Matrix Properties; the Transpose
    Handout 9 More matrix properties; the transpose Square matrix properties These properties only apply to a square matrix, i.e. n £ n. ² The leading diagonal is the diagonal line consisting of the entries a11, a22, a33, . ann. ² A diagonal matrix has zeros everywhere except the leading diagonal. ² The identity matrix I has zeros o® the leading diagonal, and 1 for each entry on the diagonal. It is a special case of a diagonal matrix, and A I = I A = A for any n £ n matrix A. ² An upper triangular matrix has all its non-zero entries on or above the leading diagonal. ² A lower triangular matrix has all its non-zero entries on or below the leading diagonal. ² A symmetric matrix has the same entries below and above the diagonal: aij = aji for any values of i and j between 1 and n. ² An antisymmetric or skew-symmetric matrix has the opposite entries below and above the diagonal: aij = ¡aji for any values of i and j between 1 and n. This automatically means the digaonal entries must all be zero. Transpose To transpose a matrix, we reect it across the line given by the leading diagonal a11, a22 etc. In general the result is a di®erent shape to the original matrix: a11 a21 a11 a12 a13 > > A = A = 0 a12 a22 1 [A ]ij = A : µ a21 a22 a23 ¶ ji a13 a23 @ A > ² If A is m £ n then A is n £ m. > ² The transpose of a symmetric matrix is itself: A = A (recalling that only square matrices can be symmetric).
    [Show full text]
  • Multilinear Algebra and Applications July 15, 2014
    Multilinear Algebra and Applications July 15, 2014. Contents Chapter 1. Introduction 1 Chapter 2. Review of Linear Algebra 5 2.1. Vector Spaces and Subspaces 5 2.2. Bases 7 2.3. The Einstein convention 10 2.3.1. Change of bases, revisited 12 2.3.2. The Kronecker delta symbol 13 2.4. Linear Transformations 14 2.4.1. Similar matrices 18 2.5. Eigenbases 19 Chapter 3. Multilinear Forms 23 3.1. Linear Forms 23 3.1.1. Definition, Examples, Dual and Dual Basis 23 3.1.2. Transformation of Linear Forms under a Change of Basis 26 3.2. Bilinear Forms 30 3.2.1. Definition, Examples and Basis 30 3.2.2. Tensor product of two linear forms on V 32 3.2.3. Transformation of Bilinear Forms under a Change of Basis 33 3.3. Multilinear forms 34 3.4. Examples 35 3.4.1. A Bilinear Form 35 3.4.2. A Trilinear Form 36 3.5. Basic Operation on Multilinear Forms 37 Chapter 4. Inner Products 39 4.1. Definitions and First Properties 39 4.1.1. Correspondence Between Inner Products and Symmetric Positive Definite Matrices 40 4.1.1.1. From Inner Products to Symmetric Positive Definite Matrices 42 4.1.1.2. From Symmetric Positive Definite Matrices to Inner Products 42 4.1.2. Orthonormal Basis 42 4.2. Reciprocal Basis 46 4.2.1. Properties of Reciprocal Bases 48 4.2.2. Change of basis from a basis to its reciprocal basis g 50 B B III IV CONTENTS 4.2.3.
    [Show full text]
  • On Manifolds of Tensors of Fixed Tt-Rank
    ON MANIFOLDS OF TENSORS OF FIXED TT-RANK SEBASTIAN HOLTZ, THORSTEN ROHWEDDER, AND REINHOLD SCHNEIDER Abstract. Recently, the format of TT tensors [19, 38, 34, 39] has turned out to be a promising new format for the approximation of solutions of high dimensional problems. In this paper, we prove some new results for the TT representation of a tensor U ∈ Rn1×...×nd and for the manifold of tensors of TT-rank r. As a first result, we prove that the TT (or compression) ranks ri of a tensor U are unique and equal to the respective seperation ranks of U if the components of the TT decomposition are required to fulfil a certain maximal rank condition. We then show that d the set T of TT tensors of fixed rank r forms an embedded manifold in Rn , therefore preserving the essential theoretical properties of the Tucker format, but often showing an improved scaling behaviour. Extending a similar approach for matrices [7], we introduce certain gauge conditions to obtain a unique representation of the tangent space TU T of T and deduce a local parametrization of the TT manifold. The parametrisation of TU T is often crucial for an algorithmic treatment of high-dimensional time-dependent PDEs and minimisation problems [33]. We conclude with remarks on those applications and present some numerical examples. 1. Introduction The treatment of high-dimensional problems, typically of problems involving quantities from Rd for larger dimensions d, is still a challenging task for numerical approxima- tion. This is owed to the principal problem that classical approaches for their treatment normally scale exponentially in the dimension d in both needed storage and computa- tional time and thus quickly become computationally infeasable for sensible discretiza- tions of problems of interest.
    [Show full text]
  • Causalx: Causal Explanations and Block Multilinear Factor Analysis
    To appear: Proc. of the 2020 25th International Conference on Pattern Recognition (ICPR 2020) Milan, Italy, Jan. 10-15, 2021. CausalX: Causal eXplanations and Block Multilinear Factor Analysis M. Alex O. Vasilescu1;2 Eric Kim2;1 Xiao S. Zeng2 [email protected] [email protected] [email protected] 1Tensor Vision Technologies, Los Angeles, California 2Department of Computer Science,University of California, Los Angeles Abstract—By adhering to the dictum, “No causation without I. INTRODUCTION:PROBLEM DEFINITION manipulation (treatment, intervention)”, cause and effect data Developing causal explanations for correct results or for failures analysis represents changes in observed data in terms of changes from mathematical equations and data is important in developing in the causal factors. When causal factors are not amenable for a trustworthy artificial intelligence, and retaining public trust. active manipulation in the real world due to current technological limitations or ethical considerations, a counterfactual approach Causal explanations are germane to the “right to an explanation” performs an intervention on the model of data formation. In the statute [15], [13] i.e., to data driven decisions, such as those case of object representation or activity (temporal object) rep- that rely on images. Computer graphics and computer vision resentation, varying object parts is generally unfeasible whether problems, also known as forward and inverse imaging problems, they be spatial and/or temporal. Multilinear algebra, the algebra have been cast as causal inference questions [40], [42] consistent of higher order tensors, is a suitable and transparent framework for disentangling the causal factors of data formation. Learning a with Donald Rubin’s quantitative definition of causality, where part-based intrinsic causal factor representations in a multilinear “A causes B” means “the effect of A is B”, a measurable framework requires applying a set of interventions on a part- and experimentally repeatable quantity [14], [17].
    [Show full text]
  • Geometric-Algebra Adaptive Filters Wilder B
    1 Geometric-Algebra Adaptive Filters Wilder B. Lopes∗, Member, IEEE, Cassio G. Lopesy, Senior Member, IEEE Abstract—This paper presents a new class of adaptive filters, namely Geometric-Algebra Adaptive Filters (GAAFs). They are Faces generated by formulating the underlying minimization problem (a deterministic cost function) from the perspective of Geometric Algebra (GA), a comprehensive mathematical language well- Edges suited for the description of geometric transformations. Also, (directed lines) differently from standard adaptive-filtering theory, Geometric Calculus (the extension of GA to differential calculus) allows Fig. 1. A polyhedron (3-dimensional polytope) can be completely described for applying the same derivation techniques regardless of the by the geometric multiplication of its edges (oriented lines, vectors), which type (subalgebra) of the data, i.e., real, complex numbers, generate the faces and hypersurfaces (in the case of a general n-dimensional quaternions, etc. Relying on those characteristics (among others), polytope). a deterministic quadratic cost function is posed, from which the GAAFs are devised, providing a generalization of regular adaptive filters to subalgebras of GA. From the obtained update rule, it is shown how to recover the following least-mean squares perform calculus with hypercomplex quantities, i.e., elements (LMS) adaptive filter variants: real-entries LMS, complex LMS, that generalize complex numbers for higher dimensions [2]– and quaternions LMS. Mean-square analysis and simulations in [10]. a system identification scenario are provided, showing very good agreement for different levels of measurement noise. GA-based AFs were first introduced in [11], [12], where they were successfully employed to estimate the geometric Index Terms—Adaptive filtering, geometric algebra, quater- transformation (rotation and translation) that aligns a pair of nions.
    [Show full text]
  • Matrices and Tensors
    APPENDIX MATRICES AND TENSORS A.1. INTRODUCTION AND RATIONALE The purpose of this appendix is to present the notation and most of the mathematical tech- niques that are used in the body of the text. The audience is assumed to have been through sev- eral years of college-level mathematics, which included the differential and integral calculus, differential equations, functions of several variables, partial derivatives, and an introduction to linear algebra. Matrices are reviewed briefly, and determinants, vectors, and tensors of order two are described. The application of this linear algebra to material that appears in under- graduate engineering courses on mechanics is illustrated by discussions of concepts like the area and mass moments of inertia, Mohr’s circles, and the vector cross and triple scalar prod- ucts. The notation, as far as possible, will be a matrix notation that is easily entered into exist- ing symbolic computational programs like Maple, Mathematica, Matlab, and Mathcad. The desire to represent the components of three-dimensional fourth-order tensors that appear in anisotropic elasticity as the components of six-dimensional second-order tensors and thus rep- resent these components in matrices of tensor components in six dimensions leads to the non- traditional part of this appendix. This is also one of the nontraditional aspects in the text of the book, but a minor one. This is described in §A.11, along with the rationale for this approach. A.2. DEFINITION OF SQUARE, COLUMN, AND ROW MATRICES An r-by-c matrix, M, is a rectangular array of numbers consisting of r rows and c columns: ¯MM..
    [Show full text]
  • 28. Exterior Powers
    28. Exterior powers 28.1 Desiderata 28.2 Definitions, uniqueness, existence 28.3 Some elementary facts 28.4 Exterior powers Vif of maps 28.5 Exterior powers of free modules 28.6 Determinants revisited 28.7 Minors of matrices 28.8 Uniqueness in the structure theorem 28.9 Cartan's lemma 28.10 Cayley-Hamilton Theorem 28.11 Worked examples While many of the arguments here have analogues for tensor products, it is worthwhile to repeat these arguments with the relevant variations, both for practice, and to be sensitive to the differences. 1. Desiderata Again, we review missing items in our development of linear algebra. We are missing a development of determinants of matrices whose entries may be in commutative rings, rather than fields. We would like an intrinsic definition of determinants of endomorphisms, rather than one that depends upon a choice of coordinates, even if we eventually prove that the determinant is independent of the coordinates. We anticipate that Artin's axiomatization of determinants of matrices should be mirrored in much of what we do here. We want a direct and natural proof of the Cayley-Hamilton theorem. Linear algebra over fields is insufficient, since the introduction of the indeterminate x in the definition of the characteristic polynomial takes us outside the class of vector spaces over fields. We want to give a conceptual proof for the uniqueness part of the structure theorem for finitely-generated modules over principal ideal domains. Multi-linear algebra over fields is surely insufficient for this. 417 418 Exterior powers 2. Definitions, uniqueness, existence Let R be a commutative ring with 1.
    [Show full text]
  • Multilinear Algebra in Data Analysis: Tensors, Symmetric Tensors, Nonnegative Tensors
    Multilinear Algebra in Data Analysis: tensors, symmetric tensors, nonnegative tensors Lek-Heng Lim Stanford University Workshop on Algorithms for Modern Massive Datasets Stanford, CA June 21–24, 2006 Thanks: G. Carlsson, L. De Lathauwer, J.M. Landsberg, M. Mahoney, L. Qi, B. Sturmfels; Collaborators: P. Comon, V. de Silva, P. Drineas, G. Golub References http://www-sccm.stanford.edu/nf-publications-tech.html [CGLM2] P. Comon, G. Golub, L.-H. Lim, and B. Mourrain, “Symmetric tensors and symmetric tensor rank,” SCCM Tech. Rep., 06-02, 2006. [CGLM1] P. Comon, B. Mourrain, L.-H. Lim, and G.H. Golub, “Genericity and rank deficiency of high order symmetric tensors,” Proc. IEEE Int. Con- ference on Acoustics, Speech, and Signal Processing (ICASSP), 31 (2006), no. 3, pp. 125–128. [dSL] V. de Silva and L.-H. Lim, “Tensor rank and the ill-posedness of the best low-rank approximation problem,” SCCM Tech. Rep., 06-06 (2006). [GL] G. Golub and L.-H. Lim, “Nonnegative decomposition and approximation of nonnegative matrices and tensors,” SCCM Tech. Rep., 06-01 (2006), forthcoming. [L] L.-H. Lim, “Singular values and eigenvalues of tensors: a variational approach,” Proc. IEEE Int. Workshop on Computational Advances in Multi- Sensor Adaptive Processing (CAMSAP), 1 (2005), pp. 129–132. 2 What is not a tensor, I • What is a vector? – Mathematician: An element of a vector space. – Physicist: “What kind of physical quantities can be rep- resented by vectors?” Answer: Once a basis is chosen, an n-dimensional vector is something that is represented by n real numbers only if those real numbers transform themselves as expected (ie.
    [Show full text]