DOCSLIB.ORG

Matrix Calculus - Notes on the Derivative of a Trace Johannes Traa

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Matrix Calculus - Notes on the Derivative of a Trace Johannes Traa Matrix Calculus - Notes on the Derivative of a Trace Johannes Traa This write-up elucidates the rules of matrix calculus for expressions involving the trace of a function of a matrix X: f tr £g ( X )¤ . (1) Æ We would like to take the derivative of f with respect to X: @f ? . (2) @X Æ One strategy is to write the trace expression as a scalar using index notation, take the derivative, and re-write in matrix form. An easier way is to reduce the problem to one or more smaller problems where the results for simpler derivatives can be applied. It’s brute- force vs bottom-up. MATRIX-VALUED DERIVATIVE M N The derivative of a scalar f with respect to a matrix X R £ can be written as: 2 1 2 3 @f @f @f 6 @X11 @X12 @X1N 7 6 ¢¢¢ 7 6 7 6 @f @f @f 7 @f 6 @X @X @X 7 6 21 22 ¢¢¢ 2N 7 6 7 (3) @X Æ 6 . 7 6 . .. 7 6 7 6 7 4 @f @f @f 5 @XM1 @XM2 ¢¢¢ @XMN So, the result is the same size as X. MATRIX AND INDEX NOTATION It is useful to be able to convert between matrix notation and index notation. For example, the product AB has elements: X [AB]ik Ai j B jk , (4) Æ j and the matrix product ABCT has elements: £ T ¤ X £ T ¤ X X XX ABC il Ai j BC jl Ai j B jkClk Ai j B jkClk . (5) Æ j Æ j k Æ j k FIRST-ORDER DERIVATIVES EXAMPLE 1 Consider this example: f tr [AXB] . (6) Æ We can write this using index notation as: X XX XX X XXX f [AXB]ii Ai j [XB]ji Ai j X jk Bki Ai j X jk Bki . (7) Æ i Æ i j Æ i j k Æ i j k Taking the derivative with respect to X jk , we get: @f X Ai j Bki [BA]k j . (8) @X jk Æ i Æ The result has to be the same size as X, so we know that the indices of the rows and columns must be j and k, respectively. This means we have to transpose the result above to write the derivative in matrix form as: @tr [AXB] AT BT . (9) @X Æ 2 EXAMPLE 2 Similarly, we have: £ T ¤ XXX f tr AX B Ai j Xk j Bki , (10) Æ Æ i j k so that the derivative is: @f X Ai j Bki [BA]k j , (11) @Xk j Æ i Æ The X term appears in (10) with indices k j, so we need to write the derivative in matrix form such that k is the row index and j is the column index. Thus, we have: @ tr £AXT B¤ BA . (12) @X Æ MULTIPLE-ORDER Now consider a more complicated example: f tr £AXBXCT ¤ (13) Æ XXXXX Ai j X jk Bkl XlmCim . (14) Æ i j k l m The derivative has contributions from both appearances of X. TAKE 1 In index notation: @f XXX £ T ¤ Ai j Bkl XlmCim BXC A k j , (15) @X jk Æ i l m Æ @f XXX £ T ¤ Ai j X jk BklCim C AXB ml . (16) @Xlm Æ i j k Æ Transposing appropriately and summing the terms together, we have: @ tr £AXBXCT ¤ AT CXT BT BT XT AT C . (17) @X Æ Å TAKE 2 We can skip this tedious process by applying (9) for each appearance of X: £ ¤ £ ¤ @ tr AXBXCT @ tr [AXD] @ tr EXCT AT DT ET C . (18) @X Æ @X Å @X Æ Å 3 where D BXCT and E AXB. So we just evaluate the matrix derivative for each appear- Æ Æ ance of X assuming that everything else is a constant (including other X’s). To see why this rule is useful, consider the following beast: f tr £AXXT BCXT XC¤ . (19) Æ We can immediately write down the derivative using (9) and (12): £ T T ¤ @ tr AXX BCX XC T T (A)T ¡XT BCXT XC¢ ¡BCXT XC¢(AX) (XC)¡AXXT BC¢ ¡AXXT BCXT ¢ (C)T @X Æ Å Å Å (20) ACT XT XCT BT X BCXT XCAX XCAXXT BC XCT BT XXT AT CT . (21) Æ Å Å Å FROBENIUS NORM The Frobenius norm shows up when we have an optimization problem involving a matrix factorization and we want to minimize a sum-of-squares error criterion: à !2 XX X 2 £ T ¤ f Xik Wi j H jk X WH F tr (X WH)(X WH) . (22) Æ i k ¡ j Æ k ¡ k Æ ¡ ¡ We can work with the expression in index notation, but it’s easier to work directly with matrices and apply the results derived earlier. Suppose we want to find the derivative with respect to W. Expanding the matrix outer product, we have: f tr £XXT ¤ tr £XHT WT ¤ tr £WHXT ¤ tr £WHHT WT ¤ . (23) Æ ¡ ¡ Å Applying (9) and (12), we easily deduce that: @ tr £(X WH)(X WH)T ¤ ¡ ¡ 2XHT 2WHHT . (24) @W Æ¡ Å LOG Consider this trace expression: µ ¶ £ T ¤ XX XX f tr V log(AXB) Vi j log Aim XmnBn j . (25) Æ Æ i j m n Taking the derivative with respect to Xmn, we get: 0 1 µ ¶ @f XX Vi j XX V @PP A AimBn j Aim Bn j . (26) @ Xmn Æ i j Aim XmnBn j Æ i j AXB i j m n Thus: 4 £ ¤ @ tr VT log(AXB) µ V ¶ AT BT . (27) @X Æ AXB Similarly: £ ¡ ¢¤ @ tr VT log AXT B µ V ¶T B A . (28) @X Æ AXT B These bare a spooky resemblance to (9) and (12). DIAG EXAMPLE 1 Consider the tricky case of a di ag ( ) operator: ¡ £ ¤ XX f tr A di ag (x)B Ai j x j B ji . (29) Æ Æ i j Taking the derivative, we have: @f X £¡ T ¢ ¤ Ai j B ji A B 1 j . (30) @x j Æ i Æ ¯ So we can write: @ tr £A di ag (x)B¤ ¡AT B¢1 . (31) @ x Æ ¯ EXAMPLE 2 Consider the following take on the last example: £ ¤ XXX f tr J A di ag (x)B Ai j x j B jk , (32) Æ Æ i j k where J is the matrix of ones. TAKE 1 Taking the derivative, we have: à !à ! @f XX X X £ T ¤ Ai j B jk Ai j B jk A 1 B1 j . (33) @x j Æ i k Æ i k Æ ¯ So we can write: @ tr £J A di ag (x)B¤ AT 1 B1 . (34) @ x Æ ¯ 5 TAKE 2 We could have derived this result from the previous example using the rotation property of the trace operator: f tr £J A di ag (x)B¤ tr £11T A di ag (x)B¤ tr £1T A di ag (x)B1¤ tr £aT di ag (x)b¤ , Æ Æ Æ Æ (35) where we have defined a AT 1 and b B1. Applying (31), we have: Æ Æ @ tr £aT di ag (x)b¤ (a b)1 AT 1 B1 . (36) @ x Æ ¯ Æ ¯ EXAMPLE 3 Consider a more complicated example: µ ¶ £ T ¡ ¢¤ XX X f tr V log A di ag (x)B Vi j log Aim xmBm j . (37) Æ Æ i j m Taking the derivative with respect to xm, we have: 0 1 @f XX Vi j @P A AimBm j (38) @xm Æ i j Aim xmBm j m " µ ¶ # X X V Aim Bm j (39) Æ j i A di ag (x)B i j ·µ µ V ¶ ¶ ¸ AT B 1 . (40) Æ A di ag (x)B ¯ m The final result is: £ ¡ ¢¤ @ tr VT log A di ag (x)B µ µ V ¶ ¶ AT B 1 . (41) @ x Æ A di ag (x)B ¯ EXAMPLE 4 How about when we have a trace composed of a sum of expressions, each of which depends on what row of a matrix B is chosen: " # µ µ ¶ ¶ X T ¡ ¢ XXX X X f tr V log A di ag (Bk: X)C Vi j log Aim Bkn Xnm Cm j . (42) Æ k Æ k i j m n Taking the derivative, we get: 6 0 1 @f XXXB Vi j C B µ ¶ C AimBknCm j (43) @Xnm Æ k i j @P P A Aim Bkn Xnm Cm j m n µ ¶ X XX V Bkn AimCm j (44) Æ k i j A di ag (Bk: X)C i j · µµ ¶T ¶¸ X T V T Bkn 1 A C , (45) Æ k A di ag (Bk: X)C ¯ km so we can write: 2 3 µ³ ´T ¶ · ¸ 1T V A CT ¡ ¢ 6 A di ag(B1: X)C 7 @ tr PVT log A di ag (B X)C 6 ¯ 7 k: 6 7 k T 6 . 7 B 6 . 7 . (46) @X Æ 6 7 6 7 4 µ³ ´T ¶5 1T V A CT A di ag(BK : X)C ¯ CONCLUSION We’ve learned two things. First, if we don’t know how to find the derivative of an expres- sion using matrix calculus directly, we can always fall back on index notation and convert back to matrices at the end. This reduces a potentially unintuitive matrix-valued problem into one involving scalars, which we are used to. Second, it’s less painful to massage an expression into a familiar form and apply previously-derived identities.
Load more

Recommended publications
  • Mathematics for Dynamicists - Lecture Notes Thomas Michelitsch
    Mathematics for Dynamicists - Lecture Notes Thomas Michelitsch To cite this version: Thomas Michelitsch. Mathematics for Dynamicists - Lecture Notes. Doctoral. Mathematics for Dynamicists, France. 2006. cel-01128826 HAL Id: cel-01128826 https://hal.archives-ouvertes.fr/cel-01128826 Submitted on 10 Mar 2015 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. Mathematics for Dynamicists Lecture Notes c 2004-2006 Thomas Michael Michelitsch1 Department of Civil and Structural Engineering The University of Sheffield Sheffield, UK 1Present address : http://www.dalembert.upmc.fr/home/michelitsch/ . Dedicated to the memory of my parents Michael and Hildegard Michelitsch Handout Manuscript with Fractal Cover Picture by Michael Michelitsch https://www.flickr.com/photos/michelitsch/sets/72157601230117490 1 Contents Preface 1 Part I 1.1 Binomial Theorem 1.2 Derivatives, Integrals and Taylor Series 1.3 Elementary Functions 1.4 Complex Numbers: Cartesian and Polar Representation 1.5 Sequences, Series, Integrals and Power-Series 1.6 Polynomials and Partial Fractions
    [Show full text]
  • Estimations of the Trace of Powers of Positive Self-Adjoint Operators by Extrapolation of the Moments∗
    Electronic Transactions on Numerical Analysis. ETNA Volume 39, pp. 144-155, 2012. Kent State University Copyright 2012, Kent State University. http://etna.math.kent.edu ISSN 1068-9613. ESTIMATIONS OF THE TRACE OF POWERS OF POSITIVE SELF-ADJOINT OPERATORS BY EXTRAPOLATION OF THE MOMENTS∗ CLAUDE BREZINSKI†, PARASKEVI FIKA‡, AND MARILENA MITROULI‡ Abstract. Let A be a positive self-adjoint linear operator on a real separable Hilbert space H. Our aim is to build estimates of the trace of Aq, for q ∈ R. These estimates are obtained by extrapolation of the moments of A. Applications of the matrix case are discussed, and numerical results are given. Key words. Trace, positive self-adjoint linear operator, symmetric matrix, matrix powers, matrix moments, extrapolation. AMS subject classifications. 65F15, 65F30, 65B05, 65C05, 65J10, 15A18, 15A45. 1. Introduction. Let A be a positive self-adjoint linear operator from H to H, where H is a real separable Hilbert space with inner product denoted by (·, ·). Our aim is to build estimates of the trace of Aq, for q ∈ R. These estimates are obtained by extrapolation of the integer moments (z, Anz) of A, for n ∈ N. A similar procedure was first introduced in [3] for estimating the Euclidean norm of the error when solving a system of linear equations, which corresponds to q = −2. The case q = −1, which leads to estimates of the trace of the inverse of a matrix, was studied in [4]; on this problem, see [10]. Let us mention that, when only positive powers of A are used, the Hilbert space H could be infinite dimensional, while, for negative powers of A, it is always assumed to be a finite dimensional one, and, obviously, A is also assumed to be invertible.
    [Show full text]
  • 5 the Dirac Equation and Spinors
    5 The Dirac Equation and Spinors In this section we develop the appropriate wavefunctions for fundamental fermions and bosons. 5.1 Notation Review The three dimension differential operator is : ∂ ∂ ∂ = , , (5.1) ∂x ∂y ∂z We can generalise this to four dimensions ∂µ: 1 ∂ ∂ ∂ ∂ ∂ = , , , (5.2) µ c ∂t ∂x ∂y ∂z 5.2 The Schr¨odinger Equation First consider a classical non-relativistic particle of mass m in a potential U. The energy-momentum relationship is: p2 E = + U (5.3) 2m we can substitute the differential operators: ∂ Eˆ i pˆ i (5.4) → ∂t →− to obtain the non-relativistic Schr¨odinger Equation (with = 1): ∂ψ 1 i = 2 + U ψ (5.5) ∂t −2m For U = 0, the free particle solutions are: iEt ψ(x, t) e− ψ(x) (5.6) ∝ and the probability density ρ and current j are given by: 2 i ρ = ψ(x) j = ψ∗ ψ ψ ψ∗ (5.7) | | −2m − with conservation of probability giving the continuity equation: ∂ρ + j =0, (5.8) ∂t · Or in Covariant notation: µ µ ∂µj = 0 with j =(ρ,j) (5.9) The Schr¨odinger equation is 1st order in ∂/∂t but second order in ∂/∂x. However, as we are going to be dealing with relativistic particles, space and time should be treated equally. 25 5.3 The Klein-Gordon Equation For a relativistic particle the energy-momentum relationship is: p p = p pµ = E2 p 2 = m2 (5.10) · µ − | | Substituting the equation (5.4), leads to the relativistic Klein-Gordon equation: ∂2 + 2 ψ = m2ψ (5.11) −∂t2 The free particle solutions are plane waves: ip x i(Et p x) ψ e− · = e− − · (5.12) ∝ The Klein-Gordon equation successfully describes spin 0 particles in relativistic quan- tum field theory.
    [Show full text]
  • A Some Basic Rules of Tensor Calculus
    A Some Basic Rules of Tensor Calculus The tensor calculus is a powerful tool for the description of the fundamentals in con- tinuum mechanics and the derivation of the governing equations for applied prob- lems. In general, there are two possibilities for the representation of the tensors and the tensorial equations: – the direct (symbolic) notation and – the index (component) notation The direct notation operates with scalars, vectors and tensors as physical objects defined in the three dimensional space. A vector (first rank tensor) a is considered as a directed line segment rather than a triple of numbers (coordinates). A second rank tensor A is any finite sum of ordered vector pairs A = a b + ... +c d. The scalars, vectors and tensors are handled as invariant (independent⊗ from the choice⊗ of the coordinate system) objects. This is the reason for the use of the direct notation in the modern literature of mechanics and rheology, e.g. [29, 32, 49, 123, 131, 199, 246, 313, 334] among others. The index notation deals with components or coordinates of vectors and tensors. For a selected basis, e.g. gi, i = 1, 2, 3 one can write a = aig , A = aibj + ... + cidj g g i i ⊗ j Here the Einstein’s summation convention is used: in one expression the twice re- peated indices are summed up from 1 to 3, e.g. 3 3 k k ik ik a gk ∑ a gk, A bk ∑ A bk ≡ k=1 ≡ k=1 In the above examples k is a so-called dummy index. Within the index notation the basic operations with tensors are defined with respect to their coordinates, e.
    [Show full text]
  • 18.700 JORDAN NORMAL FORM NOTES These Are Some Supplementary Notes on How to Find the Jordan Normal Form of a Small Matrix. Firs
    18.700 JORDAN NORMAL FORM NOTES These are some supplementary notes on how to find the Jordan normal form of a small matrix. First we recall some of the facts from lecture, next we give the general algorithm for finding the Jordan normal form of a linear operator, and then we will see how this works for small matrices. 1. Facts Throughout we will work over the field C of complex numbers, but if you like you may replace this with any other algebraically closed field. Suppose that V is a C-vector space of dimension n and suppose that T : V → V is a C-linear operator. Then the characteristic polynomial of T factors into a product of linear terms, and the irreducible factorization has the form m1 m2 mr cT (X) = (X − λ1) (X − λ2) ... (X − λr) , (1) for some distinct numbers λ1, . , λr ∈ C and with each mi an integer m1 ≥ 1 such that m1 + ··· + mr = n. Recall that for each eigenvalue λi, the eigenspace Eλi is the kernel of T − λiIV . We generalized this by defining for each integer k = 1, 2,... the vector subspace k k E(X−λi) = ker(T − λiIV ) . (2) It is clear that we have inclusions 2 e Eλi = EX−λi ⊂ E(X−λi) ⊂ · · · ⊂ E(X−λi) ⊂ .... (3) k k+1 Since dim(V ) = n, it cannot happen that each dim(E(X−λi) ) < dim(E(X−λi) ), for each e e +1 k = 1, . , n. Therefore there is some least integer ei ≤ n such that E(X−λi) i = E(X−λi) i .
    [Show full text]
  • Matrix Calculus
    Appendix D Matrix Calculus From too much study, and from extreme passion, cometh madnesse. Isaac Newton [205, §5] − D.1 Gradient, Directional derivative, Taylor series D.1.1 Gradients Gradient of a differentiable real function f(x) : RK R with respect to its vector argument is defined uniquely in terms of partial derivatives→ ∂f(x) ∂x1 ∂f(x) , ∂x2 RK f(x) . (2053) ∇ . ∈ . ∂f(x) ∂xK while the second-order gradient of the twice differentiable real function with respect to its vector argument is traditionally called the Hessian; 2 2 2 ∂ f(x) ∂ f(x) ∂ f(x) 2 ∂x1 ∂x1∂x2 ··· ∂x1∂xK 2 2 2 ∂ f(x) ∂ f(x) ∂ f(x) 2 2 K f(x) , ∂x2∂x1 ∂x2 ··· ∂x2∂xK S (2054) ∇ . ∈ . .. 2 2 2 ∂ f(x) ∂ f(x) ∂ f(x) 2 ∂xK ∂x1 ∂xK ∂x2 ∂x ··· K interpreted ∂f(x) ∂f(x) 2 ∂ ∂ 2 ∂ f(x) ∂x1 ∂x2 ∂ f(x) = = = (2055) ∂x1∂x2 ³∂x2 ´ ³∂x1 ´ ∂x2∂x1 Dattorro, Convex Optimization Euclidean Distance Geometry, Mεβoo, 2005, v2020.02.29. 599 600 APPENDIX D. MATRIX CALCULUS The gradient of vector-valued function v(x) : R RN on real domain is a row vector → v(x) , ∂v1(x) ∂v2(x) ∂vN (x) RN (2056) ∇ ∂x ∂x ··· ∂x ∈ h i while the second-order gradient is 2 2 2 2 , ∂ v1(x) ∂ v2(x) ∂ vN (x) RN v(x) 2 2 2 (2057) ∇ ∂x ∂x ··· ∂x ∈ h i Gradient of vector-valued function h(x) : RK RN on vector domain is → ∂h1(x) ∂h2(x) ∂hN (x) ∂x1 ∂x1 ··· ∂x1 ∂h1(x) ∂h2(x) ∂hN (x) h(x) , ∂x2 ∂x2 ··· ∂x2 ∇ .
    [Show full text]
  • Matrix Calculus
    D Matrix Calculus D–1 Appendix D: MATRIX CALCULUS D–2 In this Appendix we collect some useful formulas of matrix calculus that often appear in finite element derivations. §D.1 THE DERIVATIVES OF VECTOR FUNCTIONS Let x and y be vectors of orders n and m respectively: x1 y1 x2 y2 x = . , y = . ,(D.1) . xn ym where each component yi may be a function of all the xj , a fact represented by saying that y is a function of x,or y = y(x). (D.2) If n = 1, x reduces to a scalar, which we call x.Ifm = 1, y reduces to a scalar, which we call y. Various applications are studied in the following subsections. §D.1.1 Derivative of Vector with Respect to Vector The derivative of the vector y with respect to vector x is the n × m matrix ∂y1 ∂y2 ∂ym ∂ ∂ ··· ∂ x1 x1 x1 ∂ ∂ ∂ ∂ y1 y2 ··· ym y def ∂x ∂x ∂x = 2 2 2 (D.3) ∂x . . .. ∂ ∂ ∂ y1 y2 ··· ym ∂xn ∂xn ∂xn §D.1.2 Derivative of a Scalar with Respect to Vector If y is a scalar, ∂y ∂x1 ∂ ∂ y y def ∂ = x2 .(D.4) ∂x . . ∂y ∂xn §D.1.3 Derivative of Vector with Respect to Scalar If x is a scalar, ∂y def ∂ ∂ ∂ = y1 y2 ... ym (D.5) ∂x ∂x ∂x ∂x D–2 D–3 §D.1 THE DERIVATIVES OF VECTOR FUNCTIONS REMARK D.1 Many authors, notably in statistics and economics, define the derivatives as the transposes of those given above.1 This has the advantage of better agreement of matrix products with composition schemes such as the chain rule.
    [Show full text]
  • Trace Inequalities for Matrices
    Bull. Aust. Math. Soc. 87 (2013), 139–148 doi:10.1017/S0004972712000627 TRACE INEQUALITIES FOR MATRICES KHALID SHEBRAWI ˛ and HUSSIEN ALBADAWI (Received 23 February 2012; accepted 20 June 2012) Abstract Trace inequalities for sums and products of matrices are presented. Relations between the given inequalities and earlier results are discussed. Among other inequalities, it is shown that if A and B are positive semidefinite matrices then tr(AB)k ≤ minfkAkk tr Bk; kBkk tr Akg for any positive integer k. 2010 Mathematics subject classification: primary 15A18; secondary 15A42, 15A45. Keywords and phrases: trace inequalities, eigenvalues, singular values. 1. Introduction Let Mn(C) be the algebra of all n × n matrices over the complex number field. The singular values of A 2 Mn(C), denoted by s1(A); s2(A);:::; sn(A), are the eigenvalues ∗ 1=2 of the matrix jAj = (A A) arranged in such a way that s1(A) ≥ s2(A) ≥ · · · ≥ sn(A). 2 ∗ ∗ Note that si (A) = λi(A A) = λi(AA ), so for a positive semidefinite matrix A, we have si(A) = λi(A)(i = 1; 2;:::; n). The trace functional of A 2 Mn(C), denoted by tr A or tr(A), is defined to be the sum of the entries on the main diagonal of A and it is well known that the trace of a Pn matrix A is equal to the sum of its eigenvalues, that is, tr A = j=1 λ j(A). Two principal properties of the trace are that it is a linear functional and, for A; B 2 Mn(C), we have tr(AB) = tr(BA).
    [Show full text]
  • Lecture 28: Eigenvalues Allowing Complex Eigenvalues Is Really a Blessing
    Math 19b: Linear Algebra with Probability Oliver Knill, Spring 2011 cos(t) sin(t) 2 2 For a rotation A = the characteristic polynomial is λ − 2 cos(α)+1 − sin(t) cos(t) which has the roots cos(α) ± i sin(α)= eiα. Lecture 28: Eigenvalues Allowing complex eigenvalues is really a blessing. The structure is very simple: Fundamental theorem of algebra: For a n × n matrix A, the characteristic We have seen that det(A) = 0 if and only if A is invertible. polynomial has exactly n roots. There are therefore exactly n eigenvalues of A if we count them with multiplicity. The polynomial fA(λ) = det(A − λIn) is called the characteristic polynomial of 1 n n−1 A. Proof One only has to show a polynomial p(z)= z + an−1z + ··· + a1z + a0 always has a root z0 We can then factor out p(z)=(z − z0)g(z) where g(z) is a polynomial of degree (n − 1) and The eigenvalues of A are the roots of the characteristic polynomial. use induction in n. Assume now that in contrary the polynomial p has no root. Cauchy’s integral theorem then tells dz 2πi Proof. If Av = λv,then v is in the kernel of A − λIn. Consequently, A − λIn is not invertible and = =0 . (1) z =r | | | zp(z) p(0) det(A − λIn)=0 . On the other hand, for all r, 2 1 dz 1 2π 1 For the matrix A = , the characteristic polynomial is | | ≤ 2πrmax|z|=r = . (2) z =r 4 −1 | | | zp(z) |zp(z)| min|z|=rp(z) The right hand side goes to 0 for r →∞ because 2 − λ 1 2 det(A − λI2) = det( )= λ − λ − 6 .
    [Show full text]
  • Approximating Spectral Sums of Large-Scale Matrices Using Stochastic Chebyshev Approximations∗
    Approximating Spectral Sums of Large-scale Matrices using Stochastic Chebyshev Approximations∗ Insu Han y Dmitry Malioutov z Haim Avron x Jinwoo Shin { March 10, 2017 Abstract Computation of the trace of a matrix function plays an important role in many scientific com- puting applications, including applications in machine learning, computational physics (e.g., lat- tice quantum chromodynamics), network analysis and computational biology (e.g., protein fold- ing), just to name a few application areas. We propose a linear-time randomized algorithm for approximating the trace of matrix functions of large symmetric matrices. Our algorithm is based on coupling function approximation using Chebyshev interpolation with stochastic trace estima- tors (Hutchinson’s method), and as such requires only implicit access to the matrix, in the form of a function that maps a vector to the product of the matrix and the vector. We provide rigorous approximation error in terms of the extremal eigenvalue of the input matrix, and the Bernstein ellipse that corresponds to the function at hand. Based on our general scheme, we provide algo- rithms with provable guarantees for important matrix computations, including log-determinant, trace of matrix inverse, Estrada index, Schatten p-norm, and testing positive definiteness. We experimentally evaluate our algorithm and demonstrate its effectiveness on matrices with tens of millions dimensions. 1 Introduction Given a symmetric matrix A 2 Rd×d and function f : R ! R, we study how to efficiently compute d X Σf (A) = tr(f(A)) = f(λi); (1) i=1 arXiv:1606.00942v2 [cs.DS] 9 Mar 2017 where λ1; : : : ; λd are eigenvalues of A.
    [Show full text]
  • Math 217: True False Practice Professor Karen Smith 1. a Square Matrix Is Invertible If and Only If Zero Is Not an Eigenvalue. Solution Note: True
    (c)2015 UM Math Dept licensed under a Creative Commons By-NC-SA 4.0 International License. Math 217: True False Practice Professor Karen Smith 1. A square matrix is invertible if and only if zero is not an eigenvalue. Solution note: True. Zero is an eigenvalue means that there is a non-zero element in the kernel. For a square matrix, being invertible is the same as having kernel zero. 2. If A and B are 2 × 2 matrices, both with eigenvalue 5, then AB also has eigenvalue 5. Solution note: False. This is silly. Let A = B = 5I2. Then the eigenvalues of AB are 25. 3. If A and B are 2 × 2 matrices, both with eigenvalue 5, then A + B also has eigenvalue 5. Solution note: False. This is silly. Let A = B = 5I2. Then the eigenvalues of A + B are 10. 4. A square matrix has determinant zero if and only if zero is an eigenvalue. Solution note: True. Both conditions are the same as the kernel being non-zero. 5. If B is the B-matrix of some linear transformation V !T V . Then for all ~v 2 V , we have B[~v]B = [T (~v)]B. Solution note: True. This is the definition of B-matrix. 21 2 33 T 6. Suppose 40 2 05 is the matrix of a transformation V ! V with respect to some basis 0 0 1 B = (f1; f2; f3). Then f1 is an eigenvector. Solution note: True. It has eigenvalue 1. The first column of the B-matrix is telling us that T (f1) = f1.
    [Show full text]
  • Sam Roweis' Notes on Matrix Identities
    matrix identities sam roweis (revised June 1999) note that a,b,c and A,B,C do not depend on X,Y,x,y or z 0.1 basic formulae A(B + C) = AB + AC (1a) (A + B)T = AT + BT (1b) (AB)T = BT AT (1c) 1 1 1 if individual inverses exist (AB)− = B− A− (1d) 1 T T 1 (A− ) = (A )− (1e) 0.2 trace, determinant and rank AB = A B (2a) j j j jj j 1 1 A− = (2b) j j A j j A = evals (2c) j j Y Tr [A] = evals (2d) X if the cyclic products are well defined; Tr [ABC :::] = Tr [BC ::: A] = Tr [C ::: AB] = ::: (2e) rank [A] = rank AT A = rank AAT (2f) biggest eval condition number = γ = r (2g) smallest eval derivatives of scalar forms with respect to scalars, vectors, or matricies are indexed in the obvious way. similarly, the indexing for derivatives of vectors and matrices with respect to scalars is straightforward. 1 0.3 derivatives of traces @Tr [X] = I (3a) @X @Tr [XA] @Tr [AX] = = AT (3b) @X @X @Tr XT A @Tr AXT = = A (3c) @X @X @Tr XT AX = (A + AT )X (3d) @X 1 @Tr X− A 1 T 1 = X− A X− (3e) @X − 0.4 derivatives of determinants @ AXB 1 T T 1 j j = AXB (X− ) = AXB (X )− (4a) @X j j j j @ ln X j j = (X 1)T = (XT ) 1 (4b) @X − − @ ln X(z) 1 @X j j = Tr X− (4c) @z @z T @ X AX T T 1 T T T 1 j j = X AX (AX(X AX)− + A X(X A X)− ) (4d) @X j j 0.5 derivatives of scalar forms @(aT x) @(xT a) = = a (5a) @x @x @(xT Ax) = (A + AT )x (5b) @x @(aT Xb) = abT (5c) @X @(aT XT b) = baT (5d) @X @(aT Xa) @(aT XT a) = = aaT (5e) @X @X @(aT XT CXb) = CT XabT + CXbaT (5f) @X @ (Xa + b)T C(Xa + b) = (C + CT )(Xa + b)aT (5g) @X 2 the derivative of one vector y with respect to another vector x is a matrix whose (i; j)th element is @y(j)=@x(i).
    [Show full text]