January 20, 2011 14:43 World Scientific Book - 9in x 6in book Chapter 1 Matrix Calculus 1.1 Definitions and Notation We assume that the reader is familiar with some basic terms in linear alge- bra such as vector spaces, linearly dependent vectors, matrix addition and matrix multiplication (Horn and Johnson [30], Laub [39]). Throughout we consider matrices over the field of complex numbers C or real number R. Let z 2 C with z = x + iy and x; y 2 R: Thenz ¯ = x − iy: In some cases we restrict the underlying field to the real numbers R: The matrices are denoted by A; B; C; D; X; Y . The matrix elements (entries) of the matrix A are denoted by ajk. For the column vectors we write u, v, w. The zero column vector is denoted by 0. Let A be a matrix. Then AT denotes the transpose and A¯ is the complex conjugate matrix. We call A∗ the adjoint matrix, where A∗ := A¯T . A special role is played by the n × n matrices, i.e. the square matrices. In this case we also say the matrix is of order n. In denotes the n × n unit matrix (also called identity matrix). The zero matrix is denoted by 0. Let V be a vector space of finite dimension n, over the field R of real num- bers, or the field C of complex numbers. If there is no need to distinguish between the two, we speak of the field F of scalars.A basis of V is a set f g n e1; e2;:::; en of n linearly independent vectors of V , denoted by (ej)j=1. Every vector v 2 V then has the unique representation Xn v = vjej j=1 the scalars vj, which we will sometimes denote by (v)j, being the compo- nents of the vector v relative to the basis (e)j. As long as a basis is fixed 1 MATRIX CALCULUS AND KRONECKER PRODUCT - A Practical Approach to Linear and Multilinear Algebra (Second Edition) © World Scientific Publishing Co. Pte. Ltd. http://www.worldscibooks.com/mathematics/8030.html January 20, 2011 14:43 World Scientific Book - 9in x 6in book 2 Matrix Calculus and Kronecker Product unambiguously, it is thus always possible to identify V with Fn. In matrix notation, the vector v will always be represented by the column vector 0 1 v1 B C Bv2 C v = B . C @ . A vn while vT and v∗ will denote the following row vectors T ∗ v = (v1; v2; : : : ; vn); v = (¯v1; v¯2;:::; v¯n) whereα ¯ is the complex conjugate of α. The row vector vT is the transpose of the column vector v, and the row vector v∗ is the conjugate transpose of the column vector v. Definition 1.1. Let Cn be the familiar n dimensional vector space. Let u; v 2 Cn. Then the scalar (or inner) product is defined as Xn (u; v) := u¯jvj: j=1 Obviously (u; v) = (v; u) and (u1 + u2; v) = (u1; v) + (u2; v): Since u and v are considered as column vectors, the scalar product can be written in matrix notation as (u; v) ≡ u∗v: Definition 1.2. Two vectors u; v 2 Cn are called orthogonal if (u; v) = 0: Example 1.1. Let ( ) ( ) 1 1 u = ; v = : 1 −1 Then (u; v) = 0. | MATRIX CALCULUS AND KRONECKER PRODUCT - A Practical Approach to Linear and Multilinear Algebra (Second Edition) © World Scientific Publishing Co. Pte. Ltd. http://www.worldscibooks.com/mathematics/8030.html January 20, 2011 14:43 World Scientific Book - 9in x 6in book Matrix Calculus 3 The scalar product induces a norm of u defined by p kuk := (u; u): In section 1.14 a detailed discussion of norms is given. Definition 1.3. A vector u 2 Cn is called normalized if (u; u) = 1. Example 1.2. The vectors ( ) ( ) 1 1 1 1 u = p ; v = p 2 1 2 −1 are normalized and form an orthonormal basis in the vector space R2. | Let V and W be two vector spaces over the same field, equipped with n m bases (ej)j=1 and (fi)i=1, respectively. Relative to these bases, a linear transformation A : V ! W is represented by the matrix having m rows and n columns 0 1 a11 a12 ··· a1n B C B a21 a22 ··· a2n C A = B . C : @ . .. A am1 am2 ··· amn The elements aij of the matrix A are defined uniquely by the relations Xm Aej = aijfi; j = 1; 2; : : : ; n: i=1 Equivalently, the jth column vector 0 1 a1j B C B a2j C B . C @ . A amj A m of the matrix A represents the vector ej relative to the basis (fi)i=1. We call (ai1 ai2 : : : ain) the ith row vector of the matrix A. A matrix with m rows and n columns is called a matrix of type (m; n), and the vector space over the field F consisting of matrices of type (m; n) with elements in F is denoted by Am;n. MATRIX CALCULUS AND KRONECKER PRODUCT - A Practical Approach to Linear and Multilinear Algebra (Second Edition) © World Scientific Publishing Co. Pte. Ltd. http://www.worldscibooks.com/mathematics/8030.html January 20, 2011 14:43 World Scientific Book - 9in x 6in book 4 Matrix Calculus and Kronecker Product A column vector is then a matrix of type (m; 1) and a row vector a matrix of type (1; n). A matrix is called real or complex according whether its elements are in the field R or the field C. A matrix A with elements aij is written as A = (aij) the first index i always designating the row and the second, j, the column. Definition 1.4. A matrix with all its elements 0 is called the zero matrix or null matrix. ∗ Definition 1.5. Given a matrix A 2 Am;n(C), the matrix A 2 An;m(C) denotes the adjoint of the matrix A and is defined uniquely by the relations ∗ n m (Au; v)m = (u;A v)n for every u 2 C ; v 2 C ∗ which imply that (A )ij =a ¯ji. T Definition 1.6. Given a matrix A = Am;n(R), the matrix A 2 An;m(R) denotes the transpose of a matrix A and is defined uniquely by the relations T n m (Au; v)m = (u;A v)n for every u 2 R ; v 2 R T which imply that (A )ij = aji. To the composition of linear transformations there corresponds the multi- plication of matrices. Definition 1.7. If A = (aij) is a matrix of type (m; l) and B = (bkj ) of type (l; n), their matrix product AB is the matrix of type (m; n) defined by Xl (AB)ij = aikbkj: k=1 We have (AB)T = BT AT ; (AB)∗ = B∗A∗: Note that AB =6 BA, in general, where A and B are n × n matrices. Definition 1.8. A matrix of type (n; n) is said to be square, or a matrix of order n if it is desired to make explicit the integer n; it is convenient to speak of a matrix as rectangular if it is not necessarily square. Definition 1.9. If A = (aij) is a square matrix, the elements aii are called diagonal elements, and the elements aij; i =6 j, are called off-diagonal ele- ments. MATRIX CALCULUS AND KRONECKER PRODUCT - A Practical Approach to Linear and Multilinear Algebra (Second Edition) © World Scientific Publishing Co. Pte. Ltd. http://www.worldscibooks.com/mathematics/8030.html January 20, 2011 14:43 World Scientific Book - 9in x 6in book Matrix Calculus 5 Definition 1.10. The identity matrix (also called unit matrix) is the square matrix I := (δij): Definition 1.11. A square matrix A is invertible if there exists a matrix (which is unique, if it does exist), written as A−1 and called the inverse of the matrix A, which satisfies AA−1 = A−1A = I: Otherwise, the matrix is said to be singular. Recall that if A and B are invertible matrices then (AB)−1 = B−1A−1; (AT )−1 = (A−1)T ; (A∗)−1 = (A−1)∗: Definition 1.12. A square matrix A is symmetric if A is real and A = AT . The sum of two symmetric matrices is again a symmetric matrix. Definition 1.13. A square matrix A is skew-symmetric if A is real and A = −AT . Every square matrix A over R can be written as sum of a symmetric matrix S and a skew-symmetric matrix T , i.e. A = T + S. Thus 1 1 S = (A + AT ); T = (A − AT ) : 2 2 Definition 1.14. A square matrix A over C is Hermitian if A = A∗. The sum of two Hermitian matrices is again a Hermitian matrix. Definition 1.15. A square matrix A over C is skew-Hermitian if A = −A∗. The sum of two skew-Hermitian matrices is again a skew-Hermitian matrix. Definition 1.16. A Hermitian n × n matrix A is positive semidefinite if u∗Ax ≥ 0 for all nonzero u 2 Cn. Example 1.3. The 2 × 2 matrix ( ) 1 1 A = 1 1 is positive semidefinite. | MATRIX CALCULUS AND KRONECKER PRODUCT - A Practical Approach to Linear and Multilinear Algebra (Second Edition) © World Scientific Publishing Co. Pte. Ltd. http://www.worldscibooks.com/mathematics/8030.html January 20, 2011 14:43 World Scientific Book - 9in x 6in book 6 Matrix Calculus and Kronecker Product Let B be an arbitrary m × n matrix over C.