Matrix Differential Calculus with Applications in Statistics and Econometrics WILEY SERIES IN PROBABILITY AND STATISTICS Established by Walter E. Shewhart and Samuel S. Wilks Editors: David J. Balding, Noel A. C. Cressie, Garrett M. Fitzmaurice, Geof H. Givens, Harvey Goldstein, Geert Molenberghs, David W. Scott, Adrian F. M. Smith, and Ruey S. Tsay Editors Emeriti: J. Stuart Hunter, Iain M. Johnstone, Joseph B. Kadane, and Jozef L. Teugels The Wiley Series in Probability and Statistics is well established and author- itative. It covers many topics of current research interest in both pure and applied statistics and probability theory. Written by leading statisticians and institutions, the titles span both state-of-the-art developments in the field and classical methods. Reflecting the wide range of current research in statistics, the series encom- passes applied, methodological, and theoretical statistics, ranging from ap- plications and new techniques made possible by advances in computerized practice to rigorous treatment of theoretical approaches. This series provides essential and invaluable reading for all statisticians, whether in academia, in- dustry, government, or research. A complete list of the titles in this series can be found at http://www.wiley.com/go/wsps Matrix Differential Calculus with Applications in Statistics and Econometrics Third Edition Jan R. Magnus Department of Econometrics and Operations Research, Vrije Universiteit Amsterdam, The Netherlands and Heinz Neudecker† Amsterdam School of Economics, University of Amsterdam, The Netherlands This third edition first published 2019 c 2019 John Wiley & Sons Edition History John Wiley & Sons (1e, 1988) and John Wiley & Sons (2e, 1999) All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, except as permitted by law. Advice on how to obtain permission to reuse material from this title is available at http://www.wiley.com/go/permissions. The right of Jan R. Magnus and Heinz Neudecker to be identified as the authors of this work has been asserted in accordance with law. Registered Offices John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, USA John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, UK Editorial Office 9600 Garsington Road, Oxford, OX4 2DQ, UK For details of our global editorial offices, customer services, and more information about Wiley products visit us at www.wiley.com. Wiley also publishes its books in a variety of electronic formats and by print-on-demand. Some content that appears in standard print versions of this book may not be available in other formats. Limit of Liability/Disclaimer of Warranty While the publisher and authors have used their best efforts in preparing this work, they make no representations or warranties with respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including without limitation any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives, written sales materials or promotional statements for this work. The fact that an organization, website, or product is referred to in this work as a citation and/or potential source of further information does not mean that the publisher and authors endorse the information or services the organization, website, or product may provide or recommendations it may make. This work is sold with the understanding that the publisher is not engaged in rendering professional services. The advice and strategies contained herein may not be suitable for your situation. You should consult with a specialist where appropriate. Further, readers should be aware that websites listed in this work may have changed or disappeared between when this work was written and when it is read. Neither the publisher nor authors shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages. Library of Congress Cataloging-in-Publication Data applied for ISBN: 9781119541202 Cover design by Wiley Cover image: c phochi/Shutterstock Typeset by the author in LATEX 10987654321 Contents Preface ....................................xiii Part One — Matrices 1 Basic properties of vectors and matrices 3 1 Introduction............................. 3 2 Sets ................................. 3 3 Matrices: addition and multiplication . 4 4 Thetransposeofamatrix ..................... 6 5 Squarematrices........................... 6 6 Linearformsandquadraticforms. 7 7 Therankofamatrix........................ 9 8 Theinverse ............................. 10 9 Thedeterminant .......................... 10 10 Thetrace .............................. 11 11 Partitionedmatrices ........................ 12 12 Complexmatrices ......................... 14 13 Eigenvaluesandeigenvectors . 14 14 Schur’sdecompositiontheorem . 17 15 TheJordandecomposition. 18 16 Thesingular-valuedecomposition . 20 17 Furtherresultsconcerningeigenvalues . 20 18 Positive(semi)definitematrices . 23 19 Three further results for positive definite matrices . 25 20 Ausefulresult ........................... 26 21 Symmetricmatrixfunctions . 27 Miscellaneous exercises .......................... 28 Bibliographical notes ........................... 30 2 Kronecker products, vec operator, and Moore-Penrose inverse 31 1 Introduction............................. 31 2 TheKroneckerproduct ...................... 31 3 EigenvaluesofaKroneckerproduct. 33 4 Thevecoperator .......................... 34 5 TheMoore-Penrose(MP)inverse . 36 v vi Contents 6 ExistenceanduniquenessoftheMPinverse . 37 7 SomepropertiesoftheMPinverse . 38 8 Furtherproperties ......................... 39 9 The solution of linear equation systems . 41 Miscellaneous exercises .......................... 43 Bibliographical notes ........................... 45 3 Miscellaneous matrix results 47 1 Introduction............................. 47 2 Theadjointmatrix......................... 47 3 ProofofTheorem3.1........................ 49 4 Bordereddeterminants. .. .. .. .. .. .. .. .. .. 51 5 The matrix equation AX =0 ................... 51 6 TheHadamardproduct ...................... 52 7 The commutation matrix Kmn .................. 54 8 The duplication matrix Dn .................... 56 9 Relationship between Dn+1 and Dn, I .............. 58 10 Relationship between Dn+1 and Dn, II.............. 59 11 Conditions for a quadratic form to be positive (negative) subjecttolinearconstraints . 60 12 Necessary and sufficient conditions for r(A : B)= r(A)+ r(B) 63 13 TheborderedGramianmatrix . 65 14 The equations X1A + X2B′ = G1,X1B = G2 .......... 67 Miscellaneous exercises .......................... 69 Bibliographical notes ........................... 70 Part Two — Differentials: the theory 4 Mathematical preliminaries 73 1 Introduction............................. 73 2 Interiorpointsandaccumulationpoints . 73 3 Openandclosedsets........................ 75 4 TheBolzano-Weierstrasstheorem. 77 5 Functions .............................. 78 6 Thelimitofafunction....................... 79 7 Continuousfunctionsandcompactness . 80 8 Convexsets............................. 81 9 Convexandconcavefunctions. 83 Bibliographical notes ........................... 86 5 Differentials and differentiability 87 1 Introduction............................. 87 2 Continuity.............................. 88 3 Differentiability and linear approximation . 90 4 Thedifferentialofavectorfunction. 91 5 Uniquenessofthedifferential . 93 6 Continuity of differentiable functions . 94 Contents vii 7 Partialderivatives ......................... 95 8 Thefirstidentificationtheorem . 96 9 Existenceofthedifferential,I . 97 10 Existenceofthedifferential,II . 99 11 Continuous differentiability . 100 12 Thechainrule ...........................100 13 Cauchyinvariance .........................102 14 The mean-value theorem for real-valued functions . 103 15 Differentiablematrixfunctions . 104 16 Someremarksonnotation. .106 17 Complexdifferentiation . .108 Miscellaneous exercises ..........................110 Bibliographical notes ...........................110 6 The second differential 111 1 Introduction.............................111 2 Second-orderpartialderivatives . 111 3 TheHessianmatrix.........................112 4 Twice differentiability and second-order approximation, I . 113 5 Definition of twice differentiability . 114 6 Theseconddifferential.. .. .. .. .. .. .. .. .. ..115 7 SymmetryoftheHessianmatrix . .117 8 Thesecondidentificationtheorem . 119 9 Twice differentiability and second-order approximation, II . 119 10 ChainruleforHessianmatrices . 121 11 Theanalogforseconddifferentials . 123 12 Taylor’stheoremforreal-valuedfunctions . 124 13 Higher-orderdifferentials. .125 14 Realanalyticfunctions. .125 15 Twice differentiable matrix functions . 126 Bibliographical notes ...........................127 7 Static optimization 129 1 Introduction.............................129 2 Unconstrainedoptimization . .130 3 Theexistenceofabsoluteextrema . 131 4 Necessaryconditionsforalocalminimum . 132 5 Sufficient conditions for a local minimum: first-derivative test . 134 6 Sufficient conditions for a local minimum: second-derivative test 136 7 Characterization of differentiable convex functions . 138 8 Characterization of twice differentiable convex functions . 141 9 Sufficient conditions for an absolute minimum