From Algebraic Structures to Tensors Matrices and Tensors in Signal Processing Set coordinated by Gérard Favier Volume 1 From Algebraic Structures to Tensors Edited by Gérard Favier First published 2019 in Great Britain and the United States by ISTE Ltd and John Wiley & Sons, Inc. Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced, stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers, or in the case of reprographic reproduction in accordance with the terms and licenses issued by the CLA. Enquiries concerning reproduction outside these terms should be sent to the publishers at the undermentioned address: ISTE Ltd John Wiley & Sons, Inc. 27-37 St George’s Road 111 River Street London SW19 4EU Hoboken, NJ 07030 UK USA www.iste.co.uk www.wiley.com © ISTE Ltd 2019 The rights of Gérard Favier to be identified as the author of this work have been asserted by him in accordance with the Copyright, Designs and Patents Act 1988. Library of Congress Control Number: 2019945792 British Library Cataloguing-in-Publication Data A CIP record for this book is available from the British Library ISBN 978-1-78630-154-3 Contents Preface ........................................ xi Chapter 1. Historical Elements of Matrices and Tensors . 1 Chapter 2. Algebraic Structures ........................ 9 2.1. A few historical elements . 9 2.2. Chapter summary . 11 2.3. Sets . 12 2.3.1. Definitions . 12 2.3.2. Sets of numbers . 13 2.3.3. Cartesian product of sets . 13 2.3.4. Set operations . 14 2.3.5. De Morgan’s laws . 15 2.3.6. Characteristic functions . 15 2.3.7. Partitions . 16 2.3.8. σ-algebras or σ-fields . 16 2.3.9. Equivalence relations . 16 2.3.10. Order relations . 17 2.4. Maps and composition of maps . 17 2.4.1. Definitions . 17 2.4.2. Properties . 18 2.4.3. Composition of maps . 18 2.5. Algebraic structures . 18 2.5.1. Laws of composition . 18 2.5.2. Definition of algebraic structures . 22 2.5.3. Substructures . 24 2.5.4. Quotient structures . 24 2.5.5. Groups . 24 vi From Algebraic Structures to Tensors 2.5.6. Rings . 27 2.5.7. Fields . 32 2.5.8. Modules . 33 2.5.9. Vector spaces . 33 2.5.10. Vector spaces of linear maps . 38 2.5.11. Vector spaces of multilinear maps . 39 2.5.12. Vector subspaces . 41 2.5.13. Bases . 43 2.5.14. Sum and direct sum of subspaces . 45 2.5.15. Quotient vector spaces . 47 2.5.16. Algebras . 47 2.6. Morphisms . 49 2.6.1. Group morphisms . 49 2.6.2. Ring morphisms . 51 2.6.3. Morphisms of vector spaces or linear maps . 51 2.6.4. Algebra morphisms . 56 Chapter 3. Banach and Hilbert Spaces – Fourier Series and Orthogonal Polynomials ............................. 57 3.1. Introduction and chapter summary . 57 3.2. Metric spaces . 59 3.2.1. Definition of distance . 60 3.2.2. Definition of topology . 60 3.2.3. Examples of distances . 61 3.2.4. Inequalities and equivalent distances . 62 3.2.5. Distance and convergence of sequences . 62 3.2.6. Distance and local continuity of a function . 62 3.2.7. Isometries and Lipschitzian maps . 63 3.3. Normed vector spaces . 63 3.3.1. Definition of norm and triangle inequalities . 63 3.3.2. Examples of norms . 64 3.3.3. Equivalent norms . 68 3.3.4. Distance associated with a norm . 69 3.4. Pre-Hilbert spaces . 69 3.4.1. Real pre-Hilbert spaces . 70 3.4.2. Complex pre-Hilbert spaces . 70 3.4.3. Norm induced from an inner product . 72 3.4.4. Distance associated with an inner product . 75 3.4.5. Weighted inner products . 76 3.5. Orthogonality and orthonormal bases . 76 3.5.1. Orthogonal/perpendicular vectors and Pythagorean theorem . 76 3.5.2. Orthogonal subspaces and orthogonal complement . 77 3.5.3. Orthonormal bases . 79 3.5.4. Orthogonal/unitary endomorphisms and isometries . 79 Contents vii 3.6. Gram–Schmidt orthonormalization process . 80 3.6.1. Orthogonal projection onto a subspace . 80 3.6.2. Orthogonal projection and Fourier expansion . 80 3.6.3. Bessel’s inequality and Parseval’s equality . 82 3.6.4. Gram–Schmidt orthonormalization process . 83 3.6.5. QR decomposition . 85 3.6.6. Application to the orthonormalization of a set of functions . 86 3.7. Banach and Hilbert spaces . 88 3.7.1. Complete metric spaces . 88 3.7.2. Adherence, density and separability . 90 3.7.3. Banach and Hilbert spaces . 91 3.7.4. Hilbert bases . 93 3.8. Fourier series expansions . 97 3.8.1. Fourier series, Parseval’s equality and Bessel’s inequality . 97 3.8.2. Case of 2π-periodic functions from R to C ............. 97 3.8.3. T -periodic functions from R to C .................. 102 3.8.4. Partial Fourier sums and Bessel’s inequality . 102 3.8.5. Convergence of Fourier series . 103 3.8.6. Examples of Fourier series . 108 3.9. Expansions over bases of orthogonal polynomials . 117 Chapter 4. Matrix Algebra ............................ 123 4.1. Chapter summary . 123 4.2. Matrix vector spaces . 124 4.2.1. Notations and definitions . 124 4.2.2. Partitioned matrices . 125 4.2.3. Matrix vector spaces . 126 4.3. Some special matrices . 127 4.4. Transposition and conjugate transposition . 128 4.5. Vectorization . 130 4.6. Vector inner product, norm and orthogonality . 130 4.6.1. Inner product . 130 4.6.2. Euclidean/Hermitian norm . 131 4.6.3. Orthogonality . 131 4.7. Matrix multiplication . 132 4.7.1. Definition and properties . 132 4.7.2. Powers of a matrix . 134 4.8. Matrix trace, inner product and Frobenius norm . 137 4.8.1. Definition and properties of the trace . 137 4.8.2. Matrix inner product . 138 4.8.3. Frobenius norm . 138 4.9. Subspaces associated with a matrix . 139 viii From Algebraic Structures to Tensors 4.10. Matrix rank . 141 4.10.1. Definition and properties . 141 4.10.2. Sum and difference rank . 143 4.10.3. Subspaces associated with a matrix product . 143 4.10.4. Product rank . 144 4.11. Determinant, inverses and generalized inverses . 145 4.11.1. Determinant . 145 4.11.2. Matrix inversion . 148 4.11.3. Solution of a homogeneous system of linear equations . 149 4.11.4. Complex matrix inverse . 150 4.11.5. Orthogonal and unitary matrices . 150 4.11.6. Involutory matrices and anti-involutory matrices . 151 4.11.7. Left and right inverses of a rectangular matrix . 153 4.11.8. Generalized inverses . 155 4.11.9. Moore–Penrose.