Mathematical Methods
Total Page:16
File Type:pdf, Size:1020Kb
LECTURE NOTES IMPERIAL COLLEGE LONDON DEPARTMENT OF COMPUTING Mathematical Methods Marc Deisenroth, Mahdi Cheraghchi Version: December 1, 2017 Contents 1 Course Overview3 2 Sequences6 2.1 The Convergence Definition.......................6 2.2 Illustration of Convergence.......................7 2.3 Common Converging Sequences.....................8 2.4 Combinations of Sequences.......................9 2.5 Sandwich Theorem............................9 2.6 Ratio Tests for Sequences........................ 11 2.7 Proof of Ratio Tests............................ 12 2.8 Useful Techniques for Manipulating Absolute Values......... 13 2.9 Properties of Real Numbers....................... 14 3 Series 15 3.1 Geometric Series............................. 16 3.2 Harmonic Series............................. 16 3.3 Series of Inverse Squares......................... 17 3.4 Common Series and Convergence.................... 18 3.5 Convergence Tests............................ 19 3.6 Absolute Convergence.......................... 22 3.7 Power Series and the Radius of Convergence.............. 23 3.8 Proofs of Ratio Tests........................... 24 4 Power Series 27 4.1 Basics of Power Series.......................... 27 4.2 Maclaurin Series............................. 27 4.3 Taylor Series............................... 31 4.4 Taylor Series Error Term......................... 33 4.5 Deriving the Cauchy Error Term..................... 34 4.6 Power Series Solution of ODEs..................... 34 5 Linear Algebra 36 5.1 Linear Equation Systems......................... 38 5.2 Groups................................... 39 5.2.1 Definitions............................ 39 5.2.2 Examples............................. 40 2 Table of Contents CONTENTS 5.3 Matrices.................................. 40 5.3.1 Matrix Multiplication....................... 40 5.3.2 Inverse and Transpose...................... 42 5.3.3 Multiplication by a Scalar.................... 43 5.3.4 Compact Representations of Linear Equation Systems..... 43 5.4 Gaussian Elimination........................... 43 5.4.1 Example: Solving a Simple Linear Equation System...... 44 5.4.2 Elementary Transformations................... 45 5.4.3 The Minus-1 Trick for Solving Homogeneous Equation Systems 50 5.4.4 Applications of Gaussian Elimination in Linear Algebra.... 51 5.5 Vector Spaces............................... 53 5.5.1 Examples............................. 54 5.5.2 Vector Subspaces......................... 55 5.6 Linear Independence........................... 56 5.7 Generating Sets, Basis and Dimension................. 60 5.7.1 Generating Sets.......................... 60 5.7.2 Basis................................ 60 5.7.3 Rank................................ 64 5.8 Intersection of Subspaces........................ 65 5.9 Linear Mappings............................. 67 5.9.1 Image and Kernel (Null Space)................. 68 5.9.2 Matrix Representation of Linear Mappings........... 70 5.9.3 Basis Change........................... 72 5.10 Determinants............................... 76 5.11 Eigenvalues................................ 79 5.11.1 Characteristic Polynomial.................... 80 5.11.2 Example: Eigenspace Computation............... 81 5.11.3 Eigenvalues in Practical Applications.............. 82 5.12 Diagonalization.............................. 84 5.12.1 Applications............................ 88 5.12.2 Cayley-Hamilton Theorem.................... 88 5.13 Inner Products.............................. 89 5.13.1 Lengths, Distances, Orthogonality................ 90 5.13.2 Applications of Inner Products.................. 91 5.14 Orthogonal Projections.......................... 92 5.14.1 Projection onto 1-Dimensional Subspaces (Lines)....... 93 5.14.2 Projection onto General Subspaces............... 95 5.14.3 Applications of Projections.................... 97 5.15 Affine Subspaces............................. 98 5.15.1 Intersection of Affine Subspaces................. 100 5.16 Affine Mappings............................. 104 5.17 Rotations................................. 105 5.17.1 Rotations in the Plane...................... 106 5.17.2 Rotations in Three Dimensions................. 106 5.17.3 Rotations in n Dimensions.................... 107 3 CONTENTS Table of Contents 5.17.4 Properties of Rotations...................... 108 4 CONTENTS Contents Preface These lecture notes are intended for undergraduate students of computer science at Imperial College London and cover basic mathematical concepts that are required in other courses. The lecture notes are based on Jeremy Bradley’s material from 2014/15, but the course has been partially restructured since then. Many people have helped improving these lecture notes by providing comments, suggestions and corrections. In particular, we want to thank Romain Barnoud, Ruhi Choudhury and Tony Field for valuable feedback and additions. Marc Deisenroth and Mahdi Cheraghchi August 2017 1 Contents CONTENTS 2 Chapter 1 Course Overview This introductory course covers background that is essential to many other courses in the Department of Computing. We will approximately split this course into two parts: Analysis and Linear Algebra. In Analysis, we will cover topics, including power series and convergence of sequences and series. The part on Linear Algebra will cover linear equation systems (and their solution), linear mappings and their matrix representations, vector spaces, eigenvalues, affine mappings, and applications, e.g., projections and rotations. There are many courses in the department that require an understanding of math- ematical concepts. These courses range from optimization, operations research and complexity to machine learning, robotics, graphics and computer vision. The department offers additional courses that cover other and more advanced math- ematical concepts: Probability and Statistics (2nd year), Computational Techniques (2nd year), Mathematics for Inference and Machine Learning (4th year). Figures 1.1–1.2 give an overview of the relevance of this course to other courses in the Department of Computing. Figure 1.1 details the topics from this course that are relevant in other courses, whereas Figure 1.2 shows which courses in Years 2–4 rely on material covered in this introductory course. 3 Chapter 1. Course Overview Computational Techniques Mathematics for Inference and Machine Learning Information and Coding Theory Robotics Linear Equation Systems Simulation & Modelling Machine Learning Operations Research Learning in Autonomous Systems Computational Optimisation Graphics Groups Information and Coding Theory Advanced Robotics Medical Image Computing Computational Optimisation Mathematics for Inference and Machine Learning Data Analysis and Probabilistic Inference Advanced Robotics Graphics Robotics Linear Algebra Machine Learning Eigenvalues Operations Research Computer Vision Advanced Statistical Machine Learning and Pattern Recognition Quantum Computing Computational Techniques Advanced Computer Graphics Computational Optimisation Learning in Autonomous Systems Computational Techniques Advanced Computer Graphics Robotics Linear Mappings Mathematical Machine Learning Advanced Robotics Methods Information and Coding Theory Quantum Computing Medical Image Computing Medical Image Computing Computational Optimisation Advanced Robotics Computer Vision Advanced Computer Graphics Affine Mappings Robotics Information and Coding Theory Graphics Graphics Power Series Advanced Computer Graphics Operations Research Algorithms Convergence Computational Optimisation Analysis Computational Techniques Algorithms Discrete Mathematics Sequences/Series Computational Optimisation Introduction to AI Figure 1.1: Overview of the relevance of topics covered in this course to other courses in the Department of Computing. 4 Chapter 1. Course Overview Computational Techniques Mathematics for Inference and Machine Learning Information and Coding Theory Robotics Linear Equation Systems Simulation & Modelling Machine Learning Operations Research Learning in Autonomous Systems Computational Optimisation Graphics Groups Information and Coding Theory Advanced Robotics Medical Image Computing Computational Optimisation Mathematics for Inference and Machine Learning Data Analysis and Probabilistic Inference Advanced Robotics Graphics Robotics Linear Algebra Machine Learning Eigenvalues Operations Research Computer Vision Advanced Statistical Machine Learning and Pattern Recognition Quantum Computing Computational Techniques Advanced Computer Graphics Computational Optimisation Learning in Autonomous Systems Computational Techniques Advanced Computer Graphics Robotics Linear Mappings Mathematical Machine Learning Advanced Robotics Methods Information and Coding Theory Quantum Computing Medical Image Computing Medical Image Computing Computational Optimisation Advanced Robotics Computer Vision Advanced Computer Graphics Affine Mappings Robotics Information and Coding Theory Graphics Graphics Power Series Advanced Computer Graphics Operations Research Algorithms Convergence Computational Optimisation Analysis Computational Techniques Algorithms Discrete Mathematics Sequences/Series Computational Optimisation Introduction to AI Figure 1.2: Overview of courses (ordered by year) in the Department of Computing that rely on topics covered in this course. 5 Chapter 2 Sequences 2.1 The Convergence Definition Given a sequence of real numbers a1;a2;:::, we would like to have a precise definition of what it means for this sequence to converge to a limit l. You may already