Introduction to Applied Linear Algebra Vectors, Matrices, and Least Squares Julia Language Companion Stephen Boyd and Lieven Vandenberghe DRAFT March 28, 2021 Contents Preface vii Getting started with Julia ix 1 Vectors 1 1.1 Vectors ................................... 1 1.2 Vector addition .............................. 9 1.3 Scalar-vector multiplication ........................ 9 1.4 Inner product ............................... 14 1.5 Complexity of vector computations .................... 15 2 Linear functions 17 2.1 Linear functions .............................. 17 2.2 Taylor approximation ........................... 18 2.3 Regression model ............................. 19 3 Norm and distance 23 3.1 Norm .................................... 23 3.2 Distance .................................. 25 3.3 Standard deviation ............................. 26 3.4 Angle .................................... 27 3.5 Complexity ................................. 28 4 Clustering 31 4.1 Clustering ................................. 31 4.2 A clustering objective ........................... 31 4.3 The k-means algorithm .......................... 32 4.4 Examples .................................. 35 4.5 Applications ................................ 39 5 Linear independence 41 5.1 Linear dependence ............................. 41 5.2 Basis .................................... 41 5.3 Orthonormal vectors ............................ 42 5.4 Gram–Schmidt algorithm ......................... 43 iv Contents 6 Matrices 47 6.1 Matrices .................................. 47 6.2 Zero and identity matrices ........................ 52 6.3 Transpose, addition, and norm ...................... 56 6.4 Matrix-vector multiplication ........................ 58 6.5 Complexity ................................. 61 7 Matrix examples 63 7.1 Geometric transformations ........................ 63 7.2 Selectors .................................. 63 7.3 Incidence matrix .............................. 65 7.4 Convolution ................................ 67 8 Linear equations 69 8.1 Linear and affine functions ........................ 69 8.2 Linear function models .......................... 71 8.3 Systems of linear equations ........................ 73 9 Linear dynamical systems 75 9.1 Linear dynamical systems ......................... 75 9.2 Population dynamics ........................... 76 9.3 Epidemic dynamics ............................ 77 9.4 Motion of a mass ............................. 77 9.5 Supply chain dynamics .......................... 78 10 Matrix multiplication 81 10.1 Matrix-matrix multiplication ....................... 81 10.2 Composition of linear functions ...................... 82 10.3 Matrix power ............................... 83 10.4 QR factorization .............................. 84 11 Matrix inverses 87 11.1 Left and right inverses ........................... 87 11.2 Inverse ................................... 88 11.3 Solving linear equations .......................... 89 11.4 Examples .................................. 92 11.5 Pseudo-inverse ............................... 93 12 Least squares 95 12.1 Least squares problem ........................... 95 12.2 Solution .................................. 96 12.3 Solving least squares problems ...................... 97 12.4 Examples .................................. 98 13 Least squares data fitting 101 13.1 Least squares data fitting ......................... 101 13.2 Validation ................................. 107 13.3 Feature engineering ............................ 110 Contents v 14 Least squares classification 115 14.1 Classification ................................ 115 14.2 Least squares classifier ........................... 116 14.3 Multi-class classifiers ........................... 117 15 Multi-objective least squares 123 15.1 Multi-objective least squares ....................... 123 15.2 Control ................................... 126 15.3 Estimation and inversion ......................... 126 15.4 Regularized data fitting .......................... 127 15.5 Complexity ................................. 129 16 Constrained least squares 131 16.1 Constrained least squares problem .................... 131 16.2 Solution .................................. 133 16.3 Solving contrained least squares problems ................ 134 17 Constrained least squares applications 137 17.1 Portfolio optimization ........................... 137 17.2 Linear quadratic control .......................... 139 17.3 Linear quadratic state estimation ..................... 145 18 Nonlinear least squares 147 18.1 Nonlinear equations and least squares .................. 147 18.2 Gauss–Newton algorithm ......................... 147 18.3 Levenberg–Marquardt algorithm ..................... 149 18.4 Nonlinear model fitting .......................... 154 18.5 Nonlinear least squares classification ................... 157 19 Constrained nonlinear least squares 159 19.1 Constrained nonlinear least squares .................... 159 19.2 Penalty algorithm ............................. 159 19.3 Augmented Lagrangian algorithm ..................... 161 19.4 Nonlinear control ............................. 163 Appendices 165 A The VMLS package 167 A.1 Utility functions .............................. 167 A.2 Algorithms ................................. 168 A.3 Data sets .................................. 169 vi Contents Preface This Julia Language Companion accompanies our book Introduction to Applied Linear Algebra: Vectors, Matrices, and Least Squares (referred to here as VMLS). It is meant to show how the ideas and methods in VMLS can be expressed and implemented in the programming language Julia. We assume that the reader has installed Julia, or is using Juliabox online, and understands the basics of the language. We also assume that the reader has care- fully read the relevant sections of VMLS. The organization of the Julia Language Companion follows VMLS, section by section. You will see that mathematical notation and Julia syntax are pretty close, but not the same. You must be careful to never confuse mathematical notation and Julia syntax. In these notes we write mathematical notation (as in VMLS) in standard mathematics font, e.g., y = Ax. Julia code, expressions, and snippets are written in a fixed-width typewriter font, e.g., y = A*x. We encourage you to cut and paste our Julia code snippets into a Julia interactive session or notebook, to test them out, and maybe modify them and run them again. You can also use our snippets as templates for your own Julia code. At some point we will collect the Julia snippets in this companion document into Julia notebooks that you can easily run. Julia is a very powerful language, but in this companion document we use only a small and limited set of its features. The code snippets in this companion are written so as to be transparent and simple, and to emphasize closeness to the concepts in VMLS. Some of the code snippets could be written in a much more compact way, or in a way that would result in faster or more efficient execution. The code snippets in this companion document are compatible with Julia 1.0, which is not quite the same as earlier versions, like Julia 0.6. Some of the functions we use are in the standard packages LinearAlgebra, SparseArrays, and Plots, and a few others. We have created a simple and small Julia package VMLS, which includes a small number of simple Julia functions that are useful for studying the material in VMLS. The next section, Getting started with Julia, explains how to install these Julia packages. We consider this companion to be a draft. We’ll be updating it occasionally, adding more examples and fixing typos as we become aware of them. So you may wish to periodically check whether a newer version is available. Stephen Boyd Stanford, California Lieven Vandenberghe Los Angeles, California Getting started with Julia Installing Julia. Download Julia 1.0 or higher from its website, and then follow the instructions to install it on your platform. You’ll want to make sure it’s working before proceeding to install additional packages, as described below. Installing packages. While most of the Julia code we use in this companion is from the base or core of the Julia language, several important functions are con- tained in other packages that you must explicitly install. Here we explain how to do this. The installation has to be done only once. To add (install) a Julia package you use the package manager system, which is entered by typing ] to the Julia prompt. You’ll then see the package manager prompt, (@v1.6) pkg>. If you type ?, you’ll see a list of package manager com- mands. You exit the package manager by typing Ctrl-C (Control-C), which returns you to the julia prompt. To add a package called PackageName, type add PackageName to the package control prompt. It may take a few minutes to get the required packages, compile, and install them. If you type status, you’ll see what packages are installed. If you type up, installed packages will be upgraded, if needed. This companion will use functions from the following packages: • LinearAlgebra • SparseArrays • Plots On a few occasions we use functions from the packages Random and DSP, but we will mention this in the text when we use these functions. Here’s what it looks like when you install the Plots package. julia> ] (@v1.6) pkg> add Plots Updating registry at `~/.julia/registries/General` Updating git-repo `https://github.com/JuliaRegistries/General.git` Resolving package versions... Updating `~/.julia/environments/v1.6/Project.toml`