ECE 302: Lecture 1.4 Linear Algebra (Optional) Prof Stanley Chan School of Electrical and Computer Engineering Purdue University c Stanley Chan 2020. All Rights Reserved. 1 / 15 Outline 1.1 Infinite Series 1.1.1. Geometric Series 1.1.2. Binomial Series 1.2 Approximations 1.2.1. Taylor Approximation 1.2.2. Exponential Series 1.2.3. Logarithmic Approximation 1.3 Integration 1.3.1. Odd and Even Functions 1.3.2. Fundamental Theorem of Calculus 1.4 Linear Algebra (Optional) 1.4.1. Inner Products (Optional) 1.4.2. Matrix Calculus (Optional) 1.4.3. Matrix Inversion (Optional) 1.5 Combinatorics 1.5.1. Permutation 1.5.2. Combination c Stanley Chan 2020. All Rights Reserved. 2 / 15 Basic Notation n Vector: x 2 R m×n Matrix: A 2 R ; Entries are aij or [A]ij . Transpose: 2| aT |3 2 3 1 j j j aT 6| 2 |7 A = a a ::: a ; and AT = 6 7 : 4 1 2 n5 6 . 7 j j j 4 . 5 aT | n | Column: ai is the i-th column of A Identity matrix I All-one vector 1 and all-zero vector 0 Standard basis ei . c Stanley Chan 2020. All Rights Reserved. 3 / 15 Inner Product Definition T T Let x = [x1; x2;:::; xN ] and y = [y1; y2;:::; yN ] be two vectors. The inner product xT y is T Example. Let x = [x1; x2] . The inner product xT x = c Stanley Chan 2020. All Rights Reserved. 4 / 15 Inner product Figure: Geometric interpretation of inner product: We project one vector onto the other vector. The projected distance is the inner product. c Stanley Chan 2020. All Rights Reserved. 5 / 15 Weighted Inner Product Example. a 0 Let x = [x ; x ]T , µ = [µ ; µ ] and C = . The product 1 2 1 2 0 b (x − µ)T C(x − µ) is c Stanley Chan 2020. All Rights Reserved. 6 / 15 The `2-norm Also called the Euclidean norm: Definition v u n x uX 2 k k2 = t xi : (1) i=1 The set Ω = fx j kxk2 ≤ rg defines a circle: x x 2 2 2 Ω = f j k k2 ≤ rg = f(x1; x2) j x1 + x2 ≤ r g: x x x x 2 f ( ) = k k2 is not the same as f ( ) = k k2. Triangle inequality holds: kx + yk2 ≤ kxk2 + kyk2: c Stanley Chan 2020. All Rights Reserved. 7 / 15 Outline 1.1 Infinite Series 1.1.1. Geometric Series 1.1.2. Binomial Series 1.2 Approximations 1.2.1. Taylor Approximation 1.2.2. Exponential Series 1.2.3. Logarithmic Approximation 1.3 Integration 1.3.1. Odd and Even Functions 1.3.2. Fundamental Theorem of Calculus 1.4 Linear Algebra (Optional) 1.4.1. Inner Products (Optional) 1.4.2. Matrix Calculus (Optional) 1.4.3. Matrix Inversion (Optional) 1.5 Combinatorics 1.5.1. Permutation 1.5.2. Combination c Stanley Chan 2020. All Rights Reserved. 8 / 15 Matrix Calculus Definition n n Let f : R ! R be a scalar field. The gradient of f with respect to x 2 R is defined as 2 @f (x) 3 @x1 6 . 7 rx f (x) = 6 . 7 : (2) 4 @f (x) 5 @xn Example 1. f (x) = aT x. In this case, the gradient is 2 @f (x) 3 2 @ Pn 3 2 3 aj xj a1 @x1 @x1 j=1 aT x 6 . 7 6 . 7 6 . 7 a rx = 6 . 7 = 4 . 5 = 4 . 5 = : (3) 4 @f (x) 5 @ Pn aj xj an @xn @xn j=1 c Stanley Chan 2020. All Rights Reserved. 9 / 15 More Examples Example 2. f (x) = xT Ax. Then, 2 @f (x) 3 2 @ Pn 3 aij xi xj @x1 @x1 i;j=1 xT Ax 6 . 7 6 . 7 rx = 6 . 7 = 4 . 5 4 @f (x) 5 @ Pn aij xi xj @xn @xn i;j=1 2Pn 3 2Pn 3 j=1 a1;j xj i=1 ai;1xi 6 . 7 6 . 7 Ax AT x = 4 . 5 + 4 . 5 = + Pn Pn j=1 an;j xj i=1 ai;nxi T If A is symmetric so that A = A then rx f (x) = 2Ax c Stanley Chan 2020. All Rights Reserved. 10 / 15 More Examples Example 3. f (x) = kAx − yk2. The gradient is 2 T T T T rx kAx − yk = rx x A Ax − 2y Ax + y y T T T T = rx x A Ax − 2rx y Ax + rx y y = 2AT Ax − 2AT y + 0 = 2AT (Ax − y): Definition n The Hessian of f with respect to x 2 R is defined as 2 @2f (x) @2f (x) 3 2 ::: @x @x @x1 1 n 2 x 6 . 7 rx f ( ) = 6 . .. 7 : (4) 4 5 @2f (x) @2f (x) ::: 2 @xn@x1 @xn c Stanley Chan 2020. All Rights Reserved. 11 / 15 Outline 1.1 Infinite Series 1.1.1. Geometric Series 1.1.2. Binomial Series 1.2 Approximations 1.2.1. Taylor Approximation 1.2.2. Exponential Series 1.2.3. Logarithmic Approximation 1.3 Integration 1.3.1. Odd and Even Functions 1.3.2. Fundamental Theorem of Calculus 1.4 Linear Algebra (Optional) 1.4.1. Inner Products (Optional) 1.4.2. Matrix Calculus (Optional) 1.4.3. Matrix Inversion (Optional) 1.5 Combinatorics 1.5.1. Permutation 1.5.2. Combination c Stanley Chan 2020. All Rights Reserved. 12 / 15 Determinant and Inverse Definition a b Determinant Let Σ = , the determinant of Σ is c d Definition (Inverse) a b Let Σ = , the inverse of Σ is c d c Stanley Chan 2020. All Rights Reserved. 13 / 15 Questions? c Stanley Chan 2020. All Rights Reserved. 14 / 15.