ECE 302: Lecture 1.4 Linear Algebra (Optional)

Prof Stanley Chan

School of Electrical and Computer Engineering Purdue University

1.1 Inﬁnite Series 1.1.1. Geometric Series 1.1.2. Binomial Series 1.2 Approximations 1.2.1. Taylor Approximation 1.2.2. Exponential Series 1.2.3. Logarithmic Approximation 1.3 Integration 1.3.1. Odd and Even Functions 1.3.2. Fundamental Theorem of Calculus 1.4 Linear Algebra (Optional) 1.4.1. Inner Products (Optional) 1.4.2. Matrix Calculus (Optional) 1.4.3. Matrix Inversion (Optional) 1.5 Combinatorics 1.5.1. Permutation 1.5.2. Combination c Stanley Chan 2020. All Rights Reserved. 2 / 15 Basic Notation

n Vector: x ∈ R m×n Matrix: A ∈ R ; Entries are aij or [A]ij . Transpose:

— aT —   1 | | | aT — 2 — A = a a ... a , and AT =   .  1 2 n  .  | | |  .  aT — n —

Column: ai is the i-th column of A Identity matrix I All-one vector 1 and all-zero vector 0

Standard basis ei .

Deﬁnition T T Let x = [x1, x2,..., xN ] and y = [y1, y2,..., yN ] be two vectors. The inner product xT y is

T Example. Let x = [x1, x2] . The inner product xT x =

Figure: Geometric interpretation of inner product: We project one vector onto the other vector. The projected distance is the inner product.

Example. a 0 Let x = [x , x ]T , µ = [µ , µ ] and C = . The product 1 2 1 2 0 b (x − µ)T C(x − µ) is

The set Ω = {x | kxk2 ≤ r} deﬁnes a circle: x x 2 2 2 Ω = { | k k2 ≤ r} = {(x1, x2) | x1 + x2 ≤ r }. x x x x 2 f ( ) = k k2 is not the same as f ( ) = k k2. Triangle inequality holds:

kx + yk2 ≤ kxk2 + kyk2.

Definition n n Let f : R → R be a scalar field. The gradient of f with respect to x ∈ R is defined as  ∂f (x)  ∂x1  .  ∇x f (x) =  .  . (2)  ∂f (x)  ∂xn

Example 1. f (x) = aT x. In this case, the gradient is

 ∂f (x)   ∂ Pn    aj xj a1 ∂x1 ∂x1 j=1 aT x  .   .   .  a ∇x =  .  =  .  =  .  = . (3)  ∂f (x)  ∂ Pn aj xj an ∂xn ∂xn j=1

Example 2. f (x) = xT Ax. Then,

 ∂f (x)   ∂ Pn  aij xi xj ∂x1 ∂x1 i,j=1 xT Ax  .   .  ∇x =  .  =  .   ∂f (x)  ∂ Pn aij xi xj ∂xn ∂xn i,j=1 Pn  Pn  j=1 a1,j xj i=1 ai,1xi  .   .  Ax AT x =  .  +  .  = + Pn Pn j=1 an,j xj i=1 ai,nxi

T If A is symmetric so that A = A then ∇x f (x) = 2Ax

Example 3. f (x) = kAx − yk2. The gradient is

2 T T T T ∇x kAx − yk = ∇x x A Ax − 2y Ax + y y

T T T T = ∇x x A Ax − 2∇x y Ax + ∇x y y = 2AT Ax − 2AT y + 0 = 2AT (Ax − y).

Deﬁnition n The Hessian of f with respect to x ∈ R is deﬁned as

 ∂2f (x) ∂2f (x)  2 ... ∂x ∂x ∂x1 1 n 2 x  . . .  ∇x f ( ) =  . .. .  . (4)   ∂2f (x) ∂2f (x) ... 2 ∂xn∂x1 ∂xn

Deﬁnition a b Determinant Let Σ = , the determinant of Σ is c d

Deﬁnition (Inverse) a b Let Σ = , the inverse of Σ is c d