Linear Programming Lecture 1: Linear Algebra Review Lecture 1: Linear Algebra Review Linear Programming 1 / 24 1 Linear Algebra Review 2 Linear Algebra Review 3 Block Structured Matrices 4 Gaussian Elimination Matrices 5 Gauss-Jordan Elimination (Pivoting) Lecture 1: Linear Algebra Review Linear Programming 2 / 24 Matrices in Rm×n A 2 Rm×n columns rows 2 3 2 3 a11 a12 ::: a1n a1• 6 a21 a22 ::: a2n 7 6 a2• 7 A = 6 7 = a a ::: a = 6 7 6 . .. 7 •1 •2 •n 6 . 7 4 . 5 4 . 5 am1 am2 ::: amn am• 2 3 2 T 3 a11 a21 ::: am1 a•1 T 6 a12 a22 ::: am2 7 6 a•2 7 AT = 6 7 = 6 7 = aT aT ::: aT 6 . .. 7 6 . 7 1• 2• m• 4 . 5 4 . 5 T a1n a2n ::: amn a•n Lecture 1: Linear Algebra Review Linear Programming 3 / 24 Matrix Vector Multiplication A column space view of matrix vector multiplication. 2 a11 a12 ::: a1n 3 2 x1 3 2 a11 3 2 a12 3 2 a1n 3 6 a21 a22 ::: a2n 7 6 x2 7 6 a21 7 6 a22 7 6 a2n 7 6 7 6 7 = x 6 7+ x 6 7+ ··· + x 6 7 6 . .. 7 6 . 7 1 6 . 7 2 6 . 7 n 6 . 7 4 . 5 4 . 5 4 . 5 4 . 5 4 . 5 am1 am2 ::: amn xn am1 am2 amn = x1 a•1 + x2 a•2 + ··· + xn a•n A linear combination of the columns. Lecture 1: Linear Algebra Review Linear Programming 4 / 24 The Range of a Matrix Let A 2 Rm×n (an m × n matrix having real entries). Range of A m n Ran (A) = fy 2 R j 9 x 2 R such that y = Ax g Ran (A) = the linear span of the columns of A Lecture 1: Linear Algebra Review Linear Programming 5 / 24 Two Special Subspaces n Let v1;:::; vk 2 R . The linear span of v1;:::; vk : Span [v1;:::; vk ] = fy j y = ξ1v1 + ξ2v2 + ··· + ξk vk ; ξ1; : : : ; ξk 2 Rg The subspace orthogonal to v1;:::; vk : ? n fv1;:::; vk g = fz 2 R j z • vi = 0; i = 1;:::; k g ? ? Facts: fv1;:::; vk g = Span [v1;:::; vk ] ? h ?i Span [v1;:::; vk ] = Span [v1;:::; vk ] Lecture 1: Linear Algebra Review Linear Programming 6 / 24 Matrix Vector Multiplication A row space view of matrix vector multiplication. 2 3 2 3 2 3 2 Pn 3 a11 a12 ::: a1n x1 a1• • x i=1 a1i xi Pn 6 a21 a22 ::: a2n 7 6 x2 7 6 a2• • x 7 6 i=1 a2i xi 7 6 7 6 7 = 6 7 = 6 7 6 . .. 7 6 . 7 6 . 7 6 . 7 4 . 5 4 . 5 4 . 5 4 . 5 Pn am1 am2 ::: amn xn am• • x i=1 ami xi The dot product of x with the rows of A. Lecture 1: Linear Algebra Review Linear Programming 7 / 24 Fundamental Theorem of the Alternative: ? ? Nul (A) = Ran AT Ran (A) = Nul AT The Null Space of a Matrix Let A 2 Rm×n (an m × n matrix having real entries). Null Space of A n Nul (A) = fx 2 R j Ax = 0g Nul (A) = subspace orthogonal to the rows of A ? = Span [a1•; a2•;:::; am•] ? = Ran AT Lecture 1: Linear Algebra Review Linear Programming 8 / 24 The Null Space of a Matrix Let A 2 Rm×n (an m × n matrix having real entries). Null Space of A n Nul (A) = fx 2 R j Ax = 0g Nul (A) = subspace orthogonal to the rows of A ? = Span [a1•; a2•;:::; am•] ? = Ran AT Fundamental Theorem of the Alternative: ? ? Nul (A) = Ran AT Ran (A) = Nul AT Lecture 1: Linear Algebra Review Linear Programming 8 / 24 2 3 −4 1 1 0 0 3 6 2 2 0 0 1 0 7 6 7 B I3×3 =6 −1 0 0 0 0 1 7= 6 7 02×3 C 4 0 0 0 2 1 4 5 0 0 0 1 0 3 where 2 3 −4 1 3 2 1 4 B = 2 2 0 ; C = 4 5 1 0 3 −1 0 0 Block Structured Matrices 2 3 −4 1 1 0 0 3 6 2 2 0 0 1 0 7 6 7 A=6 −1 0 0 0 0 1 7 6 7 4 0 0 0 2 1 4 5 0 0 0 1 0 3 Lecture 1: Linear Algebra Review Linear Programming 9 / 24 B I = 3×3 02×3 C where 2 3 −4 1 3 2 1 4 B = 2 2 0 ; C = 4 5 1 0 3 −1 0 0 Block Structured Matrices 2 3 −4 1 1 0 0 3 2 3 −4 1 1 0 0 3 6 2 2 0 0 1 0 7 6 2 2 0 0 1 0 7 6 7 6 7 A=6 −1 0 0 0 0 1 7=6 −1 0 0 0 0 1 7 6 7 6 7 4 0 0 0 2 1 4 5 4 0 0 0 2 1 4 5 0 0 0 1 0 3 0 0 0 1 0 3 Lecture 1: Linear Algebra Review Linear Programming 9 / 24 Block Structured Matrices 2 3 −4 1 1 0 0 3 2 3 −4 1 1 0 0 3 6 2 2 0 0 1 0 7 6 2 2 0 0 1 0 7 6 7 6 7 B I3×3 A=6 −1 0 0 0 0 1 7=6 −1 0 0 0 0 1 7= 6 7 6 7 02×3 C 4 0 0 0 2 1 4 5 4 0 0 0 2 1 4 5 0 0 0 1 0 3 0 0 0 1 0 3 where 2 3 −4 1 3 2 1 4 B = 2 2 0 ; C = 4 5 1 0 3 −1 0 0 Lecture 1: Linear Algebra Review Linear Programming 9 / 24 Can we exploit the structure of A? Multiplication of Block Structured Matrices Consider the matrix product AM, where 2 1 2 3 2 3 −4 1 1 0 0 3 0 4 2 2 0 0 1 0 6 7 6 7 6 −1 −1 7 A = 6 −1 0 0 0 0 1 7 and M = 6 7 6 7 6 2 −1 7 6 0 0 0 2 1 4 7 6 7 4 5 6 4 3 7 0 0 0 1 0 3 4 5 −2 0 Lecture 1: Linear Algebra Review Linear Programming 10 / 24 Multiplication of Block Structured Matrices Consider the matrix product AM, where 2 1 2 3 2 3 −4 1 1 0 0 3 0 4 2 2 0 0 1 0 6 7 6 7 6 −1 −1 7 A = 6 −1 0 0 0 0 1 7 and M = 6 7 6 7 6 2 −1 7 6 0 0 0 2 1 4 7 6 7 4 5 6 4 3 7 0 0 0 1 0 3 4 5 −2 0 Can we exploit the structure of A? Lecture 1: Linear Algebra Review Linear Programming 10 / 24 BI X A = 3×3 so take M = ; 02×3 C Y 2 1 2 3 2 2 −1 3 where X = 4 0 4 5 ; and Y = 4 4 3 5 : −1 −1 −2 0 Multiplication of Block Structured Matrices Consider the matrix product AM, where 2 1 2 3 2 3 −4 1 1 0 0 3 0 4 2 2 0 0 1 0 6 7 6 7 6 −1 −1 7 A = 6 −1 0 0 0 0 1 7 and M = 6 7 6 7 6 2 −1 7 6 0 0 0 2 1 4 7 6 7 4 5 6 4 3 7 0 0 0 1 0 3 4 5 −2 0 Can we exploit the structure of A? Lecture 1: Linear Algebra Review Linear Programming 11 / 24 X so take M = ; Y 2 1 2 3 2 2 −1 3 where X = 4 0 4 5 ; and Y = 4 4 3 5 : −1 −1 −2 0 Multiplication of Block Structured Matrices Consider the matrix product AM, where 2 1 2 3 2 3 −4 1 1 0 0 3 0 4 2 2 0 0 1 0 6 7 6 7 6 −1 −1 7 A = 6 −1 0 0 0 0 1 7 and M = 6 7 6 7 6 2 −1 7 6 0 0 0 2 1 4 7 6 7 4 5 6 4 3 7 0 0 0 1 0 3 4 5 −2 0 Can we exploit the structure of A? BI A = 3×3 02×3 C Lecture 1: Linear Algebra Review Linear Programming 11 / 24 Multiplication of Block Structured Matrices Consider the matrix product AM, where 2 1 2 3 2 3 −4 1 1 0 0 3 0 4 2 2 0 0 1 0 6 7 6 7 6 −1 −1 7 A = 6 −1 0 0 0 0 1 7 and M = 6 7 6 7 6 2 −1 7 6 0 0 0 2 1 4 7 6 7 4 5 6 4 3 7 0 0 0 1 0 3 4 5 −2 0 Can we exploit the structure of A? BI X A = 3×3 so take M = ; 02×3 C Y 2 1 2 3 2 2 −1 3 where X = 4 0 4 5 ; and Y = 4 4 3 5 : −1 −1 −2 0 Lecture 1: Linear Algebra Review Linear Programming 11 / 24 2 2 2 −11 3 2 2 −1 3 3 6 4 2 12 5 + 4 4 3 5 7 6 7 6 −1 −2 −2 0 7 = 6 7 6 7 6 7 4 4 1 5 −4 −1 2 4 −12 3 6 6 15 7 6 7 = 6 1 −2 7 : 6 7 4 4 1 5 −4 −1 Multiplication of Block Structured Matrices BI X BX + Y AM = 3×3 = 02×3 C Y CY Lecture 1: Linear Algebra Review Linear Programming 12 / 24 Multiplication of Block Structured Matrices BI X BX + Y AM = 3×3 = 02×3 C Y CY 2 2 2 −11 3 2 2 −1 3 3 6 4 2 12 5 + 4 4 3 5 7 6 7 6 −1 −2 −2 0 7 = 6 7 6 7 6 7 4 4 1 5 −4 −1 2 4 −12 3 6 6 15 7 6 7 = 6 1 −2 7 : 6 7 4 4 1 5 −4 −1 Lecture 1: Linear Algebra Review Linear Programming 12 / 24 The solution set is either empty, a single point, or an infinite set n If a solution x0 2 R exists, then the set of solutions is given by x0 + Nul (A) : Solving Systems of Linear equations Let A 2 Rm×n and b 2 Rm.