Linear Programming =1Author: James Burke, University of Washington

Linear Programming 1

1 Author: James Burke, University of Washington Linear Algebra Review

Linear Algebra Review

Block Structured Matrices

Gaussian Elimination Matrices

Gauss-Jordan Elimination (Pivoting) columns rows   a1•  a2•  = a a ... a =   •1 •2 •n  .   .  am•

   T  a11 a21 ... am1 a•1 T  a12 a22 ... am2   a•2  AT =   =   = aT aT ... aT  . . .. .   .  1• 2• m•  . . . .   .  T a1n a2n ... amn a•n

Matrices in Rm×n

m×n A ∈ R

  a11 a12 ... a1n  a21 a22 ... a2n  A =    . . .. .   . . . .  am1 am2 ... amn rows   a1•  a2•  =    .   .  am•

Matrices in Rm×n

m×n A ∈ R

columns   a11 a12 ... a1n  a21 a22 ... a2n  A =   = a a ... a  . . .. .  •1 •2 •n  . . . .  am1 am2 ... amn    T  a11 a21 ... am1 a•1 T  a12 a22 ... am2   a•2  AT =   =   = aT aT ... aT  . . .. .   .  1• 2• m•  . . . .   .  T a1n a2n ... amn a•n

Matrices in Rm×n

m×n A ∈ R

columns rows     a11 a12 ... a1n a1•  a21 a22 ... a2n   a2•  A =   = a a ... a =    . . .. .  •1 •2 •n  .   . . . .   .  am1 am2 ... amn am•  T  a•1 T  a•2  =   = aT aT ... aT  .  1• 2• m•  .  T a•n

Matrices in Rm×n

m×n A ∈ R

columns rows     a11 a12 ... a1n a1•  a21 a22 ... a2n   a2•  A =   = a a ... a =    . . .. .  •1 •2 •n  .   . . . .   .  am1 am2 ... amn am•

  a11 a21 ... am1  a12 a22 ... am2  AT =    . . .. .   . . . .  a1n a2n ... amn T T T = a1• a2• ... am•

Matrices in Rm×n

m×n A ∈ R

   T  a11 a21 ... am1 a•1 T  a12 a22 ... am2   a•2  AT =   =    . . .. .   .   . . . .   .  T a1n a2n ... amn a•n Matrices in Rm×n

m×n A ∈ R

   T  a11 a21 ... am1 a•1 T  a12 a22 ... am2   a•2  AT =   =   = aT aT ... aT  . . .. .   .  1• 2• m•  . . . .   .  T a1n a2n ... amn a•n       a11 a12 a1n  a21   a22   a2n  = x  + x  + ··· + x   1  .  2  .  n  .   .   .   .  am1 am2 amn

= x1 a•1 + x2 a•2 + ··· + xn a•n

A linear combination of the columns.

Matrix Vector Multiplication

A column space view of matrix vector multiplication.

    a11 a12 ... a1n x1  a21 a22 ... a2n   x2       . . .. .   .   . . . .   .  am1 am2 ... amn xn     a12 a1n  a22   a2n  + x  + ··· + x   2  .  n  .   .   .  am2 amn

= x1 a•1 + x2 a•2 + ··· + xn a•n

A linear combination of the columns.

Matrix Vector Multiplication

A column space view of matrix vector multiplication.

      a11 a12 ... a1n x1 a11  a21 a22 ... a2n   x2   a21      = x    . . .. .   .  1  .   . . . .   .   .  am1 am2 ... amn xn am1   a1n  a2n  + ··· + x   n  .   .  amn

= x1 a•1 + x2 a•2 + ··· + xn a•n

A linear combination of the columns.

Matrix Vector Multiplication

A column space view of matrix vector multiplication.

        a11 a12 ... a1n x1 a11 a12  a21 a22 ... a2n   x2   a21   a22      = x  + x    . . .. .   .  1  .  2  .   . . . .   .   .   .  am1 am2 ... amn xn am1 am2 = x1 a•1 + x2 a•2 + ··· + xn a•n

A linear combination of the columns.

Matrix Vector Multiplication

A column space view of matrix vector multiplication.

          a11 a12 ... a1n x1 a11 a12 a1n  a21 a22 ... a2n   x2   a21   a22   a2n      = x  + x  + ··· + x    . . .. .   .  1  .  2  .  n  .   . . . .   .   .   .   .  am1 am2 ... amn xn am1 am2 amn A linear combination of the columns.

Matrix Vector Multiplication

A column space view of matrix vector multiplication.

          a11 a12 ... a1n x1 a11 a12 a1n  a21 a22 ... a2n   x2   a21   a22   a2n      = x  + x  + ··· + x    . . .. .   .  1  .  2  .  n  .   . . . .   .   .   .   .  am1 am2 ... amn xn am1 am2 amn

= x1 a•1 + x2 a•2 + ··· + xn a•n Matrix Vector Multiplication

A column space view of matrix vector multiplication.

= x1 a•1 + x2 a•2 + ··· + xn a•n

A linear combination of the columns. Range of A

m n Ran (A) = {y ∈ R | ∃ x ∈ R such that y = Ax }

Ran (A) = the linear span of the columns of A

The Range of a Matrix

m×n Let A ∈ R (an m × n matrix having real entries). Ran (A) = the linear span of the columns of A

The Range of a Matrix

m×n Let A ∈ R (an m × n matrix having real entries).

Range of A

m n Ran (A) = {y ∈ R | ∃ x ∈ R such that y = Ax } The Range of a Matrix

m×n Let A ∈ R (an m × n matrix having real entries).

Range of A

m n Ran (A) = {y ∈ R | ∃ x ∈ R such that y = Ax }

Ran (A) = the linear span of the columns of A I The linear span of v1,..., vk :

Span [v1,..., vk ] = {y | y = ξ1v1 + ξ2v2 + ··· + ξk vk , ξ1, . . . , ξk ∈ R}

I The subspace orthogonal to v1,..., vk :

⊥ n {v1,..., vk } = {z ∈ R | z • vi = 0, i = 1,..., k }

⊥ ⊥ Facts: {v1,..., vk } = Span [v1,..., vk ] ⊥ h ⊥i Span [v1,..., vk ] = Span [v1,..., vk ]

Two Special Subspaces

n Let v1,..., vk ∈ R . I The subspace orthogonal to v1,..., vk :

⊥ n {v1,..., vk } = {z ∈ R | z • vi = 0, i = 1,..., k }

⊥ ⊥ Facts: {v1,..., vk } = Span [v1,..., vk ] ⊥ h ⊥i Span [v1,..., vk ] = Span [v1,..., vk ]

Two Special Subspaces

n Let v1,..., vk ∈ R .

I The linear span of v1,..., vk :

Span [v1,..., vk ] = {y | y = ξ1v1 + ξ2v2 + ··· + ξk vk , ξ1, . . . , ξk ∈ R} ⊥ ⊥ Facts: {v1,..., vk } = Span [v1,..., vk ] ⊥ h ⊥i Span [v1,..., vk ] = Span [v1,..., vk ]

Two Special Subspaces

n Let v1,..., vk ∈ R .

I The linear span of v1,..., vk :

Span [v1,..., vk ] = {y | y = ξ1v1 + ξ2v2 + ··· + ξk vk , ξ1, . . . , ξk ∈ R}

I The subspace orthogonal to v1,..., vk :

⊥ n {v1,..., vk } = {z ∈ R | z • vi = 0, i = 1,..., k } ⊥ ⊥ {v1,..., vk } = Span [v1,..., vk ] ⊥ h ⊥i Span [v1,..., vk ] = Span [v1,..., vk ]

Two Special Subspaces

n Let v1,..., vk ∈ R .

I The linear span of v1,..., vk :

Span [v1,..., vk ] = {y | y = ξ1v1 + ξ2v2 + ··· + ξk vk , ξ1, . . . , ξk ∈ R}

I The subspace orthogonal to v1,..., vk :

⊥ n {v1,..., vk } = {z ∈ R | z • vi = 0, i = 1,..., k }

Facts: ⊥ h ⊥i Span [v1,..., vk ] = Span [v1,..., vk ]

Two Special Subspaces

n Let v1,..., vk ∈ R .

I The linear span of v1,..., vk :

Span [v1,..., vk ] = {y | y = ξ1v1 + ξ2v2 + ··· + ξk vk , ξ1, . . . , ξk ∈ R}

I The subspace orthogonal to v1,..., vk :

⊥ n {v1,..., vk } = {z ∈ R | z • vi = 0, i = 1,..., k }

⊥ ⊥ Facts: {v1,..., vk } = Span [v1,..., vk ] Two Special Subspaces

n Let v1,..., vk ∈ R .

I The linear span of v1,..., vk :

Span [v1,..., vk ] = {y | y = ξ1v1 + ξ2v2 + ··· + ξk vk , ξ1, . . . , ξk ∈ R}

I The subspace orthogonal to v1,..., vk :

⊥ n {v1,..., vk } = {z ∈ R | z • vi = 0, i = 1,..., k }

⊥ ⊥ Facts: {v1,..., vk } = Span [v1,..., vk ] ⊥ h ⊥i Span [v1,..., vk ] = Span [v1,..., vk ] Matrix Vector Multiplication

A row space view of matrix vector multiplication.

       Pn  a11 a12 ... a1n x1 a1• • x i=1 a1i xi Pn  a21 a22 ... a2n   x2   a2• • x   i=1 a2i xi      =   =    . . .. .   .   .   .   . . . .   .   .   .  Pn am1 am2 ... amn xn am• • x i=1 ami xi

The dot product of x with the rows of A. Null Space of A

n Nul (A) = {x ∈ R | Ax = 0}

Nul (A) = subspace orthogonal to the rows of A ⊥ = Span [a1•, a2•,..., am•] ⊥ = Ran AT

Fundamental Theorem of the Alternative: ⊥ ⊥ Nul (A) = Ran AT Ran (A) = Nul AT

The Null Space of a Matrix

m×n Let A ∈ R (an m × n matrix having real entries). Nul (A) = subspace orthogonal to the rows of A ⊥ = Span [a1•, a2•,..., am•] ⊥ = Ran AT

Fundamental Theorem of the Alternative: ⊥ ⊥ Nul (A) = Ran AT Ran (A) = Nul AT

The Null Space of a Matrix

m×n Let A ∈ R (an m × n matrix having real entries). Null Space of A

n Nul (A) = {x ∈ R | Ax = 0} ⊥ = Span [a1•, a2•,..., am•] ⊥ = Ran AT

Fundamental Theorem of the Alternative: ⊥ ⊥ Nul (A) = Ran AT Ran (A) = Nul AT

The Null Space of a Matrix

m×n Let A ∈ R (an m × n matrix having real entries). Null Space of A

n Nul (A) = {x ∈ R | Ax = 0}

Nul (A) = subspace orthogonal to the rows of A ⊥ = Ran AT

Fundamental Theorem of the Alternative: ⊥ ⊥ Nul (A) = Ran AT Ran (A) = Nul AT

The Null Space of a Matrix

m×n Let A ∈ R (an m × n matrix having real entries). Null Space of A

n Nul (A) = {x ∈ R | Ax = 0}

Nul (A) = subspace orthogonal to the rows of A ⊥ = Span [a1•, a2•,..., am•] Fundamental Theorem of the Alternative: ⊥ ⊥ Nul (A) = Ran AT Ran (A) = Nul AT

The Null Space of a Matrix

m×n Let A ∈ R (an m × n matrix having real entries). Null Space of A

n Nul (A) = {x ∈ R | Ax = 0}

Nul (A) = subspace orthogonal to the rows of A ⊥ = Span [a1•, a2•,..., am•] ⊥ = Ran AT The Null Space of a Matrix

m×n Let A ∈ R (an m × n matrix having real entries). Null Space of A

n Nul (A) = {x ∈ R | Ax = 0}

Nul (A) = subspace orthogonal to the rows of A ⊥ = Span [a1•, a2•,..., am•] ⊥ = Ran AT

Fundamental Theorem of the Alternative: ⊥ ⊥ Nul (A) = Ran AT Ran (A) = Nul AT  3 −4 1 1 0 0   2 2 0 0 1 0    B I3×3 = −1 0 0 0 0 1 =   02×3 C  0 0 0 2 1 4  0 0 0 1 0 3

where  3 −4 1  2 1 4 B = 2 2 0 , C =   1 0 3 −1 0 0

Block Structured Matrices

 3 −4 1 1 0 0   2 2 0 0 1 0    A= −1 0 0 0 0 1     0 0 0 2 1 4  0 0 0 1 0 3 B I = 3×3 02×3 C

where  3 −4 1  2 1 4 B = 2 2 0 , C =   1 0 3 −1 0 0

Block Structured Matrices

 3 −4 1 1 0 0   3 −4 1 1 0 0   2 2 0 0 1 0   2 2 0 0 1 0      A= −1 0 0 0 0 1 = −1 0 0 0 0 1       0 0 0 2 1 4   0 0 0 2 1 4  0 0 0 1 0 3 0 0 0 1 0 3 where  3 −4 1  2 1 4 B = 2 2 0 , C =   1 0 3 −1 0 0

Block Structured Matrices

 3 −4 1 1 0 0   3 −4 1 1 0 0   2 2 0 0 1 0   2 2 0 0 1 0      B I3×3 A= −1 0 0 0 0 1 = −1 0 0 0 0 1 =     02×3 C  0 0 0 2 1 4   0 0 0 2 1 4  0 0 0 1 0 3 0 0 0 1 0 3 Block Structured Matrices

 3 −4 1 1 0 0   3 −4 1 1 0 0   2 2 0 0 1 0   2 2 0 0 1 0      B I3×3 A= −1 0 0 0 0 1 = −1 0 0 0 0 1 =     02×3 C  0 0 0 2 1 4   0 0 0 2 1 4  0 0 0 1 0 3 0 0 0 1 0 3

where  3 −4 1  2 1 4 B = 2 2 0 , C =   1 0 3 −1 0 0 Can we exploit the structure of A?

Multiplication of Block Structured Matrices

Consider the matrix product AM, where

 1 2   3 −4 1 1 0 0  0 4 2 2 0 0 1 0      −1 −1  A =  −1 0 0 0 0 1  and M =      2 −1   0 0 0 2 1 4       4 3  0 0 0 1 0 3   −2 0 Multiplication of Block Structured Matrices

Consider the matrix product AM, where

 1 2   3 −4 1 1 0 0  0 4 2 2 0 0 1 0      −1 −1  A =  −1 0 0 0 0 1  and M =      2 −1   0 0 0 2 1 4       4 3  0 0 0 1 0 3   −2 0

Can we exploit the structure of A? BI X A = 3×3 so take M = , 02×3 C Y

 1 2   2 −1  where X =  0 4  , and Y =  4 3  . −1 −1 −2 0

Multiplication of Block Structured Matrices

Consider the matrix product AM, where

Can we exploit the structure of A? X so take M = , Y

 1 2   2 −1  where X =  0 4  , and Y =  4 3  . −1 −1 −2 0

Multiplication of Block Structured Matrices

Consider the matrix product AM, where

Can we exploit the structure of A?

BI A = 3×3 02×3 C  1 2   2 −1  where X =  0 4  , and Y =  4 3  . −1 −1 −2 0

Multiplication of Block Structured Matrices

Consider the matrix product AM, where

Can we exploit the structure of A?

BI X A = 3×3 so take M = , 02×3 C Y Multiplication of Block Structured Matrices

Consider the matrix product AM, where

Can we exploit the structure of A?

BI X A = 3×3 so take M = , 02×3 C Y

 1 2   2 −1  where X =  0 4  , and Y =  4 3  . −1 −1 −2 0 BX + Y = CY

  2 −11   2 −1     2 12  +  4 3      −1 −2 −2 0  =        4 1  −4 −1  4 −12   6 15    =  1 −2  .    4 1  −4 −1

Multiplication of Block Structured Matrices

BI X AM = 3×3 02×3 C Y   2 −11   2 −1     2 12  +  4 3      −1 −2 −2 0  =        4 1  −4 −1  4 −12   6 15    =  1 −2  .    4 1  −4 −1

Multiplication of Block Structured Matrices

BI X BX + Y AM = 3×3 = 02×3 C Y CY  4 −12   6 15    =  1 −2  .    4 1  −4 −1

Multiplication of Block Structured Matrices

BI X BX + Y AM = 3×3 = 02×3 C Y CY

  2 −11   2 −1     2 12  +  4 3      −1 −2 −2 0  =        4 1  −4 −1 Multiplication of Block Structured Matrices

BI X BX + Y AM = 3×3 = 02×3 C Y CY

  2 −11   2 −1     2 12  +  4 3      −1 −2 −2 0  =        4 1  −4 −1  4 −12   6 15    =  1 −2  .    4 1  −4 −1 The solution set is either empty, a single point, or an inﬁnite set

n If a solution x0 ∈ R exists, then the set of solutions is given by

x0 + Nul (A) .

Solving Systems of Linear equations

m×n m Let A ∈ R and b ∈ R . n Find all solutions x ∈ R to the system Ax = b.

. empty, a single point, or an inﬁnite set

n If a solution x0 ∈ R exists, then the set of solutions is given by

x0 + Nul (A) .

Solving Systems of Linear equations

m×n m Let A ∈ R and b ∈ R . n Find all solutions x ∈ R to the system Ax = b.

The solution set is either . a single point, or an inﬁnite set

n If a solution x0 ∈ R exists, then the set of solutions is given by

x0 + Nul (A) .

Solving Systems of Linear equations

m×n m Let A ∈ R and b ∈ R . n Find all solutions x ∈ R to the system Ax = b.

The solution set is either empty, . or an inﬁnite set

n If a solution x0 ∈ R exists, then the set of solutions is given by

x0 + Nul (A) .

Solving Systems of Linear equations

m×n m Let A ∈ R and b ∈ R . n Find all solutions x ∈ R to the system Ax = b.

The solution set is either empty, a single point, . n If a solution x0 ∈ R exists, then the set of solutions is given by

x0 + Nul (A) .

Solving Systems of Linear equations

m×n m Let A ∈ R and b ∈ R . n Find all solutions x ∈ R to the system Ax = b.

The solution set is either empty, a single point, or an inﬁnite set. Solving Systems of Linear equations

m×n m Let A ∈ R and b ∈ R . n Find all solutions x ∈ R to the system Ax = b.

The solution set is either empty, a single point, or an inﬁnite set.

n If a solution x0 ∈ R exists, then the set of solutions is given by

x0 + Nul (A) . This process is called Gaussian elimination.

The three elementary row operations. 1. Interchange any two rows. 2. Multiply any row by a non-zero constant. 3. Replace any row by itself plus a multiple of any other row.

These elementary row operations can be interpreted as multiplying the augmented matrix on the left by a special matrix.

Gaussian Elimination and the 3 Elementary Row Operations

We solve the system Ax = b by transforming the augmented matrix

[ A | b]

into upper echelon form using the three elementary row operations. The three elementary row operations. 1. Interchange any two rows. 2. Multiply any row by a non-zero constant. 3. Replace any row by itself plus a multiple of any other row.

These elementary row operations can be interpreted as multiplying the augmented matrix on the left by a special matrix.

Gaussian Elimination and the 3 Elementary Row Operations

We solve the system Ax = b by transforming the augmented matrix

[ A | b]

into upper echelon form using the three elementary row operations.

This process is called Gaussian elimination. 1. Interchange any two rows. 2. Multiply any row by a non-zero constant. 3. Replace any row by itself plus a multiple of any other row.

These elementary row operations can be interpreted as multiplying the augmented matrix on the left by a special matrix.

Gaussian Elimination and the 3 Elementary Row Operations

We solve the system Ax = b by transforming the augmented matrix

[ A | b]

into upper echelon form using the three elementary row operations.

This process is called Gaussian elimination.

The three elementary row operations. 2. Multiply any row by a non-zero constant. 3. Replace any row by itself plus a multiple of any other row.

These elementary row operations can be interpreted as multiplying the augmented matrix on the left by a special matrix.

Gaussian Elimination and the 3 Elementary Row Operations

We solve the system Ax = b by transforming the augmented matrix

[ A | b]

into upper echelon form using the three elementary row operations.

This process is called Gaussian elimination.

The three elementary row operations. 1. Interchange any two rows. 3. Replace any row by itself plus a multiple of any other row.

These elementary row operations can be interpreted as multiplying the augmented matrix on the left by a special matrix.

Gaussian Elimination and the 3 Elementary Row Operations

We solve the system Ax = b by transforming the augmented matrix

[ A | b]

into upper echelon form using the three elementary row operations.

This process is called Gaussian elimination.

The three elementary row operations. 1. Interchange any two rows. 2. Multiply any row by a non-zero constant. These elementary row operations can be interpreted as multiplying the augmented matrix on the left by a special matrix.

Gaussian Elimination and the 3 Elementary Row Operations

We solve the system Ax = b by transforming the augmented matrix

[ A | b]

into upper echelon form using the three elementary row operations.

This process is called Gaussian elimination.

The three elementary row operations. 1. Interchange any two rows. 2. Multiply any row by a non-zero constant. 3. Replace any row by itself plus a multiple of any other row. Gaussian Elimination and the 3 Elementary Row Operations

We solve the system Ax = b by transforming the augmented matrix

[ A | b]

into upper echelon form using the three elementary row operations.

This process is called Gaussian elimination.

The three elementary row operations. 1. Interchange any two rows. 2. Multiply any row by a non-zero constant. 3. Replace any row by itself plus a multiple of any other row.

These elementary row operations can be interpreted as multiplying the augmented matrix on the left by a special matrix. Multiplying any 4 × n matrix on the left by the exchange matrix

 1 0 0 0   0 0 0 1     0 0 1 0  0 1 0 0

will exchange the second and fourth rows of the matrix.

(multiplication of a m × 4 matrix on the right by this exchanges the second and fourth columns.) A permutation matrix is obtained by permuting the columns of the identity matrix.

Exchange and Permutation Matrices

An exchange matrix is given by permuting any two columns of the identity. (multiplication of a m × 4 matrix on the right by this exchanges the second and fourth columns.) A permutation matrix is obtained by permuting the columns of the identity matrix.

Exchange and Permutation Matrices

An exchange matrix is given by permuting any two columns of the identity.

Multiplying any 4 × n matrix on the left by the exchange matrix

 1 0 0 0   0 0 0 1     0 0 1 0  0 1 0 0

will exchange the second and fourth rows of the matrix. A permutation matrix is obtained by permuting the columns of the identity matrix.

Exchange and Permutation Matrices

An exchange matrix is given by permuting any two columns of the identity.

Multiplying any 4 × n matrix on the left by the exchange matrix

 1 0 0 0   0 0 0 1     0 0 1 0  0 1 0 0

will exchange the second and fourth rows of the matrix.

(multiplication of a m × 4 matrix on the right by this exchanges the second and fourth columns.) Exchange and Permutation Matrices

An exchange matrix is given by permuting any two columns of the identity.

Multiplying any 4 × n matrix on the left by the exchange matrix

 1 0 0 0   0 0 0 1     0 0 1 0  0 1 0 0

will exchange the second and fourth rows of the matrix.

(multiplication of a m × 4 matrix on the right by this exchanges the second and fourth columns.) A permutation matrix is obtained by permuting the columns of the identity matrix. Left Multiplication of A: When multiplying A on the left by an m × m matrix M, it is often useful to think of this as an action on the rows of A.

For example, left multiplication by a permutation matrix permutes the rows of the matrix.

However, mechanically, left multiplication corresponds to matrix vector multiplication on the columns.

MA = [Ma•1 Ma•2 ··· Ma•n]

Notes on Matrix Multiplication

m×n Let A = [aij ]m×n ∈ R . When multiplying A on the left by an m × m matrix M, it is often useful to think of this as an action on the rows of A.

For example, left multiplication by a permutation matrix permutes the rows of the matrix.

However, mechanically, left multiplication corresponds to matrix vector multiplication on the columns.

MA = [Ma•1 Ma•2 ··· Ma•n]

Notes on Matrix Multiplication

m×n Let A = [aij ]m×n ∈ R .

Left Multiplication of A: For example, left multiplication by a permutation matrix permutes the rows of the matrix.

However, mechanically, left multiplication corresponds to matrix vector multiplication on the columns.

MA = [Ma•1 Ma•2 ··· Ma•n]

Notes on Matrix Multiplication

m×n Let A = [aij ]m×n ∈ R .

Left Multiplication of A: When multiplying A on the left by an m × m matrix M, it is often useful to think of this as an action on the rows of A. However, mechanically, left multiplication corresponds to matrix vector multiplication on the columns.

MA = [Ma•1 Ma•2 ··· Ma•n]

Notes on Matrix Multiplication

m×n Let A = [aij ]m×n ∈ R .

Left Multiplication of A: When multiplying A on the left by an m × m matrix M, it is often useful to think of this as an action on the rows of A.

For example, left multiplication by a permutation matrix permutes the rows of the matrix. Notes on Matrix Multiplication

m×n Let A = [aij ]m×n ∈ R .

Left Multiplication of A: When multiplying A on the left by an m × m matrix M, it is often useful to think of this as an action on the rows of A.

For example, left multiplication by a permutation matrix permutes the rows of the matrix.

However, mechanically, left multiplication corresponds to matrix vector multiplication on the columns.

MA = [Ma•1 Ma•2 ··· Ma•n] Right Multiplication of A: When multiplying A on the right by an n × n matrix N, it is often useful to think of this as an action on the columns of A.

For example, right multiplication by a permutation matrix permutes the columns of the matrix.

 a N  However, mechanically, right 1•  a2•N  multiplication corresponds to AN =    .  left matrix vector multiplication  .  on the rows. am•N

Notes on Matrix Multiplication

m×n Let A = [aij ]m×n ∈ R . When multiplying A on the right by an n × n matrix N, it is often useful to think of this as an action on the columns of A.

For example, right multiplication by a permutation matrix permutes the columns of the matrix.

 a N  However, mechanically, right 1•  a2•N  multiplication corresponds to AN =    .  left matrix vector multiplication  .  on the rows. am•N

Notes on Matrix Multiplication

m×n Let A = [aij ]m×n ∈ R .

Right Multiplication of A: For example, right multiplication by a permutation matrix permutes the columns of the matrix.

 a N  However, mechanically, right 1•  a2•N  multiplication corresponds to AN =    .  left matrix vector multiplication  .  on the rows. am•N

Notes on Matrix Multiplication

m×n Let A = [aij ]m×n ∈ R .

Right Multiplication of A: When multiplying A on the right by an n × n matrix N, it is often useful to think of this as an action on the columns of A.  a N  However, mechanically, right 1•  a2•N  multiplication corresponds to AN =    .  left matrix vector multiplication  .  on the rows. am•N

Notes on Matrix Multiplication

m×n Let A = [aij ]m×n ∈ R .

Right Multiplication of A: When multiplying A on the right by an n × n matrix N, it is often useful to think of this as an action on the columns of A.

For example, right multiplication by a permutation matrix permutes the columns of the matrix.   a1•N  a2•N  AN =    .   .  am•N

Notes on Matrix Multiplication

m×n Let A = [aij ]m×n ∈ R .

Right Multiplication of A: When multiplying A on the right by an n × n matrix N, it is often useful to think of this as an action on the columns of A.

For example, right multiplication by a permutation matrix permutes the columns of the matrix.

However, mechanically, right multiplication corresponds to left matrix vector multiplication on the rows. Notes on Matrix Multiplication

m×n Let A = [aij ]m×n ∈ R .

Right Multiplication of A: When multiplying A on the right by an n × n matrix N, it is often useful to think of this as an action on the columns of A.

For example, right multiplication by a permutation matrix permutes the columns of the matrix.

 a N  However, mechanically, right 1•  a2•N  multiplication corresponds to AN =    .  left matrix vector multiplication  .  on the rows. am•N This can be accomplished by left matrix multiplication as follows.

Gaussian Elimination Matrices

The key step in Gaussian elimination is to transform a vector of the form  a   α  , b

k n−k−1 where a ∈ R , 0 6= α ∈ R, and b ∈ R , into one of the form  a   α  . 0 Gaussian Elimination Matrices

The key step in Gaussian elimination is to transform a vector of the form  a   α  , b

k n−k−1 where a ∈ R , 0 6= α ∈ R, and b ∈ R , into one of the form  a   α  . 0

This can be accomplished by left matrix multiplication as follows. a α 0

  =   .

Gaussian Elimination Matrices

k n−k−1 a ∈ R , 0 6= α ∈ R, and b ∈ R

    Ik×k 0 0 a  0 1 0   α  −1 0 −α b I(n−k−1)×(n−k−1) b a α 0

Gaussian Elimination Matrices

k n−k−1 a ∈ R , 0 6= α ∈ R, and b ∈ R

      Ik×k 0 0 a  0 1 0   α  =   . −1 0 −α b I(n−k−1)×(n−k−1) b α 0

Gaussian Elimination Matrices

k n−k−1 a ∈ R , 0 6= α ∈ R, and b ∈ R

      Ik×k 0 0 a a  0 1 0   α  =   . −1 0 −α b I(n−k−1)×(n−k−1) b 0

Gaussian Elimination Matrices

k n−k−1 a ∈ R , 0 6= α ∈ R, and b ∈ R

      Ik×k 0 0 a a  0 1 0   α  =  α  . −1 0 −α b I(n−k−1)×(n−k−1) b Gaussian Elimination Matrices

k n−k−1 a ∈ R , 0 6= α ∈ R, and b ∈ R

      Ik×k 0 0 a a  0 1 0   α  =  α  . −1 0 −α b I(n−k−1)×(n−k−1) b 0 This matrix is invertible with inverse   Ik×k 0 0  0 1 0  . −1 0 α b I(n−k−1)×(n−k−1)

Note that a Gaussian elimination matrix and its inverse are both lower triangular matrices.

Gaussian Elimination Matrices

The matrix   Ik×k 0 0  0 1 0  −1 0 −α b I(n−k−1)×(n−k−1)

is called a Gaussian elimination matrix. Note that a Gaussian elimination matrix and its inverse are both lower triangular matrices.

Gaussian Elimination Matrices

The matrix   Ik×k 0 0  0 1 0  −1 0 −α b I(n−k−1)×(n−k−1)

is called a Gaussian elimination matrix.

This matrix is invertible with inverse   Ik×k 0 0  0 1 0  . −1 0 α b I(n−k−1)×(n−k−1) Gaussian Elimination Matrices

The matrix   Ik×k 0 0  0 1 0  −1 0 −α b I(n−k−1)×(n−k−1)

is called a Gaussian elimination matrix.

This matrix is invertible with inverse   Ik×k 0 0  0 1 0  . −1 0 α b I(n−k−1)×(n−k−1)

Note that a Gaussian elimination matrix and its inverse are both lower triangular matrices. n×n n×n A subset S of R is said to be a sub-algebra of R if n×n I S is a subspace of R , I S is closed wrt matrix multiplication, and −1 I if M ∈ S is invertible, then M ∈ S.

Matrix Sub-Algebras

n×n Lower (upper) triangular matrices in R are said to form a n×n sub-algebra of R . n×n I S is a subspace of R , I S is closed wrt matrix multiplication, and −1 I if M ∈ S is invertible, then M ∈ S.

Matrix Sub-Algebras

n×n Lower (upper) triangular matrices in R are said to form a n×n sub-algebra of R .

n×n n×n A subset S of R is said to be a sub-algebra of R if I S is closed wrt matrix multiplication, and −1 I if M ∈ S is invertible, then M ∈ S.

Matrix Sub-Algebras

n×n Lower (upper) triangular matrices in R are said to form a n×n sub-algebra of R .

n×n n×n A subset S of R is said to be a sub-algebra of R if n×n I S is a subspace of R , −1 I if M ∈ S is invertible, then M ∈ S.

Matrix Sub-Algebras

n×n Lower (upper) triangular matrices in R are said to form a n×n sub-algebra of R .

n×n n×n A subset S of R is said to be a sub-algebra of R if n×n I S is a subspace of R , I S is closed wrt matrix multiplication, and Matrix Sub-Algebras

n×n Lower (upper) triangular matrices in R are said to form a n×n sub-algebra of R .

n×n n×n A subset S of R is said to be a sub-algebra of R if n×n I S is a subspace of R , I S is closed wrt matrix multiplication, and −1 I if M ∈ S is invertible, then M ∈ S. 1 1 2 0 2 − 2 0 2 5

 1 1 2  A =  2 4 2  . −1 1 3

 1 0 0  Eliminate the ﬁrst column with G1 =  −2 1 0  a Gaussian elimination matrix. 1 0 1

 1 0 0   1 1 2    G1A =  −2 1 0   2 4 2  =   1 0 1 −1 1 3

Gaussian Elimination in Practice

Transformation to echelon (upper triangular) form. 1 1 2 0 2 − 2 0 2 5

 1 0 0  Eliminate the ﬁrst column with G1 =  −2 1 0  a Gaussian elimination matrix. 1 0 1

 1 0 0   1 1 2    G1A =  −2 1 0   2 4 2  =   1 0 1 −1 1 3

Gaussian Elimination in Practice

 1 1 2  Transformation to echelon A =  2 4 2  . (upper triangular) form. −1 1 3 1 1 2 0 2 − 2 0 2 5

 1 0 0  G1 =  −2 1 0  1 0 1

 1 0 0   1 1 2    G1A =  −2 1 0   2 4 2  =   1 0 1 −1 1 3

Gaussian Elimination in Practice

 1 1 2  Transformation to echelon A =  2 4 2  . (upper triangular) form. −1 1 3

Eliminate the ﬁrst column with a Gaussian elimination matrix. 1 1 2 0 2 − 2 0 2 5

 1 0 0   1 1 2    G1A =  −2 1 0   2 4 2  =   1 0 1 −1 1 3

Gaussian Elimination in Practice

 1 1 2  Transformation to echelon A =  2 4 2  . (upper triangular) form. −1 1 3

 1 0 0  Eliminate the ﬁrst column with G1 =  −2 1 0  a Gaussian elimination matrix. 1 0 1 1 1 2 0 2 − 2 0 2 5

  =  

Gaussian Elimination in Practice

 1 1 2  Transformation to echelon A =  2 4 2  . (upper triangular) form. −1 1 3

 1 0 0  Eliminate the ﬁrst column with G1 =  −2 1 0  a Gaussian elimination matrix. 1 0 1

 1 0 0   1 1 2  G1A =  −2 1 0   2 4 2  1 0 1 −1 1 3 1 1 2 0 2 − 2 0 2 5

Gaussian Elimination in Practice

 1 1 2  Transformation to echelon A =  2 4 2  . (upper triangular) form. −1 1 3

 1 0 0  Eliminate the ﬁrst column with G1 =  −2 1 0  a Gaussian elimination matrix. 1 0 1

 1 0 0   1 1 2    G1A =  −2 1 0   2 4 2  =   1 0 1 −1 1 3 1 2 0 2 − 2 0 2 5

Gaussian Elimination in Practice

 1 1 2  Transformation to echelon A =  2 4 2  . (upper triangular) form. −1 1 3

 1 0 0  Eliminate the ﬁrst column with G1 =  −2 1 0  a Gaussian elimination matrix. 1 0 1

 1 0 0   1 1 2   1  G1A =  −2 1 0   2 4 2  =   1 0 1 −1 1 3 2 0 2 − 2 0 2 5

Gaussian Elimination in Practice

 1 1 2  Transformation to echelon A =  2 4 2  . (upper triangular) form. −1 1 3

 1 0 0  Eliminate the ﬁrst column with G1 =  −2 1 0  a Gaussian elimination matrix. 1 0 1

 1 0 0   1 1 2   1 1  G1A =  −2 1 0   2 4 2  =   1 0 1 −1 1 3 0 2 − 2 0 2 5

Gaussian Elimination in Practice

 1 1 2  Transformation to echelon A =  2 4 2  . (upper triangular) form. −1 1 3

 1 0 0  Eliminate the ﬁrst column with G1 =  −2 1 0  a Gaussian elimination matrix. 1 0 1

 1 0 0   1 1 2   1 1 2  G1A =  −2 1 0   2 4 2  =   1 0 1 −1 1 3 2 − 2 0 2 5

Gaussian Elimination in Practice

 1 1 2  Transformation to echelon A =  2 4 2  . (upper triangular) form. −1 1 3

 1 0 0  Eliminate the ﬁrst column with G1 =  −2 1 0  a Gaussian elimination matrix. 1 0 1

 1 0 0   1 1 2   1 1 2  G1A =  −2 1 0   2 4 2  =  0  1 0 1 −1 1 3 − 2 0 2 5

Gaussian Elimination in Practice

 1 1 2  Transformation to echelon A =  2 4 2  . (upper triangular) form. −1 1 3

 1 0 0  Eliminate the ﬁrst column with G1 =  −2 1 0  a Gaussian elimination matrix. 1 0 1

 1 0 0   1 1 2   1 1 2  G1A =  −2 1 0   2 4 2  =  0 2  1 0 1 −1 1 3 0 2 5

Gaussian Elimination in Practice

 1 1 2  Transformation to echelon A =  2 4 2  . (upper triangular) form. −1 1 3

 1 0 0  Eliminate the ﬁrst column with G1 =  −2 1 0  a Gaussian elimination matrix. 1 0 1

 1 0 0   1 1 2   1 1 2  G1A =  −2 1 0   2 4 2  =  0 2 − 2  1 0 1 −1 1 3 Gaussian Elimination in Practice

 1 1 2  Transformation to echelon A =  2 4 2  . (upper triangular) form. −1 1 3

 1 0 0  Eliminate the ﬁrst column with G1 =  −2 1 0  a Gaussian elimination matrix. 1 0 1

 1 0 0   1 1 2   1 1 2  G1A =  −2 1 0   2 4 2  =  0 2 − 2  1 0 1 −1 1 3 0 2 5 1 1 2 0 2 − 2 0 0 7

 1 0 0  G2 =  0 1 0  0 −1 1

 1 0 0   1 1 2     0 1 0   0 2 −2  =   0 −1 1 0 2 5

Gaussian Elimination in Practice

Now do Gaussian eliminiation on the second column.  1 1 2   0 2 −2  0 2 5 1 1 2 0 2 − 2 0 0 7

 1 0 0   1 1 2     0 1 0   0 2 −2  =   0 −1 1 0 2 5

Gaussian Elimination in Practice

Now do Gaussian eliminiation on the second column.  1 1 2   1 0 0   0 2 −2  G2 =  0 1 0  0 2 5 0 −1 1 1 1 2 0 2 − 2 0 0 7

Gaussian Elimination in Practice

Now do Gaussian eliminiation on the second column.  1 1 2   1 0 0   0 2 −2  G2 =  0 1 0  0 2 5 0 −1 1

 1 0 0   1 1 2     0 1 0   0 2 −2  =   0 −1 1 0 2 5 1 2 0 2 − 2 0 0 7

Gaussian Elimination in Practice

Now do Gaussian eliminiation on the second column.  1 1 2   1 0 0   0 2 −2  G2 =  0 1 0  0 2 5 0 −1 1

 1 0 0   1 1 2   1   0 1 0   0 2 −2  =   0 −1 1 0 2 5 1 2 2 − 2 0 7

Gaussian Elimination in Practice

Now do Gaussian eliminiation on the second column.  1 1 2   1 0 0   0 2 −2  G2 =  0 1 0  0 2 5 0 −1 1

 1 0 0   1 1 2   1   0 1 0   0 2 −2  =  0  0 −1 1 0 2 5 0 2 2 − 2 0 7

Gaussian Elimination in Practice

Now do Gaussian eliminiation on the second column.  1 1 2   1 0 0   0 2 −2  G2 =  0 1 0  0 2 5 0 −1 1

 1 0 0   1 1 2   1 1   0 1 0   0 2 −2  =  0  0 −1 1 0 2 5 0 2 − 2 0 7

Gaussian Elimination in Practice

Now do Gaussian eliminiation on the second column.  1 1 2   1 0 0   0 2 −2  G2 =  0 1 0  0 2 5 0 −1 1

 1 0 0   1 1 2   1 1 2   0 1 0   0 2 −2  =  0  0 −1 1 0 2 5 0 − 2 0 7

Gaussian Elimination in Practice

Now do Gaussian eliminiation on the second column.  1 1 2   1 0 0   0 2 −2  G2 =  0 1 0  0 2 5 0 −1 1

 1 0 0   1 1 2   1 1 2   0 1 0   0 2 −2  =  0 2  0 −1 1 0 2 5 0 0 7

Gaussian Elimination in Practice

Now do Gaussian eliminiation on the second column.  1 1 2   1 0 0   0 2 −2  G2 =  0 1 0  0 2 5 0 −1 1

 1 0 0   1 1 2   1 1 2   0 1 0   0 2 −2  =  0 2 − 2  0 −1 1 0 2 5 0 Gaussian Elimination in Practice

Now do Gaussian eliminiation on the second column.  1 1 2   1 0 0   0 2 −2  G2 =  0 1 0  0 2 5 0 −1 1

 1 0 0   1 1 2   1 1 2   0 1 0   0 2 −2  =  0 2 − 2  0 −1 1 0 2 5 0 0 7 0 1 0

  =   .

What is the inverse of this matrix?   Ik×k a 0  0 α 0  . 0 b I(n−k−1)×(n−k−1)

Gauss-Jordan Elimination, or Pivot Matrices

What happens in the following multiplication?

 −1    Ik×k −α a 0 a  0 α−1 0   α  −1 0 −α b I(n−k−1)×(n−k−1) b 0 1 0

What is the inverse of this matrix?   Ik×k a 0  0 α 0  . 0 b I(n−k−1)×(n−k−1)

Gauss-Jordan Elimination, or Pivot Matrices

What happens in the following multiplication?

 −1      Ik×k −α a 0 a  0 α−1 0   α  =   . −1 0 −α b I(n−k−1)×(n−k−1) b 1 0

What is the inverse of this matrix?   Ik×k a 0  0 α 0  . 0 b I(n−k−1)×(n−k−1)

Gauss-Jordan Elimination, or Pivot Matrices

What happens in the following multiplication?

 −1      Ik×k −α a 0 a 0  0 α−1 0   α  =   . −1 0 −α b I(n−k−1)×(n−k−1) b 0

What is the inverse of this matrix?   Ik×k a 0  0 α 0  . 0 b I(n−k−1)×(n−k−1)

Gauss-Jordan Elimination, or Pivot Matrices

What happens in the following multiplication?

 −1      Ik×k −α a 0 a 0  0 α−1 0   α  =  1  . −1 0 −α b I(n−k−1)×(n−k−1) b What is the inverse of this matrix?   Ik×k a 0  0 α 0  . 0 b I(n−k−1)×(n−k−1)

Gauss-Jordan Elimination, or Pivot Matrices

What happens in the following multiplication?

 −1      Ik×k −α a 0 a 0  0 α−1 0   α  =  1  . −1 0 −α b I(n−k−1)×(n−k−1) b 0   Ik×k a 0  0 α 0  . 0 b I(n−k−1)×(n−k−1)

Gauss-Jordan Elimination, or Pivot Matrices

What happens in the following multiplication?

 −1      Ik×k −α a 0 a 0  0 α−1 0   α  =  1  . −1 0 −α b I(n−k−1)×(n−k−1) b 0

What is the inverse of this matrix? Gauss-Jordan Elimination, or Pivot Matrices

What happens in the following multiplication?

 −1      Ik×k −α a 0 a 0  0 α−1 0   α  =  1  . −1 0 −α b I(n−k−1)×(n−k−1) b 0

What is the inverse of this matrix?   Ik×k a 0  0 α 0  . 0 b I(n−k−1)×(n−k−1)