Gauss-Jordan Elimination Method

The following row operations on the augmented of a system produce the of an equivalent system, i.e., a system with the same solution as the original one.

• Interchange any two rows.

• Multiply each element of a row by a nonzero constant.

• Replace a row by the sum of itself and a constant multiple of another row of the matrix.

For these row operations, we will use the following notations.

• Ri ↔ Rj means: Interchange row i and row j.

• αRi means: Replace row i with α times row i.

• Ri + αRj means: Replace row i with the sum of row i and α times row j. The Gauss-Jordan elimination method to solve a system of linear equations is described in the following steps.

1. Write the augmented matrix of the system.

2. Use row operations to transform the augmented matrix in the form described below, which is called the reduced row echelon form (RREF).

(a) The rows (if any) consisting entirely of zeros are grouped together at the bottom of the matrix. (b) In each row that does not consist entirely of zeros, the leftmost nonzero element is a 1 (called a leading 1 or a pivot). (c) Each column that contains a leading 1 has zeros in all other entries. (d) The leading 1 in any row is to the left of any leading 1’s in the rows below it.

3. Stop process in step 2 if you obtain a row whose elements are all zeros except the last one on the right. In that case, the system is inconsistent and has no solutions. Otherwise, finish step 2 and read the solutions of the system from the final matrix.

Note: When doing step 2, row operations can be performed in any order. Try to choose row opera- tions so that as few fractions as possible are carried through the computation. This makes calculation easier when working by hand.

1 Example 1. Solve the following system by using the Gauss-Jordan elimination method.  x + y + z = 5  2x + 3y + 5z = 8 4x + 5z = 2

Solution: The augmented matrix of the system is the following.

 1 1 1 5   2 3 5 8  4 0 5 2

We will now perform row operations until we obtain a matrix in reduced row echelon form.

 1 1 1 5   1 1 1 5  R2−2R1  2 3 5 8  −−−−−→  0 1 3 −2  4 0 5 2 4 0 5 2

 1 1 1 5  R3−4R1 −−−−−→  0 1 3 −2  0 −4 1 −18

 1 1 1 5  R3+4R2 −−−−−→  0 1 3 −2  0 0 13 −26

  1 1 1 1 5 13 R3 −−−→  0 1 3 −2  0 0 1 −2

 1 1 1 5  R2−3R3 −−−−−→  0 1 0 4  0 0 1 −2

 1 1 0 7  R1−R3 −−−−→  0 1 0 4  0 0 1 −2

 1 0 0 3  R1−R2 −−−−→  0 1 0 4  0 0 1 −2

From this final matrix, we can read the solution of the system. It is

x = 3, y = 4, z = −2.

2 Example 2. Solve the following system by using the Gauss-Jordan elimination method.  x + 2y − 3z = 2  6x + 3y − 9z = 6 7x + 14y − 21z = 13

Solution: The augmented matrix of the system is the following.

 1 2 −3 2   6 3 −9 6  7 14 −21 13

Let’s now perform row operations on this augmented matrix.

 1 2 −3 2   1 2 −3 2  R2−6R1  6 3 −9 6  −−−−−→  0 −9 9 −6  7 14 −21 13 7 14 −21 13

 1 2 −3 2  R3−7R1 −−−−−→  0 −9 9 −6  0 0 0 −1

We obtain a row whose elements are all zeros except the last one on the right. Therefore, we conclude that the system of equations is inconsistent, i.e., it has no solutions.

Example 3. Solve the following system by using the Gauss-Jordan elimination method.  4y + z = 2  2x + 6y − 2z = 3 4x + 8y − 5z = 4

Solution: The augmented matrix of the system is the following.

 0 4 1 2   2 6 −2 3  4 8 −5 4

3 We will now perform row operations until we obtain a matrix in reduced row echelon form.

 0 4 1 2   2 6 −2 3  R1↔R2  2 6 −2 3  −−−−−→  0 4 1 2  4 8 −5 4 4 8 −5 4

 2 6 −2 3  R3−2R1 −−−−−→  0 4 1 2  0 −4 −1 −2

 2 6 −2 3  R3+R2 −−−−→  0 4 1 2  0 0 0 0

  1 2 6 −2 3 4 R2 −−−→  0 1 1/4 1/2  0 0 0 0

 2 0 −7/2 0  R1−6R2 −−−−−→  0 1 1/4 1/2  0 0 0 0

  1 1 0 −7/4 0 2 R1 −−−→  0 1 1/4 1/2  0 0 0 0

This last matrix is in reduced row echelon form so we can stop. It corresponds to the augmented matrix of the following system. ( 7 x − 4 z = 0 1 1 y + 4 z = 2 We can express the solutions of this system as

7 1 1 x = 4 z, y = 2 − 4 z. Since there is no specific value for z, it can be chosen arbitrarily. This means that there are infinitely many solutions for this system. We can represent all the solutions by using a parameter t as follows.

7 1 1 x = 4 t, y = 2 − 4 t, z = t

Any value of the parameter t gives us a solution of the system. For example,

1 t = 4 gives the solution (x, y, z) = (7, − 2 , 4)

7 t = −2 gives the solution (x, y, z) = (− 2 , 1, −2).

4 Example 4. Solve the following system by using the Gauss-Jordan elimination method.  A + B + 2C = 1  2A − B + D = −2 A − B − C − 2D = 4  2A − B + 2C − D = 0 Solution: We will perform row operations on the augmented matrix of the system until we obtain a matrix in reduced row echelon form.  1 1 2 0 1   1 1 2 0 1   1 1 2 0 1 

 2 −1 0 1 −2  R2−2R1  0 −3 −4 1 −4  R3−R1  0 −3 −4 1 −4    −−−−−→   −−−−→    1 −1 −1 −2 4   1 −1 −1 −2 4   0 −2 −3 −2 3  2 −1 2 −1 0 2 −1 2 −1 0 2 −1 2 −1 0

 1 1 2 0 1   1 1 2 0 1 

R4−2R1  0 −3 −4 1 −4  R4−R2  0 −3 −4 1 −4  −−−−−→   −−−−→    0 −2 −3 −2 3   0 −2 −3 −2 3  0 −3 −2 −1 −2 0 0 2 −2 2

 1 1 2 0 1   1 1 2 0 1  − 1 R R2↔R3  0 −2 −3 −2 3  2 2  0 1 3/2 1 −3/2  −−−−−→   −−−−→    0 −3 −4 1 −4   0 −3 −4 1 −4  0 0 2 −2 2 0 0 2 −2 2

 1 1 2 0 1   1 1 2 0 1 

R3+3R2  0 1 3/2 1 −3/2  2R3  0 1 3/2 1 −3/2  −−−−−→   −−→    0 0 1/2 4 −17/2   0 0 1 8 −17  0 0 2 −2 2 0 0 2 −2 2

 1 1 2 0 1   1 1 2 0 1  − 1 R R4−2R3  0 1 3/2 1 −3/2  18 4  0 1 3/2 1 −3/2  −−−−−→   −−−−→    0 0 1 8 −17   0 0 1 8 −17  0 0 0 −18 36 0 0 0 1 −2

 1 1 2 0 1   1 1 2 0 1 

R3−8R4  0 1 3/2 1 −3/2  R2−R4  0 1 3/2 0 1/2  −−−−−→   −−−−→    0 0 1 0 −1   0 0 1 0 −1  0 0 0 1 −2 0 0 0 1 −2

 1 1 2 0 1   1 0 0 0 1  R − 3 R 2 2 3  0 1 0 0 2  R1−2R3,R1−R2  0 1 0 0 2  −−−−−→   −−−−−−−−−−−→    0 0 1 0 −1   0 0 1 0 −1  0 0 0 1 −2 0 0 0 1 −2 From this final matrix, we can read the solution of the system. It is A = 1,B = 2,C = −1,D = −2.

5

Chapter 04.08 Gauss-Seidel Method

After reading this chapter, you should be able to: 1. solve a set of equations using the Gauss-Seidel method, 2. recognize the advantages and pitfalls of the Gauss-Seidel method, and 3. determine under what conditions the Gauss-Seidel method always converges.

Why do we need another method to solve a set of simultaneous linear equations? In certain cases, such as when a system of equations is large, iterative methods of solving equations are more advantageous. Elimination methods, such as , are prone to large round-off errors for a large set of equations. Iterative methods, such as the Gauss-Seidel method, give the user control of the round-off error. Also, if the physics of the problem are well known, initial guesses needed in iterative methods can be made more judiciously leading to faster convergence. What is the algorithm for the Gauss-Seidel method? Given a general set of n equations and n unknowns, we have

a11 x1 + a12 x2 + a13 x3 + ... + a1n xn = c1

a21 x1 + a22 x2 + a23 x3 + ... + a2n xn = c2 ......

an1 x1 + an2 x2 + an3 x3 + ... + ann xn = cn If the diagonal elements are non-zero, each equation is rewritten for the corresponding

unknown, that is, the first equation is rewritten with x1 on the left hand side, the second

equation is rewritten with x2 on the left hand side and so on as follows

04.08.1 04.08.2 Chapter 04.08

c1 − a12 x2 − a13 x3  − a1n xn x1 = a11

c2 − a21 x1 − a23 x3  − a2n xn x2 = a22 

cn−1 − an−1,1 x1 − an−1,2 x2  − an−1,n−2 xn−2 − an−1,n xn xn−1 = an−1,n−1

cn − an1 x1 − an2 x2 − − an,n−1 xn−1 xn = ann These equations can be rewritten in a summation form as n c1 − ∑ a1 j x j j=1 j≠1 x1 = a11 n c2 − ∑ a2 j x j j=1 j≠2 x2 = a22 . . . n cn−1 − ∑ an−1, j x j j=1 j≠n−1 xn−1 = an−1,n−1 n cn − ∑anj x j j=1 j≠n xn = ann Hence for any row i , n ci − ∑ aij x j j=1 j≠i xi = ,i =1,2,,n. aii

Now to find xi ’s, one assumes an initial guess for the xi ’s and then uses the rewritten equations to calculate the new estimates. Remember, one always uses the most recent estimates to calculate the next estimates, xi . At the end of each iteration, one calculates the

absolute relative approximate error for each xi as Gauss-Seidel Method 04.08.3

x new − x old ∈ = i i × a i new 100 xi new old where xi is the recently obtained value of xi , and xi is the previous value of xi . When the absolute relative approximate error for each xi is less than the pre-specified tolerance, the iterations are stopped.

Example 1 The upward velocity of a rocket is given at three different times in the following table

Table 1 Velocity vs. time data. Time, t (s) Velocity, v (m/s) 5 106.8 8 177.2 12 279.2

The velocity data is approximated by a polynomial as 2 v(t) = a1t + a2t + a3 , 5 ≤ t ≤12

Find the values of a1 , a2 , and a3 using the Gauss-Seidel method. Assume an initial guess of the solution as

a1  1   =   a2  2     a3  5 and conduct two iterations. Solution

The polynomial is going through three data points (t1,v1 ), (t2 ,v2 ), and (t3 ,v3 ) where from the above table

t1 = 5, v1 = 106.8

t2 = 8, v2 = 177.2

t3 = 12, v3 = 279.2 2 Requiring that v(t) = a1t + a2t + a3 passes through the three data points gives 2 v(t1 ) = v1 = a1t1 + a2t1 + a3 2 v(t2 ) = v2 = a1t2 + a2t2 + a3 2 v(t3 ) = v3 = a1t3 + a2t3 + a3

Substituting the data (t1,v1 ), (t2 ,v2 ), and (t3 ,v3 ) gives 2 a1(5 )+ a2 (5)+ a3 =106.8 2 a1(8 )+ a2 (8)+ a3 =177.2 2 a1(12 )+ a2 (12)+ a3 = 279.2 04.08.4 Chapter 04.08 or

25a1 + 5a2 + a3 =106.8

64a1 + 8a2 + a3 =177.2

144a1 +12a2 + a3 = 279.2

The coefficients a1, a2 , and a3 for the above expression are given by

 25 5 1 a1  106.8     =    64 8 1 a2  177.2       144 12 1 a3  279.2 Rewriting the equations gives 106.8 − 5a − a a = 2 3 1 25 177.2 − 64a − a a = 1 3 2 8 279.2 −144a −12a a = 1 2 3 1 Iteration #1 Given the initial guess of the solution vector as

a1  1   =   a2  2     a3  5 we get 106.8 − 5(2) − (5) a = 1 25 = 3.6720 177.2 − 64(3.6720)− (5) a = 2 8 = −7.8150 279.2 −144(3.6720)−12(− 7.8510) a = 3 1 = −155.36

The absolute relative approximate error for each xi then is 3.6720 −1 ∈ = ×100 a 1 3.6720 = 72.76% − 7.8510 − 2 ∈ = ×100 a 2 − 7.8510 = 125.47% −155.36 − 5 ∈ = ×100 a 3 −155.36 = 103.22% Gauss-Seidel Method 04.08.5

At the end of the first iteration, the estimate of the solution vector is

a1   3.6720    = −  a2   7.8510     a3  −155.36 and the maximum absolute relative approximate error is 125.47%.

Iteration #2 The estimate of the solution vector at the end of Iteration #1 is

a1   3.6720    = −  a2   7.8510     a3  −155.36 Now we get 106.8 − 5(− 7.8510)− (−155.36) a = 1 25 = 12.056 177.2 − 64(12.056)− (−155.36) a = 2 8 = −54.882 279.2 −144(12.056)−12(− 54.882) a = 3 1 = − 798.34

The absolute relative approximate error for each xi then is 12.056 − 3.6720 ∈ = ×100 a 1 12.056 = 69.543% − 54.882 − (− 7.8510) ∈ = ×100 a 2 − 54.882 = 85.695% − 798.34 − (−155.36) ∈ = ×100 a 3 − 798.34 = 80.540% At the end of the second iteration the estimate of the solution vector is

a1   12.056    = −  a2   54.882    a3  − 798.54 and the maximum absolute relative approximate error is 85.695%. Conducting more iterations gives the following values for the solution vector and the corresponding absolute relative approximate errors.

04.08.6 Chapter 04.08

∈ ∈ ∈ Iteration a1 a 1 % a2 a 2 % a3 a 3 % 1 3.6720 72.767 –7.8510 125.47 –155.36 103.22 2 12.056 69.543 –54.882 85.695 –798.34 80.540 3 47.182 74.447 –255.51 78.521 –3448.9 76.852 4 193.33 75.595 –1093.4 76.632 –14440 76.116 5 800.53 75.850 –4577.2 76.112 –60072 75.963 6 3322.6 75.906 –19049 75.972 –249580 75.931

As seen in the above table, the solution estimates are not converging to the true solution of

a1 = 0.29048

a2 =19.690

a3 =1.0857 The above system of equations does not seem to converge. Why? Well, a pitfall of most iterative methods is that they may or may not converge. However, the solution to a certain classes of systems of simultaneous equations does always converge using the Gauss-Seidel method. This class of system of equations is where the coefficient matrix [A] in [A][X ] = [C] is diagonally dominant, that is n aii ≥ ∑ aij for all i j=1 j≠i n aii > ∑ aij for at least one i j=1 j≠i If a system of equations has a coefficient matrix that is not diagonally dominant, it may or may not converge. Fortunately, many physical systems that result in simultaneous linear equations have a diagonally dominant coefficient matrix, which then assures convergence for iterative methods such as the Gauss-Seidel method of solving simultaneous linear equations.

Example 2 Find the solution to the following system of equations using the Gauss-Seidel method.

12x1 + 3x2 − 5x3 = 1

x1 + 5x2 + 3x3 = 28

3x1 + 7x2 + 13x3 = 76 Use

x1  1   =   x2  0    x3  1 as the initial guess and conduct two iterations. Solution The coefficient matrix Gauss-Seidel Method 04.08.7

12 3 − 5   [A] =  1 5 3   3 7 13  is diagonally dominant as

a11 = 12 = 12 ≥ a12 + a13 = 3 + − 5 = 8

a22 = 5 = 5 ≥ a21 + a23 = 1 + 3 = 4

a33 = 13 = 13 ≥ a31 + a32 = 3 + 7 = 10 and the inequality is strictly greater than for at least one row. Hence, the solution should converge using the Gauss-Seidel method. Rewriting the equations, we get 1− 3x + 5x x = 2 3 1 12 28 − x − 3x x = 1 3 2 5 76 − 3x − 7x x = 1 2 3 13 Assuming an initial guess of

x1  1   =   x2  0     x3  1 Iteration #1 1− 3(0)+ 5(1) x = 1 12 = 0.50000 28 − (0.50000)− 3(1) x = 2 5 = 4.9000 76 − 3(0.50000)− 7(4.9000) x = 3 13 = 3.0923 The absolute relative approximate error at the end of the first iteration is 0.50000 −1 ∈ = ×100 a 1 0.50000 =100.00% 4.9000 − 0 ∈ = ×100 a 2 4.9000 =100.00% 3.0923 −1 ∈ = ×100 a 3 3.0923 = 67.662% 04.08.8 Chapter 04.08

The maximum absolute relative approximate error is 100.00% Iteration #2 1− 3(4.9000)+ 5(3.0923) x = 1 12 = 0.14679 28 − (0.14679)− 3(3.0923) x = 2 5 = 3.7153 76 − 3(0.14679)− 7(3.7153) x = 3 13 = 3.8118 At the end of second iteration, the absolute relative approximate error is 0.14679 − 0.50000 ∈ = ×100 a 1 0.14679 = 240.61% 3.7153− 4.9000 ∈ = ×100 a 2 3.7153 = 31.889% 3.8118 − 3.0923 ∈ = ×100 a 3 3.8118 = 18.874% The maximum absolute relative approximate error is 240.61%. This is greater than the value of 100.00% we obtained in the first iteration. Is the solution diverging? No, as you conduct more iterations, the solution converges as follows.

∈ ∈ ∈ Iteration x1 a 1 % x2 a 2 % x3 a 3 % 1 0.50000 100.00 4.9000 100.00 3.0923 67.662 2 0.14679 240.61 3.7153 31.889 3.8118 18.874 3 0.74275 80.236 3.1644 17.408 3.9708 4.0064 4 0.94675 21.546 3.0281 4.4996 3.9971 0.65772 5 0.99177 4.5391 3.0034 0.82499 4.0001 0.074383 6 0.99919 0.74307 3.0001 0.10856 4.0001 0.00101

This is close to the exact solution vector of

x1  1   =   x2  3     x3  4

Example 3 Given the system of equations

3x1 + 7x2 + 13x3 = 76 Gauss-Seidel Method 04.08.9

x1 + 5x2 + 3x3 = 28

12x1 + 3x2 - 5x3 = 1 find the solution using the Gauss-Seidel method. Use

x1  1   =   x2  0     x3  1 as the initial guess. Solution Rewriting the equations, we get 76 − 7x −13x x = 2 3 1 3 28 − x − 3x x = 1 3 2 5 1−12x − 3x x = 1 2 3 − 5 Assuming an initial guess of

x1  1   =   x2  0     x3  1 the next six iterative values are given in the table below.

∈ ∈ ∈ Iteration x1 a 1 % x2 a 2 % x3 a 3 % 1 21.000 95.238 0.80000 100.00 50.680 98.027 2 –196.15 110.71 14.421 94.453 –462.30 110.96 3 1995.0 109.83 –116.02 112.43 4718.1 109.80 4 –20149 109.90 1204.6 109.63 –47636 109.90 5 2.0364×105 109.89 –12140 109.92 4.8144×105 109.89 6 –2.0579×106 109.89 1.2272×105 109.89 –4.8653×106 109.89

You can see that this solution is not converging and the coefficient matrix is not diagonally dominant. The coefficient matrix  3 7 13    [A] =  1 5 3  12 3 − 5 is not diagonally dominant as

a11 = 3 = 3 ≤ a12 + a13 = 7 + 13 = 20 Hence, the Gauss-Seidel method may or may not converge. However, it is the same set of equations as the previous example and that converged. The only difference is that we exchanged first and the third equation with each other and that made the coefficient matrix not diagonally dominant. 04.08.10 Chapter 04.08

Therefore, it is possible that a system of equations can be made diagonally dominant if one exchanges the equations with each other. However, it is not possible for all cases. For example, the following set of equations

x1 + x2 + x3 = 3

2x1 + 3x2 + 4x3 = 9

x1 + 7x2 + x3 = 9 cannot be rewritten to make the coefficient matrix diagonally dominant.

Key Terms: Gauss-Seidel method Convergence of Gauss-Seidel method Diagonally dominant matrix

Gaussian Elimination CS6015 :

Ack: 1. LD Garcia, MTH 199, Sam Houston State University 2. Linear Algebra and Its Applications - Gilbert Strang The Gaussian Elimination Method

• The Gaussian elimination method is a technique for solving systems of linear equations of any size. • The operations of the Gaussian elimination method are: 1.Interchange any two equations. 2.Replace an equation by a nonzero constant multiple of itself. 3.Replace an equation by the sum of that equation and a constant multiple of any other equation. Example • Solve the following system of equations:

Solution • First, we transform this system into an equivalent system in which the coefficient of in the first equation is 1:

Multiply the equation by 1/2 Example • Solve the following system of equations:

Solution • First, we transform this system into an equivalent system in which the coefficient of in the first equation is 1:

Multiply the equation by 1/2 Example • Solve the following system of equations:

Solution • Next, we eliminate the variable from all equations except the first:

Replace by the sum of – 3 x (the first equn.) + (second equation) Example • Solve the following system of equations:

Solution • Next, we eliminate the variable from all equations except the first:

Replace by the sum of – 3 x (the first equn.) + (second equation) Example • Solve the following system of equations:

Solution • Next, we eliminate the variable from all equations except the first:

Replace by the sum of the (first equation) + (third equation) Example • Solve the following system of equations:

Solution • Next, we eliminate the variable from all equations except the first:

Replace by the sum of the (first equation) + (third equation) Example • Solve the following system of equations:

Solution • Then we transform so that the coefficient of in the second equation is 1:

Multiply the second equation by 1/2 Example • Solve the following system of equations:

Solution • Then we transform so that the coefficient of in the second equation is 1:

Multiply the second equation by 1/2 Example • Solve the following system of equations:

Solution • We now eliminate from all equations except the second:

Replace by the sum of the first equation + (–2) x (the second equation) Example • Solve the following system of equations:

Solution • We now eliminate from all equations except the second:

Replace by the sum of the first equation + (–2) x (the second equation) Example • Solve the following system of equations:

Solution • We now eliminate from all equations except the second:

Replace by the sum of the third equation + (–3) x (the second equation) Example • Solve the following system of equations:

Solution • We now eliminate from all equations except the second:

Replace by the sum of the third equation + (–3) x (the second equation) Example • Solve the following system of equations:

Solution • Now we transform so that the coefficient of in the third equation is 1:

Triangular system Multiply the third equation by 1/11 Example • Solve the following system of equations:

Solution • Now we transform so that the coefficient of in the third equation is 1:

Multiply the third equation by 1/11 Example • Solve the following system of equations:

Solution • We now eliminate from all equations except the third:

Back substitution Putting value of z in the first equation Example • Solve the following system of equations:

Solution • We now eliminate from all equations except the third:

Back substitution Putting value of z in the first equation Example • Solve the following system of equations:

Solution • We now eliminate from all equations except the third:

Back substitution Putting value of z in the second equation Example • Solve the following system of equations:

Solution • We now eliminate from all equations except the third:

Back substitution Putting value of z in the second equation Example • Solve the following system of equations:

Solution • Thus, the solution to the system is , , and . Row-Reduced Form of a Matrix

• Each row consisting entirely of zeros lies below all rows having nonzero entries. • The first nonzero entry in each nonzero row is 1 (called a leading 1). • In any two successive (nonzero) rows, the leading 1 in the lower row lies to the right of the leading 1 in the upper row. • If a column contains a leading 1, then the other entries in that column are zeros. Row Operations

1. Interchange any two rows.

2. Replace any row by a nonzero constant multiple of itself.

3. Replace any row by the sum of that row and a constant multiple of any other row. Terminology for the Gaussian Elimination Method

Unit Column • A column in a coefficient matrix is in unit form if one of the entries in the column is a 1 and the other entries are zeros.

Pivoting • The sequence of row operations that transforms a given column in an augmented matrix into a unit column. Notation for Row Operations

• Letting 𝑖 denote the -th row of a matrix, we write Operation 1: to mean: 𝑖 𝑗 Interchange row with row .

Operation 2: 𝑖 to mean: replace row with times row .

Operation 3: 𝑖 𝑗 to mean: Replace row with the sum of row and times row . Example • Pivot the matrix about the circled element

Solution: 𝟏 𝑹 𝑹𝟐 −𝟐𝑹𝟏 𝟑 𝟏 The Gaussian Elimination Method

1. Write the augmented matrix corresponding to the linear system. 2. Interchange rows, if necessary, to obtain an augmented matrix in which the first entry in the first row is nonzero. Then pivot the matrix about this entry. 3. Interchange the second row with any row below it, if necessary, to obtain an augmented matrix in which the second entry in the second row is nonzero. Pivot the matrix about this entry. 4. Continue until the final matrix is in row-reduced form. Augmented Matrices • Matrices are rectangular arrays of numbers that can aid us by eliminating the need to write the variables at each step of the reduction. • For example, the system

may be represented by the augmented matrix

Augmented matrix [C|B] Coefficient Matrix [C] Matrices and Gaussian Elimination • Every step in the Gaussian elimination method can be expressed with matrices, rather than systems of equations, thus simplifying the whole process: • Steps expressed as systems of equations:

• Steps expressed as augmented matrices: 2𝑥 + 4𝑦 + 6𝑧 =22 24622 3𝑥 + 8𝑦 + 5𝑧 =27 38527 −𝑥+𝑦+2𝑧=2 −1122 𝟏 𝑹 = 𝑹 𝟏 𝟐 𝟏 𝑥 + 2𝑦 + 3𝑧 =11 12311 3𝑥 + 8𝑦 + 5𝑧 =27 38527 −𝑥+𝑦+2𝑧=2 −1122 𝑹𝟐 =𝑹𝟐 −𝟑𝑹𝟏 𝑥 + 2𝑦 + 3𝑧 =11 12311 2𝑦 − 4𝑧 =−6 0 2 −4 −6 −𝑥+𝑦+2𝑧=2 −1 1 2 2 𝑹𝟑 =𝑹𝟑 +𝑹𝟏 𝑥 + 2𝑦 + 3𝑧 =11 12 3 11 2𝑦 − 4𝑧 =−6 0 2 −4 −6 3𝑦 + 5𝑧 =13 03 5 13 𝟏 𝑹 = 𝑹 𝟐 𝟐 𝟐 𝟏 𝑹 = 𝑹 𝟐 𝟐 𝟐 𝑥 + 2𝑦 + 3𝑧 =11 12 3 11 𝑦 − 2𝑧 =−3 0 1 −2 −3 3𝑦 + 5𝑧 = 13 03 5 13 𝑹𝟏 =𝑹𝟏 −𝟐𝑹𝟐 𝑥 + 7𝑧 = 11 10 7 17 𝑦 − 2𝑧 = −3 0 1 −2 −3 3𝑦 + 5𝑧 = 13 03 5 13 𝑹𝟑 =𝑹𝟑 −𝟑𝑹𝟐 𝑥 + 7𝑧 = 11 10 7 17 𝑦 − 2𝑧 = −3 0 1 −2 −3 11𝑧 = 22 0 0 11 22 𝟏 𝑹𝟑 = 𝑹𝟑 𝑥 + 7𝑧 = 11 10 7 17 𝟏𝟏 𝑦 − 2𝑧 = −3 0 1 −2 −3 𝑧 = 2 00 1 2 𝑹𝟏 =𝑹𝟏 −𝟕𝑹𝟑 𝑹𝟏 =𝑹𝟏 −𝟕𝑹𝟑 𝑥 = 3 10 0 3 𝑦 − 2𝑧 = −3 0 1 −2 −3 𝑧 = 2 00 1 2 𝑹𝟐 =𝑹𝟐 +𝟐𝑹𝟑 𝑥 = 3 1003 𝑦 = −3 0101 𝑧 = 2 0012 Row Reduced Form of the Matrix

Thus, the solution to the system is , , and . Gaussian Elimination in the case of unique solution

• With a full set of 𝑛 pivots, there is only one solution. • The system is non singular, and it is solved by forward elimination and back-substitution. Under what circumstances could the process break down? • But if a zero appears in a pivot position, elimination has to stop—either temporarily or permanently. The system might or might not be singular. • If the first coefficient is zero, in the upper left corner, the elimination of u from the other equations will be impossible. • The same is true at every intermediate stage. Notice that a zero can appear in a pivot position, even if the original coefficient in that place was not zero. • Roughly speaking, we do not know whether a zero will appear until we try, by actually going through the elimination process. • In many cases this problem can be cured, and elimination can proceed. Such a system still counts as nonsingular; it is only the algorithm that needs repair. • In other cases a breakdown is unavoidable. Those incurable systems are singular, they have no solution or else infinitely many, and a full set of pivots cannot be found. Systems of Linear Equations: Underdetermined and Overdetermined systems A System of Equations with an Infinite Number of Solutions • Solve the system of equations given by 𝑥+2𝑦−3𝑧=−2 3𝑥−𝑦−2𝑧=1 2𝑥 + 3𝑦 − 5𝑧 =−3 Solution 1 2 −3 −2 1 2 −3 −2 3−1−21 0−77 7 2 3 −5 −3 𝑹𝟐 −𝟑𝑹𝟏 0−11 1 𝑹 −𝟐𝑹 𝟑 𝟏 𝟏 − 𝑹 𝟕 𝟐 10−10 1 2 −3 −2 0 1 −1 −1 0 1 −1 −1

00 0 0 𝑹𝟏 −𝟐𝑹𝟐 0−11 1 𝑹𝟑 +𝑹𝟐 A System of Equations with an Infinite Number of Solutions • Solve the system of equations given by 𝑥+2𝑦−3𝑧=−2 3𝑥 − 𝑦 − 2𝑧 =1 2𝑥 + 3𝑦 − 5𝑧 =−3 Solution

• Observe that row three reads , which is true but of no use to us. A System of Equations with an Infinite Number of Solutions • Solve the system of equations given by 𝑥+2𝑦−3𝑧=−2 3𝑥 − 𝑦 − 2𝑧 =1 2𝑥 + 3𝑦 − 5𝑧 =−3 Solution • This last augmented matrix is in row-reduced form. • Interpreting it as a system of equations gives a system of two equations in three variables , , and :

𝑥−𝑧=0 𝑦−𝑧=−1 A System of Equations with an Infinite Number of Solutions • Solve the system of equations given by 𝑥+2𝑦−3𝑧=−2 3𝑥 − 𝑦 − 2𝑧 =1 2𝑥 + 3𝑦 − 5𝑧 =−3 Solution • Let’s single out a single variable –say, – and solve for and in terms of it.

• If we assign a particular value of –say, we obtain and , giving the solution

𝑥−𝑧=0 𝑥 = 𝑧 = 0=0 𝑦−𝑧=−1 𝑦=𝑧−1= 0−1=−1 A System of Equations with an Infinite Number of Solutions • Solve the system of equations given by 𝑥+2𝑦−3𝑧=−2 3𝑥 − 𝑦 − 2𝑧 =1 2𝑥 + 3𝑦 − 5𝑧 =−3 Solution • Let’s single out a single variable –say, – and solve for and in terms of it. • If we instead assign , we obtain the solution .

𝑥−𝑧=0 𝑥 = 𝑧 = 1=1 𝑦−𝑧=−1 𝑦=𝑧−1= 1−1=0 A System of Equations with an Infinite Number of Solutions • Solve the system of equations given by 𝑥+2𝑦−3𝑧=−2 3𝑥 − 𝑦 − 2𝑧 =1 2𝑥 + 3𝑦 − 5𝑧 =−3 Solution • Let’s single out a single variable –say, – and solve for and in terms of it. • In general, we set , where represents any real number (called the parameter) to obtain the solution .

𝑥−𝑧=0 𝑥 = 𝑧 = 𝑡=𝑡 𝑦−𝑧=−1 𝑦=𝑧−1= 𝑡−1=𝑡−1 A System of Equations That Has No Solution • Solve the system of equations given by

Solution 1111 11 1 1 3−1−14 0−4−41 15 5−1𝑹𝟑 −𝟑𝑹𝟏 04 4−2 𝑹𝟑 −𝑹𝟏 𝑹𝟑 +𝑹𝟐

11 1 1 0−4−41 00 0−1 A System of Equations That Has No Solution • Solve the system of equations given by

Solution

• Observe that row three reads or ! • We therefore conclude the system is inconsistent and has no solution. Systems with no Solution

• If there is a row in the augmented matrix containing all zeros to the left of the vertical line and a nonzero entry to the right of the line, then the system of equations has no solution. Theorem

a. If the number of equations is greater (over- determined system) than or equal to the number of variables in a linear system, then one of the following is true: i. The system has no solution. ii. The system has exactly one solution. iii. The system has infinitely many solutions.

b. If there are fewer equations than variables (under-determined system) in a linear system, then the system either has no solution or it has infinitely many solutions. Cost of elimination How many separate arithmetical operations does elimination require, for n equations in n unknowns? • Ignore the right-hand sides of the equations, and count only the operations on the left. These operations are of two kinds. • We divide by the pivot to find out what multiple (say ℓ) of the pivot equation is to be subtracted. • When we do this subtraction, we continually meet a “multiply-subtract” combination; the terms in the pivot equation are multiplied by ℓ, and then subtracted from another equation. • We call each division, and each multiplication-subtraction, one operation. • In column 1, it takes 𝒏 operations for every zero we achieve—one to find the multiple ℓ, and the other to find the new entries along the row. Cost of elimination (contd.) • There are 𝑛−1rows underneath the first one, so the first stage of elimination needs 𝑛(𝑛 − 1) =𝑛 −𝑛operations.

𝟏 • If 𝒏 is at all large, a good estimate for the number of operations is 𝒏𝟑. 𝟑 • Back-substitution is considerably faster. The last unknown is found in only one operation (a division by the last pivot). The second to last unknown requires two operations, and so on. Then the total for back-substitution is 1 + 2 +··· +𝑛. Cost of elimination (contd.)

Special Notes :

There now exists a method that requires only 𝐶𝑛 multiplications! It depends on a simple fact: Two combinations of two vectors in two-dimensional space would seem to take 8 multiplications, but they can be done in 7. That lowered the exponent from log 8, which is 3, to log 7≈2.8. This discovery produced tremendous activity to find the smallest possible power of n. The exponent finally fell (at IBM) below 2.376. Fortunately for elimination, the constant C is so large and the coding is so awkward that the new method is largely (or entirely) of theoretical interest. The newest problem is the cost with many processors in parallel.