Development Team

Material science

Paper No. : Mathematical tools for materials Module : Indirect methods for linear equations

Development Team

Prof. Vinay Gupta, Department of Physics and Astrophysics, Principal Investigator University of Delhi, Delhi

Prof. P. N. Kotru ,Department of Physics, University of Jammu, Paper Coordinator Jammu-180006

Content Writer Prof. V. K. Gupta , Department of Physics, University of Delhi, Delhi-110007

Prof Mahavir Singh Department of Physics, Himachal Pradesh Content Reviewer University, Shimla

Crystallography & crystal growth Material science Experimental methods for x-ray diffraction

Description of Module Subject Name Physics

Paper Name Mathematical tools for physics

Module Name/Title Indirect methods for linear Equations

NA-2 Module Id

Crystallography & crystal growth Material science Experimental methods for x-ray diffraction

TABLE OF CONTENTS 1. Introduction 2. Vector Norms 2.1 Distance between vectors in ℝ푛 2.2 Convergent sequences 3. Matrix norms and distances 4. The Jacobi iterative method 4.1 The matrix form 5. The Gauss-Seidel method 5.1 The matrix form 6. Convergence of iterative techniques 6.1Criterion for convergence

Crystallography & crystal growth Material science Experimental methods for x-ray diffraction

LEARNING OBJECTIVES

1 Various vector norms are introduced. The associated notion of vector distance is also introduced. 2 Idea of convergence is defined and limit of a sequence of vectors introduced. 3 Next, matrix norms and distances. 4 The Jacobi iterative method is described and also put in the matrix form. 5 Next the Gauss-Seidel method is described. This is also put in the matrix form as well. 6 Convergence of iterative techniques is discussed and simple criterion for convergence described.

Crystallography & crystal growth Material science Experimental methods for x-ray diffraction

Indirect methods for solution of linear Equations 1. Introduction In this unit we describe iterative techniques for solving linear systems. Jacobi and the Gauss-Seidel iterative methods are the two classic iterative methods in this class. Iterative techniques are seldom used for solving linear systems of small dimension since the time required for sufficient accuracy exceeds that required for direct techniques such as Gauss elimination. For large systems with a high percentage of 0 entries, however, these techniques are often much more efficient. Systems of this type arise frequently in circuit analysis and in the numerical solution of boundary-value problems and partial-differential equations. 2. Vector norms Before we begin to describe these methods, we introduce certain theoretical ideas about norms of matrices and vectors which are useful in the analysis of the Jacobi and the Gauss-Seidel iterative techniques. We already introduced the idea of a vector norm earlier. Now we will introduce two types of vector norms, the l2 and l∞ T norms of a vector x={x1, x2, …..xn} [the vector x is a column; it is being written as the transpose of a row only to save space.] The l2 and l∞ norms of a vector are respectively

푛 2 1/2 ‖퐱‖2 = {∑푖=1 푥푖 } (1)

‖퐱‖∞ = max |푥푖| (2) 푖=1,푛

The l2 norm is what we called the Euclidean norm of the vector x. It is not difficult to show that the norms defined above do satisfy the properties required of a norm that we described in the unit on matrices. Example 푇 The l2 and l∞ norms of the vector 퐱 = {1,2, −2} are respectively 푛 2 1/2 ‖퐱‖2 = {∑푖=1 푥푖 } = 3

‖퐱‖∞ = max |푥푖| = 2 푖=1,푛 2.1 Distance between vectors in ℝ푛 The distance between two vectors is defined as the norm of the difference of the vectors. Thus

푛 2 1/2 ‖퐱 − 퐲‖2 = {∑푖=1(푥푖 − 푦푖) } (3)

‖퐱 − 퐲‖∞ = max |푥푖 − 푦푖| (4) 푖=1,푛 Example Let 퐱 = {1,3,5}푇; 퐲 = {3,2,3}푇. Then

푛 2 1/2 ‖퐱 − 퐲‖2 = {∑푖=1(푥푖 − 푦푖) } = 3

‖퐱 − 퐲‖∞ = max |푥푖 − 푦푖| = 2 푖=1,푛 2.2 Convergent sequence

Crystallography & crystal growth Material science Experimental methods for x-ray diffraction

The concept of distance in ℝ푛 is also used to define a limit of a sequence of vectors in this space. A sequence of 푘 ∞ 푛 vectors {퐱 }푘=1 in ℝ is said to converge to x with respect to the norm ‖. ‖ if, given any ε > 0, there exists an integer N(ε) such that ‖퐱(푘) − 퐱‖ < 휀 for all 푘 ≥ 푁(휀) Regarding the convergence of sequences of vectors, we have the following useful theorems. Theorem-1 푘 ∞ 푛 The sequence of vectors {퐱 }푘=1 converges to x in ℝ with respect to the l∞ norm if, and only if, 푘 lim 푥 = 푥푖; 푖 = 1,2, ⋯ 푛 푘→∞ 푖 Proof (k) Suppose {x } converges to x with respect to the l∞ norm. Then, given any ε > 0, there exists an integer n(ε) such that for all k > n(ε)

(푘) (푘) max |푥 − 푥푖| = ‖퐱 − 퐱‖ < 휀 푖=1 푡표 푛 푖 ∞

(푘) 푘 This implies that |푥 − 푥푖| < 휀 ⟹ lim (푥푖) = 푥푖 for each 푖 푖 푘→∞ The converse can be proved in a similar manner. Example 2 3 Consider the sequence of vectors {1, 2 + , 3 + , 4푒−푘}푇. For 푘 → ∞ it converges to {1,2,3,0}T with respect to 푘 푘2 the l∞ norm. Further, given any ε > 0, there exists an integer N(ε/2) with the property that 휀 휖 ‖퐱(푘) − 퐱‖ ≤ whenever 푘 ≥ 푁( ) ∞ 2 2 By the theorem-1 above, this implies that

2휖 ‖퐱(퐤) − 퐱‖ ≤ √4‖퐱(푘) − 퐱‖ ≤ = 휀 2 ∞ 2

휖 (k) when 푘 ≥ 푁 ( ). So x converges to x with respect to l2 norm as well. 2 Theorem-2 For each

푛 푥 ∈ ℝ , ‖퐱‖∞ ≤ ‖퐱‖2 ≤ √푛‖퐱‖∞. Proof Let j be such that |xj| is the maximum of |xi| for i = 1, 2, ...n. Then

2 2 2 푛 2 2 ‖퐱‖∞ = |푥푗| = 푥푗 ≤ ∑푖=1 푥푖 = ‖퐱‖2 Or

‖퐱‖∞ ≤ ‖퐱‖2 6

Crystallography & crystal growth Material science Experimental methods for x-ray diffraction

Further, since xj is the maximum in magnitude

2 푛 2 푛 2 2 2 ‖퐱‖2 = ∑푖=1 푥푖 ≤ ∑푖=1 푥푗 = 푛푥푗 = 푛‖퐱‖∞ ⟹ ‖퐱‖2 ≤ √푛‖퐱‖∞ Combining the two inequalities we get the desired result

‖퐱‖∞ ≤ ‖퐱‖2 ≤ √푛‖퐱‖∞. QED 3. Matrix norms and distances A matrix norm on the set of all n × n matrices is a real-valued function, ‖. ‖ defined on this set, satisfying the following properties for all n × n matrices A and B and all real numbers α:

(i) ‖퐀‖ ≥ 0, ‖퐀‖ = 0 if, and only if, the matrix 퐀 is a null matrix.

(ii) ‖α퐀‖ = |훼|‖퐀‖

(iii) ‖퐀 + 퐁‖ ≤ ‖퐀‖ + ‖퐁‖

(iv) ‖퐀퐁‖ ≤ ‖퐀‖‖퐁‖

The distance between n × n matrices A and B with respect to this matrix norm is ‖퐀 − 퐁‖. Although matrix norms can be obtained in various ways, the norms that we consider are those that follow from the vector norms l2 and l∞. These norms are defined using the following theorem: Theorem-3 For every vector norm ‖. ‖ ‖퐀‖ = max ‖퐀퐱‖ (5) ‖퐱‖=1 is a matrix norm. Thus every vector norm defines a matrix norm. In addition there could be, and are, matrix norms which have no counterpart in vector norms. A matrix norm defined by a vector norm is called the natural or induced matrix norms associated with that vector norm. If 퐱 ≠ 0, the vector 퐳 = 퐱/|퐱| is a unit vector. Hence

퐱 ‖퐀퐱‖ max‖퐀퐳‖ = max ‖퐀( )‖ = max ‖퐳‖=1 ‖퐱‖≠0 |퐱| ‖퐱‖≠0 ‖퐱‖ We can therefore alternately write

‖퐀퐱‖ ‖퐀‖ = max (6) ‖퐱‖≠0 ‖퐱‖ From this result follows the following corollary: for any vector 퐱 ≠ 0, any A, and any natural matrix norm ‖퐀퐱‖ ≤ ‖퐀‖‖퐱‖ (7)

From the l2 and l∞ vector norms, there follow the l∞ matrix norm

‖퐀‖∞ = max ‖퐀퐱‖∞ (8) ‖퐱‖∞=1

and the l2 matrix norm 7

Crystallography & crystal growth Material science Experimental methods for x-ray diffraction

‖퐀‖2 = max ‖퐀퐱‖2 (9) ‖퐱‖2=1 Theorem-4 The l∞ norm of a matrix can be easily computed by the use of the following theorem: If A = (aij) is an n x n matrix, 푛 ‖퐀‖∞ = max ∑푗=1 |푎푖푗|. (10) 푖=1,2,⋯푛 Proof Let x be an n-dimensional vector with ‖퐱‖∞ = max |푥푖| = 1. Since Ax is also a vector, 푖=1,⋯푛 푛 푛 푛 ‖퐴퐱‖∞ = max |(퐀퐱)푖| = max |∑푗=1 푎푖푗푥푗| ≤ max ∑푗=1 |푎푖푗| max |∑푗=1 푥푗| 푖=1,⋯푛 푖=1,⋯푛 푖=1,⋯푛 푗=1,⋯푛

푛 But max |∑푗=1 푥푗| = ‖퐱‖∞ = 1, so that 푗=1,⋯푛

‖퐀퐱‖∞ ≤ max ‖퐀퐱‖∞ ‖퐱‖∞=1 And consequently

‖퐀‖∞ = ‖퐀퐱‖∞ ≤ max ‖퐀퐱‖∞ ‖퐱‖∞=1 Let l be an integer such that 푛 ∑푗=1 |푎푙푗| = max |푎푖푗| 푖=1,⋯푛 and x be a vector with components

1 푖푓 푎푙푗 ≥ 0 푥푗 = { −1 푖푓 푎푙푗 < 0

Then ⟦퐱⟧∞ = 1; 푎푙푗푥푗 = |푎푙푗|; 푗 = 1,2, ⋯ 푛, and this implies

푛 ‖퐀‖∞ = ‖퐀퐱‖∞ ≥ max |∑푗=1 푎푖푗| 푖=1,⋯푛 On combining the two inequalities, we get

푛 ‖퐀‖∞ = max |∑푗=1 푎푖푗| QED 푖=1,⋯푛 Example 1 2 −1 For the matrix 퐴 = [2 −1 2 ], 3 −2 −2 3 ∑푗=1 |푎1푗| = |1| + |2| + |−1| = 4 8

Crystallography & crystal growth Material science Experimental methods for x-ray diffraction

3 ∑푗=1 |푎2푗| = |2| + |−1| + |2| = 5

3 ∑푗=1 |푎3푗| = |3| + |−2| + |−2| = 7 Hence

‖퐀‖∞ = max{4,5,7} = 7 Theorem-5 The l2 norm of a matrix is related to the spectral radius of the matrix. The spectral radius ρ of a matrix was defined earlier in the units on matrices:

휌 = max(|휆푖|), 휆푖, 푖 = 1,2, ⋯ 푛 (11)

λi being eigenvalues of A. Since even real matrices can have complex eigenvalues, some of the λi may be 2 2 complex: λ = λ1 + iλ2. In that case |휆| = √휆1 + 휆2. The l2 norm of a matrix is related to its spectral radius by

푇 (i) ‖퐀‖2 = √휌(퐀 퐀) (12) (ii) 휌(퐀) ≤ ‖퐀‖ for any natural norm of 퐀 (13)

Proof of (ii) Proof of part (i), equation (12) is rather involved and we do not present it here. Proof of part (ii) is trivial. Let λ be an eigenvalue of A and let the corresponding normalized eigenvector be x: ‖퐱‖ = 1. Then Ax = λx and |휆| = |휆|‖퐱‖ = ‖휆퐱‖ = ‖퐀퐱‖ ≤ ‖퐀‖‖퐱‖ = ‖퐀‖ Since for each λ, |휆| ≤ ‖퐀‖, it follows that 휌(퐴) = max 휆 ≤ ‖퐀‖ QED  If A is symmetric then A and its transpose have the same eigenvalues. Hence part (i) of the theorem above implies as a corollary that if A is symmetric, ‖퐀‖2 = 휌(퐀). Example-1 0 −1 Let 퐀 = [ ] 1 1 0 1 0 1 0 −1 1 1 Then 퐀푇 = [ ]; 퐀푇퐀 = [ ] [ ] = [ ] −1 1 −1 1 1 1 1 2

1 1 3±√5 3+√5 3+√5 Eigenvalues of [ ] are 휆 = . Hence 휌(퐀푇퐀) = and ‖퐀‖ = √ 1 2 2 2 2 2 Example-2 2 1 0 Let 퐀 = [1 2 0] 0 0 3 This is a symmetric matrix with eigenvalues 1,3,3. Hence

‖퐀‖2 = 휌(퐀) = 3 9

Crystallography & crystal growth Material science Experimental methods for x-ray diffraction

4. The Jacobi iterative method An iterative technique to solve the n × n linear system Ax = b starts with an initial approximation x(0) to the ∞ solution x and generates a sequence of vectors {퐱(푘)} that converges to x. How from the given iteration x(i) 푘=1 the next iteration x(i + 1) is to be obtained depends on the method employed. In the Jacobi iterative method we solve the ith equation in Ax = b for xi to obtain (provided 푎푖푖 ≠ 0)

푛 푎푖푗푥푗 푏푖 푥푖 = ∑푗=1(− ) + ; 푖 = 1,2, ⋯ , 푛 (14) 푎푖푖 푎푖푖 푗≠푖

(k + 1) (k + 1) (k) Now for each k ≥ 0, generate the components xi of x from the components of x by

(푘+1) 1 푛 (푘) 푥푖 = [∑푗=1 (−푎푖푗푥푗 ) + 푏푖] ; 푖 = 1,2, ⋯ , 푛; 푘 = 0,1, ⋯ (15) 푎푖푖 푗≠푖

(k + 1) (k) The iterative process is continued till difference between xi and xi becomes less than the desired accuracy. The accuracy is determined by the l∞ norm:

‖퐱(푘+1)−퐱(퐤)‖ ∞ < desired accuracy. ‖퐱(푘)‖ ∞

 Because of division by aii in equations (14) and (15) we require that aii be nonzero, for each i = 1, 2, . . , n.  If one of the aii entries is 0 and the system is nonsingular, a reordering of the equations can be performed so that no aii = 0.  To speed convergence, the equations should be arranged so that aii is as large as possible.

Example Consider the system of three equations 10푥1 − 푥2 = 9

−푥1 + 10푥2 − 2푥3 = 7

−2푥2 + 10푥3 = 8

(0) T The correct solution is x1 = x2 = x3 = 1. Let us start with x = {0,0,0} . From the first, second and third equations we obtain respectively,

푥(0)+9 푥(0)+2푥(0)+7 2푥(0)+8 푥(1) = 2 = 0.9; 푥(1) = 1 3 = 0.7; 푥(1) = 2 = 0.8; ‖퐱(1)‖ = 0.9 1 10 2 10 3 10 ∞

푥(1)+9 푥(1)+2푥(1)+7 2푥(1)+8 푥(2) = 2 = 0.97; 푥(2) = 1 3 = 0.95; 푥(2) = 2 = 0.94; ‖퐱(2)‖ = 0.97 1 10 2 10 3 10 ∞

‖퐱(2)−퐱(ퟏ)‖ ‖퐱(2) − 퐱(ퟏ)‖ = 0.25 ⟹ ∞ ≅ 0.25 ∞ ‖퐱(1)‖ ∞

푥(2)+9 푥(2)+2푥(2)+7 2푥(2)+8 푥(3) = 2 = 0.995; 푥(3) = 1 3 = 0.985; 푥(3) = 2 = 0.99; ‖퐱(2)‖ = 0.995 1 10 2 10 3 10 ∞

Crystallography & crystal growth Material science Experimental methods for x-ray diffraction

‖퐱(2)−퐱(ퟏ)‖ ‖퐱(3) − 퐱(ퟐ)‖ = 0.05 ⟹ ∞ ≅ 0.05 ∞ ‖퐱(1)‖ ∞ For more accuracy than this, we need to calculate x(4), x(5), … as well.

4.1 The matrix form In general, iterative techniques for solving linear systems involve a process that converts the system Ax = b into an equivalent system of the form x = Tx+c for some fixed matrix T and vector c. After the initial vector x(0) is selected, the sequence of approximate solution vectors is generated by computing 퐱(푘+1) = 퐓퐱(푘) + 퐜 (16) The Jacobi method that we are considering can also be put in this form. For this purpose we split the matrix A into three parts; the diagonal, the upper triangular and the lower triangular parts: A = D – L – U (17)

−푎 푎11 −푎 −푎12 ⋯ 1푛 푎 21 ⋮ 퐷 = [ 22 ⋯ ] ; 퐿 = [ ] ; 푈 = [ ⋱ ] ⋮ ⋱ −푎푛−1,푛 푎푛푛 −푎푛,푛−1 −푎푛1 ⋯ Using this splitting, the equation Ax = b is rewritten as Dx = (L + U)x + b

If none of the aii is zero, then the matrix D is invertible and we have x = D-1(L + U)x + D-1b (18) This is the matrix form of the Jacobi method:

퐱(푘+1) = 퐃−1(퐋 + 퐔)퐱(푘) + 퐃−1퐛; 푘 = 0,1,2, ⋯ If we call 퐃−1(퐋 + 퐔) = 퐓; 퐃−1퐛 = 퐜, (19) we see that Jacobi method is put in the form

퐱(푘+1) = 퐓퐱(푘) + 퐜; 푘 = 0,1,2, ⋯  Equations (16-19) are useful for theoretical analysis; however, for actual computations we employ equations (14) and (15).

Example The system of equations

Crystallography & crystal growth Material science Experimental methods for x-ray diffraction

5푥1 + 3푥2 + 푥3 = 14 10푥 + 5푥 + 푥 = 24 1 2 4 푥1 + 5푥3 + 푥4 = 20 5푥1 + 3푥2 + 푥3 + 3푥4 = 26 can be rewritten as

3 1 14 푥 = − 푥 − 푥 + 1 5 2 5 3 5 1 24 푥 = 2푥 − 푥 + 2 1 5 4 5 푥 푥 푥 = − 1 − 4 + 4 3 5 5 5 1 26 푥 = − 푥 − 푥 − 푥 + 4 3 1 2 3 3 3 Hence, we have

0 −3/5 −1/5 0 14/5 2 0 −1/5 24/5 푇 = [ 0 ] ; 푐 = [ ] −1/5 0 0 −1/5 4 −5/3 −1 −1/3 0 26/3 5. The Gauss-Seidel Method The Jacobi iteration method can be improved upon by a simple variation of the procedure. In equation (15) to find the (k + 1)th iterate for xi, we use on the right hand side the kth iterates for all xj. This is alright for i = 1. But (k + 1) (k + 1) once x1 has been determined, to determine x2 we can use the new improved (expectedly better (k + 1) (k ) approximation) x1 on the right hand side, instead of the old x1 . For the rest, we have to use the old iterate (k ) (푘+1) (푘+1) (푘+1) (k + 1) xi . Continuing in this fashion, once 푥1 , 푥2 , ⋯ , 푥푙−1 have been determined, to determine xl use (푘+1) (푘+1) (푘+1) 푥1 , 푥2 , ⋯ , 푥푙−1 on the right hand side since they are already known. For the rest we use (푘) (푘) (푘) 푥푙+1, 푥푙+2, ⋯ , 푥푛 . Thus equation (15) is simply replaced by

(푘+1) 1 푖−1 (푘+1) 푛 (푘) 푥푖 = [∑푗=1 (−푎푖푗푥푗 ) + ∑푗=푖+1 (−푎푖푗푥푗 ) + 푏푖] ; 푖 = 1,2, ⋯ , 푛; 푘 = 0,1, ⋯ (20) 푎푖푖 This modification is called the Gauss-Seidel iterative method. Example Let us look at the earlier example once again:

10푥1 − 푥2 = 9

−푥1 + 10푥2 − 2푥3 = 7

−2푥2 + 10푥3 = 8

(0) T The correct solution is x1 = x2 = x3 = 1. Let us start with x = {0,0,0} . From the first, second and third equations we obtain respectively,

( ) ( ) ( ) ( ) 푥 0 +9 푥 1 +2푥 0 +7 2푥 1 +8 푥(1) = 2 = 0.9; 푥(1) = 1 3 = 0.79; 푥(1) = 2 = 0.958; 1 10 2 10 3 10 12

Crystallography & crystal growth Material science Experimental methods for x-ray diffraction

‖퐱(1)‖ = 0.958 ∞

( ) ( ) ( ) ( ) 푥 1 +9 푥 2 +2푥 1 +7 2푥 2 +8 푥(2) = 2 = 0.979; 푥(2) = 1 3 = 0.9895; 푥(2) = 2 = 0.9979; 1 10 2 10 3 10

‖퐱(2)−퐱(ퟏ)‖ ‖퐱(2)‖ = 0.9979 ⟹ ‖퐱(2) − 퐱(ퟏ)‖ = 0.1795 ⟹ ∞ ≅ 0.18 ∞ ∞ ‖퐱(1)‖ ∞

( ) ( ) ( ) ( ) 푥 2 +9 푥 3 +2푥 2 +7 2푥 2 +8 푥(3) = 2 = 0.99895; 푥(3) = 1 3 = 0.999475; 푥(3) = 2 = 0.999895; 1 10 2 10 3 10

‖퐱(3)−퐱(ퟐ)‖ ‖퐱(3)‖ = 0.999895 ⟹ ‖퐱(3) − 퐱(ퟐ)‖ = 0.01995 ⟹ ∞ ≅ 0.02 ∞ ∞ ‖퐱(2)‖ ∞ For more accuracy than this, we need to calculate x(4), x(5), … as well.  There is a clear cut improvement as compared to the Jacobi case. The improvement is the least (k + 1) for x1, because for x1 we use all the previous iterates. The improvement increases for x2, x3, ...

and is most pronounced for xn for which all the (k + 1)th iterates are used on the right hand side.

5.1 The matrix form Let us now put the Gauss-Sidel method also in the matrix form. For this purpose we multiply equation (20) by aii and take all the (k + 1)th iterate terms on the left hand side: (푘+1) 푖−1 (푘+1) 푛 (푘) 푎푖푖푥푖 + ∑푗=1 (−푎푖푗푥푗 ) = ∑푗=푖+1 (−푎푖푗푥푗 ) + 푏푖; 푖 = 1,2, ⋯ , 푛; 푘 = 0,1, ⋯ Writing the equations for i = 1, 2, ….n, in detail we have

(푘+1) (푘) (푘) (푘) 푎11푥1 = −푎12푥2 − 푎13푥3 − ⋯ − 푎1푛푥푛 + 푏1 (푘+1) (푘+1) (푘) (푘) 푎21푥1 + 푎22푥2 = −푎23푥3 − ⋯ − 푎2푛푥푛 + 푏2 ⋯ ⋯ ⋯ = ⋯ ⋯ ⋯ ⋯

(푘+1) (푘+1) (푘+1) 푎푛1푥1 + 푎푛2푥2 − ⋯ − 푎푛푛푥푛 = 푏푛 Define the matrices D, L and U as we did for the Jacobi case. Then the matrix form of the Gauss-Sidel method is

(퐃 − 퐋)퐱(푘+1) = 퐔퐱(푘) + 퐛 Now multiply by (D – L)-1 on both sides and we have

퐱(푘+1) = (퐃 − 퐋)−1퐔퐱(푘) + (퐃 − 퐋)−1퐛 With 퐓 = (퐃 − 퐋)−1퐔; 퐜 = (퐃 − 퐋)−1퐛, (21) we have 퐱(푘+1) = 퐓퐱(푘) + 풄 (22) 13

Crystallography & crystal growth Material science Experimental methods for x-ray diffraction

 For the matrix (D – L) to be nonsingular no additional condition is required: if none of the aii is zero, then (D – L) is nonsingular.

 There is a word of caution here. It is generally true, and is borne by the example that we have chosen above, that the Gauss-Seidel method converges faster than the Jacobi method. However there are linear systems for which the Jacobi method converges but the Gauss-Seidel method does not.

6. Convergence of iterative techniques We will now study certain general properties of the iterative techniques for the solution of linear systems, in particular the question of convergence of the iterative series. The question of convergence is crucial for all problems where iterative techniques are employed. There is always a possibility that a series of iterates diverges or converges so slowly that it is of no practical use.

First of all we define a convergent matrix. An n x n matrix A is said to be convergent if

푘 lim (퐀 )푖푗 = ퟎ, 푖 = 1,2, ⋯ , 푛; 푗 = 1,2, ⋯ , 푛. 푘→∞ Theorem-6 For a convergent matrix the following statements are equivalent; that is each follows from the others: (i) A is a convergent matrix. (ii) lim ‖퐀푘‖ = 0 for all natural matrix norms. 푘→∞ (iii) ρ(A) < 1 (iv) lim 퐀k퐱 = 0 for all x. 푘→∞ Now we come to the general iteration method, equation (21).

Theorem-7 First of all we have the following result: If the spectral radius of T satisfies ρ(T) < 1, then (I – T)-1 exists and satisfies the relation (퐈 − 퐓)−1 = 퐈 + 퐓 + 퐓ퟐ + ⋯

Proof If Tx = λx then (I – T)x = (1 – λ)x. Thus if λ is an eigenvalue of T, then 1 − λ is an eigenvalue of I − T. But |λ| ≤ ρ(T) < 1, which implies that λ = 1 is not an eigenvalue of T, and 0 cannot be an eigenvalue of I − T. Hence, (I – T)-1 exists. QED Theorem-8 ∞ Further, for any x(0), the sequence {퐱(푘)} given by 푘=1

퐱(푘+1) = 퐓퐱(푘) + 풄; 푘 ≥ 0

converges to a unique solution x = Tx + c if, and only if, ρ(T) < 1. 14

Crystallography & crystal growth Material science Experimental methods for x-ray diffraction

Proof First assume that ρ(T) < 1. Then iteratively

퐱(푘+1) = 퐓퐱(푘) + 퐜 = 푻(퐓퐱(푘−1) + 퐜) + 퐜 = 퐓2퐱(푘−1) + (퐈 + 퐓)퐜

Continuing in this way, we have

퐱(푘+1) = 퐓푘+1퐱(0) + (퐈 + 퐓 + ⋯ = 퐓푘+1)퐜

Because ρ(T) < 1, theorem (6) implies T is convergent and lim 퐓푘퐱(0) = 0. Hence 푘→∞

(푘+1) 푘+1 (0) ∞ 푖 ∞ 푖 −1 lim 퐱 = lim 퐓 퐱 + ∑푖=0 퐓 퐜 = ∑푖=0 퐓 퐜 = (퐈 − 퐓) 퐜 푘→∞ 푘→∞ So x(k + 1) converges to a unique vector x given by

퐱 = (퐈 − 퐓)−1퐜 ⟹ 퐱 = 퐓퐱 + 퐜 QED

We omit the proof of the converse of the theorem.

Theorem-9 Next we have the theorem which gives the error bounds on the various iterates: For any vector c and for any ∞ natural matrix norm ||T|| < 1, the sequence {푥(푘)} given by 퐱(푘+1) = 퐓퐱(푘) + 풄; 푘 ≥ 0 converges to a 푘=1 vector x with x = Tx +c, for any x(0). Also the following error bounds hold:

‖퐱 − 퐱(푘)‖ ≤ ‖퐓‖푘 ‖퐱 − 퐱(0)‖ (23)

‖퐓‖푘 ‖퐱 − 퐱(푘)‖ ≤ ‖퐱(1) − 퐱(0)‖ (24) 1−‖퐓‖ Jacobi and Gauss-Seidel techniques Both Jacobi and Gauss-Seidel techniques can be written in the form [See equations (16),19, 21 and (22)]

(푘+1) (푘) (푘+1) (푘) 퐱 = 퐓퐽퐱 + 퐜퐽; 퐱 = 퐓퐺퐱 + 퐜퐺

−1 −1 −1 −1 퐓퐽 = 퐃 (퐋 + 퐔); 퐜 = 퐃 퐛; 퐓퐺 = (퐃 − 퐋) 퐔; 퐜 = (퐃 − 퐋) 퐛

The subscripts, J and G stand for Jacobi and Gauss-Seidel respectively. If the spectral radius ρ of TJ or TG is less than unity, the corresponding sequence will converge to the solution x of x = Ax + b. For the Jacobi method we had

퐱(푘+1) = 퐃−1(퐋 + 퐔)퐱(푘) + 퐃−1퐛 If the sequence converges to x, then

퐱 = 퐃−1(퐋 + 퐔)퐱 + 퐃−1퐛 ⟹ 퐃퐱 = (퐋 + 퐔)퐱 + 퐛 ⟹ (퐃 − 퐋 − 퐔)퐱 = 퐛 Or

Crystallography & crystal growth Material science Experimental methods for x-ray diffraction

Ax = b We can demonstrate the result for Gauss-Seidel also in the same fashion. 6.1 Criterion for convergence What we need is an executable criterion for the convergence and the rapidity of convergence of the two techniques. To provide such a criterion, we first introduce a strictly diagonally dominant matrix. An n x n matrix A is said to be strictly diagonally dominant if

푛 |푎푖푖| > ∑푗=1|푎푖푗| holds for each 푖 = 1,2, ⋯ 푛 푗≠푖 If the equality is also allowed in the above equation, then the matrix is said to be diagonally dominant. Strict diagonal dominance implies that each diagonal element is so large that its magnitude is greater than the sum of magnitudes of all other elements in that row.  A strictly diagonally dominant matrix A is nonsingular.  Moreover, in this case, Gaussian elimination can be performed on any linear system Ax = b to obtain its unique solution without row or column interchanges, and the computations will be stable with respect to the growth of round-off errors.

Theorem-10 If A is strictly diagonally dominant, then for any choice of x(0), both the Jacobi and Gauss-Seidel methods give ∞ sequences {퐱(푘)} that converge to the unique solution of Ax = b. The relationship of the rapidity of 푘=1 convergence to the spectral radius of the iteration matrix T is given by equations (23) and (24). The inequalities hold for any natural matrix norm, so it follows from theorem-5 that ‖퐱(푘) − 퐱‖ ≈ 휌(퐓)푘‖퐱(0) − 퐱‖

Thus, for any given A and b, we would like to select the procedure for which ρ(T) is the least. However, no general results exist to tell which of the two techniques, Jacobi or Gauss-Seidel, will be more successful for a particular system. In some cases, however, the answer is known.

Crystallography & crystal growth Material science Experimental methods for x-ray diffraction

SUMMARY

 Before taking up the iterative methods of solving linear systems, we introduce various vector norms and the associated notion of vector distances.  Next we introduce the idea of convergence for vectors and limit of a sequence of vectors.  Then we define matrix norms and distances in the same vein.  After these preliminaries, we describe the Jacobi iterative method for solving linear system of equations. The equations are put in a matrix form which is more useful for theoretical analysis.  Next we describe the Gauss-Seidel method, often but not always more rapidly converging than the Jacobi method. Here also the equations are put in a matrix form as well.  Finally we briefly discuss convergence of these iterative techniques and describe simple criterion for convergence.

Crystallography & crystal growth Material science Experimental methods for x-ray diffraction