Matrix Application - Truss Matrix Iterative Methods Iterative Methods Jacobi Iteration Gauss-Seidel Iteration SOR Method Numerical Analysis and Computing Lecture Notes #16 — Matrix Algebra — Norms of Vectors and Matrices Eigenvalues and Eigenvectors Iterative Techniques Joe Mahaffy, [email protected] Department of Mathematics Dynamical Systems Group Computational Sciences Research Center San Diego State University San Diego, CA 92182-7720 Joe Mahaffy, [email protected]://www-rohan.sdsu.edu/ Matrix Algebrajmahaffy — (1/51) ∼ Matrix Application - Truss Matrix Iterative Methods Iterative Methods Jacobi Iteration Gauss-Seidel Iteration SOR Method Outline 1 Matrix Application - Truss 2 Matrix Iterative Methods Basic Definitions 3 Iterative Methods Example 4 Jacobi Iteration 5 Gauss-Seidel Iteration 6 SOR Method Joe Mahaffy, [email protected] Matrix Algebra — (2/51) Matrix Application - Truss Matrix Iterative Methods Iterative Methods Jacobi Iteration Gauss-Seidel Iteration SOR Method Matrix Application - Truss Trusses are lightweight structures capable of carrying heavy loads, e.g., roofs. TRUSS 2 f 1 f f 4 3 π F /4 π/6 1 1 3 4 f f F 2 10,000 N 5 2 F3 Joe Mahaffy, [email protected] Matrix Algebra — (3/51) Matrix Application - Truss Matrix Iterative Methods Iterative Methods Jacobi Iteration Gauss-Seidel Iteration SOR Method Physics of Trusses The truss on the previous slide has the following properties: 1 Fixed at Joint 1 2 Slides at Joint 4 3 Holds a mass of 10,000 N at Joint 3 4 All the Joints are pin joints 5 The forces of tension are indicated on the diagram Joe Mahaffy, [email protected] Matrix Algebra — (4/51) Matrix Application - Truss Matrix Iterative Methods Iterative Methods Jacobi Iteration Gauss-Seidel Iteration SOR Method Static Equilibrium At each joint the forces must add to the zero vector. Joint Horizontal Force Vertical Force 1 F + √2 f + f = 0 √2 f F = 0 − 1 2 1 2 2 1 − 2 2 √2 f + √3 f = 0 √2 f f 1 f = 0 − 2 1 2 4 − 2 1 − 3 − 2 4 3 f + f = 0 f 10, 000 = 0 − 2 5 3 − 4 √3 f f = 0 1 f F = 0 − 2 4 − 5 2 4 − 3 This creates an 8 8 linear system with 47 zero entries and 17 × nonzero entries. Sparse matrix – Solve by iterative methods Joe Mahaffy, [email protected] Matrix Algebra — (5/51) Matrix Application - Truss Matrix Iterative Methods Iterative Methods Basic Definitions Jacobi Iteration Gauss-Seidel Iteration SOR Method Earlier Iterative Schemes Earlier we used iterative methods to find roots of equations f (x) = 0 or fixed points of x = g(x) The latter requires g (x) < 1 for convergence. | ′ | Want to extend to n-dimensional linear systems. Joe Mahaffy, [email protected] Matrix Algebra — (6/51) Matrix Application - Truss Matrix Iterative Methods Iterative Methods Basic Definitions Jacobi Iteration Gauss-Seidel Iteration SOR Method Basic Definitions Definition A Vector norm on Rn is a function mapping Rn R with the ||·|| → following properties: (i) x 0 for all x Rn || || ≥ ∈ (ii) x = 0 if and only if x = 0 || || (iii) αx = α x for all α R and x Rn (scalar || || | ||| || ∈ ∈ multiplication) (iv) x + y x + y for all x, y Rn (triangle inequality) || || ≤ || || || || ∈ Joe Mahaffy, [email protected] Matrix Algebra — (7/51) Matrix Application - Truss Matrix Iterative Methods Iterative Methods Basic Definitions Jacobi Iteration Gauss-Seidel Iteration SOR Method Common Norms The l1 norm is given by n x = xi || ||1 | | i =1 The l2 norm or Euclidean norm is given by 1 n 2 x = x2 || ||2 i i =1 The l norm or Max norm is given by ∞ x = max xi || ||∞ 1 i n | | ≤ ≤ The Euclidean norm represents the usual notion of distance (Pythagorean theorem for distance). Joe Mahaffy, [email protected] Matrix Algebra — (8/51) Matrix Application - Truss Matrix Iterative Methods Iterative Methods Basic Definitions Jacobi Iteration Gauss-Seidel Iteration SOR Method Triangle Inequality We need to show the triangle inequality for . ||·||2 Theorem (Cauchy-Schwarz) For each x, y Rn ∈ n n 1/2 n 1/2 t 2 2 x y = xi yi x y = x y ≤ i i || ||2 · || ||2 i i i =1 =1 =1 Joe Mahaffy, [email protected] Matrix Algebra — (9/51) Matrix Application - Truss Matrix Iterative Methods Iterative Methods Basic Definitions Jacobi Iteration Gauss-Seidel Iteration SOR Method Triangle Inequality - Proof Cauchy-Schwarz. This result gives for each x, y Rn ∈ n 2 2 x + y = (xi + yi ) || || i=1 n n n 2 2 = xi + 2 xi yi + yi i i i =1 =1 =1 x 2 + 2 x y + y 2 ≤ || || || |||| || || || Taking the square root of the above gives the Triangle Inequality Joe Mahaffy, [email protected] Matrix Algebra — (10/51) Matrix Application - Truss Matrix Iterative Methods Iterative Methods Basic Definitions Jacobi Iteration Gauss-Seidel Iteration SOR Method Distance Definition n If x, y R , the l2 and l distances between x and y is a function ∈ ∞ mapping Rn R with the following properties:are defined by ||·|| → n 1/2 2 x y = (xi yi ) || − ||2 − i =1 x y = max1 i n xi yi || − ||∞ ≤ ≤ | − | Joe Mahaffy, [email protected] Matrix Algebra — (11/51) Matrix Application - Truss Matrix Iterative Methods Iterative Methods Basic Definitions Jacobi Iteration Gauss-Seidel Iteration SOR Method Convergence Also, we need the concept of convergence in n-dimensions. Definition A sequence of vectors x(k) in Rn is said to converge to x { }k∞=1 with respect to norm if given any ǫ> 0 there exists an integer ||·|| N(ǫ) such that x(k) x < ǫ for all k N(ǫ). || − || ≥ Joe Mahaffy, [email protected] Matrix Algebra — (12/51) Matrix Application - Truss Matrix Iterative Methods Iterative Methods Basic Definitions Jacobi Iteration Gauss-Seidel Iteration SOR Method Basic Theorems Theorem The sequence of vectors x(k) x in Rn with respect to { }k∞=1 → if and only if ||·||∞ (k) lim x = xi for each i = 1, 2,..., n. k i →∞ Theorem For each x Rn ∈ x x 2 √n x . || ||∞ ≤ || || ≤ || ||∞ It can be shown that all norms on Rn are equivalent. Joe Mahaffy, [email protected] Matrix Algebra — (13/51) Matrix Application - Truss Matrix Iterative Methods Iterative Methods Basic Definitions Jacobi Iteration Gauss-Seidel Iteration SOR Method Matrix Norm We need to extend our definitions to include matrices. Definition A Matrix Norm on the set of all n n matrices is a real-valued × function , defined on this set satisfying for all n n matrices A ||·|| × and B and all real numbers α. (i) A 0 || || ≥ (ii) A = 0 if and only if A is 0 (all zero entries) || || (iii) αA = α A (scalar multiplication) || || | ||| || (iv) A + B A + B (triangle inequality) || || ≤ || || || || (v) AB A B || || ≤ || |||| || The distance between n n matrices A and B with respect to this × matrix norm is A B . || − || Joe Mahaffy, [email protected] Matrix Algebra — (14/51) Matrix Application - Truss Matrix Iterative Methods Iterative Methods Basic Definitions Jacobi Iteration Gauss-Seidel Iteration SOR Method Natural Matrix Norm Theorem If is a vector norm on Rn, then ||·|| A = max Ax || || x =1 || || || || is a matrix norm. This is the natural or induced matrix norm associated with the vector norm. z For any z = 0, x = z is a unit vector || || z Az max Ax = max A = max || || x =1 || || z =0 z z =0 z || || || || || || || || || || Joe Mahaffy, [email protected] Matrix Algebra — (15/51) Matrix Application - Truss Matrix Iterative Methods Iterative Methods Basic Definitions Jacobi Iteration Gauss-Seidel Iteration SOR Method Matrix Action The natural norm describes how a matrix stretches unit vectors relative to that norm. (Think eigenvalues!) Theorem If A = aij is an n n matrix, then { } × n A = max aij (largest row sum) || ||∞ 1 i n | | ≤ ≤ j =1 Joe Mahaffy, [email protected] Matrix Algebra — (16/51) Matrix Application - Truss Matrix Iterative Methods Iterative Methods Basic Definitions Jacobi Iteration Gauss-Seidel Iteration SOR Method Matrix Mapping An n m matrix is a function that takes m-dimensional vectors × into n-dimensional vectors. For square matrices A, we have A : Rn Rn. → Certain vectors are parallel to Ax, so Ax = λx or (A λI )x = 0. − These values λ, the eigenvalues, are significant for convergence of iterative methods. Joe Mahaffy, [email protected] Matrix Algebra — (17/51) Matrix Application - Truss Matrix Iterative Methods Iterative Methods Basic Definitions Jacobi Iteration Gauss-Seidel Iteration SOR Method Eigenvalues and Eigenvectors Definition If A is an n n matrix, the characteristic polynomial of A is × defined by p(λ) = det(A λI ) − Definition If p is the characteristic polynomial of the matrix A, the zeroes of p are eigenvalues (or characteristic values) of A. If λ is an eigenvalue of A and x = 0 satisfies (A λI )x = 0, then x is an − eigenvector (or characteristic vector) of A corresponding to the eigenvalue λ. Joe Mahaffy, [email protected] Matrix Algebra — (18/51) Matrix Application - Truss Matrix Iterative Methods Iterative Methods Basic Definitions Jacobi Iteration Gauss-Seidel Iteration SOR Method Geometry of Eigenvalues and Eigenvectors If x is an eigenvector associated with λ, then Ax = λx, so the matrix A takes the vector x into a scalar multiple of itself. If λ is real and λ> 1, then A has the effect of stretching x by a factor of λ. If λ is real and 0 <λ< 1, then A has the effect of shrinking x by a factor of λ. If λ< 0, the effects are similar, but the direction of Ax is reversed. Joe Mahaffy, [email protected] Matrix Algebra — (19/51) Matrix Application - Truss Matrix Iterative Methods Iterative Methods Basic Definitions Jacobi Iteration Gauss-Seidel Iteration SOR Method Spectral Radius The spectral radius, ρ(A), provides a valuable measure of the eigenvalues, which helps determine if a numerical scheme will converge.