Solving Sparse Linear Systems Part 2: Trilinos Solver Overview Michael A. Heroux Sandia National LaBoratories http://www.users.csBsju.edu/~mheroux/HartreeTutorialPart2.pdF Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, 1 for the United States Department of Energy under contract DE-AC04-94AL85000. Outline § Sources oF sparse linear systems. § Sparse solver ecosystem. § Details aBout sparse data structures. § Overview oF Trilinos. § Overview oF Trilinos linear solvers. 2 If You Know Some Linear Algebra § If this intro material is not new, visit the Trilinos Tutorial Site: w https://github.com/trilinos/Trilinos_tutorial/wiki/TrilinosHandsOnTutorial w Lots of examples. w A Web portal for compiling, linking and running Trilinos examples. w Pointers to Trilinos User Group videos: Coverage of many Trilinos topics. § If you know Linux well: w Start installing Trilinos from the instructions. w Install these packages: • Kokkos, Tpetra, Epetra, EpetraExt, AztecOO, ML, Belos, Muelu, Ifpack, Ifpack2, Zoltan, Zoltan2, Amesos • Rob Allan has already done this on Tutorial systems, so he can assist you. § If you are into Docker: w Go https://hub.docker.com/r/sjdeal/webtrilinos/ for image. § If you want to start on the Epetra, AztecOO example: w Scroll down the tutorial page and start using and studying them. 3§ Explore… Setting up Linux Environment § Commands: w module avail # (look for Trilinos install) w module load intel # (get Intel compilers) w module load trilinos/12.2.1 w env # (look for path to Trilinos) § Copy and paste sample makefile from Hands-on page w http://trilinos.org/oldsite/Export_Makefile_example.txt w Get rid of references to libmyappLib w Define: MYAPP_TRILINOS_DIR=/gpfs/stfc/local/apps/intel/trilinos/12.2.1 § Try to make. 4 Sparse Linear Systems 5 Matrices § Matrix (defn): (not rigorous) An m-by-n, 2 dimensional array of numbers. § Examples: 1.0 2.0 1.5 A = 2.0 3.0 2.5 1.5 2.5 5.0 a11 a12 a13 A = a21 a22 a23 a31 a32 a33 6 6 7 Sparse Matrices § Sparse Matrix (defn): (not rigorous) An m-by-n matrix with enough zero entries that it makes sense to keep track of what is zero and nonzero. § Example: a11 a12 0 0 0 0 a21 a22 a23 0 0 0 A = 0 a32 a33 a34 0 0 0 0 a43 a44 a45 0 0 0 0 a54 a55 a56 0 0 0 0 a65 a66 7 Dense vs Sparse Costs § What is the cost of storing the tridiagonal matrix with all entries? § What is the cost if we store each of the diagonals as vector? § What is the cost of computing y = Ax for vectors x (known) and y (to be computed): w If we ignore sparsity? w If we take sparsity into account? 8 9 Origins of Sparse Matrices § In practice, most large matrices are sparse. Specific sources: w Differential equations. • Encompasses the vast majority of scientific and engineering simulation. • E.g., structural mechanics. – F = ma. Car crash simulation. w Stochastic processes. • Matrices describe probability distribution functions. w Networks. • Electrical and telecommunications networks. • Matrix element aij is nonzero if there is a wire connecting point i to point j. w 3D imagery for Google Earth • Relies on SuiteSparse (via the Ceres nonlinear least squares solver developed by Google). w And more… 9 10 Example: 1D Heat Equation (Laplace Equation) § The one-dimensional, steady-state heat equation on the interval [0,1] is as follows: u ''(x) = 0, 0 < x <1. u(0) = a. u(1) = b. § The solution u(x), to this equation describes the distribution of heat on a wire with temperature equal to a and b at the left and right endpoints, respectively, of the wire. 10 11 Finite Difference Approximation § The following formula provides an approximation of u”(x) in terms of u: u(x + h) − 2u(x) + u(x − h) u"(x) = + O(h2 ) 2h2 § For example if we want to approximate u”(0.5) with h = 0.25: u(0.75) − 2u(0.5) + u(0.25) u"(0.5) ≈ 2(1 4)2 11 1D Grid 12 Note that it is impossiBle to Find u(x) For all values oF x. Instead we: Create a “grid” with n points. Then Find an approximate to u at these grid points. IF we want a Better approximation, we increase n. u(1)=b u(0)=a u(0.25)=u u(0.5)=u u(0.75)=u 1 2 3 = u4 u(x): = u0 Interval: x: x0= 0 x1= 0.25 x2= 0.5 x3= 0.75 x4= 1 Note: We know u0 and u4 . We know a relationship Between the ui via the Finite diFFerence equations. We need to Find ui for i=1, 2, 3. 12 13 What We Know Left endpoint: 1 (u − 2u + u ) = 0, or 2u − u = a. h2 0 1 2 1 2 Right endpoint: 1 (u − 2u + u ) = 0, or 2u − u = b. h2 2 3 4 3 2 Middle points: 1 (u − 2u + u ) = 0, or − u + 2u − u = 0 for i = 2. h2 i−1 i i+1 i−1 i i+1 13 14 Write in Matrix Form ⎡ 2 −1 0 ⎤ ⎡u1 ⎤ ⎡a⎤ ⎢ 1 2 1⎥ ⎢u ⎥ ⎢0⎥ ⎢− − ⎥ ⎢ 2 ⎥ = ⎢ ⎥ ⎣⎢ 0 −1 2 ⎦⎥ ⎣⎢u3 ⎦⎥ ⎣⎢b⎦⎥ Notes: 1. This system was assembled From pieces oF what we know. 2. This is a linear system with 3 equations and three unknowns. 3. We can easily solve. 4. Note that n=5 generates this 3 equation system. 5. In general, for n grid points on [0, 1], we will have n-2 equations and unknowns. 14 15 General Form of 1D Finite Difference Matrix ⎡ 2 −1 0 0 0 ⎤ ⎡ u1 ⎤ ⎡a⎤ ⎢ 1 2 1 0 0 ⎥ ⎢ u ⎥ ⎢0⎥ ⎢− − ⎥ ⎢ 2 ⎥ ⎢ ⎥ ⎢ 0 0 ⎥ ⎢ u3 ⎥ = ⎢0⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ −1⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ 0 0 0 −1 2 ⎦ ⎣un−1 ⎦ ⎣b⎦ 15 16 A View of More Realistic Problems § The previous example is very simple. § But basic principles apply to more complex problems. § Finite difference approximations exist for any differential equation. § Finite volume is widely used. § Sandia primarily used finite elements. § Leads to far more complex matrix patterns. § Fun example… 16 “Tapir” Matrix (John Gilbert) 17 17 Corresponding Mesh 18 18 Spot Quiz Define sparse matrix. 19 20 Sparse Linear Systems: Problem Definition § A frequent requirement for scientific and engineering computing is to solve: Ax = b where A is a known large (sparse) matrix a linear operator, b is a known vector, x is an unknown vector. NOTE: We are using x differently than before. Previous x: Points in the interval [0, 1]. New x: Vector of u values. § Goal: Find x. § Question: How do we solve this problem? 20 WHY AND HOW TO USE SPARSE SOLVER LIBRARIES 21 2 1 § A farmer had chickens and pigs. There was a total of 60 heads and 200 feet. How many chickens and how many pigs did the farmer have? § Let x be the number of chickens, y be the number of pigs. § Then: x + y = 60 2x + 4y = 200 § From first equation x = 60 – y, so replace x in second equation: 2(60 – y) + 4y = 200 § Solve for y: 120 – 2y + 4y = 200 2y = 80 y = 40 § Solve for x: x = 60 – 40 = 20. § The farmer has 20 chickens and 40 pigs. 22 § A restaurant owner purchased one box of frozen chicken and another box of frozen pork for $60. Later the owner purchased 2 boxes of chicken and 4 boxes of pork for $200. What is the cost of a box of frozen chicken and a box of frozen pork? § Let x be the price of a box of chicken, y the price of a box of pork. § Then: x + y = 60 2x + 4y = 200 § From first equation x = 60 – y, so replace x in second equation: 2(60 – y) + 4y = 200 § Solve for y: 120 – 2y + 4y = 200 2y = 80 y = 40 § Solve for x: x = 60 – 40 = 20. § A box of chicken costs $20 and a box of pork costs $40. 23 ProBlem Statement § A restaurant owner purchased one box of frozen chicken and another box of frozen pork for $60. Later the owner purchased 2 boxes of chicken and 4 boxes of pork for $200. What is the cost of a box of frozen chicken and a box of frozen pork? § Let x be the price of a box of chicken, y the price of a box of pork. Variables § Then: ProBlem Setup x + y = 60 2x + 4y = 200 § From first equation x = 60 – y, so replace x in second equation: 2(60 – y) + 4y = 200 Solution Method § Solve for y: 120 – 2y + 4y = 200 2y = 80 y = 40 § Solve for x: x = 60 – 40 = 20. § A box of chicken costs $20. A box of pork costs $40. Translate Back 24 Why Sparse Solver LiBraries? § Many types oF proBlems. § Similar Mathematics. § Separation oF concerns: w ProBlem Statement. w Translation to Math. App w Set up proBlem. w Solve ProBlem. w Translate Back. SuperLU 25 Importance oF Sparse Solver Libraries § Computer solution oF math proBlems is hard: w Floating point arithmetic not exact: • 1 + ε = 1, For small ε > 0. • (a + b) + c not always equal to a + (b + c). w High Fidelity leads to large proBlems: 1M to 10B equations. w Clusters require coordinated solution across 100 – 1M processors. § Sophisticated solution algorithms and liBraries leveraged: w Solver expertise highly specialized, expensive. w Write code once, use in many settings. § Trilinos is a large collection oF state-of-the-art work: w The latest in scientiFic algorithms. w Modern soFtware design and architecture. 26 Sparse Direct Methods § Construct L and U, lower and upper triangular, resp, s.t.