Iterative Methods for Nonsymmetric

COMPUTER PROJECT ITERATIVE METHODS FOR NONSYMMETRIC LINEAR SYSTEMS

Due December 3, 2013 100 points

Purpose: To expose you to the matrix collections at Matrix Market and University of Florida, to literature searches using MathSciNet, and to other internet resources and to have you gain experience with using Krylov subspace iterative methods intended for non-symmetric systems. Recall from our discussion at the beginning of the semester that these iterative methods were identified by the computer scientist Jack Dongarra and others to be among the top ten algorithms that have had “the greatest influence on the development and practice of science and engineering in the 20th century.”

This project will require that you use internet or the library to access and read parts of a book about iterative methods, use the library or internet to look up one or more journal articles, , download test matrices the University of Florida (UF) Sparse Matrix Collection (and potentially from the Matrix Market) and carry out some test runs in Matlab that compare two iterative method that you select or study one method using different parameter selections (such as restart in gmres). A typed report of approximately five to ten pages is required. You may do this assignment with a partner so that, if you wish, one report can be turned for two people.

The steps that you need to follow are:

1. Look at the book Templates for the Solution of Linear Systems: Building Blocks for Iterative Methods, by Richard Barrett, Michael Berry, et. al., SIAM, Philadelphia, 1994. The book is available on line at http://www.netlib.org/linalg/html_templates/Templates.html The book is part of the netlib library which is an extensive collection of high quality programs and packages related to scientific computing. You might want to open www.netlib.org and browse around the library You can browse through the book by clicking on items appearing in blue. The section “Computational Aspects of the Methods”, might be most useful to you. After reading through this section (and after classroom discussion) you will need to choose one or two of the six methods GMRES, BiCG, QMR, CGS BiCGSTAB or LSQR to study further. You will probably want to go back to “Nonstationary Iterative Methods” to read more details about the method you choose and to note the references mentioned relating to the method. LSQR is not discussed in the Templates book but is discussed in some of the course handouts and elsewhere. Also current versions of Matlab have additional Krylov subspace methods – TFQMR and BiCGSTABL – that you might explore.

2. Next you should go to MathSciNet to look at references for one of the method that you have chosen. Also you may wish to look at references in the templates book or in Watkins text book. You can access MathSciNet through the SJSU library (http://www.ams.org.libaccess.sjlibrary.org/mathscinet/ ). From off campus you may need to type in the bar code from the back of your student ID and you will also need to know you library PIN (get a PIN sooner rather than later if you don’t already have one) . After you enter MathSciNet click on "Start Full Search ". Scroll down to the “anywhere” and type in a keyword of phrase for your search. For example you might enter GMRES to search for articles about the GMRES method. Now click on start search. You should get a list of references that have this name somewhere in the abstract of the reference. One of the challenges in searching a data base is narrowing the number of hits to a set that is of interest to you. This is an art which we won't pursue here. In this case you need to find only one article to discuss (see below) but you will probably need to look at several articles to choose an appropriate article. MathSciNet will give you the abstract to the article (click on the weird number, for example 99c:65243).

3. You can read the complete article by searching on line, through the library or through a Google search. For example from MathSciNet clicking on “the original article” may work. However perhaps a password will be required and a better alternative may be to go through our library. To access the full text of an article through our library go to the site http://www.sjlibrary.org/gateways/academic/ and search the library catalog, databases or e-journals. For example if you select “journal titles” and type “SIAM” in the search window you will get a list of available on line SIAM journals. Many of the articles related to Krylov subspaces are in SIAM journals such as SIAM Journal on Matrix Analysis and Applications, SIAM Journal on Scientific Computing, or SIAM Review. The library has access to these journals over certain periods of time. I have access to all the SIAM journals and if you can’t find the article you want on line or through the library you can email me the citation and I can email you back the article. Another good way to find the text of articles is to go to the author’s web page. Often their articles will be posted on their web page. Of course search engines, especially “google scholar,” can be very useful. If none of the above works one can also request articles through interlibrary loan (http://www.sjlibrary.org/services/request/ill.htm ). This can be somewhat slow.

You should look up several of the references that relate to the method that you choose. Look for an article or articles that have examples comparing your method with other methods. Sometimes you may be able to find an article that uses a matrix that appears in the Matrix Market or UF site and you can try out the same example.

4. In addition you should get on internet to try to locate the examples and the University of Florida (UF) Sparse Matrix Collection ( http://www.cise.ufl.edu/research/sparse/matrices ) or, potentially, from the Matrix Matrix Market (http://math.nist.gov/MatrixMarket/ ). These are marvelous collections of matrices that come from real world applications that, in many cases, lead to large matrices. The University of Florida site is more comprehensive and easier to access from Matlab and therefore that site is preferable to the Matrix Market.. Also the SJSU Singular Matrix Database (http://www.math.sjsu.edu/singular/matrices/ ) is nice collection if you want to look at singular matrices. Many of the matrices are much bigger than 1000 by 1000 and indeed one matrix in the UF collection is over 118,000,000 by 118,000,000.

From Matlab 7 you can download the matrices from the UF Sparse site by two methods. Either directly from the web site by clicking on the appropriate link (see below) or you can use the UFget utility (http://www.cise.ufl.edu/research/sparse/mat/UFget.html) to download matrices from the University of Florida sparse matrix collection or the SJget utility (http://www.math.sjsu.edu/singular/matrices/SJget.html) to download matrices from the SJSU singular matrix database. These interfaced make it easy to loop through a large number of matrices which are identified by properties that you select. See the documentation at the above sites or my compare_methods program (item 65 on my web page) for examples.

I recommend that you download all your matrices in the same folder. If you plan to use several computers this is most easily done by using a thumb drive that you can move between computers. To get to this folder from Matlab, you will need to navigate from inside Matlab to the folder using the “Current Folder:” (or similar, depending on your version of Matlab) window in the Matlab menu bar.

You might want to begin by downloading the Market’s Laplace matrix. This 900 by 900 sparse (that is lots of zeros) matrix is similar to the matrix in the five point star example we will do in class (with the -1 -1 4 -1 -1 coefficient pattern). It is slightly different since a fancier approximation (the "nine point star" which has a -1 -1 -1 -1 8 -1 -1 -1 -1 coefficient pattern) is used. The gr_30_30 matrix is symmetric positive definite (SPD) and so all the methods should work well.

To download this by visiting the University Florida Web site, go to http://www.cise.ufl.edu/research/sparse/matrices, search for gr_30_30 using a utility in your browser that searches the current web page, from the line corresponding to the gr_30_30 matrix right click (I am using Chrome for my browser) on the MAT link, select “save link as” (or similar in browsers other than Chrome), and save the gr_30_30.mat in your target folder. Next from inside Matlab, after moving to the target folder, type “load gr_30_30.mat”. This will load a Matlab structure called Problem. Problem contains the matrix and other information about the matrix. Now from inside Matlab type “A = Problem.A” or, better, download my routine mat_init.m in item 65 at the course web site to your target folder and type, from inside Matlab, “mat_init.m”. The routine mat_init creates the matrix A from the structure Problem and initializes some convenient values including the right hand side, b, of a system of equations Ax=b.

An alternate, and indeed better, method to download the matrix is to use the UFget utility. Here are instruction for downloading the UFget utility: % FYI, instruction to download and install UFget: % (1) go to http://www.cise.ufl.edu/research/sparse/mat/UFget.html . % and download either UFget.zip or UFget.tar.gz and unzip the file % in a folder of your choice. UFget.zip may be easier to unzip in % Windows and UFget.tar.gz easier in Linux. In Matlab 7 there is an % unzip routine, your browser may have unzip routine or, in Windows, % windows explorer may have an unzip routine. % (2) In Matlab Move to the folder containing the file UFget.m (it may be % in a UFget subfolder). From that folder you can access UFget. % (3) If you want access to UFget from folder other than your current % folder type % addpath(pwd); savepath; % Depending on the write access you have on your computer savepath % may or may not work. If it does not you need to move to the folder % containing UFget.m to access UFget.

After installing the UFget utility type, from inside Matlab, “Problem = UFget(159); mat_init” (since gr_30_30 is matrix number 159 in the UF sparse matrix collection) or “Problem = UFget(‘HB/gr_30_30’), mat_init” (since the full name of gr_30_30, including its “group” is HB/gr_30_30).

UFget is a superior method for downloading matrices because UFget allows one to choose sets of matrices with selected properties using selectUF (item 65 in the course web page) or compare two iterative methods over sets of matrices using compare_methods (item 65 on the course web page). Use of compare_methods will be illustrated below.

After the matrix has been loaded into Matlab and will be called A. In addition the routine mat_init initialize the true solution, xtrue, and the right hand side b = A * xtrue, an intial guess x0 of all zeros and an identity matrix M which is the same size as A. We can now run the various iterative methods For example to run conjugate gradients method

max_it = 5000 tol = 1.e-8 % For your runs I recommend tol = 1.e-8 or less tic, [x,flag,relres,iter,resvec] =pcg(A,b,tol,max_it); toc semilogy(1:length(resvec),resvec1/norm(b),'--o'), grid title('pcg'), xlabel('iteration'), ylabel('norm(b-A*x) / norm(b)')

This returns a time of approximately 0.01 seconds, a number of iterations (iter1) = 51, relres1 = 'norm(b-A*x) / norm(b) = 7.7919e-009 < tol and flag1 = 0 (pcg succeeded without encountering troubles). The plot, on the right below, shows the convergence history of the method.

p c g j a c o b i 0 0 1 0 1 0

-2 -2 1 0 1 0 ) ) b b ( ( -4

-4 m m r 1 0 r 1 0 o o n n

) ) x x * * A A - - -6 b

b -6 ( ( 1 0 1 0 m m r r o o n n

-8 -8 1 0 1 0

-10 -10 1 0 1 0 0 2 0 0 4 0 0 6 0 0 8 0 0 1 0 0 0 1 2 0 0 1 4 0 0 1 6 0 0 1 8 0 0 0 1 0 2 0 3 0 4 0 5 0 6 0 i t e r a t i o n i t e r a t i o n The plot on the left shows the convergence history of Jacobi’s method an older, alternative iterative method. Note how much faster pcg is than Jacobi. Jacobi requires more than 1700 iterations and 0.15 seconds. We won’t study Jacobi’s method further, because it is relatively slow.

You can compare two methods on sets of matrices using the routine compare_methods (see item 65 on the course web page). For example, we can compare bicg with gmres on square matrices with maximum dimension between 1 and 1000 and an estimated condition number between 1 and 1,000,000. To do this, in a Matlab folder that contains compare_methods, its associated files and access to UFget, we assign method1 = ‘bicg’ method2 = ‘gmres’

We next edit one line of compare_methods. In Matlab type “edit compare_methods”, move to line number 94, change that line to selectUF_string = selectUF_string13; % or use another string above and save the file. After this, type “compare_methods”.

Here is the summary information for the run, which includes results for 168 matrices:

Run times, mean values (sample size = 168): sample1: 0.1654 sample2: 1.7187 t-test statistic for paired sample t test is: -7.418 As a rough guideline this statistics should be at least 2 in magnitude to conclude (with 5% chance or error) that two samples are different. An estimate of the probability that the two samples are the same and the difference in the run times is due to random chance: 5.7181e-12

Run times when flag is zero for both samples, mean values (sample size = 161): sample1: 0.13612 sample2: 1.7192 t-test statistic for paired sample t test is: -7.2566 As a rough guideline this statistics should be at least 2 in magnitude to conclude (with 5% chance or error) that two samples are different. An estimate of the probability that the two samples are the same and the difference in the run times is due to random chance: 1.6196e-11

Success rates (flag = 0): for method1 soln: 161 of 168 samples. for method2 soln: 168 of 168 samples An estimate of the probability that the two samples are the same and the difference in the success rate is due to random chance: (assuming appropriate assumptions): 0.015625

The summary information also includes pictures of the cumulative distribution functions (CDF’s) for (1) the run times over all runs, (2) the run times over runs where both methods converges, (3) the number of iterations over all runs and (4) the number of iterations when both methods converge. A CDF shows information about all the observations on a single plot. For example the plot in the upper left corner below, shows the probability, F(x), that the run time for bicg (in blue) and gmres (in green) is less than x, where x is the run time. From this graph we see that the median run time (the x where F(x) = 0.5) for bicg is about 0.04 seconds and the median run time for gmres is about 0.2 seconds. Note that the x axis has a logarithmic scale. In the top two graphs the CDF on the left will correspond to the method that is fastest. Bicg in this case. On the bottom two graphs the CDF to the left will show which method requires fewer iterations. Gmres in this case. C D F C D F w h e n b o t h m e t h o d s c o n v e r g e 1 1 x x

= 0 . 8 = 0 . 8 < < n n o o i i t t a a

v 0 . 6 v 0 . 6 r r e e s s b b o o

b 0 . 4 b 0 . 4 o o r r p p = = ) ) x x ( 0 . 2 t i m e m e t h o d 1 ( 0 . 2 t i m e m e t h o d 1 F F t i m e m e t h o d 2 t i m e m e t h o d 2 0 0 -3 -2 -1 0 1 2 -3 -2 -1 0 1 2 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 x x

C D F C D F w h e n b o t h m e t h o d s c o n v e r g e 1 1

x i t e r s m e t h o d 1 x i t e r s m e t h o d 1

= 0 . 8 = 0 . 8 < <

n i t e r s m e t h o d 2 n i t e r s m e t h o d 2 o o i i t t a a

v 0 . 6 v 0 . 6 r r e e s s b b o o

b 0 . 4 b 0 . 4 o o r r p p = = ) ) x x ( 0 . 2 ( 0 . 2 F F

0 0 0 1 2 3 4 0 1 2 3 4 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 x x

The conclusion from the above statistical tests, summary information and CDF plots is that, for this sample, bicg is clearly faster than gmres, but that gmres is more reliable in the sense that it converges for more matrices.

There are several tips that make use of compare_methods more convenient: 1. It is difficult to predict how long compare_methods will run. However, if one launches a run that seems to be taking forever, the run can be terminated by typing ctrl-C (hold down the ctrl key and type c). Once compare_methods stops running, type “compare_methods_stats” to get summary information for the run. 2. One can also define the optional parameter time_limit. The compare_methods program will stop running when it finishes a computation after more than time_limit seconds have passed. 3. Occasionally Matlab will encounter a large matrix and seem to get “stuck” on the computations of that matrix. While Matlab is computing inside a routine, ctrl-C may not work. If you do not want to wait, you can stop Matlab using the task manager in Windows (type ctrl alt del) or the kill command in Linux. All is not lost! You can still recover the work that has already been completed by restarting Matlab, moving to the proper folder, typing “load” and then typing “compare_methods_stats”. 4. The routines compare_methods and compare_methods_stats store summary information for the run in a file with a “.mat” extension. The filename contains the names of the methods used in the run and the selectUF values. If you load the file load (type or copy the filename here) % for example: load cm_bicg__gmres__selectUF(1,1000,3,1,1.e6).mat and run compare_methods_stats, the summary information for the run will be displayed. It is not necessary to rerun all the matrices again. 5. Sometimes Matlab’s graphics utilities produce error messages while running compare_methods_stats. Typing the Matlab command “close all” and rerunning compare_methods_stats seems to cure the problem. 6. To aid in displaying summary information using the stored “.mat” file mentioned in 4, the commands !dir /od *.mat % in Windows !ls –ltr *.mat % in Linux are useful. They show the existing “.mat” files ordered so that the most recent file is last. Since retyping these filenames can be tedious, it is convenient to cut and paste the file names from the list.

In addition to runs on individual matrices you should compare two methods over larger sets of matrices. This can be done with my compare_methods on set(s) of matrices of interest to you. Here is some criteria that you could use (this is from “help selectUF’):.

min_dim - select square matrices with min_dim <= dimension (default: 1). Here dimension means the number of rows or columns. max_dim - select square matrices whose dimension <= max_dim (default: inf, except when nargin >= 4, max_dim = the dimension of the largest (about 100,000 x 100,000) matrix with condest stored in UFcondest) sym_spd (default: 3) - 1: symmetric positive definite (Hermitian pos. def. for complex A) 2: symmetric (for real A) or Hermitian matrices (for complex A) 3: any square matrix 4: symmetric indefinite matrices (Hermitian, not positive definite for complex A) 5: non-symmetric square matrices (non-Hermitian for complex A) 6: non-SPD square matrices (not Hermitian pos. def., complex A) 11 - 16: only select real matrices, ones digit is same as above min_condest - select square matrices with min_condest <= estimated condition number, calculated by Matlab's condest, (default:1) max_condest - select square matrices whos estimated condition number, calculated by Matlab's condest, is <= max_condest (default:inf) If any of the above variables are <= 0 choose the default value. group - (a string) select only matrices in the matching UF group. Ignore if the string does not match any group. For UF groups see http://www.cise.ufl.edu/research/sparse/matrices/groups.html . kind - (a string) select only matrices in the matching UF kind (or application area). For description of UF kind names see http://www.cise.ufl.edu/research/sparse/matrices/kind.html . The kind string can be leading portion of complete UF kind name.

Also lines 72 to 92 of compare_methods have some sample choices for the parameters in selectUF.

For the project I would like you to download approximately half a dozen matrices and look at them individually, similar to the way we examined gr_30_30, except you won’t use Jacobi in your comparisons. Look at the following examples 1. Begin with gr_30_30, but do not select other SPD matrices since all the methods should work well for SPD matrices. 2. Next choose one or more examples where the method(s) you selected work(s) well in the sense that it converges relatively quickly. 3. Also try to find an example where the method(s) you chose does not work very well. 4. Include preconditioning on at least one example. 5. Try at least on matrix that is at at least 10,000 by 10,000.

In addition to the above individual matrices you should run compare_methods over various sets of matrices. Of course method1 and method2 should be chosen to illustrate the focus of your investigation. Do the following runs:

1. One run should have restricted condition numbers (<= 1000 or <= 1,000,000?). 2. One run should allow matrices with unrestricted condition numbers. 3. You should try at least one run with matrices up to 10,000 by 10,000 or indeed bigger. If the runs take too long, it is ok to stop the run using ctrl-C or by killing Matlab. You can still recover your results as discussed above. Your other runs can focus on smaller matrices (but up to at least 1000 by 1000), if you wish 4. One run should include preconditioning (even if that is not a focus in report).

Other than those guidelines you are free to choose your matrices however you wish (except don’t choose only SPD matrices). For example, if you have an interest, you could select matrices from certain applications (using kind) or matrix source (using group).

5. Write a report about the method(s) that you selected. Your report should begin by describing the method(s) that you selected and the question that you are looking at. Next describe some of the theoretical and practical properties of the iterative method that you selected. Some useful material will be discussed in class or you may want to use the Template book. Other useful references are Iterative Methods for Sparse Linear Systems by Yousef Saad (PWS, Boston, 1996), Numerical Linear Algebra by Lloyd Trefethen and David Bau (SIAM, Philadelphia, 1997) as well as Watkins text book and course notes. Make sure that you reference any material that you use from these sources (especially direct quotes). Following this I would suggest that you summarize the computer runs that you did. Tables are nice way to present you data. You should mention some of the properties of the matrices that you have selected. (Use UFweb to see information about a matrix. For example type UFweb(159) for information about gr_30_30. UFweb is part of the UFget utility.) Following this you should have a conclusion that summarizes what you discovered about the question that you examined. Also your report should try to describe the main conclusion (not necessarily details or proofs) of one journal article. This could be integrated into the report of done separately. The body of your report should be five to ten pages with at least a page summarizing or discussing the journal article. In an appendix to your report include output from your runs and any code that you wrote. Turn in a printed copy of your report and a copy of the article that you discussed. Also email me a file containing your report (for example if you used word to create your report the file might be called “KrylovReport.doc”).