ORNL/TM-2007/44 Leadership Computing Facility National Center for Computational Sciences Oak Ridge National Laboratory COMPUTATIONAL SCIENCE REQUIREMENTS FOR LEADERSHIP COMPUTING Douglas Kothe Ricky Kendall Date Published: July 2007 Prepared by OAK RIDGE NATIONAL LABORATORY P.O. Box 2008 Oak Ridge, Tennessee 37831-6254 managed by UT-Battelle, LLC for the U.S. DEPARTMENT OF ENERGY under contract DE-AC05-00OR22725 Computational Science Requirements Contents CONTENTS FIGURES .................................................................................................................................................. v TABLES ................................................................................................................................................... vii ABBREVIATIONS AND ACRONYMS ................................................................................................. ix EXECUTIVE SUMMARY ...................................................................................................................... xiii INTRODUCTION .................................................................................................................................... 1 SCIENCE DRIVERS ......................................................................................................................... 3 REFERENCES ................................................................................................................................... 5 SCIENCE QUALITY AND PRODUCTIVITY REQUIREMENTS ....................................................... 7 MODEL AND ALGORITHM REQUIREMENTS ........................................................................... 9 SOFTWARE REQUIREMENTS ............................................................................................................. 13 RUNTIME REQUIREMENTS................................................................................................................. 17 DATA ANALYSIS AND DATA MANAGEMENT REQUIREMENTS ............................................... 21 SUMMARY AND RECOMMENDATIONS ........................................................................................... 27 SCIENCE ........................................................................................................................................... 27 MODELS AND ALGORITHMS ....................................................................................................... 27 SOFTWARE ...................................................................................................................................... 28 RUNTIME FOOTPRINT ................................................................................................................... 29 DATA ANALYSIS AND DATA MANAGEMENT ......................................................................... 29 ACKNOWLEDGMENTS ........................................................................................................................ 31 APPENDIX A: GLOSSARY OF APPLICATION CODES .................................................................... 37 APPENDIX B: PROJECT ALLOCATIONS AND USAGE ON THE NCCS LCF SYSTEMS IN 2006 ............................................................................................................................ 43 APPENDIX C: APPLICATIONS REQUIREMENT COUNCIL ............................................................ 51 APPENDIX D: ASCAC CODE PROJECT QUESTIONNAIRE ............................................................. 61 APPENDIX E: SURVEY OF ACCEPTANCE AND EARLY ACCESS SCIENCE APPLICATIONS ................................................................................................................................ 67 iii Computational Science Requirements Figures FIGURES B.1. Job size distribution of allocated science applications on the NCCS LCF systems in the January to September 2006 time period ............................................................................. 45 B.2. Percentage of total NCCS LCF system utilization for each project receiving a 2006 allocation award in the January to September 2006 time period ........................................ 46 B.3. Month-by-month change in the job size distribution of the allocated science applications on the NCCS Cray XT3 (Jaguar) system in the January to September 2006 time period. ....................................................................................................... 46 B.4. Month-by-month change in the job size distribution of the allocated science applications on the NCCS Cray X1E (Phoenix) system in the January to September 2006 time period ........................................................................................................ 47 B.5. FY 06 utilization on the Cray XT3 (Jaguar) system by scientific discipline. .............................. 47 B.6. FY 06 utilization on the Cray X1E (Phoenix) system by scientific discipline. ........................... 48 E.1. AORSA on the Cray XT series Jaguar system compared with an IBM Power3 ......................... 69 E.2. Performance of the CAM 3.1 atmospheric model. ...................................................................... 71 E.3. Explicit Eulerian hydrodynamics. VH-1 weak scaling. ............................................................... 76 E.4. FLASH exhibited good scaling. ................................................................................................... 78 E.5. Good scaling was achieved on up to 5000 processors ................................................................. 80 E.6. GYRO scaling studies on various computers. ............................................................................. 84 E.7. LAMMPS parallelize efficiently for large problems. .................................................................. 87 E.8. MADNESS shows good overall scaling and scalability of the component algorithms. .............. 91 E.9. The codes for the Cray XTE will be optimized ........................................................................... 97 E.10. Single fuel assembly of a sodium-cooled, fast-spectrum nuclear reactor. ................................... 101 E.11. Benchmarks of the DFT code on various architectures. .............................................................. 106 E.12. PFLOTRAN has exhibited linear (strong) scaling on up to 2048 processors on Jaguar and good (though nonlinear) scaling to 4096 processors ............................................................. 110 E.13. Parallel Ocean Program (POP) 1.4.3: 0.1-degree benchmark, logarithmic axes. ........................ 113 v National Center for Computational Sciences E.14. Parallel Ocean Program (POP) 1.4.3: 0.1-degree benchmark, linear axes................................... 113 E.15. Strong scaling Qbox results on BlueGene/L for 1000 molybdenum atoms with 1 (non-zero) k-point ............................................................................................................. 116 E.16. S3D scaling demonstrated with a weak-scaling test. ................................................................... 121 vi Computational Science Requirements Tables TABLES 1. Science drivers projects receiving a 2006 allocation on LCF systems at the NCCS .................. 3 2. Science investigations and achievements possible on a 1-PF LC system for specific application codes in relevant science domains ............................................................................ 7 3. Increase in science simulation fidelity possible with a 1-PF LC system for specific application codes in various science domains .............................................................................. 8 4. Examples of how physical model attributes might change on a 1-PF LC system for specific application codes in various science domains ................................................................ 10 5. The “seven dwarfs” categorization of algorithms employed by specific application codes in various science domains ................................................................................................ 11 6. Functional software requirements (and options) for specific application codes in various science domains .......................................................................................................... 13 7. Typical features and associated suggested requirements for components of an LC system software stack .................................................................................................................. 14 8. Proposed solutions to specific requirements for components of the NCCS LCF software stack .............................................................................................................................. 15 9. Science application behavioral and algorithmic drivers for LC system attributes ...................... 18 10. Three-tier prioritization of 12 system attributes for relevant science domains ........................... 19 11. Typical development characteristics and runtime requirements of a single simulation (job) for selected application codes on the NCCS LCF systems circa June 2006 ...................... 20 12. Typical per-simulation I/O requirements for the largest data-producing application codes on the NCCS LCF systems ................................................................................................ 21 13. Prescription for estimating local storage bandwidth requirements for science applications on LC systems ........................................................................................................