SAND97-2551 Distribution Unlimited Release Category UC-705 Printed October 1997 The Development and Performance of a Message-Passing Version of the PAGOSA Shock-Wave Physics Code David R. Gardner Parallel Computational Sciences Department Courtenay T. Vaughan Parallel Computing Science Department P.O. Box 5800 Sandia National Laboratories Albuquerque, NM 87185-111 Prepared for The Computational Mechanics and Material Modeling Technology Coordination Group of the Joint DoD/DOE Munitions Technology Development Program Abstract A message-passing version of the PAGOSA shock-wave physics code has been developed at Sandia National Laboratories for multiple-instruction, multiple-data stream (MIMD) computers. PAGOSA is an explicit, Eulerian code for modeling the three-dimensional, high-speed hydrodynamic flow of fluids and the dynamic deformation of solids under high rates of strain. It was originally developed at Los Alamos National Laboratory for the sin- gle-instruction, multiple-data (SIMD) Connection Machine parallel com- puters. The performance of Sandia’s message-passing version of PAGOSA has been measured on two MIMD machines, the nCUBE 2 and the Intel Paragon XP/S. No special efforts were made to optimize the code for ei- ther machine. The measured scaled speedup (computational time for a sin- gle computational node divided by the computational time per node for fixed computational load) and grind time (computational time per cell per time step) show that the MIMD PAGOSA code scales linearly with the number of computational nodes used on a variety of problems, including the simulation of shaped-charge jets perforating an oil well casing. Scaled parallel efficiencies for MIMD PAGOSA are greater than 0.70 when the available memory per node is filled (or nearly filled) on hundreds to a thou- sand or more computational nodes on these two machines, indicating that the code scales very well. Thus good parallel performance can be achieved for complex and realistic applications when they are first implemented on MIMD parallel computers. 1 Acknowledgments The PAGOSA code was developed at Los Alamos National Laboratory under the direction of Dr. J. W. Hopson. We thank Dr. Hopson for making the data-parallel version of PAGOSA available to us, and we thank Dr. D. B. Kothe of Los Alamos National Laborato- ry for providing technical advice concerning its structure. We thank the Advanced Com- puting Laboratory of Los Alamos National Laboratory, Los Alamos, NM 87545, for providing computing resources necessary to complete this work. This work was supported under the Joint DoD/DOE Munitions Technology Development Program, and sponsored by the Office of Munitions of the Secretary of Defense. We thank T. Mack Stallcup of Intel Corporation for providing technical information about the Paragon computer. We thank Martin W. Lewitt, formerly of nCUBE Corporation, for technical information about the nCUBE 2 computer. UNIX® is a trademark of AT&T. CM-2, and CM-5 are trademarks of Thinking Machines Corporation. iPSC®, i860, and Paragon™ are trademarks of Intel Corporation. This work was performed at Sandia National Laboratories supported by the U.S. Depart- ment of Energy under contract number DE-AC04-94AL85000. We thank the Massively Parallel Computing Research Laboratory at Sandia for providing computing resources necessary to complete this work. 2 Table of Contents Abstract................................................................................................................................1 Acknowledgments................................................................................................................2 Introduction..........................................................................................................................9 Issues in Parallel Computing .............................................................................................11 Parallel Code Performance Measurements ........................................................................13 Development of the MIMD PAGOSA Code.......................................................................18 Features of PAGOSA 5.5 ........................................................................................18 Development of MIMD PAGOSA 5.5 from SIMD PAGOSA 5.5...........................19 Features of MIMD PAGOSA 5.5............................................................................22 The Test Simulations .........................................................................................................22 The Finned Projectile Simulations.........................................................................22 The Explosive Welding Simulation.......................................................................23 The Oil-Well Perforation Simulation.....................................................................24 The Test Conditions...........................................................................................................27 The Performance of MIMD PAGOSA ..............................................................................28 Message-Passing Performance on the nCUBE 2 ...................................................29 Message-Passing Performance on the Intel Paragon .........................................36 Performance of MIMD PAGOSA on the nCUBE 2 and the Intel Paragon Message-Passing Computers .............................................................................................46 Summary and Conclusions ................................................................................................52 References..........................................................................................................................54 Appendices.........................................................................................................................56 A MIMD PAGOSA 5.5 Input Guide.....................................................................56 B Test Problem Input Sets...................................................................................90 B1 Input Set for the Finned Projectile Problem with the Hydrodynamic Constitutive Model, fp1 .............................................................................90 3 B2 Input Set for the Finned Projectile Problem with the Elastic, Perfectly Plastic Constitutive Model, fp2 .................................................................94 B3 Input Set for the Explosive Welding Problem, ew ....................................98 B4 Input Set for the Oil-Well Perforation Problem, owp..............................102 4 List of Figures 1 Illustration of the speedup surface, the fixed-size speedup curve, and the scaled speedup curve.................................................................................................15 2 The fixed-size speedup and scaled speedup curves projected on the P-S plane ...................................................................................................................17 3 Illustration of the decomposition of the global computational domain into subdomains, in two spatial dimensions.....................................................................20 4 Program fragments illustrating the translation from CM Fortran to Fortran 77.......20 5 Illustration of shift_left local inter-node communication between two nodes for two-dimensional subdomains for a message-passing code.......................................21 6 Simulation of a finned tungsten projectile obliquely impacting a stainless steel plate (hydrodynamic constitutive model) .................................................................23 7 Simulation of a finned tungsten projectile obliquely impacting a stainless steel plate (elastic, perfectly plastic constitutive model)...................................................23 8 Simulation of the explosive welding of a copper tube to a stainless steel plate. ......24 9 Simulation of the perforation of a steel oil-well casing by a shaped charge jet. ......26 10 Scaled speedup and grind time for the fp1 simulation on the nCUBE 2. .................33 11 Scaled speedup and grind time for the fp2 simulation on the nCUBE 2. .................33 12 Scaled speedup and grind time for the ew simulation on the nCUBE 2...................34 13 Scaled speedup and grind time for the owp simulation on the nCUBE 2.................34 14 Scaled parallel efficiency as a function of the number of nodes on the nCUBE 2...35 15 Scaled speedup and grind time for the fp1 simulation on the Paragon.................43 16 Scaled speedup and grind time for the fp2 simulation on the Paragon.................43 17 Scaled speedup and grind time for the ew simulation on the Paragon..................44 18 Scaled speedup and grind time for the owp simulation on the Paragon. ..............44 19 Scaled parallel efficiency on the Paragon .............................................................45 20 Effect of subdomain size on the scaled parallel efficiency on the Paragon for the ew and owp simulations......................................................................................45 5 Intentionally Blank Page 6 List of Tables 1 Compiler Versions and Options for Compiling MIMD PAGOSA..............................27 2 Grind Time Repeatability ............................................................................................27 3 Performance of MIMD PAGOSA on the nCUBE 2 for fp1..........................................31 4 Performance of MIMD PAGOSA on the nCUBE 2 for fp2..........................................31