TACC: A Powerhouse in the TeraGrid TACC makes major contributions to the nation’s largest experiment in building a capable grid as part of the scientific infrastructure

The TeraGrid has been in the news again as a new software, and instrument and visualization resources. infusion of funding fuels an ambitious program to The other resource providers are the University of broaden and deepen the use of grid resources in Chicago/Argonne National Laboratory; Indiana scientific research. Begun four years ago with funding University; the National Center for Supercomputing from the National Science Foundation (NSF), the Ter- Applications; Oak Ridge National Laboratory; Pitts- aGrid is a visionary project, a partnership of people burgh Supercomputing Center; ; and resources that provides comprehensive cyberin- and the San Diego Supercomputer Center. Currently frastructure to enable discovery in U.S. science and about 1000 investigators are using the TeraGrid; in engineering research. five years, Catlett hopes that figure will rise by a fac- tor of 5 to 10. The fresh round of funding from NSF brings $10 mil- lion to The University of Texas at Austin for the Texas TACC Contributions to the TeraGrid Advanced Computing Center to enhance its partici- pation in the TeraGrid. “TACC is proud to represent “TACC’s role in the TeraGrid is very broad,” says the university in working with some of the nation’s Boisseau. “We bring into the mix a blend of commod- leading institutions to develop, operate, and evolve ity and leading-edge machinery,knowledge resources, the TeraGrid,” says Dr. Jay Boisseau, TACC director software technologies, and expertise that is unique in and the principal investigator for the university on the world of supercomputing.” the project. High-Performance Computing: “TeraGrid users can Through high-performance network connections, take advantage of our Cray/Dell Linux cluster, called grid-computing software, and coordinated services, Lonestar,” says Chris Hempel, TACC associate direc- the TeraGrid integrates a distributed set of the high- tor for resources and services and site lead for the est capability computational, data management, and TeraGrid. “Lonestar is a massively parallel machine visualization resources to make U.S. researchers with 1024 compute processors,” he says. The peak more productive. With Science Gateway collabora- performance of the machine is 6.2 teraflops (trillions tions, links to large-scale instruments like telescopes of floating-point operations per second), and the total and particle accelerators, and an array of education memory capacity of the machine is 1 terabyte (1 TB, and outreach programs, the TeraGrid also broadens one trillion bytes). Local storage and global work the scientific communities that can make good use space are 18 TB and 3 TB, respectively. An IBRIX of its resources. TeraGrid director Charlie Catlett of Global File System is available via high-speed access Argonne National Laboratory and the University of from every node, with a total capacity of 6.2 TB. A Chicago says that the project “will be a persistent Myrinet-2000 switch fabric, employing PCI-X inter- infrastructure whose capabilities deepen and evolve. faces, interconnects all compute nodes with a point- As scientists come to depend upon it, the nature of to-point, unidirectional bandwidth of 2 gigabits per their research and the questions they can ask will also second. broaden and deepen.” Researchers at The University of Texas at Austin and TACC is one of eight U.S. supercomputing centers around the country, via TeraGrid, have been using that house the TeraGrid’s computational, storage, Lonestar for large-scale calculations in physics and

Texas Advanced Computing Center | Feature Story For more info, contact: Faith Singer-Villalobos, Public Relations, [email protected], 512.232.5771 Page 1 of 3 geophysics (including final gravity-field calculations surges, and tsunamis, via the TeraGrid Flood Model- for one of NASA’s most exacting earth science mis- ing Science Gateway. Like the other TeraGrid science sions, being conducted by the university’s Center for gateways, the Flood Modeling system will unite the Space Research), biochemistry (including massive data and tools used by an entire community of sci- projects in RNA sequence/structure relations and entists and emergency managers. Data used through phylogenetics), chemistry (including intensive quan- the gateway will include LiDAR land elevations tum dynamics calculations for chemical reactions), and NEXRAD precipitation data, as well as satellite engineering (including first-principles calculations data on storms and hurricanes. Models developed of turbulent fluid behavior), and other disciplines. by CRWR will enable flood prediction in real time “As one of the many powerful high-end compute for wide areas subject to sudden flooding. Tomislav resources available via the TeraGrid,” Hempel says, Urban, interim manager of TACC’s Data and Informa- “Lonestar has been finely tuned to assist researchers tion Services group, is the liaison for the Flood Model- in getting optimum turnaround and efficiency.” ing Science Gateway.

Remote and Collaborative Visualization Capability: Software Technology Development: TACC’s exper- “This machine was designed with remote visualiza- tise in the development of user-friendly interfaces for tion capability in mind,” Boisseau says of Maverick, high-end resource monitoring and interaction is being a Sun E25K platform available to university and put to work in the development of the TeraGrid User TeraGrid researchers. Maverick has 64 dual-core 1.05 Portal. The portal is a tool for users that will offer GHz processors and 512 gigabytes of shared mem- grid information and interactivity through a standard ory. A special feature of Maverick is the additional web browser. The initial goals for the TeraGrid portal presence of 16 graphics processor cards exactly like are to simplify account creation for users by hiding the systems that power fast video games. “We give and automating grid credential creation; to provide large-memory and interactive jobs priority on Mav- a web interface for viewing allocation usage on all erick,” says Gregory S. Johnson of TACC, manager of TeraGrid resources; and to enable access to dynamic TACC’s Visualization and Data Analysis group. Mav- grid information about the resources, including load, erick can also be used from remote locations to drive status, and job queue tables. Other features, such as high-end display equipment, especially important interfaces for data management and job submission, for collaborative research. “This makes it one of the will be added incrementally throughout the life of most versatile platforms on the TeraGrid for scientific the project. Software developer Eric Roberts of TACC visualization,” Johnson says. leads the TeraGrid portals working group and also participates in and contributes to the science gateways Database Hosting: TACC is making data collections effort. from four major university research centers avail- able to users of the TeraGrid. The data come from the A further software technology development contrib- Bureau of Economic Geology, Center for Research in uted to the TeraGrid is GridShell, a facility for creat- Water Resources, Center for Space Research, and the ing a common environment for managing scientific High Resolution X-ray Computed Tomography Facil- jobs on all the different resources available on the ity. TeraGrid users can access high-resolution digital TeraGrid. GridShell was developed by Dr. Edward terrain data, worldwide hydrological data, global Walker of TACC in collaboration with other TACC gravity data, and computed tomography of living researchers and scientists at the University of Wis- and fossil specimens. These data sets are invaluable consin, Caltech, and the Pittsburgh Supercomputing research tools for scientists in environmental, geologi- Center. GridShell enables transparent access to grid cal, climate, and biological research programs. resources via software agents, and it has been used very successfully in the TACC UT Grid project to Science Gateway : Under the joint direction of Dr. schedule computational runs. Gordon L. Wells of the university’s Center for Space Research and Dr. David Maidment of the Center for TeraGrid: A Vision and a Promise Research in Water Resources (CRWR), geologists and hydrologists will be able to develop facilities to “The contributions TACC is making to the TeraGrid, anticipate and model the effects of flash floods, storm listed above, are central results of the research pro-

Texas Advanced Computing Center | Feature Story For more info, contact: Faith Singer-Villalobos, Public Relations, [email protected], 512.232.5771 Page 2 of 3 grams in which TACC has been active for several of what is possible, and TACC is playing a leading years. A pure facility without science-driven research role in this transformation of scientific capabilities.” and technology would not have been able to tackle these tasks, and TACC has taken a visionary path ______here in service to the sciences,” says Dr. Kelly Gaither, TACC associate director for research and develop- Try Out the TeraGrid ment. “We expect to expand our research and devel- opment programs so we can continue meeting the Researchers may request access to the TeraGrid by challenge of building a national infrastructure for the submitting proposals for a Development Allocation computational sciences.” (DAC) start-up account—up to 30,000 service units of “roaming” allocation (similar to cell phone roaming) Ultimately, Boisseau says, “the TeraGrid is defining to use on any TeraGrid resource. a new vision of the potential offered by integrating 1. Go to: www..org. Click on “New Alloca- computing resources, visualization systems, data col- tions” and “More on DAC Accounts.” lections, and instruments into a ‘’ 2. Submit an abstract, as instructed, describing your that enhances the capabilities of researchers. Scientists science, applications you are using, and which Ter- from The University of Texas at Austin and elsewhere aGrid resources you want to explore. around the country will tackle new challenges, scaling 3. If you have questions, please send an email to: up applications in ways that would have been impos- [email protected]. Or, please call Chris Hempel, Ter- sible before now. The TeraGrid serves as an exemplar aGrid Site Lead for TACC, at: 512-475-9479.

Texas Advanced Computing Center | Feature Story For more info, contact: Faith Singer-Villalobos, Public Relations, [email protected], 512.232.5771 Page 3 of 3