ACCELERATING SCIENTIFIC DISCOVERY RESCUED HISTORY Five years ago, XSEDE was looking forward to ac- 2 Digital humanities project looks into lives complishing a great deal, and it has. XSEDE has of historical African American women WELCOME supported over 17,000 publications, collected dur- to the annual highlights publication of the Extreme ing the program period. Analysis of these publica- Science and Engineering Discovery Environment tions has indicated that publications supported by HAVE = HAD + CHANGE XSEDE are, on average, cited twice as often as oth- 4 XSEDE makes efforts to help teach America’s (XSEDE). As the fifth issue of this publication, it is exciting and gratifying to see the stories that have er publications in the same journal, indicting the teachers computational science been selected and how they have continued to greater impact these publications are having in the represent the larger collection of world class re- various domains of research supported by XSEDE. LEARNING MORE ABOUT ECSS search and education activities supported by Thus far (July 2011-April 2016), over $2.13 billion in 6 XSEDE ECSS is a helpful resource XSEDE. There was a tremendous portfolio of ar- funded research projects have had allocations on to novice supercomputer users ticles from which to choose. This issue represents XSEDE resources to support those research activi- John Towns the work of more than 8,000 researchers par- ties. The growth in size of the community is com- XSEDE PI and ticipating in more than 2,700 projects, producing plemented by a greater diversity of domains from CIREN TO THE RESCUE Project Director 8 more than 3,500 publications during the past year. which those researchers come. Researchers from Car crash simulations produce better computational finance, machine learning, textual data than crash test dummies analysis, phylogeny, genomics, archaeology, and PARTNER INSTITUTIONS XSEDE has matured significantly over the five years it has been providing resources and servic- digital humanities have begun to make use of the Cornell University Center national cyberinfrastructure ecosystem with the for Advanced Computing TINY ZAPS, BIG RESULTS es to the community. The organization has come 10 support of XSEDE. Indiana University Nanoscale laser bursts can change a great distance from the early days of a freshly the structure of certain materials forged team of partners without significant experi- Jülich Supercomputing Centre Looking forward, XSEDE has submitted a proposal Michigan State University ence in such close collaboration to an organiza- tion well aligned with its goals and committed to to the National Science Foundation (NSF) to con- National Center for Atmospheric Research CIPRES OPENS GATEWAY attaining a shared vision: a world of digitally-en- tinue support of XSEDE for an additional five years. National Center for Supercomputing 12 Science Gateways allows researchers That proposal suggests a number of changes in- Applications - University of Illinois at abled scholars, researchers, and engineers par- Urbana-Champaign new insight into new tree of life ticipating in multidisciplinary collaborations while duced by the lessons we have learned over the National Institute for Computational seamlessly accessing computing resources and past five years in deploying and supporting an Sciences - University of Tennessee THE SCIENCE OF JET NOISE sharing data to tackle society’s grand challenges. ever-advancing infrastructure within the broader Knoxville/Oak Ridge National Laboratory cyberinfrastructure ecosystem. While currently 14 Making jet engines quieter is a science More pointedly, research now requires more than Ohio Supercomputer Center - under consideration by the NSF, it is hoped that The Ohio State University problem rooted in design just supercomputers, and XSEDE represents a by the time this publication is released, some very Pittsburgh Supercomputing Center - step toward a more comprehensive and cohesive Carnegie Mellon University/ set of advanced digital services through our mis- positive news can be shared with the community University of Pittsburgh INVESTIGATING AAAs sion: to substantially enhance the productivity of about the future availability of the resources and Purdue University 16 Abdominal aneurysms modeled in order a growing community of scholars, researchers, services XSEDE has provided to support the na- Rice University to improve medical decisions and engineers through access to advanced digital tional research enterprise. San Diego Supercomputer Center - services that support open research; and to co- University of California, San Diego XSEDE looks forward to an opportunity to con- ordinate and add significant value to the leading Shodor Education Foundation THE TURTLE TREE OF LIFE tinue to deliver value and to accelerate scientific 18 cyberinfrastructure resources funded by the NSF Southeastern Universities Research Turtles have so many varied characteristics, discovery! Association studying lineage is difficult and other agencies. Texas Advanced Computing Center - The University of Texas at Austin XSEDE HELPS CONFIRM University of California, Berkeley 20 University of Chicago LIGO DISCOVERY University of Southern California XSEDE provided resources to help find first University of Virginia gravitational waves detected by LIGO Each story in this @XSEDEscience Highlights book comes w/ its own tweet! Enjoy the stories and share with your friends on social media! RESCUED HISTORY

“We’re hoping, in the next stage, to ramp bers mean. They have a whole theory We say, “History is written by the victors.” But it’s probably up and check these topics against the behind how you go about interpreting larger corpus of works,” Mendenhall things as they relate to the larger soci- more true to say it’s written by the people who have the adds. “Do the ‘recovery’ part.” ety—that’s really interesting to me.” opportunity to write. Van Moer’s data visualization work is Another goal will be to create a compu- building ways of displaying results in tational toolbox that researchers in many One example of this phenomenon is riodicals,” says Mendenhall. “The num- cause the tools we’re using need huge a way that helps humans make more fields can use to search various texts for the study of Black women, their lives ber is greater than what you would nor- amounts of shared memory,” a strength intuitive sense of them. A “tree map” topics of interest—and how they relate. and their experiences. Many earlier mally be able to look at during an entire of that system, says Mark Van Moer, an displays key words in boxes that cor- documents don’t mention them direct- career.” XSEDE staff member at the National respond to each word’s frequency. A “We’re generally interested in Black ly, though these works may offer clues. Center for Supercomputing Applications “network graph” charts how often key women and their life experience,” Men- Those that do mention Black women Powering Up at Illinois who worked as the team’s vi- words appear close to each other, offer- denhall says. “But we also see this as a often are historically obscure. Until re- sualization specialist. ing insight based on context. Yet another tool that social scientists and people in cently researchers had no good way of To search tens or even hundreds of visualization technique plots key terms the humanities can use to study many recovering this “lost history.” thousands of books, articles and letters, The researchers analyzed 20,000 docu- in histographs, allowing users to track a topics.” the researchers needed considerably ments known to contain information given topic’s prominence over time. Ruby Mendenhall, an associate profes- more computing power than available about Black women in the HathiTrust sor of Sociology, African American Stud- on a typical campus “cluster” of com- and JSTOR databases to create a com- Making Sense of the Numbers ies, Urban and Regional Planning and modity computers. They consulted putational model. They’re now using this Social Work at the University of Illinois with colleagues on campus who were model on Greenfield to study the entire An initial finding confirmed that docu- at Urbana-Champaign, is leading a col- XSEDE members, identifying the now- 800,000 documents in both databases. ments that referenced the post-World laboration of social scientists and digital retired XSEDE resource Blacklight at -War-I Black Women’s Club movement researchers that hopes to harness the the Pittsburgh Supercomputing Center Making Sense of Words via also mentioned the New Negro move- power of high-performance comput- (PSC) as a good fit for their project. Numbers, Graphics ment. The Illinois researchers hope to ing to find and understand the historical begin answering questions about how experiences of Black women. They’re With help from PSC’s Sergiu Sanielevici, To make sense of the numbers, the in- these movements interacted in their using XSEDE resources to analyze two head of XSEDE’s New and Innovative vestigators turned to two sets of com- current work on XSEDE systems. massive databases of written works Projects group in the Extended Col- putational techniques: topic modeling from the 18th- through 20th centuries, laborative Support Service (ECSS), they and data visualization. “The beauty of computation and Big along the way creating a common tool- adapted their work to Blacklight. They Data lies in how it complements the box that can help other digital humani- have since moved the project to PSC’s Topic modeling looks at how often cer- traditional close reading,” says Nicole ties projects. interim Greenfield system, a precursor to tain key words associate with each oth- Brown, a postdoctoral fellow in Men- the new XSEDE resource at PSC, Bridg- er. For example, a book that contains the denhall’s group who is interpreting the “With a Big Data approach we get a es. word “negro”—at the time considered computational results in light of Black chance to make use of hundreds of the most respectful term to describe feminist theory. “The two methods com- thousands of texts—journals, books, pe- “We chose Blacklight specifically -be Black men and women—the word “vote” plement each other to give you a full Above: Sculptor Edmondia Lewis and the word “women” might offer clues picture of what’s going on.” (1844-1907) was the first woman of African- and Native-American descent about Black women’s participation in to achieve notoriety in the fine arts the women’s suffrage movement. Mike “Working with the social science and world. She spent most of her career in Researchers use XSEDE resources to recover Black, then at Illinois and currently at the humanities people has been a real eye Rome. Credit: Henry Rocher – National opener,” adds Van Moer. “Previously, I Portrait Gallery, Smithsonian Institution, Black women’s “lost history” by analyzing University of Massachusetts, headed Public Domain this topic modeling project for the team. worked with physical scientists. Human- th th literary works from 18 to 20 centuries. ities and social science researchers are Left: Harriet Tubman, famous as an worried about more than what the num- abolitionist, Underground Railroad leader and women’s suffrage pioneer. Credit: H. B. Lindsley – National Portrait Gallery, Smithsonian Institution, Public Domain (PD-1875) 2 3 HAVE = HAD + CHANGE Everybody has a favorite teacher from high school or professor from college, and these folks teach eager minds about every subject imaginable. But who prepares these instructors to excite, energize, and educate the future generations of American scientists, technologists, engineers, mathematicians, and scholars in all fields of study? XSEDE’s education team, which in- “We welcome inquiries from educators Below is a sampling of the range of edu- cludes Steve Gordon and Kate Cahill in all fields of study to discuss opportu- cation workshops hosted by XSEDE: from the Ohio Supercomputer Center, nities for extending this impact to class- • Computational Biology for and Bob Panoff and Aaron Weeden from rooms across the nation,” said Cahill. Biology Educators the Shodor Education Foundation, are leading this charge across the country. Panoff added, “Our workshops are • Computational Chemistry for faculty members indicate that they have for more than 50 undergraduate engi- tions and more. My students create Chemistry Educators They work side-by-side with faculty reaching a national audience of faculty used materials from the workshops in neering students over the past couple spreadsheets to show what can happen across the country to help them incor- and K-12 teachers, who in turn teach • Computing MATTERS: Inquiry-Based their classes. of years.” when variables change in the equations porate computational and data-enabled the future practitioners, innovators and Science and Mathematics Enhanced we use and the use of graphs of this technologies, resources, and methods leaders within a diverse mix of institu- by Computational Thinking “I incorporated the information about Eric Frost at San Diego State University data to visualize the changes as they into the curriculum in all fields of study. tions including two- and four-year col- • Computational Physics for Use in the Vensim that I learned at the seminar into even uses XSEDE materials for a gradu- vary. And we also have a Lego robotics They conduct this effort through a col- leges and universities, minority serving Undergraduate Modern Physics Course my Modeling and Simulation (CSCI 323) ate class on Homeland Security, which class in 7th and 8th grade and this helps lection of campus visits, strategic plan- institutions, and K-12 schools.” • Intermediate Parallel Computing: course that I subsequently taught,” said shows XSEDE’s value outside of physics with computational thinking as well, at a ning sessions, tutorials, workshops and Introduction to Parallel Computing Don Allison, a faculty member at SUNY- or chemistry courses. “Materials were younger age.” online courses. Even though computation has become Oneonta in the computer science de- used as people try to understand how • Introduction to Computational Thinking the third pathway to discovery—along partment. “I also purchased one of the modeling, simulation, and statistics are In summary, Gordon concluded, “The “The mission of the XSEDE education with theory and experimentation—many • Kinder, Gentler Supercomputing: Using recommended books on computational used in regard to public safety and se- key is that faculty have used the ex- program is to prepare a diverse commu- faculty have not integrated the use of HPC Resources and Visualization Tools science to serve as a starting point to curity. Where lives are at stake for mak- amples from the workshops to make nity of researchers, scholars, educators, computation into their classes. “Our • LittleFe Curriculum Module Buildout develop a future course in the topic for ing real decisions and computational substantial changes in their courses and and practitioners in the use of data anal- workshops provide explicit examples of our institution.” sciences can assist, XSEDE has been curriculum to include computational ysis and management, modeling, simu- computational models and tools that “What we’ve seen is educators want— wonderfully helpful in being able to be science. Those changes often take sev- lation, and visualization techniques,” can easily be assimilated into a variety and their students need—this type of ma- Stephen Angel, Washburn University: “I a bridge from mathematical complexity eral years beyond the workshops to get said Gordon. of classroom situations,” said Panoff. terial for a changing and evolving work- attended an introduction to computa- to practical decision-making and confi- put into place. However, they clearly force. Computational science, including tional chemistry course many years ago dence in approximating results.” have made significant impacts, in some parallel computing and basic modeling and developed a freshman lab exercise cases resulting in entire new courses, knowledge, is essential for the future of upon return. It has been well-received Leonard Freidhof at Eastland Vocational for example on modeling or compu- .@XSEDEscience helps educate nation’s all fields of scholarly research and appli- by students and faculty.” School in northwest Illinois said, “I have tational chemistry, while in others us- cations in industry,” said Gordon. been using information I picked up in ing computational science to enhance educators: from junior high to graduate classes, Melanie Correll, University of Florida: “I my high school chemistry and physics mathematics education for a wide range The response from the educators who use materials for my ABE 4662 –Quanti- classes to show virtual events in kinet- of students.” in STEM subjects and even national security. have attended these courses has been fication of Biological Processes for proj- ics, thermodynamics, equilibrium and overwhelmingly positive, and numerous ects and homework assignment ideas velocity, acceleration, pseudoderiva-

4 5 parallelization of key loops, and restruc- the same time that they are planning Within ECSS Communities: turing of the code to greatly reduce the their computational objectives. ECSS • Extended Support for Community Codes number of linear algebra operations,” can be requested with small startup al- (ESCC), led by John Cazes of Texas Ad- said Robert Sinkovits, who leads SDSC’s locations as well as at any point in an vanced Computing Center (TACC): ESCC’s scientific applications efforts. “The end allocation lifecycle as needs develop. focus is on deploying, hardening, and result was that calculations which previ- ECSS collaborations can last from a optimizing software systems necessary ously took nearly twelve hours to finish number of weeks up to one year, and for extensive research communities to could be completed in just a few min- renewals are possible. create new knowledge using XSEDE re- utes, with a total speedup of 167x rela- sources and related technologies. ESCC staff also evaluate software that might tive to the original code.” At the top level, ECSS is divided into be appropriate for installation on XSEDE two main areas: ECSS Projects, led by resources. Today, the expertise of ECSS staff mem- Roskies, where work is done primar- bers extends to science gateway de- ily with individual research groups, and • Extended Support for Science Gateways velopment and workflow systems, the ECSS Communities, led by Wilkins- (ESSGW), led by Marlon Pierce of Indiana use of R, Java, and MATLAB, database Diehr, where work benefits larger sci- University: This group’s primary mission LEARNING MORE ABOUT ECSS is to provide support to existing and new optimization, cloud computing, data entific communities. The areas work scientific communities that would like mining, and more. Moreover, while ECSS together closely, with ECSS staff often to use XSEDE resources through Science One of the most robust services offered by XSEDE is its was established to provide research- shifting back and forth between the two Gateways. ESSGW does not directly ers the support and expertise needed areas. Sub groups include: develop and support gateways, but to conduct computationally-based sci- supports science gateway operators via Extended Collaborative Support Service, or ECSS. This service Within ECSS Projects: ence, the service also gives them an integration with XSEDE resources and provides expert staff to assist research teams for either a short opportunity to learn new computational • Extended Support for Research Teams choice of technologies. (ESRT), led by Lonnie Crosby of National techniques. • Extended Support for Training, Education Institute for Computational Sciences and Outreach (ESTEO), led by Jay Alam- or an extended period using large-scale computing resources (NICS): This is a collaborative effort “Most ECSS staff members have ad- eda of NCSA: ESTEO and Training staff between an XSEDE user group and one vanced degrees in scientific fields as survey existing online training materials or more ECSS staff members, with the in order to advance the team’s work. delivered by all service providers and well as technology expertise, and that goal of enhancing the research group’s identify new areas where materials might Any U.S. researcher with an XSEDE al- scale resources, the group also includes One recent example of ECSS’s assis- means they speak the language of capability to transform knowledge using benefit the community. Online training the researchers,” according to PSC’s XSEDE-allocated resources and related location can request ECSS support. Re- staff from other institutions who bring tance involved a project led by Ramon is increasingly popular with the user Roskies. “While our domain expertise technologies. These projects currently searchers can receive help in areas such valuable expertise in areas such as digi- Huerta, a research scientist with UC San community when both time and travel comprise about 60% of all ECSS projects. as code optimization, optimizing com- tal humanities, workflows, and science Diego’s BioCircuits Laboratory, to place includes areas traditionally associated budgets are limited. plex workflows, porting to new system gateways. chemical sensors – known as “e-noses” with high-end computing such as as- • Novel and Innovative Projects (NIP), led More information about ECSS can be architectures, and using of accelerators – in the homes of elderly participants in trophysics, computational chemistry, by Sergiu Sanielevici of PSC: This group found at https://www.xsede.org/ecss. such as GPUs for detailed visualizations, “We find this expertise increasingly nec- an effort to continuously and non-intru- climate modeling, and molecular dy- seeks out projects by researchers in com- all at no cost to their grants. essary as the domain areas, the pro- sively monitor their living environments. namics, we are also prepared to assist munities that don’t typically use high- gramming methods, and the advanced Huerta’s lab developed software that researchers in emerging disciplines – performance computing and advanced cyberinfrastructure – areas that may Comprising roughly one-third of cyberinfrastructure resources allocated could detect anomalous behavior that areas that haven’t traditionally used ad- include bioinformatics, economics, the XSEDE’s annual budget, the ECSS staff by XSEDE continue to diversify,” said vanced computation such as genomics, would suggest a change in a resident’s social sciences, and arts and humanities. covers a wide variety of expertise, and is Wilkins-Diehr. health status. This software was paired machine learning, or predictive analyt- sized to support about 50 ECSS projects with sensor data and was ported to SD- ics.” and 50 training events per year. Nancy The ECSS program, which evolved from SC’s Gordon supercomputer. Wilkins-Diehr of San Diego Supercom- the Advanced Support for TeraGrid Ap- How ECSS Works puter Center (SDSC) and Ralph Roskies plications (ASTA) in the NSF program “We were able to improve the perfor- .@XSEDEscience Extended Collab. Support of Pittsburgh Supercomputing Center that preceded XSEDE, historically fo- mance of the code through a combi- ECSS assistance is requested through Service helps researchers across diff. domains (PSC), both co-PIs of XSEDE, lead ECSS. cused on improving code performance nation of smart compiler choices and the XSEDE allocations process, the While most ECSS staff members come on supercomputers, including perfor- options, replacement of calls to the same process used to request CPU, use advanced computational tools from research sites that have also re- mance analysis, optimization, debug- generic math library with calls to Intel’s data, and other resources. Researchers ceived NSF grants to operate large- ging, and on the use of specific comput- highly tuned MKL (Math Kernel Library), are encouraged to identify areas where to advance their work. ing architectures. they might need specific assistance at

6 7 CIREN TO THE RESCUE More than 33,000 Americans die in motor vehicle crashes annually, according to the Centers for Disease Control and Prevention. Modern restraint systems save lives, but some deaths and injuries remain—and restraints themselves can cause some injuries. “Crash-test dummies” help engineers us much more, predicting injuries in or- dict fractures with much more detail design safer cars, but provide only lim- gans that aren’t in that dummy, such as than do dummies.” For example, she ited information about forces the body lung contusions.” adds, the model helps identify which experiences during impact. Computer specific areas of the rib cage are expe- models of vehicle crashes, on the other Turning on the CIREN riencing forces that can break bones, hand, provide more sophisticated infor- instead of only calculating the overall This simulated real-world crash uses a force on the chest. human-body model. Credit: Wake Forest mation on how to improve restraints and With Joel Stitzel and graduate students University Center for Injury Biomechanics other safety systems. The models also and staff from the Center for Injury Bio- help researchers simulate the effects mechanics, Weaver used as their start- Stepping up to XSEDE power of thousands of variables that would be ing point a database of real-world vehi- impossible to assess in physical crash cle crashes from the National Highway Initially, the team used the DEAC Cluster tests. Traffic Safety Administration-sponsored at Wake Forest University for their work. Crash Injury Research and Engineering The simulations used virtual versions of “There’s really limited information you Network (CIREN). the Toyota Camry and Chevrolet Cobalt. can get from a crash-test dummy,” says Ashley A. Weaver, an assistant professor “Wake Forest is a CIREN center, enroll- But the researchers found they needed at the Virginia Tech-Wake Forest Univer- ing four to five cases per month,” Weaver more computational power than they up the cyberinfrastructure and work- sity Center for Injury Biomechanics and says. “Engineers and physicians discuss could get from a campus computa- flows they needed to run the simula- causing stress moves from the foot to a former NSF graduate research fellow. the injuries and how the vehicles may tional cluster like DEAC. Assistance tions. Using the Total Human Model for the lower leg as a driver’s head comes “The [virtual] human body model gives have contributed. The result is a data- from XSEDE Extended Collaborative Safety, developed by the Toyota Central forward into a frontal airbag. They also “I would like to see our work used to base that researchers can use to model Support Service (ECSS) expert David Research and Development Laboratory, discovered that more reclined seating evaluate safety changes,” Weaver says. injuries computationally.” O’Neal at Pittsburgh Supercomputing Weaver and her team showed that sim- positions can lead to a higher risk of “One of the things we would be interest- Center (PSC) was instrumental in their ulations can reproduce real-world injury head and chest injuries. ed in is helping in the design of adaptive CIREN’s anatomical detail allowed the success. The architecture of XSEDE re- patterns and predict details crash-test .@ restraints that can be tuned to the age XSEDEscience scientists to detect injuries to soft-tissue source Blacklight at PSC, they found, dummies can’t provide. Weaver and her colleagues published and size of the occupant.” Such a system organs not present in mechanical dum- would be optimal for rapid assessment resources power study their results in Traffic Injury Prevention might shift the stress of an impact to a mies. of the large numbers of simulations they Results: Tailored restraints? in October 2015. The findings can help part of an occupant’s body that is better of virtual “crash-test needed to run in parallel. auto manufacturers design safer ve- able to sustain the shock, whether a frail “Lung injuries are really important in The simulations allowed the research- hicles and restraint systems. They have elder, a child or a healthy adult. dummies,” pointing respect to elderly vehicle occupants,” The ECSS team—staff with expertise ers to quantify the sensitivity and uncer- since moved their work to PSC’s interim Weaver says. “They put patients at risk in advanced computing as well as the tainty of the injury risk predictions based to safer passenger Greenfield system, a precursor to the “Of course we’d need to test new re- for pneumonia or acute respiratory dis- scientific background needed to under- on occupant position, which is difficult to new XSEDE resource at PSC, Bridges. straints computationally before moving tress syndrome. In addition, the human stand each user’s research at a deep restraints. determine in real-world motor vehicle into the crash-test phase,” she adds. body model gives us the ability to pre- level—helped the Wake Forest team set crashes. They demonstrated how injury-

8 9 TINY ZAPS, BIG RESULTS

An illustration of some of the results of large-scale When nanotechnology researchers make ‘small talk,’ it’s atomistic computer simulations of laser-induced structural modification of silver targets irradiated by 200 femtosecond (one quadrillionth of a second) extremely small. They are interested in physical phenomena laser pulses. The simulations make predictions regarding the surface and explain the experimental observation of surface swelling. The experiments at one-billionth of a meter – a million times shorter than were done in the group of Peter Balling at Aarhus University, Denmark. Illustration courtesy of Leonid the length of an ant, or 100,000 times thinner than a human Zhigilei, the University of Virginia. hair. Despite the size of their subject, the benefits to society are huge. In fact, according to the National Nano- reducing friction, erosion, and contami- Zhigilei’s group uses the Darter super- technology Initiative, more than 800 nation on items that need to be kept computer at the National Institute for everyday commercial products rely on clean, such as roof tiles and skyscraper Computational Sciences (NICS) and nanoscale materials and processes for a windows. Stampede at the Texas Advanced Com- variety of applications in medicine, en- puting Center (TACC). High-perfor- ergy, information technology, and many Because laser-induced processes are mance computing has played a crucial other areas. complex and happen so fast, experi- role in many of their projects because mentation cannot provide a detailed the systems they simulate can consist Going forward, Zhigilei’s group will study One method of nanotechnology re- understanding of the structural trans- of up to a billion atoms. how metal nanoparticles efficiently con- search involves the use of short laser formations triggered by the rapid laser vert laser energy absorbed at a surface, pulses at minuscule fractions of a sec- energy deposition, Zhigilei said. “Unlike in real experiments,” Zhigilei ex- a capability that is key to a growing ond to produce structural changes in plained, “the analysis of non-equilibrium number of imaging and therapeutic bio- thin, localized surface regions of mate- Chengping Wu, a member of Zhigilei’s processes in molecular dynamics can medical techniques, Wu said. rials like gold, silver, or silicon. Leonid computational materials group, add- be performed with unlimited atomic- Zhigilei of the University of Virginia said ed: “Our atomistic simulations, on the level resolution, providing complete in- “For example, researchers have dem- that what attracted him to this type of other hand, provide clear visual repre- formation of the phenomena of interest.” onstrated that laser irradiation of gold research is the ability of lasers to excite sentations, or ‘atomic movies.’ They are other XSEDE resources, of course,” he use of SAWs has broad implications for or silver nanoparticles attached to gene and change materials in ways not pos- well-suited to reveal the relationships XSEDE made Zhigilei’s compute alloca- said. “We use VisIt software for visualiza- applications in which heating must be markers and delivered to specific cells sible with any other technique. between the properties of laser-treated tions on Darter and Stampede possible, tions. In addition, many of my students avoided. can be used for selective killing of can- regions of the targets and the underly- as well as provided additional benefits. enjoy the file transfer service, Globus cer cells or bacteria,” Wu explained. For example, short-pulse laser process- ing microscopic mechanisms of laser- “When we write our allocation requests, Online, which is very efficient in moving Collaboration with the experimental “In the area of drug delivery, doping ing can transform a surface from very induced target modification.” we often ask for time on different com- large files that we are generating in our group of Peter Balling of Aarhus Uni- the walls of microcapsules with metal water attractant to very water repellant, puters, and we also take advantage of simulations.” versity, Denmark, led to an article titled nanoparticles opens a way for the re- “Generation of subsurface voids and a mote release of encapsulated mate- During 2015, Zhigilei’s group published nanocrystalline surface layer in fem- rial into specific cells by targeting metal two papers in the journal Physical Review tosecond laser irradiation of a single- nanoparticles with near-infrared laser with different collaborators. Working crystal Ag target.” The research showed irradiation.” XSEDE supports research in nanotechnology, where complex with Henry Helvajian of The Aerospace that the volume of a metal on a surface laser processes occur at lightning speed. And although Corporation, they discovered they could could be increased, thereby providing Yet another illustration of big implica- use surface acoustic waves (SAWs) to new opportunities for tailoring surface tions for humanity found at the na- the scale is tiny, the results for society are huge. move tiny particles of gold on a silicon properties to the needs of practical ap- noscale. surface. In the article, they note that the plications.

10 11 gen-free wetlands. Our current view of CIPRES OPENS GATEWAY the tree of life consists of three major branches: Eukaryotes, including ani- mals, plants, fungi and protozoans; fa- An outline for a new tree of life, depicting the evolution of life on miliar bacteria such as Escherchia coli; this planet that included more than 1,000 new types of bacteria and the Archaea. The revolution in DNA sequencing dur- and Archaea lurking in the Earth’s nooks and crannies, was ing the past couple of decades, however, has led to an explosion of data offering yet more complete descriptions of the made possible with the help of supercomputing resources genetic relationships among species. Researchers began sequencing whole available through XSEDE, along with a phylogenetics communities or organisms at once and picking out the individual groups based “gateway” available on those resources. on their genes alone. This metagenomic sequencing revealed whole new groups The new tree, published in April 2016 in and planetary science, policy, and man- ing researchers investigate relationships of bacteria and Archaea, many of them the new journal Nature Microbiology and agement, and the study’s principal in- by comparing DNA sequences informa- from extreme environments, such as the widely publicized throughout the gen- vestigator. “The new depiction will be tion between species. This type of anal- toxic puddles in abandoned mines, the eral press, reinforces once again that the of use not only to biologists who study ysis is becoming more powerful as the dirt under toxic waste sites, and the hu- life we see around us – plants, animals, microbial ecology, but also biochemists number of DNA sequences available is man gut. Some of these had been de- humans and other so-called eukaryotes searching for novel genes and research- increasing rapidly, with new, larger data tected before, but nothing was known – represents but a tiny percentage of ers studying evolution and earth history.” sets requiring higher levels of computa- about them because they wouldn’t sur- the world’s biodiversity. tional power. Researchers used the CIPRES (CyberIn- vive when isolated in a lab dish. “The tree of life is one of the most im- frastructure for Phylogenetic RESearch), In using the CIPRES gateway, the re- For the new paper, Banfield and Hug, portant organizing principles in biol- a web-based portal or gateway that al- searchers relied on two other XSEDE along with more than a dozen other ogy,” said Jill Banfield, a University of lows researchers to explore evolution- resources: Gordon, the first high-perfor- muscle, and without XSEDE there is no researchers who have sequenced new California, Berkeley professor of earth ary relationships between species. mance supercomputer to use massive work done, but Miller and his role with microbial species, gathered 1,011 pre- amounts of flash-based SSD (solid state CIPRES provides the entry point for viously unpublished genomes to add “The CIPRES Science Gateway was criti- drive) storage, and Comet, a petascale those resources. to already known genome sequences cal to our work,” said Laura Hug, who supercomputer designed to transform of organisms representing the major computed the trees at the University of advanced scientific computing by ex- Charles Darwin first sketched a tree of groups of life on Earth. A new tree of life California, Berkeley and is now a biology panding access and capacity among life in 1837 as he sought ways to show faculty member at the University of Wa- showing 1000 new traditional as well as non-traditional re- how animals and plants are related to Their investigation, representing the to- terloo (Canada). “Previous attempts to search domains. The two jobs ran for a one another. The idea took root in the tal diversity among all sequenced ge- types of bacteria infer the trees presented severe prob- total of about five days, using 48 cores. 19th century, with the tips of the twigs nomes, produced a tree with branches lems with run time, memory allocation Both Gordon and Comet, the result of representing life on Earth today, while dominated by bacteria, especially by developed using and a lack of parallelized implementa- two NSF grants, are housed at the San the branches connecting them to the uncultivated bacteria. A second view tion of the RAxML (Randomized Axeler- @XSEDEscience Diego Supercomputer Center (SDSC) at trunk implied evolutionary relationships of the tree grouped organisms by their ated Maximum Likelihood), a popular the University of California, San Diego. among these creatures. evolutionary distance from one another resources including program for phylogenetic analysis of rather than current taxonomic defini- large datasets. No run had success- Mark Miller, principal investigator for Since then, researchers have built on tions, making clear that about one-third the CIPRES fully finished prior to our introduction to CIPRES, thinks of the science gateway Darwin’s initial sketch, gradually add- of all biodiversity comes from cultivated CIPRES.” as a broker for XSEDE: he helps bring ing twigs and branches to the origi- phylogenetics bacteria, one-third from uncultivated people in to use CIPRES through one nal, including bacteria and Archaea bacteria, and one-third from Archaea gateway. Access to supercomputers also was a XSEDE account. XSEDE and its infra- — microbes that live in extreme envi- and Eukaryotes. key part of completing this study, help- structure provide all the computational ronments such as hot springs and oxy-

12 13 XSEDE’S Extended Collaborative formance, mostly through vectorization. What’s next for Bodony and Support Service (ECSS) We found that the data encoding and team? the original algorithm weren’t well suit- Through XSEDE’s Extended Collabora- ed to the Xeon Phi… there was a lot of According to Bodony, future comput- tive Support Service (ECSS) program, out-of-order memory access, which you ers are going to look a lot different than researchers have access to cyberinfra- can’t vectorize very easily.” Stampede or any of the other NSF- structure experts with a variety of exper- funded systems. Now, they’re not using tise. At first, Wilson executed a simple - per ECSS to focus on performance; they’re formance analysis of the code and iden- working with ECSS to change how the Bodony has used the ECSS program for tified the algorithmic weaknesses to find code is programmed. a variety of projects. When asked if he better ways to express the algorithms. would recommend the ECSS program Sometimes the algorithms needed new As part of the exascale applications to other researchers, his response was data structures, sometimes an entirely group, Bodony and his team are fo- a quick, “Yes, wholeheartedly. The ECSS new algorithm needed to be imple- cused on building scalable algorithms. experts are able to look at the code and mented to perform the same operation, “How you program on future machines understand the hardware and software and sometimes the researchers needed is going to be very different from how very quickly to make a diagnosis.” to rewrite part of the code that made we program for Stampede,” Bodony said. that algorithm no longer necessary. “What that means is that the codes that Currently, through XSEDE’s ECSS pro- we have now may not run on future ma- gram, Bodony works primarily with It was a team effort among Bodony, Wil- chines. We’re trying to rewrite the code Luke Wilson from the Texas Advanced son and several people at Illinois, but in such a way that it’s ready for those fu- THE SCIENCE OF JET NOISE Computing Center (TACC). Wilson, who Wilson was instrumental in taking the ture machines. Luke and I are working serves as the technical expert—with cumulative view of the code and speed- together to figure out how to fix our cur- Daniel Bodony’s love of science began with a love of airplanes. knowledge about the hardware, and ing it up by a factor of seven. How? He rent code and transform it into one that’s how the software interacts with the figured out where the performance bot- useful at exascale.” hardware—is helping Bodony and his tlenecks were. On weekends, he worked for one of his dad’s colleagues who team achieve real performance results “Most people think that Knight’s Landing on their code. “We still have a long way to go with jet (a second generation Xeon Phi product had an airplane. “I would mow his grass and he would let me noise and we’re going to continue to using a 14 nm process) is a preview of “When we run our code we have a basic follow jet noise for the foreseeable fu- what processors will look like going for- fly,” Bodony remembers fondly. idea of what its weaknesses are, and we ture,” Bodony said. “We think the flow ward as we push toward exascale—lots try to identify the biggest weakness that that exits the jet engines contains the of concurrency, many cores in a single Those early boyhood days fueled the Champaign, is looking into the science is more by accident than by design, and impacts our ability to run efficiently on information that we need to figure out package, the memory footprint per thread fire for Bodony as he committed himself surrounding the aeroacoustics of jet we’re trying to change that, but it relies XSEDE systems, including being able to how to make jet engines quieter. We is going to be very small—so we will com- to a career as a military pilot but, at that engines and researching how to make on bigger calculations, bigger codes, utilize Stampede’s Intel® Xeon Phi™ pro- just haven’t probed it in the right way. pletely rethink the way we solve our prob- time, a pilot who wore glasses was not them quieter. and more complex computing capabili- cessors,” Bodony says. We’ve been working on tools to extract lems. It’s safe to say that every time a new allowed. Taking that in stride, Bodony ties.” that information. Our current hypothesis processor comes out it’s a completely decided he would instead design air- A veteran user of NSF high performance According to Wilson, “The goal has al- has shown that our idea has merit using new challenge,” Wilson concluded. planes, only to experience another shift computing (HPC) resources since 2008, The computational challenges that ways been to get this code up and run- small scale simulations and now we’re in his early career. Bodony said: “The reason we use super- Bodony and his team face invariably in- ning on the Intel® Xeon Phi™, and we applying these ideas to the full scale jet computers is because in aeroacoustics volve turbulence, which is an unsteady, started out looking for some simple noise problem.” “When I got to college and started to there is no simple relationship that re- chaotic motion of a fluid. places we could target to improve per- design airplanes I realized that I liked lates an unsteady flow field to the sound the science behind the design more it creates. So we have to resort to elabo- “It’s a classical multi-scale problem,” than I liked the design itself,” he said. rate experiments or simulations to try to Bodony said. “Computational research come up with the contextual underpin- is required to resolve all of those scales Quiet the sonic boom! @XSEDEscience researchers are finding innovative Bodony, the Blue Waters Associate nings that relate cause and effect. And which requires us to use the largest Professor in Aerospace Engineering we still haven’t done it. The fact that air- computers to which we have access, ways of studying aeroacoustics in order to make jet engines quieter. at the University of Illinois at Urbana- craft has gotten quieter over the years XSEDE’s Stampede being one of them.”

14 15 in the abdomen. Abdominal aneurysms are more common and when they rup- INVESTIGATING AAAs ture, they are often lethal. Abdominal aortic aneurysm (AAA) is Because aneurysms are stressed by the a vascular disease that affects tens of XSEDE ECSS supports Yale researchers continuous action of changing blood thousands of people around the world pressures, methods of engineering that each year. The disease occurs most studying vascular disease were developed to study the flow of vis- commonly in men over 60, and is char- cous fluids (e.g., blood) and the failure of acterized by a dilation of the abdominal structures (rupture of the aorta) can be aortic wall and often a persistent blood used to help understand the complex clot. When they rupture, abdominal an- history of these deadly lesions. Once an eurysms can be lethal. AAA ruptures, death is nearly inevitable To understand why some AAAs rupture due to extreme internal bleeding, signi- and others do not, researchers must fying the critical need to predict which better understand how they form and aneurysms are most vulnerable. enlarge, digging deep into both the mi- Using computational models in com- cro and macro levels to understand the bination with biological experiments, complex interactions that define the dis- researchers in the lab can study AAA ease. development at both the micro and Mechanically and hemodynamically-driven growth and remodeling models of disease progression could help improve clinical decision-making by providing indications of the future course of disease. One group attempting to accomplish macro scales. Jay Humphrey, professor this task is the Continuum Biomechan- of biomedical engineering, emphasizes ics Lab in the Department of Biomedi- that, “The biological and mechanical partnered with Yifeng Cui at the San Di- cases, which in practice translated into cal Engineering at Yale University. To complexity of AAAs demands a multi- ego Supercomputing Center (SDSC) to a four-time speedup. explore the hemodynamics in AAAs, the disciplinary team to advance our under- optimize their code and improve simu- Working with Cui, the team also ran fluid dynamics of blood flows, the re- standing.” George Tellides, Yale profes- lations. They found the program very their code on TACC’s most powerful su- searchers rely on a mix of computational sor of surgery and Andrew Sherman at helpful to adapting their new codes to a percomputer, Stampede, and used its resources from Yale and XSEDE. the Yale Center for Research Comput- supercomputing environment. ing, teamed up with Humphrey to en- visualization nodes to create realistic “Access to XSEDE resources is a key as- able the development of patient-spe- “I would say that’s the biggest advan- depictions of AAAs. The team also used pect of our research. The code we start- cific computational models for multiple tage of ECSS, talking with experts to get Trestles, which had been at SDSC. ed working with was already equipped aspects of AAAs. real insight into how our codes can be To advance their research, the team is to work on supercomputers, but having improved. It’s the best way for us to opti- using this information to develop com- access to high performance computers In collaboration with post-doctoral fel- mize our codes, because we don’t have prehensive models to track the pro- enables us to explore additional paral- lows and Ph.D. students, they have re- time to read all the literature on these gression of the disease. Ultimately, they lel capabilities,” said Paolo Di Achille, a cently put forth new hypotheses on new architectures,” said Di Achille. hope this research will not only increase researcher in the group. factors that drive intraluminal (interior areas of the artery) clotting as well as In this particular case, according to Di our basic understanding of AAAs, but will also help clinicians make more in- Computer simulations of the fluid dynamics of blood flow. Researchers integrate this Looking In the mechanobiological stability of the Achille, after discussing with the ECSS information with other blood flow metrics to investigate why thrombus forms in aneurysms. diseased aortas. expert about the limitations of a particle formed decisions when treating pa- tracking code that they developed, the tients. The aorta is the largest artery in the body; To take full advantage of XSEDE’s capa- researchers introduced a different load it allows oxygenated blood to be distrib- bilities for this research, the researchers balancing scheme that better exploit- “Clinical decisions are extremely com- uted to organs and tissues throughout Abdominal aortic aneurysm affects thousands applied to XSEDE’s Extended Collabo- ed the capabilities of large computers plex,” Di Achille noted. “If our modeling the body. Aneurysms, or localized dila- ration and Support Service (ECSS), a and high performance file systems (e.g. results could help in this decision pro- each year. @XSEDEscience ECSS supports tations, can form in the ascending or de- program that pairs researchers with ex- parallel I/O libraries). With these modi- cess and improve outcomes that would scending portions of the aorta within the Yale researchers investigating the disease. pert staff members in advanced cyber- fications, the group brought their code be very satisfying.” chest or within the descending portion infrastructure. The researchers at Yale to scale almost linearly for typical use

16 17 THE TURTLE TREE OF LIFE

The researchers were, for example, able The project also took the initiative of as- “I think that naming the Archelosauria In all of the animal kingdom, it’s hard to top the evolutionary to show that softshell turtles have been signing a name to the grouping of what grouping will solidify the idea of this evolving on their own branch of the tree previous research has suggested to be clade (a group of organisms believed to longevity and unique features of turtles. These reptiles have for a very long time. The findings also turtles’ nearest relatives, crocodiles and have evolved from a common ancestor) illustrate that sea turtles are closely dinosaurs (including birds). The name in people’s minds,” Parham said. existed for more than 200 million years, and their combination related to particular groups of North they coined is Archelosauria, a combi- American freshwater turtles (mud and nation of archosaurs (crocodiles and di- Details of the research are contained in of shells, beaks, widely ranging body sizes, and other unusual snapping turtles) and, therefore, likely nosaurs) and chelonians (turtles). “A phylogenomic analysis of turtles,” in originated in North America as well. the journal Molecular Phylogenetics and traits has been an endless source of fascination. Evolution. “We compared our tree of life to the tim- From the freshwater-loving variety to team composed of turtle expert James Through an allocation from XSEDE, ing and geography of the fossil record those of the sea, to the bizarre ones Parham, geneticist Nicholas Crawford, Crawford was able to make use of the to provide a very complete picture of that sport soft shells and can move at and other co-researchers performed large RAM nodes on the Gordon system turtles around the world through time,” speeds of up to 15 miles an hour, turtles an investigation to trace the changes at the San Diego Supercomputer Cen- Parham said. have adopted so many varied charac- in evolutionary history, or phylogeny, of ter (SDSC) to complete the UCE work in teristics during their existence through turtles. parallel during the course of only a few the ages that their lineage has been a days. The project also applied the Black- longstanding question in biology. “In some instances, previous molecu- light system (now decommissioned) at lar [genetic] analyses conflicted with the Pittsburgh Supercomputing Center What does the turtle tree of life morphological [form, structure, feature] (PSC) via XSEDE. look like? analyses, but it wasn’t clear if this was because morphological analyses were The research team discovered the In an effort to add further knowledge wrong or if there simply wasn’t enough strongest genetic evidence to date for and perspective to the existing body molecular data,” said Crawford, a post- how turtles evolved, and the phyloge- of scientific literature, a collaborative doctoral fellow in the Department of nomic tree they developed from their Genetics at the University of Pennsylva- findings can be used by paleontologists nia. “So we sequenced 1,000-plus indi- as a starting point to greatly advance vidual parts from 29 turtle genomes to the study of turtle fossils and evolution, get a bigger picture and hopefully less- according to Parham. With XSEDE, biased molecular perspective of turtle researchers have relationships.” “Over the past 25 years or so, ideas about how different groups of turtles probed the lineage Crawford explained that he needed to were related to one another—such as assemble numerous libraries of what which group of freshwater turtles is of turtles and found are called ultraconserved elements most closely related to sea turtles, or the strongest genetic (UCEs), across a variety of parameters. which turtles have been evolving inde- UCEs, according to a website called pendently on their own branch of the evidence yet for how ultraconserved.org, can be defined as tree for the longest amount of time— “highly conserved regions of organismal have been changing,” Parham said. “By these animals with genomes shared among evolutionary virtue of the huge amounts of data that eclectic features distant taxa [organism populations]—for we bring to the problem, we can provide instance, birds share many UCEs with strong evidence to bear on competing likely evolved. humans.” hypotheses.”

18 19 A technician works on one of LIGO’s optics. At each observatory, the 2 1/2-mile long L-shaped LIGO interferometer uses laser light split into two beams that travel back and forth down the arms. The beams are used to monitor the distance between mirrors precisely The discovery of gravitational waves was LIGO people doing code optimization; signals to ensure that the search is work- positioned at the ends of the arms. According to Einstein’s theory, made by researchers at the NSF-funded the directions we chose were formed by ing properly,” said Brown. “Once we have the distance between the mirrors will change when a gravitational twin Laser Interferometer Gravitational our interactions with TACC staff and we made detections, extracting the physics wave passed by the detector. Credit: LIGO Laboratory Wave Observatory (LIGO) detectors in wouldn’t have been as successful with- from these waves requires careful mod- September 2015, but scientists wanted out them,” Couvares said. eling of the black holes. The waves that to be sure of what they detected on the LIGO detected were produced by very two detectors, which are about 1,900 LIGO used Comet’s new Virtual Cluster strong gravitational fields. By comparing miles apart in Louisiana and Washing- interface for the analysis of their present calculations of the detected waves with ton State. data. The total data set was five tera- our observations, we can explore the bytes (TB), stored at the Holland Com- nature of black holes and gravity itself.” LIGO researchers used the OSG as a puting Center at University of Nebras- The factor of 8x speedup in code perfor- gateway to XSEDE’s Stampede and ka–Lincoln (UNL). Distributing the 5TB mance allowed LIGO to publish the re- Comet supercomputers. “We also lev- dataset from UNL to the many clusters sult much earlier than would have been eraged resources at other institutions on OSG, including Stampede and Comet, possible with their original code. in the OSG, but Stampede and Comet required a total transfer of one petabyte were among the most used,” said Pe- of data, which was accomplished using Couvares noted that LIGO’s comput- ter Couvares, the Advanced LIGO Data GridFTP. Comet is housed at the San Di- ing demands are very much in flux. “If Analysis Computing Manager. “For LIGO ego Supercomputer Center (SDSC) at nature is kind and we see a lot more especially, OSG and XSEDE are highly UC San Diego. events than we expected, we’d need a complementary and becoming critical lot more computing power than antici- to our research efforts.” Couvares said the discovery has opened pated. We wanted to have capabilities a new window on astronomy and astro- that went beyond our in-house comput- Staff at the Texas Advanced Computing physics. “We’re now seeing things in the ing resources, so XSEDE and OSG were Center (TACC) at the University of Texas universe that we’ve never seen before, an excellent option,” he said. at Austin, including those from XSEDE’s and we have a medium for information Extended Collaborative Support Ser- that we’ve never had before. In addition Moreover, he believes that XSEDE is at vice (ECSS) program, were able to rec- to telescopes that provide electromag- the forefront of defining what large scale reate the computing environment that netic observations, we now have gravi- computing and these national facilities LIGO researchers were accustomed tational wave observations and that’s will look like in the future. to on Stampede, which was valued for very powerful.” its large number of cores. The bulk of “LIGO is going to move into an era of data analysis requires large scale high- Since LIGO scientists don’t know what regular detections where we’re record- throughput computing with parallel the waves look like before they detect ing more than one event,” he said. “At the workflows at the scale of tens of thou- them, they have to search over a large same time, we’ll be pushing the limits of sands of cores for long periods of time. space of possible signals. “The gravita- physics, astrophysics, and engineering.” In addition, TACC staff were able to im- tional waves depend sensitively on the XSEDE HELPS CONFIRM LIGO DISCOVERY prove the performance of the code by black hole’s masses and spins, making a factor of four, and with the techniques the search computationally challeng- learned from TACC, LIGO staff were able ing,” said Duncan Brown, the Charles Scientists used XSEDE Scientists for the first time detected gravitational waves— to improve that by another factor of Brightman Professor of Physics at Syra- two. The result was that LIGO’s original cuse University and a gravitational-wave supercomputing ripples in the fabric of space-time hypothesized by Albert estimate for their computing needs go- astrophysicist who studies the mergers ing forward were reduced by a factor of of binary black holes. resources to verify Einstein a century ago—in a landmark discovery announced eight, which obviated an investment of the discovery tens of millions of dollars in computer LIGO’s discovery of gravitational waves hardware. from the binary black hole required of gravitational waves in early 2016 that opened a new era in astrophysics while large-scale data analysis to validate “The staff at TACC was instrumental the discovery claim. “This includes theorized by Albert relying on XSEDE and Open Science Grid (OSG) resources to in helping us guide where we looked measuring how significant the signal is for optimization; ultimately, we ended compared to noise in the detector, and Einstein 100 years ago. verify the findings. up with a team of almost five full-time re-analyzing the data with simulated

20 21 XSEDE is supported by the National Science Foundation.

xsede.org | [email protected] | /XSEDEscience | @XSEDEscience