Getting “High Performance” out of High Performance Computing in a Liberal Arts and Science Setting

Carol A. Parish

Department of Chemistry University of Richmond Richmond, VA 23227 [email protected] University of Richmond • 3000 undergraduate students; 500 graduate students (law, business) • Significant resources for teaching and research ($1.4B endowment) • Very strong research culture • Significant support for science • 600 sq ft labs • expansion University of Richmond • Chemistry 15 TT faculty, 4 Ph.D support faculty, 5 staff, 75% have external funding • Graduate 20-35 majors each year - 25- 40% go on to graduate school Current Research Students: Dio Saldano-Greco, „10 Research Group Greg Springsted, ‟10 Tyler Steele, ‟10 Kendra Cunningham, ‟11 Sally Fisher, ‟11 Jenna Landers, ‟11 Angela Xie, ‟11 Anna Parker, ‟11 Reggie Gooden, ‟11 John Mancini, ‟11 Hao Zhong, ‟12 Matt Fanelli -PB Justin Cook, ‟12 Julia James – PD Alex Hahn, „13 Chris Ashman – PD Conor Phelan, „13 Sara Booth – HS student Financial Support: ACS-PRF (GB, UFS, B) NSF (RUI, REU, MRI) Collaborators: DOE-BES Dreyfus Foundation Roald Hoffmann – Cornell Jeffress Foundation Inmaculada Robina – University of Seville Beckman Foundation KC Russell – Northern Kentucky U Pfizer SURF program Hans Lischka – University of Vienna HHMI Foundation Kumaresh Ghosh – University of Kalyani, India Merck-AAAS Program Uses of HPC • Approach – hardware and problem dependent – Many individual jobs on many individual nodes (I/I) • Really coarse grained – monitored/maintained by students using logins, web-based boards and ganglia • Coarse grained – using server software that spawns multiple individual jobs and collects up final product – Individual job on 1 multi-core node (I/1MC) – Individual job run across a large number of multiple nodes (I/M) – needs high speed interconnects (expensive) Uses of HPC • Methods – need the right tool for the job – Quantum mechanics (I/I, I/1MC, I/M; depending on level of rigor and system size) – Conformational Searching (I/I), Docking (I/I) – , Monte Carlo, Free Energy Perturbation (I/I, I/1MC, I/M) • Software – different algorithms in different packages – Schrodinger (MacroModel, , Glide, QSite) $$ – $ Q-Chem $ in-house SOM Uses of HPC • Software – free! – Columbus, Gamess, NWChem – , Amber, Charmm – Freeware is sometimes a bit harder to use resulting in more investment in student training How a chemist obtains access to HPC

• Write external grants to get $ – Theory is not cheap • hardware is almost “disposable” particularly if used heavily – start thinking about replacement almost as soon as acquisition • HW/SW advances happen quickly • New methods demand new hardware • As processors get faster and cheaper, the “cutting edge” in system size gets larger and more rigorous • The larger the research group the more processors and GUI workstations are necessary How a chemist obtains access to HPC

1st Cluster Fun to build Hard to maintain

16 single AMD Athlon 800 MHz procs, 100 Mbps switch

Dreyfus Special Grant Program $16K Patchett Family Foundation $14K Power and cooling HWS Cluster computing on your own • Need to learn or be knowledgeable about – Latest in hardware development – HW/SW configuration, compatibility – Interaction with vendors, cost/performance – Intranet and internet, security – Operating systems – Back up devices, software – To queue or not to queue – Power/cooling, space/footprint, noise – User accounts – Troubleshooting hardware failure How a chemist obtains access to HPC

3rd cluster – 96 cores 48 dual Opteron 2.2GHz, gigabit 512RAM each node, 160/20 HD $150K – UR relocation Location – Data Center

2nd cluster – 46 cores, 23 - dual AMD athlons MP 2200+ procs, 100Mbps, 160 GB HD 3rd cluster – 80 cores and 512 MB RAM on head & 10-2.33GHz Intel Xeon 20GB HD and 512 MB RAM Dual Quad Core nodes on nodes Location – lab -NSF/PRF-G, $48K Dedicated, ventilated -Power and cooling HWS cooling -Hired CS student to help Gigabit connects chem students to build $40K NSF cluster More control = more work

If you have good IT/IS people – trust them to take care of your hardware How a chemist obtains access to HPC • Obtain institutional buy-in – Support for purchasing new hardware and software – Support for on-going technical support – Plan for replacement – Donation of phased out institutional hardware • Is it “new‟ enough to warrant the time to build a homemade cluster? • Can it be used to replace workstations particularly if workstations are used mainly for job-setup and ? The importance of talented, local support personnel

• HWS – no support • UR – significant support – René Kanters - chemistry HW/SW support – Sasko Stefanovski – IT linux support specialist – Clovis Khoury – Data Center Manager – Diane Kellogg – chemistry facilities liaison – Kathy Monday – VP for Information Services How a chemist obtains access to HPC • Use TeraGrid – NSF supported – largest cyberinfrastructure facility available for non-classified use in the US. – Network of supercomputers • 100+ teraflops of computing power • 15 petabytes of storage • High speed interconnects How a chemist obtains access to HPC • Use TeraGrid – Network of supercomputers • Academic partners - Indiana, Pitt, Purdue, San Diego, Texas, Chicago/Argonne • National lab partners –Center for Supercomputer Applications (NCSA), Oak Ridge, Center for Atmospheric Research (NCAR) – Free, relatively easy to get allocation – Initial app (30K hours) – 1 page proposal – c.v. How a chemist obtains access to HPC • Use TeraGrid – So why doesn‟t everybody use it? • Difficult to use • Lack of systematic configuration across sites • Undergraduates can chew up compute time without necessarily generating publishable results particularly during training – Campus Champions Program – Point of contact (René) – Liaison to facilitate access – Attends training How a chemist obtains access to HPC

• Join or form a consortium – The Molecular Education and Research Consortium in Undergraduate (MERCURY) – Summon the forces of many to compete for $ – Original request was for $780K MERCURY Founding Members

Ramona Taylor Maria Gomez Marc Zimmer College of the Holy Cross Vassar College Connecticut College Worcester, MA Poughkeepsie, NY New London, CT

George Shields Jeff Greathouse Carol Parish Hamilton College St. Lawrence Univ. Clinton, NY Hobart & Wm Smith College Canton, NY Geneva, NY How a chemist obtains access to HPC • Join or form a consortium – Consortium has grown to 12+ members from all over the U.S. – Since 2001 we have submitted two more MRI proposals for an additional $329K. – We have used the funds to purchase hardware and to support an annual meeting – The initial grant paid for a full-time system administrator at Hamilton College (Jenn Strum); after that granting period Hamilton has picked up the cost (Steve Young) How we have used MERCURY $ • Rack #1 - 32 cpu SGI Origin • Rack #2 - 38 – 4 cpu nodes • Rack #3 - 32 cpu SGI Altix) • Rack #4 - ( 2 – 16 cpu and 1-2 cpu SGI Altix) • Rack #5 - Server equipment rack (NetApp SAN, Tape Library) Benefits and Effects • Faculty-to-faculty mentoring and support – Writing and reviewing proposals and papers – Student training, expectations – Identifying grants that will support post-grad hiring – Negotiating with administrators – Dealing with (ignoring? ) local politics – Recruiting students – Mentoring students in the pursuit of awards • significant undergraduate involvement • a mix of new and established faculty participants Benefits and Effects Since the MERCURY consortium was established • our collective publication rate has more than doubled • our external grant awards have more than tripled (submission rates are even higher) • the number of students pursuing graduate work in chemistry has increased by ~25% • we have mentored an increasingly large number of underrepresented students Benefits and Effects • Student and faculty networking and mentoring • Student awards: (1) Rhodes Scholar, (1) Gates- Millennium Scholar, (4) ACS Scholars, (1) UNCF-Merck Scholar, (8) Barry M. Goldwater Scholars, (4) Fulbright Scholars • Students accepted into high quality graduate programs: UC Berkeley Georgia Pennsylvania UC Santa Barbara Georgia Tech Penn State Cambridge Johns Hopkins SUNY – Buffalo (2) Oxford Michigan Syracuse Colorado Minnesota Texas - Austin Columbia Missouri Wisconsin Connecticut New Hampshire Washington Duke North Carolina Wyoming Illinois Northwestern Yale (5) The Annual MERCURY Meeting • Focus on undergraduate computational chemistry • National in scope • Modeled after Gordon conferences • Brings together undergraduate researchers and their faculty mentors Carlos Simmerling SUNY Stony Brook • Plenary talks by leaders in the field • Talks directed at an undergraduate audience • Undergraduate poster session – Sanibel style The Annual MERCURY Meeting

• Opening and closing banquet • Common dining experiences • Evenings in the Adirondack pub • Lodging on campus Students meeting with speakers • Informal “chat sessions” with speakers • 3 days / 2 nights of scientific exchange, networking and camaraderie Lessons Learned • Local resources and talented technical support are key – ideal situation – technical support also knowledgeable in specific academic field • It‟s not critical that I retain root control – surrendering this has allow me to do more CHEMISTRY • Resources must match needs • Don‟t forget costs associated with power, cooling, maintenance, etc • Consortia are important • National grid resources can be useful and will hopefully become more useful • Effective and frequent communication and well-written S.O.P.s are critical in order for a PI to use non-local resources effectively