Vijay S. Pande1–4 Ian Baker1 Atomistic Protein Folding Jarrod Chapman4,* Sidney P. Elmer1 Simulations on the Siraj Khaliq5 Submillisecond Time Scale Stefan M. Larson2 Using Worldwide Distributed Young Min Rhee1 Michael R. Shirts1 Computing Christopher D. Snow2 Eric J. Sorin1 Bojan Zagrovic2 1 Department of Chemistry, 4 Stanford Synchrotron Stanford University, Radiation Laboratory, Stanford, CA 94305-5080 Stanford University, Stanford, CA 94305-5080 2 Biophysics Program, Stanford University, 5 Department of Computer Stanford, CA 94305-5080 Science, Stanford University, 3 Department of Structural Stanford, CA 94305-5080 Biology, Received 27 March 2002; Stanford University, accepted 22 May 2002 Stanford, CA 94305-5080 Abstract: Atomistic simulations of protein folding have the potential to be a great complement to experimental studies, but have been severely limited by the time scales accessible with current computer hardware and algorithms. By employing a worldwide distributed computing network of tens of thousands of PCs and algorithms designed to efficiently utilize this new many-processor, highly heterogeneous, loosely coupled distributed computing paradigm, we have been able to simulate hundreds of microseconds of atomistic molecular dynamics. This has allowed us to directly Correspondence to: Vijay S. Pande; email: pande@stanford. edu *Present address: Physics Department, University of California, Berkeley, CA Contact grant sponsor: ACS PRF, NSF MRSEC CPIMA, NIH BISTI, ARO, and Stanford University Contact grant numbers: 36028-AC4 (ACS PRF), DMR-9808677 (NSF MRSEC CPIMA), IP20 6M64782-01 (NIH BISTI), 41778- LS-RIP (ARO) Biopolymers, Vol. 68, 91–109 (2003) © 2002 Wiley Periodicals, Inc. 91 92 Pande et al. simulate the folding mechanism and to accurately predict the folding rate of several fast-folding proteins and polymers, including a nonbiological helix, polypeptide ␣-helices, a ␤-hairpin, and a three-helix bundle protein from the villin headpiece. Our results demonstrate that one can reach the time scales needed to simulate fast folding using distributed computing, and that potential sets used to describe interatomic interactions are sufficiently accurate to reach the folded state with exper- imentally validated rates, at least for small proteins. © 2002 Wiley Periodicals, Inc. Biopolymers 68: 91–109, 2003 Keywords: atomistic protein folding; microsecond time scale; computer hardware; computer algorithms; molecular dynamics; distributed computing; villin; beta hairpin INTRODUCTION Developing such large-scale parallelization meth- ods is very difficult, and current parallelization Understanding the sequence–structure relationship of schemes cannot scale to the level of even thousands of proteins will play a pivotal role in the postgenomic processors (i.e., cannot use so many processors effi- era, and will have great impact on genetics, biochem- ciently). To understand why scalability to thousands 1–3 istry, and pharmaceutical chemistry. A detailed of processors is so difficult, consider an analogy to a picture of the folding process itself will be important graduate student thesis. A typical thesis takes 1500 in understanding diseases, such as Alzheimer’s and graduate student days. If one employed 1500 graduate variant Creutzfeldt–Jacob disease, believed to be re- students to accomplish this goal, would it be possible 4 lated to protein misfolding. Finally, an understanding to complete a thesis in a single day? Clearly not—the of protein folding mechanisms will be important in overhead of communication between students, as well protein design and nanotechnology, in which self- as the inability to devise an “algorithm” to divide the assembling nanomachines may be designed using labor evenly, would make 100% efficiency impossible synthetic polymers with protein-like folding proper- 5 in this case. At this level of scaling, it is likely that the ties. work would actually take longer. These issues, in Unfortunately, current computational techniques to particular balancing communication time against time tackle protein folding simulations are fundamentally spent actually doing work, are mirrored in the division limited by the long time scales (from a simulation of labor between computer processors. Clearly, the point of view) needed to study the dynamics of inter- only way to efficiently utilize such a large number of est. For example, while the fastest proteins fold on the processors is to divide work in such a way that re- order of tens of microseconds, current single com- quires minimal communication. puter processors can only simulate on the order of a Even with an algorithm with perfect scalability nanosecond of real-time folding in full atomic detail per CPU day—a 10,000-fold-computational gap. (e.g., with a 10,000-fold increase in speed using Great strides in traditional parallel molecular dynam- 10,000 processors), we are still left with the problem ics (MD), utilizing many processors to speed a single of obtaining a 10,000-processor supercomputer. For dynamics simulation, have been made and have par- comparison, the largest unclassified supercomputer in tially overcome this divide. A tour-de-force parallel- the world (the SP at NERSC) has 2500 processors, ization of simulation code for supercomputers by Duan and of course this resource must be shared between and Kollman has previously led to the simulation many (hundreds) different research groups. Recently, of 1 ␮s of dynamics for the villin headpiece three- another approach has been developed to bridge this helix bundle,6 demonstrating that parallelization enormous computational gap: worldwide distributed 7 schemes using hundreds of processors can be used to computing. There are hundreds of millions of idle make significant progress at closing this computa- PCs potentially available for use at any given time, tional gap. However, such methods have fundamental the majority of which are vastly underused. These drawbacks: in particular, these methods require com- computers could be used to form the most powerful plex, expensive supercomputers due to the need for supercomputer on the planet by several orders of fast communication between processors. Moreover, magnitude.7 However, to tap into this resource effi- due to the stochastic nature of folding, in order to ciently and productively, we must employ nontradi- study the folding of a 10-␮s folder, one must simulate tional parallelization techniques.8,9 Indeed, we wish to hundreds of microseconds, requiring computing accomplish a seemingly impossible goal: to push the power equal to thousands or tens of thousands of scalability of MD simulations to previously unattain- today’s processors. able levels (i.e., the efficient use of tens of thousands Protein Folding Simulations on the Submillisecond Time Scale 93 of processors) using an extremely heterogeneous net- along the pathway (which have been found in many folding work of processors that are loosely coupled by rela- and unfolding simulations19). Second, these methods re- tively low-tech networking (primarily modems). quire knowledge of the native state as an end goal and in a In this review, we first present the details of our sense apply a field to the trajectories to reach this native method to simulate protein folding using distributed state. Finally, in the end, the heart of the protein folding problem lies in sampling, and even with the great benefits of computing, and then summarize our folding simula- path sampling methodology, a tremendous degree of sam- tion results for several small, fast folding proteins and pling (in this case, in path space) must still be performed. polymers. Specifically, we demonstrate the applica- Thus, the main goal of the method we have developed is bility of our method by simulating the folding of to simulate folding dynamics starting purely from the pro- 10,11 protein helices in atomic detail. Next, as an addi- tein sequence and an atomistic force field, without using any tional quantitative test of our methodology, we exam- knowledge of the native state in the folding simulation. It is ine the folding rate and folding time distribution of a important that our method can successfully tackle the issue nonbiological helix,12 for which results of traditional of lingering in metastable free energy minima. To achieve MD simulations13 and experiments14 are known. We this, we use the following algorithm (“ensemble dynam- finally apply these methods to larger and more com- ics”). Consider running M independent simulations started plex proteins, including a ␤-hairpin15,16 and a three- from a given initial condition (each run starts from the same coordinates, but with different velocities). We next wait for helix bundle.6 We conclude with an assessment of the the first simulation to cross the free energy barrier (see validity of our methods including a quantitative com- Figure 1). Since the average time for the first of M simula- parison of these results with experimental measure- tions to cross a single barrier is M times less than the ments of folding rates and equilibrium constants, and average time for all the simulations (assuming an exponen- a discussion of what we have learned about folding tial distribution of barrier crossing times8,9; see below for mechanisms. details), we can use M processors to effectively achieve an M times speedup of a dynamical simulation, thus avoiding the waiting in free energy minima. In a sense, we distribute the waiting to each processor in parallel, rather than in METHODS series, as in traditional parallel MD. Given