Parallel Molecular Dynamics Simulations of Biomolecular Systems Alexander Lyubartsev and Aatto Laaksonen Department of Physical Chemistry Arrhenius Lab oratory University of Sto ckholm S Sto ckholm Sweden Abstract We describ e a general purp ose parallel molecular dynamics co de for simulations of arbitrary mixtures of exible molecules in so lution The program allows us to simulate molecular systems describ ed by standard force elds like AMBER GROMOS or CHARMM con taining terms for shortrange interactions of the LennardJones typ e electrostatic interactions covalent b onds covalent angles and torsional angles and a few other optional terms The stateoftheart molecular dy namics techniques are implemented constanttemp erature and constant pressure simulations optimized Ewald metho d for treatment of electro static forces double time step algorithm for separate integration of fast and slow motions The program is written in standard Fortran and uses MPI library for communications b etween no des The scalable prop erties of the program do not dep end on the complexity of the studied system and are determined mainly by the hardware and communication sp eed Examples of a few molecular systems diering by the comp osi tion will b e given Ionic water solutions large DNA fragments in water solution with counter ions a phospholipid membrane system Keywords parallel algorithms molecular dynamics computer simula tions Intro duction Computer simulation metho ds such as Molecular Dynamics MD and Monte Carlo MC have now b ecome imp ortant techniques to study uids and solids These metho ds provide a link b etween theory and exp eriment and they are also the only way to study complex manyb o dy systems when b oth exp erimental techniques and analytical theories are unavailable The MD metho d provides a numerical solution of classical Newtons equa tions of motion 2 2 m d r dt F r r i i i 1 N where force F r r acting on particle i is dened by the interaction i 1 N p otential or force eld U r r 1 N F r r U r r i 1 N 1 N r i The most timeconsuming part sometimes up to of the cpu time of the MD simulations is the calculations of the forces F r r In a typical case i 1 N of pair interactions the cpu time may scale with numb er of particles N b etween 2 O N and O N dep ending on the algorithm and the typ e of interaction p o tential note that scaling as O N may b e reached only at a very large N and for systems with shortrange interactions Now it is a standard routine pro cedure to simulate molecular systems con sisting of order of particles which in some cases eg simple liquids is sucient to give a go o d description of corresp onding macrosystem For other systems a larger numb er of particles is needed in order to describ e them in a realistic way Complex bio and organic molecules eg proteins nucleic acids membranes carb ohydrates etc immersed into a solvent increase the numb er of involved atoms one to two or more orders of magnitude Also larger the molecu lar systems grow longer simulations are needed to follow lowamplitude motions and slow conformational transitions It is clear that the rate of the progress to wards more complex molecular mo dels is set to a large extent by advances in micropro cessor technology and computer architecture as well as by development of appropriate software Computer simulations of manyparticle systems are well suited for parallel computer systems The basic reason for this is that the forces acting on each par ticle can b e calculated indep endently in dierent pro cessors However the most optimal parallel scheme for a particular problem dep ends b oth on the hardware in hand and on the system under investigation size typ e of interaction etc Electrostatic interactions fast intramolecular motions due to explicit mo deling of hydrogens angle and torsional angle forces of macromolecules all these kinds of forces require a sp ecial treatment to create an eective parallel co de We have develop ed a general purp ose molecular dynamics co de MDynaMix for simulations of arbitrary mixtures of rigid or exible molecules employing the most mo dern simulation techniques double time step algorithm for fast and slow mo des optimized Ewald metho d for electrostatic interactions constant temp erature constant pressure algorithm The program can b e used for sim ulation of mixtures of molecules interacting with AMBER or CHARMMlike force elds The co de is highly universal and well suitable for simula tion of b oth simple molecules and complex biological macromolecules The co de uses only standard Fortran statements b eing able to run on any parallel sys tem with MPIlibrary installed In the latest version additional features were included separate pressure control in dierent directions for simulation of anisotropic systems generalized reaction eld metho d for electrostatic in teractions truncated o ctahedron or hexagonal simulation cell parallel SHAKE algorithm for constrained dynamics dierent typ es of torsional angle p otentials and a few other options Below we describ e details of the program organization parallelization algorithm and p erformance Parallel Molecular Dynamics General Organization Complex molecular systems are often describ ed by force elds like AMBER or CHARMM These force elds contain terms for following interactions atomatom shortrange interactions LennardJones p otential atomatom electrostatic interactions intramolecular interactions covalent b onds covalent angles and torsional angles optional terms hydrogen b onds anharmonic b ond p otential other sp ecic interactions The general functional form of such a force eld is N N X X A q q B ij i j ij U r r 1 N 6 12 r r r ij ij ij ij =1 ij =1 X X 0 2 0 2 K r r K bond ij ang a ij a bonds ang les X 0 K C osm U tor s t t optional t tor sions where r is the distance b etween atoms i and j and other constants dene ij force eld parameters for each chemical atom typ e In principle calculations of pairwise atomatom interactions LennardJones and electrostatic forces ie the rst and the second term in expression require a double sum over all the atom pairs Usually this is the most time consuming part of the force calculations The application of a cuto radius for these interactions allows one to considerably reduce the cpu time by eectively setting the interactions b etween particles separated by distances larger than the cuto distance to zero Two problems however emerge here First the electrostatic interactions are longranged and strictly no cuto without a sp ecial treatment can b e applied Attempts to simulate molecular systems with electrostatic interactions using simple spherical cuto even of a rather large radius may lead to serious artifacts Second in order to decide whether to calculate the forces b etween a given atom pair or not eg whether the atoms are within or out the cuto radius one still should know the distances b etween all atomic pairs or at least to have a list of atom pairs with distances less than the cuto list of neighb ours One of the most eective and p opular metho ds of treatment of electrostatic interactions is the Ewald summation metho d The Ewald metho d splits up the total force into a longrange and short range comp onent The longrange part is calculated in the recipro cal space by the Fourier transform while the shortrange part is treated alongside with the LennardJones forces The convergence of the two parts of the Ewald sum is regulated by a parameter The optimal choice 32 of the convergence parameter leads to a scaling of the cpu time as O N The optimized Ewald metho d is implemented in the program Recently a new version of the Ewald metho d Particle Mesh Ewald has app eared which 4 scales as N l nN for large N We are planning to implement the Particle Mesh Ewald metho d in a future program release Creation of the neighb our lists is another problem In liquids this list should b e up dated p erio dically In the linkedcell metho d the search of neighb oring pairs can b e limited only by the current and the touching cells this leads to an O N algorithm However the true O N algorithm is achieved only at very 3 4 large N for the average size systems of particles p erio dical eg each MD steps up date of neigb ours list by lo oking through all the atom pairs o ccurs eective enough Although the cpu time of this blo ck scales as 2 O N the co ecient is small and for example in a test run with H O 2 molecules atoms this part of the program consumes ab out of the cpu time Irresp ectively to the parallelization an essential saving of cpu time may b e achieved by applying the multiple time scale algorithm Dierent kinds of forces in the system uctuate at dierent characteristic time scales In systems with explicit treatment of hydrogens covalent b ond forces LennardJones and elec trostatic interactions at short distances b etween atoms require an up dated force calculation after fs whereas long range parts of LennardJones and elec trostatic interactions which consume most of cpu time may b e calculated after fs In the present program the two time scale algorithm is applied The covalent forces and atomatom forces for atoms closer than A are calculated at every short time step Forces b etween atoms with distance from A to cuto and recipro cal part of the Ewald sum are calculated at every long time step Corresp ondingly two lists of neighb ours are calculated in the program for fast and for slow forces Parallelization Strategy There are two main strategies in parallelization