Parallel Molecular Dynamics Simulations of
Biomolecular Systems
Alexander Lyubartsev and Aatto Laaksonen
Department of Physical Chemistry Arrhenius Lab oratory
University of Sto ckholm S Sto ckholm Sweden
Abstract We describ e a general purp ose parallel molecular dynamics
co de for simulations of arbitrary mixtures of exible molecules in so
lution The program allows us to simulate molecular systems describ ed
by standard force elds like AMBER GROMOS or CHARMM con
taining terms for short range interactions of the Lennard Jones typ e
electrostatic interactions covalent b onds covalent angles and torsional
angles and a few other optional terms The state of the art molecular dy
namics techniques are implemented constant temp erature and constant
pressure simulations optimized Ewald metho d for treatment of electro
static forces double time step algorithm for separate integration of fast
and slow motions The program is written in standard Fortran and
uses MPI library for communications b etween no des The scalable prop
erties of the program do not dep end on the complexity of the studied
system and are determined mainly by the hardware and communication
sp eed Examples of a few molecular systems di ering by the comp osi
tion will b e given Ionic water solutions large DNA fragments in water
solution with counter ions a phospholipid membrane system
Keywords parallel algorithms molecular dynamics computer simula
tions
Intro duction
Computer simulation metho ds such as Molecular Dynamics MD and Monte
Carlo MC have now b ecome imp ortant techniques to study uids and solids
These metho ds provide a link b etween theory and exp eriment and they are also
the only way to study complex many b o dy systems when b oth exp erimental
techniques and analytical theories are unavailable
The MD metho d provides a numerical solution of classical Newton s equa
tions of motion
2 2
m d r dt F r r
i i i 1 N
where force F r r acting on particle i is de ned by the interaction
i 1 N
p otential or force eld U r r
1 N
F r r U r r
i 1 N 1 N
r i
The most time consuming part sometimes up to of the cpu time of the
MD simulations is the calculations of the forces F r r In a typical case
i 1 N
of pair interactions the cpu time may scale with numb er of particles N b etween
2
O N and O N dep ending on the algorithm and the typ e of interaction p o
tential note that scaling as O N may b e reached only at a very large N and
for systems with short range interactions
Now it is a standard routine pro cedure to simulate molecular systems con
sisting of order of particles which in some cases e g simple liquids
is su cient to give a go o d description of corresp onding macrosystem For other
systems a larger numb er of particles is needed in order to describ e them in a
realistic way Complex bio and organic molecules e g proteins nucleic acids
membranes carb ohydrates etc immersed into a solvent increase the numb er of
involved atoms one to two or more orders of magnitude Also larger the molecu
lar systems grow longer simulations are needed to follow low amplitude motions
and slow conformational transitions It is clear that the rate of the progress to
wards more complex molecular mo dels is set to a large extent by advances in
micropro cessor technology and computer architecture as well as by development
of appropriate software
Computer simulations of many particle systems are well suited for parallel
computer systems The basic reason for this is that the forces acting on each par
ticle can b e calculated indep endently in di erent pro cessors However the most
optimal parallel scheme for a particular problem dep ends b oth on the hardware
in hand and on the system under investigation size typ e of interaction etc
Electrostatic interactions fast intramolecular motions due to explicit mo deling
of hydrogens angle and torsional angle forces of macromolecules all these kinds
of forces require a sp ecial treatment to create an e ective parallel co de
We have develop ed a general purp ose molecular dynamics co de MDynaMix
for simulations of arbitrary mixtures of rigid or exible molecules employing
the most mo dern simulation techniques double time step algorithm for fast and
slow mo des optimized Ewald metho d for electrostatic interactions constant
temp erature constant pressure algorithm The program can b e used for sim
ulation of mixtures of molecules interacting with AMBER or CHARMM like
force elds The co de is highly universal and well suitable for simula
tion of b oth simple molecules and complex biological macromolecules The co de
uses only standard Fortran statements b eing able to run on any parallel sys
tem with MPI library installed In the latest version additional features
were included separate pressure control in di erent directions for simulation
of anisotropic systems generalized reaction eld metho d for electrostatic in
teractions truncated o ctahedron or hexagonal simulation cell parallel SHAKE
algorithm for constrained dynamics di erent typ es of torsional angle p otentials
and a few other options Below we describ e details of the program organization
parallelization algorithm and p erformance
Parallel Molecular Dynamics
General Organization
Complex molecular systems are often describ ed by force elds like AMBER
or CHARMM These force elds contain terms for following interactions
atom atom short range interactions Lennard Jones p otential
atom atom electrostatic interactions
intramolecular interactions covalent b onds covalent angles and torsional
angles
optional terms hydrogen b onds anharmonic b ond p otential other sp eci c
interactions
The general functional form of such a force eld is
N N
X X
A q q B
ij i j ij
U r r
1 N
6 12
r r r
ij
ij ij
i j =1 i j =1
X X
0 2 0 2
K r r K
bond ij ang a
ij a
bonds ang les
X
0
K C os m U
tor s t t optional
t
tor sions
where r is the distance b etween atoms i and j and other constants de ne
ij
force eld parameters for each chemical atom typ e
In principle calculations of pairwise atom atom interactions Lennard Jones
and electrostatic forces i e the rst and the second term in expression
require a double sum over all the atom pairs Usually this is the most time
consuming part of the force calculations The application of a cut o radius for
these interactions allows one to considerably reduce the cpu time by e ectively
setting the interactions b etween particles separated by distances larger than the
cut o distance to zero
Two problems however emerge here First the electrostatic interactions are
long ranged and strictly no cut o without a sp ecial treatment can b e applied
Attempts to simulate molecular systems with electrostatic interactions using
simple spherical cuto even of a rather large radius may lead to serious artifacts
Second in order to decide whether to calculate the forces b etween a given
atom pair or not e g whether the atoms are within or out the cuto radius
one still should know the distances b etween all atomic pairs or at least to have
a list of atom pairs with distances less than the cut o list of neighb ours
One of the most e ective and p opular metho ds of treatment of electrostatic
interactions is the Ewald summation metho d The Ewald metho d splits up the
total force into a long range and short range comp onent The long range part is
calculated in the recipro cal space by the Fourier transform while the short range
part is treated alongside with the Lennard Jones forces The convergence of the
two parts of the Ewald sum is regulated by a parameter The optimal choice
3 2
of the convergence parameter leads to a scaling of the cpu time as O N
The optimized Ewald metho d is implemented in the program Recently a new
version of the Ewald metho d Particle Mesh Ewald has app eared which
4
scales as N l nN for large N We are planning to implement the Particle
Mesh Ewald metho d in a future program release
Creation of the neighb our lists is another problem In liquids this list should
b e up dated p erio dically In the linked cell metho d the search of neighb oring
pairs can b e limited only by the current and the touching cells this leads to
an O N algorithm However the true O N algorithm is achieved only at very
3 4
large N for the average size systems of particles p erio dical e g
each MD steps up date of neigb ours list by lo oking through all the atom
pairs o ccurs e ective enough Although the cpu time of this blo ck scales as
2
O N the co e cient is small and for example in a test run with H O
2
molecules atoms this part of the program consumes ab out of the cpu
time
Irresp ectively to the parallelization an essential saving of cpu time may b e
achieved by applying the multiple time scale algorithm Di erent kinds of forces
in the system uctuate at di erent characteristic time scales In systems with
explicit treatment of hydrogens covalent b ond forces Lennard Jones and elec
trostatic interactions at short distances b etween atoms require an up dated force
calculation after fs whereas long range parts of Lennard Jones and elec
trostatic interactions which consume most of cpu time may b e calculated after
fs In the present program the two time scale algorithm is applied The
covalent forces and atom atom forces for atoms closer than A are calculated at
every short time step Forces b etween atoms with distance from A to cut o
and recipro cal part of the Ewald sum are calculated at every long time step
Corresp ondingly two lists of neighb ours are calculated in the program for fast
and for slow forces
Parallelization Strategy
There are two main strategies in parallelization of molecular dynamics programs
which called Replicated data and Domain decomp osition In the replicated
data metho d all the no des know p ositions of all the particles in the system
while calculation on di erent contributions to forces go es in parallel The ad
vantages of this metho d is a relative simplicity to distribute force calculations
over pro cessors di erent contributions to the sum have a form of a sum of
a large numb er of similar indep endent terms i e sums over atom pairs cova
lent b ond covalent angles etc Such calculations can b e done in parallel on
di erent pro cessors equally e ective for virtually all kinds of the force elds and
molecular structures The weak p oint of the replicated data approach is that it
requires relatively extensive communications b etween pro cessors to collect data
on contribution to the forces from all the no des and to distribute new particle s
p ositions to all the no des In the Domain decomp osition approach particles are
distributed b etween the no des usually each no de b eing resp onsible for the par
ticles in the corresp onding sub cell The communication overhead may b e lower
in this case comparing to the replicated data metho d but other sources of over
heads arise due to necessity to monitor moving of particles through the sub cells
to implement Ewald summation of electrostatic interactions to get use of the
Newton s rd law etc An addition of extra functionality to the program in the
domain decomp osition metho d may also lead to essential increase of cpu time
b oth for calculations and for communications
It is b elieved that the domain decomp osition metho d is more e ective for
massively parallel computer systems to run simulation programs for particles
interacting with simple short range p otentials for parallel simulation of complex
biomolecular systems on computers with several or several dozens pro cessors the
domain decomp osition metho d is preferable In our molecular dynamics program
the replicated data metho d is implemented
Parallelization of Sp eci c Parts of the Program
For an e ective parallelization of non b onded interactions rst and second sum
in the whole list of atom pairs I J N I J should b e uniformly di
vided b etween the available numb er of no des The condition I J is set to avoid
a double count but it makes di cult to divide equally atom pairs b etween the
no des To avoid this the following scheme was applied All particles are divided
equally b etween pro cessors lo op do I T AS K I D N N U M T AS K where
T AS K I D no de numb er N U M T AS K total numb er of available no des For
each I we have an inner lo op over J with the following condition I J is even
for I J and J I is o dd for I J So the atom pairs o ccur uniformly dis
tributed b etween no des and each one is treated only once Then for each no de
lists of closest r A and far r R neighb ours are calculated These
cut
lists are lo cal for each no de and are recalculated after ab out long time steps
They are used for the Lennard Jones and electrostatic real space part force
calculations Using of lo cal list of neighb ours on each no de makes it p ossible to
use the same co de b oth for parallel and single pro cessor architecture Moreover
since the list of neighb ours is the biggest data structure in the program dis
tributing it over pro cessors reduce the total memory requirements or makes it
p ossible to simulate much bigger molecular systems with the same memory limit
p er pro cessor
Parallelization of the recipro cal part of the Ewald sum is more or less straight
forward This contribution is expressed as a sum over recipro cal space vectors
Each no de calculates force contributions due to a certain xed for this no de
group of recipro cal vectors
Calculations of forces due to covalent b onds angles and torsions can b e
done indep endently for each b ond angle or dihedral angle At the b eginning of
simulations all the b onds covalent angles and torsions are distributed uniformly
over available no des Each no de gets its own list of b onds angles and torsions and
corresp onding contributions to the forces and energies are calculated in parallel
Di erent forces acting on each atom from other atoms are accumulated rst
on each no de separately After completing the calculations on each no de all
the forces acting on a given atom are transfered to the no de corresp onding to
this atom and summed up The whole op eration is made by a single MPI call
MPI REDUCE SCATTER which dep ending on the implementation may work
essentially faster than consecutive summing up forces REDUCE followed by
their distribution over the no des SCATTER Then on each no de the atoms are
moved according to a chosen integration scheme constant energy NVE con
stant temp erature NVT or constant pressure and temp erature NPT molecu
lar dynamics The double time step algorithm is easily integrated into this
scheme After completing an MD step new p ositions of atoms are broadcasted
to all no des
Portability
The program is written in standard Fortran and highly p ortable b eing able to
run on any parallel system with MPI library installed It can b e also easily made
run on a single pro cessor computer So far the program was tested on parallel
systems IBM SP Cray T E PC linux clusters employing MPI calls through
an ethernet network as well as on single pro cessor systems IBM RISC
DEC Alpha PC linux Porting the co de to another computer system normally
requires only small changes of the Make le
Examples
The p erformance of the program was tested in details on several molecular sys
tems
A p erio dic fragment of DNA in ionic aqueous solution one helical turn
of DNA atoms water molecules N a and C l ions atoms
totally In other simulations Na ions was substituted by Li or Cs ions
An NaCl ion solution NaCl ion pairs and water molecules
atoms totally
A lipid bilayer DPPC lipid molecules atoms each and water
molecules atoms totally
The three molecular systems are qualitatively di erent on the comp osition
the rst one contains one macromolecule in solution the second is a mixture
of small molecules the third one is an inhomogeneous system consisting of a
numb er of middle size molecules and waters
The dep endence of the total computation time on the numb er of no des in
each case is given in Table In all the cases the double time step algorithm with
a short step of fs and a long step of fs was used The long range electrostatic
interactions were taken into account by the Ewald metho d The cut o in the
recipro cal space was determined from the conditions that the remaining terms
were less then relative to the total electrostatic energy
The calculations were p erformed on IBM SP Strindb erg at PDC KTH and
on CRAY T E in the National Sup ercomputer Center Linko ping The program
shows go o d scaling prop erties in all cases running on up to pro cessors
Table CPU time in min for ps simulation of di erent molecular systems
dep ending on numb er of pro cessors Box sizes and cuto distance R are given in A
cut
no des
System Num atoms Box sizes R
cut
IBM SP Mhz pro c
DNA x x
Ion solution x x
lipid bilayer x x
Cray T E
DNA x x
lipid bilayer x x
Pentium Mhz
lipid bilayer x x
We have used the MDynaMix program for practical simulations of these
molecular systems on the nanosecond time scale Moreover for the DNA solu
tion simulations were carried out with several di erent counterion typ es Both
structural and dynamical prop erties of the molecules are studied The reader is
referred to the corresp onding articles for details
Acknowledgements
We are grateful to the Center for parallel Computing at the Royal Institute of
Technology Sto ckholm and National Sup ercomputing Center Linko ping for
granting the computer facilities and the Swedish National Science Foundation
NFR and the Foundation for Strategic Research SSF for nancial supp ort
References
http www fos su se physical sasha md prog html
Weiner S J Kollman P A Ngyuen D T Case D A An All Atom Force Field
for Simulations of Proteins and Nucleic Acids J Comput Chem
MacKerell A D Wiorkiewicz Kuczera J Karplus M An All Atom Empirical
Energy Function for the Simulation of Nucleic Acids J Am Chem So c
Tironi I G Sp erb R Smith P E van Gunsteren W F A Generalized Reaction
Field Metho d for Molecular Dynamics Simulations J Chem Phys
Allen M P Tildesley D J Computer Simulation of Liquids Clarendon Oxford
Darden T York D Pedersen L Particle Mesh Ewald A N l nN Metho d for
Ewald Sums in Large Systems J Chem Phys
Martyna G I Tuckerman M E Tobais D J Klein M L Explicit Reversible
Integrators for Extended Systems Dynamics Mol Phys
Tuckerman M E Berne B J Martyna G J Reversible Multiple Time Scale
Molecular Dynamics J Chem Phys
Lyubartsev A P Laaksonen A Concentration E ects in Aqueous NaCl Solu
tions A Molecular Dynamics Simulations J Phys Chem
Lyubartsev A P Laaksonen A Molecular Dynamics Simulations of DNA in Pres
ence of Di erent Counterions J Biomolec Structure and Dynamics in press