Summer School CEA-EDF-INRIA Toward petaflop numerical simulation on parallel hybrid architectures BigDFT and GPU INRIA CENTRE DE RECHERCHE SOPHIA ANTIPOLIS,FRANCE Atomistic Simulations DFT Ab initio codes BigDFT Wavelet-Based DFT calculations on Massively Properties BigDFT and GPUs Parallel Hybrid Architectures Code details BigDFT and HPC GPU Luigi Genovese Practical cases Discussion Messages L_Sim – CEA Grenoble June 9, 2011 Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim Luigi Genovese Outline 1 Review of Atomistic Simulations Density Functional Theory Ab initio codes BigDFT and 2 The BigDFT project GPU Formalism and properties Atomistic Simulations The needs for hybrid DFT codes DFT Main operations, parallelisation Ab initio codes BigDFT 3 Properties Performance evaluation BigDFT and GPUs Code details Evaluating GPU gain BigDFT and Practical cases HPC GPU Practical cases 4 Concrete examples Discussion Messages Messages Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim Luigi Genovese Review of Atomistic Simulations A interdisciplinary domain Theory – Experiment – Simulation Hardware – Computers BigDFT and Algorithms GPU Atomistic Simulations Different Atomistic Simulations DFT Ab initio codes Force fields (interatomic potentials) BigDFT Properties Tight Binding Methods BigDFT and GPUs Code details Hartree-Fock BigDFT and HPC Density Functional Theory GPU Practical cases Configuration interactions Discussion Messages Quantum Monte-Carlo Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim Luigi Genovese Quantum mechanics for many particle systems Can we do quantum mechanics on systems of many atoms? Decoupling of the nuclei and electron dynamics Born-Oppenheimer approximation: The position of the nuclei can be considered as fixed, BigDFT and obtaining the potential “felt” by the electrons GPU n Za Atomistic Vext(r;fR1;··· ;Rng) = − ∑ Simulations a=1 jr − Raj DFT Ab initio codes BigDFT Electronic Schrödinger equation Properties BigDFT and GPUs The system properties are described by the ground state Code details BigDFT and wavefunction y(r1;··· ;rN ), which solves Schrödinger HPC equation GPU Practical cases H [fRg]y = Ey Discussion Messages The quantum hamiltonian depends on the set of the atomic positions fRg. Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim Luigi Genovese Atomistic Simulations Two intrinsic difficulties for numerical atomistic simulations, related to complexity: Interactions The way that atoms interact is known: ¶Ψ BigDFT and i = H Ψ H y = E0y GPU } ¶t Exploration of the configuration space Atomistic Simulations DFT Ab initio codes BigDFT Properties BigDFT and GPUs Code details BigDFT and HPC GPU Practical cases Discussion Messages Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim Luigi Genovese Atomistic Simulations Two intrinsic difficulties for numerical atomistic simulations, related to complexity: Interactions The way that atoms interact is known: ¶Ψ BigDFT and i = H Ψ H y = E0y GPU } ¶t Exploration of the configuration space Atomistic Simulations E DFT pot Ab initio codes BigDFT Properties BigDFT and GPUs Code details BigDFT and R1 HPC GPU Practical cases Discussion Messages Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim Luigi Genovese Atomistic Simulations Two intrinsic difficulties for numerical atomistic simulations, related to complexity: Interactions The way that atoms interact is known: ¶Ψ BigDFT and i = H Ψ H y = E0y GPU } ¶t Exploration of the configuration space Atomistic Simulations E DFT pot Ab initio codes BigDFT Properties BigDFT and GPUs Code details BigDFT and R1 HPC GPU Practical cases R2 Discussion Messages Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim Luigi Genovese Atomistic Simulations Two intrinsic difficulties for numerical atomistic simulations, related to complexity: Interactions The way that atoms interact is known: ¶Ψ BigDFT and i = H Ψ H y = E0y GPU } ¶t Exploration of the configuration space Atomistic Simulations E DFT pot Ab initio codes BigDFT Properties BigDFT and GPUs Code details BigDFT and R1 HPC GPU Practical cases R3 R2 Discussion Messages Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim Luigi Genovese Atomistic Simulations Two intrinsic difficulties for numerical atomistic simulations, related to complexity: Interactions The way that atoms interact is known: ¶Ψ BigDFT and i = H Ψ H y = E0y GPU } ¶t Exploration of the configuration space Atomistic Simulations E DFT pot Ab initio codes R1000 BigDFT Properties BigDFT and GPUs Code details BigDFT and R1 HPC GPU R41 R3 Practical cases Rn R2 Discussion Messages Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim Luigi Genovese Choice of Atomistic Methods 3 criteria 1! Generality (elements, alloys) G P 2! Precision (∆r, ∆E) 3! System size (N, ∆t) BigDFT and S GPU Atomistic Chemistry and Physics Simulations DFT Ab initio codes Force Fields BigDFT G P G P G P Properties Tight Binding BigDFT and GPUs Code details S S S Hartree-Fock BigDFT and HPC DFT GPU G P G P G P Practical cases Conf. Inter. Discussion Messages S S S Quantum Monte-Carlo Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim Luigi Genovese Choice of Atomistic Methods 3 criteria 1! Generality (elements, alloys) G P 2! Precision (∆r, ∆E) 3! System size (N, ∆t) BigDFT and S GPU Atomistic Chemistry and Physics Simulations DFT Ab initio codes Force Fields BigDFT G P G P G P Properties Tight Binding BigDFT and GPUs Code details S S S Hartree-Fock BigDFT and HPC DFT GPU G P G P G P Practical cases Conf. Inter. Discussion Messages S S S Quantum Monte-Carlo Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim Luigi Genovese Choice of Atomistic Methods 3 criteria 1! Generality (elements, alloys) G P 2! Precision (∆r, ∆E) 3! System size (N, ∆t) BigDFT and S GPU Atomistic Chemistry and Physics Simulations DFT Ab initio codes Force Fields BigDFT G P G P G P Properties Tight Binding BigDFT and GPUs Code details S S S Hartree-Fock BigDFT and HPC DFT GPU G P G P G P Practical cases Conf. Inter. Discussion Messages S S S Quantum Monte-Carlo Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim Luigi Genovese Choice of Atomistic Methods 3 criteria 1! Generality (elements, alloys) G P 2! Precision (∆r, ∆E) 3! System size (N, ∆t) BigDFT and S GPU Atomistic Chemistry and Physics Simulations DFT Ab initio codes Force Fields BigDFT G P G P G P Properties Tight Binding BigDFT and GPUs Code details S S S Hartree-Fock BigDFT and HPC DFT GPU G P G P G P Practical cases Conf. Inter. Discussion Messages S S S Quantum Monte-Carlo Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim Luigi Genovese Choice of Atomistic Methods 3 criteria 1! Generality (elements, alloys) G P 2! Precision (∆r, ∆E) 3! System size (N, ∆t) BigDFT and S GPU Atomistic Chemistry and Physics Simulations DFT Ab initio codes Force Fields BigDFT G P G P G P Properties Tight Binding BigDFT and GPUs Code details S S S Hartree-Fock BigDFT and HPC DFT GPU G P G P G P Practical cases Conf. Inter. Discussion Messages S S S Quantum Monte-Carlo Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim Luigi Genovese Choice of Atomistic Methods 3 criteria 1! Generality (elements, alloys) G P 2! Precision (∆r, ∆E) 3! System size (N, ∆t) BigDFT and S GPU Atomistic Chemistry and Physics Simulations DFT Ab initio codes Force Fields BigDFT G P G P G P Properties Tight Binding BigDFT and GPUs Code details S S S Hartree-Fock BigDFT and HPC DFT GPU G P G P G P Practical cases Conf. Inter. Discussion Messages S S S Quantum Monte-Carlo Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim Luigi Genovese The Hohenberg-Kohn theorem A tremendous numerical problem N 1 1 1 H = − ∇2 + V (r ;fRg) + ∑ ri ext i ∑ i=1 2 2 i6=j jri − rj j BigDFT and The Schrödinger is very difficult to solve for more than two GPU electrons! Another approach is imperative Atomistic Simulations The fundamental variable of the problem is however not the DFT Ab initio codes wavefunction, but the electronic density BigDFT Z ∗ Properties r(r) = N dr2 ···drN y (r;r2;··· ;rN )y(r;r2;··· ;rN ) BigDFT and GPUs Code details BigDFT and Hohenberg-Kohn theorem (1964) HPC GPU Practical cases The ground state density r(r) of a many-electron system Discussion uniquely determines (up to a constant) the external potential . Messages The external potential is a functional of the density Vext = Vext[r] Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim Luigi Genovese The Kohn-Sham approximation Given the H-K theorem, it turns out that the total electronic energy is an unknown functional of the density E = E[r] =) Density Functional Theory DFT (Kohn-Sham approach) BigDFT and Mapping of a interacting many-electron system into a system GPU with independent particles moving into an effective potential. Atomistic Simulations DFT Find a set of orthonormal orbitals Ψ (r) that minimizes: Ab initio codes i BigDFT Properties N=2 1 Z 1 Z BigDFT and GPUs E = − Ψ∗(r)∇2Ψ (r)dr + r(r)V (r)dr Code details ∑ i i H 2 i=1 2 BigDFT and HPC Z GPU + Exc[r(r)]+ Vext (r)r(r)dr Practical cases Discussion N=2 Messages ∗ 2 r(r) = 2 ∑ Ψi (r)Ψi (r) ∇ VH (r) = −4pr(r) i=1 Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim Luigi Genovese Ab initio calculations with DFT Several advantages Main limitations 4 Ab initio: No 8 Approximated approach adjustable parameters 8 Requires high computer 4 DFT: Quantum power, limited to few mechanical BigDFT and hundreds atoms in most GPU (fundamental) cases Atomistic treatment Simulations DFT Ab initio codes Wide range of applications: nanoscience, biology, materials BigDFT Properties BigDFT and GPUs Code details BigDFT and HPC GPU Practical cases Discussion Messages Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim Luigi Genovese Performing a DFT calculation A self-consistent equation Then r = 2∑i hyi jyi i, where jyi i satisfies 1 2 ∇ + VH [r] + Vxc[r] + Vext + Vpseudo jyi i = ∑Λi;j jyj i ; 2 j BigDFT and GPU Now in practice: implementing a DFT code Atomistic Simulations (Kohn-Sham) DFT “Ingredients” DFT Ab initio codes An XC potential, functional of the density BigDFT Properties several approximations exists (LDA,GGA,.