Electronic Structure Calculation Methods on Accelerators, Oak Ridge BigDFT Formalism Wavelets Algorithms Daubechies wavelets as a basis set for density Poisson Solver Features functional pseudopotential calculation Applications Publications Si clusters Boron clusters Conclusion T. Deutsch, L. Genovese, D. Caliste L_Sim – CEA Grenoble February 7, 2012 Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim T. Deutsch, L. Genovese, D. Caliste A basis for nanosciences: The BigDFT project STREP European project: BigDFT(2005-2008) Four partners, 15 contributors: CEA-INAC Grenoble (T. D.), U. Basel (S. Goedecker), U. Louvain-la-Neuve (X .Gonze), U. Kiel (R. Schneider) BigDFT Formalism Aim: To develop an ab-initio DFT code Wavelets based on Daubechies Wavelets Algorithms Poisson Solver and integrated in ABINIT (Gnu-GPL). Features Applications BigDFT 1.0 −! January 2008 Publications Si clusters Boron clusters Conclusion . Why have we done this? Test the potential advantages of a new formalism * A lot of outcomes and interesting results A lot can be done starting from present know-how Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim T. Deutsch, L. Genovese, D. Caliste Outline 1 Formalism Wavelets Algorithms Poisson Solver BigDFT Features Formalism Wavelets Algorithms 2 Applications Poisson Solver Features Publications Applications Si clusters Publications Si clusters Boron clusters Boron clusters Conclusion 3 Conclusion Next talk: Performances (MPI+OpenMP+OpenCL) Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim T. Deutsch, L. Genovese, D. Caliste A DFT code based on Daubechies wavelets BigDFT: a PSP Kohn-Sham code A Daubechies wavelets basis has unique properties for DFT usage Systematic, Orthogonal BigDFT Localised, Adaptive Formalism Wavelets Algorithms Kohn-Sham operators are analytic Poisson Solver Daubechies Wavelets Features Short, Separable convolutions 1.5 Applications φ(x) Publications ~c = a c ` ∑j j `−j 1 ψ(x) Si clusters Boron clusters 0.5 Peculiar numerical properties Conclusion 0 Real space based, highly flexible -0.5 -1 Big & inhomogeneous systems -1.5 -6 -4 -2 0 2 4 6 8 x Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim T. Deutsch, L. Genovese, D. Caliste A brief description of wavelet theory Two kind of basis functions A Multi-Resolution real space basis The functions can be classified following the resolution level they span. BigDFT Formalism Scaling Functions Wavelets Algorithms The functions of low resolution level are a linear combination Poisson Solver Features of high-resolution functions Applications Publications Si clusters = + Boron clusters m Conclusion f(x) = ∑ hj f(2x − j) j=−m Centered on a resolution-dependent grid: fj = f0(x − j). Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim T. Deutsch, L. Genovese, D. Caliste A brief description of wavelet theory Wavelets They contain the DoF needed to complete the information which is lacking due to the coarseness of the resolution. = 1 + 1 BigDFT 2 2 m m Formalism (2x) = h~ (x − j) + g~ (x − j) Wavelets f ∑ j f ∑ j y Algorithms j=−m j=−m Poisson Solver Features Applications Increase the resolution without modifying grid space Publications Si clusters Boron clusters SF + W = same DoF of SF of higher resolution Conclusion m y(x) = ∑ gj f(2x − j) j=−m All functions have compact support, centered on grid points. Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim T. Deutsch, L. Genovese, D. Caliste Wavelet properties: Adaptivity Adaptivity Resolution can be refined following the grid point. The grid is divided in BigDFT Low (1 DoF) and High Formalism (8 DoF) resolution points. Wavelets Algorithms Points of different resolution Poisson Solver belong to the same grid. Features Applications Empty regions must not be Publications Si clusters “filled” with basis functions. Boron clusters Conclusion Localization property,real space description Optimal for big & inhomogeneous systems, highly flexible Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim T. Deutsch, L. Genovese, D. Caliste Adaptivity of the mesh Atomic positions (H2O), hgrid BigDFT Formalism Wavelets Algorithms Poisson Solver Features Applications Publications Si clusters Boron clusters Conclusion Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim T. Deutsch, L. Genovese, D. Caliste Adaptivity of the mesh Fine grid (high resolution, frmult) BigDFT Formalism Wavelets Algorithms Poisson Solver Features Applications Publications Si clusters Boron clusters Conclusion Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim T. Deutsch, L. Genovese, D. Caliste Adaptivity of the mesh Coarse grid (low resolution, crmult) BigDFT Formalism Wavelets Algorithms Poisson Solver Features Applications Publications Si clusters Boron clusters Conclusion Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim T. Deutsch, L. Genovese, D. Caliste No integration error Orthogonality, scaling relation Z 1 m dx fk (x)fj (x) = dkj f(x) = p ∑ hj f(2x − j) 2 j=−m BigDFT The hamiltonian-related quantities can be calculated up to Formalism machine precision in the given basis. Wavelets Algorithms The accuracy is only limited by the basis set ( h14 ) Poisson Solver O grid Features Applications Publications Exact evaluation of kinetic energy Si clusters Boron clusters Obtained by convolution with filters: Conclusion 2 f (x) = ∑c`f`(x); ∇ f (x) = ∑~c` f`(x) ; ` ` Z 2 ~c` = ∑cj a`−j ; a` ≡ f0(x)¶x f`(x) ; j Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim T. Deutsch, L. Genovese, D. Caliste Separability in 3D The 3-dim scaling basis is a tensor product decomposition of 1-dim Scaling Functions/ Wavelets. e ;e ;e e f x y z (x;y;z) = fex (x)f y (y)fez (z) jx ;jy ;jz jx jy jz BigDFT With (jx ;jy ;jz ) the node coordinates, (0) (1) Formalism fj and fj the SF and the W respectively. Wavelets Algorithms Poisson Solver Features Gaussians and wavelets Applications Publications The separability of the basis allows us to save computational Si clusters Boron clusters time when performing scalar products with separable Conclusion functions (e.g. gaussians): Initial wavefunctions (input guess) 1 Poisson solver (tensor decomposition of r ) Non-local pseudopotentials Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim T. Deutsch, L. Genovese, D. Caliste Wavelet families used in BigDFT code Daubechies f (x) = ∑` c`f`(x) Interpolating f (x) = ∑j fj jj (x) R Orthogonal c` = dx f`(x)f (x) Dual to dirac deltas fj = f (j) LEAST ASYMMETRIC DAUBECHIES-16 1.5 INTERPOLATIN !"E#LAURIERS-"$%$&'-( scaling function 1 wavelet scaling function wavelet 1 0.5 0.5 BigDFT 0 0 Formalism -0.5 Wavelets -0.5 Algorithms -1 Poisson Solver Features -1.5 -1 -6 -4 -2 0 2 4 6 -4 -2 0 2 4 Applications Used for wavefunctions, Used for charge density, Publications Si clusters scalar products function products Boron clusters Conclusion “Magic Filter” method (A. Neelov, S. Goedecker) The passage between the two basis sets can be performed without losing accuracy. Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim T. Deutsch, L. Genovese, D. Caliste Operations performed The SCF cycle Real Space Daub. Wavelets Orbital scheme: Hamiltonian Preconditioner BigDFT Coefficient Scheme: Formalism Wavelets Overlap matrices Algorithms Poisson Solver Features Orthogonalisation Applications Publications Comput. operations Si clusters Boron clusters Convolutions Conclusion BLAS routines FFT (Poisson Solver) Why not GPUs? Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim T. Deutsch, L. Genovese, D. Caliste Kohn-Sham Equations: Computing Energies Calculate different integrals E[r] = K [r] + U[r] BigDFT 2 Z 2 ~ ∗ 2 Formalism K [r] = − ∑ dr yi ∇ yi Wavelets 2 me i V Algorithms Poisson Solver Features Z Z 0 1 0 r(r)r(r ) Applications U[r] = dr Vext (r)r(r)+ dr dr 0 + Exc[r] Publications V 2 V jr − r j | {z } Si clusters | {z } exchange−correlation Boron clusters Hartree Conclusion We minimise with the variables yi (r) and fi Z with the constraint dr r(r) = Nel . V Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim T. Deutsch, L. Genovese, D. Caliste inv FWT + Magic Filter occ 2 r(r) = ∑ jyi (r)j Fine non-adaptive mesh: < 8Nf i Poisson solver V (r) = R G(:)r Vxc [r(r)] H VNL(fyi g) Veffective FWT 1 2 − 2 ∇ FWT Kinetic Term i ¶Etotal j dca = − ∗ + ∑Λij ca ¶ci (a) j Λij =< yi jHjyj > i Stop when dca small preconditioning Steepest Descent, new;i i i ca = ca + hstepdca DIIS Direct Minimisation: Flowchart i yi = ∑cafa Basis: adaptive mesh (0;:::;Nf) a Orthonormalized BigDFT Formalism Wavelets Algorithms Poisson Solver Features Applications Publications Si clusters Boron clusters Conclusion Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim T. Deutsch, L. Genovese, D. Caliste Poisson solver V (r) = R G(:)r Vxc [r(r)] H VNL(fyi g) Veffective FWT 1 2 − 2 ∇ FWT Kinetic Term i ¶Etotal j dca = − ∗ + ∑Λij ca ¶ci (a) j Λij =< yi jHjyj > i Stop when dca small preconditioning Steepest Descent, new;i i i ca = ca + hstepdca DIIS Direct Minimisation: Flowchart i yi = ∑cafa Basis: adaptive mesh (0;:::;Nf) a Orthonormalized inv FWT + Magic Filter occ 2 BigDFT r(r) = ∑ jyi (r)j Fine non-adaptive mesh: < 8Nf i Formalism Wavelets Algorithms Poisson Solver Features Applications Publications Si clusters Boron clusters Conclusion Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim T. Deutsch, L. Genovese, D. Caliste FWT 1 2 − 2 ∇ FWT Kinetic Term i ¶Etotal j dca = − ∗ + ∑Λij ca ¶ci (a) j Λij =< yi jHjyj > i Stop when dca small preconditioning Steepest Descent, new;i i i ca = ca + hstepdca DIIS Direct Minimisation: Flowchart i yi = ∑cafa Basis: adaptive mesh (0;:::;Nf) a Orthonormalized inv FWT + Magic Filter occ 2 BigDFT r(r) = ∑ jyi (r)j Fine non-adaptive mesh: < 8Nf i Formalism Wavelets Algorithms Poisson solver Poisson Solver Features V (r) = R G(:)r Applications Vxc [r(r)] H VNL(fyi g) Veffective Publications Si clusters Boron clusters Conclusion Laboratoire de Simulation Atomistique http://inac.cea.fr/L_Sim T.