INTRODUCTION TO FREEFEM WITH AN EMPHASIS ON PARALLEL COMPUTING Pierre Jolivet http://jolivet.perso.enseeiht.fr/FreeFem-tutorial Browser shortcuts: ◦ Ctrl + f fit to width/height ◦ Ctrl + g go to a page # ◦ Ctrl + PgDwn next page ◦ Ctrl + PgUp previous page v2021.08 INTRODUCTION ACKNOWLEDGEMENTS I ◦ University of Tsukuba, Tokyo, Japan ◦ Institute of Mathematics, University of Seville, Spain ◦ CMAP, École Polytechnique, France 1 ACKNOWLEDGEMENTS II ◦ FreeFEM BDFL Frédéric Hecht ◦ HPDDM feedbacks • Pierre-Henri Tournier • Pierre Marchand ◦ PETSc/SLEPc feedbacks • Johann Moulin • Julien Garaud 2 PREREQUISITES (THE MORE THE BETTER) ◦ FreeFEM [Hecht 2012] https://github.com/FreeFem/FreeFem-sources ◦ Gmsh [Geuzaine and Remacle 2009] http://gmsh.info ◦ ParaView https://www.paraview.org ◦ MPI https://www.mpi-forum.org ◦ HPDDM [Jolivet, Hecht, Nataf, et al. 2013] https://github.com/hpddm/hpddm ◦ PETSc [Balay et al. 1997] https://gitlab.com/petsc/petsc ◦ SLEPc [Hernandez et al. 2005] https://gitlab.com/slepc/slepc 3 PROGRAM OF THE LECTURE 1. Introduction 6. Overlapping Schwarz methods 2. Finite elements 7. Substructuring methods 3. FreeFEM 8. PETSc 4. Shared-memory parallelism 9. SLEPc 5. Distributed-memory parallelism 10. Applications FINITE ELEMENTS u = gD on ΓD @nu = gN on ΓN ◦ essential boundary conditions ◦ natural boundary conditions MODEL PROBLEM y −∆u = f in Ω Ω x 5 @nu = gN on ΓN ◦ natural boundary conditions MODEL PROBLEM y −∆u = f in Ω ΓD = Γ u gD on D Ω x ◦ essential boundary conditions 5 MODEL PROBLEM ΓN y −∆u = f in Ω ΓD = Γ u gD on D Ω @nu = gN on ΓN x ◦ essential boundary conditions ◦ natural boundary conditions 5 ◦ unknown function u ◦ test function v VARIATIONAL FORMULATION Green’s theorem Find u 2 H1(Ω) such that Z Z Z r · r = + ; 8 2 1 (Ω) u v f v gN v v HΓD Ω Ω ΓN u = gD on ΓD 6 VARIATIONAL FORMULATION Green’s theorem Find u 2 H1(Ω) such that Z Z Z r · r = + ; 8 2 1 (Ω) u v f v gN v v HΓD Ω Ω ΓN u = gD on ΓD ◦ unknown function u ◦ test function v 6 1 uh 2 H (Ωh) () uh is continuous ◦ f gNh basis functions 'i i=1 MESH AND FINITE ELEMENTS ◦ Ω discretized by Ω with n elements h P h ◦ Nh u discretized by uh = i=1 uh(i)'i, smooth w.r.t. Ωh 7 ◦ f gNh basis functions 'i i=1 MESH AND FINITE ELEMENTS ◦ Ω discretized by Ω with n elements h P h ◦ Nh u discretized by uh = i=1 uh(i)'i, smooth w.r.t. Ωh 1 uh 2 H (Ωh) () uh is continuous 7 MESH AND FINITE ELEMENTS ◦ Ω discretized by Ω with n elements h P h ◦ Nh u discretized by uh = i=1 uh(i)'i, smooth w.r.t. Ωh 1 uh 2 H (Ωh) () uh is continuous ◦ f gNh basis functions 'i i=1 7 Right-hand side Z Z 8 2 J K i 1; Nh ; bi = f 'i + gN 'i h Ωh ΓN =) numerical integration using quadrature rules ASSEMBLY PROCEDURES Matrix Z 8 2 J K2 r · r (i; j) 1; Nh ; Aij = 'j 'i Ωh =) only integrate in the intersection of both supports 8 ASSEMBLY PROCEDURES Matrix Z 8 2 J K2 r · r (i; j) 1; Nh ; Aij = 'j 'i Ωh =) only integrate in the intersection of both supports Right-hand side Z Z 8 2 J K i 1; Nh ; bi = f 'i + gN 'i h Ωh ΓN =) numerical integration using quadrature rules 8 ◦ symmetric elimination " #" # " # A 0 x b − A gD Ax = GDGD GD = GD GDGD 0 IGDGD xGD gD ◦ penalization " #" # " # A A x b Ax = GDGD GDGD GD = GD + 30 30 AGDGD AGDGD 10 IGDGD xGD 10 gD ESSENTIAL BOUNDARY CONDITIONS GD subset of unknowns associated to Dirichlet BC ◦ nonsymmetric elimination " #" # " # A A x b Ax = GDGD GDGD GD = GD 0 IGDGD xGD gD 9 ◦ penalization " #" # " # A A x b Ax = GDGD GDGD GD = GD + 30 30 AGDGD AGDGD 10 IGDGD xGD 10 gD ESSENTIAL BOUNDARY CONDITIONS GD subset of unknowns associated to Dirichlet BC ◦ nonsymmetric elimination " #" # " # A A x b Ax = GDGD GDGD GD = GD 0 IGDGD xGD gD ◦ symmetric elimination " #" # " # A 0 x b − A gD Ax = GDGD GD = GD GDGD 0 IGDGD xGD gD 9 ESSENTIAL BOUNDARY CONDITIONS GD subset of unknowns associated to Dirichlet BC ◦ nonsymmetric elimination " #" # " # A A x b Ax = GDGD GDGD GD = GD 0 IGDGD xGD gD ◦ symmetric elimination " #" # " # A 0 x b − A gD Ax = GDGD GD = GD GDGD 0 IGDGD xGD gD ◦ penalization " #" # " # A A x b = GDGD GDGD GD = GD Ax 30 30 AG G AGDGD + 10 IGDGD xGD 10 gD D D 9 FREEFEM [LINK TO THE EXAMPLES] HISTORY 1987 MacFem/PCFem by Pironneau 1992 FreeFem by Pironneau, Bernardi, Hecht, and Prud’homme 1996 FreeFem+ by Pironneau, Bernardi, and Hecht 1998 FreeFem++ by Hecht, Pironneau, and Ohtsuka 2008 version 3 2014 version 3.34 with distributed-memory parallelism 2018 moving to GitHub 2019 version 4, rebranded as FreeFEM 10 =) https://community.freefem.org INSTALLATION ◦ packages available for most OSes • Windows • macOS • Debian ◦ compilation from the sources for more flexibility • custom PETSc/SLEPc installation • develop branch ◦ on the cloud • https://www.rescale.com • http://qarnot.com 11 INSTALLATION ◦ packages available for most OSes • Windows • macOS • Debian ◦ compilation from the sources for more flexibility • custom PETSc/SLEPc installation • develop branch ◦ on the cloud • https://www.rescale.com • http://qarnot.com =) https://community.freefem.org 11 STANDARD COMPILATION PROCESS I ◦ remove any previous instance of FreeFEM ◦ install gcc/clang and gfortran ◦ make sure you have a working MPI implementation 12 STANDARD COMPILATION PROCESS II > git clone https://github.com/FreeFem/FreeFem-sources > cd FreeFem-sources > git checkout develop > autoreconf -i > ./configure --enable-download --without-hdf5 --prefix=${PWD} > cd 3rdparty/ff-petsc > make petsc-slepc > cd - > ./reconfigure > make 12 BINARIES ◦ FreeFem++ ◦ FreeFem++-mpi ◦ ffglut ◦ ffmedit ◦ ff-c++ Basic parameters ◦ -ns script not printed ◦ -nw/-wg graphical output deactivated ◦ -v 0 level of verbosity 13 mesh AND mesh3 [EXAMPLE2.EDP] Structured meshes ◦ in 2D, square ◦ in 3D, cube Unstructured meshes ◦ buildmesh ◦ interface with Gmsh, TetGen [Si 2013], and MMG Online visualization ◦ mostly for debugging purposes ◦ prefer medit over plot in 3D 14 fespace [EXAMPLE3.EDP] ◦ formal relationship between a mesh and a FE ◦ one accessible member .ndof ◦ may be used to define finite element functions 15 fespace [EXAMPLE3.EDP] ◦ formal relationship between a mesh and a FE ◦ one accessible member .ndof ◦ may be used to define finite element functions 15 varf AND on [EXAMPLE4.EDP] ◦ bilinear forms ◦ linear forms ◦ boundary conditions ◦ dynamic definition ◦ instantiated to assemble matrices or vectors ◦ qforder to change integration rules 16 varf AND on [EXAMPLE4.EDP] ◦ bilinear forms ◦ linear forms ◦ boundary conditions ◦ dynamic definition ◦ instantiated to assemble matrices or vectors ◦ qforder to change integration rules 16 Essential boundary conditions ◦ tgv = -1 for nonsymmetric elimination ◦ tgv = -2 for symm. elim. (careful about the RHS) ◦ tgv = 1e+30 for penalization matrix AND set [EXAMPLE5.EDP] ◦ assemble a varf using a pair of fespaces ◦ variable sym to assemble the upper triangular part ◦ matrix–vector products ◦ specify a solver for linear systems with set 17 matrix AND set [EXAMPLE5.EDP] ◦ assemble a varf using a pair of fespaces ◦ variable sym to assemble the upper triangular part ◦ matrix–vector products ◦ specify a solver for linear systems with set Essential boundary conditions ◦ tgv = -1 for nonsymmetric elimination ◦ tgv = -2 for symm. elim. (careful about the RHS) ◦ tgv = 1e+30 for penalization 17 real[int] AND OTHER ARRAYS [EXAMPLE6.EDP] ◦ real[int,int] ◦ real[int][int] ◦ complex[int] ◦ matrix[int], string[int] ◦ formal array of finite element functions Common members and methods ◦ .n and .m ◦ .resize ◦ =, +=, /= 18 ADDITIONAL PLUGINS [EXAMPLE7.EDP] ◦ keyword load ◦ load "something" =) ff-c++ -auto something.cpp ◦ FreeFEM objects manipulated in C++/Fortran 19 func [EXAMPLE7.EDP] ◦ user-defined functions ◦ may be passed to external codes ◦ useful for matrix-free computations ◦ LinearCG, EigenValue 20 macro [EXAMPLE7.EDP] ◦ evaluated when parsing input files ◦ defined on the command line -DmacroName=value ◦ conditional statements using IFMACRO 21 SUBMESH [EXAMPLE8.EDP] ◦ trunc optional parameter new2old ◦ + restrict to go from one fespace to another ◦ useful to avoid (costly) interpolations ◦ meshes in multiphysics, e.g., solid + fluid domains 22 SHARED-MEMORY PARALLELISM [LINK TO THE EXAMPLES] MOTIVATION 80 60 40 Time (s) 20 0 Linear solve assembly Mesh generationRight-handMatrix side assembly Solution exporting 23 MOTIVATION 80 60 40 Time (s) 20 0 Linear solve assembly Mesh generationRight-handMatrix side assembly Solution exporting Parallelism is key for performance 23 IMPACT ON THE FINITE ELEMENT METHOD [EXAMPLE1.EDP] Shared-memory parallelism (pthreads, OpenMP) ◦ global address space ◦ minimize critical sections ◦ mostly suited to small-scale architectures 24 IMPACT ON THE FINITE ELEMENT METHOD [EXAMPLE1.EDP] Shared-memory parallelism (pthreads, OpenMP) ◦ global address space ◦ minimize critical sections ◦ mostly suited to small-scale architectures 24 IMPACT ON THE FINITE ELEMENT METHOD [EXAMPLE1.EDP] Shared-memory parallelism (pthreads, OpenMP) ◦ global address space ◦ minimize critical sections ◦ mostly suited to small-scale architectures Color #1: 10 triangles 24 IMPACT ON THE FINITE ELEMENT METHOD [EXAMPLE1.EDP] Shared-memory parallelism (pthreads, OpenMP) ◦ global address space ◦ minimize critical sections ◦ mostly suited to small-scale architectures Color #1: 10 triangles Color #2: 11 triangles 24 IMPACT ON THE FINITE ELEMENT METHOD [EXAMPLE1.EDP] Shared-memory parallelism (pthreads, OpenMP) ◦ global address space ◦ minimize critical sections ◦ mostly suited to small-scale architectures Color #1: 10 triangles Color #2: 11 triangles Color #3: 8 triangles 24 IMPACT ON THE FINITE ELEMENT