EOF-Library: Open-source Elmer FEM and OpenFOAM
coupler for electromagnetics and fluid dynamics
(CC BY-SA 4.0) Motivation for MHD - controlling liquid metals
Liquid metal pump
rotating permanent pipe magnets
F1
F2
F
FL
FL Mechanically FL FL Electromagnetically Required physics
Modelling coupled
● Electromagnetics ● Fluid dynamics
Extra
● Free surface ● Heat transfer Liquid metal pump - modelling requires two-way ● Solidification / melting coupled electromagnetics and fluid dynamics
1 Required numerical models
Fluid dynamics ● 3D, transient Electromagnetics ● High Reynolds ● Complex geometries, turbulence models multi-regions ● Complex A−V ● Volume of Fluid magnetic vector ● Advanced pre- and potential (VOF) for free post-processing tools surface modelling formulation ● Parallelization with ● Conductivity ● Viscosity good scaling dependence on dependence on temperature ● User community, temperature documentation 2 Existing Open Source codes
Best for Best for Electromagnetics (EM) Fluid dynamics (FD)
FEM FVM getDP OpenFOAM Elmer SU2 ...... 3 Choosing Electromagnetics solver (1)
GetDP + Designed for EM in mind, great for eddy-current problems + Uses PETSc iterative solvers, PETSc is MPI parallelized
- Matrix assembly is not parallel (neither MPI nor OpenMP) - Weak integration with external post-processing tools
MaxFEM, FreeFem++, deal.II, … Pictures from getdp.info ● I have no opinion 4 Choosing Electromagnetics solver (2)
Elmer by CSC (Finland)
+ Multiphysics out-of-the-box (>40 solvers) + Build-in linear & nonlinear iterative solvers, fully parallelized + Build-in computation of 3D electrical currents resulting from VxB term - very important for MHD and rotating magnets + Extensive documentation, active forum, responsive developers
- Adding new functionality requires knowledge in FEM and Fortran
5 Choosing Fluid dynamics solver
● In terms of popularity and functionality OpenFOAM is leader + Most important - it has Volume of Fluid (VOF) + OpenFOAM = solvers + examples + community + impact
6 No one code solves our problem
.., but two coupled codes can!
Elmer + OpenFOAM
● Both can simulate real stuff ● Common pre/post processing (Salome, Paraview) ● Straight-forward mesh interpolation
● Most important - Parallel MPI-based coupling is feasible 7 Elmer OpenFOAM = EOF-Library
Two-way coupling
File based ● SLOW and does not SCALE on parallel computers
MPI (Message Passing Interface) based ● Both codes already use MPI for parallelization ● Probably the most EFFICIENT coupling solution 8 MPI - Message Passing Interface
Running on 16 cores: mpirun -n 16 Elmer : -n 16 OpenFOAM
“Wrapper Elmer OpenFOAM (16 processes) (16 processes) that starts applications MPI_SEND( MPI_RECV( data, data, in parallel” type, type, size, size, to whom) from whom)
MPI_RECV(...) MPI_SEND(...)
9 In the code
Interpolating and sending fields from Elmer to OpenFOAM
Checks whether Elmer sends message, if not then sleeps 10ms (saves CPU resources) Projector matrix (interpolation) is applied on variable
10 Structure of EOF-Library
Patch for splitting MPI_COMM_WORLD on Coupler’s OpenFOAM part OpenFOAM side (every OF version has its own patch) ├── coupleElmer │ ├── Elmer.C │ ├── Elmer.H ├── commSplit │ └── Make │ ├── allReduce.H │ ├── files │ ├── allReduceTemplates.C │ └── options │ ├── Make │ │ ├── files │ │ └── options Coupler’s Elmer part │ ├── PstreamGlobals.C │ ├── PstreamGlobals.H ├── Elmer2OpenFOAM │ ├── UIPread.C │ └── Elmer2OpenFOAM.F90 │ ├── UOPwrite.C └── OpenFOAM2Elmer │ └── UPstream.C └── OpenFOAM2Elmer.F90
11 Example for modified OpenFOAM solver
├── mhdInterFoam │ ├── alphaCourantNo.H │ ├── alphaEqn.H Current field value │ ├── alphaEqnSubCycle.H │ ├── correctPhi.H Field value when │ ├── createFields.H Elmer was called │ ├── Make last time │ │ ├── files │ │ └── options │ ├── mhdInterFoam.C │ ├── pEqn.H │ ├── setDeltaT.H │ ├── setRDeltaT.H │ └── UEqn.H
User is responsible about physics 12 Simulation workflow using open-source software
EOF-Library
CAD & meshing Post-processing
Salome Elmer FEM OpenFOAM Paraview Possible* coupling combinations * not all of them are tested
OpenFOAM Elmer FEM
Liquid Gas Heat Induction Microwaves f(T)
Parametric Rarified Particle Steady & dependence gas tracking transient Radiation Application examples
Electromagnetic levitation melting Surface wave generation using low frequency EM field Interpolation test - unit cube
Elmer & OpenFOAM Elmer-to-OpenFOAM meshes are different Tet size 0.1 0.05 0.025
Relative 8.0 2.3 0.57 Refinement level Error % ● 0.1 (4.7k tets) ● 0.05 (32k tets) ● 0.025 (245k tets) OpenFOAM-to-Elmer
Tet size 0.1 0.05 0.025 Reference scalar field Relative 11 4.0 1.5 2 sin[2πx] cos[2πy] Error % Strong scaling test
13.1M cells / elements
Conclusion: Coupler makes a negligible performance overhead comparing to computation time Finally - obtaining the code
https://github.com/jvencels
I typically use Salome for CAD and meshing
Useful salome script for exporting mixed (tet+hex) meshes directly to Elmer
Similar script for OpenFOAM https://github.com/nicolasedh/s alomeToOpenFOAM
13 EOF-Library.com
● Research ● Development ● Support