Gkyl Documentation Release 2.0-Alpha

gkyl Documentation Release 2.0-alpha

Ammar Hakim

Sep 24, 2021

Contents

1 Installing Gkeyll 3 1.1 gkyl install...... 4 1.2 Postgkyl install...... 7 1.3 Note on the Windows Linux Subsystem (WSL)...... 9

2 Quickstart 13 2.1 A ﬁrst Gkeyll simulation...... 13 2.2 Vlasov example...... 18 2.3 Gyrokinetic example...... 26 2.4 Fluid example...... 39

3 gkyl Reference 47 3.1 Using gkyl...... 47 3.2 gkyl Apps...... 54 3.3 App plugins...... 74 3.4 gkyl Tools...... 91

4 Postgkyl reference 97 4.1 Using Postgkyl...... 97 4.2 Examples...... 108 4.3 Function/Command reference...... 121

5 Publications and theses 175 5.1 Doctoral Dissertations...... 175 5.2 Algorithms papers...... 175 5.3 Physics papers...... 176

6 Presentations 179

7 Developer notes 181 7.1 On use of the Maxima CAS...... 181 7.2 Modal basis functions...... 182 7.3 The recovery Maxima code...... 184 7.4 Strong-Stability preserving Runge-Kutta time-steppers...... 184 7.5 Normalized units for the Vlasov-Maxwell system...... 185 7.6 From normalized to physical units in Vlasov and multi-ﬂuid simulations...... 188 7.7 The eigensystem of the Maxwell equations with extension to perfectly hyperbolic Maxwell equations 190

i 7.8 The eigensystem of the Euler equations...... 192 7.9 The eigensystem of the ten-moment equations...... 193 7.10 Handling two-ﬂuid ﬁve-moment and ten-moment source terms...... 197

Bibliography 199

ii gkyl Documentation, Release 2.0-alpha

“Magic Chicken Software Framework” – Artificial ‘Intelligence’ view on Gkeyll “Don’t Panic” – The Hitchhiker’s Guide to the Galaxy Gkeyll v2.0 (pronounced as in the book “The Strange Case of Dr. Jekyll and Mr. Hyde”) is a computational plasma physics code mostly written in C/C++ and LuaJIT. Gkeyll contains solvers for gyrokinetic equations, Vlasov-Maxwell equations, and multi-fluid equations. Gkeyll contains ab-initio and novel implementations of a number of algorithms, and perhaps is unique in using a JIT compiled typeless dynamic language for as its main implementation language. The Gkeyll package contains two major parts: the gkyl simulation framework and the the postgkyl post-processing package. Here you will find documentation for the full Gkeyll package. For license and authors see License and Authors.

Contents 1 gkyl Documentation, Release 2.0-alpha

2 Contents CHAPTER 1

Installing Gkeyll

“Mostly Painless” – Petr Cagas Installation instructions for both the gkyl simulation framework and the postgkyl postprocessing tool are provided below.

Contents

• Installing Gkeyll – gkyl install

* Installing from source (preferred) · Installing using “machine ﬁles” (recommended as part of installation from source) · Machine ﬁles for non-native systems · Installing from source manually

* Installing with Conda * Troubleshooting – Postgkyl install

* Installing with Conda (preferred for non-developers) * Installing from source (preferred for developers) * Switching from Conda version to repository – Note on the Windows Linux Subsystem (WSL)

* Enabling and installing WSL * Interaction between Windows and Linux; GUI * Known issues

3 gkyl Documentation, Release 2.0-alpha

1.1 gkyl install

There are two options for installing gkyl. One can install directly from the source code, or via the Conda package. Installing directly from the source is the preferred option, as this method gives users more control over the installation process. For many users who will wish to run gkyl on a cluster, which will have cluster-built versions of the Message Passing Interface (MPI) for parallel simulations, and potentially other gkyl depedencies, the source build will allow users to set the appropriate paths to the cluster installations of these dependencies.

1.1.1 Installing from source (preferred)

To install gkyl from source, ﬁrst clone the GitHub repository using: git clone https://github.com/ammarhakim/gkyl

Navigate into the gkyl directory to begin. In many cases, an installation of gkyl will involve building most of gkyl’s dependencies only require a modern C/C++ compiler and Python 3. The full list of dependencies is: • C/C++ compiler with C++17 support (But NOT Clang >= 12.0 provided by Xcode 12) • Python 3 >=3.6 • MPI compiler with MPI3 support (>=openmpi 3.0 or >=mpich 3.0) • LuaJIT 2.1.0 • ADIOS 1.13.1 (But NOT >=ADIOS 2.0) • Eigen 3.3.7 • CUDA Toolkit >=10.2 (if building with GPU support, NOT FULLY SUPPORTED) The following instructions assume that at minimum the user has both a C/C++ compiler with C++17 support and Python 3.

Installing using “machine ﬁles” (recommended as part of installation from source)

For systems on which gkyl has been built before, the code can be built in three steps using scripts found in the machines/ directory. 1. Install dependencies using a mkdeps script from the machines/ directory:

./machines/mkdeps.[SYSTEM].sh where [SYSTEM] should be replaced by the name of the system you are building on, such as macosx or eddy. By default, installations will be made in ~/gkylsoft/. Even on systems which have installations of gkyl dependencies such as MPI, the mkdeps script must be run ﬁrst to build other gkyl dependencies such as LuaJIT. 2. Conﬁgure waf using a configure script from the machines/ directory:

./machines/configure.[SYSTEM].sh

NOTE: Steps 1 and 2 should only need to be done on the ﬁrst build, unless one wishes to change the dependencies. A successful waf conﬁgure, on a system without GPU support, will look like:

4 Chapter 1. Installing Gkeyll gkyl Documentation, Release 2.0-alpha

bash$ ./machines/configure.macosx.sh ./wafCC=clang CXX=clang++ MPICC=/Users/junoravin/gkylsoft/openmpi/bin/mpicc MPICXX=/

˓→Users/junoravin/gkylsoft/openmpi/bin/mpicxx --out=build --prefix=/Users/junoravin/

˓→gkylsoft/gkyl --cxxflags=-O3,-std=c++17 --luajit-inc-dir=/Users/junoravin/gkylsoft/

˓→luajit/include/luajit-2.1 --luajit-lib-dir=/Users/junoravin/gkylsoft/luajit/lib --

˓→luajit-share-dir=/Users/junoravin/gkylsoft/luajit/share/luajit-2.1.0-beta3 --enable-

˓→mpi --mpi-inc-dir=/Users/junoravin/gkylsoft/openmpi/include --mpi-lib-dir=/Users/

˓→junoravin/gkylsoft/openmpi/lib --mpi-link-libs=mpi --enable-adios --adios-inc-dir=/

˓→Users/junoravin/gkylsoft/adios/include --adios-lib-dir=/Users/junoravin/gkylsoft/

˓→adios/lib configure Setting top to : /Users/junoravin/gkyl Setting out to : /Users/junoravin/gkyl/build Checking for 'clang'(C compiler) : clang Checking for 'clang++'(C++ compiler) : clang++ Setting dependency path: : /Users/junoravin/gkylsoft Setting prefix: : /Users/junoravin/gkylsoft/gkyl Checking for LUAJIT : Found LuaJIT Checking for MPI : Found MPI Checking for ADIOS : Found ADIOS Checking for EIGEN : Found EIGEN Checking for Sqlite3 : Using Sqlite3 Checking for NVCC compiler : Not found NVCC 'configure' finished successfully(0.843s)

3. Build the code using:

./waf build install

The ﬁnal result will be a gkyl executable located in the ~/gkylsoft/gkyl/bin/ directory. Feel free to add this directory to your PATH environment variable or create an alias so you can simply call gkyl. Note that if MPI was built as well as the part of the installation, ~/gkylsoft/openmpi/bin/ needs to be added to the PATH as well. Finally, on some distributions, it is required to add ~/gkylsoft/openmpi/lib/ to the LD_LIBRARY_PATH environmental variable.

Machine ﬁles for non-native systems

For systems that do not already have corresponding ﬁles in the machines/ directory, we encourage you to add ﬁles for your machine. Instructions can be found in machines/README.md.

Note: Using Gkeyll on IBM Power9 systems (like Summit or Traverse) is not recommended. This is due to incomplete support for the LuaJIT compiler on Power9.

Installing from source manually

If machine files are not available, the dependencies, configuration, and build can be done manually. The first step is to build the dependencies. Depending on your system, building dependencies can be complicated. On a Mac or Linux machine you can simply run the mkdeps.sh script in the install-deps directory. To build dependencies cd to: cd gkyl/install-deps

First, please check details by running:

1.1. gkyl install 5 gkyl Documentation, Release 2.0-alpha

./mkdeps.sh-h

On most supercomputers you will likely need to use the system recommended compilers and MPI libraries. In this case, you should pass the appropriate compilers to mkdeps.sh as follows:

./mkdeps.sh CC=cc CXX=cxx MPICC=mpicc MPICXX=mpicxx....

You should only build libraries not provided by the system. In practice, this likely means LuaJIT, ADIOS and, perhaps Eigen. (Many supercomputer centers at DOE already offer ADIOS builds and should be preferred instead of your own builds). A typical command will be:

./mkdeps.sh--build-adios=yes--build-luajit=yes--build-eigen=yes

(in addition, you may need to specify compilers also). By default, the mkdeps.sh script will install dependencies in $HOME/gkylsoft directory. If you install it elsewhere, you will need to modify the instructions below accordingly. Once you have all dependencies installed, you can build gkyl itself by cd-ing to the top-directory in the source. gkyl uses the Waf build system. You do NOT need to install waf as it is included with the distribution. However, waf depends on Python (included on most systems). Waf takes a number of options. To get a list do

./waf--help

There are two build scenarios: ﬁrst, all dependencies are installed in $HOME/gkylsoft directory, and second, you are using some libraries supplied by your system. If you have installed all dependencies in the gkylsoft directory you can simply run:

./waf configure CC=mpicc CXX=mpicxx where CC and CXX are names of the MPI compilers to use. Note that in some cases the full path to the compiler may need to be specified. If the compilers are already in your path, then you can omit all flags. If you need to use system supplied builds, you need to specify more complex set of paths. Although you can do this by passing options to the waf build script, it is easiest to follow these steps: • Copy the configure-par.sh-in script to configure-par.sh • Modify this script to reflect the locations of various libraries on your machine. In particular, if you are using pre-built libraries you will likely need to change the information about MPI and ADIOS. • Run the configure-par.sh script Once the configuration is complete, run the following command to build and install (note: if you are working on a cluster and using environment modules, you may need to load them at this point):

./waf build install

The builds are created in the ‘build’ directory and the executable is installed in $HOME/gkylsoft/gkyl/bin, unless you speciﬁed a different install preﬁx. The executable can only be run from the install directory1. If you need to clean up a build do:

./waf clean

If you need to uninstall do: 1 The reason for this is that gkyl is in reality a LuaJIT compiler extended with MPI. Hence, for the compiler to ﬁnd Lua modules (i.e. gkyl speciﬁc code) certain paths need to be set which is done relative to the install location.

6 Chapter 1. Installing Gkeyll gkyl Documentation, Release 2.0-alpha

./waf uninstall

LuaJIT builds easily on most machines with standard GCC compiler. Often, you may run into problems on older gcc as they do not include the log2 and exp2 functions unless c99 standard is enabled. To get around this, modify the src/Makeﬁle in LuaJIT. To do this, change the line:

CC= $(DEFAULT_CC) to:

CC= $(DEFAULT_CC) -std=gnu99

To build on Mac OS X Mojave and beyond you must set the following env flag: export MACOSX_DEPLOYMENT_TARGET=10.YY where YY is the version number of the OSX operating system. For example, to build on OS X Mojave the env flag is: export MACOSX_DEPLOYMENT_TARGET=10.14 and to build on OS X Catalina the env flag is: export MACOSX_DEPLOYMENT_TARGET=10.15

1.1.2 Installing with Conda

The gkyl package is also available to be installed via Conda, although this gives less ﬂexibility for keeping the code up-to-date as gkyl development continues. Once Conda is installed and added to the PATH, gkyl can be obtained with: conda install-c gkyl gkeyll

Note, that this will also install all dependencies into the Conda install directory. Often this may lead to some conﬂicts, particularly for the MPI installation, specially if there is another version of MPI already located in the system. gkyl should be run using the MPI provided by Conda. In general, having Conda and source-built gkyl on the same machine can cause confusion. In that case please use explicit paths to the mpiexec and gkyl executable you wish to use when running simulations.

1.1.3 Troubleshooting

Having trouble building? We will try to compile a list of suggestions and common error messages in this troubleshooting site.

1.2 Postgkyl install

There are two options for getting Postgkyl. The ﬁrst option is to get the Conda package and the second is to clone the source code repository. Installing via Conda is preferred for the majority of users as it requires literally a single Conda command. What is more, Conda is already the suggested way of installing all the dependencies. On the other hand, the later option has an advantage of always having the most up-to-date version and is generally required for users that want to contribute to the code.

1.2. Postgkyl install 7 gkyl Documentation, Release 2.0-alpha

1.2.1 Installing with Conda (preferred for non-developers)

Postgkyl can be installed with Conda with literally a single command: conda install -c gkyl postgkyl

Note that the ﬂag for the Gkeyll channel, -c gkyl, is required even for updating. However, it can be permanently added. conda config --add channels gkyl conda install postgkyl

Updates can be downloaded with: conda update -c gkyl postgkyl

Note: To install a new package, users need the write permission for the Anaconda directory. If this is not the case (e.g. on a computing cluster), one can either create a Conda environment (see tip below) or install Conda into the $HOME directory.

Tip: To create a Conda environment for postgkyl, use conda create -n pgkylenv python=3

Then activate the environment with conda activate pgkylenv and install postgkyl using the commands above (or the ones below to install from source). After install, one must always have the pgkylenv environment activated in order to use postgkyl.

1.2.2 Installing from source (preferred for developers)

Postgkyl source code is hosted in a GitHub repository. To get Postgkyl running, one ﬁrst needs to clone the repository and install dependencies. First, clone the repository using: git clone https://github.com/ammarhakim/postgkyl

Postgkyl has these dependencies, which are readily available thru Conda: • Click • Matplotlib >= 3.0 • NumPy >=1.13 • PyTables • SciPy • SymPy • Bokeh

8 Chapter 1. Installing Gkeyll gkyl Documentation, Release 2.0-alpha

• Adios • MessagePack for python (msgpack-python) All these dependencies can be easily obtained from the Gkeyll Conda channel, via conda install -c gkyl postgkyl --only-deps

Once the dependencies are installed, postgkyl can be installed by navigating into the postgkyl repository and running python setup.py install python setup.py develop

Note that these commands only ever need to be run once (even if one is modifying source code). Changes to the source code will be automatically included because we have installed in development mode. Adios can also be built manually from the source code. Note that for the manual build, Adios needs to be already installed and its bin directory added to the PATH (the default Gkeyll location is ~/gkylsoft/adios/bin/ ). The standard location for the wrapper in the Gkeyll installation is gkyl/install-deps/adios-x.x.x/ wrappers/numpy/. After navigating to that directory, build and install adiospy via make python python setup.py install

This currently does not work out of the box with the clang compiler because of a deprecated library. This can be overcome removing the -lrt ﬂag from the line 33 of the Makefile. The edited lines 32 and 33 should look like this: adios.so: python setup.py build_ext

This will allow to complete the adiospy build successfully and it has no know consequences for Postgkyl.

1.2.3 Switching from Conda version to repository

While the Conda build of Postgkyl is the suggested version for the majority of users, the source code repository is required for any code contributions. We should stress that when switching between the different version, it is strongly advised to remove the other version. Having both may lead to an unforeseen behavior based on the relative order of components in the PATH and PYTHONPATH. The Conda version can be uninstalled with: conda uninstall postgkyl

1.3 Note on the Windows Linux Subsystem (WSL)

Historically, Gkeyll was supported only for Unix-like operating systems. However, since the Windows 10 Anniversary Update in August 2016, it became possible to run Gkeyll even on Windows although with reduced performance. This changed with the introduction of WSL 2 in Spring/Summer 2020. With WSL 2, the performance on Unix and Windows is indistinguishable. Requirements: • For x64 systems: Version 1903 or higher, with Build 18362 or higher

1.3. Note on the Windows Linux Subsystem (WSL) 9 gkyl Documentation, Release 2.0-alpha

• For ARM64 systems: Version 2004 or higher, with Build 19041 or higher • For WSL 2 (recommended): Build 18362 and a CPU supporting Hyper-V virtualization

1.3.1 Enabling and installing WSL

First, WSL needs to be enabled. This can be done either using GUI or PowerShell. For the former, go to Turn Windows features on or off in the Control Panel and check Windows Subsystem for Linux at the very bottom.

Alternatively, this can be done in a terminal (running the PowerShell as an Administrator): dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /

˓→norestart

The Linux distribution itself is available thru Microsoft Store. Currently, the following ﬂavors are available: • Ubuntu 16.04 LTS • Ubuntu 18.04 LTS • Ubuntu 20.04 LTS • openSUSE Leap 15.1 • SUSE Linux Enterprise Server 12 SP5 • SUSE Linux Enterprise Server 15 SP1 • Kali Linux • Debian GNU/Linux • Fedora Remix for WSL • Pengwin

10 Chapter 1. Installing Gkeyll gkyl Documentation, Release 2.0-alpha

• Pengwin Enterprise • Alpine WSL It is strongly recommended to then enable WSL 2 for much better performance. First, enable virtualization in Admin- istrator PowerShell dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart

Download and install the kernel update package. And ﬁnally, set WSL 2 as the default (again using PowerShell as Administrator). wsl --set-version 2 wsl --set-default-version 2

1.3.2 Interaction between Windows and Linux; GUI

As of September 2020, WSL2 does not directly support GUI; however, this is currently supposed to be in the works. Until the ofﬁcial release, this can be overcome with a 3rd party X-server like VcXsrv (this is our recommended option as the other option does not seem to work on some conﬁgurations) or Xming. Note that when using VcXsrv, the Disable access control checkbox needs to be marked when setting XLaunch. Otherwise, the X11 forwarding would not work properly.

Finally, the $DISPLAY environmental variable needs to be set up on the Linux side. In WSL 1 this was as simple as export DISPLAY=:0

WSL 2, however, uses virtualization and the forwarding address changes which each launch of the system. The address is stored in /etc/resolv.conf. The $DISPLAY can be set up with:

1.3. Note on the Windows Linux Subsystem (WSL) 11 gkyl Documentation, Release 2.0-alpha

export DISPLAY=$(cat /etc/resolv.conf | grep nameserver | awk '{print $2}'):0

Another issue might be caused by tools that want to open a browser, for example, the commonly used jupyter-lab. This can be easily overcome by launching the server without a browser using jupyter-lab --no-browser and then copying the address including the token to a Windows browser. Finally, it is often useful to access the Linux ﬁles from Windows. Using WLS 2, the Linux root, /, is located at \wsl$\.

1.3.3 Known issues

There is currently a known issue where Windows and Linux clocks might get desynchronized when the computer sleeps. This might cause issues with Git and update installation using sudo apt update. There is a workaround that works until this issue gets patched and that is manually calling sudo hwclock -s to manually synchronize the time.

12 Chapter 1. Installing Gkeyll CHAPTER 2

Quickstart

Once you have a working installation of Gkeyll, you can get to know the code and learn how to use it by following these quickstart examples. For additional examples one can browse the regression tests folder in the gkyl code or repository (gkyl/ Regression/). We also began compiling input ﬁles for simulations reported by publications in this repository. Lastly, more examples and in depth information about the gkyl Apps can be found in the gkyl Reference.

2.1 A ﬁrst Gkeyll simulation

Gkeyll supports several models and numerous capabilities. Before diving into those, here’s a short overview of an input ﬁle, how to run it and plot its results.

Contents

• A first Gkeyll simulation – Physics model – Input file – Running your first simulation – Plotting – Additional quick-start examples

13 gkyl Documentation, Release 2.0-alpha

2.1.1 Physics model

Consider the collisionless Landau damping of an electrostatic wave in a plasma. We simulate it with the Vlasov- Maxwell model:

휕푓푠 푞푠 + v · ∇푓푠 + (E + v × B) · ∇v푓 = 0, 휕푡 푚푠 휕B + ∇ × E = 0, 휕푡 휕E 휖 휇 − ∇ × B = −휇 J. 0 0 휕푡 0 More information about the model can be found in the Vlasov-Maxwell documentation. In this case we consider an electrostatic wave, and there are subtleties regarding electrostatic simulation using Maxwell’s induction equations. So we initialize the electron density as having a small sinusoidal perturbation:

푛푒(푥, 푡 = 0) = 푛0(1 + 퐴 cos(푘푥)) and initialize the electric ﬁeld in a manner consistent with the Poisson equation (using quasineutrality of the equilibrium electron and ion densities, 푛푖(푥, 푡 = 0) = 푛0): 퐴 퐸 (푥, 푡 = 0) = − sin(푘푥) 푥 푘

2.1.2 Input ﬁle

This simulation is setup using Normalized units for the Vlasov-Maxwell system in a short Lua input ﬁle, which begins with the Preamble: local Plasma= require("App.PlasmaOnCartGrid").VlasovMaxwell() permitt= 1.0 -- Permittivity of free space. permeab= 1.0 -- Permeability of free space. eV= 1.0 -- Elementary charge, or Joule-eV conversion factor. elcMass= 1.0 -- Electron mass. ionMass= 1.0 -- Ion mass. nElc= 1.0 -- Electron number density. nIon= nElc -- Ion number density. Te= 1.0 -- Electron temperature. Ti= Te -- Ion temperature. vtElc= math.sqrt(eV *Te/elcMass) -- Electron thermal speed. vtIon= math.sqrt(eV *Ti/ionMass) -- Ion thermal speed. wpe= math.sqrt((eV^2) *nElc/(permitt*elcMass)) -- Plasma frequency. lambdaD= vtElc/wpe -- Debye length.

-- Amplitude and wavenumber of sinusoidal perturbation. pertA= 1.0e-3 pertK= .750/lambdaD

-- Maxwellian in (x,vx)-space, given the density (denU), bulk flow -- velocity (flowU), mass and temperature (temp). local function maxwellian1D(x, vx, den, flowU, mass, temp) local v2= (vx- flowU)^2 local vtSq= temp/mass return (den/math.sqrt(2*math.pi*vtSq))*math.exp(-v2/(2*vtSq)) end

14 Chapter 2. Quickstart gkyl Documentation, Release 2.0-alpha

The Preamble typically consists of a line loading the Gkeyll plasma ‘App’ to be used (in this case VlasovMaxwell), and a specification of a number of user input parameters and simple derived quantities. One can also create user-defined functions, like maxwellian1D in this case, which may be used in the preamble or in the Gkeyll App that follows. For this specific simulation the Gkeyll app is created by the following: plasmaApp= Plasma.App { tEnd= 20.0/wpe, -- End time. nFrame= 20, -- Number of output frames. lower={-math.pi/pertK}, -- Lower boundary of configuration space. upper={ math.pi/pertK}, -- Upper boundary of configuration space. cells={64}, -- Configuration space cells. polyOrder=1, -- Polynomial order. periodicDirs={1}, -- Periodic directions.

elc= Plasma.Species { charge=-eV, mass= elcMass, lower={-6.0 *vtElc}, -- Velocity space lower boundary. upper={ 6.0 *vtElc}, -- Velocity space upper boundary. cells={64}, -- Number of cells in velocity space. init= function (t, xn) -- Initial conditions. local x, v= xn[1], xn[2] return (1+pertA*math.cos(pertK*x))*maxwellian1D(x, v, nElc, 0.0, elcMass, Te) end, evolve= true, -- Evolve species? },

ion= Plasma.Species { charge= eV, mass= ionMass, lower={-6.0 *vtIon}, -- Velocity space lower boundary. upper={ 6.0 *vtIon}, -- Velocity space upper boundary. cells={64}, -- Number of cells in velocity space. init= function (t, xn) -- Initial conditions. local x, v= xn[1], xn[2] return maxwellian1D(x, v, nIon, 0.0, ionMass, Ti) end, evolve= true, -- Evolve species? },

field= Plasma.Field { epsilon0= permitt, mu0= permeab, init= function (t, xn) -- Initial conditions. local Ex, Ey, Ez=-pertA *math.sin(pertK*xn[1])/pertK, 0.0, 0.0 local Bx, By, Bz= 0.0, 0.0, 0.0 return Ex, Ey, Ez, Bx, By, Bz end, evolve= true, -- Evolve field? }, }

The Gkeyll App typically consists of three sections: • Common: a declaration of parameters that control the (configuration space) discretization, and time advancement. This first block of code in Plasma.App may specify the periodic directions, the MPI decomposition, and the frequency with which to output certain diagnostics. • Species: Definition of the species to be considered in the simulation. Each species gets its own Lua table, in which one provides the velocity-space domain and discretization of that species (for kinetic models), initial condition, diagnostics, boundary conditions, and whether to evolve it or not (evolve).

2.1. A ﬁrst Gkeyll simulation 15 gkyl Documentation, Release 2.0-alpha

• Fields: A field table, which tells the App whether to evolve the electric and/or magnetic fields according to the field equations of the model. In this table we also specify the initial condition of the fields. In some applications other sections of the Plasma.App may be necessary, for example, to specify the geometry. Finally, an input file concludes with an invocation of the App’s run method: plasmaApp:run()

2.1.3 Running your ﬁrst simulation

Now that we have a Gkeyll input ﬁle (named vm-damp.lua), simply run the simulation by typing gkyl vm-damp.lua

You should see the program printing to screen like this: wsName:gkyldir gabriel$ gkyl vm-damp.lua Tue Sep 15 2020 16:16:44.000000000 Gkyl built with b0b8203670c7+ Gkyl built on Sep 14 2020 16:29:40 Initializing PlasmaOnCartGrid simulation ... ** WARNING: timeStepper not specified, assuming rk3 Using CFL number0.333333 Initializing completed in 0.0629927 sec

Starting main loop of PlasmaOnCartGrid simulation ...

Step0 at time0. Time step0.00727108. Completed0% 0123456789 Step 276 at time2.00698. Time step0.00727174. Completed 10% 0123456789 Step 551 at time4.00677. Time step0.00727214. Completed 20% 0123456

Gkeyll prints a number every 1% of the simulation, and a longer message with the total number of time steps taken, the simulation time and the latest time step size every 10% of the simulation. This particular simulation ran in 74 seconds on a 2015 MacBookPro. As it progressed it wrote out diagnostic ﬁles.

2.1.4 Plotting

In this case we did not request additional diagnostics, so the only ones provided are default ones: • Distribution functions: vm-damp_elc_#.bp and vm-damp_ion_#.bp. • Electromagnetic ﬁelds: vm-damp_field_#.bp. • Field energy: vm-damp_fieldEnergy.bp. Fields that are larger (in memory) like the distribution function, get written out periodically, not every time step. These snapshots (frames) are labeled by the number # at the end of the ﬁle name. In order to plot the initial distribution function of the electrons we will use the Gkeyll post-processing tool (postgkyl), invoked by the pgkyl command as follows pgkyl vm-damp_elc_0.bp interpolate plot

This produces the 2D plot of the initial Maxwellian distribution given below.

16 Chapter 2. Quickstart gkyl Documentation, Release 2.0-alpha

Fig. 1: Initial electron distribution function, 푓푒(푥, 푣, 푡 = 0).

We can also examine the electrostatic energy in the simulation. This most clearly exhibits the wave energy decaying as the collisionless damping takes effect. For this purpose we use the following postgkyl command (we select the x-component, and pg_cmd-plot can use a log scale, as well as add labels): pgkyl vm-damp_fieldEnergy.bp select -c0 plot --logy -x 'time' -y '$|E_x|^2$' resulting in the following ﬁgure of the (normalized) electrostatic energy as a function of time

2 Fig. 2: Normalized electrostatic ﬁeld energy ∝ |퐸푥| as a function of time (normalized to 휔푝푒).

2.1.5 Additional quick-start examples

The above example used a Vlasov-Maxwell simulation to showcase how to setup, run and postprocess a Gkeyll simulation. In addition to Vlasov-Maxwell there are also Gyrokinetic and (ﬂuid) Moment models. Each of these have slightly different features and ways of using them. Quick examples for each of these are found below: Vlasov example Gyrokinetic example Fluid example

2.1. A ﬁrst Gkeyll simulation 17 gkyl Documentation, Release 2.0-alpha

2.2 Vlasov example

In this example, we extend the Vlasov-Maxwell input file shown in A first Gkeyll simulation to simulate a more general kinetic plasma. Because the Vlasov-Maxwell system of equations is widely applicable in plasma physics, this example is intended to illustrate some of the functionality of the Vlasov-Maxwell solver a user may desire for production science. For more extensive documentation on all of the available options for Vlasov-Maxwell simulations, we refer the reader to VlasovMaxwell App: Vlasov-Maxwell equations on a Cartesian grid. This simulation is based on the studies of [Skoutnev2019] and [Juno2020] and concerns the evolution of instabilities driven by counter-streaming beams of plasma. This example demonstrates the flexibility of the Vlasov-Maxwell solver by showing how one extends Vlasov-Maxwell simulations to higher dimensionality, in this case 2x2v. The input file for this example is also a standard performance benchmark for Gkeyll and timings for this input file with varying grid resolution can be found in the gkyl repo in the Benchmarks/ folder.

Contents

• Vlasov example – Physics model and initial conditions – Input ﬁle – Running the simulation – Postprocessing – References

2.2.1 Physics model and initial conditions

This example solves the Vlasov-Maxwell system of equations

휕푓푠 푞푠 + v · ∇푓푠 + (E + v × B) · ∇v푓 = 0, 휕푡 푚푠 휕B + ∇ × E = 0, 휕푡 휕E 휖 휇 − ∇ × B = −휇 J, 0 0 휕푡 0 in two spatial dimensions and two velocity dimensions (2x2v). For this example, the ions are taken to be a stationary, neutralizing background, and therefore do not contribute to the plasma current J. The electrons are initialized as two equal density, equal temperature, counter-propagating, Maxwellian beams

[︂ (︂ 2 2 )︂ (︂ 2 2 )︂]︂ 푚푒푛0 (푣푥) + (푣푦 − 푢푑) (푣푥) + (푣푦 + 푢푑) 푓푒(푥, 푦, 푣푥, 푣푦) = exp −푚푒 + exp −푚푒 . 2휋푇푒 2푇푒 2푇푒 The electromagnetic fields are initialized as a bath of fluctuations in the electric and magnetic fields in the two configuration space dimensions,

푁,푁 (︂ )︂ ∑︁ ˜ 2휋푛푥푥 2휋푛푦푦 ˜ 퐵푧(푡 = 0) = 퐵푛푥,푛푦 sin + + 휑푛푥,푛푦 , 퐿푥 퐿푦 푛푥,푛푦 =−푁,−푁

˜ ˜ where 푁 = 16 and 퐵푛푥,푛푦 and 휑푛푥,푛푦 are random amplitudes and phases. The electric ﬁelds, 퐸푥, 퐸푦 are initialized similarly. Note that the sum goes from −푁 to 푁 so as to initialize phases from 0 degrees to 180 degrees.

18 Chapter 2. Quickstart gkyl Documentation, Release 2.0-alpha

2.2.2 Input ﬁle

The full Lua input ﬁle (found here) has three components: the App dependencies, the Preamble, and the App. In the App dependencies section, we load the necessary components of Gkeyll to perform a Vlasov-Maxwell simulation, as well as any additional functionality we require:

------App dependencies ------Load the Vlasov-Maxwell App local Plasma= require("App.PlasmaOnCartGrid").VlasovMaxwell() -- Pseudo-random number generator from SciLua package for initial conditions. -- Specific rng is the mrg32k3a (multiple recursive generator) of Ecuyer99. local prng= require"sci.prng" local rng= prng.mrg32k3a()

The Preamble to set up the initial conditions is:

------Preamble ------Constants permitt= 1.0 -- Permittivity of free space permeab= 1.0 -- Permeability of free space lightSpeed= 1.0/math.sqrt(permitt *permeab) -- Speed of light chargeElc=-1.0 -- Electron charge massElc= 1.0 -- Electron mass

-- Initial conditions -- 1 = Right-going beam; 2 = Left-going beam. nElc1= 0.5 nElc2= 0.5 ud= 0.1 -- Drift velocity of beams uxElc1= 0.0 uyElc1= ud uxElc2= 0.0 uyElc2=-ud

R= 0.1 -- Ratio of thermal velocity to drift

˓→velocity TElc1= massElc *(R*ud)^2 TElc2= massElc *(R*ud)^2 vthElc1= math.sqrt(TElc1/massElc) vthElc2= math.sqrt(TElc2/massElc) k0_TS= 6.135907273413176 -- Wavenumber of fastest growing two-

˓→stream mode theta= 90.0/180.0 *math.pi -- 0 deg is pure Weibel, 90 deg is pure ˓→two-stream kx_TS= k0_TS *math.cos(theta) ky_TS= k0_TS *math.sin(theta) k0_Weibel= 2.31012970008316 -- Wavenumber of fastest growing Weibel

˓→mode theta= 0.0/180.0 *math.pi -- 0 deg is pure Weibel, 90 deg is pure ˓→two-stream kx_Weibel= k0_Weibel *math.cos(theta) (continues on next page)

2.2. Vlasov example 19 gkyl Documentation, Release 2.0-alpha

(continued from previous page) ky_Weibel= k0_Weibel *math.sin(theta) kx= k0_Weibel ky= k0_TS/3.0 perturb_n= 1e-8 -- Perturbing the first 16 wave modes with random amplitudes and phases. -- Note that loop goes from -N to N to sweep all possible phases. N=16 P={} for i=-N,N,1 do P[i]={} for j=-N,N,1 do P[i][j]={} for k=1,6,1 do P[i][j][k]=rng:sample() end end end

-- Domain size and number of cells Lx=2 *math.pi/kx Ly=2 *math.pi/ky Nx= 16 Ny= 16 vLimElc=3 *ud -- Maximum velocity in velocity space NvElc= 16

-- Maxwellian in 2x2v local function maxwellian2D(n, vx, vy, ux, uy, vth) local v2= (vx- ux)^2+ (vy- uy)^2 return n/(2*math.pi*vth^2)*math.exp(-v2/(2*vth^2)) end

The Preamble deﬁnes the constants in the normalization standard outlined in Normalized units for the Vlasov-Maxwell system and sets the parameters and perturbations to the wave modes of interest for the study. Note that because the dimensionality of the simulation is now 2x2v, the normalization of the Maxwellian has correspondingly changed from the 1x1v Langmuir wave simulation described in A ﬁrst Gkeyll simulation. The App can be further subdivided into a number of sections plasmaApp= Plasma.App { ------Common ------...

------Species ------...

------Fields ------... } ------(continues on next page)

20 Chapter 2. Quickstart gkyl Documentation, Release 2.0-alpha

(continued from previous page) -- Run application ------plasmaApp:run()

The Common section of the App defines input parameters which will be utilized by all solvers in the simulation. For example, the configuration space extents and number of configuration space cells (lower, upper, cells), as well as what directions, if any, utilize periodic boundary conditions (periodicDirs), and how to parallelize the simulation (decompCuts,useShared).

------Common ------logToFile= true, tEnd= 50.0, -- End time nFrame=1, -- Number of output frames lower={0.0,0.0}, -- Lower boundary of configuration space upper= {Lx,Ly}, -- Upper boundary of configuration space cells= {Nx,Ny}, -- Configuration space cells basis="serendipity", -- One of "serendipity", "maximal-order", or

˓→"tensor" polyOrder=2, -- Polynomial order timeStepper="rk3s4", -- One of "rk2", "rk3", or "rk3s4"

-- MPI decomposition for configuration space decompCuts={1,1}, -- Cuts in each configuration direction useShared= true, -- If using shared memory

-- Boundary conditions for configuration space periodicDirs={1,2}, -- periodic directions (both x and y)

-- Integrated moment flag, compute integrated quantities 1000 times in simulation calcIntQuantEvery= 0.001,

The Species section of the App deﬁnes the species-speciﬁc inputs for the Vlasov-Maxwell simulation within a Plasma.Species table. For example, the velocity space extents and number of velocity space cells (lower, upper, cells), the function which prescribes the initial condition, and the types of diagnostics. More discussion of diagnostic capabilities can be found in VlasovMaxwell App: Vlasov-Maxwell equations on a Cartesian grid.

------Electrons ------elc= Plasma.Species { charge= chargeElc, mass= massElc, -- Velocity space grid lower={-vLimElc,-vLimElc}, upper= {vLimElc, vLimElc}, cells= {NvElc, NvElc}, -- Initial conditions init= function (t, xn) local x, y, vx, vy= xn[1], xn[2], xn[3], xn[4] local fv= maxwellian2D(nElc1, vx, vy, uxElc1, uyElc1, vthElc1)+ maxwellian2D(nElc2, vx, vy, uxElc2, uyElc2, vthElc2) return fv end, evolve= true, (continues on next page)

2.2. Vlasov example 21 gkyl Documentation, Release 2.0-alpha

(continued from previous page) diagnosticMoments={"M0","M1i","M2ij","M3i"}, diagnosticIntegratedMoments={"intM0","intM1i","intM2Flow","intM2Thermal"}, },

Note that for this particular simulation the ions are a stationary, neutralizing background that does not contribute to the plasma current, so we only require a species table for the electrons. The Field section if the final section of the App and specifies the input parameters for the field equation, in this case Maxwell’s equation, in the Plasma.Field table. For example, similar to the Plasma.Species table, the Plasma.Field table contains the initial condition for the electromagnetic field.

------Field solver ------field= Plasma.Field { epsilon0= permitt, mu0= permeab, init= function (t, xn) local x, y= xn[1], xn[2] local E_x, E_y, B_z= 0.0, 0.0, 0.0 for i=-N,N,1 do for j=-N,N,1 do if i~=0 or j~=0 then E_x= E_x+ perturb_n *P[i][j][1]*math.sin(i*kx*x+j*ky*y+2*math. ˓→pi*P[i][j][2]) E_y= E_y+ perturb_n *P[i][j][3]*math.sin(i*kx*x+j*ky*y+2*math. ˓→pi*P[i][j][4]) B_z= B_z+ perturb_n *P[i][j][5]*math.sin(i*kx*x+j*ky*y+2*math. ˓→pi*P[i][j][6]) end end end return E_x, E_y, 0.0, 0.0, 0.0, B_z end, evolve= true, },

2.2.3 Running the simulation

The input ﬁle vm-tsw-2x2v.lua can be run using the gkyl executable

~/gkylsoft/gkyl/bin/gkyl vm-tsw-2x2v.lua assuming gkyl has been installed in the user’s home directory. When running this simulation, a user should see the following output

Wed Sep 16 2020 11:38:54.000000000 Gkyl built with a4430cbb5d93 Gkyl built on Sep 16 2020 01:25:31 Initializing Vlasov-Maxwell simulation ... Initializing completed in 1.39731 sec

Starting main loop of Vlasov-Maxwell simulation ... Step0 at time0. Time step0.0360652. Completed0% 0123456789 Step 139 at time5.01307. Time step0.0360652. Completed 10% 01234

22 Chapter 2. Quickstart gkyl Documentation, Release 2.0-alpha

The full screen output can be found here, which includes performance details for the simulation. This example was run with a single core of a 10th gen Intel i9 (Comet Lake) processor. Increasing the resolution to 322 × 322 and now running the simulation using all 10 cores of the Intel i9 using

~/gkylsoft/openmpi/bin/mpirun -n 10 ~/gkylsoft/gkyl/bin/gkyl vm-tsw-2x2v.lua

we obtain the following performance with useShared=true and the installed MPI from the Gkeyll build.

2.2.4 Postprocessing

The output of this simulation is the following set of files: • Distribution functions: vm-tsw-2x2v_elc_#.bp. • Electromagnetic fields: vm-tsw-2x2v_field_#.bp. • Diagnostic moments: vm-tsw-2x2v_elc_M0_#.bp, vm-tsw-2x2v_elc_M1i_#.bp, vm-tsw-2x2v_elc_M2ij_#.bp, and vm-tsw-2x2v_elc_M3i_#.bp. • Field energy: vm-tsw-2x2v_fieldEnergy.bp. • Diagnostic integrated moments: vm-tsw-2x2v_elc_intM0.bp, vm-tsw-2x2v_elc_intM1i.bp, vm-tsw-2x2v_elc_intM2Flow.bp, and vm-tsw-2x2v_elc_intM2Thermal.bp. Snapshots (frames) are labeled by the number # at the end of the file name, while integrated diagnostics that are computed as a time-series, such as the field energy, are written out as a single file. Since nFrame=1 in the input file, the only frames that are output are 0, corresponding to the initial condition, and 1, corresponding to the end of the simulation. Since this simulation has two configuration space dimensions, postgkyl creates pcolor plots when run from the com- 2 2 mand line with pgkyl. We can compare the initial condition and final state of the magnetic field, 퐵푧, (of the 32 ×32 simulation) in two separate figures with the pgkyl command:

pgkyl vm-tsw-2x2v_field_0.bp vm-tsw-2x2v_field_1.bp interp sel --comp5 plot -b --fix-

˓→aspect

−1 Fig. 3: 퐵푧 magnetic ﬁeld at 푡 = 0 (left) and 푡 = 50휔푝푒 , the end of the simulation (right).

The default postgkyl colorbar is sequential and useful for visualizing data such as distribution functions, which will vary from 0 (zero phase space density/no particles) to some number (corresponding to a local increase in phase space density). However, we can see that the colorbar for the magnetic field varies between roughly equal positive and negative numbers, and thus a diverging colormap may yield a more useful representation of the data. In addition, we can utilize the flexibility of the interpolate command to interpolate the discontinuous Galerkin data onto an even finer mesh

2.2. Vlasov example 23 gkyl Documentation, Release 2.0-alpha

pgkyl vm-tsw-2x2v_field_0.bp vm-tsw-2x2v_field_1.bp interp -i6 sel --comp5 plot -b -

˓→-fix-aspect --diverging --xlabel '$x (d_e) $' --ylabel '$y (d_e) $'

−1 Fig. 4: 퐵푧 magnetic field at 푡 = 0 (left) and 푡 = 50휔푝푒 , the end of the simulation (right), now with a diverging colorbar, finer interpolation, and labels. where we have now added labels with the normalized units in 푥 and 푦. Note that the default interpolation level for polynomial order 2 is 3 (polyOrder + 1). We can likewise visualize diagnostic moments such as the first velocity moment elc_M1i

pgkyl vm-tsw-2x2v_elc_M1i_1.bp interp -i6 plot --fix-aspect --diverging --xlabel '$x

˓→(d_e) $' --ylabel '$y (d_e) $'

−1 Fig. 5: M1푥 ﬁrst velocity moment (left) and M1푦 ﬁrst velocity moment (right) at 푡 = 50휔푝푒 , the end of the simulation.

Note that elc_M1i has two components due to the fact that this simulation has two velocity dimensions, and both components are visualized when this pgkyl command is utilized. The left plot is the 푣푥 velocity moment and the right plot is the 푣푦 velocity moment. Further details on the diagnostics available and their deﬁnitions can be found in VlasovMaxwell App: Vlasov-Maxwell equations on a Cartesian grid. We can also visualize the distribution function from this simulation. However, for this simulation the distribution function if four-dimensional, two conﬁguration space and two velocity space dimensions. Postgkyl offers a number of options for down-selecting the data to be more amenable to visualizing. For example, we can read-in a subset of the data and visualize the distribution function in velocity space 푣푥 − 푣푦 in the lower left corner of the domain

pgkyl vm-tsw-2x2v_elc_1.bp --z00 --z10 interp -i6 sel --z00.0 --z10.0 plot --

˓→xlabel '$v_x (v_{th_e}) $' --ylabel '$v_y (v_{th_e}) $' --vmin0.0

24 Chapter 2. Quickstart gkyl Documentation, Release 2.0-alpha

Note that the immediate --z0 0 --z1 0 tells postgkyl to read in only the first 푥 and 푦 configuration space grid cells (while still reading in all of velocity space). Because we are then interpolating the data onto a finer mesh, the data is still four dimensional so we pass the abbreviated select command sel to finally down-select to the lower-left corner of the configuration space domain. These selective read-in commands are vital for very large arrays where the cost in memory and CPU time can be quite large to read-in and manipulate the data structure of interest. Alternatively, if we do want to read-in the whole array, we can perform other manipulations to the distribution function such we can still easily visualize the data. For example, we can use the integrate command to integrate the distribution function over 푥 and 푣푥 to produce a 푦 − 푣푦 plot of the electron distribution function. pgkyl vm-tsw-2x2v_elc_1.bp interp integrate0,2 plot --xlabel '$y (d_e) $' --ylabel'

˓→$v_y (v_{th_e}) $' --vmin0.0

Finally, since we performed this simulation at two different resolutions, and interesting diagnostic to look at is a comparison of integrated quantities between the two simulations. For ease of plotting we have moved the data from the two simulations to two different folders, res1 (162 × 162) and res2 (322 × 322). Here, we are being agnostic on what a user might have named these two different simulations and labeling them ourselves with postgkyl. pgkyl res1/*fieldEnergy.bp -l '$16^2 \times 16^2$' res2/*fieldEnergy.bp -l '$32^2 ˓→\times 32^2$' select --comp5 plot --logy --xlabel '$t (\omega_{pe}^{-1})$'--

˓→ylabel '$\int B_z^2$' -f0

2.2. Vlasov example 25 gkyl Documentation, Release 2.0-alpha

2 Fig. 6: Integrated magnetic ﬁeld energy, |퐵푧| , plotted as a function of time comparing the lower resolution calculation, 162 × 162 (blue), and higher resolution calculation, 322 × 322 (orange).

2.2.5 References

2.3 Gyrokinetic example

The gyrokinetic system is used to study turbulence in magnetized plasmas. Gkeyll’s Gyrokinetic App is specialized to study the edge/scrape-off-layer region of fusion devices, which requires handling of large fluctuations and open-field- line regions. In this example, we will set up a gyrokinetic problem on open magnetic field lines (e.g. in the tokamak scrape-off layer). Using specialized boundary conditions along the field line that model the sheath interaction between the plasma and conducting end plates, we will see that we can model the self-consistent formation of the sheath potential. This simple test case can then be used as a starting point for full nonlinear simulations of the tokamak SOL.

Contents

• Gyrokinetic example – Background – Input ﬁle

* Electron density * Sheath potential * Particle balance * Divertor Fluxes – References

2.3.1 Background

Gyrokinetics intro?

2.3.2 Input ﬁle

The full Lua input file (gk-sheath.lua) for this simulation is a bit longer than the one in A first Gkeyll simulation, but not to worry, we will go through each part of the input file carefully.

26 Chapter 2. Quickstart gkyl Documentation, Release 2.0-alpha

To set up a gyrokinetic simulation, we ﬁrst need to load the Gyrokinetic App package and other dependencies. This should be done at the top of the input ﬁle, via

------App dependencies ------local Plasma= require("App.PlasmaOnCartGrid").Gyrokinetic() -- load the

˓→Gyrokinetic App local Constants= require"Lib.Constants" -- load some

˓→physical Constants

Here we have also loaded the Constants library, which contains various physical constants that we will use later. The next block is the Preamble, containing input paramters and simple derived quantities:

------Preamble ------Universal constant parameters. eps0= Constants.EPSILON0 eV= Constants.ELEMENTARY_CHARGE qe=-eV -- electron charge qi= eV -- ion charge me= Constants.ELECTRON_MASS -- electron mass mi= 2.014 *Constants.PROTON_MASS -- ion mass (deuterium ions)

-- Plasma parameters. Te0= 40 *eV -- reference electron temperature, used to set ˓→up electron velocity grid [eV] Ti0= 40 *eV -- reference ion temperature, used to set up ion ˓→velocity grid [eV] n0= 7e18 -- reference density [1/m^3]

-- Geometry and magnetic field parameters. B_axis= 0.5 -- magnetic field strength at magnetic axis [T] R0= 0.85 -- device major radius [m] a0= 0.15 -- device minor radius [m] R= R0+ a0 -- major radius of flux tube simulation domain

˓→[m] B0= B_axis *(R0/R) -- magnetic field strength at R [T] Lpol= 2.4 -- device poloidal length (e.g. from bottom

˓→divertor plate to top) [m]

-- Parameters for collisions. nuFrac= 0.1 -- use a reduced collision frequency (10% of

˓→physical) -- Electron collision freq. logLambdaElc= 6.6- 0.5 *math.log(n0/1e20)+ 1.5 *math.log(Te0/eV) nuElc= nuFrac *logLambdaElc*eV^4*n0/(6*math.sqrt(2)*math.pi^(3/2)*eps0^2*math. ˓→sqrt(me)*(Te0)^(3/2)) -- Ion collision freq. logLambdaIon= 6.6- 0.5 *math.log(n0/1e20)+ 1.5 *math.log(Ti0/eV) nuIon= nuFrac *logLambdaIon*eV^4*n0/(12*math.pi^(3/2)*eps0^2*math.sqrt(mi)*(Ti0)^(3/ ˓→2))

-- Derived parameters vti= math.sqrt(Ti0/mi) -- ion thermal speed vte= math.sqrt(Te0/me) -- electron thermal speed (continues on next page)

2.3. Gyrokinetic example 27 gkyl Documentation, Release 2.0-alpha

(continued from previous page) c_s= math.sqrt(Te0/mi) -- ion sound speed omega_ci= math.abs(qi *B0/mi) -- ion gyrofrequency rho_s= c_s/omega_ci -- ion sound gyroradius

-- Simulation box size Lx= 50 *rho_s -- x = radial direction Ly= 100 *rho_s -- y = binormal direction Lz=4 -- z = field-aligned direction

This simulation also requires a source, which models plasma crossing the separatrix. The next part of the Preamble initializes some source parameters, along with some functions that will be used later to set up the source density and temperature proﬁles.

-- Source parameters P_SOL= 3.4e6 -- total SOL power, from experimental heating

˓→power [W] P_src= P_SOL *Ly*Lz/(2*math.pi*R*Lpol) -- fraction of total SOL power into flux tube ˓→domain [W] xSource=R -- source peak radial location [m] lambdaSource= 0.005 -- source radial width [m]

-- Source density and temperature profiles. -- Note that source density will be scaled to achieve desired source power. sourceDensity= function (t, xn) local x, y, z= xn[1], xn[2], xn[3] local sourceFloor= 1e-10 if math.abs(z)< Lz/4 then -- near the midplane, the density source is a Gaussian return math.max(math.exp(-(x-xSource)^2/(2*lambdaSource)^2), sourceFloor) else return 1e-40 end end sourceTemperature= function (t, xn) local x, y, z= xn[1], xn[2], xn[3] if math.abs(x-xSource)<3 *lambdaSource then return 80*eV else return 30*eV end end

This concludes the Preamble. We now have everything we need to initialize the Gyrokinetic App. In this input ﬁle, the App initialization consists of 4 sections:

------App initialization ------plasmaApp= Plasma.App { ------Common ------...

------Species (continues on next page)

28 Chapter 2. Quickstart gkyl Documentation, Release 2.0-alpha

(continued from previous page) ------...

------Fields ------...

------Geometry ------... }

• The Common section includes a declaration of parameters that control the (conﬁguration space) discretization, and time advancement. This ﬁrst block of code in Plasma.App may specify the periodic directions, the MPI decomposition, and the frequency with which to output certain diagnostics.

------Common ------logToFile= true, -- will write simulation output log to gk-sheath_

˓→0.log tEnd= 10e-6, -- simulation end time [s] nFrame= 10, -- number of output frames for diagnostics lower={R- Lx/2,-Ly/2,-Lz/2}, -- configuration space domain lower bounds, {x_

˓→min, y_min, z_min} upper={R+ Lx/2, Ly/2, Lz/2}, -- configuration space domain upper bounds, {x_

˓→max, y_max, z_max} cells={4,1,8}, -- number of configuration space cells, {nx, ny,

˓→nz} basis="serendipity", -- basis type (only "serendipity" is supported

˓→for gyrokinetics) polyOrder=1, -- polynomial order of basis set (polyOrder = 1

˓→fully supported for gyrokinetics, polyOrder = 2 marginally supported) timeStepper="rk3", -- timestepping algorithm cflFrac= 0.4, -- fractional modifier for timestep calculation

˓→via CFL condition restartFrameEvery= .2, -- restart files will be written after every 20%

˓→of simulation

-- Specification of periodic directions -- (1-based indexing, so x-periodic = 1, y-periodic = 2, etc) periodicDirs={2}, -- Periodic in y only (y = 2nd dimension)

• The Species section sets up the species to be considered in the simulation. Each species gets its own Lua table, in which one provides the velocity-space domain and discretization of the species, initial conditions, sources, collisions, boundary conditions, and diagnostics. In this input ﬁle, we initialize gyrokinetic electron and ion species. Since this section is the most involved part of the input ﬁle, we will discuss various parts in detail below.

------Species ------(continues on next page)

2.3. Gyrokinetic example 29 gkyl Documentation, Release 2.0-alpha

(continued from previous page) -- Gyrokinetic electrons electron= Plasma.Species { evolve= true, -- evolve species? charge= qe, -- species charge mass= me, -- species mass

-- Species-specific velocity domain lower={-4 *vte,0}, -- velocity space domain lower bounds, ˓→{vpar_min, mu_min} upper={4 *vte, 12*me*vte^2/(2*B0)}, -- velocity space domain upper bounds, ˓→{vpar_max, mu_max} cells={8,4}, -- number of velocity space cells, {nvpar,

˓→ nmu}

-- Initial conditions init= Plasma.MaxwellianProjection { -- initialize a Maxwellian with the

˓→specified density and temperature profiles -- density profile density= function (t, xn) -- The particular functional form of the initial density profile -- comes from a 1D single-fluid analysis (see Shi thesis), which

˓→derives -- quasi-steady-state initial profiles from the source parameters. local x, y, z, vpar, mu= xn[1], xn[2], xn[3], xn[4], xn[5] local Ls= Lz/4 local floor= 0.1 local effectiveSource= math.max(sourceDensity(t,{x,y,0}), floor) local c_ss= math.sqrt(5/3 *sourceTemperature(t,{x,y,0})/mi) local nPeak=4 *math.sqrt(5)/3/c_ss*Ls*effectiveSource/2 local perturb=0 if math.abs(z) <= Ls then return nPeak*(1+math.sqrt(1-(z/Ls)^2))/2*(1+perturb) else return nPeak/2*(1+perturb) end end, -- temperature profile temperature= function (t, xn) local x= xn[1] if math.abs(x-xSource)<3 *lambdaSource then return 50*eV else return 20*eV end end, scaleWithSourcePower= true, -- when source is scaled to achieve

˓→desired power, scale initial density by same factor },

-- Collisions parameters coll= Plasma.LBOCollisions { -- Lenard-Bernstein model collision

˓→operator collideWith={'electron'}, -- only include self-collisions with

˓→electrons frequencies= {nuElc}, -- use a constant (in space and time)

˓→collision freq. (calculated in Preamble) }, (continues on next page)

30 Chapter 2. Quickstart gkyl Documentation, Release 2.0-alpha

(continued from previous page)

-- Source parameters source= Plasma.MaxwellianProjection { -- source is a Maxwellian with the

˓→specified density and temperature profiles isSource= true, -- designate as source density= sourceDensity, -- use sourceDensity function

˓→(defined in Preamble) for density profile temperature= sourceTemperature, -- use sourceTemperature function

˓→(defined in Preamble) for temperature profile power= P_src/2, -- sourceDensity will be scaled to

˓→achieve desired power },

-- Non-periodic boundary condition specification bcx= {Plasma.Species.bcZeroFlux, Plasma.Species.bcZeroFlux}, -- use zero-flux

˓→boundary condition in x direction bcz= {Plasma.Species.bcSheath, Plasma.Species.bcSheath}, -- use sheath-

˓→model boundary condition in z direction

-- Diagnostics diagnosticMoments={"GkM0","GkUpar","GkTemp"}, diagnosticIntegratedMoments={"intM0","intM1","intKE","intHE","intSrcKE"}, diagnosticBoundaryFluxMoments={"GkM0","GkUpar","GkEnergy"}, diagnosticIntegratedBoundaryFluxMoments={"intM0","intM1","intKE","intHE"}, },

-- Gyrokinetic ions ion= Plasma.Species { evolve= true, -- evolve species? charge= qi, -- species charge mass= mi, -- species mass

-- Species-specific velocity domain lower={-4 *vti,0}, -- velocity space domain lower bounds, ˓→{vpar_min, mu_min} upper={4 *vti, 12*mi*vti^2/(2*B0)}, -- velocity space domain upper bounds, ˓→{vpar_max, mu_max} cells={8,4}, -- number of velocity space cells, {nvpar,

˓→ nmu}

-- Initial conditions init= Plasma.MaxwellianProjection { -- initialize a Maxwellian with the

2.3. Gyrokinetic example 31 gkyl Documentation, Release 2.0-alpha

(continued from previous page) return nPeak*(1+math.sqrt(1-(z/Ls)^2))/2*(1+perturb) else return nPeak/2*(1+perturb) end end, -- temperature profile temperature= function (t, xn) local x= xn[1] if math.abs(x-xSource)<3 *lambdaSource then return 50*eV else return 20*eV nuIon= nuFrac *logLambdaIon*eV^4*n0/(12*math.pi^(3/2)*eps0^2*math.sqrt(mi)*(Ti0)^(3/ ˓→2))

-- Derived parameters vti= math.sqrt(Ti0/mi) -- ion thermal speed vte= math.sqrt(Te0/me) -- electron thermal speed c_s= math.sqrt(Te0/mi) -- ion sound speed omega_ci= math.abs(qi *B0/mi) -- ion gyrofrequency rho_s= c_s/omega_ci -- ion sound gyroradius

-- Simulation box size Lx= 50 *rho_s -- x = radial direction Ly= 100 *rho_s -- y = binormal direction Lz=4 -- z = field-aligned direction

-- Source parameters P_SOL= 3.4e6 -- total SOL power, from experimental heating

32 Chapter 2. Quickstart gkyl Documentation, Release 2.0-alpha

(continued from previous page) return 30*eV end end

This concludes the Preamble. We now have everything we need to initialize the Gyrokinetic App. In this input ﬁle, the App initialization consists of 4 sections:

------App initialization ------plasmaApp= Plasma.App { ------Common ------...

------Species ------... scaleWithSourcePower= true, -- when source is scaled to achieve

˓→desired power, scale initial density by same factor },

-- Collisions parameters coll= Plasma.LBOCollisions { -- Lenard-Bernstein model collision operator collideWith={'ion'}, -- only include self-collisions with ions frequencies= {nuIon}, -- use a constant (in space and time) collision

˓→freq. (calculated in Preamble) },

-- Source parameters source= Plasma.MaxwellianProjection { -- source is a Maxwellian with the

˓→specified density and temperature profiles isSource= true, -- designate as source density= sourceDensity, -- use sourceDensity function

˓→(defined in Preamble) for density profile temperature= sourceTemperature, -- use sourceTemperature function

˓→(defined in Preamble) for temperature profile power= P_src/2, -- sourceDensity will be scaled to

˓→achieve desired power },

-- Non-periodic boundary condition specification bcx= {Plasma.Species.bcZeroFlux, Plasma.Species.bcZeroFlux}, -- use zero-flux

˓→boundary condition in x direction bcz= {Plasma.Species.bcSheath, Plasma.Species.bcSheath}, -- use sheath-

˓→model boundary condition in z direction

The initial condition for this problem is given by a Maxwellian. This is speciﬁed using init = Plasma.

2.3. Gyrokinetic example 33 gkyl Documentation, Release 2.0-alpha

MaxwellianProjection { ... }, which is a table with entries for the density and temperature profile functions (we could also specify the driftSpeed profile) to be used to initialze the Maxwellian. In this simulation, the initial density profile takes a particular form that comes from a 1D single-fluid analysis (see [Shi2019]), which derives quasi-steady-state initial profiles from the source parameters. The sources also take the form of Maxwellians, specified via source = Plasma.MaxwellianProjection { isSource = true, ... }. For the density and temperature profile functions, we use the sourceDensity and sourceTemperature functions defined in the Preamble. We also specify the desired source power. The source density is then scaled so that the integrated power in the source matches the desired power. Therefore, sourceDensity only controls the shape of the source density profile, not the amplitude. Since the initial conditions are related to the source, we also scale the initial species density by the same factor as the source via the scaleWithSourcePower = true flag in the initial conditions. Self-species collisions are included using a Lenard-Bernstein model collision operator via the coll = Plasma. LBOCollisions { ... } table. For more details about collision models and options, see Collisions. Non-periodic boundary conditions are specified via the bcx and bcz tables. For this simulation, we use zero-flux boundary conditions in the 푥 (radial) direction, and sheath-model boundary conditions in the 푧 (field-aligned) direction. Finally, we specify the diagnostics that should be outputted for each species. These consist of various moments and integrated quantities. For more details about available diagnostics, see Gyrokinetic App: Electromagnetic gyrokinetic model for magnetized plasmas. • The Fields section specifies parameters and options related to the field solvers for the gyrokinetic potential(s).

------Fields ------Gyrokinetic field(s) field= Plasma.Field { evolve= true, -- Evolve fields? isElectromagnetic= false, -- use electromagnetic GK by including magnetic vector

˓→potential A_parallel?

-- Non-periodic boundary condition specification for electrostatic potential phi -- Dirichlet in x. phiBcLeft={T="D",V= 0.0}, phiBcRight={T="D",V= 0.0}, -- Periodic in y. -- -- No BC required in z (Poisson solve is only in perpendicular x,y directions) },

• The Geometry section speciﬁes parameters related to the background magnetic ﬁeld and other geometry parameters.

------Geometry ------Magnetic geometry funcField= Plasma.Geometry { -- Background magnetic field profile -- Simple helical (i.e. cylindrical slab) geometry is assumed bmag= function (t, xn) local x= xn[1] return B0*R/x end,

34 Chapter 2. Quickstart gkyl Documentation, Release 2.0-alpha

quantity is controlled by the nFrame parameter in the input ﬁle. We can use the Gkeyll post-processing tool (postgkyl) to visualize the outputs.

Electron density

First, let’s examine the initial conditions, which are given in output ﬁle sending in _0.bp. The initial electron density 푛푒(푥, 푦, 푧) is found in gk-sheath_electron_GkM0_0.bp, where GkM0 is the label for the density moment. Let’s look at this ﬁle as a function of the 푥 and 푧 coordintes by taking a line-out at 푦 = 0 via

pgkyl gk-sheath_electron_GkM0_0.bp interp sel --z10. pl -x '$x$' -y '$z$'

where we have used the interp (interpolate) command to interpolate the DG data onto the grid, and the sel --z1 0. (select) command to make the line-out at 푦 = 0 (--z1 refers to the 푦 coordinate here). The resulting plot looks like

Fig. 7: Initial electron density 푛푒(푥, 푦 = 0, 푧, 푡 = 0)

We ran this simulation for 10 휇s, and since nframe=10 we have an output frame for each 휇s of the simulation. Let’s look at the ﬁnal state now, at 푡 = 10휇s.

pgkyl gk-sheath_electron_GkM0_10.bp interp sel --z10. pl -x '$x$' -y '$z$'

gives

Fig. 8: Electron density 푛푒(푥, 푦 = 0, 푧, 푡 = 10휇s)

2.3. Gyrokinetic example 35 gkyl Documentation, Release 2.0-alpha

Sheath potential

Now let’s look at the electrostatic potential, 휑. We’d like to see if the sheath potential formed self-consistently due to our conducting-sheath boundary conditions. Let’s look at 휑 along the ﬁeld line (i.e. along the 푧 coordinate) by taking line-outs at 푥 = 1.0 and 푦 = 0. pgkyl gk-sheath_phi_10.bp interp sel --z01. --z10. pl -x '$z$' gives

Fig. 9: Electrostatic potential 휑(푥 = 1, 푦 = 0, 푧, 푡 = 10휇s)

Indeed, at the domain ends in 푧, we have a sheath potential 휑푠ℎ = 90 V. We can also make an animation of the evolution of the sheath potential via pgkyl "gk-sheath_phi_[0-9]*.bp" interp sel --z01. --z10. anim -x '$z$'

Particle balance

We can examine particle balance between the sources and sinks (from end losses to the wall via the sheath) by looking at the electron_intM0.bp (integrated electron density) file and other related files. By using the ev (evaluate) command, we can combine various quantities. ev is extremely useful and flexible, but it can lead to some complicated pgkyl commands. For this plot, the full command that we’ll use is pgkyl gk-sheath_electron_intM0.bp -l 'total' gk-sheath_electron_intSrcM0.bp -l

˓→'sources' \ gk-sheath_electron_intM0FluxZlower.bp gk-sheath_electron_intM0FluxZupper.bp \ ev -g -l 'sinks' 'f[2] f[3] + -1 *' ev -g -l 'sources + sinks' 'f[1] f[-1] +' \ ev -g -l 'total - (sources + sinks)' 'f[0] f[-1] -' activate -i0,1,-3,-2,-1 plot -x

˓→'time (s)' -f0

Note: The above pgkyl command could use tags instead of dataset indices as follows: pgkyl gk-sheath_electron_intM0.bp -l 'total' -t tot gk-sheath_electron_intSrcM0.bp -l

˓→'sources' -t src \ gk-sheath_electron_intM0FluxZlower.bp -t fluxL gk-sheath_electron_intM0FluxZupper.bp

˓→-t fluxU \ ev -g -l 'sinks' -t sinks 'fluxL fluxU + -1 *' ev -g -l 'sources + sinks' -t ˓→srcPsinks 'src sinks +' \ ev -g -l 'total - (source + sinks)' -t bal 'tot srcPsinks -' activate -t tot,src,

˓→sinks,srcPsinks,bal pl -f0

36 Chapter 2. Quickstart gkyl Documentation, Release 2.0-alpha

Let’s break this command down a bit. We ﬁrst load all the data ﬁles that we need: pgkyl gk-sheath_electron_intM0.bp -l 'total' gk-sheath_electron_intSrcM0.bp -l

˓→'sources' \ gk-sheath_electron_intM0FluxZlower.bp gk-sheath_electron_intM0FluxZupper.bp \

Here gk-sheath_electron_intM0.bp is the (total) integrated electron density, gk-sheath_electron_intSrcM0.bp is the integrated electron source density, gk-sheath_electron_intM0FluxZlower.bp is the integrated particle flux to the lower divertor plate, and gk-sheath_electron_intM0FluxZupper.bp is the integrated particle flux to the upper plate. We’ve used the -l flag to label the first two of these as 'total' and 'sources'. Next, we use the ev command to sum the fluxes and change the sign so that the result is negative: ev -g -l 'sinks' 'f[2] f[3] + -1 *'

Here, f2 refers to the 3rd loaded file (active dataset 2, with 0-based indexing) and f3 the 4th loaded file (active dataset 3); these are the two Flux files. The ev command uses reverse Polish notation, so that this command translates to -(f2 + f3). This creates a new dataset at the end of the stack, which can be indexed as dataset -1. We label this dataset as 'sinks'. Next, we want to sum the sources and the sinks. To do this, we sum the 'source' dataset (dataset 1 from the original loading) and the 'sinks' dataset (dataset -1, which we just created with ev), via ev -g -l 'sources + sinks' 'f[1] f[-1] +'

This pushes another, new dataset to the stack, which we label as 'sources and sinks'. This becomes dataset -1 and pushes the 'sinks' dataset back to dataset -2. Next, we use ev to compute the difference between the 'total' dataset (dataset 0) and the 'sources + sinks' dataset (dataset -1), via ev -g -l 'total - (sources + sinks)' 'f[0] f[-1] -'

Again, this pushes another dataset to the stack, which we label as 'total - (sources + sinks)'. Now we have computed everything we need. We just need to activate all the datasets we would like to plot, and plot them. We do this with activate -i0,1,-3,-2,-1 pl -x 'time (s)' -f0 with the -f0 ﬂag to put all the lines on the same ﬁgure. The end result is

Fig. 10: Electron particle balance

The ﬂat purple line shows that electron density is conserved after accounting for sources and sinks.

2.3. Gyrokinetic example 37 gkyl Documentation, Release 2.0-alpha

Divertor Fluxes

pgkyl gk-sheath_ion_GkM0FluxZlower_10.bp interp ev 'f[0] 1,2 avg' pl -x '$x$'

Here we use ev to average in the 푦 and 푧 direction (for boundary ﬂuxes, an average in the boundary direction is always required). This results in

Fig. 11: Ion particle ﬂux to lower divertor at t=10 휇s

The ion energy (heat) ﬂux proﬁle can similarly be plotted via

pgkyl gk-sheath_ion_GkEnergyFluxZlower_10.bp interp ev 'f[0] 1,2 avg' pl -x '$x$'

Fig. 12: Ion heat ﬂux to lower divertor at t=10 휇s

Suppose instead of the instantaneous ﬂux, we want the time-averaged ﬂux over some period of time, perhaps from 5-10 휇s. To compute this, we can use

pgkyl "gk-sheath_ion_GkEnergyFluxZlower_*.bp" interp collect \ sel --z05:10 ev 'f[0] 0,2,3 avg' pl -x '$x$'

This uses the collect command to aggregate the frames into a time dimension, which becomes coordinate 0. We then use sel --z0 5:10 to select frames 5-10. Then we use ev 'f[-1] 0,2,3 avg' to average the data in the 0th (time), 2nd (푦), and 3rd (푧) dimensions. This gives

38 Chapter 2. Quickstart gkyl Documentation, Release 2.0-alpha

Fig. 13: Time-averaged ion heat ﬂux to lower divertor (t= 5-10 휇s)

2.3.3 References

2.4 Fluid example

Contents

• Fluid example – Physics model and initial conditions – Input ﬁle – Running the simulation – Postprocessing

* Plotting all physical quantities in a frame * Plotting one physical quantity * Plotting integrated quantities over time – References

2.4.1 Physics model and initial conditions

This simulation models 2d fast magnetic reconnection in a two-fluid (electron-ion) plasma (see [Birn2001] for the classical paper on this topic and [Wang2015] for our earlier application). The electron and ion fluid spcies are evolved using the five-moment model 휕휌 푠 + ∇ · (휌 u ) = 0, 휕푡 푠 푠 휕 (휌 u ) 푠 푠 + ∇푝 + ∇ · (휌 u u ) = 푛 푞 (E + u × B) , 휕푡 푠 푠 푠 푠 푠 푠 휕ℰ 푠 + ∇ · [u (푝 + ℰ )] = 푛 푞 u · E, 휕푡 푠 푠 푠 푠 푠 푠

2.4. Fluid example 39 gkyl Documentation, Release 2.0-alpha

3 1 2 where 휌푠, u푠, 푝푠, and ℰ = 2 푝푠 + 2 휌푠u푠 are the mass density, velocity, thermal pressure, and total energy of the species 푠. The electromagnetic ﬁelds are evolved and coupled to the plasma species by the usual Maxwell’s equations

1 휕E ∑︁ = ∇ × B − 휇 J , 푐2 휕푡 0 푠 푠 휕B = −∇ × E. 휕푡 The algorithms are summarized in more detail in Moments App: Multifluid-moment-Maxwell model, as well as in [Hakim2006], [Hakim2008], and [Wang2020]. The simulation input file is adapted from those used in our earlier work presented in [Wang2015], though only a five- moment version is presented here. We employ a 2D simulation domain that is periodic in 푥 (−퐿푥/2 < 푥 < 퐿푥/2) and √︁ 2 is bounded by conducting walls in 푦 at 푦 = ±퐿푦/2. Here 퐿푥 = 100푑푖0, 퐿푦 = 50푑푖0, where 푑푖0 = 푚푖/휇0푛0 |푒| is the ion inertia length due to 푛0. The initial equilibrium is a single Harris sheet where magnetic field and number densities are specified by

B0 = 퐵0 tanh (푦/휆퐵) xˆ,

and

2 푛푒 = 푛푖 = 푛0sech (푦/휆퐵) + 푛푏,

respectively. The total current, J0 = ∇ × B0/휇0, is decomposed according to 퐽푧푒0/퐽푧푖0 = 푇푖0/푇푒0, where the initial temperatures 푇푒0 and 푇푖0 are constant. A sinusoid perturbation is then applied on the magnetic ﬁeld according to 훿B = zˆ × ∇휓, where

휓 = 훿휓 cos (2휋푥/퐿푥) cos (휋푦/퐿푦) .

2.4.2 Input ﬁle

1 ------

2 -- App dependencies

3 ------

4 -- Load the App and equations to be used

5 local Moments= require("App.PlasmaOnCartGrid").Moments()

6 local Euler= require"Eq.Euler"

9 ------

10 -- Preamble

11 ------

12 -- Basic constants

13 local gasGamma= 5./3. -- gas gamma

14 local elcCharge=-1.0 -- electron charge

15 local ionCharge= 1.0 -- ion charge

16 local elcMass= 1.0 -- electron mass

17 local lightSpeed= 1.0 -- light speed

18 local mu0= 1.0 -- mu_0

20 -- Problem-specific characteristic parameters

21 local n0= 1.0 -- characteristic density variation

22 local nb_n0= 0.2 -- background density vs. varation density

23 local plasmaBeta0= 1.0 -- thermal pressure vs. magnetic pressure

24 local lambda_di0= 0.5 -- transition layer thickness over ion inertial length (continues on next page)

40 Chapter 2. Quickstart gkyl Documentation, Release 2.0-alpha

(continued from previous page)

25 local Ti0_Te0= 5.0 -- ion temperature vs. electron temperature

26 local massRatio= 25.0 -- ion mass vs. electron mass

27 local vAe0_lightSpeed= 0.5 -- electron Alfven speed vs. c

28 local perturbationLevel= 0.1 -- relative perturbation magnitude

30 -- Derived parameters for problem setup or for diagnostic information

31 local epsilon0= 1.0/ lightSpeed^2/ mu0 -- epsilon_0 32 local ionMass= elcMass * massRatio -- ion mass 33 local vAe0= lightSpeed * vAe0_lightSpeed -- electron Alfven speed 34 local vAlf0= vAe0 * math.sqrt(elcMass/ ionMass) -- plasma Alfven speed 35 local B0= vAlf0 * math.sqrt(n0 * ionMass) -- background B field 36 local OmegaCi0= ionCharge * B0/ ionMass -- ion cyclotron frequency 37 local OmegaPe0= math.sqrt(n0 * elcCharge^2/ (epsilon0 * elcMass)) 38 -- electron plasma frequency

39 local de0= lightSpeed/ OmegaPe0 -- electron inertial length 40 local OmegaPi0= math.sqrt(n0 * ionCharge^2/ (epsilon0 * ionMass)) 41 -- ion plasma frequency

42 local di0= lightSpeed/ OmegaPi0 -- ion inertial length 43 local lambda= lambda_di0 * di0 -- transition layer thickness 44 local psi0= perturbationLevel * B0 -- potential of perturbation B ˓→field

46 -- Domain and time control 47 local Lx, Ly= 25.6 * di0, 12.8 * di0 -- domain lengths 48 local Nx, Ny= 128, 64 -- grid size

49 local tEnd= 25.0/ OmegaCi0 -- end of simulation

50 local nFrame=5 -- number of output frames at t=tEnd

53 ------

54 -- App construction

55 ------

56 local momentApp= Moments.App {

57 ------

58 -- Common

59 ------

60 logToFile= true, -- will log to rt-5m-gem_0.log

61 tEnd= tEnd, -- stop the simulation at t=tEnd

62 nFrame= nFrame, -- number of output frames at t=tEnd

63 lower={-Lx/2,-Ly/2}, -- lower limit of domain in each direction

64 upper= {Lx/2, Ly/2}, -- upper limit of domain in each direction

65 cells= {Nx, Ny}, -- number of cells in each direction

66 timeStepper="fvDimSplit", -- always use fvDimSplit for fluid simulations,

67 -- i.e., finite-volume dimensional-splitting

68 periodicDirs={1}, -- periodic boundary condition directions

70 ------

71 -- Species

72 ------

73 -- electrons

74 elc= Moments.Species {

75 charge= elcCharge, mass= elcMass, -- charge and mass

76 equation= Euler { gasGamma= gasGamma },

77 -- default equation to be evolved

78 -- using 2nd-order scheme

79 equationInv= Euler { gasGamma= gasGamma, numericalFlux="lax"},

80 -- backup equation to be solved to (continues on next page)

2.4. Fluid example 41 gkyl Documentation, Release 2.0-alpha

(continued from previous page)

81 -- retake the time step in case of

82 -- negative density/pressure

83 init= function (t, xn) -- initial condition to set the

84 -- moments of this species

85 local x, y= xn[1], xn[2]

87 local TeFrac= 1.0/(1.0+ Ti0_Te0)

88 local sech2=(1.0/ math.cosh(y/ lambda))^2 89 local n= n0 * (sech2+ nb_n0) 90 local Jz=-(B0/ lambda) * sech2 91 local Ttotal= plasmaBeta0 * (B0 * B0)/ 2.0/n0 92 93 local rho_e=n * elcMass 94 local rhovx_e= 0.0

95 local rhovy_e= 0.0 96 local rhovz_e= (elcMass/ elcCharge) * Jz * TeFrac 97 local thermalEnergy_e=n * Ttotal * TeFrac/ (gasGamma-1.0) 98 local kineticEnergy_e= 0.5 * rhovz_e * rhovz_e/ rho_e 99 local e_e= thermalEnergy_e+ kineticEnergy_e

100

101 return rho_e, rhovx_e, rhovy_e, rhovz_e, e_e

102 end,

103 evolve= true, -- whether to evolve this species

104 -- in the hyperbolic step; it could

105 -- still be evolved in the source

106 -- update step, though

107 bcy= { Euler.bcWall, Euler.bcWall }, -- boundary conditions along

108 -- non-periodic directions; in this

109 -- case, the y direction

110 },

111

112 -- ions

113 ion= Moments.Species {

114 charge= ionCharge, mass= ionMass,

115 equation= Euler { gasGamma= gasGamma },

116 equationInv= Euler { gasGamma= gasGamma, numericalFlux="lax"},

117 init= function (t, xn)

118 local x, y= xn[1], xn[2]

119

120 local TiFrac= Ti0_Te0/(1.0+ Ti0_Te0)

121 local sech2=(1.0/ math.cosh(y/ lambda))^2 122 local n= n0 * (sech2+ nb_n0) 123 local Jz=-(B0/ lambda) * sech2 124 local Ttotal= plasmaBeta0 * (B0 * B0)/ 2.0/ n0 125 126 local rho_i=n *ionMass 127 local rhovx_i= 0.0

128 local rhovy_i= 0.0 129 local rhovz_i= (ionMass/ ionCharge) * Jz * TiFrac 130 local thermalEnergy_i=n * Ttotal * TiFrac/ (gasGamma-1) 131 local kineticEnergy_i= 0.5 * rhovz_i * rhovz_i/ rho_i 132 local e_i= thermalEnergy_i+ kineticEnergy_i

133

134 return rho_i, rhovx_i, rhovy_i, rhovz_i, e_i

135 end,

136 evolve= true,

137 bcy= { Euler.bcWall, Euler.bcWall }, (continues on next page)

42 Chapter 2. Quickstart gkyl Documentation, Release 2.0-alpha

(continued from previous page)

138 },

139

140 ------

141 -- Electromagnetic field

142 ------

143 field= Moments.Field {

144 epsilon0= epsilon0, mu0= mu0,

145 init= function (t, xn)

146 local x, y= xn[1], xn[2]

147 local pi= math.pi

148 local sin= math.sin

149 local cos= math.cos

150

151 local Ex, Ey, Ez= 0.0, 0.0, 0.0

152 153 local Bx= B0 * math.tanh(y/ lambda) 154 local By= 0.0

155 local Bz= 0.0

156 157 local dBx=-psi0 *(pi/ Ly) * cos(2 * pi * x/ Lx) * sin(pi * y/ Ly) 158 local dBy= psi0 * (2 * pi/ Lx) * sin(2 * pi * x/ Lx) * cos(pi * y/ Ly) 159 local dBz= 0.0

160

161 Bx= Bx+ dBx

162 By= By+ dBy

163 Bz= Bz+ dBz

164

165 return Ex, Ey, Ez, Bx, By, Bz

166 end,

167 evolve= true, -- whether to evolve the field in the hyperbolic step; even

168 -- if this is set to false, the E field could still be

169 -- evolved in the source update step

170 bcy= { Moments.Field.bcReflect, Moments.Field.bcReflect },

171 -- boundary conditions along the nonperiodic directions,

172 -- in this case, the y direction.

173 },

174

175 ------

176 -- Source equations that couple the plasma fluids and EM fields

177 ------

178 emSource= Moments.CollisionlessEmSource {

179 species={"elc","ion"}, -- plasma species involved

180 timeStepper="direct", -- time-stepper; one of "time-centered",

181 -- "direct", and "exact"

182 },

183

184 }

185

186

187 ------

188 -- Run the App

189 ------

190 momentApp:run()

2.4. Fluid example 43 gkyl Documentation, Release 2.0-alpha

2.4.3 Running the simulation

The input ﬁle rt-5m-gem.lua can be run using the gkyl executable

mpirun -n4 ~/gkylsoft/gkyl/bin/gkyl inputFiles/rt-5m-gem.lua

assuming gkyl has been installed in the user’s home directory. When running this simulation, a user should see the following output

Fri Sep 18 2020 15:53:11.000000000 Gkyl built with e0fd133198bd+ Gkyl built on Sep 11 2020 17:59:49 Initializing PlasmaOnCartGrid simulation ... Using CFL number1 Initializing completed in 0.0723601 sec

Starting main loop of PlasmaOnCartGrid simulation ...

** Time step too large! Aborting this step! ** Time step 1250 too large! Will retake ˓→with dt1 Step0 at time0. Time step1. Completed0% 0123456789 Step 125 at time 125. Time step1. Completed 10% 0123456789 Step 250 at time 250. Time step1. Completed 20% 0123456789 Step 375 at time 375. Time step1. Completed 30% 0123

2.4.4 Postprocessing

The output of this simulation is the following set of files: • Electron fluid moments: rt-5m-gem_elc_#.bp. • Ion fluid moments: rt-5m-gem_elc_#.bp. • Electromagnetic fields: rt-5m-gem_field_#.bp. • Field energy: rt-5m-gem_fieldEnergy.bp. • Diagnostic integrated moments: rt-5m-gem_elc_intMom.bp and rt-5m-gem_in_intMom.bp. Snapshots (frames) are labeled by the number # at the end of the file name, while integrated diagnostics that are computed as a time-series, such as the field energy, are written out as a single file. Since nFrame=1 in the input file, the only frames that are output are 0, corresponding to the initial condition, and 1, corresponding to the end of the simulation.

Plotting all physical quantities in a frame

The following command plot all ﬁve moment term in the 5th electron output frame. Note that the ﬁve moment terms are ordered as 휌, 휌푣푥 휌푣푦, 휌푣푧, ℰ.

pgkyl rt-5m-gem_elc_5.bp plot --fix-aspect

Plotting one physical quantity

The following command plot the 3rd moment term in the 5th electron output frame., The 3rd moment is the out-of- plane momentum, which differs from the electron out-of-plane current by an coefﬁcient 푞푒/푚푒 .

44 Chapter 2. Quickstart gkyl Documentation, Release 2.0-alpha

Fig. 14: All ﬁve moment terms of the electron ﬂuid.

pgkyl rt-5m-gem_elc_5.bp -c3 plot --fix-aspect

Plotting integrated quantities over time

2.4.5 References

2.4. Fluid example 45 gkyl Documentation, Release 2.0-alpha

Fig. 15: Out-of-plane electron current, 휌푢푒,푧, in the 5th (last) output frame.

2 Fig. 16: Integrated magnetic ﬁeld energy, |퐵푧| , plotted as a function of time.

46 Chapter 2. Quickstart CHAPTER 3

gkyl Reference

Below is the (hopefully) complete reference to the gkyl simulation tool, whose data can be post-processed with postgkyl. For instructions on installing postgkyl see gkyl install.

3.1 Using gkyl

Contents

• Using gkyl – Run simulations

* Serial simulations * Parallel simulation * Running on GPUs – Restarts – Using the fromFile option – Handy perks

* Run Lua with gkyl * gkyl Tools * ADIOS tools · bpls · bpdump

We now cover the basics of runing gkyl on desktops, clusters, and GPUs. Gkyl can also be used be used to run any Lua scripts, as well as tools provided within gkyl. We also comment on some useful tools provided by ADIOS.

47 gkyl Documentation, Release 2.0-alpha

Additional details on the contents of the input files can be found in the Input file file basics page and the pages for the Vlasov, Gyrokinetic and Moment Apps. The installation placed the gkyl executable in /gkylsoft/gkyl/bin/ (where the default is home, ~), so one typically needs to call gkyl as /gkylsoft/gkyl/bin/ gkyl. However most users create an alias so one can simply call gkyl. The documentation assumes such alias unless specified otherwise. The gkyl command has a built-in help menu. Access it with

gkyl -h

3.1.1 Run simulations

There are three ways of running simulations with gkyl: • Serial: using a single core/processes/CPU. • Parallel: running a multi-core simulation (using MPI). • GPUs: using graphical processing units (GPUs). The input ﬁle has to have some knowledge of which of these modalities you will use. We provide some examples of each of these below.

Serial simulations

Suppose you have the kbm.lua input file for a linear kinetic ballooning mode (KBM) calculation with gyrokinetics. In the Common section of the App declaration (i.e. between plasmaApp = Plasma.App { and electron = Plasma.Species {) there are two variables, decompCuts and useShared. The refer to the number of MPI decompositions and the use of MPI shared memory, respectively. For serial simulations one can remove these from the input file, or useShared must be set to false, and decompCuts must be a table with as many 1’s as there are configuration space dimensions (three in this case). That’s why the input file contains:

plasmaApp= Plasma.App { ... decompCuts={1,1,1},-- Cuts in each configuration direction. useShared= false,-- If to use shared memory. ... }

Then one can run the input ﬁle in serial with the simple command:

gkyl kbm.lua

By the time it completes, after 54 seconds on a 2015 MacbookPro, this simulation will produce the following output to screen:

1 Thu Sep 17 2020 22:20:16.000000000

2 Gkyl built with 1b66bd4a21e5+

3 Gkyl built on Sep 17 2020 22:20:05

4 Initializing Gyrokinetic simulation ...

5 Initializing completed in 12.9906 sec

7 Starting main loop of Gyrokinetic simulation ... (continues on next page)

48 Chapter 3. gkyl Reference gkyl Documentation, Release 2.0-alpha

(continued from previous page)

9 Step0 at time0. Time step1.11219e-08. Completed0%

10 0123456789 Step 27 at time3.00286e-07. Time step1.11215e-08. Completed 10%

11 0123456789 Step 54 at time6.00559e-07. Time step1.1121e-08. Completed 20%

12 0123456789 Step 80 at time8.89697e-07. Time step1.11204e-08. Completed 30%

13 0123456789 Step 107 at time1.18994e-06. Time step1.11197e-08. Completed 40%

14 0123456789 Step 133 at time1.47904e-06. Time step1.11189e-08. Completed 50%

15 0123456789 Step 160 at time1.77924e-06. Time step1.11179e-08. Completed 60%

16 0123456789 Step 186 at time2.06828e-06. Time step1.11165e-08. Completed 70%

17 0123456789 Step 213 at time2.3684e-06. Time step1.11145e-08. Completed 80%

18 0123456789 Step 239 at time2.65735e-06. Time step1.11121e-08. Completed 90%

19 0123456789 Step 266 at time2.94849e-06. Time step2.27109e-09. Completed 100%

20 0

21 Total number of time-steps 267

22 Solver took 25.14505 sec(0.094176 s/step)(46.493%)

23 Solver BCs took2.14804 sec(0.008045 s/step)(3.972%)

24 Field solver took0.58969 sec(0.002209 s/step)(1.090%)

25 Field solver BCs took0.20732 sec(0.000776 s/step)(0.383%)

26 Function field solver took0.00000 sec(0.000000 s/step)(0.000%)

27 Moment calculations took 18.12544 sec(0.067886 s/step)(33.514%)

28 Integrated moment calculations took4.57880 sec(0.017149 s/step)(8.466%)

29 Field energy calculations took0.03020 sec(0.000113 s/step)(0.

˓→056%)

30 Collision solver(s) took0.00000 sec(0.000000 s/step)(0.000%)

31 Collision moments(s) took0.00000 sec(0.000000 s/step)(0.000%)

32 Source updaters took0.00000 sec(0.000000 s/step)(0.000%)

33 Stepper combine/copy took1.39611 sec(0.005229 s/step)(2.581%)

34 Time spent in barrier function 0.14791 sec(0.000554 s/step)(0.

˓→273%)

35 [Unaccounted for]1.86320 sec(0.006978 s/step)(3.445%)

37 Main loop completed in 54.08386 sec(0.202561 s/step)(

˓→100%)

39 Thu Sep 17 2020 22:21:23.000000000

These simulation logs contain the following:

Line 1: start date and time. Lines 2-3: gkyl repository revision with which this simulation was run, and the date on which the executable was built. Line 9: report the initial time step number, time and initial time step size. Lines 10-19: report progress every 1% of the simulation (ﬁrst column). Then, every 10% of the simulation time, give the number of time steps taken so far, simulation time transcurred, and the latest time step size. Lines 21-37: give various metrics regarding the time-steps and wall-clock time taken by the simulation, and the time spent on various parts of the calculation. Line 39: Date and time when the simulation ﬁnished.

Also, by default gkyl produces a log ﬁle with the format _0.log. If you wish to disable this set logToFile = false, in the Common section of the App.

3.1. Using gkyl 49 gkyl Documentation, Release 2.0-alpha

Parallel simulation

For large problems running on a single CPU can lead to impractical runtimes. In those cases one benefits from parallelizing the simulation over many CPUs. This is accomplished in gkyl by decomposing the (phase) space into MPI domains. Therefore, in order to run parallel simulations you must have a parallel installation of gkyl, as most installations typically are. Suppose one wishes to run the kinetic ballooning mode (KBM) calculation in the previous section on a node with 16 cores, using 4 MPI processes along 푦 and 4 along 푧. In this case one must edit the variable decompCuts in the Common of the input file to reflect this decomposition: plasmaApp= Plasma.App { ... decompCuts={1,4,4},-- Cuts in each configuration direction. useShared= false,-- If to use shared memory. ... }

Once decompCuts and the rest of the input ﬁle is set appropriately, you can run the simulation with the MPI executable provided by your cluster or MPI implementation (e.g. mpirun, mpiexec, srun, ibrun). For example, with mpirun we would run the simulation as mpirun-n 16 gkyl kbm.lua

The argument following -n is the total number of MPI processes to launch, in this case 4 × 4 = 16. This clearly requires that your computer/node/job has access to at least 16 cores.

Note: The number of decompCuts in any dimension should not exceed the number of cells in that dimension.

Note: •( This feature may be superseeded soon) One can request additional parallelism in velocity space for kinetic simulations by setting useShared = true. This enables MPI shared memory. In this case the decompCuts must specify the number of nodes and not number of processors. That is, the total number of processors will be determined from decompCuts and the number of threads per node.

On many computer clusters where one may run parallel simulations one must submit scripts in order to submit a job. This jobscript causes the simulation to be queued so that it runs once resources (i.e. cores, nodes) become available. When resources are ﬁnally available the simulation runs in a compute node (instead of the login node). Jobscripts for some machines are provided below. Note that the installation instructions point to machine scripts for building gkyl on each of these computers. If you need assistance with setting up gkyl in a new cluster, see this or feel free to contact the developers. Sample submit scripts: • NERSC’s Cori. • TACC’s Stampede2. • TACC’s Frontera. • MIT’s Engaging. • Princeton’s Eddy. • Princeton’s Adroit.

50 Chapter 3. gkyl Reference gkyl Documentation, Release 2.0-alpha

• Princeton’s Stellar.

Running on GPUs

Gkyl is also capable of running on graphical processing units (GPUs) with minimal modifiation of an input file that you would use to run on CPUs. Our implementation of GPU capabilities uses CUDA. At the moment, if gkyl was built with CUDA and the node one is performing the computation in has a GPU, it will default to running the calculation in a GPU. So given an input file cudaInputFile.lua, we would simply run it with

gkyl cudaInputFile.lua

On clusters is often common to submit scripts that queue the job for running on compute nodes (when the resources become available). In fact this is often preferable to ssh-ing into a node if that is even possible. Some sample job scripts for running parallel (CPU) jobs were given in the previous section, and below we provide some sample jobscripts for submitting GPU jobs: • PPPL’s Portal. • Princeton’s Adroit. Some usage and development notes regarding gkyl’s GPU capabilities can be found in this repository.

3.1.2 Restarts

Sometimes a simulations ends prematurely (e.g. your job’s wallclock time allocation ran out), or perhaps it ended successfully but now you wish to run it longer. In these cases one can restart the simulation. The first simulation prints out a number of restart files, those ending in _restart.bp. In order to begin a second simulation from where the first left off, check the tEnd and nFrame variables in the input file. These are defined as absolute times/number of frames, that is, they specify the final simulation time and number of ouput frames from the beginning of the first simulation, not relative to the previous simulation. So suppose we run simulation 1 with the following in the App’s Common section:

momentApp= Moments.App { ... tEnd= 10.0, nFrame= 100, ... }

There are two restart scenarios: • If the simulation completes successfully, one must increase tEnd and nFrame in order to run the second, restart simulation. Otherwise it will just initialize, realize it does not need to advance any further, and terminate. • The first simulation ended prematurely, so tEnd=10.0 was not reached. One can restart the simulation with the same tEnd and nFrame and it will simply try to get there this second time. Or one can increase tEnd and nFrame so the second simulation goes farther than the first one intended to. Once you’ve made the appropriate edits to the input file the second, restart simulation is run by simply appending the word restart after the input file, like This second, restart simulation will use the _restart.bp files of the first simulation to construct an initial condition. Note that it will look for the restart files in the same directory in which the restart simulation is being run, so typically we run restarts in the same directory as the first simulation.

3.1. Using gkyl 51 gkyl Documentation, Release 2.0-alpha

3.1.3 Using the fromFile option

The fromFile option can be used to read data from a file on initialization. This can be used for initial conditions, sources, and geometry data. The file to be read must have the same prefix as the input file but can otherwise be named as desired, including the extension (it might be useful to use a different extension, such as .read, to avoid accidentally deleting needed files if one does rm *.bp).

3.1.4 Handy perks

Run Lua with gkyl

One can use gkyl to run (almost?) any Lua code. Say for example I ﬁnd code in the interverse which promises to compute the factors of “Life, the Universe, and Everything” (who wouldn’t want that?). We can take such code, put it in an input ﬁle named factors.lua and run it with gkyl factors.lua

Try it! It’s free! gkyl Tools

A number of additional tools that users and developers may find useful as part of their (Gkeyll) workflow are shipped as gkyl Tools. One such tool, for example, allows us to compare BP (ADIOS) files. Suppose you ran the plasma beach simulation with the Moment App, using the momBeach.lua input file which contains a variable local J0= 1.0e-12-- Amps/m^3. in the collisionless electromagnetic source. Let’s assume you were scanning this variable, so you may choose to create another input file momBeachS.lua which increases J0 to local J0= 1.0e-10-- Amps/m^3.

If after running momBeachS you are not sure if the results changed at all, you can use the comparefiles tool. For example, compare the electromagnetic ﬁelds produced at the end of both simulations with the following command: gkyl comparefiles-a momBeach_field_100.bp-b momBeachS_field_100.bp

In this particular example the tool would then print the following to screen:

Checking attr numCells in momBeach_field_100.bp momBeach_field_100s.bp ...... comparing numCells Checking attr lowerBounds in momBeach_field_100.bp momBeach_field_100s.bp ...... comparing lowerBounds Checking attr upperBounds in momBeach_field_100.bp momBeach_field_100s.bp ...... comparing upperBounds Checking attr basisType in momBeach_field_100.bp momBeach_field_100s.bp ...... comparing basisType Checking attr polyOrder in momBeach_field_100.bp momBeach_field_100s.bp ...... comparing polyOrder Files are different!

So we know that increasing J0 by a factor of a 100 did change the simulation. Additional documentation of these tools is found in the gkyl Tools reference.

52 Chapter 3. gkyl Reference gkyl Documentation, Release 2.0-alpha

ADIOS tools

ADIOS has two handy tools that one may use to explore data files produced by a gkyl simulation. These are bpls and bpdump. We give a brief example of each here, and expanded descriptions of their capabilities can be found in the ADIOS documentation, or using the bpls -h and bpdump -h commands. Note that these tools are complimentary to postgkyl’s info command. bpls bpls provides a simple view of the structure and contents of a .bp file. For example, in the previous section we discussed a 5-moment calculation of the plasma beach problem. Such simulation produced the file momBeach_field_1.bp. We can explore this file with

bpls momBeach_field_1.bp

which outputs

double time scalar integer frame scalar double CartGridField {400,8}

It tells us that this file contains three variables, the simulation time at which this snapshot was produced, the frame number, and a Cartesian grid field (CartGridField) for 400 cells which contains 8 electromagnetic components (3 for electric field, 3 for magnetic field, and the other 2 are used in gkyl’s algorithms). One may dump one of these variables with the additional -d flag. So if we wish to know the simulation time of this frame, we would use

bpls momBeach_field_1.bp time-d

and see it output

double time scalar 5.1e-11

Note that for large variables (e.g. CartGridField) dumping can overwhelm the terminal/screen. One can also slice the dataset and only dump part of it, see bpls -h. There are also a number of attributes (smaller pieces of time-constant data), which one can see with the -a ﬂag:

ws:dir jill$ bpls momBeach_field_1.bp -a double time scalar integer frame scalar double CartGridField {400, 8} string changeset attr string builddate attr string type attr string grid attr integer numCells attr double lowerBounds attr double upperBounds attr string basisType attr integer polyOrder attr string inputfile attr

and you can peek the value of an attribute with bpls -a -d.

3.1. Using gkyl 53 gkyl Documentation, Release 2.0-alpha bpdump

The -d ﬂag in the previous dumps the values of a variable onto the screen. There’s a separate command to do just that called bpdump. You can dump a speciﬁc variable with bpdump-d

3.2 gkyl Apps

Gkeyll Apps are top-level objects that solve a class of problems. Apps make it easy to setup a problem as the steps in the algorithm are pre-packaged. Although the apps are usually flexible enough to support most use cases, there may be problems for which a more fine-grained control over the simulation is required. In this case, the user has the option of directly instantiating various Gkeyll objects (grids, fields etc) and calling updaters in a custom hand-coded time-stepping loop1. This process can be complicated and is described elsewhere. The following apps are available in Gkeyll.

3.2.1 Input ﬁle basics

Contents

• Input file basics – Input file structure – The input file Common

Input ﬁle structure

Although this has been covered in the quickstart, we remind the reader that most input files have a similar structure regardless of which model (App) is being used. On the whole, input files consist of four parts: the App dependencies, the Preamble, the App initialization, and the App run. So most input files look like

------App dependencies. ...

------Preamble.

...

------App initialization.

... (continues on next page)

1 The concept of apps was introduced in Gkeyll 2.0. In the 1.0 version of the code the input ﬁles contained the complete simulation cycle. Although Gkeyll can still be used in that fashion, it is best to switch to the app model as it simpliﬁes the user input considerably.

54 Chapter 3. gkyl Reference gkyl Documentation, Release 2.0-alpha

(continued from previous page)

------App run.

...

We expand on the description of each part in the following sections: local Plasma= require("App.PlasmaOnCartGrid").VlasovMaxwell()-- Load the Vlasov App. local Plasma= require("App.PlasmaOnCartGrid").Gyrokinetic()-- Load the

˓→Gyrokinetic App. local Plasma= require("App.PlasmaOnCartGrid").Moments()-- Load the Moments

˓→App. local Constants= require"Lib.Constants"-- Load universal physical Constants. permitt= 1.0-- Permittivity of free space. permeab= 1.0-- Permeability of free space. eV= 1.0-- Elementary charge, or Joule-eV conversion factor. elcMass= 1.0-- Electron mass. ionMass= 1.0-- Ion mass. nElc= 1.0-- Electron number density. nIon= nElc-- Ion number density. Te= 1.0-- Electron temperature. Ti= Te-- Ion temperature. vtElc= math.sqrt(eV *Te/elcMass)-- Electron thermal speed. vtIon= math.sqrt(eV *Ti/ionMass)-- Ion thermal speed. wpe= math.sqrt((eV^2) *nElc/(permitt*elcMass))-- Plasma frequency. lambdaD= vtElc/wpe-- Debye length.

-- Amplitude and wavenumber of sinusoidal perturbation. pertA= 1.0e-3 pertK= .750/lambdaD

------Species. ...

------Fields. (continues on next page)

3.2. gkyl Apps 55 gkyl Documentation, Release 2.0-alpha

(continued from previous page) ...

------ExternalFields. ...

------Extras. ... } plasmaApp:run()

The input ﬁle Common

As mentioned in the previous section, the App initialization has a section called the Common, which contains parameters common to all the Apps. Here we describe what these possible entries are and their default value (if a default value is not given it means that the user must provide this parameter).

56 Chapter 3. gkyl Reference gkyl Documentation, Release 2.0-alpha

Table 1: Parameters in the App’s Common Parameter Description Default tEnd Final simulation time. lower Table with configuration space coordinates of lower boundaries. upper Table with configuration space coordinates of upper boundaries. cells Table with number of cells along configuration space each direction. nFrame Number of frames of data to write. Initial conditions are always written. For more fine-grained control over species and field output, see below. periodicDirs Periodic directions. Note: X is 1, Y is 2 and Z is 3. {} basis Basis functions to use. One of "serendipity", "tensor" or "serendipity" "maximal-order". polyOrder Polynomial order of the basis. 0 basis Basis functions to use. One of "serendipity", "tensor" or "serendipity" "maximal-order". decompCuts For parallel simulations: Table with number of processors to use in {} each configuration space direction. useShared For parallel simulations: Set to true to use MPI shared memory. false maximumDt Largest time step size allowed. tEnd-tStart suggestedDt Initial suggested time-step. Adjusted as simulation progresses. maximumDt cflFrac Fraction (usually 1.0) to multiply CFL determined time-step. 1.0, or 2.0 for timeStepper = "rk3s4". cfl CFL number to use in determining the time step. This parameter cflFrac/ should be avoided and cflFrac used instead. (2*polyOrder+1) timeStepper One of "rk1" (first order Runge-Kutta), "rk2" (SSP-RK2), "rk3" "rk3" (SSP-RK3) or “rk3s4” (SSP-RK3 with 4 stages) or "fvDimSplit". restartFrameEvery Frequency with which to write restart files, given as a decimal frac- max(0.05, tion. Default is every 5% (=0.05) of the simulation, or as frequently 1./nFrame) as frames are outputted (whichever is largest). ioMethod Method to use for file output. One of "MPI" or "POSIX". When "MPI" "POSIX" is selected, each node writes to its own file in a sub- directory. logToFile If set to true, log messages are written to log file. true

Note: • In general, you should not specify cfl or cflFrac, unless either doing tests or explicitly controlling the time-step. The app will determine the time-step automatically. • The "rk3s4" time-stepper allows taking twice the time-step as "rk2" and "rk3" at the cost of an additional RK stage. Hence, with this stepper a speed-up of 1.5X can be expected. •( This feature may be superseeded soon) One can request additional parallelism in velocity space for kinetic simulations by setting useShared = true. This enables MPI shared memory. In this case the decompCuts must specify the number of nodes and not number of processors. That is, the total number of processors will be determined from decompCuts and the number of threads per node.

3.2. gkyl Apps 57 gkyl Documentation, Release 2.0-alpha

3.2.2 VlasovMaxwell App: Vlasov-Maxwell equations on a Cartesian grid

The VlasovMaxwell app solves the Vlasov equation on a Cartesian grid.

휕푓푠 ∑︁ + ∇ · (v푓 ) + ∇ · (a 푓 ) = 퐶[푓 , 푓 ′ ] 휕푡 x 푠 v 푠 푠 푠 푠 푠′ where 푓푠 the particle distribution function, a푠 is particle acceleration and 퐶[푓푠, 푓푠′ ] are collisions. This app uses a version of the discontinuous Galerkin (DG) scheme to discretize the phase-space advection, and Strong-Stability preserving Runge-Kutta time-steppers (SSP-RK) to discretize the time derivative. We use a matrix and quadrature free version of the algorithm described in [Juno2018] which should be consulted for details of the numerics the properties of the discrete system. For neutral particles, the acceleration can be set to zero. For electromagnetic (or electrostatic) problems, the acceleration is due to the Lorentz force:

푞푠 a푠 = (E + v × B) 푚푠 The electromagnetic ﬁelds can be either speciﬁed, or determined from Maxwell or Poisson equations.

Contents

• VlasovMaxwell App: Vlasov-Maxwell equations on a Cartesian grid – Overall structure of app – Basic parameters – Species parameters – Electromagnetic ﬁeld parameters – Functional ﬁeld parameters – App output – Examples – References

Overall structure of app

By CDIM we mean conﬁguration space dimension and by VDIM we mean velocity space dimension. NDIM = CDIM+VDIM is the phase-space dimension. Note that we must have VDIM>=CDIM. The overall structure of the app is as follows local Vlasov=(require"App.PlasmaOnCartGrid").VlasovMaxwell() vlasovApp= Vlasov.App { -- basic parameters

-- description of each species: names are arbitrary elc= Vlasov.Species { -- species parameters },

(continues on next page)

58 Chapter 3. gkyl Reference gkyl Documentation, Release 2.0-alpha

(continued from previous page) -- fields (optional, can be omitted for neutral particles) field= Vlasov.Field { -- field parameters }, } -- run application vlasovApp:run()

Note that if the app’s run method is not called, the simulation will not be run, but the simulation will be initialized and the initial conditions will be written out to ﬁle.

Basic parameters

The app takes the following basic parameters. Parameters that have default values can be omitted.

Table 2: Basic Parameters for VlasovMaxwell() Parameter Description Default logToFile If set to true, log messages are written to log file true tEnd End time of simulation suggestedDt Initial suggested time-step. Adjusted as simulation progresses. tEnd/nFrame nFrame Number of frames of data to write. Initial conditions are always written. For more fine-grained control over species and field output, see below. lower CDIM length table with lower-left configuration space coordinates upper CDIM length table with upper-right configuration space coordinates cells CDIM length table with number of configuration space cells basis Basis functions to use. One of “serendipity” or “maximal-order” polyOrder Basis function polynomial order cfl CFL number to use. This parameter should be avoided and cfl- Determined from Frac used instead. cflFrac cflFrac Fraction (usually 1.0) to multiply CFL determined time-step. Determined from timeStepper timeStepper One of “rk2” (SSP-RK2), “rk3” (SSP-RK3) or “rk3s4” (SSP-RK3 “rk3” with 4 stages). For the last, cflFrac is 2.0 ioMethod Method to use for file output. One of “MPI” or “POSIX”. When “MPI” “POSIX” is selected, each node writes to its own file in a sub- directory. Depending on your system “MPI_LUSTRE” may be available and, if so, should be preferred. decompCuts CDIM length table with number of processors to use in each con- {} figuration space direction. useShared Set to true to use shared memory. false periodicDirs Periodic directions. Note: X is 1, Y is 2 and Z is 3. {} field Type of field solver to use. See details below. This is optional and nil if not specified no force terms will be evolved, i.e. the particles will be assumed to be neutral. species-name Species objects. There can be more than one of these. See details below.

Note:

3.2. gkyl Apps 59 gkyl Documentation, Release 2.0-alpha

• In general, you should not specify cfl or cflFrac, unless either doing tests or explicitly controlling the time-step. The app will determine the time-step automatically. • When useShared=true the decompCuts must specify the number of nodes and not number of processors. That is, the total number of processors will be determined from decompCuts and the number of threads per node. • The “rk3s4” time-stepper allows taking twice the time-step as “rk2” and “rk3” at the cost of an additional RK stage. Hence, with this stepper a speed-up of 1.5X can be expected.

Note that the field object must be called “field”. You can also omit the field object completely. In this case, it will be assumed that you are evolving neutral particles and the acceleration will be set to zero (i.e. a푠 = 0 in the Vlasov equation). Only one field object (if not omitted) is required. At present, the app supports a EM field evolved with Maxwell equations, or a EM field specified as a time-dependent function.

Species parameters

The Vlasov app works with arbitrary number of species. Each species is described using the Vlasov.Species objects. By default every species in the app is evolved. However, species evolution can be turned off by setting the evolve ﬂag to false. Species can be given arbitrary names. As the species names are used to label the output data ﬁles, reasonable names should be used. elc= Vlasov.Species { -- species parameters },

Table 3: Parameters for Vlasov.Species Parameter Description Default nDistFuncFrame These many distribution function outputs will be written during sim- nFrame from top- ulation. If not specified, top-level nFrame parameter will be used level nDiagnosticFrame These many diagnostics outputs (moments etc) will be written dur- nFrame from top- ing simulation. If not specified, top-level nFrame parameter will level be used charge Species charge (ignored for neutral particles) mass Species mass (ignored for neutral particles) lower VDIM length table with lower-left velocity space coordinates upper VDIM length table with upper-right velocity space coordinates cells VDIM length table with number of velocity space cells decompCuts VDIM length table with number of processors to use in each veloc- {} ity space direction. init Function with signature function(t,xn) that initializes the species distribution function. This function must return a single value, 푓(푥, 푣, 푡 = 0) at xn, which is a NDIM vector. bcx Length two table with BCs in X direction. See details on BCs below. {} bcy Length two table with BCs in Y direction. Only needed if CDIM>1 {} bcz Length two table with BCs in Z direction. Only needed if CDIM>2 {} evolve If set to false the species distribution function is not evolved. In true this case, only initial conditions for this species will be written to file. diagnosticMoments List of moments to compute for diagnostics. See below for list of {} moments supported.

60 Chapter 3. gkyl Reference gkyl Documentation, Release 2.0-alpha

The supported diagnostic moments are, “M0”, “M1i”, “M2ij”, “M2” and “M3i” deﬁned by ∫︁ 푀0 = 푓 푑푣 ∫︁ 푀1푖 = 푣푖푓 푑푣 ∫︁ 푀2푖푗 = 푣푖푣푗푓 푑푣 ∫︁ 푀2 = 푣2푓 푑푣 ∫︁ 2 푀3푖 = 푣 푣푖푓 푑푣

In these diagnostics, the index 푖, 푗 run over 1 . . . 푉 퐷퐼푀. Additionally, Gkeyll can calculate the weak moments: bulk velocity “u” and the square of thermal velocity “vtSq”. The boundary conditions (if not periodic) are speciﬁed with the bcx etc. tables. Each table must have exactly two entries, one for BC on the lower edge and one for the upper edge. The supported values are

Table 4: Boundary conditions for Vlasov.Species Parameter Description Vlasov.Species.bcAbsorb All outgoing particles leave the domain, and none reenter. Vlasov.Species.bcCopy A zero-gradient BC, approximating an open domain Vlasov.Species.bcReflect Particles are specularly reflected (i.e. billiard ball reflection)

Note that often “reﬂection” boundary condition is used to specify a symmetry for particles. For example, for a 1x simulation, to specify that the left boundary is a reﬂector, while the right an absorber use:

bcx= { Vlasov.Species.bcReflect, Vlasov.Species.bcAbsorb }

Electromagnetic ﬁeld parameters

The EM ﬁeld object is used as follows

field= Vlasov.Field { -- field parameters },

3.2. gkyl Apps 61 gkyl Documentation, Release 2.0-alpha

Table 5: Parameters for EM field objects Parameter Description Default nFrame These many field outputs will be written during simulation. If not nFrame from top- specified, top-level nFrame parameter will be used level epsilon0 Vacuum permittivity (휖0) mu0 Vacuum permeability (휇0) mgnErrorSpeedFactor Factor specifying speed for magnetic field divergence error correc- 0.0 tion elcErrorSpeedFactor Factor specifying speed for electric field divergence error correction 0.0 hasMagneticField Flag to indicate if there is a magnetic field true init Function with signature function(t,xn) that initializes the field. This function must return 6 values arranged as 퐸푥, 퐸푦, 퐸푧, 퐵푥, 퐵푦, 퐵푧 at 푡 = 0 at xn, which is a CDIM vector. bcx Length two table with BCs in X direction. See details on BCs below. {} bcy Length two table with BCs in Y direction. Only needed if CDIM>1 {} bcz Length two table with BCs in Z direction. Only needed if CDIM>2 {} evolve If set to false the field is not evolved. In this case, only initial true conditions will be written to file.

Note: When doing an electrostatic problem with no magnetic field, set the hasMagneticField to false. This will choose specialized solvers that are much faster and can lead to significant gain in efficiency. The boundary conditions (if not periodic) are specified with the bcx etc. tables. Each table must have exactly two entries, one for BC on the lower edge and one for the upper edge. The supported values are

Table 6: Boundary conditions for Vlasov.Field Parameter Description Vlasov.Field.bcCopy A zero-gradient BC, approximating an open domain Vlasov.Field.bcReﬂect Perfect electrical conductor wall

Functional ﬁeld parameters

To peform “test-particle” simulation one can specify a time-dependent electromagnetic ﬁeld which does not react to particle currents. externalField= Vlasov.ExternalField { -- field parameters },

Table 7: Parameters for functional field objects Parameter Description Default nFrame These many field outputs will be written during simulation. If not nFrame from top- specified, top-level nFrame parameter will be used level emFunc Function with signature function(t, xn) that specifies time- dependent EM field. It should return six values, in order, 퐸푥, 퐸푦, 퐸푧, 퐵푥, 퐵푦, 퐵푧. evolve If set to false the field is not evolved. In this case, only initial true conditions will be written to file.

62 Chapter 3. gkyl Reference gkyl Documentation, Release 2.0-alpha

App output

The app will write distribution function for each species and the EM fields at specified time intervals. Depending on input parameters specified to the species and field block, different number of distribution functions, fields and diagnostics (moments, integrated quantities) will be written. The output format is ADIOS BP files. Say your input file is called “vlasov.lua” and your species are called “elc” and “ion”. Then, the app will write out the following files: • vlasov_elc_N.bp • vlasov_ion_N.bp • vlasov_field_N.bp Where N is the frame number (frame 0 is the initial conditions). Note that if a species or the field is not evolved, then only initial conditions will be written. In addition to the above, optionally diagnostic data may also be written. For example, the moments files are named: • vlasov_elc_M0_N.bp • vlasov_ion_M0_N.bp • vlasov_elc_M1i_N.bp • vlasov_ion_M1i_N.bp etc, depending on the entries in the diagnosticMoments table for each species. In addition, integrated moments for each species are written: • vlasov_elc_intMom_N.bp This file has the time-dependent “M0”, three contributions of kinetic energy and the “M2” (integrated over configuration space) stored in them. 2 2 2 2 2 2 For the field, the electromagnetic energy components 퐸푥, 퐸푦 , 퐸푧 , 퐵푥, 퐵푦 , and 퐵푧 (integrated over configuration space) are stored in the file: • vlasov_fieldEnergy.bp These can be plotted using postgkyl in the usual way.

Examples

• Advection in a potential well (Field not evolved) • Landau damping of Langmuir waves • Two-stream instability • Three species electrostatic shock problem. See [Pusztai2018] for full problem description. • Advection of particles in a constant magnetic ﬁeld. (Field not evolved) • Weibel instability in 1x2v. See [Cagas2017] for full problem description.

References

3.2.3 Gyrokinetic App: Electromagnetic gyrokinetic model for magnetized plasmas

The Gyrokinetic App solves the (electromagnetic) gyrokinetic system on a Cartesian grid.

3.2. gkyl Apps 63 gkyl Documentation, Release 2.0-alpha

Contents

• Gyrokinetic App: Electromagnetic gyrokinetic model for magnetized plasmas – Overall structure of app – Species parameters

* Diagnostics · A note on boundary ﬂux diagnostics – References

Overall structure of app

To set up a gyrokinetic simulation, we ﬁrst need to load the Gyrokinetic App package. This should be done at the top of the input ﬁle, via local Plasma=(require"App.PlasmaOnCartGrid").Gyrokinetic()

This creates a table Plasma that loads the gyrokinetic species, ﬁelds, etc. packages. The general structure of the input ﬁle is then

------App dependencies. local Plasma=(require"App.PlasmaOnCartGrid").Gyrokinetic() ...

------Preamble. ...

------App initialization. plasmaApp= Plasma.App { ------Common ...

------Species electron= Plasma.Species { -- GkSpecies parameters ... },

-- other species, e.g. ions

------Fields field= Plasma.Field { -- GkField parameters ... },

(continues on next page)

64 Chapter 3. gkyl Reference gkyl Documentation, Release 2.0-alpha

(continued from previous page) ------ExternalFields extField= Plasma.Geometry { -- GkGeometry parameters ... }, } ------App run. plasmaApp:run()

Species parameters

The Gyrokinetic App works with an arbitrary number of species. Each species should be declared as

------Species species_name= Plasma.Species { -- GkSpecies parameters ... },

The species name (species_name here) is arbitrary, but will be used for naming in diagnostic ﬁles, so names like ion or electron are common. Here we describe all possible parameters used to specify a gyrokinetic species. Parameters that have default values can be omitted. Units are arbitrary, but often SI units are used. In the following, VDIM refers to the velocity space dimension, and CDIM refers to the conﬁguration space dimension. The gyrokinetic app works for 1X1V (CDIM=1, VDIM=1), 1X2V, 2X2V, and 3X2V. The velocity coordinates are (푣‖, 휇). See [Shi2017] for details.

3.2. gkyl Apps 65 gkyl Documentation, Release 2.0-alpha

Table 8: GkSpecies Parameters Parameter Description Default charge Species charge 1.0 mass Species mass 1.0 lower VDIM-length table with lower-left velocity space coordinates upper VDIM-length table with upper-right velocity space coordinates cells VDIM-length table with number of velocity space cells decompCuts NOT CURRENTLY SUPPORTED, no processor decomposition in velocity space allowed init Specifies how to initialize the species distribution function. Use a Projection plugin (see Projections), or a function with signature function(t,xn) that return a single value, 푓(푡 = 0, 푥푛[0], 푥푛[1], ...), where xn is a NDIM vector. evolve If set to false the species distribution function is not evolved. In true this case, only initial conditions for this species will be written to file. bcx Length two table with BCs in X direction. See details on BCs below. {} bcy Length two table with BCs in Y direction. Only needed if CDIM>1 {} bcz Length two table with BCs in Z direction. Only needed if CDIM>2 {} coll Collisions plugin. See Collisions models in Gkeyll. source Specifies a source that is added to the RHS on every timestep. Use a Projection plugin (see Projections), or a function with signature function(t,xn) that return a single value, 푆(푡, 푥푛[0], 푥푛[1], ...), where xn is a NDIM vector. diagnosticMoments List of moments to compute for diagnostics. See below for list of {} moments supported. diagnosticIntegratedMomentsList of integrated moments to compute for diagnostics. See below {} for list of integrated moments supported. diagnosticBoundaryFluxMomentsList of boundary flux moments to compute for diagnostics. See {} below for list of moments supported. diagnosticIntegratedBoundaryFluxMomentsList of integrated boundary flux moments to compute for diagnos- {} tics. See below for list of integrated moments supported.

Note: • In general, you should not specify cfl or cflFrac, unless either doing tests or explicitly controlling the time-step. The app will determine the time-step automatically. • When useShared=true the decompCuts must specify the number of nodes and not number of processors. That is, the total number of processors will be determined from decompCuts and the number of threads per node. • The “rk3s4” time-stepper allows taking twice the time-step as “rk2” and “rk3” at the cost of an additional RK stage. Hence, with this stepper a speed-up of 1.5X can be expected.

Diagnostics

There are species-speciﬁc diagnostics available, which mainly consist of moments of the distribution function and integrals (over conﬁguration-space) of these moments. There are also additional species diagnostics which serve as metrics of positivity and collisions-related errors.

66 Chapter 3. gkyl Reference gkyl Documentation, Release 2.0-alpha

Currently there are four types of diagnostic moments, defined below. Note that in these definitions dw = d푣‖ or dw = (2휋퐵0/푚)d푣‖d휇 depending on whether it is a 1V or a 2V simulation. We also use the notation 푑푣 to signify 2 2 the number of physical velocity-space dimensions included, i.e. 푑푣 = 1 for 1V and 푑푣 = 3 for 2V. Also, 푣 = 푣‖ for 2 2 1V and 푣 = 푣‖ + 2휇퐵0/푚 for 2V. • diagnosticMoments Velocity moments of the distribution function, written as functions of configuration- space position on each diagnostic frame. The options are ∫︀ – GkM0: number density, 푛 = 푀0 = dw 푓. ∫︀ – GkM1: parallel momentum density, 푀1 = dw 푣‖푓. ∫︀ 2 – GkM2: energy density, 푀2 = dw 푣 푓.

– GkUpar: parallel ﬂow velocity, 푢‖ = 푀1/푛.

– GkTemp: temperature, 푇 = (푚/푑푣)(푀2 − 푀1푢‖)/푛 2 – GkBeta: plasma beta, 훽 = 2휇0푛푇/퐵 • diagnosticIntegratedMoments Velocity moments integrated over conﬁguration-space, written as time- series. The options are – intM0: particle number, 푁 = ∫︀ dxdw 푓 ∫︀ – intM1: parallel momentum, 푈 = dxdw 푣‖푓 – intM2: ∫︀ dxdw 푣2푓 ∫︀ 2 – intKE: kinetic energy, ℰ퐾 = (푚/2) dxdw 푣 푓 ∫︀ 2 – intHE: total (kinetic + potential) energy, ℰ퐻 = dxdw 퐻푓, where 퐻 = 푚푣 /2+푞휑 is the Hamiltonian.

• diagnosticBoundaryFluxMoments Moments of the (phase-space) fluxes Γz through the (non-periodic) boundaries of configuration-space. The options are ∫︀ – GkM0: particle flux through boundary, dw Γz. ∫︀ – GkUpar: parallel momentum flux through boundary, dw 푣‖Γz. ∫︀ – GkEnergy: total (kinetic + potential) energy flux through boundary, dw 퐻Γz. • diagnosticIntegratedBoundaryFluxMoments Boundary flux moments integrated over configuration space. – intM0: integrated particle flux through the boundary. – intM1: integrated momentum flux through the boundary. – intKE: integrated kinetic energy flux through the boundary. – intHE: integrated total (kinetic + potential) energy flux through the boundary.

A note on boundary ﬂux diagnostics

The boundary ﬂuxes are computed via integrals of the time rates of change computed in the ghost cells. If we consider a simple phase-space advection equation in 2X2V without any forces 휕푓 + v · ∇푓 = 0 휕푡 the weak form used by the algorithm is obtained by multiplying this equation by a test function 휓 and integrating over phase space in a single cell. After an integration by parts one obtains

∫︁ 휕푓 ∫︁ ⃒푥푖+1/2 ∫︁ ⃒푦푗+1/2 ∫︁ ⃒ ⃒ dz 휓 + dv d푦 푣[푥푓휓⃒ + dv d푥 푣[푦푓휓⃒ − dz v · (∇휓)푓 = 0 휕푡 푥푖−1/2 푦푗−1/2

3.2. gkyl Apps 67 gkyl Documentation, Release 2.0-alpha

where the hat means that a numerical ﬂux is constructed, and dz = dx dv. In ghost cells only the surface terms corresponding to ﬂuxes through the physical domain boundaries are computed. This means tha in the ghost cell at the upper boundary along 푥, for example

∫︁ 휕푓 ∫︁ ⃒푦푗+1/2 ⃒ dz 휓 = − dv d푥 푣[푦푓휓⃒ (3.1) 휕푡 푦푗−1/2

This is phase-space flux through the upper 푥 boundary during a stage of the PDE solver. For Runge-Kutta steppers one must form a linear combination of these fluxes from every stage in the same manner as the time rates of change are combined for forward time stepping. For the sake of simplicity here we just assume a single forward Euler step, and define phase-space flux during a single time step through the upper 푥 boundary as

1 ∫︁ ⃒푦푗+1/2 Γ = − dv d푥 푣[푓휓⃒ z,푥+ 푦 ⃒ 푉 푦푗−1/2

where the volume factor 푉 arises from the phase-space integral on the left side of equation (3.1). Note that these integrals are over a single cell, and that the quantity Γz,푥+ is phase-space field, Γz,푥+ = Γz,푥+ (x, v). With this boundary flux in mind, if one requests the particle density of the boundary flux through diagnosticBoundaryFluxMoments={GkM0} the diagnostic would be computed as

′ ∫︁ ∫︁ ∫︁ ⃒푦푗+1/2 1 ′ ′ dv Γz,푥 = − dv dv d푥 푣[푓휓⃒ + 푦 ⃒ ′ 푉 푦푗−1/2

This yields the rate of number density crossing the upper 푥 boundary (per cell-length in the 푥 direction of the ghost cell). In order to compute the number of particles per unit time crossing the upper 푥 boundary (diagnosticIntegratedBoundaryFluxMoments={intM0}) we simply integrate the above quantity over 푦 (and multiply it by the 푥-cell length of the ghost cell)

′ ∫︁ ∫︁ ∫︁ ⃒푦푗+1/2 1 ′ ′ (∆푥) dv d푦 Γz,푥 = −(∆푥) dv d푦 dv d푥 푣[푓휓⃒ + 푦 ⃒ ′ 푉 푦푗−1/2

The final detail is that the files created by these diagnostics contain the fluxes through the boundary accumulated since the last snapshot (frame), not since the beginning of the simulation.

References

Contents

• Moments App: Multiﬂuid-moment-Maxwell model – Summary of model equations – Overall structure of the Moments app – Examples – Basic parameters – Species parameters – Electromagnetic ﬁeld parameters – App output – References

68 Chapter 3. gkyl Reference gkyl Documentation, Release 2.0-alpha

3.2.4 Moments App: Multiﬂuid-moment-Maxwell model

Summary of model equations

The Moment app solves high-moment multifluid equations on a Cartesian grid, coupled to Maxwell’s equations through the Lorentz force. • Each fluid could be either five-moment, where the plasma pressure is assumed to a scalar (see [Hakim+2006]), or ten-moment, where the full anisotropic and nongyrotropic plasma pressure tensors (see [Hakim2008]) are evolved. • The hyperbolic system is solved in converative forms of the five-moment (Euler) equations, and/or the ten- moment equations, and the perfectly hyperbolic Maxwell’s equations. • The sources that couples the plasma species momenta and electromagnetic fields are described here and more comprehensively in [Wang+2020]. This App solves the hyperbolic and source parts parts of the coupled system separately and apply high accuracy schemes on both. • Ignoring sources, the homogeneous equations of the fluid moments and electromagneic fields are solved separately using a high-resolution wave-propagation finite-volume method described in [Hakim+2006]. The main time-step size constraint comes from the speed of light. • The sources are evolved using a locally implicit, time-centered solver to step over the constraining scales like the Debye length and plasma frequency, etc. See Handling two-fluid five-moment and ten-moment source terms or [Wang+2020] for more details. • Additonial sources can be added if needed, for example, for the ten-moment model, we may apply the closure to relax the pressure tensor towards a scalar pressure (see [Wang+2015]). • We then apply an Strang-type operator-splitting sequence to combine the hyperbolic and source parts to achive second-order accuracy in time:

exp (ℒ푆∆푡/2) exp (ℒ퐻 ∆푡) exp (ℒ푆∆푡/2) .

Here, we represent the homogeneous update schematically as the operator exp (ℒ퐻 ∆푡) and the source update as exp (ℒ푆∆푡).

Overall structure of the Moments app

------Preamble ------The Moments app wraps fluid and field objects, and tells the -- the program how to evolve and couple them local Moments= require("App.PlasmaOnCartGrid").Moments() local TenMoment= require"Eq.TenMoment" -- TenMoment or Euler

-- Create the app momentApp= Moments.App { ------COMMON ------basic parameters, e.g., time step, grid, domain decomposition

-- Description of each species: names are arbitrary electron= Moments.Species { (continues on next page)

3.2. gkyl Apps 69 gkyl Documentation, Release 2.0-alpha

(continued from previous page) -- species parameters, equations, and boundary conditions },

-- Repeat to add more species hydrogen= Moments.Species { ... }, oxygen= Moments.Species { ... },

-- EM fields (optional, can be omitted for neutral fluids) field= Moments.Field { -- EM field parameters, equations, and boundary conditions },

-- Basic source that couple the fluids and EM fields emSource= Moments.CollisionlessEmSource { -- names of the species to be coupled species={"electron","hydorgen","oxygen"}, -- other specifications },

-- Additional sources if needed elc10mSource= Moments.TenMomentRelaxSource { species={"elctron"}, -- other specifications }, }

-- run the app momentApp:run()

Examples

• Five-moment modeling of the GEM challenge magnetic reconnection problem. • Ten-moment modeling of the GEM challenge magnetic reconnection problem. This simulation uses a simpliﬁed closure appropriate for this problem.

70 Chapter 3. gkyl Reference gkyl Documentation, Release 2.0-alpha

Basic parameters

Table 9: Basic Parameters for PlasmaOnCartGrid.Moments Parameter Description Default logToFile If set to true, log messages are written to log file true tEnd End time of simulation suggestedDt Initial suggested time-step. Adjusted as simulation progresses. tEnd/nFrame nFrame Number of frames of data to write. Initial conditions are always written. For more fine-grained control over species and field output, see below. lower CDIM length table with lower-left configuration space coordinates upper CDIM length table with upper-right configuration space coordinates cells CDIM length table with number of configuration space cells cfl CFL number to use. This parameter should be avoided and ‘‘cfl- Determined from Frac‘‘ used instead. cflFrac cflFrac Fraction (usually 1.0) to multiply CFL determined time-step. Determined from timeStepper maximumDt Hard limit of time step size. tEnd-tStart timeStepper The multifluid-Maxwell model currently only supports "fvDimSplit" the dimensional- splitting finite-volume method, i.e., "fvDimSplit". decompCuts CDIM length table with number of processors to use in each con- {} figuration space direction. useShared Set to true to use shared memory. false periodicDirs Periodic directions. Note: X is 1, Y is 2 and Z is 3. E.g., {2} sets {} the Y direction to be periodic.

Note: • In general, you should not specify cfl or cflFrac, unless either doing tests or explicitly controlling the time-step. The app will determine the time-step automatically. • When useShared=true the decompCuts must specify the number of nodes and not number of processors. That is, the total number of processors will be determined from decompCuts and the number of threads per node.

3.2. gkyl Apps 71 gkyl Documentation, Release 2.0-alpha

Species parameters

Table 10: Parameters for Moments.Species Parameter Description Default charge Species charge (ignored for neutral particles) mass Species mass (ignored for neutral particles) equation The type of default moment equation for this species, e.g., Èuler {gasGamma=5/3}`, èquation = TenMoment {}`. If domain invariance is violated, i.e., negative density/pressure occurs, the step is retaken using the èquationInv` method that is supposed to guarantee positivity but is more diffusive. equationInv Backup equation that guarantees positivity in case it is violated when the default èquation` is used. Examples are: èquationInv = Euler { gasGamma = gasGamma, numericalFlux = 'lax' }`, èquationInv = TenMoment { numericalFlux = "lax" }`. init Function with signature function(t,xn) that initializes the species moments. This function return n values, where n is the number of moments for this species. bcx Length two table with BCs in X direction. See details on BCs below. {} bcy Length two table with BCs in Y direction. Only needed if CDIM>1 {} bcz Length two table with BCs in Z direction. Only needed if CDIM>2 {} evolve If set to false the moments are not evolved in the hyperbolic part, true but could be modified in the source updater. In this case, by default only initial conditions for this species will be written to file. To force writing to file as usual, set the forceWrite option to true. forceWrite If set to true the moments are written to file even if evolve is set false to false.

72 Chapter 3. gkyl Reference gkyl Documentation, Release 2.0-alpha

Electromagnetic ﬁeld parameters

Table 11: Parameters for Moments.Field derived from App. Field.MaxwellField Parameter Description Default nFrame These many field outputs will be written during simulation. If not nFrame from top- specified, top-level nFrame parameter will be used level epsilon0 Vacuum permittivity (휖0) mu0 Vacuum permeability (휇0) mgnErrorSpeedFactorFactor specifying speed for magnetic field divergence error correc- 0.0 tion elcErrorSpeedFactorFactor specifying speed for electric field divergence error correction 0.0 init Function with signature function(t,xn) that initializes the field. This function must return 6 values arranged as 퐸푥, 퐸푦, 퐸푧, 퐵푥, 퐵푦, 퐵푧 at 푡 = 0 at xn, which is a CDIM vector. bcx Length two table with BCs in X direction. See details on BCs below. {} bcy Length two table with BCs in Y direction. Only needed if CDIM>1 {} bcz Length two table with BCs in Z direction. Only needed if CDIM>2 {} evolve If set to false the field is not evolved. In this case, only initial true conditions will be written to file. forceWrite If set to true the moments are written to file even if evolve is set false to false.

App output

The app will write snapshots of moments for each species and the EM fields at specified time intervals. Diagnostics like integrated fluid moments and field energy are recorded for each time-step and written in one file for each species/field object. The output format is ADIOS BP files. Say your input file is called “5m.lua” and your species are called “elc” and “ion”. Then, over specified time invertals the app will write out the following files: • 5m_elc_N.bp • 5m_ion_N.bp • 5m_field_N.bp Where N is the frame number (frame 0 is the initial conditions). Note that if a species or the field is not evolved, then only initial conditions will be written unless the forceWrite option is set to true. In addition, integrated moments for each species are written: • vlasov_elc_intMom_N.bp 2 2 2 2 2 2 For the field, the electromagnetic energy components 퐸푥, 퐸푦 , 퐸푧 , 퐵푥, 퐵푦 , and 퐵푧 (integrated over configuration space) are stored in the file: • vlasov_fieldEnergy_N.bp These can be plotted using postgkyl in the usual way.

3.2. gkyl Apps 73 gkyl Documentation, Release 2.0-alpha

References

3.3 App plugins

There are additional features that may be plugged into various Apps.

3.3.1 Collision models in Gkeyll

In Gkeyll we currently have two different collision operators for use in kinetic models: the Bhatnagar–Gross–Krook (BGK) and the Dougherty operators. We referred to the latter as the LBO for the legacy of Lenard-Bernstein. Its implementation in Gkeyll is detailed in [Hakim2020][Francisquez2020].

Contents

• Collision models in Gkeyll – BGK collisions – Dougherty collisions – Collisions in Gkeyll input ﬁles

* Constant collisionality * Spatially varying collisionality · Option A · Option B · Option C – Comments on stability – Examples

* Example 1: 1x1v collisional relaxation * Example 2: 1x2v collisional Landau damping – References

BGK collisions

The BGK operator [Gross1956] for the effect of collisions on the distribution of species 푓푠 is (︂ )︂ 휕푓푠 ∑︁ = 휈 (푓 − 푓 ) 휕푡 푠푟 푀푠푟 푠 푐 푟 where the sum is over all the species. The distribution functon 푓푀푠푟 is the Maxwellian

[︃ 2 ]︃ 푛푠 (v − u푠푟) 푓푀푠푟 = exp − 2 푑푣 /2 2푣2 (2휋푣푡푠푟) 푡푠푟

2 with the primitive moments u푠푟 and 푣푡푠푟 properly deﬁned to preserve some properties (such as conservation), and 푑푣 2 2 is the number of velocity-space dimensions. For self-species collisions u푠푟 = u푠 and 푣푡푠푟 = 푣푡푠. For multi-species

74 Chapter 3. gkyl Reference gkyl Documentation, Release 2.0-alpha collisions we follow an approach similar to [Greene1973] and deﬁne the cross-species primitive moments as

훼퐸 푚푠 + 푚푟 u푠푟 = u푠 − (u푠 − u푟) 2 푚푠푛푠휈푠푟 [︂ ]︂ 2 2 1 훼퐸 (︀ 2 2 )︀ 2 훼퐸 2 2 푣푡푠푟 = 푣푡푠 − 푑푣 푚푠푣푡푠 − 푚푟푣푡푟 − 푚푟 (u푠 − u푟) + 4 (푚푠 + 푚푟) (u푠 − u푟) 푑푣 푚푠푛푠휈푠푟 푚푠푛푠휈푠푟 but contrary to Greene’s definition of 훼퐸, we currently use in Gkeyll the following expression 1 + 훽 훼퐸 = 푚푠푛푠휈푠푟훿푠 . 푚푠 + 푚푟 Little guidance is provided by Greene as to how to choose 훽, although it seems clear that −1 < 훽. In Gkeyll the default value is 훽 = 0, but the user can specify it in the input file (explained below). We have introduced the additional quantity 훿푠 (which Greene indirectly assumed to equal 1) defined as

2푚푠푛푠휈푠푟 훿푠 = 푚푠푛푠휈푠푟 + 푚푟푛푟휈푟푠 The BGK operator can be used with both the Vlasov-Maxwell solver and the gyrokinetic solver.

Dougherty collisions

The Doughery (LBO) model for collisions [Dougherty1964] in Gkeyll is given by (︂ )︂ [︂ ]︂ 휕푓푠 ∑︁ 휕 2 휕푓푠 = 휈푠푟 · (v − u푠푟) 푓푠 + 푣 . (3.2) 휕푡 휕v 푡푠푟 휕v 푐 푟 In this case we compute the cross-primitive moments by a process analogous to Greene’s with the BGK operator, yielding the following formulas for the cross ﬂow velocity and thermal speed:

훼퐸 푚푠 + 푚푟 u푠푟 = u푠 + (u푟 − u푠) 2 푚푠푛푠휈푠푟 훼 푚 + 푚 1 [︂ 푚 1 ]︂ 푣2 = 푣2 + 퐸 푠 푟 푣2 − 푠 푣2 + (u − u )2 푡푠푟 푡푠 푚푠 푡푟 푡푠 푠 푟 2 푚푠푛푠휈푠푟 1 + 푚푟 푑푣 푚푟 with 훼퐸 deﬁned in the BGK section above. The LBO used by the gyrokinetic solver is

(︂ )︂ {︂ [︂ ]︂ [︂ 2 ]︂}︂ 휕푓푠 ∑︁ 휕 휕푓푠 휕 푚푠푣 휕푓푠 = 휈 (︀푣 − 푢 )︀ 푓 + 푣2 + 2휇푓 + 2 푡푠푟 휇 휕푡 푠푟 휕푣 ‖ ‖푠푟 푠 푡푠푟 휕푣 휕휇 푠 퐵 휕휇 푐 푟 ‖ ‖

Collisions in Gkeyll input ﬁles

Users can specify collisions in input files by adding an additional Lua table within each species one wishes to add collisions to. The collision frequency can be constant, varying in space and time, or it can also have a user-defined profile.

Constant collisionality

An example of adding LBO collisions (for BGK collisions simply replace LBOcollisions with BGKCollisions) to a species named ‘elc’ is

3.3. App plugins 75 gkyl Documentation, Release 2.0-alpha

elc= Plasma.Species { charge= q_e, mass= m_e, -- Velocity space grid. ... -- Initial conditions. ... evolve= true, -- Collisions. coll= Plasma.LBOCollisions { collideWith={"elc"}, frequencies= { nu_ee }, }, },

If there were another species, say one named ‘ion’, this ‘elc’ species could be made to collide with ‘ion’ by adding ‘ion’ to the collideWidth table:

coll= Plasma.LBOCollisions { collideWith={"elc","ion"}, frequencies= { nu_ee, nu_ei }, },

The constant collision frequencies nu_ee and nu_ei need to be previously computed/speciﬁed in the input ﬁle. 2 The user can specify the value of 훽 in the above formulas for the cross-species primitive moments (u푠푟 and 푣푡푠푟) by specifying the variable betaGreene in the collisions table (if the user does not specify it, betaGreene=0.0 is assumed) like

coll= Plasma.LBOCollisions { collideWith={"elc","ion"}, frequencies= { nu_ee, nu_ei }, betaGreene= 0.9 },

In some cases the user may be interested in colliding species ‘elc’ with species ‘ion’, but not collide species ‘ion’ with species ‘elc’. Gkeyll supports this combination, but since the formulas for cross-species primitive moments involve both 휈푒푖 and 휈푖푒, the code will default to assuming 휈푖푒 = 푚푒휈푒푖/푚푖. Note however that this scenario is not energy conserving: for exact energy conservation, one must include the effect of binary collisions on both species. It is also possible to specify both LBO and BGK collisions between different binary pairs in a single input ﬁle. For example, if there are three species ‘elc’, ‘ion’ and ‘neut’, the ‘elc’ species could be made collide with both ‘ion’ and ‘neut’ as follows:

cColl= Plasma.LBOCollisions { collideWith={"elc","ion"}, frequencies= { nu_ee, nu_ei }, }, nColl= Plasma.BGKCollisions { collideWith={"neut"}, frequencies= { nu_en }, },

If no collisionality is speciﬁed in the input ﬁle, it is assumed that the user desires Gkeyll to build a spatially-varying collisionality from scratch using a Spitzer-like formula for 휈푠푟 (explained below).

76 Chapter 3. gkyl Reference gkyl Documentation, Release 2.0-alpha

Spatially varying collisionality

Currently there are three ways to run simulations with a spatially varying collisionality. All of these options lead to a spatially varying, cell-wise constant collisionality. We will be adding support for variation of the collisionality within a cell in the future.

Option A

The simplest way to run with spatially varying collisionality is to not specify the table frequencies. In this case the code computes 휈푠푟 according to 푛 (︂ 1 1 )︂ 푞2푞2 log Λ 1 휈 = 휈 푟 + 푠 푟 푠푟 푠푟 frac 3/2 2 3/2 푚푠 푚푠 푚푟 3(2휋) 휖 2 2 0 (푣푡푠 + 푣푡푟) where 휈frac is a scaling factor, the Coulomb logarithm is deﬁned as ⎧ −1 ⎫ (︃ 2 2 )︃ [︂ (︂ )︂]︂−2 1 ⎨ ∑︁ 휔푝훼 + 휔푐훼 |푞푠푞푟| ⎬ log Λ = ln 1 + max , ~ 푠푟 푇 푇 2 1/2 2 훼 + 3 푠 4휋휖0푚푠푟푢 2푒 푚푠푟푢 ⎩ 훼 푚훼 푚푠 ⎭ and the 훼-sum is over all the species. For Vlasov-Maxwell simulations we do not add the correction due to gy- 2 2 2 romotion (휔푐훼 = 0 here). The relative velocity here is computed as 푢 = 3푣푡푟 + 3푣푡푠, the reduced mass is 푚푠푟 = 푚푠푚푟/ (푚푠 + 푚푟), and 휔푝훼 is the plasma frequency computed with the density and mass of species 훼. Simpler formulas for the Coulomb logarithm can be easily generated by developers if necessary. The formulas above assume all the plasma quantities and universal constants are in SI units. The user can provide a different value for these variables by passing them to the collisions table in the input ﬁles, as shown here:

coll= Plasma.LBOCollisions { collideWith={"elc","ion"}, epsilon0= 1.0, -- Vacuum permitivity. elemCharge= 1.0, -- Elementary charge value. hBar= 1.0, -- Planck's constant h/2pi. },

Additionally the user can pass the scaling factor 휈frac by specifying nuFrac in the collisions table.

Option B

Another way to use a spatially varying collisionality is to pass a reference collisionality normalized to a combination of the density and thermal speed of the colliding species. This normalized collisionality, is deﬁned as (︀ 2 2 )︀3/2 휈푠푟푁 = 휈푠푟0 푣푡푠0 + 푣푡푟0 /푛푟0 and one provides through normNu in the collisions table as shown below: elc= Plasma.Species { ... coll= Plasma.LBOCollisions { collideWith={"ion"}, normNu= { nu_ei *((vte^2+vti^2)^(3/2))/n_i0 } }, },

where nu_ei, vte, vti, n_e0 are computed in the Preamble of the input ﬁle and it is up to the user to ensure that these all have consistent units. Then, in each time step, the collisions will be applied with the following collisionality

푛푟(푥, 푡) 휈푠푟(푥) = 휈frac휈푠푟푁 . 2 2 3/2 (푣푡푠(푥, 푡) + 푣푡푟(푥, 푡))

3.3. App plugins 77 gkyl Documentation, Release 2.0-alpha

Note that if one is using the normNu feature for self-species collisions, one must still use these formulas. In this case one would specify electron-electron collisions like elc= Plasma.Species { ... coll= Plasma.LBOCollisions { collideWith={"elc"}, normNu= { nu_ee *((2*(vte^2))^(3/2))/n_e0 } }, },

Option C

The user may also wish to specify their own collisionality proﬁle, so for this purpose one can pass functions into the frequencies table in the collisions table. For example, suppose that one would like to run a simulation with a collisionality that decays exponentially in x. In this case we could create a exponentially decaying function in the preamble and pass it as the collision frequency as follows: local Plasma= require("App.PlasmaOnCartGrid").VlasovMaxwell local Constants= require"Lib.Constants" eps0= Constants.EPSILON0 eV= Constants.ELEMENTARY_CHARGE me= Constants.ELECTRON_MASS n0= 7e19 -- Number density [1/m^3]. Te0= 100 *eV -- Electron temperature [J].

-- Reference electron collision frequency (at x=0). logLambdaElc= 24.0- 0.5 *math.log(n0/1e6)+ math.log(Te0/eV) nu_ee= logLambdaElc *(eV^4)*n0 /(12*math.sqrt(2)*(math.pi^(3/2))*(eps0^2)*math.sqrt(me)*(Te0^(3/2))) local function nu_eeProfile(t, xn) local x= xn[1] return nu_ee*math.exp(-x) end vlasovApp= Plasma.App { ... elc= Plasma.Species { ... -- Collisions. coll= Plasma.LBOCollisions { collideWith={"elc"}, frequencies= { nu_eeProfile }, } } ... } -- Run application. vlasovApp:run()

At present all the frequencies must either be constant numbers or functions. We do not yet support having a combination of the two in the same collisions table.

78 Chapter 3. gkyl Reference gkyl Documentation, Release 2.0-alpha

Comments on stability

The are known issues with the implementation of the collision operators in Gkeyll. One of them, for example, is that we do not have a positivy preseving algorithm for the LBO. Positivity issues are often accompanied by large ﬂows or negative temperatures and/or densities. For this reason we have taken three precautions: 2 1. Calculation of primitive moments u푠푟 and 푣푡푠푟 is carried out using cell-average values if the number density is non-positive at one of the corners of that cell.

2. The collision term is turned off locally if the ﬂow velocity u푠푟 is greater than the velocity limits of the domain, 2 or if 푣푡푠푟 is negative. 2 3. The collision frequency 휈푠푟 is locally set to zero if the cell-average values of 푛푟 or 푣푡푠푟 are negative. We track the number of cells in which precaution 2 is used, and for stable simulations this is typically small (a few percent or less). Further discussion of why these precautions are necessary appears in [Hakim2020].

Examples

We offer two full examples of the use of collisions. One in Vlasov-Maxwell and one in Gyrokinetics.

Example 1: 1x1v collisional relaxation

Consider an initial distribution function in 1x1v phase space given by a Maxwellian and a large bump in its tail [︃ ]︃ [︃ ]︃ 푛 (푣 − 푢 )2 푛 (푣 − 푢 )2 1 푓(푥, 푣, 푡 = 0) = 0 exp − 0 + 푏 exp − 푏 (3.3) 2 1/2 2푣2 2 1/2 2푣2 (푣 − 푢 )2 + 푠2 (2휋푣푡0) 푡0 (2휋푣푡푏) 푡푏 푙 푏 Suppose we wish to collisionally relax this initial state, without the influence of collisionless terms. That is, we wish to evolve this distribution function according to equation (3.2). In this case our input file will use the VlasovMaxwell App (for 1x1v it would be equivalent to use the Gyrokinetic App), and we define the distribution in equation (3.3) in the Preamble via the function

-- Maxwellian with a Maxwellian bump in the tail. local function bumpMaxwell(x,vx,n,u,vth,bN,bU,bVth,bL,bS) local vSq= ((vx-u)/(math.sqrt(2.0) *vth))^2 local vbSq= ((vx-bU)/(math.sqrt(2.0) *bVth))^2 return (n/math.sqrt(2.0*math.pi*vth))*math.exp(-vSq) +(bN/math.sqrt(2.0*math.pi*bVth))*math.exp(-vbSq)/((vx-bL)^2+bS^2) end

In this case we chose constants for all densities, flow speed and temperatures. We also set the charge to 0. Under these conditions the collisionless terms have no effect, but we can explicitly turn them off with the evolveCollisionless flag. We will also request the total integrated bulk flow energy (intM2Flow) and the total thermal energy (intM2Thermal) as diagnostics. plasmaApp= Plasma.App { tEnd= 80, -- End time. nFrame= 80, -- Number of frames to write. lower={0.0}, -- Configuration space lower coordinate. upper={1.0}, -- Configuration space upper coordinate. cells={8}, -- Configuration space cells. polyOrder=2, -- Polynomial order. periodicDirs={1}, -- Periodic directions. -- Neutral species with a bump in the tail. (continues on next page)

3.3. App plugins 79 gkyl Documentation, Release 2.0-alpha

(continued from previous page) bump= Plasma.Species { charge= 0.0, mass= 1.0, -- Velocity space grid. lower={-8.0 *vt0}, upper={ 8.0 *vt0}, cells={32}, -- Initial conditions. init= function (t, xn) local x, v= xn[1], xn[2] return bumpMaxwell(x,v,n0,u0,vt0,nb,ub,vtb,uL,sb) end, evolve= true, -- Evolve species? evolveCollisionless= false, -- Evolve collisionless terms? diagnosticIntegratedMoments={"intM2Flow","intM2Thermal"}, -- Collisions. coll= Plasma.LBOCollisions { collideWith={'bump'}, frequencies= {nu}, }, }, }

We run this input ﬁle with the call gkyl lboRelax.lua

On a 2015 MacBookPro this ran in 1.5 seconds and produced a screen output like this one. We can start looking at the data by ﬁrst, for example, making a movie of the distribution function as function of time with pgkyl: pgkyl "lboRelax_bump_[0-9]*.bp" interp sel --z00. anim -x '$v$' -y '$f(x=0,v,t)$'

(note that postgkyl allows abbreviations, so interp = interpolate, sel = select, anim = animate) This command produces the movie given below. We can see that from the initial, bump-in-tail state the distribution relaxes to a Maxwellian. The Maxwellian by the way is the analytic steady state of this operator. Such relaxation should also take place without breaking momentum or energy conservation. We can examine the evolution of the total energy in the system by adding intM2Flow and intM2Thermal and plotting it as a function of time. This is achieved in pgkyl via: pgkyl lboRelax_bump_intM2Flow.bp lboRelax_bump_intM2Thermal.bp ev 'f[0] f[1] +' pl -x

˓→'time' -y 'energy'

As we can see in the ﬁgure below, and in particular in the 10−14 scale of it, the total particle energy is conserved very well. The changes in energy over a collisional period are of the order of machine precision.

Example 2: 1x2v collisional Landau damping

We now explore the modiﬁcation of Landau damping by inclusion of Dougherty collisions. Speciﬁcally, we will consider ion acoustic waves with adiabatic electrons. This means that the electron number density simply follows

(︂ 푒휑 )︂ 푛푒(푥, 푡) = 푛0 1 + (3.4) 푇푒0

80 Chapter 3. gkyl Reference gkyl Documentation, Release 2.0-alpha

Fig. 1: Normalized particle energy vs. time as an initial bump-in-tail distribution is relaxed to a Maxwellian by the Dougherty collision operator.

and our gyrokinetic Poisson equation is simply replaced by the quasineutrality

(︂ 푒휑 )︂ ∫︁ 푛0 1 + = 푛푖(푥, 푡) = 2휋퐵 d푣‖ d휇 푓푖(푥, 푣‖, 휇, 푡). 푇푒0 So there is no need to evolve the electron distribution function. In the Gyrokinetic App we can specify an adiabatic species using Plasma.AdiabaticSpecies:

plasmaApp = Plasma.App { ... adiabaticElectron = Plasma.AdiabaticSpecies { charge = -1.0, mass = mElc, temp = Te, -- Initial conditions.. use ion background so that background is exactly

˓→neutral. init = function (t, xn) return nElc end, evolve = false, -- Evolve species? }, ... }

This simulation then only needs to solve the electrostatic gyrokinetic equations for ions (︂ )︂ 휕퐵푓푖 (︁ ˙ )︁ 휕 (︀ )︀ 휕퐵푓푖 + ∇ · 퐵푓푖R + 퐵푓푖푣˙‖ = (3.5) 휕푡 휕푣‖ 휕푡 푐 and we do so with an initial condition that contains a sinusoidal perturbation (wavenumber 푘 = 0.5) in the ion density:

[︃ 2 ]︃ 푛 [1 + 훼 cos(푘푥)] 푣 + 2휇퐵/푚푖 푓 (푥, 푣 , 휇, 푡 = 0) = 푖0 exp − ‖ 푖 ‖ √︀ 2 2푣2 2휋푣푡푖0 푡푖0 If the right side of this equation (3.5) were zero, this ion acoustic wave would damp at the collisionless rate calculated by Landau (well he did electron Langmuir waves). But collisions will change the picture and we wish to numerically find out how. This simulation is setup in the ionSound.lua input file. This input file calls for discretizing the ion phase space 2 [−휋/푘, 휋/푘] × [−6푣푡, 6푣푡] × [0, 푚푖(5푣푡 )/(2퐵0)] using 64 × 128 × 16 cells and a piecewise linear basis. With a collisionality of 휈=0.005, the simulation ran on a 2015 MacbookPro in 41 minutes, while a collisionality of 휈 = 0.05 required 1.35 hours. They were run the command

3.3. App plugins 81 gkyl Documentation, Release 2.0-alpha

gkyl ionSound.lua

and produced this screen output. Note that this is really a linear problem, that is, one can sufficiently model it with a linearized version of equation (3.5), using 푓푖 = 푓푖0 + 푓푖1, where the fluctuation 푓푖1 is small compared to the equilibrium (Maxwellian) 푓푖0. Users may wish to output this fluctuation in time: in order to to this specify the background with the initBackground table:

ion= Plasma.Species { ... -- Specify background so that we can plot perturbed distribution and moments. initBackground={"maxwellian", density= function (t, xn) return nIon end, temperature= function (t, xn) return Ti end, }, ... },

This will output the fluctuation to a file with the name format __f1_#.bp, where # stands for the frame number. So for example, in this ionSound.lua case it creates files named ionSound_ion_f1_#.bp. We can plot this fluctuation along 푣‖ at t=5 with

pgkyl"ionSound_ion_f1_10.bp" interp sel--z0 0.0--z2 0.0 pl-x'$v_\parallel$'-y'

˓→$f_{i1}(x=0,v_\parallel,\mu=0,t=5)$'

(note that postgkyl allows abbreviations, so interp = interpolate, sel = select, pl = plot) which produces the following image

Fig. 2: Fluctuation in the ion distribution function 푓푖1 along 푣‖ at time 푡 = 5. The fluctuation is defined as the instantaneous 푓푖 minust the equilibrium 푓푖0 defined in the input file (a Maxwellian).

Perhaps most valuable to the physics of this simulation is to see a signature of the decay of the ion acoustic wave. This simulation produced the integrated squared electrostatic potential, ∫︀ d푥 |휑|2, which we take as a measure of the wave energy. It is stored in a ﬁle with the name format _phiSq.bp. If we had run two simulations, ionSound.lua with 휈 = 0.005 and ionSoundH.lua with 휈 = 0.05, we could plot both electrostatic energies in time with the following pgkyl command:

82 Chapter 3. gkyl Reference gkyl Documentation, Release 2.0-alpha

pgkyl ionSound_phi2.bp-l'$ \nu=0.005$' ionSoundH_phi2.bp-l'$ \nu=0.05$' pl--logy-

˓→f0-x'time'-y'Integrated $|\phi|^2$'

Notice that we are giving each file a label to use in the plot with the -l flag. Postgkyl then produces the following figure

Fig. 3: Electrostatic ﬁeld energy as a function of time for two collisionalities in 1x2v ion-sound wave damping simulation with gyrokinetics.

We thus see that the wave energy is decaying as a function of time (the envelope of the curve is going down), and that the rate at which this happens decreases with collisionality. That is, for this case increasing collisionality decreased the damping rate. From this curve we can also read the period of the wave, using the spacing between the dips.

References

3.3.2 Neutral models in Gkeyll

Simpliﬁed models of plasma-neutral interactions available in Gkeyll. The neutral species is always modeled as a kinetic Boltzmann species using the Vlasov-Maxwell solver, while the plasma species to which it is coupled can be either Vlasov-Maxwell or gyrokinetic.

Contents

• Neutral models in Gkeyll – Electron-impact Ionization – Charge exchange – Wall recycling boundary conditions – Neutral species and gyrokinetic plasma species coupling – Neutral interactions in Gkeyll input ﬁles

* Electron-impact ionization * Charge exchange * Wall recycling – Examples

* 1X1V Vlasov simulation

3.3. App plugins 83 gkyl Documentation, Release 2.0-alpha

* 1X2V gyrokinetic + 1X3V Vlasov simulation – References

Electron-impact Ionization

− + − This process is given by 푒 + 푛 → 푖 + 2푒 − 퐸푖푧, where 퐸푖푧 is the ionization energy and modeled by collision terms on the RHS of the dynamical equations, as in [Wersal2015]: 푑푓 푒 = −푛 ⟨푣 휎 ⟩(2푓 − 푓 ), (3.6) 푑푡 푛 푒 푖푧 푀,푖푧 푒 푑푓 푖 = 푛 ⟨푣 휎 ⟩푓 , (3.7) 푑푡 푒 푒 푖푧 푛 푑푓 푛 = −푛 ⟨푣 휎 ⟩푓 , (3.8) 푑푡 푒 푒 푖푧 푛 2 where 푓푀,푖푧 = 푓푀,푖푧(푛푒, u푛, 푣푡ℎ,푖푧) is a Maxwellian distribution function that accounts for the lower-energy elelec- trons resulting from this process. (See Neutral species and gyrokinetic plasma species coupling for definitions of Maxwellian distribution function on the Vlasov-Maxwell and gyrokinetic grids.) 푛푒 is the electron density, u푛 is the 2 neutral fluid velocity, and 푣푡ℎ,푖푧 is defined by

2 2 푣푡ℎ,푒 퐸푖푧 푣푡ℎ,푖푧 = − . 2 3푚푒

Currently the ionization rate ⟨푣푒휎푖푧⟩ is approximated using the ﬁtting function from [Voronov1997]

1 + 푃 (퐸 /푇 )1/2 (︂퐸 )︂퐾 푖푧 푒 푖푧 −퐸푖푧 /푇푒 −6 3 ⟨푣푒휎푖푧⟩ = 퐴 푒 × 10 m /s, 푋 + 퐸푖푧/푇푒 푇푒 where 퐴, 퐾, 푃 and 푋 are tabulated for elements up to 푍 = 28. To avoid unphysical negative temperatures, when 2 푣푡ℎ,푖푧 < 0 the ionization rate is set to zero in the code. A similar model of ionization was previously used in Gkeyll and was presented in [Cagas2017].

Charge exchange

This process is given by 푖+ + 푛 → 푛 + 푖+, and the simplifed model of this process contained within Gkeyll is based on [Meier2012]. Collision terms appear in the ion and neutral equations as: 푑푓 푖 = 휎 푉 (푛 푓 − 푛 푓 ), (3.9) 푑푡 푐푥 푐푥 푖 푛 푛 푖 푑푓 푛 = −휎 푉 (푛 푓 − 푛 푓 ), (3.10) 푑푡 푐푥 푐푥 푖 푛 푛 푖 where

√︂ 4 4 푉 ≡ 푣2 + 푣2 + 푣2 , 푐푥 휋 푡,푖 휋 푡,푛 푖푛

푣푖푛 ≡ |u푖 − u푛|.

The cross section is approximated by a ﬁtting function. For hydrogen and Deuterium these are given by, respectively

−18 −20 휎푐푥,퐻 = 1.12 × 10 − 7.15 × 10 ln(푉푐푥), −18 −20 휎푐푥,퐷 = 1.09 × 10 − 7.15 × 10 ln(푉푐푥).

84 Chapter 3. gkyl Reference gkyl Documentation, Release 2.0-alpha

Wall recycling boundary conditions

A model for wall recycling has been implemented at the boundaries where field lines terminate. These boundary conditions provide a source of neutrals from the targets that depends on the flux of outgoing ions. Consider a simulation with one configuration space dimension, parallel to the magnetic field, and three velocity space dimensions (1x3v). Since 푥 is parallel to the magnetic field, 푣푥 is the parallel velocity for neutrals. We define the neutral distribution function in the ghost cell at the lower ($x_min$) boundary as

푓푛(푣푥, 푣푦, 푣푧, 푥 = 푥푔ℎ표푠푡) = 퐶푟푒푐푓푀,푟푒푐(푇 = 푇푛,푟푒푐).

The Maxwellian function for recycled neutrals 푓푀,푟푒푐 is defined by a zero mean flow and a temperature that is set in the user input file to model the Franck-Condon atoms coming from the wall. The Maxwellian is scaled such that the magnitude of the flux of incoming neutrals is equal to the magnitude of the flux of outgoing ions. At this time, angular dependency is not included in the model of wall recycling.

Neutral species and gyrokinetic plasma species coupling

Neutral species are always evolved on the Vlasov grid. For a Vlasov-Maxwell plasma species, the neutrals and ions are evolved on identical phase-space grids. Thus, the ion-neutral interaction terms in Eqs. (3.7), (3.9), and (3.10) are straightforward. However, when the plasma species are evolved using the gyrokinetic model, the ion and neutral velocity-space grids are no longer identical, and it becomes necessary to pass information between two different phase- 2 space grids. This is accomplished by taking fluid moments, 푛, u, and 푣푡ℎ, of the species distribution function and using them to project a Maxwellian distribution function on the destination phase-space grid. This is valid assuming that ion and neutral distribution functions are approximately Maxwellian. In the Vlasov-Maxwell formulation, a Maxwellian distribution is defined [︃ ]︃ 푛 (v − u)2 푓푀,푣푚(x, v) = exp − , 2 푑푣 /2 2푣2 (2휋푣푡ℎ) 푡ℎ where 푑푣 is the velocity-space dimension. In the gyrokinetic formulation a Maxwellian distribution function is defined [︃ (︀ )︀2 ]︃ 푛 푣‖ − 푢‖ 퐵휇 푓푀,푔푘(x, 푣‖, 휇) = exp − − , 2 3/2 2푣2 푚푣2 (2휋푣푡ℎ) 푡ℎ 푡ℎ where we have assumed the gyrokinetic grid is either 1X2V or 3X2V. Note that in the gyrokinetic formulation, the fluid velocity moment contains only one component, 푢‖, which is along the magnetic field line. However, the neutral fluid velocity contains 3 components. It is assumed that once a neutral particle is ionized, the perpendicular components are immediately “smeared out” by the gyro-motion. Thus, only the 푧-component of the neutral fluid velocity moment is included in the Maxwellian projection on the gyrokinetic grid. Conversely, the ion fluid velocity moment contains only one component. Thus, the ion Maxwellian distribution function on the 3V Vlasov grid contains the fluid moment u푖 = (푢푥 = 0, 푢푦 = 0, 푢푧 = 푢‖,푖). The collision terms in this gyrokinetic-Vlasov coupling become 푑 풥 푓 (R, 푣 , 휇, 푡) = 푛 ⟨휎 푣 ⟩풥 푓 (푛 , 푢 , 푣2 ) + 휎 푉 [푛 풥 푓 (푛 , 푢 , 푣2 ) − 푛 풥 푓 ], 푑푡 푖 ‖ 푒 푖푧 푒 푀,푔푘 푛 푧,푛 푡ℎ,푛 푐푥 푐푥 푖 푀,푔푘 푛 푧,푛 푡ℎ,푛 푛 푖 푑 푓 (x, v, 푡) = 푛 푓 ⟨휎 푣 ⟩ − 휎 푉 [푛 푓 − 푛 푓 (푛 , 푢 , 푣2 )], 푑푡 푛 푒 푛 푖푧 푒 푐푥 푐푥 푖 푛 푛 푀,푣푚 푖 ‖,푖 푡ℎ,푖 where 풥 is the Jacobian for the gyrokinetic model.

Neutral interactions in Gkeyll input ﬁles

3.3. App plugins 85 gkyl Documentation, Release 2.0-alpha

Electron-impact ionization

Below is an example of adding ionization to a Vlasov-Maxwell simulation:

------App dependencies ------local Plasma=(require"App.PlasmaOnCartGrid").VlasovMaxwell()

... plasmaApp= Plasma.App { ------Common ------...

------Species ------Vlasov-Maxwell electrons elc= Plasma.Species { evolve= true, charge= qe, mass= me, ... -- Ionization ionization= Plasma.Ionization { collideWith={"neut"}, -- species to collide with electrons="elc", -- define name for electron species neutrals="neut", -- define name for neutral species elemCharge= eV, -- define elementary charge elcMass= me, -- electron mass plasma="H", -- ion species element }, ... },

-- Vlasov-Maxwell ions ion= Plasma.Species { evolve= true, charge= qi, mass= mi, ... -- Ionization ionization= Plasma.Ionization { collideWith={"neut"}, -- species to collide with electrons="elc", -- define name for electron species neutrals="neut", -- define name for neutral species elemCharge= eV, -- define elementary charge elcMass= me, -- electron mass plasma="H", -- ion species element }, ... },

-- Vlasov neutrals (continues on next page)

86 Chapter 3. gkyl Reference gkyl Documentation, Release 2.0-alpha

(continued from previous page) neut= Plasma.Species { evolve= true, charge=0, mass= mi, ... -- Ionization ionization= Plasma.Ionization { collideWith={"elc"}, -- species to collide with electrons="elc", -- define name for electron species neutrals="neut", -- define name for neutral species elemCharge= eV, -- define elementary charge elcMass= me, -- electron mass plasma="H", -- ion species element }, ... }, },

In order to add ionization to a gyrokinetic simulation and include neutral particles which are evolved using the Vlasov solver, deﬁne the Gyrokinetic App in the dependencies as local Plasma = (require "App. PlasmaOnCartGrid").Gyrokinetic(). Then replace the neutral Lua table above with neut= Plasma.Vlasov { evolve= true, charge=0, mass= mi, init= Plasma.VmMaxwellianProjection { ... } -- initial conditions (and source)

˓→defined using Vlasov app ... -- Ionization ionization= Plasma.Ionization { collideWith={"elc"}, -- species to collide with electrons="elc", -- define name for electron species neutrals="neut", -- define name for neutral species elemCharge= eV, -- define elementary charge elcMass= me, -- electron mass plasma="H", -- ion species element }, ... bcx= {Vlasov.Species.bcReflect, Vlasov.Species.bcReflect} -- boundary conditions

˓→defined using Vlasov app },

Note that Plasma.Species became Plasma.Vlasov and Plasma.MaxwellianProjection became Plasma.VmMaxwellianProjection but the ionization Lua table remains ionization = Plasma. Ionization. The latter remains as is since the ionization calculation is carried out from within the Gyrokinetic App but other parameters such as initial conditions, source, and boundary conditions are deﬁned using the Vlasov App, which gets called from within the Gyrokinetic App.

Charge exchange

Charge exchange can be added much in the same way as ionization was included above, though the former only affects the ion and neutral species. For the case of gyrokinetic plasma species with Vlasov neutrals, include the following in the Species section of the input ﬁle.

3.3. App plugins 87 gkyl Documentation, Release 2.0-alpha

-- Gyrokinetic ions ion= Plasma.Species { evolve= true, charge= qi, mass= mi, ... -- Charge exchange chargeExchange= Plasma.ChargeExchange { collideWith={"neut"}, -- species to collide with ions="ion", -- define ion species name neutrals="neut", -- define neutral species name ionMass= mi, -- ion mass neutMass= mi, -- neutral mass plasma="H", -- ion species element charge= qi, -- species charge }, ... },

-- Vlasov neutrals neut= Plasma.Vlasov { evolve= true, charge=0, mass= mi, ... -- Charge exchange chargeExchange= Plasma.ChargeExchange { collideWith={"ion"}, -- species to collide with ions="ion", -- define ion species name neutrals="neut", -- define neutral species name ionMass= mi, -- ion mass neutMass= mi, -- neutral mass plasma="H", -- ion species element charge=0, -- species charge }, ... },

Wall recycling

Wall recycling boundary conditions can be included for the neutral Vlasov species by including the following in the neutral table for a simulation in one configuration-space dimension. Since particle fluxes necessary for wall recycling are stored in boundary flux diagnostics, diagnosticBoundaryFluxMoments must be included in all species table as shown below.

...

-- Gyrokinetic electrons elc= Plasma.Species { evolve= true, ... diagnosticBoundaryFluxMoments={"GkM0","GkUpar","GkEnergy"}, }

-- Gyrokinetic ions (continues on next page)

88 Chapter 3. gkyl Reference gkyl Documentation, Release 2.0-alpha

(continued from previous page) ion= Plasma.Species { evolve= true, ... diagnosticBoundaryFluxMoments={"GkM0","GkUpar","GkEnergy"}, }

-- Vlasov neutrals neut= Plasma.Vlasov { evolve= true, charge=0, mass= mi, ... bcx= {Plasma.Vlasov.bcRecycle, Plasma.Vlasov.bcRecycle},

-- Recycle elements recycleTemp= 10 *eV, recycleFrac= 0.5, recycleIon="ion", recycleTime= 100e-6, ...

diagnosticBoundaryFluxMoments={"M0","u","M2Flow","M2Thermal"}, },

Additional flags are required including recycleTemp which defines 푇푛,푟푒푐, recycleFrac which defines the wall recycling fraction 훼푟푒푐, and recycleIon which defines the ion species name in the input file. An opptional flag recycleTime provides a time dependency for the wall recycling fraction, which gradually ramps up to the desired value 훼푟푒푐,0 according to the equation 1 (1 + tanh(푡/휏 − 1))훼 , 2 푟푒푐 푟푒푐 where 휏푟푒푐 is given by recycleTime.

Examples

Two examples of simulations with neutral interactions are presented here. The ﬁrst uses the Vlasov-Maxwell solver for the plasma species and includes electron-impact ionization. The second uses the gyrokinetic solver with both electron-impact ionization and charge exchange.

1X1V Vlasov simulation

A simple Vlasov-Maxwell test case in 1X1V with spatially constant ﬂuid moments for all species and periodic boundary conditions can be set up to test conservation properties of this model. Simply run the included input ﬁle vlaso- vIz.lua using standard procedures detailed here. The simulation completes in about 12 seconds on a 2019 MacBook Pro. Then use the Postgkyl command-line tool to check particle and energy conservation. To plot the sum of the integrated particle densities of ions and electrons, use the following command. pgkyl vlasovIz_ion_intM0.bp vlasovIz_neut_intM0.bp ev 'f[0] f[1] +' plot -x 'time' -y

˓→'particles'

This produces the plot shown below, illustrating conservation of particle number. Next plot the sum of integrated thermal energy of ions and neutrals with the following command.

3.3. App plugins 89 gkyl Documentation, Release 2.0-alpha

Fig. 4: Sum of ion and neutral integrated particle densities vs. time.

pgkyl vlasovIz_ion_intM2Thermal.bp vlasovIz_neut_intM2Thermal.bp ev 'f[0] f[1] +'

˓→plot -x 'time' -y 'thermal energy'

This produces the plot shown below which demonstrates the conservation of thermal energy.

Fig. 5: Sum of ion and neutral integrated thermal energy vs. time.

1X2V gyrokinetic + 1X3V Vlasov simulation

This example is based on a simplified model of a scrape-off layer plasma, the open-field line region in a fusion device. Parameters were chosen based on previous Gkeyll simulations described in [Shi2015]. Gyrokinetic ion and electron species are coupled to Vlasov neutrals via electron-impact ionization and charge exchange interactions. Sheath model boundary conditions are used for the plasma species and reflecting boundary conditions are used for neutrals. The gyrokinetic species are evolved using two velocity-space dimensions, (푣‖, 휇). The Vlasov species are run using three velocity-space dimensions, (푣푥, 푣푦, 푣푧), where the subscripts (푥, 푦, 푧) correspond to the non-orthogonal field-line following coordinate system used in the gyrokinetic solver. Thus, 푣‖ in the gyrokinetic system is identical to the 푣푧 Vlasov coordinate. The simulation can be run with the input file 1x2vSOL.lua, which is currently set to run in parallel on 4 processors (decompCuts = {4}). On a 2019 Macbook Pro, this simulation takes approximately 15 minutes to complete. The output can be analyzed with the Postgkyl tools. For example, the anim command can be used to observe changes in the electron density profile, as shown below.

pgkyl "1x2vSOL_elc_GkM0_[0-9]*.bp" interp anim -x '$x$' -y '$n_e$'

90 Chapter 3. gkyl Reference gkyl Documentation, Release 2.0-alpha

This command produces the following animation of the evolution of the electron density proﬁle in time.

References

3.3.3 Projection plugin

3.4 gkyl Tools

Contents

• gkyl Tools – Solving the exact Reimann problem for Euler equations: exacteulerrp – Linear dispersion relation solver: multimomlinear – References

Tools are pre-packaged programs that are often useful but not part of the main App system. For example, there are tools to run and query the the regression test system and its output, solve the exact Reimann problem for the Euler equations and compute linear dispersion solvers for multi-moment multiﬂuid equations, and coming soon, linear dispersion solvers for kinetic equations. To get a list of all the supported tools do:

gkyl-t

This will give the list of tools and provide a one-line description of what the tool does. Each tool comes with its own embedded help system. For example, to see how to run the exact Euler Reimann solver tool do:

gkyl exacteulerrp-h

This will print the help and the command options for the tool to the terminal. On this page the main tools are documented.

3.4.1 Solving the exact Reimann problem for Euler equations: exacteulerrp

The Reimann problem is a fundamental problem in the theory of hyperbolic PDEs (like Euler and ideal MHD equations) which brings out the essential nonlinear structure of the underlying physics. In particular, the solution to the Reimann problem shows the shocks, contact discontinuities and rarefactions waves supported by a system of hyperbolic equations. The exacteulerrp solves the Reimann problem exactly for the 1D Euler equations

⎡ 휌 ⎤ ⎡ 휌푢 ⎤ ⎢휌푢⎥ ⎢ 휌푢2 + 푝 ⎥ 휕 ⎢ ⎥ 휕 ⎢ ⎥ ⎢휌푣 ⎥ + ⎢ 휌푢푣 ⎥ = 0 휕푡 ⎢ ⎥ 휕푥 ⎢ ⎥ ⎣휌푤⎦ ⎣ 휌푢푤 ⎦ 퐸 (퐸 + 푝)푢 where 푝 1 퐸 = + 휌(푢2 + 푣2 + 푤2) 훾 − 1 2 is the total energy contained in the ﬂuid. To get help for this tool do:

3.4. gkyl Tools 91 gkyl Documentation, Release 2.0-alpha

gkyl exacteulerrp-h

Note that the exacteulerrp tool itself can print an example input ﬁle to shell that you can then redirect to a ﬁle

gkyl exacteulerrp-e> sod-shock.lua

To run this tool make an input ﬁle (or modify the one produced by the -e option) with the left and right initial states, the time to compute the solution at and a grid resolution. (The grid is used to determine where the solution is computed. The solution at each grid location is exact and does not depend on the resolution).

lower=-1.0 -- left-edge of domain upper= 1.0 -- right-edge of domain location= 0.0 -- location of shock ncell= 100 -- Number of cells to use tEnd= 0.2 -- Time at which to compute RP gasGamma= 1.4 -- Gas adiabatic index

-- left/right states { density, velocity, pressure } leftState={ 1.0, 0.0, 1.0} rightState={ 0.125, 0.0, 0.1}

If this ﬁle was saved as “sod-shock.lua” you can run the tool:

gkyl exacteulerrp-i sod-shock.lua

This will produce a set of BP ﬁles with the solution to the Riemann problem:

sod-shock_density.bp sod-shock_pressure.bp sod-shock_internalenergy.bp sod-shock_velocity.bp

You can now use postgkyl to plot the solution. For example, to plot all the ﬁles in a single ﬁgure do:

pgkyl"sod-shock_ *.bp" pl-b

to produce the following plot. For a comprehensive set of 1D Riemann problems used to benchmark two ﬁnite-volume schemes see this note

3.4.2 Linear dispersion relation solver: multimomlinear

The multimomlinear allows solving linear dispersion equations for multi-moment multifluid equations and will eventually be extended to full kinetic equations. This tool allows arbitrary number of species, each of which can be either an isothermal fluid, a five-moment fluid or a ten-moment fluid. The fields can be computed from Maxwell equations or Poisson equations, with the option of some species “ignoring” the background fields. Certain forms of closures, including non-local Hammett-Perkins Landau fluid closures, can be used. For the list of equations and a brief overview of the algorithm used, please see this technical note. Essentially, the key idea of this algorithm is to convert the problem of finding the dispersion relation to an eigenvalue problem and then use a standard linear algebra package (Eigen in this case) to compute the eigensystem. This allows great flexibility as there is no need to directly find complex nonlinear polynomial roots or even formulate the dispersion relation explicitly. A more careful set of benchmarks and applications are presented in this document. To run this tool prepare an input file with the species you wish to include, the field and the set of wave-numbers at which the dispersion relation should be computed. An example input file for cold electron and ions is given below.

92 Chapter 3. gkyl Reference gkyl Documentation, Release 2.0-alpha

Fig. 6: The exact solution to the sod-shock Riemann problem computed using the exacteulerrp tool.

3.4. gkyl Tools 93 gkyl Documentation, Release 2.0-alpha

local Species= require"Tool.LinearSpecies"

-- Electrons elc= Species.Isothermal { mass= 1.0, -- mass charge=-1.0, -- charge density= 1.0, -- number density velocity={0.0, 0.0, 0.0}, -- velocity vector temperature= 0.0, -- temperature }

-- Ions ion= Species.Isothermal { mass= 25.0, -- mass charge= 1.0, -- charge density= 1.0, -- number density velocity={0.0, 0.0, 0.0}, -- velocity vector temperature= 0.0, -- temperature }

-- EM field field= Species.Maxwell { epsilon0= 1.0, mu0= 1.0,

electricField={0.0, 0.0, 0.0}, -- background electric field magneticField={1.0, 0.0, 0.75}, -- background magnetic field }

-- list of species to include in dispersion relation speciesList= { elc, ion }

-- List of wave-vectors for which to compute dispersion relation kvectors={} local kcurr, kmax, NK= 0.0, 4.0, 401 dk= (kmax-kcurr)/(NK-1) for i=1, NK do kvectors[i]= {kcurr, 0.0, 0.0} -- each k-vector is 3D kcurr= kcurr+ dk end

Any number of species can be specified and the field can be either Species.Maxwell or Species.Poisson. The wave-vectors at which to compute the dispersion are specified in the kvector table, which is a list of three element tables (with components 푘푥, 푘푦, 푘푧). To run this input file (say it is saved in cold-plasma.lua): gkyl multimomlinear-i cold-plasma.lua

This will create a output ﬁle named cold-plasma_frequencies.bp, with the eigenvalues stored in a Gkeyll “DynVector” object. For each element in the dynvector, the ﬁrst three components are the components of the wave-vector and the rest the corresponding 휔푛(k) with real and imaginary parts stored separately (next to each other). You can plot the real part of the frequencies as function of wave-vector (say 푘푥) as: pgkyl cold-plasma_frequencies.bp val2coord-x0-y3::2 pl-s-f0--xlabel"k"--

˓→ylabel'$\omega_r$'--markersize=2

94 Chapter 3. gkyl Reference gkyl Documentation, Release 2.0-alpha

And the imaginary parts as:

pgkyl cold-plasma_frequencies.bp val2coord-x0-y4::2 pl-s-f0--xlabel"k"--

˓→ylabel'$\omega_r$'--markersize=2

Often, it is useful to plot the eigenvalues in the complex plane (real part on X-axis and imaginary part on the Y-axis). For this do:

pgkyl cold-plasma_frequencies.bp val2coord-x3::2-y4::2 pl-s-f0--xlabel'$\omega_

˓→r$'--ylabel'$\omega_i$'--markersize=2

Note that the frequencies are not outputed in any particular order. Hence it is not possible to easily extract a single “branch” of the dispersion relation from the output. Please see pgkyl help to understand what the val2coord and pl (short for plot) do and how to use them. Example of the real frequency for the cold plasma waves is shown below

Fig. 7: Cold plasma dispersion computed using multimomlinear tool. Seen are the L-mode branch, the two branches of the R-mode, and the low-frequency ion-scale waves.

The species objects can be one of Species.Isothermal, Species.Euler or Species.Tenmoment. The input parameters accepted by each of these objects are given below. Note that the input parameters can either be dimensional or dimensionless. The tool itself does not use any non-dimensionalization. The Species.Isothermal takes the following parameters:

3.4. gkyl Tools 95 gkyl Documentation, Release 2.0-alpha

Table 12: Parameters for Species.Isothermal Parameter Description Default mass Mass of particle charge Charge on particle density Number density velocity Velocity vector {푣푥, 푣푦, 푣푧} temperature Fluid temperature (set to zero for cold ﬂuid) ignoreBackgroundFieldDo not consider background electromagnetic ﬁeld on species. false

The Species.Euler takes the following parameters:

Table 13: Parameters for Species.Euler Parameter Description Default mass Mass of particle charge Charge on particle density Number density velocity Velocity vector {푣푥, 푣푦, 푣푧} pressure Fluid pressure gasGamma Gas adiabatic index 5/3 ignoreBackgroundFieldDo not consider background electromagnetic ﬁeld on species. false

The Species.Tenmoment takes the following parameters:

Table 14: Parameters for Species.TenMoment Parameter Description Default mass Mass of particle charge Charge on particle density Number density velocity Velocity vector {푣푥, 푣푦, 푣푧} pressureTensor Pressure in the ﬂuid frame as a table {푃푥푥, 푃푥푦, 푃푥푧, 푃푦푦, 푃푦푧, 푃푧푧} useClosure Flag to turn on various collisionless closures false chi Multiplicative factor in closure. √︀4/9휋. ignoreBackgroundFieldDo not consider background electromagnetic ﬁeld on species. false

Note: • Presently, a unmagnetized Hammett-Perkins closure is implemented. See tech-note linked above and [Ng2017]. This is not always very useful and accurate, specially for strongly magnetized problems. We hope to implement better closures soon. • The ignoreBackgroundField allows a species to “ignore” the applied background electromagnetic ﬁelds. This is often useful when one species is unmagnetized.

3.4.3 References

96 Chapter 3. gkyl Reference CHAPTER 4

Postgkyl reference

“I love pgkyl! However, Petr, the next feature I want is . . . ” – Ammar Hakim Below is the living reference to the Postgkyl tool for post-processing gkyl simulation data. The main aspiration of Postgkyl is to create a postprocessing suit which would: 1. Make simple tasks very easy 2. Make complex tasks possible The documentation is split into three sections. The ﬁrst section introduces the key concepts of using Postgkyl and goes into details of loading data. Particularly the former one is a strongly recommended reading for new users. The second part provides many examples of using Postgkyl, which aim to supplement the examples already presented in the Quickstart chapter. Finally, the last part consists of detailed documentation for each individual available command.

4.1 Using Postgkyl

Postgkyl is, at its core, a Python library. When properly installed (see pg_install), it can be loaded in any Python script or interactive environment like IPython or JupyterLab. import postgkyl

Postgkyl can be used to read all the Gkeyll data outputs (including the legacy Gkeyll 1 files), can transform the raw expansion coefficients of Gkeyll basis functions to finite-volume style data, and also contains many postprocessing functions. See the following sections for more details.

4.1.1 Key concepts

There are a few basic concepts of using pgkyl in a command line. Together, they allow to quickly and easily create quite complex diagnostics, which would otherwise require writing custom postpocessing scripts. Note that there are often multiple ways to achieve the same thing. Sometimes, they are analogous, othertimes, one is superior. This page makes an attempt to explain these key concepts to allow user to choose the best solution for each situation.

97 gkyl Documentation, Release 2.0-alpha

Contents

• Key concepts – Dataset – Command chaining – Tags – Active and inactive datasets – Overwriting vs. new dataset

Dataset

Each data file which is loaded creates a dataset. Additionally, some commands can create new datasets during the flow. When files are loaded using wildcard characters, each match creates its own dataset. Therefore, assuming there are two files, file1.bp and file2.bp, located in the current directory, the two following commands will have the same result; both will create two datasets:

pgkyl file1.bp file2.bp pgkyl file?.bp

The info command with the -c ﬂag is useful to list all available datasets.

Command chaining

Commands are evaluated from left to right. Each command, by default applies to everything to the left of it. For example, this command chain combines loading data ﬁles and interpolate command:

pgkyl file1.bp interpolate file2.bp

file1.bp is loaded first, its DG expansion coefficients are then interpolated on a finer uniform mesh, and, finally, file2.bp is loaded. interpolate will not be applied on file2.bp. This particular example can be used, for example, to simultaneously work with finite-element and finite-volume data. Commands that should not be applied on all the datasets can be further controlled using tags and by designating some datasets as inactive. Note that there are some commands, e.g., collect, which switch their inputs to inactive themselves. It is worth noting that there is no limit on how many commands can be chained. See, for example, the particle balance section the the gyrokinetic quickstart page.

Tags

During loading, optional ﬂag, --tag or -t, can be used to assign a tag to the dataset(s). The default behavior of most of the commands is agnostic to the tags. For example, the following two commands will lead to the same result:

pgkyl file1.bp file2.bp plot pgkyl file1.bp -t 'f1' file2.bp -t 'f2' plot

98 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

However, most of the commands can take the --use or -u ﬂag to limit them only to the datasets with the speciﬁed tag. Similar to the example above, this can be useful when working with different types of data:

pgkyl file1.bp -t 'f1' file2.bp -t 'f2' interpolate -u f1 plot

Here, interpolate will be used only on the file1.bp even though it follows loading both of the ﬁles. The plot command will then apply to both the datasets. Note that multiple comma-separated tags can be used:

pgkyl file1.bp -t 'f1' file2.bp -t 'f2' file3.bp -t 'f3' interpolate -u f1,f2 plot

Additionally, there are some commands like collect or animate are by default tag-aware and separate datasets with different tags from each other. When no tag is speciﬁed, the default tag is assigned.

Warning: When using tags together with wildcard characters, it is important to use quotes, e.g.: pgkyl 'file?.bp' -t name

Without the quotes, the string is replaced with all the matches, pgkyl treats them as separate load commands, and the speciﬁed tag is applied only to the last match.

Active and inactive datasets

In addition to specifying tags, the ﬂow of a pgkyl command chain can be controlled by activating and deactivat- ing datasets. By default, all loaded datasets are active. This can be changed with the pair of pg_cmd_activate and pg_cmd_deactivate commands. In addition, commands that create a new dataset, e.g., collect, leave only the output active. The motivation behind this is that these commands change the nature of data and user would typically want to keep working only with the result. The aforementioned collect turns N-dimensional data to (N+1)-dimensional data. With the inputs inactive, commands can be easily chained, e.g.,

pgkyl 'file*.bp' collect plot

pg_cmd_activate can either take in indices, tags, or both. When no inputs are speciﬁed, everything is activated. The two following commands provide yet another way to to achieve the same result as in the tag example above:

pgkyl file1.bp -t f1 file2.bp -t f2 activate -t f1 interpolate activate plot pgkyl file1.bp file2.bp activate -i0 interpolate activate plot

In both cases only the file1.bp is active and, therefore, the interpolate command is applied only on the first file. The second activate then reactivates the second file again so the plot command is going to plot both. The info command can be useful when working with multiple active/inactive datasets. Its --compact option shows only identifiers for each dataset, thus removes some clatter, and --allsets adds even the currently inactive datasets.

Overwriting vs. new dataset

There are two basic ways commandsinteract with inputs. The first type modifies its inputs and pushes data down the chain. A typical example is the interpolate command, which takes expansion coefficients of DG finite-element data and interpolates them on a finer uniform mesh, essentially creating finite-volume like data.

pgkyl file1.bp interpolate plot

4.1. Using Postgkyl 99 gkyl Documentation, Release 2.0-alpha

In this case the original information is lost after the interpolate command (lost within this command chain, nothing happens to the data ﬁle itself). The other type does not overwrite its inputs but rather creates a new dataset. As a rule of thumb, these are commands that take (or can take) multiple inputs and/or change the nature of data. Note that these commands often make the result the only active dataset to simplify the ﬂow. A typical example is the ev command:

pgkyl file1.bp file2.bp ev 'f[0] f[1] -' plot

As a result of this chain, there will be three datasets; however, only the result of ev will be active, so the plot command will create just one figure. There are instances when user does not want to overwrite the inputs. For example, when we want to use select to create multiple slices of data. For this purpose, the commands that would normally overwrite data have the optional --tag or -t flag which instead creates a new dataset with specified tag. Note that in this case, the resulting dataset will not be the only one active.

pgkyl file1.bp -t input select -u input --z0 -1. -t planes \ select -u input --z01. -t planes plot -u planes

4.1.2 Data loading

“I disagree strongly with whatever work this quote is attached to.” – Randall Munroe One can argue that loading data is the most important part of a postprocessing tool. In Postgkyl, it is handled by the postgkyl.data.Data class (there is a postgkyl.Data shortcut). It load data on initialization and serves as an input for all the other parts of Postgkyl. Examples are provided simultaneously for scripting and command line using output ﬁles of an electrostatic two-stream instability simulation [two-stream.lua].

Contents

• Data loading – Accessing a Gkeyll ﬁle – Loading multiple datasets – Partial loading – Tags and labels – Loading data with c2p mapping

Accessing a Gkeyll ﬁle

Gkeyll ﬁles are loaded in Postgkyl by creating a new instance of the Data class with the ﬁle name as the parameter.

Listing 1: Script import postgkyl as pg data= pg.Data('filename')

Next, getGrid() and getValues() can be used to return the grid and values as NumPy arrays. For structured meshes, the getGrid() return a Python list of 1D NumPy arrays which represent the nodal points of the grid in

100 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha each dimension. Note that since these are nodal points, these arrays will always have one more cell in each dimension in comparison to the value array. Another important note is that the value array always have one extra dimension for components. Components can represent many things from vector elements to discontinuous Galerkin expansion coefﬁcients. As a rule, this extra dimension is always retained even if there is just one component. import postgkyl as pg data= pg.Data('two-stream_elc_0.bp') print(data.getGrid()) [array([-6.283185307179586,-6.086835766330224,-5.890486225480862, -5.6941366846315,-5.497787143782138,-5.301437602932776, -5.105088062083414,-4.908738521234052,-4.71238898038469, -4.516039439535327,-4.319689898685965,-4.123340357836604, -3.9269908169872414,-3.730641276137879,-3.5342917352885173, -3.3379421944391554,-3.141592653589793,-2.945243112740431, -2.748893571891069,-2.552544031041707,-2.356194490192345, -2.1598449493429825,-1.9634954084936211,-1.7671458676442588, -1.5707963267948966,-1.3744467859455343,-1.178097245096172, -0.9817477042468106,-0.7853981633974483,-0.589048622548086, -0.3926990816987246,-0.1963495408493623, 0., 0.1963495408493623, 0.3926990816987246, 0.589048622548086, 0.7853981633974483, 0.9817477042468106, 1.178097245096172, 1.3744467859455343, 1.5707963267948966, 1.767145867644258, 1.9634954084936211, 2.1598449493429825, 2.356194490192344, 2.552544031041707, 2.7488935718910685, 2.9452431127404317, 3.141592653589793, 3.3379421944391545, 3.5342917352885177, 3.730641276137879, 3.9269908169872423, 4.123340357836604, 4.319689898685965, 4.516039439535328, 4.71238898038469, 4.908738521234051, 5.105088062083414, 5.301437602932776, 5.497787143782137, 5.6941366846315, 5.890486225480862, 6.086835766330225, 6.283185307179586 ]), array([-6.,-5.8125,-5.625,-5.4375,-5.25,-5.0625,-4.875, -4.6875,-4.5,-4.3125,-4.125,-3.9375,-3.75,-3.5625, -3.375,-3.1875,-3.,-2.8125,-2.625,-2.4375,-2.25, -2.0625,-1.875,-1.6875,-1.5,-1.3125,-1.125,-0.9375, -0.75,-0.5625,-0.375,-0.1875, 0., 0.1875, 0.375, 0.5625, 0.75, 0.9375, 1.125, 1.3125, 1.5, 1.6875, 1.875, 2.0625, 2.25, 2.4375, 2.625, 2.8125, 3., 3.1875, 3.375, 3.5625, 3.75, 3.9375, 4.125, 4.3125, 4.5, 4.6875, 4.875, 5.0625, 5.25, 5.4375, 5.625, 5.8125, 6. ])] print(data.getValues()) [[[ 1.6182154425614533e-127 2.2497634664678846e-136 2.1705614015952743e-127... 1.4466223559100639e-127 7.7862978418103503e-137 2.0112020871650523e-136] [ 7.2163320153412515e-118 1.0032681083505769e-126 9.6785762877207286e-118... 6.4497610162539372e-118 3.4719259660326997e-127 8.9669370964188083e-127] [ 1.3363156717841295e-108 1.8578453383418215e-117 1.7920360303344134e-108... 1.1940080895062958e-108 6.4284392330301674e-118 1.6599988152412963e-117] ... print(data.getGrid()[0].shape) (65,) print(data.getGrid()[1].shape) (65,) print(data.getValues().shape) (64, 64,8)

4.1. Using Postgkyl 101 gkyl Documentation, Release 2.0-alpha

It is also possible to create an empty instance and ﬁll it using the push function. In the command line mode, a data ﬁle is loaded by simply adding it to the pgkyl script chain at any position.

Listing 2: Command line pgkyl filename

Note: Under the hood, Postgkyl calls a hidden load command to load the file. When provided string does not match any command but is matching a file, the load command is invoked and the file name is passed to it. The load command should not be called manually but it can be used to access the help.

pgkyl load --help

Currently, Postgkyl supports h5 file that were used in Gkeyll 1, Gkeyll 2 ADIOS bp files, and Gkeyll 0 gkyl binary files. Many of the advanced functions like loading only partial data and some quality of life features like storing the polynomial order of DG representation are currently available only for the ADIOS bp files.

Loading multiple datasets

Loading multiple ﬁles in a script is straightforward; one creates more instances of the Data class. Postgkyl does naturally support loading any number of ﬁles.

pgkyl two-stream_elc_0.bp two-stream_elc_1.bp

All the commands are then generally batch performed on all the data sets and the plot command creates a separate figure for each data set (this can be modified with plot options like -f0). When batch application of commands is not the desired behavior, some data files can be loaded later in the chain, loaded dataset can be changed from active to inactive (pg_cmd_activate/pg_cmd_deactivate), or the command scope can be limmited by specifying tags. The Key concepts section provides examples where one desired behavior is achieved in multiple ways. It is left up to the user to chose the preferred one. Postgkyl also allows for loading with a wild card characters:

pgkyl 'two-stream*.bp'

Warning: While the quotes are entirely optional when loading a single file, they change behavior when used with wild card characters. With quotes, a single load command is performed and the wild card matching is done internally by Postgkyl. Without quotes, the wild card is replaced before calling Postgkyl which results in several load command calls. This leads to several key differences: 1. With quotes, Postgkyl orders files correctly, i.e., file_2 will be before file_10. 2. With quotes, tags, labels, etc., are applied to all the matching files, not just the last one. 3. Some wildcard characters like [0-9] are not supported by every shell.

Using wild card characters might lead to unexpected situations. For example in the two-stream case, the query two-stream_elc_* is going to return two-stream_elc_0.bp but also the moment ﬁles like two-stream_elc_M0_0.bp. If we want to load just the distribution functions, we can limit the query. For example:

102 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

pgkyl 'two-stream_elc_[0-9]*.bp'

This requires the first character to be a number between 0 and 9, which effectively eliminates all the outputs except for the distribution functions themselves. Following are details on load parameters which alter the behavior. Here, we would like to mention that these can be specified individually for each file of as the global options of the pgkyl script itself. For example, the partial loading flag --z0 (see bellow) can be applied to one file (file_0):

pgkyl file_0 --z00 file_1

Or it can be applied globally to all the ﬁles:

pgkyl --z00 file_0 file_1

This is analogous to:

pgkyl file_0 --z00 file_1 --z00

Partial loading

Gkeyll output ﬁles, especially the higher dimensional ones, can be large. Therefore, Postgkyl allows to load just a smaller subsection of each ﬁle. This is done with the optional z0 to z5 parameters for coordinates and comp for components. Each can be either an integer number or a string in the form of start:end. Note that this does follow the Python convention so the last index is excluded, i.e., 1:5 will load only the indices/components 1, 2, 3, and 4. This functionality is supported both in the script mode and the command line mode.

Listing 3: Script import postgkyl as pg data= pg.Data('two-stream_elc_0.bp', z1='1:3', comp=0)

Listing 4: Command line pgkyl two-stream_elc_0.bp --z11:3 -c0

Note that the select command has a similar use. In addition, it allows to specify a coordinate value instead of an index. However, it requires the whole ﬁle to be loaded into memory.

Tags and labels

Datasets can be decorated with tags and labels. The former serve mostly to specify the scope of commands (see tags) in the command line mode while the later one allows to add custom labels for plots and print-outs. When no labels are specified, Postgkyl attempts to find the shortest unique identifier and uses it as a label. For example:

pgkyl two-stream_elc_0.bp two-stream_elc_1.bp info -c 0(default#0) 1(default#1)

pgkyl two-stream_elc_0.bp two-stream_field_0.bp info -c elc(default#0) field(default#1)

4.1. Using Postgkyl 103 gkyl Documentation, Release 2.0-alpha

pgkyl two-stream_elc_0.bp two-stream_field_1.bp info -c elc_0(default#0) field_1(default#1)

These labels, can be customized and can include LaTeX syntax, which will be properly rendered in a plot legend.

pgkyl two-stream_elc_0.bp -l '$t\omega_{pe}=0$' two-stream_elc_1.bp -l '$t\omega_{pe}

˓→=0.5$' info -c $t\omega_{pe}=0$(default#0) $t\omega_{pe}=0.5$(default#1)

Note, the in all these examples, both datasets have the default tag and are indexed 0 and 1. These can be manually speciﬁed.

pgkyl two-stream_elc_0.bp -t 'el' two-stream_field_0.bp -t 'em' info -c elc(el#0) field(em#0)

Loading data with c2p mapping

Warning: This feature was introduced in 1.6.7 and currently only works with gkyl binary ﬁles.

Postgkyl supports the c2p mapping used in Gkeyll. The ﬁle with the map can be speciﬁed using the --c2p keyword. Following are two plots where a Maxwellian particle distribution is evaluated in cylindrical coordinates with and without c2p map provided to Postgkyl.

pgkyl rt_eval_on_nodes_f-ser.gkyl interpolate -bms -p2 plot -a

pgkyl rt_eval_on_nodes_f-ser.gkyl --c2p rt_eval_on_nodes_rtheta-ten.gkyl interpolate -

˓→bms -p2 plot -a

Gkeyll stores the c2p coordinate information as expansion coefficients of a finite element representation independent of the representation of the data itself. It is converted to plotting nodal points during the interpolate command when the information about the data is provided. However, the interpolate command is never used when working with finite-volume data. For this instance, the --fv flag is available which converts the expansion coefficients to nodal values immediately after loading.

pgkyl euler_axis_sodshock-euler_0.gkyl --c2p euler_axis_sodshock-mapc2p.gkyl --fv

˓→select -c0 plot -a

4.1.3 Note on the colors

What makes Postgkyl quite unique, is the wrapping of all the functions into command line commands using the Click Python package. In other words, almost the full functionality of Postgkyl can be used directly in any terminal. This has beneﬁcts for everyday work where typing a single command is faster than writing a Python script and also for work with remote machines and supercomputers, which are primarily accessed through a terminal. However, the classical way of using Postgkyl in a script still provides more control and is well suited, for example, for long and complex scripts for publication level ﬁgures. The command line executable for Postgkyl is called pgkyl and its call has the following synopsis:

104 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

Fig. 1: Plot of Maxwellian distribution in cylindrical coordinates without a c2p map.

4.1. Using Postgkyl 105 gkyl Documentation, Release 2.0-alpha

Fig. 2: Plot of Maxwellian distribution in cylindrical coordinates with a c2p map provided with --c2p.

106 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

Fig. 3: Plot of ﬁnite-volume data with --c2p provided and the --fv ﬂag on.

4.1. Using Postgkyl 107 gkyl Documentation, Release 2.0-alpha

pgkyl[OPTIONS] COMMAND1[ARGS]...[COMMAND2[ARGS]...]...

Here, the COMMAND represents either data to load or an opperation to be performed on the loaded data. All the available options can be listed using the inbuilt help, pgkyl --help or pgkyl -h for short. Note that Postgkyl supports an arbitrary number of commands. Similar to piping in Linux, results of one command are passed to another. This allows Postgkyl to perform even a rather complex diagnostics and create complicated plots straight up from the terminal. However, this puts a responsibility on the user to ensure that the commands are called in a logical order. Similarly to the main part of pgkyl, --help can be called for each individual command which will provide additional information. Finally, it is worth mentioning that it is not necessary to write the full names of each command and the shortest unique sequence is good enough. Still, full names will be used through this documentation for clarity.

4.2 Examples

The Gkeyll workﬂow typically involves a mixture of command line and script mode use of postgkyl. It is useful to become versed in both of these, and to that end we provide a number of examples below.

4.2.1 Commandline examples

There are more examples in the pages describing each of the pgkyl commands and functions. For a brief set of examples we here consider the output of a two-stream instability Vlasov-Maxwell simulation.

List outputs

We can list the different kinds of ﬁles outputted by the simulation with the listoutputs command

pgkyl list two-stream_elc two-stream_elc_M0 two-stream_elc_M1i two-stream_elc_M2ij two-stream_elc_M3i two-stream_field

Notice that we used the abbreviation list instead of listoutputs; this is allowed and whenever convenient we will use abbreviations if the meaning is clear. This writes out a list of the unique ﬁlename stems of the ﬁles outputted by a simulation.

Obtain ﬁle information

Now suppose we wish to know more about one ﬁle in particular, say the electron initial condition. We can load the data set and probe it with the info command:

pgkyl two-stream_elc_0.bp info Set(default#0) Time:0.000000e+00 Frame:0 Number of components:8 Number of dimensions:2 Grid:(uniform) (continues on next page)

108 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

(continued from previous page) Dim0: Num. cells: 64; Lower: -6.283185e+00; Upper:6.283185e+00 Dim1: Num. cells: 64; Lower: -6.000000e+00; Upper:6.000000e+00 Maximum:3.804653e+00 at(31, 26) component0 Minimum: -6.239745e-01 at(31, 38) component2 DG info: Polynomial Order:2 Basis Type: serendipity(modal) Created with Gkeyll: Changeset: 82231b06678c+ Build Date: Feb7 2021 09:14:06

The output shows (in order): • Simulation time of this snapshot. • Frame number of this snapshot. • Number of degrees of freedom per cell (components). In this case 8 basis functions for the second order 2D Serendipity basis ([Arnold2011], see Modal basis functions for more details). • Number of dimensions (in this case 2 because it is a 1x1v simulation, 1D in position and 1D in velocity). • The grid resolution and extents. • The maximum value of the dataset (in this case largest DG coefﬁcient). • The minimum value of the dataset (in this case smallest DG coefﬁcient). • The polynomial order and type of basis. • The Gkeyll build used to produce this simulation.

Plotting (interpolated data or single coefﬁcients)

In the case of DG data one is most frequently interested in plotting the actual function instead of its expansion (DG) coefﬁcients. To do so we construct ﬁnite-volume-like dataset on a uniform mesh with higher resolution using the interpolate command. If we interpolate the same dataset (electron initial condition) as in the previous example and inspect it with info, we get pgkyl two-stream_elc_0.bp interpolate info Set(default#0) Time:0.000000e+00 Frame:0 Number of components:1 Number of dimensions:2 Grid:(uniform) Dim0: Num. cells: 192; Lower: -6.283185e+00; Upper:6.283185e+00 Dim1: Num. cells: 192; Lower: -6.000000e+00; Upper:6.000000e+00 Maximum:1.970512e+00 at(96, 79) Minimum: -1.177877e-08 at(94, 60) DG info: Polynomial Order:2 Basis Type: serendipity(modal) Created with Gkeyll: Changeset: 82231b06678c+ Build Date: Feb7 2021 09:14:06

Note that now there is only one degree of freedom (component) per cell but there are 3X more cells; we have interpolated the function onto a ﬁner mesh.

4.2. Examples 109 gkyl Documentation, Release 2.0-alpha

We can then plot the distribution function evaluated on this ﬁner mesh with the plot command

pgkyl two-stream_elc_0.bp interp plot

Note the allowed abbreviation of interpolate to interp. This command produces the ﬁgure below:

In some cases one may also be interested in plotting single expansion coefficients. The most common use is to, for example, plot the cell-averaged value (which is the zeroth coefficient times a constant). We can select specific coefficients in all cells with the select and the -c flag. So, to plot the cell averaged electron distribution function (times a factor) we would use

pgkyl two-stream_elc_0.bp select -c0 plot

Which produces the ﬁgure below. Notice how the values are slightly different and the resolution is coarser than in the previous plot of interpolated data.

Plot data slices

The select command introduced in the previous example can also be used with the --z ﬂag in order to select a data slice which we may subsequently plot. In the two stream instability example, the electron initial condition is clearly independent of position, so we can plot as a function of velocity space at 푥 = −2.0 with

pgkyl two-stream_elc_0.bp interp select --z0 -2. plot

110 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

4.2. Examples 111 gkyl Documentation, Release 2.0-alpha

112 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

producing the following plot Postgkyl is currently limited to 1D and 2D plots, so in order to visualize datasets that have more than 3 dimensions one may need to select several slices at once. You can do that with multiple --z ﬂags. For example, if we have a 2x2v simulation producing four-dimensional distribution functions, we can select a slice at 푣푥 = 0. and 푣푦 = 1. with --z2 0. --z3 1..

Create animations

Another useful operation is to load multiple datasets consisting of consecutive frames in a time-dependent simulation, plot them and stitch them together into a movie. This can be accomplished by the animate command. We ﬁrst load all the frames containing the electron momentum densities in time (i.e. elc_M1i), then interpolate them onto a ﬁner mesh, and put together all the frames with the animate command:

pgkyl "two-stream_elc_M1i_[0-9]*.bp" interp animate -x 'x' -y 'Momentum'

Here we are also using the -x and -y ﬂags to the animate command in order to place labels in the ﬁgure. The product is the movie given below: Of course, one can use command chaining to slice higher dimensional data prior to calling the animate command. For example, creating an animation of the distribution function along velocity-space at 푥 = 0 would be accomplished with

pgkyl "two-stream_elc_[0-9]*.bp" interp sel --z00. animate

Here we have used the abbreviations interp and sel in favor of interpolate and select, respectively. Such command produces this animation:

Averaging and integrating

A common diagnostic need is to perform averages and integrations over a dimension or over time. In general averages can be performed either by using the ev command with the avg operation, or using the integrate and later dividing by the corresponding space/time segment (with the ev command). We use the ﬁnal electron distribution function (frame 100) as an example. Let’s ﬁrst plot it in phase-space to get a sense of it:

pgkyl two-stream_elc_100.bp interp pl

Suppose we wish to average over the central region 푥 ∈ [−2, 2] where a velocity-space ‘hole’ forms. We can plot such 푥 integral as follows

pgkyl two-stream_elc_100.bp interp sel --z0 -2.:2. ev 'f[0] 0 avg' pl

As mentioned above, we can also do this with the integrate command. We accomplish that with

pgkyl two-stream_elc_100.bp interp sel --z0 -2.:2. integrate0 ev 'f[0] 4. /' pl

Another useful application of ev (with avg) and integrate is to average or integrate quantities over time. Consider the evolution of the electron distribution function along velocity space at 푥 = 0 in the previous example. The action starts after the 60th frame approximately, so if we wish to time-average the distribution function at 푥 = 0 we could use the ev command as follows:

pgkyl "two-stream_elc_[0-9]*.bp" activate -i '59:' interp sel --z00. collect ev ˓→'f[0] 0 avg' pl

4.2. Examples 113 gkyl Documentation, Release 2.0-alpha

114 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

4.2. Examples 115 gkyl Documentation, Release 2.0-alpha

A different way to accomplish the same time average over frames 59-100 and dividing by the corresponding time period (휏 = 50 − 29.5018 = 20.4982): pgkyl "two-stream_elc_[0-9]*.bp" activate -i '59:' interp sel --z00. collect ˓→integrate0 ev 'f[0] 20.4982 /' pl

To break this last approach down, the command does the following (in order): • Load all frames of the electron distribution function. • Activate only frames 59-100. • Interpolate each frame onto a ﬁner mesh and slice at 푥 = 0. • Collect all the slices into a single dataset. This produces a 2D dataset with time along the 0th dimension and velocity-space along the 1st dimension. • Use the integrate command to integrate along the 0th dimension (time). • Use ev to divide the time-integrated quantity by the appropriate time window. • Plot. The product of either of these comands is shown below:

116 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

Plot differences between datasets

It is common to have to evaluate the difference between two datasets. These could be two frames from the same simulation, or two datasets from different simulations. There are also various ways to discern differences, and below we show how to plot them in a single ﬁgure or how to plot their actual difference. Suppose we wish to see how the electron distribution function has changed along 푥 between 푡 = 0 (0th frame) and 푡 = 50 (100th frame) at 푣푥 = 0. We can plot both of these datasets as follows pgkyl two-stream_elc_0.bp two-stream_elc_100.bp interp sel --z10. plot -f0 -x '$x$'-

˓→y '$f_e(x,v_x=0)$' where we have speciﬁed the ﬁgure with -f0 so they are both plotted together, and we have used -x and -y to place labels. The plot produced is

Another alternative is to compute the actual difference of the two data sets with ev: pgkyl two-stream_elc_0.bp two-stream_elc_100.bp interp sel --z10. ev 'f[1] f[0] -' pl

Note that these operations also work with 2D datasets. So we could’ve have taken the whole distribution function in phase space at 푡 = 0, 50, subtracted them and plot them with pgkyl two-stream_elc_0.bp two-stream_elc_100.bp interp ev 'f[1] f[0] -' plot --

˓→diverging which thanks to the --diverging ﬂag, produces the following image:

4.2. Examples 117 gkyl Documentation, Release 2.0-alpha

118 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

4.2. Examples 119 gkyl Documentation, Release 2.0-alpha

Saving plots/animations to a ﬁle

Any of the figures above can be saved to a file by appending either --save, or --saveas followed by the desired filename. For example the diverging 2D colorplot in the previous section can be saved to a file with

pgkyl two-stream_elc_0.bp two-stream_elc_100.bp interp ev 'f[1] f[0] -' plot --

˓→diverging --saveas 'two-stream_elc_fr0m100.png'

Fileformats supported depend on matplotlib, but likely include .png, .pdf and .eps. The animate command is also capable of saving animations to a ﬁle. It requires an ffmpeg installation, and once that is available we can create an animation described earlier as

pgkyl "two-stream_elc_[0-9]*.bp" interp sel --z00. animate --saveas 'two-stream_elc_ ˓→z0eq0p0.mp4'

Extracting input ﬁle

In our commitment to reproducibility, Gkeyll output files store the Lua input file used to produce that data. This input file can be extracted using the extractinput command, as follows

pgkyl two-stream_elc_0.bp extractinput

By default, this commands simply prints the input ﬁle to screen. However, this could be easily piped into a new ﬁle with

pgkyl two-stream_elc_0.bp extractinput > newInputFile.lua

Reference

4.2.2 Script mode examples

Examples of using the script mode of postgkyl. Simple loading of data in grids:

import postgkyl as pg

def func_data(ionDensityData): ionDensityInterp= pg.data.GInterpModal(ionDensityData,1,'ms') interpGrid, ionDensityValues= ionDensityInterp.interpolate()

# get cell center coordinates CCC=[] for j in range(0,len(interpGrid)): CCC.append((interpGrid[j][1:]+ interpGrid[j][:-1])/2)

x_vals= CCC[0] y_vals= CCC[1] z_vals= CCC[2] X, Y= np.meshgrid(x_vals, y_vals) ionDensityGrid= np.transpose(ionDensityValues[:,:,z_slice,0]) return x_vals,y_vals,X,Y,ionDensityGrid

(continues on next page)

120 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

(continued from previous page) ionDensity=DIR+'_ion_GkM0_%d'%data_num[i]+'.bp' ionDensityData= pg.data.GData(ionDensity) x_vals,y_vals,X,Y,ionDensityGrid= func_data(ionDensityData)

Evaluating derivative of data on grids: import postgkyl as pg def func_data(phiData): phiInterp= pg.data.GInterpModal(phiData,1,'ms') interpGrid, phiValues= phiInterp.interpolate()

exValues=- np.gradient(phiValues,dx,axis=0) dexdxValues= np.gradient(exValues,dx,axis=0) eyValues=- np.gradient(phiValues,dy,axis=1)

# get cell center coordinates CCC=[] for j in range(0,len(interpGrid)): CCC.append((interpGrid[j][1:]+ interpGrid[j][:-1])/2)

x_vals= CCC[0] y_vals= CCC[1] z_vals= CCC[2] X, Y= np.meshgrid(x_vals, y_vals) exGrid= np.transpose(exValues[:,:,z_slice,0]) eyGrid= np.transpose(eyValues[:,:,z_slice,0]) dexdxGrid= np.transpose(dexdxValues[:,:,z_slice,0])

return x_vals,y_vals,X,Y,exGrid,eyGrid,dexdxGrid phi=DIR+'_phi_%d'%data_num[i]+'.bp' phiData= pg.data.GData(phi) x_vals,y_vals,X,Y,exGrid,eyGrid,dexdxGrid= func_data(phiData)

4.3 Function/Command reference

This is the reference for Postgkyl functions and commands. In the most cases, the commands are simple click wrappers of the Python function. However, where a command addresses something that is easilly done in a script, there is no Python function equivalent. Examples are provided simultaneously for the analogous functions and commands using output ﬁles of an electrostatic two-stream instability simulation [two-stream.lua].

Contents

• Function/Command reference – Output

4.3. Function/Command reference 121 gkyl Documentation, Release 2.0-alpha

– Data selection – Data manipulation – Diagnostics – Command control

4.3.1 Output animate

Create movies (animations) by stitching together ﬁgures created by plotting multiple datasets, typically originating from a data load with wildcard/regex. Saving the animation to a ﬁle requires an ffmpeg installation.

Command line pgkyl animate -h Usage: pgkyl anim[OPTIONS]

Animate the actively loaded dataset and show resulting plots in a loop. Typically, the datasets are loaded using wildcard/regex feature of the -f option to the main pgkyl executable. To save the animation ffmpeg needs to be installed.

Options: -u, --use TEXT Specify a tag to plot. -s, --squeeze Squeeze the components into one panel. -b, --subplots Make subplots from multiple datasets. --nsubplotrow INTEGER Manually set the number of rows for subplots. --nsubplotcol INTEGER Manually set the number of columns for subplots. --transpose Transpose axes. -c, --contour Make contour plot. -q, --quiver Make quiver plot. -l, --streamline Make streamline plot. -g, --group[0|1] Switch to group mode. -s, --scatter Make scatter plot. --markersize FLOAT Set marker size for scatter plots. --style TEXT Specify Matplotlib style file(default: Postgkyl).

-d, --diverging Switch to diverging colormesh mode. --arg TEXT Additional plotting arguments, e.g.,' *--'. -a, --fix-aspect Enforce the same scaling on both axes. --logx Set x-axis to log scale. --logy Set y-axis to log scale. --logz Set values of 2D plot to log scale. --xscale FLOAT Value to scale the x-axis(default:1.0). --yscale FLOAT Value to scale the y-axis(default:1.0). --vmax FLOAT Set maximal value of data for plots. --vmin FLOAT Set minimal value of data for plots. -f, --float Choose min/max levels based on current frame (i.e., each frame uses a different color range). (continues on next page)

122 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

(continued from previous page)

--xlim TEXT Set limits for the x-coordinate(lower,upper) --ylim TEXT Set limits for the y-coordinate(lower,upper). --legend / --no-legend Show legend. --force-legend Force legend even when plotting a single dataset.

-x, --xlabel TEXT Specify a x-axis label. -y, --ylabel TEXT Specify a y-axis label. --clabel TEXT Specify a label for colorbar. --title TEXT Specify a title. -i, --interval INTEGER Specify the animation interval. --save Save figure as PNG. --saveas TEXT Name to save the plot as. --fps INTEGER Specify frames per second for saving. --dpi INTEGER DPI(resolution) for output. -e, --edgecolors TEXT Set color for cell edges(default: None) --showgrid / --no-showgrid Show grid-lines(default: True) --collected Animate a dataset that has been collected, i.e. a single dataset with time taken to be the first index.

--hashtag Turns on the pgkyl hashtag! --show / --no-show Turn showing of the plot ON and OFF(default: ON).

-h, --help Show this message and exit.

Suppose we extend the simulation time of the gyrokinetic ion acoustic wave :simulation to tEnd=50 and the number of frames to nFrame=100, and run it with gkyl gk-ionSound-1x2v-p1.lua

Note: This simulation took 582 seconds running with decompCuts=8 (8 cores) on a 2020 MacBookPro. and plot the electrostatic potential from all frames with pgkyl "gk-ionSound-1x2v-p1_phi_[0-9]*.bp" interp collect pl -x '$x$' -y '$\phi$'-- ˓→group1 --clabel 'time' this produces the pretty picture below, showing the electrostatic potential as a function of 푥 in each frame, with the color indicating the time stamp of that frame. This kind of plot can become impractical if there are many frames, or if one would instead like to see a time-dependent movie of the evolution of the potential. For that purpose we can create an animation of 휑(푥, 푡) using the following command pgkyl "gk-ionSound-1x2v-p1_phi_[0-9]*.bp" interp anim -x '$x$' -y '$\phi$' --saveas ˓→'gk-ionSound-1x2v-p1_phi.mp4'

Note: In order to save the animation to an .mp4 ﬁle, simply append --saveas fileName.mp4 to the end of the above command. and this will produce the animation below:

4.3. Function/Command reference 123 gkyl Documentation, Release 2.0-alpha

124 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

One can clearly see the amplitude of the wave decaying as Landau damping ruins the fun. extractinput

Data files produced by Gkeyll may have the input file used to generate that data saved in them as an attribute. We can extract the input file and print it to screen or to a new file with extractinput. Certain versions of ADIOS (the I/O library Gkeyll uses) have trouble saving very long files. For this reason saving the input file in data files is sometimes disabled. In those cases extractinput may fail or return nothing.

Command line pgkyl extractinput -h Usage: pgkyl extractinput[OPTIONS]

Extract embedded input file from compatible BP files

Options: -u, --use TEXT Specify a 'tag' to apply to(default all tags). -h, --help Show this message and exit.

If we run the ion acoustic gyrokinetic simulation, it would produce save the initial ion distribution function in the file gk-ionSound-1x2v-p1_ion_0.bp. We can extract the input file used to generate this data with pgkyl gk-ionSound-1x2v-p1_ion_0.bp extract where we have shortened extractinputfile into extract for simplicity. This would print it to screen. Suppose you wish to write it to a new file because, say, you lost the original input file and wish to recreate or restart this simulation. You could write the saved input file to a new input file with: pgkyl gk-ionSound-1x2v-p1_ion_0.bp extract > gk-ionSound-1x2v-p1.lua

If you are using the same name as the original input ﬁle you’ll be able to restart the simulation. If the original input ﬁle is still in the directory it will be overwritten. info info is a function of the postgkyl.data.Data class that returns information about the current dataset (see Data loading for more details about the class itself). The function doesn’t take any arguments and it is wrapped into the info command.

Command line pgkyl info --help Usage: pgkyl info[OPTIONS]

Print info of active datasets.

Options: -u, --use TEXT Specify a 'tag' to apply to(default all tags). -c, --compact Show in compact mode (continues on next page)

4.3. Function/Command reference 125 gkyl Documentation, Release 2.0-alpha

(continued from previous page) -a, --allsets All data sets. -h, --help Show this message and exit. info always prints information about time stamp, number of components (these are typically individual expansion coefﬁcients and/or vector components), number of dimensions with number of cells. A useful capability of info when working with complex pgkyl commands (especially if using tags) is the -ac ﬂags. For example, if we had been working on the time averaged potential in the integrate page, and instead of plotting we placed info -ac at the end: pgkyl gk-ionSound-1x2v-p1_phi_0.bp -l '$t=0$' -t phi0 \ "gk-ionSound-1x2v-p1_phi_[0-9]*.bp" -t phis interp collect -u phis -t phiC \ integrate -u phiC -t phiInt0 ev -l 'Time average' -t phiAvg 'phiInt 10. /' \ activate -t phi0,phiAvg info -ac we would get the following output

Set $t=0$(phi0#0) Set0(phis#0) Set1(phis#1) ... Set 50(phis#50) Set collect(phiC#0) Set(phiInt#0) Set Time average(phiAvg#0)

Showing the label, tag and index of each dataset.

Script mode import postgkyl as pg data= pg.data.Data('two-stream_elc_0.bp') print(data.info())

Note that info() produces a single string output. Therefore, it is recommended to use the print() function for readable output. listoutputs

You can list the unique filename stems of all the files written by a simulation. This is useful if you are unsure about which outputs were produced, or want to browse all possible files/diagnostics. This command doesn’t take any arguments, and can be used with the appreviation list

Command line pgkyl list --help Usage: pgkyl list[OPTIONS]

List Gkeyll filename stems in the current directory

Options: (continues on next page)

126 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

(continued from previous page) -e, --extension TEXT Output file extension[default: bp] -h, --help Show this message and exit.

In the directory where the simulation was run type pgkyl list and you will obtain a list of the unique ﬁlename stems in that directory. For example, if I run the gk-alfven-p1.lua simulation I would get gk-alfven-p1_allGeo gk-alfven-p1_apar gk-alfven-p1_dApardt gk-alfven-p1_electron gk-alfven-p1_electron_GkM0 gk-alfven-p1_electron_GkM1 gk-alfven-p1_electron_f0 gk-alfven-p1_electron_f1 gk-alfven-p1_ion gk-alfven-p1_ion_GkM0 gk-alfven-p1_ion_GkM1 gk-alfven-p1_laplacianWeight gk-alfven-p1_modifierWeight gk-alfven-p1_phi

Note that if multiple simulations with different input ﬁles have been run in the same directory it will list the unique ﬁlename stems for all such simulations. plot postgkyl.output.plot plots 1D and 2D data using Matplotlib.

4.3. Function/Command reference 127 gkyl Documentation, Release 2.0-alpha

Table 1: plot arguments and parameters Parameter Description data Gkeyll Data object to be plotted args An optional tuple of additional Matplotlib parameters, .e.g., -- for dashed lines scatter (1D mode) Only shows points without lines streamline(2D mode) Shows streamlines of vector data quiver (2D mode) Shows vector arrows countour (2D mode) Shows contours instead of area diverging (2D mode) Changes colormap to diverging (red to blue) and centers value range around 0 group (2D mode) Splits 2D data into 1D lines and plots them on top of each other xscale Multiply x-axis by the set number (default: 1.0) yscale Multiply y-axis by the set number (default: 1.0) logx Changes x-axis to logaritmic logy Changes y-axis to logaritmic logz Changes 2D value axis to logaritmic style Chages the Matplotlib style ﬁle legend Turns the legend on and off (default: on) labelPrefixAllows to pass a preﬁx for the automated label xlabel Sets the lable for the x-axis (accepts LaTeX) ylabel Sets the label for the y-axis (accepts LaTeX) title Sets the plot title (accepts LaTeX) fixaspect Fixes the aspect ratio between the x and y axies vmin Sets the minimum value for a 2D plot vmax Sets the maximum value for a 2D plot xlim Manually sets the x-axis limits ylim Manually sets the y-axis limits showgrid Turns the grid on and off (default: on) hashtag Adds a little #pgkyl label to the bottom right corner xkcd Turns on the xkcd mode color Manually sets the color for 1D plots markersizeOverwrites the marker size value

The Python function is wrapped into the plot command.

$ pgkyl plot -h Usage: pgkyl plot[OPTIONS]

Plot active datasets, optionally displaying the plot and/or saving it to PNG files. Plot labels can use a sub-set of LaTeX math commands placed between dollar($) signs.

Options: -f, --figure TEXT Specify figure(integer) to plot in. -s, --squeeze Squeeze the components into one panel. -b, --subplots Make subplots from multiple datasets. --arg TEXT Additional plotting arguments like' *--'. -c, --contour Draw contour plot. -q, --quiver Draw quiver plot. -l, --streamline Make streamline plot. -s, --scatter Plot data in scatter-plot mode. --markersize FLOAT Set marker size for scatter plots. -d, --diverging Switch to inverted color mode. -g, --group[0|1] Switch to group mode. (continues on next page)

128 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

(continued from previous page) --style TEXT Specify Matplotlib style file(default: Postgkyl). -a, --fix-aspect Enforce the same scaling on both axes. --logx Set x-axis to log scale. --logy Set y-axis to log scale. --logz Set values of 2D plot to log scale. --xscale FLOAT Value to scale the x-axis(default:1.0). --yscale FLOAT Value to scale the y-axis(default:1.0). --vmax FLOAT Set maximal value of data for plots. --vmin FLOAT Set minimal value of data for plots. --xlim TEXT Set limits for the x-coordinate(lower,upper) --ylim TEXT Set limits for the y-coordinate(lower,upper). --legend / --no-legend Show legend. --force-legend Force legend even when plotting a single dataset. --show / --no-show Turn showing of the plot ON and OFF(default: ON). --color TEXT Set color when available. -x, --xlabel TEXT Specify a x-axis label. -y, --ylabel TEXT Specify a y-axis label. -t, --title TEXT Specify a title. --save Save figure as PNG file. --saveas TEXT Name of figure file. --dpi INTEGER DPI(resolution) for output -e, --edgecolors TEXT Set color for cell edges to show grid outline (default: None) --showgrid / --no-showgrid Show grid-lines(default: True) --xkcd Turns on the xkcd style! --hashtag Turns on the pgkyl hashtag! -h, --help Show this message and exit.

Contents

• plot – Default plotting – Plotting data with multiple components – Plotting multiple datasets – Plotting modes

* Countour * Diverging * Group – Formating

* Labels * Axes and values – Storing

4.3. Function/Command reference 129 gkyl Documentation, Release 2.0-alpha

Default plotting plot automatically regnizes the dimensions of data and creates either 1D line plot or 2D pcolormesh plot using the Postgkyl style ﬁle (Inferno color map). Here is an example of 2D particle distribution function from the two-stream instability simulation.

Listing 5: Script import postgkyl as pg data= pg.Data('two-stream_elc_80.bp') dg= pg.GInterpModal(data) dg.interpolate(stack=True) pg.output.plot(data)

Listing 6: Command line pgkyl two-stream_elc_80.bp interpolate plot

Note that in this case the data does not contain the values of the distribution function directly but rather the expansion components of the basis functions. Therefore, interpolate was added to the ﬂow to show the distribution function itself.

Fig. 4: The default behavior of plot for 2D data

130 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

1D plots are created in a similar manner. For example, here is the electron density correfponding to the ﬁgure above.

Listing 7: Script import postgkyl as pg data= pg.Data('two-stream_elc_M0_80.bp') dg= pg.GInterpModal(data) dg.interpolate(stack=True) pg.output.plot(data)

Listing 8: Command line pgkyl two-stream_elc_M0_80.bp interpolate plot

Fig. 5: The default behavior of plot for 1D data

Plotting data with multiple components

Gkeyll data can contain multiple components. Typically, these are basis function expansion coefficients but can also correspond to components of a vector array like electromagnetic field or momentum. By default, Postgkyl plots each component into a separate subplot. This can be seen if we do not use the interpolation from the previous example and let Postgkyl plot the expansion coefficients.

4.3. Function/Command reference 131 gkyl Documentation, Release 2.0-alpha

Listing 9: Script import postgkyl as pg data= pg.Data('two-stream_elc_M0_80.bp') pg.output.plot(data)

Listing 10: Command line pgkyl two-stream_elc_M0_80.bp plot

Fig. 6: Plotting data with multiple components

Postgkyl automatically adds labels with component indices to each subplot. If there are some labels already (either custom or when working with multiple data sets), the component indices are appended. Postgkyl also automatically calculates the numbers of rows and columns (it tries to make a square). This can be overridden with nSubplotRow or nSubplotCol. The default behavior of putting each component to an individual subplot can be supressed with the squeeze parameter. This is useful, for example, for comparing magnitudes. Note that the magnitues of the expansion coefﬁcients are quite different so this is not the best example of the functionality.

132 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

Listing 11: Script import postgkyl as pg data= pg.Data('two-stream_elc_M0_80.bp') pg.output.plot(data, squeeze=True)

Listing 12: Command line pgkyl two-stream_elc_M0_80.bp plot --squeeze

Fig. 7: Plotting data with multiple components with squeeze=True

Plotting multiple datasets

Postgkyl in a terminal can easily load multiple ﬁles (see Data loading for more details). By default, each data set creates its own ﬁgure.

4.3. Function/Command reference 133 gkyl Documentation, Release 2.0-alpha

Listing 13: Command line pgkyl two-stream_elc_70.bp two-stream_elc_80.bp interp plot

Postgkyl automatically parses the names of the files and creates labels from the unique part of each one. Note that the labels can specified manually during Data loading. This behavior can be supressed by specifying the figure to plot in. When the same figure is specified, data sets are plotted on top of each other.

Listing 14: Script import postgkyl as pg data1= pg.Data('two-stream_elc_M0_70.bp') dg= pg.GInterpModal(data1) dg.interpolate(stack=True) data2= pg.Data('two-stream_elc_M0_80.bp') dg= pg.GInterpModal(data2) dg.interpolate(stack=True) pg.output.plot(data1, figure=0) pg.output.plot(data2, figure=0)

134 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

Listing 15: Command line pgkyl two-stream_elc_M0_70.bp two-stream_elc_M0_80.bp interp plot -f0

Fig. 8: Plotting multiple data set with specifying figure=0

Finally, the data sets can be added into subplots.

Listing 16: Command line pgkyl two-stream_elc_70.bp two-stream_elc_80.bp interp plot -f0 --subplots

The same behavior can be achieved in a script as well but it requires slightly more manual control.

Listing 17: Script import postgkyl as pg data1= pg.Data('two-stream_elc_M0_70.bp') dg= pg.GInterpModal(data1) dg.interpolate(stack=True) data2= pg.Data('two-stream_elc_M0_80.bp') dg= pg.GInterpModal(data2) dg.interpolate(stack=True) pg.output.plot(data1, figure=0, numAxes=2) (continues on next page)

4.3. Function/Command reference 135 gkyl Documentation, Release 2.0-alpha

Fig. 9: Plotting multiple data set with specifying figure=0 and subplots

136 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

(continued from previous page) pg.output.plot(data2, figure=0, numAxes=2, startAxes=1)

Plotting modes

Appart from the default line 1D plots and continuous 2D plots, Postgkyl offers some additional modes.

Countour

Listing 18: Script import postgkyl as pg data= pg.Data('two-stream_elc_80.bp') dg= pg.GInterpModal(data) dg.interpolate(stack=True) pg.output.plot(data, contour=True)

Listing 19: Command line pgkyl two-stream_elc_80.bp interpolate plot --contour

Diverging

Diverging mode is similar to the default plotting mode but the colormap is changed to a red-white-blue and the range is set to the plus-minus maximum absolute value. It is particulary useful for visualizing changes, both in time and around a mean value. Here we use the ev command to visualize the change from the initial conditions.

Group

In the group mode (maybe not the best name :-/), one direction (either 0 or 1) is retained and the other is split into individual lineouts which are then plot over each other. The lines are color-coded with the inferno colormap, i.e., from black to yellow as the coordinate increases. This could provide an additional insight into variation along one coordinate axis. In the example, the 2D distribution function is ﬁrst limited in the ﬁrst coordinate, z0 (in this case corresponding to x), from 1.5 to 2.0 using the select command (otherwise there would be too many lines). Then the plot with group=True is used.

Listing 20: Script import postgkyl as pg data= pg.Data('two-stream_elc_80.bp') dg= pg.GInterpModal(data) dg.interpolate(stack=True) pg.data.select(data, z0='1.5:2.0', stack=True) pg.output.plot(data, group=1)

4.3. Function/Command reference 137 gkyl Documentation, Release 2.0-alpha

Fig. 10: Plotting multiple data set with contour=True

138 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

Fig. 11: diverging mode is used to visualize changes from the initial conditions

4.3. Function/Command reference 139 gkyl Documentation, Release 2.0-alpha

Listing 21: Script pgkyl two-stream_elc_80.bp interpolate select --z01.5:2.0 plot --group1

Fig. 12: Plotting the distribution function limited to 1.5

Formating

While Postgkyl is not necesarily meant for the production level figures for publications, it includes a decent amount of formating options. The majority of a look of each figure, e.g., grid line style and thickness or colormap, is set in a stule file. Custom matplotlib style files can be specified with style keyword. The default Postgkyl style is the following:

figure.facecolor : white lines.linewidth :2 font.size : 12 axes.labelsize : large axes.titlesize : 14 image.interpolation : none image.cmap : inferno image.origin : lower (continues on next page)

140 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

(continued from previous page) grid.linewidth :0.5 grid.linestyle : :

Labels

Postgkyl allows to specify all the axis labels and the plot title.

Listing 22: Script import postgkyl as pg data= pg.Data('two-stream_elc_80.bp') dg= pg.GInterpModal(data) dg.interpolate(stack=True) pg.output.plot(data, xlabel=r'$x$', ylabel=r'$v_x$', title=r'$k=\frac{1}{2}$')

Listing 23: Command line pgkyl two-stream_elc_80.bp interpolate \ plot --xlabel '$x$' --ylabel '$v_x$' \ --title '$k=\frac{1}{2}$'

Axes and values

Postgkyl supports the logaritmic axes using the keywords logx and logy. In the example, the electric ﬁeld energy is plotted using the logarithmic y-axis to show the region of the linear growth of the two stream instability. Note that 2 2 2 2 2 2 Gkeyll stores 퐸푥, 퐸푦 , 퐸푧 , 퐵푥, 퐵푦 , and 퐵푧 into six components of the fieldEnergy.bp ﬁle. Therefore, the select 2 command is used to plot only the 퐸푥, which is the only component growing in this case.

Listing 24: Script import postgkyl as pg data= pg.Data('two-stream_fieldEnergy.bp') pg.data.select(data, comp=0, stack=True) pg.output.plot(data, logy=True)

Listing 25: Command line pgkyl two-stream_fieldEnergy.bp select -c0 plot --logy

Note that Postgkyl also incudes a diagnostic growth command that allows to ﬁt the data with an exponential to get the growth rate.

Storing

Plotting outputs can be save as a PNG files using the save parameter which uses the data set name(s) to put together the name of the image. Alternativelly, saveas can be used to specify the custom file name (without the extension, all the files are saved as .png). DPI of the result can be controlled with the dpi parameter.

4.3. Function/Command reference 141 gkyl Documentation, Release 2.0-alpha

Fig. 13: Postgkyl allows to specify axis labels and the ﬁgure title

142 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

Fig. 14: Plotting ﬁeld energy with logy

4.3. Function/Command reference 143 gkyl Documentation, Release 2.0-alpha pr

Print data to screen.

Command line pgkyl pr -h Usage: pgkyl pr[OPTIONS]

Print the data

Options: -u, --use TEXT Specify a 'tag' to apply to(default all tags). -h, --help Show this message and exit.

This command allows us to print a dataset to screen. Large data sets like time-traces or multidimensional data might ﬁll up the screen, so this might not be a good way to explore those. However, since postgkyl does not plot 0D data (single points), it is useful to print those to screen. For example, we could integrate the initial number density in the two stream instability simulation and print out the total number of particles to the screen with pgkyl two-stream_elc_M0_0.bp interp integrate0 pr and produce the following output:

25.132741228817842

4.3.2 Data selection collect

Assemble multiple active datasets into a new, single dataset. It is also possible to collect datasets into chunks (multiple datasets) of a speciﬁed size rather than into a single dataset.

Command line pgkyl collect --help Usage: pgkyl collect[OPTIONS]

Collect data from the active datasets and create a new combined dataset. The time-stamp in each of the active datasets is collected and used as the new X-axis. Data can be collected in chunks, in which case several datasets are created, each with the chunk-sized pieces collected into each new dataset.

Options: -s, --sumdata Sum data in the collected datasets(retain components) -p, --period FLOAT Specify a period to create epoch data instead of time data

(continues on next page)

144 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

(continued from previous page) --offset FLOAT Specify an offset to create epoch data instead of time data[default:0.0]

-c, --chunk INTEGER Collect into chunks with specified length rather than into a single dataset

-u, --use TEXT Specify a 'tag' to apply to(default all tags). -t, --tag TEXT Specify a 'tag' for the result. -l, --label TEXT Specify the custom label for the result. -h, --help Show this message and exit.

There are many uses of the collect command. One such use, for example, is to plot quantities over time. If we consider the two stream instability Vlasov-Maxwell simulation we can gather the 푥-component of the electric ﬁeld from all frames and plot it as a function of time and space with

pgkyl "two-stream_field_[0-9]*.bp" interp sel -c0 collect pl -x 'time' -y 'x'

The field is very small at first but is exponentially growing, so in the above figure we mostly see the saturated electric field at later times. The collect command is also able to collect datasets into chunks of a specified length with the -c flag. So suppose we wish to compute a time average of the electric field over three separate periods in the two stream instability simulation (we will set nFrame=98 in the gkyl input file for this example so there are a total of 99 frames), we can collect the field data into three separate datasets with 33 frames each and use the ev command to average in time (0th dimension)

4.3. Function/Command reference 145 gkyl Documentation, Release 2.0-alpha

as follows:

pgkyl "two-stream_field_[0-9]*.bp" interp sel -c0 collect -c 33 ev -l 'frames 0-32' ˓→'f[99] 0 avg' \ ev -l 'frames 33-65' 'f[100] 0 avg' ev -l 'frames 66-98' 'f[101] 0 avg' activ -

˓→i102:105 \ pl -x '$x$' -y '$\langle E_x\rangle_t$'

producing the following three plots

Clearly the ﬁeld amplitude averaged over 30 frame intervals is increasing as the instability develops. Finally, collect allows a transformation of the time dimension so that instead of time t it instead becomes (t-offset)/period via the --offset and --period flags. This is used, for example, in astronomy for variable stars and in creating Poincare plots (see http://ammar-hakim.org/sj/je/je32/ je32-vlasov-test-ptcl.html). select

Select a subset of one or multiple datasets. This subset can can be created by • Choosing a cell along a direction (--z# with an integer). • Choosing a range of cells along a direction (--z# with a slice). • Choosing a specific component (-c), which may be an expansion coefficient, fluid moment or, if used along with interpolate, a vector component. Multiple components can be selected with a slice of floats or with comma- separated values. • Evaluating the function at a specific coordinate in one direction (using interpolate and select --z# with a float) or segment in one dimension (using interpolate and select --z# with a slize of floats).

Note: Postgkyl retains the number of dimensions and component index. This means that, for example, given a 16x16 2D dataset with 8 components per cell ﬁxing the second coordinate and selecting one component will produce a dataset with a shape (16, 1, 1). For plotting purposes postgkyl treats such data as 1D.

Command line

pgkyl select --help Usage: pgkyl select [OPTIONS]

Subselect data from the active dataset(s). This command allows, for (continues on next page)

146 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

(continued from previous page) example, to choose a specific component of a multi-component dataset, select a index or coordinate range. Index ranges can also be specified using python slice notation(start:end:stride).

Options: --z0 TEXT Indices for 0th coord(either int, float, or slice) --z1 TEXT Indices for 1st coord(either int, float, or slice) --z2 TEXT Indices for 2nd coord(either int, float, or slice) --z3 TEXT Indices for 3rd coord(either int, float, or slice) --z4 TEXT Indices for 4th coord(either int, float, or slice) --z5 TEXT Indices for 5th coord(either int, float, or slice) -c, --comp TEXT Indices for components(either int, slice, or coma- separated)

-u, --use TEXT Specify a 'tag' to apply to(default all tags). -t, --tag TEXT Optional tag for the resulting array -h, --help Show this message and exit.

Perhaps the two most common uses of select are to choose a vector component or to evaluate a function at a specific coordinate. Consider the data produced by a two stream instability Vlasov-Maxwell simulation. The file two-stream_field_0.bp contains 24 components per cell. That is because it contains 8 vector components (3 components of the electric and magnetic fields each and 2 correction potentials, see http://ammar-hakim.org/sj/ maxwell-eigensystem.html) and 3 degrees of freedom per component (for piecewise linear basis in 1D). If we wish to only see 퐸푥 then we can select it and plot it with pgkyl two-stream_field_0.bp interp select -c0 pl which yields the following figure We could also select multiple components with a comma-separated list pgkyl two-stream_field_0.bp interp select -c0,3 pl -x '$x$' which yields the following plot of 퐸푥 and 퐵푥 Or we could select all the components of the magnetic field in all frames with pgkyl "two-stream_field_[0-9]*.bp" interp select -c3:6 pl -f0 -x '$x$' --no-legend -- ˓→nsubplotrow1 to show that this is an electrostatic simulation.

As a demonstration of using select to evaluate functions at a given coordinate we could evaluate the above 퐸푥 at 푥 = −휋, but that would yield a single number. We could instead load all the field data files, evaluate the 푥-component of the electric field at 푥 = −휋 in each frame, collect all those points into a single dataset, and plot them as a function of time. This would be accomplished with the following command pgkyl "two-stream_field_[0-9]*.bp" interp sel -c0 --z0 -3.14159 collect pl -x 'time'- ˓→y '$E_x(x=-\pi,t)$'

We will sometimes use the abbreviation sel instead of select for convenience. The resulting figure (above) shows how the amplitude of the electric field at this point drastically increases as the instability develops. There is an oscillatory behavior that is lost in this exponential growth; if one zoomed into the earlier times of this plot, such oscillations would become visible. One can, and must, select coordinates at which to evaluate higher (than 2) dimensional datasets. Take the initial and final distribution functions of the electrons as an example; we can evaluate them at 푥 = 0 and plot their variation along 푣푥 with

4.3. Function/Command reference 147 gkyl Documentation, Release 2.0-alpha

148 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

Fig. 15: x-component of the electric and magnetic ﬁelds.

4.3. Function/Command reference 149 gkyl Documentation, Release 2.0-alpha

Fig. 16: Components of the magnetic ﬁeld.

150 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

4.3. Function/Command reference 151 gkyl Documentation, Release 2.0-alpha

pgkyl two-stream_elc_0.bp -l '$t=0$' two-stream_elc_100.bp -l '$t=50$' interp \ sel --z00. pl -x '$v_x$' -y '$f_e(x=0,v_x)$' -f0 producing the following plot:

If we were interested in investigating the distribution function in a speciﬁc cell along 푥, say the ﬁrst (0th in 0-index), we could use pgkyl two-stream_elc_100.bp sel --z00 interp pl -x '$x$' -y '$v_x$'

Notice that this produces a 2D colorplot because it takes the expansion coefficients in the 0th cell along 푥 and all cells along 푣푥, interpolates them onto a finer mesh and plots them (so the 푥 extent of this plot is a single cell of the simulation). If we instead wanted a 1D plot of the distribution function along 푣푥, we could first interpolate onto a finer mesh and then evaluate it at the 0th cell of the finer mesh using pgkyl two-stream_elc_100.bp interp sel --z00 pl -x '$x$' -y '$f_e(x=-6.25046,v_x)$' or interpolate it and evaluate it at the lower boundary of the domain along 푥, which is located at 푥 = −2휋, with pgkyl two-stream_elc_100.bp interp sel --z0 -6.283185 pl -x '$x$' -y '$f_e(x=-2\pi,v_

˓→x)$'

These two commands are evaluating the distribution function at slightly different 푥 coordinates (∆푥/(푝 + 1)/2 apart to be precise, where ∆푥 is the cell length of the simulation, and 푝 the polynomial order of the basis). We can discern the difference between the two by plotting them together using the following command:

152 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

4.3. Function/Command reference 153 gkyl Documentation, Release 2.0-alpha

pgkyl two-stream_elc_100.bp -t fe interp sel -t zfl --z0 -6.283185 sel -u fe -t zint -

˓→-z00 \ pl -u zfl,zint -f0 -x '$v_x$' -y '$f_e$'

This commmand used tags to indicate which dataset to perform the interpolation on, and to name the interpolated datasets. The result is

Lastly, we show that the select command can also be used to restrict and interpolated dataset to a segement along one direction when the --z# ﬂag is used with a slice of ﬂoats. For example, if one wants to plot the initial electron distribution function at 푥 = 0 for positive velocities only, then one could employ pgkyl two-stream_elc_0.bp interp sel --z00. --z10.: pl -x '$x$' -y '$f_e(x=0,v_x,

˓→t=0)$'

154 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

4.3. Function/Command reference 155 gkyl Documentation, Release 2.0-alpha

Script mode

Table 2: Parameters for select Parameter Description Default data (GData) Data to subselect. coord0 (int, float, or Index corresponding to the first coordinate for the partial load. Ei- None slice) ther integer, float, or Python slice (e.g., ‘2:5’). coord1 (int, float, or Index corresponding to the second coordinate for the partial load. None slice) Either integer, float, or Python slice (e.g., ‘2:5’). coord2 (int, float, or Index corresponding to the third coordinate for the partial load. Ei- None slice) ther integer, float, or Python slice (e.g., ‘2:5’). coord3 (int, float, or Index corresponding to the fourth coordinate for the partial load. None slice) Either integer, float, or Python slice (e.g., ‘2:5’). coord4 (int, float, or Index corresponding to the fifth coordinate for the partial load. Ei- None slice) ther integer, float, or Python slice (e.g., ‘2:5’). coord5 (int, float, or Index corresponding to the sixth coordinate for the partial load. Ei- None slice) ther integer, float, or Python slice (e.g., ‘2:5’). comp (int, slice, or Index corresponding to the component for the partial load. Either None multiple) integer, Python slice (e.g., ‘2:5’), or multiple.

Unlike for the partial load parameters (see :ref:), float point numbers can be specified instead of just integers. In that case, Postgkyl treats it as a grid value and automatically finds and index of a grid point with the closest value. This works both for the single index and for specifying a slice.

4.3.3 Data manipulation differentiate

Compute the derivatives of a function along each direction. This returns a dataset with as many components as there are dimensions in the original dataset, i.e. it returns the gradient, unless the -d ﬂag is used to specify a direction. This command takes the basis expansion in a cell, differentiates it analytically in the desired directions, and then interpolates it onto a ﬁner mesh in the same fashion as the interpolate command. Note that differentiation is also possible with the ev command and the grad operation.

Command line pgkyl differentiate -h Usage: pgkyl differentiate[OPTIONS]

Interpolate a derivative of DG data on a uniform mesh

Options: -b, --basistype[ms|ns|mo] Specify DG basis -p, --polyorder INTEGER Specify polynomial order -i, --interp INTEGER Interpolation onto a general mesh of specified amount

-d, --direction INTEGER Direction of the derivative(default: calculate all) (continues on next page)

156 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

(continued from previous page)

-r, --read BOOLEAN Read from general interpolation file -t, --tag TEXT Specify a 'tag' to apply to(default all tags). -o, --outtag TEXT Optional tag for the resulting array -h, --help Show this message and exit.

Let’s take a gyrokinetic ion acoustic wave simulation as an example. We can examine the initial electrostatic potential generated by the initial conditions with pgkyl gk-ionSound-1x2v-p1_phi_0.bp interp pl -x '$x$' -y '$\phi$' giving the plot shown below on the left.

Suppose we wished to know what the initial electric ﬁeld is, then we would differentiate the potential and multiply it by -1 as follows pgkyl gk-ionSound-1x2v-p1_phi_0.bp diff -d0 ev 'f[1] -1 *' pl -x '$x$' -y '$- ˓→\partial_x\phi$'

Note we we have abbreviated differentiate -> diff, either is allowed. This command produces the electric ﬁeld above on the right. It is cellwise constant because we use a piecewise linear basis function. Now suppose you wish to examine the gradient of the ion distribution function at 푡 = 0 and 푥 = 0. This can be accomplished with the following command pgkyl gk-ionSound-1x2v-p1_ion_0.bp diff sel --z00. pl -x '$v_\parallel$' -y '$\mu$'

4.3. Function/Command reference 157 gkyl Documentation, Release 2.0-alpha

in order to produce:

Starting with the top left and going clockwise, this plot provides the gradient in 푓푖(푥, 푣‖, 휇) along 푥, 푣‖ and 휇, all three evaluated at 푥 = 0. ev ev is a special command that enables simple operations like adding or multiplying datasets directly in a terminal. As these operations are generally simple to do in a script mode, ev is available only in the command line interface.

Contents

• ev – Reverse Polish notation – Using ev on simple datasets – Using ev on datasets with tags – Examples of speciﬁc ev operations

* grad * int

158 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

* avg

Table 3: ev supported operations Operator Description + Sum datasets. - Subtract the second dataset from the ﬁrst. * Multiply datasets. / Divide ﬁrst dataset by the second. sqrt Square root of a dataset. sin Compute 푠푖푛(푓) where 푓 is a dataset. cos Compute 푐표푠(푓) where 푓 is a dataset. tan Compute 푡푎푛(푓) where 푓 is a dataset. abs Absolute value of a dataset avg Average of a dataset along one or more directions. log Calculate the natural log of a dataset. log10 Calculate the base-10 log of a dataset. max Calculate the maximum of a dataset (returns a single number) min Calculate the minimum of a dataset (returns a single number) mean Calculate the mean of a dataset (returns a single number) len pow Raise a dataset to a given power. sq Square a dataset. exp Compute 푒푓 where 푓 is a dataset. grad Compute the derivative of a dataset along a given direction. int Integrate a dataset along one or more directions. div curl

Reverse Polish notation

ev uses reverse Polish notation (RPN; also know as postﬁx notation) to input operations. Unlike in the more common inﬁx notation, in RPN, the operators follow operands. For example 4 - 2 is written in RPN as 4 2 -; 4 - (2 / 5) is 4 2 5 / -. This can be further demonstrated on the sequence that processes this expression. 1. 4 >>> 4 (Add 4 to the stack) 2. 4 2 >>> 4 2 (Add 2 to the stack) 3. 4 2 5 >>> 4 2 5 (Add 5 to the stack) 4. 4 2 5 / >>> 4 0.4 (Divide the last two elements with each other) 5. 4 2 5 / - >>> 3.6 (Subtract the last two elements from each other) More information about RPN can be found, for example, on Wikipedia.

Using ev on simple datasets

The ev command has one required argument, which is the operation sequence. Datasets are specified in the sequence with f[set_index][component]. Note that Python indexing conventions are used. Specifically, when no indices are specified, everything is used, i.e., f[0][:] and f[0] are treated as identical calls. The indices need to be integers unless a global indexing mode -g is used, which ignores active and inactive sets. Then, Python slice syntax

4.3. Function/Command reference 159 gkyl Documentation, Release 2.0-alpha

can be used as well including negative indices and strides. For example, f[2:-1:2] will select every other dataset, starting with the third one (zero-indexed), and ending with one before the last (by Python conventions, the upper bound is not included; in this example it might end with the third element from the end because of the stride).

Note: Dataset selection in ev internally uses the same code as the pg_cmd_activate/pg_cmd_deactivate commands, so the following commands produce similar results (ev actually copies the datasets instead of just activating some).

pgkyl two-stream_elc_?.bp activate -i '2:-1:2' pgkyl two-stream_elc_?.bp ev -g 'f[2:-1:2]'

However, this is probably a fringe application of ev.

The simplest example of ev is a numerical operation performed on a dataset, e.g., dividing the values by the insidious factor of 2:

pgkyl two-stream_elc_0.bp ev 'f[0] 2 /'

This can be also combined with the fact that ev can access dataset metadata as long as they are included (which is a new feature in Gkeyll introduced in January 2021). An example of this can be plotting number density from a ﬂuid simulation (Gkeyll outputs mass density).

pgkyl 5m_fluid_elc_0.bp ev 'f[0][0] f[0].mass /' plot

Note that on top of dividing by mass, only the ﬁrst component, which corresponds to density, was selected. This can be easily extended to apply on multiple datasets and create an animation using the animate command

pgkyl '5m_fluid_elc_[0-9]*.bp' ev -g 'f[:][0] f[:].mass /' animate

The capabilities are not limited to operations with ﬂoat factors. As an example, ev can be used to visualize differences (--diverging mode of the plot command is well suited for this)

pgkyl two-stream_elc_0.bp two-stream_elc_80.bp interpolate ev 'f[1] f[0] -' plot --

˓→diverging

Note: info command, especially with the --compact -c ﬂag can be useful to print indices for available datasets.

The same concept can be used to calculate bulk velocity from the ﬁrst two moments:

pgkyl two-stream_elc_M0_0.bp two-stream_elc_M1i_0.bp interpolate ev 'f[1] f[0] /' plot

Finally, it is worth noting that this syntax cannot be used when there are datasets with more than one tag active.

Using ev on datasets with tags

The ev command is tag-aware. Tagged datasets use the following notation t. tag_name[set_index][component]. Using this, the previous example can be reproduced:

pgkyl two-stream_elc_M0_0.bp -t dens two-stream_elc_M1i_0.bp -t mom interp ev 'mom[0]

˓→dens[0] /' plot

However, unlike the previous example, this can be naturally extended for batch loading and animate:

160 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

Fig. 17: Visualizing difference between two datasets

4.3. Function/Command reference 161 gkyl Documentation, Release 2.0-alpha

pgkyl 'two-stream_elc_M0_[0-9].bp' -t dens 'two-stream_elc_M1i_[0-9]*.bp' -t mom ˓→interp ev -g 'mom dens /' animate

Examples of speciﬁc ev operations

In this section we provide examples of some ev operations that are less trivial or intuitive.

grad

This operation differentiates a along a direction given by the second operand. So, for example, given the data from an ion sound wave gyrokinetic simulation we can plot the initial electrostatic potential with

pgkyl gk-ionSound-1x2v-p1_phi_0.bp interp pl -x '$x$' -y '$\phi$'

and compute the parallel electric ﬁeld by differentiating the potential along 푥 as follows:

pgkyl gk-ionSound-1x2v-p1_phi_0.bp interp ev 'f[0] 0 grad -1 *' pl -x '$x$' -y '$\phi$ ˓→'

These produce the following plots:

int

Integrate a dataset along a direction speciﬁed by the second operand, or along multiple directions speciﬁed by a comma- separated list. If we once again take the ion sound wave gyrokinetic simulation data, we can examine the number of particles in the simulation (should be conserved) by taking the time trace of the integrated ion number density (intM0) and taking its mean:

pgkyl gk-ionSound-

˓→1x2v-p1_ion_intM0.bp ev 'f[0] mean' pr

which prints out

12.566370614358858

If we instead use ev to integrate the initial and/or the ﬁnal number density GkM0, we should get roughly the same an- swer. We can check that this is the case by typing

pgkyl gk-ionSound-1x2v-

˓→p1_ion_GkM0_10.bp interp ev 'f[0] 0 int' pr

which produces

12.566370614358522

and we have shown that the number of particles at the end is roughly the same as the mean number of particles throughout the simulation.

162 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

avg

Average a dataset along a direction speciﬁed by the second operand, or along multiple directions speciﬁed by a comma-separated list.

integrate

Integrate a dataset over speciﬁed direction(s).

Command line

$ pgkyl integrate --help Usage: pgkyl integr[OPTIONS] AXIS

Integrate data over a specified axis or axes

Options: -u, --use TEXT Specify the tag to integrate -t, --tag TEXT Optional tag for the resulting array -h, --help Show this message and exit.

Consider the gyrokinetic simulation of an ion acoustic wave as an example. It outputs the integrated particle density over time, which can plot as follows:

pgkyl gk-ionSound-1x2v-p1_ion_intM0.bp pl

We can see from the values on the y-axis that the total number of particles is 12.566. The number of particles should be conserved, to machine precision. We can check this another way by integrating the particle density along 푥 (the 0th dimension) at the end of the simulation with pgkyl:

pgkyl gk-ionSound-1x2v-p1_ion_GkM0_1.bp interp integr0 print

where we have abbreviated integrate with integr, and we use the print command to print the result of the integral to screen. The output of this command is simply

12.566370614359034

The integrate command can also be used to integrate higher dimensional datasets in one or more directions. We could take the ion distribution function and integrated along the 푣‖ and 휇 directions (1st and 2nd dimensions, respectively) with

pgkyl gk-ionSound-

˓→1x2v-p1_ion_0.bp gk-ionSound-1x2v-p1_ion_GkM0_0.bp interp \ activate ˓→-i0 integr1,2 ev -l 'integrate 1,2' 'f[0] 6.283185 *' \ activate -i1,2 pl -f0 -x 'x' -y 'Number density, $n$'

Note: In this command we: • First load the ion distribution function (*_ion_0.bp) and its number density (*_ion_GkM0_0.bp) at 푡 = 0.

4.3. Function/Command reference 163 gkyl Documentation, Release 2.0-alpha

164 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

• Integrate the distribution function over velocity space with activate -i 0 integr 1,2.

• Multiply such integral by 2휋퐵0/푚푖 (퐵0 = 푚푖 = 1 here) with ev -l 'integrate 1,2' 'f[0] 6.283185 *'. • Activate the number denstiy and integrated distribution function data sets and plot them with activate -i1,2 pl -f0.

and this should give approximately the same number density as the GkM0 diagnostic outputted by the simulation, as shown below.

Another useful application of the integrate command is to integrate, or average, over time (although note that the ev command has a avg operation that may make this easier). Usually this requires collecting multiple frames into a single dataset with the collect command, and then integrating over the 0th dimension (time). So if we increase the tEnd of the gyrokinetic ion sound wave simulation to 10 and the number of frames to 50 we could plot the electrostatic potential as a function of time and position with

pgkyl "gk-ionSound-1x2v-p1_phi_[0- ˓→9]*.bp" interp collect pl -x 'time' -y 'x' --clabel '$\phi$'

We can integrate this potential in time and plot it on top of the initial potential with

4.3. Function/Command reference 165 gkyl Documentation, Release 2.0-alpha

166 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

pgkyl gk-ionSound-1x2v-p1_phi_0.bp -l '$t=0$' -t phi0 \ "gk-ionSound-1x2v- ˓→p1_phi_[0-9]*.bp" -t phis interp collect -u phis -t phiC \ integrate -u phiC

˓→-t phiInt0 ev -l 'Time average' -t phiAvg 'phiInt 10. /' \ activate -t phi0,phiAvg pl -f0 -x '$x$' -y '$\phi$'

This command uses tags to select which dataset to perform an operation on. The end result is the plot below

showing that the time averaged potential is lower amplitude due to the collisionless Landau damping of the wave.

interpolate

Gkeyll’s simulation may employ higher-order methods with more than one degree of freedom per cell, typically expansion coefﬁcients or nodal values. The interpolate command interpolates this higher-order representation onto a ﬁner mesh with more points than the number of cells in the Gkeyll simulation.

4.3. Function/Command reference 167 gkyl Documentation, Release 2.0-alpha

Command line pgkyl interpolate --help Usage: pgkyl interp[OPTIONS]

Interpolate DG data onto a uniform mesh.

Options: -b, --basistype[ms|ns|mo|mt] Specify DG basis. -p, --polyorder INTEGER Specify polynomial order. -i, -

˓→-interp INTEGER Interpolation onto a general mesh of specified amount.

-u, --use

˓→TEXT Specify a 'tag' to apply to(default all tags).

-t, -

˓→-tag TEXT Optional tag for the resulting array -r, --

˓→read BOOLEAN Read from general interpolation file. -n, --new for testing purposes -h, --help Show this message and exit.

A Gkeyll simulation can have several degrees of freedom per cell. If we simply invoke the plot command we would then obtain a plot of all the coefficients in each cell. So for example, take the electron distribution function at the end of a two stream instability simulation and plot it with pgkyl two-stream_elc_100.bp pl -x '$x$' -y '$v_x$' we would obtain figure below on the left, with all 8 DG expansion coefficients plotted in phase space. Most commonly we are interested in plotting the actual function that the expansion coefficients represent, rather than plotting such coefficients. We can do that by interpolating onto a finer mesh with the interpolate command: pgkyl two-stream_elc_100.bp interp pl -x '$x$' -y '$v_x$' which results in the figure below on the right.

Fig. 18: DG coefﬁcients.

168 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

We can compare the cell average of the electron distribution function with the interpolated distribution function with the following command pgkyl two-stream_elc_100.bp -t fe interp

˓→-u fe -t fInterp sel -u fe -c0 -t c0 \ ev -l 'cell average' -t cellAv

˓→'c0 2 /' activ -t fInterp,cellAv pl -b which divides the zeroth DG coefﬁcient by 2 in order to obtain the cell average (1r 2D piecewise quadratic basis), and produces the following ﬁgure

Fig. 19: Interpolated distribution function.

Notice how the cell average (right) is naturally coarser grained, and the interpolated function (left) offers a smoother plot. By default the interpolate command interpolates onto a uniform mesh that subdivides each cell in the simulation into 푝 + 1 cells in each direction, where 푝 is the polynomial order of the simulation. It is also possible to interpolate onto finer meshes with the -i flag in order to obtain smoother plots. However note that interpolating onto finer meshes can also augment local maxima and/or minima. Below we compare the final electron distribution function interpolated onto a mesh with 3 subcells per cell in each direction (default for 푝 = 2)

4.3. Function/Command reference 169 gkyl Documentation, Release 2.0-alpha

and interpolating onto a mesh with 8 subcells per cell in each direction:

pgkyl two-stream_elc_100.bp -t fe interp -t i3 interp -i8 -u fe -t i8 activ -t i3,i8

˓→pl -b -x '$x$' -y '$v_x$'

This example also shows the use of tags in order to tag datsets and to instruct interpolate which datasets to operate on (via the -u/--use flag). In order to request that interpolate operates on a given tagged dataset, one must pass -u followed by the dataset we wish to interpolate. And in order to create a new dataset outof the interpolated data one must use the -t flag followed by the name (tag) of the new dataset. In the above example the first interpolate operates on the input data (no -u necessary because it immediately precedes it and there is only one dataset at that point in the chain) and creates a dataset tagged i3. The second interpolate operates on the input data (-u fe) and creates a dataset tagged i8.

Script mode interpolate uses the GInterpModal and GInterpNodal classes based on the DG mode.

170 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

Table 4: Initialization parameters for GInterpModal and GInterpNodal Parameter Description Default gdata (GData) A GData object to be used. polyOrder (int) The polynomial order of the discontinuous Galerkin discretization. basis (str) The polynomial basis. Currently supported options are 'ns' for nodal Serendipity, 'ms' for modal Serendipity, and 'mo' for the maximal order basis.

After the initialization, both GInterpModal and GInterpNodal can be used to interpolate data on a uniform grid and to calculate derivatives

Table 5: Members of GInterpModal and GInterpNodal Member Description interpolate(int component, bool stack) -> Interpolates the selected component (default is 0) of the DG data on narray, narray a uniform grid derivative(int component, bool stack) -> Calculates the derivative of the DG data narray, narray

When the stack parameter is set to true (it is false by default), the grid and values are pushed to the GData stack rather than returned. An example of the usage:

import postgkyl as pg data= pg.data.GData('bgk_neut_0.bp') interp= pg.data.GInterpModal(data,2,'ms') iGrid, iValues= interp.interpolate()

4.3.4 Diagnostics

growth

Measure the growth rate in the time trace of a quantity, assuming that it is a squared quantity (e.g. 휑2, |E|2). If one applies this to a non-squared quantity then the growth-rate would be off by a factor of two. The command works by ﬁtting 퐴푒(2훾푡) to the data and returning 훾.

Command line

$ pgkyl growth --help Usage: pgkyl growth[OPTIONS]

Attempts to compute growth rate(i.e. fit e^(2x)) from DynVector data, typically an integrated quantity like electric or magnetic field energy.

Options: -u, --use TEXT Specify a 'tag' to apply to(default all tags).

-g, --guess ... Specify initial guess (continues on next page)

4.3. Function/Command reference 171 gkyl Documentation, Release 2.0-alpha

(continued from previous page) -p, --plot Plot the data and fit --minn INTEGER Set minimal number of points to fit --maxn INTEGER Set maximal number of points to fit -i, --instantaneous Plot instantaneous growth rate vs time -h, --help Show this message and exit.

The two tream instability simulation produces a ﬁeld energy time trace that can be plotted with pgkyl two-stream_fieldEnergy.bp sel -c0 pl --logy

We have selected the 퐸푥 component because all other ﬁeld components are essentially zero. We can then measure the growth rate in the ﬁeld energy with the following command pgkyl two-stream_fieldEnergy.bp sel -c0 growth

Such command produces the output below fitGrowth: fitting region 100 -> 7916 gamma= +3.79836e-01(current +5.814e-02 R^2=7.407e-01) 99.99% done [======] gamma= +3.79836e-01

This output indicates that it made the measurement taking into account all the data between the 100th and the 7916th point (if you examine the file two-stram_fieldEnergy.bp you’d find that it has 7916 data points), and arrived at the growth rate 훾 = 0.379836, the the 푅2 of the fit being 0.7407.

172 Chapter 4. Postgkyl reference gkyl Documentation, Release 2.0-alpha

The growth command also allows for specifying a window in which to perform the measurement via the --minn and --maxn flags. We could then limit the window between the 2000th and the 6000th point with pgkyl two-stream_fieldEnergy.bp sel -c0 growth --minn 2000 --maxn 6000 and the output would be fitGrowth: fitting region 2000 -> 6000 gamma= +3.79836e-01(current +4.322e-01 R^2=9.998e-01) 99.98% done [======] gamma= +3.79836e-01 giving the same result obtained above. There is also an option for specifying a guess to 퐴 and 훾 in the fit, via the -g flag: pgkyl two-stream_fieldEnergy.bp sel -c0 growth --minn 2000 --maxn 6000 -g1.0.36 fitGrowth: fitting region 2000 -> 6000 gamma= +3.79836e-01(current +4.322e-01 R^2=9.998e-01) 99.98% done [======] gamma= +3.79836e-01

4.3.5 Command control

4.3. Function/Command reference 173 gkyl Documentation, Release 2.0-alpha

174 Chapter 4. Postgkyl reference CHAPTER 5

Publications and theses

A good source of various benchmarks and other tests is A. Hakim’s Simulation Journal and its github webpage. We have also started compiling input ﬁles for the simulations reported in publications in this repository.

5.1 Doctoral Dissertations

• Mandell, N. R. (2021, March 26) “Magnetic Fluctuations in Gyrokinetic Simulations of Scrape-Off Layer Tur- bulence”. Ph.D. dissertation, Princeton University, 2021. arXiv:2103.16062 • Juno, J. (2020, March 27) “A Deep Dive into the Distribution Function: Understanding Phase Space Dynamics Using Continuum Vlasov-Maxwell Simulations”. Ph. D. dissertation, University of Maryland, College Park, 2020. arXiv:2005.13539 • Bernard, T. N. “Discontinuous Galerkin Modeling of Plasma Turbulence in a Simple Magnetized Torus”. Ph. D. dissertation, The University of Texas at Austin, 2019. PDF • Ng, J. “Fluid closures for the modeling of reconnection and instabilities in magnetotail current sheets”. Ph.D. dissertation, Princeton University, 2018. PDF • Cagas, P. (2018, July 30). “Continuum kinetic simulations of plasma sheaths and instabilities”. Ph.D. dissertation, Virginia Polytechnic Institute and State University, 2018. https://vtechworks.lib.vt.edu/handle/10919/ 84979 • Shi, E. L. (2017, August 24). “Gyrokinetic Continuum Simulation of Turbulence in Open-Field-Line Plasmas”, Ph.D. dissertation, Princeton University, 2017 arXiv:1708.07283

5.2 Algorithms papers

• Cagas, P and Hakim, A and Srinivasan, B. (2021) “A boundary value “reservoir problem” and boundary conditions for multi-moment multiﬂuid simulations of sheaths.” Physics of Plasmas 28.1. https://doi.org/10.1063/5. 0024510

175 gkyl Documentation, Release 2.0-alpha

• Hakim, A and Juno, J. “Alias-free, Matrix-free, and Quadrature-free Discontinuous Galerkin Algorithms for (Plasma) Kinetic Equations”. Accepted to Supercomputing 2020 arXiv:2004.09019 • Francisquez, M., Bernard, T. N., Mandell, N. R., Hammett, G. W., Hakim, A. (2020). “Conservative discontinuous Galerkin scheme of a gyro-averaged Dougherty collision operator”, Nuclear Fusion, 60, (9). https://doi.org/10.1088%2F1741-4326%2Faba0c9 • Hakim, A., Francisquez, M., Juno, J., & Hammett, G. W. (2020). “Conservative discontinuous Galerkin schemes for nonlinear Dougherty–Fokker–Planck collision operators”, Journal of Plasma Physics, 86, (4). https://doi. org/10.1017/S0022377820000586 • Wang, L., Hakim, A., Ng, J., Dong, C., & Germaschewski, K. (2020). “Exact and locally implicit source term solvers for multiﬂuid-Maxwell systems”, Journal of Computational Physics, 415, 109510. https://doi.org/10. 1016/j.jcp.2020.109510 • Cagas, P., Hakim, A., & Srinivasan, B. (2020). “Plasma-material boundary conditions for discontinuous Galerkin continuum-kinetic simulations, with a focus on secondary electron emission”, Journal of Computa- tional Physics, 406, 109215. https://doi.org/10.1016/j.jcp.2019.109215 • Mandell, N. R., Hakim, A., Hammett, G. W., & Francisquez, M. (2020). “Electromagnetic full-f gyrokinetics in the tokamak edge with discontinuous Galerkin methods”, Journal of Plasma Physics, 86. https://doi.org/10. 1017/S0022377820000070 • Juno, J., Hakim, A., TenBarge, J., Shi, E., & Dorland, W. (2018). “Discontinuous Galerkin algorithms for fully kinetic plasmas”, Journal of Computational Physics, 353, 110–147. https://doi.org/10.1016/j.jcp.2017.10.009 • Hakim, A., Hammett, G. W., Shi, E. L. (2014). “On discontinuous Galerkin discretizations of second-order derivatives”, arXiv:1405.5907

5.3 Physics papers

• Ng, J., Hakim, A., Wang, L., & Bhattacharjee, A. (2020). “An improved ten-moment closure for reconnection and instabilities”, Physics of Plasmas, 27, 082106. https://doi.org/10.1063/5.0012067 • Juno, J., Swisdak, M. M., TenBarge. J. M., Skoutnev, V., & Hakim, A. (2020). “Noise-induced magnetic field saturation in kinetic simulations”, Journal of Plasma Physics, 86, (4). https://doi.org/10.1017/ S0022377820000707 • Ng, J., Chen, L.-J., Hakim, A., & Bhattacharjee, A. (2020). “Reconstruction of electron and ion distribution functions in a magnetotail reconnection diffusion region”, Journal of Geophysical Research: Space Physics, 125, e2020JA027879. https://doi.org/10.1029/2020JA027879 • Francisquez, M., Bernard, T. N., Zhu, B., Hakim, A., Rogers, B. N., & Hammett, G. W. (2020). “Fluid and gyrokinetic turbulence in open field-line, helical plasmas”, Physics of Plasmas, 27, 082301. https://doi.org/10. 1063/5.0005333 • Bernard, T. N., Stoltzfus-Dueck, T., Gentle, K. W., Hakim, A., Hammett, G. W., & Shi, E. L. (2020). “Investi- gating shear flow through continuum gyrokinetic simulations of limiter biasing in the Texas Helimak”, Physics of Plasmas, 27, 062304. https://doi.org/10.1063/5.0003904 • Hakim, A. H., Mandell, N. R., Bernard, T. N., Francisquez, M., Hammett, G. W., & Shi, E. L. “Continuum electromagnetic gyrokinetic simulations of turbulence in the tokamak scrape-off layer and laboratory devices”, Physics of Plasmas, 27, 042304. https://doi.org/10.1063/1.5141157 • Pusztai, I., Juno, J., Brandenburg, A., Tenbarge, J. M., Hakim, A., Francisquez, M., & Sundström, A. (2020). “Dynamo in Weakly Collisional Nonmagnetized Plasmas Impeded by Landau Damping of Magnetic Fields”, Physical Review Letters, 124, 255102. https://link.aps.org/doi/10.1103/PhysRevLett.124.255102

176 Chapter 5. Publications and theses gkyl Documentation, Release 2.0-alpha

• TenBarge, J. M., Ng, J., Juno, J., Wang, L., Hakim, A. & Bhattacharjee, A. (2019). “An extended MHD study of the 16 October 2015 MMS diffusion region crossing”, Journal of Geophysical Research: Space Physics, 124, 8474-8487. https://doi.org/10.1029/2019JA026731 • Ng, J., Hakim, A., Juno, J., & Bhattacharjee, A. (2019). Drift instabilities in thin current sheets using a two- fluid model with pressure tensor effects. Journal of Geophysical Research: Space Physics, 124, 3331-3346. https://doi.org/10.1029/2018JA026313 • Dong, C., Wang, L., Hakim, A., Bhattacharjee, A., Slavin, J. A., DiBraccio, G. A., & Germaschewski, K. (2019). “A Novel Ten-Moment Multifluid Model for Mercury: From the Planetary Conducting Core to the Dynamic Magnetosphere”, Geophysical Review Letters, 46, 11584-11596. https://doi.org/10.1029/2019GL083180 • Shi, E. L., Hammett, G. W., Stoltzfus-Dueck, T., & Hakim, A. (2019). “Full-f gyrokinetic simulation of turbulence in a helical open-field-line plasma”, Physics of Plasmas, 26, 012307. https://doi.org/10.1063/1.5074179 • Bernard, T. N., Shi, E. L., Gentle, K. W., Hakim, A., Hammett, G. W., Stoltzfus-Dueck, T., & Taylor, E. I. (2019). “Gyrokinetic continuum simulations of plasma turbulence in the Texas Helimak”, Physics of Plasmas, 26, 042301. https://doi.org/10.1063/1.5085457 • Skoutnev, V., Hakim, A., Juno, J., & TenBarge, J. M. (2019). “Temperature-Dependent Saturation of Weibel- Type Instabilities in Counter-streaming Plasmas”, Astrophysical Journal Letters, 872, (2). https://doi.org/10. 3847%2F2041-8213%2Fab0556 • Sundström, A., Juno, J., TenBarge, J. M., & Pusztai, I. (2019). “Effect of a weak ion collisionality on the dynamics of kinetic electrostatic shocks”, Journal of Plasma Physics, 85. https://doi.org/10.1017/S0022377819000023 • Srinivasan, B. and Hakim, A. (2018). “Role of electron inertia and electron/ion finite Larmor radius effects in low-beta, magneto-Rayleigh-Taylor instability”, Physics of Plasmas, 25, 092108. https://doi.org/10.1063/1. 5046098 • Ng, J., Hakim, A., & Bhattacharjee, A. (2018). “Using the maximum entropy distribution to describe electrons in reconnecting current sheets”, Physics of Plasmas, 25, 082113. https://doi.org/10.1063/1.5041758 • Wang, L., Germaschewski, K., Hakim, A., Dong, C., Raeder, J., & Bhattacharjee, A. (2018). “Electron Physics in 3-D Two-Fluid 10-Moment Modeling of Ganymede’s Magnetosphere”, Journal of Geophysical Research: Space Physics, 41 (A3), 8688–16. https://doi.org/10.1002/2017JA024761 • Pusztai, I., TenBarge, J. M., Csapó, A. N., Juno, J., Hakim, A., Yi, K & Fülöp, T. (2018). “Low Mach-number collisionless electrostatic shocks and associated ion acceleration”, Plasma Physics and Controlled Fusion, 60 (3), 035004–11. https://doi.org/10.1088/1361-6587/aaa2cc • Shi, E. L., Hammett, G. W., Stolzfus-Dueck, T., Hakim, A. (2017). “Gyrokinetic continuum simulation of turbulence in a straight open-field-line plasma”, Journal of Plasma Physics, 83, 1–27. https://doi.org/10.1017/ S002237781700037X • Cagas, P., Hakim, A., Scales, W., Srinivasan, B. (2017). “Nonlinear saturation of the Weibel instability”, Physics of Plasmas, 24 (11), 112116. https://doi.org/10.1063/1.4994682 • Ng, J., Hakim, A., Bhattacharjee, A., Stanier, A., & Daughton, W. (2017). “Simulations of anti-parallel reconnection using a nonlocal heat flux closure”, Physics of Plasmas, 24 (8), 082112. https://doi.org/10.1063/1. 4993195 • Stanier, A., Daughton, W., Simakov, A. N., Chacón, L., Le, A., Karimabadi, H., Ng, J., & Bhattacharjee, A. (2017). “The role of guide field in magnetic reconnection driven by island coalescence”, Physics of Plasmas, 24, 022124. https://doi.org/10.1063/1.4976712 • Cagas, P., Hakim, A., Juno, J., Srinivasan, B. (2017). “Continuum kinetic and multi-fluid simulations of classical sheaths”, Physics of Plasmas, 24 (2), 022118. https://doi.org/10.1063/1.4976544 • Ng, J., Huang, Y.-M., Hakim, A., Bhattacharjee, A., Stanier, A., Daughton, W., Wang, L., & Germaschewski, K. (2015). “The island coalescence problem: Scaling of reconnection in extended fluid models including higher- order moments”, Physics of Plasma, 22, 112104. https://doi.org/10.1063/1.4935302

5.3. Physics papers 177 gkyl Documentation, Release 2.0-alpha

• Stanier, A., Daughton, W., Chacón, L., Karimabadi, H., Ng, J., Huang, Y.-M., Hakim, A., & Bhattacharjee, A. (2015). “Role of Ion Kinetic Physics in the Interaction of Magnetic Flux Ropes”, Physical Review Letters, 115, 175004. https://doi.org/10.1103/PhysRevLett.115.175004 • Wang, L., Hakim, A. H., Bhattacharjee, A., & Germaschewski, K. (2015). “Comparison of multi-ﬂuid moment models with particle-in-cell simulations of collisionless magnetic reconnection”, Physics of Plasmas, 22 (1), 012108. https://doi.org/10.1063/1.4906063

178 Chapter 5. Publications and theses CHAPTER 6

Presentations

You can browse a folder of pdf / PowerPoint / Keynote ﬁles of Gkeyll presentations, or click on links below.

Note: Google Drive often does not display the slides properly. Please download the slides on your local machine to view them properly.

2021 • “Modelling AUG scrape-off-layer plasma with full-f continuum Electromagnetic Gyrokinetic simulation”, Ru- pak Mukherjee, Virtual 25th Joint EU-US TTF Meeting, September 2021. pdf • “Simulation of AUG scrape-off-layer plasma with full-f continuum Electromagnetic Gyrokinetic simulation”, Rupak Mukherjee, Sherwood Fusion Theory conference, August 2021. pdf, Keynote (with movies), video recording of talk • “Initial Gkeyll simulations of Scrape-Off-Layer Turbulence in ASDEX-U”, Rupak Mukherjee, MPPC Meeting, January 2021. pdf, Keynote (with movies), video recording of talk 2020 • “Electromagnetic full-f gyrokinetic simulation of ASDEX SOL turbulence with discontinuous Galerkin method”, Rupak Mukherjee, APS DPP Annual Meeting, November 2020. pdf, Keynote (with movies), video recording of talk • “Investigating magnetic ﬂuctuations in gyrokinetic simulations of tokamak SOL turbulence”, Noah Mandell, APS DPP Annual Meeting, November 2020. pdf, Keynote (with movies), video recording of talk • “A Deep Dive into the Distribution Function: Understanding Phase Space Dynamics Using Continuum Vlasov- Maxwell Simulations”, Jimmy Juno, APS DPP Annual Meeting, November 2020. Keynote • “Balancing Flexibility and Usability in the Gkeyll Simulation Framework”, Jimmy Juno, APS DPP Annual Meeting, November 2020 Keynote • “Studies of plasma sheaths using novel numerical schemes with self-consistent emitting walls and Fokker- Planck collisions”, Petr Cagas, APS DPP Annual Meeting, November 2020. pdf, video recording

179 gkyl Documentation, Release 2.0-alpha

• “Kinetic Boltzmann modeling of neutral transport for a continuum gyrokinetic code”, Tess Bernard, APS DPP Annual Meeting, November 2020. pdf • “Alias-free, Matrix-free, and Quadrature-free Discontinuous Galerkin Algorithms for (Plasma) Kinetic Equa- tions”, Ammar Hakim. SuperComputing 2020, November 2020. ppt • “Investigating magnetic fluctuations in gyrokinetic simulations of tokamak SOL turbulence”, Noah Mandell, PPPL Theory Research & Review Seminar, October 2020. pdf, Keynote (with movies) • “Investigating magnetic fluctuations in tokamak SOL turbulence using Gkeyll gyrokinetic simulations”, Noah Mandell, PPPL Monthly Research Meeting, October 2020. pdf, Keynote (with movies) • “Magnetic fluctuations in gyrokinetic simulations of tokamak SOL turbulence”, Noah Mandell, Journal of Plasma Physics Frontiers colloquium series, April 2020. pdf, Keynote (with movies) • “Initial SOL turbulence results from the Gkeyll code, including first electromagnetic effects”, Greg Hammett, AUG Seminar, Garching, January 2020. pdf, ppt (with movies) 2019 • “Continuum Electromagnetic Gyrokinetic Simulations of Turbulence in the Tokamak Scrape-Off Layer and Laboratory Devices”, Ammar Hakim, APS Division of Plasma Physics, Fort Lauderdale, 2019. • “Tests of a Discontinuous Galerkin scheme for Hamiltonian systems in non-canonical coordinates”, Rupak Mukherjee, APS Division of Plasma Physics, Fort Lauderdale, 2019. pdf • “Gyrokinetic continuum simulations of plasma turbulence in the Texas Helimak”, Tess Bernard, Sherwood Fusion Theory Conference, Princeton, April 2019. • “Gyrokinetic continuum simulations of plasma turbulence in the Texas Helimak”, Tess Bernard, 24th Joint US-EU Transport Task Force Meeting, Austin, March 2019. 2016 • “Full-F gyrokinetic simulations of the LAPD device with open field lines and sheath boundary conditions”, Greg W. Hammett, Eric L. Shi, Ammar Hakim, Oxford Plasma Theory Group Seminar, Nov. 17, 2016. pdf, ppt Not very complete. more to be added. . .

180 Chapter 6. Presentations CHAPTER 7

Developer notes

7.1 On use of the Maxima CAS

Throughout Gkyl a large amount of Lua and C++ code is automatically pre-generated using the Maxima computer algebra system (CAS). Maxima is free and has a vast amount of features. For some of the calculations the use of a CAS is essential as the algebra, even though relatively easy, is very tedious, needing thousands of evaluations of various integrals etc. A very pleasant front-end for Maxima is provided by the wxmaxima program. This provides a “document based” interface to Maxima and one can mix regular text (and equations) with Maxima interactions. A very comprehensive physics oriented tutorial is Maxima by Example by Edwin Woollett. All Maxima code is checked into the gkyl/cas-scripts directory. To use the Maxima code in this directory you need to tell Maxima to ﬁnd it. To do this, create the directory (if it does not exist already): mkdir $HOME/.maxima

In this create or edit the ﬁle called “maxima-init.mac” and add the following lines to it: file_search_maxima: append(file_search_maxima, ["PATH_TO_YOUR_GKYL/gkyl/cas-scripts/###.{lisp,mac}"]) $

Where “PATH_TO_YOUR_GKYL” is the full path to the location where your gkyl source lives. Start/restart Maxima. Once you do this, then the Maxima code in the cas-scripts directory can be loaded, for example as: load("modal-basis")$ load("basis-precalc/basisSer1x1v")$

This will load the code to work with Modal basis functions and the serendipity basis sets in 1x1v into your Maxima session/code. To make plots on Maxima, you can use the excellent draw2d/3d packages. Chapter 4 of this manual describes the draw packages. To get plotting to work you need to install Gnuplot and set some paths properly. On a Mac, the maxima-init.mac ﬁle looks like:

181 gkyl Documentation, Release 2.0-alpha

load("draw")$ gnuplot_command: "/Applications/Gnuplot.app/Contents/Resources/bin/gnuplot" $ set_plot_option([gnuplot_term, "qt"], [gnuplot_preamble, "set object rectangle from screen 0,0 to screen 1,1 behind

˓→fillcolor rgb 'white' fillstyle solid noborder"] )$

set_draw_defaults(terminal=qt, user_preamble="set object rectangle from screen 0,0 to screen 1,1 behind fillcolor

˓→rgb 'white' fillstyle solid noborder", nticks=200, line_width=2 )$

file_search_maxima: append(file_search_maxima, ["PATH_TO_YOUR_GKYL/gkyl/cas-scripts/###.{lisp,mac}"]) $

Again, remember “PATH_TO_YOUR_GKYL” is the full path to the location where your gkyl source lives. On Linux or Windows you will need to experiment with paths and settings to get plots to work.

7.1.1 Note to developers on maximum default available memory with sbcl LISP compiler

Depending on which LISP compiler Maxima is using, by default, the compiler may not be able to claim/use all of the RAM on your computer. To change this a developer needs to edit the “maxima” executable themselves. On MacOS go to the following directory (assuming you have installed Maxima in your Applications):

cd/Applications/Maxima.app/Contents/Resources/opt/bin

Once in this directory, with your favorite text editor, open up the ﬁle maxima. Search for the speciﬁc LISP compiler being used. For example, the most common LISP compiler amongst the Gkeyll development team is sbcl. You should see a conditional statement like:

elif ["$MAXIMA_LISP"="sbcl" ]; then

within the conditional add:

MAXIMA_LISP_OPTIONS+="--dynamic-space-size 100000"

or a larger number depending on the size of your RAM.

7.2 Modal basis functions

Gkyl uses orthonormal modal basis functions. The basis functions are deﬁned on a d-dimensional hypercube 퐼푑 = 푑 [−1, 1] . Let 휓푘(x), 푘 = 1, . . . , 푁, with 휓1 a constant, be the basis. Then the basis satisfy

⟨휓푘휓푚⟩ = 훿푘푚

where the angle brackets indicate integration over the hypercube 퐼푑. In this note I describe some common operations that are needed while working with these basis sets. (All of this relatively straightforward stuff, but it is good to write it down somewhere).

182 Chapter 7. Developer notes gkyl Documentation, Release 2.0-alpha

Contents

• Modal basis functions – Pre-computed basis functions – Computing cell averages – Convolution of two functions

7.2.1 Pre-computed basis functions

Precomputed modal basis sets on phase-space for 1x1v, 1x2v, 1x3v, 2x2v, 2x3v, 3x2v, and 3x3v phase-space are stored as Lisp ﬁles in gkyl/cas-scripts/basis-precalc directory. Both maximal-order and serendipity basis sets are computed for polynomial orders 1, 2, 3 and 4. Computing orthonormal basis set in higher dimensions is time-consuming and so these pre-computed lisp ﬁles should be used. For example:

load("basis-precalc/basisSer2x3v")

will load the serendipity basis sets in 2x3v phase-space. NOTE: please read Maxima notes to get this command to work. The ﬁles deﬁne the following Maxima variables:

varsC, varsP, basisC, basisP

The varsC are the conﬁguration-space independent variables (푥, 푦 in 2X) and varsP are the phase-space independent variables (푥, 푦, 푣푥, 푣푦, 푣푧 in 2X3V). The i-th entry in the list basisC and basisP contain the basis sets of polynomial order 푖. Hence, to access the phase-space basis for 푝 = 2 do something like:

basisP2 : basisP[2].

7.2.2 Computing cell averages

Consider some function that is expanded in the basis: ∑︁ (︀ )︀ 푓(x) = 푓푘휓푘 휂(x) 푘 where 휂(x) maps the physical space to logical space. Then, the cell-average is deﬁned as 1 1 푓 = ⟨푓⟩ = 푓 휓 ⟨1⟩ = 푓 휓 . 2푑 2푑 1 1 1 1

2 where 휓1 is constant. By orthonormality we have ⟨휓1⟩ = 1 which indicates that 1 휓1 = √ . 2푑 This means that the cell-average is given by 푓 푓 = √1 2푑

7.2. Modal basis functions 183 gkyl Documentation, Release 2.0-alpha

7.2.3 Convolution of two functions

Now consider two functions, 푓 and 푔 that are both expanded in the basis. The inner product of the two functions is: 1 푓푔 = ⟨푓푔⟩ 2푑 Note the the bar on the LHS, i.e. this expression gives the average of the inner product on a single cell. Now, as the basis functions are orthonormal, this leads to the particularly simple expression

1 ∑︁ 푓푔 = 푓 푔 2푑 푘 푘 푘 This makes it very easy to compute things like electromagnetic energy, and other quadratic quantities.

7.3 The recovery Maxima code

A major algorithm used in many parts of Gkeyll is the recovery procedure. This is used to construct DG (hyper)diffusion equation solvers, the LBO diffusion kernels and even some very high-order hyperbolic solvers. We have attempted to abstract the recovery procedure (which is rather complicated) into three very simple to use functions.

7.4 Strong-Stability preserving Runge-Kutta time-steppers

The Gkyl DG solvers use SSP-RK time-steppers. Three steppers are implemented: SSP-RK2, SSP-RK3 and a four- stage SSP-RK3 that allows twice the CFL (for the cost of additional memory) as the other schemes. See [DurranBook] page 56. The schemes are described below. Here, the symbol ℱ is used to indicate a ﬁrst-order Euler update:

ℱ[푓, 푡] = 푓 + ∆푡ℒ[푓, 푡] where ℒ[푓] is the RHS operator from the spatial discretization of the DG scheme.

Contents

• Strong-Stability preserving Runge-Kutta time-steppers – SSP-RK2 – SSP-RK3 – Four stage SSP-RK3 – Region of absolute stability – References

7.4.1 SSP-RK2

푓 (1) = ℱ[푓 푛, 푡푛] 1 1 푓 푛+1 = 푓 푛 + ℱ[푓 (1), 푡푛 + ∆푡] 2 2 with 퐶퐹 퐿 ≤ 1.

184 Chapter 7. Developer notes gkyl Documentation, Release 2.0-alpha

7.4.2 SSP-RK3

푓 (1) = ℱ[푓 푛, 푡푛] 3 1 푓 (2) = 푓 푛 + ℱ[푓 (1), 푡푛 + ∆푡] 4 4 1 2 푓 푛+1 = 푓 푛 + ℱ[푓 (2), 푡푛 + ∆푡/2] 3 3 with 퐶퐹 퐿 ≤ 1. As this scheme has three stages instead of two, it will take about 1.5푋 longer to run than the SSP-RK2 scheme.

7.4.3 Four stage SSP-RK3

1 1 푓 (1) = 푓 푛 + ℱ[푓 푛, 푡푛] 2 2 1 1 푓 (2) = 푓 (1) + ℱ[푓 (1), 푡푛 + ∆푡/2] 2 2 2 1 1 푓 (3) = 푓 푛 + 푓 (2) + ℱ[푓 (2), 푡푛 + ∆푡] 3 6 6 1 1 푓 푛+1 = 푓 (3) + ℱ[푓 (3), 푡푛 + ∆푡/2] 2 2 with 퐶퐹 퐿 ≤ 2. Note that this scheme has four stages, but allows twice the time-step that SSP-RK2 and SSP-RK3, hence will result in a speed up of 1.5푋 compared to the three-stage SSP-RK3 scheme.

7.4.4 Region of absolute stability

For each of the above schemes, I have plotted below the region of absolute stability. Note that only the RK3 schemes are stable when there is no diffusion in the system, and hence should be prefered.

7.4.5 References

7.5 Normalized units for the Vlasov-Maxwell system

The Gkyl design philosophy involves the implementation of unit-full systems of equations, i.e., Gkyl simulations can be run with real parameters for direct comparison with experiments, with universal constants speciﬁed by using values provided by the National Institute of Standards and Technology. For example, in the absence of collisions, the Vlasov-Maxwell system of equations in S.I. units is as follows,

휕푓푠 푞푠 + v · ∇x 푓푠 + (E + v × B)·∇v 푓푠 = 0, 휕푡 푚푠 휕B + ∇ × E = 0, ∇ · B = 0, 휕푡 x x 휕E 휌푐 휖0휇0 − ∇x × B = −휇0J, ∇x · E = 휕푡 휖0 ∑︁ ∫︁ ∞ ∑︁ ∫︁ ∞ 휌푐 = 푞푠 푓푠 푑v, J = 푞푠 v푓푠 푑v. 푠 −∞ 푠 −∞ However, one may not always wish to run simulations with the unit-full system. Instead, one can consider a normalized set of equations. A natural choice for the normalization of the Vlasov-Maxwell system of equations would redeﬁne all

7.5. Normalized units for the Vlasov-Maxwell system 185 gkyl Documentation, Release 2.0-alpha

Fig. 1: Absolute stability regions for a equation 푦˙ = (휆 + 푖휔)푦 for SSP-RK2 (red), SSP-RK3 (black) and four stage SSP-RK3 (magenta). When there is no diffusion (휆 = 0) the SSP-RK2 scheme is slightly unstable as it has no intercept on the imaginary axis. Hence, the third order schemes should be preferred.

186 Chapter 7. Developer notes gkyl Documentation, Release 2.0-alpha

the relevant quantities as follows, −1 푡 = 휔푝푒 휏,

x = 푑푒 휒, v = 푐 휈,

푞푠 = 푒 푞˜푠,

푚푠 = 푚푒 푚˜푠,

푛푠 = 푛0 푛,˜ 푒푛 푑 E = 0 푒 E˜, 휖0 푒푛 B = 0 B˜ , 휖0휔푝푒 where 1 푐 = √ , 휖0휇0 √︃ 푒2푛 휔푝푒 = , 푚푒휖0 푐 푑푒 = , 휔푝푒 are the speed of light, electron plasma frequency, and the electron skin depth respectively. Note that the charge normalization means that in a proton-electron plasma, 푞˜푠 = ±1, and the density normalization is such that in a quasi-neutral plasma, the initial density of each species is 1.0. We can also check that the electric and magnetic ﬁeld normalizations are reasonable by making sure that the normalization has the correct units for the electric and magnetic ﬁelds in S.I. units, 1 푒푛푑 퐶 3 푚 푁 푒 → 푚 = 휖 퐶2 퐶 0 푁푚2 1 푒푛 퐶 3 푘푔 → 푚 = = 푇. 휖 휔 퐴2푠4 1 퐴푠2 0 푝푒 푘푔푚3 푠 With these normalizations, the Vlasov-Maxwell system of equations then becomes,

휕푓푠 푞˜푠 + 휈 · ∇휒 푓푠 + (E˜ + 휈 × B˜ )·∇휈 푓푠 = 0, 휕휏 푚˜ 푠 휕B˜ + ∇ × E˜ = 0, ∇ · B˜ = 0, 휕휏 휒 휒 휕E˜ − ∇ × B˜ = −J˜, ∇ · E =휌 ˜ 휕휏 휒 휒 푐 ∑︁ ∫︁ ∞ ∑︁ ∫︁ ∞ 휌˜푐 = 푞˜푠 푓푠 푑휈, J˜ = 푞˜푠 휈푓푠 푑휈. 푠 −∞ 푠 −∞

This system of equations has the obvious advantage that universal constants, such as 휖0, are eliminated. In doing so, one does not need to worry about the propagation of round off error from, for example, the accumulation of the 푛+1 푛 푛+1 푛 current to the electric ﬁeld in the Ampere-Maxwell law, 퐸 = 퐸 + 훿푡J/휖0 becomed 퐸 = 퐸 + 훿푡J˜. Given Gkyl’s unit-full representation, a simple way to force the Vlasov-Maxwell solver to “use” these units is to specify the following parameters be equal to 1.0,

휖0 = 1.0, 휇0 = 1.0, 푐 = 1.0,

푚푒 = 1.0, 푞푖 = 1.0, 푞푒 = −1.0,

휔푝푒 = 1.0, 푑푒 = 1.0,

푛0 = 1.0.

7.5. Normalized units for the Vlasov-Maxwell system 187 gkyl Documentation, Release 2.0-alpha

With the above parameters set to 1.0, then Vlasov-Maxwell simulations require only a few parameters to be completely determined. In a proton-electron plasma, these are, the proton-to-electron mass ratio, 푚푝/푚푒, the proton-to-electron temperature ratio, 푇푝/푇푒, the ratio of some characteristic velocity, such as the electron Alfv’en speed, to the speed of

light, 푣퐴푒 /푐, and the plasma beta of either the protons or the electrons, 훽. It is often convenient with this normalized system to use the combination of the ratio of the electron Alfv’en speed to the speed of light and the plasma beta to derive the temperature in normalized units, like so, √ 푣 |B|/ 푛 푚 휇 퐴푒 = 푒 푒 0 → 푣˜ = |B˜ |, 푐 푐 퐴푒 2푛 푇 휇 훽 = 푒 푒 0 → 푇˜ = 훽˜ 푣˜ 2/2.0, 푒 |B|2 푒 푒 퐴푒

assuming the plasma is quasineutral and thus, 푛0 = 1.0 for both the protons and electrons. The proton beta and proton temperature then follow from the speciﬁed proton-to-electron temperature ratio. It is recommended that the user initialize Maxwellian distribution functions using this derived temperature, so as to avoid the ambiguity of the user’s deﬁnition of the thermal velocity,

(︂ 2 )︂ 푛˜푠 (휈 − u˜푠) 푓maxwellian = √︁ exp −푚˜푠 . 2푇˜푠 2휋푇˜푠/푚˜푠 √︀ √︀ Whether the user ultimately elects to use 푣푡ℎ푠 = 2푇푠/푚푠 or 푣푡ℎ푠 = 푇푠/푚푠 is of no consequence to the initialization of the simulation, and likely only to manifest in the user’s speciﬁcation of the velocity space extents.

7.6 From normalized to physical units in Vlasov and multi-ﬂuid simulations

Very often we setup a problem in terms of non-dimensional units. An example of one such non-dimensional units is given in Normalized Units notes. Here, I describe one approach to “denormalize” the setup and create a set of dimensional values that are perhaps easier to interpret than the non-dimensional units. This denormalization is not unique, expressing the fact that plasma (and ﬂuid) equations are essentially scale-free. As an example, consider the following fragment of a Gkeyll input ﬁle, describing a typical problem setup.

local Constants= require"Lib.Constants" -- Universal constants. epsilon0= 1.0 -- permittivity of free space. mu0= 1.0 -- permeability of free space. lightSpeed= 1.0/math.sqrt(mu0 *epsilon0) -- speed of light.

-- User inputs. massRatio= 1836.153 tau= 1.0 -- Ratio of ion to electron temperature. B0= 0.25 -- Driven magnetic field amplitude. beta= 0.01 -- Total plasma beta.

ionMass= massRatio -- Ion mass in simulation. elcMass= 1.0 -- Electron mass in simulation. ionCharge= 1.0 -- Ion charge in simulation. elcCharge=-1.0 -- Electron charge in simulation.

n= 0.01 -- Plasma density, same for ions and electrons. Te= beta *(B0^2)/(2.0*mu0*(1.0+tau)) -- Electron temperature. Ti= tau *Te -- Ion temperature.

(continues on next page)

188 Chapter 7. Developer notes gkyl Documentation, Release 2.0-alpha

(continued from previous page) -- Derived parameters. vtElc= math.sqrt(2.0 *Te/elcMass) vtIon= math.sqrt(2.0 *Ti/ionMass)

-- cyclotron frequency omegaCe= ionCharge *B0/elcMass

-- gyro radius rhoe= vtElc/omegaCe

Lx= 200.0 *math.pi*rhoe nuElc= 0.1 *w0*(math.pi^2) -- Electron collision frequency. nuIon= nuElc/math.sqrt(ionMass *(tau^3)) -- Ion collision frequency. wpe= math.sqrt(n *elcCharge^2/(elcMass*epsilon0)) -- plasma frequency

What do these numbers mean? For example, in the above we set 푛 = 0.01. Obviously, this does not mean that there are 1/100 particles per meter! If it did, then if our cell-size was say 1/10 of a meter, then we would have hardly any particles in a single cell. In fact, we could no longer treat the problem with a Vlasov equation but would need to use a N-body description instead. So how to interpret these numbers? First, recall that these are non-dimensional quantities. Hence, these numbers do not have any units and can’t be interpreted as if they have units. To do a meaningful interpretation, we must denormalize the numbers by picking some reference values. There are several reasonable choices one can make. For example, one can choose a reference frequency. Given the speed of light in SI units we can then determine the reference length scale. Then, using SI units values for electron mass, fundamental charge and other quantities, allows us to completely denormalize all values. Of course, other reasonable choices are possible too. For example, we can select a reference length and then use the speed of light to determine the reference time (frequency) unit. 10 −1 In the following fragment, we select our plasma-frequency as 휔푝푒 = 10 푠 , and the compute a reference frequency and, using speed of light, a length-scale:

wpePhys= 1e10 -- Plasma-frequency in [1/s] WDIM= wpePhys/wpe -- Reference frequency [1/s] CDIM= Constants.SPEED_OF_LIGHT -- Reference speed LDIM= CDIM/WDIM -- Reference length scale

Now, all other quantities can be computed:

nPhys= wpePhys^2 *Constants.ELECTRON_MASS*Constants.EPSILON0/Constants.ELEMENTARY_ ˓→CHARGE^2 lambdaDPhys= vtElc/wpe *LDIM OmegaCePhys= omegaCe *WDIM B0Phys= OmegaCePhys *Constants.ELECTRON_MASS/Constants.ELEMENTARY_CHARGE vAlfPhys= B0Phys/math.sqrt(Constants.MU0 *Constants.ELECTRON_MASS*massRatio*nPhys)

Note we use physical SI unit values of electron mass, elementary charge, 휖0 and 휇0. With this, the various physical values are:

Number density 3.14208e+16[#/m^3] Electron thermal speed 5.29963e+06 [m/s] Ion thermal speed 123678 [m/s] Debye length 0.000529963 [m] Electron gyro-radius 0.000211985 [m] Domain length 0.133194 [m] (continues on next page)

7.6. From normalized to physical units in Vlasov and multi-ﬂuid simulations 189 gkyl Documentation, Release 2.0-alpha

(continued from previous page) Plasma parameter 4.67686e+06[#] B0 0.142141 [T] vAlf/c 0.0583426

These numbers appear perfectly reasonable. For example, the plasma parameter, i.e. the number of particles inside a 3 6 Debye sphere, is computed as 푛휆퐷 = 4.7 × 10 , showing that the plasma approximation is perfectly valid. Of course, other choices of the initial plasma-frequency (or another choice of a particular physical parameter like the domain size or number-density) would give a different set of values. However, of course, independent of the choice, the physics remains unchanged as long as all physical dimensions are scaled consistently. (Which is of course the virtue of the non-dimensional units in the ﬁrst place).

7.7 The eigensystem of the Maxwell equations with extension to perfectly hyperbolic Maxwell equations

7.7.1 Eigensystem of Maxwell equations

In this document I list the eigensystem of the Maxwell equations. Maxwell’s equations consist of the curl equations 휕B + ∇ × E = 0 휕푡 휕E 휖 휇 − ∇ × B = −휇 J 0 0 휕푡 0 along with the divergence relations 휚 ∇ · E = 푐 휖0 ∇ · B = 0.

Here, E is the electric field, B is the magnetic flux density, 휖0, 휇0 are permittivity and permeability of free space, and 1/2 J and 휚푐 are specified currents and charges respectively. The speed of light is determined from 푐 = 1/(휇0휖0) . These are linear equations and hence the eigensytem is independent of the value of the electromagnetic fields. In 1D Maxwell equations can be written as, ignoring sources, ⎡ ⎤ ⎡ ⎤ 퐸푥 0 2 ⎢퐸푦 ⎥ ⎢ 푐 퐵푧 ⎥ ⎢ ⎥ ⎢ 2 ⎥ 휕 ⎢퐸푧 ⎥ 휕 ⎢−푐 퐵푦⎥ ⎢ ⎥ + ⎢ ⎥ = 0. 휕푡 ⎢퐵푥⎥ 휕푥 ⎢ 0 ⎥ ⎢ ⎥ ⎢ ⎥ ⎣퐵푦⎦ ⎣ −퐸푧 ⎦ 퐵푧 퐸푦

The eigenvalues of this system are {−푐, −푐, 푐, 푐, 0, 0}. The right eigenvectors of the ﬂux Jacobian are given by the columns of the matrix ⎡ 0 0 0 0 1 0⎤ ⎢ 1 0 1 0 0 0⎥ ⎢ ⎥ ⎢ 0 1 0 1 0 0⎥ 푅 = ⎢ ⎥ . ⎢ 0 0 0 0 0 1⎥ ⎢ 1 1 ⎥ ⎣ 0 푐 0 − 푐 0 0⎦ 1 1 − 푐 0 푐 0 0 0

190 Chapter 7. Developer notes gkyl Documentation, Release 2.0-alpha

The left eigenvectors are the rows of the matrix

⎡ 1 푐 ⎤ 0 2 0 0 0 − 2 1 푐 ⎢0 0 2 0 2 0 ⎥ ⎢ 1 푐 ⎥ ⎢0 2 0 0 0 2 ⎥ 퐿 = ⎢ 1 푐 ⎥ . ⎢0 0 0 − 0 ⎥ ⎢ 2 2 ⎥ ⎣1 0 0 0 0 0 ⎦ 0 0 0 1 0 0

7.7.2 Eigensystem of Perfectly Hyperbolic Maxwell equations

The perfectly hyperbolic Maxwell equations are a modiﬁcation of the Maxwell equations that take into account the divergence relations. The modiﬁed equations explicitly “clean” divergence errors and are a hyperbolic generalization of the Hodge project method commonly used in electromagnetism to correct for charge conservation errors. See [munz_2000], [munz_2000b], [munz_2000c] for details. These equations are written as 휕B + ∇ × E + 훾∇휓 = 0 휕푡 휕E 휖 휇 − ∇ × B + 휒∇휑 = −휇 J 0 0 휕푡 0 1 휕휑 휚 + ∇ · E = 푐 휒 휕푡 휖0 휖 휇 휕휓 0 0 + ∇ · B = 0. 훾 휕푡

Here, 휓 and 휓 are correction potentials for the electric and magnetic ﬁeld respectively and 휒 and 훾 are dimensionless factors that control the speed at which the errors are propagated. In 1D these equations can be written as, ignoring sources,

⎡ ⎤ ⎡ 2 ⎤ 퐸푥 휒푐 휑 2 ⎢퐸푦 ⎥ ⎢ 푐 퐵푧 ⎥ ⎢ ⎥ ⎢ 2 ⎥ ⎢퐸푧 ⎥ ⎢−푐 퐵푦⎥ ⎢ ⎥ ⎢ ⎥ 휕 ⎢퐵푥⎥ 휕 ⎢ 훾휓 ⎥ ⎢ ⎥ + ⎢ ⎥ = 0. 휕푡 ⎢퐵푦⎥ 휕푥 ⎢ −퐸푧 ⎥ ⎢ ⎥ ⎢ ⎥ ⎢퐵푧 ⎥ ⎢ 퐸푦 ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ 휑 ⎦ ⎣ 휒퐸푥 ⎦ 2 휓 훾푐 퐵푥

The eigenvalues of this system are {−푐훾, 푐훾, −푐휒, 푐휒, −푐, −푐, 푐, 푐}. The right eigenvectors of the ﬂux Jacobian are given by the columns of the matrix

⎡ 0 0 1 1 0 0 0 0 ⎤ ⎢ 0 0 0 0 1 0 1 0 ⎥ ⎢ ⎥ ⎢ 0 0 0 0 0 1 0 1 ⎥ ⎢ ⎥ ⎢ 1 1 0 0 0 0 0 0 ⎥ 푅 = ⎢ 1 1 ⎥ . ⎢ 0 0 0 0 0 푐 0 − 푐 ⎥ ⎢ 1 1 ⎥ ⎢ 0 0 0 0 − 푐 0 푐 0 ⎥ ⎢ 1 1 ⎥ ⎣ 0 0 − 푐 푐 0 0 0 0 ⎦ −푐 푐 0 0 0 0 0 0

7.7. The eigensystem of the Maxwell equations with extension to perfectly hyperbolic Maxwell 191 equations gkyl Documentation, Release 2.0-alpha

The left eigenvectors are the rows of the matrix

⎡ 1 1 ⎤ 0 0 0 2 0 0 0 − 2 푐 1 1 ⎢0 0 0 2 0 0 0 2 푐 ⎥ ⎢ 1 푐 ⎥ ⎢ 2 0 0 0 0 0 − 2 0 ⎥ ⎢ 1 푐 ⎥ ⎢ 2 0 0 0 0 0 2 0 ⎥ 퐿 = ⎢ 1 푐 ⎥ . ⎢0 2 0 0 0 − 2 0 0 ⎥ ⎢ 1 푐 ⎥ ⎢0 0 2 0 2 0 0 0 ⎥ ⎢ 1 푐 ⎥ ⎣0 2 0 0 0 2 0 0 ⎦ 1 푐 0 0 2 0 − 2 0 0 0

7.8 The eigensystem of the Euler equations

In this document I list the eigensystem of the Euler equations valid for a general equation of state. The formulas are taken from [Kulikovskii2001], Chapter 3, section 3.1. The Euler equations can be written in conservative form as

⎡ 휌 ⎤ ⎡ 휌푢 ⎤ ⎢휌푢⎥ ⎢ 휌푢2 + 푝 ⎥ 휕 ⎢ ⎥ 휕 ⎢ ⎥ ⎢휌푣 ⎥ + ⎢ 휌푢푣 ⎥ = 0 (7.1) 휕푡 ⎢ ⎥ 휕푥 ⎢ ⎥ ⎣휌푤⎦ ⎣ 휌푢푤 ⎦ 퐸 (퐸 + 푝)푢 where 1 퐸 = 휌휀 + 휌푞2 2 is the total energy and 휀 is the internal energy of the ﬂuid and 푞2 = 푢2 + 푣2 + 푤2. The pressure is given by an equation of state (EOS) 푝 = 푝(휀, 휌). For an ideal gas the EOS is 푝 = (훾 − 1)휌휀. The eigenvalues are {푢 − 푐, 푢, 푢, 푢, 푢 + 푐}. The right eigenvectors of the ﬂux Jacobian are given by the columns of the matrix ⎡ 1 0 0 1 1 ⎤ ⎢ 푢 − 푐 0 0 푢 푢 + 푐 ⎥ ⎢ ⎥ 푅 = ⎢ 푣 1 0 푣 푣 ⎥ (7.2) ⎢ ⎥ ⎣ 푤 0 1 푤 푤 ⎦ ℎ − 푢푐 푣 푤 ℎ − 푐2/푏 ℎ + 푢푐

here ℎ = (퐸 + 푝)/휌 √︃ 휕푝 푝 휕푝 푐 = + 휕휌 휌2 휕휀

is the enthalpy and the sound speed respectively. Also, 1 휕푝 푏 = . 휌 휕휀 Note that for ideal gas EOS we have

푐2 1 ℎ = + 푞2 훾 − 1 2 √︂훾푝 푐 = 휌

192 Chapter 7. Developer notes gkyl Documentation, Release 2.0-alpha

and 푏 = 훾 − 1. Hence, in this case the term ℎ − 푐2/푏 in (7.2) is just 푞2/2. The left eigenvectors are the rows of the matrix ⎡휃 + 푢푐/푏 −푢 − 푐/푏 −푣 −푤 1 ⎤ ⎢−2푣푐2/푏 0 2푐2/푏 0 0 ⎥ 푏 ⎢ ⎥ 퐿 = ⎢−2푤푐2/푏 0 0 2푐2/푏 0 ⎥ (7.3) 2푐2 ⎢ ⎥ ⎣2ℎ − 2푞2 2푢 2푣 2푤 −2⎦ 휃 − 푢푐/푏 −푢 + 푐/푏 −푣 −푤 1 where 퐸 휕푝/휕휌 휃 = 푞2 − + 휌 휌 휕푝/휕휀 which, for an ideal gas EOS reduces to 푞2/2. 푇 Now consider the problem of splitting a jump vector ∆ ≡ [훿0, 훿1, 훿2, 훿3, 훿4] into coefficients neeeded in computing the Riemann problem. The coefficients are given by 퐿∆. For an ideal gas law EOS, after some algebra we can show that an efficient way to compute these are 훾 − 1 훼 = [︀(ℎ − 푞2)훿 + 푢훿 + 푣훿 + 푤훿 − 훿 ]︀ 3 푐2 0 1 2 3 4 훼1 = −푣훿0 + 훿2

훼2 = −푤훿0 + 훿3 (7.4) 1 훼 = [훿 + (푐 − 푢)훿 − 푐훼 ] 4 2푐 1 0 3 훼0 = 훿0 − 훼3 − 훼4.

7.8.1 References

7.9 The eigensystem of the ten-moment equations

In this document I list the eigensystem of the ten-moment equations. These equations are derived by taking moments of the Boltzmann equation and truncating the resulting inﬁnite series of equations by assuming the heat ﬂux tensor vanishes. In non-conservative form these equations are

휕푡푛 + 푛휕푗푢푗 + 푢푗휕푗푛 = 0 1 푞 휕 푢 + 휕 푃 + 푢 휕 푢 = (퐸 + 휖 푢 퐵 ) 푡 푖 푚푛 푗 푖푗 푗 푗 푖 푚 푖 푘푚푖 푘 푚 푞 휕 푃 + 푃 휕 푢 + 휕 푢 푃 + 푢 휕 푃 = 퐵 휖 푃 푡 푖푗 푖푗 푘 푘 푘 [푖 푗]푘 푘 푘 푖푗 푚 푚 푘푚[푖 푗푘] In these equations square brackets around indices represent the minimal sum over permutations of free indices within the bracket needed to yield completely symmetric tensors. Note that there is one such system of equations for each species in the plasma. Here, 푞 is the species charge, 푚 is the species mass, 푛 is the number density, 푢푗 is the velocity, 푃푖푗 the pressure tensor and E and B are the electric and magnetic ﬁeld respectively. Also 휕푡 ≡ 휕/휕푡 and 휕푖 ≡ 휕/휕푥푖. To determine the eigensystem of the homogeneous part of this system we ﬁrst write, in one-dimension, the left-hand side of of these equations in the form

휕푡v + A휕1v = 0 where v is the vector of primitive variables and A is the quasilinear coefﬁcient matrix1. For the ten-moment system we have [︀ ]︀푇 v = 휌, 푢1, 푢2, 푢3, 푃11, 푃12, 푃13, 푃22, 푃23, 푃33

1 There is no standard name for this matrix. I choose to call it the quasilinear coefﬁcient matrix instead of the incorrect term “primitive ﬂux Jacobian”.

7.9. The eigensystem of the ten-moment equations 193 gkyl Documentation, Release 2.0-alpha

where 휌 ≡ 푚푛 and ⎡ ⎤ 푢1 휌 0 0 0 0 0 0 0 0 ⎢ 0 푢1 0 0 1/휌 0 0 0 0 0 ⎥ ⎢ ⎥ ⎢ 0 0 푢1 0 0 1/휌 0 0 0 0 ⎥ ⎢ ⎥ ⎢ 0 0 0 푢1 0 0 1/휌 0 0 0 ⎥ ⎢ ⎥ ⎢ 0 3푃11 0 0 푢1 0 0 0 0 0 ⎥ A = ⎢ ⎥ ⎢ 0 2푃12 푃11 0 0 푢1 0 0 0 0 ⎥ ⎢ ⎥ ⎢ 0 2푃13 0 푃11 0 0 푢1 0 0 0 ⎥ ⎢ ⎥ ⎢ 0 푃22 2푃12 0 0 0 0 푢1 0 0 ⎥ ⎢ ⎥ ⎣ 0 푃23 푃13 푃12 0 0 0 0 푢1 0 ⎦ 0 푃33 0 2푃13 0 0 0 0 0 푢1

The eigensystem of this matrix needs to be determined. It is easiest to use a computer algebra system for this. I prefer the open source package Maxima for this. The right-eigenvectors returned by Maxima need to massaged a little bit to bring them into a clean form. The results are described below. The eigenvalues of the system are given by

1,2 √︀ 휆 = 푢1 − 푃11/휌 3,4 √︀ 휆 = 푢1 + 푃11/휌 5 √︀ 휆 = 푢1 − 3푃11/휌 6 √︀ 휆 = 푢1 + 3푃11/휌 7,8,9,10 휆 = 푢1

To maintain hyperbolicity we must hence have 휌 > 0 and 푃11 > 0. In multiple dimensions, in general, the diagonal elements of the pressure tensor must be positive. When 푃11 = 0 the system reduces to the cold fluid equations which is known to be rank deficient and hence not hyperbolic as usually understood2. Also notice that the eigenvalues do not √︀ √︀ include the usual fluid sound-speed 푐푠 = 5푝/3휌 but instead have two different propagation speeds 푐1 = 푃11/휌 √︀ and 푐2 = 3푃11/휌. This is because the (neutral) ten-moment system does not go to the correct limit of Euler equations in the absence of collisions. In fact, it is collisions that drive the pressure tensor to isotropy. These collision terms should also be included in the plasma ten-moment system. In this case, however, the situation is complicated due to the presence of multiple species of very different masses which leads to inter-species collision terms that need to be computed carefully. For a two-species plasma, for example, see the paper by Green [Green1973] in which the relations for relaxation of momentum and energy are used to derive a simplified collision integral for use in the Boltzmann equation. The right eigenvectors (column vectors) are given below.

⎡ 0 ⎤ ⎡ 0 ⎤ ⎢ 0 ⎥ ⎢ 0 ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ∓푐1 ⎥ ⎢ 0 ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ 0 ⎥ ⎢ ∓푐1 ⎥ ⎢ ⎥ ⎢ ⎥ 1,3 ⎢ 0 ⎥ 2,4 ⎢ 0 ⎥ r = ⎢ ⎥ r = ⎢ ⎥ ⎢ 푃11 ⎥ ⎢ 0 ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ 0 ⎥ ⎢ 푃11 ⎥ ⎢ ⎥ ⎢ ⎥ ⎢2푃12⎥ ⎢ 0 ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ 푃13 ⎦ ⎣ 푃12 ⎦ 0 2푃13

2 For hyperbolicity the quasilinear matrix must posses real eigenvalues and a complete set of linearly independent right eigenvectors. For the cold ﬂuid system we only have a single eigenvalue (the ﬂuid velocity) and a single eigenvector. This can lead to generalized solutions like delta shocks.

194 Chapter 7. Developer notes gkyl Documentation, Release 2.0-alpha

and ⎡ ⎤ 휌푃11 ⎢ ∓푐2푃11 ⎥ ⎢ ⎥ ⎢ ∓푐2푃12 ⎥ ⎢ ⎥ ⎢ ∓푐2푃13 ⎥ ⎢ 2 ⎥ 5,6 ⎢ 3푃 ⎥ r = ⎢ 11 ⎥ ⎢ 3푃11푃12 ⎥ ⎢ ⎥ ⎢ 3푃11푃13 ⎥ ⎢ 2 ⎥ ⎢ 푃11푃22 + 2푃 ⎥ ⎢ 12 ⎥ ⎣푃11푃23 + 2푃12푃13⎦ 2 푃11푃33 + 2푃13 and ⎡1⎤ ⎡0⎤ ⎡0⎤ ⎡0⎤ ⎢0⎥ ⎢0⎥ ⎢0⎥ ⎢0⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢0⎥ ⎢0⎥ ⎢0⎥ ⎢0⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢0⎥ ⎢0⎥ ⎢0⎥ ⎢0⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ 7 ⎢0⎥ 8 ⎢0⎥ 9 ⎢0⎥ 10 ⎢0⎥ r = ⎢ ⎥ r = ⎢ ⎥ r = ⎢ ⎥ r = ⎢ ⎥ ⎢0⎥ ⎢0⎥ ⎢0⎥ ⎢0⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢0⎥ ⎢0⎥ ⎢0⎥ ⎢0⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢0⎥ ⎢1⎥ ⎢0⎥ ⎢0⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣0⎦ ⎣0⎦ ⎣1⎦ ⎣0⎦ 0 0 0 1

We can now compute the left eigenvectors (row vectors) by inverting the matrix with right eigenvectors stored as columns. This ensures the normalization l푝r푘 = 훿푝푘, where the l푝 are the left eigenvectors. On performing the inversion we have [︂ ]︂ 1,3 푃12 1 푃12 1 l = 0 ± ∓ 0 − 2 0 0 0 0 2푐1푃11 2푐1 2푃11 2푃11 [︂ ]︂ 2,4 푃13 1 푃13 1 l = 0 ± 0 ∓ − 2 0 0 0 0 2푐1푃11 2푐1 2푃11 2푃11 and [︂ ]︂ 5,6 1 1 l = 0 ∓ 0 0 2 0 0 0 0 0 2푐2푃11 6푃11 and [︂ ]︂ 7 1 l = 1 0 0 0 − 2 0 0 0 0 0 3푐1 [︂ 2 ]︂ 8 4푃12 − 푃11푃22 2푃12 l = 0 0 0 0 2 − 0 1 0 0 3푃11 푃11 [︂ ]︂ 9 4푃12푃13 − 푃11푃23 푃13 푃12 l = 0 0 0 0 2 − − 0 1 0 3푃11 푃11 푃11 [︂ 2 ]︂ 10 4푃13 − 푃11푃33 2푃13 l = 0 0 0 0 2 0 − 0 0 1 3푃11 푃11

7.9.1 The eigensystem of the equations written in conservative form

In the wave-propagation scheme the quasilinear equations can be updated. However, the resulting solution will not be conservative. This actually might not be a problem for the ten-moment system as the system (as written) can not be put

7.9. The eigensystem of the ten-moment equations 195 gkyl Documentation, Release 2.0-alpha

into a homogeneous conservation law form anyway. However, most often for numerical simulations the eigensystem of the conservation form of the homogeneous system is needed. This eigensystem is related to the eigensystem of the quasilinear form derived above. To see this consider a conservation law

휕푡q + 휕1f = 0 where f = f(q) is a ﬂux function. Now consider an invertible transformation q = 휙(v). This transforms the conservation law to

′ −1 ′ 휕푡v + (휙 ) 퐷f 휙 휕1v = 0 where 휙′ is the Jacobian matrix of the transformation and 퐷f ≡ 휕f/휕q is the ﬂux Jacobian. Comparing this to the quasilinear form we see that the quasilinear matrix is related to the ﬂux Jacobian by

A = (휙′)−1 퐷f 휙′

This clearly shows that the eigenvalues of the ﬂux Jacobian are the same as those of the quasilinear matrix while the right and left eigenvectors can be computed using 휙′r푝 and l푝(휙′)−1 respectively. For the ten-moment system the required transformation is

⎡ 휌 ⎤ ⎢ 휌푢1 ⎥ ⎢ ⎥ ⎢ 휌푢2 ⎥ ⎢ ⎥ ⎢ 휌푢3 ⎥ ⎢ ⎥ ⎢휌푢1푢1 + 푃11⎥ q = 휙(v) = ⎢ ⎥ ⎢휌푢1푢2 + 푃12⎥ ⎢ ⎥ ⎢휌푢1푢3 + 푃13⎥ ⎢ ⎥ ⎢휌푢2푢2 + 푃22⎥ ⎢ ⎥ ⎣휌푢2푢3 + 푃23⎦ 휌푢3푢3 + 푃33

For this transformation we have ⎡ 1 0 0 0 0 0 0 0 0 0⎤ ⎢ 푢1 휌 0 0 0 0 0 0 0 0⎥ ⎢ ⎥ ⎢ 푢2 0 휌 0 0 0 0 0 0 0⎥ ⎢ ⎥ ⎢ 푢3 0 0 휌 0 0 0 0 0 0⎥ ⎢ ⎥ ′ ⎢푢1푢1 2휌푢1 0 0 1 0 0 0 0 0⎥ 휙 (v) = ⎢ ⎥ ⎢푢1푢2 휌푢2 휌푢1 0 0 1 0 0 0 0⎥ ⎢ ⎥ ⎢푢1푢3 휌푢3 0 휌푢1 0 0 1 0 0 0⎥ ⎢ ⎥ ⎢푢2푢2 0 2휌푢2 0 0 0 0 1 0 0⎥ ⎢ ⎥ ⎣푢2푢3 0 휌푢3 휌푢2 0 0 0 0 1 0⎦ 푢3푢3 0 0 2휌푢3 0 0 0 0 0 1

The inverse of the transformation Jacobian is ⎡ 1 0 0 0 0 0 0 0 0 0⎤ ⎢−푢1/휌 1/휌 0 0 0 0 0 0 0 0⎥ ⎢ ⎥ ⎢−푢2/휌 0 1/휌 0 0 0 0 0 0 0⎥ ⎢ ⎥ ⎢−푢3/휌 0 0 1/휌 0 0 0 0 0 0⎥ ⎢ ⎥ ′ −1 ⎢ 푢1푢1 −2푢1 0 0 1 0 0 0 0 0⎥ (휙 ) = ⎢ ⎥ ⎢ 푢1푢2 −푢2 −푢1 0 0 1 0 0 0 0⎥ ⎢ ⎥ ⎢ 푢1푢3 −푢3 0 −푢1 0 0 1 0 0 0⎥ ⎢ ⎥ ⎢ 푢2푢2 0 −2푢2 0 0 0 0 1 0 0⎥ ⎢ ⎥ ⎣ 푢2푢3 0 −푢3 −푢2 0 0 0 0 1 0⎦ 푢3푢3 0 0 −2푢3 0 0 0 0 0 1

196 Chapter 7. Developer notes gkyl Documentation, Release 2.0-alpha

7.9.2 References

7.10 Handling two-ﬂuid ﬁve-moment and ten-moment source terms

The two-fluid system treats a plasma as a mixture of electron and ion fluids, coupled via the electromagnetic field and collisions. In the ideal two-fluid system collisions and heat-flux are neglected, leading to a closed system of coupled PDEs: one set of fluid equations for each of the fluids, and Maxwell equations for the electromagnetic field. Non- neutral effects, electron interia as well as displacement currents are retained. Further, the fluid pressures can be treated as either a scalar (five-moment model) or a symmetric 3 × 3 tensor (ten-moment model), or a combination. While solving the two-fluid system there are two distinct solves: the hyperbolic update, which can be performed for each equation system separately and the source update, which couples the fluids and the fields together. In this note I only focus on the source updates, for both the five as well as the ten-moment equations. When written in non- conservation law form, the only sources in the system are the Lorentz force in the momentum equation, the plasma currents in the electric field equation and, in the ten-moment model, the rotation of the pressure tensor due to the magnetic field. The latter equation is uncoupled from the source update for the momentum and the electric field, and can be treated separately.

7.10.1 Five-moment source updates

The equations for the ﬁve-moment source updates are

푑v푠 푞푠 = (E + v푠 × B) 푑푡 푚푠 푑E ∑︁ 휖 = − 푞 푛 v 0 푑푡 푠 푠 푠 푠

where, for the plasma species 푠, 푛푠 is the number density, v푠 is the velocity, 푞푠 and 푚푠 are the charge and mass respectively. Further, E is the electric ﬁeld, and 휖0 is permittivity of free space. In these equations the magnetic ﬁeld and number density are constants, as there are no source terms for these quantities.

It is more convenient to work in terms of the plasma current J푠 ≡ 푞푠푛푠v푠, which leads to the coupled system 푑J 푠 = 휔2휖 E + J × Ω 푑푡 푠 0 푠 푠 푑E ∑︁ 휖 = − J 0 푑푡 푠 푠 √︀ 2 where Ω푠 ≡ 푞푠B/푚푠 and 휔푠 ≡ 푞푠 푛푠/휖0푚푠 is the species plasma frequency. This is a system of linear, constant- coefﬁcient ODEs for the 3푠 + 3 unknowns J푠 and E.

Implicit solution

To solve the system of ODEs we replace the time-derivatives with time-centered differences. This leads to the discrete equations J푛+1/2 − J푛 푠 푠 = 휔2휖 E푛+1/2 + J푛+1/2 × Ω ∆푡/2 푠 0 푠 푠 E푛+1/2 − E푛 ∑︁ 휖 = − J푛+1/2 0 ∆푡/2 푠 푠

푛+1/2 푛+1 푛 푛+1/2 푛+1 푛 where J푠 = (J푠 +J푠 )/2 and E = (E +E )/2. This is a system of 3푝+3 system of linear equations 푛+1/2 푛+1/2 for the 3푝 + 3 unknowns J푠 , 푠 = 1, . . . , 푝 and E푠 and can be solved with any linear algebra routine.

7.10. Handling two-ﬂuid ﬁve-moment and ten-moment source terms 197 gkyl Documentation, Release 2.0-alpha

7.10.2 Ten-moment source updates

The ten-moment equations have identical sources for currents and electric ﬁeld. For these terms the same implicit algorithm can be used. In addition, there are source terms in the pressure equation. In non-conservative form these can be written as the linear system of equations

⎡ ⎤ 푃푥푥 ⎢푃푥푦⎥ ⎢ ⎥ 푑 ⎢푃푥푧 ⎥ 푞 ⎢ ⎥ = (02 퐵푧 푑푡 ⎢푃푦푦 ⎥ 푚 ⎢ ⎥ ⎣푃푦푧 ⎦ 푃푧푧

−2 퐵푦0 00

−퐵푧0

퐵푥퐵푧

−퐵푦0

퐵푦 − 퐵푥 00

퐵푧 − 퐵푦

0 − 2 퐵푧 00

2 퐵푥0

0퐵푦

−퐵푧 − 퐵푥

0퐵푥 00

2 퐵푦0

−2 퐵푥0 ⎡ ⎤ 푃푥푥 ⎢푃푥푦⎥ ⎢ ⎥ ⎢푃푥푧 ⎥ ⎢ ⎥ . ⎢푃푦푦 ⎥ ⎢ ⎥ ⎣푃푦푧 ⎦ 푃푧푧

This system can be updated using a similar time-centered implicit method as use for the currents and the electric field. Note that unlike the sources for the currents and the electric field, the pressure source terms are uncoupled from the other fluid and electromagnetic terms.

198 Chapter 7. Developer notes Bibliography

[Skoutnev2019] Skoutnev, V., Hakim, A., Juno, J., & TenBarge, J. M. (2019). “Temperature-Dependent Saturation of Weibel-Type Instabilities in Counter-streaming Plasmas”, Astrophysical Journal Letters, 872, (2). https: //doi.org/10.3847%2F2041-8213%2Fab0556 [Juno2020] Juno, J., Swisdak, M. M., TenBarge. J. M., Skoutnev, V., & Hakim, A. “Noise-induced magnetic field saturation in kinetic simulations”, Journal of Plasma Physics, 86, (4). https://doi.org/10.1017/ S0022377820000707 [Shi2019] Shi, E. L., Hammett, G. W., Stoltzfus-Dueck, T., & Hakim, A. (2019). “Full-f gyrokinetic simulation of turbulence in a helical open-field-line plasma”, Physics of Plasmas, 26, 012307. https://doi.org/10.1063/1. 5074179 [Hakim2006] Hakim, A., Loverich, J., & Shumlak, U. (2006). A high resolution wave propagation scheme for ideal Two-Fluid plasma equations. Journal of Computational Physics, 219, 418–442. https://doi.org/10.1016/j. jcp.2006.03.036 [Hakim2008] Hakim, A. H. (2008). Extended MHD modeling with the ten-moment equations. Journal of Fusion Energy, 27, 36–43. https://doi.org/10.1007/s10894-007-9116-z [Wang2020] Wang, L., Hakim, A. H., Ng, J., Dong, C., & Germaschewski, K. (2020). Exact and locally implicit source term solvers for multifluid-Maxwell systems. Journal of Computational Physics. https://doi.org/10. 1016/j.jcp.2020.109510 [Birn2001] Birn, L., et al. (2001). Geospace Environmental Modeling ({GEM}) magnetic reconnection challenge. Journal of Geophysical Research, 106, 3715. [Wang2015] Wang, L., Hakim, A. H. A. H., Bhattacharjee, A., & Germaschewski, K. (2015). Comparison of multifluid moment models with particle-in-cell simulations of collisionless magnetic reconnection. Physics of Plasmas, 22(1), 012108. https://doi.org/10.1063/1.4906063 [Juno2018] Juno, J., Hakim, A., TenBarge, J., Shi, E., & Dorland, W.. “Discontinuous Galerkin algorithms for fully kinetic plasmas”, Journal of Computational Physics, 353, 110–147, 2018. http://doi.org/10.1016/j.jcp.2017. 10.009 [Pusztai2018] I Pusztai, J M TenBarge, A N Csapó, J Juno, A Hakim, L Yi and T Fülöp “Low Mach-number collisionless electrostatic shocks and associated ion acceleration”. Plasma Phys. Control. Fusion 60 035004, 2018. https://doi.org/10.1088/1361-6587/aaa2cc [Cagas2017] P. Cagas, A. Hakim, W. Scales, and B. Srinivasan, “Nonlinear saturation of the Weibel instability. Physics of Plasmas”, 24 (11), 112116–8, 2017 http://doi.org/10.1063/1.4994682

199 gkyl Documentation, Release 2.0-alpha

[Shi2017] Shi, E. L., Hammett, G. W., Stolzfus-Dueck, T., & Hakim, A. (2017). Gyrokinetic continuum simulation of turbulence in a straight open-field-line plasma. Journal of Plasma Physics, 83, 1–27. http://doi.org/10. 1017/S002237781700037X [Hakim+2006] Hakim, A., Loverich, J., & Shumlak, U. (2006). A high resolution wave propagation scheme for ideal Two-Fluid plasma equations. Journal of Computational Physics, 219, 418–442. https://doi.org/10.1016/j. jcp.2006.03.036 [Hakim2008] Hakim, A. H. (2008). Extended MHD modeling with the ten-moment equations. Journal of Fusion Energy, 27, 36–43. https://doi.org/10.1007/s10894-007-9116-z [Wang+2020] Wang, L., Hakim, A. H., Ng, J., Dong, C., & Germaschewski, K. (2020). Exact and locally implicit source term solvers for multifluid-Maxwell systems. Journal of Computational Physics. https://doi.org/10. 1016/j.jcp.2020.109510 [Wang+2015] Wang, L., Hakim, A. H. A. H., Bhattacharjee, A., & Germaschewski, K. (2015). Comparison of multifluid moment models with particle-in-cell simulations of collisionless magnetic reconnection. Physics of Plasmas, 22(1), 012108. https://doi.org/10.1063/1.4906063 [Gross1956] E. P. Gross & M. Krook. Model for collision precesses in gases: small-amplitude oscillations of charged two-component systems. Physical Review, 102(3), 593–604 (1956). [Greene1973] J. M. Greene. Improved Bhatnagar-Gross-Krook model of electron-ion collisions. Physics of Fluids, 16(11), 2022–2023 (1973). [Dougherty1964] J. P. Dougherty. Model Fokker-Planck Equation for a Plasma and Its Solution. Physics of Fluids, 7(11), 1788–1799 (1964). [Hakim2020] A. Hakim, et al. (2020). Conservative Discontinuous Galerkin Schemes for Nonlinear Fokker-Planck Collision Operators. Journal of Plasma Physics Vol 86 No. 4, 905860403 (2020), arXiv:1903.08062. [Francisquez2020] M. Francisquez, et al. (2020). Conservative discontinuous Galerkin scheme of a gyro-averaged Dougherty collision operator. Nuclear Fusion 60 No. 9, 096021 (2020), arxiv:2009.06660. [Wersal2015] Wersal, C., & Ricci, P. (2015). A first-principles self-consistent model of plasma turbulence and kinetic neutral dynamics in the tokamak scrape-off layer. Nuclear Fusion, 55(12), 123014. [Voronov1997] Voronov, G. S. (1997). A practical fit formula for ionization rate coefficients of atoms and ions by electron impact: Z = 1-28. Atomic Data and Nuclear Data Tables, 65(1), 1–35. [Cagas2017] Cagas, P., Hakim, A., Juno, J., & Srinivasan, B. (2017). Continuum kinetic and multi-fluid simulations of classical sheaths. Phys. Plasmas, 24(2), 22118. [Meier2012] Meier, E. T., & Shumlak, U. (2012). A general nonlinear fluid model for reacting plasma-neutral mix- tures. Physics of Plasmas, 19(7). [Shi2015] Shi, E. L., Hakim, A. H., & Hammett, G. W. (2015). A gyrokinetic one-dimensional scrape-off layer model of an edge-localized mode heat pulse. Physics of Plasmas, 22(2). [Ng2017] J. Ng, A. Hakim, A. Bhattacharjee, A. Stanier, & W. Daughton “Simulations of anti-parallel reconnection using a nonlocal heat flux closure”. Physics of Plasmas, 24 (8), 082112–5. (2017) http://doi.org/10.1063/ 1.4993195 [Arnold2011] Arnold, D. N. and Awanou, G. “The serendipity family of finite elements.” Foundations of Computa- tional Mathematics 11.3 (2011): 337-344. [DurranBook] Dale E. Durran, “Numerical Methods for Fluid Dynamics”, Springer. Second Edition. [munz_2000] C.-D Munz, P. Omnes, R. Schneider and E. Sonnendruer and U. Voss, “Divergence Correction Tech- niques for Maxwell Solvers Based n a Hyperbolic Model”, Journal of Computational Physics, 161, 484- 511, 2000.

200 Bibliography gkyl Documentation, Release 2.0-alpha

[munz_2000b] C.-D Munz, P. Omnes, and R. Schneider, “A three-dimensional ﬁnite-volume solver for the Maxwell equations with divergence cleaning on unstructured meshes”, Computer Physics Communications, 130, 83-117, 2000. [munz_2000c] C.-D Munz and U. Voss, “A Finite-Volume Method for the Maxwell Equations in the Time Domain”, SIAM Journal of Scientiﬁc Computing, 22, 449-475, 2000. [Kulikovskii2001] Andrei G. Kulikoviskii and Nikolai V. Pogorelov and Andrei Yu. Semenov, Mathematical Aspects of Numerical Solutions of Hyperbolic Systems, Chapman and Hall/CRC, 2001. [Green1973] John M. Greene. Improved Bhatnagar-Gross-Krook model of electron-ion collisions. The Physics of Fluids, 16(11):2022-2023, 1973.

Bibliography 201