Multi-Particle Simulation Techniques I Ji Qiang Accelerator Modeling Program Accelerator Technology & Applied Physics Division Lawrence Berkeley National Laboratory CERN Accelerator School, Thessaloniki, Greece Nov. 15, 2018 Outline • Introduction of the particle-in-cell method for multi-particle simulation • Deposition/interpolation • Self-consistent field calculations • FFT based Green function method for open boundary condition • Multigrid method for irregular shape boundary condition • Particle advance Introduction • Particle-in-cell method: “the method amounts to following the trajectories of charged particles in self-consistent electromagnetic (or electrostatic) fields computed on a fixed mesh” • Particle-in-cell codes are widely used in accelerator physics community: • PARMELA, ASTRA, GPT, IMPACT-T, IMPACT-Z, GENESIS, GINGER…. • An example of 2D particle-in-cell simulation an mismatched beam transport through a FODO Courtesy of R. D. Ryne Governing Equations in Space-Charge Simulation f (r, p,t) f (r, p,t) f (r, p,t) r p 0 t r p 2 / f (r, p,t)d 3 p wi (r ri )( p pi ) One Step Particle-In-Cell Method particle deposition field calculation field particle advance field interpolation Particle Deposition/Field Interpolation Particle deposition: p p p qi w(xi xp ) i q p i p J p qi w(xi x p )v i i Field interpolation: Ei E p w(xi xp ) p Bi B p w(xi xp ) p • Grid reduces the computational cost compared with direct N-body point-to-point interaction • Grid also provides smoothness to the shot noise and close collision Weight Function for Deposition/Interpolation • Spatial localization of errors - At particle separations large compared with the mesh spacing, the field error should be small • Smoothness - The charge assigned to the mesh from a particle and the force interpolated to a particle a particle from the mesh should smoothly vary as the particle moves across the mesh • Momentum conservation - No self force Weight Function for Deposition/Interpolation • Spatial localization of errors q m m j (x j ) wp (xi )G(x j xp ) wp (xi )G(x j xi xi xp ) p1 p1 m m (x x )n d nG(x x ) p p (x ) w (x )G(x x ) w (x ) i p j i j p i j i p i n p1 p1 n1 n! dx qi m m p p w (x )(x x )n const wp (xi ) 1 p i i p p1 p1 • Smoothness: • Continuity of weight function value • Continuity of derivative -1 0 x 1 -h 0 h • Momentum conservation m m Fself (xi ) d(xp ; xp' )wdep (xi x p' )wint (xi xp ) p1 p'1 d(x p ; x p' ) d(x p'; x p ) Weight Functions for Deposition/Interpolation NGP: x x w (x) ( p ) p h CIC: x x w (x) 1 | p | p h TSC: 3 x xp 2 ( ) 4 h wp (x) 1 3 x x ( | p |) 2 2 2 h x R. W. Hockney and J. W. Eastwood, Computer Simulation Using Particles Self-Consistent Field Calculations 2 / f (r, p)d 3 p Different Boundary/Beam Conditions Need Different Efficient Numerical Algorithms O(Nlog(N)) or O(N) FFT based Green function method: • Standard Green function: low aspect ratio beam • Shifted Green function: separated particle and field domain • Integrated Green function: large aspect ratio beam • Non-uniform grid Green function: 2D radial non-uniform beam Fully open boundary conditions Spectral-finite difference method: Transverse regular pipe with longitudinal open/periodic Multigrid spectral-finite difference method: TransverseTransverse irregularirregular pipepipe Field Calculation with Open Boundary Conditions Green Function Solution of Poisson’s Equation (r) G(r,r')(r')dr' ; r = (x, y,z) N (ri) hG(ri ri' )(ri' ) i '1 G(x, y, z) 1/ (x2 y2 z 2 ) Direct summation of the convolution scales as N2 !!!! N – total number of grid points Hockney’s Algorithm or Zero Padding Gc(r) G(r) G(-r) G(r) G(-r) (r) L • This is different from a real periodic system • The real calculation is done in discrete coordinate instead of continuous coordinate R. W. Hockney and J. W. Eastwood, Computer Simulation Using Particles Hockney’s Algorithm or Zero Padding A Schematic Plot of an e- Beam and Its Image Charge y + e o e- z cathode Shifted Green function Algorithm: F (r) Gs(r,r')(r')dr' Gs(r,r') G(r rs,r') J. Qiang, M. Furman, and R. Ryne, Phys. Rev. ST Accel. Beams, vol 5, 104402 (October 2002). Test of Image Space-Charge Calculation Numerical Solution vs. Analytical Solution Shifted-green function Analytical solution Integrated Green Function for Large Aspect Ratio Beam G(r) • Lack of resolution along longer side if same number of grids are used for both sides • Brute force: use more grid points along longer side • Better way: break the original convolution integral into a sum of small cell integral and use integrated Green’s function within each cell Integrated Green Function 2N c(ri) Gi(ri ri' )c(ri' ) i '1 Gi(r,r') Gs(r,r')dr' J. Qiang, S. Lidia, R. D. Ryne, and C. Limborg-Deprey, Phys. Rev. ST Accel. Beams, vol 9, 044204 (2006). A Comparison Example: Aspect Ratio = 30 Poisson Solver with Finite Boundary Conditions j-1 j j+1 Where j = 1, …,N Finite Difference Poisson Solver: Iterative Methods A: is a sparse matrix • Direct Gaussian elimination: O(N3) • Iterative method: O(mN) Where S is an approximation of A-1, i = 1,…,m Finite Difference Poisson Solver: Iterative Methods • Classical iterative methods: – Jacobi – Damped Jacobi – Gauss-Seidel – Successive Over Relaxation • Problems of classical iterative methods: – slow convergence Weighted Jacobi Chosen to Damp High Frequency Error Initial error “Rough” Lots of high frequency components Norm = 1.65 Error after 1 Jacobi step “Smoother” Less high frequency component Norm = 1.055 Error after 2 Jacobi steps “Smooth” Little high frequency component Norm = .9176, won’t decrease much more Ref: J. Demmel’s class note Multigrid Motivation • Classical sparse-matrix-vector-multiply-based algorithms: o low frequency error decreases slowly after a few iterations o move information one grid at a time o take N1/d steps to get information across grid o matrix-vector multiplications are done on the full fine grid • Multigrid algorithm: o smooths out the numerical errors of different frequencies on different scales using multiple grids o moves the information across grid by W(log n) steps o most multipications are done on coarse grids Comparison of Convergent Time for an Example: SOR vs. Multi-Grid Multi-Grid Iteration Method • Basic Algorithm: – Replace correction problem on fine grid by an approximation on a coarser grid – Solve the coarse grid problem approximately, and use the solution as a correction to the fine grid problem and build a new starting guess for the fine-grid problem, which is then iteratively updated – Solve the coarse grid problem recursively, i.e. by using a still coarser grid approximation, etc. • Success depends on coarse grid solution being a good approximation to the fine grid Multigrid Sketch on a Regular 1D Mesh • Consider a 2m+1 grid in 1D for simplicity • Let P(i) be the problem of solving the discrete Poisson equation on a 2i+1 grid in 1D • Write linear system as A(i) * x(i) = b(i) • P(m) , P(m-1) , … , P(1) is a sequence of problems from finest to coarsest Multigrid Sketch on a Regular 2D Mesh • Consider a 2m+1 by 2m+1 grid • Let P(i) be the problem of solving the discrete Poisson equation on a 2i+1 by 2i+1 grid in 2D o Write linear system as A(i) * x(i) = b(i) • P(m) , P(m-1) , … , P(1) is a sequence of problems from finest to coarsest Basic Operators of Multigrid Iteration • For problem P(i) : o b(i) is the RHS and o x(i) is the current estimated solution both live on grids of size 2i-1 o (A(i) is implicit in the operators below.) • All the following operators just average values on neighboring grid points o Neighboring grid points on coarse problems are far away in fine problems, so information moves quickly on coarse problems • The restriction operator R(i) maps P(i) to P(i-1) o Restricts problem on fine grid P(i) to coarse grid P(i-1) by sampling or averaging o b(i-1)= R(i) (b(i)) o Graphic representation: • The prolongation operator P(i-1) maps an approximate solution x(i-1) to an x(i) o Interpolates solution on coarse grid P(i-1) to fine grid P(i) o x(i) = P(i-1)(x(i-1)) o Graphic representation: • The smooth operator S(i) takes P(i) and computes an improved solution x(i) on same grid o Uses “weighted” Jacobi or Gauss-Seidel o x improved (i) = S(i) (b(i), x(i)) o Graph representation: Restriction Operator R(i) - Details • The restriction operator, R(i), takes o a problem P(i) with RHS b(i) and o maps it to a coarser problem P(i-1) with RHS b(i-1) • Simplest way: sampling • Averaging values of neighbors is better; in 1D this is o xcoarse(i) = 1/4 * xfine(i-1) + 1/2 * xfine(i) + 1/4 * xfine(i+1) • In 2D, average with all 8 neighbors (N,S,E,W,NE,NW,SE,SW) Prolongation/Interpolation Operator • The prolongation/interpolation operator P(i-1), takes a function on a coarse grid P(i-1) , and produces a function on a fine grid P(i) • In 1D, linearly interpolate nearest coarse neighbors o xfine(i) = xcoarse(i) if the fine grid point i is also a coarse one, else o xfine(i) = 1/2 * xcoarse(left of i) + 1/2 * xcoarse(right of i) • In 2D, interpolation requires averaging with 2 or 4 nearest neighbors (NW,SW,NE,SE) Two-Grid Iteration • Pre-smoothing: compute approximated solution by applying n1 steps of a relaxation method on fine grid: i1 i i X (2) X (2) S(B AX (2)); i 1,,n1 • Construct residual vectors: r(2) B AX n1 1(2) • Restrict the residual to coarser grid: r(1) R(r(2)) • Solve exactly on the coarse grid for the error vector: E(1) A1R(r(1)) • Prolongate/Interpolate the error vector to fine grid: E(2) P(E(1)) • Compute