Nonhydrostatic Modeling with NICAM HIROFUMI TOMITA Research Institute for Global Change, JAMSTEC Advanced Institute for Computational Science, RIKEN Contents Numerical method on NICAM Dynamical core Horizontal discretization • Icosahedral grid – Modified by spring dynamics( Tomita et al. 2001, 2002 JCP) Nonhydrostatic framework • Mass and total energy conservation – ( Satoh 2002, 2003 MWR ) Computational strategy Domain decomposition in parallel computer Example of computational performance Problems Current problems Future problems Summary NICAM project NICAM project ( ~2000 ) Aim : • Construction of GCRM for climate simulation – Suitable to the future computer environment » Massively parallel supercomputer » First target : Earth Simulator 1 40TFLOPS in peak 640 computer nodes Strategy: • Horizontal discretization : – Use grid of icosahedarl grid (quasi-homogeneous) – Anyway, 2nd accuracy over the globe! • Dynamics: – Non-hydrostatic » Mass & total energy conservation • Physics : – Import from MIROC ( one of Japan IPCC models ) except for microphysics. Horizontal discretization Grid arrangement Arakawa A-grid type Velocity, mass • triangular vertices Control volume • Connection of center of triangles – Hexagon – Pentagon at the icosahedral vertices Advantage Easy to implement no computational mode • Same number of grid points for vel. and mass Disadvantage Non-physical Glevel-3 grid & control volume 2-grid scale structure • E.g. bad geostrophic adjustment Horizontal differential operator e.g. Divergence 1. ector : given at VPi u()Pi 2. Interpolation of u at Qi u()()()αPPP+ β u+ γ u u()Q ≈ 0 i 1+ mod(i , 6 ) i α+ β + γ 3. Gauss theorem 6 1 u()()QQi +1 u + mod(i , 6 ) ∇ •u()P0 ≈ ∑bi •ni AP()0 i=1 2 2nd order accuracy? NO Æ Allocation points is not gravitational center ( default grid ) Modified Icosahedral Grid (1) Reconstruction of grid by spring dynamics To reduce the grid-noise 1. STD-grid : Generated by the recursive grid division. 2. SPRING DYNAMICS : Connection of gridpoints by springs 6 dw0 ∑k()− die d i −α w0 M = i=1 dt dr w = 0 0 dt SPR-grid: Solve the spring dynamics Æ The system calms down to the static balance Modified Icosahedral Grid (2) Gravitational-Centered Relocation To make the accuracy of numerical operators higher 1. SPR-grid: Generated by the spring dynamics. Æ ● 2. CV: Defined by connecting the GC of triangle elements. Æ ▲ 3. SPR-GC-grid: The grid points are moved to the GC of CV. Æ ● Æ The 2nd order accuracy of numerical operator is perfectly guaranteed at all of grid points. Improvement of accuracy of operator est functionT Heikes & Randall(1995) Error norm Global error 1 / 2 I(,)(,)λ() x θ− x λ2 θ l() x = { [ T ]} 2 2 1 / 2 I{}[] x(,)Tλ θ Local error maxallλ,λx θ ( θ ,− )T x λ ( θ , ) l∞() x = maxallλ, θx(,)λ T θ STD-grid SPR-GC-grid L_2 norm Almost 2nd-order(●) Perfect 2nd-order(○) l_inf norm 2nd order(Not ▲) Perfect 2nd-order(△) Nonhydrostatic framework Design of our non-hydrostatic modeling Governing equation Full compressible system • Acoustic wave Æ Planetary wave Flux form • Finite Volume Method • Conservation of mass and energy Deep atmosphere • Including all metrics terms and Coriolis terms Solver Split explicit method • Slow mode : Large time step • Fast mode : small time step HEVI ( Horizontal Explicit & Vertical Implicit ) • 1D-Helmholtz equation Governing Equations Å MODE STAFL.H.S. : Æ Å MODE R.H.S. : SLOWÆ ∂ Vh ∂ ⎛ W 3 Vh ⎞ +R ∇h ⋅ +⎜ +G ⋅ ⎟ = 0 ⎜ 1 / 2 ⎟ (1) ∂t γ∂ ξ⎝ G γ ⎠ ∂ P ∂ ⎛ 3 P ⎞ +Vh ∇ h + ⎜G ⎟ =ADVh + Coriolis F (2) ∂t γ∂ ξ⎝ ⎠ γ ∂ ∂ ⎛ P ⎞ W + γ 2 ⎜ +⎟ Rg =ADV + F (3) ⎜ 1 / 2⎟ 2 z Coriollis ∂t ∂ξ⎝ G γ ⎠ ∂ ⎛ Vh ⎞ ∂ ⎡ ⎛ W 3 Vh ⎞⎤ Eh ⎜ h ⎟ h⎜ G ⎟ + ∇⎜ ⋅ ⎟ + ⎢ ⎜ 1 /+ 2 ⋅ ⎟⎥ ∂t ⎝ γ⎠ ∂ ξ⎣ ⎝ G γ ⎠⎦ Vh ⎡ P ∂ ⎛ PW3 ⎞⎤ 2 ∂ ⎛ P ⎞ − ⋅ ∇h +⎜G ⎟ − γ ⎜ +⎟Wg = Q ⎢ ⎜ ⎟⎥ ⎜ 1 / 2⎟ 2 heat (4) R ⎣ γ∂ ξ⎝ ⎠ γ⎦ R ∂ξ⎝ G γ ⎠ Prognostic variables Metrics • density RG=γ2 1 / 2ρ 1 / 2 ⎛ ∂z ⎞ G = ⎜ ⎟ 2 1 / 2 ⎝ ∂ξ ⎠ • horizontal momentum Vh =γG ρvh x, y 3 • vertical momentum W=γ2 G 1 / 2ρ w G=() ∇hξ z 2 1 / 2 H() z− z • internal energy E=γ G ρ in e ξ = s H− s z Temporal Scheme ( in the case of RK2) Assumption : the variable at t=A is known. ¾ Obtain the slow mode tendency S(A). HEVI solver 1. 1st step : Integration of the prog. var. by using S(A) from A to B. ¾ Obtain the tentative values at t=B. ¾ Obtain the slow mode tendency S(B) at t=B. 2. 2nd step : Returning to A, Integration of the prg.var. from A to C by using S(B). Æ Obtain the variables at t=C Small Step Integration step integration, there In small are 3 steps: 1. Horizontal Explicit Step ¾ of horizontal momentumUpdate HEVI 2. Implicit StepVertical ¾ of vertical Updates momentum and density. 3. Energy Correction Step ¾ Update of energy Horizontal Explicit Step • explicitly momentum is updated byHorizontal ⎡ t+ n Δτ [t , or+ t Δ / 2t ⎤ ] +t( n + 1 )τ Δ t+ n Δτ ⎛ P ∂ ⎛ 3 P ⎞⎞ ⎛ ∂Vh ⎞ Vh =Vh + Δτ ⎢⎜ −h ∇⎜G − ⎟⎟ + ⎜ ⎟ ⎥ ⎜ γ∂ ξ⎜ ⎟ γ⎟ ∂t ⎣⎢⎝ ⎝ ⎠⎠ ⎝ slow⎠ mode⎦⎥ Fast mode Slow mode : given Small Step Integration (2) Step Implicit Vertical • The equations of R,W, and E can be written as: RR+t( n + 1 )τ Δ− t+ n Δτ ∂ ⎛W+t( n + 1 )τ Δ⎞ + ⎜ ⎟ = G ⎜ 1 / 2 ⎟ R (6) Δτ ∂ξ ⎝ G ⎠ W+t( n + 1 )τ Δ−Wt+ n Δτ ∂ ⎛ P+t( n + 1 )τ Δ⎞ + γ 2 ⎜ ⎟R+ +t( n + 1 )τ Δg= G (7) ⎜ 1 / 2 2⎟ z Δτ ∂ξ⎝ G γ ⎠ PP+t( n + 1 )τ Δ− t+ n Δτ ∂ ⎡⎛W+t( n + 1 )τ Δ⎞ ⎤ R R + ⎜ ⎟c2t+ n Δτ + dW+t( n + 1 )τ Δ~ g = d G (8) ⎢⎜ 1 / 2 ⎟ s ⎥ E Δτ ∂ξ ⎣⎝ G ⎠ ⎦ CV CV – Eqs.(6), (7), Coupling and (8), we can obtain the 1D-Helmholtz equation for W： W+t( n + 1 )τ Δ ∂ ⎡ 1 ∂ ⎛ W+t( n + 1 )τ Δ⎞⎤ ⎡ ∂ ⎛ R W+t( n + 1 )τ Δ⎞⎤ g∂ ⎛ W+t( n + 1 )τ Δ⎞ − ⎜Δτ2c t+ 2 n Δτ ⎟ − ⎜Δτ 2 d g~ ⎟+ Δτ 2 ⎜ ⎟ 2 ⎢1 / 2 2 ⎜ s 1 / 2 ⎟⎥ ⎢ ⎜ 1 / 2 2 ⎟⎥ 2 ⎜ 1 / 2 ⎟ γ ξ∂ ⎣G γ∂ ⎝ ξ G ⎠⎦ ⎣∂ξ ⎝ CV G γ ⎠⎦ γ∂ ξ⎝ G ⎠ = R.H.S.(source term) (9) • Eq.(9) Æ W • Eq.(6) Æ R • Eq.(8) Æ E Small Step Integration (3) Energy Correction Step （Total eng.）＝（Internal eng.）＋（Kinetic eng.）＋（Potential eng.） • consider the equation of total energyWe ∂ ⎡ Vh ⎤ ∂ ⎡ ⎛ W 3 Vh ⎞⎤ +E ∇ ⋅h h + k + Φ+ +h + k Φ⎜ G +⎟ ⋅ = 0 total ⎢()⎥ ⎢()⎜ 1 / 2 ⎟⎥ (10) ∂t ⎣ γ⎦ ∂ ξ⎣ ⎝ G γ ⎠⎦ where 2 1 / 2 Etotal = ργG(+ ein + k ) Φ • Additionally, Eq.(10) is solved as +t( n + 1 )τ Δ t+ n Δτ Etotal = Etotal +t( n + 1 )τ Δ ⎡ ⎡ Vh ⎤ ∂ ⎡ ⎛ W 3 Vh ⎞⎤⎤ − Δτ ∇h ⋅h + k ++ Φ +h + k Φ⎜ G +⎟ ⋅ ⎢ ⎢()⎥ ⎢()⎜ 1 / 2 ⎟⎥⎥ ⎣⎢ ⎣ γ⎦ ∂ ξ⎣ ⎝ G γ ⎠⎦⎦⎥ – Written by a flux form. • The kinetic energy and potential energy: Æ known by previous step. • Recalculate the internal energy: +t( n + 1 )τ Δ +t( + n 1 Δ )τ(t 1 )+ n +τ 2 Δ 1 / 2+t n + (τ Δ 1 ) E =Etotal ρ − G γ ( k + Φ) Large Step Integration Large step tendecy has 2 main parts: 1. Coliolis term ¾ Formulated straightforward. 2. Advection term • We should take some care to this term because of curvature of the earth Advection of momentum the Use of a catesian coordinate in which the origin is center of the earth. The advection of term Ｖｈ and W is calculated as follows. 1. Construct the 3-dimensional momentum V using Vh and W. 2. this vector as 3 components as (Express V1,V2,V3) in a fixed coordinate. ¾ These components are scalars. 3. Obtain a vector which contains 3 divergences as its components. Æ 1 / 2 2 ∇ ⋅()vVVV1 ,, ∇ v ⋅2 3 ∇ v ⋅ wherevi= V i ( γ G / ρ) 4. Split again to a horizontal vector and a vertial components. Æ ADVh , ADVz Computational strategy and performance Computational strategy(1) (0) region division level 0 (1) region division level 1 Domain decomposition 1. By connecting two neighboring icosahedral triangles, 10 rectangles are constructed. (rlevel-0) 2. For each of rectangles, 4 sub-rectangles are generated by connecting (2) region division level 2 (3) region division level 3 the diagonal mid-points. （rlevel-1) 3. The process is repeated. (rlevel-n) Computational strategy(2) Load balancing Example ( rlevel-1 ) # of region : 40 # of process : 10 Situation: Polar region: Less computation Equatorial region: much computation Each process manage same color regions Cover from the polar region and equatorial region. Avoid the load imbalance Computational strategy(3) Vectorization Structure in one region Icosahedral grid Æ Unstructured grid? Treatment as structured grid Æ Fortran 2D array Æ vectorized efficiently! 2D array Æ 1D array Higher vector operation length Computational Performance (1) Computational performance Depend on… • Computer architecture, degree of code tuning….. Performance on the old Earth Simulator Earth Simulator • Massively parallel super-computer based on NEC SX-6 architecture. – 640 computational nodes. – 8 vector-processors in each of nodes. – Peak performance of 1CPU : 8GFLOPS – Total peak performance： 8X8X640 = 40TFLOPS – Crossbar network Target simulations for the measurement • 1 day simulation of Held & Suarez dynamical core experiment Computational Performance (2) Scalability of our model (NICAM) --- strong scaling Configuration • Horizontal resolution : glevel-8 •30km resolution • Vertical layers : 100 Fixed • Computer nodes : increases from 10 to 80. Results Green ： ideal speed-up line Red ： actual speed-up line Computational Performance (3) Performance against the horizontal resolution --- weak scaling The elapse time should increase by a factor of 2. ｇlevel Number of PNs Elapse Time Average Time GFLOPS (grid intv.) (peak performance) [sec] [msec] (ratio to peak[%]) 6 5 48.6 169 140 (120km) (320GFLOPS) (43.8) 7 20 97.4 169 558 (60km) (1280GFLOPS) (43.6) 8 80 195 169 2229 (30km) (5120GFLOPS) (43.5) 9 320 390 169 8916 (15km) (20480GFLOPS) Configuration As the grid level increases, # of gridpoints : X 4 Results # of CPUs : X 4 The elapse time increases by a factor of 2.