A New Dynamical Core of the Global Environmental Multiscale (GEM) Model with a Height-Based Terrain-Following Vertical Coordinate

JULY 2019 H U S A I N E T A L . 2555

SYED ZAHID HUSAIN,CLAUDE GIRARD, AND ABDESSAMAD QADDOURI Atmospheric Numerical Prediction Research Section, Meteorological Research Division, Environment and Climate Change Canada, Dorval, Quebec, Canada

ANDRÉ PLANTE Canadian Meteorological Centre, Environment and Climate Change Canada, Dorval, Quebec, Canada

(Manuscript received 20 December 2018, in ﬁnal form 7 May 2019)

ABSTRACT

A new dynamical core of Environment and Climate Change Canada’s Global Environmental Multiscale (GEM) atmospheric model is presented. Unlike the existing log-hydrostatic-pressure-type terrain- following vertical coordinate, the proposed core adopts a height-based approach. The move to a height- based vertical coordinate is motivated by its potential for improving model stability over steep terrain, which is expected to become more prevalent with the increasing demand for very high-resolution forecasting systems. A dynamical core with height-based vertical coordinate generally requires an iterative solution approach. In addition to a three-dimensional iterative solver, a simpliﬁed approach has been devised allowing the use of a direct solver for the new dynamical core that separates a three- dimensional elliptic boundary value problem into a set of two-dimensional independent Helmholtz problems. The issue of dynamics–physics coupling has also been studied, and incorporating the physics tendencies within the discretized dynamical equations is found to be the most acceptable approach for the height-based vertical coordinate. The new dynamical core is evaluated using numerical experiments that include two-dimensional nonhydrostatic theoretical cases as well as 25-km resolution global forecasts. For a wide range of horizontal grid resolutions—from a few meters to up to 25 km—the results from the direct solution approach are found to be equivalent to the iterative approach for the new dynamical core. Furthermore, results from the different numerical experiments conﬁrm that the new height-based dynamical core is equivalent to the existing pressure-based core in terms of solution accuracy.

1. Introduction 2014). The existing pressure-type coordinate system then permits the use of a direct solver for the discretized The dynamical core of the Global Environmental Mul- EBV problem to resolve the dynamical component of tiscale (GEM) model, used operationally by Environ- the flow. The direct solver starts by separating the EBV ment and Climate Change Canada (ECCC) for numerical problem vertically in terms of the vertical eigenvec- weather prediction (NWP), employs a log-hydrostatic- tors of the part of the coefficient matrix that only in- pressure-type terrain-following vertical coordinate. The cludes the discretized difference and average operators system of nonlinear model equations is linearized in the vertical direction (Qaddouri and Lee 2010). For N around a basic state and is then reduced to an elliptic number of model vertical levels, the resulting N verti- boundary value (EBV) problem through numerical cally decoupled two-dimensional Helmholtz problems discretization and elimination of variables (Girard et al. are then separated along the longitude, leading to a system of tridiagonal problems depending only on the Denotes content that is immediately available upon publica- latitude. The tridiagonal problems are finally solved tion as open access. using LU decomposition without pivoting. Such an approach is computationally more efficient than most it- Corresponding author: Syed Zahid Husain, syed.husain@ erative methods, particularly for the spatial resolutions canada.ca of the current operational NWP systems at ECCC.

DOI: 10.1175/MWR-D-18-0438.1 For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/ PUBSReuseLicenses). Unauthenticated | Downloaded 09/25/21 04:19 PM UTC 2556 MONTHLY WEATHER REVIEW VOLUME 147

One of the principal incentives for the adoption of has been found to lead to any meaningful stability im- the existing pressure-type vertical coordinate in GEM provement for steep orography. is the computational advantage of the direct solution Although the instability induced by steep orogra- approach that is permissible with such a coordinate. phy is often characterized as a limitation of the terrain- However, numerical experiments carried out at ECCC following nature of the vertical coordinate itself, have revealed that vertical separability, which is an Smolarkiewicz et al. (2007) were successful in re- imperative for the direct solution approach, can be- solving flow around the Pentagon with a model in- come quite restrictive for very high spatial resolutions volving height-based TFC where the maximum slope (e.g., for subkilometer horizontal grid spacing). Fur- was well above the widely acknowledged 458 threshold. thermore, the reduction of the 2D Helmholtz problems Previous experience with the Mesoscale Compress- into the 1D tridiagonal problems requires projection ible Community model at ECCC also suggests that a of right hand side (rhs) of the 2D problems along dynamical core with height-based TFC does not suf- the pertinent eigenvectors which is based on Fourier fer from similar severe orography-induced instability transformation. This necessitates transposing the (Girard et al. 2005). Apparently, a dynamical core coefficient matrix that involves global communica- with a height-based TFC can lead to improved nu- tions, and therefore, becomes inefficient for very large merical stability through better implicit treatment of number of processor cores. As a result, the direct solver the metric terms arising from the vertical coordinate is found to lose scalability with increasing number transformation through iterative solvers. More impor- of processor cores. Initial research at ECCC as well tantly, conventional numerical approximation of the as published research literature (Müller and Scheichl horizontal gradients in a TFC becomes less accurate 2014) reveal that optimized three-dimensional iterative with increasing terrain slope as well as with increasing solvers may possess better scalability in these circum- vertical resolution close to the model surface (Mahrer stances. The different limitations of the existing di- 1984). In this context, Zängl (2012) argues that the rect solver, particularly its potential lack of scalability pressure gradient term, in particular, becomes suscep- for future generations of massively parallel supercom- tible to triggering numerical instability when the height puters, therefore warrants the development of more difference between two adjacent grid points along a scalable iterative solvers at ECCC. A height-based terrain-following vertical level is much larger than the vertical coordinate is considered to be more amenable vertical grid resolution adjacent to the level. Numeri- to such iterative solvers as the metric terms originating cal approximation of the horizontal gradient terms in from the vertical coordinate transformation appear ex- theTFC,however,canbesignificantly improved fol- plicitly in the discretized EBV. lowing the approach proposed by Mahrer (1984). This Another, but more challenging, problem pertaining approach requires determination of the modified hor- to the existing GEM dynamical core is the fact that the izontal differencing stencils associated with each grid- current model exhibits strong numerical instability over point location that minimize the error in the metric steep orography. Tests conducted at ECCC have de- corrections for the terrain-following nature of the co- termined the maximum permissible terrain slope for ordinate. The existing pressure-type TFC varies with maintaining stability to be approximately 458 (Vionnet time and, therefore, would require determination of et al. 2015). This has been considered to be a limitation the pertinent gridpoint locations in the vertical for the inherent to the terrain-following coordinate (TFC) sys- modified differencing stencils at every time step. On tems (Zängl 2012). With a growing demand for very the contrary, the height-based TFC is time-invariant high-resolution operational NWP systems, steep oro- and thus would require the determination of these graphic slopes are expected to become more preva- gridpoint locations only once at the beginning of the lent in the near future. Improving model stability over time integration. Therefore, from a computational ef- steep mountains is therefore of critical importance for ficiency standpoint, a height-based TFC is more suitable the future development of subkilometer NWP systems. for implementing improved numerical approximation of A number of approaches have been investigated to im- horizontal gradients to address instabilities induced by prove numerical stability with the existing pressure-type steep orography. vertical coordinate in GEM. These include increased The aforementioned challenges associated with the off-centering in the discretized vertical momentum existing log-hydrostatic-pressure-type TFC motivated equation, a vertically variable basic state temperature the development of a new dynamical core for the GEM profile, and modifications to the nonhydrostatic contri- model that utilizes a height-based TFC. The primary butions in the linear system arising from the discretized objective of the present study is to demonstrate that, GEM formulation. However, none of these approaches for the model configurations where orography-induced

Unauthenticated | Downloaded 09/25/21 04:19 PM UTC JULY 2019 H U S A I N E T A L . 2557 numerical instability is not relevant—that is, for hori- d lnr ›u ›y ›w tanf d lnr 1 1 1 2 y 5 , (4) zontal grid resolutions within the hydrostatic regime— dt ›x ›y ›z a dt phys the new dynamical core developed at ECCC with height- based TFC makes predictions that are equivalent to d lnT d lnp d lnT 2 k 5 , (5) those from the existing model. Another major objec- dt dt dt phys tive is to propose a strategy that enables the utiliza- tion of the direct solver to make the new dynamical where Eqs. (1)–(5) govern the evolutions of the u, y, core comparable to the existing one in terms of com- and w components of velocity, mass, and energy, re- putational performance for spatial resolutions asso- spectively. The spatial coordinates in the above equa- ciated with the different GEM-based operational tions are denoted by (x, y, z), which are related to the NWP systems. The present study also explores the spherical coordinate (l, f, r) through the differential appropriate strategy for coupling the operational relations given by Physics Parameterization Package (PPP) of RPN 5 5 5 (Recherche en prévision numérique) with the new dx a cosfdl, dy adf, dz dr, (6) height-based dynamical core. Different setups for such that u, y, and w are the physical wind components. numerical experiments, covering both hydrostatic and In Eq. (6), a denotes Earth’s radius. The Lagrangian nonhydrostatic scenarios, are utilized to compare the derivative in this case can be expressed as two dynamical cores. The experiments include two- dimensional theoretical cases (Robert 1993; Schär d › › › › [ 1 u 1 y 1 w . (7) et al. 2002) as well as three-dimensional global fore- dt ›t ›x ›y ›z casts over a Yin–Yang grid (Qaddouri and Lee 2011). Relevant background information on the GEM dy- In addition to the four independent variables (x, y, z, t), namical core with the proposed height-based TFC— the system of five Eqs. (1)–(5) involves six dependent from the spatial and temporal discretizations to the variables, namely, the velocity components u, y, and w, derivation of the elliptic boundary value problem—is the temperature T, the pressure p, and the density r. presented in section 2. The different solution methods Also, in the above equations, f is the Coriolis parameter utilized for the discretized elliptic problem are discussed and k 5 R/cp, where R is the total gas constant and cp is in section 3. The issue of coupling between the dynam- the specific heat of moist air at constant pressure. The ical core and the parameterized physics forcings is pre- terms on the rhs of Eqs. (1)–(5) with subscript ‘‘phys’’ sented in section 4. Section 5 contains the comparisons denote the various physical forcings. Depending on the between the existing and the proposed dynamical cores equation, these physical forcings may arise from differ- in the context of two-dimensional theoretical bench- ent sources that include friction, diabatic heating and mark cases as well as three-dimensional deterministic frictional dissipation of kinetic energy. A sixth equation global predictions. The conclusions are then summa- is required to close the system described by the six de- rized in section 6. pendent variables, and is provided by the equation of state, given by 5 2. Model description p rRT . (8) a. Governing equations It is important to note that the atmospheric sub- stance does not only contain dry air but also water The GEM model equations originate from the Euler vapor and different types of hydrometeors. The dis- equations. With the traditional shallow atmosphere ap- placement and evolution of water vapor and hydro- proximation (Phillips 1966), the system of equations in a meteors in the atmosphere are governed by their own spherical coordinate (l, f, r) can be expressed as follows: evolution equations. However, they will also affect the du tanf 1 ›p du rhs terms through fluxes of water vapor and precipita- 2 f 1 u y 1 5 , (1) dt a r ›x dt tion which constitute sources of mass. The total air phys density in the presence of water in its different forms dy tanf 1 ›p dy can be expressed as 1 f 1 u u 1 5 , (2) dt a r ›y dt phys r 5 r 1 r 5 r (1 1 r ), (9) d w d w dw 1 ›p dw 1 1 g 5 , (3) where rd is the dry air density, rw is the density of water dt r ›z dt phys vapor and hydrometeors, and rw 5 rw/rd is the mixing

Unauthenticated | Downloaded 09/25/21 04:19 PM UTC 2558 MONTHLY WEATHER REVIEW VOLUME 147 ratio for the total water content of the atmosphere. The where z is the true geometric height, zs and zT are the equation of state is then strictly given by surface and the model top level heights, and H is a scaling constant. Equation (14) can be further general- 5 1 p (rdRd ryRy)T , (10) ized as where ry and Ry are the density and gas constant of 5 1 z A BzS , (15) water vapor. Equation (10) can then be further rearranged in terms of the total air density r as where A 5 (zT/H)z and B 5 (1 2 z/H). Assigning H 5 zT 5 5 implies zT zT and, as a result, Eq. (15) becomes p rRdTy , (11) 5 1 z z BzS , (16) where Ty is the virtual temperature of moist air which is given by which is similar to Eq. (13) in form. The vertical co- R ordinate for the proposed dynamical core in the present 1 1 y r study, however, is modified as R y T 5 d T , (12) y 1 1 r 5 1 1 2 w z z B1zSL B2(zS zSL), (17) 5 where ry ry/rd is the water vapor mixing ratio. Re- which follows the concept of the smooth level vertical writing the dynamical Eqs. (1)–(5) in terms of Ty is (SLEVE)-like coordinate system proposed by Schäretal. helpful as the equations can then be expressed using (2002),wherezSL denotes the large-scale components of only the dry gas constant that does not vary due to the the orography. The vertical coordinate defined by Eq. (17) atmospheric water content. It also allows to implicitly permits separate rates of flattening for the large and account for the effects of water vapor buoyancy and small-scale contributions of the orography on the terrain- condensed water loading. Furthermore, from the adia- following vertical coordinate with changing elevation batic point of view, the introduction of Ty has no con- through the metric terms B1 and B2 that are defined as sequence since the water content is conserved during r dynamical transport. z 2 z n B 5 T , (18) n 2 b. Vertical coordinate zT zS The log-hydrostatic-pressure-type terrain-following where rn 5 [rn,max 2 (rn,max 2 rn,min)lk] and lk 5 (z1 2 vertical coordinate of the operational GEM model zk)/(z1 2 zS). The values of rn,min and rn,max together (Girard et al. 2014) has the following form: determine the rate of flattening of the vertical levels with lnp 5 j 1 Bs, (13) increasing height. The subscript k of l indicates the model vertical level number. The value of k decreases with in- where j defines the terrain-following vertical co- creasing height above the surface such that k 5 1in- ordinate, p denotes the hydrostatic pressure, B is a dicates the top-most model level. The values of rn,min and metric term prescribing the rate of flattening of the rn,max are selected to maximize the latter and minimize vertical coordinate with elevation, and s 5 ln(ps/pref) the former in such a way that allows quick flattening of with ps being the hydrostatic pressure at the surface the terrain effects on the coordinate at the upper levels 3 and pref 5 10 hPa is a reference pressure. The definition while ensuring that the vertical levels are not excessively of this vertical coordinate follows the concept of the gen- squeezed near the surface over complex terrain. eralized hydrostatic-pressure-type hybrid coordinate pro- Henceforth, in this paper, the two dynamical cores with posed by Laprise (1992). Further details regarding the vertical coordinates based on log-hydrostatic-pressure log-hydrostatic-pressure-type vertical coordinate are pro- and height are referred to as GEM-P and GEM-H, re- vided by Girard et al. (2014) and Husain and Girard (2017). spectively. Different aspects of the GEM-P dynamical The present study proposes to develop a GEM dy- core, including the model formulation, discretization and namical core where the vertical coordinate, given by numerical solution of the discretized problem along with Eq. (13), is replaced by a height-based TFC. The tradi- the various modifications to the formulation over the past tional formulation for height-based TFC, as proposed by years, have been discussed in detail in the existing liter- Gal-Chen and Somerville (1975), can be expressed as ature (Yeh et al. 2002; Qaddouri and Lee 2011; Girard et al. 2014; Husain and Girard 2017). The following z 2 z 5 S subsections therefore only present the relevant details z(z) H 2 , (14) zT zS of the proposed GEM-H dynamical core.

Unauthenticated | Downloaded 09/25/21 04:19 PM UTC JULY 2019 H U S A I N E T A L . 2559 c. GEM-H formulation the square of the speed of sound. In Eq. (23), k is replaced by k 5 R /c as an approximation. It is also important to The development of the GEM-H formulation requires d d pd note that, the physical forcings associated with the mod- further modifications to the system of Eqs. (1)–(5). First, ified continuity equation, given by Eq. (23), now includes virtual temperature, given by Eq. (12), is introduced in the same diabatic heating term that appears in the ther- the system of equations along with an isothermal basic modynamic equation, given by Eq. (24). state temperature T such that T 5 T0 1 T ,whereT0 is * y y * y In the above equations, the terms J , J , and J appear the temperature deviation. The corresponding basic state X Y z due to the vertical coordinate transformation where pressure p is defined hydrostatically as * JX 5 ›z/›X, JY 5 ›z/›Y, and Jz 5 ›z/›z. It is, however, important to note that the coordinate transformation ›(lnp ) * 52g/(R T ). (19) used to derive the Eqs. (20)–(24) is incomplete and only d * ›z the following derivative operators have been transformed: It is important to note that the standard value of T used * › › J › in GEM is 240 K, whereas the value of p at z 5 0 is set 5 2 X , (25) * ›x ›X J ›z to 105 Pa. The equation of state, given by Eq. (11),is z r then used to eliminate density as a prognostic variable › › J › followed by a transformation of the resulting equations 5 2 Y , (26) ›y ›Y J ›z from the geometric height coordinate to the terrain- z following z coordinate. The vertical coordinate trans- › 1 › formation leads to the replacements of the independent 5 , (27) ›z J ›z variables (x, y, z) associated with the z coordinate by z (X, Y, z) that are defined in the z coordinate. As a result, d › › › › 5 1 u 1 y 1 z_ . (28) the system of Eqs. (1)–(5) is modified as follows: dt ›t ›X ›Y ›z ! dy tanf T ›q J ›q du As a result of the incomplete transformation, the origi- 2 f 1 u y 1 y 2 X 5 , dt a T ›X J ›z dt nal vertical velocity w has not been completely elimi- * z phys nated from the system. The system of equations in the (20) z coordinate has its own vertical velocity in the form ! of z_ 5 dz/dt, whereas w 5 dz/dt remains in the system dy tanf T ›q J ›q dy 1 f 1 u u 1 y 2 Y 5 , as a kinematic relation that needs to be dealt with ex- dt a T ›Y J ›z dt * z phys plicitly. Charron et al. (2014) have demonstrated that (21) such an approach is perfectly equivalent to a full co- ! ordinate transformation. As the treatment of advection dw T 1 ›q T0 dw in GEM is based on the semi-Lagrangian approach 1 y 2 g y 5 , dt T J ›z T dt (Husain and Girard 2017), the kinematic relation de- * z y phys fining w is also solved semi-Lagrangially. However, for (22) convenience, the kinematic relation is modified as d q ›u 1 ›(cosfy) ›z_ g 1 lnJ 1 1 1 2 w d _ dt c2 z ›X cosf ›Y ›z c2 (z 2 z) 1 z 2 w 5 0, (29) * * dt d ln(rT ) 5 y , (23) where Eq. (29) along with the Eqs. (20)–(24) constitute the dt phys complete system of equations for the GEM-H formulation. " # One important aspect of any NWP model is how the d T q N2 d lnT ln y 2 1 *w 5 y , effects of the physics forcings, as presented in the rhs dt T c T g dt * pd * phys of the Eqs. (20)–(24), are accounted for as the model (24) equations are integrated at each time step. One way is to resolve the dynamical equations in the absence of 5 where q RdT* ln(p/p*) is a measure of the pressure any physics forcing and then modify the solution with _ 5 deviation from the basic-state pressure p*, z dz/dt the parameterized forcing as adjustments outside the is the vertical motion with respect to the transformed dynamics step. Another possible approach is to com- z 2 5 2 coordinate, N* g /(cpdT*) is the square of the basic- pute the physics forcing and combine their effects as ä ä ä 2 5 2 k state Brunt–V is l frequency, and c* RdT*/(1 d)is tendencies within the dynamics step. This important

Unauthenticated | Downloaded 09/25/21 04:19 PM UTC 2560 MONTHLY WEATHER REVIEW VOLUME 147

FIG. 1. (a) Vertical Charney–Phillips grid. (b) Horizontal Arakawa C grid. aspect of the GEM model with a particular focus on its dF i 2 G 5 P , (30) impact pertaining to the GEM-H dynamical core is dt i i discussed in further details in section 4. where Fi denotes the advected quantity for an individual d. Spatial grid and discretization equation i within the system, Gi is the associated dy- The primary objective of this project has been to namics source term with linear and nonlinear compo- implement the option of a dynamical core based on nents, and Pi denotes the corresponding physics forcing. height-type vertical coordinate in addition to the existing Similarly to GEM-P, treating the advection terms in a pressure-type coordinate in the GEM model. The strategy semi-Lagrangian way and applying a two-time-level has been to add the new coordinate option with min- Crank–Nicholson temporal discretization leads to imal changes to the dynamical core and the rest of the FA 2 FD 1 1 b 1 2 b GEM model source code. Therefore, the spatial grid i i 2 iGA 2 iGD 5 s P , (31) Dt 2 i 2 i c i structures in GEM-H, both in the horizontal and the vertical, are kept the same as those in GEM-P, which where Dt indicates the time-step length, and the super- implies a staggered Arakawa C grid (Arakawa 1988) scripts A and D denote the arrival and departure posi- in the horizontal and a staggered Charney–Phillips tions of the air parcels at the current time t and the grid (Charney and Phillips 1953) in the vertical. The previous time (t 2Dt), respectively. The integrals of the horizontal and vertical grid structures are presented in source terms Gi for the different dynamical equations Fig. 1. are approximated by trajectory averages. The parameter

In addition to being similar to GEM-P for the limited- bi denotes the off-centering weight factor for the aver- area model (LAM) grids, the global grid system is also aging of the dynamics source terms. When bi 5 0, the kept unchanged in GEM-H, and is therefore, based on a averaging of the source term is fully centered, whereas

Yin–Yang grid system (Qaddouri and Lee 2011). The bi . 0 implies additional weight placed on the implicit Yin–Yang system combines two overlapping latitude– component of the discretized source term. longitude LAM grids to form a global grid following the Historically, off-centering was implemented in GEM-P Schwarz method for nonmatching domain decomposition primarily to address spurious resonance originating (Qaddouri et al. 2008) and thus avoids pole-related singu- from stationary orographic forcing (Rivest et al. 1994). larity and convergence issues associated with a conventional However, it also suppresses computational noise and global lat-lon grid. Further details on the Yin–Yang grid are improves numerical stability. These other beneﬁcial provided by Qaddouri and Lee (2011). impacts have been found to be equally important in the current and previous implementations of GEM-P e. Discretization in time for the different operational NWP systems at ECCC. The general form of an individual equation in the As the principal objective of this study is to have a system comprising Eqs. (20)–(24) and (29) can be ex- GEM-H dynamical core that is equivalent to GEM-P pressed as for the different operational GEM-based NWP systems,

Unauthenticated | Downloaded 09/25/21 04:19 PM UTC JULY 2019 H U S A I N E T A L . 2561 off-centering has been retained in GEM-H. Also, fol- (2017), similar consistent trajectory calculation ap- lowing the latest implementation of GEM-P (Husain and proach is adopted in GEM-H, that is, trapezoidal rule Girard 2017), a differential approach for off-centering has for evaluating the integral of the source term in Eq. (32) been adopted for GEM-H where bi varies depending on along with cubic interpolation to determine the wind the dynamical equation denoted by the subscript i.At ﬁeld at the departure positions. present, the system of equations, given by Eqs. (20)–(24) and (29), are separated into three groups with an associ- g. The elliptic problem ated off-centering parameter for each group as follows: To solve the system of equations associated with the

(i) bm for the horizontal momentum equations [Eqs. (20) GEM-H formulation, each equation of the form in and (21)], Eq. (31) is rearranged to separate the linear and

(ii) bh for the continuity and thermodynamic equations nonlinear components of the implicit part and is ex- [Eqs. (23) and (24)], and pressed as (iii) b for the vertical momentum and kinematic equa- nh 5 2 tions [Eqs. (22) and (29)]. Li Ri Ni , (33)

On the rhs of Eq. (31), the term P denotes the pa- 5 A 2 A i where Li (Fi /ti Gi )linear includes the linearized rameterized physics source term and the parameter s 5 A 2 A 2 c lhs terms, Ni Fi /t Gi Li contains the remaining indicates the mode of coupling between physics and 5 D 1 D 1 5 nonlinear terms, and Ri Fi /ti biGi scPi with bi dynamics. Depending on the chosen method for (1 2 bi)/(1 1 bi) and ti 5Dt(1 1 bi)/2. It is important to dynamics–physics coupling, the value of sc can be either note that, based on the choice of the linear terms in Li, 0 or 1. Also, the approach for dynamics–physics coupling the nonlinear Ni terms may include some linear residuals determines how the contribution of Pi is accounted for as well. Equations (20)–(24) and (29) are rearranged as in the model. Further discussions regarding the coupling in Eq. (33), giving of the parameterized physics forcing with the dynamical u Xz core is presented in section 4. L 5 1 d q 2 s J J21d q , (34) u t X i X z z f. Trajectory calculations m y Yz Semi-Lagrangian treatment of advection requires the L 5 1 d q 2 s J J21d q , (35) y t Y i Y z z solution of kinematic displacement equations of the m following form: w T0 L 5 1 (s J21 1 s )d q 2 g y , (36) w t i z d z dX dY nh Ty 5 u, 5 y , (32a) dt dt 1 q 1 L 5 1 lnJ 1 d u 1 d (cosf ) dz c t c2 z X cosf Y y 5 z_ , (32b) h * dt 1 _ 2 z dzz «w , (37) to determine the departure positions of the air parcels. ! In the context of GEM-P, Husain and Girard (2017) 1 T0 qz L 5 y 2 1 mw, (38) have shown that the consistency in the numerical dis- T t T c T h y pd * cretizations between the dynamical and trajectory equations is important for accurate solution of the advection z 2 z L 5 1 z_ 2 w, (39) problem. To be numerically consistent, similarly to the z tnh treatment of the dynamics source term in Eq. (30), the averaging of the velocities in Eq. (32) needs to be where the symbol dX denotes the finite difference op- done using the trapezoidal rule. Furthermore, the in- erator along the X direction, and the overline operator terpolation scheme employed to determine the wind ()X implies spatial averaging in the X coordinate. For field at the departure positions for the trajectory calcu- convenience of notation, the superscript A has been A lations ideally should be the same as the one applied to removed from the Fi terms in the above expressions for 2 2 determine the source terms in the dynamical equations Li. Also, the terms g/c* and N*/g have been replaced by at the departure positions. In the case of GEM-P, cubic « and m, respectively. The corresponding nonlinear interpolation is used for both the wind field and the components Ni associated with the discretized forms of dynamical source terms to achieve numerical consis- the Eqs. (20)–(24) and (29) as well as the associated rhs tency. Following the conclusions of Husain and Girard terms Ri are provided in appendix A. The parameters si

Unauthenticated | Downloaded 09/25/21 04:19 PM UTC 2562 MONTHLY WEATHER REVIEW VOLUME 147 and s in the above equations denote the choice of the 1 ›q T0 d 5 g y (41) solver for the dynamical core, where the subscripts i Jz ›z Ty and d stand for iterative and direct, respectively. Based on the selected solution approach, these parameters can as a hydrostatic approximation. The value of q at the be either 1 or 0, and are mutually exclusive such that different model levels are then obtained by integrat- si 5 1 2 sd. Further discussion on these solution ap- ing Eq. (41) where at the surface, due to the hydro- ^ é 5 proaches is presented in section 3. As shown by Cot static approximation, qs RdT* ln(ps/p*). The initial and Staniforth (1988), the solution of the system of value of z_ is computed by assuming ›r/›t 5 0 at time t 5 equations of the type Eq. (33) requires nonlinear itera- 0 in the continuity equation. In the z coordinate, this tions for convergence, where the nonlinear terms Ni takes the form are reevaluated during each iteration using the latest ! values of the prognostic variables. Furthermore, Crank– › ›z 1 › ›z › ›z ru 1 cosfry 1 rz_ 5 0, Nicholson iterations are required, where the R terms i ›X ›z cosf ›Y ›z ›z ›z are reevaluated during each iteration at the departure (42) positions calculated using the latest velocity estimates. At present, the GEM-based operational NWP systems which is then discretized to compute the initial value at ECCC utilize two Crank–Nicholson iterations, and of z_.Oncez_ is known, the initial value of w is obtained two nonlinear iterations are carried out within each from its definition in the z coordinate: w [ dz/dx 5 Crank–Nicholson step. As a result, irrespective of the _ uJX 1 yJY 1 zJz. solution approach, the solver is called four times during The boundary conditions for the upper and lower each dynamical time step. _ 5 _ 5 boundaries are given by zT zS 0. This implies that The discretized equations with the left hand sides (lhs) the vertical motion in the z coordinate vanishes at the given by Eqs. (34)–(39) are then reduced into a single surface and the model top which is flat. For LAM elliptic boundary value (EBV) problem through elimi- problems, GEM-H also requires lateral boundary con- nation of variables, where the lhs of the final elliptic ditions which are obtained from the driving fields. As problem takes the form the global Yin–Yang system is based on two interacting 1 geometrically identical LAM domains, it therefore L000 5 d A 1 d ½cosfB 1 d C 2 «C z 2 gq, c X cosf Y z similarly requires lateral boundary conditions. In this (40) case, the boundary conditions for one subdomain Xz (Yin or Yang) depend on the solution of the other. 21 where A 5 dX q 2 siJX J dzq , B 5 ðdY q 2 Yz z Thus the solution of the global problem is obtained by 21 C 5G 21 1 2 z siJY Jz dzq Þ,and [(siJz sd)dzq mq ]. Also, iteratively solving the two subproblems separately and g 5 2 t t G5 t t 1 2 t t in Eq. (40), 1/(c* h m) and 1/( m/ nh N* h m). updating the values in the overlapping region until a 5 A B It is important to note that, with si 1, the terms , , certain convergence criterion is satisfied (Qaddouri C 5 and (with m 0) are simply the components of the and Lee 2011). gradients in the terrain-following coordinate system. The sequence of steps involved in deriving the EBV 000 3. Solution of the EBV problem problem (i.e., the final form of Lc ) is provided in appendix B. The EBV problem to be resolved by GEM-H at each h. Initial and boundary conditions model time step can be expressed as =2 1 M 5 R As with the case of any initial value problem, in zq q , (43) order to initiate integration in time, the GEM-H dy- 000 =2 1 M namical core requires initial values of all the prog- by replacing Lc in Eq. (40) with ( z )q, where =2 5 1 nostic variables. At present, ECCC’s operational z [dXX (1/cosf)dY (cosfdY )] is a discretized two- data assimilation system provides analyzed initial dimensional horizontal operator in the z coordinate, M 000 values for the horizontal wind components u and y, contains all the remaining terms of Lc that include the virtual temperature Ty, and surface pressure ps.The discretized difference and averaging operators in the remaining prognostic variables w, z_,andq are com- horizontal and vertical dimensions, q is the unknown puted in a diagnostic manner at time t 5 0. The initial and R includes the explicit rhs terms as well as the im- value of q is obtained from the analyzed surface plicit nonlinear terms. It is important to note that the pressure ps with a hydrostatic approximation. Substitut- nonlinear terms in R require iterations for convergence, ing dw/dt 5 0 in Eq. (22) gives irrespective of the solution approach. At present, the

Unauthenticated | Downloaded 09/25/21 04:19 PM UTC JULY 2019 H U S A I N E T A L . 2563

GEM dynamical core uses two iterations for the suffi- margin for the configurations of the various GEM-based cient convergence of the nonlinear terms and two iter- NWP systems running operationally. The ‘‘simplified ations for trajectories. As a result, the solver is called approach’’ thus makes the application of GEM-H for into action four times during each dynamical step. such configurations comparable to GEM-P in terms of Once Eq. (43) is solved to obtain the unknown q, the computational performance, and makes the option of a other prognostic variables are obtained through back future replacement of the GEM-P core with GEM-H substitution as presented in appendix C. As has been feasible for the operational NWP systems. The simpli- mentioned earlier, two general approaches are available fied approach, however, is applicable as long as the for solving the elliptic problem—direct and iterative. vertical–horizontal coupling, imparted through the met- The selection of these approaches depends on which ric terms, is not too significant so that these terms can be terms are included in M, and is determined by the values treated efficiently through nonlinear iterations. This of the terms si and sd in Eqs. (34)–(38). approach works provided the maximum terrain slope is not too steep (less than 408) which is generally the case a. The direct solver for the different operational GEM-based NWP systems. The direct solver works by first decoupling Eq. (43) in However, with increasing spatial resolution, the slopes the vertical. It is achieved through the expansion of in gridscale orography also increase, particularly over com- the unknown q and the rhs R in terms of the eigenvec- plex terrain, which leads to increased vertical–horizontal tors that diagonalizes the operator M (Qaddouri and coupling and, at one point, makes the ‘‘simplified approach’’ Lee 2011). This is only possible when the operator M inapplicable. does not include contributions from the metric terms arising from the vertical coordinate transformation that b. The iterative solver involves horizontally variable coefficients imparting When all the metric terms in A and B [see Eq. (40)] horizontal coupling. Therefore, the implementation of are included in M, the resulting vertical–horizontal direct solver in GEM-H requires a ‘‘simplified ap- coupling makes the problem nonseparable. This is proach’’ where all the metric terms of the relevant dis- done by setting si 5 1. In such a scenario, the three- cretized equations are treated as nonlinear terms. This dimensional equation of the form in (43) can only be 5 is achieved by setting sd 1. For Nk number of vertical solved at each time step by using an iterative solver. levels used in the model, vertical separation reduces The current iterative solver for the EBV problem in Eq. (43) to a set of Nk independent horizontal Helmholtz GEM-H is based on the flexible generalized minimal problems of the form residual (FGMRES) method (Saad 1993; Qaddouri and Lee 2010). =2~ 1 ~ 5 R~ z q mq , (44) The fully discretized system of equations of the form in (43) can be further generalized as ~ where q~ and R are the vertical projections of q and R, M respectively, and m is the eigenvalue of the operator . Aq 5 R, (45) The horizontal solution of the algorithm then proceeds ~ by expanding q~ and R in terms of the eigenvectors that where the coefficient matrix A contains the discretized =2 1 M diagonalize the X-component of the two-dimensional operator ( z ). The FGMRES method approxi- =2 operator z. For Ni number of grid points along the mates the solution in a Krylov subspace of small di- longitude X, this leads to Ni independent tridiagonal mension by minimizing the Euclidean norm of its problems of Nj dimension for each model vertical level, residual. A major advantage of such a method is that where Nj denotes the number of grid points along the instead of explicitly generating the coefficient matrix latitude Y. The total number of tridiagonal problems to A, one only needs to compute the vector resulting from 3 =2 1 M solve is therefore Nk Ni. Solution to the tridiagonal the action of the underlying operator ( z ) on the problems is computed by Gaussian elimination without vector q. Efficient functioning of such an iterative pivoting, and afterward, the final three-dimensional so- solver, however, requires a preconditioner. At present, lution q is obtained (Qaddouri and Lee 2010). the preconditioner is based on the block Jacobi itera- The ‘‘simplified approach’’ has been primarily im- tion for the EBV in Eq. (43), where all the metric terms plemented in GEM-H to take advantage of the com- in M are absent. This preconditioner improves the con- putational performance of the direct solver. The direct vergence rate of the FGMRES solver. However, the solver uses Fast Fourier Transform to compute the time of execution is still high compared to the fast direct horizontal solution q~ and outperforms any iterative solver. Significant research is currently underway at approach implemented at the ECCC by a substantial ECCC to devise iterative solvers that are competitive

Unauthenticated | Downloaded 09/25/21 04:19 PM UTC 2564 MONTHLY WEATHER REVIEW VOLUME 147 with the direct approach and will work for both GEM-P is carried out in the split mode between the dynamics and GEM-H. At present, the current implementation and physics steps where the diffusion scheme has ac- of the iterative solver in GEM-H—although not as ef- cess to the three-dimensional outputs of the dynamics ficient—provides the necessary reference for the direct substep. solver approach. a. Different coupling approaches in GEM Particularly in the context of GEM-P, two approaches 4. Dynamics–physics coupling are presently available to couple the RPN Physics Along with the dynamical core, parameterization of Package with the GEM dynamical core. A brief dis- the subgrid-scale physical processes constitutes the cussion on these methods will be helpful in establishing other fundamental component of any NWP model. the rationale behind the approach selected for the Coupling between the dynamical core and the parame- GEM-H dynamical core. terized subgrid-scale physical processes is of critical (i) Split method: As has been mentioned earlier, sc importance. How to devise the most appropriate cou- determines the mode of coupling between dynamics pling strategy is still an unsettled question (Beljaars et al. and physics. If sc 5 0 then the dynamical equations 2018). This issue is being actively studied at ECCC. are resolved in the absence of any physical forcing However, it is not the objective of this study to delve and at the end of the dynamics step their contribu- deep into the fundamental questions around dynamics– tions are incorporated as adjustments in the so- physics coupling. Rather, in this section, the issue of called split mode. In the absence of physics forcing, coupling the RPN Physics Package with the GEM dy- Eq. (31) becomes namical core is discussed in order to determine which approach is the most feasible for GEM-H among the A* 2 D 1 2 F F 1 b A 1 b options that are available for GEM-P. i i 2 iG * 2 iGD 5 0, (47) Dt 2 i 2 i It is important to note that, during every model time A* step in GEM, the RPN Physics Package is executed af- where Fi is the interim solution of the dynamics ter the dynamical equations have been resolved by step. Once Eq. (47) is resolved, the physics source the dynamical core and thus the physics schemes utilize term is then applied as gridpoint adjustments as the outputs of the dynamics step as inputs. However, follows irrespective of the vertical coordinate used in the dynamical core, the physics schemes use a traditional 5 A 2 A* 5D (dF)phys Fi Fi tPi . (48) s coordinate in the vertical, defined as Thus, in the split method, the dynamics step predicts p s 5 , (46) an inviscid and adiabatic solution that is modified ps through adjustments attributable to the parameterized physics forcings in order to obtain the complete p where is the hydrostatic pressure and the subscript s solution at the end of each model time step. denotes the surface. Also, the physics schemes work (ii) Tendency method: The second option for dynamics– within a one-dimensional configuration where each physics coupling works by directly incorporating the processor core only has access to the vertical structure impact of physics forcings as tendencies into the of the meteorological fields associated with a single discretized dynamical equations, and hence is re- horizontal grid point. The various physical processes ferred to as the ‘‘tendency method.’’ This method are parameterized sequentially where the tendencies treats the physics source terms explicitly by setting estimated by one parameterization scheme affect the 5 D sc 1 and replacing Pi by Pi . As a result, it involves ones that follow. The parameterization sequence dur- solution of equations of the following form at each ing each physics step initiates with the radiation scheme model time step: and is followed by the parameterizations of the surface processes, gravity wave drag, boundary layer turbu- FA 2 FD 1 1 b 1 2 b lence, convection and gridscale condensation. At the i i 2 iGA 2 iGD 5 PD . (49) Dt 2 i 2 i i end of the physics step, the tendencies from the different physical parameterization schemes are aggregated to In this approach, the physics tendency Pi from the compute the gridscale tendencies for the wind compo- previous time step is combined with the rhs term D nents, temperature, water vapor and the hydrometeors. Ri, followed by the determination of Ri at the It is important to note that horizontal diffusion in GEM departure positions in a semi-Lagrangian way.

Unauthenticated | Downloaded 09/25/21 04:19 PM UTC JULY 2019 H U S A I N E T A L . 2565 b. Coupling in GEM-H that are permissible with the semi-implicit semi- Lagrangian models. Similar conclusions were drawn Although both of the aforementioned coupling from a theoretical analysis with the two-time-level methods are available for GEM-P, it is important to note scheme by Staniforth et al. (2002). Figure 2a shows the that all the operational NWP systems based on GEM-P geopotential height contours at 400 hPa from a 72-h at present utilize the split method for dynamics–physics global forecast with ECCC’s 25-km resolution Global coupling. Nevertheless, there exists strong concern about Deterministic Prediction System (GDPS) using the the split method in general, and a brief discussion GEM-P dynamical core. The results correspond to split highlighting the pertinent issues will be helpful. method for dynamics–physics coupling. Although the In the context of model formulations for both GEM-P distribution of geopotential height for the meso and and GEM-H, the term F does not necessarily match i large scales does not reveal any issue of immediate the prognostic model variables. For example, in the case concern, when one looks at the smallest scales (i.e., of GEM-P in its hydrostatic mode, as presented by scales of about a few grid lengths), some spurious com- Girard et al. (2014): putational noise is visible (see Fig. 2b). Historically, the operational NWP systems at ECCC have been utilizing ›B T F 5 u, y, Bs 1 ln 1 1 ,ln y 2 k Bs , (50a) spatial filters over the model-predicted meteorological ›j T d * fields of interest, like the geopotential height, to smooth out any computational noise in the model outputs. As a d P 5 (u, y,lnr,lnT ) , (50b) result, the kind of computational noise shown in Fig. 2b dt y phys has not been troublesome for the meteorologists using ECCC’s operational NWP outputs. Nevertheless, the whereas the prognostic variables are u, y, s, j_, and T . y computational noise associated with the split method Therefore, in the actual model implementation of the remains as a concern. However, as shown in Fig. 2c, split method, the prediction from the dynamics step is when the split method for coupling is replaced by the utilized to compute the interim state of the prognostic tendency method, the noise in the geopotential height variables and adjustments are then applied to these disappears. It should be noted that, although the ten- variables at the end of the physics step. It is important to dency approach for coupling can impose stability limi- note that in this case the only adjustments that have tations in terms of the acceptable length of time steps, been found to not result in any issue of major concern ECCC’s operational NWP system configurations are are (du) ,(dy) , and (dT ) . Also in GEM-P, phys phys y phys found to function with the tendency method without adjustments are required for density, as in effect requiring any adjustment to the time-step lengths. d lnr d Similarly to GEM-P, the Fi terms do not match with [ ln(1 1 r ). (51) the prognostic variables for all of the GEM-H model dt dt w phys equations, where ( ) Although water vapor and hydrometeors are updated q T q through physical parameterizations, the only variable F 5 u, y, w, 1 lnJ ,ln y 2 , (53a) c2 z T c T that could be updated in the split mode appears to be s * * pd * through a surface pressure adjustment dp that may be s d computed as P 5 [u, y, w, ln(rT ), lnT ] . (53b) dt y y ð phys 5 1 dps d ln(1 rw) dp, (52) Particularly, the presence of the physics forcing term

[d ln(rTy)/dt]phys in the modified continuity Eq. (23) which takes into account the net inflow/outflow of mass poses an additional challenge for the GEM-H formula- through Earth’s surface at every model grid point. Un- tion, as far as the split method is concerned. If one fortunately, the vertical distribution of this change in attempts to apply this tendency as a hydrostatic adjust- mass through water vapor and precipitation fluxes can- ment to pressure in the split mode then of course its not be correctly accounted for in the split mode, and effect is not limited to the continuity equation alone. is found to produce considerable noise in the wind Rather, applying the adjustments to pressure (i.e., chang- forecast. ing q in the case of GEM-H) also affects the thermo- Furthermore, in the context of three-time-level dis- dynamic equation. Such an adjustment leads to over cretization, Caya et al. (1998) have shown that the split compensation and is found to result in spurious bias in method can lead to erroneous results for long time steps the temperature in upper troposphere and stratosphere,

Unauthenticated | Downloaded 09/25/21 04:19 PM UTC 2566 MONTHLY WEATHER REVIEW VOLUME 147

FIG. 2. (a) Geopotential height (dam) at 400 hPa after 72 h for a 25-km global forecast initiated at 1200 UTC 25 Jan 2015 obtained with GEM-P using the ‘‘split method’’ for dynamics–physics coupling (contour intervals of 6 dam). (b) The enlarged view over a small section of the global domain (contour intervals of 0.5 dam). (c) As in (b), but with ‘‘tendency method’’ for the dynamics–physics coupling. which is unacceptable (not shown). Also, the jet-level hybrid approach with GEM-H produces results that are wind is found to be adversely affected. As a result, the equivalent to those obtained with the split method for physics contributions in GEM-H are accommodated GEM-P. However, questions remain about the numer- through the tendency method by including them as ex- ical consistency of the hybrid split-tendency method. plicit gridscale tendencies in the rhs of the discretized It is important to mention that the current imple- dynamical equations. For a fair comparison between mentation of the RPN Physics Package, when coupled the new and the existing dynamical core, the results for with the GEM-P (GEM-H) dynamical core through both GEM-P and GEM-H presented in the rest of this the tendency method, leads to some deterioration in paper are obtained with dynamics–physics coupling temperature bias in the upper troposphere compared based on the tendency approach. It is also important to to the split (split-tendency) method. This implies that mention that, even though parameterization of physi- further research is necessary to have a more consis- cal processes like boundary layer turbulence can mod- tent dynamics–physics coupling with the tendency ap- ify vertical motion w, at present the impact of physics on proach. Particularly, the gridscale condensation scheme w is neglected. This is the case for both GEM-P and in GEM, for 10 km or coarser horizontal resolutions, GEM-H. has been found to exhibit large sensitivity with the A hybrid ‘‘split-tendency method’’ has also been tendency method which leads to underprediction of tested with GEM-H, where the physics contributions to clouds (R. McTaggart-Cowan, ECCC, 2018, personal the thermodynamic and horizontal momentum equa- communication). This implies that some process-speciﬁc tions are accounted for through the ‘‘split method’’ adjustments in the computation of the relevant physics while the contributions to the continuity equation is tendencies may be required to improve the overall accommodated using the ‘‘tendency method.’’ Such a dynamics–physics coupling. The challenges imposed by

Unauthenticated | Downloaded 09/25/21 04:19 PM UTC JULY 2019 H U S A I N E T A L . 2567 the process-specific issues are also being explored by (7 and 10 min), is presented in Fig. 3 for both GEM-P other operational NWP centers (Beljaars et al. 2018). and GEM-H. The bubbles predicted by the two cores Currently, work is underway at ECCC to explore the initially deform into a somewhat mushroom-like shape various issues within the coupling interface as well as (see at t 5 7 min) and then are deformed further (at the parameterizations of the different physical processes t 5 10 min). Overall, the predictions from GEM-P and to improve the dynamics–physics coupling in general. GEM-H are equivalent for the entire range of scales– Further discussion on this issue is, however, beyond the from the largest to the smallest resolved scales. Such scope of this paper. a good resemblance between the two predictions implies negligible impact of the choice of the vertical coordinate and the other modifications in model formulations in 5. Evaluation of GEM-H the absence of any orographic variation at the model One of the most important objectives of this study surface. It also indicates that the representation of the has been to develop a dynamical core for the GEM advection and buoyancy effects are comparable be- model that utilizes a height-based TFC and is capable tween the two GEM cores. It is important to note that, of producing predictions that are equivalent to the re- due to considerable differences in the model formula- sults obtained by the pressure-based dynamical core and tions and spatiotemporal discretizations, it is difficult to with a comparable computational cost. To evaluate compare the evolution of the bubbles between two the consistency, accuracy and performance of the new completely separate models in a quantitative manner. dynamical core, a number of numerical experiments Only qualitative comparisons are feasible. Therefore, covering a wide range of scales—ranging from micro- the lesser resemblance between the results from Robert scales to the meso and synoptic scales—have been car- (1993) and the GEM dynamical cores are not unusual. ried out. These include two-dimensional theoretical Although the predictions from the two GEM cores have test cases involving bubble convection (Robert 1993) some large-scale resemblance to the results presented and nonhydrostatic mountain waves (Schär et al. 2002) by Robert (1993), significant differences appear at the as well as three-dimensional global NWP. The two- upper half of the bubble–particularly at t 5 10 min. dimensional test cases selected in this paper have be- However, the upper structure of the bubble compares come ubiquitous tools in testing the consistency and better with the predictions by Smolarkiewicz and performance of nonhydrostatic dynamical cores. Also, Pudykiewicz (1992). Also, because of the implicit dissi- the availability of the GEM-P core provides the op- pation associated with the semi-Lagrangian approach, portunity to have reference solutions for all these cases. the GEM solution does not suffer from computational noise like some models based on Eulerian advection a. Robert’s bubble convection case (Juang 2000). Overall, as GEM-P is being used opera- Robert (1993) presented a two-dimensional theo- tionally at ECCC, the resemblance between GEM-H retical case involving the evolution of a warm bubble and GEM-P is of more significant importance, as it within a dry isentropic atmosphere. Initially, the bubble confirms consistency of the GEM-H formulation and a has a diameter of 500 m and is placed 10 m above a flat neutral impact of the vertical coordinate modification surface within a 1 km 3 1 km computational domain, on buoyancy and advection. and has a uniform potential temperature of 30.58C. b. Schär’s mountain case Also, the basic-state atmosphere is at rest under a hydrostatic equilibrium with an isentropic basic-state Schär et al. (2002) presented a linear two-dimensional temperature of 308C.Asthebubblehasapotential theoretical test case of mountain waves which is an temperature excess of 0.58C compared to the sur- excellent benchmark for verifying nonhydrostatic rounding atmosphere, it rises due to the action of the dynamical cores, particularly in determining the pres- buoyancy force. The absence of any orographic vari- ence of possible inconsistencies in the numerical details ation makes this experiment an excellent benchmark (Husain and Girard 2017; Melvin et al. 2010; Girard to test the functioning of advection and buoyancy et al. 2005; Klemp et al. 2003). The bottom boundary within a dynamical core during the early stages of its profile of the idealized mountain for this case is de- development. fined by The numerical experiment for bubble convection is carried out with a spatial grid resolution of 10 m and a 5 2(x/a)2 2 zs z0e cos (px/lx), (54) time step of 5 s. No explicit numerical diffusion is used.

The resulting evolution of the bubble, in terms of its where z0 5 250 m, lx 5 4 km, a 5 5 km, and p is the potential temperature distribution at two different times conventional mathematical constant. The upstream ﬂow

Unauthenticated | Downloaded 09/25/21 04:19 PM UTC 2568 MONTHLY WEATHER REVIEW VOLUME 147

FIG. 3. Potential temperature (8C) distribution for the rising bubble with: (a) GEM-P at t 5 7 min, (b) GEM-H at t 5 7 min, (c) GEM-P at t 5 10 min, and (d) GEM-H at t 5 10 min. conditions are given by uniform upstream wind U 5 much higher altitudes. As can be seen in Fig. 4, the so- 21 10 m s , upstream surface temperature Tsurf 5 288 K, lutions with both types of vertical coordinates gener- upstream surface pressure p0 5 1000 hPa, and a constant ate these two regimes of the mountain waves and are ä ä ä 5 21 Brunt–V is l frequency N* 0.01 s . All other con- very similar. Results shown in this figure represent ditions for the simulations of this test case are similar to simulations that have been carried out without any off- those presented by Husain and Girard (2017). centering in the discretized dynamical equations and One major advantage of Schär’s mountain case is the with consistent trajectory calculations (i.e., integration availability of a steady-state analytical solution of the of the wind field in the trajectory equations is based corresponding linearized problem as a reference (Schär on the trapezoidal rule while the wind field at the de- et al. 2002). The simulated quasi-steady vertical velocity parture positions is obtained with cubic interpolation). obtained after 4 h of integration are presented in Fig. 4 Inconsistent trajectory calculations were found to pro- for both the GEM-P and GEM-H dynamical cores. The duce similar distortion in the large-scale hydrostatic analytical solution of the problem, as presented by Schär waves with GEM-H (not shown) as has been found et al. (2002), is a combination of rapidly decaying small- for GEM-P earlier by Husain and Girard (2017).Al- scale nonhydrostatic mountain waves close to the sur- though the GEM-H results presented here correspond face and large-scale hydrostatic waves extending to to the direct solver based on the simplified approach,

Unauthenticated | Downloaded 09/25/21 04:19 PM UTC JULY 2019 H U S A I N E T A L . 2569

21 FIG. 4. Steady-state vertical velocity contours (at 0.1 m s intervals) for Schär’s mountain case predicted by: (a) GEM-P and (b) GEM-H. No off-centering is used in the discretized dynamical and trajectory equations. results with the iterative solver are found to be almost but the distortions are only completely eliminated identical (not shown). when bm 5 bh 5 bnh 5 0 (not shown). This conforms to Figure 5 reveals that off-centering in the discretized the conclusions drawn by Husain and Girard (2017) in dynamical equations leads to distortions in the vertical the context of GEM-P. Furthermore, as has been velocity distribution, irrespective of the type of the shown by Husain and Girard (2017), consistent tra- vertical coordinate. The solutions presented here are jectory calculations in the presence of off-centering obtained with uniform off-centering involving bm 5 necessitates off-centered averaging applied to the in- bh 5 bnh 5 0.2, which are the standard values used in the tegrals of the source term on the rhs of Eq. (32).With current GEM-based operational NWP systems. The uniform off-centering applied to the discretized dy- results correspond to Dt 5 32 s, for which the maximum namical equations, the discretized trajectory equa- Courant number is approximately 0.76. Reducing the tions also require uniform off-centering of the same time step to even 4 s is unable to remove these distor- degree. Figure 6 reveals that in the presence of con- tions. Also, reducing the level of off-centering is found sistent off-centering in the trajectory calculations, to reduce the level of distortion in the mountain waves, the distortions in the vertical velocity distribution are

FIG.5.AsinFig. 4, but with uniform off-centering (bm 5 bh 5 bnh 5 0.2) used in the discretized dynamical equations. The absence of off-centering in the discretized trajectory equations leads to inconsistent trajectory calculations and distortions in the mountain waves.

Unauthenticated | Downloaded 09/25/21 04:19 PM UTC 2570 MONTHLY WEATHER REVIEW VOLUME 147

FIG.6.AsinFig. 4, but with consistent off-centering (i.e., off-centering is applied to both the discretized dynamical and trajectory equations). eliminated for the both dynamical cores. In the pres- coupled with dynamics through the tendency method ence of differential off-centering (i.e., with different for both GEM-H and GEM-P for all the tests presented values of bh and bnh for hydrostatic and nonhydrostatic in this section. contributions in the system of dynamical equations), First, comparisons are made in the spectral space a similar differential approach is required for off- by comparing the variance spectra associated with the centering in the discretized trajectory equations. As different meteorologically important fields. To compute has been shown for GEM-P by Husain and Girard the spectral variance of any meteorological field, it (2017), in order to achieve numerical consistency, is first interpolated from the Yin–Yang grid to a global the off-centering in the discretized source terms in Gaussian grid. Afterward, variance spectra of the field Eqs. (32a)and(32b) needs to be equal to the values are calculated by decomposing it at a given pressure of bh and bnh used in the discretized dynamical equations, level using the spherical harmonics. Figure 7 shows the respectively. spectral variance of the geopotential height and temperature fields for 120-h forecasts at three different c. Global deterministic prediction pressure levels averaged over 10 cases from the winter A series of 5-day global forecasts, with 25-km hori- period. The results show spectral similarity between zontal grid spacing, has been carried out covering GEM-P and GEM-H for the entire range of scales re- winter and summer periods for the Northern Hemi- solved by the global model—from synoptic to me- sphere to compare the predictions by GEM-H and soscales. The spectra of kinetic energy and vertical GEM-P from NWP standpoint. Each seasonal period velocity are presented in Fig. 8. The spectral slope of includes 44 cases where the initialization between kinetic energy is critical for accurate representation two consecutive cases is apart by 36 h. The first summer of atmospheric dynamics, and as can be seen in this and winter cases start at 0000 UTC 25 June 2014 and figure, both GEM-H and GEM-P have the same 19 December 2014, respectively. The global predic- spectral slope at the synoptic and mesoscales. The tions have been obtained with uniform off-centering vertical motion is also important, particularly for in the discretized dynamical equations (bm 5 bh 5 physical processes like convection. Figure 8 shows bnh 5 0.2). Husain and Girard (2017) have shown that close spectral similarity between the vertical motions inconsistent off-centering has negligible effects for from the two dynamical cores. This implies that three-dimensional NWP applications. Therefore, the changing the vertical coordinate has negligible sensi- global forecasts are carried out without any off- tivity to the extremely important physical processes centering in the discretized trajectory equations. Sim- like convection and convection-driven precipitation. ulations have been carried out using 84 vertical levels Also, the comparisons in the spectral space confirm with the model top set approximately at 0.1 hPa and that the height-based TFC does not lead to any spu- 65 km for GEM-P and GEM-H, respectively. Further- rious noise or damping in the meteorological fields for more, as has been mentioned in section 4, physics is any model-resolved length scale.

Unauthenticated | Downloaded 09/25/21 04:19 PM UTC JULY 2019 H U S A I N E T A L . 2571

FIG. 7. Spectral variance of (left) geopotential height (GZ) and (right) temperature (TT) for 120-h global forecasts with 25-km horizontal grid spacing. Results are presented for three different pressure levels [(top) 100 hPa, (middle) 500 hPa, (bottom) 850 hPa] that are obtained by averaging the spectra for 10 Northern Hemi- sphere winter forecasts.

Objective forecast scores are computed by comparing predictions for the individual cases as well as for the the model predictions against radiosonde observations average of the 44 cases covering each seasonal period. at different pressure levels. The evaluation is based on Figure 9 presents the vertical proﬁle of error in the the bias and standard deviation of error (SDE) in model predictions from GEM-P (blue) and GEM-H (red) for

Unauthenticated | Downloaded 09/25/21 04:19 PM UTC 2572 MONTHLY WEATHER REVIEW VOLUME 147

FIG. 8. Spectral variance of (left) kinetic energy (KE) and (right) vertical velocity (WW) for 120-h global forecasts. All other conditions are as in Fig. 7. the winter period. These figures represent global aver- statistical confidence scores at the different pressure age scores of 120-h forecast from 44 winter cases for levels along the left and right vertical axes of the indi- zonal wind (UU), wind speed (UV), geopotential height vidual subplots for bias and SDE, respectively. A con- (GZ), and temperature (TT). An important thing to fidence value (in %) shaded in blue (red) color implies note while reading this figure is the presence of the statistically significant improvement obtained with the

Unauthenticated | Downloaded 09/25/21 04:19 PM UTC JULY 2019 H U S A I N E T A L . 2573

FIG. 9. Comparison of 120-h 25-km GDPS forecasts obtained with GEM-P (blue) and GEM-H (red) dynamical cores against radiosonde observations for zonal wind (UU), wind speed (UV), temperature (TT), and geopotential height (GZ). The dashed and solid lines, respectively, indicate bias and standard deviation of error (SDE). The scores are obtained by averaging over 44 Northern Hemisphere winter cases. The red and blue shaded numbers along the left (right) vertical axes within each panel indicate the confidence in percentage in the statistically significant improvements in bias (SDE) for the dynamical core associated with each color. Significance for bias and SDE are computed using t and F test, respectively.

GEM-P (GEM-H) core with respect to the other. The parameterizations can be sensitive to the position of the absence of the statistical significance value of a score at a vertical levels and this is apparently responsible for the pressure level implies that the difference between the small bias differences in Fig. 9. Even though small dif- two dynamical cores is statistically insignificant with ferences in bias are present, the objective scores from respect to that score. The confidence scores for the av- GEM-P and GEM-H can be safely assumed to be sta- erage of SDE and bias are estimated by applying the F tistically equivalent as a whole. The two dynamical cores test and t test, respectively. Figure 9 reveals that al- have also been found to be similarly equivalent for the though there are small differences in the bias for the summer period (not shown). average of the 44 winter cases, there is no statistically The geopotential height at 1000 hPa in Fig. 9 shows a significant difference in the SDE. When tested in the negative bias for both dynamical cores. This indicates a absence of physics forcings, no statistically significant loss of mass conservation, which is a consequence of the difference is found between the two dynamical cores in nonconservative nature of semi-Lagrangian advection. either bias or SDE (not shown). The meteorological This can be improved by introducing a simple global fields are interpolated from the TFC of the dynamics to mass fixer that works by conserving the global mean the s coordinate for physics through vertical interpola- surface pressure after each dynamics step in the model. tion which can lead to small differences in the verti- The scores in the presence of a global mass fixer are cal for the different definitions of the TFC. Physical shown in Fig. 10, where the bias at 1000 hPa is eliminated

Unauthenticated | Downloaded 09/25/21 04:19 PM UTC 2574 MONTHLY WEATHER REVIEW VOLUME 147

FIG. 10. As in Fig. 9, but with a global mass fixer in the simulations for both GEM-P and GEM-H to improve mass conservation. for both dynamical cores. The overall scores for the two dynamical core (Husain et al. 2019). This theoretical test cores are again, as expected, found to be equivalent in case is internally referred to as the ‘‘no flow’’ case. In an the presence of a global mass fixer. ideal situation, when marched forward in time, the at- Overall, the results presented in the Figs. 3–10 clearly mosphere in the ‘‘no flow’’ case is supposed to maintain demonstrate that the implementation of GEM-H as zero (or very negligible) motion. However, the differ- presented in this paper produces results that are equiv- ence between temperature of the atmosphere T0 and the alent to the existing GEM-P dynamical core. semi-implicit basic-state temperature T* generates numerical noise that grows with increasing steepness of the d. Stability over steep terrain mountain. This can lead to spurious vertical motions, A systematic study demonstrating the benefits of and with sufficiently steep terrain, the integration can the height-based vertical coordinate over a mass-based become unstable. Results from this test case (not shown) coordinate is beyond the scope of this paper. Without reveal that both GEM-P and GEM-H remain stable for implementing any special approach dedicated to im- terrain slopes below 528. However, GEM-P generally proving model stability (e.g., improving the approxi- leads to a more noisy and nonnegligible vertical motion mation of horizontal pressure gradient) the GEM-H field, particularly as the maximum terrain slope is in- dynamical core at its current form is not expected to creased beyond 458. With increasing terrain slope, the lead to any significant stability improvement. Never- noise level in the vertical motion field resolved by theless, the initial results (not shown) obtained for a GEM-P increases substantially and becomes unstable two-dimensional test case—involving a steep isolated as the maximum slope goes beyond 608. Regarding mountain under an isothermal atmosphere at rest— GEM-H, the simplified approach remains stable up to a point to stability improvements with the GEM-H maximum terrain slope of 528, whereas the maximum

Unauthenticated | Downloaded 09/25/21 04:19 PM UTC JULY 2019 H U S A I N E T A L . 2575 slope limit can be extended to almost 708 with the iter- equations are carried out using cubic interpolation. Al- ative approach. Overall, the height-based GEM-H is though it has not been shown explicitly, similarly to found to be less susceptible to noisy solutions and nu- GEM-P, in the absence of any off-centering in the dis- merical instability over steep terrain when compared cretized dynamical equations, numerical consistency in to the mass-based GEM-P dynamical core. These are, GEM-H requires a trapezoidal averaging of the source however, preliminary results. Caution needs to be ap- terms in the discretized trajectory equations along with plied when trying to extend the conclusions drawn from cubic interpolation for the wind fields at the departure the ‘‘no flow’’ test to three-dimensional NWP simula- positions. Furthermore, in the presence of off-centering tions with physics. For example, it should be noted that in the discretized dynamical equations, Schär’s moun- the slope limits can be somewhat dependent on the test tain case shows that the discretized sources terms in case. Therefore, a more systematic detailed study is re- the trajectory equations also necessitate off-centered quired to analyze and document the stability response averaging for the sake of consistent numerics in both of the two dynamical cores. GEM-H and GEM-P. In general, the knowledge ac- quired over years regarding the different numerical aspects of the GEM-P dynamical core is proven to be 6. Summary equally applicable to GEM-H. Comparisons between A newly developed dynamical core for ECCC’s GEM GEM-H and GEM-P for global deterministic predic- model with a height-based terrain-following vertical tions are also presented. The results are found to be coordinate is presented. The operational GEM-P dy- equivalent in the spectral space confirming that the namical core, which is based on a log-hydrostatic- new vertical coordinate does not lead to any spurious pressure-type vertical coordinate, suffers from strong noise or damping over the model-resolved scales. When numerical instability when subjected to steep orogra- compared against upper-air radiosonde observations, phy which becomes more prevalent with very high- except for small differences in bias, GEM-H and GEM-P resolution (subkilometer) NWP simulations. Also, the are found to produce statistically equivalent results. scalability limitation of the direct solver employed in The rationale behind the choice of the solution GEM-P is driving extensive research at ECCC aimed approach for the discretized EBV resulting from the at developing optimized three-dimensional iterative GEM-H formulation as well as the method for dynamics– solvers for future generations of massively parallel su- physics coupling is discussed in detail. Although the percomputers. A dynamical core with height-based TCF general structure of the EBV originating from the is expected to be better placed with respect to these GEM-H discretized system of equations require an ongoing NWP challenges at ECCC. The principal ob- iterative approach, a simplified approach has been jective of this paper is to provide information pertaining devised where the horizontally variable metric terms— to the different aspects of the new height-based dynamical attributable to the vertical-coordinate transformation— core that include the changes to the model formulation, are coupled with the nonlinear terms and are treated numerical discretizations, the solvers for the discretized with nonlinear iterations. This makes the EBV vertically problems and the strategy for coupling the new dynamical separable and allows the use of a direct solver as in core with the RPN physics package. Another important GEM-P. The fact that the GEM-H dynamical core objective is to demonstrate that the new GEM-H core is can utilize the direct solver approach for global and capable of making meteorological predictions that are regional-scale model resolutions makes it as efficient equivalent to the existing GEM-P dynamical core. as the operational GEM-P core. This renders GEM-H Numerical experiments have been conducted feasible for near-future implementations of opera- throughout the different stages of GEM-H develop- tional NWP systems at ECCC. Furthermore, a three- ment. Initially, the bubble convection test revealed that dimensional iterative solver based on FGMRES is the advection and the buoyancy effects in GEM-H are also developed for situations when the simplified ap- accurately represented and are producing results that proach is not applicable. are equivalent to GEM-P. When tested for the ideal- Improving any NWP model is a continuous process. ized Schär’s mountain case, the nonhydrostatic and hy- As a stable GEM-H dynamical core is now developed, a drostatic components of the mountain waves predicted number of other relevant issues are currently being by GEM-H are found to be very close to the GEM-P studied. The objective is to improve the GEM model in predictions as well as the analytical solution. The dy- general and the GEM-H dynamical core in particular. namics source terms in GEM are averaged over the air One of the most important short-term goal in this regard parcel trajectories using the trapezoidal method and is to devise a more numerically consistent and accurate the calculation of the rhs terms in the dynamical coupling between RPN Physics and GEM dynamics

Unauthenticated | Downloaded 09/25/21 04:19 PM UTC 2576 MONTHLY WEATHER REVIEW VOLUME 147 ! Yz which will beneﬁt both of the dynamical cores. Also, f T 5 1 tan XY XY 1 y 2 extensive research is being carried out to develop highly Ny f u u 1 dY q a T* optimized three-dimensional iterative solvers that can ! be competitive against the operationally used direct Yz T 2 Yz solver while scaling better for very large number of 2 y 2 s J J 1d q , (A2) T i Y z z processor cores for the future generations of massively * parallel supercomputers. Currently, the Yin–Yang sys- 0 T 2 T T tem uses the Schwarz method where the global solution N 5 y 2 s J 1 2 s d q 2 y 2 1 g y , (A3) w T i z d z T T is produced by iteratively solving two elliptic sub- * * y problems for the two subdomains (Yin and Yang) sep- N 5 0, (A4) arately and updating the solutions in the overlapping c regions until a certain convergence criterion is satisﬁed. 1 T T0 N 5 ln y 2 y , (A5) One promising solution to reduce the execution time of T t T T the iterative solver for the Yin–Yang system is to re- h * y move the Schwarz iterations and to solve the two elliptic N 5 0: (A6) subproblems as one by using FGMRES (Zerroukat and z Allen 2012). Also, preconditioners based on other types The corresponding rhs terms, in the absence of physics of methods (e.g., incomplete factorization, block Gauss– forcing, take the following forms: Seidel or multigrid method) could be used in order to " improve the convergence rate of the FGMRES solver. f 5 u 2 2 1 tan XY Finally, on the GEM-H front, another important short- Ru bm f u y tm a term objective is to improve its numerical stability over # Xz steep orography by implementing more accurate nu- T Xz 1 y d q 2 J J21d q merical approximation of the horizontal gradients in the ( X X z z ) , (A7) T* discretized dynamical equations. " y tanf Acknowledgments. The authors would like to sin- R 5 2 b f 1 uXY uXY y t m a cerely thank the members of the Numerical Methods m # Research Group at RPN for all their thoughtful com- Yz T Yz ments and suggestions during the course of GEM-H 1 y (d q 2 J J21d q ) , (A8) T Y Y z z development. The authors would like to particularly * thank Michel Desgagné,Stéphane Gaudreault, Rabah w T T0 Aider, and Vivian Lee for their valuable contributions R 5 2 b y J21d q 2 g y , (A9) w t nh T z z T during the implementation of the GEM-H source code. nh * y Also, comments from the internal reviewer, Dr. Chris- 1 q 1 topher Subich, have helped to considerably improve the R 5 1 lnJ 2b d u 1 d (cosfy) c t c2 z X cosf Y overall presentation of the paper. h * 1 _ 2 z dzz «w , (A10) APPENDIX A " # 1 T qz R 5 ln y 2 2b[mw], (A11) Nonlinear and rhs Components of the Discretized T t T c T h * pd * Equations z 2 z The N terms associated with Eqs. (20)–(24) and (29), R 5 2 b [z_ 2 w]. (A12) i z t nh corresponding to the linear components given by Eqs. nh (34)–(39), are provided below: ! Xz f T APPENDIX B 52 1 tan XY 1 y 2 Nu f u y 1 dX q a T* ! Deriving the EBV Problem Xz T Xz 2 y 2 s J J21d q , (A1) The lhs of the EBV problem to be solved at every T i X z z * time step in GEM-H is derived by manipulating the

Unauthenticated | Downloaded 09/25/21 04:19 PM UTC JULY 2019 H U S A I N E T A L . 2577 discretized system of equations of the form in Eq. (33). Beljaars, A., and Coauthors, 2018: The numerics of the physical The sequence of operations to derive the EBV is pro- parametrization in the ECMWF model. Front. Earth Sci., 6, vided below in terms of its linear components: 137, https://doi.org/10.3389/feart.2018.00137. Caya, A., R. Laprise, and P. Zwack, 1998: Consequences of using the splitting method for implementing physical forcings in a 1 1 lnJ L0 5 d L 1 d (cosfL )2 L 2 z , (B1) semi-implicit semi-Lagrangian model. Mon. Wea. Rev., 126, c X u Y y c , cosf tm th 1707–1713, https://doi.org/10.1175/1520-0493(1998)126 1707: COUTSM.2.0.CO;2. z 2 z Charney, J., and N. A. Phillips, 1953: Numerical integration of the L0 5 L 2 , (B2) z z t quasi-geostrophic equations for barotropic and simple baro- nh clinic flows. J. Meteor., 10, 71–99, https://doi.org/10.1175/1520- 0469(1953)010,0071:NIOTQG.2.0.CO;2. L0 5 L 1 L0 /t , (B3) w w z nh Charron, M., A. Zadra, and C. Girard, 2014: Four-dimensional tensor equations for a classical fluid in an external gravita- 0 5 1 0 LT LT mLz , (B4) tional field. Quart. J. Roy. Meteor. Soc., 140, 908–916, https:// doi.org/10.1002/qj.2185. 00 5G 0 1 0 ^ é LT (gthLT Lw), (B5) Cot , J., and A. Staniforth, 1988: A two-time-level semi-Lagrangian semi-implicit scheme for spectral models. Mon. Wea. Rev., 116, « z 2003–2012, https://doi.org/10.1175/1520-0493(1988)116,2003: L00 5 L0 1 L0 , (B6) c c z ATTLSL.2.0.CO;2. tm Gal-Chen, T., and C. Somerville, 1975: On the use of a coordinate z transformation for the solution of the Navier–Stokes equa- 000 5 00 1 00 2 00 Lc Lc dzLT «LT . (B7) tions. J. Comput. Phys., 17, 209–228, https://doi.org/10.1016/ 0021-9991(75)90037-6. Similar manipulations are also applied to the non- Girard, C., R. Benoit, and M. Desgagné, 2005: Finescale topogra- linear and rhs components of the discretized equations phy and the MC2 dynamics kernel. Mon. Wea. Rev., 133, 1463– to obtain R0 , N0 , R0 , N0 , R0 , N0 , R0 , N0 , R00 , N00 , R00, N00, 1477, https://doi.org/10.1175/MWR2931.1. c c z z w w T T T T c c ——, and Coauthors, 2014: Staggered vertical discretization of the 000 000 Rc , and Nc . Canadian Environmental Multiscale (GEM) model using a coordinate of the log-hydrostatic-pressure type. Mon. Wea. Rev., 142, 1183–1196, https://doi.org/10.1175/MWR-D-13-00255.1. APPENDIX C Husain, S., and C. Girard, 2017: Impact of consistent semi- Lagrangian trajectory calculations on numerical weather pre- Back Substitution to Obtain the Other Prognostic diction performance. Mon. Wea. Rev., 145, 4127–4150, https:// Variables doi.org/10.1175/MWR-D-17-0138.1. ——, ——, A. Qaddouri, and A. Plante, 2019: Global Environ- The solution of the EBV computes the unknown q. mental Multiscale model with a new terrain-following Workshop on PDEs on The rest of the variables are then calculated from the vertical coordinate based on height. the Sphere, Montréal, QC, Canada, Canadian Meteorological discretized dynamical equations through back sub- Centre, Environment and Climate Change Canada, http:// stitution in the following sequence: collaboration.cmc.ec.gc.ca/science/pdes-2019/pdfs/PDEs-2019_ paper_11.pdf. Xz u 5 t [R 2 N 2 (d q 2 s J J21d q )], (C1) Juang, H.-M. H., 2000: The NCEP mesoscale spectral model: A m u u X i X z z revised version of the nonhydrostatic regional spectral model. Mon. Wea. Rev., 128, 2329–2362, https://doi.org/10.1175/1520- Yz 5 2 2 2 21 0493(2000)128,2329:TNMSMA.2.0.CO;2. y tm[Ry Ny (dY q siJY Jz dzq )], (C2) Klemp, J. B., W. C. Skamarock, and O. Fuhrer, 2003: Numerical z_ 5 t fR00 2 N00 2G[(s J21 1 s )d q 2 mqz]g, (C3) consistency of metric terms in terrain-following coordinates. m T T i z d z Mon. Wea. Rev., 131, 1229–1239, https://doi.org/10.1175/1520- 0493(2003)131,1229:NCOMTI.2.0.CO;2. 5 _ 2 0 w z Lz , (C4) Laprise, R., 1992: The Euler equations of motion with hydrostatic pressure as an independent variable. Mon. Wea. Rev., 120, _ 21 197–207, https://doi.org/10.1175/1520-0493(1992)120,0197: z 2 T 5 gT g 2 1 (s J 1 1 s )d q 2 R0 2N0 . (C5) TEEOMW.2.0.CO;2. y * t i z d z w w nh Mahrer, Y., 1984: An improved numerical approximation of the horizontal gradients in a terrain-following coordinate system. Mon. Wea. Rev., 112, 918–922, https://doi.org/10.1175/1520- , . REFERENCES 0493(1984)112 0918:AINAOT 2.0.CO;2. Melvin, T., M. Dubal, N. Wood, A. Staniforth, and M. Zerroukat, Arakawa, A., 1988: Finite-difference methods in climate modeling. 2010: An inherently mass-conserving iterative semi-implicit Physically-Based Modelling and Simulation of Climate and semi-Lagrangian discretization of the non-hydrostatic verti- Climatic Change, M. E. Schlesinger, Ed., Springer, 79–168, cal-slice equations. Quart. J. Roy. Meteor. Soc., 136, 799–814, https://doi.org/10.1007/978-94-009-3041-4. https://doi.org/10.1002/qj.603.

Unauthenticated | Downloaded 09/25/21 04:19 PM UTC 2578 MONTHLY WEATHER REVIEW VOLUME 147

Müller, E. H., and R. Scheichl, 2014: Massively parallel solvers for Schär, C., D. Leuenberger, O. Fuhrer, D. Lüthi, and C. Girard, elliptic partial differential equations in numerical weather and 2002: A new terrain-following vertical coordinate formulation climate prediction. Quart. J. Roy. Meteor. Soc., 140, 2608– for atmospheric prediction models. Mon. Wea. Rev., 130, 2624, https://doi.org/10.1002/qj.2327. 2459–2480, https://doi.org/10.1175/1520-0493(2002)130,2459: Phillips, N. A., 1966: The equations of motion for a shallow rotating ANTFVC.2.0.CO;2. atmosphere and the ‘‘traditional approximation.’’ J. Atmos. Sci., Smolarkiewicz, P. K., and J. A. Pudykiewicz, 1992: A class of semi- 23, 626–628, https://doi.org/10.1175/1520-0469(1966)023,0626: Lagrangian approximations for ﬂuids. J. Atmos. Sci., 49, TEOMFA.2.0.CO;2. 2082–2096, https://doi.org/10.1175/1520-0469(1992)049,2082: Qaddouri, A., and V. Lee, 2010: The elliptic solvers in the Cana- ACOSLA.2.0.CO;2. dian limited area forecasting model GEM-LAM. Modeling, ——, R. Sharman, J. Weil, S. G. Perry, D. Heist, and G. Bowker, Simulation, and Optimization—Tolerance and Optimal Con- 2007: Building resolving large-eddy simulations and compar- trol, S. Cakaj, Ed., InTech, 109–124, https://doi.org/10.5772/ ison with wind tunnel experiments. J. Comput. Phys., 227, 633– 9038. 653, https://doi.org/10.1016/j.jcp.2007.08.005. ——, and ——, 2011: The Canadian Global Environmental Mul- Staniforth, A., N. Wood, and J. Cot^ é, 2002: Analysis of the nu- tiscale model on the Yin-Yang grid system. Quart. J. Roy. merics of physics-dynamics coupling. Quart. J. Roy. Meteor. Meteor. Soc., 137, 1913–1926, https://doi.org/10.1002/qj.873. Soc., 128, 2779–2799, https://doi.org/10.1256/qj.02.25. ——, L. Laayouni, S. Loisel, J. Cot^ é, and M. J. Gander, 2008: Vionnet, V., S. Bélair, C. Girard, and A. Plante, 2015: Wintertime Optimized Schwarz methods with an overset grid for the shallow- subkilometer numerical forecasts of near-surface variables in water equations: Preliminary results. Appl.Numer.Math., 58, the Canadian Rocky Mountains. Mon. Wea. Rev., 143, 666– 459–471, https://doi.org/10.1016/j.apnum.2007.01.015. 686, https://doi.org/10.1175/MWR-D-14-00128.1. Rivest, C., A. Staniforth, and A. Robert, 1994: Spurious resonant Yeh, K.-S., J. Cot^ é, S. Gravel, A. Méthot, A. Patoine, M. Roch, and response of semi-Lagrangian discretizations to orographic A. Staniforth, 2002: The CMC-MRB Global Environnemental forcing: Diagnosis and solution. Mon. Wea. Rev., 122, 366–376, Multiscale (GEM) model. Part III: Nonhydrostatic formula- https://doi.org/10.1175/1520-0493(1994)122,0366:SRROSL. tion. Mon. Wea. Rev., 130, 339–356, https://doi.org/10.1175/ 2.0.CO;2. 1520-0493(2002)130,0339:TCMGEM.2.0.CO;2. Robert, A., 1993: Bubble convection experiments with a semi- Zängl, G., 2012: Extending the numerical stability limit of terrain- implicit formulation of the Euler equations. J. Atmos. Sci., 50, following coordinate models over steep slopes. Mon. Wea. 1865–1873, https://doi.org/10.1175/1520-0469(1993)050,1865: Rev., 140, 3722–3733, https://doi.org/10.1175/MWR-D-12- BCEWAS.2.0.CO;2. 00049.1. Saad, Y., 1993: A ﬂexible inner-outer preconditioned GMRES Zerroukat, M., and T. Allen, 2012: On the solution of elliptic algorithm. SIAM J. Sci. Comput., 14, 461–469, https://doi.org/ problems on overset/Yin–Yang grids. Mon. Wea. Rev., 140, 10.1137/0914028. 2756–2767, https://doi.org/10.1175/MWR-D-11-00272.1.

Unauthenticated | Downloaded 09/25/21 04:19 PM UTC