MAY 2001 GRIFFIES ET AL. 1081
Tracer Conservation with an Explicit Free Surface Method for z-Coordinate Ocean Models
STEPHEN M. GRIFFIES AND RONALD C. PACANOWSKI NOAA/Geophysical Fluid Dynamics Laboratory, Princeton, New Jersey
MARTIN SCHMIDT Baltic Sea Research Institute WarnemuÈnde, Rostock Warnemunde, Germany
V. B ALAJI Cray/SGI, NOAA/Geophysical Fluid Dynamics Laboratory, Princeton, New Jersey
(Manuscript received 2 December 1999, in ®nal form 24 March 2000)
ABSTRACT This paper details a free surface method using an explicit time stepping scheme for use in z-coordinate ocean models. One key property that makes the method especially suitable for climate simulations is its very stable numerical time stepping scheme, which allows for the use of a long density time step, as commonly employed with coarse-resolution rigid-lid models. Additionally, the effects of the undulating free surface height are directly incorporated into the baroclinic momentum and tracer equations. The novel issues related to local and global tracer conservation when allowing for the top cell to undulate are the focus of this work. The method presented here is quasi-conservative locally and globally of tracer when the baroclinic and tracer time steps are equal. Important issues relevant for using this method in regional as well as large-scale climate models are discussed and illustrated, and examples of scaling achieved on parallel computers provided.
1. Introduction similar to that considered by Marsigli in the seventeenth century (see Gill 1982), and a global ocean simulation. An explicit free surface method has been developed Before detailing the algorithm and showing examples, for the z-coordinate Geophysical Fluid Dynamics Lab- it is useful to provide some context for the present work. oratory (GFDL) Modular Ocean Model (MOM) (for documentation of the model, see Pacanowski and Grif- ®es 2000). The scheme allows for the use of a free a. Tracer conservation and freshwater forcing surface with substantially improved stability properties For a Boussinesq ¯uid, volume conservation over a over the older approach of Killworth et al. (1991, re- vertical column yields the free surface height equation ferred to as KSWP in the following), as well as for
(U ϩ qw, (1 ´ t ϭϪ١ץ greatly improved conservation of tracers, such as heat and salt, relative to either the KSWP or Dukowicz and where is the deviation of the ¯uid surface elevation Smith (1994) approach. The result is an algorithm suit- able for climate as well as regional ocean models. from its resting position at z ϭ 0; U ϭ #ϪH u dz is the The purpose of this paper is to present the method vertically integrated horizontal velocity, also known as and to discuss its physical and numerical properties. the transport; and qw ϭ P Ϫ E ϩ R represents a volume Solution examples are provided of the Goldsbrough± ¯ux per unit area (with units of a velocity) of freshwater Stommel circulation (see Huang 1993) driven by surface passing across the air±sea interface due to precipitation, freshwater forcing, a baroclinic adjustment problem evaporation, and/or runoff. It is through this balance of volume that dynamical assumptions, which affect and U, determine the ability of a model to faithfully rep- resent freshwater forcing. Many ocean models employ a constant volume. For Corresponding author address: Dr. Stephen M. Grif®es, NOAA/ ,t ϭ 0ץ Geophysical Fluid Dynamics Laboratory, P.O. Box 308, Forrestal example, the rigid-lid approximation sets Campus, Princeton, NJ 08542. thence eliminating fast barotropic waves. With this as- E-mail: [email protected] sumption, volume conservation via Eq. (1) leads to the
᭧ 2001 American Meteorological Society 1082 MONTHLY WEATHER REVIEW VOLUME 129
t salinity is constant. For a free surface model, h ϭ |z1 | ϩ , where z ϭ z1 Ͻ 0 is the bottom of the surface tsץ(tracer cell, and so salt conservation means (|z1 | ϩ -t. When |z1 | k ||, this equation is often apץϭϪs t (e.g., KSWP; Dukowiczץts ϭϪsץ| proximated by |z1 and Smith 1994; Wolff et al. 1997; Marshall et al. 1997), which generally precludes salt conservation whenever the surface height changes, as through the addition of freshwater [see Barnier (1998) or Gordon et al. (2000) for a discussion of the virtual salt ¯uxes applied to con- stant volume ocean models]. Such limitations are also relevant for heat and other tracers. Our aim here is to relax this approximation and to provide a conservative scheme valid over a wide range of vertical resolutions,
including high resolutions where |z1 | ϳ ||. In principle, implementation of tracer forcing in ¯ux
FIG. 1. Schematic of rectangular surface tracer and velocity cells form is suf®cient to ensure that tracer is globally con- with the free surface. The tracer points are denoted by a solid dot served. However, with time-dependent cell thicknesses, and the tracer cells are enclosed by solid lines; a velocity point is details related to the discrete time stepping scheme, in- denoted by an ϫ and is enclosed by dashed lines. These points have cluding the use of time ®lters necessary with the three- ®xed vertical position z ϭ z1/2 Ͻ 0, regardless of the value for the t t time-level leapfrog scheme, serve to preclude strict con- free surface height. The thickness of a tracer cell is h ϭϪz1 ϩ , t servation. Additionally, global conservation is neither where zkϭ0 ϭ is the prognostic surface height determined through volume conservation [i.e., the vertically integrated continuity equa- necessary nor suf®cient to ensure locally conservative u u tion (1)]. The thickness of a velocity cell is h ϭϪz1 ϩ , where behavior. For example, consider an ocean at rest with u is determined by taking the minimum height of the four surround- globally constant tracer concentration. Then allow a ing tracer cells, as prescribed by the partial cell approach of Paca- nowski and Gnanadesikan (1998). Thicknesses of the cells are as- wind to blow so to initiate velocity and surface height t sumed to be positive, so that Ϫz1 ϩ Ͼ 0. In models without an ¯uctuations, yet do not introduce any tracer surface ¯ux- explicit representation of tides, this constraint in practice means that es. In addition to global tracer content remaining ®xed, |z1 | must be greater than roughly 2 m. For models with tides, |z1 | may the tracer concentration locally at every point in the need to be somewhat larger, depending on the tidal range considered. ocean will maintain the same constant value. This be- havior provides an example of ``local'' tracer conser-
-U, which forms the basis for the work vation, which is not a necessary result of global con ´ ١ balance qw ϭ of Huang (1993). The corresponding barotropic dynam- servation. Instead, it arises so long as there is compat- ics require both a streamfunction and velocity potential, ibility between tracer and volume budgets. thus introducing two elliptic problems. Elliptic problems are generally inef®cient to solve, especially in the pres- b. Splitting between fast and slow dynamics ence of realistic geometry, topography, and surface forc- ing, and moreso on parallel computers (see section 5). For computational ef®ciency, primitive equation Bryan (1969) made the further assumption that the ver- ocean models aim to exploit the timescale split between tically integrated velocity has zero divergence, resulting the fast barotropic gravity waves, which can propagate in a zero vertical velocity at the ocean surface: w(z ϭ at some 200 m sϪ1, and the much slower (some 100 -U ϭ 0. This assumption eliminates the ve- times slower) remaining dynamics. Such models ap ´ ϭϪ١ (0 locity potential, yet at the cost of precluding a direct proximate the barotropic mode by the depth-averaged freshwater forcing. motion, and the baroclinic mode is approximated by the A free surface model, in principle, should trivially al- deviation from the depth average. In a rigid-lid ocean low for the introduction of freshwater, since it only re- model, vertical averaging provides a clean split between quires the addition of qw to the free surface height equa- the fast and slow modes. In contrast, vertical averaging tion (1). However, doing so consistently and conserva- does not completely separate out the fast dynamics in tively requires some changes to the baroclinic and tracer a free surface model, since in this case the barotropic equations that are not commonly made. To illustrate this mode is weakly depth dependent. The papers by KSWP, point, consider what conservation means for total salt in Dukowicz and Smith (1994), Higdon and Bennett a single grid cell in the special case of zero salt ¯uxes, (1996), Higdon and de Szoeke (1997), and Hallberg but generally nonzero fresh water ¯uxes. With rectan- (1997) provide discussions of these issues, which can gular model cells (Fig. 1), salt is conserved if be quite subtle yet important for the purpose of provid- ing a stable split between the fast and slow modes in (hts) ϭ 0, (2) ץ t realistic ocean models. Additionally, details of this split where ht is the thickness of a model tracer cell. Salt have crucial implications for the tracer conservation
.t ϭ 0, means properties of the modelץ conservation with a rigid lid, in which MAY 2001 GRIFFIES ET AL. 1083 c. Modeling context and summary of key results [though advances are available, as documented by Gu- yon et al. (2000)]. MOM has traditionally been a rigid-lid ocean model, where the streamfunction algorithm of Bryan (1969) was used to solve for the barotropic component of the d. Contents of this paper dynamics. In the 1990s, important alternatives have This paper consists of the following sections. Section been added, including the explicit free surface of KSWP, 2 describes the model's discrete tracer equations. It high- the rigid lid±surface pressure of Smith et al. (1992) and lights points concerning tracer conservation in the pres- Dukowicz et al. (1993), and the implicit free surface of ence of a time-dependent surface cell. Section 3 de- Dukowicz and Smith (1994). Version 3 of MOM (MOM scribes the model's discrete equations of motion and 3) includes all of these algorithms within a single model details the time stepping algorithm for the barotropic framework. There are numerous additional develop- and baroclinic systems. Section 4 exhibits results from ments using a free surface in z-coordinate ocean models, idealized numerical examples that provide some prac- such as that used in the Ocean Circulation and Climate tical illustrations of the method. Section 5 discusses Advanced Modelling project (OCCAM; Webb et al. issues related to using the method in a parallel com- 1998), where the KSWP scheme is re®ned, as well as putational environment. Section 6 ®nishes the paper the Hamburg Ocean Primitive Equation (Wolff et al. with conclusions. 1997), Massachusetts Institute of Technology (Marshall et al. 1997), and OceÂan ParalleÂlise (OPA; Roullet and Madec 2000) ocean models, each of which employ an 2. Discrete tracer budget implicit time stepping algorithm. The present paper is The purpose of this section is to derive the discrete the result of an effort to assess each of these free surface tracer budgets within the free surface ocean model with schemes, and possible alternatives such as those of a time-dependent top cell volume. For orientation and Blumberg and Mellor (1987) and Mellor (1996), who notation, Fig. 1 details a zonal-depth cross section of use a terrain-following coordinate model, and Bleck and the rectangular control volumes over which the model's Smith (1990) and Hallberg (1997) who employ isopyc- surface equations are discretized. Notably, the position nal layered models. of a grid point within a cell is assumed to be ®xed in When testing the various free surface methods, we time, hence maintaining the Eulerian nature of MOM. kept the following goals in mind: 1) MOM's dynamical However, the corresponding grid cell volume generally core should provide a faithful representation of large- changes according to temporal undulations of the sur- scale ocean dynamics in an ef®cient manner for use in face height. We do not make the approximation that the both coarse (Ͼ1Њ) and ®ne (Ͻ1Њ) resolution global and surface height is small relative to any other length scale regional experiments. 2) Because of the increasing im- in the model, including the ocean depth. portance of parallel computation, the model should scale well as the number of computer processors increases. 3) The input of freshwater should occur naturally and a. Continuous time tracer budget allow tracer conservation. As described in this paper, Omitting meridional gradients for brevity, the con- our preferred approach is to use an explicit free surface tinuous time budget for the total amount of tracer within method, which is shown here to satisfy these three goals. a discrete tracer cell (Fig. 1) is given by There are important differences between the free sur- t ÄÄxx z z (t(Ah T) ϭϪdy(FFiϩ1 Ϫ ik) Ϫ A( FFϪ1 Ϫ k), (3ץ face method as described in this paper and the other free surface methods currently in use with z-coordinate where T is the tracer per unit volume, that is, a tracer models. First, the KSWP, Dukowicz and Smith (1993), concentration; A ϭ dx dy is the horizontal area of the t t Wolff et al. (1997), and Marshall et al. (1997) ap- tracer cell; h ϭϪz1 ϩ is the thickness of the tracer proaches assume the top model box to have ®xed vol- cell; i denotes the zonal grid point and k the vertical; ume for purposes of the tracer and baroclinic momentum and these labels are exposed only when needed. Also, budgets, and so do not satisfy the third goal above. FÄ x is the thickness weighted horizontal advective and Second, the OCCAM model (Webb et al. 1998) relaxes diffusive ¯uxes, which are computed as in Pacanowski this constraint, although details of their method are not and Gnanadesikan (1998) to account for the generally documented. As shown in this paper, allowing the top space±time-dependent grid cell thicknesses. z box to change in time is necessary, but not suf®cient, In Eq. (3),F k is the vertical turbulent and advective to globally and locally conserve tracer when using a tracer ¯ux that passes through the cell interface k, with z free surface method. Details of the time stepping scheme F k Ͼ 0 representing an upward ¯ux of tracer. In par- z are crucial and warrant careful analysis. Third, the OPA ticular,F kϭ0 represents the tracer ¯ux through the time- t model also relaxes the ®xed top cell volume constraint, dependent sea surface zkϭ0 ϭ . This ¯ux involves ver- yet it employs an implicit approach that is arguably less tical tracer advection relative to the undulating sea sur- straightforward than an explicit approach, in addition face, which is induced by freshwater ¯ux through the does not tend to perform as well on parallel machines sea surface, as well as parameterized turbulent ¯ux: 1084 MONTHLY WEATHER REVIEW VOLUME 129
zz,turb FFkϭ0 ϭϪkϭ0 qwTkϭ0. (4) c. Compatibility between volume and tracer budgets z Here,F kϭ0 must be calculated from a boundary con- Recall the example considered in section 1a, in which dition that equates this ¯ux with the total tracer ¯ux QT an ocean at rest with an initially constant tracer con- entering the ocean from other component models, such centration is perturbed via a wind stress yet without as atmosphere, river, and/or ice models. Generally QT tracer sources. Again, the solution should maintain the has a contribution from parameterized turbulence as same constant tracer concentration locally at each grid well as a tracer ¯ux with freshwater, cell, even as the surface cells undulate. To do so, it is Q ϭϪQturb q T , (5) necessary for there to be compatibility between the vol- T T w w ume and tracer budgets. In particular, upon setting the where Tw is the tracer concentration in the freshwater. tracer concentration to a constant, the tracer budget in Although Tw and Tkϭ0 may be of the same order of the surface grid cells must reduce to the discretized magnitude, the terms qwTw and qwTkϭ0 stand for different surface height equation (1). Compatibility between the physical processes. The term qwTw represents the budgets thus imposes a constraint on how to time step amount of tracer passing through the air±sea interface the surface height, and such is detailed in section 3c. with freshwater, whereas qwTkϭ0 represents the advec- tion of tracer in the ocean relative to the sea surface. Speci®cation of the surface tracer ¯ux involves details d. Total tracer content in the discrete model of how the air±sea interface is modeled. A simple ``clo- What is desired is the temporal combination of thick- sure'' for the turbulence term, which is often used for ness and tracer that measures the evolution of total tracer turb ocean-only simulations, is a restoring condition QT ϭ content in the model. Details of the time stepping t ␥h (T1 Ϫ T*), where T* is the reference tracer, ␥ is an scheme for both the tracer and surface height determine inverse damping time, and T1 is the time-lagged surface the form of such a measure; indeed, whether or not it tracer concentration. For a more re®ned coupling of exists. To illustrate these points, consider, as in the in- turb ocean and atmosphere, the ¯uxQT and the tracer Tw troduction, a single tracer cell in the absence of tracer have to be prescribed, for example, by a boundary layer ¯uxes. Doing so leads to the balance model. For salt, which does not pass the sea surface and ( h() ϭ 0, (7ץ( T() ϩ T(ץ(turb h( is conserved exactly, QT ϭ 0 and Tw ϭ 0 is appropriate (salt is considered in detail in section 2g). For other where the superscript on the tracer cell thickness is sup- tracers, approximations depend on the details of the pressed (h ϭ ht). Use of a leapfrog scheme for both boundary layer model [a simple approximation is given tracer and surface height, with equal time step , renders in section 2f; see also Pacanowski and Grif®es (2000) the discrete conservation law or Large and Pond (1981) for more discussion]. h()T( ϩ⌬) ϩ T()h( ϩ⌬) b. Time discretization ϭ h( Ϫ⌬)T() ϩ T( Ϫ⌬)h(). (8) The question arises whether it is preferable to time Alternative time stepping schemes for either tracer and/ step the thickness-weighted tracer concentration htT or or thickness will necessarily lead to distinct discrete the tracer concentration T. Tests with both approaches conservation laws, or even to the inability to identify were performed and showed negligible differences. No- such at all. We have explored a number of possibilities, tably, as shown below, both approaches have the same with the choice of leapfrog for both tracer and thickness temporal order of accuracy, and so choosing one is a being the most natural way to provide compatibility matter of convenience. As time stepping T involved the between the tracer and volume budgets. least changes to the existing MOM code, we detail that As previously mentioned, an alternative is to leap frog approach in the following. A similar reasoning applies the thickness-weighted tracer hT, which in the absence to the momentum equation time discretization discussed of boundary tracer ¯uxes leads to the discrete conser- in section 3. vation law: Dividing the tracer budget (3) by the tracer cell cross- h( ϩ⌬)T( ϩ⌬) ϭ T( Ϫ⌬)h( Ϫ⌬). (9) sectional area, and performing the product rule on t (t(h T), yields the tracer concentration budget Importantly, both of the conservation laws (8) and (9ץ are equivalent to second-order accuracy in ⌬. That is, t Ä x zz (t ϩϪFFkϪ1 k), (6ץxF Ϫ (␦k1TkץtT ϭϪץ h t(hT) ϭ 0 at the same rate as ⌬ץ they both reduce to where ␦k1 is the Kronecker delta, which is nonzero only is reduced. Hence, although nonstandard, the conser- for the surface cell when k ϭ 1. This budget is time vation law (8) resulting from separately time stepping stepped with a leapfrog scheme along with a Robert tracer concentration and thickness is just as valid as the time ®lter (e.g., Haltiner and Williams 1980) to suppress alternative (9). Again, our choice for (8) is based on time splitting. When needing the cell thickness, the convenience. thickness is set to that value at the current time step More generally, manipulation of the tracer concen- ht(). tration budget (6), again assuming that both the tracer MAY 2001 GRIFFIES ET AL. 1085 concentration and surface height are time stepped via a to the affects that freshwater has on the convergence U. This approximation simpli®es the setup of ´ leapfrog, leads, in the absence of boundary ¯uxes, to Ϫ١ conservation of the total tracer content as measured by boundary conditions for the tracer ¯ux. However, in general T and T are distinct and so T may need to A [h ()T ( ϩ⌬) ϩ ␦ T ()h ( ϩ⌬)], (10) w 1 w ij k k k1 kk be speci®ed explicitly from data or another component ijk model. where Aij is the horizontal area of a tracer cell. Again, this measure of total tracer content is the same, to sec- ond-order accuracy in ⌬, as the analogous measure g. The special case of salt realized via time stepping hT instead of T. The air±sea interface effectively acts as an impene- trable barrier for salts due to their large hydration en- e. Local compatibility versus global conservation ergy. Therefore, neglecting the formation of sea spray The compatibility condition, which leads to a locally and the interchange of salt with sea ice, and in the ab- conservative scheme, and global tracer conservation sence of salt entering through the solid earth boundaries, cannot be simultaneously maintained with a stable three- total ocean salt is constant. In turn, the salt concentration time-level discretization. For example, use of a time os within a grid cell, where s is salinity, changes only ®lter on either the tracer (via the Robert ®lter) or the through advective and turbulent ¯uxes from the interior surface height (via a time averaging procedure described ocean, as well as time tendencies in the volume of the in section 3), preclude strict conservation of the total grid cell. From the ¯ux expression (5), a zero salt ¯ux tracer content given by Eq. (10). However, these ®lters through the air±sea interface formally represents a bal- do not preclude local compatibility between volume and ance tracer budgets. Alternatively, omitting the time ®lters FFzz,ϭϪturb q s ϭ Q ϭ 0 (12) on the surface height and tracer in the tracer budget (11) kϭ0 kϭ0 w kϭ0 s allows for exact global conservation, yet such breaks between the turbulent ¯ux and advective ¯ux. Discus- the local compatibility so long as a time ®lter is used sion of this balance has been given by Huang (1993) in the surface height equation (11). We provide two and Dewar and Huang (1996), where the turbulent ¯ux examples in section 4 that aim to quantify these points. was called an antiadvective ¯ux. On the atmosphere side, Given the inability to provide exact discrete local and both the advective and turbulent ¯ux components van- turb global conservation properties with an undulating free ish, sw ϭ 0 and Qs ϭ 0. In relation (11), this result surface and the leapfrog scheme, we generally believe indicates that surface salinity is affected only through t term. A more thorough treatment of the salinityץit prudent that conservation properties should be as- the s1 sessed on a case by case basis. surface boundary condition is given by Beron-Vera et Alternative time stepping schemes exist, such as two- al. (1999). time-level schemes (e.g., Hallberg 1995, 1997), which With large salinity deviations, as may occur in the provide potential remedies to the issues raised here. presence of realistic freshwater ¯uxes, a more general However, pursuit of such alternatives goes beyond our approach than that discussed by Bryan and Cox (1972) present scope. appears necessary to compute ocean density. Bryan and Cox approximate the full United Nations Educational, f. Comments on surface ¯uxes Scienti®c and Cultural Organization (UNESCO) equa- tion of state by different cubic polynomials for each It is useful to further elucidate the different terms model depth level. This approach is warranted for mod- describing the surface tracer ¯uxes. For this purpose, els in which salinity is close to 35 psu and for which consider use of the free surface equation (1) and the the model's vertical cell thickness is independent of hor- surface tracer ¯ux expression (5), which provides the izontal position. The latter assumption is broken by the relation Pacanowski and Gnanadesikan (1998) partial bottom ϩ Q cells, and the former is broken by realistic freshwater ץ ϩϭFz T ץ T 1 t kϭ0 1 t T ¯uxes. Hence, for the purpose of computing the model's turb -U ϩ qw(T1 Ϫ Tw). (11) hydrostatic pressure, we employ the full UNESCO equa ´ ϭϪQT T1١ tion of state as formulated by Jackett and McDougall Notably, neither the tracer Tkϭ0 nor the turbulent ¯ux z,turb (1995). In this approach, in situ density is computed as F kϭ0 are explicitly needed. Relatedly, the distinction should be made between qwT1, which naturally appears () ϭ [(), s(), p( Ϫ⌬)], (13) here in the relation (11), and qwTkϭ0, which appears in z the expression forF kϭ0 in Eq. (4). where is the model's potential temperature and s is Assuming that the tracer concentration in the fresh- salinity. The pressure p is lagged one time step instead water is the same as in the surface cell, Tw ഠ T1, makes of performing the iterative approach described by Dewar the tracer time tendency independent of any explicit et al. (1998). The added cost of computing density via freshwater forcing. Instead, the dependence is restricted Jacket and McDougall's UNESCO formulation has been 1086 MONTHLY WEATHER REVIEW VOLUME 129 found to be a modest 5%±10% depending on the model momentum ¯ux with freshwater. Although the total mo- u con®guration. mentum ¯ux is continuous at zkϭ0 ϭ , the two com- ponents need not be. For example, the freshwater ve- locity u may be different from u , although most 3. Discrete momentum budget w kϭ0 climate models assume they are equal and take them
In this section, we formulate the discrete momentum equal to the horizontal velocity ukϭ1 in the top model budget, which shares much in common with the tracer grid cell. This point is further discussed below. budget just discussed, and then present a solution al- On the B grid used in MOM, the offset in the hori- gorithm for the model's approximated barotropic and zontal between velocity and tracer/density points means baroclinic modes. that the hydrostatic pressure at the western face of a surface velocity cell is given by y atm a. Continuous time momentum budget piiϭ g (|z1|/2 ϩ ii) ϩ p , (17) y Ignoring meridional gradients for brevity, the contin- where␣ is a meridional grid average and i is on the uous time budget for the zonal momentum of a Bous- tracer point. The ®rst piece of this pressure, pb ϵ y sinesq ¯uid occupying a rectangular velocity cell is giv- (g|z1 |/2)i , represents a discretization of the baroclinic en by pressure in the surface cell, whose horizontal gradients arise from baroclinicity in the density ®eld. Note that u u u (t(o Ah u)i ϭ (o Ah f)i Ϫ h dy(piϩ1 Ϫ piץ z1 has been assumed independent of horizontal position, ÄÄxx hence its removal from the meridional average operator. Ϫ ody(FFiϩ1 Ϫ i ) y The second contribution, ps ϵ gii, represents a dis- zz Ϫ o A(FFkϪ1 Ϫ k ). (14) cretization of the surface pressure, whose gradients arise In this equation, A is the horizontal area of the velocity from those in the density-weighted free surface height. atm cell, hu is its thickness (see Fig. 1 for the relation be- Here, p represents an atmospheric pressure contri- u t bution, which is dropped in the following for brevity, tween h and h ), and pi is the hydrostatic pressure acting on the vertical face of the velocity cell. As with the yet is maintained in the numerical code when coupling zonal tracer ¯ux, the momentum ¯ux FÄ x is weighted by to an atmospheric model. For Boussinesq models, it is common to approximate the generally time-dependent thickness of the cell. Met- y the surface pressure as ps ഠ g0 i , rather than the ric terms arising from momentum advection and friction y on a sphere (e.g., Grif®es and Hallberg 2000) have been hydrostatic formgii . With this approximation, sur- omitted for brevity, but they are present in the numerical face pressure gradients arise solely from gradients in the code.1 free surface height. To maintain self-consistency with Here, F z represents the vertical advective and tur- the hydrostatic baroclinic pressure ®eld, while incurring only trivial computational expense, we maintain the hy- bulent momentum ¯ux passing across the horizontal cell y z drostatic form ps ϭ giiin the model. faces. In particular,F kϭ0 represents the momentum ¯ux u passing through the ocean surface at zkϭ0 ϭ . It con- sists of the usual vertical turbulent momentum ¯ux and b. General strategy for the time stepping a contribution from the vertical advection of momentum relative to the moving sea surface, Manipulations of the budget (14) lead to the zonal velocity equation zz,turb FFkϭ0 ϭϪkϭ0 qwukϭ0. (15) Ä (xps/o ϩ Gk, (18ץtuk ϭ f k Ϫץ The surface ¯uxF z must be prescribed by the bound- kϭ0 where GÄ represents the forcing ary condition that the total vertical momentum ¯ux k u uuÄ ץthrough the air±sea interface is continuous at zkϭ0 ϭ . hGkkϭ hG kkϪ ␦ k1ukt That is, F z ϭ M, where the momentum ¯ux M, which kϭ0 huxp / FÄ x ץb oϪ ץarises from another component model such as an at- ϭϪ kx mospheric boundary layer, takes the form zz (). (19 ץϪ (FkϪ1 Ϫ Fkkϩ ␦ 1ukt Ϫ1 x u M ϭϪ(o winds ϩ qwuw), (16) Notably, the cell thicknesses h used for computing the which has contributions from wind stress as well as terms in Gk are taken at baroclinic time , as are the other inviscid contributions such as pressure and ad- vection.
The freshwater velocity uw in the surface momentum 1 Additionally, the constant Boussinesq density, o, is not set to 1 ¯ux (16) requires additional efforts to specify the com- gcmϪ3, as previously done in the Bryan±Cox±Semtner model (Bryan plete boundary conditions. However, the momentum 1969; Cox 1984; Semtner 1974) or previous versions of MOM (Pa- Ϫ3 ¯ux with freshwater is in many cases smaller than the canowski et al. 1991). Instead, o ϭ 1.035 g cm , from which density in the World Ocean generally deviates by less than 2% (Gill 1982, uncertainty in the parameterized wind stress. So, simple Ϫ3 p. 47), whereas using 0 ϭ 1.0gcm is less accurate. approximations for uw may be exploited to reduce the MAY 2001 GRIFFIES ET AL. 1087
1) ALGORITHM BASICS Allowing the grid cell thicknesses to evolve intro- duces a fundamentally new element to the traditional algorithms relevant for constant cells in z-coordinate FIG. 2. Schematic of the split-explicit time stepping scheme. Time models (e.g., Bryan 1969; Semtner 1974; Cox 1984; increases to the right. The baroclinic time steps are denoted by Ϫ ⌬, , ϩ⌬, and ϩ 2⌬. The curved line represents a baroclinic Killworth et al. 1991; Dukowicz and Smith 1994). How- leapfrog time step, and the smaller barotropic time steps N⌬t ϭ 2⌬ ever, since we impose positive cell thicknesses for all are denoted by the zigzag line. First, a baroclinic leapfrog time step cells, including the surface, modi®cations to the con- updates the baroclinic mode to ϩ⌬. Then, using the vertically stant cell approach should be relatively modest. That is integrated forcing G() [Eq. (21)] computed at baroclinic time step , a forward±backward time stepping scheme integrates the surface a goal of the proposed algorithm. height and vertically integrated velocity from to ϩ 2⌬ using N The basic idea is directly analogous to the approach barotropic time steps of length ⌬t, while keeping G() and the ocean of Bryan (1969), Semtner (1974), and Cox (1984). That depth D() ®xed. Time averaging the barotropic ®elds over the N ϩ is, the velocity at an arbitrary depth level k and baro- 1 time steps (endpoints included) centers the vertically integrated clinic time Јϭ ϩ⌬ is split into two components: velocity and free surface height at baroclinic time step ϩ⌬. Note that for accurate centering, N is set to an even integer. uk(Ј) ϭ Bkm()um(Ј) ϩ [␦km Ϫ Bkm()]um(Ј)
ϵ uà k(, Ј) ϩ u(, Ј). (22) model complexity. Using the free surface height equa- This equation is an identity valid for any baroclinic z tion (1), and the surface ¯ux F kϭ0 ϭ M, with M given times Ј and , with Јϭ ϩ⌬ chosen in the fol- by Eq. (16), leads to lowing. The utility of the identity relies on the form of the ``baroclinicity operator'' used to split the velocity z Ϫ1 x U ®eld ´ t ϩϭϪϪFkϭ0 o winds u1١ץu1
Ϫ1 u ϩ qw(u1 Ϫ uw). (20) Bkm() ϭ ␦km Ϫ D() hm(), (23)
Hence, the approximation uw ഠ u1 removes the explicit where ␦km is the Kronecker delta, summation over the dependence of baroclinic momentum on the freshwater repeated vertical level index m is implied, and ¯ux. Freshwater does in¯uence baroclinic momentum D() ϭ huu() ϭ D ϩ () (24) indirectly, however, through its affects on the conver- ko k U of the vertically integrated velocity. The ´ gence Ϫ١ approximation uw ഠ u1 should be well justi®ed for many is the ocean depth at baroclinic time over a column cases, and generalizations for special cases such as of velocity points, with Do the resting ocean depth. heavy rainfall are straightforward. This baroclinicity operator projects out the depth-in- For the barotropic mode, MOM time steps the ver- dependent part of a ®eld and keeps its approximate bar- oclinic portion, where the projection is based on the tically integrated transport U ϭ⌺k hkuk, as is common in shallow water models. Its evolution takes the form distribution of cell thicknesses at time . Introduction of two baroclinic time labels and Ј to Eq. (22) is p / ϩ G, (21) necessitated by the freedom afforded the ocean depth ץu ϭ fVϪ D ץ ϩ h ץ U ϭ u ץ t 1 tktkxso k to change in time. That is, Ј represents the baroclinic time of the full velocity ®eld u (Ј), whereas repre- u , the vertically inte- k ץ where Eq. (18) was used for t k sents the time used to de®ne the baroclinicity operator grated forcing is given by G ϭ⌺ h G with G de®ned k k k k B () and which de®nes the split. in Eq. (19), and D ϭ⌺ h is the time-dependent ocean km k k If the split introduced in Eq. (22) is successful, depth. Once the transport and ocean depth are updated, uà (, Ј) will evolve on a slow timescale, ⌬, and the updated barotropic velocity can be diagnosed k u(, Ј) will evolve on the fast timescale, ⌬t ϭ 2N Ϫ1⌬, through u( ϩ⌬) ϭ U( ϩ⌬)/D( ϩ⌬). with N determined by the ratio of external to internal gravity wave speeds. The method therefore proceeds by c. Time stepping algorithm separately updating uà k(, Ј) and u(, Ј) by exploiting the timescale split. Upon doing so, the right-hand side The focus in this section is on time and depth dis- of the identity (22) will be speci®ed, hence allowing for cretization, with Fig. 2 summarizing the following al- an update of the full velocity ®eld via gorithm. For purposes of brevity, the horizontal spatial u ( ϩ⌬) ϭ uà (, ϩ⌬) ϩ u(, ϩ⌬). (25) discretization discussed in the previous subsection will k k not be exposed. Discrete baroclinic times and time steps Upon updating the velocity cell thickness to hu( ϩ will be denoted by the Greek and ⌬, respectively, ⌬), the baroclinic and barotropic velocities uà k( ϩ⌬) whereas the barotropic analogs will use the Latin t and and u( ϩ⌬) can then be diagnosed. The following ⌬t. subsections detail this approach. 1088 MONTHLY WEATHER REVIEW VOLUME 129
2) COMPUTING uà k(, ϩ⌬) tn ϭ ϩ n⌬t, (29) By construction, evolution of the velocity ®eld with n ∈ [0, N]. For stability purposes, we found it uà k(, Ј) is unaffected by vertically independent forces, important to take the initial condition *(,tnϭ0) as the such as those from surface pressure gradients. There- time average *() computed from the previous baro- fore, it is suf®cient to update the ``primed'' velocity: tropic integration taking place over Ϫ⌬ to ϩ⌬ [time averaging is de®ned by Eq. (32) discussed below]. R Ä uЈkk( ϩ⌬) ϭ u ( Ϫ⌬) ϩ 2⌬[ f k() ϩ Gk()], (26) Note that it is assumed that the freshwater ¯ux qw is which represents a temporal discretization of the full constant over the small barotropic time steps, since the momentum equation (18), yet without the surface pres- hydrological ¯uxes are typically updated on each bar- sure gradient, thus allowing for use of the baroclinic oclinic time step. R Having the surface height * updated to the new time step. The lagged velocityuk ( Ϫ⌬) ϭ uk( Ϫ R barotropic time step allows for an update of the transport ⌬) ϩ (␣/2)[uk() Ϫ 2uk( Ϫ⌬) ϩ uk ( Ϫ 2⌬)] is a Robert time-®ltered version of the full velocity ®eld. V*(, t ) ϩ V*(, t ) A weak form of such ®ltering, with ␣ ϭ 0.01, has been U*(, t ) ϭ U*(, t ) ϩ f⌬t nnϩ1 found suf®cient to suppress splitting between the two nϩ1 n []2 leapfrog branches (e.g., Haltiner and Williams 1980). It (pÄ*(, t )], (30 ץ(ϩ⌬t[G() Ϫ D( is also useful to dampen fast dynamics that may partially xs nϩ1 leak through the baroclinicity operator due to the gen- wherepÄ*s (,tnϩ1) ϭ g*(,tnϩ1)()/o is the surface erally imperfect separation between the slow and fast pressure normalized by the Boussinesq density. The dynamics. Coriolis force is computed using a Crank±Nicholson
The baroclinic piece of the primed velocityuЈk ( ϩ semi-implicit time stepping scheme, which was also ⌬) yields used by KSWP (see also Haltiner and Williams 1980). After N barotropic time steps, the vertical transport uà (, ϩ⌬) ϭ B ()(uЈ ϩ⌬), (27) k km m and surface height are time averaged to produce as can be shown by adding the missing surface pressure 1 N gradient to Eq. (26), where it is annihilated by appli- U*( ϩ⌬) ϭ U*(, t ) (31) n N ϩ 1 nϭ0 cation of Bkm(). At this point, we have speci®ed one- half of the updated full velocity given by Eq. (25). 1 N *( ϩ⌬) ϭ *(, t ). (32) n N ϩ 1 nϭ0 3) COMPUTING u(, ϩ⌬) The time-averaged ®elds are centered on the baroclinic The computation of u(, ϩ⌬) constitutes a most time step ϩ⌬, so long as N is an even integer. The crucial part of any free surface algorithm, since details time-averaged surface height *( ϩ⌬) is used to largely determine the stability and conservative aspects initialize the next suite of barotropic integrations from of the scheme. The following approach, which uses a ϩ⌬ to ϩ 3⌬, and U*( ϩ⌬) is used to update time averaging over barotropic time steps integrated u(, ϩ⌬) via from to ϩ 2⌬, is commonly used in explicit free u(, ϩ⌬) ϭ U*( ϩ⌬)/D(). (33) surface methods. However, its details differ somewhat from other approaches, and so prompts a reasonably Having u(, ϩ⌬) then allows for the full velocity thorough presentation. ®eld uk( ϩ⌬) to be updated according to Eq. (25). The barotropic ``subcycling'' uses a forward±back- ward time stepping scheme, also used by KSWP (see also, e.g., Haltiner and Williams 1980). In this approach, 4) COMPUTING ( ϩ⌬) AND u( ϩ⌬) surface height is ®rst time stepped using the small bar- Although the updated full velocity ®eld uk( ϩ⌬) otropic time step ⌬t: is now known, the updated surface height ( ϩ⌬) *(, t ) ϭ *(, t ) and barotropic velocity u( ϩ⌬) ϭ U( ϩ⌬)/D( nϩ1 n ϩ⌬) remain to be determined. A number of options
U*(, tnw) Ϫ q ()]. (28) were tried, with the following two approaches initially ´ ١]Ϫ⌬t attempted. (a) Stop the barotropic subcycling at ϩ⌬ The asterisk is used to denote intermediate values of the and use the unaveraged barotropic ®elds: barotropic ®elds, each of which are updated on the bar- otropic time steps. The time label on * and U* de- ( ϩ⌬) ϭ *(, ϩ⌬) and (34) notes the baroclinic time at which the vertically inte- U( ϩ⌬) ϭ U*(, ϩ⌬). (35) grated forcing G(), the tracer ®elds, and the total depth of the ocean D() are held for the duration of the bar- This choice was originally used by KSWP, yet we, and otropic time stepping from to ϩ 2⌬. This is also others such as the OCCAM group (see Webb et al. the time that sets the barotropic time steps via 1998), have found it to result in a very unstable algo- MAY 2001 GRIFFIES ET AL. 1089 rithm and so it is not suitable. (b) Use of the time- d. Comments on the algorithm averaged barotropic ®elds given by Eqs. (31) and (32), 1) GENERALITY ( ϩ⌬) ϭ *( ϩ⌬) and (36) Although the immediate application of the algorithm U( ϩ⌬) ϭ U*( ϩ⌬), (37) is for the free surface method, the algorithm allows any thickness within a vertical column to vary in time, so allows for a substantial improvement in stability, and it long as it remains positive. In particular, it is thought is for this reason that OCCAM chooses this approach. that this approach will ®nd use for models with a time- However, this choice is not self-consistent in the sense varying bottom boundary layer thickness, such as that that U( ϩ⌬) so de®ned will not generally be equiv- proposed by Killworth and Edwards (1999). u alent to ⌺k hk ( ϩ⌬)uk( ϩ⌬) due to time-dependent thicknesses. Additionally, use of the time average for the surface height precludes compatibility between the 2) STABILITY tracer and volume budgets as discussed in section 2c, and it also does not allow for a speci®cation of a dis- As expected, the baroclinic and barotropic time steps cretely conserved total tracer as described in section 2d. are determined by the usual Courant±Freidrichs±Lewy Therefore, we propose the following alternative ap- constraints set by waves and advection. Notably, time proach that satis®es our goals. averaging has not been found to adversely affect the We ®rst update the thicknesses, and then diagnose the propagation of barotropic waves relative to the results integrated transport. To do so, compatibility as discussed using the KSWP approach. We comment further on this in section 2c indicates that since the tracer concentration point when presenting a version of the Marsigli problem is time stepped with a leapfrog scheme, so too must the in section 4c. surface height budget: Time averaging over the barotropic steps has been found to stabilize the model so that stretching of tracer ( ϩ⌬) ϭ *( Ϫ⌬) time steps to values larger than the baroclinic time step is readily available. The stretching of tracer time steps U() Ϫ qw()]. (38) is ubiquitous when spinning up to a thermodynamic ´ ١]Ϫ 2⌬ equilibrium coarse-resolution, rigid-lid, z-coordinate As the surface height generally evolves on a fast time- ocean models (e.g., Bryan 1984; Killworth et al. 1984; scale, use of this proposed baroclinic leapfrog time step Danagasoglu et al. 1996). Therefore, we consider the is generally unstable. To maintain stability, as well as added stability of the present scheme to be of great general smoothness of the free surface ®eld, it has been practical value. found necessary to use the time-averaged surface height *( Ϫ⌬) rather than a more traditional Robert ®ltered height. Doing so does not effect the compatibility con- 3) COMPUTATIONAL TIMING dition, and so has been found suitable. Using this ``big leapfrog'' step to de®ne ( ϩ⌬) then allows for an Because of the ability to match the baroclinic and update of the surface tracer and velocity cell thicknesses tracer time steps to those commonly used in rigid-lid tu hhkk( ϩ⌬) and ( ϩ⌬). models, the present scheme has been found to be com- Given the updated cell thicknesses, the updated trans- putationally comparable to the rigid lid on a single com- port U( ϩ⌬) can be diagnosed puter processor. Furthermore, due to the absence of the topographic instability present in rigid-lid models, U( ϩ⌬) ϭ hu( ϩ⌬)u ( ϩ⌬), (39) kk which was described by Killworth (1987) and Dukowicz k et al. (1993), the free surface is more economical with thus ensuring self-consistency between the updated full nontrivial topography. Section 4a provides an example to support these comments, and section 5 presents a velocity uk( ϩ⌬) and the updated vertically inte- grated velocity. Finally, to complete the time stepping, discussion of computational aspects relevant for parallel the updated barotropic velocity is diagnosed through machines. Relatedly, when using partial bottom cells of Paca- u( ϩ⌬) ϭ U( ϩ⌬)/D( ϩ⌬), (40) nowski and Gnanadesikan (1998), as commonly used now in MOM for representing the bottom topography, u where D( ϩ⌬) ϭ⌺hk ( ϩ⌬) is the new depth of use of the time-dependent top model cells engenders a velocity cell column. The updated baroclinic velocity only a trivial added cost relative to the case of constant follows from its de®nition uà k( ϩ⌬) ϭ uk( ϩ⌬) top model cell volumes. This experience is in contrast Ϫ u( ϩ⌬). Note that a similar reconciliation or re- to the 10% added cost of allowing the surface cell to adjustment of the velocities is discussed by Hallberg vary in the implicit method of Roullet and Madec (1995, p. 221) for use in his isopycnal model. (2000). 1090 MONTHLY WEATHER REVIEW VOLUME 129
-zw ϭ 0 over a rectץu ϩ ´ ١ GRID SPLITTING of the continuity equation (4 angular top model grid cell: As pointed out in KSWP, the surface height on a B grid is prone to grid splitting, which manifests as a (u. (41 ´ w ϭ w() ϩ dz١ checkerboard pattern. The present scheme is less prone kϭ1 ͵ to this splitting than other algorithms we tried, and nu- z1 merous tests indicate that the splitting is easily sup- Use of the surface kinematic boundary condition pressed with mild ®ltering as described by KSWP or ( ϭ w() ϩ qw, (42[١´(t ϩ u(ץ] other ®lters described in Pacanowski and Grif®es ١z ϭ 0, renders Notably, as the nonlinear free surface described and Leibnitz's rule with .(2000) here incorporates the surface height undulations into the 1 tracer and momentum equations, suppression of the grid dzu ´ ١ Ϫ q ϩץw ϭ splitting is more important than in the linearized free kϭ1 tw ͵ surface methods. z1 (dzu, (43 ´ ١ U ϩ ´ ϭϪ١ 5) VOLUME CONSERVATION ͵ z1 To conserve the model volume, it is suf®cient to en- where the last step used the vertically integrated conti- sure that the surface height evolves in a conservative nuity equation (1). This exact expression for wkϭ1 is ap- manner. Both explicit time stepping discretizations (28) proximated in the model with a discretized version of and (38) trivially satisfy such conservation. Therefore, (hu), (44)´ ١ U ϩ ´ all model tests indicate that volume is conserved to wkϭ1 ഠ Ϫ١ within computer roundoff. where h ϭ ϩ |z1 | is the surface cell thickness and the horizontal velocity u is that in the surface cell k ϭ 1. 6) ENERGETIC CONSISTENCY Notably, unlike most z-coordinate free surface models, there has been no linearization of the surface kinematic Energetic consistency of the methods used here have boundary condition made by dropping the surface height .١´(been shown elsewhere. First, the arguments given by advection u( -U, di ´ Bryan (1969) account for the momentum advection Given an expression for the convergence Ϫ١ terms, which are discretized using second-order cen- agnosing wkϭ1 in this manner allows for the remaining tered differences. The result is a redistribution of local interior vertical velocities to be successively found kinetic energy, yet a preservation of its global integral. through further integration of the continuity equation Second, the arguments in the appendix of Pacanowski downward through a vertical column. On a B grid, (U is centered on a tracer point. Hence, Eq. (44 ´ and Gnanadesikan (1998) show that the partial cell Ϫ١ methods ensure that the change in energy due to hori- yields the vertical velocity on the bottom face of the zontal pressure forces balances potential energy change surface tracer cell. The vertical velocity on the sur- when density is linearly dependent on temperature and rounding velocity cells is constructed as a volume con- salinity. Importantly, this consistency is realized only serving average of the surrounding tracer cell vertical when incorporating the undulating surface grid cell velocities. thickness into both the tracer and baroclinic velocity In MOM, the bottom of the ocean on tracer cells is equations, as done here. Third, appendix B of Dukowicz a ¯at surface, representing the ``lopped off'' surfaces and Smith (1994) provides a complementary analysis of topography.2 Hence, the vertical velocity must vanish of the energetic consistency within their implicit free at this location. A self-consistency check on how ac- surface method, much of which is relevant for the pre- curately the model's numerics conserve volume amounts sent considerations. to testing how well this property is satis®ed when in- tegrating downward from the ocean surface, starting from the vertical velocity given by Eq. (44). Adding 7) VERTICAL VELOCITIES freshwater to the model provides a nontrivial test of Vertical velocities are diagnosed at baroclinic time these properties. The present scheme produces zero ver- steps using volume conservation within a grid cell. In tical velocities at the ocean bottom on tracer cells, to particular, volume conservation over a surface cell in- within computer roundoff, regardless of the topography dicates that the vertical velocity at the bottom face of or surface forcing. this cell arises from the horizontal convergence of vol- ume in this cell, time tendencies in the thickness of this cell, and volume passing across the top face from fresh- 2 The bottom velocity cell generally does not sit on the ocean water ¯uxes. bottom, and so can support a vertical velocity due to sloping topog- It is useful to see precisely how these velocities are raphy. Details of how MOM handles this velocity are given in Webb determined. For this purpose, consider a vertical integral (1995) as well as in Pacanowski and Grif®es (2000). MAY 2001 GRIFFIES ET AL. 1091
8) ORDER OF THE SOLUTION METHOD quite low, often requiring only a single scan to update the streamfunction. The method presented above solves for the updated More realistic bottom topography, surface forcing, ``quasi-baroclinic'' velocity uà ( , ) prior to the k ϩ⌬ and mesoscale eddies will greatly increase the number updated quasi-barotropic velocity u(, ϩ⌬). The of elliptic solver scans required by the rigid-lid model. pre®x quasi is used since these velocities are de®ned In practice, more realistic situations often prompt one with respect to the ocean depth at time instead of to greatly reduce or loosen the criteria used for deter- ϩ⌬. This approach is motivated largely because it mining a solution to the elliptic problem. Indeed, in allows for minimal fundamental changes to the MOM some cases it is dif®cult to argue that a ``solution'' has 3 structure, in which the baroclinic and tracer equations even been found. In contrast, the ratio of barotropic to are solved prior to the barotropic equations. However, baroclinic wave speeds is independent of model reso- a more straightforward approach is to solve for the up- lution and surface forcing. Rather, it is determined by dated surface height ( ϩ⌬) and barotropic velocity the bottom topography and strati®cation. Consequently, u( ϩ⌬), and then to update the baroclinic and tracer for the explicit free surface method, there is no issue equations. This latter approach is being pursued in regarding convergence to a solution, as convergence is MOM 4. guaranteed by construction. As with KSWP and Dukowicz and Smith (1994), we 4. Numerical examples check numerical integrity of the free surface solution by comparing with the more highly tested rigid-lid re- For the purpose of numerically illustrating the al- sults. For example, Fig. 3 provides a sample of the free gorithm, we present selected results from four experi- surface solution; the rigid-lid results are nearly identical. ments. The ®rst and second are from an idealized, ¯at- Since the vertical velocity at the ocean surface vanishes bottom, coarse-resolution, sector model driven by time- at equilibrium in a constant forced, coarse-resolution independent buoyancy and momentum forcing. Details free surface model without freshwater forcing, the two of the con®guration are provided in the caption to Fig. methods should indeed provide nearly identical answers. 3. The third example is that of a higher-resolution re- A detailed comparison (not shown) reveals negligible gional model with two basins connected by a channel differences for all aspects of the simulations. Other mod- with a shallow sill. This con®guration is shown in detail el con®gurations also have been run, with similar com- later (Fig. 5). The ®nal example is a coarse-resolution parisons. global model that quanti®es the issues of local versus We conclude from these tests that the explicit free global conservation. surface method maintains the convenience of the rigid lid for purposes of idealized coarse-resolution modeling. a. Free surface compared to rigid lid The solutions are likewise nearly the same when run under the same forcing. Both of these conclusions have We start by providing a direct comparison between remained valid with more realistic climate model ex- the explicit free surface and the rigid-lid streamfunction periments now routinely run at GFDL. method in the sector model. Each experiment used the same tracer time step of 1 day and the same baroclinic time step of 1 h. The rigid-lid experiment used a 1 h b. Goldsbrough±Stommel circulation time step for the streamfunction, whereas the free sur- As discussed by Huang (1993), the Goldsbrough± face used 300 s for the barotropic time step. With the Stommel circulation arises from freshwater forcing, and time averaging used with the free surface method, there this circulation is absent in traditional rigid-lid models are a total of 24 barotropic time steps integrated for that use the virtual salt ¯ux. Figure 4 illustrates this each baroclinic time step. Both the free surface and circulation when run with the free surface for 4000 yr. rigid-lid methods used a virtual salt ¯ux and zero fresh- The freshwater forcing, as used by Huang, is given by water forcing. The experiments were run for 4000 tracer years, at q0 Ϫ q ϭ wS1 Ϫ 2 , (45) which point both reached a quasi-equilibrium. When run w cosϪ on a single processor on GFDL's Cray T90, the free NS 0 Ϫ1 surface method took a few percent longer than the rigid- where qw ϭ 1myr and N and S are the northern lid method. Although this comparison is a function of and southern latitudinal boundaries, respectively. The model con®guration and computer details, as well as domain integral of this water ¯ux vanishes. There is no elliptic solver algorithm, the key point is that such a wind forcing nor temperature forcing, and temperature comparison is impressive for the free surface model, is uniform. The barotropic circulation is quite weak and since the rigid-lid model is ideally suited for such a is comparable to that obtained by Huang. The baroclinic simple con®guration. That is, the ¯at-bottom, time-in- circulation, as documented through the overturning dependent forcing, and absence of mesoscale eddies streamfunction, is strong and saline direct, yet somewhat means that the number of elliptic solver scans can be weaker than that of Huang, perhaps because of our use 1092 MONTHLY WEATHER REVIEW VOLUME 129
FIG. 3. The sector model con®guration employed here consists of a ¯at-bottom 60Њϫ60Њ sector from 5Њ to 65ЊN. The grid is locally square, so that ⌬ ϭ⌬ cos, where ⌬ ϭ 2.5Њ is the longitudinal resolution. There are 18 unevenly spaced grid points in the vertical. The vertical tracer diffusivity is 0.5 cm 2 sϪ1, the vertical momentum viscosity is 10 cm2 sϪ1, the horizontal tracer diffusivity is 107 cm2 sϪ1, and horizontal momentum viscosity is 2 ϫ 109 cm2 sϪ1. The Quick scheme of Leonard (1979), as detailed in Holland et al. (1998), is used for tracer advection. Temperature in the surface level of 35-m resting thickness is restored to the linear temperature pro®le given by Cox and Bryan (1984). A 35-day timescale is used, which delivers roughly a 47 W mϪ2 (ЊC)Ϫ1 ¯ux sensitivity. Salinity is uniform. Winds are zonal as given by Bryan (1987). The rigid-lid model used a conjugate gradient solver. Shown here are results from the free surface model after 4000 tracer years; the rigid-lid results are virtually identical. (upper left). Zonally averaged temperature. (lower left) Overturning streamfunction (Sv ϭ 106 m3 sϪ1). (upper right) Barotropic stream- function (Sv). (lower right) Free surface height (cm). Note that the surface height diagnosed from the rigid-lid solution is quite close to that shown here from the free surface method. of half the vertical diffusivity. It remains unclear how When integrating the model to reach the solution in important this circulation is for realistic climate simu- Fig. 4, we used 1-day tracer and 1-h baroclinic velocity lations, since it is often easily masked in the presence time steps as in the previous experiment. Although nei- of wind and heat forcing. Regardless, its presence is a ther salt nor water were added to the model, the total natural consequence of the use of freshwater forcing salt does not remain precisely constant during the in- with the free surface method. tegration. The reason, as discussed in section 2, is that MAY 2001 GRIFFIES ET AL. 1093
FIG. 5. The model con®guration consists of a domain from roughly 51Њ to 66.5ЊN, 35Њ in long, and 100-m depth. The resolution is 0.3Њ long, 0.15Њ lat, with uniform 4-m-thick boxes. A sill is placed in the middle of the domain, running from south to north and reaching to within 20 m of the surface. Partial bottom cells of Pacanowski and Gnanadesikan (1998) are used to represent the sill. Tracer and bar- oclinic momentum time steps are 240 s, and the barotropic time step is 60 s. The vertical tracer diffusivity is 1.0 cm2 sϪ1, the vertical momentum viscosity is 10 cm2 sϪ1, the horizontal tracer diffusivity is 107 cm2 sϪ1, and the horizontal momentum viscosity is 5 ϫ 108 cm2 sϪ1. The model is initialized from rest with zero sea level ele- vation and with a uniform linear temperature pro®le from 10ЊCat the surface and 5ЊC at the bottom. Salinity varies vertically from 33 to 35 psu west of the sill, and from 10 to 12 psu east of the sill. Shown here is the surface height and barotropic velocity ®elds (only every third velocity point is shown) after 1 day of adjustment from the initial conditions. Note the strong dipole sitting on top of the sill, and the predominantly northward barotropic currents next to the outer walls as remnants of barotropic Kelvin waves. A vector representing a current speed of 15 cm sϪ1 is indicated at the bottom of the ®gure.
c. Marsigli problem The purpose of this section is to directly compare the new free surface method to that of KSWP. For this pur- pose, we consider an unforced baroclinic adjustment problem motivated from the system ®rst discussed for the Bosporus by Marsigli in 1681 (see Gill 1982, p. 96). The experimental details are given in the caption to Fig. 5. A strong baroclinic pressure gradient arising from a salinity front rapidly piles a sea level difference of about 40 cm between the two basins. After about two model days, barotropic and average baroclinic pressure FIG. 4. Illustrating the Goldsbrough±Stommel circulation. (top) The gradients balance each other, and the sea level difference barotropic streamfunction (Sv). (bottom) Overturning streamfunction, with each contour representing 10 Sv. These results should be com- becomes roughly time independent. A weak cross-chan- pared to Figs. 6a and 7a in Huang (1993). nel geostrophic circulation persists that prevents a sa- linity exchange between both basins. Although the setup appears arti®cial, a similar balance can be observed in the tracer and baroclinic momentum time steps were the Baltic Sea during periods of weak wind forcing. unequal. The consequences on the equilibrium solution Because the sea level changes rapidly and there is a reached at 4000 yr have not been assessed. Doing so strong interbasin salinity gradient, this adjustment prob- requires rerunning the experiment with equal tracer and lem provides a severe test for how well the model con- baroclinic time steps. serves total salt. 1094 MONTHLY WEATHER REVIEW VOLUME 129
are four time series over a 3-month period that track how salinity deviates from its initial value.
The time series l1 results from running the free surface of KSWP, with a constant cell thickness in the tracer and baroclinic velocity equations. The tracer and bar- oclinic time steps are both 240 s and the barotropic time step is 60 s. During the ®rst two model days the sea level in the western basin is decreasing rapidly but the sea level in the eastern basin is increasing. Since the upper model boundary with the KSWP scheme is at z ϭ 0, saline water is entering the model area in the west- ern basin but brackish water leaves the model area through the level z ϭ 0 in the eastern basin. Conse- quently, the domain-averaged salinity in the model area is increasing. As long as the sea surface salinity is uni- form, the salt gain in the western basin and the salt loss in the eastern basin would be canceled exactly by a sea
FIG. 6. Time series of the deviation of domain-averaged salinity level variation in the opposite direction. However, such from its initial condition, where each model time step is plotted (note cancellation is not general, and so nonconservation of the logarithmic vertical axis). All model versions start with the same total salt is the norm. initial conditions. The time series l1 results from running a linearized Since the sea surface height is known, the salt be- free surface, with a constant cell thickness in the tracer and baroclinic velocity equations. The tracer and baroclinic time steps are both 240 tween the upper model boundary at z ϭ 0 and the sea level z can be included in the diagnostic for domain s. The time series l2 employs exactly the same linearized free surface ϭ scheme but the salt between the upper model boundary at z ϭ 0 and tracer content. As an example, the time series l2 employs the sea level z ϭ is included in the domain tracer integral. The exactly the same linearized free surface scheme as l1 time series n , which shows negligible deviation from zero, uses the 1 but uses this improved tracer diagnostic. Compared with nonlinear free surface discussed in this paper with tracer and baro- l1, the conservation of salt is better; however, there is clinic time steps both equal to 240 s. The time series n2 uses the same nonlinear free surface, yet with a tracer time step of 720 s, and a clear trend in the domain-averaged salinity. This gain baroclinic time step of 240 s. All cases used 60 s for the barotropic of salt stems from undulations of the sea level in cor- time step. respondence with a varying sea surface salinity.
The time series n1, which shows minor deviation from zero, uses the new free surface discussed in this paper Figure 5 shows the sea level and barotropic current with tracer and baroclinic time steps both equal to 240 after 1 day using the nonlinear free surface method. The s, with a 60-s barotropic time step. The conservation of current between the basins is de¯ected to the right by tracer is improved substantially when compared with the Coriolis force. There is a strong dipole sitting on the linearized free surface results. The deviation from top of the sill, with a high to the southwest and a low zero is less than 10Ϫ5 psu. to the northeast. The barotropic ¯ow at the outer edges The time series n 2 uses the same free surface, yet of the subbasins is directed northward in both basins, with unequal baroclinic and tracer time steps, with a which indicates that this current is the result of baro- tracer time step of 720 s, and baroclinic time step of tropic Kelvin waves encircling the subbasins. Such 240 s. As discussed in section 2, tracer conservation is waves, and the remaining currents, cannot be repre- not ensured in this case, as is indeed shown by the sented in a rigid-lid model. roughly 0.04 psu salinity drift. In general, the solution for the sea level, velocity, temperature, and salinity differ only slightly between d. Quantifying local versus global conservation the KSWP and the new approach during the adjustment. The close agreement of these features provides an im- The Marsigli example provided an illustration of the portant positive assessment of the new method. In par- global conservation properties of the new algorithm ticular, it indicates that the time averaging approach used when using a discretization that maintains local com- here does not overly damp the barotropic waves, at least patibility between the volume and tracer budgets. As as compared to the KSWP algorithm. mentioned in section 2e, there is an alternative approach As there are no surface water ¯uxes, the total volume that is available for maintaining more exact global con- of the ocean domain should remain constant, as indeed servation properties, yet at the cost of sacri®cing the it does to within computer roundoff (not shown). Like- local compatibility. We present here an example that wise, total salt and domain-averaged salinity (total salt compares the two approaches and concludes that the divided by total volume) should remain constant, where scheme that performs superior for global conservation total salt is computed via Eq. (10). An accounting of is preferable for climate modeling purposes. the domain-averaged salinity is shown in Fig. 6. Shown We consider a coarse-resolution global model run MAY 2001 GRIFFIES ET AL. 1095 with two idealized conditions and using two different erable approach for longer integrations where conser-
.t term appearing vation is keyץforms for the discretization of the T in the tracer concentration budget (6). Discretization (a) uses T()[( ϩ⌬) Ϫ *( Ϫ⌬)], as used in the Marsigli example just presented. This form provides for 5. Computational aspects local self-consistency between the tracer and volume The purpose of this section is to provide an overview budgets as discussed when deriving Eq. (38). However, of computational issues related to the use of a particular as discussed in section 2d, it will not provide for exact barotropic algorithm in a parallel computational envi- global conservation. Discretization (b) drops the time ronment. The papers by Dukowicz et al. (1993), Webb average on the lagged surface height and instead uses (1996), Webb et al. (1997), Marshall et al. (1997), and T()[( ϩ⌬) Ϫ ( Ϫ⌬)]. This expression leads Guyon et al. (2000) are complementary to the following. to exact global conservation in the case where there is no Robert ®ltering on the tracer, yet it is not locally self-consistent with a volume budget that uses a time- a. Fundamental considerations ®ltered surface height *( Ϫ⌬) for stability. The dominant feature of current supercomputing tech- The global model has roughly three degrees of hor- nology is the large and rapidly growing disparity be- izontal resolution, uses 15 vertical levels down to tween processor clock speeds and the memory band- 5000-m depth, and is run with a 30-min time step for width and latency. The rate at which processors are able the tracer and baroclinic momentum and 30-s barotropic to execute their instructions upon data far outstrips the time step. The ®rst experiment initialized the ocean at ability of memory to deliver data to the processor. All rest with constant 25ЊC water and then added a cli- current research into supercomputing architecture (see, matological wind stress yet no heat ¯uxes. As presented e.g., Culler and Singh 1998) is directed at resolving this in section 2c, the solution should maintain constant 25ЊC problem. water everywhere. For an inde®nite integration period, Traditional supercomputers, such as the Cray vector discretization a indeed remains precisely at 25ЊC. In machines, are largely dependent on prohibitively ex- contrast, discretization b shows water after 30 days with pensive custom memory to deliver data at a rate suf- a temperature of 24.999 95ЊC. Although not exact, this ®cient to keep vector pipelines busy. The microproces- is quite a small deviation even when extrapolated to sor-based computers currently being produced rely in- integrations of centuries to millennia. stead on commodity memory, but implement a deep The second experiment initialized the temperature memory hierarchy, where multiple levels of smaller and ®eld with a zonally averaged version of the Levitus faster data caches are placed between the main memory (1982) analysis. The surface ¯ux consisted of the same and the processor. For performance comparable to vec- climatological wind stress as the previous example as Ϫ2 tor systems, scalable cache-based microprocessor su- well as a uniform and constant 10 W m surface heat percomputers achieve speedup through massive parallel ¯ux. We assess here whether the heat input through the partitioning of the problem, where the problem is dis- ocean surface is equivalent to the heat absorbed by the tributed among many processors working concurrently ocean as deduced by measuring the ocean's total heat and exchanging data on a network. This architectural content via Eq. (10). We focus on a single model day model leads to measures of algorithm performance sub- since this is a typical time period over which winds and stantially different from the vector model. Notably, the surface tracer ¯ux are held ®xed in coupled climate following issues become central. models. After 1 day for discretization a, the heat input through 1) Temporal locality. Given the relatively high cost of the ocean surface minus the change in ocean heat con- memory access, algorithms that have a high rate of tent was equivalent to a spatially averaged error over data reuse are preferable. The computational inten- the globe of Ϫ0.5WmϪ2. Note that over the ®rst half sity, de®ned as the ratio of ¯oating-point operations of the day, the error was Ϫ1.6 W mϪ2, which diminished per memory request, is required to be as high as to Ϫ0.37 W mϪ2 after 2 days. These are nontrivial errors, possible. The match between the size of the data upward of 10%±20% of the imposed heating from the subset allotted to a processor, and the size of its atmosphere. In contrast, discretization b, along with cache, is a key element in this aspect of performance. Robert ®ltering on the tracer, reduced the error from 2) Spatial locality. In the interests of minimizing com- Ϫ0.5 W mϪ2 to a negligible Ϫ0.003 W mϪ2. Removing munication among processors, algorithms that main- the Robert ®lter brings the error to 10Ϫ9 WmϪ2, which tain spatial locality of data references are most use- arises from computer roundoff. ful. Both solution methods showed negligible qualitative 3) Communication pro®le. The communication perfor- differences for the structure of the surface height over mance of the interprocessor hardware interconnect the 30-day integration period. However, given the small is measured in terms of its latency (startup time per errors incurred by discretization b for both the ®rst and message) and bandwidth (the inverse of the trans- second set of experiments, we conclude that it is a pref- mission rate). Communication latency can be a cause 1096 MONTHLY WEATHER REVIEW VOLUME 129
TABLE 1. MOM 3 performance on a Cray T3E-900 for a coarse- TABLE 2. Same as Table 1 but for a more re®ned grid of 1Њ Mercator resolution model. The model has one-dimensional domain decom- resolution with 360 ϫ 112 ϫ 40 grid points, tracer time step of 4320 position along latitude rows, runs with the MOM 3 memory window, s, baroclinic time step of 4320 s, and explicit free surface time step and uses Cray-SHMEM communication. Results here are from a of 60 s. Southern Hemisphere con®guration using a 4ЊMercator grid extend- ing from the equator to 74ЊS with 90 ϫ 28 ϫ 40 grid points. The Nj/N Main fs Total Scale (%) M¯op/s model used a tracer time step of 43 200 s, baroclinic time step of 1 112 1793.41 142.29 1935.72 100 39.77 4320 s, and explicit free surface time step of 216 s. Model run times 2 56 1088.09 144.38 1232.48 98 77.96 are given in s. Shown are the number of T3E computer processors 4 28 545.62 78.79 624.43 97 153.87 (N), number of computed latitude rows per processor (j/N), time for 7 16 316.80 50.85 367.42 94 261.51 the main model loop (main), time for the free surface portion of the 8 14 277.57 45.07 322.66 94 297.78 model (fs), total model run time (total), scaling (scale), and the me- 14 8 162.72 31.53 194.27 89 494.53 ga¯ops per second (M¯op/s). Scaling is de®ned to be the total run 16 7 143.55 27.96 172.34 88 557.52 time taken for an experiment to complete on one processor, divided 28 4 86.04 22.99 109.05 79 881.10 by the time taken for N processors as scaled by the number of pro- cessors: i.e., scale ϭ T1/(NTN).
Nj/N Main fs Total Scale (%) M¯op/s vergence. For example, the conjugate gradient (CG) 1 28 27.10 1.10 28.22 100 32.86 solver computes the optimal vector along which to con- 2 14 15.18 0.40 15.58 90 59.51 verge. Computation of this vector involves a global sum 4 7 10.94 0.27 11.22 63 82.65 7 4 7.48 0.21 7.70 53 120.43 across the entire domain; hence, the data dependencies 14 2 1.45 0.19 1.65 122 562.00 are global, and involve global communication at each step in the solver iteration. It is observed in MOM that the number of iterations for the CG method is ϳ100 for for concern on most commercially available ma- a1Њ model with static boundary conditions, and goes chines today, barring a few exceptions such as the up with increasing resolution. Additionally, the iteration Cray T3E. The amount of data that needs to be trans- count is highly dependent on the model's representation ferred between processors is a function of the spatial of bottom topography and surface forcing, with more locality of the algorithm and the problem size. Al- realistic topography and time-dependent surface forcing gorithms that can accomplish the maximum data generally requiring much larger iteration counts. Parallel transfer in the fewest possible messages are prefer- implementation of the CG requires communication at able. each iteration. Other, faster methods, with better parallel performance, such as multigrid methods, are known to exist, but also require increased steps as the resolution b. Algorithms available in MOM 3 goes up. As mentioned in the introduction, there are three basic The explicit free surface has improved data locality, algorithms in MOM 3 for solving the barotropic mode: since the computations involve only local derivatives the rigid-lid streamfunction of Bryan (1969), with mod- and no elliptic problem. The ratio of the baroclinic to i®cations to the rigid-lid surface pressure method of the barotropic time step is given by the ratio of the phase Smith et al. (1992) and Dukowicz et al. (1993); the speeds of the barotropic gravity wave to the ®rst bar- implicit free surface method of Dukowicz and Smith oclinic wave. This ratio is determined by ocean depth (1994); and the explicit free surface method of KSWP and strati®cation and, so, is independent of grid reso- as well as that discussed here. lution. Hence, the method's computational performance The rigid-lid model, although parallelized (Redler et is generally independent of grid resolution. al. 1998), has not been the focus of recent algorithm research at GFDL, largely because of the physical issues c. Examples of performance on scalable systems previously raised in this paper as well as problems with global data dependencies related to island integrals [see MOM 3 is parallelized in one dimension, along lat- Bryan (1969) as well as Smith et al. 1992 for discus- itude rows only. Scalability is thus measured in terms sion]. of latitude rows per processor. We show scaling results The implicit free surface method, in contrast, has a from a Cray T3E at two different resolutions using the relatively long history of use on parallel machines. This explicit free surface algorithm documented here. Table algorithm solves the barotropic system implicitly in 1 shows scaling results for a Southern Hemisphere con- time, hence allowing for the barotropic mode time step ®guration using a 4Њ Mercator grid. The superlinear scal- to be lengthened to that of the baroclinic mode. As such, ing above seven rows per processing element (PE) is a there is no subcycling of the barotropic mode as required result of arrays becoming small enough to ®t in cache. with the explicit approach. In so doing, however, the Table 2 shows results for the same domain at 1Њ reso- algorithm requires the solution of an elliptic problem. lution. The elliptic solvers used in the implicit free surface The results for the 1Њ model are more appropriate for method are variants of the basic iterative Jacobi solver, current and future needs of parallel ocean modeling. where methods are used to accelerate the rate of con- They indicate that one can increase the number of pro- MAY 2001 GRIFFIES ET AL. 1097 cessors in MOM 3 until 8 rows/PE without signi®cant Blumberg, A. F., and G. L. Mellor, 1987: A description of a three- loss of scalability. Note that with 4 rows/PE, scalability dimensional coastal ocean circulation model. Three-Dimensional Coastal Ocean Models, N. Heaps, Ed., Coastal and Estuarine is 80%, whereas it has been found to be about 60% for Sciences, Vol. 4, Amer. Geophys. Union, 1±16. MOM's version of the implicit free surface method. Fur- Bryan, F. O., 1987: Parameter sensitivity of primitive equation ocean thermore, as noted previously, absolute times for the general circulation models. J. Phys. Oceanogr., 17, 970±985. implicit free surface method are highly dependent on Bryan, K., 1969: A numerical method for the study of the circulation the conjugate gradient iteration count needed to achieve of the world ocean. J. Comput. Phys., 4, 347±376. , 1984: Accelerating the convergence to equilibrium of ocean- convergence to the subjectively chosen tolerance level. climate models. J. Phys. Oceanogr., 14, 666±673. , and M. D. Cox, 1972: An approximate equation of state for numerical models of the ocean circulation. J. Phys. Oceanogr., 6. Conclusions 2, 510±514. Some key conclusions of this paper were noted in the Cox, M. D., 1984: A primitive equation, 3-dimensional model of the ocean. GFDL Ocean Group Tech. Rep. 1, 143 pp. introduction, with the central ones highlighted here. , and K. Bryan, 1984: A numerical model of the ventilated ther- By adapting the partial cell framework of Pacanowski mocline. J. Phys. Oceanogr., 14, 674±687. and Gnanadesikan (1998) to the undulating top model Culler, D. E., and J. P. Singh, 1998: Parallel Computer Architecture: grid cell, the explicit free surface scheme described here A Hardware/Software Approach. Morgan Kaufmann, 1100 pp. fully incorporates the effects of the surface height into Danabasoglu, G., J. C. McWilliams, and W. G. Large, 1996: Approach to equilibrium in accelerated global oceanic models. J. Climate, the tracer and baroclinic momentum equations. Doing 9, 1092±1110. so allows for the surface boundary conditions to be Dewar, W. K., and R. X. Huang, 1996: On the forced ¯ow of salty formulated in a physically consistent and meaningful water in a loop. Phys. Fluids, 8, 954±970. manner, to treat tracer and freshwater ¯uxes consistently , Y. Hsueh, T. J. McDougall, and D. Yuan, 1998: Calculation of and quasi-conservatively, and to maintain energetic con- pressure in ocean simulations. J. Phys. Oceanogr., 28, 577±588. Dukowicz, J. K., and R. D. Smith, 1994: Implicit free-surface method sistency in which buoyancy effects in the density equa- for the Bryan±Cox±Semtner ocean model. J. Geophys. Res., 99, tion balance the work done by pressure from the mo- 7991±8014. mentum equation. , , and R. C. Malone, 1993: A reformulation and imple- The explicit free surface method is stable and allows mentation of the Bryan±Cox±Semtner ocean model on the con- long tracer time steps. This result represents a key prac- nection machine. J. Atmos. Oceanic Technol., 10, 195±208. Gill, A. E., 1982: Atmosphere±Ocean Dynamics. Academic Press, tical feature of the algorithm for use in coarse-resolution 662 pp. climate studies. Relatedly, when run on parallel com- Gordon, C., C. Cooper, C. A. Senior, H. Banks, J. M. Gregory, T. C. puters, the absence of an elliptic problem, present in the Johns, J. F. B. Mitchell, and R. A. Wood, 2000: The simulation rigid-lid and implicit free surface methods, enhances the of SST, sea ice extents and ocean heat transports in a version of the Hadley Centre coupled model without ¯ux adjustments. Cli- processor scaling of the explicit free surface method. mate Dyn., 16, 147±168. Grif®es, S. M., and R. W. Hallberg, 2000: Biharmonic friction with Acknowledgments. We thank members of the Ocean, a Smagorinsky viscosity for use in large-scale eddy-permitting Climate, and Prediction Groups at GFDL for testing this ocean models. Mon. Wea. Rev., 128, 2935±2946. scheme, and earlier versions, in various model con®g- Guyon, M., G. Madec, F. X. Roux, M. Imbard, C. Herbaut, and P. urations. In particular, we thank Jeff Anderson, Bob Fronier, 1999: Parallelization of the OPA ocean model. Calcu- lateurs Paralleles, 11, 499±517. Hallberg, Matt Harrison, George Mellor, Thomas Neu- Hallberg, R., 1995: Some aspects of the circulation in ocean basins mann, Young-Gyu Park, Igor Polyakov, Tony Rosati, with isopycnals intersecting the sloping boundaries. Ph.D. thesis, Torsten Seifert, Mike Spelman, Eli Tziperman, David University of Washington, Seattle, WA, 244 pp. Webb, Mike Winton, and Bruce Wyman for enjoyable , 1997: Stable split time stepping schemes for large-scale ocean modeling. J. Comput. Phys., 135, 54±65. conversations and useful suggestions. Bob Hallberg and Haltiner, G. J., and R. T. Williams, 1980: Numerical Prediction and Igor Polyakov deserve special thanks for extensive sug- Dynamic Meteorology. John Wiley, 477 pp. gestions and useful critiques. Comments from the anon- Higdon, R. L., and A. F. Bennett, 1996: Stability analysis of operator ymous reviewers are also greatly appreciated. We thank splitting for large-scale ocean modeling. J. Comput. Phys., 123, Jerry Mahlman, the director of GFDL, for his support 311±329. , and R. A. de Szoeke, 1997: Barotropic±baroclinic time splitting and encouragement. for ocean circulation modeling. J. Comput. Phys., 135, 30±53. Holland, W. R., J. C. Chow, and F. O. Bryan, 1998: Application of a third-order upwind scheme in the NCAR ocean model. J. Cli- REFERENCES mate, 11, 1487±1493. Barnier, B., 1998: Forcing the oceans. Ocean Modeling and Param- Huang, R. X., 1993: Real freshwater ¯ux as a natural boundary con- eterization, E. P. Chassignet and J. Verron, Eds., NATO Ad- dition for the salinity balance and thermohaline circulation vanced Study Institute, Kluwer Academic, 45±80. forced by evaporation and precipitation. J. Phys. Oceanogr., 23, Beron-Vera, F. J., J. Ochoa, and P. Ripa, 1999: A note on boundary 2428±2446. conditions for salt and freshwater balances. Ocean Modelling, Jackett, D. R., and T. J. McDougall, 1995: Minimal adjustment of 1, 111±118. hydrographic pro®les to achieve static stablilty. J. Atmos. Oce- Bleck, R., and L. T. Smith, 1990: A wind-driven isopycnic coordinate anic Technol., 12, 381±389. model of the north and equatorial Atlantic Ocean. 1. Model Killworth, P. D., 1987: Topographic instabilities in level model development and supporting experiments. J. Geophys. Res., 95 OGCM's. Ocean Modelling (unpublished manuscript), 75, 9±12. (C3), 3273±3285. , and N. R. Edwards, 1999: A turbulent bottom boundary layer 1098 MONTHLY WEATHER REVIEW VOLUME 129
code for use in numerical models. J. Phys. Oceanogr., 29, 1221± model user guide. GFDL Ocean Group Tech. Rep. 2, Geophys- 1238. ical Fluid Dynamics Laboratory, Princeton, NJ, 16 pp. , J. M. Smith, and A. E. Gill, 1984: Speeding up ocean circulation Redler, R., K. Ketelsen, J. Dengg, and C. W. BoÈning, 1998: A high- models. Ocean Modelling (unpublished manuscript), 56, 1±5. resolution numerical model for the circulation of the Atlantic , D. Stainforth, D. J. Webb, and S. M. Paterson, 1991: The Ocean. Proceedings of the Fourth European CRAY-SGI MPP development of a free-surface Bryan±Cox±Semtner ocean mod- Workshop, H. Lederer and F. Hertweck, Eds., Max-Planck-In- el. J. Phys. Oceanogr., 21, 1333±1348. stitut fuÈr Plasmaphysik, 95±108. Large, W. G., and S. Pond, 1981: Open ocean ¯ux measurements in Roullet, G., and G. Magec, 2000: Salt conservation, free surface and moderate to strong winds. J. Phys. Oceanogr., 11, 324±336. varying volume. A new formulation for Ocean GCMs. J. Geo- Leonard, B. P., 1979: A stable and accurate convective modelling phys. Res., 105, 23 927±23 947. procedure based on quadratic upstream interpolation. Comput. Semtner, A. J., Jr., 1974: An oceanic general circulation model with Methods Appl. Mech. Eng., 19, 59±98. bottom topography. Numerical Simulation of Weather and Cli- Levitus, S., 1982: Climatological Atlas of the World Ocean. NOAA mate, Tech. Rep. 9, Department of Meteorology, University of Prof. Paper 13, U.S. Government Printing Of®ce, Washington, California, Los Angeles. DC, 173 pp. Smith, R. D., J. K. Dukowicz, and R. C. Malone, 1992: Parallel ocean Marshall, J., A. Adcroft, C. Hill, L. Perelman, and C. Heisey, 1997: general circulation modeling. Physica D, 60, 38±61. A ®nite-volume, incompressible Navier±Stokes model for stud- Webb, D. J., 1995: The vertical advection of momentum in Bryan± ies of the ocean on parallel computers. J. Geophys. Res., 102, Cox±Semtner ocean general circulation models. J. Phys. Ocean- 5753±5766. ogr., 25, 3186±3195. Mellor, G. L., 1996: User's guide for a three-dimensional, primitive , 1996: An ocean model code for array processor computers. equation, numerical ocean model. Program in Atmospheric and Comput. Geophys., 22, 569±578. Oceanic Studies, Princeton University, Princeton, NJ, 40 pp. , A. C. Coward, B. A. de Cuevas, and C. S. Gwilliam, 1997: A [Available from Program in Atmospheric and Oceanic Sciences, multiprocessor ocean general circulation model using message Princeton University, Princeton, NJ 08542.] passing. J. Atmos. Oceanic Technol., 14, 175±183. Pacanowski, R. C., and A. Gnanadesikan, 1998: Transient response , B. A. de Cuevas, and A. C. Coward, 1998: The ®rst main run in a z-level ocean model that resolves topography with partial of the OCCAM global ocean model. Southampton Oceanography cells. Mon. Wea. Rev., 126, 3248±3270. Centre Internal Doc. 34, 44 pp. , and S. M. Grif®es, 2000: The MOM 3.1 manual. NOAA/Geo- Wolff, J.-O., E. Maier-Reimer, and S. Legutke, 1997: The Hamburg physical Fluid Dynamics Laboratory, Princeton, NJ, 680 pp. Ocean Primitive Equation Model HOPE. DKRZ Tech. Rep. 13, , K. Dixon, and A. Rosati, 1991: The GFDL modular ocean 98 pp.