Downloaded 09/25/21 04:59 AM UTC 2252 MONTHLY WEATHER REVIEW VOLUME 148

JUNE 2020 S T I N I S E T A L . 2251

Improving Solution Accuracy and Convergence for Stochastic Physics Parameterizations with Colored Noise

PANOS STINIS Advanced Computing, Mathematics and Data Division, Paciﬁc Northwest National Laboratory, Richland, Washington

HUAN LEI Department of Computational Mathematics, Science and Engineering, and Department of Statistics and Probability, Michigan State University, East Lansing, Michigan

JING LI Advanced Computing, Mathematics and Data Division, Paciﬁc Northwest National Laboratory, Richland, Washington

HUI WAN Atmospheric Sciences and Global Change Division, Paciﬁc Northwest National Laboratory, Richland, Washington

(Manuscript received 5 June 2019, in ﬁnal form 20 January 2020)

ABSTRACT

Stochastic parameterizations are used in numerical weather prediction and climate modeling to help capture the uncertainty in the simulations and improve their statistical properties. Convergence issues can arise when time integration methods originally developed for deterministic differential equations are applied naively to stochastic problems. In previous studies, it has been demonstrated that a correction term, known in stochastic analysis as the Ito^ correction, can help improve solution accuracy for various deterministic numerical schemes and ensure convergence to the physically relevant solution without substantial computational overhead. The usual formulation of the Ito^ correction is valid only when the stochasticity is represented by white noise. In this study, a generalized formulation of the Ito^ correction is derived for noises of any color. The formulation is applied to a test problem described by an advection–diffusion equation forced with a spectrum of fast processes. We present numerical results for cases with both constant and spatially varying advection velocities to show that, for the same time step sizes, the introduction of the generalized Ito^ correction helps to substantially reduce time integration error and signiﬁcantly improve the convergence rate of the numerical solutions when the forcing term in the governing equation is rough (fast varying); alternatively, for the same target accuracy, the generalized Ito^ correction allows for the use of signiﬁcantly longer time steps and, hence, helps to reduce the computational cost of the numerical simulation.

1. Introduction impact on the large-scale ﬂow motions need to be accounted for using parameterizations (see, e.g., McFarlane 2011). Physical and chemical processes happening in Earth’s In recent years, stochastic parameterizations have atmosphere span many orders of magnitude in terms become an active area of research [see review by Berner of their spatial and temporal scales, which presents great et al. (2017) and Leutbecher et al. (2017)]. The funda- challenges to numerical modeling. For example, in mental principle behind the stochastic formulation is general circulation models, motions or phenomena that that the state of the unresolved processes at any instant cannot be resolved in space or time but have signiﬁcant is not entirely determined by the state of the resolved processes. Thus, an element of randomness needs to be Denotes content that is immediately available upon publica- introduced to account for this indeterminacy. This ran- tion as open access. domness can act as a source of roughness in the temporal evolution of the governing equations’ right-hand-side Corresponding author: Panos Stinis, [email protected] terms as well as in the evolution of the solution.

DOI: 10.1175/MWR-D-19-0178.1 Ó 2020 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses). Unauthenticated | Downloaded 09/25/21 04:59 AM UTC 2252 MONTHLY WEATHER REVIEW VOLUME 148

Deterministic time integration schemes used in numer- in Kloeden and Platen 1992). The two most-studied dis- ical weather prediction and climate projection models, cretization methods in stochastic analysis are: (i) using however, typically assume temporal smoothness of the the left endpoint of each subinterval (which leads to underlying solutions. When such schemes are applied the Ito^ integral or Ito^ interpretation) and (ii) using naively to stochastic parameterizations, the conditions the middle point of each subinterval (which leads to the for solution convergence might no longer be satisfied. Stratonovich integral or Stratonovich interpretation). There is a large body of work dedicated to the de- The Stratonovich interpretation leads to ordinary cal- velopment of stochastic numerical schemes and sto- culus while the Ito^ interpretation does not (see e.g., chastic versions of deterministic numerical schemes, sections 3.9 and 4 in Øksendal 2003). Because the fea- some especially targeted for weather and climate ap- sibility of weather forecasting and climate projection plications (see e.g., Rüemelin 1982; Sura and Penland depends fundamentally on the predictable (determinis- 2002; Sardeshmukh et al. 2003; Ewald et al. 2004; Ewald tic) components of the weather and climate systems, one and Temam 2005; Hansen and Penland 2006). The can argue that the Stratonovich calculus is more relevant purpose of our work is more modest. We want to study for the primary targets of these applications. However, the use of a correction term that can help improve the there are also certain aspects of weather and climate solution accuracy of deterministic schemes when part of dynamics (e.g., the impact of individual cloud/aerosol the variables of the model is replaced by noise and in particles on the evolution of cloud systems and the im- particular colored noise. pact of tropical convection on the global circulation), As shown in Hodyss et al. (2013), the convergence which contain characteristics that can be usefully de- issue of deterministic numerical schemes when applied scribed by Ito^ calculus. The present study is targeted at to stochastic parameterizations can be investigated the first type of applications, and hence aim at en- through the use of tools from stochastic analysis [see suring the discrete numerical solutions of the model e.g., section 3.3 in Øksendal (2003) and section 4.9 in equations are consistent with the Stratonovich in- Kloeden and Platen (1992)]. In particular, for the cases terpretation. For a more general discussion about when an unresolved process is replaced by a rough the Ito–Stratonovich^ dilemma, see for example, random process (e.g., white noise), it is not difficult to chapter 7 in Kloeden and Platen (1992), the review construct examples for which popular deterministic paper Mannella and McClintock (2012), and Moon numerical schemes (e.g., Euler forward and backward, and Wettlaufer (2014). Adams–Bashforth) will no longer converge to the It is important to note that many popular time inte- physically relevant solution except for special cases [e.g., gration schemes designed for deterministic problems the second-order Runge–Kutta scheme analyzed by will converge to the Ito^ solution when applied to sto- Hodyss et al. (2013)]. Multiple examples relevant for chastic problems driven by white noise (Kloeden and atmospheric modeling can be found in Fig. 3 in the paper Platen 1992). In other words, naively describing an un- of Hodyss et al. (2013). Here, ‘‘physically relevant so- resolved process by white noise and solving the stochastic lution’’ refers to the one corresponding to ordinary equation with a deterministic numerical scheme can lead calculus (see discussion below). In the study by Hodyss to erroneous results even in the limit of infinite temporal et al. (2014), ensemble simulations of Hurricane Isaac resolution. Fortunately, the Ito^ and Stratonovich inter- in the year 2012 were conducted using the Navy pretations are related, and this relationship can help Operational Global Atmospheric Prediction System; it recover, at least to some extent, the convergence of was shown that the choice of numerical scheme for the deterministic numerical schemes to the physically rele- stochastic term can lead to failure in predicting the vant Stratonovich solution. The connection between the correct ensemble mean of hurricane intensity. two interpretations comes in the form of a correction The mathematical reason for the lack of convergence term called the Ito^ correction. When the Ito^ correction is that when we replace an unresolved process with white is added to the equation, the numerical solution under noise, the equations describing the phenomena under the Ito^ interpretation converges to the Stratonovich investigation make sense only in integral form (not in solution. the usual differential form). The integral form of the While the abovementioned Ito–Stratonovich^ corre- equations contains a temporal integral of an expression spondence is a basic concept in stochastic analysis, the involving the white noise process. If we try to estimate widely known form of the Ito^ correction applies only to such an integral through a limiting process involving the case of white noise. A key feature of white noise is progressively refined subintervals, different answers will that it has zero autocorrelation (and hence no memory). be obtained depending on the manner we choose to Given the typical time step size of seconds to an hour discretize the interval of integration (see e.g., section 4.9 in weather and climate models, some parameterized

Unauthenticated | Downloaded 09/25/21 04:59 AM UTC JUNE 2020 S T I N I S E T A L . 2253 processes (e.g., turbulence and cumulus convection) can from the simple but relevant test problem provide a have characteristic time scales equivalent to multiple proof of concept, which motivates further exploration time steps. Therefore, colored noise, which has nonzero of the generalized Ito^ correction for the purpose of autocorrelation length, can provide a better description helping improve the solution accuracy and efficiency of such processes. In fact, the state of the art in ac- in atmospheric models of various complexity, in- counting for model uncertainties of Earth systems points cluding general circulation models using stochastic to the need of stochastic processes with spatiotemporal parameterizations. correlations (see e.g., sections 3 and 5 in Leutbecher The remainder of the paper is organized as follows: et al. 2017), which makes our construction more relevant section 2 presents the derivation of the generalized Ito^ for applications. correction. Section 3 contains a presentation of the test A fundamental difference between colored noise and problem, the advection–diffusion equation with con- white noise is that colored noise is in principle resolvable stant and spatially varying advection velocity, along with while white noise is not. In other words, if one could use analytical results (supplemental details can be found in small enough step sizes, there would be no distinction the appendix). Section 4 contains numerical results. between the Ito^ and Stratonovich interpretations for Finally, section 5 contains a discussion of our results as the case of colored noise. All deterministic numerical well as suggestions for future work. schemes will eventually converge to the Stratonovich solution. But even in simple examples, let alone the very 2. The generalized Ito^ correction complex and expensive systems encountered in weather and climate prediction, the critical time step that re- We consider the following deterministic differential covers convergence to the Stratonovich solution can be equation: prohibitively small. As a result, for realistically afford- ›u ^ 5 D(u) 1 P(u), (1) able time step sizes, the dichotomy between Ito and ›t Stratonovich interpretations practically exists and needs to be addressed also for the cases of colored noise. In where D(u) and P(u) are the resolved dynamics and other words, the long time steps in practical applications parameterized physics, respectively. Here we focus on motivate us to find time-stepping methods with higher the special case where P(u) takes the following form: accuracy. 5 Another point worth mentioning is that, as Hodyss P(u) g(u)H(t). (2) et al. (2013) and Hodyss et al. (2014) have pointed This form results from the attempt to eliminate a fast- out, certain deterministic numerical schemes (e.g., the evolving physical quantity from the original equations second-order Runge–Kutta scheme) have (an unaveraged and replace it by a time-dependent process. We can version of) the Ito^ correction already ‘‘built-in’’ and consider a more general form where H depends also on hence perform better for stochastic problems. Such the spatial variable, but that generalization will not alter schemes are typically multistage schemes that require the derivation of the generalized Ito^ correction below, multiple evaluations of the right-hand side of the govern- hence we restrict our attention to the case where H ing equations, making them very expensive for weather depends only on t. and climate models. The Ito^ correction, in contrast, al- If the time scales associated with H(t) are substantially lows for the use of single-stage schemes (e.g., the Euler shorter than the time scales of D(u), we can approximate forward scheme) to be complemented by a correction P(u) by its stochastic counterpart P (u) defined as constructed only for the stochastic term, and hence can s be cost effective. 5 _ Ps(u) g(u)R(t), (3) For these reasons, we present in this paper a generalization of the Ito^ correction that is valid for noises where R_(t) represents a general noise term. We note of any color. We use an advection-diffusion equation here that replacing H(t) by the noise term, R_(t), may with constant or spatially varying advection velocity to include a limiting process where the function g(u) may demonstrate that, for both white and colored noises, the be also modified [see e.g., section 6.4 in Gardiner (1985) generalized Ito^ correction can accelerate convergence to and Papanicolaou and Kohler (1974)].This does not al- the Stratonovich solution when added to the Euler for- ter the main line of our derivation and we keep the ward scheme. We demonstrate that improved conver- notation g(u) for the multiplicative factor. gence means higher accuracy for the same step size or, Using Eq. (3), we get the following stochastic coun- alternatively, larger step size (and, hence, lower compu- terpart of the deterministic equation originally given tational cost) for the same target accuracy. These results by Eq. (1):

Unauthenticated | Downloaded 09/25/21 04:59 AM UTC 2254 MONTHLY WEATHER REVIEW VOLUME 148

›u that we derive below is not tied to the forward Euler 5 D(u) 1 P (u). (4) ›t s scheme. We will come back to this point in section 2c. It is well known in stochastic analysis that in the white Without loss of generality, we assume E[R_(t)] [ 0, where noise case, the Stratonovich integral can be written as E[] denotes the mean over different realizations of the the sum of an Ito^ integral and a correction term called _ _ 0 0 noise process. If E[R(t)R(t )] 5 d(t 2 t ) where d()is ^ Ø _ the Ito correction (see, e.g., ksendal 2003). Below we Dirac’s delta function, then R(t) is white noise and R(t) show that the same is true for colored noise, although E _ _ 0 d 2 0 _ is a Wiener process; when [R(t)R(t )] 6¼ (t t ), R(t) the Ito^ correction needs to be generalized. is a colored noise. For t*j 5 (tj 1 tj11)/2, performing a Taylor expansion of We focus on how to numerically solve Eq. (4) after its g[u(t*j )] about tj and expressing ›u/›t using Eq. (4) gives form has been derived; how to construct a good Ps(u)to approximate the original P(u) is a separate topic that is Dt dg(u) ›u Dt2 d2g[u(t)] * 5 1 1 g[u(tj )] g[u(tj)] 2 outside the scope of the current work. 2 du ›t 8 dt j tj a. Derivation (8) Let us take the integral over an arbitrary time window 5 1 1 dg(u) D (t1, t2) on both sides of Eq. (4). For Ps(u), we discretize g[u(tj)] D[u] t 2 du t D j the time interval into J bins of equal length t and de- D D 2 2 note the increment of R in each bin as R. We use t*j to 1 1 dg(u) _ D 1 t d g[u(t)] g[u]R(t) t 2 , denote the discretization point inside the jth bin [i.e., 2 du 8 dt j tj the instant where the value of P (u) is evaluated for s (9) numerical integration]. With this notation, the integral of Eq. (4) can be written as where j 2 [tj,(tj 1 tj11)/2]. For small Dt, we write ð t 2 R_(t )Dt ’DR . (10) u(t ) 2 u(t ) 5 D[u(t)] dt 1 limåg[u(t*)]DR . (5) j j 2 1 Dt/0 j j t1 j Hence, Eq. (9) can be approximated as In the white noise case [i.e., R(t) 5 B(t)whereB(t) is the Wiener process], the choice of discretization 1 dg(u) g[u(t*)] ’ g[u(t )] 1 D(u) Dt point for the integral can lead to different results j j 2 du tj (Øksendal 2003). The two most popular choices are 1 dg(u) Dt2 d2g[u(t)] defined as 1 D 1 g[u] Rj 2 . 2 du t 8 dt j ð j Ito^ integral: g(u) dB 5 limåg[u(t*)]DB where (11) D / j j t 0 j 5 Assuming g(u) is sufficiently smooth and Dt is small, t*j tj(left endpoint), (6) one can neglect the second and fourth terms on the and right-hand side of Eq. (11) but, in general, not the third ð term. Therefore, with the Stratonovich interpretation of the stochastic integral in Eq. (5), we have Stratonovich integral: g(u)+dB 5 limåg[u(t*)]DB D / j j t 0 j ð ð t2 t2 t 1 t D 2 5 1 j j11 t u(t2) u(t1) D(u) dt g(u) dR where t*5 5 t 1 (midpoint). (7) t t j 2 j 2 |fflfflfflfflfflfflffl{zfflfflfflfflfflfflffl}1 |fflfflfflfflfflfflffl{zfflfflfflfflfflfflffl}1 traditional integral Ito^ integral D 5 2 Here Bj Bj11 Bj. Because the physical processes 1 dg(u) 2 represented by the deterministic Eq. (1) are assumed 1 limå g[u] (DR ) . (12) Dt/ j 0 j 2 du t continuous, the Stratonovich integral should be used in j our case (see section 3.3 in Øksendal 2003). The mathematical expectation of the last term in Before proceeding further with the derivation, we note Eq. (12) is that the Ito^ interpretation for the stochastic integral as shown in Eq. (6) coincides with how the Euler forward 1 dg(u) 2 approach would treat the stochastic term, but the corre- limå g[u] E[(DR ) ], (13) Dt/ j 0 j 2 du t spondence between Ito^ and Stratonovich interpretations j

Unauthenticated | Downloaded 09/25/21 04:59 AM UTC JUNE 2020 S T I N I S E T A L . 2255 which is the generalized Ito^ correction in its integral the stochastic integral makes the evolution of the sto- form. The exact form of the expectation in expression chastic process R_(t) independent of the solution u(t). The (13) depends on the formulation of R. For example, Ito^ correction serves as a way to account for the inter- _ when R is the Wiener process, the increment DRj is a action of R(t) and u(t) during the interval Dt, similar to Gaussian random variable with mean 0 and variance Dt: the role played by memory terms in model reduction, which account for the interaction between resolved and E D 2 5D [( Rj) ] t, (14) unresolved variables. c. Applicability hence (13) becomes ð It has been stated earlier in section 2a that our gen- t 2 1 dg(u) eralized Ito^ correction (13) is not tied to the specific g(u) dt, (15) 2 du discretization method (e.g., Euler forward) that is cho- t1 sen for the time integral of the stochastic term in Eq. (4). which is the integral form of the traditional Ito^ correc- The reason is that for any discretization, as long as an tion (see, e.g., section 3.3 in Øksendal 2003). analysis similar to Eqs. (9)–(12) reveals that the dis- The generalized Ito^ correction (13) can be extended cretized integral converges to the Ito^ integral, expres- to the case of multiple partial differential equations sion (13) can be used to obtain numerical results that (PDEs) each containing multiple noise processes. Let us converge to the Stratonovich solution. assume a system of n PDEs for the functions u 5 (u1, As an example, the method of analysis demon- u2, ..., un): strated by Eqs. (9)–(12) canalsobeappliedtothe ‘‘decentered’’ Euler method commonly used in at- ›u p i 5 D (u) 1 å g (u)R_ (t), for i 5 1, ... , n, (16) mospheric models. Since this method approximates › i il l t l51 time integrals (or derivatives) using the discretization point t*5 (1 2 l)t 1 lt 1 5 t 1 lDt where 0 # l # 1, _ _ 5 _ j j j 1 j where Rl(t)isthelth component of R(t) [R1(t), Eq. (5) becomes R_ (t), ... , R_ (t)], a p-dimensional vector noise pro- 2 p ð ð cess with independent components. Then, the expres- t2 t2 2 5 1 sion for the generalized Ito^ correction for the equation u(t2) u(t1) D(u) dt g(u) dR t1 t1 for ui is given by |fflfflfflfflfflfflffl{zfflfflfflfflfflfflffl} |fflfflfflfflfflfflffl{zfflfflfflfflfflfflffl} traditional integral Ito^ integral p n › 1 g (u) 2 dg(u) 2 å limå å il g [u] E[(DR ) ]. (17) 1 limå l g[u] (DR ) . (18) D / › kl lj Dt/ j l51 t 0 j 2 k51 u 0 j du t k tj j where DRlj 5 Rl,j11 2 Rlj is the increment of the lth noise A comparison of Eq. (18) with Eq. (12) suggests that the process Rl. correction term linking the ‘‘decentered’’ integral and the Stratonovich integral is b. Remarks 1 dg(u) We want to make two remarks concerning the deri- limå 2 l g[u] E[(DR )2], (19) D / j vation of the generalized Ito^ correction (13). First, there t 0 j 2 du tj is an alternative way to derive the generalized Ito^ correction. In particular, under the assumption that the which is a further generalization of expression (13) for correlation time of the noise is short, one can employ nonzero l. In the special case of l 5 0, the ‘‘decentered’’ the expansion devised by Stratonovich [see section 4.8 scheme gives the Ito^ interpretation of the stochastic in- in Stratonovich (1963)], through which a stochastic tegral and we recover (13); the case of l 5 1/2 corre- equation driven by colored noise can be rewritten as an sponds to the Stratonovich interpretation of the integral effective stochastic equation driven by white noise. and the correction vanishes. For the case when R is the Then, one can compute the traditional Ito^ correction for Wiener process, expression (19) corresponds to the the resulting white noise driven equation. correction formula for white noise that is found in Second, the Ito^ correction, in its traditional or gen- Hodyss et al. (2013) [see Eqs. (2.7) and (3.21) therein] as eralized form, can be interpreted as a memory term well as in section 3.5 of Kloeden and Platen (1992). encountered in model reduction formalisms [see e.g., The example of decentered schemes discussed above Chorin and Stinis (2007)]. By using as discretization point can be generalized even further: for a generic dis- the left endpoint of each interval, the Ito^ interpretation of cretization method, an analysis similar to Eqs. (9)–(12),

Unauthenticated | Downloaded 09/25/21 04:59 AM UTC 2256 MONTHLY WEATHER REVIEW VOLUME 148 followed by a comparison with the desired interpreta- where Nf, C, v0, and vm are parameters of the noise tion of the equation [i.e., Eq. (12) for Stratonovich or process n(t) (cf. appendix A). The expression for C Eq. (12) without the last right-hand-side term for Ito],^ contains a parameter a that controls the color of the can lead to the correction term needed to obtain nu- Fourier spectrum of n(t), with a 5 0 corresponding merical results converging to the desired type of solution to white noise, and larger a values corresponding to (Stratonovich or Ito).^ noise spectra that are more red. Before we proceed, we want to make an important remark d. Integral versus differential form about the formula for I.Forthecaseofwhitenoisen ^ a 5 v 2 1 The expressions for the generalized Ito correction ( 0), weo obtain limNf /‘(1/Nf ) [C( 0) /2] Nf v 2 5 ^ presented so far have been obtained from the integral åm51C( m) 1, and we recover the usual Ito cor- form of the stochastic equation, while in the literature on rection expression. However, for the case of colored stochastic analysis, the Ito^ correction conventionally noise (a 6¼ 0) with exponentiallyn decaying spectrum,o we 2 Nf 2 /‘ v 1 å v 5 : denoted by the symbol I is typically the term that is have limNf (1/Nf ) [C( 0) /2] m51C( m) 0 added to the differential form of the stochastic equation. This result is not surprising. As we have explained also The differential form of our generalized Ito^ correction in the introduction, in the case of colored noise, the in the test problem discussed below is given in section 3 distinction between the Ito^ and Stratonovich interpre- and in appendix A. An example showing how the dif- tations disappears in the limit of infinite temporal ferential form is obtained from the integral form can be resolution. The reason is that in the limit of infinite found in appendix A [Eqs. (A9)–(A13)]. temporal resolution a nonwhite colored noise is resolved and thus all discretizations of the integral of the 3. Test problem stochastic term give the same answer. However, any numerical experiment that one conducts always has In the remainder of the paper, we use an example to finite temporal resolution. In this case, the generalized demonstrate the impact of the generalized Ito^ correc- Ito^ correction for the case of colored noise is no longer tion. We consider the following stochastic differential zero. Moreover, as we show with our numerical results, equation: it can play a significant role in restoring or accelerating h i ›u « ›u ›2u convergence to the Stratonovich solution. This remark 52 c 1 cos(x) 1 m 1 g(u)n(t), (20) ›t 2 ›x ›x2 is particularly pertinent for weather and climate applications where due to computational limitations we 5 with initial condition u(x,0) u0(x) and periodic are always forced to use larger timesteps than the p boundary conditions on [0, 2 ]. In the context of at- shortest time scales present in the solution. mosphere modeling, the first two terms on the right- To derive analytical solutions for the test problem, hand side represent the resolved dynamics and the last we express u(x, t) in the form of a superposition of term represents fast varying physics parameterizations. Fourier modes: When the parameter « is set to 0, we recover the advection-diffusion equation with constant advection 5 å u(x, t) Fk(t) exp(ikx), (24) velocity discussed in Hodyss et al. (2013). The inclusion k2Z of « cos(x)/2 in the first right-hand-side term makes the advection velocity spatially varying. In section 4, nu- and transform Eq. (20) into a system of stochastic dif- 2 merical results are shown for both « 5 0 and « 5 10 3. ferential equations. Here i is the imaginary unit. Like in Following Hodyss et al. (2013), we let Hodyss et al. (2013), we assume the initial condition contains only one mode: c 5 1, m 5 0:1, (21) u(x,0)5 cos(k x) with k 5 1: (25) ›u 0 0 g(u) 5 r with r 5 0:2: (22) ›x a. Case with constant advection velocity (« 5 0) The stochastic noise process n(t) is the same as described in appendix A of Hodyss et al. (2013) (also described in As pointed out by Hodyss et al. (2013), when the ad- appendix A of this paper). For this choice of g(u) and vection velocity is constant, the Fourier modes are un- n(t), the generalized Ito^ correction is given by coupled. The ordinary differential equation (ODE) for F (t) reads " # k N 1 ›2u 1 C(v )2 f 5 r2 0 1 å v 2 dF I 2 lim C( m) , (23) k 52 2 m 2 1 r 2 ›x N /‘N 2 5 ikcF k F ik n(t)F . (26) f f m 1 dt k k k

Unauthenticated | Downloaded 09/25/21 04:59 AM UTC JUNE 2020 S T I N I S E T A L . 2257

The analytical solution of Eq. (26) takes the following b. Case with spatially varying advection velocity form: (« 6¼ 0) ð t For cases with nonzero «, even if the initial condition F (t) 5 A exp 2(ick 1 mk2)t 1 irk n(t0) dt0 , (27) k has a single Fourier mode, the spatially dependent 0 component of the advection velocity causes the repre- with A being any complex constant. Initial condition sentation of the solution to require more than one mode. (25) implies that A 5 1 in Eq. (27); only one Fourier This is an elementary way to introduce coupling between mode (the one corresponding to k 5 1) is sufﬁcient to different Fourier modes but still keep Eq. (20) linear. represent the solution (the Fourier mode for k 521is We truncate the Fourier series in Eq. (24) to retain also needed but due to the solution of (20) being real, it only modes with appreciable magnitudes and denote is the complex conjugate of the solution for the Fourier the largest remaining wavenumber as Nx. Substituting mode with k 5 1). Eq. (24) for the unknown u in (20) gives n N N h i x dF (t) x « å k 5 å 2 1 exp(ikx) ikFk(t) exp(ikx) c cos(x) 52 dt 52 2 k Nx k Nx

2mk2F (t) exp(ikx) 1 ikrF (t) exp(ikx)n(t) (28) k k g.

By multiplying Eq. (28) with exp(2ikx)andinte- 4. Numerical results grating over [0, 2p], we get the following coupled equations: In this section we use numerical results to show how noise n(t) of different color (roughness in time) can affect d 52 1 ... 2 for k Nx 1, , Nx 1, the solution convergence. We also demonstrate how the ^ dF (t) i(k 1 1)« inclusion of the generalized Ito correction can help in k 5 (2ick 2 mk2)F (t) 2 F (t) dt k 4 k11 restoring and/or accelerating convergence. As explained in section 2c, the validity of our generalized Ito^ correction i(k 2 1)« 2 F (t) 1 ikrF (t)n(t); (29) is not tied to any specific time-stepping method. For 4 k21 k simplicity, we use in our numerical experiments the for- d for k 5 Nx, ward Euler scheme as an illustrating example. dF (t) i k 2 « a. Definition of solution error k 5 2 2 m 2 2 ( 1) ( ick k )Fk(t) Fk21(t) dt 4 The error of a numerical solution is evaluated after two 1 r ik Fk(t)n(t); (30) time units of integration using the L2 norm (Hodyss et al. 2013, and D. Hodyss 2018, personal communication): d 52 for k Nx, ð 2p 1/2 dF (t) i(k 1 1)« E(Dt) 5 [u^(x, t 5 2) 2 u(x, t 5 2)]2 dx . (34) k 5 (2ick 2 mk2)F (t) 2 F (t) dt k 4 k11 0 1 r ^ ik Fk(t)n(t). (31) Here u and u are the discrete and analytical solutions, respectively. To ensure the accuracy of the analytical solution Using the notation defined in appendix B, we can write computed for our error evaluation, the time integral of the the above stochastic ODE system for the Fourier mode noise process in Eqs. (27) and (33) is calculated analytically. coefficients Fk in matrix form as b. Case with constant advection velocity (« 5 0) dF For the case with « 5 0, the discretization of Eq. (26) using 5 [D 1 rn(t)H]F. (32) dt forward Euler with the Ito^ correction included is given by

The analytical solution reads F^ (t ) 2 F^ (t ) k j11 k j ^ 2 ^ ð 52ikcF (t ) 2 mk F (t ) t Dt k j k j 5 D 1 Hr 0 0 F(t) exp t n(t ) dt F(0). (33) 1 ikrF^ (t )n(t ) 1 I (t ), (35) 0 k j j k j

Unauthenticated | Downloaded 09/25/21 04:59 AM UTC 2258 MONTHLY WEATHER REVIEW VOLUME 148

FIG. 1. Error in the numerical solution of the 1D advection–diffusion equation with constant advection velocity [« 5 0inEq.(20)]andthedependencyontimestepsize(x axis) and characteristics of the noise term 2 2 2 (a 5 0, 10 6,10 5,10 4, or 1, shown in different colors). Results obtained using the forward Euler scheme (left) without and (right) with the generalized Ito^ correction, respectively, are shown. Simulations were performed

for 100 realizations of the noise process and the l2 solution error was calculated separately for each realiza- tion using Eq. (34). The thick dots are the mean error of the 100 realizations; the vertical bars denote the standard deviation around the mean. The two dashed lines are reference lines indicating convergence rates of 0.5 (upper) and 1.0 (lower), respectively.

where n(tj) is the colored noise at t 5 tj and Ik(tj) is the We make two observations. First, for the case of white ^ Ito correction for Fk at tj: noise (a 5 0, purple line), the Euler scheme without the " # Ito^ correction fails to converge to the analytical solution N 1 1 C(v )2 f no matter how small the step size is [it converges to the I (t ) 52 r2k2F^ (t ) 0 1 å C(v )2 . (36) k j k j m Ito^ solution, cf. section 10.2 in Kloeden and Platen 2 Nf 2 m51 (1992)]. Second, for the case of colored noise (a 6¼ 0, We note that although the expression for the Ito^ cor- blue, green, orange, and red lines), the Euler scheme rection in Eq. (23) involves a limiting process, the limit is without Ito^ correction will start converging to the ana- not present in the expression in Eq. (36) because we lytical solution with order 1 [as predicted by determin- have discretized the equation and thus have picked a istic numerical analysis, see e.g., chapter I.7 in Hairer ﬁnite time step. et al. (1992)] when the step size becomes smaller than Figure 1a [which appears also in Hodyss et al. (2013)] some critical step size that depends on the color of the shows the effect of different noises on the conver- noise (value of a). The more red the noise is (larger a), gence of the Euler scheme without the Ito^ correction. the larger is the critical step size [see also Hodyss et al.

The thick dots are the l2 error of the numerical so- (2013) for a discussion and estimation of the critical lution averaged over 100 realizations of the noise stepsize]. process; the error bars denote the standard deviation Figure 1b shows the effect of including the Ito^ cor- around the average. Using the terminology of sto- rection. We want to make again two observations. First, chastic analysis, this plot (and the rest of them in the for the case of white noise (a 5 0, purple line), the Euler paper) shows the strong convergence of the numeri- scheme with the Ito^ correction does converge to the cal solution.1 analytical solution with order 1/2 (see Hu et al. (2016) for an explanation of this convergence rate). We note that this numerical result was mentioned in Hodyss et al.

1 (2013) although not illustrated by any graphic there. As a reminder, we note that strong convergence is measured by a the mean of the solution error of individual realizations of the Second, for the case of colored noise ( 6¼ 0, blue, green, stochastic equation while weak convergence is measured by the orange and red lines), the Euler scheme with the error of the mean solution. generalized Ito^ correction starts converging to the

Unauthenticated | Downloaded 09/25/21 04:59 AM UTC JUNE 2020 S T I N I S E T A L . 2259

23 FIG.2.AsinFig. 1, but for the case of « 5 10 in Eq. (20). analytical solution with order 1 for larger step sizes G 5 2 2 2 2 2 1 2 ... Diagf ( Nx) , ( Nx 1) , , than the Euler scheme without the Ito^ correction. 2 2 2 2 2 Thus, the addition of the Ito^ correction can help re- (Nx 1) , Nx g. (39) store and/or accelerate convergence of the forward Euler scheme. The truncation wavenumber Nx was chosen empirically: a test simulation was conducted using Eq. (37) with a c. Case with spatially varying advection velocity large Nx; an inspection of the magnitude of the resulting « ^ ( 6¼ 0) Fk revealed Nx 5 5 was sufficient to retain all modes ^ . 24 5 We continue with the case of a spatially dependent with jFkj 10 . Hence, Nx 5 was then used to obtain 2 advection velocity with « 5 10 3 A small value was the results shown in Fig. 2. a 5 chosen for « because the forward Euler scheme is ex- Figure 2a shows that for the case of white noise ( 0, ^ plicit and only first order. As such, it needs a very large purple line), the forward Euler scheme without the Ito number of steps in order to reach the asymptotic con- correction fails to converge to the analytical solution as ^ vergence regime for larger values of « due to the need to expected. Figure 2b shows how the inclusion of the Ito resolve steepening gradients associated with the oscil- correction can restore convergence with order 1/2. We latory nature of the spatial perturbation of the advection note that the standard deviation bars around the mean « 5 velocity. Moreover, the cost of evaluation of the noise appear larger than in the case with 0 because of the n(t), which depends quadratically on the number of time logarithmic scale of the plot. This figure demonstrates a steps, becomes very large when « is large. For practical that for the case of colored noise ( 6¼ 0), the use of the ^ purposes (computational cost), we chose a small « for generalized Ito correction again accelerates the estab- the demonstration here. lishment of the order 1 convergence regime predicted by We discretized Eq. (32) using the Euler scheme with deterministic numerical analysis. the generalized Ito^ correction: ^ 2 ^ 5. Conclusions F(t 1 ) F(t ) j 1 j 5 DF^(t ) 1 rn(t )HF^(t ) 1 I(t ), (37) Dt j j j j Stochastic parameterizations are increasing in pop- ularity in numerical weather prediction and climate ^ where the Ito correction reads modeling as a way to improve the statistical represen- " # tation of the studied phenomena. Naive implementation N r2 1 C(v )2 f 5 0 1 å v 2 G^ of such parameterizations with deterministic numerical I(tj) C( m) F(tj), (38) 2 Nf 2 m51 time integration schemes can cause serious convergence issues. Such issues can be alleviated by the addition of with the matrix G being certain correction terms (known as the Ito^ correction

Unauthenticated | Downloaded 09/25/21 04:59 AM UTC 2260 MONTHLY WEATHER REVIEW VOLUME 148 in stochastic analysis) to the deterministic numerical U.S. Department of Energy (DOE), Office of Science, schemes. However, the Ito^ correction was originally de- Office of Advanced Scientific Computing Research (ASCR) rived only for the special case when the stochastic process and Office of Biological and Environmental Research is represented by white noise. For numerical weather (BER), Scientific Discovery through Advanced Computing prediction and climate modeling it will be useful to (SciDAC) program. Pacific Northwest National Laboratory have the option to properly handle colored noise. is operated by Battelle Memorial Institute for DOE We have derived a generalized Ito^ correction for the under Contract DE-AC05-76RL01830. case of colored noise and applied it to a test problem of an advection-diffusion equation driven by noise of different colors. Our results indicate that the generalized APPENDIX A Ito correction can substantially reduce the time discretization error, significantly improve the convergence Colored Noise and the Corresponding Ito^ Correction rate of the numerical solutions and therefor allow for the Following Hodyss et al. (2013), we use the following use of significantly larger step sizes. specification of the noise process n(t): While our derivation started from a stochastic differ- ( ential equation, the fact that colored noise is in principle N 1 b f 5 qffiffiffiffiffiffiffiffiffiffiffi v 0ffiffiffi 1 å v resolvable by sufficiently small step sizes implies that the n(t) C( 0)p C( m) D 2 m51 generalized Ito^ correction can also be useful for deter- Nf t ministic problems for the purpose of improving solution ) convergence, accuracy, and efficiency. Compared to 3[a sin(v t) 1 b cos(v t)] , (A1) higher-order schemes like the Runge–Kutta family, the m m m m Ito^ correction is less costly in terms of computing time; 2 compared to implicit methods that may provide better C(v) 5 e2av , (A2) stability, the Ito^ correction is less intrusive in terms of 2pm the code modification it requires. v 5 , (A3) m 2 D In the future, we plan to apply the current framework (N 1) t to more realistic atmospheric modeling problems (e.g., N 5 (N 2 1)/2, (A4) simplified versions of the atmospheric general circu- f lation models or their parameterizations). We ac- where N is the number of discrete time levels per unit knowledge the fact that the parameterizations will likely time, including the starting and ending time levels. a is a not be given directly in the multiplicative form required parameter controlling the color of the Fourier spectrum by our formulation. However, there is hope that new of n(t)(a 5 0 corresponds to white noise while a 6¼ 0 approaches [e.g., training a neural network to represent to colored noise). To construct different realizations of the function g(u)inEq.(2)] will allow us to still use our the noise process, we sample, for m 5 0, ... , N construction for more involved parameterizations. f the coefficients a and b independently from the The generalized Ito^ correction for the case of colored m m normal distribution N (0, 1). It should be noted that n(t) noise still assumes that the subgrid phenomena evolve is an approximate random noise. The difference between on a significantly shorter time scale than the phenomena n(t) and the noise term R_(t)insection 2 is that R_(t)would we resolve explicitly. However, as was mentioned in contain an infinite number of Fourier modes while n(t) section 2b, the Ito^ correction can be interpreted as a only has a finite number of modes. Nevertheless, in nu- memory term in model reduction formalisms. This merical modeling, we can use n(t) to approximate R_(t). opens the possibility of exploiting more sophisticated Let us define types of memory terms that correspond to more nu- ð anced and realistic noise processes. t b d 0 0 t n(t ) dt (A5) 0 Acknowledgments. We want to thank Drs. Christopher J. Vogl (LLNL), Carol S. Woodward (LLNL), and and Shixuan Zhang (PNNL) for helpful discussions on the Db 5 b 2 b numerical examples shown in this paper, and Dr. Daniel 1D , (A6) j tj t tj Hodyss (NRL) for clarifications regarding his earlier work that inspired our study. We also want to thank the and consider Dbj as an approximation to DRj (recall that D 5 2 anonymous referees for their insightful comments Rj Rtj1Dt Rtj is the increment of the stochastic and suggestions. This work was supported by the process R). For the above-defined n(t), we find

Unauthenticated | Downloaded 09/25/21 04:59 AM UTC JUNE 2020 S T I N I S E T A L . 2261

FIG. A1. (a) Autocorrelation of the noise process n(t) for different values of a. (b) The e-folding time of the noise process n(t) for different values of a.

2( Also note that per construction, we have ð t 1Dt Nf 2 4 j 1 b E[(Db ) ] 5 E qffiffiffiffiffiffiffiffiffiffiffi C(v )p0ffiffiffi 1 å C(v ) E 2 [ E 2 [ j 0 m [am] 1, [bm] 1, (A8) t D 2 m51 j Nf t 3 ) for any m 5 0, 1, ..., Nf. Therefore, ! 2 ( 5 2 2 Nf 3 [a sin(v t) 1 b cos(v t)] dt . (Dt) C(v ) m m m m E Db 2 5 0 1 å v 2 2 v [( j) ] C( m) [sin ( mtj) N Dt 2 5 f ) m 1

D E Db 2 1 cos2(v t )] (A9) For small t we can approximate [( j) ]as m j 2( " # 2 Nf Nf v 1 C( 0) 2 4 1 b0 5D 1 å v E qffiffiffiffiffiffiffiffiffiffiffi C(v )pffiffiffi 1 å C(v )[a sin(v t ) t C( m) . (A10) 0 m m m j N 2 m51 D 2 m51 f Nf t ) 3 E Db 2 ! 2 The expression for [( j) ] can be used to obtain the integral form of the generalized Ito^ correction, 1 v D 5 bm cos( mtj)] t . namely 1 dg(u) limå g[u] E[(DR )2] (A11) D / j Taking into account the independence among the co- t 0 j 2 du tj efficients a and b , we have m m " # N " # v 2 f 2 1 dg(u) 1 C( ) 2 1 b 5 limå g[u] 0 1 å C(v ) Dt. E Db 2 5 E v 0ffiffiffi D D / m [( ) ] C( )p t t 0 j 2 du t N 2 m51 j D 0 j f Nf t 2 (A12) N 1 f 1 å E[C(v )2a2 sin2(v t )(Dt)2] ^ D m m m j Thus, the differential form of the generalized Ito cor- N tm51 f rection reads Nf 1 2 2 2 2 " # 1 å E v v D N [C( m) bm cos ( mtj)( t) ] 2 f N Dt 5 1 dg 1 C(v ) f m 1 I 5 g(u) lim 0 1 å C(v )2 . (A13) N /‘ m (A7) 2 du f Nf 2 m51

Unauthenticated | Downloaded 09/25/21 04:59 AM UTC 2262 MONTHLY WEATHER REVIEW VOLUME 148

The reason we keep in the ﬁnal formula the limit Nf / ‘ APPENDIX B is that Nf depends on the length of the subinterval Dt. D / / ‘ So, when 0 the number of frequencies Nf .We Advection-Diffusion Equation with Spatially have provided in the main text a short discussion on the Varying Advection Velocity behavior of this limit. We have computed the autocorrelation and the Let us deﬁne e-folding time of the noise process n(t) for different 5 ... T a F (F2 , F2 1 , , F 2 , F ) , (B1) values of the parameter (see Fig. A1). Nx Nx 1 Nx 1 Nx H 5 2 2 1 ... 2 i Diagf Nx, Nx 1, , Nx 1, Nxg, (B2)

and

2 i(N 2 1)« 6 icN 2 mN2 x 00 6 x x 4 6 6 6 iN « i(N 2 2)« 6 x ic(N 2 1) 2 m(N 2 1)2 x 0 6 x x 6 4 4 6 6 2 « 2 « 6 i(Nx 1) 2 i(Nx 3) 6 0 ic(N 2 2) 2 m(N 2 2) D 5 6 4 x x 4 6 6 6 6 6 6 0 6 6 6 4 0 0

3 00 0 7 7 00 0 7 7 7 ... 7 7 7 2 « 2 « 7 i(Nx 3) 2 i(Nx 1) 7 2 2ic(N 2 2) 2 m(N 2 2) 2 0 7 4 x x 4 7 7 7 i(N 2 2)« iN « 7 0 2 x 2ic(N 2 1) 2 m(N 2 1)2 2 x 7 x x 7 4 4 7 5 i(N 2 1)« 00 2 x 2icN 2 mN2 4 x x

This notation allows us to write Eqs. (29)–(31) as Ewald, B., and R. Temam, 2005: Numerical analysis of stochastic Eq. (28) and the analytical solution as Eq. (33). schemes in geophysics. SIAM J. Numer. Anal., 42, 2257–2276, https://doi.org/10.1137/S0036142902418333. ——, C. Penland, and R. Temam, 2004: Accurate integration of ñ REFERENCES stochastic climate models with application to El Ni o. Mon. Wea. Rev., 132, 154–164, https://doi.org/10.1175/ Berner, J., and Coauthors, 2017: Stochastic parameterization: 1520-0493(2004)132,0154:AIOSCM.2.0.CO;2. Toward a new view of weather and climate models. Bull. Gardiner, C. W., 1985: Handbook of Stochastic Methods. 2nd ed. Amer. Meteor. Soc., 98, 565–588, https://doi.org/10.1175/ Springer, 442 pp. BAMS-D-15-00268.1. Hairer, E., S. P. Nørsett, and G. Wanner, 1992: Solving Ordinary Chorin, A. J., and P. Stinis, 2007: Problem reduction, renormali- Differential Equations I: Nonstiff Problems. Springer, 528 pp. zation, and memory. Commun. Appl. Math. Comput. Sci., 1, Hansen, J. A., and C. Penland, 2006: Efﬁcient approximate 1–27, https://doi.org/10.2140/CAMCOS.2006.1.1. techniques for integrating stochastic differential equations.

Unauthenticated | Downloaded 09/25/21 04:59 AM UTC JUNE 2020 S T I N I S E T A L . 2263

Mon. Wea. Rev., 134, 3006–3014, https://doi.org/10.1175/ McFarlane, N., 2011: Parameterizations: Representing key processes MWR3192.1. in climate models without resolving them. Wiley Interdiscip. Rev.: Hodyss, D., K. C. Viner, A. Reinecke, and J. A. Hansen, 2013: The Climate Change, 2, 482–497, https://doi.org/10.1002/WCC.122. impact of noisy physics on the stability and accuracy of Moon, W., and J. Wettlaufer, 2014: On the interpretation of physics-dynamics coupling. Mon. Wea. Rev., 141, 4470–4486, Stratonovich calculus. New J. Phys., 16, 055017, https://doi.org/ https://doi.org/10.1175/MWR-D-13-00035.1. 10.1088/1367-2630/16/5/055017. ——, J. G. McLay, J. Moskaitis, and E. A. Serra, 2014: Inducing Øksendal, B., 2003: Stochastic Differential Equations: An Introduction tropical cyclones to undergo Brownian motion: A comparison With Applications. Springer, 379 pp. between Ito^ and Stratonovich in a numerical weather predic- Papanicolaou, G. C., and W. Kohler, 1974: Asymptotic theory of tion model. Mon. Wea. Rev., 142, 1982–1996, https://doi.org/ mixing stochastic ordinary differential equations. Commun. 10.1175/MWR-D-13-00299.1. Pure Appl. Math., 27, 641–668, https://doi.org/10.1002/ Hu, Y., Y. Liu, and D. Nualart, 2016: Rate of convergence and cpa.3160270503. asymptotic error distribution of Euler approximation schemes Rüemelin, W., 1982: Numerical treatment of stochastic differential for fractional diffusions. Ann. Appl. Probab., 26, 1147–1207, equations. SIAM J. Numer. Anal., 19, 604–613, https://doi.org/ https://doi.org/10.1214/15-AAP1114. 10.1137/0719041. Kloeden, P. E., and E. Platen, 1992: Numerical Solution of Sardeshmukh, P. D., C. Penland, and M. Newman, 2003: Drifts Stochastic Differential Equations. Springer, 636 pp. induced by multiplicative red noise with application to cli- Leutbecher, M., and Coauthors, 2017: Stochastic representations of mate. Europhys. Lett., 63, 498–504, https://doi.org/10.1209/epl/ model uncertainties at ECMWF: State of the art and future i2003-00550-y. vision. Quart. J. Roy. Meteor. Soc., 143, 2315–2339, https:// Stratonovich, R. L., 1963: Topics in the Theory of Random Noise. doi.org/10.1002/qj.3094. Gordon and Breach, 329 pp. Mannella, R., and P. V. McClintock, 2012: Ito^ versus Stratonovich: Sura, P., and C. Penland, 2002: Sensitivity of a double-gyre ocean 30 years later. Fluct. Noise Lett., 11, 1240010, https://doi.org/ model to details of stochastic forcing. Ocean Modell., 4, 327– 10.1142/S021947751240010X. 345, https://doi.org/10.1016/S1463-5003(02)00008-2.

Unauthenticated | Downloaded 09/25/21 04:59 AM UTC