Extensions of the Novikov-Furutsu Theorem, Obtained by Using Volterra Functional Calculus

Extensions of the Novikov-Furutsu theorem, obtained by using Volterra functional calculus

G.A. Athanassoulis and K.I. Mamis [email protected], [email protected]

School of Naval Architecture & Marine Engineering, National Technical University of Athens, 9 Iroon Polytechniou st., 15780 Zografos, GREECE

Abstract. Novikov-Furutsu (NF) theorem is a well-known mathematical tool, used in stochastic dynamics for correlation splitting, that is, for evaluating the mean value of the product of a random functional with a Gaussian argument multiplied by the argument itself. In this work, the NF theorem is extended for mappings (function-functionals) of two arguments, one being a random variable and the other a random function, both of which are Gaussian, may have non-zero mean values, and may be correlated with each other. This extension allows for the study of random differential equations under coloured noise excitation, which may be correlated with the random initial value. Applications in this direction are briefly discussed. The proof of the extended NF theorem is based on a more general result, also proven herein by using Volterra functional calculus, stating that: The mean value of a general, nonlinear function-functional having random arguments, possibly non-Gaussian, can be expressed in terms of the characteristic functional of its arguments. Generalizations to the multidimensional case (multivariate random arguments) are also presented.

Keywords: Novikov-Furutsu theorem, Volterra-Taylor expansion, correlation splitting, random differential equation, coloured noise, characteristic functional

TABLE OF CONTENTS

Abstract 1 1. Introduction 2

2. The mean value of a random, nonlinear function-functional 3

3. The function-functional shift operator and its exponential form 6 4. Proof of Theorem 1 7 5. Proof of the extended NF Theorem 8 6. Application of the extended NF Theorem to the study of random differential equations under coloured Gaussian excitation 12 7. First example and validation: The exact response pdf evolution equation corresponding to a linear random differential equation 15 8. Generalization of Theorems 1 and 2 to the multidimensional case 17 9. Summary and conclusions 19 Appendix A. Volterra‟s principle of passing from discrete to continuous. Volterra calculus 20 Appendix B. Proof of Lemmata 1 – 6 23 References 28

1 1. Introduction

In their works (Furutsu, 1963; Novikov, 1965), on electromagnetic waves and turbulence respectively, K. Furutsu and E. Novikov derived independently the following formula (1)

t t F [(;)]  t t 0 (;)tCF [(  t sds ;)](,)  ()() , t 0   (;)s  t 0  where ( ;) is a zero-mean Gaussian random function with autocovariance function t  Ct()() s (,) , F [ (; )]  is a functional of ( ;) over [,]tt0 , [] denotes the t 0 mean value operator, and /( ;s )  is the Volterra functional derivative with respect to ( ;) at s . The above equation, called henceforth the Novikov-Furutsu (NF) theorem, has been extensively used in turbulent diffusion (Cook, 1978; Hyland, McKee and Reeks, 1999; Shrimpton, Haeri and Scott, 2014; Krommes, 2015; Martins Afonso, Mazzino and Gama, 2016), random waves (Sobczyk, 1985; Rino, 1991; Bobryk, 1993; Konotop and Vazquez, 1994; Creamer, 2008), and stochastic dynamics (Protopopescu, 1983; Klyatskin, 2005, 2015; Scott, 2013; Kliemann and Sri Namachchivaya, 2018). One particular use of the NF theorem lies in the formulation of partial differential equations, called generalized Fokker-Planck-Kolmogorov (genFPK) equations, that govern the evolution of the probability density function (pdf) of the response to a random differential equation (RDE) excited by coloured noise (Sancho et al., 1982; Cetto and de la Peña, 1984; Fox, 1986; Hyland, McKee and Reeks, 1999; Venturi et al., 2012).

In this work, the classical NF theorem is extended (Theorem 2 in Section 2) to random function- t functionals ( RFF ) F [();(;)]X 0  (function on the scalar random variable X 0 () t 0 and functional on the scalar random function (;) over the interval [,]tt0 ). RFF ‟s arguments X () , (;) are considered correlated and jointly Gaussian, with mean values m , 0 X 0 m (), autocovariances, C , C (,) , and cross-covariance C (). This exten- () XX00 ()() X 0 () sion of the NF theorem is useful for the formulation of pdf equations corresponding to RDEs in which both the initial value and the excitation are Gaussian random elements, correlated to each other; see Section 6.

What is more, the above extension of the NF theorem is derived as a special case of a more general result (Theorem 1 in Section 2), whose importance has not been highlighted before, to the best of our knowledge. Theorem 1 expresses the mean value of a general, nonlinear RFF in terms of the joint characteristic function-functional ( j.Ch.FF ) of its arguments, which can be non-Gaussian.

Theorems 1, 2 are proved in Sections 4 and 5 respectively, by using the mathematical tools introduced by Vito Volterra, namely the concept of passing from the discrete to continuous and his eponymous calculus for functionals, which are presented briefly in Appendix A. An application of the extended NF Theorem to the derivation of an evolution equation governing the response

(1)  denotes the stochastic argument.

2 probability density function of ordinary differential equations under coloured excitation is briefly presented in Section 6. In Section 7, we formulate and study the exact response pdf evolution equation for a linear RDE under Gaussian coloured noise correlated to its Gaussian initial value. The unique solution of this equation has been found, and it turns to be identical with the correct Gaussian response pdf (obtained through the corresponding linear moment problem), as expected. Since the derivation of the said evolution equation is based on the extended NF Theorem 2, this example also constitutes a first validation for Theorem 2. In Section 8, we state (without complete proofs) the generalizations of Theorems 1 and 2 to the multidimensional case, i.e. when

X 0 () and Ξ( ;) are a random vector and a random vector function, respectively. In the concluding Section 9, a recapitulation of the results is given, while the merits of the novel proof of NF theorem presented herein and further generalizations, are discussed.

2. The mean value of a random, nonlinear function-functional

Before proceeding with the study of the RFF , we shall introduce its deterministic counterpart and state the basic analytical assumptions. Consider a real-valued function-functional, t G[;()]: u Î , where Î C[,] t0 t is the space of continuous func- t 0   t tions. The function-functional G[;()] u is assumed to have derivatives of any order, both t 0 with respect to the scalar argument  and the function argument u (), and it is expandable in

Volterra-Taylor series, jointly with respect to  and u (), around a fixed pair  00;()u  . Here, functional derivatives are considered in the sense of Volterra (Volterra, 1930), which are reconsidered in a rigorous manner by (Donsker and Lions, 1962). For the sake of brevity, the t above smoothness conditions of G[;()] u will be denoted as C  henceforth. t 0

The RFF is obtained by replacing the argument  ;()u  by the random element

 X 0 ();(;)  . The latter is fully described by the infinite-dimensional joint probability measure P , defined over the Borel   algebra () . Probability measure P X 0 () BÎ X 0 () is assumed to be continuously distributed, i.e. it possesses a well-defined probability density t functional. Since X 0 ();(;):     Î is Borel measurable, and G[;()] u is   t 0  t C , the composition G[();(;)]X 0  is also BÎ()measurable. t 0

t Then, by definition, the mean value of G[();(;)]X 0  is expressed as t 0

 [();(;)][;()]()Xt    P d   d  . (1) P X  ( ) GG0t X 0  ( )   0 0   Î

However, the above definition involves an integral over an infinite-dimensional space, which, in most of the cases, is rather difficult to calculate. As a more easily computable alternative, we express, in the following theorem, the mean value of a RFF , via the probabilistic structure of its arguments, as described by its joint characteristic function-functional ( j.Ch.FF ) :

3 t t   X ( )0[  ;ui  ( X )] iP  s exp u s ( ds ) ( ; ) ( ) . (2) 0 t 0 X 0  ( )  t 0

Note that, the random elements X 0 () , (;)  may be dependent, having any prescribed probability distribution.

Theorem 1 [Mean value of an RFF ]: The mean value of the random function-functional t G[();(;)]X 0  is expressed by the formula t 0

tt  [Xu ();( ;)];    [;( )] GG0 ttXˆ ˆ ()   m 000 iu i () X 0  um()()  () (3) t  t  exp Xˆ () dsˆ (;)  su [;()] 0 G t   m  0 X 0  us() um()() t 0  ()

 where    P   , m X and ms()() are the mean values of X 0 () and (;)  , X 0  () 0 respectively, /  denotes partial differentiation with respect to  , and /()us denotes Volterra functional differentiation with respect to the function u () at s . Further, the quantities

XXˆ ()() m , ˆ (;)(;)()s   s  m s (4a,b) 00X 0 ()

are the fluctuations of the random elements X 0 () and  (;)s  around their mean values, and

ˆ ˆ [;()]u is the j.Ch.FF of the said fluctuations. The exact meaning of the operator X 0 () appearing in the right-hand side of Eq. (3) will be defined below, during the derivation of Eq. (3). ■

Remark 1. By comparing formula (3) to Eq. (1), we observe that, by Theorem 1, we compute an integration over an infinite-dimensional space by the action of a pseudo-differential operator. A similar practice of integration by differentiation in functional spaces, albeit with quite different applications, has been recently introduced by Kempf, Jackson and Morales (2014, 2015); Jia, Tang and Kempf (2017) in quantum field theory.

Remark 2. A formula similar to Eq. (3) for a random functional of the form G[(;)]

GG12[(;)][(;)] is given by (Klyatskin, 2005), Ch. 4. The proof given by Klyatskin is concise and unmotivated, thus difficult to follow for a reader not well-acquainted with Volter- ra functional calculus. Eq. (3) includes Klyatskin‟s result as a special case. The proof presented herein is fairly detailed and motivated by easily understood analogues in the discrete setting. Then, the Volterra technique of passing from the discrete to continuous reveals the appropriate forms in the function-functional setting. The latter can be directly proved by using Volterra functional calculus.

4 The proof of Theorem 1 is completed in Section 4, and is based on the expansion of the RFF t G[();(;)]X 0  around the mean values m X and m (), by employing the function- t 0 0 functional shift operator. The latter is introduced in Section 3, via Volterra‟s concept of passing from the discrete to continuous. This principle, discussed concisely in Appendix A, motivates a calculus for functionals that is analogous to the calculus for functions of many variables, which can be (and has been) rigorous; see also Appendix A for relevant discussion and references.

Now, by setting in Eq. (3):

[();(;)][]X t   GG0 t 0 (5) t  (;)[();(;)](;)[],tFF Xt0      t 0 specifying the j.Ch.FF [;()]u as a Gaussian one (see Appendix A for its deriva- X 0 () tion): Gauss t X ()[;()]u  0 t 0 t t t 1 expi m ( s ) u ( s ) d s  C(,)()() s s u s u s ds ds () 2   ( )()  1 2 1 221 t0 t 0 t 0 t 1 2  expi mXX  CXX   exp,   C()() s) u( s ds (6) 02 0 0  0 t 0  and calculating the action of the operator  Xˆ ˆ (); on the function-functional 0 iu i () t u()[;()] tF  u , the following extension of the NF theorem for RFF s is obtained. t 0

Theorem 2 [The extended Novikov-Furutsu theorem]: For a sufficiently smooth RFF of the t form FF[();(;)][]X 0 , whose arguments X 0 () ,  (;) are jointly Gaussi- t 0 an, the following formula holds true

 (;)[]t  F   (7)  F [] m ()[]t C ()t  ()() F  X 0   X 0 () t  F []  C()() (,). t s ds ■   (;)s  t 0

Detailed proof of Theorem 2 is given in Section 5. Note that the extension of the classical NF theorem for a non-zero mean  (;)t  uncorrelated to X 0 () , that gives rise to the term

5  mt()( )[ ] F  in Eq. (7), has also been given in (Hänggi, 1989; Cáceres, 2017, Sec.

1.12.2). The general form of Eq. (7), corresponding to correlated X 0 () and  (t ;) , is novel, to the best of our knowledge.

3. The function-functional shift operator and its exponential form

The shift operator for function-functionals is constructed by using Volterra's passing from the discrete to continuous, as explained in Appendix A. As a first step, we introduce its discrete analogue, the multivariate shift operator Txˆ for functions of many variables, g ()xx0 ˆ

Txˆ g ()x 0 , which is in fact a compact form of the Taylor series expansion. N 1 Let gC()(). Then, the Taylor expansion of gg(xx )() x 0 ˆ around the N 1 point x 0  , reads as 1 NN 11()m   m gx(x xxxgˆˆ )  1( ˆ  ) . (8) 00   nn1 m mx! x nn m1 n1  1 n m  1 1 m

To write the above equation in a more concise form, the following identity is used

m NNNN1m  1  1()m  1 hn u n h n u n  h n h n  u n u n , (9)  ii   11mm n1i  1 nim  1 n1  1 n  1 which is valid under the assumption that all symbols hu, commute, or their order is definite- nnii ly prescribed, as dictated by the specific case. Setting hxnn ˆ and uxnn /  , and imposing the order convention that all xˆ n precede of all / x n , we can apply Eq. (9) to express the differential operator appearing in the right-hand side of Eq. (8), recasting the latter in the following compact form:

m N 1 1  g()()x xˆˆ x g x . (10) 00mx! n  mn01n

N 1 0

In Eq. (10) the additional (trivial) convention xxˆ nn/   1 is used. Then, we observe  n  1  that the infinite series appearing in the right-hand side of Eq. (10), under the said order assump- N 1 tion, is identified as the Taylor series of the symbol expxxˆ nn / , leading to the   n  1  nice formula

N 1  g(x xˆ )  exp x ˆ g ( x )  expx ˆ    g ( x ) . (11) 0 n  x 0x 0 n  1 n

Comparing Eq. (11) with gg()()x00 xˆ T xˆ x , we see that the symbol exp xˆ x  can be identified as the shift operator for the case of functions of many variables. This is the multivari-

6 ate analogue of the translation (shift) operator exp()xdˆ / dx for functions of one variable, first introduced by Lagrange; see also (Glaeske, Prudnikov and Skòrnik, 2006, Sec. 1.1).

Denoting the first component of x by (the scalar)  , and the remaining ones with the N  dimensional vector u , Eq. (11) is recast as

G(;)(;)(;)u GG 0  ˆ u 00  uuˆ 0 T ˆ uˆ N   (12) exp(ˆˆ ; )  exp(uˆˆn  ;  GG), 0  uu 00 0  u u  u n  1 n  providing us with the shift operator for a function of a scalar and a vector argument, G(;). u

Now, by setting in the above equation uunn t (), where ut() is a continuous function, and applying Volterra‟s approach to pass from the discrete to continuous (see Appendix A), we ob- t serve that the function G(;) u tends to the function-functional G[ ;u ()] and the sum t 0

N t uuˆ nn/ tends to the integral uˆ ()/() su s ds , where /(us ) is the Volterra  n  1 t 0 functional derivative. Thus, Eq. (12) suggests the following exponential form of the shift opera- t tor T when applied to the function-functional G[;()] u : ˆ uˆ() t 0

t t t GG[;()][;()()]u 00  ˆ u  uˆ  t0 t 0 t 0 t T G[;()] 00u (13) ˆ uˆ() t 0 t  t expˆ uˆ ( s ) dsG [00 ; u ( )].   us() t 0 t 0

By omitting the scalar argument  , Eq. (13) defines the functional shift operator Tuˆ() , derived in (Sobczyk 1985, chap.I, Eq. 1.116'), where Volterra‟s passing was also employed. A rigorous derivation of Eq. (13) can be obtained by direct use of the Volterra-Taylor series expansion of t G[;()] u ; see Appendix A. t 0

4. Proof of Theorem 1

Equation (13) is the essential deterministic prerequisite for the proof of Theorem 1. Having it at our disposal, the proof of Theorem 1 is straightforward. Substituting, in Eq. (13), the arguments

 ,()u by the random arguments X 0 () , (;) , we obtain the following representation of t the random RFF G[();(;)]X 0  : t 0

tˆ tˆ t GG[();(;)][();()(;)]X0   mX  X 0  m ( )     t00 t 0 t 0

t TGˆ ˆ ()[;()] mmX () X 0 () 0 t 0

7 t ˆ  ˆ t expX 0( ( ) ) ds ( sm ; ) m G [X ; ( )]. (14)   us() 0 t 0 t 0

Recall that m , m () are the mean values, and Xˆ () , ˆ ( ;) are the fluctuations of the X 0 () 0 random elements X 0 () , ( ;) around their mean values; see Eq. (4).

Eq. (14) provides us with a decomposition of the random RFF G[();(;)]X 0  in the form of a random shift operator “times” the deterministic function-functional [;()].mm By averaging both sides of Eq. (14), we obtain G X 0 ()  t G[();(;)]X 0  t 0 t (15)  ˆ  ˆ t exp  X0( ( ) ) ds ( sm ; ) m G [X ; ( )].   us() 0 t 0 t 0

 The term exp  , in the right-hand side of Eq. (15), is called the averaged shift operator. Finally, recalling the form of the j.Ch.FF , Eq. (2), we see that Eq. (15) can be also written as

 tt GG[();(;)];[;()]Xm0( ) m   ˆ ˆ X  , tt00X 0 ()0 i i u() s which is exactly Eq. (3). Thus, the proof of Theorem 1 is completed. In addition, the term

ˆ ˆ  /;()i/ i u s  is identified as the averaged shift operator for RFF s. X 0 ()

5. Proof of the extended NF Theorem

Since, in Theorem 2, the arguments X 0 () , (;)s  are assumed jointly Gaussian, their fluctuations Xˆ ()() X m , ˆ (;)(;)()s   s  m s are also jointly Gaussian with 00X 0 () zero mean values and the same central moments CCˆˆ  XX , C ˆ ˆ () C () and XX00 00 X 0 () X 0 ()

CCˆˆ()() (,)(,) ()() . Thus, the j.Ch.FF of the random fluctuations around mean t values, ˆ ˆ [;()]u , is now specified, by using Eq. (6), as X 0 () t 0 tt Gauss t 1 ˆ ˆ [ ;u ( )] exp C()() (,)()() s1 s 2 u s 1 u s22 ds 1 ds  X 0 ( ) t 0 2  tt00 t 1 2  exp CXXX  exp  C()()() s u s ds . (16) 2 00  0 t 0

8 Accordingly, in this case, the averaged shift operator ˆ ˆ  /iu s ;(/ )i  can be ex- X 0 () pressed as the product of three operators

  ˆˆˆˆ ˆ ˆ; ˆ ˆ TTT , (17) XX0 ( X )( ) X 0 0 0   ()()  i i u() s   defined by

1  2 T XXˆˆ  expC XX 2 , (18a) 00 2 00

t  T ˆ ˆ  exp CX ()( sds) , (18b) X 0 ()  0  us( ) t 0

tt 1 2 T  expC( s, sds) ds . (18c) ˆˆ()()  ( )() 1 21 2 2(us12)()us tt00

Operators T can be considered as second-order versions of the shift operators, defined in Sec- tion 3. Thus, they will be called quadratic averaged shift operators. Before proceeding further (to complete the proof of Theorem 2), we shall study the basic properties of the said operators.

Properties of operators T . On C  function-functionals, operators T are well-defined and they have the following properties, which are needed subsequently for the proof of the extended NF theorem.

Lemma 1: Operators T are linear.

t t That is, for any two C  function-functionals, G[;()] u , F [;()] u it holds true that t 0 t 0

t t t t TTGFGF[;()][;()][;()][;()] u   T   u     u   u ,  t0  t 0   t 0 t 0 (19)

where T stands for any of the three operators TTTˆ ˆ,, ˆˆ ˆ ˆ , and  ,  are XXX0 0 0 () ()() scalars or scalar functions having argument(s) different than the differentiation argument(s) appearing in the corresponding operator T .

Lemma 2: Operators T commute with   and us() differentiations.

 t That is, for a C function-functional G[;()] u , and for TTTT ˆ ˆ,, ˆˆ ˆ ˆ , t 0  XXX0 0 0 () ()() 

 [;()] u t  t G t TTG[;()] u  0 , (20a) t 0  

9 and [;()] u t t G t TTG[;()] u  0 . (20b) u( su )( s ) t 0  

Lemma 3: Operators T commute with each other.

t That is, for any C  function-functional G[ ;u () ] t 0

tt TTTGTTTGˆ ˆ ˆˆˆ ˆ ˆ[;()][;()]uu ˆ ˆ ˆ ˆ ˆ XXXXXX0 0 0()()()()()() tt00 0 0 0   t  TTTGˆ ˆ ˆ ˆ ˆ ˆ [;()] u . (21) ()() XXX0 0 0  () t 0

In other words, the product of the three operators T , under any permutation of their order, has t the same action on G[ ;u () ] . Proofs of Lemmata 1-3 are presented in Appendix B. t 0

Using Eqs. (17) and (18), Eq. (3), for jointly Gaussian arguments, takes the form

 tt    GTTTG[();(;)][;()]Xu0  ˆ()() ˆXXX ˆˆ () ˆ ˆ    m X (22) tt00  0 0 0  0 um()()  ()

For the case GF[](;)[] t  , Eq. (5), as assumed in Theorem 2, Eq. (22) reads

 (;)[]t  F  t (23)  u()[;()] t u TTTFˆ()() ˆXXX ˆˆ () ˆ ˆ t   m  0 0 0 0  X 0 um()()  ()

The main part of the proof for the extended NF theorem lies in the successive applications of the t three operators T on u()[;()] tu F  . Since, by virtue of Lemma 3, the order of applica- t 0 tion of operators T does not alter the final result, we choose the order shown in Eq. (23). The action of each operator T is given by the following lemmata, the proofs of which are presented in Appendix B.

t Lemma 4. The action of operator T ˆˆ on u()[;()] tF  u is given by XX00 t 0

tt    TFTFˆ ˆu()[;()]()[;()] t u u t ˆ ˆ u . (24) XXXX0 0tt00  0 0  

In the right-hand side of Eq. (24), TFˆˆ  [;()] u  is a new function-functional, which is XX00 denoted by F1 [;()] u . By applying operator T ˆ ˆ on both sides of Eq. (24) we obtain X 0 ()

tt    TTFTFˆˆˆ ˆ ˆu()[;()]()[;()] t u ˆ u t1 u . (25) XXXX0()() 0 0tt00  0  

The right-hand side of Eq. (25) is calculated by using the following result:

10 t Lemma 5. The action of operator T ˆ ˆ on u( tu )F1 [ ; ()] is given by X 0 () t 0

 tt Tˆˆˆˆu()[;()]()[;() t FT11 u F  u t u ] XX00( )( ) tt00 t (26)  F1 [;() u ] t 0  C X ()(t ) T ˆ ˆ . 0 X 0 ()  

Denoting the term TFˆ ˆ 1 [;()] u by F2 [;()] u , and combining Eq. (26) and Eq. X 0 ()  (25), we obtain tt Tˆˆ T ˆ ˆ u()[;()]()[;()] t FF u u tu2 XXX0() 0 0 tt00 t  F1 [;() u ] t 0  CtX ()()T ˆ ˆ . 0 X 0 ()  

Applying the operator T ˆˆ()() on both sides of the above equation, leads to

tt    TTTFTFˆ ˆ ˆˆ ˆ ˆˆ ˆu()[;()]()[;()] t u u t2 u ( )  ( )( )XXX (0 ) () 0 0 tt00    t (27)  F1 [;()] u t 0  Ct() . TTˆ()() ˆX 0 () X ˆ ˆ () 0  

We shall now elaborate on the two terms appearing in the right-hand side of Eq. (27). By using the linearity of T ˆˆ (Lemma 1) and the commutation of T ˆˆ with the   derivative ()() XX00

(Lemma 2) in conjunction with the definition of F1 [;()] u , the second term in the right-hand side of Eq. (27) is expressed in terms of the function-functional F [;()] u as follows:

t  F1 [;()] u t 0 Ct()  TTˆ()() ˆX 0 () X ˆ ˆ () 0   (28) t  F [;()] u t 0  CtX ()()TTTˆ ˆ ˆˆ ˆ ˆ . 0 ()() XXX0() 0 0  

Concerning the first term in the right-hand side of Eq. (27), we need the following Lemma:

t Lemma 6. The action of operator T ˆˆ on u()[;()] tF2  u is given by ()() t 0

11  tt Tˆ ˆˆ ˆ u()[;()]()[;()] t FT22 u F  u tu ( )  ( )( ) ( ) tt00 t [;()] u t (29) F2 t  C( t , sds ).T 0  ()() ˆˆ()() us() t 0 

Having now at our disposal the results of the previous lemmata, we are able to proceed to the

Finalization of the proof of Theorem 2: Substituting Eqs. (28) and (29) into Eq. (27), recalling that FTF21[ ;uu ( )][ ;ˆ (ˆ )]  and FT1 [ ;uu ( F )][ ;ˆˆ ( )] , and X 0 ()  XX00 employing the commutation of operators T ˆ ˆ , T ˆˆ with the Volterra derivative X 0 () XX00 /(us ) (Lemma 2), we obtain

t TTTFˆ ˆ ˆˆ ˆ ˆ u()[;()] t u  ()() XXX0() 0 0 t 0 t u()[;()] tu TTTFˆ ˆ ˆˆ ˆ ˆ  ()() XXX0() 0 0 t 0 t  F [;()] u t 0 (30) CtX ()()TTTˆ ˆ ˆˆ ˆ ˆ 0 ()() XXX0() 0 0   t t F [;()] u t 0  Ct()() (,)s TTTˆ ˆ ˆˆ ˆ ˆ ds.  ()() X 0 () XX00us() t 0 

By setting   m , and um( )( ) in Eq. (30), and applying Eq. (22) to each term of the X 0 () form TTTˆ ˆ ˆˆ ˆ ˆ   in both sides of Eq. (30), we obtain the extended NF theorem, ()() XXX0() 0 0 Eq. (7). The proof is now completed.

6. Application of the extended NF Theorem to the study of random differential equations under coloured Gaussian excitation

Let us consider the jointly Gaussian X 0 () and (;)t  to be the initial value and the excitation to the nonlinear random differential equation (RDE):

X(;)((;))(;) t h X t     t  , X(;)() t00 X , (31a,b) where hx() is a deterministic, continuous, nonlinear function, and  is a constant. The response pdf, fxXt()(), of RDE (31) can be represented as the average of a random delta function (Hänggi, 1978; Sancho and San Miguel, 1980; Cetto and de la Peña, 1984; Fox, 1986; Venturi et al., 2012; Wang, Tartakovsky and Tartakovsky, 2013; Athanassoulis, Kapelonis and Mamis, 2018):

 fXt()()((;)) x x X t . (32)

By differentiating both sides of Eq. (32) with respect to time and employing RDE (30), we obtain an evolution equation for fxXt()()

 fxXt()()   h() x fXt()  ()(;)( xt  x  X t (;))   , (33) t  xx called the corresponding stochastic Liouville equation (SLE). By treating the solution Xt( ;) of RDE (31) as a function of the initial value X 0 () and a functional on the excitation ( ;) over the time interval [,]tt0 , the averaged term appearing in the right-hand side of SLE (33) is

 t expressed as (;)[t  x (); X X (0 ;)] , which is a form suitable for the ap-  t 0  plication for the extended NF theorem, Eq. (7), with

tt F [Xx00 ();( X X  ;)][  ();( ;)] . (34) tt0  0 

Under the use of the extended NF Theorem, and the chain rule for the derivatives of delta function, SLE (33) is transformed into

 fx() Xt()   h()()() x  m()() t fXt x  tx t 2 XX[();(;)]  0 t Ct()   ((;))x X t  0 X 0 () x 2 X () 0 t t 2 XX[();(;)]  0 t C(,)((;)). t s   x X tds  0 (35) xs2  ()() (;) t 0 

For the case of zero mean excitation ( mt()( ) 0 ), uncorrelated to the initial value ((Ct )) , transformed SLE (35) reduces to the SLE derived in (Sancho et al., 1982; Hänggi X 0 () et al., 1985; Fox, 1987; Peacock-López, West and Lindenberg, 1988; Hänggi and Jung, 1995).

By employing a current-time approximation scheme (see references below) for the averaged terms in the right-hand side of the transformed SLE (35), we can obtain an approximate, yet closed equation governing the evolution of response pdf fxXt()(). Such pdf evolution equations are commonly called generalized Fokker-Planck-Kolmogorov (genFPK) equations, since they constitute the counterpart of the classical Fokker-Planck-Kolmogorov equation for RDEs under coloured noise excitation. The transformed SLE (35) is preferred over the original SLE (33) as a starting point for the formulation of genFPK equations, since: i) In transformed SLE (35) the mean effect of excitation is explicitly present in the drift coefficient (the additional term  mt()() which is hidden in the original SLE (33)),

13 ii) Inside the averaged terms of SLE (35), the derivatives of the response with respect to initial value X[();(;)]/() XX00 and excitation X[();(;)]/(;) Xs0   ap- pear. These derivatives can be expressed via the solution of the variational problem associated with the nonlinear RDE (31). Since the variational problem is always linear, the said derivatives can be easily expressed in the following closed form for any RDE; see e.g. (Mamis, Athanassoulis and Papadopoulos, 2018),

t XX[();(;)]0  t t 0   exph X ( u ; ) du , (36a)  X 0 () t 0 t XX[();(;)]0  t t 0   exph X ( u ; ) du , (36b) (;)s  s with the prime denoting the derivative of the function with respect to its argument. In the litera- ture, various current-time approximations have been proposed for the transformed SLE, the most widely-used being the small correlation time (Sancho et al., 1982), Hänggi‟s ansatz (Hänggi et al., 1985) and Fox‟s approximation (Fox, 1986). In our recent works (Athanassoulis, Kapelonis and Mamis, 2018; Mamis, Athanassoulis and Kapelonis, 2018), a new approximation scheme, based on random Volterra-Taylor expansion series, was employed for the closure of Eq. (35). This approximation led to a family of novel genFPK equations which, for the non-restrictive rep- N resentation h( xg )( ) xk  1 kk in Eq. (31), where  k ‟s are constants, g1 () xx  , model- ing the linear part, and gxk (), kN 2, , are nonlinear functions, reads

 fx() Xt()   h()()() x  m()() t fXt x  tx

2 M φ α x;{()} R t  efftt  eff    gk ()  2 D0 Rh ( )(),(),(), t D m R h ( ) t f X ( t ) x  x tt00    α! mm1 |α | (37)

  where M  , denoting the order of the approximation used, Rsh()() h X(;) s   ,  R( s )( ; ) g X s  are moments of the response, α (,,) is a multi-index, gk () k   2 N and

φ x;{()};(),,;() Rg()()() t2 x R g  tN x R g  t , (38a)  k    2   N  with ()()()x g x R t . (38b) k k k gk () 

eff t Last, coefficients Dmh[(),] R() t , called the (ordered) generalized effective noise intensi- t 0 ties, are

14 t eff t  m Dm[ R hh( )( ( )0 ), tR ] u exp  ( )(duCt ) X ()() t t t 0  0 t 0 (39) tt  2 expR ( u ) du C ( t , s ) ( t s )m ds . h()()() t 0 s

Eq. (37) has been solved numerically and validated via Monte Carlo simulations (Athanassoulis, Kapelonis and Mamis, 2018; Mamis, Athanassoulis and Kapelonis, 2018), for a wide range of values of the coloured noise correlation time (up to five times the relaxation time of the ODE (31)) and noise intensity. In all these cases, the results obtained by solving Eq. (37) are in excel- lent agreement with MC simulations both in the transient and in the long-time regime.

Remark 3. The formulation of genFPK equations for RDEs with coloured excitation correlated with the random initial value has not been derived before, to the best of our knowledge, and it is the topic of some recent works of ours (Athanassoulis, Kapelonis and Mamis, 2018; Mamis, Athanassoulis and Papadopoulos, 2018). Note that, it is a feature worth including for certain applications, as mentioned in (Hänggi, 1989, Sec. 9.4; Wio, Colet and San Miguel, 1989).

Remark 4. The observant reader has already pointed out the discrepancy between the require- ment, in NF theorem, for the RFF being C  with respect to its random arguments, and the use of random delta function, Eq. (34), as the said RFF . A rigorous treatment of this point requires the formulation of the results in the context of generalized random functions, see (Gelfand and Vilenkin, 1964) Chapter 3, and their proof by reduction to the dual space of C  spaces. This analysis involves Gelfand-Shilov spaces and will not be performed herein.

7. First example and validation: The exact response pdf evolution equation corresponding to a linear random differential equation

By considering h() xx   , Eq. (31) becomes the linear random differential equation:

X(;)(;)(;) t  X t     t  , X(;)() t00 X . (40a,b)

Since hx() is linear, the variational derivatives, Eqs. (36a,b), are independent from the re-

t  ()tt0 t  ()ts sponse; X[();(;)]/() XX00 e   , X[();(;)]/(;) Xs0  e    , t 0 t 0 and thus the transformed SLE (35) is, in fact, the exact response pdf evolution equation:

f()() x 2 f x X()() teff X t x  m()()()()() t fXt x  D t 2 , (41) t  x  x where the term Dteff (), called the effective noise intensity, is given by

t Deff()(,) t  e ()t  t 0 Ct( )  2 e (ts ) C t s ds , (42) X 0 ( )  ()( ) t 0 along with the initial Gaussian condition:

2 11()x  m X f(( x)) f x  exp  0 . (43) X() t00 X 2 2 2  2  X  X 0  0  where m ,  2 are the initial mean value and initial variance respectively. It is a well-known X 0 X 0 result that the response process of any linear system, with Gaussian initial distribution, to an ad- ditive Gaussian excitation (either coloured or white) is also a Gaussian process. Furthermore, the 2 mean value mtX ()() and variance  X ()()t of the response Xt( ;) can be determined as the solutions to the respective moment equations, derived directly from RDE (40) (see e.g. (Sun, 2006) ch. 8, or (Athanassoulis, Tsantili and Kapelonis, 2015)):

m( tm )( )( )t m t , m() t m , (44a,b) XX( )( )( )  X () 0 X 0 t  22(tt ) 2 ( ) 2C (t ) e (tt 0 ) 2 2) C t, u e (tu du , (45a) XX( ) ( )X 0  ( )( )  ( ) t 0 22()t  , (45b) X ( ) 0 X 0 with dot denoting the time differentiation. Note that Eq. (45a) can expressed equivalently, using Eq. (42), as

22eff  XX()((tt ) 2 ) ( )  2D (t ) . (45a′)

Solving initial value problems (44), (45), results into

t m( t )( ) m e ()tt 0  m  e(t  ) d  , (46) X ( )( ) X 0   t 0 and t 22()t e 2 (tt 0 ) 2 Deff() e 2 (t  ) d . (47) X ( ) X 0  t 0

Returning to Eq. (41), and following (Sun, 2006) sec. 6.3, this pdf evolution equation can be solved by employing the Fourier transform; fˆ (,)() y t ei y x f x dx , which leads to  Xt()

ˆˆ f(,)(,) y t f y t eff 2 ˆ y  imtyDtyfyt()()()(,)   , (48a) ty supplemented with the transformed initial condition

ˆ 1 2 2 f( y , t0 ) exp i mXX y y . (48b) 002

Initial value problem (48) is solved (Polyanin, Zaitsev and Moussiaux, 2001) sec. 4.1, by first determining the characteristic curve w(,) y t y e t as the solution of the characteristic equa-

16 t tion dt dy/() y . Then, we seek a solution of the form g( wh ) exp( w t , dt ) , where t 0 gw() is a function of the characteristic curve, to be defined by the initial condition (48b), and tteff 2 2 ˆ hwt(,)()() im () twe D twe , that is the coefficient multiplying fy(,) t in Eq. (48a), rewritten in terms of w , t . Finally, by returning to the original variables y , t , we obtain the solution t fˆ ( y , t ) exp( i ) m e ()tt 0  m  e()t d  y X 0  () t 0 (49) t 1 2 (tt ) exp2   (2 ) e0 Deff e 2 (t ) d y 2 2 X 0  t 0 which, by employing the inverse Fourier transform, and utilizing the formulae (46), (47) for 2 mtX ()() and  X ()()t , results in

2 11 x  m X ()()t  f (x ) exp , (50) Xt() 2 2 2 (t ) 2 X () (t ) X ()  which is the expected Gaussian distribution.

Remark 5. The uniqueness of Gaussian solution (50) is ensured by the injectivity of Fourier transform for absolutely integrable functions, and the uniqueness of solution for transformed problem (48a,b), see (Polyanin, Zaitsev and Moussiaux, 2001), Sec. 10.1.2.

Remark 6. To sum up, in this section, we have proven that response pdf evolution Eq. (41), corresponding to the linear RDE (40a,b), reproduces the expected Gaussian solution, Eq. (50), cor- rectly. This result also constitutes a first validation for the novel, extended NF theorem, Eq. (7), since the derivation of Eq. (41) is based solely on random delta formalism (32) and the extended NF theorem.

8. Generalization of Theorems 1 and 2 to the multidimensional case

In this section, we concisely present the generalizations of Theorems 1 and 2 for the case where 0 XX0 ()() is an N  dimensional random vector and Ξ(;) is a K  dimensional random function. Note that, in general, dimensions N and K do not necessarily agree. Since the proofs of the generalized theorems are similar to those of Theorems 1, 2, their details are omit- ted.

Theorem 3: Under appropriate smoothness assumptions, the mean value of the RFF G[();(;)]X 0 Ξ is expressed as

 0  t GG[();(;)];[;()]X Ξ   ˆˆ0 υ u X Ξ()t υm 0 i  υui () 0 X  um()() Ξ()

17 NKt  t  ˆ 0 ˆ (51) exp  Xdsnk ()(;) s [;( )] G υu t υm  0 X 0 nk11 nkus() um()() t 0 Ξ()

ˆ 00 ˆ where XXm( )( ) 0 , ΞΞm(sss ; ) ( ; )( ) are the fluctuations of the ran- X Ξ() dom elements X 0 () and Ξ(s ;) around their mean values, and  [;()]υu is the Xˆˆ0 Ξ() j.Ch.FF of the said fluctuations. ■

As in Theorem 1, Theorem 3 is proved by identifying the similarity between the average of the random shift operator ()[] for the multidimensional case, defined by TXˆˆ0 Ξ()

00tt t ˆˆ GG[();(;)][();()(;)]X Ξm  X m0  Ξ Ξ()  tt00 t 0 X

t TGˆˆ00()[;()] mmΞ() X ΞX() t 0

NKt ˆ 0  ˆ t expXnk ( )  ds  ( s ; )G [mm0 ;Ξ() ( )], (52)  X t 0 nk11 nkus() t 0 and the definition relation of j.Ch.FF

NKt [υu ; ( )]  expi Xˆ 0 (  )   i ˆ ( s ;  ) u ( s ) ds . (53) Xˆˆ0 Ξ() n n k k nk11 t 0

Eqs. (52), (53) can be obtained using Volterra‟s passing from the discrete to continuous, as has been done before for their scalar counterparts, Eqs. (14), (2) respectively.

Theorem 4: the extended Novikov-Furutsu (NF) theorem for the multidimensional case. t Under appropriate smoothness assumptions, for the RFF FF[();(;)][]X 0 Ξ  , t 0 with jointly Gaussian arguments X 0 () , Ξ(;) , the following formulae hold true   k (;)[]t  F N  (54)  F [] mt ()()[] F  Ct0 () 0  k  X n  k ()  X () n  1 n K t   F []  C(,). t s ds ■   k ()()  1  (;)s  t 0 

0 t Eq. (54) is proved by substituting GF[();(;)](;)[]X Ξ k t  in Eq. (51), and t 0 employing the j.Ch.FF of jointly Gaussian Xˆ 0 () , Ξˆ (;) :

18 Gauss t  ˆˆ0 [;()]υu  X Ξ() t 0

KK tt 1 exp  C() ( ) (,)( s12 s ukk()) s1 u s21 ds ds 2   kk12 12 2 kk1 1 (55) 12tt00 NN NKt 1  expexp   CC0 00 s u s ds ()() . XX n12 nk n X  ()  n12n  n k 2 nn11 nk 11 12 t 0

Eq. (55) can be obtained by using Volterra‟s approach. For this purpose, a procedure similar to the proof of Theorem 2 in Section 5 is followed, by defining and studying the appropriate operators T .

9. Summary and conclusions

In this work, an extension of the classical NF theorem for the case of RFF s with jointly Gauss- ian arguments has been presented (Theorem 2). In fact, the said extension of the NF theorem is based on a representation of the mean value of any RFF via the j.Ch.FF of its (possibly non- Gaussian) arguments (Theorem 1). The latter, more general, result is an alternative to the series expansions for averaged random functionals given by (Bochkov, Dubkov and Malakhov, 1977). Extensions of the two aforementioned theorems to the multidimensional case have also been formulated as Theorem 4 and Theorem 3, respectively. In all cases, the random arguments assumed to have arbitrary mean values, and they can be mutually dependent.

Apart from the extensions, the proof of NF theorem presented herein is significantly different from the ones presented by other authors, and it is easily generalizable. The original proof by Novikov (Novikov, 1965), found also in (Ishimaru, 1978 Appendix 20B; Konotop and Vazquez, 1994; Scott, 2013) lies in expanding the functional F [(;)]  into a Volterra-Taylor series with respect to (;) around zero, multiplying both sides by (;)t  and then taking the average. For the completion of the proof, Isserlis‟ theorem (Isserlis, 1918) for computing higher order moments in terms of the autocorrelation for the Gaussian case is invoked. Thus, Novikov‟s proof is essentially restricted to the Gaussian case. Our derivation is based on a generic result (Theorem 1), which is valid for any type of random arguments, and the specification of the characteristic (function-)functional as Gaussian, allowing for generalizations to other cases. In (Sobczyk, 1985; Klyatskin, 2005) the starting point of the proof is the functional shift operator in exponential form, as in the present work. However, a different operator formalism is used after- wards, making the proof less transparent, to the authors‟ opinion. Recall that all existing proofs refers to “pure” random functionals, while our results cover the case of random function- functionals, permitting the study of RDEs with random initial values, possibly correlated with coloured noise excitation(s).

19 Appendix A. Volterra’s principle of passing from discrete to continuous. Volterra calculus

A naïve, yet useful, way to describe Volterra‟s principle of passing from the discrete to continuous is the following:

Consider the discrete quantities  n , u n , nN 1(1) , obtained by sampling the continuous functions (),()t u t over the interval [,]ab:

n  (t n ),( ),[ u n , ] u t n t n a b .

Assume further that the points t n are ordered, a t12  ttt  b nN    , and, as

N , all increments tttnnn  1 , nN1(1)(1) , tta01 and t N bt N , tend to zero. Under this assumption, Volterra‟s principle of passing from the discrete to continuous consists of the replacement of the sums

N NNN   (tut ),( u  ) t (  ) nnn  nn n  n1 nnn  111 by NN (),()()s  t  s u s  t , (2) n n n n n nn11 and their interpretation (in the limit, as N ) as Riemann integrals, bb (),()()s d s s u s d s . aa This line of thought is especially useful for the probabilistic study of random functions (in continuous time), since the primitive probabilistic information associated with a random function (;)t  is the joint distribution (or the joint characteristic function) of the random variables

nn()(;)   t , where the time instances t12 t   tnN   t are distributed over the (common) domain of definition of path functions. For example, considering the joint characteristic function of the random variables X 0 () , (;)s1  , (;),s 2  ,  (;)s N  , N  ;us ( ), , us ( )  exp iX (  )   i  ( suss ;  ) ( )  , X01( s ) ( sN ) 1 N  0  n n n  1

(A1) and applying the above described Volterra's passing from the discrete to continuous, we obtain the j.Ch.FF of Xt0 (),(;) : t t   X ( )[  ;u ( )]P exp i X 0 (  )   i  ( s ;  ) u ( s ) ds . (A2) 0 t 0 X 0  ( )  t 0

2 ( ) sn [,] t n t n  1 . Any choice of s n leads to the same results, since the functions  (),()t u t are assumed to be continuous.

20 Another interesting application of Volterra's passing from the discrete to continuous is the con- struction of the j.Ch.FF of a jointly Gaussian scalar random variable and scalar random function, which is of fundamental importance for the proof of the extended NF theorem. In order to find this j.Ch.FF we begin from its discrete analogue, the joint characteristic function of a random variable X 0 () and a random vector Ξ() . This is obtained by simple manipulations of a (1N ) dimensional Gaussian characteristic function (see e.g. (Lukacs and Laha, 1964), Sec. 2.1), and reads as follows:

NN N Gauss 1 XnnΞΞΞΞ m ;u  expi m uu u C  0 nnm 2   nn11 m  1 (A3) N 1 2 expexp,i m XX C XX     C Ξ u n 002 00   n n  1 where m , C are the mean value and the variance of the random variable X () , m , X 0 XX00 0 Ξ C are the mean value and autocovariance of the random vector Ξ() , and C denotes their ΞΞ X 0 Ξ cross-covariance. By setting nn()(;)   t and unn u() t and considering the Volterra‟s limiting process as described above, we obtain the Gaussian j.Ch.FF

Gauss t X ()[;()]u  0 t 0 t t t 1 expi m ( s ) u ( s ) d s C(,)()() s s u s u s ds ds  () 2   ( )()  1 2 1 221 t0 t 0 t 0 t 1 2  expi mXX  CXX   exp.   C()() s) u( s ds (A4) 02 0 0  0 t 0

Gauss t In Eq. (A4), we observe that X ()[;()]u is the product of the Gaussian characteristic 0 t 0 functional of (;) , the Gaussian characteristic function of X 0 () , as well as a term encap- sulating their probabilistic dependence via their cross-correlation function Cs(). X 0 ()

Passing from discrete to continuous is only a minor trick of Volterra‟s approach. The vigor of the latter is amply revealed only when consider the corresponding functional calculus. Volterra functional derivative /()us is the continuous analogue of the (usual) partial derivative  /  u n , and leads to a well-structured calculus on functionals (Volterra, 1930; Volterra and Pérès, 1936; Averbukh and Smolyanov, 1967). An example illustrating the rigorousness of Volterra calculus, as well as the relation between Volterra functional derivative and the more widely known Gâ- teaux derivative, is the following derivation of the of Taylor‟s theorem for functionals, called the Volterra-Taylor theorem:

21 t Let us consider the functional G[u ()] , assumed to be M 1 times Gâteaux differentiable, t 0 with the goal of expanding it around u 0 (). By assuming u (), u 0 () as fixed, we define the tt real-valued function of one scalar variable guu( )(G )(0 ) , where tt00

uuu( )( )( ) 0 . Obviously, g () is well-defined and continuous in [0,1], has the properties

t t gu(0)[ G ()] 0 , and gu(1)[ ()]G , t 0 t 0 and it is M 1 times differentiable, in the usual sense, in the interval [0,1]. Its Taylor expansion, for g (1) around g (0) , reads as

M 1dmM gd ( )1( g ) 1 g (1) , 01 . (A5)  mM1 m  0 m!( dM 1)!   0 d 

t Rewriting Eq. (A5) in terms of the functional G[u ()] we obtain t 0

M m t1   t t GG[()]()()u u0  u  t0 m  t 0 t 0 m  0 m!    0 (A6) M 1 1  tt G uu 0 (  ) ( ) , 0 1. M 1 tt00 (M  1)!  

m  tt The terms G uu0 ()()  are identified as the Gâteaux functional deriva- m tt00    0 tives (Averbukh and Smolyanov, 1968), of G calculated at u 0 () in the direction of u (), de- m tt noted by G uu0 ();() . Thus, Eq. (A6) can be written as tt00

M t1 m  t t GG[()]();()u u0 u t0  t 0 t 0 m  0 m! (A7) M 1 1  tt G uu0 ( )  ( ) , 0   1, M 1 tt00 (M  1)!  

Eq. (A7) is the Gâteaux-Taylor series expansion for functionals. Under some technical assumptions (see (Hall, 1978; Hamilton, 1980)), it can be proved that Gâteaux derivatives are expressed in terms of Volterra derivatives by formulae

t t G[()]u 0 tt t 0 G u0 ();()() u u s ds , tt00 us() t 0 for the first derivative and, in general,

ttm t ()m G[()]u 0 m tt t 0 G u01( ) ; uu ( s 1)( ) u s ( ds ) ds mm tt00 u()() s1 u s m tt00 for the mth derivative. Using the above equation, the Gâteaux-Taylor expansion, Eq. (A7), is rewritten as a Volterra-Taylor expansion in the form

ttm t M ()m G[()]u 0 t 1 t 0 G[uu ( s ) ]( u) s ds( ) ds 11mm t 0   m  0 m!()() u s1 u s m (A8) tt00  Residual term .

t By considering now that G[u ()] is infinitely Gâteaux-differentiable, and the functional se- t 0 ries in the right-hand side of Eq. (A8) converges, we obtain the infinite Volterra-Taylor series expansion of the functional,

m t  tt ()m G[()]u 0 t 1 t 0 G[uu ( s )]( ) u s ds( ) ds 11mm. (A9) t 0   m  0 m!()() u s1 u s m tt00

The structure of Eq. (A9) is the same with the Taylor expansion for a function G ()u of an N  dimensional vector u around u 0 :

 1 NN()m  m G ()u Gu()u  u 0 , (A10) m!    u u nn1 m m0 n1  1 n m  1 nn1 m

where  uuun n0 n . It is easy to see that Eq. (A9) can be also obtained from Eq. (A10) by applying the process of passing from the discrete to continuous, as described in the beginning of this section. The above analysis gives a clear indication of the validity of Volterra‟s principle (under some technical assumptions) and, in addition, establishes the fact the Volterra functional derivative is the continuous analogue of the (discrete) partial derivative.

Appendix B. Proofs of Lemmata 1-6

As the proofs in (Jia, Tang and Kempf, 2017), where integration by differentiation is also performed, Lemmata 1-6 are proven using operators T in series form, which are obtained by expanding the exponential in the right-hand sides of Eqs. (18a,b,c) of the main paper:

 2 p 11 p  T ˆˆ  C XX , (B1a) XX00  pp00 2 p  0 p!2 

T ˆ ˆ  X 0 () tt  1 ()p  pp (B1b)  C()() s(1) C s (pp )ds (1) ds ( ) ,  XX00()() pp(1) ( ) p  0 pu!( s)() u s tt00

23 tt  11 (2p ) T C(,)(,) s(1) s (1) C s (pp ) s ( ) ˆˆ()()  p  ()()1   2 ()()1 2 p  0 p!2 tt00 (B1c) 2 p  (1) (1) (pp ) ( )  (1) (1) (pp ) ( ) ds1 ds 2 ds 1 ds 2 . u()()()() s1 u s 2 u s 1 u s 2 

The proofs of Lemmata 1-3 are based on the series expansions given above, in conjunction with the linearity of integrals and derivatives.

Proof of Lemma 1: Operators T are linear. The action of operator T ˆ ˆ on X 0 () tt GF[  ;uu (  ) ][ ; ( ) ] is expressed via Eq. (B1b) as tt00

[;()][;()] uutt   TGFXˆ ˆ () tt 0 00

tt pp tt  ()p [;()][;( uu   )] GFtt 1 (1) (pp )(1) ( ) 00  CXX()()() s C ()s ds ds .  00 pp(1) ( ) p  0 pu!( ) u ( s )  s tt00

Employing the linearity of derivatives and integrals, in conjunction with the assumption that  ,

 are independent from the differentiation arguments  and u () of operator T ˆ ˆ , each X 0 () term of the right-hand side of the above equation is linearly decomposed, resulting in the linearity of the operator T ˆ ˆ . Thus, Lemma 1 is proved for operator T ˆ ˆ . The proofs for X 0 () X 0 ()

T ˆˆ, T ˆˆ are similar. ■ XX00 ()()

Proof of Lemma 2: Operators T commute with   and us() differentiations. Using Eq. (B1b), we may write

 t T ˆ ˆ G[;()] u   X 0 () t 0 tt pp t  ()p  G[;()] u  1 (1) (pp )t 0 (1) ( )  C ()()sCs ds ds  XX00()() pp(1) ( ) p  0 p!(u s)() u s tt00 using the linearity and continuity of the derivative tt pp1 t  ()p  G[;()] u 1 (1) (pp )t 0 (1) ( )  C()( s C s ) ds ds  XX00()() pp1 (1) ( ) p  0 p!()() u s u s tt00 which is identified, via Eq. (B1b), as the infinite series of t G[;()] u t 0  T ˆ ˆ . X 0 ()  

24 t Proof of the lemma for T ˆ ˆ G[;()] u is similar, as it is also for the other two us()X 0 () t 0 operators T ˆˆ, T ˆˆ. ■ XX00 ()()

Proof of Lemma 3: Operators T commute with each other. For the sake of simplicity, we shall prove the commutativity of operators T ˆ ˆ and T ˆˆ. We begin from X 0 () XX00

t TTˆˆˆ ˆ G[;()] u  XX00 X() 0 t 0 by using Eqs. (B1a,b) tt  1 ()p Cs( (1))()Cs (p )  XX00()() p  0 p! tt00

2m t pp   [;()] u  11 G t C m 0 ds(1) ds (p ) ppus()()!(1) mmus ( )2  m 2 XX00 m  0 rearranging the order of summations and using the linearity of derivatives and integrals  11  C m   m XX00 m  0 m!2

tt pp t 2m  ()p  [;()] u  1 G t  C()() s(1) C s (pp )0 ds (1) ds ( ) 2(1)mp  ( ) XX p00()() p  0 pu!( ) ( ) s u s  tt00 

which, by Eqs. (B1a,b), is identified as

t  TTˆ ˆ ˆ ˆ G[;()] u . ■ XXX0 0 0 () t 0

In Lemmata 4-6, the actions of operators T on the product u()[;()] tu F  are determined.

For the actions of T ˆ ˆ T ˆˆ, the following product rule for Volterra derivatives is also X 0 () ()() needed

k t k t u()[;()] tF  u F [;()] u t 0 t ut() 0 u()()()() s(1) u s (kk ) u s (1) u s ( )

k 1 t (B2) k [;()] u F t   ()t  s ()n 0 ,  k () n  1   1 us()  n

k( ) (1) ( n 1) ( n 1) ( k ) with   1 u()()()()() s u s u s u s u s . Eq. (B2) is easily proven  n via mathematical induction on index k , commencing from the relation for k  1 (product rule for the first-order derivative):

u()[;()] t u t t F t F [;()] u 0 t 0 ut() t u( tu )[ ; ( )] F  , u( su )( s ) u ( s ) t 0 in which u()/( t u s)   (t  s). Having formulated Eq. (B2), we move on to the proofs of Lemmata 4-6.

t Proof of Lemma 4: The action of operator T ˆˆ on u( tu )F [ ; ()] is given by XX00 t 0

tt    TFTFˆ ˆu()[;()]()[;()] t u u t ˆ ˆ u . (B3) XXXX0 0tt00  0 0  

Eq. (B3) holds true due to the linearity of T ˆˆ, since the factor ut() is independent from  , XX00 which is the differentiation argument of T ˆˆ. ■ XX00

Proof of Lemma 5. By using the series expansion of Eq. (B1b), and employing the product rule, t Eq. (B2), the action of operator T ˆ ˆ on u( tu )F [ ; ()] is expressed as X 0 () t 0

t TFˆ ˆ u()[;()]() t u u t  C X ()()t , (B4) X 0 ()t 0 0 where  ,  are

tt pp t  ()p  F [;()] u 1 (1) (pp )t 0 (1) ( )  C()()s C s ds ds ,  XX00()() pp(1) ( ) p  0 p!()() u s u s tt00

tt p p1 t  p (p 1)  [;()] u 1 F t    pC ()sds( m )( ) 0  p m . m1 Xm0  ( )1 pp () pu!(m nm n   s ) pn11tt m  1 00 mn

By the symmetry of integration arguments, all terms of the n  sum are equal, thus,

tt p p 1 t  (p  1)  [;()] u 1 F t  Cs()s(1)C () (1)pp0 ds (1) ds (1) .  XX00 ()() pp(1) ( 1) p  1 (1p)!()( u su s ) t 00t

In view of Eq. (B1b), series  is identified as

t TFˆ ˆ [;()] u . (B5) X 0 ()t 0

For series  , we perform the index change kp1, resulting in

tt t  ()k k k  [;()] u 1  F t  C ()()s(1)C s (kk )0 ds (1) ds ( )  XX00()() kk(1) ( ) k  0 k!()()uuss tt00  (B6)

26 Via Eq. (B1b), the right-hand side of Eq. (B6) is identified as

t  F [;()] u t 0 T ˆ ˆ . (B7) X 0 () 

Substitution of Eqs. (B5), (B7) into Eq. (B4) results in Eq. (25) of the main paper for t t F [ ;u ()]  F1 [;()] u , completing thus the proof of Lemma 5 ■ t 0 t 0

t Proof of Lemma 6. By expanding TFˆˆ u()[;()] tu  in series using Eq. (B1c), ()() t 0 and employing the product rule, Eq. (B2), we obtain

t TFˆˆ u()[;()]() tuu t      , (B8) ()() t 0 where  is

tt  11 (2p )   C(,)(,) s(1) s (1) C s (pp ) s ( )  p  ()()1  2  ()()1  2 p  0 p!2 tt00

2 p [;()] u t  F t  0 ds(1) ds (1) ds (pp ) ds ( )  , u()()()() s(1) u s (1) u s (pp ) u s ( ) 1 2 1 2  1 2 1 2  while  can be written as follows, after taking into account the symmetry of autocorrelation function Cs( ) s ( )(,) 1 2 :

tt  112 p (2p 1)    C(,)(,) t s()()()n p C s m s m p  ()()  2m  1  ()()1  2 pn11p!2  mn tt00

21p [;()] u t  F t  0 ds()()()n p ds m ds m . ()()()n p m m 2m  1 1 2  u()()() s2 m  1 u s 1 u s 2 mn mn 

By virtue of Eq. (B1c), series  is identified as

t TFˆˆ [;()] u . (B9) ()() t 0

We turn now our attention to the series  . By performing, in each term of the n  sum 2 p , the change of integration variables:  n  1 ()n ()()m ( ) (m 1) ss 2 and ssii for mn , ssii for mn , i  1, 2 , we obtain:

p1 ( ) ( ) p ( m ) ( m ) 1()()1C  (,)(,) s s 2 m  1 C  ()()1  s s 2 , mn

27 pp1 ( ) m ( )( m ) ( ) 1u( s 1 ) u ( su 21 )(s 1m ) u s ( 2 ) , mn pp1 ( ) m ( )( m ) ( ) 1ds 1 dsds 21 1 dsm 2 . mn Now, it is easy to see that all 2 p terms in n  sum are equal, resulting in

tt  11 ( 2p  1)  C(,)(,)(,) t s C s(1) s (1) C s (1)pp s (1)   p 1  ()()   ()()1  2  ()()1  2 p  1 (p  1)! 2 tt00

21p  [;()] u t  F t  0 ds ds(1) ds (1) ds (1)pp ds (1) . u()()()()() s u s(1) u s (1) u s (1)pp u s (1) 1 2 1 2  1 2 1 2  Further, we perform the index change kp1 and interchange s  integration with summation, resulting in

t t t  11 (2k )   C(,)(,)(,) t s C s(1) s (1) C s (kk ) s ( ) ()()  k     ()()12   ()()12  k  0 k!2 t0 t 0 t 0

t 2k [;()] u  F t  0 ds(1) ds (1) ds (kk ) ds ( )  ds. u()()()()() s(1) u s (1) u s (kk ) u s ( )  u s 1 2 1 2  1 2 1 2   (B10)

The sum in the right-hand side of Eq. (B10) is identified, via Eq. (B1c), as

t [;()] u t F t  C(,) t sdsT 0 . (B11)  ()() ˆˆ()() us() t 0 

By substituting Eqs. (B9), (B11) into Eq. (B8), we obtain Eq. (28) of the main paper for t t F [;()] u  F2 [;()] u . Thus, proof of Lemma 6 is completed. ■ t 0 t 0

References

Athanassoulis, G. A., Kapelonis, Z. G. and Mamis, K. I. (2018) „Numerical solution of generalized FPK equations corresponding to random differential equations under colored noise excitation. The transient case‟, in 8th Conference on Computational Stochastic Mechanics. Paros, Greece.

Athanassoulis, G. A., Tsantili, I. C. and Kapelonis, Z. G. (2015) „Beyond the Markovian assumption: response- excitation probabilistic solution to random nonlinear differential equations in the long time‟, Proc.R.Soc.A, 471:(20150501).

Averbukh, V. I. and Smolyanov, O. G. (1967) „The theory of differentiation in linear topological spaces‟, Russian Mathematical Surveys, 22(6), pp. 201–258.

Averbukh, V. I. and Smolyanov, O. G. (1968) „The various definitions of the derivative in linear topological spaces‟, Russian Mathematical Surveys, 23(4), pp. 67–113.

Bobryk, R. V. (1993) „Singular perturbation method for short waves in random media‟, Waves in Random Media,

28 3(4), pp. 267–277. doi: 10.1088/0959-7174/3/4/003.

Bochkov, G. N., Dubkov, A. A. and Malakhov, A. N. (1977) „Structure of the correlation dependence of nonlinear stochastic functionals‟, Radiophys. Quantum Electron., 20, pp. 276–280. doi: 10.1007/ BF01039470.

Cáceres, M. O. (2017) Non-equilibrium Statistical Physics with Application to Disordered Systems. Springer.

Cetto, A. M. and de la Peña, L. (1984) „Generalized Fokker-Planck Equations for Coloured, Multiplicative, Gaussian Noise‟, Revista Mexicana de Física, 31(1), pp. 83–101.

Cook, I. (1978) „Application of the Novikov-Furutsu theorem to the random acceleration problem‟, Plasma Physics, 20(4), pp. 349–359. doi: 10.1088/0032-1028/20/4/006.

Creamer, D. B. (2008) „On closure schemes for polynomial chaos expansions of stochastic differential equations‟, Waves in Random and Complex Media, 18(2), pp. 197–218. doi: 10.1080/17455030701639701.

Donsker, M. D. and Lions, J. L. (1962) „Fréchet-Volterra variational equations, boundary value problems, and function space integrals‟, Acta Mathematica, 108, pp. 147–228.

Fox, R. F. (1986) „Functional-calculus approach to stochastic differential equations‟, Physical Review A, 33(1), pp. 467–476.

Fox, R. F. (1987) „Stochastic calculus in physics‟, Journal of Statistical Physics, 46(5–6), pp. 1145–1157. doi: 10.1007/BF01011160.

Furutsu, K. (1963) „On the Statistical Theory of Electromagnetic Waves in a Fluctuating Medium (I)‟, Journal of research of the National Bureau of Standards-D. Radio Propagation, 67(D), pp. 303–323.

Gelfand, I. M. and Vilenkin, N. Y. (1964) Generalized Functions, Volume 4: Applications of Harmonic Analysis. AMS Chelsea Publishing.

Glaeske, H.-J., Prudnikov, A. P. and Skòrnik, K. A. (2006) Operational Calculus and Related Topics. Taylor & Francis Group, LLC.

Hall, R. L. (1978) „On certain functional derivatives‟, Journal of Optimization Theory and Applications, 26(3), pp. 373–378.

Hamilton, E. P. (1980) „A new definition of variational derivative‟, Bull. Austral. Math. Soc., 22, pp. 205–210.

Hänggi, P. (1978) „Correlation functions and Masterequations of generalized (non-Markovian) Langevin equations‟, Zeitschrift fur Physik B, 31, pp. 407–416.

Hänggi, P. et al. (1985) „Bistability driven by colored noise: Theory and experiment‟, Physical Review A, 32(1), pp. 695–698. doi: 10.1103/PhysRevA.32.695.

Hänggi, P. (1989) „Colored noise in continuous dynamical systems: a functional calculus approach‟, in Moss, F. and McClintock, P. V. E. (eds) Noise in nonlinear dynamical systems, vol. 1: Theory of continuous Fokker-Planck systems. Cambridge University Press, pp. 307–328.

Hänggi, P. and Jung, P. (1995) „Colored Noise in Dynamical Systems‟, Advances in Chemical Physics, 89, pp. 239– 326.

Hyland, K. E., McKee, S. and Reeks, M. W. (1999) „Derivation of a pdf kinetic equation for the transport of particles in turbulent flows‟, J. Phys. A: Math. Gen., 32, pp. 6169–6190.

Ishimaru, A. (1978) Wave Propagation and Scattering in Random Media (Volume 2). Academic Press. doi: 10.1109/5.104210.

Isserlis, L. (1918) „On a formula for the product-moment coefficient of any order of a normal frequency distribution in any number of variables‟, Biometrika, 12(1–2), pp. 134–139.

Jia, D., Tang, E. and Kempf, A. (2017) „Integration by differentiation: new proofs, methods and examples‟, Journal of Physics A: Mathematical and Theoretical. IOP Publishing, 50, p. 235201. doi: 10.1088/1751-8121/aa6f32.

Kempf, A., Jackson, D. M. and Morales, A. H. (2014) „New Dirac delta function based methods with applications to perturbative expansions in quantum field theory‟, Journal of Physics A: Mathematical and Theoretical. IOP Publishing, 47, p. 415204. doi: 10.1088/1751-8113/47/41/415204.

Kempf, A., Jackson, D. M. and Morales, A. H. (2015) „How to (path-) integrate by differentiating‟, Journal of Physics: Conference Series, 626, p. 012015. doi: 10.1088/1742-6596/626/1/012015.

Kliemann, W. and Sri Namachchivaya, N. (eds) (2018) Nonlinear Dynamics and Stochastic Mechanics. (reissued). CRC Press.

Klyatskin, V. I. (2005) Stochastic Equations through the eye of the Physicist. Elsevier.

Klyatskin, V. I. (2015) Stochastic Equations: Theory and Applications in Acoustics, Hydrodynamics, Magnetohydrodynamics, and Radiophysics, Volume 1. Springer.

Konotop, V. V. and Vazquez, L. (1994) Nonlinear Random Waves. World Scientific.

Krommes, J. A. (2015) „A tutorial introduction to the statistical theory of turbulent plasmas, a half-century after Kadomtsev‟s Plasma Turbulence and the resonance-broadening theory of Dupree and Weinstock‟, Journal of Plasma Physics, 81(6). doi: 10.1017/S0022377815000756.

Lukacs, E. and Laha, R. G. (1964) Applications of characteristic functions. London: C. Griffin.

Mamis, K. I., Athanassoulis, G. A. and Kapelonis, Z. G. (2018) „A systematic path to non-Markovian dynamics: New generalized FPK equations for dynamical systems under coloured noise with arbitrary correlation function‟, (submitted). Available at: https://arxiv.org/abs/1811.12383.

Mamis, K. I., Athanassoulis, G. A. and Papadopoulos, K. E. (2018) „Generalized FPK equations corresponding to systems of nonlinear random differential equations excited by colored noise. Revisitation and new directions‟, Procedia Computer Science, 136(C), pp. 164–173.

Martins Afonso, M., Mazzino, A. and Gama, S. (2016) „Combined role of molecular diffusion, mean streaming and helicity in the eddy diffusivity of short-correlated random flows‟, Journal of Statistical Mechanics: Theory and Experiment, 2016(10). doi: 10.1088/1742-5468/2016/10/103205.

Novikov, E. A. (1965) „Functionals and the Random Force method in Turbulence theory‟, Soviet Physics Jetp, 20(5), pp. 1290–1294.

Peacock-López, E., West, B. J. and Lindenberg, K. (1988) „Relations among effective Fokker-Planck for systems driven by colored noise‟, Physical Review A, 37(9), pp. 3530–3535.

Polyanin, A. D., Zaitsev, V. F. and Moussiaux, A. (2001) Handbook of First Order Partial Differential Equations. Taylor & Francis.

Protopopescu, V. (1983) Advances in Nuclear Science and Technology 15. USA. Edited by J. Lewins and M. Martin Becker. Springer.

Rino, C. L. (1991) „On propagation in continuous random media‟, Waves in Random Media, 1(1), pp. 59–72. doi: 10.1088/0959-7174/1/1/006.

Sancho, J. M. et al. (1982) „Analytical and numerical studies of multiplicative noise‟, Physical Review A, 26(3), pp. 1589–1609.

Sancho, J. M. and San Miguel, M. (1980) „External non-white noise and nonequilibrium phase transitions‟, Zeitschrift fur Physik B, 36, pp. 357–364.

Scott, M. (2013) Applied Stochastic Processes in Science and Engineering. University of Waterloo.

Shrimpton, J. S., Haeri, S. and Scott, J. S. (2014) Statistical Treatment of Turbulent Polydisperse Particle Systems. Springer.

Sobczyk, K. (1985) Stochastic Wave Propagation. Elsevier.

Sun, J. Q. (2006) Stochastic Dynamics and Control. Elsevier.

Venturi, D. et al. (2012) „A computable evolution equation for the joint response-excitation probability density function of stochastic dynamical systems‟, Proc.R.Soc.A, 468, pp. 759–783.

Volterra, V. (1930) Theory of Functionals and of Integral and Integro-differential Equations. Phoenix. Dover Publications (2005 reprint).

30 Volterra, V. and Pérès, J. (1936) Théorie Générale des Fonctionnelles, tome premier. Paris: Gauthier-Villars.

Wang, P., Tartakovsky, A. M. and Tartakovsky, D. M. (2013) „Probability density function method for langevin equations with colored noise‟, Physical Review Letters, 110(140602), pp. 1–4. doi: 10.1103/PhysRevLett.110.140602.

Wio, H. S., Colet, P. and San Miguel, M. (1989) „Path-integral formulation for stochastic processes driven by colored noise‟, Physical Review A, 40(12), pp. 7312–7324.