Master Equations and the Theory of Stochastic Path Integrals

Review Article

Master equations and the theory of stochastic path integrals

Markus F. Weber and Erwin Frey Arnold Sommerfeld Center for Theoretical Physics and Center for NanoScience, Department of Physics, Ludwig-Maximilians-Universität München, Theresienstraße 37, 80333 München, Germany. E-mail: [email protected]

Abstract. This review provides a pedagogic and self-contained introduction to master equations and to their representation by path integrals. Since the 1930s, master equations have served as a fundamental tool to understand the role of fluctuations in complex biological, chemical, and physical systems. Despite their simple appearance, analyses of masters equations most often rely on low- noise approximations such as the Kramers-Moyal or the system size expansion, or require ad-hoc closure schemes for the derivation of low-order moment equations. We focus on numerical and analytical methods going beyond the low-noise limit and provide a unified framework for the study of master equations. After deriving the forward and backward master equations from the Chapman-Kolmogorov equation, we show how the two master equations can be cast into either of four linear partial differential equations (PDEs). Three of these PDEs are discussed in detail. The first PDE governs the time evolution of a generalized probability generating function whose basis depends on the stochastic process under consideration. Spectral methods, WKB approximations, and a variational approach have been proposed for the analysis of the PDE. The second PDE is novel and is obeyed by a distribution that is marginalized over an initial state. It proves useful for the computation of mean extinction times. The third PDE describes the time evolution of a “generating functional”, which generalizes the so-called Poisson representation. Subsequently, the solutions of the PDEs are expressed in terms of two path integrals: a “forward” and a “backward” path integral. Combined with inverse transformations, one obtains two distinct path integral representations of the conditional probability distribution solving the master equations. We exemplify both path integrals in analysing elementary chemical reactions. Moreover, we show how a well-known path integral representation of averaged observables can be recovered from them. Upon expanding the forward and the backward path integrals around stationary paths, we then discuss and extend a recent method for the computation of rare event probabilities. Besides, we also derive path integral representations for processes with continuous state spaces whose forward and backward master equations admit Kramers-Moyal expansions. A truncation of the backward expansion at the level of a diffusion approximation recovers a classic path integral representation of the (backward) Fokker-Planck equation. One can rewrite this path integral in terms of an Onsager-Machlup function and, for purely diffusive Brownian motion, it simplifies to the path integral of Wiener. To make this review accessible to a broad community, we have used the language of probability theory rather than quantum

arXiv:1609.02849v2 [cond-mat.stat-mech] 2 Apr 2017 (field) theory and do not assume any knowledge of the latter. The probabilistic structures underpinning various technical concepts, such as coherent states, the Doi-shift, and normal-ordered observables, are thereby made explicit. Master equations and the theory of stochastic path integrals 2

PACS numbers: 02.50.-r, 02.50.Ey, 02.50.Ga, 02.70.Hm, 05.10.Gg, 05.10.Cc, 05.40.-a

Keywords: Stochastic processes, Markov processes, master equations, path integrals, path summation, spectral analysis, rare event probabilities

Preprint of the article: M. F. Weber and E. Frey. Master equations and the theory of stochastic path integrals. Rep. Prog. Phys., 80(4):046601, 2017. doi:10.1088/1361-6633/aa5ae2. CONTENTS 3

Contents

1 Introduction 5 1.1 Scope of this review ...... 5 1.2 Organization of this review ...... 6 1.3 Continuous-time Markov processes and the forward and backward master equations ...... 8 1.4 Analytical and numerical methods for the solution of master equations 12 1.5 History of stochastic path integrals ...... 15 1.6 Résumé ...... 16

2 The probability generating function 17 2.1 Flow of the generating function ...... 19 2.2 Bases for particular stochastic processes ...... 21 2.2.1 Random walks...... 21 2.2.2 Chemical reactions...... 22 2.2.3 Intermezzo: The unit vector basis...... 24 2.2.4 Processes with locally excluding particles...... 25 2.3 Methods for the analysis of the generating function’s flow equation . . 27 2.3.1 A spectral method for the computation of stationary distributions. 27 2.3.2 WKB approximations and related approaches...... 28 2.3.3 A variational method...... 29 2.4 Résumé ...... 29

3 The marginalized distribution and the probability generating functional 30 3.1 Flow of the marginalized distribution ...... 30 3.2 Bases for particular stochastic processes ...... 31 3.2.1 Random walks...... 32 3.2.2 Chemical reactions...... 32 3.3 Mean extinction times ...... 33 3.4 Flow of the generating functional ...... 34 3.5 The Poisson representation ...... 35 3.6 Résumé ...... 36

4 The backward path integral representation 37 4.1 Derivation ...... 38 4.2 Simple growth and linear decay ...... 39 4.3 Feynman-Kac formula for jump processes ...... 39 4.4 Intermezzo: The backward Kramers-Moyal expansion ...... 40 4.4.1 Processes with continuous state spaces...... 41 4.4.2 Path integral representation of the backward Kramers-Moyal expansion...... 41 4.4.3 Path integral representation of the backward Fokker-Planck equation...... 42 4.4.4 The Onsager-Machlup function...... 42 4.4.5 Alternative discretization schemes...... 43 4.4.6 Wiener’s path integral...... 43 4.5 Further exact solutions and perturbation expansions ...... 43 4.5.1 Pair generation...... 44 4.5.2 Diffusion on networks...... 44 4.5.3 Diffusion in continuous space...... 46 4.5.4 Diffusion and decay...... 46 4.6 Résumé ...... 48

5 The forward path integral representation 48 5.1 Derivation ...... 48 5.2 Linear processes ...... 49 5.3 Intermezzo: The forward Kramers-Moyal expansion ...... 49 5.4 Résumé ...... 50

6 Path integral representation of averaged observables 50 6.1 Derivation ...... 51 6.2 Intermezzo: Alternative derivation based on coherent states ...... 52 6.3 Perturbation expansions ...... 53 6.4 Coagulation ...... 54 6.5 Résumé ...... 55

7 Stationary paths 55 7.1 Forward path integral approach ...... 56 7.2 Binary annihilation ...... 57 7.3 Backward path integral approach ...... 58 7.4 Binary annihilation ...... 60 7.4.1 Leading order...... 60 7.4.2 Beyond leading order...... 60 7.5 Résumé ...... 61

8 Summary and outlook 62

A Proof of the Feynman-Kac formula in section 1.3 64

B Proof of the path summation representation in section 1.4 65

C Solution of the random walk in sections 2.2.1 and 3.2.1 65

D Details on the evaluation of the backward path integral representa- tion in section 4.3 65 Master equations and the theory of stochastic path integrals 5

1. Introduction gene expression [39]. All of these approaches can be treated within a unified framework. Knowledge 1.1. Scope of this review about this common framework makes it possible to The theory of continuous-time Markov processes is systematically search for new ways of solving the largely built on two equations: the Fokker-Planck [1–4] master equation. and the master equation [4,5]. Both equations assume Before outlining the organization of this review, that the future of a system depends only on its let us note that by focusing on the above path current state, memories of its past having been wiped integral representations of master and Fokker-Planck out by randomizing forces. This Markov assumption equations, we neglect several other “stochastic” or is sufficient to derive either of the two equations. “statistical” path integrals that have been developed. Whereas the Fokker-Planck equation describes systems These include Edwards path integral approach to that evolve continuously from one state to another, the turbulence [40, 41], a path integral representation master equation models systems that perform jumps in of Haken [42], path integral representations of non- state space. Markov processes [43–58] and of polymers [59–62], Path integral representations of the master and a representation of “hybrid” processes [63–65]. equation were first derived around 1980 [6–15], shortly The dynamics of these stochastic hybrid processes are after such representations had been derived for the piecewise-deterministic. Moreover, we do not discuss Fokker-Planck equation [16–19]. Both approaches were the application of renormalization group techniques, heavily influenced by quantum theory, introducing despite their significant importance. Excellent texts such concepts as the Fock space [20] with its “bras” exploring these techniques in the context of non- and “kets” [21], coherent states [22–24], and “normal- equilibrium critical phenomena [66–68] are provided by ordering” [25] into non-equilibrium theory. Some the review of Täuber, Howard, and Vollmayr-Lee [69] of these concepts are now well established and the as well as the book by Täuber [70]. Our main interest original “bosonic” path integral representation has been lies in a mathematical framework unifying the different complemented with a “fermionic” counterpart [26– approaches from the previous paragraph and in two 31]. Nevertheless, we feel that the theory of these path integrals that are based on this framework. Both “stochastic” path integrals may benefit from a step back of these path integrals provide exact representations and a closer look at the probabilistic structures behind of the conditional probability distribution solving the the integrals. Therefore, the objects imported from master equation. We exemplify the use of the path quantum theory make place for their counterparts from integrals for elementary processes, which we choose for probability theory in this review. For example, the their pedagogic value. Most of these processes do not coherent states give way to the Poisson distribution. involve spatial degrees of freedom but the application Moreover, we use the bras and kets as particular basis of the presented methods to processes on spatial functionals and functions whose choice depends on the lattices or networks is straightforward. A process stochastic process at hand (a functional maps functions with diffusion and linear decay serves as an example to numbers). Upon choosing the basis functions as of how path integrals can be evaluated perturbatively Poisson distributions, one can thereby recover both using Feynman diagrams. The particles’ linear decay a classic path integral representation of averaged is treated as a perturbation to their free diffusion. observables as well as the Poisson representation of The procedure readily extends to more complex Gardiner and Chaturvedi [32, 33]. The framework processes. Moreover, we show how the two path presented in this review integrates a variety of different integrals can be used for the computation of rare event approaches to the master equation. Besides the probabilities. Let us emphasize that we only consider Poisson representation, these approaches include a Markov processes obeying the Chapman-Kolmogorov spectral method for the computation of stationary equation and associated master equations [4]. It probability distributions [34], WKB approximations may be interesting to extend the discussed methods and other “semi-classical” methods for the computation to “generalized” or “physical” master equations with of rare event probabilities [35–38], and a variational memory kernels [71–74]. approach that was proposed in the context of stochastic Master equations and the theory of stochastic path integrals 6

Path integral representation via the Linear PDEs of the… forward Kramers-Moyal expansion (258) Generating function (45)

Path summation ⎜g(τ ⎜⋅)〉 = ⎜n〉τ p(τ, n ⎜⋅) 5.1 representation (19) n 5.3 2.1 2.3 Spectral method of Walczak et al. for 1.4 stationary distributions [34] Forward path integral Forward master Spectral formulation of Assaf and representation (247)-(249) t) equation (10) Meerson [36, 37] − ⎜ 〈 ⎜ ⎜ 〉 p(t, n t0, n0) = n t e n0 t0 ∂τ p(τ, n ⎜t0, n0) = … Variational method of Eyink [83] and [t0 1.3 of Sasai and Wolynes [39] 7.1 3.4 Path integral Chapman-Kolmogorov 〈 p(⋅ ⎜τ)⎜ = p(⋅ ⎜τ, n0) 〈n0 ⎜τ in (149) Rare event theory of representation for equation (6) n0 Elgart and Kamenev [35], observables (275) t] † and its extension − 〈 〉 〈〈 〉〉 3.4 Generating functional (143) A(n) = e A x(t) (t0 〈 ⎜ ⎜ 〈 ⎜ ⎜ 1.3 g(τ ⋅) = n τ p(τ, n ⋅) 7.3 n Backward master 3.5 6.1 equation (12) Poisson representation (152) [32, 33] Backward path integral ∂−τ p(t, n ⎜τ, n0) = … representation (170)-(172) t] − † p(t, n ⎜t0, n0) = 〈n0 ⎜t e ⎜n〉t 4.4 3.1 Marginalized distribution (110) 0 (t0 ⎜p(⋅ ⎜τ)〉 = p(⋅ ⎜τ, n0) ⎜n0〉τ n0 4.1 3.3

Path integral representation via the Mean extinction times ⎧ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎩ backward Kramers-Moyal expansion (188) Perturbation theory and renormalization

Figure 1. Roadmap and summary of the methods considered in this review. The arrows represent possible routes for derivations. Labelled arrows represent derivations that are explicitly treated in the respective sections. For example, the forward and backward master equations are derived from the Chapman-Kolmogorov equation in section 1.3. In this section, we also discuss the path summation representation (19) of the conditional probability distribution p(τ, n|t0, n0). This representation can be derived by examining the stochastic simulation algorithm (SSA) of Gillespie [75–78] or by performing a Laplace transformation of the forward master equation (10) (cf. appendix B). In sections 2 and 3, the forward and the backward master equations are cast into four linear PDEs, also called “flow equations”. These equations are obeyed by a probability generating function, a probability generating functional, a marginalized distribution, and a further series expansion. The flow equations can be solved in terms of a forward and a backward path integral as shown in sections 4 and 5. Upon performing inverse transformations, the path integrals provide two distinct representations of the conditional probability distribution solving the master equations. Moreover, they can be used to represent averaged observables as explained in section 6. Besides the methods illustrated in the figure, we discuss path integral representations of processes with continuous state spaces whose master equations admit Kramers-Moyal expansions (sections 4.4 and 5.3). A truncation of the backward Kramers-Moyal expansion at the level of a diffusion approximation results in a path integral representation of the (backward) Fokker-Planck equation whose original development goes back to works of Martin, Siggia, and Rose [16], de Dominicis [17], Janssen [18, 19], and Bausch, Janssen, and Wagner [19]. The representation can be rewritten in terms of an Onsager-Machlup function [79], and it simplifies to Wiener’s path integral [80, 81] for purely diffusive Brownian motion [82]. Renormalization group techniques are not considered in this review. Information on these techniques can be found in [69, 70].

1.2. Organization of this review equation evolves the distribution p(t, n|τ, n0) backward in time, starting out at time τ = t. Both The organization of this review is summarized in master equations can be derived from the Chapman- figure 1 and is as follows. In the next section 1.3, Kolmogorov equation (cf. left side of figure 1). In we introduce the basic concepts of the theory of section 1.4, we discuss two explicit representations continuous-time Markov processes. After discussing of the conditional probability distribution solving the the roles of the forward and backward Fokker-Planck two master equations. Moreover, we comment on equations for processes with continuous sample paths, various numerical methods for the approximation of we turn to processes with discontinuous sample paths. this distribution and for the generation of sample The probability of finding such a “jump process” paths. Afterwards, section 1.5 provides a brief in a generic state n at time τ > t , given that 0 historical overview of contributions to the development the process has been in state n at time t , is 0 0 of stochastic path integrals. represented by the conditional probability distribution The main part of this review begins with section 2. p(τ, n|t , n ). Whereas the forward master equation 0 0 We first exemplify how a generalized probability evolves this probability distribution forward in time, generating function can be used to determine the starting out at time τ = t , the backward master 0 stationary probability distribution of an elementary Master equations and the theory of stochastic path integrals 7 chemical reaction. This example introduces the bra- be evaluated in terms of a perturbation expansion. ket notation used in this review. In section 2.1, we The summands of the expansion are expressed by formulate conditions under which a general forward Feynman diagrams. Besides, we derive a path integral master equation can be transformed into a linear representation for Markov processes with continuous partial differential equation (PDE) obeyed by the state spaces in the “intermezzo” section 4.4. This generating function. This function is defined as representation is obtained by performing a Kramers- the sum of the conditional probability distribution Moyal expansion of the backward master equation and p(τ, n|t0, n0) over a set of basis functions {|ni}, the it comprises a classic path integral representation [16– “kets” (cf. middle column of figure 1). The explicit 19] of the (backward) Fokker-Planck equation as a choice of the basis functions depends on the process special case. One can rewrite the representation of being studied. We discuss different choices of the the Fokker-Planck equation in terms of an Onsager- basis functions in section 2.2, first for a random walk, Machlup function [79] and, for purely diffusive afterwards for chemical reactions and for processes Brownian motion [82], the representation simplifies to whose particles locally exclude one another. Several the path integral of Wiener [80, 81]. Moreover, we methods have recently been proposed for the analysis recover a Feynman-Kac like formula [85], which solves of the PDE obeyed by the generating function. the (backward) Fokker-Planck equation in terms of an These methods include the variational method of average over the paths of an Itô stochastic differential Eyink [83] and of Sasai and Wolynes [39], the WKB equation [86–88] (or of a Langevin equation [89]). approximations [84] and spectral methods of Elgart In section 5, we complement the backward path and Kamenev [35] and of Assaf and Meerson [36–38], integral representation with a forward path integral and the spectral method of Walczak, Mugler, and representation. Its derivation in section 5.1 starts out Wiggins [34]. We comment on these methods in from the PDE obeyed by the generalized generating section 2.3. function (cf. right side of figure 1). The forward In section 3.1, we formulate conditions under path integral representation can, for example, be used which a general backward master equation can be to compute the generating function of generic linear transformed into a novel, backward-time PDE obeyed processes as we demonstrate in section 5.2. Besides, by a “marginalized distribution”. This object is we briefly outline how a Kramers-Moyal expansion defined as the sum of the conditional probability of the (forward) master equation can be employed distribution p(t, n|τ, n0) over a set of basis functions to derive a path integral representation for processes {|n0i} (cf. middle column of figure 1). If the basis with continuous state spaces in section 5.3. This path function |n0i is chosen as a probability distribution, integral can be expressed in terms of an average over the marginalized distribution also constitutes a true the paths of an SDE proceeding backward in time. Its probability distribution. Different choices of the potential use remains to be explored. basis function are considered in section 3.2. In Before proceeding, let us briefly point out some section 3.3, the use of the marginalized distribution properties of the forward and backward path integral is exemplified in the calculation of mean extinction representations. First, the paths along which these times. Afterwards, in section 3.4, we derive yet path integrals proceed are described by real variables another linear PDE, which is obeyed by a “probability and all integrations are performed over the real generating functional”. This functional is defined as line. Grassmann path integrals [26–31] for systems the sum of the conditional probability distribution whose particles locally exclude one another are not p(τ, n|t0, n0) over a set of basis functionals {hn|}, the considered. It is, however, explained in section 2.2.4 “bras”. In section 3.5, we show that the way in which how such systems can be treated without the need the generating functional “generates” probabilities for Grassmann variables, based on a method recently generalizes the Poisson representation of Gardiner and proposed by van Wijland [90]. Second, transformations Chaturvedi [32, 33]. of the path integral variables such as the “Doi-shift” [91] Sections 4 and 5 share the goal of representing are implemented on the level of the basis functions the master equations’ solution by path integrals. In and functionals. Third, our derivations of the forward section 4.1, we first derive a novel backward path and backward path integral representations do not integral representation from the PDE obeyed by the involve coherent states or combinatoric relations for marginalized distribution (cf. right side of figure 1). the commutation of exponentiated operators. Last, Its use is exemplified in sections 4.2, 4.3, and 4.5 in the path integrals allow for time-dependent rate which we solve several elementary processes. Although coefficients of the stochastic processes. we do not discuss the application of renormalization In section 6, we derive a path integral representa- group techniques, section 4.5.4 includes a discussion tion of averaged observables (cf. right side of figure 1). of how the backward path integral representation can This representation can be derived both from the back- Master equations and the theory of stochastic path integrals 8 ward and forward path integral representations (cf. sec- equation. Before going into the mathematical details, tion 6.1), and by representing the forward master equa- let us explain when a system’s time evolution can be tion in terms of the eigenvectors of creation and an- modelled as a continuous-time Markov process with nihilation matrices (“coherent states”; cf. section 6.2). discontinuous sample paths and what that phrase The duality between these two approaches resembles actually means. the duality between the wave [92] and matrix [93–95] First of all, if the evolution of a system is to formulations of quantum mechanics. Let us note that be modelled as a continuous-time Markov process, it our resulting path integral does not involve a “second- must be possible to describe the system’s state by quantized” or “normal-ordered” observable [96]. In some variable n. In fact, it must be possible to do fact, we show that this object agrees with the aver- so at every point τ in time throughout an observation age of an observable over a Poisson distribution. In period [t0, t]. A variable n ∈ Z could, for example, section 6.3, we then explain how the path integral can represent the position of a molecular motor along a be evaluated perturbatively using Feynman diagrams. cytoskeletal filament, or a variable n ∈ R≥0 the price Such an evaluation is demonstrated for the coagulation of a stock between the opening and closing times of reaction 2 A → A in section 6.4, restricting ourselves an exchange. The assumption of a continuous time to the “tree level” of the diagrams. parameter τ is rather natural and conforms to our In section 7, we review and extend a recent everyday experience. Still, a discrete time parameter method of Elgart and Kamenev for the computation may sometimes be preferred, for example, to denote of rare event probabilities [35]. As explained in individual generations of an evolving population [97]. section 7.1, this method evaluates a probability By allowing τ to take on any value between the initial distribution by expanding the forward path integral time t0 and the final time t, we can choose it to be representation from section 5 around “stationary”, arbitrarily close to one of those times. Below, this or “extremal”, paths. In a first step, one thereby possibility will allow us to describe the evolution of acquires an approximation of the ordinary probability the process in terms of a differential equation. generating function. In a second step, this generating The (unconditional) probability of finding the sys- function is transformed back into the underlying tem in state n at time τ is represented by the “single- probability distribution. The evaluation of this back time” probability distribution p(τ, n). Upon demand- transformation typically involves an additional saddle- ing that the system has visited some state n0 at an point approximation. In section 7.2, we demonstrate earlier time t0 < τ, the probability of observing state n both of the steps for the binary annihilation reaction at time τ is instead encoded by the conditional probab- 2 A → ∅, improving an earlier approximation of the ility distribution p(τ, n|t0, n0). If the conditional prob- process by Elgart and Kamenev [35] by terms of ability distribution is known, the single-time distribu- sub-leading order. In section 7.3, we then extend tion can be inferred from any given initial distribu- the “stationary path method” to the backward path tion p(t , n ) via p(τ, n) = P p(τ, n|t , n )p(t , n ). 0 0 n0 0 0 0 0 integral representation from section 4. The backward A stochastic process is said to be Markovian if a dis- path integral provides direct access to a probability tribution conditioned on multiple points in time ac- distribution without requiring an auxiliary saddle- tually depends only on the state that was realized point approximation. However, the leading order term most recently. In other words, a conditional distri- of its expansion is not normalized. We demonstrate bution p(t, n|τk, mk; ··· ; τ1, m1; t0, n0) must agree with 1 the procedure for the binary annihilation reaction p(t, n|τk, mk) whenever t > τk > τk−1, . . . , t0. There- in section 7.4. fore, a Markov process is fully characterized by the Finally, section 8 closes with a summary of single-time distribution p(t0, n0) and the conditional the different approaches discussed in this review and distribution p(τ, n|t0, n0). The latter function is com- outlines open challenges and promising directions for monly referred to as the “transition probability” for future research. Markov processes [98]. The stochastic dynamics of a system can be 1.3. Continuous-time Markov processes and the modelled in terms of a Markov process if the system forward and backward master equations has no memory. Let us explain this requirement with the example of a Brownian particle suspended in a Our main interest lies in a special class of stochastic fluid [82]. Over a very short time scale, the motion processes, namely in the class of continuous-time of such a particle is ballistic and its velocity highly Markov processes with discontinuous sample paths. auto-correlated [99]. But as the particle collides with These processes are also called “jump processes”. In the following, we outline the mathematical theory 1 Note that we do not distinguish between random variables and of jump processes and derive the central equations their outcomes. Moreover, we stick to the physicists’ convention of ordering times in descending order. In the mathematical obeyed by them: the forward and the backward master literature, the reverse order is more common, see e.g. [4]. Master equations and the theory of stochastic path integrals 9 molecules of the fluid, that memory fades away. A δ(x − x0). The corresponding equation is called significant move of the particle due to fluctuations the Kolmogorov backward or backward Fokker-Planck in the isotropy of its molecular bombardment then equation. It has the generic form appears to be completely uncorrelated from previous 1 2 ∂−τ p(t, x|τ, x0) = ατ (x0)∂x p + βτ (x0)∂ p . (2) moves (provided that the observer does not look too 0 2 x0 closely [100]). Thus, on a sufficiently coarse time scale, The forward and backward Fokker-Planck equa- the motion of the particle looks diffusive and can be tions provide information about the conditional prob- modelled as a Markov process. However, the validity ability distribution but not about the individual paths of the Markov assumption does not extend beyond the of a Brownian particle. The general theory of how coarse time scale. partial differential equations connect to the individual The Brownian particle exemplifies only two of sample paths of a stochastic process goes back to works the properties that we are looking for: its position of Feynman and Kac [85,107].2 Their theory allows us is well-defined at every time τ and its movement is to write the solution of the backward Fokker-Planck effectively memoryless on the coarse time scale. But equation (2) in terms of the following Kolmogorov for- the paths of the Brownian particle are continuous, mula, which constitutes a special case of the Feynman- meaning that it does not spontaneously vanish and Kac formula [108–110]: then reappear at another place. If the friction of
the fluid surrounding the Brownian particle is high p(t, x|τ, x0) = δ x − x(t) W . (3) (over-damped motion), the probability of observing The brackets hh·iiW represent an average over realiza- the particle at a particular place can be described tions of a Wiener process W , which evolves through by the Smoluchowski equation [101]. This equation uncorrelated Gaussian increments dW . The Wiener coincides with the simple diffusion equation when the process drives the evolution of the sample path x(s) particle is not subject to an external force. In the from x(τ) = x0 to x(t) via the Itô stochastic differen- general case, the probability of observing the particle tial equation (SDE) [86–88] at a particular place with a particular velocity obeys p the Klein-Kramers equation [102, 103] (the book of dx(s) = αs(x(s)) ds + βs(x(s)) dW (s) . (4)

Risken [104] provides a pedagogic introduction to The diffusion coefficient βs must be non-negative these equations). From a mathematical point of view, because x(s) describes the position of a real particle. all of these equations constitute special cases of the Otherwise, the sample path heads√ off into imaginary (forward) Fokker-Planck equation [1–4, 104]. For a space (for a multivariate process, βs may be chosen as single random variable x ∈ R, e.g. the position of the the unique symmetric and positive-semidefinite square Brownian particle, this equation has the generic form root of βs [111]). Algorithms for the numerical solution 1 of SDEs are provided in [108]. In the physical sciences, ∂ p(τ, x|t , x ) = −∂ α (x)p + ∂2β (x)p . (1) τ 0 0 x τ 2 x τ SDEs are often written as Langevin equations [89].3 The initial condition of this equation is given by For a discussion of stochastic differential equations the the Dirac delta distribution (or generalized function) reader may refer to a recent report on progress [112]. After this brief detour to continuous-time Markov p(t0, x|t0, x0) = δ(x − x0). Here we used the letter x for the random variable because the letter n would processes with continuous sample paths, let us return to jump processes, whose sample paths are suggest a discrete state space. The function ατ is discontinuous. A system that can be modelled as often called a drift coefficient and βτ a diffusion coefficient (note, however, that in the context of such a process are motor proteins on cytoskeletal filaments [113–115]. The uni-directional walk of a population genetics, βτ describes the strength of random genetic “drift” [105, 106]). For reasons molecular motor such as myosin, kinesin, or dynein addressed below, the diffusion coefficient must be along an actin filament or a microtubule is driven by non-negative at every point in time for every value the hydrolysis of adenosine triphosphate (ATP) and is intrinsically stochastic [116]. Once a sufficient amount of x (for a multivariate process, βτ represents a positive-semidefinite matrix). In the mathematical of energy is available, one of the two “heads” of the community, the Fokker-Planck equation is better motor unbinds from its current binding site on the known as the Kolmogorov forward equation [105], 2 Due to the central importance of the Feynman-Kac formula, honouring Kolmogorov’s fundamental contributions to we provide a brief proof of it in appendix A. We also the theory of continuous-time Markov processes [4]. encounter the formula in section 4 in evaluating a path integral Whereas the above Fokker-Planck equation evolves the representation of the (backward) master equation. 3 The Langevin equation corresponding to the SDE (4) reads conditional probability distribution forward in time, p ∂sx(s) = αs(x(s)) + βs(x(s))η(s), with the Gaussian white one can also evolve this distribution backward in time, noise η(s) having zero mean and the auto-correlation function 0 0 starting out from the final condition p(t, x|t, x0) = hη(s)η(s )i = δ(s − s ). Master equations and the theory of stochastic path integrals 10

filament and moves to the next binding site. Each equation binding site can only be occupied by a single head. On X p(t, n|t0, n0) = p(t, n|τ, m)p(τ, m|t0, n0) . (6) a coarse-grained level, the state of the system at time m τ is therefore characterized by the occupation of its Letting p(t|t0) denote the matrix with elements binding sites. With only a single cytoskeletal filament p(t, n|t0, n0), the Chapman-Kolmogorov equation can whose binding sites are labelled as {0, 1, 2,...} =· L, | | also be written as p(t|t0) = p(t|τ)p(τ|t0) (semigroup the variable n ≡ (n0, n1, n2,...) ∈ {0, 1} L can be property). Note that the matrix notation requires a used to represent the occupied and unoccupied binding mapping between the state space of n and n0 and an sites. Here, | | denotes the total number of binding L appropriate index set I ⊂ N. However, we also make sites along the filament and ni = 1 signifies that the use of this notation when the state space is countably i-th binding site is occupied. Since the state space infinite. of all the binding site configurations is discrete, a To derive the (forward) master equation from the change in the binding site configuration involves a Chapman-Kolmogorov equation (6), we define “jump” in state space. Provided that the jumps are p(τ + ∆t, n|τ, m) − δn,m uncorrelated from one another (which needs to be Qτ,∆t(n, m) ·= (7) verified experimentally), the dynamics of the system ∆t for all values of n and m and assume the existence and can be described by a continuous-time Markov process finiteness of the limits with discontinuous sample paths. Before addressing further systems for which this is the case, let us derive Qτ (n, m) ·= lim Qτ,∆t(n, m) . (8) ∆t→0 the fundamental equations obeyed by these processes: These are the elements of the transition (rate) matrix the forward and the backward master equation. Qτ , which is also called the infinitesimal generator In his classic textbook [110], Gardiner presents of the Markov process or is simply referred to as a succinct derivation of both the (forward) master the Qτ -matrix. Its off-diagonal elements wτ (n, m) ·= and the (forward) Fokker-Planck equation by distin- Qτ (n, m) denote the rates at which probability flows guishing between discontinuous and continuous con- from a state m to a state n 6= m. The “exit rates” tributions to sample paths. In the following, we are wτ (m) ·= −Qτ (m, m), on the other hand, describe only interested in the master equation, which governs the rates at which probability leaves state m. Both the evolution of systems whose states change discon- wτ (n, m) and wτ (m) are non-negative for all n and m. tinuously. To prevent the occurrence of continuous All of the processes considered here shall conserve the changes, we assume that the state of our system is P total probability, requiring that Qτ (n, m) = 0 or, represented by a discrete variable n and that the space P n equivalently, wτ (m) = wτ (n, m) (with wτ (m, m) · of all states is countable. With the state space chosen n = 0). The finiteness of the exit rate wτ (m) and the as the set of integers Z, n could, for example, represent conservation of total probability imply that we consider the position of a molecular motor along a cytoskeletal a stable and conservative Markov process [117]. In filament. On the other hand, n ∈ N0 could represent the natural sciences, the master equation is commonly the number of molecules in a chemical reaction. The written in terms of w , but most mathematicians prefer minimal jump size is one in both cases. By keeping the τ Qτ . These matrices can be converted into one another explicit role of n unspecified, the following considera- by employing tions also apply to systems harbouring different kind of A B C 3 Qτ (n, m) = wτ (n, m) − δn,mwτ (m) . (9) molecules (e.g. n ≡ (n , n , n ) ∈ N0), and to systems whose molecules perform random walks in a (discrete) We refer to both of the matrices as transition (rate) matrices and to their (identical) off-diagonal elements space (e.g. n ≡ {ni ∈ N0}i∈Z). To derive the master equation, we start out as transition rates. The transition rates fully specify by marginalizing the joint conditional distribution the stochastic process. p(t, n; τ, m|t0, n0) over the state m at the intermediate Assuming that the limit in (8) interchanges time τ (t ≥ τ ≥ t0), resulting in with a sum over the state m, the (forward) master X equation now follows from the Chapman-Kolmogorov p(t, n|t0, n0) = p(t, n; τ, m|t0, n0) . (5) equation (6) as m ∂ p(τ, n|t , n ) (10) Whenever the range of a sum is not specified, it shall τ 0 0 p(τ + ∆t, n|t , n ) − p(τ, n|t , n ) cover the whole state space of its summation variable. = lim 0 0 0 0 The above equation holds for arbitrary stochastic ∆t→0 ∆t processes. But for a Markov process, one can employ X p(τ + ∆t, n|τ, m) − δn,m = lim p(τ, m|t0, n0) ∆t→0 ∆t the relation between joint and conditional distributions m to turn the equation into the Chapman-Kolmogorov X = Qτ (n, m)p(τ, m|t0, n0) . m Master equations and the theory of stochastic path integrals 11

Thus, the master equation constitutes a set of intense research and have been analysed using path coupled, linear, first-order ordinary differential equa- integrals (see, for example, [96, 130–133] and the tions (ODEs). The time evolution of the distribu- references in section 1.5). Master equations, and tion starts out from p(t0, n|t0, n0) = δn,n0 . In matrix simulations algorithms based on master equations, notation, the equation can be written as ∂τ p(τ|t0) = are now being used in numerous fields of research. Qτ p(τ|t0). In terms of wτ , it assumes the intuitive They are being applied in the contexts of spin gain-loss form dynamics [134–137], gene regulatory networks [34,138– 143], the spreading of diseases [144–147], epidermal ∂τ p(τ, n|t0, n0) (11) homeostasis [148], nucleosome repositioning [149], X  = wτ (n, m)p(τ, m|·) − wτ (m, n)p(τ, n|·) . ecological [150–163] and bacterial dynamics [158, 164– m 167], evolutionary game theory [168–179], surface The dot inside the probability distribution’s argument growth [180], and social and economic processes [181– abbreviates the initial parameters t0 and n0, which are 184]. Queuing processes are also often modelled in of secondary concern right here. That will change be- terms of master equations, but in this context, the low in the derivation of the backward master equa- equations are typically referred to as Kolmogorov tion. An omission of the parameters also makes it equations [185]. Moreover, master equations and impossible to distinguish the conditional distribution the SSA have helped to understand the formation p(τ, n|t0, n0) from the single-time distribution p(τ, n). of traffic jams on highways [186, 187], the walks of The single-time distribution obeys the master equa- molecular motors along cytoskeletal filaments [113– tion as well, as can inferred directly from the rela- 115, 188–190], and the condensation of bosons in tion p(τ, n) = P p(τ, n|t , n )p(t , n ) or by sum- n0 0 0 0 0 driven-dissipative quantum systems [178, 191, 192]. ming the above master equation over an initial dis- The master equation that was found to describe the tribution p(t0, n0). In fact, the single-time distribu- coarse-grained dynamics of these bosons coincides with tion would even obey the master equation if the pro- the master equation of the (asymmetric) inclusion cess was not Markovian, but without providing a com- process [193–196]. Transport processes are commonly plete characterization of the process [70,74]. The mas- modelled in terms of the (totally) asymmetric simple ter equation (10) or (11) is particularly interesting for exclusion process (ASEP or TASEP) [197–200]. The transition rates causing an imbalance between forward ASEP describes the biased hopping of particles along a and backward transitions along closed cycles of states, one-dimensional lattice, with each lattice site providing i.e. for rates violating Kolmogorov’s criterion [118] for space for at most one particle. The ASEP and detailed balance [70]. Such systems are truly out of the TASEP are regarded as paradigmatic models thermal equilibrium. If detailed balance is instead ful- in the field of non-equilibrium statistical mechanics, filled, the system eventually converges to a stationary with many exact mathematical results having been Boltzmann-Gibbs distribution with vanishing probab- established [201–212]. Some of these results were ility currents between states [70]. Whether or not de- established by applying the Bethe ansatz to the master tailed balance is actually fulfilled is, however, not rel- equation of the ASEP [205, 209]. The review of Chou, evant for the methods discussed in this review. Inform- Mallick, and Zia provides a comprehensive account ation on the existence and uniqueness of an asymptotic of the ASEP and of its variants [190]. The master stationary distribution of the master equation can be equation of the TASEP with Langmuir kinetics was found in [117]. recently used to understand the length regulation of The name “master equation” was originally coined microtubules [213]. by Nordsieck, Lamb, and Uhlenbeck [5] in their study Unlike deterministic models, the master equations of the Furry model of cosmic rain showers [119]. describing the dynamics of the above systems take into Shortly before, Feller applied an equation of the account that discrete and finite populations are prone same structure to the growth of populations [120] to “demographic fluctuations”. The populations of the and Delbrück to well-mixed, auto-catalytic chemical above systems consist of genes or proteins, infected reactions [121]. Delbrück’s line of research was persons, bacteria or cars and they are typically small, followed by several others [122–125], most notably by at least compared to the number of molecules in a mole McQuarrie [126–128] (see also the books [98,110,129]). of gas. For example, the copy number of low abundance In these articles, several elementary chemical reactions proteins in Escherichia coli cytosol was found to be in are solved by methods that also appear later in this the tens to hundreds [214]. Therefore, the presence review. When the particles engaging in a reaction or absence of a single protein is much more important can also diffuse in space, their density may exhibit than the presence or absence of an individual molecule dynamics that are not expected from observations in a mole of gas. A demographic fluctuation may even made in well-mixed environments. Hence, reaction- be fatal for a system, for example, when the copy diffusion master equations have been the focus of Master equations and the theory of stochastic path integrals 12 number of an auto-catalytic reactant drops to zero. (recall that p(τ|τ0) is the matrix with elements The master equation (10) provides a useful tool to p(τ, n|τ0, n0)). The Chapman-Kolmogorov equa- describe such an effect. tion (6) is also solved by the distribution. Although Up to this point, we have only considered the the matrix exponential inside this solution can in prin- forward master equation. But just as the (forward) ciple be evaluated in terms of the (convergent) Taylor k Fokker-Planck equation (1) is complemented by the P∞ (τ−τ0) k expansion k=0 k! Q , the actual calculation of backward Fokker-Planck equation (2), the (forward) this series is typically infeasible for non-trivial pro- master equation (10) is complemented by a backward cesses, both analytically and numerically (a trunca- master equation. This equation can be derived from tion of the Taylor series may induce severe round-off the Chapman-Kolmogorov equation (6) as errors and serves as a lower bound on the perform-

∂−τ p(t, n|τ, n0) (12) ance of algorithms in [217]). Consequently, alternat- ive numerical algorithms have been developed to eval- p(t, n|τ − ∆t, n0) − p(t, n|τ, n0) = lim uate the matrix exponential. Moler and Van Loan re- ∆t→0 ∆t viewed “nineteen dubious ways” of computing the ex- X p(τ, m|τ − ∆t, n0) − δm,n = lim p(t, n|τ, m) 0 ponential in [217, 218]. Algorithms that can deal with ∆t→0 ∆t m very large state spaces are considered in [219,220]. For X time-dependent transition rates, the matrix exponen- = p(t, n|τ, m)Qτ (m, n0) . m tial generalizes to a Magnus expansion [221, 222]. Here, the transition rate is obtained in the limit In the previous paragraph, we restricted the dynamics to a finite state space to ensure the existence lim∆t→0 Qτ−∆t,∆t(m, n0) (cf. (7)). In matrix notation, of the matrix exponential in (15). Provided that the backward master equation reads ∂−τ p(t|τ) = the supremum sup |Q(m, m)| of all the exit rates is p(t|τ)Qτ . In terms of wτ , it assumes the form (cf. (9)) m finite (uniformly bounded Q-matrix), the validity of ∂−τ p(t, n|τ, n0) (13) the above solution extends to state spaces comprising a X  = p(·|τ, m) − p(·|τ, n0) wτ (m, n0) . countable number of states [223]. To see that, we define −1 m the left stochastic matrix P ·= 1 + λ Q, with the In this equation, the dots abbreviate the final paramet- parameter λ being larger than the above supremum. Q∆t −λ∆t λ∆tP ers t and n. The backward master equation evolves the Writing e = e e with ∆t ·= τ − τ0, the conditional probability distribution backward in time, matrix exponential can be evaluated in terms of the starting out from the final condition p(t, n|t, n0) = convergent Taylor series δ . Just as the backward Fokker-Planck equation, ∞ n,n0 X e−λ∆t(λ∆t)k the backward master equation proves useful for the p(τ|τ ) = P k . (16) 0 k! computation of mean extinction and first passage times k=0 (see [110,215] and section 3.3). Furthermore, it follows Effectively, one has thereby decomposed the continuous- from the backward master equation (12) that the (con- time Markov process with transition matrix Q into · P ditional) average hAi(t|τ, n0) ·= n A(n)p(t, n|τ, n0) a discrete-time Markov chain with transition matrix of an observable A fulfils an equation of just the same P , subordinated to a continuous-time Poisson process form, namely with rate coefficient λ (the Poisson process acts as a X “clock” with sufficiently high ticking rate λ). Such a ∂−τ hAi(t|τ, n0) = hAi(t|τ, m)Qτ (m, n0) . (14) m decomposition is called a uniformization or random- The final condition of the equation is given by ization and was first proposed by Jensen [224]. The series (16) can be evaluated via numerically stable al- hAi(t|t, n0) = A(n0). The validity of equation (14) is the reason why we later employ a “backward” path gorithms and truncation errors can be bounded [225, integral to represent the average hAi (cf. section 6). 226]. Nevertheless, the uniformization method requires the computation of the powers of a matrix having as many rows and columns as the system has states. Con- 1.4. Analytical and numerical methods for the sequently, a numerical implementation of the method is solution of master equations only feasible for sufficiently small state spaces. Further If the dynamics of a system are restricted to a information on the method and on its improvements finite number of states and if its transition rates can be found in [224–229]. are independent of time, both the forward master The mathematical study of the existence and equation ∂τ p(τ|t0) = Qp(τ|t0) and the backward uniqueness of solutions of the forward and backward master equations was pioneered by Feller and Doob in master equation ∂−τ0 p(t|τ0) = p(t|τ0)Q are solved by [216] the 1940s [230, 231]. Feller derived an integral recur-

Q(τ−τ0) rence formula [117,230], which essentially constitutes a p(τ|τ0) = e 1 (15) Master equations and the theory of stochastic path integrals 13 single step of the “path summation representation” that p(t, n ⎜t0, n0) we derive further below. In the following, we assume that the forward and the backward master equations probability to jump have the same unique solution and we restrict our at- to n1: w(n1, n0)⁄w(n0) tention to processes performing only a finite number n0 n3 of jumps during any finite time interval. These con- probability to remain in state n = n: (t − t ) ditions, and the conservation of total probability, are, n1 J n J nJ n for example, violated by processes that “explode” after n2 4 a finite time. More information on such processes is nJ−1 provided in [117, 216]. probability to wait In the following, we complement the above for τ0 = t1 − t0: n0(τ0) representations of the master equations’s solution with a “path summation representation”. This t0 t1 t2 t3 … tJ−1 tJ t representation can be derived by examining the steps of the stochastic simulation algorithm (SSA) of Gillespie Figure 2. Illustration of the stochastic simulation algorithm (SSA) and of the path summation representation (its “direct” version) [75, 76, 78] or by performing of the probability distribution p(t, n|t0, n0). In a numerical a Laplace transformation of the forward master implementation of the SSA, the system is prepared in state equation. Here we follow the former, qualitative, n0 at time t0. After a waiting time τ0 that is drawn from the exponential distribution W (τ ) = w(n )e−w(n0)τ0 Θ(τ ), approach. A formal derivation of the representation via n0 0 0 0 the system transitions into a new state. An arrival state n1 the master equation’s Laplace transform is provided in is chosen with probability w(n1, n0)/w(n0). The procedure is appendix B. Although the basic elements of the SSA repeated until after J steps, the current time tJ ≤ t plus an had already been known before Gillespie’s work [230– additional waiting time exceeds t. The sample path has thus 236], its popularity largely increased after Gillespie resided in state nJ at time t. This information is recorded in a histogram approximation of p(t, n|t0, n0). The path summation applied it to the study of chemical reactions. As the representation of p(t, n|t0, n0) requires nJ to coincide with n. SSA is restricted to time-independent transition rates, The probability that the system has remained in state n over the last time interval [tJ , t] is given by the survival probability so is the following derivation. −w(n)(t−t ) Sn(t − tJ ) = e J Θ(t − tJ ). The total probability of To derive the path summation representation, we the generated path, integrated over all possible waiting times, is prepare a system in state n0 at time t0 as illustrated represented by pτ (PJ ) in (20). A summation of this probability in figure 2. Since the process is homogeneous in time, over all possible sample paths results in the path summation representation (19). we may choose t0 = 0. The total rate of leaving state n is given by the exit rate w(n ) = P w(n , n ). 0 0 n1 1 0 In order to determine how long the system actually visit state nJ = n at some time tJ ≤ t. The stays in state n0, one may draw a random waiting total time τJ−1 + ··· + τ0 until the jump to state time τ from the exponential distribution W (τ ) · 0 n0 0 · n occurs is distributed by the convolutions of the −w(n0)τ0 = w(n0)e Θ(τ0). The Heaviside step function J−1 individual waiting time distributions, i.e. by ? Wn . Θ prevents the sampling of negative waiting times j=0 j and is here defined as Θ(τ) = 1 for τ ≥ 0 and For example, τ ·= τ1 + τ0 is distributed by Θ(τ) = 0 for τ < 0. Thus far, we only know that the (W ? W )(τ) = dτ W (τ − τ )W (τ ) (17) system leaves n0 but not where it ends up. It could n1 n0 ˆ 0 n1 0 n0 0 R end up in any state n1 for which the transition rate −w(n0)τ −w(n1)τ
w(n1)w(n0) e − e w(n1, n0) is positive. The probability that a particular = Θ(τ) . (18) state n1 is realized is given by w(n1, n0)/w(n0). In w(n1) − w(n0) a numerical implementation of the SSA, the state The probability that the system still resides in state n1 is determined by drawing a second (uniformly- nJ = n at time t is determined by the “survival −w(n)(t−tJ ) distributed) random number. Our goal is to derive an probability” Sn(t − tJ ) = e Θ(t − tJ ). After analytic representation of the probability p(t, n|t0, n0) putting all of these pieces together, we arrive at of finding the system in state n at time t. Thus, the following path summation representation of the after taking J − 1 further steps, the sample path conditional probability distribution: PJ ·= {nJ ← · · · ← n1 ← n0} should eventually ∞ X X p(t, n|t , n ) = p (P ) with (19) 4 Just as a population whose growth is described by the 0 0 t−t0 J 2 J=0 deterministic equation ∂τ n = n explodes after a finite time, {PJ } so does a population whose growth is described by the master   J−1 w(nj+1, nj)  equation (11) with transition rate w(n, m) = δn,m+1m(m − pτ (PJ ) = Sn ? ? Wnj (τ) . (20) 1) [117]. This transition rate models the elementary reaction j=0 w(nj) 2 A → 3 A as explained in section 1.5. An explosion also occurs P P P 2 Here, ·= ··· generates every path for the rate w(n, m) = δn,m+1m . {PJ } n1 nJ−1 with J jumps between n0 and nJ = n. The probability Master equations and the theory of stochastic path integrals 14 of such a path, integrated over all possible waiting of sample paths with the correct probability of times, is represented by pτ (PJ ). By an appropriate occurrence. Thus, just a few sample paths generated choice of integration variables, the probability pτ (PJ ) with the SSA are often sufficient to infer the “typical” can also be written as dynamics of a process. A look at individual paths J−1 τ may, for example, reveal that the dynamics of a Y w(nj+1, nj)  p (P ) = dτ W (τ ) (21) system are dominated by some spatial pattern, e.g. τ J ˆ j w(n ) nj j j=0 0 j by spirals [168]. Efficient variations of the above
“direct” version of the SSA are, for example, described ·Sn τ − (τJ−1 + ... + τ0) . in [78, 247–249]. Algorithms for the fast simulation The survival probability Sn is included in these in- of biochemical networks or processes with spatial tegrations. Without the integrations, the expression degrees of freedom are implemented in the simulation would represent the probability of a path with J jumps packages [250–257]. and fixed waiting times. That probability is, for ex- P The evaluation of the average hAi = n A(n)p(t, n|·) ample, used in the master equation formulation of of an observable A typically requires the computa- stochastic thermodynamics in associating an entropy tion of a larger number of sample paths. However, to individual paths [237]. In appendix B, we formally since the occurrence probability of sample paths gen- derive the path summation representation (19) from erated with the SSA is statistically correct, such an the Laplace transform of the forward master equa- average typically converges comparatively fast. Fur- tion (11). thermore, each path can be sampled independently of The path summation representation (19) does not every other path. Therefore, the computation of paths only form the basis of the SSA but also of some can be distributed to individual processing units, sav- alternative algorithms [238–243]. These algorithms ing real time, albeit no computation time. A distrib- either infer the path probability pτ (PJ ) numerically uted computation of the sample paths is most often re- from its Laplace transform or evaluate the convolutions quired, but possibly not even sufficient, if one wishes to in (20) analytically. The analytic expressions that compute the full probability distribution p(t, n|t0, n0). arise are, however, rather cumbersome generalizations Vastly more sample paths are required for this purpose, of the convolution in (18) [244, 245]. They simplify especially if “rare event probabilities” in the distribu- only for the most basic processes (e.g. for a random tion’s tails are sought for. In particular, if the prob- walk or for the Poisson process). Moreover, care has to ability of finding a system in state n at time t is only be taken when the analytic expressions are evaluated 10−10, an average of 1010 sample paths are needed to numerically because they involve pairwise differences observe that event just once. Moreover, the probab- of the exit rate w(n) (cf. (18)). When these exit ility of observing any particular state decreases with rates differ only slightly along a path, a substantial the size of a system’s state space. Thus, the sampling loss of numerical significance may occur due to finite of full distributions becomes less and less feasible as precision arithmetic. Future studies could explore how systems become larger. Various other challenges re- the convolutions of exponential distributions in (20) main open as well; for example, the efficient simulation can be approximated efficiently (for example, in terms of processes evolving on multiple time scales. These of a Gamma distribution or by analytically determining processes are typically simulated using approximative the Laplace transform of (20), followed by a saddle- techniques such as τ-leaping [78, 249, 258–267]. An- point approximation [246] of the corresponding inverse other challenge is posed by the evaluation of processes Laplace transformation). In general, both the SSA with time-dependent transition rates [268–270]. as well as its competitors suffer from the enormous For completeness, let us mention yet another number of states of non-trivial systems, as well as from numerical approach to the (forward) master equation. the even larger number of paths connecting different Since the master equation (10) constitutes a set of states. In [240], these paths were generated using a coupled linear first-order ODEs, it can of course be deterministic depth-first search algorithm, combined treated as such and be integrated numerically. The with a filter to select the paths that arrive at the integration is, however, only feasible if the state space right place at the right time. In [242], a single path is sufficiently small (or appropriately truncated) and was first generated using the SSA and then gradually if all transitions occur on comparable time scales changed into new paths through a Metropolis Monte (otherwise, the master equation is quite probably Carlo scheme. Thus far, the two methods have only stiff [271]). Nevertheless, a numerical integration of been applied to relatively simple systems and their the master equation may be preferable over the use prevalence is low compared to the prevalence of the of the SSA if the full probability distribution is to be SSA. Further research is needed to explore how relevant computed. paths can be sampled more efficiently. Neither the matrix exponential representation The true power of the SSA lies in its generation Master equations and the theory of stochastic path integrals 15

Q(t−t0) p(t|t0) = e 1 of the conditional probability dis- 1.5. History of stochastic path integrals tribution, nor its path summation representation (19) The oldest path integral, both in the theory of is universally applicable. Moreover, even if the require- stochastic processes and beyond [107, 285–287], is pre- ments of these solutions are met, the size of the state sumably Wiener’s integral for Brownian motion [80, space or the complexity of the transition matrix may 81]. The path integrals we consider here were devised make it infeasible to evaluate them. In the next sec- somewhat later, namely in the 1970s and 80s: first tions, we formulate conditions under which the con- for the Fokker-Planck (or Langevin) equation [16–19] ditional probability distribution can be represented in and soon after for the master equation. For the mas- terms of the “forward” path integral ter equation, the theoretical basis of these “stochastic” t) −S path integrals was developed by Doi [6, 7]. He first p(t, n|t0, n0) = hn| e |n0i (22) t t0 expressed the creation and annihilation of molecules [t0 in a chemical reaction by the corresponding operators and in terms of the “backward” path integral for the quantum harmonic oscillator [288] (modulo a t] −S† normalization factor), introducing the concept of the p(t, n|t0, n0) = hn0| e |ni . (23) t0 t Fock space [20] into non-equilibrium theory. Further- (t0 more, he employed coherent states [22–24] to express The meaning of the integral signs and of the bras hn| averaged observables. Similar formalisms as the one of and kets |ni will become clear over the course of this Doi were independently developed by Zel’dovich and review. Let us only note that the integrals do not Ovchinnikov [8], as well as by Grassberger and Scheu- proceed along paths of the discrete variable n, but over nert [10]. Introductions to the Fock space approach, the paths of two continuous auxiliary variables that which are in part complementary to our review, are, are introduced for this purpose. The relevance of each for example, provided in [70, 91, 289–291]. The re- path is weighed by the exponential factors inside the view of Mattis and Glasser [289] provides a chrono- integrals. logical list of contributions to the field. These contri- Besides these exact representations of the condi- butions include Rose’s renormalized kinetic theory [9], tional probability distribution solving the master equa- Mikhailov’s path integral [11, 12, 14], which is based tions, there exist powerful ways of approximating this on Zel’dovich’s and Ovchinnikov’s work, and Golden- distribution and the values of averaged observables. feld’s extension of the Fock space algebra to polymer These methods include the Kramers-Moyal [103, 272] crystal growth [13]. Furthermore, Peliti reviewed the and the system-size expansion [98, 273], as well as the Fock space approach and provided derivations of path derivation of moment equations. Information on these integral representations of averaged observables and of methods can be found in classic text books [98,110] and the probability generating function [15]. Peliti also ex- in a recent review [274]. Although moment equations pressed the hope that future “rediscoveries” of the path encode the complete information about a stochastic integral formalism would be unnecessary in the future. process, they typically constitute an infinite hierarchy However, we believe that the probabilistic structures whose evaluation requires a truncation by some closure behind path integral representations of stochastic pro- scheme [275–283]. On the other hand, the Kramers- cesses have not yet been clearly exposed. As illustrated Moyal and the system-size expansion approximate the in figure 1, we show that the forward and backward master equation in terms of a Fokker-Planck equation. master equations admit not only one but two path in- Both expansions work best if the system under con- tegral representations: the forward representation (22) sideration is “large” (more precisely, they work best and the novel backward representation (23). Although if the dynamics are centred around a stable or meta- the two path integrals resemble each other, they dif- stable state at a distance N  1 from a potentially ab- fer conceptually. While the forward path integral rep- sorbing state; the standard deviation of its surround- √ resentation provides a probability generating function ing distribution is then of order N). An extension in an intermediate step, the backward representation of the system-size expansion to absorbing boundaries provides a distribution that is marginalized over an ini- has recently been proposed in [284]. In sections 4.4 tial state. Both path integrals can be used to represent and 5.3, we show how Kramers-Moyal expansions of the averaged observables as shown in section 6. The back- backward and forward master equations can be used to ward path integral, however, will turn out to be more derive path integral representations of processes with convenient for this purpose (upon choosing a Poisson continuous state spaces. When the expansion of the n −x distribution as the basis function, i.e. |ni ·= x e , backward master equation stops (or is truncated) at · n! the level of a diffusion approximation, one recovers a the representation is obtained by summing (23) over classic path integral representation of the (backward) an observable A(n)). Let us note that even though we Fokkers-Planck equation [16–19]. adopt some of the notation of quantum theory, our re- view is guided by the notion that quantum (field) the- Master equations and the theory of stochastic path integrals 16 ory is “totally unnecessary” for the theory of stochastic “Fermionic” path integrals, on the other hand, have path integrals. Michael E. Fisher once stated the same been developed for systems in which the particles about the relevance of quantum field theory for renor- exclude one another. Thus, the number of particles malization group theory [292] (while acknowledging in these systems is locally restricted to the values that in order to do certain types of calculations, fa- 0 and 1. For systems with excluding particles, the miliarity with quantum field theory can be very useful; master equation’s solution may be represented in the same applies to the theory of stochastic path in- terms of a path integral whose underlying creation tegrals). and annihilation operators fulfil an anti-commutation Thus far, path integral representations of the relation [26–31]. However, van Wijland recently master equation have primarily been applied to showed that the use of operators fulfilling the bosonic processes whose transition rates can be decomposed commutation relation is also possible [90]. These additively into the rates of simple chemical reactions. approaches are considered in section 2.2.4. Simple means that the transition rate of such a reaction We do not intend to delve further into the is determined by combinatoric counting. Consider, historic development and applications of stochastic for example, a reaction of the form k A → l A in path integrals at this point. Doing so would require a which k ∈ N0 molecules of type A are replaced by proper introduction into renormalization group theory, l ∈ N0 molecules of the same type. Assuming that which is of pivotal importance for the evaluation of the k reactants are drawn from an urn with a total the path integrals. Readers can find information on number of m molecules, the total rate of the reaction the application of renormalization group techniques in should be proportional to the falling factorial (m)k · the review of Täuber, Howard, and Vollmayr-Lee [69] = m(m − 1) ··· (m − k + 1). The global time scale and in the book of Täuber [70]. Introductory texts of reaction is set by the rate coefficient γτ , which we are also provided by Cardy’s (lecture) notes [91, 293]. allow to depend on time. Thus, the rate at which the Roughly speaking, path integral representations of chemical reaction k A → l A induces a transition from the chemical master equation (24) have been used to state m to state n is wτ (n, m) = γτ (m)kδn,m−k+l. The assess how a macroscopic law of mass action changes Kronecker delta inside the transition rate ensures that due to fluctuations, both below [96, 130, 132, 294–305] k molecules are indeed replaced by l new ones. Note and above the (upper) critical dimension [295, 306, that the number of particles in the system can never 307], using either perturbative [14, 70, 130–133, 294– become negative, provided that the initial number of 301, 303–305, 308–316] or non-perturbative [306, 307, particles was non-negative. Hence, the state space of 317–319] techniques. All of these articles focus on n is N0. Insertion of the above transition rate into the stochastic processes with spatial degrees of freedom forward master equation (11) results in the “chemical” for which alternative analytical approaches are scarce. master equation Path integral representations of these processes, combined with renormalization group techniques, ∂τ p(τ, n|t0, n0) (24)   have been pivotal in understanding non-equilibrium = γτ (n − l + k)kp(τ, n − l + k|·) − (n)kp(τ, n|·) . phase transitions and they contributed significantly Microphysical arguments for its applicability to to the classification of these transitions in terms chemical reactions can be found in [77]. According of universality classes [67, 70, 320]. Moreover, path to the chemical master equation, the mean particle integral representations of master equations have P number hni = n n p(τ, n|·) obeys the equation recently been employed in such diverse contexts as ∂τ hni = γτ (l − k)h(n)ki. For a system with a large the study of neural networks [321–323], of ecological number of particles (n  k), fluctuations can often populations [156, 157, 162, 163, 324, 325], and of the be neglected in a first approximation, leading to the differentiation of stem-cells [326]. deterministic rate equation k 1.6. Résumé ∂τ n¯ = γτ (l − k)¯n , (25) obeyed by a continuous particle number n¯(τ) ∈ R. Continuous-time Markov processes with discontinuous Path integral representations of the chemical sample paths describe a broad range of phenomena, master equation (24) are sometimes said to be ranging from traffic jams on highways [186] and on “bosonic”. First, because an arbitrarily large number cytoskeletal filaments [113–115, 190] to novel forms of molecules may in principle be present in the system of condensation in bosonic systems [178, 191, 192]. (both globally and, upon extending the system to a In the introduction, we laid out the mathematical spatial domain, locally). Second, because the path theory of these processes and derived the fundamental integral representations are typically derived with the equations governing their evolution: the forward and help of “creation and annihilation operators” fulfilling the backward master equations. Whereas the forward a “bosonic” commutation relation (see section 2.2.2). master equation (10) evolves a conditional probability Master equations and the theory of stochastic path integrals 17 distribution forward in time, the backward master probability of observing n ∈ N0 such molecules obeys equation (12) evolves the distribution backward in the equation time. In the following main part of this review,   ∂τ p(τ, n|t0, n0) = γ p(τ, n − 1|·) − p(τ, n|·) (28) we represent the conditional probability distribution   solving the master equations in terms of path integrals. + µ (n + 1)p(τ, n + 1|·) − np(τ, n|·) , The framework upon which these path integrals are with initial condition p(t0, n|t0, n0) = δn,n0 . This based unifies a broad range of approaches to the master master equation respects the fact that the number equations, including, for example, the spectral method of molecules cannot become negative through the of Walzcak, Mugler, and Wiggins [34] and the Poisson reaction A → ∅. By differentiating the probability representation of Gardiner and Chaturvedi [32, 33]. generating function g(τ; q|t0, n0) in (26) with respect to the current time τ, one finds that it obeys the flow 2. The probability generating function equation ∂ g = (q − 1)(γ − µ∂ )g . (29) The following two sections 2 and 3 are devoted to τ q n0 mapping the forward and backward master equations Its time evolution starts out from g(t0; q|t0, n0) = q . (10) and (12) to linear partial differential equations Instead of solving the flow equation right away, let us (PDEs). For brevity, we refer to such linear PDEs as first simplify it by changing the basis function qn of the “flow equations”. In sections 4 and 5, the derived flow generating function (26). As a first step, we change it equations are solved in terms of path integrals. to (q + 1)n, turning the corresponding flow equation It has been known since at least the 1940s that into ∂τ g = q(γ − µ∂q)g. As a second step, we multiply − γ (q+1) the (forward) master equation can be cast into a flow the new basis function by e µ and arrive at the equation obeyed by the ordinary probability generating simplified flow equation function ∂ g = −µq∂ g . (30) X τ q g(τ; q|t , n ) ·= qnp(τ, n|t , n ) , (26) 0 0 · 0 0 The generating function obeying this equation reads n∈N0 X n − γ (q+1) at least when the corresponding transition rate g(τ; q|·) = (q + 1) e µ p(τ, n|·) . (31) describes a simple chemical reaction [122–128, 327]. n∈N0 The generating function effectively replaces the As before, the dots inside the functions’ arguments discrete variable n by the continuous variable q. The abbreviate the initial parameters t0 and n0. absolute convergence of the sum in (26) is ensured (at The simplified flow equation (30) is now readily least) for |q| ≤ 1. The generating function “generates” solved by separation of variables. But before doing so, probabilities in the sense that let us introduce some new notation. From now on, we 1 write the basis function as n p(τ, n|t0, n0) = ∂q g(τ; q|t0, n0) . (27) n − γ (q+1) n! q=0 · µ |niq ·= (q + 1) e (32) This inverse transformation from g to p involves the and the corresponding generalized probability generat- · application of the (real) linear functional Ln[f] ·= ing function (31) as 1 ∂nf(q) , which maps a (real-valued) function n! q q=0 X |g(τ|·)i ·= |ni p(τ, n|·) . (33) f to a (real) number. Moreover, it fulfils Ln[f + q · q n∈ αg] = Ln[f] + αLn[g] for two functions f and g, N0 and α ∈ R. A more convenient notation for linear In quantum mechanics, an object written as |·i is functionals is introduced shortly. In the following, called a “ket”, a notation that was originally introduced we generalize the probability generating function (26) by Dirac [21]. In the above two expressions, the and formulate conditions under which the generalized kets simply represent ordinary functions. For brevity, function obeys a linear PDE, i.e. a flow equation. But we write the arguments of the kets as subscripts before proceeding to the general case, let us exemplify and occasionally drop these subscripts altogether. In the use of a generalized probability generating function principle, the basis function could also depend on time for a specific process (for brevity, we often drop the (i.e. |niτ,q). terms “probability” and “generalized” in referring to Later, in section 2.2, we introduce various basis this function). functions for the study of different stochastic processes, · inq As the example, we consider the bi-directional including the Fourier basis function |niq ·= e for the reaction ∅ A in which molecules of type A form solution of a random walk (with n ∈ Z). Moreover, we at rate γ ≥ 0 and degrade at per capita rate µ > 0. consider a “linear algebra” approach in which |ni ·= eˆn According to the chemical master equation (24), the represents the unit column vector in direction n ∈ N0 (this vector equals one at position n + 1 and is zero Master equations and the theory of stochastic path integrals 18 everywhere else; cf. section 2.2.3). The generating separation of variables. Making the ansatz |gi = function (33) corresponding to this “unit vector basis” f(τ)h(q), one obtains an equation whose two sides coincides with the column vector p(τ|t0, n0) of the depend either on f or on h but not on both. The −kµ(τ−t0) k probabilities p(τ, n|t0, n0). The unit vector basis will equation is solved by f(τ) = e and h(q) = q , prove useful later on in recovering a path integral with k being a non-negative parameter. The non- representation of averaged observables in section 6.2. negativity of k ensures the finiteness of the initial

In our present example and in most of this review, condition |g(t0|t0, n0)iq = |n0iq in the limit q → 0. By however, the generating function (33) represents an the completeness of the polynomial basis, the values ordinary function and obeys a linear PDE. For the basis of k can be restricted to N0. It proves convenient function (32), this PDE reads to represent also the standard polynomial basis in · ∂ |gi = −µq∂ |gi . (34) terms of bras and kets, namely by defining hhk|f ·= τ q 1 k · k k! ∂q f(q)|q=0 and |kiiq ·= q . These bras and kets Its time evolution starts out from |g(t0|t0, n0)i = |n0i. are again orthogonal to one another in the sense of In order to recover the conditional probability hhk|lii = δk,l and they also fulfil a completeness relation distribution from the generating function (33), we now P ( k |kiihhk| represents a Taylor expansion around q = 0 complement the “kets” with “bras”. Such a bra is and thus acts as an identity on analytic functions). written as h·| and represents a linear functional in our Using the auxiliary bras and kets, the solution of the present example. In particular, we define a bra hm| flow equation (34) can be written as for every m ∈ N0 by its following action on a test X −kµ(τ−t0) function f: |g(τ|·)i = |kii e hhk|n0i . (38) 1  γ m k∈N0 hm|f ·= ∂q + f(q)|q=−1 . (35) m! µ We wrote the expansion coefficient in this solution as hhk|n i to respect the initial condition |g(t |·)i = |n i. The evaluation at q = −1 could also be written in 0 0 0 The conditional probability distribution can be integral form as dq δ(q + 1)(···). The functional hm| R recovered from the generating function (38) via the is obviously linear and it maps the basis function (32) ´ inverse transformation (37) as to X −kµ(τ−t0) p(τ, n|t0, n0) = hn|kii e hhk|n0i . (39) hm|ni = δm,n . (36) k∈N0 Thus, the “basis functionals” in {hm|} are m∈N0 The coefficients hn|kii and hhk|n i can be computed orthogonal to the basis functions in {|ni} . The 0 n∈N0 recursively as explained in [328]. Here we are interested orthogonality condition can be used to recover the in the asymptotic limit τ → ∞ of the distribution (39) conditional probability distribution from (33) via for which only the k = 0 “mode” survives. Therefore, p(τ, n|·) = hn|g(τ|·)i . (37) the distribution converges to the stationary Poisson Besides being orthogonal to one another, the distribution kets (32) and bras (35) fulfil the completeness relation (γ/µ)ne−γ/µ P 5 p(∞, n|t0, n0) = hn|0iihh0|n0i = . (40) n |niqhn|f = f(q) with respect to analytic functions n! (note that hn|f as defined in (35) is just a real number The above example illustrates how the master and does not depend on q). Above, we mentioned that equation can be transformed into a linear PDE obeyed we later introduce alternative basis kets, including the by a generalized probability generating function and inq Fourier basis function |niq = e for the solution of a how this PDE simplifies for the right basis function. random walk (with n ∈ Z), and the unit column vector The explicit choice of the basis function depends on |ni = eˆn with n ∈ N0. These kets can also be com- the problem at hand. Moreover, the above example plemented to obtain orthogonal and complete bases, introduced the bra-ket notation used in this review. In namely by complementing the Fourier basis function section 4.2, the reaction ∅ A will be reconsidered · π dq −imq with the basis functional hm|f ·= −π 2π e f(q) using a path integral representation of the probability (m ∈ Z), and by complementing the unit´ column vec- distribution. We will then see that this process is not · | tor with the unit row vector hm| ·= eˆm (m ∈ N0). For only solved by a Poisson distribution in the stationary the unit vector basis {|ni, hm|}n,m∈N0 , the complete- limit, but actually for all times (at least, if the P 1 ness condition n |nihn| = involves the (infinitely number of molecules in the system was initially Poisson large) unit matrix 1. distributed). Thanks to the new basis function (32), the In the remainder of this section, as well as in simplified flow equation (34) can be easily solved by section 3, we generalize the above approach and derive flow equations for the following four series expansions 5 As before, sums whose range is not specified cover the whole state space. (with dynamic time variable τ ∈ [t0, t]): Master equations and the theory of stochastic path integrals 19

X X n could describe the position of a molecular motor |nip(τ, n|t , n ) , (41) hn|p(τ, n|t , n ) , (43) 0 0 0 0 along a cytoskeletal filament (n ∈ Z), the copy n n number of a molecule (n ∈ ), the local copy X X N0 p(t, n|τ, n0)|n0i , (42) p(t, n|τ, n0)hn0| . (44) numbers of the molecule on a lattice (n ≡ {ni ∈ n n 0 0 N0}i∈Z), or the copy numbers of multiple kinds Apparently, the series (41) coincides with the A B C 3 of molecules (n ≡ (n , n , n ) ∈ N0). For generalized probability generating function (33). In the multivariate configurations, the basis function is the next section 2.1, we formulate general conditions typically decomposed into a product |n1i|n2i · · · of under which this function obeys a linear PDE. individual basis functions |nii, each depending on its The remaining series (42)–(44) may not be as own variable qi. A process with spatial degrees of familiar. We call the series (42) a “marginalized freedom is considered in section 4.5.2. Besides, we also distribution”. It will be shown in section 3 that consider a system of excluding particles in section 2.2.4. this series does not only solve the (forward) master There, the (local) number of particles n is restricted to equation, but that it also obeys a backward-time the values 0 and 1. Note that the sum in (45) extends PDE under certain conditions. The marginalized over the whole state space. distribution proves useful in the computation of mean The definition of the generating function (45) extinction times as we demonstrate in section 3.3. In assumes the existence of a set {|niτ } of basis functions section 3.4, we consider the “probability generating for every time τ ∈ [t0, t]. In addition, we assume that functional” (43). For a “Poisson basis function”, the there exists a set {hm|τ } of linear basis functionals for inverse transformation, which maps this functional every time τ ∈ [t0, t]. These bras shall be orthogonal to the conditional probability distribution, coincides to the kets in the sense that at each time point τ, they with the Poisson representation of Gardiner and act on the kets as Chaturvedi [32,33]. The potential use of the series (44) hm| |ni = δ (O) remains to be explored. τ τ m,n The goal of the subsequent sections 4 and 5 lies (for all m and n). Here we note the possible time- in the solution of the derived flow equations by path dependence of the basis because the (O)rthogonality integrals. In section 4, we first solve the flow equations condition will only be required for equal times of the obeyed by the marginalized distribution (42) and by bras and kets. In addition to orthogonality, the basis the generating functional (43) in terms of a “backward” shall fulfil the (C)ompleteness condition path integral. Afterwards, in section 5, the flow X |ni hn| f = f , (C) equations obeyed by the generating function (41) and τ τ by the series expansion (44) are solved in terms of n a “forward” path integral. Inverse transformations, where f represents an appropriate test function. The such as (37), will then provide distinct path integral completeness condition implies that the function f representations of the forward and backward master can be decomposed in the basis functions |ni with equations. expansion coefficients hn|f. In the introduction to this section, we introduced various bases fulfilling both the orthogonality and the completeness condition. As in 2.1. Flow of the generating function the introductory example, the orthogonality condition We now formulate general conditions under which the allows one to recover the conditional probability forward master equation (10) can be cast into a linear distribution via the inverse transformation PDE obeyed by the generalized probability generating p(τ, n|t , n ) = hn|g(τ|t , n )i . (48) function 0 0 0 0 X Before deriving the flow equation obeyed by |g(τ|t0, n0)i ·= |ni p(τ, n|t0, n0) . (45) q τ,q the generating function, let us note that the n (O)rthogonality condition differs slightly from the The basis function |ni is a function of the variable q τ,q corresponding conditions used in most other texts on and possibly of the time variable τ. But unless one of stochastic path integrals (see, for example, [13, 15] or these variables is of direct relevance, its corresponding the “exclusive scalar product” in [10]). Typically, the subscript will be dropped. The explicit form of the orthogonality condition includes an additional factorial basis function depends on the problem at hand and n! on its right hand side. The inclusion of this factorial is chosen so that the four conditions (O), (C), (E), is advantageous in that it accentuates a symmetry and (Q) below are satisfied (the conditions (O) and (C) between the bases that we consider in sections 2.2.2 concern the orthogonality and completeness of the and 3.2.2 for the study of chemical reactions. Its basis, which we already required in the introductory inclusion would be rather unusual, however, for the example). The variable n again represents some Fourier basis introduced in sections 2.2.1 and 3.2.1. state from a countable state space. For example, Master equations and the theory of stochastic path integrals 20

The Fourier basis will be used to solve a simple only speak of “operators” with respect to differential random walk. Moreover, the factorial obscures operators, but not with respect to matrices). Just a connection between the probability generating like the basis (E)volution operator, Qτ (q, ∂q) should functional introduced in section 3.4 and the Poisson be polynomial in ∂q. In section 5, it will be assumed representation of Gardiner and Chaturvedi [32,33]. We that Qτ (q, ∂q) can be expanded in powers of both q and discuss this connection in section 3.5. ∂q. In a constructive approach, one could also define To derive the flow equation obeyed by the gener- the operator as ating function |gi, we differentiate its definition (45) X Q ·= |miQ (m, n)hn| . (53) with respect to the time variable τ. The resulting time τ · τ m,n derivative of the conditional probability distribution This constructive definition does not guarantee, p(τ, n|t0, n0) can be replaced by the right-hand side of the forward master equation (10). In matrix notation, however, that Qτ (q, ∂q) has the form of a differential operator. This property is, for example, not this equation reads ∂τ p(τ|t0) = Qτ p(τ|t0). Eventually, one finds that immediately clear for the Fourier basis function |niq = X X  einq (with n ∈ Z), which we complemented with the ∂ |gi = ∂ |ni + |miQ (m, n) p(τ, n|·) . (49) π τ τ τ functional hn|f = dq e−inqf(q) in the introduction n m −π 2π to this section. Most´ of the processes that we solve Our goal is to turn this expression into a partial in later sections have polynomial transition rates. differential equation for |gi. For this purpose, we Suitable bases and operators for these processes are require two differential operators. First, we require provided in the next section 2.2. It remains an a differential operator Eτ (q, ∂q) encoding the time open problem for the field to find such bases and evolution of the basis function. In particular, this operators for processes whose transition rates have operator should fulfil, for all values of n, different functional forms. That is, for example, the Eτ |ni = ∂τ |ni . (E) case for transition rates that saturate with the number of particles and have the form of a Hill function. By the (O)rthogonality condition, one could also define Provided that one has found a transition op- this operator in a “constructive” way as erator Qτ and a basis (E)volution operator Eτ for X
Eτ ·= ∂τ |ni hn| . (51) a (C)omplete and (O)rthogonal basis, it follows n from (49) that the generalized generating function 6 We call Eτ the basis (E)volution operator. In order |g(τ|t0, n0)i obeys the flow equation to arrive at a proper PDE for |gi, Eτ (q, ∂q) should ∂τ |gi = (Eτ + Qτ )|gi =· Qeτ |gi . (54) be polynomial in ∂q (later, in section 2.2.4 we also encounter a case in which it constitutes a power series Its initial condition reads |g(t0|t0, n0)i = |n0i, with the possibly time-dependent basis function |n i being with infinitely high powers of ∂q). For now, the pre- 0 2 evaluated at time t0. Although time-dependent bases factors of 1, ∂q, ∂q ,. . . may be arbitrary functions of q. Later, in our derivation of a path integral in prove useful in section 7, we mostly work with time- section 5, we will also require that the pre-factors independent bases in the following. The operators Qeτ can be expanded in powers of q. Note that for and Qτ then agree because Eτ is zero. Therefore, we a multivariate configuration n ≡ (nA, nB,...), the refer to both Qeτ and Qτ as transition (rate) operators. In our above derivation, we assumed that the ket derivative ∂q represents individual derivatives with respect to q ≡ (qA, qB,...). |ni represents an ordinary function and that the bra The actual dynamics of a jump process are hn| represents a linear functional. To understand why we chose similar letters for the Q -matrix and the Q - encoded by its transition rate matrix Qτ (see τ τ section 1.3). The off-diagonal elements of this matrix operator, it is insightful to consider the unit column vectors |ni ·= eˆ and the unit row vectors hn| ·= eˆ| are the transition rates wτ (n, m) from a state m to · n · n a state n, and its diagonal elements are the negatives (with m, n ∈ N0). For these vectors, the right hand P side of the transition operator (53) simply constitutes of the exit rates wτ (m) = n wτ (n, m) from a state a representation of the Q -matrix. Hence, Q is not m (with wτ (m, m) = 0; see (9)). We encode the τ τ a differential operator in this case but coincides with information stored in Qτ by a second differential the Q -matrix. This observation does not come as a operator called Qτ (q, ∂q). This operator should fulfil, τ for all values of n, 6 Later, in our derivation of the forward path integral X in section 5.1, we employ the finite difference approximation Q |ni = |miQ (m, n) . (Q) τ τ   m lim |g(τ + ∆t|·)i − |g(τ|·)i /∆t = lim Qeτ,∆t|g(τ|·)i . ∆t→0 ∆t→0 In analogy with the transition (rate) matrix Qτ , we This scheme conforms with the derivation of the forward master call Qτ the transition (rate) operator (note that we equation (10). Master equations and the theory of stochastic path integrals 21 surprise because we already noted that the generating An appropriate choice of the orthogonal and function (45) represents the vector p(τ|t0, n0) of all complete basis proves to be the time-independent probabilities in this case. Moreover, the corresponding Fourier basis flow equation (54) does not constitute a linear PDE π dq |ni ·= einq and hn|f ·= e−inqf(q) (56) but a vector representation of the forward master q · · ˆ−π 2π equation (10). with n ∈ Z and test function f. For this Following Doi, the transition operator Qeτ could basis, the generalized generating function |gi = be called a “time evolution” or “Liouville” operator [6, P inq inq n e p(τ, n|·) = he i coincides with the character- 7], or, following Zel’dovich and Ovchinnikov, a istic function. Moreover, the corresponding orthogon- “Hamiltonian” [8]. The latter name is due to ality condition hm|ni = δm,n agrees with a common the formal resemblance of the flow equation (54) to integral representation of the Kronecker delta. The the Schrödinger equation in quantum mechanics [92] completeness of the basis can be shown with the help (this resemblance holds for any linear PDE that is of a Fourier series representation of the “Dirac comb” first order in ∂ ). The name “Hamiltonian” has τ X 1 X gained in popularity throughout the recent years, X(q) = δ(q − 2πn) = einq . (57) 2π possibly because the path integrals derived later in n∈Z n∈Z this review share many formal similarities with the It thereby follows that path integrals employed in quantum mechanics [107, π X 0 0 0 285–287]. Nevertheless, let us point out that Qeτ |niqhn|f = dq X(q − q )f(q ) = f(q) . (58) ˆ−π is generally not Hermitian and that the generating n∈Z function |gi does not represent a wave function and also Since the Fourier basis function is time-independent, not a probability (unless one chooses the unit vectors the condition (E) is trivially fulfilled for the evolution | · |ni = eˆn and hn| = eˆn as basis). In this review, we operator Eτ ·= 0. The only piece still missing is the stick to the name transition (rate) operator for Qeτ (and transition operator Qτ . Its defining condition (Q) Qτ ) to emphasize its connection to the transition (rate) requires knowledge of the transition matrix Qτ whose matrix Qτ . elements

Qτ (m, n) (59) 2.2. Bases for particular stochastic processes = rτ (δm,n+1 − δm,n) + lτ (δm,n−1 − δm,n) In the previous section, we formulated four conditions can be inferred by comparing the master equation (55) under which the master equation (10) can be cast with its general form (10). The condition (Q) therefore into the linear PDE (54) obeyed by the generalized reads generating function (45). In the following, we illustrate Q |ni = r |n + 1i − |ni
+ l |n − 1i − |ni
. (60) how these conditions can be met for various stochastic τ τ τ processes. One can construct a transition operator with this property with the help of the functions c(q) ·= eiq and · −iq 2.2.1. Random walks. A simple process that can be a(q) ·= e . The function c shifts the basis function inq solved by the method from the previous section is the |niq = e to the right via c|ni = |n + 1i, and the one-dimensional random walk. We model this process function a shifts it to the left via a|ni = |n − 1i. Thus, in terms of a particle sitting at position n of the one- an operator with the property (60) can be defined as     dimensional lattice L ·= Z. The particle may jump to Qτ (q, ∂q) ·= rτ c(q) − 1 + lτ a(q) − 1 . (61) the nearest lattice site on its right with the (possibly This operator can also be inferred from its constructive time-dependent) rate rτ > 0, and to the site on its left definition (53) by making use of the Dirac comb. with the rate lτ > 0. Given that the particle was at After putting the above pieces together, one finds position n0 at time t0, the probability of finding it at that the generating function |g(τ|t0, n0)iq obeys the position n at time τ obeys the master equation flow equation   ∂τ p(τ, n|t0, n0) = rτ p(τ, n − 1|·) − p(τ, n|·) (55)  iq −iq  ∂τ |gi = rτ (e − 1) + lτ (e − 1) |gi (62)   + lτ p(τ, n + 1|·) − p(τ, n|·) , for t0 ≤ τ ≤ t with the initial value |g(t0|t0, n0)i = |n i = ein0q. The flow equation is readily solved for |gi with initial condition p(t0, n|t0, n0) = δn,n0 . One 0 can solve this equation by solving the associated flow by τ τ equation (54). But for this purpose, we first require an  iq −iq  (O)rthogonal and (C)omplete basis, as well as a basis exp (e − 1) ds rs + (e − 1) ds ls |n0i . ˆt ˆt (E)volution operator E and a transition operator Q 0 0 τ τ The conditional probability distribution can be (condition (Q)). recovered from this generating function by performing Master equations and the theory of stochastic path integrals 22 the inverse Fourier transformation (48), i.e. by decomposition of a process into elementary reactions evaluating p(τ, n|t0, n0) = hn|gi. A series expansion of the form k A → l A is also possible in the of all of the involved exponentials and some laborious contexts of growing polymer crystals [13], aggregation rearrangement of sums eventually result in a Skellam phenomena [294], and predator-prey ecosystems [163]. distribution as the solution of the process [329]. As explained in section 1.5, the transition rate We outline the derivation of this distribution in wτ (m, n) = γτ δm,n−k+l(n)k of the reaction k A → appendix C. The distribution’s mean is µ = n0 + l A is determined by combinatorial counting. Its τ 2 τ ds (rs − ls) and its variance σ = ds (rs + ls). proportionality to the falling factorial (n)k = n(n − t0 t0 These´ moments also follow from the´ fact that the 1) ··· (n − k + 1) derives from picking k molecules out Skellam distribution describes the difference of two of a population of n molecules. The overall time scale Poisson random variables; one for jumps to the right, of the reaction is set by the rate coefficient γτ (this the other for jumps to the left. The two moments can, coefficient may also absorb a factorial k! accounting also be obtained more easily by deriving equations for for the indistinguishability of molecules). The above their time evolution from the master equation. Those transition rate guarantees that the number of molecules equations do not couple for the simple random walk. (or “particles”) in the system never drops below zero, Let us also note that if the two jump rates rτ and lτ provided that it was non-negative initially. Thus, the agree, the Skellam distribution tends to a Gaussian for state space of n is N0 and we require basis functions large times. |ni and basis functionals hn| only for such values. Before specifying the (C)omplete and (O)rthogonal 2.2.2. Chemical reactions. As a second example, basis as well as the basis (E)volution operator Eτ , we turn to processes whose transition rates can be let us first specify an appropriate transition operator decomposed additively into the transition rates of Qτ . Its corresponding condition (Q) depends on the simple chemical reactions. In such a reaction, k1, transition matrix Qτ , whose elements Qτ (m, n) = k2, . . . molecules of types A1, A2, . . . come together to γτ (n)k(δm,n−k+l − δm,n) can be inferred from the rate be replaced by l1, l2, . . . molecules of the same types wτ (m, n) with the help of the relation (9). Con- sequently, the condition (Q) reads (with kj, lj ∈ N0). Besides reacting with each other,
the molecules could also diffuse in space, which can be Qτ |ni = γτ (n)k |n − k + li − |ni . (63) modelled in terms of hopping processes on a regular, This condition is met by the transition operator d-dimensional lattice such as ·= d. Upon labelling L Z l k k particles on different lattice sites by their positions, the Qτ (c, a) ·= γτ (c − c )a , (64) hopping of a molecule of type A1 from lattice site i ∈ L provided that there exist a “creation” operator c and to lattice site j ∈ L could be regarded as the chemical an “annihilation” operator a acting as (i) (j) reaction A1 → A1 . We consider the “chemical” c|ni = |n + 1i and (65) master equation associated to such hopping processes in section 4.5.2. a|ni = n|n − 1i . (66) To demonstrate the generating function approach The second of these relations ensures that basis from section 2.1, we focus on a system with only functions with n < 0 do not appear because a|0i a single type of molecule A engaging in the “well- vanishes. Both c and a may depend on time, just as mixed” reaction k A → l A (k, l ∈ N0). Since the the basis functionals and functions hn| and |ni may do. forward master equation (10) is linear in the transition It follows from (65) and (66) that the two operators rate Qτ (n, m), the following considerations readily fulfil the commutation relation extend to networks of multiple reactions, multiple [aτ , cτ ] ·= aτ cτ − cτ aτ = 1 (67) types of molecules, and processes with spatial degrees of freedom. The basis functions and functionals at a fixed time τ. The commutation relation is introduced below can, for example, be used to study meant with respect to functions that can be expanded branching and annihilating random walks, which are in the basis functions yet to be defined. For commonly modelled in terms of diffusing particles that brevity, we commonly drop the subscript τ of the engage in the binary annihilation reaction 2 A → ∅ and creation and annihilation operators. Together with the linear growth process A → (1 + m)A [96, 132, 317, the orthogonality condition, the commutation relation 330, 331]. For an odd number of offspring, these walks implies that the operators act on the basis functionals exhibit an absorbing state phase transition falling into as the universality class of directed percolation (according hn|c = hn − 1| and (68) to perturbative calculations in one and two spatial hn|a = (n + 1)hn + 1| . (69) dimensions [96, 132], according to non-perturbative calculations in up to six dimensions [318]). The In quantum mechanics, creation and annihilation operators prove useful in solving the equation of motion Master equations and the theory of stochastic path integrals 23 of a quantum particle in a quadratic potential (i.e. dependent functions and ζ 6= 0 is a free parameter. in solving the Schrödinger equation of the quantum This parameter only becomes relevant in section 6 in harmonic oscillator) [22, 332]. There, the ket |ni recovering a path integral representation of averaged is interpreted as carrying n quanta of energy ~ω observables. There, its value is set to ζ ·= i but in addition to the ground state energy ~ω/2 of the typically we choose ζ ·= 1. For the latter choice, the oscillator’s “vacuum state” |0i (with Dirac constant basis function (70) simplifies to ~ and angular frequency ω). The creation operator n −x˜(q+˜q) |niq = (q +q ˜) e . (71) then adds a quantum√ of energy to state |ni via the relation c|ni = n + 1|n + 1i, and the annihilation Alternatively, the parameter ζ could be used to rescale the variable q by a system size parameter N if such a operator√ removes an energy quantum via a|ni = n|n − 1i (a also destroys the vacuum state). These parameter is available. relations differ from the ones in (65) and (66) The two functions q˜ and x˜ may prove helpful in because in quantum mechanics, the creation and simplifying the flow equation obeyed by the generating · · annihilation operators are defined in terms of self- function. For example, we chose q˜ ·= 1 and x˜ ·= γ/µ adjoint position and momentum operators, forcing the in the introduction to section 2 to simplify the flow former operators to be hermitian adjoints of each equation of the reaction ∅ A (with growth rate other (i.e. c = a† and a = c†). Consequently, the coefficient γ and decay rate coefficient µ). Later, in † section 7, q˜ and x˜ will act as the stationary paths relations√ corresponding to√ (68) and (69) read hn|a = nhn − 1| and hn|a = n + 1hn + 1|. Nevertheless, of a path integral with q and an auxiliary variable x the commutation relation (67) also holds in the being deviations from them. If both q˜ and x˜ are chosen quantum world in which its validity ultimately derives as zero, the basis function (71) simplifies to the basis from the non-commutativity of the position operator Q function n and the momentum operator P (i.e. from [Q, P ] = i~, |niq = q (72) which follows from P being the generator of spatial of the ordinary probability generating function. displacements in state space [332]). Besides, let us note For the general basis function (70), creation that in quantum field theory, the creation operator and annihilation operators with the properties (65) is interpreted as adding a bosonic particle to an and (66) can be defined as energy state, and the annihilation operator as removing one [333]. c(q, ∂q) ·= ζq +q ˜ and (73) One may wonder whether the above interpreta- a(q, ∂q) ·= ∂ζq +x ˜ . (74) tions can be transferred to the theory of stochastic Apparently, these operators are not each other’s processes in which the creation and annihilation oper- n adjoints. For the basis function |niq = q of the ators need not be each other’s adjoints. For example, ordinary probability generating function, the operators the creation operator in (65) could be interpreted as simplify to c = q and a = ∂q. The corresponding adding a molecule to a system, and the corresponding l transition operator (64) reads Qτ (q, ∂q) = γτ (q − annihilation operator (66) as removing a molecule. The qk)∂k, resulting in the flow equation commutation relation (67) may then be interpreted as q l k k that the addition of a particle to the system (one way ∂τ |gi = γτ (q − q )∂q |gi . (75) to do it) and the removal of a particle (many ways to Thus far, we have not specified the basis do it) do not commute. Let us, however, note that functionals. These functionals can be defined these interpretations only apply to the processes dis- using the annihilation operator (74). In particular, cussed in the present section, which can be decomposed we complement the basis function (70) with the into reactions of the form k A → l A (with possibly functionals multiple types of particles and spatial degrees of free- n a hn|f ·= f(q) , (76) dom). The interpretations do not apply, for example, n! q=−q/ζ˜ to the random walk of a single particle with state space where f represents a test function. The (O)rthogonality , which we discussed in the previous section (there, Z of the basis {hm|, |ni} follows directly from the the “shift operators” with actions c|ni = |n + 1i and m,n∈N0 action of the annihilation operator on the basis func- a|ni = |n − 1i commute). tion |ni, and from the vanishing of this function at Whether creation and annihilation operators with q = −q/ζ˜ , except for n = 0. The (C)ompleteness of the properties (65) and (66) exist depends on the choice the basis is also readily established.7 of the basis functions. For the study of chemical The only piece still missing is the basis (E)volution reactions, a useful choice proves to be the basis function operator Eτ to encode the time-dependence of the basis · n −x˜(ζq+˜q) |niq ·= (ζq +q ˜) e . (70) 7 n n x˜(ζq+˜q) Use (∂q + ζx˜) f(q)|q=−q/ζ˜ = ∂q (e f(q))|q=−q/ζ˜ and Here, q˜(τ) and x˜(τ) are arbitrary, possibly time- the analyticity of ex˜(ζq+˜q)f(q) for an analytic test function f. Master equations and the theory of stochastic path integrals 24 function (70). Differentiation of this function with with n ∈ N0. Hermite polynomials constitute an respect to time and use of the creation and annihilation Appell sequence, i.e. they fulfil ∂qHen(q) = nHen−1(q) operators (73) and (74) show that this operator is given (18.9.27 in [335]). With a ·= ∂q, this property coincides by with the defining relation a|ni = n|n − 1i of the annihilation operator in (66). Furthermore, Hermite Eτ (c, a) ·= (∂τ q˜)(a − x˜) − (∂τ x˜)c . (77) polynomials obey the recurrence relation He (q) = Let us illustrate the use of the above basis for a n+1 qHe (q)−nHe (q) (18.9.1 in [335]). Combined with process whose transition operator can be reduced to n n−1 the Appell property, c ·= q − ∂ therefore fulfils the a mere constant by an appropriate choice of q˜ and · q defining relation c|ni = |n + 1i of a creation operator x˜. This simplification is, however, bought by making in (65). After complementing the basis function (81) the basis functions time-dependent. In particular, with the functional we consider the simple growth process ∅ → A for a ∞ −q2/2 time-independent growth rate coefficient γ. By our 1 e hm|f ·= dq √ Hem(q)f(q) , (82) previous discussions, one can readily verify that the m! ˆ−∞ 2π ordinary probability generating function with basis the (O)rthogonality and (C)ompleteness of the basis n function |niq = q obeys the flow equation ∂τ |gi = are also established (18.3 and 18.18.6 in [335]). Thus, γ(q − 1)|gi for this process. This flow equation can the chemical master equation (24) can be transformed be simplified by redefining the basis function of the into a flow equation obeyed by a generating function · n −x˜(q+1) generating function as |niτ,q ·= (q + 1) e , with based on Hermite polynomials. To our knowledge, x˜ solving the rate equation ∂τ x˜ = γ of the process. however, no stochastic process has thus far been solved Hence, the basis function depends explicitly on time or been approximated along these lines. Besides through its dependence on x˜(τ) =x ˜(t0) + γ(τ − Hermite polynomials, Ohkubo also proposed the use t0). Upon combining the transition operator (64) of Charlier polynomials and mentioned their relation with the basis evolution operator (77), one finds that with certain birth-death processes [334]. the generating function now obeys the flow equation ∂τ |gi = −γ|gi. The equation is readily solved 2.2.3. Intermezzo: The unit vector basis. One −γ(τ−t0) by |g(τ|t0, n0)i = e |n0i . The conditional t0 may wonder why we actually bother with explicit probability distribution can be recovered via the representations of the bras hn| and kets |ni. Often, inverse transformation (48) as these objects are introduced only formally as the basis p(τ, n|t , n ) = e−γ(τ−t0)hn| |n i . (78) 0 0 τ 0 t0 of a “bosonic Fock space” [6, 7, 91, 289, 336], leaving the impression that the particles under consideration The coefficient hn|τ |n0it can be evaluated by 0 are in fact bosonic quantum particles. Although this determining how the functional hn|τ acts at time t0. impression takes the analogy with quantum theory too Using aτ −x˜(τ) = at0 −x˜(t0) and the binomial theorem, one can rewrite the action of this functional as far, the analogy has helped in developing new methods n
n−m for solving the master equations. In the previous X γ(τ − t0) hn| f = hm| f . (79) sections, we showed how the forward master equation τ (n − m)! t0 m=0 can be cast into a linear PDE obeyed by a probability By the (O)rthogonality condition, the conditional generating function. Later, in section 5, this PDE probability distribution (78) therefore evaluates to the is solved in terms of a path integral by which we shifted Poisson distribution then recover a path integral representation of averaged n−n −γ(τ−t0)
0 observables in section 6. This “analytic” derivation of e γ(τ − t0) p(τ, n|t0, n0) = Θn−n , (80) the path integral is, however, not the only possible way. (n − n )! 0 0 In the following, we sketch the mathematical basis of an · · where Θn ·= 1 for n ≥ 0 and Θn ·= 0 otherwise. The alternative approach, which employs the unit vectors validity of this solution can be verified by solving the eˆn as the basis. The approach ultimately results corresponding flow equation of the ordinary generating in the same path integral representation of averaged function. observables as we show in section 6.2. The duality In our above approach, we first established the between the two approaches resembles the duality form of the transition operator because the basis between the matrix mechanics formulation of quantum function (70) is not the only possible choice. Ohkubo mechanics by Heisenberg, Born, and Jordan [93–95] recently proposed the use of orthogonal polynomials and its analytic formulation in terms of the Schrödinger for this purpose [334]. In analogy to the eigenfunctions equation [92]. of the quantum harmonic oscillator [22, 332], one can, The alternative derivation of the path integral for example, choose the basis function as the Hermite also starts out from the forward master equation polynomial ∂τ p(τ|t0) = Qp(τ|t0). As before, p(τ|t0) represents the · n q2/2 n −q2/2 |niq ·= Hen(q) = (−1) e ∂q e , (81) matrix of the conditional probabilities p(τ, n|t0, n0), Master equations and the theory of stochastic path integrals 25 and Q is the transition rate matrix. For simplicity, we the transition operator (64). Both of them are normal- assume the transition rate matrix to be independent ordered polynomials in c and a. This property will of time. Moreover, we focus on processes that help us in section 6.2 in recovering a path integral can be decomposed additively into chemical reactions representation of averaged observables. of the form k A → l A with time-independent rate coefficients. The elements of the transition 2.2.4. Processes with locally excluding particles. As matrix associated to this reaction read Q(m, n) = noted in the introductory section 1.3, one can model γ(n)k(δm,n−k+l − δm,n). Since the particle numbers the movement of molecular motors along a cytoskeletal n and n0 may assume any values in N0, the probability filament in terms of a master equation [113–115]. matrix p(τ|t0) and the transition matrix Q have In its simplest form, such a model describes the infinitely many rows and columns. movement of mutually excluding motors as a hopping In the introductory section 1.4, we formulated process on a one-dimensional lattice L ⊂ Z, with conditions under which the forward master equation is attachment and detachment of motors at certain Q(τ−t0) solved by the matrix exponential p(τ|t0) = e 1 boundaries. To respect the mutual exclusion of motors, (in the sense of the expansion (16), the state space their local number ni on a lattice site i ∈ L is must countable and the exit rates bounded). For restricted to 0 and 1. Various other processes can the above chemical reaction, those conditions are not be modelled in similar ways, for example, aggregation necessarily met, and thus the matrix exponential may processes [26], adsorption processes [337, 338], and Q(τ−t0) not exist. Nevertheless, we regard p(τ|t0) = e 1 directed percolation [29]. In the following, we as a “formal” solution in the following. With the help show how the generating function approach from of certain mathematical “tricks”, this solution will be section 2.1 and its complementary approach from cast into a path integral in section 6. But before, section 2.2.3 can be applied to such processes. To let us rewrite the transition matrix in the exponential illustrate the mathematics behind these approaches in terms of creation and annihilation matrices. For while not burdening ourselves with too many indices, this purpose, we employ the (infinitely large) unit we demonstrate the approaches for the simple, non- column vectors |ni ·= eˆn and their orthogonal unit row spatial telegraph process [339, 340]. This process · | vectors hn| ·= eˆn as basis (with n ∈ N0). Individual describes a system that randomly switches between probabilities can therefore be inferred from the above two states that are called the “on” and “off” states, or, solution as for brevity, the “1” and “0” states. The rate at which Q(τ−t0) the system switches from state 1 to state 0 is denoted p(τ, n|t0, n0) = hn|e |n0i . (83) as µ, and the rate of the reverse transition as γ. For For multivariate or spatial processes, the basis vectors simplicity, we assume these rate coefficients to be time- can be generalized to tensors (i.e. |ni = |n1i ⊗ |n2i ⊗ independent. Consequently, the master equation of the ...). telegraph process reads The orthogonality of the basis allows us to rewrite  −µ γ  the elements of the transition rate matrix of the ∂τ p = p , (86) reaction k A → l A as µ −γ 
 with p(τ|·) = (p(τ, 1|·), p(τ, 0|·))| being the probability Q(m, n) = hm| γ (n)k |n − k + li − |ni . (84) vector. Alternatively, the master equation can be P A matrix with these elements can be written as written as ∂τ p(τ, n|·) = m∈{1,0} Q(n, m)p(τ, m|·) Q = γ(cl − ck)ak , (85) with transition matrix elements with c acting as a “creation matrix” and a acting as Q(n, m) = µ(δn,0 − δn,1)δm,1 + γ(δn,1 − δn,0)δm,0 . (87) an “annihilation matrix”. In particular, we define c in One can solve the above master equation in various terms of its sub-diagonal (1, 1, 1,...) and a in terms ways, for example, by evaluating the matrix expo- Q(τ−τ0) of its super-diagonal (1, 2, 3,...). All of their other nential in p(τ|τ0) = e 1 after diagonalizing the matrix elements are set to zero. It is readily shown transition matrix. In the following, the telegraph pro- that these matrices fulfil c|ni = |n + 1i and a|ni = cess serves as the simplest representative of processes n|n − 1i with respect to the basis vectors. These with excluding particles and it helps in explaining how relations coincide with our previous relations (65) the methods from the previous sections are applied to and (66). Besides, the matrices also fulfil hn|c = such processes. The inclusion of spatial degrees of free- hn − 1| and hn|a = (n + 1)hn + 1|, as well as the dom into the procedure is straightforward. commutation relation [a, c] = 1. Further properties To apply the generating function technique from of the matrices are addressed in section 6.2. By our section 2.1, we require orthogonal and complete basis previous comments in section 2.1, it is not a surprise functions {|0i, |1i} and basis functionals {h0|, h1|}, that the transition matrix (85) has the same form as Master equations and the theory of stochastic path integrals 26 as well as a transition operator Q(q, ∂q) fulfilling not fulfilled by c ·= q and a ·= ∂q, at least not if q condition (Q), i.e., for n ∈ {0, 1}, represents an ordinary real variable. However, ∂qq + Q|ni = µ|0i − |1i)δ + γ|1i − |0i)δ . (88) q∂q = 1 holds true if q represents a Grassmann variable n,1 n,0 (see [341, 342] for details). Grassmann variables An operator with this property can be constructed in commute with real and complex numbers (i.e. [q, α] = 0 (at least) two ways: using creation and annihilation for α ∈ C), but they anti-commute with themselves operators fulfilling the “bosonic” commutation relation and with other Grassmann variables (i.e. {q, q˜} = 0). [a, c] = ac − ca = 1, or using operators fulfilling Grassmann variables have, for example, proven useful the “fermionic” anti-commutation relation {a, c} ·= in calculating the correlation functions of kinetic Ising ac + ca = 1. models [343], even when the system is driven far from Let us first outline an approach based on the equilibrium [344]. commutation relation. The approach was recently Instead of using Grassmann variables for the basis, proposed by van Wijland [90]. To illustrate the let us consider a representation of the basis defined by approach, we take the “analytic” point of view the unit row vectors h0| = (0, 1) and h1| = (1, 0), and with c(q, ∂q) and a(q, ∂q) being differential operators. by the unit column vectors |0i = (0, 1)| and |1i = However, the following considerations also apply to (1, 0)|. Obviously, these vectors fulfil the orthogonality the approach from the previous section where c and a P 1 condition hm|ni = δm,n, and n |nihn| = is the represented creation and annihilation matrices. To cast 2-by-2 unit matrix. The anti-commutation relation a master equation such as (86) into a path integral, van {a, c} = 1 is met by the creation and annihilation Wijland employed creation and annihilation operators matrices with the same actions as in section 2.2.2, i.e. c|ni =  0 1   0 0  c ·= σ+ = and a ·= σ− = . (91) |n + 1i and a|ni = n|n − 1i. Besides complying with · 0 0 · 1 0 the commutation relation [a, c] = 1, these relations imply that the “number operator” N ·= ca fulfils The creation matrix acts as c|0i = |1i and c|1i = 0, N |ni = n|ni. This operator may be used to define and the annihilation matrix as a|1i = |0i and a|0i = 0 the “Kronecker operator” (zero vector). Consequently, the matrix product ca π fulfils ca|0i = 0 and ca|1i = |1i, and thus it serves du iu(N −m) δN ,m ·= e (89) the same purpose as the Kronecker delta δn,1 in (88). ˆ−π 2π Analogously, the operator ac serves the same purpose in terms of a series expansion of its exponential. The as δn,0. The master equation (86) can therefore be Kronecker operator acts on the basis functions as written in a form resembling the flow equation (90), δN ,m|ni = δn,m|ni, and thus it can be used to replace namely as the Kronecker deltas in (88). One thereby arrives at ∂ p = µ(a − 1)ca + γ(c − 1)acp . (92) the flow equation τ In this form, the stochastic process mimics a spin-1/2 ∂ |gi = Q|gi = µ(a − 1)δ + γ(c − 1)δ |gi . (90) τ N ,1 N ,0 problem (especially, if the creation and annihilation On the downside, the flow equation now involves power matrices are written in terms of the Pauli spin series with arbitrarily high derivatives with respect to q matrices σx, σy, and σz). More information on (upon choosing the operators as c = q and a = ∂q). In spin-representations of master equations is provided section 5, we show how the solution of a flow equation in [66, 337, 338, 345–348]. A spin-representation has, such as (90) can be expressed in terms of a path for example, been employed in the analysis of reaction- integral. The derivation of the path integral requires diffusion master equations via the density matrix that the transition operator Q(q, ∂q) is normal-ordered renormalization group [349, 350]. In their study of with respect to q and ∂q, i.e. that all the q are to directed percolation of excluding particles on the the left of all the ∂q in every summand. This order one-dimensional lattice Z, Brunel, Oerding, and van can be achieved by the repeated use of [∂q, q]f(q) = Wijland performed a Jordan-Wigner transformation of · + f(q). More information on the procedure can be found the spatially extended “spin” matrices ci ·= σi and · − in [90]. Van Wijland applied the resulting path integral ai ·= σi (i ∈ Z) [29]. The transformed operators to the asymmetric diffusion of excluding particles on a fulfill anti-commutation relations not only locally but one-dimensional lattice. Moreover, Mobilia, Georgiev, also non-locally, and thus represent the stochastic and Täuber have employed the method in studying process in terms of a “fermionic” (field) theory. the stochastic Lotka-Volterra model on d-dimensional While the Jordan-Wigner transformation provides an lattices [153]. exact reformulation of the stochastic process, its An alternative to the above approach lies in applicability is largely limited to systems with one the use of operators fulfilling the anti-commutation spatial dimension. Moreover, the transformation relation {a, c} = ac + ca = 1. This relation is clearly requires that the stochastic process is first rewritten Master equations and the theory of stochastic path integrals 27 in terms of a spin-1/2 chain (typically possible only method and affect its stability. Information on how for processes with a single species). Based on their values are chosen is provided in [34]. For our the (coherent) eigenstates of the resulting fermionic present purposes, the values of the parameters remain creation and annihilation operators, Brunel et al. then unspecified. By differentiating the generating function derived a path integral representation of averaged with respect to time, one finds that the generating observables. The paths of these integrals proceed function obeys the flow equation ∂τ |gi = (Q0 + Q1)|gi along the values of Grassmann variables. Further with the transition operators information on the Jordan-Wigner transformation, X Q ·= − µ q ∂ and (95) Grassmann path integrals, and alternative approaches 0 · i i qi can be found in [26–28, 30, 31, 351–353]. i∈{1,2} X Q1 ·= − qi(¯γi − γˆi) . (96) 2.3. Methods for the analysis of the generating i∈{1,2} function’s flow equation Here, the two new operators γˆ1 and γˆ2 are defined In the following, we outline various approaches that in terms of their actions γˆi|n1i = γi(n1)|n1i on the have recently been proposed for the analysis of the basis functions. Surprisingly, the explicit form of these generating function’s flow equation. operators is not needed. Thus, the spectral method even allows for non-polynomial growth rates γi(n1). The operator Q has the same form as the 2.3.1. A spectral method for the computation of sta- 0 transition operator of the bi-directional reaction ∅ tionary distributions. In their study of a linear tran- A from the introductory example. Without the scriptional regulatory cascade of genes and proteins, perturbation Q , the above flow equation could thus Walczak, Mugler, and Wiggins developed a spectral 1 be solved by extending the previous ansatz (38) to two method for the computation of stationary probability species. To accommodate Q as well, it proves useful distributions [34]. They described the regulatory cas- 1 to generalize that ansatz to cade on a coarse-grained level in terms of the copy num- X bers of certain chemical “species” at its individual steps. |g(τ|·)i = |kiiGk(τ|·) , (97) 2 By imposing a Markov approximation, the dynamics of k∈N0 the cascade was reduced to a succession of two-species with yet to be determined expansion coefficients master equations. The solution of each master equa- Gk(τ|·). The auxiliary ket is defined as |kiiq ·= tion served as input for the next equation downstream. |k ii |k ii with |k ii = qki and is orthogonal to Every of the reduced master equations allowed for the 1 q1 2 q2 i qi i · · 1 ki the bra hhk| ·= hhk1|hhk2| with hhki|f ·= ∂q f(qi)|qi=0. following processes: First, each of the master equa- ki! i These bras can be used to extract the expansion tion’s two species i ∈ {1, 2} is produced in a one-step coefficient via G (τ|·) = hhk|g(τ|·)i. Differentiation process whose rate γ (n ) depends only on the copy k i 1 of this coefficient with respect to time and imposing number n of the species coming earlier along the cas- 1 stationarity eventually results in a recurrence relation cade. Second, each of the two species degrades at a for G that can be solved iteratively. The steady- constant per-capita rate µ . With n ·= (n , n )| ∈ 2, k i · 1 2 N0 state probability distribution of the stochastic process the corresponding master equation reads is then recovered from the generating function via the ∂τ p(τ, n|·) = γ1(n1 − 1)p(τ, n − eˆ1|·) − γ1(n1)p(τ, n|·) inverse transformation (48).   + γ2(n1) p(τ, n − eˆ2|·) − p(τ, n|·) (93) In [34], Walczak et al. computed the steady- state distribution of the transcriptional regulatory + µ (n + 1)p(τ, n + eˆ |·) − n p(τ, n|·) 1 1 1 1 cascade by solving the recurrence relation for Gs   k + µ2 (n2 + 1)p(τ, n + eˆ2|·) − n2p(τ, n|·) . numerically. The resulting distribution was compared to distributions acquired via an iterative method Here, the unit vector eˆi points in the direction of the i-th species. The master equation can be cast into and via the stochastic simulation algorithm (SSA) a flow equation for the generating function |g(τ|·)i = of Gillespie. The spectral method was found to be P about 108 times faster than the SSA in achieving n p(τ, n|·)|ni by following the steps in section 2.1. For this purpose, we choose the basis function as a the same accuracy. But as we already mentioned multivariate extension of the basis function from the in the introductory section 1.4, the SSA generally introduction to section 2, i.e. as |ni ·= |n i |n i performs poorly in the estimation of full distributions, q · 1 q1 2 q2 with especially in the estimation of their tails. As the γ¯i spectral method has only been applied to cascades ni − µ (qi+1) |nii ·= (qi + 1) e i . (94) qi · with steady-state copy numbers below n ≈ 30 thus The values of the auxiliary parameters γ¯1 and γ¯2 are far, it may be challenged by a direct integration only specified in a numerical implementation of the of the two-species master equation (after introducing Master equations and the theory of stochastic path integrals 28 a reasonable cut-off in the copy numbers). The n∞ integration of ∼ 1000 coupled ODEs does not pose a problem for modern integrators and the integration 60 0.03 provides the full temporal dynamics of the process 0.02 (see [271] for efficient algorithms). In its current 40 formulation, the spectral method is limited to the Probability 0.01 16 evaluation of steady-state distributions and to simple τ = 10

Particle number 20 0 one-step birth-death dynamics. It would be interesting 0 20 40 60 80 100 120 to advance the method for the application to more Particle number n 0 complex processes (possibly with spatial degrees of 10−3 10 105 109 1013 1017 freedom), and to extend its scope to the temporal Time τ evolution of distributions. Figure 3. Comparison of the deterministic particle number 2.3.2. WKB approximations and related approaches. n¯(τ) (blue dashed line) and the mean particle number hni(τ) of The probability distribution describing the transcrip- the stochastic model (orange line) of the combined processes 2 A → ∅ and A → 2 A. The annihilation rate coefficient tional regulatory cascade from the previous section ap- was set to µ = 1, the growth rate coefficient to γ = 150. proaches a non-trivial stationary shape in the asymp- The deterministic trajectory started out from n¯(0) = 40, the totic time limit τ → ∞ (cf. figure 2 in [34]). Of- numerical integration of the master equation from p(0, n|0, 40) = ten, however, a non-trivial shape of the probability δn,40. The deterministic trajectory converged to n¯∞ = 75 for very large times, whereas the mean particle number approached distribution persists only transiently and is said to a quasi-stationary value close to n¯∞ before converging to the be quasi-stationary or metastable. The lifetime and absorbing state n = 0. The inset shows the conditional shape of such a distribution can often be approxim- probability distribution at time τ = 1016. At this time, a significant share of particles was already in the absorbing state ated in terms of a WKB approximation [84]. A WKB and the distribution was bimodal. approximation of a jump process starts out from an exponential (eikonal) ansatz for the shape of the meta- stable probability distribution (“real-space” approach) WKB approximation [357], and by Assaf and Meerson or for the generating function discussed in the pre- upon combining the generating function technique vious sections (“momentum-space” approach). In- with Sturm-Liouville theory [366] (the latter method formation on the real-space approach can be found was developed in [36, 37]; see also [367]). Using in [63,64,146,171,172,177,354–363]. The recent review this method, Assaf and Meerson also succeeded in of Assaf and Meerson provides an in-depth discussion computing the shape of the metastable distribution. of the applicability of the real- and momentum-space Moreover, Assaf et al. showed how a momentum-space approaches [364]. WKB approximation can be used to determine mean In the following, we outline the momentum-space extinction times for processes with time-modulated WKB approximation for a system in which particles rate coefficients [38]. of type A annihilate in the binary reaction 2 A → ∅ Upon rescaling time as γ τ → τ, the chemical with rate coefficient µ, and are replenished in the master equation (24) of the combined processes A → linear reaction A → 2 A with rate coefficient γ  µ. 2 A and 2 A → ∅ translates into the flow equation

According to a deterministic model of the combined h 2 1 2 2i 2 ∂τ |gi = (q − q)∂q + (1 − q )∂q |gi (98) processes with rate equation ∂τ n¯ = γn¯ − 2µn¯ , 2¯n∞ the particle number n¯ converges to an asymptotic of the ordinary generating function g(τ; q|·) = value n¯ = γ/(2µ)  1. However, a numerical ∞ P qnp(τ, n|·) (cf. sections 2.1 and 2.2.2). A WKB integration of the (truncated) master equation of n approximation of the flow equation can be performed the stochastic process shows that the mean particle by inserting the ansatz number stays close to n¯∞ only for a long but finite S(τ,q) time (cf. figure 3). Asymptotically, all particles become g(τ; q|·) = e (99) trapped in the “absorbing” state n = 0. Consequently, with the “action” the conditional probability distribution converges to ∞ X Sk(τ, q) p(∞, n|t0, n0) = δn,0. Up to a pre-exponential factor, S(τ, q) = 2n ¯ (100) ∞ n¯k the time after which the absorbing state is reached k=0 ∞ can be readily estimated using a momentum-space into the equation, followed by a successive analysis of WKB approximation as shown below [35]. The terms that are of the same order with respect to the value of the pre-exponential factor was determined power of the small parameter 1/n¯∞. Note that the by Turner and Malek-Mansour using a recurrence exponential ansatz (99) often includes a minus sign in relation [365], by Kessler and Shnerb using a real-space front of the action, which we neglect for convenience. Master equations and the theory of stochastic path integrals 29

along characteristic curves (the minus signs are only 0.6 included to emphasize a similarity with an action 0.5 encountered later in section 7.1). According to Elgart

x 0.4 and Kamenev, the (negative) value of the action (100) 0.3 along the optimal path to extinction determines the 0.2 logarithm of the mean extinction time τ¯ from the Variable metastable state in leading order of n¯ [35]. As this 0.1 ∞ path proceeds along a zero-energy line, the action (100) 0 evaluates to −0.1 0 q dq 0 0.2 0.4 0.6 0.8 1 1.2 S ≈ 2¯n∞S0 = 2¯n∞ = −2¯n∞(1 − ln 2) . (106) Variable q ˆ1 1 + q Upon returning to the original time scale via τ → γ τ, Figure 4. Phase portrait of Hamilton’s equations (103) the mean extinction time follows as and (104). The Hamiltonian H(q, x) in (102) vanishes along −1 2¯n (1−ln 2) “zero-energy” lines (orange dotted lines). These lines connect τ¯ = A γ e ∞ (107) fixed points of Hamilton’s equations (green disks). Hamilton’s with pre-exponential factor A. The value of this factor equations reduce to the (rescaled) rate equation ∂ x = x − 2x2 s has been determined in [357, 365, 366] and reads, in along the line connecting the fixed points (1, 0) and (1, 1/2). p The zero-energy line connecting the fixed point (1, 1/2) to (0, 0) terms of our rate coefficients, A = 2 π/n¯∞. More denotes the “optimal path to extinction” and can be used to information on the above procedure can be found approximate the mean extinction time of the process. in [35, 38]. The method has also been applied to classic epidemiological models [369] and to a variant The pre-factor “2” of the action (100) is also included of the Verhulst logistic model [370] (exact results on just for convenience. Upon inserting the ansatz (99) this model with added immigration have been obtained into the flow equation (98), one obtains the equality in [371] using the generating function technique). ∂ S + O(1/n¯ ) = H(q, ∂ S ) + O(1/n¯ ) (101) τ 0 ∞ q 0 ∞ 2.3.3. A variational method. The generating func- with the “Hamiltonian” tion’s flow equation can also be analysed using a vari- 2 2 2 ational method as proposed by Sasai and Wolynes [39, H(q, x) ·= (q − q)x + (1 − q )x . (102) 372]. Their approach is based on a method that Thus, at leading order of 1/n¯∞, we have obtained Eyink had previously developed (primarily) for Fokker- a closed equation for S0, which has the form of Planck equations [83]. Instead of dealing with flow a Hamilton-Jacobi equation [368]. A Hamilton- equation ∂τ |gi = Qeτ |gi of the generating function in Jacobi equation can be solved by the method of its differential form (or in terms of its spin matrix rep- characteristics [368], with the characteristic curves q(s) resentation section 2.2.4), the variational method in- and x(s) obeying Hamilton’s equations volves a functional variation of the “effective action” t L R R ∂H(q, x) 2 Γ = dτ hψ |(∂τ − Qeτ )|ψ i with |ψ i = |gi. To ∂ x = = (2q − 1)x − 2qx and (103) t0 s ∂q perform´ this variation, one requires an ansatz for the objects hψL| and |ψRi, being parametrized by the vari- ∂H(q, x) 2 2 ∂−sq = = (q − q) + 2(1 − q )x . (104) L R ∂x ables {αi }i=1,...,S and {αi }i=1,...,S. The values of these parameters are determined by requiring that hψL| These equations are, for example, solved by q(s) = is an extremum of the action. Besides its application 1 and x(s) being a solution of ∂ x = x − 2x2. s to networks of genetic switches [39], the variational Note that this equation corresponds to a rescaled method has been applied to signalling in enzymatic rate equation. The Hamiltonian vanishes along the cascades in [372]. An extension of the method to mul- characteristic curve because H(1, x) = 0. Further tivariate processes is described in [373]. Whether or “zero-energy” lines of the Hamiltonian are given by not the variational method provides useful information x = 0 and by x(q) = q . These lines partition the 1+q mainly depends on making the right ansatz for hψL| phase portrait of Hamilton’s equations into separate and |ψRi. The method itself does not suggest their regions as shown in figure 4. The path (q, q ) from 1+q choice. It remains to be seen whether the method can 1 the “active state” (q, x) = (1, 2 ) to the “passive state” be applied to processes for which only little is known (0, 0) constitutes the “optimal path to extinction” (see about the generating function. below) [35, 38, 369]. In addition to Hamilton’s equations, the method 2.4. Résumé of characteristics implies that the action S0 obeys d In the present section, we formulated general condi- S = −x ∂ q − H(q, x)
(105) ds 0 −s tions under which the forward master equation (10) can Master equations and the theory of stochastic path integrals 30 be transformed into a linear partial differential equa- easily derived from the backward master equation (12). tion, a “flow equation”, obeyed by the probability gen- The equation proves useful in the computation of erating function mean extinction times. Moreover, it provides a X most direct route to path integral representations |g(τ|t , n )i = |ni p(τ, n|t , n ) . (108) 0 0 0 0 of the conditional probability distribution and of n averaged observables. To our knowledge, the flow First, the conditions (C) and (O) require that there equation of the marginalized distribution has not exists a complete and orthogonal basis comprising been considered thus far. In section 3.4, we then a set of basis functions {|ni} and a set of basis introduce a probability generating “functional” whose functionals {hn|}. The basis functionals recover the flow equation is derived from the forward master conditional probability distribution via p(τ, n|t0, n0) = equation. The transformation mapping the functional hn|g(τ|t0, n0)i and the right choice of the basis to the conditional probability distribution is shown to functions may help to obtain a simplified flow equation. generalize the Poisson representation of Gardiner and We introduced different bases for the study of random Chaturvedi [32, 33]. walks, of chemical reactions, and of processes with locally excluding particles in section 2.2. Moreover, 3.1. Flow of the marginalized distribution the conditions (E) and (Q) require that there exist two differential operators: a basis evolution operator The generalized probability generating function (45) Eτ encoding the possible time-dependence of the basis, was defined by summing the conditional probability and a transition operator Qτ encoding the actual distribution p(τ, n|t0, n0) over a set of basis functions, dynamics of the process. The generating function (108) the kets {|ni}. In the following, we consider the then obeys the flow equation distribution p(t, n|τ, n0) instead, i.e. we fix the final time t while keeping the initial time τ variable. By ∂ |gi = (E + Q )|gi = Q |gi . (109) τ τ τ eτ summing this distribution over a set of basis functions Various methods have recently been proposed for the {|n0iτ }, one can define the series study of such a flow equation and were outlined in · X section 2.3. These methods include the variational |p(t, n|τ)i ·= p(t, n|τ, n0)|n0i . (110) approach of Eyink [83] and of Sasai and Wolynes [39], n0 WKB approximations and spectral formulations of As before, the subscript denoting the time-dependence Elgart and Kamenev and of Assaf and Meerson [35–37], of the basis function is typically dropped. The and the spectral method of Walczak, Mugler, and variables n and n0 again represent states from Wiggins [34]. Thus far, most of these methods have some countable state space. Assuming that the only been applied to systems without spatial degrees basis functions {|ni} and appropriately chosen basis of freedom and with only one or a few types of functionals {hn|} form a (C)omplete and (O)rthogonal particles. Future research is needed to overcome these basis, the conditional probability distribution can be limitations. Later, in section 5, we show how the recovered from (110) via solution of the flow equation (109) can be represented p(t, n|τ, n0) = hn0|p(t, n|τ)i . (111) by a path integral. The evaluation of the path integral We call the function |p(t, n|τ)i a “marginalized is demonstrated in computing the generating function distribution” because it proves most useful when the of general linear processes. Moreover, we explain in summation in (110) constitutes a marginalization of section 7 how this path integral connects to a recent the conditional probability distribution p(t, n|τ, n0) method of Elgart and Kamenev for the computation over a probability distribution |n0i. The marginalized of rare event probabilities [35]. One can also use the distribution then represents a “single-time distribution” path integral to derive a path integral representation with respect to the random variable n in the sense of averaged observables. But a simpler route to of section 1.3. A basis function to which these this representation starts out from a different flow considerations apply is the “Poisson basis function” equation, a flow equation obeyed by the “marginalized xn0 e−x |n0i ·= . We make heavy use of this x · n0! distribution”. basis function in the study of chemical reactions in section 3.2.2. Since the definition of the marginalized 3. The marginalized distribution and the distribution in (110) does not affect the random probability generating functional variable n, the marginalized distribution of course solves the forward master equation ∂t|p(t, n|τ)i = In the following, we discuss two further ways of P Qt(n, m)|p(t, m|τ)i with the initial condition casting the forward and backward master equations m |p(τ, n|τ)i = |n0iτ . In the following, we formulate into linear PDEs. The first of these flow equations conditions under which |pi also obeys a linear PDE is obeyed by a “marginalized distribution” and is evolving backward in time. Master equations and the theory of stochastic path integrals 31

Before proceeding, let us briefly note that if Provided that both a basis evolution operator Eτ † the basis function of the marginalized distribution is and an adjoint transition operator Qτ are found for not chosen as a probability distribution, the name a particular process, it follows from the backward- marginalized “distribution” is somewhat of a misnomer; time equation (112) that the marginalized distribution that is, for example, the case for the Fourier basis |p(t, n|τ)i obeys the flow equation8 |n i ·= ein0x, which we consider in section 3.2.1. 0 x · ∂ |pi = (−E + Q† )|pi =· Q† |pi . (116) The derivation of the linear PDE obeyed by the −τ τ τ · eτ marginalized distribution proceeds analogously to the The evolution of this equation proceeds backward in derivation in section 2.1. But instead of employing time, starting out from the final condition |p(t, n|t)i = the forward master equation, we now employ the |ni. backward master equation ∂−τ p(t|τ) = p(t|τ)Qτ In section 4, we show how the flow equation (116) for this purpose (recall that p(t|τ) is the matrix can be solved in terms of a “backward” path integral. with elements p(t, n|τ, n0)). Upon differentiating the Provided that the basis function |ni is chosen as a marginalized distribution (110) with respect to the probability distribution, this path integral represents time parameter τ, one obtains the equation a true probability distribution: the marginalized distribution. The fact that the backward path integral ∂−τ |pi (112)   represents a probability distribution distinguishes X X | = p(·|τ, n0) −∂τ |n0i + |miQτ (m, n0) . it from the “forward” path integral in section 5. n0 m The forward path integral represents the probability | The rate Qτ (m, n0) = Qτ (n0, m) represents an element generating function (45). Both the forward path | of the transposed transition matrix Qτ . integral and the backward path integral can be used As in section 2.1, two differential operators are to derive a path integral representation of averaged required to turn the above expression into a linear observables, as we show in section 6. The derivation of PDE. First, we require a basis evolution operator this representation from the backward path integral, Eτ (x, ∂x) fulfilling Eτ |n0i = ∂τ |n0i for all values of however, is significantly easier. In fact, the path n0. The previous evolution operator (E) serves this integral representation of the average of an observable † purpose. A second differential operator Qτ (x, ∂x) A will follow directly by summing the backward path is required to encode the information stored in the integral representation of the marginalized distribution P transition matrix. This operator should be a power over A(n), i.e. via hAi = n A(n)|p(t, n|τ)i. Note that series in ∂x and fulfil, for all n0, this average also obeys the flow equation (116). In X section 3.3, we demonstrate how the flow equation can Q† |n i = |miQ|(m, n ) . (Q|) τ 0 τ 0 be used to compute mean extinction times. m Before introducing bases for the analysis of By the (C)ompleteness of the basis, one could also different stochastic processes, let us briefly note that define this operator constructively as if a process is homogeneous in time, its marginalized X † | distribution depends only on the difference t − τ. The Qτ ·= |miQτ (m, n0)hn0| . (114) m,n0 above flow equation can then be rewritten so that it We wrote these expressions in terms of the transposed evolves |p(τ, n|0)i forward in time τ, starting out from transition matrix because for the unit column vectors the initial condition |p(0, n|0)i = |ni. In section 3.3, we | † make use of this property to compute mean extinction |mi = eˆm and the unit row vectors hn| = eˆn, Qτ | times. and Qτ coincide (with m, n ∈ N0). As we mostly consider bases whose kets and bras represent functions † and functionals, we call Qτ the “adjoint” transition 3.2. Bases for particular stochastic processes † operator in the following. Often, an operator O (x, ∂x) To demonstrate the application of the marginalized is said to be the adjoint of an operator O(x, ∂x) if distribution (110), let us reconsider the random the following relation holds with respect to two test walk from section 2.2.1 and the chemical reaction functions f and g: from section 2.2.2. The “Poisson basis function”  †  introduced in the latter section will be employed in the dx O (x, ∂x)f(x) g(x) (115) ˆ computation of mean extinction times in section 3.3.

  8 = dx f(x) O(x, ∂x)g(x) . In the derivation of the backward path integral in section 4.1, ˆ we use the finite difference approximation However, whether an operator Q complementing the † τ lim (|p(·|τ − ∆t)i − |p(·|τ)i)/∆t = lim Qe |p(·|τ)i . † ∆t→0 ∆t→0 τ−∆t,∆t above Qτ actually exists will not be important in the following (except in our discussion of the Poisson The discretization scheme conforms with the derivation of the representation in section 3.5). backward master equation (12). Master equations and the theory of stochastic path integrals 32

Moreover, it will allow us to recover the Poisson As discussed in section 2.2.2, the elements of representation of Gardiner and Chaturvedi [32, 33] in the transition rate matrix of the reaction k A → l A section 3.5. with rate coefficient γτ are given by Qτ (m, n) = γτ (n)k(δm,n−k+l − δm,n). The condition (Q|) on the 3.2.1. Random walks. The following solution of the adjoint transition operator therefore reads random walk largely parallels the previous derivation †
Qτ |n0i = γτ (n0−l+k)k|n0 − l + ki−(n0)k|n0i .(121) in section 2.2.1. In particular, we again use the inx This condition is met by orthogonal and complete Fourier basis |nix = e and π dx −inx † · k l k hn|f = −π 2π e f(x) with n ∈ Z. Due to the time- Qτ (c, a) ·= γτ c (a − a ) , (122) independence´ of the basis, the (E)volution operator E τ provided that there exist operators c and a fulfilling is zero. The condition (Q|) on the adjoint transition † operator Qτ is specified by the transition matrix (59) c|ni = (n + 1)|n + 1i and (123) of the process and reads a|ni = |n − 1i , (124) † Qτ |n0i = rτ (|n0 − 1i−|n0i)+lτ (|n0 + 1i−|n0i) .(117) respectively. We again call c the creation and a As before, we employ the operators c(x) = eix and the annihilation operator, even though the pre-factors a(x) = e−ix, which act on the basis functions as c|ni = in the above relations differ from the ones in (65) |n + 1i and a|ni = |n − 1i, respectively. Therefore, the and (66). Similar relations also hold with respect to the adjoint transition operator with the above property can basis functionals, namely hn|c = nhn − 1| and hn|a = be defined as hn + 1| (assuming the orthogonality of the basis). The operators also fulfil the commutation relation [a, c] = 1. † ·
Qτ (x, ∂x) ·= rτ a(x) − 1) + lτ (c(x) − 1 . (118) The actions of the creation and annihilation As this operator does not contain any derivatives, it is operators on the basis functions hint at how the self-adjoint in the sense of (115). Due to a mismatch basis functions and functionals from section 2.2.2 can in signs, it is, however, not the adjoint of the previous be adapted to meet the present requirements. In operator Qτ in (61). This mismatch could be corrected particular, the relations (123) and (124) can be fulfilled · −inx by redefining the above basis function as |nix ·= e . by moving the factorial from the basis functional (76) Ignoring this circumstance, the flow equation of the to the basis function (70) so that marginalized distribution follows as (ζx +x ˜)ne−q˜(ζx+˜x) ·  −ix ix  |nix ·= and (125) ∂−τ |pi = rτ (e − 1) + lτ (e − 1) |pi , (119) n! · n and is solved by hn|f ·= a f(x) x=−x/ζ˜ . (126) t t Let us briefly note that we changed the argument of the  −ix ix  |pi = exp (e −1) ds rs+(e −1) ds ls |ni .(120) basis function from q to x as compared to section 2.2.2 ˆτ ˆτ because both the generating function approach and The conditional probability distribution is recovered the marginalized distribution approach thereby result via the inverse Fourier transformation p(t, n|τ, n ) = 0 in the same path integral representation of averaged hn |pi. Upon inserting the explicit representation of 0 observables (cf. section 6). Apart from this notational the basis, the derivation proceeds as in appendix C change, the basis evolution operator (with the substitutions x → −q, τ → t0 and t → τ). Eventually, one recovers a Skellam distribution as the Eτ (c, a) ·= (∂τ x˜)(a − q˜) − (∂τ q˜)c (127) solution of the process. keeps the form it had in (77). The creation and annihilation operators also keep their previous forms 3.2.2. Chemical reactions. To prepare the computa- in (73) and (74), i.e. tion of mean extinction times in the next section as c(x, ∂ ) ·= ζx +x ˜ and (128) well as the derivation of the Poisson representation in x · section 3.5, we now reconsider processes that can be a(x, ∂x) ·= ∂ζx +q ˜ . (129) decomposed additively into chemical reactions of the † Despite their similar appearance, the operator Qτ form k A → l A. Our later derivation of a path integral in (122) is not the adjoint of the operator Qτ in (64). representation of averaged observables is also restric- Nevertheless, the two operators fulfil Qτ (q, x) = ted to such processes. The state space of the number † Qτ (x, q) for scalar arguments. This relation is of molecules is again N0. Analogous to section 2.2.2, we essentially the reason why we interchanged the letters require an (O)rthogonal and (C)omplete basis, as well x and q as compared to section 2.2.2. Both the as an (E)volution operator Eτ and an adjoint trans- generating function approach and the marginalized † | ition operator Qτ (condition (Q )) to specify the flow distribution approach thereby lead to the same path equation of the marginalized distribution. Master equations and the theory of stochastic path integrals 33 integral representation of averaged observables, as will marginalized distribution has the mathematical form be shown in section 6.1. of a backward Fokker-Planck equation. That is, for The choice of the parameters ζ 6= 0, x˜(τ), and q˜(τ) example, the case for the coagulation reaction 2 A → in the basis function (125) depends on the problem at A. For the Poisson basis function, the marginalized hand. In section 7, x˜ and q˜ will act as “stationary” distribution |p(t, n|τ)ix of this process obeys the flow or “extremal” paths, with x and an auxiliary variable equation q being deviations from them. For ζ ·=q ˜ ·= 1 and · · 1 2 · ∂−τ |pi = ατ (x)∂x|pi + βτ (x)∂ |pi , (135) x˜ ·= 0, the basis function instead simplifies to the 2 x Poisson distribution with the drift coefficient α (x) ·= −µ x2 and the n −x τ · τ x e 2 diffusion coefficient βτ (x) ·= −2µτ x . Unlike the |nix = . (130) n! diffusion coefficient of the “true” backward Fokker- In the following, we make heavy use of this “Poisson Planck equation (2), this diffusion coefficient may basis function”. It will play a crucial role in be negative (e.g. for x ∈ R\{0}). The final the formulation of a path integral representation of xne−x condition |p(t, n|t)ix = n! and the Feynman-Kac averaged observables in section 6 and in recovering the formula (A.3) imply that the above flow equation is Poisson representation in section 3.5. solved by Let us demonstrate the use of the Poisson basis n −x(t) function for the linear decay process A → ∅ with DDx(t) e EE |p(t, n|τ)ix = . (136) W rate coefficient µτ . For the above choice of ζ, x˜, and n! q˜, the creation operator (128) reads c = x and the Here, x(s) obeys the Itô SDE annihilation operator (129) reads a = ∂ + 1. With x dx(s) = α (x(s)) ds + pβ (x(s)) dW (s) , (137) the transition operator (122), the flow equation (116) s s which evolves x(s) from x(τ) = x to x(t). The obeyed by the marginalized distribution |p(t, n|τ)ix follows as symbol hh·iiW represents an average over realizations of the Wiener process W . Since the diffusion coefficient ∂−τ |pi = −µτ x∂x|pi . (131) 2 βτ (x) = −2µτ x of the coagulation process can take This equation is solved by the Poisson distribution on negative values, the sample paths of the SDE may

n −αt,τ x acquire imaginary components. This circumstance (αt,τ x) e |p(t, n|τ)i = (132) does not prevent the use of (136) for the calculation x n! of |p(t, n|τ)i, although it may complicate numerical whose mean αt,τ x decays proportionally to αt,τ ·= − t ds µ evaluations. Recently, Wiese attempted the evaluation e τ s . To interpret this solution, let us recall ´ of the average in (136) via the generation of sample that the sum in the definition of the marginalized paths [336]. Over a short time interval [τ, t], he found a distribution (110) does not affect the particle number good agreement between the resulting distribution and n. Therefore, upon relabelling the time parameters, a distribution effectively acquired via the stochastic the above solution (132) also solves the forward master simulation algorithm (SSA) in section 1.3. Over equation of the process, namely larger time intervals, however, the integration of the

∂τ |p(τ, n|t0)ix (133) SDE encountered problems regarding its numerical
convergence. Future research is needed to overcome = µτ (n + 1)|p(τ, n + 1|t0)i − n|p(τ, n|t0)i . x x this limitation. Moreover, it remains an open challenge Unlike the conditional distribution p(τ, n|t0, n0), how- to specify the boundary conditions of the PDE (135) to ever, the marginalized distribution |p(τ, n|t0)ix de- enable its direct numerical integration (an analogous scribes the dynamics of a population whose particle problem is encountered for the flow equation of the number at time t0 is Poisson distributed with mean x. generating function, cf. [366]). The conditional distribution is recovered from it via the inverse transformation p(τ, n|t , n ) = hn |p(τ, n|t )i 0 0 0 0 3.3. Mean extinction times n0 with hn0|f = (∂x + 1) f(x)|x=0. This transformation results in the Binomial distribution One often wishes to know the mean time at which n  n n −n a process first hits some target in state space. Such p(τ, n|t , n ) = 0 α
1 − α
0 . (134) 0 0 n τ,t0 τ,t0 a target could, for example, be a state in which

τ no more particles are left in the system. If the − ds µs Both the mean value e t0 n0 and the variance particles only replenish through auto-catalysis, the τ τ ´ − ds µs − ds µs t0 t0 (1 − e ´ ) e ´ n0 of this distribution decay process will then come to a halt. The mean time exponentially for large times, provided that µs > 0. after which that happens is called the mean extinction If at most two reactants and two products are time. For Markov processes with continuous sample involved in a reaction, the flow equation (116) of the paths, mean extinction times and, more generally, Master equations and the theory of stochastic path integrals 34

first-passage times, are commonly calculated with the Mean x of the initial Poisson distribution help of the backward Fokker-Planck equation (2). For 0 5 10 15 20 〉

jump processes, one can use the backward master τ 〈 equation for this purpose. The calculation, however, μ 3 is typically feasible only for one-step processes and involves the solution of a recurrence relation [110, 2 215]. In [374], Drummond et al. recently showed how the calculation can be simplified using the Poisson representation of Gardiner and Chaturvedi [32, 33]. 1 We introduce this representation in section 3.5. In the following, we outline how mean extinction times Mean extinction time 0 can instead be inferred in an analogous way using the 0 5 10 15 20 marginalized distribution from the previous sections. Initial number of particles n0 xn0 e−x For the Poisson basis function |n0i = , x n0! Figure 5. Mean extinction time µhτi of the linear decay the marginalized distribution is a true (single-time) process A → ∅ with decay rate coefficient µ. The blue line probability distribution and is more easily interpreted represents the mean extinction time if the number of particles is than the integral kernel of the Poisson representation. initially Poisson distributed with mean x ∈ R≥0 (µhτix = ln x + We again consider a process that can be γ−Ei(−x)). The red circles represent the mean extinction time if the initial number of particles is set to n ∈ (µhτi = H ). decomposed additively into chemical reactions of the 0 N0 n0 n0 form k A → l A. In addition, the transition † † matrix Qτ = Q of the process shall now be time- The last equality follows from the fact that a certain independent and the particles in the system shall be fraction of all particles, namely e−x, has already Poisson distributed with mean x at time τ = 0. been in the absorbing state initially. The above Consequently, the probability of finding n particles derivation readily extends to higher moments of the in the system at time τ ≥ 0 is described by the mean extinction time. marginalized distribution |p(τ, n|0)ix, provided that For the linear decay process A → ∅ with decay xn0 e−x its basis function is chosen as |n0i ·= (cf. rate coefficient µ, the equation (141) for the mean x · n0! the definition (110) of the marginalized distribution). extinction time reads Since the Poisson basis function is independent of time −x µ∂xhτix = (1 − e )/x . (142) and the process under consideration homogeneous in This equation implies that the mean extinction time time, the marginalized distribution |p(τ, n|0)ix obeys the forward-time flow equation (cf. (116)) µhτix increases logarithmically with the particles’ mean initial distance x from the absorbing state (see ∂ |pi = Q†|pi . (138) τ figure 5). The explicit solution of the equation is We define the probability of finding the system in µhτix = ln x + γ − Ei(−x) with Euler’s constant an “active” state with n > 0 particles at time τ ≥ 0 as γ and the exponential integral Ei (6.2.6 in [335]; ∞ · X the exponential integral ensures that hτi0 = 0). α(τ; x) ·= |p(τ, n|0)ix = 1 − |p(τ, 0|0)ix . (139) The application of the functional hn0|f = (∂x + n=1 1)n0 f(x) to the solution returns the mean Over time, this probability flows into the “absorbing” x=0 extinction time of particles whose initial number is state n = 0 at rate f(τ, x) ·= −∂τ α(τ; x). Since f(τ, x) dτ is the probability of becoming absorbed not Poisson distributed but that is fixed to some during the time interval [τ, τ + ∆t], one can define the value n0 ≥ 0. The corresponding mean extinction ∞ · time µhτin0 is given by the harmonic number Hn0 ·= mean extinction time as hτix ·= dτ τf(τ, x). An 0 Pn0 1 . One can also infer this result directly from the integration by parts transforms this´ average into i=1 i ∞ ∞ backward master equation using the methods discussed hτi = − dτ τ∂ α(τ; x) = dτ α(τ; x) . (140) in [110, 215]. x ˆ τ ˆ 0 0 Drummond et al. have extended the above The boundary terms of the integration by parts computation to chemical reactions with at most two vanished because we assume that all particles reactants and two products [374]. For these reactions, are eventually absorbed (in particular, we assume the flow equation (138) has the mathematical form of limτ→∞ τα(τ; x) = 0). The definition of the probab- a (forward) Fokker-Planck equation. ility α(τ; x) in (139) implies that it fulfils the same forward-time flow equation as the marginalized dis- 3.4. Flow of the generating functional tribution |p(τ, n|0)ix. Since limτ→∞ α(τ; x) = 0, the mean extinction time (140) therefore obeys Both the probability generating function (45) and † −x −Q (x, ∂x)hτix = α(0; x) = 1 − e . (141) the marginalized distribution (110) were defined by Master equations and the theory of stochastic path integrals 35 summing the conditional probability distribution over As a side note, let us remark that the flow equation a set of basis functions, either at the final or at the ∂τ |gi = (Eτ + Qτ )|gi of the generating function in (54) initial time. In addition, one can define the “probability also admits a functional counterpart. In particular, the generating functional” series X · X hg(τ|t0, n0)| ·= hn|p(τ, n|t0, n0) (143) hp(t, n|τ)| ·= p(t, n|τ, n0)hn0| (149) n n0 by summing the conditional distribution over the obeys the flow equation set {hn|} of basis functionals. The definition of ∂−τ hp| = hp|(Eτ + Qτ ) = Qeτ hp| , (150) the generating functional is meant with respect to test functions that can be expanded in the basis with final value hp(t, n|t)| = hn|. The corres- functions {|ni}. As before, we assume that the ponding inverse transformation reads p(t, n|τ, n0) = basis functions and functionals constitute a (C)omplete hp(t, n|τ)|n0i. Both the flow equation obeyed by the and (O)rthogonal basis. The probability generating generating function and the above flow equation can functional “generates” probabilities in the sense that be used to derive the “forward” path integral repres- entation in section 5. Further uses of the series (149) p(τ, n|t0, n0) = hg(τ|t0, n0)|ni . (144) remain to be explored. In the next section, we show how this inverse transformation reduces to the Poisson representation 3.5. The Poisson representation of Gardiner and Chaturvedi [32,33] upon choosing the basis function |ni as a Poisson distribution. Assuming that the action of the generating functional The derivation of the (functional) flow equation hg| on a function f can be expressed in terms of an of hg| proceeds analogously to the derivations in sec- integral kernel (also called g) as tions 2.1 and 3.1. Differentiation of its definition (143) ∞ hg(τ|t0, n0)|f = dx g(τ; x|t0, n0)f(x) , (151) with respect to τ and use of the forward master equa- ˆ−∞ tion ∂τ p(τ|t0) = Qτ p(τ|t0) result in n −x the insertion of the Poisson basis function |ni = x e   x n! X X | ∂τ hg| = ∂τ hn| + Qτ (n, m)hm| p(τ, n|·) . (145) into the inverse transformation (144) results in n m ∞ xne−x This equation can be cast into a linear PDE by p(τ, n|t0, n0) = dx g(τ; x|t0, n0) . (152) ˆ−∞ n! using the basis (E)volution operator Eτ and the † | A representation of the probability distribution of this adjoint transition operator Qτ (condition (Q )). In form is called a “Poisson representation” and was first particular, since the Kronecker delta hn|mi = δn,m proposed by Gardiner and Chaturvedi [32, 33]. Since is independent of time, it holds that hn|∂τ |mi = the integration in (152) proceeds along the real line, (−∂τ hn|)|mi. Consequently, the basis evolution operator in condition (E) fulfils the above representation is referred to as a “real” Poisson representation [375]. Although the use of a hn|Eτ = −∂τ hn| (146) real variable may seem convenient, its use typically with respect to functions that can be expanded in the results in the kernel being a “generalized function”, basis functions. Moreover, the (O)rthogonality and i.e. a distribution. For example, the initial condition

(C)ompleteness of the basis imply that the adjoint p(t0, n|t0, n0) = δn,n0 is recovered for the integral n0 transition operator in condition (Q|) acts on basis kernel g(t0; x|t0, n0) = δ(x)(∂x+1) . By the definition functionals as ∞ ∞ j
j dx ∂ δ(x) f(x) ·= dx δ(x)(−∂x) f(x) (153) † X | x · hn|Qτ = Qτ (n, m)hm| . (147) ˆ−∞ ˆ−∞ m of distributional derivatives (1.16.12 in [335], j ∈ N0), Both of the above expressions hold for all values of n. n0 this kernel can also be written as [(1 − ∂x) δ(x)]. If the two differential operators E and Q† exist, the τ τ The integral kernel δ(x − x0) instead results for a generating functional hg(τ|t , n )| obeys the functional 0 0 Poisson distribution with mean x0. In [32], Gardiner flow equation and Chaturvedi used the real Poisson representation † † to calculate steady-state probability distributions ∂τ hg| = hg|(−Eτ + Qτ ) = hg|Qeτ , (148) of various elementary reactions. Furthermore, with initial condition hg(t0|t0, n0)| = hn0|. Note that the real Poisson representation was employed by this flow equation employs the same operator as the Elderfield [376] to derive a stochastic path integral flow equation (116) of the marginalized distribution. representation. Droz and McKane [377] later argued Both equations can be used to derive the “backward” that this representation is equivalent to a path integral path integral representation considered in section 4. representation based on Doi’s Fock space algebra. Master equations and the theory of stochastic path integrals 36

Our discussion of the backward and forward path numerical integration of an SDE with potentially integral representations in sections 4 and 5 clarifies negative diffusion coefficient was attempted for the bi- the similarities and differences between these two directional reaction 2 A ∅ in [379]. The value of the approaches. integration has remained inconclusive. Recently, an To circumvent the use of generalized functions, exponential ansatz for the integral kernel g(τ; x|·) has various alternatives to the real Poisson representation been considered to approximate its flow equation in the have been proposed: the “complex” and “positive” limit of weak noise [380] (cf. section 2.3.2). Moreover, Poisson representations [110, 375] as well as the a gauge Poisson representation was recently employed “gauge” Poisson representation [378]. The former two in a study of the coagulation reaction 2 A → A [381]. representations as well as the real representation are discussed in the book of Gardiner [110]. The complex 3.6. Résumé Poisson representation is obtained by continuing the Poisson basis function in (152) into the complex In the previous section, we formulated general domain and performing the integration around a closed conditions under which the master equation can path C around 0 (once in counter-clockwise direction). be transformed into a partial differential equation Upon using Cauchy’s differentiation formula to redefine obeyed by the probability generating function (45). the basis functional as In the present section, we complemented this flow x equation by a backward-time flow equation obeyed n x n! e hn|f ·= ∂ e f(x)|x=0 = dx f(x) , (154) by the marginalized distribution (110) and by a x ˛ 2πi xn+1 C functional flow equation obeyed by the probability it becomes apparent that the initial condition generating functional (143). Whereas the marginalized p(t0, n|t0, n0) = δn,n0 is then recovered for the kernel x distribution g(t ; x|t , n ) = n0! e . As the interpretation of 0 0 0 2πi xn0+1 X this kernel is also not straightforward, we refrain from |p(t, n|τ)i = p(t, n|τ, n0)|n0i (158) calling it a “quasi-probability distribution” [32]. n0 Let us exemplify the flow equation obeyed by the was defined as the sum of the conditional probability integral kernel for the reaction k A → l A with rate distribution over a set of basis functions, the generating coefficient γτ . The flow equation can be inferred from functional the corresponding flow equation (148) of the generating X hg(τ|t0, n0)| = hn|p(τ, n|t0, n0) (159) functional and the kernel’s definition in (151). Since n we employ the Poisson basis function, the results was defined as the sum of the distribution over a set from section 3.2.2 imply that the adjoint transition of basis functionals. In section 3.2, we introduced operator (122) reads (O)rthogonal and (C)omplete basis functions and † k l k
Qτ = γτ x (∂x + 1) − (∂x + 1) . (155) functionals for the study of different stochastic processes, including the Poisson basis for the study Consequently, one finds that the integral kernel n −x of chemical reactions (|ni = x e and hm|f = g(τ; x|·) obeys the flow equation x n! m (∂x + 1) f(x) ). Provided that there also exist a ∂ g = Q (x, ∂ )g (156) x=0 τ τ x basis (E)volution operator Eτ and an adjoint transition l k
k † | = γτ (1 − ∂x) − (1 − ∂x) x g . (157) operator Qτ (condition (Q )), the marginalized distribution obeys the backward-time flow equation To arrive at this equation, we performed repeated † † integrations by parts while ignoring any potential ∂−τ |pi = (−Eτ + Qτ )|pi = Qeτ |pi (160) boundary terms (cf. the definition of the adjoint with final condition |p(t, n|t)i = |ni. Moreover, the operator in (115)). The importance of boundary terms probability generating functional (143) then obeys is discussed in [378]. the functional flow equation (148). The inverse Thus far, most studies employing the Poisson transformation p(τ, n|t0, n0) = hg(τ|t0, n0)|ni of representation have focused on networks of bimolecular this functional generalizes the Poisson representation reactions P k A → P l A with P k ≤ 2 and j j j j j j j j of Gardiner and Chaturvedi [32, 33] as shown in P l ≤ 2. For these networks, the flow equation (156) j j section 3.5. In section 3.3, we showed how the assumes the mathematical form of a forward Fokker- flow equation (160) obeyed by the marginalized Planck equation with the derivatives being of at most distribution can be used to compute mean extinction second order (the corresponding diffusion coefficient times. Furthermore, the equation will prove useful may be negative). It has been attempted to map in the derivation of path integral representations of the resulting equation to an Itô SDE [32], but it the master equation and of averaged observables in should be explored whether this procedure is supported sections 4 and 6. Future studies could explore by the Feynman-Kac formula in appendix A. The whether the flow equation obeyed by the marginalized Master equations and the theory of stochastic path integrals 37 distribution can be evaluated in terms of WKB spatial degrees of freedom is considered in sections 4.5.2 approximations or spectral methods. to 4.5.4. After deriving the backward path integral repres- 4. The backward path integral representation entation in section 4.1, we exemplify how this repres- entation can be used to solve the bi-directional reac- In the previous two sections, we showed how the tion ∅ A (section 4.2), the pair generating process forward and backward master equations can be cast ∅ → 2 A (section 4.5.1), and a process with diffusion into four linear PDEs for the series expansions (41)– and linear decay (sections 4.5.2 to 4.5.4). The forward (44). In this section, as well as in section 5, path integral representation is derived in section 5 and the solutions of these four equations are expressed is exemplified in deriving the generating function of in terms of two path integrals. Upon applying linear processes A → l A with l ≥ 0. For the linear inverse transformations, the path integrals provide growth process A → 2 A, we recover a negative Bi- distinct representations of the conditional probability nomial distribution as the solution of the master equa- distribution solving the master equations. The flow tion. Observables of the particle number are considered equations obeyed by the generating function (41) and in section 6. by the series (44) will lead us to the “forward” path As an intermezzo, we show in sections 4.4 and 5.3 integral representation how one can derive path integral representations t) for jump processes with continuous state spaces (or −S p(t, n|t0, n0) = hn| e |n0i , (161) processes whose transition rates can be extended to t t0 [t0 such spaces). The corresponding derivations are based and the flow equations obeyed by the marginalized on Kramers-Moyal expansions of the backward and distribution (42) and by the generating functional (43) forward master equations. Since the backward and to the “backward” path integral representation forward Fokker-Planck equations constitute special t] cases of these expansions, we recover a classic path −S† p(t, n|t0, n0) = hn0| e |ni . (162) integral representation whose development goes back t0 t (t0 to works of Martin, Siggia, and Rose [16], de t) t] Dominicis [17], Janssen [18, 19], and Bausch, Janssen, The meanings of the integral signs and , as well [t0 (t0 and Wagner [19]. Moreover, we recover the Feynman- as of the exponential weights are explainedffl ffl below. We Kac formula from appendix A, an Onsager-Machlup choose the above terms for the two representations representation [79], and Wiener’s path integral for because the forward path integral representation Brownian motion [80, 81]. propagates the basis function |n0i to time t, where t0 Before starting out with the derivations of the it is then acted upon by the functional hn| . Counter- t two path integral representations (161) and (162), intuitively, however, this procedure requires us to solve let us note that these representations only apply if a stochastic differential equation proceeding backward their underlying transition operators Q or Q† can in time (cf. sections 5.2 and 5.3). Analogously, the eτ eτ be written as power series in their arguments (x backward path integral representation propagates the and ∂x for the backward path integral and q and basis function |nit backward in time to t0, where one ∂q for the forward path integral; the names of the then applies the functional hn0| . This procedure t0 variables differ because both path integrals then lead requires us to solve an ordinary, forward-time Itô SDE to the same path integral representation of averaged (cf. section 4.3). The name “adjoint” path integral observables in section 6 without requiring a change representation may also be used for the backward of variable names). In addition, the power series representation. As we will see below, its “action” S† need to be “normal-ordered”, meaning that in every involves the adjoint transition operator Q† . eτ summand, all the x are to the left of all the ∂ (all To make the following derivations as explicit x the q to the left of all the ∂ ). This order can be as possible, we now assume the discrete variables q established by the repeated use of the commutation n and n to be one-dimensional. Likewise, the 0 relation [∂ , x]f(x) = f(x) (or, more directly, by variables x and q of the associated flow equations x invoking (∂ x)nf(x) = Pn  n+1 xm∂mf(x) or are one-dimensional real variables. The derivation x m=0 m+1 x (x∂ )nf(x) = Pn  n xm∂m, with the curly braces below employs an exponential representation of the x m=0 m x Dirac delta function. Although such a representation representing Stirling numbers of the second kind; cf. section 26.8 in [335]). The transition operator Qτ = also exists for Grassmann variables [382], we do not l k k γτ (c −c )a in (64) and the adjoint transition operator consider that case. The extension of the following † k l k derivations to processes with multiple types of particles Qτ = γτ c (a − a ) in (122) of the chemical reaction is straightforward and proceeds analogously to the k A → l A are already in their normal-ordered forms inclusion of spatial degrees of freedom. A process with (with the creation and annihilation operators in (73) Master equations and the theory of stochastic path integrals 38 and (74), or (128) and (129), respectively). with the abbreviation l l Y dxj dqj 4.1. Derivation ·= . (167) ˆ 2 2π k j=k R We first derive the backward path integral representa- † † tion (162) because its application to actual processes is To proceed, we now replace Lτ = 1 + Qeτ ∆t + 2
Q† ∆t more intuitive than the application of the forward path O (∆t) by the exponential e eτ (for brevity, we integral representation. For this purpose, we consider drop the correction term in the following). Note that † Q† ∆t the flow equation ∂−τ |p(·|τ)i = Qeτ |p(·|τ)i in (116) the exponential e eτ does not involve differential obeyed by the marginalized distribution. Its final con- operators because those were all replaced by the new dition reads |p(t, n|t)i = |ni. The conditional prob- variables iqj. Whether the exponentiation can be made ability distribution is recovered from the marginalized mathematically rigorous is an open question and may distribution via p(t, n|t0, n0) = hn0|p(t, n|t0)i. As the possibly be answered positively only for a restricted first step, we split the time interval [t0, t] into N pieces class of stochastic processes. Upon performing the t0 ≤ t1 ≤ ... ≤ tN ·= t of length ∆t ·= (t − t0)/N  1. exponentiation, one obtains the following discrete- Over the time interval [t0, t1], the flow equation is then time path integral representation of the marginalized solved by9 distribution: † N † |p(t, n|t0)ix = Lt (x0, ∂x0 )|p(t, n|t1)ix (163) −S 0 0 0 |p(t, n|t )i = e N |ni with (168) 0 x0 t,xN † † 2
1 with the generator Lτ ·= 1 + Qeτ ∆t + O (∆t) . N Alternatively, the following derivation could be † X  xj − xj−1 †  S ·= ∆t iqj − Qe (xj−1, iqj) . (169) performed by solving the flow equation (148) obeyed N ∆t tj−1 by the generating functional over the time interval j=1 [tN−1, t] as (cf. figure 6) The final condition |p(t, n|t)ix = |nit,x is trivially † fulfilled because only N = 0 time slices fit between † hg(t|t0, n0)| = hg(tN−1|t0, n0)|Lt (xN−1, ∂xN−1 ) . (164) N−1 t and t. The object SN is called an “action”. The −S† Here, the generating functional hg| acts on functions exponential factor e N weighs the contribution of f(xN−1), meets the initial condition hg(t0|t0, n0)| = every path (x1, q1) → (x2, q2) → ... → (xN , qN ). hn0|, and is transformed back into the conditional prob- As the final step of the derivation, we take ability distribution via p(t, n|t0, n0) = hg(t0|t0, n0)|ni. the continuous-time limit N → ∞, or ∆t → 0, Both of the above approaches result in the same path at least formally. The following continuous-time integral representation. In the following, we use the expressions effectively serve as abbreviations of the equation (163) for this purpose. above discretization scheme with the identifications As the next step of the derivation, we insert the x(t0 + j∆t) ·= xj and q(t0 + j∆t) ·= qj. integral representation of a Dirac delta between the Further comments on the discretization scheme are generator L† and the marginalized distribution |pi provided in section 4.4. Combined with the in (163), turning the right-hand side of this expression inverse transformation (111), we thus arrive at the into following backward path integral representation of the dx1 dq1 conditional probability distribution: L† (x , ∂ ) e−iq1(x1−x0)|p(t, n|t )i . (165) t0 0 x0 1 x1 ˆ 2 2π R p(t, n|t , n ) = hn | |p(t, n|t )i (170) 0 0 0 t0,x(t0) 0 x(t0) The integrations over x1 and q1 are both performed t] † along the real line from −∞ to +∞. If the adjoint with |p(t, n|t )i = e−S |ni (171) 0 x(t0) t,x(t) † † (t transition operator Qeτ , and thus also Lτ , are normal- 0 ordered power series, we can replace ∂ by iq in t x0 1 †  †  † and S ·= dτ iq∂ x − Q (x, iq) . (172) the above expression and pull L to the right of · τ eτ t0 ˆt0 the exponential. Making use of the final condition Here we included x(t0) as an argument of the functional |p(t, n|t)ix = |nit,x , the above steps can be repeated N N hn0| to express that this functional acts on until (165) reads t0,x(t0) t] x(τ) only for τ = t0. Moreover, we defined ·= N  N  (t0 Y −iqj (xj −xj−1) † N ffl e L (xj−1, iqj) |ni , (166) tj−1 t,xN limN→∞ 1 to indicate that the path integral involves 1 j=1 integrationsffl over x(t) and q(t), but not over x(t0) and q(t ). 9 This discretization of time conforms with the Itô prescription 0 for forward-time SDEs and with the derivation of the backward The evaluation of the above path integral master equation (12) (see section 4.3 and also the footnote on representation for an explicit process involves two page 31). steps. First, the path integral (171) provides the Master equations and the theory of stochastic path integrals 39

Forward master Flow equation (148) of equation (10) 3.4 the generating functional 4.1 ∂τ p(τ, n ⎜⋅) = … ∂τ 〈g(τ ⎜⋅) ⎜ = … Backward path integral Evaluation in terms of an representation (170)-(172) 4.3 average over the paths of t] − † the Itô SDE (181) ⎜ 〈 ⎜ ⎜ 〉 p(t, n t0, n0) = n0 t0 e n t Backward master Flow equation (116) of the (t0 equation (12) 3.1 marginalized distribution 4.1 ∂−τ p(⋅ ⎜τ, n0) = … ∂−τ ⎜p(⋅ ⎜τ)〉 = …

Figure 6. Outline of the derivation of the backward path integral representation and of its evaluation in terms of an average over the paths of an Itô stochastic differential equation.

marginalized distribution |p(t, n|t0)i. Second, this above Poisson distribution. For µ > 0, both of them marginalized distribution is mapped to the conditional converge to the asymptotic value γ/µ. In the next probability distribution p(t, n|t0, n0) by the action of section and in section 4.5, we generalize these results the functional hn0|. to more general processes. The marginalized distribution (174) does not only 4.2. Simple growth and linear decay solve the master equation of the reaction ∅ A, but it also establishes the link between the path integral Let us demonstrate the above two steps for the variable x and the moments of the particle number bi-directional reaction ∅ A. The stationary n. In particular, the mean particle number evaluates distribution of this process was already derived in to hni(t) = x(t), while higher order moments can the introduction to section 2. Both the growth k Pk k l determined via hn i(t) = l=0 l x(t) . The curly rate coefficient γ and the decay rate coefficient µ braces denote a Stirling number of the second kind (cf. shall be time-independent, but this assumption is section 26.8 in [335]). easily relaxed. As the first step, the backward path The conditional probability distribution can integral (170) requires us to choose an appropriate be calculated from the marginalized distribution basis. The time-independent Poisson basis with the by applying the functional hn |f = (∂ + xne−x 0 x(t0) basis function |nix = n! proves to be convenient n0 1) f(x(t0))|x(t0)=0 to the latter (cf. (126) and (129)). for this purpose (cf. section 3.2.2). The adjoint In the limit of a vanishing decay rate (µ → 0), for transition operator follows from (122), (128), and (129) which x(τ) = x(t0) + γ(τ − t0), we thereby recover the † as Qe (x, ∂x) = (γ − µx)∂x. Insertion of this operator shifted Poisson distribution (80), i.e. into the action (172) results in e−γ(t−t0)(γ(t − t ))n−n0 t 0 p(t, n|t0, n0) = Θn−n0 . (175) S† = dτ iq ∂ x − (γ − µx) (173) (n − n0)! ˆ τ t0 Thus, the mean and variance of the conditional The integration over the path q(τ) from τ = t0 probability distribution grow linearly with time. If the to τ = t is performed most easily in the discrete- particles decay but are not replenished (µ > 0 but γ = time approximation. The marginalized distribution 0), we instead recover the Binomial distribution (134). |p(t, n|t )i with x(t ) ≡ x thereby follows as the 0 x(t0) 0 0 following product of Dirac deltas: 4.3. Feynman-Kac formula for jump processes N Y
 We now show how the backward path integral dxj δ xj − xj−1 − (γ − µ xj−1)∆t |ni . ˆ t,xN representation (171) of the marginalized distribution j=1 R can be expressed in terms of an average over the The function |ni is also integrated over. Upon t,xN paths of an Itô stochastic differential equation (SDE). taking the continuous-time ∆t → 0 and performing The resulting expression bears similarities with the the integration over the path x(τ), one finds that Feynman-Kac formula (3), especially when the adjoint the marginalized distribution is given by the Poisson † transition operator Qe (x, ∂x) of the stochastic process distribution τ under consideration is quadratic in ∂x. In the general x(t)ne−x(t) case, however, functional derivatives act on the average |p(t, n|t0)i = |ni = , (174) x(t0) t,x(t) n! over paths. These derivatives can, for example, be with x(τ) solving ∂τ x = γ − µx. This equation evaluated in terms of perturbation expansions as we coincides with the rate equation of the process ∅ A. demonstrate in section 4.5.4. The procedure outlined −µ(τ−t0) γ γ below serves as a general starting point for the The solution x(τ) = e (x(t0) − µ ) + µ of the rate equation specifies the mean and the variance of the Master equations and the theory of stochastic path integrals 40 exact or approximate evaluation of the backward path on the “source” X(τ 0) only for times τ 0 < τ, and integral (171). δ t 0 0 0 that δQ(τ) t dτ Q(τ )f(τ ) = f(τ) holds for all τ ∈ In the following, we consider a stochastic process 0 (t0, t]. Moreover,´ let us note that the (Q, X)-generating whose adjoint transition operator has the generic form functional depends on qN and x(t0) but not on xN . † 1 2 † These properties follow from the derivation of the Qe (x, ∂x) = ατ (x)∂x + βτ (x)∂ + P (x, ∂x) . (176) τ 2 x τ representation (179), which we outline in appendix D. As before, we call ατ a drift and βτ a diffusion An important special case of the above represent- † ation is constituted by processes whose perturbation coefficient. The object Pτ is referred to as the (adjoint) † perturbation operator, or simply as the perturbation. operator Pτ is zero. For Q = X = 0, the generating iqN x(t) The perturbation operator absorbs all the terms of functional (179) then simplifies to Z0,0 = e and higher order in ∂x and possibly also terms of lower the marginalized distribution (177) follows in terms of † the Feynman-Kac like fomula order. Thus, the above form of Qeτ is not unique † and ατ , βτ , and P should be chosen so that the τ |p(t, n|t0)ix(t ) = |nit,x(t) . (182) evaluation of the expressions below becomes as simple 0 W † Here, x(τ) solves the Itô SDE as possible. If the perturbation operator Pτ is zero, those expressions simplify considerably. p dx(τ) = ατ (x) dτ + βτ (x) dW (τ) (183) As the first step, let us rewrite the backward path integral representation (171) of the marginalized with the initial value x(t0). For the Poisson basis xne−x distribution as function |nix = n! , the above representation of the marginalized distribution coincides with our dxN dqN |p(t, n|t )i = e−iqN xN Z |ni , (177) 0 x(t0) 0,0 t,xN earlier result (136), which we encountered in the ˆ 2 2π R discussion of the coagulation reaction 2 A → A. where Z represents a value of the (Q, X)- Q=0,X=0 For the bi-directional reaction ∅ A from the generating functional previous section with adjoint transition operator t) † † t Qe (x, ∂x) = (γ − µx)∂x, the Itô SDE (183) simplifies iqN xN −S + dτ [Q(τ)x(τ)+X(τ)iq(τ)] ZQ,X ·= e t0 . (178) ´ to the deterministic rate equation ∂τ x = γ − (t0 µx. Thus, no averaging is required to evaluate the Hence, we have singled out the integrations over xN marginalized distribution (182) and one recovers our and qN before performing the continuous-time limit earlier solution (174). (cf. (168) and (171)). We call ZQ,X a generating Our above discussion only applies to processes functional because a functional differentiation of ZQ,X in well-mixed environments with a single type of with respect to Q(τ) generates a factor x(τ), and a particles. A generalization of the results to processes functional differentiation of ZQ,X with respect to X(τ) with multiple types of particles or with spatial generates a factor iq(τ). degrees of freedom is straightforward. A spatial Given the action (172) with the adjoint transition process will be discussed in section 4.5. In a operator (176), the above properties of the (Q, X)- multivariate generalization of the above procedure, the generating functional can now be used to rewrite this adjoint transition operator (176) includes a vector- function as (cf. appendix D) valued drift coefficient ατ (x) and a diffusion matrix δ t † δ δ iqN + dτ P ( , ) 0 βτ (x). Moreover, the derivation of the corresponding Z = e δQ(t) t0 τ δQ(τ) δX(τ) Z (179) Q,X ´ Q,X generating functional (179) requires that there exists t √ 0 dτQ(τ)x(τ) · t0 a matrix β ·= γ fulfilling γ γ| = β . If the with ZQ,X ·= e´ W . (180) τ · τ τ τ τ diffusion matrix β is positive-semidefinite, its positive- Here, hh·ii represents the average over realizations of a τ √ W semidefinite and symmetric square root β can be Wiener process W (τ). This Wiener process influences τ determined via diagonalization [111]. the evolution of the path x(τ) through the Itô SDE In section 4.5, we exemplify the use of our above   p dx(τ) = ατ (x) + X(τ) dτ + βτ (x) dW (τ) . (181) results for various well-mixed and spatial processes. But before, let us show how the results from the The temporal evolution of x(τ) starts out from previous sections can be used to derive a path integral x(t ), which is determined by the argument of the 0 representation for processes with continuous state marginalized distribution (177). In section 4.5.4, we spaces. demonstrate how the marginalized distribution can be evaluated in terms of a perturbation expansion of the (Q, X)-generating functional (179) for a process 4.4. Intermezzo: The backward Kramers-Moyal of diffusing particles that are also decaying. In order expansion to perform the perturbation expansion, let us already The transition rate wτ (m, n) denotes the rate at which note that the value x(τ) of the path at time τ depends probability flows from a state n to a state m, or, Master equations and the theory of stochastic path integrals 41 considering an individual sample path, the rate at The backward analogue of the master equa- which particles jump from state n to state m. Thus, tion (184) reads κτ (∆n, n) ·= wτ (n + ∆n, n) denotes the rate at which ∂−τ p(t, x|τ, x0) (187) the state n is left via jumps of size ∆n (with n ∈ N0   and ∆n ∈ Z). Thus far, we only considered jumps = d∆x p(·|τ, x0 + ∆x) − p(·|τ, x0) κτ (∆x, x0) , ˆ between the states of a discrete state space. But in the R following, we derive a path integral representation for with the final condition p(t, x|t, x0) = δ(x−x0). Given processes having a continuous state space. the analyticity of p(·|τ, x0) in x0, the backward master equation can be rewritten in terms of the backward 4.4.1. Processes with continuous state spaces. We Kramers-Moyal expansion consider a process whose state is characterized by † ∂−τ p(·|τ, x0) = Qeτ (x0, ∂x0 )p(·|τ, x0) (188) a continuous variable x ∈ R≥0. The change from the letter n to the letter x is purely notational and with the adjoint transition operator emphasizes that the state space is now continuous (the ∞ 1 † · X (m) m Qe (x0, ∂x0 ) ·= M (x0)∂ . (189) change also highlights a formal similarity between the τ m! τ x0 linear PDEs discussed so far and the ones derived m=1 below). The conditional probability distribution This operator is the adjoint of the operator in the for- p(τ, x|t0, x0) describing the system shall be normalized ward Kramers-Moyal expansion (185) in the sense that  †    as dx p(τ, x|·) = 1 and obey the master equation dx Qeτ (x, ∂x)f(x) g(x) = dxf(x) Qeτ (x, ∂x)g(x) R ´ ´(provided that all boundary terms´ in the integrations ∂ p(τ, x|·) = d∆x κ (∆x, x − ∆x)p(τ, x − ∆x|·) by parts vanish). τ ˆ τ R  − κτ (∆x, x)p(τ, x|·) (184) 4.4.2. Path integral representation of the backward with the initial condition p(t0, x|t0, x0) = δ(x − x0). Kramers-Moyal expansion. The adjoint transition op- † The structure of this master equation is equivalent to erator Qeτ (x0, ∂x0 ) of the backward Kramers-Moyal ex- the structure of the master equation (11). Provided pansion (189) is normal-ordered with respect to x0 and that the product κτ (∆x, x)p(τ, x|·) is analytic in x, ∂x0 . Moreover, the backward Kramers-Moyal expan- one can perform a Taylor expansion of the above sion (188) has the same form as the flow equation (116) master equation to obtain the (forward) Kramers- obeyed by the marginalized distribution. One can Moyal expansion [1, 103, 272] therefore follow the steps in section 4.1 to represent ∞ the backward Kramers-Moyal expansion by the path X (−1)m ∂ p(τ, x|·) = ∂mM(m)(x)p(τ, x|·) . (185) integral τ m! x τ m=1 t] −S† (m) p(t, x|t0, x0) = e δ(x − x(t)) (190) The “jump moments” M are defined as x(t0)=x0 τ (t0 (m) m with the action M (x) ·= d∆x (∆x) κτ (∆x, x) . (186) τ ˆ t R S† ·= dτ iq(τ)∂ x(τ) − Q† (x(τ), iq(τ)) . (191) By Pawula’s theorem [104, 383, 384], the positivity of · τ eτ ˆt0 the conditional probability distribution requires that The integral sign in (190) is again defined as the the Kramers-Moyal expansion either stops at its first continuous-time limit of (167) and traces out all paths or second summand, or that it does not stop at all. If of x(τ) and q(τ) for τ ∈ (t , t] (note that x(τ) the expansion stops at its second summand, it assumes 0 differs from x and x , which are fixed parameters; the form of a (forward) Fokker-Planck equation. The 0 for brevity, however, x(τ) is occasionally abbreviated drift coefficient of this Fokker-Planck equation is given (1) as x below). A diagrammatic computation of multi- by Mτ (x) and its non-negative diffusion coefficient time correlation functions based on a path integral (2) by Mτ (x). The sample paths of the Fokker-Planck representation equivalent to (190) has recently been equation are continuous, however, contradicting our considered in [385]. earlier assumption of the process making discontinuous Let us specify the path integral representa- jumps in state space. For a jump process, the tion (190) for a model of the chemical reaction k A → Kramers-Moyal expansion cannot stop. Nevertheless, a l A with rate coefficient γτ (and k, l ∈ N0). As we truncation of the Kramers-Moyal expansion at the level assume the particle “number” x to be continuous, the of a Fokker-Planck equation often provides a decent model is only reasonable in an approximate sense for approximation of a process, provided that fluctuations large values of x. In defining the transition rate of the cause only small relative changes of its state x. reaction, one has to ensure the non-negativity of x and Master equations and the theory of stochastic path integrals 42

† the conservation of probability. A possible choice of not involve a perturbation operator Pτ . Therefore, one the transition rate is can follow the steps in that section to evaluate the path k
integral (190) in terms of an average over the paths of κτ (∆x, x) ·= γτ x Θ(x − k)δ ∆x − (l − k) . (192) an Itô SDE. This procedure shows that the backward Here, the Heaviside step function Θ(x−k) ensures that Fokker-Planck equation is solved by a sufficient number of particles are present to engage p(t, x|t , x ) = δ(x − x(t)) , (197) in a reaction (and it prevents the loss of probability to 0 0 W negative values of x). with x(τ) solving the Itô SDE Assuming a smooth approximation of the Heav- p dx(τ) = ατ (x(τ)) dτ + βτ (x(τ)) dW (τ) (198) iside step function that vanishes for x < 0, the jump with initial value x(t ) = x . Hence, we have recovered moments (186) follow as 0 0 the Feynman-Kac formula (3) (apart from a notational (m) m k Mτ (x) = γτ (l − k) x Θ(x − k) . (193) change in the time parameters). The corresponding adjoint transition operator evalu- ates to 4.4.4. The Onsager-Machlup function. The connec- tion between the above path integral representation Q† (x, iq) = γ (xe−iq)k(eiq)l − (eiq)kΘ(x − k) . (194) eτ τ of the (backward) Fokker-Planck equation and the Path integrals with such an operator have been noted work of Onsager and Machlup [79] becomes apparent in [386–388], but their potential use remains to be fully upon the completion of a square (as in a Hubbard- explored. Curiously, upon ignoring the step function, Stratonovich transformation [391, 392]). For this pur- the operator (194) has the same structure as the pose, the diffusion coefficient βτ in the transition oper- transition operator (122) upon identifying xe−iq with ator (195) must not only be non-negative but positive(- the creation operator c, and eiq with the annihilation definite). One can then complete the square in the operator a (Cole-Hopf transformation [386–390]). variable q and perform the path integration over this Note, however, that the stochastic processes associated variable. Returning to the discrete-time approxima- to the two transition operators differ from each other tion (169) of the action, one obtains the following rep- (discrete vs. continuous state space; different transition resentation of the conditional probability distribution rates). The connection between the two associated p(t, x|t0, x0) in terms of convolutions of Gaussian dis- path integrals (171) and (190) should be further tributions: explored. N−1  Y  lim dxj+1 Gµ ,σ2 (xj+1 −xj) δ(x−xN ) .(199) N→∞ ˆ j j 4.4.3. Path integral representation of the backward j=0 R Fokker-Planck equation. With the drift coefficient The Dirac delta function is included in the integrations. · (1) · 2 · ατ (x) ·= Mτ (x) and the non-negative diffusion Moreover, µj ·= αtj (xj)∆t acts as the mean and σj ·= (2) β (x )∆t as the variance of the Gaussian distribution coefficient βτ (x) ·= Mτ (x), a truncation of tj j 2 2 the adjoint transition operator (189) at its second e−(x−µ) /(2σ ) summand reads Gµ,σ2 (x) ·= √ . (200) 2πσ2 † 1 2 Qe (x0, ∂x0 ) = ατ (x0)∂x0 + βτ (x0)∂ . (195) As before, ∆t = (t − t0)/N denotes the time intervals τ 2 x0 between t ≤ t ≤ ... ≤ t = t. Thus, the corresponding Kramers-Moyal expan- 0 1 N Upon identifying x(t0 + j∆t) with xj, the sion (188) recovers the backward Fokker-Planck equa- representation (199) can be rewritten as tion (2). Since the action (191) evaluates to t] t −S†  1  p(t, x|t0, x0) = e δ(x − x(t)) . (201) † ·   2 x(t0)=x0 S ·= dτ iq ∂τ x − ατ (x) + βτ (x)q , (196) (t0 ˆt 2 0 This integral proceeds only over paths x(τ) with the corresponding path integral (190) coincides with a τ ∈ (t0, t] because the q-variables have already been classic path integral representation of the (backward) integrated over. Paths are weighed by the exponential Fokker-Planck equation. The original development † factor e−S with the action of this representation goes back to works of Martin, N−1  2 Siggia, and Rose [16], de Dominicis [17], Janssen [18, † X xj+1 − xj − αtj (xj)∆t S ·= lim . (202) 19], and Bausch, Janssen, and Wagner [19]. The N→∞ 2β (x )∆t j=0 tj j application of this path integral to stochastic processes One may abbreviate the continuous-time limit of this is, for example, discussed in the book of Täuber [70]. action by the integral The transition operator (195) of the backward 2 Fokker-Planck equation has the same form as the t ∂ x − α (x) S† = dτ τ τ . (203) transition operator (176) in section 4.3, but it does ˆt0 2βτ (x) Master equations and the theory of stochastic path integrals 43

The integrand of this action is called an Onsager- The Itô version of this action with κ = 0 has proved Machlup function [79,393] and the representation (201) to be convenient in perturbation expansions of path may, consequently, be called an Onsager-Machlup rep- integrals because so called “closed response loops” can resentation (or “functional” in the sense of functional be omitted right from the start (see section 4.5 in [70]). integration). Note that the Onsager-Machlup repres- Moreover, this prescription does not require a change entation involves the limit of variables and makes the connection to the Feynman- t] N−1 Kac formula in appendix A most apparent. The Y dxj+1 ·= lim . (204) Stratonovich version of the action with κ = 1 is, for N→∞ ˆ p2πβ (x )∆t 2 (t0 j=0 R tj j example, employed in Seifert’s review on stochastic Mathematically rigorous formulations of the Onsager- thermodynamics [237]. For a recent, more thorough Machlup representation have been attempted in [394– discussion of the above discretization schemes, as well 396]. For the other path integrals discussed in this as of conflicting approaches, we refer the reader to [398] review, such attempts have not yet been made to our (the above action is discussed in the appendices). knowledge. 4.4.6. Wiener’s path integral. Before returning to 4.4.5. Alternative discretization schemes. Up to this processes with a discrete state space, let us briefly note point, we have discretized time in such a way that the how the Onsager-Machlup representation (201) relates evaluation of the path integrals eventually proceeds via to Wiener’s path integral of Brownian motion [80, 81], the solution of an Itô stochastic differential equation as it is discussed in [399]. It turns out that the (cf. section 4.3). Alternative discretization schemes Onsager-Machlup representation in fact coincides with have been proposed as well and are, for example, that path integral for one-dimension Brownian motion, employed in stochastic thermodynamics [237]. To for which the drift coefficient vanishes and the diffusion illustrate these schemes, let us, for simplicity, assume coefficient D = βτ is constant. Since the convolution that the drift coefficient α(x) does not depend on of two Gaussian distributions is again a Gaussian distribution, with means and variances being summed, time and that the diffusion coefficient D ·= βτ is constant. A general discretization scheme — called the the solution of the process follows as p(t, x|t0, x0) = G (x − x ). α-scheme but here we use a κ — consists of shifting the 0,D(t−t0) 0 argument xj of the drift coefficient in the action (202) to x¯j ·= κ xj+1 + (1 − κ)xj by writing 4.5. Further exact solutions and perturbation expansions xj =x ¯j − κ(xj+1 − xj) (205) (with κ ∈ [0, 1]). Afterwards, the drift coefficient is After this detour to processes with continuous state expanded in powers of x − x , which is of order spaces, let us return to the evaluation of the backward √ j+1 j path integral representation (171). In the following, ∆t along relevant paths. Following [397], it suffices we show how the method introduced in section 4.3 can to keep only the first two terms of the expansion of be applied to several elementary jump processes. We α(x ) so that j already applied a simplified version of the method in xj+1 − xj − α(xj)∆t (206) section 4.2 to solve the bi-directional reaction ∅ A.  0  ≈ 1 + κ α (¯xj)∆t (xj+1 − xj) − α(¯xj)∆t . We now consider a generic reaction k A → l A with rate coefficient γτ . Using the Poisson basis function As the next step, the integration variables x ,..., x n −x 1 N |ni = x e , the adjoint transition operator of this are transformed according to x n! reaction can be written as (cf. section 3.2.2)  0  1 + κ α (¯xj)∆t (xj+1 − xj) 7→ xj+1 − xj . (207) † † k l k Qeτ (x, ∂x) = Q (c, a) = γτ c (a − a ) (209) The Jacobian of this transformation vanishes every- ∂2 where except on its diagonal and sub-diagonal. Its de- = γ (l − k)xk∂ + γ l(l − 1) − k(k − 1)
xk x τ x τ 2 terminant therefore contributes an additional factor to † the path integral and requires a redefinition of the ac- + Pτ (x, ∂x) . (210) tion (202) as Here we employed the creation operator c = x and the N−1  2 annihilation operator a = ∂x + 1, and we performed † X  xj+1 − xj − α(¯xj)∆t 0  S ·= lim + κ α (¯xj)∆t . a series expansion with respect to ∂x. Terms of N→∞ 2D∆t j=0 third and higher order in ∂x were shoved into the perturbation operator P†(x, ∂ ). In the following, One may abbreviate this limit by τ x we approximate the marginalized distribution of the 2 t ∂ x − α(x)  reaction k A → l A by first dropping both the diffusion S† = dτ τ + κ α0(x) . (208) coefficient and the perturbation operator from (210). ˆt0 2D Master equations and the theory of stochastic path integrals 44

Afterwards, we reintroduce the diffusion coefficient x(τ), the Itô SDE is readily solved by x(t) = x(t0) + t t √ and show how the marginalized distribution follows dτ2γτ + dW (τ) 2γτ . After introducing the t0 t0 as the average of a Poisson distribution over the (rather´ daunting)´ parameter paths of an Itô SDE. For the pair generation process t t √ DD −( dτ γτ + dW (τ) 2γτ ) · t0 t0 ∅ → 2 A, this representation is exact because the ηk ·= e ´ ´ (213) perturbation operator associated to this process is zero √ t t k  1/2 dW (τ) 2γτ  EE (cf. section 4.5.1). Later, in section 4.5.4, we solve a t0 · dτ 2γτ + , † ´ 1/2 W process with a non-vanishing perturbation operator P . ˆt0 t τ dτ 2γτ t0 As a first approximation of the reaction k A → l A, ´ let us drop all the terms of the adjoint transition one can employ the binomial theorem to rewrite the operator (210) except for the drift coefficient ατ (x) = right-hand side of the marginalized distribution (211) k γτ (l − k)x . The SDE (181) then simplifies to the as k t n deterministic rate equation ∂τ x = γτ (l − k)x of the −(x(t0)+ t dτ γτ )   t e 0 X n  k/2 reaction. Its solution x(τ) acts as the mean of the ´ dτ 2γ η x(t )n−k . n! k ˆ τ k 0 marginalized distribution, which, according to (177)– k=0 t0 (180), is again given by the Poisson distribution If the rate coefficient γ is independent of time, one |p(t, n|t )i = |ni in (174). 0 x(t0) t,x(t) can show that the parameter ηk coincides with the k- Going one step further, one may keep the diffusion th moment of a Gaussian distribution with zero mean coefficient in (210). The marginalized distribution then and unit variance (i.e. ηk = 0 for odd k and ηk = reads (k − 1)!! for even k; note that the sum of two Gaussian DDx(t)ne−x(t) EE random variables is again a Gaussian random variable, |p(t, n|t0)ix(t ) = (211) 0 n! W with its mean and variance following additively). The marginalized distribution |p(t, n|t0)i therefore with x(τ) solving the Itô SDE x(t0) follows as k dx = γτ (l − k)x dτ (212) t k bn/2c
n−2k t dτ γτ q −(x(t0)+ dτ γτ ) X t0 (x(t0))   k e t0 . (214) + γτ l(l − 1) − k(k − 1) x dW (τ) . ´ ´ k! (n − 2k)! k=0 Hence, the Poisson distribution in (211) is averaged Here, bn/2c represents the integral part of n/2. over all possible sample paths of the SDE (212), The marginalized distribution (214) solves the master whose evolution starts out from the initial value x(t0) equation of the pair generation process and is initially (cf. (136)). The above expression has also recently of Poisson shape. For x(t0) = 0, the distribution been derived by Wiese [336], although based on the effectively keeps that shape, although only on the set forward path integral that we will discuss in section 5. of all even numbers (the reaction ∅ → 2 A cannot Apparently, the expression under the square root create an odd number of particles when starting out in (212) is non-negative only if k ∈ {0, 1} or if from zero particles). Using Mathematica by Wolfram l ≥ k ≥ 2. If that is not the case, x(τ) strays Research, we verified that the distribution also applies off into the complex domain. SDEs with imaginary when γτ depends on time. The evolution of the noise (or the corresponding Langevin equations) have distribution is shown in figure 7 for the rate coefficient been studied in several recent articles, most notably √ γτ = 1/(1 + τ). We refrain from computing for the binary annihilation reaction 2 A → ∅ and for the conditional distribution from the marginalized the coagulation process 2 A → A [336, 400, 401]. The distribution because the computation is unwieldy and numerical evaluation of such SDEs, however, often does not shed any more light on the path integral encountered severe convergence problems [336, 379]. approach. The appearance of imaginary noise has been linked to the anti-correlation of particles in spatial systems [400], 4.5.2. Diffusion on networks. The backward path but as (212) shows, no spatial degrees of freedoms integral can also be used to solve spatial processes. In are actually required for its emergence. As Wiese fact, stochastic path integrals have been most useful pointed out, imaginary noise generally appears when, in the study of such processes. If particles engaging over time, the marginalized distribution (211) becomes in a chemical reaction can also diffuse in space, their narrower than a Poisson distribution [336]. density may evolve in ways that are not expected from the well-mixed, non-spatial limit. That is, for example, 4.5.1. Pair generation. For the pair generation the case for particles that annihilate one another in process ∅ → 2 A with growth rate coefficient γτ , the the reaction 2 A → ∅ while diffusing along a one- drift and the (squared) diffusion coefficients agree: dimensional line. From the well-mixed limit, one would ατ = βτ = 2γτ . As neither of them depends on expect that the particle density decays asymptotically Master equations and the theory of stochastic path integrals 45

then obeys the master equation t = 22

〉 0.20 ) X X 

0 ∂τ p(τ, n|·) = ετ,i (ni + 1)p(τ, n + eˆi − eˆj|·) ⎜ t = 24 n 0.15 i∈L j∈Ni ( t , 6 p  ⎜ t = 2 − nip(τ, n|·) . (215) 0.10 8 t = 2 Here, eˆi represents a unit vector that points in 10 t = 2 12 direction i ∈ L. Moreover, ετ,i acts as a hopping rate 0.05 t = 2 and may depend both on time and on the node from Probability which a particle departs. Apparently, the above master 0.00 equation has the same structure as the chemical master 0 50 100 150 200 250 300 equation (24). It may therefore be cast into a linear Number of particles n PDE by extending the operators and basis functions Figure 7. Evolution of the marginalized distribution from section 3.2.2 to multiple variables. In particular, |p(t, n|t0)i in (214), which solves the pair generation process we extend the Poisson basis function to ∅ → 2 A (with t ·= 0). The initial mean of the distribution 0 · Y xni e−xi was chosen as x(t0) = 0 and the rate coefficient of the reaction · i √ |nix ·= (216) as γτ = 1/(1 + τ). ni! i∈L and employ the creation and annihilation operators as t−1 with time (see section 7), but instead it decays as c ·= x and a ·= ∇ + 1 . (217) t−1/2 [130,294,402]. The reason behind this surprising decay law is the rapid condensation of the system into a According to the flow equation (116), the marginalized † state in which isolated particles are separated by large distribution evolves via ∂−τ |p(t, n|τ)i = Qeτ |p(t, n|τ)i voids. From that point on, further decay of the particle with the adjoint transition operator density requires that two particles first find each other † X Qeτ (x, ∇) = (∆ετ,ixi)∂xi . (218) through diffusion. Therefore, the process is “diffusion- i∈L limited” and exhibits a slower decay of the particles Here we introduced the discrete Laplace operator ∆fi · (see e.g. [403] on the related “first-passage” problem). P = (fj − fi), which acts both on ετ,i and on xi. Path integral representations of the master equation, j∈Ni By following the steps in section 4.3, one finds combined with renormalization group techniques, have that the marginalized distribution is given by the been successfully applied to the computation of decay multivariate Poisson distribution laws in spatially extended systems, both regarding ni −xi(t) Y xi(t) e the (universal) exponents and the pre-factors of these |p(t, n|t )i = . (219) 0 x(t0) laws [130–132,294,306,404]. For a broader discussion of ni! i∈L systems exhibiting a transition into an absorbing state Its mean x(t) solves the discrete diffusion equation see [68]. Before turning to the particle density and, X
more generally, to observables of the particle number, ∂τ xi = ετ,jxj − ετ,ixi = ∆ετ,ixi , (220) we now show how the backward path integral helps in j∈Ni computing the full probability distribution of a spatial with the initial condition x(t0). Let us note process. that the marginalized distribution solves the master As a first step, we consider a pure diffusion process equation (215), but with the initial number of particles with particles hopping between neighbouring nodes of being Poisson distributed locally. a network . For now, the topology of the network L The conditional distribution p(t, n|t0, n0) follows may be arbitrary, being either random or regular. In from the marginalized distribution (219) by applying the next section, the network topology will be chosen the functional as a regular lattice. In the limit of a vanishing lattice hY n0,i i hn | f = ∂ + 1
f(x(t )) (221) 0 x(t0) xi(t0) 0 spacing (and an infinite number of lattice sites), a “field x(t0)=0 theory” will be obtained. In section 4.5.4, the particles i∈L will also be allowed to decay. to it (cf. (126) with (129)). The evaluation requires a The configuration of particles on the network may prior solution of the discrete diffusion equation (220). |L| This equation can be written in matrix form as be represented by the vector n ∈ N0 , with |L| being the total number of network nodes. The configuration ∂τ x = Mτ x with Mτ,ij = (Aij − |Ni|δij)ετ,j. Here, A represents the symmetric adjacency matrix of the changes whenever a particle hops from some node i ∈ L network and |N | represents the number of neighbours to a neighbouring node j ∈ Ni ⊂ L. The probability i of node i ∈ . The matrix equation can in principle p(τ, n|t0, n0) of finding the system in configuration n L be solved through a Magnus expansion. The solution has the generic form x(τ) = G(τ|t0)x(t0) with the Master equations and the theory of stochastic path integrals 46

−d “propagator” G solving ∂τ G(τ|t0) = Mτ G(τ|t0). We path integral variable as q(τ, r) ·= l qr(τ) so that in write the elements of the propagator as G(τ, i|t0, j). the limit l → 0, Its flow starts out from G(t0|t0) = 1. For later t † X
 purposes, let us note the time-reversal property in S = dτ iqr ∂τ xr − ∆ετ,rxr (226) ˆt terms of the matrix inverse G(t|τ)−1 = G(τ|t) and also 0 r∈L t the conservation law 1|G(τ|t0) = 1|.
→ dτ dr iq ∂τ x − ∆Dτ x . (227) It proves insightful to consider the master ˆ ˆ d t0 R equation (215) of the multi-particle hopping process The above rescaling of x and q is not unique for the random walk of a single particle on the one- and depends on the problem at hand. Instead of dimensional lattice = , which we already considered d L Z dividing qr by the volume factor l , this factor is in sections 2.2.1 and 3.2.1 (with symmetric hopping sometimes employed to cast xr and the particle number rates ε = l = r ). The presence of only a single τ,i τ τ nr into densities [130]. In the study of branching particle can be enforced by choosing n0 as n0,k = 1 for and annihilating random walks with an odd number one k ∈ Z and n0,j = 0 for all j 6= k. By following of offspring, yet another kind of rescaling may bring the above steps, one eventually finds that the master the action into a Reggeon field theory [70,405–407] like equation (215) is solved by the conditional probability form [96]. The critical behaviour of these random walks distribution falls into the universality class of directed percolation XY
(DP) [96, 132]. Information on this universality class p(t, n|t0, n0) = δnj ,0 δni,1G(t, i|t0, k) , (222) can be found in the book [68], as well as in the original i∈Z j6=i articles of Janssen and Grassberger, which established with the propagator G solving the master equation (55) the extensive scope of the DP class [408, 409]. of the simple random walk. Hence, the propagator evaluates to a Skellam distribution. 4.5.4. Diffusion and decay. Let us exemplify how one can accommodate a non-vanishing perturbation 4.5.3. Diffusion in continuous space. To make the operator P† in the adjoint transition operator (176). transition to a field theory, one may specify the network τ As in the section before the last, we consider a as the the d-dimensional lattice = (l )d with the L Z system of particles that are hopping between the lattice spacing l > 0 going to zero. In order to take nodes of an arbitrary network . The corresponding this limit, we define the variable x(τ, r) ·= x (τ), L · r diffusive transition operator (218) shall specify the the rescaled Laplace operator ∆ ·= ∆/l2, and the · drift coefficient α (x) = ∆ε x in the multivariate rescaled diffusion coefficient D (r) ·= l2ε for r ∈ . τ,i τ,i i τ · τ,r L extension of (176). The coefficient β (x) is zero. Assuming that these definitions can be continued to τ,ij Besides allowing for diffusion, we now allow the r ∈ d, the discrete diffusion equation (220) becomes R particles to decay in the linear reaction A → ∅. For a PDE for the “field” x(τ, r) in the limit l → 0, namely the sake of brevity, we assume that the corresponding ∂ x(τ, r) = ∆(D (r)x(τ, r)). Here, ∆ represents τ τ decay rate coefficient µ is spatially homogeneous. the ordinary Laplace operator.10 The solution of the τ This assumption can be relaxed but the equations that PDE acts as the mean of the multivariate Poisson result cannot be written in matrix form and involve distribution (219) whose extension to r ∈ is, R many indices. We treat the adjoint transition operator however, not quite obvious. If the diffusion coefficient of the decay process as the perturbation, which reads, is homogeneous in space, the PDE is solved by according to the flow equation (131), † X x(τ, r) = dr0 G(τ, r|t0, r0)x(t0, r0) , (224) Pτ (x, ∇) = −µτ xi∂xi . (228) ˆ d R i∈L with the Gaussian kernel The combined process could also be solved directly −(r−r0)2/4 τ ds D e τ0 s by treating the decay process as part of the drift 0 0 ´ G(τ, r|τ , r ) = q . (225) coefficient α (x). For pedagogic reasons, however, we 4π τ ds D τ,i τ 0 s wish to outline a perturbative solution using Feynman ´ The action (172) can also be extended into continuous diagrams. space. For that purpose, one may rescale the second The first step of the derivation is to solve the differential equation (181) for x, which now reads 10 d With Nr denoting the neighbouring lattice site of r ∈ (l ) , Z ∂τ xi = ∆ετ,ixi + Xi(τ). The homogeneous solution of the ordinary Laplace operator follows as the limit of a finite- · difference approximation, i.e. as this equation is given by xh(τ) ·= G(τ|t0)x(t0), with G being the propagator from the end of section 4.5.2. d X f 0 − fr X r → ∂2 f(r) = ∆f(r) . (223) The solution of the full equation can be written as l2 ri 0 i=1 τ r ∈Nr 0 0 0 x(τ) = xh(τ) + dτ G(τ|τ )X(τ ) . (229) ˆt0 Master equations and the theory of stochastic path integrals 47

As the next step, the evaluation of the marginalized t dτ1G(t|τ1) xh(τ1) t0 distribution (177) requires us to compute the (Q, X)- ´ . (235) generating functional (179) for Q = X = 0. iqN · −µτ1 By performing a series expansion of its leading exponential, this function can be written as Note that dangling outgoing lines are not permitted because the derivative δ ln Z0 /δX(τ) vanishes for ∞ k Q,X X X 1  δ l Q = X = 0. Z = iq · (230) 0,0 l! N δQ(t) For more complex, non-linear processes, the k=0 l=0 Feynman diagrams may contain multiple kinds of t τ2 † † 0 vertices, each representing an individual summand · dτk−l P ··· dτ1 P Z , ˆ τk−l ˆ τ1 Q,X Q=X=0 † t0 t0 of Pτ . If there exist vertices with more than two † δ δ legs, the diagrams may exhibit internal loops (see with Pτ = · (−µτ ) . To evaluate Z0,0, it δX(τ) δQ(τ) section 6.4). It is then usually impossible to evaluate helps to note that ln Z0 = t dτ Q · x, from which Q,X t0 the full perturbation series and it needs to be truncated it follows that for t0 ≤ τ ≤ t: ´ at a certain order in the number of loops. Further 0 δ ln ZQ,X information about these techniques, and about how = xh(τ) and (231) renormalization group theory comes into play, is δQ(τ) Q=X=0 provided, for example, by the book of Täuber [70]. δ2 ln Z0 Q,X 0 0 Information on a non-perturbative renormalization 0 = G(τ, i|τ , j)Θ(τ − τ ) . (232) δQi(τ)δXj(τ ) Q=X=0 group technique can be found in [410]. For the simple diffusion process with decay, The Heaviside step function is used with Θ(0) ·= 0 to 0 all the summands of (230) are readily cast into take into account that xh(τ) depends on X(τ ) only for τ 0 < τ (see appendix D). All other derivatives of Feynman diagrams. However, it turns out that 0 individual summands of the expansion may be ln ZQ,X , as well as itself, vanish for Q = X = 0. Upon inserting the perturbation (228) into (230), associated to multiple diagrams that are not connected one observes that every summand of the expansion to one another. Furthermore, diagrams that can be written in terms of a combination of the represent summands of lower order keep reappearing as unconnected components of summands of higher order. derivatives (231) and (232), and a terminal factor iqN . For example, the summand with k = 2 and l = 1 reads Hence, there appears to be redundant information involved. This redundancy is removed by a classical δ t δ δ iq · dτ ·(−µ ) Z0 , (233) theorem from diagrammatic analysis. This theorem N δQ(t) ˆ 1 δX(τ ) τ1 δQ(τ ) Q,X t0 1 1 states that the logarithm ln Z0,0 is given by the sum with Q = X = 0 being taken after the evaluation of only the connected diagrams [411], i.e. by of the functional derivatives. The evaluation of these ln Z0,0 = + derivatives results in (236) t + .... iq · dτ G(t|τ )(−µ )x (τ ) . (234) + N ˆ 1 1 τ1 h 1 t0 This sum can be evaluated with the help of One can represent this expression graphically by a G(t|τ)G(τ|t0) = G(t|t0) as Feynman diagram according to the following rules. ∞ X t First, every diagram ends in a sink , which ln Z = iq · dτ G(t|τ )(−µ ) (237) 0,0 N ˆ k k τk contributes the factor iqN . The incoming line, or k=0 t0 δ “leg”, of the sink represents the derivative . This τ2 δQ(t) ··· dτ G(τ |τ )(−µ )x (τ ) leg may either be left dangling, resulting in a factor 1 2 1 τ1 h 1 ˆt0 xh(t) according to (231), or it may be “contracted” t − dτ µτ t0 with the outgoing line of a vertex . The = iqN · xh(t)e ´ . (238) two legs of this vertex reflect the two derivatives in After inserting this expression into the marginalized † δ δ the perturbation Pτ = δX(τ) · (−µτ ) δQ(τ) . According distribution (177), one recovers the multivariate to (232), the contraction results in a propagator Poisson distribution (219). Its mean, however, has now t − dτ µτ G(t, i|τ, j) and the vertex itself contributes a factor t0 acquired the pre-factor e ´ , reflecting the decay −µτ . The incoming leg of the vertex may again be of the particles. left dangling or it may be connected to another vertex. Therefore, each Feynman diagram is a straight line for the linear decay process. The expression (234) can therefore be represented graphically by Master equations and the theory of stochastic path integrals 48

4.6. Résumé 5. The forward path integral representation

In the present section, we introduced the novel Thus far, we have focused on the backward path backward path integral representation integral (162). This integral will be used again in t] section 6 to derive a path integral representation of −S† |p(t, n|t0)i = e |nit (239) averaged observables. Moreover, we use it in section 7.4 (t 0 to approximate the binary annihilation reaction 2 A → of the marginalized distribution (cf. section 3) ∅. In the present section, however, we shift our focus to X the forward path integral (161). Its derivation proceeds |p(t, n|t0)i = p(t, n|t0, n0)|n0i . (240) t0 analogously to the derivation of the backward solution, n0 so we keep it brief. When the basis function |nix is chosen as a probability n −x distribution (e.g. |ni = x e for all n ∈ x n! 5.1. Derivation N0), the backward path integral represents a true probability distribution: the marginalized distribution The forward path integral can be derived both from the |p(t, n|t0)i. This distribution solves the forward master flow equation (54) obeyed by the generating function or equation (10) for the initial condition |p(t0, n|t0)i = from the flow equation (150) obeyed by the series (149) |ni and it transforms into the conditional probability (cf. figure 8). As it is more convenient to work distribution as p(t, n|τ, n0) = hn0|p(t, n|τ)i. In with functions than with functionals, we use the flow section 4.3, we showed how the backward path equation of the generating function for this purpose. integral (239) can be expressed in terms of an The derivation parallels a derivation of Elgart and average over the paths of an Itô stochastic differential Kamenev [35]. As in section 4.1, we first split the time equation. This method provided the exact solutions interval [t0, t] into N pieces t0 ≤ t1 ≤ ... ≤ tN ·= t of various elementary stochastic processes, including of length ∆t. Over a sufficiently small interval ∆t, the the simple growth, the linear decay, and the pair flow equation (54) is then solved by generation processes. Moreover, we showed how the |g(t|t0, n0)i = Lt (qN , ∂q )|g(tN−1|t0, n0)i , (241) path integral can be evaluated perturbatively using qN N−1 N qN 2
Feynman diagrams for a process with diffusion and with the generator Lτ ·= 1 + Qeτ ∆t + O (∆t) . After linear particle decay. We hope that the backward path inserting the integral form of a Dirac delta between L integral (239) will prove useful in the study of reaction- and |gi, the right-hand side of the equation reads diffusion processes. Thus far, the critical behaviour dq dx N−1 N−1 −ixN−1(qN−1−qN ) of such processes could only be approached via path LtN−1 (qN , ∂qN ) e ˆ 2 2π integral representations of averaged observables or of R · |g(tN−1|·)i . (242) the generating function [69, 412]. In section 6, we qN−1 show how the former representation readily follows Assuming that the transition operator Qeτ , and from the backward path integral (239) upon summing therefore also Lτ , are normal-ordered with respect to q the marginalized distribution over an observables A(n). and ∂q, we may replace ∂q by ixN−1 and interchange The corresponding representation is commonly applied N Lt with the exponential. This procedure is repeated in the study of diffusion-limited reactions (e.g. [69, N−1 N times before invoking the exponentiation Lτ = 96, 130–132], but we here show how it can be freed exp (Qeτ ∆t). Using the inverse transformation (48) and of some of its quantum mechanical ballast (such the initial condition |g(t0|t0, n0)i = |n0i, one obtains as “second-quantized” or “normal-ordered” observables the discrete-time path integral representation and coherent states). Besides, we showed in section 4.4 p(t, n|t , n ) = hn| |g(t|t , n )i (243) how one can derive a path integral representation 0 0 t,qN 0 0 qN of processes with continuous state spaces whose N−1 with |g(t|t , n )i = e−SN |n i and (244) (backward) master equations admit a Kramers-Moyal 0 0 qN 0 t0,q0 0 expansion. Provided that this expansion stops at N−1  q − q  the level of a diffusion approximation, one recovers a · X j j+1 SN ·= ∆t ixj − Qetj (qj+1, ixj) . (245) ∆t classic path integral representation of the (backward) j=0 Fokker-Planck equation and also the Feynman-Kac formula (3). Moreover, the representation can be Here we again employed the abbreviation (167), i.e. rewritten in terms of an Onsager-Machlup function l l Y dqj dxj and, for diffusive Brownian motion, it simplifies to the = . (246) ˆ 2 2π path integral of Wiener. k j=k R

Moreover, the initial condition p(t0, n|t0, n0) = δn,n0 is again trivially fulfilled for N = 0. Upon taking Master equations and the theory of stochastic path integrals 49

Forward master Flow equation (54) of equation (10) 2.1 the generating function 5.1 ∂τ p(τ, n ⎜⋅) = … ∂τ ⎜g(τ ⎜⋅)〉 = … Forward path integral Evaluation in terms of an representation (247)-(249) 5.3 average over the paths of a

t) − backward-time SDE ⎜ 〈 ⎜ ⎜ 〉 p(t, n t0, n0) = n t e n0 t0 Backward master Flow equation (150) of [t0 equation (12) 3.4 the series expansion (149) ∂−τ p(⋅ ⎜τ, n0) = … ∂−τ 〈 p(⋅ ⎜τ)⎜ = …

Figure 8. Outline of the derivation of the forward path integral representation and of its evaluation in terms of an average over the paths of a backward-time stochastic differential equation. the continuous-time limit N → ∞, so that ∆t → 0, For the linear growth, or Yule-Furry [119, the forward path integral representation of the master 413], process A → 2 A (l = 2), the inverse equation follows as transformation (247) casts the generating function into p(t, n|t , n ) = hn| |g(t|t , n )i (247)  1  0 0 t,q(t) 0 0 q(t) n p(t, n|t0, n0) = ∂q(t)|g(t|t0, n0)iq(t) (255) t) n! q(t)=0 −S with |g(t|t0, n0)i = e |n0i (248)   t t q(t) t0,q(t0) n − 1 − dτ µτ n0 − dτ µτ n−n0 t0
t0
[t0 = e ´ 1 − e ´ (256) t n − n0   and S ·= dτ ix∂−τ q − Qeτ (q, ix) . (249) for n > 0 and into δ for n = 0. The above ˆ 0,n0 t0 distribution has the form of a negative Binomial t) N−1 t · − dτ µτ The limit ·= limN→∞ now involves t0 [t0 0 distribution with probability of success e ´ , integrations overffl x(t0) and q(t0),ffl but not over x(t) and number of failures n − n0, and number of successes t dτ µτ q(t). t0 n0 [414]. The mean value e´ n0 of the marginalized distribution (256) grows exponentially 5.2. Linear processes with time as long as µτ > 0, and so does its variance t t dτ µτ dτ µτ t0 t0 The forward path integral (248) can, for example, be (e´ − 1) e´ n0. used to derive the generating function of the generic For the linear decay process A → ∅ (l = 0), the linear process A → l A with rate coefficient µτ (and inverse transformation in (255) instead recovers the Binomial distribution (134). l ∈ N0). For that purpose, we choose the basis function n as |niτ,q = q so that |gi coincides with the ordinary generating function. Since the basis function does 5.3. Intermezzo: The forward Kramers-Moyal not depend on time, (64) alone specifies the transition expansion operator The above procedure can be generalized to processes l Qeτ (q, ∂q) = µτ (q − q)∂q (250) whose transition operator is of the form of the flow equation ∂ |g(τ|·)i = Q (q, ∂ )|g(τ|·)i. τ eτ q 1 2 Qeτ (q, ∂q) = ατ (q)∂q + βτ (q)∂ + Pτ (q, ∂q) . (257) Consequently, the action 2 q t  l  The derivation proceeds analogously to section 4.3 and S = dτ ix ∂−τ q − µτ (q − q) (251) ˆt0 appendix D. The probability distribution is thereby is linear in ix. To evaluate the path integral (248), it expressed as an average over the paths of a stochastic helps to reconsider the discrete-time approximation differential equation proceeding backward in time. The N−1 merit of such a representation remains to be explored. X  l  In the following, we briefly outline how the procedure SN = ixj qj − (qj+1 + µtj (qj+1 − qj+1)∆t) .(252) j=0 is applied to processes with continuous sample paths. In section 4.4, we explained how the forward of the action. Upon integrating over all the qj- variables and taking the limit ∆t → 0, one obtains master equation (184) of a process with a continuous the generating function state space can be written in terms of the (forward)

n0 Kramers-Moyal expansion |g(t|t0, n0)iq(t) = q(t0) (253) ∞ m l X (−1) m (m)  with q(τ) solving ∂−τ q = µτ (q − q). The unique real ∂ p(τ, q|·) = ∂ M (q)p(τ, q|·) (258) τ m! q τ solution of this equation with final value q(t) reads m=1 q(t) with initial condition p(τ, q|t , q ) = δ(q −q ). Here we q(τ) = . (254) 0 0 0  (l−1) t ds µ 1/(l−1) q(t)l−1 + e τ s (1 − q(t)l−1) changed the letter from x to q to keep in line with the ´ Master equations and the theory of stochastic path integrals 50 notation used in sections 5.1 and 5.2. If the Kramers- Thus, the leading exponential in (265) converted the Moyal expansion stops with its second summand, it argument of the Dirac delta from the solution of coincides with the (forward) Fokker-Planck equation a final value problem to the solution of an initial 1 value problem. It is, of course, no surprise that ∂ p(τ, q|·) = −∂ M(1)(q)p + ∂2M(2)(q)p . (259) τ q τ 2 q τ the probability distribution is given by a Dirac delta To apply the procedure from the previous section to function because the process is purely deterministic. this equation, it needs to be brought into the form Another simple process that can be solved with ∂τ p(τ, q|·) = Qeτ (q, ∂q)p(τ, q|·) with a normal-ordered the help of (265) is the pure diffusion process with (1) (2) transition operator Qeτ (q, ∂q). It is readily established Mτ = 0 and Mτ = D. The derivation is performed that this operator can be expressed in the form of (257) most easily in the discrete-time approximation and with the coefficients results in the Wiener path integral (1) (2) ατ (q) ·= −Mτ + ∂qMτ , (260) p(t, q|t0, q0) = (270) · (2) N N βτ (q) ·= Mτ , and (261) Y Y  lim d˜qj−1 G0,D∆t(˜qj−1 − q˜j) δ(˜q0 − q0) , (1) 1 2 (2) N→∞ ˆ Pτ (q) ·= −∂qM + ∂ M . (262) j=1 R j=1 τ 2 q τ The Fokker-Planck equation now has the same form as with q˜N ·= q and Gaussian distribution Gµ,σ2 the flow equation obeyed by the generating function in (cf. (200)). An evaluation of the convolutions of section 5.1. Therefore, we can follow the steps in that Gaussian distributions shows that the process is solved section to represent the solution of the Fokker-Planck by G0,D(t−t0)(q − q0). Further uses of the path integral equation by the path integral representation (263) of the (forward) Fokker-Planck t) equation remain to be explored. −S p(t, q|t0, q0) = e δ(q0 − q(t0)) (263) q(t)=q [t0 5.4. Résumé t   with S ·= dτ ix∂−τ q − Qeτ (q, ix) . (264) Here we derived the forward path integral representa- ˆ t0 tion The evaluation of this path integral proceeds analog- t) ously to the derivation in section 5.2 and appendix D. |g(t|t , n )i = e−S |n i (271) 0 0 0 t0 In particular, one can rewrite the probability distribu- [t0 tion as of the probability generating function |g(t|t , n )i = t 0 0 dτ Pτ (q(τ)) t0 P |ni p(t, n|t , n ). The conditional probability p(t, q|t0, q0) = e´ δ(q0 − q(t0)) W , (265) n t 0 0 with q(τ) solving the backward-time SDE distribution is recovered from the generating func- p tion via the inverse transformation p(t, n|t0, n0) = − dq(τ) = ατ (q(τ)) dτ + βτ (q(τ)) dW (τ) . (266) hn|g(t|t0, n0)i. The path integral representation of the The time evolution of this SDE starts out from the generating function has, for example, been employed final value q(t) = q. In a discrete-time approximation, to compute rare event probabilities [35] by a method the SDE reads that we discuss in section 7. Most often, however, the q representation has only served as an intermediate step qj − qj+1 = αt (qj+1)∆t + βt (qj+1) ∆Wj . (267) j j in deriving a path integral representation of averaged The increments ∆Wj are Gaussian distributed with observables. Such a representation is considered in the mean 0 and variance ∆t. Let us note that the next section. In section 5.2, we showed how the for- two path integral representations (190) and (263) of ward path integral (271) can be evaluated along the the conditional probability distribution belong to an paths of a differential equation proceeding backward infinite class of representations [42, 104, 415] (see also in time. We thereby obtained the generating function section 4.4.5). We focus on the two representations of generic linear processes. Besides, we derived a novel that follow from the backward and forward Kramers- path integral representation of processes with continu- Moyal expansions via the step-by-step derivations in ous state spaces in section 5.3, based on the (forward) sections 4.1 and 5.1, respectively. Kramers-Moyal expansion. The potential use of this To exemplify the validity of the path integral (263) representation remains to be explored. and of the representation (265), we consider a process with linear drift and no diffusion, i.e. a process with (1) (2) 6. Path integral representation of averaged jump moments Mτ = µq and Mτ = 0. For this observables process, the representation (265) evaluates to

−µ(t−t0) −µ(t−t0) p(t, q|t0, q0) = e δ(q0 − q e ) (268) The backward and forward path integral representa- µ(t−t0) tions of the conditional probability distribution provide = δ(q − q0 e ) . (269) Master equations and the theory of stochastic path integrals 51 a full characterization of a Markov process. Yet, an in- as tuitive understanding of how a process evolves is often t] −S† attained more easily by looking at the mean particle hAix(t0) = e hhAiix(t) (275) 2 (t number hni and at its variance n− hni
. Although 0 ∞ n −x both of these averages can in principle be inferred from X x e with hhAiix ·= A(n) . (276) n! a given probability distribution, it often proves con- n=0 venient to bypass the computation of the distribution † With the adjoint transition operator Qτ (c, a) = and to focus directly on the observables. Thus, we now k l k γτ c (a − a ) of the reaction k A → l A (cf. sec- show how one can derive a path integral representation † of the average A of an observable A(n). The path in- tion 3.2.2), the action S in (172) reads t tegral representation applies to all processes that can †  †  be decomposed additively into reactions of the form S = dτ iq∂τ x − Qτ (x, iq + 1) . (277) ˆt k A → l A in a well-mixed, non-spatial environment. 0 The “+1” in the transition operator is called the “Doi- The extension of the path integral to multiple types shift” [91]. In our above derivation, this shift followed of interacting particles and to processes with spatial from choosing a Poisson distribution as the basis degrees of freedom is straightforward. In the deriv- function. The unshifted version of the path integral ation, we assume that the number of particles in the can be derived by choosing the basis function of the system is initially Poisson distributed with mean x(t ). n0 0 x(t0) marginalized distribution (110) as |n0i = , This assumption is common in the study of reaction- x(t0) n0! diffusion master equations and will allow us to focus turning the average (274) into ∞ on the marginalized distribution from section 3 in- X hAi = e−x(t0) A(n)|p(t, n|t )i . (278) stead of on the conditional probability distribution. x(t0) 0 x(t0) The derived path integral has been applied in various n=0 contexts, particularly in the analysis of decay laws of Upon rewriting the marginalized distribution in terms diffusion-limited reactions and in the identification of of the backward path integral (171), the action (277) universality classes [69, 70]. acquires the addition summand x(t0) − x(t) and now † involves the unshifted transition operator Qτ (x, iq). 6.1. Derivation The average over a Poisson distribution in (276) establishes the link between the particle number n Assuming that the number of particles is initially and the path integral variable x. For the simplest Poisson distributed with mean x(t0), the probability observable A(n) ·= n, i.e. for the particle number itself, of finding n particles at time t is p(t, n) = it holds that hhnii = x. This relation generalizes to P x p(t, n|t0, n0)p(t0, n0). Here we make use of the n0 factorial moments of order k ∈ N for which hh(n)kiix = initial (single-time) distribution k x (recall that (n)k ·= n(n − 1) ··· (n − k + 1)). The n0 −x(t0) x(t0) e computation of factorial moments may serve as an p(t0, n0) = . (272) intermediate step in obtaining ordinary moments of the n0! particle number. For this purpose, one may use the Consequently, the average value of an observable A(n) k Pk k relation n = l=0 l (n)l, where the curly braces at time t evaluates to represent a Stirling number of the second kind (cf. ∞ X section 26.8 in [335]). An extension of the path integral hAi = A(n)p(t, n) . (273) x(t0) representation (275) to multi-time averages of the form n=0 ∞ Here we emphasize that the average depends on the X A(n , n )p(τ , n ; τ , n |t , n ) (279) mean of the initial Poisson distribution. As the single- 2 1 2 2 1 1 0 0 n ,n =0 time distribution coincides with the marginalized 2 1 remains open (with τ > τ > t ). The results of distribution |p(t, n|t0)i in (110) for the Poisson basis 2 1 0 n0 −x(t0) x(t0) e Elderfield [376] may prove helpful for this purpose. function |n0i = , the above average x(t0) n0! In a slightly rewritten form, the Doi-shifted path can equivalently be written as integral (275) was, for example, employed by Lee ∞ X in his study of the diffusion-controlled annihilation hAix(t ) = A(n)|p(t, n|t0)i . (274) 0 x(t0) reaction k A → ∅ with k ≥ 2 [130, 416]. He found n=0 that below the critical dimension dc = 2/(k − 1), The path integral representation of hAi then x(t0) the particle density asymptotically decays as n ∼ follows directly from the backward path integral −d/2 Akt with a universal amplitude Ak. At the critical representation (171) of the marginalized distribution dimension, the particle density instead obeys n ∼ Master equations and the theory of stochastic path integrals 52

1/(k−1) Ak(ln t/t) . Neglecting the diffusion of particles, following section outlines this derivation for the process the above action (277) of the reaction k A → ∅ reads k A → l A with time-independent rate coefficient γ. k Moreover, we assume A(n) to be analytic in n. t  X k  S† = dτ iq∂ x + γ xk (iq)j . (280) The alternative derivation of the path integral ˆ τ τ j t0 j=1 representation (275) starts out from the exponential Besides using different names for integration variables solution of the master equation, i.e. from (cf. (83)) Q(t−t0) (iq → ψ and x → ψ), the action employed by Lee p(t, n|t0, n0) = hn|e |n0i . (283) involves an additional boundary term. This term The bras are chosen as the unit row vectors hn| = eˆ| can be introduced by rewriting the path integral n and the kets as the unit column vectors |ni = eˆ . As representation (275) as n before, the transition matrix of the reaction k A → l A t] −S† with rate coefficient γ reads (cf. (85)) hAiλ0 = e hhAiix(t) . (281) l k k [t0 Q(c, a) = γ(c − c )a . (284) This path integral involves integrations over the The creation matrix fulfils c|ni = |n + 1i and hn|c = variables q(t ) and x(t ), and the mean of the initial 0 0 hn − 1|, and the annihilation matrix a|ni = n|n − 1i Poisson distribution is denoted by λ . Consequently, 0 and hn|a = (n + 1)hn + 1|. Thus, the basis column one needs to add the boundary term iq(t )(x(t ) − λ ) 0 0 0 vectors can be generated incrementally via |ni = cn|0i to the action (280) to equate x(t ) with λ . The factor n 0 0 and the basis row vectors via hn| = h0| a . iq(t )x(t ) of the new term is, however, often dropped n! 0 0 Since the bra hn| is a left eigenvector of the number eventually [130, 416]. The convention of Lee is also matrix N ·= ca with eigenvalue n, one may write commonly used, for example, by Täuber [70]. The · A(n)hn| = hn|A(N ) for an analytic observable A. unshifted version of the path integral with transition After inserting the exponential solution (283) into the operator Q† (x, iq) is recovered via iq + 1 → iq. τ averaged observable (282), an evaluation of the sums For completeness, let us note that the path therefore results in integral representation (275) can also be derived from a Q(c,a)(t−t0) x0(c−1) the forward path integral (244), provided that the hAix0 = h0|e A(ca)e e |0i . (285) observable A(n) is analytic in n. It then suffices Here we employed the (infinitely-large) unit matrix 1. to consider the factorial moment A(n) ·= (n)k. To Moreover, we changed the variable x(t0) to x0 in perform the derivation, one may choose the basis anticipation of a discrete-time approximation. · function of the generating function (45) as |ni ·= Following the lecture notes of Cardy [91], we now n · · · (iq + 1) (insert ζ ·= i, q˜ ·= 1 and x˜ ·= 0 into (70)). perform the Doi-shift by shifting the first exponential As the first step, the forward path integral (248) is in the above expression to the right. To do so, we summed over an initial Poisson distribution with mean require certain relations, which are all based on the x(t0). The average of the factorial moment is then 1 k commutation relation [a, c] = . First, it follows from obtained via h(n)kix(t0) = ∂iq |g(t|t0; x(t0))i |qN =0. n n−1 N qN this relation that [a, c ] = nc 1 holds for all n ∈ N0, To recover the action (277), one may note that the n n−j1 † and more generally that [a, [a ··· [a, c ]]] = (n)jc operator Qτ (c, a) in (64) and the operator Qτ (c, a) † holds for nested commutators with j ≤ n annihilation in (122) fulfil Qτ (iq + 1, x) = Qτ (x, iq + 1) for scalar matrices. Nested commutators of higher order vanish. arguments. Thus, both the marginalized distribution The Hadamard lemma [417] can therefore be employed approach and the generating function approach result to write (with z ∈ and n ∈ ) in the same path integral representation of averaged C N 1 observables. A detailed derivation of the path ezacn = cn + [a, cn]z + [a, [a, cn]]z2 + ...
eza (286) integral (275) from the generating function is, for 2 n example, included in the article of Dickman and = cn + ncn−1z 1 + cn−2z2 1 + ...
eza (287) Vidigal [412] (see their equations (106) and (108)). 2 = (c + z1)neza . 6.2. Intermezzo: Alternative derivation based on This expression generalizes to the following shift coherent states operations for an analytic function f: We noted previously in section 2.2.3 that the path ezaf(c) = f(c + z1)eza and (288) integral representation (275) of the average f(a)ezc = ezcf(a + z1) . (289) ∞ x(t )n0 e−x(t0) X 0 Shifting of the first exponential in (285) to the right hAix(t0) = A(n) p(t, n|t0, n0) (282) n0! n0,n=0 therefore results in 1 can be derived without first casting the master 1
Q(c+ ,a)(t−t0) x0c hAix0 = h0|A (c + )a e e |0i . (290) equation into a linear PDE [91,130,289,336,416]. The Master equations and the theory of stochastic path integrals 53

To proceed, we split the time interval [t0, t] into relation of Touchard polynomials (see [419, 420] for N pieces of length ∆t ·= (t − t0)/N. The Trotter information on these polynomials): formula [418] can then be used to express the right j   hand side of (290) in terms of the limit X j fj+1(x) = x fi(x) with f0(x) = 1 . (297) N i lim h0|A(c + 1)a
1 + Q(c + 1, a)∆t ex0c|0i.(291) i=0 N→∞ Since the j-th Touchard polynomial fj(x) agrees with This expression is now rewritten by inserting N of the the j-th moment of a Poisson distribution with mean x, j identity matrices [15] i.e. with hhn iix as defined in (276), our above assertion ∞ holds true. X 1  dq  1 = dx (∂n xm) e−iqx |mihn| (292) The path integral representation (275) of averaged m! ˆ x ˆ 2π m,n=0 R R observables is recovered as the continuous-time limit dx dq of (296) (upon rewriting 1 + Q∆t as an exponential). = e−(iq−1)x|xiihh−iq| . (293) ˆ 2 2π The action (277) is also recovered because the R transition matrix Q(c, a) in (284) and the adjoint We obtained the expression in the second line by transition operator Q† (c, a) in (122) fulfil Q(iq+1, x) = performing integrations by parts while neglecting any τ Q†(x, iq + 1) for scalar arguments. potential boundary terms (note that the derivations in sections 4 and 5 did not involve integrations by parts). Moreover, we introduced the right eigenvector 6.3. Perturbation expansions · z(c−1) |zii ·= e |0i of the annihilation matrix (a|zii = The path integral representation (275) of factorial ? · z|zii with z ∈ C), and the left eigenvector hhz | ·= moments can be rewritten in terms of a (Q, X)- za ? ? h0|e of the creation matrix (hhz |c = zhhz |). These generating functional analogously to section 4.3. The vectors are commonly referred to as “coherent states”. resulting expression may serve as the starting point The eigenvector conditions can be easily verified by for a perturbative [15, 70] or a non-perturbative rewriting the vectors as analysis [410] of the path integral. Here we focus on the ∞ ∞ X zme−z X perturbative approach. To derive the representation, |zii = |mi and hhz?| = znhn| . (294) m! we assume that the adjoint transition operator can be m=0 n=0 split into drift, diffusion, and perturbation parts as Insertion of the identity matrices into (291) results in in (176), so that N ∂2 1 xN c † x † hAix0 = lim h0|A((c + )a)e |0i (295) Qτ (x, ∂x + 1) = ατ (x)∂x + βτ (x) + Pτ (x, ∂x) .(298) N→∞ 1 2 N By following the steps in appendix D, the path integral Y · h0|e−iqj (xj −a)1 + Q(c + 1, a)∆texj−1c|0i . representation of a factorial moment can be written as j=1 j t δ dτ P†( δ , δ ) 0 t0 τ δQ(τ) δX(τ) h(n)jix(t0) = e´ ZQ,X , This expression can be simplified with the help of the δQ(t)j Q=X=0 shift operations in (288) and (289). Since the Q- (299) matrix (284) is a normal-ordered polynomial in c and t 0 dτQ(τ)x(τ) a (i.e. all the c are to the left of all the a), and both t0 where ZQ,X = e´ W represents the h0|c and a|0i vanish, the above expression evaluates to (Q, X)-generating functional (180). Moreover, x(τ) N solves the Itô SDE 1 1 hAix0 = lim h0|A((c + )(a + xN ))|0i (296)   p N→∞ 1 dx(τ) = ατ (x) + X(τ) dτ + βτ (x) dW (τ) (300) N with initial condition x(t0). If both the diffusion and Y −iqj (xj −xj−1)  · e 1 + Q(iqj + 1, xj−1)∆t . perturbation parts of the transition operator (298) j=1 vanish, the factorial moment at time t obeys the j The factor h0|A((c + 1)(a + xN 1))|0i could be statistics of a Poisson distribution, i.e. h(n)ji = x(t) . evaluated by normal-ordering the observable with In the general case, the representation (299) may be respect to c and a before employing h0|c = 0 and evaluated perturbatively by expanding its exponential a|0i = 0 again. The resulting object is sometimes as [15] called a “normal-ordered observable” [336]. In the ∞ t τ t † 2 dτ Pτ X † † following, we show that this object agrees with the e t0 = dτm P ··· dτ1 P . (301) ´ ˆ τm ˆ τ1 average over a Poisson distribution in (276). The proof m=0 t0 t0 of this assertion is based on the observation that fj(x) · Note that this expression involves only a single sum = h0|((c + 1)(a + x1))j|0i fulfils the following defining and not a double sum as the expansion (230) did. Master equations and the theory of stochastic path integrals 54

6.4. Coagulation In general, diagrams constructed from the above building blocks exhibit internal loops. The simplest Let us exemplify the perturbation expansion of (299) connected diagram with such a loop and with a single for particles that diffuse and coagulate in the reaction sink is given by 2 A → A. This reaction exhibits the same asymptotic particle decay as the binary annihilation reaction 2 A → ∅ and thus belongs to the same universality . (305) class [294, 421]. Only a pre-factor differs between the perturbation operators of the two reactions (see below). A full renormalization group analysis of This loop is part of the m = 2 summand of (301). general annihilation reactions k A → l A with l < k Its mathematical expression is obtained by tracing the was performed by Lee [130, 416] (see also [69]). The diagram from right to left and reads procedure of Lee differs slightly from the one presented t X dτ G(t, i|τ , j)(−µ ) (306) below, but it involves the same Feynman diagrams. We 2 2 τ2,j ˆt0 restrict ourselves to the “tree level” of the diagrams. j∈L τ2 The coagulation reaction 2 A → A and the annihilation X  2  2 · dτ1 2 G(τ2, j|τ1, k) (−µτ1,k) xh,k(τ1) . reaction 2 A → ∅ eventually cause all but possibly one ˆt 0 k∈L particle to vanish from a system. The transition into an The combinatorial factor 2 stems from the two possible absorbing steady state can be prevented; for example, ways of connecting the two vertices (either outgoing by allowing for the creation of particles through the leg can connect to either incoming leg). Note that the reaction ∅ → A. The fluctuations around the ensuing sink, which corresponds to the final derivative δ/δQ(t), non-trivial steady state have been explored with the is not associated to an additional pre-factor (unlike in help of path integrals in [298, 308]. section 4.5.4). In the following, we consider particles that diffuse In the following, we focus on the contribution of on an arbitrary network L and that coagulate in the “tree diagrams” to the mean particle number. These reaction 2 A → A. Hence, all results from section 4.5.4 diagrams do not exhibit internal loops. Thus, upon apply here as well, except that the transition operator removing all the diagrams containing loops from the of the coagulation reaction now acts as a perturbation. expansion of (299), one can define the “tree-level With the local coagulation rate coefficient µτ,i at average” lattice site i ∈ L, this perturbation reads (cf. (122)) † X 2 2 Pτ (x, iq) = µτ,ixi (iqi + 1) − (iqi + 1) (302) i∈L X 2 2 2 = (−µτ,i)xi iqi + (−µτ,i)xi (iqi) . (303) n¯i(t) ·= + + 2 + .... i∈L For the binary annihilation reaction 2 A → ∅, the first rate coefficient µτ,i in this expression differs by an (307) additional pre-factor 2. Note that we allow the rate The corresponding mathematical expression reads coefficient to depend both on time and on the node n¯i(t) = xh,i(t) (308) i ∈ L. t X 2 As explained in section 4.5.4, summands of the + dτ G(t, i|τ , j)(−µ )x (τ ) ˆ 1 1 τ1,j h,j 1 expansion of (299) can be represented by Feynman t0 j∈L diagrams. For the mean local particle number hn i, i t X each of the diagrams is composed of a sink with one + 2 dτ2 G(t, i|τ2, j)(−µτ2,j)xh,j(τ2) ˆt incoming leg, and possibly of the vertices 0 j∈L τ 2 X 2 · dτ G(τ , j|τ , k)(−µ )x (τ ) ˆ 1 2 1 τ1,k h,k 1 −µ −µ . (304) t0 τ,i and τ,i k∈L + .... According to the functional derivative (232), contrac- The inclusion of diagrams with loops would correct n¯i ted lines, either between two vertices or between a to the true mean hnii. For the treatment of loops, see vertex and the sink, are associated to the propagator for example [70,130, 293, 416]. G(τ|τ 0). Dangling incoming lines, on the other hand, The tree-level average n¯i defined above fulfils the deterministic rate equation of the coagulation process, introduce a factor xh(τ), which represents the homo- i.e. it fulfils geneous solution of ∂τ xi = ∆ετ,ixi + Xi(τ) (cf. (229) 2 and (231)). ∂τ n¯i = ∆ετ,in¯i − µτ,in¯i . (309) Master equations and the theory of stochastic path integrals 55

If both the rates ετ and µτ , and also the mean path integral representation was found to be equival- x(t0) of the initial Poisson distribution are spatially ent to Doi-shifted path integrals used in the literat- homogeneous, the well-mixed analogue of (309) can ure [70, 96, 130–133]. Its unshifted version is obtained be derived by direct resummation of (308). In the for a redefined basis function. Unlike path integral general case, the validity of the rate equation (309) representations encountered in the literature, our rep- can be established by exploiting a self-similarity of resentation (311) does not involve a so-called “normal- n¯i. For that purpose, we assume that n¯i(τ) is known ordered observable”. By using a defining relation of for τ < t and we want to extend its validity up Touchard polynomials, we could show that this object to time t. To do so, we remove all the sinks from agrees with the average (312) of A(n) over a Poisson the diagrams in (307) and connect their now dangling distribution (cf. section 6.2). outgoing legs with the incoming leg of another three- As shown in section 6.3, the path integral vertex at time τ. The second incoming leg of this vertex representation (311) can be rewritten in terms of is contracted with every other tree diagram and its a perturbation expansion. We demonstrated the outgoing leg with a sink at time t. The corresponding evaluation of this expansion in section 6.4 for diffusing P 2 expression reads G(t, i|τ, j)(−µτ,j)¯nj(τ) and it particles that coagulate according to the reaction j∈L contributes to n¯i(t) for every time τ. To respect also 2 A → A. In doing so, we restricted ourselves to the the initial condition n¯i(t0) = xi(t0), we introduce the tree-level of the Feynman diagrams associated to the first diagram of (307) by hand so that in total perturbation expansion. Information on perturbative t renormalization group techniques for the treatment of X 2 n¯i(t) = xh,i(t)+ dτ G(t, i|τ, j)(−µτ,j)¯nj(τ) .(310) diagrams with loops can be found in [69,70]. Recently, ˆt 0 j∈L non-perturbative renormalization group techniques Differentiation of this equation with respect to t have been developed for the evaluation of stochastic confirms the validity of the rate equation (309) (recall path integrals [317, 410]. These techniques have the definition of the propagator in section 4.5.2). proved particularly useful in studying branching and annihilating random walks [317, 318] and annihilation processes [306, 307]. 6.5. Résumé

Path integral representations of averaged observables 7. Stationary paths have proved useful in a variety of contexts. Their use has deepened our knowledge about the critical beha- In the previous sections, we outlined how path integrals viour of diffusion-limited annihilation and coagulation can be expressed in terms of averages over the paths of reactions [69,70,130,293,294,302,304,305,316,400], of stochastic differential equations (SDEs). Corrections branching and annihilating random walks and percola- to those paths were treated in terms of perturbation tion processes [69,70,96,132,133,301,303,422,423], and expansions. In the following, we formulate an of elementary multi-species reactions [131,295–297,299, alternative method in which the variables of the 300, 309–311, 313, 424, 425]. In the present section, we path integrals act as deviations from “stationary”, derived a path integral representation of averaged ob- or “extremal”, paths. The basic equations of the servables for processes that can be decomposed addit- method have the form of Hamilton’s equations from ively into reactions of the form k A → l A. Provided classical mechanics. Their application to stochastic that the number of particles n in the system is initially path integrals goes back at least to the work of Poisson distributed with mean x(t0), we showed that Mikhailov [11]. the average of an observable A(n) can be represented More recently, Elgart and Kamenev extended the by the path integral method for the study of rare event probabilities [35] t] and for the classification of phase transitions in † hAi = e−S hhAii (311) reaction-diffusion models with a single type of x(t0) x(t) (t0 particles [320]. These studies are effectively based ∞ X xne−x on the forward path integral representation (244) with hhAii = A(n) . (312) x n! of the generating function. After reviewing how n=0 this approach can be used for the approximation We derived this representation in section 6.1 by sum- of probability distributions, we extend it to the ming the backward path integral representation (171) backward path integral representation (168) of the of the marginalized distribution over the observable marginalized distribution. Whereas the generating xne−x A(n) (using the Poisson basis function |ni = n! ). function approach requires an auxiliary saddle-point The generalization of the above path integral to re- approximation to extract probabilities from the actions with multiple types of particles and spatial generating function, the backward approach provides degrees of freedom is straightforward. The above Master equations and the theory of stochastic path integrals 56 direct access to probabilities. A proper normalization in terms of the forward path integral (248) by following of the resulting probability distribution is, however, the steps in section 5.1. Using ∆q(τ) and ∆x(τ) as only attained beyond leading order. The generating labels for the path integral variables, one arrives at the function technique respects the normalization of representation the distribution even at leading order, but this  t)  x˜(t)˜q(t) −S normalization may be violated by the subsequent |giq(t) = e e |n0it ,∆q(t ) . (317) 0 0 ∆q(t)=0 saddle-point approximation. [t0 The methods discussed in the following all apply In analogy with (246), the integral sign is defined as to the chemical master equation (24) and employ the the continuous-time limit basis functions that we introduced in sections 2.2.2 t) N−1 Y d∆xj d∆qj and 3.2.2. Moreover, we assume the number of = lim . (318) N→∞ ˆ 2 2π particles to be initially Poisson distributed with mean [t0 j=0 R x(t0). This assumption proves to be convenient in the The action (249) inside the generating function (317) analysis of explicit stochastic processes but it can be reads easily relaxed. t   S = dτ i∆x∂−τ ∆q − Qeτ (∆q, i∆x) . (319) ˆ 7.1. Forward path integral approach t0 As emphasized above, our interest lies in a Our goal lies in the approximation of the marginalized system whose initial number of particles is Poisson probability distribution distributed with mean x(t0). Thus, instead of dealing ∞ n0 −x(t0) X x(t0) e with the ordinary generating function (314), it proves |p(t, n|t0)i = p(t, n|t0, n0) .(313) x(t0) n ! convenient to work in terms of the generating function n =0 0 0 ∞ x(t )n0 e−x(t0) For this purpose, we employ the forward path integ- · X 0 |g(t|t0; x(t0))iq(t) ·= |g(t|t0, n0)iq(t) ral (244) to formulate an alternative representation of n0! n0=0 the ordinary probability generating function t) ∞ = ex˜(t)˜q(t)−x(t0) e−S . (320) X n ∆q(t)=0 · [t0 |g(t|t0, n0)iq(t) ·= q(t) p(t, n|t0, n0) . (314) n=0 To arrive at the second line, we made use of the As the first step, we rewrite the argument q(t) of path integral representation (317) while requiring that this function in terms of a deviation ∆q(t) from a the path x˜(τ) meets the initial condition x˜(t0) = “stationary path” q˜(t), i.e. q(t) =q ˜(t) + ∆q(t). The x(t0). Note that the marginalized distribution (313) path q˜(t) and an auxiliary path x˜(τ) are chosen so that is recovered from the redefined generating function via the action of the resulting path integral is free of terms 1 n |p(t, n|t0)ix(t ) = ∂q(t)|g(t|t0; x(t0))iq(t) . (321) that are linear in the path integral variables ∆q(τ) and 0 n! q(t)=0 ∆x(τ) (with τ ∈ [t , t]). Thus, the approach bears 0 Thus far, we have not yet specified the paths similarities with the stationary phase approximation q˜(τ) and x˜(τ), besides requiring that they fulfil the of oscillatory integrals [426]. In the next section, we boundary conditions q˜(t) = q(t) and x˜(t ) = x(t ). We apply the method to the binary annihilation process 0 0 specify these paths in such a way that the action (319) 2 A → ∅. becomes free of terms that are linear in the deviations In order to implement the above steps, we define ∆q and ∆x. For this purpose, let us recall the the basis function definitions of the creation operator c = ζ∆q +q ˜ and · n −x˜(τ)(ζ∆q(τ)+˜q(τ)) |niτ,∆q(τ) ·= (ζ∆q(τ) +q ˜(τ)) e (315) of the annihilation operator a = ∂ζ∆q +x ˜ from section for yet to be specified paths q˜(τ) and x˜(τ), and a free section 2.2.2, as well as the definition of the transition parameter ζ. The basis function has the same form as operator the basis function (70) but with its second argument Qeτ (∆q, ∂∆q) = Qτ (c, a) + Eτ (c, a) . (322) being written as ∆q(τ). Provided that the path q˜(τ) The basis evolution operator is specified in (77) as fulfils the final condition q˜(t) = q(t), one can trivially rewrite the generating function (314) in terms of the Eτ (c, a) = (∂τ q˜)(a − x˜) − (∂τ x˜)c . (323) above basis function as Upon performing a Taylor expansion of the operator  ∞  Q (∆q, i∆x) in the action (319) with respect to the x˜(t)˜q(t) X eτ |giq(t) = e |nit,∆q(t) p(t, n|·) . (316) ∆q(t)=0 deviations ∆q and ∆x, one observes that terms that n=0 The term in brackets has the form of the generalized generating function (45), and thus it can be rewritten Master equations and the theory of stochastic path integrals 57 are linear in the deviations vanish from the action if 11 the paths x˜(τ) and q˜(τ) fulfil x(t0)

∂Qτ (˜q, x˜) 20 x ∂ x˜ = with x˜(t ) = x(t ) and (324) ~ τ ∂q˜ 0 0

∂Qτ (˜q, x˜) ∂−τ q˜ = with q˜(t) = q(t) . (325) ∂x˜ Variable 10 These equations resemble Hamilton’s equations from classical mechanics. Just as in classical mechanics, 0 Qτ is conserved along solutions of the equations if it 0 q(t) 0.5 1 1.5 2 does not depend on time itself. This property follows ~ d d Variable q from dτ Qτ (˜q, x˜) = ∂τ Qτ (˜q, x˜), with dτ being the total time derivative. Since the ordinary generating function Figure 9. Phase portrait of Hamilton’s equations (330) obeys the flow equation ∂τ |giq = Qτ (q, ∂q)|giq, and (331) for the rate coefficient µτ = 1. The transition 2 2 the conservation of total probability requires that operator Qτ (˜q, x˜) = µτ (1 − q˜ )˜x of the binary annihilation process vanishes for x˜ = 0 and q˜ = 1 (orange dotted lines). The Qτ (1, ∂q) = 0. This condition is, for example, fulfilled by the transition operator Q (q, ∂ ) = γ (ql − qk)∂k of green disk represents the fixed point (˜q, x˜) = (1, 0) of Hamilton’s τ q τ q equations. The red dash-dotted line exemplifies a particular the generic reaction k A → l A (cf. (64)). For the final solution of Hamilton’s equations for a given time interval (t0, t), value q(t) = 1, Hamilton’s equations (324) and (325) a given initial value x(t0) and a given final value q(t) (black are solved by q˜(τ) = 1 with x˜(τ) solving the rate dashed lines). k equation ∂τ x˜ = γτ (l − k)˜x of the process. Elgart and Kamenev analysed the topology of Qτ (˜q, x˜) = 0 lines of the case for the simple growth process ∅ → A. reaction-diffusion models with a single type of particles For the binary annihilation reaction 2 A → ∅, the to classify the phase transitions of these models [320] leading-order approximation was evaluated by Elgart (they use the label p instead of q˜, and q instead of x˜). and Kamenev up to a pre-exponential factor [35]. Provided that Hamilton’s equations (324) and (325) The pre-exponential factor was later approximated by are fulfilled, the action (319) evaluates to Assaf and Meerson for large times [36]. In the next S =x ˜(t)˜q(t) − x(t0) + S˜ + ∆S (326) section, we evaluate the leading-order approximation of the binary annihilation reaction and evaluate its pre- with the definitions t exponential factor for arbitrary times. ˜   S ·= x(t0)(1 − q˜(t0)) + dτ x∂˜ −τ q˜− Qτ (˜q, x˜) (327) ˆ t0 7.2. Binary annihilation t
and ∆S ·= dτ i∆x∂−τ ∆q − ∆Qτ . (328) Let us demonstrate the use of the representation (329) ˆ t0 for the binary annihilation reaction 2 A → ∅ with The transition operator ∆Qτ absorbs all terms of the rate coefficient µτ . First, we evaluate the generating Taylor expansion of Qτ that are of second or higher function in leading order. Afterwards, the generating order in the deviations. Combined with (320), the function is cast into a probability distribution using above action results in the following path integral the inverse transformation (48). The derivatives representation of the generating function: involved in the inverse transformation are expressed by t) Cauchy’s differentiation formula, which is evaluated in −S˜ −∆S |g(t|t0; x(t0))i = e e . (329) terms of a saddle-point approximation. q(t) ∆q(t)=0 [t0 The transition operator of the binary annihilation 2 2 Although this path integral representation may seem reaction follows from (64) as Qτ (cτ , aτ ) = µτ (1−cτ )aτ daunting, our primary interest lies only in its leading- with the annihilation rate coefficient µτ . Therefore, order approximation |gi ≈ e−S˜. This approximation Hamilton’s equations (324) and (325) read is exact if ∆Qτ vanishes. That is, for example, 2 ∂τ x˜ = −2µτ q˜x˜ with x˜(t0) = x(t0) and (330) 11 If Hamilton’s equations are fulfilled, it holds that 2 ∂−τ q˜ = 2µτ (1 − q˜ )˜x with q˜(t) = q(t) . (331)  ∂Qτ ∂Qτ −1  Qeτ (∆q, i∆x) = Qτ (˜q, x˜) + ζ∆q + ζ i∆x + ∆Qτ A phase portrait of these equations is shown in figure 9. ∂q˜ ∂x˜ Equation (331) is solved by q˜ = 1, for which the  −1  + (∂τ q˜)ζ i∆x − (∂τ x˜)(ζ∆q +q ˜) previous equation simplifies to the rate equation ∂τ x˜ =   2 = −∂τ (˜xq˜) − x∂˜ −τ q˜ − Qτ (˜q, x˜) + ∆Qτ . −2µτ x˜ of the process. This rate equation is solved by x(t0) Here, ∆Qτ represents all terms of the Taylor expansion of Qτ x¯(t) ·=x ˜(t) = (332) that are of second or higher order in the deviations. 1 + x(t0)/x¯∞(t) Master equations and the theory of stochastic path integrals 58

t −1 with the asymptotic limit x¯∞(t) ·= (2 dτ µτ ) → solution (334) with respect to q(t), the saddle-point · t0 0. Since Hamilton’s equations conserve´(˜q2 − 1)˜x2, one condition can be rewritten as p 2 can rewrite the equation (331) as n 1 − q˜(t0) = x(t0) . (339) p 2p 2 q p 2 ∂τ q˜ = −2µτ x(t0) 1 − q˜(t0) 1 − q˜ . (333) s 1 − qs Here we assume q˜(t) = q(t) < 1 but the derivation can A closed equation for qs is obtained by combining this also be performed for q(t) > 1 (these inequalities are equation with the implicit solution (334). The resulting preserved along the flow). The above equation only equation is solved by qs = 1 for n =x ¯(t), i.e. if n lies on the trajectory of the rate equation. For other values allows for an implicit solution that provides q˜(t0) for a given q(t), namely of n, the equation has to be solved numerically. Once qs has been obtained, the value q˜(t0) can be inferred x(t0) p 2 arccosq ˜(t0) + 1 − q˜(t0) = arccos q(t) . (334) from (339). x¯∞(t) One piece is still missing for the numeric The first term of this equation was neglected in evaluation of the probability distribution (338), namely previous studies [35, 36] (for large times t, q˜(t0) ≈ 1). its denominator. It evaluates to Using the conservation of (˜q2 − 1)˜x2 once again, the 2 2 ˜00 qs α − n/x¯∞(t) action in the leading-order approximation |gi = n − qs S (qs) =x ¯∞(t) 2 (340) q(t) qs − 1 ˜ −S(q(t)) −1 e can be written as  x(t0)˜q(t0)  2 2 with α ·= 1 − 1 + . (341)   1 − q˜(t0) x(t0) x¯∞(t) S˜(q(t)) = 1 − q˜(t0) x(t0) + . (335) 2¯x∞(t) The pre-exponential factor of the distribution (338) We have thus fully specified the generating function computed by Assaf and Meerson [36] is recovered for in terms of its argument q(t) and the mean x(t0) large times for which x¯∞(t) → 0 and thus α → 1. The of the initial Poisson distribution. The leading- above expressions hold both for qs < 1 and qs > 1. −S˜(q(t)) Care has to be taken in evaluating the limit q → 1.A order approximation |giq(t) = e respects s the normalization of the underlying probability rather lengthy calculation employing L’Hôpital’s rule distribution because shows that the left-hand side of (340) evaluates to ∞  −3 X 2 1h x(t0) i p(t, n|t0, n0) = |giq(t)=1 = 1 . (336) x¯(t) 1 + 1 + . (342) 3 2 x¯∞(t) n=0 Here we used that for q(t) = 1, Hamilton’s The pre-factor of this expression matches the normal- equation (331) is solved by q˜(τ) = 1, implying that ization constant that Elgart and Kamenev inserted by S˜(q(t) = 1) = 0. hand [35]. The probability distribution follows from the After putting all of the above pieces together, generating function via the inverse transformation the probability distribution (338) provides a decent 1 n approximation of the binary annihilation process. |p(t, n|t0)i = ∂ |gi . This transforma- x(t0) n! q(t) q(t) q(t)=0 Figure 10 compares the distribution with a distribution tion is trivial for n = 0, for which one obtains the prob- that was obtained through a numerical integration of ability of observing an empty system. For other values the master equation. For very large times, the quality of n, the derivatives can be expressed by Cauchy’s dif- of the approximation deteriorates. In particular, the ferentiation formula so that in leading order approximation does not capture the final state of the 1 dq(t) |p(t, n|t )i = e−S˜(q(t))−n ln q(t) . (337) process in which it is equally likely to find a single 0 x(t0) 2πi ˛C q(t) surviving particle or none at all. Limiting cases of the Here, C represents a closed path around zero in the approximation (338) are provided in [35, 36]. complex domain and is integrated over once in counter- Although the above evaluation of the saddle-point clockwise direction. approximation is rather elaborate, it is still feasible. The contour integral in (337) can be evaluated in It becomes infeasible if one goes beyond the leading- a saddle-point approximation as order term of the generating function. In the section ˜ after the next, we show how higher order terms can q−ne−S(qs) |p(t, n|t )i ≈ s . (338) be included in a dual approach, which is based on the 0 x(t0) q 2 ˜00 backward path integral (171). 2π(n − qs S (qs))

The saddle-point qs is found by solving the equation 7.3. Backward path integral approach 0 n/qs = −S˜ (qs). By differentiating the implicit The above method can be readily extended to a novel path integral representation of the marginalized Master equations and the theory of stochastic path integrals 59

(a) (b) (c) 〉 〉 〉

) ) 0.20 ) 0.5

0 0.03 t = 0.001 0 t = 0.1 0 t = 1 ⎜ ⎜ ⎜

n n n 0.4 0.15 ( t , ( t , ( t ,

p 0.02 p p ⎜ ⎜ ⎜ 0.3 0.10 0.2 0.01 0.05 0.1

Probability 0.00 Probability 0.00 Probability 0.0 140 160 180 200 0 2 4 6 8 10 12 0 1 2 3 4 Number of particles n Number of particles n Number of particles n (d) 1 (e) 1 (f) 1 〉 〉 〉 ) ) )

0 −10 0 −10 0 −10

⎜ 10 ⎜ 10 ⎜ 10 n n n −20 −20 −20 ( t , 10 ( t , 10 ( t , 10 p p p ⎜ ⎜ ⎜ 10−30 10−30 10−30

−40 −40 −40 10 t = 0.001 10 t = 0.1 10 t = 1

Probability 10−50 Probability 10−50 Probability 10−50 50 100 150 200 250 300 0 5 10 15 20 25 30 35 0 2 4 6 8 10 Number of particles n Number of particles n Number of particles n

Figure 10. Solution of the binary annihilation reaction 2 A → ∅ via the numerical integration of the master equation (blue lines) and approximation of the process using the stationary path method in section 7.2 (red circles). The figures in the upper row show the marginalized probability distribution |p(t, n|0)i at times (a) t = 0.001,(b) t = 0.1, and (c) t = 1 on a linear scale. The figures in the lower row show the distribution at times (d) t = 0.001, (e) t = 0.1, and (f) t = 1 on a logarithmic scale. The rate coefficient of the process was set to µτ = 1. For the numerical integration, the master equation was truncated at n = 450. The marginalized distribution was initially of Poisson shape with mean x(0) = 250. Its leading-order approximation is given in (338). distribution As shown below, the marginalized distribution (343) ∞ n0 −x(t0) can then be expressed as X x(t0) e |p(t, n|t0)i = p(t, n|t0, n0) .(343) x(t0) n −S˜† t] n0! x˜(t) e † n0=0 −∆S |p(t, n|t0)ix(t ) = e (346) 0 n! ∆x(t0)=0 Unlike in the previous section, no saddle-point (t0 t approximation is required to evaluate |p(t, n|t0)i . x(t0) with S˜† ·= x(t ) + dτ q∂˜ x˜ − Q† (˜x, q˜)
(347) The resulting path integral can also be evaluated · 0 τ τ ˆt0 beyond leading order, at least numerically. In ζ∆x(t) h ζ∆x(t)i the next section, we outline such an evaluation for and ∆S† ·= n − ln 1 + (348) x˜(t) x˜(t) the binary annihilation reaction 2 A → ∅. On t the downside, the leading order of the “backward” †
+ dτ i∆q∂τ ∆x − ∆Qτ . approach does not respect the normalization of ˆt0 |p(t, n|t )i . The normalization of the distribution 0 x(t0) The variables ∆x and ∆q again represent deviations has to be implemented by hand or by evaluating higher from the stationary paths x˜ and q˜. The transition order terms. † operator ∆Qτ encompasses all the terms of an To represent the marginalized distribution (343) † −1 expansion of Qτ (ζ∆x+˜x, ζ i∆q+˜q) that are of second by a path integral, we require that for a given particle or higher order in the deviations. A Taylor expansion number n and for a given initial mean x(t0), there exist of the action shows that it is free of terms that are functions x˜(τ) and q˜(τ) fulfilling linear in ∆x and ∆q. ∂Q† (˜x, q˜) The derivation of the above representation ∂ x˜ = τ with x˜(t ) = x(t ) and (344) τ ∂q˜ 0 0 proceeds analogously to section 7.1. It begins by using x˜(t ) = x(t ) and the basis function |ni = ∂Q† (˜x, q˜) n 0 0 τ,∆x τ 1 n −q˜(ζ∆x+˜x) ∂−τ q˜ = with q˜(t) = . (345) (ζ∆x+˜x) e from (125) to rewrite the right- ∂x˜ x˜(t) n! hand side of the marginalized distribution (343) as † Since the transition operators Qτ in (122) and Qτ ∞ † (˜q(t0)−1)˜x(t0) X in (64) are connected via Qτ (˜x, q˜) = Qτ (˜q, x˜), e p(t, n|t0, n0)|n0i .(349) t0,∆x(t0) ∆x(t0)=0 the above equations differ from the previous equa- n0=0 tions (324) and (325) only in the final condition on q˜. The sum in this expression can be represented by the Master equations and the theory of stochastic path integrals 60 backward path integral (168), turning the expression Here we assume q˜(t) < 1 but the derivation can also into be performed for q˜(t) > 1. The equation is implicitly t] † solved by (˜q(t0)−1)˜x(t0) −S e e |nit,∆x(t) . (350) s ∆x(t0)=0 2 2 p 2 2 (t0 x˜(t) − n x˜(t) − n arccos 1 − 2 + (358) Recalling the definition of Eτ in (127), the transition x(t0) x¯∞(t) operator n † † −1 = arccos . Qeτ (∆x, i∆q) = Qτ (ζ∆x +x, ˜ ζ i∆q +q ˜) (351) x˜(t) −1 − Eτ (ζ∆x +x, ˜ ζ i∆q +q ˜) For a given initial mean x(t0), this equation can be can now be expanded in the deviations. We thereby solved numerically for x˜(t), which is then inserted obtain the action into (353). Upon normalizing the distribution by hand, t it provides a reasonable approximation of the process. S† =q ˜(t )˜x(t ) − q˜(t)˜x(t) + dτ q∂˜ x˜ − Q† (˜x, q˜) The quality of the approximation comes close to the 0 0 ˆ τ τ t0 quality of the approximation discussed in section 7.2. t †
+ dτ i∆q∂τ ∆x − ∆Qτ . (352) ˆt0 7.4.2. Beyond leading order. Instead of normalizing The path integral representation (346) follows upon the function (353) by hand, it can be normalized by inserting this action into (350) and employing the final evaluating the path integral (346). In the following, condition q˜(t) = n/x˜(t). we perform this evaluation, but only after restricting the action ∆S† in (348) to terms that are of second order in the deviations ∆x and ∆q, i.e. to 7.4. Binary annihilation (∆x(t))2 n t To complement the approximation of the binary ∆S† ≈ − + dτ i∆q∂ ∆x−∆Q†
.(359) 2 x˜(t)2 ˆ τ τ annihilation reaction 2 A → ∅ with rate coefficient t0 µτ in section 7.2, we now perform an approximation The corresponding transition operator of the process using the backward path integral (∆q)2 (∆x)2 ∆Q† ≈ i∆q∆x α − β + c (360) representation (346) of the marginalized distribution. τ τ 2 τ 2 τ We first perform the approximation in leading order, is obtained as explained in section 7.3 (with ζ ·= i). Its which amounts to the evaluation of the pre-factor coefficients read of (346). For the simple growth process ∅ → A and for · · 2 · 2 the linear decay process A → ∅, the pre-factor provides ατ ·= −4µτ x˜q˜ , βτ ·= 2µτ x˜ , cτ ·= 2µτ (˜q −1) .(361) exact solutions. The path integral can now be evaluated in one of the following two ways. † 7.4.1. Leading order. The evaluation of the leading- On the one hand, the transition operator ∆Qτ has order approximation the same form as the one in section 4.3 and thus can

˜† be treated in the same way. Following appendix D, the x˜(t)ne−S |p(t, n|t )i ≈ (353) path integral (346) can be expressed in terms of the 0 x(t0) n! following average over a Wiener process W : proceeds very much like the derivation in section 7.2. † t] Using the adjoint transition operator Q (c , a ) = −∆S† τ τ τ e (362) 2 2 ∆x(t0)=0 µτ cτ (1 − aτ ) from (122), one finds that Hamilton’s (t0 equations DD (∆x(t))2 n t (∆x)2 EE 2 = exp + dτ cτ . ∂τ x˜ = −2µτ q˜x˜ with x˜(t0) = x(t0) and (354) 2 W 2 x˜(t) ˆt0 2 2 ∂−τ q˜ = 2µτ (1 − q˜ )˜x with q˜(t) = n/x˜(t) (355) Here, ∆x(τ) solves the Itô SDE agree with (330) and (331), apart from the final d∆x = α ∆x dτ + pβ dW (τ) , (363) condition on q˜. The conservation of (˜q2 − 1)˜x2 and the τ τ t −1 with initial value ∆x(t0) = 0. Unfortunately, the asymptotic deterministic limit x¯∞(t) = (2 dτ µτ ) t0 computation of a sufficient number of sample paths to can be used to rewrite the action (347) as ´ approximate the average (362) was found to be rather s  2 2  2 2 slow. ˜† x˜(t) − n x˜(t) − n S = n+ 1− 1 − 2 x(t0)+ .(356) x(t0) 2¯x∞(t) As an alternative to the above way, one can discretize the action ∆S† and cast it into the quadratic In addition, the conservation law can be used to infer † 1 form ∆S = ξ|Aξ with ξ ·= {∆qj, ∆xj}j=1,...,N . The the flow equation 2 matrix A is symmetric and tridiagonal. If A is also p 2p 2 2 ∂τ q˜ = −2µτ 1 − q˜ x˜(t) − n . (357) positive-definite, the path integral (346) evaluates to Master equations and the theory of stochastic path integrals 61

(a) (b) (c) 〉 〉 〉

) ) 0.20 ) 0.5

0 0.03 t = 0.001 0 t = 0.1 0 t = 1 ⎜ ⎜ ⎜

n n n 0.4 0.15 ( t , ( t , ( t ,

p 0.02 p p ⎜ ⎜ ⎜ 0.3 0.10 0.2 0.01 0.05 0.1

Probability 0.00 Probability 0.00 Probability 0.0 140 160 180 200 0 2 4 6 8 10 12 0 1 2 3 4 Number of particles n Number of particles n Number of particles n (d) 1 (e) 1 (f) 1 〉 〉 〉 ) ) )

0 −10 0 −10 0 −10

⎜ 10 ⎜ 10 ⎜ 10 n n n −20 −20 −20 ( t , 10 ( t , 10 ( t , 10 p p p ⎜ ⎜ ⎜ 10−30 10−30 10−30

−40 −40 −40 10 t = 0.001 10 t = 0.1 10 t = 1

Probability 10−50 Probability 10−50 Probability 10−50 50 100 150 200 250 300 0 5 10 15 20 25 30 35 0 2 4 6 8 10 Number of particles n Number of particles n Number of particles n

Figure 11. Solution of the binary annihilation reaction 2 A → ∅ via the numerical integration of the master equation (blue lines) and approximation of the process using the stationary path method in section 7.4 (red circles). The figures in the upper row show the marginalized probability distribution |p(t, n|0)i at times (a) t = 0.001,(b) t = 0.1, and (c) t = 1 on a linear scale. The figures in the lower row show the distribution at times (d) t = 0.001, (e) t = 0.1, and (f) t = 1 on a logarithmic scale. The rate coefficient of the process was set to µτ = 1. For the numerical integration, the master equation was truncated at n = 450. The marginalized distribution√ was initially of Poisson shape with mean x(0) = 250. Its leading-order approximation (353) was corrected by the factor 1/ det A (cf. section 7.4.2). The numerical solution of the fixed point equation (358) failed for large times (missing circles in (f)).

√ 1/ det A. Curiously, we found that this factor even the marginalized distribution (343). In both cases, matches the average (362) when A is not positive-√ the stationary paths obey differential equations resem- definite. Therefore, we used the factor 1/ det A bling Hamilton’s equations from classical mechanics. to correct the approximation (353) and evaluated In sections 7.2 and 7.4, we demonstrated the two ap- the resulting expression for various times t.A proaches in approximating the binary annihilation re- comparison with distributions obtained via a numeric action 2 A → ∅. Our approximation based on the for- integration of the master equation is provided in ward path integral amends an earlier computation of figure 11. Apparently, the quality of the approximation Elgart and Kamenev [35]. The computation requires a is even better than the quality of the approximation saddle-point approximation of Cauchy’s differentiation discussed in section 7.2, at least for early times. formula to extract probabilities from the generating However, upon approaching the asymptotic limit of the function. The advantage of the approach lies in the fact process, it becomes increasingly difficult to numerically that its leading order term respects the normalization solve (358) for a fixed point x˜(t). of the underlying probability distribution. The back- ward approach does not require the auxiliary saddle- 7.5. Résumé point approximation but its leading order term is not normalized. In the approximation of the binary anni- The solution of a master equation can be approxim- hilation reaction, we demonstrated how the expansion ated by expanding its forward or backward path in- of the backward path integral can be evaluated beyond tegral representations around stationary paths of their leading order. Future studies are needed to explore respective actions. Such an expansion proves particu- whether the two approaches are helpful in analysing larly useful in the approximation of “rare-event” prob- processes with multiple types of particles and spatial abilities in a distribution’s tail. Algorithms such as degrees of freedom. The efficient evaluation of contri- the stochastic simulation algorithm (SSA) of Gillespie butions beyond the leading order approximations also typically perform poorly for this purpose. Whereas remains a challenge. the expansion of the forward path integral repres- entation provides the ordinary probability generating function (314) as an intermediate step, the expansion of the backward path integral representation provides Master equations and the theory of stochastic path integrals 62

8. Summary and outlook Alternative path summation algorithms (section 1.4) Given enough time, the SSA could be used to gener- On sufficiently coarse time and length scales, many ate every possible sample path of a process (at least complex systems appear to evolve through finite jumps. if the state space is finite). If every path is recorded Such a jump may be the production of an mRNA or just once, the corresponding histogram approximation protein in a gene regulatory network [138, 139, 142], of the conditional probability distribution p(τ, n|τ0, n0) the step of a molecular motor along a cytoskeletal would agree with the “path summation representa- filament [113–115], or the flipping of a spin [134– tion” (19). Various alternatives to the SSA have been 137]. The stochastic evolution of such processes is proposed for the numerical evaluation of this represent- commonly modelled in terms of a master equation [4,5]. ation, mostly based on its Laplace transform [238–243]. This equation provides a generic description of a Thus far, these algorithms have remained restricted to system’s stochastic evolution under the following three rather simple systems. The analytical evaluation of the conditions: First, time proceeds continuously. Second, sums and convolutions that are involved proves to be the system’s future is fully determined by the system’s a challenge. present state. Third, changes of the system’s state proceed via discontinuous jumps. Exponentiation and uniformization (section 1.4) In this work, we reviewed the mathematical If the state space of a system is finite, the (forward) theory of master equations and discussed analytical master equation ∂τ p(τ|t0) = Qp(τ|t0) is solved by and numerical methods for their solution. Special Q(τ−τ0) the matrix exponential p(τ|τ0) = e 1. Here, attention was paid to methods that apply even Q represents the transition matrix of a process and when stochastic fluctuations are strong and to the p(τ|t0) the matrix of the conditional probabilities representation of master equations by path integrals. p(τ, n|τ0, n0). If the state space of the system is In the following, we provide brief summaries of countably infinite but the exit rates from all states are the discussed methods, which all have their own bounded (i.e. supm |Q(m, m)| < ∞), the conditional merits and limitations. Information on complementary probability distribution can be represented by the approximation methods can be found in the text “uniformization” formula (16). As the evaluation of this books [98, 110] and in the review [274]. formula involves the computation of an infinite sum of matrix powers, its use is restricted to small systems and The stochastic simulation algorithm (section 1.4) to systems whose transition matrices exhibit special The SSA [75–78] and its variations [78, 247–249] symmetries. The same applies to the evaluation of come closest to being all-purpose tools in solving the the above matrix exponential (note, however, the (forward) master equation numerically. The SSA projection techniques proposed in [219, 220]). enables the computation of sample paths with the correct probability of occurrence. Since these paths Flow equations (sections 2 and 3) are statistically independent of one another, their For the chemical master equation (24) of the reaction computation can be easily distributed to individual k A → l A with rate coefficient γτ , it is readily shown processing units. Besides providing insight into a that the probability generating function system’s “typical” dynamics, the sample paths can X be used to compute a histogram approximation of |g(τ|t0, n0)i = |ni p(τ, n|t0, n0) (364) the master equation’s solution or to approximate n n observables. In principle, the SSA can be applied with basis function |niq = q obeys the linear partial to arbitrarily complex systems with multiple types differential equation, or “flow equation” of particles and possibly spatial degrees of freedom. ∂ |gi = γ (ql − qk)∂k|gi . (365) Consequently, the algorithm is commonly used in τ τ q biological studies [138, 139, 142] and extensions of Analogously, the marginalized distribution X it have been implemented in various simulation |p(t, n|τ)i = p(t, n|τ, n )|n i (366) packages [250–257]. A fast but only approximate 0 0 n alternative to the SSA for the simulation of processes 0 xn0 e−x with Poisson basis function |n0i = obeys the evolving on multiple time scales is τ-leaping [78, 249, x n0! 258–267]. Algorithms for the simulation of processes backward-time flow equation with time-dependent transition rates are discussed k l k ∂−τ |pi = γτ x (∂x + 1) − (∂x + 1) |pi . (367) in [268, 269]. Let us note, however, that for small systems, a direct numerical integration of the master After one has solved one of these equations, the equation may be more efficient than the use of the SSA. conditional probability distribution can be recovered via the inverse transformations p(τ, n|t0, n0) = hn|g(τ|t0, n0)i or p(t, n|τ, n0) = hn0|p(t, n|τ)i. In Master equations and the theory of stochastic path integrals 63 sections 2 and 3, we generalized the above approaches transitions [14, 69, 70, 96, 130–133, 162, 293–319, 321, and formulated conditions under which the forward 323–325, 400, 422–425]. A classical example of such and backward master equations can be transformed a phase transition is the transition between an active into flow equations. Besides the flow equations obeyed and an absorbing state of a system [68]. In section 6, by the generating function and the marginalized we showed that for a process whose initial number distribution, we also derived a flow equation obeyed of particles is Poisson distributed with mean x(t0), by the generating functional the average of an observable A(n) at time t can be X represented by the path integral hg(τ|t0, n0)| = hn|p(τ, n|t0, n0) . (368) t] n † hAi = e−S hhAii (369) xne−x x(t0) x(t) For the Poisson basis function |nix = n! , the (t0 inverse transformation p(τ, n|t , n ) = hg(τ|t , n )|ni ∞ n −x 0 0 0 0 X x e of this functional recovers the Poisson representation with hhAiix ·= A(n) . (370) n! of Gardiner and Chaturvedi [32, 33]. The Poisson n=0 representation can, for example, be used for the The above path integral can be derived both from the computation of mean extinction times [374], but novel backward path integral representation we found the marginalized distribution to be more t] −S† convenient for this purpose (cf. section 3.3). p(t, n|t0, n0) = hn0| e |ni , (371) t0 t Thus far, most methods that have been proposed (t0 for the analysis of the generating function’s flow of the conditional probability distribution (cf. sec- equation have only been applied to systems without tion 4), and from the forward path integral representa- spatial degrees of freedom. These methods include tion a variational approach [39, 83], spectral formulations t) −S and WKB approximations [34, 36–38, 328, 364, 366, p(t, n|t0, n0) = hn| e |n0i (372) t t0 367, 370, 427]. We outlined some of these methods [t0 in section 2.3. “Real-space” WKB approximations, (cf. section 5). The meanings of the integral signs which employ an exponential ansatz for the probability and of the actions S and S† were explained in distribution rather than for the generating function, the respective sections. We derived both of the were not discussed in this review (information on these above representations of the conditional probability approximations can be found in [63, 64, 146, 171, 172, distribution from the flow equations discussed in 177, 354–360, 362–364]). WKB approximations often the previous paragraph. An extension of the path prove helpful in computing a mean extinction time integrals to systems with multiple types of particles or a (quasi)stationary probability distribution. Future or spatial degrees of freedom is straightforward. We studies could explore whether the flow equation obeyed demonstrated the use of the integrals in solving various by the marginalized distribution can be evaluated in elementary processes, including the pair generation terms of a WKB approximation or using spectral process and a process with linear decay of diffusing methods. Moreover, further research is needed to particles. Although we did not discuss the application investigate whether the above methods may be helpful of renormalization group techniques, we showed how in studying processes with spatial degrees of freedom the above path integrals can be evaluated in terms and multiple types of particles. For that purpose, of perturbation expansions using Feynman diagrams. it will be crucial to specify satisfactory boundary Information on perturbative renormalization group conditions for the flow equations obeyed by the techniques can be found in [69,70], information on non- generating function and the marginalized distribution perturbative techniques in [410]. Besides the above (see [366] for a discussion of “lacking” boundary path integrals, we showed how one can derive path conditions in a study of the branching-annihilation integral representations for processes with continuous reaction A → 2 A and 2 A → ∅). state spaces. These path integrals were based on Recently, we have been made aware of novel ways Kramers-Moyal expansions of the respective backward of treating the generating function’s flow equation and forward master equations. Upon truncating the based on duality relations [428–432]. These approaches expansion of the backward master equation at the are not yet covered here. level of a diffusion approximation, we recovered a classic path integral representation of the (backward) Stochastic path integrals (sections 4–6) Fokker-Planck equation [16–19]. We hope that our Path integral representations of the master equation exposition of the path integrals helps in developing have proved invaluable tools in gaining analytical and new methods for the analysis of stochastic processes. numerical insight into the behaviour of stochastic Future studies may focus directly on the backward or processes, particularly in the vicinity of phase forward path integral representations (371) and (372) Master equations and the theory of stochastic path integrals 64 of the conditional probability distribution, or use the for τ ∈ [t0, t], a function u(τ, x) obeys the linear PDE representation (369) to explore how (arbitrarily high) 1 ∂ u(τ, x) = α (x)∂ u + β (x)∂2u (A.1) moments of the particle number behave in the vicinity −τ τ x 2 τ x of phase transitions. with final value u(t, x) = G(x) . (A.2)

Stationary path approximations (section 7) The function ατ is called a drift coefficient and βτ a Elgart and Kamenev recently showed how the forward diffusion coefficient. According to the Feynman-Kac path integral representation (372) can be approximated formula, the above PDE is solved by by expanding its action around “stationary paths” [35]. u(τ, x) = G(X(t)) W (A.3) The paths obey Hamilton’s equations of the form with hh·iiW representing an average over realizations ∂Qτ ∂Qτ of the Wiener process W . The function X(s) with ∂τ x˜ = and ∂−τ q˜ = , (373) ∂q˜ ∂x˜ s ∈ [τ, t] and τ ≥ t0 represents a solution of the Itô with the transition operator Qτ acting as the “Hamilto- stochastic differential equation (SDE) nian”. In section 7, we reviewed this “stationary path p dX(s) = αs(X(s)) ds + βs(X(s)) dW (s) (A.4) method” and showed how it can be extended to the backward path integral representation (371). We found with initial value X(τ) = x . (A.5) that this backward approach does not require an aux- If the initial value x of the SDE is chosen as a real iliary saddle-point approximation if the number of number, the drift coefficient αs is a real function, particles is initially Poisson distributed, but that a and the diffusion coefficient βs as a real non-negative proper normalization of the probability distribution function, the “sample path” X(s) assumes only real is only attained beyond leading order. Future work values along its temporal evolution. In a multivariate is needed to apply the method to systems with spa- extension√ of the Feynman-Kac formula, one requires a | tial degrees of freedom and multiple types of particles. matrix βs ·= γs fulfilling γsγs = βs. If the matrix βs The latter point also applies to the classification of is symmetric and positive-semidefinite, one may choose phase transitions based on phase-space trajectories of γs as its unique symmetric and positive-semidefinite the equations (373) as has been proposed in [320]. square root [111]. The solution (3) of the backward Fokker-Planck We hope that our review of the above methods equation (2) constitutes a special case of the above inspires future research on master equations and that it formula. There, the independent variable is x0 helps researchers who are new to the field of stochastic instead of x, and u(τ, x0) is the conditional probability processes to become acquainted with the theory of distribution p(t, x|τ, x0) (with x being an arbitrary stochastic path integrals. parameter). The final value of the distribution is G(x0) = δ(x − x0). The Feynman-Kac formula (A.3)

Acknowledgments then implies that δ(x−X(t)) W solves the backward Fokker-Planck equation (2). Note that in the main For critical reading of a pre-submission manuscript and text, we use a small letter to denote the sample path. for providing valuable feedback, we would like to thank To prove the Feynman-Kac formula (A.3), we Michael Assaf, Léonie Canet, Alexander Dobrinevski, assume that X(s) solves the SDE (A.4). By Itô’s Nigel Goldenfeld, Peter Grassberger, Baruch Meerson, Lemma and the PDE (A.1), it then holds that Ralf Metzler, Mauro Mobilia, Harold P. de Vladar, ∂u(s, X(s)) du(s, X(s)) = pβ (X(s)) dW (s) . (A.6) Herbert Wagner, and Royce Zia. Moreover, we ∂X(s) s would like to thank Johannes Knebel, Isabella Krämer, and Cornelius Weig for helpful discussions during As the next step, we integrate this differential from the preparation of the manuscript. This research s = τ to s = t and average the result over realizations was financially supported by the German Excellence of the Wiener process W . Since hhdW iiW = 0, it follows Initiative via the program ‘NanoSystems Initiative that hhu(τ, X(τ))iiW = hhu(t, X(t))iiW . This expression Munich’ (NIM). coincides with the Feynman-Kac formula (A.3) because the initial condition (A.5) implies that u(τ, X(τ)) = u(τ, x) does not depend on the Wiener process, A. Proof of the Feynman-Kac formula in and because the final condition (A.2) implies that section 1.3 u(t, X(t)) = G(X(t)). Here we provide a brief proof of a special case of the Feynman-Kac, or Kolmogorov, formula (see section 4.3.5 in [110]). In particular, we assume that Master equations and the theory of stochastic path integrals 65

B. Proof of the path summation C. Solution of the random walk in representation in section 1.4 sections 2.2.1 and 3.2.1

In the following, we prove that the path summation We here provide the conditional probability distribu- representation (19) solves the forward master equa- tion p(τ, n|t0, n0) solving the random walk from sec- tion (11) if the transition rate w(n, m) is independ- tions 2.2.1 and 3.2.1. In the first of these sections, we ent of time. Hence, the process is homogeneous in showed that this distribution is obtained by applying · π dq −inq time and we may choose t0 ·= 0. After defining the functional hn|f = −π 2π e f(q) to the generat- · P d(n, m) ·= δn,mw(m) = δn,m k w(k, m), we first re- ing function |g(τ|t0, n0´)i, which we derived as write the master equation as  τ τ  exp (eiq − 1) ds r + (e−iq − 1) ds l + in q . ∂ p(τ|0, n ) = (w − d)p(τ|·) (B.1) s s 0 τ 0 ˆt0 ˆt0 with the probability vector p(τ|t0, n0). Note that Multiple series expansions show that this expression τ − ds (rs+ls) the matrix notation assumes some mapping between t0 can be rewritten as e ´ times the state space of n and an index set I ⊂ N. ∞ k τ
m τ
k−m X X t ds rs t ds ls Following [239], the Laplace transform 0 0 ei(2m−k+n0)q . ´ m! ´ (k − m)! ∞ k=0 m=0 · −sτ Lf(s) ·= dτ e f(τ) (B.2) τ − ds (rs+ls) ˆ0 t0 We ignore the constant pre-factor e ´ for of the above master equation is given by now and apply the functional hn| to this expression. sLp(s|·) − eˆ = (w − d)Lp(s|·) . (B.3) After carefully noting the restrictions imposed by n0 Kronecker deltas, the resulting expression reads Here, p(0|0, n ) = eˆ represents a unit vector pointing 0 n0 ∞ τ τ
k ds rs · ds ls in direction n0. Note that s is a scalar but that w X t0 t0 ´ ´ · (C.1) and d are matrices. Making use of the unit matrix 1, k!(k + |n − n0|)! k=0 the above expression can be solved for Lp(s|·) and be rewritten as This sum can be expressed by a modified Bessel function of the first kind (10.25.2 in [335]), resulting 1 1 −1 −1 1 −1 Lp(s|·) = − (s + d) w (s + d) eˆn0 . (B.4) in τ n−n0 τ τ 1 The value of the real part of s is determined by the  ds rs  2    2  t0 I 2 ds r ds l . parameter ε in the inverse Laplace transformation ´ τ n−n0 s s ds ls ˆt ˆt t0 0 0 ε+i∞ 1 ´ τ st − ds (rs+ls) f(t) = ds e Lf(s) . (B.5) t0 Multiplied with e ´ , this expression corres- 2πi ˆε−i∞ ponds to a Skellam distribution with mean µ = n0 + τ 2 τ We assume that ε can be chosen so large that the ds (rs − ls) and variance σ = ds (rs + ls). t0 t0 first factor in the solution (B.4) can be rewritten as ´ ´ a geometric series, i.e. as D. Details on the evaluation of the backward ∞ path integral representation in section 4.3 X 1 −1 J 1 −1 Lp(s|·) = (s + d) w (s + d) eˆn0 . (B.6) J=0 In the following, we fill out the missing steps in This expression may be simplified by noting that section 4.3 and rewrite the backward path integral w(n, m) representation (171) of the marginalized distribution in (s1 + d)−1w = . (B.7) n,m s + w(n) terms of the (Q, X)-generating functional (179). Upon comparing the discrete-time approximation of the By defining P ·= P ··· P , (B.6) becomes {PJ } · n1 nJ−1 marginalized distribution (168) with its representation Lp(s, n|·) = (B.8) in (177), one observes that the corresponding (Q, X)- generating functional should read ∞ J−1  J 1 X X Y Y N−1 w(nj+1, nj) . † iqN xN −S ZN s + w(nj) n =n ZQ,X,N ·= e N e (D.1) J=0 {PJ } j=0 j=0 J · 1 Since an exponential function f(τ) ·= e−ατ Θ(τ) N N−1 −1 X X transforms as Lf(s) = (s + α) and since the Laplace with ZN ·= ∆t Qjxj−1 + ∆t Xj−1iqj . (D.2) transform converts convolutions into products, we thus j=1 j=1 find that (B.8) agrees with the Laplace transform of the A differentiation of (D.1) with respect to ∆t Qj path summation representation (19). generates a factor xj−1, a differentiation with respect to ∆t Xj−1 a factor iqj. Upon recalling the definition Master equations and the theory of stochastic path integrals 66

† of the action SN in (169), the first exponential in (D.1) kept in mind when the functional derivatives in (D.9) can be rewritten via are evaluated in continuous-time. In the continuous- † time limit, i.e. for N → ∞ and ∆t → 0, one recovers iq x − S (D.3) N N N the marginalized distribution (177) with generating N−1 X    functional (179) and Itô SDE (181). = − iqj xj − xj−1 + αtj−1 (xj−1)∆t (D.4) j=1 N−1 2 N−1 [1] A. Einstein. Über die von der molekularkinetischen X qj X † − βt (xj−1)∆t + ∆t P (xj−1, iqj) Theorie der Wärme geforderte Bewegung von in 2 j−1 tj−1 j=1 j=1 ruhenden Flüssigkeiten suspendierten Teilchen. Ann. Phys. (Berlin), 322(8):549–560, 1905. doi:10.1002/ + iqN xN−1 . andp.19053220806. The equality holds up to corrections of O(∆t) (because [2] A. D. Fokker. Die mittlere Energie rotierender elektrischer Dipole im Strahlungsfeld. Ann. Phys. of the sums, all remaining terms are of O(1)). Note (Berlin), 348(5):810–820, 1914. doi:10.1002/andp. that the right hand side of (D.4) is independent of xN . 19143480507. One can linearize the terms that are quadratic in qj by [3] M. Planck. Über einen Satz der statistischen Dynamik the completion of a square. In particular, we write und seine Erweiterung in der Quantentheorie. Sitz.-ber. Preuß. Akad. Wiss., pages 324–341, 1917.  N−1 q2  [4] A. Kolmogoroff. Über die analytischen Methoden in der X j Wahrscheinlichkeitsrechnung. Math. Ann., 104(1):415– exp − βtj−1 (xj−1)∆t (D.5) 2 458, 1931. doi:10.1007/BF01457949. j=1 [5] A. Nordsieck, W. E. Lamb Jr., and G. E. Uhlenbeck. On N−1 DD  X q EE the theory of cosmic-ray showers I: The Furry model = exp iqj βtj−1 (xj−1)∆Wj and the fluctuation problem. Physica, 7(4):344–360, W j=1 1940. doi:10.1016/S0031-8914(40)90102-1. [6] M. Doi. Second quantization representation for classical with the average being defined as many-particle system. J. Phys. A: Math. Gen., N−1 9(9):1465–1477, 1976. doi:10.1088/0305-4470/9/9/  Y 
008. · ·= d∆Wj G0,∆t(∆Wj) · . (D.6) W ˆ [7] M. Doi. Stochastic theory of diffusion-controlled reaction. j=1 R J. Phys. A: Math. Gen., 9(9):1479–1495, 1976. doi: The average employs the Gaussian distribution 10.1088/0305-4470/9/9/009. [8] Ya B. Zel’dovich and A. A. Ovchinnikov. The mass 2 e−(∆Wj ) /(2∆t) action law and the kinetics of chemical reactions G0,∆t(∆Wj) = √ (D.7) with allowance for thermodynamic fluctuations of the 2π∆t density. Sov. Phys. JETP, 47(5):829–834, 1978. with zero mean and variance ∆t (cf. (200)). The [9] H. A. Rose. Renormalized kinetic theory of nonequilib- rium many-particle classical systems. J. Stat. Phys., sequence ∆W1,..., ∆WN−1 can be interpreted as the 20(4):415–447, 1979. doi:10.1007/bf01011780. steps of a discretized Wiener process. To proceed [10] P. Grassberger and M. Scheunert. Fock-space methods for with the derivation, we now move the perturbation identical classical objects. Fortschr. Phys., 28(10):547– operator P† to the front of the (Q, X)-generating 578, 1980. doi:10.1002/prop.19800281004. [11] A. S. Mikhailov. Path integrals in chemical kinetics functional (D.1) by writing I. Phys. Lett. A, 85(4):214–216, 1981. doi:10.1016/ PN−1 † 0375-9601(81)90017-7. ∆t P (xj−1,iqj ) Z e j=1 tj−1 e N (D.8) [12] A. S. Mikhailov. Path integrals in chemical kinetics II.

PN−1 † 1 ∂ 1 ∂
Phys. Lett. A, 85(8–9):427–429, 1981. doi:10.1016/ j=1 ∆t Pt , Z = e j−1 ∆t ∂Qj ∆t ∂Xj−1 e N . 0375-9601(81)90429-1. [13] N. Goldenfeld. Kinetics of a model for nucleation- Upon combining all of the above steps, one finds that controlled polymer crystal growth. J. Phys. A: Math. Gen., 17(14):2807–2821, 1984. doi:10.1088/0305- ZQ,X,N (D.9) 4470/17/14/024. 1 ∂ PN−1 † 1 ∂ 1 ∂
[14] A. S. Mikhailov and V. V. Yashin. Quantum-field iqN + ∆t P , ∆t ∂QN j=1 tj−1 ∆t ∂Qj ∆t ∂Xj−1 0 = e ZQ,X,N methods in the theory of diffusion-controlled reactions. J. Stat. Phys., 38(1-2):347–359, 1985. doi:10.1007/ with the definition BF01017866. N−1 [15] L. Peliti. Path integral approach to birth-death processes 0 DD Y  on a lattice. J. Phys. (Paris), 46(9):1469–1483, 1985. Z ·= dxj δ xj − xj−1 (D.10) Q,X,N ˆ doi:10.1051/jphys:019850046090146900. j=1 R [16] P. C. Martin, E. D. Siggia, and H. A. Rose. Statistical n q o dynamics of classical systems. Phys. Rev. A, 8(1):423– − α (x ) + X ∆t + β (x ) ∆W tj−1 j−1 j−1 tj−1 j−1 j 437, 1973. doi:10.1103/PhysRevA.8.423. [17] C. de Dominicis. Techniques de renormalisation de la PN ∆t Q x +O(∆t)EE · e j=1 j j−1 . théorie des champs et dynamique des phénomènes W critiques. J. Phys. (Paris), 37(C1):247–253, 1976. doi: The sequence of Dirac delta functions implies that xj 10.1051/jphyscol:1976138. [18] H.-K. Janssen. On a Lagrangean for classical field depends on Xi only for i < j. This property has to be Master equations and the theory of stochastic path integrals 67

dynamics and renormalization group calculations of extinction in a time-modulated environment. Phys. dynamical critical properties. Z. Phys. B, 23(4):377– Rev. E, 78(4):041123, 2008. doi:10.1103/PhysRevE. 380, 1976. doi:10.1007/BF01316547. 78.041123. [19] R. Bausch, H.-K. Janssen, and H. Wagner. Renormalized [39] M. Sasai and P. G. Wolynes. Stochastic gene expression field theory of critical dynamics. Z. Phys. B, 24(1):113– as a many-body problem. Proc. Natl. Acad. Sci. 127, 1976. doi:10.1007/BF01312880. USA, 100(5):2374–2379, 2003. doi:10.1073/pnas. [20] V. Fock. Konfigurationsraum und zweite Quantelung. Z. 2627987100. Phys., 75(9):622–647, 1932. doi:10.1007/BF01344458. [40] S. F. Edwards. The statistical dynamics of homogeneous [21] P. A. M. Dirac. A new notation for quantum mechanics. turbulence. J. Fluid Mech., 18(2):293–273, 1964. doi: Math. Proc. Camb. Phil. Soc., 35(3):416–418, 1939. 10.1017/S0022112064000180. doi:10.1017/S0305004100021162. [41] K. R. Sreenivasan and G. L. Eyink. Sam Edwards and the [22] E. Schrödinger. Der stetige Übergang von der Mikro- zur turbulence theory. In P. Goldbart, N. Goldenfeld, and Makromechanik. Naturwissenschaften, 14(28):664–666, D. Sherrington, editors, Stealing the Gold, pages 66–85. 1926. doi:10.1007/BF01507634. Oxford University Press, 2005. doi:10.1093/acprof: [23] E. C. G. Sudarshan. Equivalence of semiclassical and oso/9780198528531.001.0001. quantum mechanical descriptions of statistical light [42] H. Haken. Generalized Onsager-Machlup function and beams. Phys. Rev. Lett., 10(7):277–279, 1963. doi: classes of path integral solutions of the Fokker-Planck 10.1103/PhysRevLett.10.277. equation and the master equation. Z. Phys. B, [24] R. J. Glauber. Coherent and incoherent states of the 24(3):321–326, 1976. doi:10.1007/BF01360904. radiation field. Phys. Rev., 131(6):2766–2788, 1963. [43] K. Wódkiewicz. Functional representation of a non- doi:10.1103/PhysRev.131.2766. Markovian probability distribution in statistical mech- [25] G. C. Wick. The evaluation of the collision matrix. Phys. anics. Phys. Lett., 84(2):56–58, 1981. doi:10.1016/ Rev., 80(2):268–272, 1950. doi:10.1103/PhysRev.80. 0375-9601(81)90589-2. 268. [44] L. Pesquera, M. A. Rodriguez, and E. Santos. Path [26] S. Sandow and S. Trimper. Aggregation processes in a integrals for non-Markovian processes. Phys. Lett. master-equation approach. Europhys. Lett., 21(8):799– A, 94(6-7):287–289, 1983. doi:10.1016/0375-9601(83) 804, 1993. doi:10.1209/0295-5075/21/8/001. 90719-3. [27] H. Patzlaff, S. Sandow, and S. Trimper. Diffusion and [45] R. F. Fox. Functional-calculus approach to stochastic correlation in a coherent representation. Z. Phys. B, differential equations. Phys. Rev. A, 33(1):467–476, 95(3):357–362, 1994. doi:10.1007/BF01343964. 1986. doi:10.1103/PhysRevA.33.467. [28] P.-A. Bares and M. Mobilia. Diffusion-limited reactions [46] R. F. Fox. Uniform convergence to an effective Fokker- of hard-core particles in one dimension. Phys. Rev. Planck equation for weakly colored noise. Phys. Rev. E, 59(2):1996–2009, 1999. doi:10.1103/PhysRevE.59. A, 34(5):4525–4527, 1986. doi:10.1103/PhysRevA.34. 1996. 4525. [29] V. Brunel, K. Oerding, and F. van Wijland. Fermionic [47] J. F. Luciani and A. D. Verga. Functional integral field theory for directed percolation in (1 + 1)- approach to bistability in the presence of correlated dimensions. J. Phys. A: Math. Gen., 33(6):1085–1097, noise. Europhys. Lett., 4(3):255–261, 1987. doi:10. 2000. doi:10.1088/0305-4470/33/6/301. 1209/0295-5075/4/3/001. [30] M. Schulz and P. Reineker. Exact substitute processes for [48] A. Förster and A. S. Mikhailov. Optimal fluctuations diffusion-reaction systems with local complete exclusion leading to transitions in bistable systems. Phys. rules. New J. Phys., 7:31, 2005. doi:10.1088/1367- Lett. A, 126(8-9):459–462, 1988. doi:10.1016/0375- 2630/7/1/031. 9601(88)90039-4. [31] É. M. Silva, P. T. Muzy, and A. E. Santana. Fock space [49] J. F. Luciani and A. D. Verga. Bistability driven by for fermion-like lattices and the linear Glauber model. correlated noise: Functional integral treatment. J. Stat. Physica A, 387(21):5101–5109, 2008. doi:10.1016/j. Phys., 50(3):567–597, 1988. doi:10.1007/BF01026491. physa.2008.04.015. [50] A. J. Bray and A. J. McKane. Instanton calculation of the [32] C. W. Gardiner and S. Chaturvedi. The Poisson escape rate for activation over a potential barrier driven representation. I. A new technique for chemical master by colored noise. Phys. Rev. Lett., 62(5):493–496, 1989. equations. J. Stat. Phys., 17(6):429–468, 1977. doi: doi:10.1103/PhysRevLett.62.493. 10.1007/BF01014349. [51] P. Hänggi. Path integral solutions for non-Markovian [33] S. Chaturvedi and C. W. Gardiner. The Poisson processes. Z. Phys. B, 75(2):275–281, 1989. doi: representation. II. Two-time correlation functions. 10.1007/BF01308011. J. Stat. Phys., 18(5):501–522, 1978. doi:10.1007/ [52] A. J. McKane. Noise-induced escape rate over a potential BF01014520. barrier: Results for general noise. Phys. Rev. A, [34] A. M. Walczak, A. Mugler, and C. H. Wiggins. A 40(7):4050–4053, 1989. doi:10.1103/PhysRevA.40. stochastic spectral analysis of transcriptional regulatory 4050. cascades. Proc. Natl. Acad. Sci. USA, 106(16):6529– [53] A. J. McKane, H. C. Luckock, and A. J. Bray. 6534, 2009. doi:10.1073/pnas.0811999106. Path integrals and non-Markov processes. I. General [35] V. Elgart and A. Kamenev. Rare event statistics in formalism. Phys. Rev. A, 41(2):644–656, 1990. doi: reaction-diffusion systems. Phys. Rev. E, 70(4):041106, 10.1103/PhysRevA.41.644. 2004. doi:10.1103/PhysRevE.70.041106. [54] A. J. Bray, A. J. McKane, and T. J. Newman. Path [36] M. Assaf and B. Meerson. Spectral formulation and integrals and non-Markov processes. II. Escape rates WKB approximation for rare-event statistics in reaction and stationary distributions in the weak-noise limit. systems. Phys. Rev. E, 74(4):041115, 2006. doi: Phys. Rev. A, 41(2):657–667, 1990. doi:10.1103/ 10.1103/PhysRevE.74.041115. PhysRevA.41.657. [37] M. Assaf and B. Meerson. Spectral theory of metastability [55] H. C. Luckock and A. J. McKane. Path integrals and and extinction in birth-death systems. Phys. Rev. non-Markov processes. III. Calculation of the escape- Lett., 97(20):200602, 2006. doi:10.1103/PhysRevLett. rate prefactor in the weak-noise limit. Phys. Rev. 97.200602. A, 42(4):1982–1996, 1990. doi:10.1103/PhysRevA.42. [38] M. Assaf, A. Kamenev, and B. Meerson. Population 1982. Master equations and the theory of stochastic path integrals 68

[56] T. G. Venkatesh and L. M. Patnaik. Effective Fokker- [74] I. Oppenheim and K. E. Shuler. Master equations and Planck equation: Path-integral formalism. Phys. Rev. Markov processes. Phys. Rev., 138(4B):B1007–B1011, E, 48(4):2402–2412, 1993. doi:10.1103/PhysRevE.48. 1965. doi:10.1103/PhysRev.138.B1007. 2402. [75] D. T. Gillespie. A general method for numerically [57] S. J. B. Einchcomb and A. J. McKane. Using path- simulating the stochastic time evolution of coupled integral methods to calculate noise-induced escape rates chemical reactions. J. Comput. Phys., 22(4):403–434, in bistable systems: The case of quasi-monochromatic 1976. doi:10.1016/0021-9991(76)90041-3. noise. In M. Millonas, editor, Fluctuations and Order [76] D. T. Gillespie. Exact stochastic simulation of coupled - The New Synthesis, Institute for Nonlinear Science, chemical reactions. J. Phys. Chem., 81(25):2340–2361, pages 139–154. Springer, New York, NY, 1996. doi: 1977. doi:10.1021/j100540a008. 10.1007/978-1-4612-3992-5_10. [77] D. T. Gillespie. A rigorous derivation of the chemical [58] C. Mahanta and T. G. Venkatesh. Damped stochastic master equation. Physica A, 188(1):404–425, 1992. system driven by colored noise: Analytical solution by doi:10.1016/0378-4371(92)90283-V. a path integral approach. Phys. Rev. E, 62(2):1509– [78] D. T. Gillespie. Stochastic simulation of chemical kinetics. 1520, 2000. doi:10.1103/PhysRevE.62.1509. Annu. Rev. Phys. Chem., 58:35–55, 2007. doi:10. [59] F. W. Wiegel. Introduction to Path-Integral Methods 1146/annurev.physchem.58.032806.104637. in Physics and Polymer Science. World Scientific [79] L. Onsager and S. Machlup. Fluctuations and irreversible Publishing Company, Singapore, 1986. processes. Phys. Rev., 91(6):1505–1512, 1953. doi: [60] M. Doi and S. F. Edwards. The Theory of Polymer 10.1103/PhysRev.91.1505. Dynamics. Number 73 in International Series of [80] N. Wiener. The average of an analytic functional. Proc. Monographs on Physics. Clarendon Press, Oxford, 1988. Natl. Acad. Sci. USA, 7(9):253–260, 1921. doi:10. [61] T. A. Vilgis. Polymer theory: path integrals and scaling. 1073/pnas.7.9.253. Phys. Rep., 336(3):167–254, 2000. doi:10.1016/S0370- [81] N. Wiener. The average of an analytic functional and 1573(99)00122-2. the Brownian movement. Proc. Natl. Acad. Sci. USA, [62] H. Kleinert. Path Integrals in Quantum Mechanics, 7(10):294–298, 1921. doi:10.1073/pnas.7.10.294. Statistics, Polymer Physics, and Financial Markets, [82] R. Brown. XXVII. A brief account of microscopical chapter 15. World Scientific Publishing Company, New observations made in the months of June, July and Jersey, 5th edition, 2009. August 1827, on the particles contained in the pollen [63] P. C. Bressloff and J. M. Newby. Path integrals and of plants; and on the general existence of active large deviations in stochastic hybrid systems. Phys. molecules in organic and inorganic bodies. Philos. Rev. E, 89(4):042701, 2014. doi:10.1103/PhysRevE. Mag. Ser. 2, 4(21):161–173, 1828. doi:10.1080/ 89.042701. 14786442808674769. [64] P. C. Bressloff. Stochastic Processes in Cell Biology, [83] G. L. Eyink. Action principle in nonequilibrium statistical volume 41 of Interdisciplinary Applied Mathematics, dynamics. Phys. Rev. E, 54(4):3419–3435, 1996. doi: chapter 10. Springer, Cham, 2014. doi:10.1007/978- 10.1103/PhysRevE.54.3419. 3-319-08488-6. [84] C. M. Bender and S. A. Orszag. Advanced Mathematical [65] P. C. Bressloff. Path-integral methods for analyzing Methods for Scientists and Engineers I, chapter 10. the effects of fluctuations in stochastic hybrid neural Springer, New York, NY, 1999. doi:10.1007/978-1- networks. J. Math. Neurosc., 5(4):1–33, 2015. doi: 4757-3069-2. 10.1186/s13408-014-0016-z. [85] M. Kac. On distributions of certain Wiener functionals. [66] H. Hinrichsen. Non-equilibrium critical phe- Trans. Amer. Math. Soc., 65(1):1–13, 1949. doi:10. nomena and phase transitions into absorb- 2307/1990512. ing states. Adv. Phys., 49(7):815–958, 2000. [86] K. Itô. Stochastic integral. Proc. Imp. Acad., 20(8):519– doi:10.1080/00018730050198152. 524, 1944. doi:10.3792/pia/1195572786. [67] G. Ódor. Universality classes in nonequilibrium lattice [87] K. Itô. On a stochastic integral equation. Proc. systems. Rev. Mod. Phys., 76(3):663–724, 2004. doi: Japan Acad., 22(2):32–35, 1946. doi:10.3792/pja/ 10.1103/RevModPhys.76.663. 1195572371. [68] M. Henkel, H. Hinrichsen, and S. Lübeck. Non- [88] K. Itô. Stochastic differential equations in a differentiable Equilibrium Phase Transitions - Volume I: Absorbing manifold. Nagoya Math. J., 1:35–47, 1950. Phase Transitions. Theoretical and Mathematical [89] D. S. Lemons. Paul Langevin’s 1908 paper “On the theory Physics. Springer, Dordrecht, 2008. doi:10.1007/978- of Brownian motion” [“Sur la théorie du mouvement 1-4020-8765-3. brownien,” C. R. Acad. Sci. (Paris) 146, 530–533 [69] U. C. Täuber, M. Howard, and B. P. Vollmayr-Lee. (1908)]. Am. J. Phys., 65(11):1079–1081, 1997. doi: Applications of field-theoretic renormalization group 10.1119/1.18725. methods to reaction-diffusion problems. J. Phys. A: [90] F. van Wijland. Field theory for reaction-diffusion Math. Gen., 38(17):R79–R131, 2005. doi:10.1088/ processes with hard-core particles. Phys. Rev. 0305-4470/38/17/R01. E, 63(2):022101, 2001. doi:10.1103/PhysRevE.63. [70] U. C. Täuber. Critical Dynamics - A Field Theory 022101. Approach to Equilibrium and Non-Equilibrium Scaling [91] J. Cardy. Reaction-diffusion processes. In S. Nazar- Behavior. Cambridge University Press, Cambridge, enko and O. V. Zaboronski, editors, Non-equilibrium 2014. Statistical Mechanics and Turbulence, London Math- [71] R. Zwanzig. Memory effects in irreversible thermodynam- ematical Society Lecture Note Series, pages 108–161. ics. Phys. Rev., 124(4):983–992, 1961. doi:10.1103/ Cambridge University Press, Cambridge, 2008. doi: PhysRev.124.983. 10.1017/CBO9780511812149. [72] R. Zwanzig. On the identity of three generalized master [92] E. Schrödinger. An undulatory theory of the mechanics equations. Physica, 30(6):1109–1123, 1964. doi:10. of atoms and molecules. Phys. Rev., 28(6):1049–1070, 1016/0031-8914(64)90102-8. 1926. doi:10.1103/PhysRev.28.1049. [73] H. Mori. Transport, collective motion, and Brownian [93] W. Heisenberg. Über quantentheoretische Umdeutung motion. Prog. Theor. Phys., 33(3):423–455, 1965. doi: kinematischer und mechanischer Beziehungen. Z. 10.1143/PTP.33.423. Phys., 33(1):879–893, 1925. doi:10.1007/BF01328377. Master equations and the theory of stochastic path integrals 69

[94] M. Born and P. Jordan. Zur Quantenmechanik. Z. Phys., compartments. Phys. Rev. Lett., 87(10):108101, 2001. 34(1):858–888, 1925. doi:10.1007/BF01328531. doi:10.1103/PhysRevLett.87.108101. [95] M. Born, W. Heisenberg, and P. Jordan. Zur Quanten- [114] A. Parmeggiani, T. Franosch, and E. Frey. Phase coexist- mechanik. II. Z. Phys., 35(8-9):557–615, 1926. doi: ence in driven one-dimensional transport. Phys. Rev. 10.1007/BF01379806. Lett., 90(8):086601, 2003. doi:10.1103/PhysRevLett. [96] J. L. Cardy and U. C. Täuber. Field theory of branching 90.086601. and annihilating random walks. J. Stat. Phys., [115] A. Parmeggiani, T. Franosch, and E. Frey. Totally 90(1/2):1–56, 1998. doi:10.1023/A:1023233431588. asymmetric simple exclusion process with Langmuir [97] B. Sinervo and C. M. Lively. The rock-paper-scissors game kinetics. Phys. Rev. E, 70(4):046101, 2004. doi:10. and the evolution of alternative male strategies. Nature, 1103/PhysRevE.70.046101. 380(6571):240–243, 1996. doi:10.1038/380240a0. [116] J. Howard. Molecular motors: structural adaptations to [98] N. G. van Kampen. Stochastic Processes in Physics and cellular functions. Nature, 389(6651):561–567, 1997. Chemistry. North-Holland Personal Library. Elsevier, doi:10.1038/39247. Amsterdam, 3rd edition, 2007. [117] W. J. Anderson. Continuous-Time Markov Chains - An [99] R. Huang, I. Chavez, K. M. Taute, B. Lukić, S. Jeney, Applications-Oriented Approach. Springer Series in M. G. Raizen, and E.-L. Florin. Direct observation of Statistics - Probability and its Applications. Springer, the full transition from ballistic to diffusive Brownian New York, NY, 1991. doi:10.1007/978-1-4612-3038- motion in a liquid. Nat. Phys., 7(7):576–580, 2011. 0. doi:10.1038/nphys1953. [118] A. Kolmogoroff. Zur Theorie der Markoffschen Ketten. [100] T. Franosch, M. Grimm, M. Belushkin, F. M. Mor, Math. Ann., 112(1):155–160, 1936. doi:10.1007/ G. Foffi, L. Forró, and S. Jeney. Resonances BF01565412. arising from hydrodynamic memory in Brownian [119] W. H. Furry. On fluctuation phenomena in the passage motion. Nature, 478(7367):85–88, 2011. doi:10.1038/ of high energy electrons through lead. Phys. Rev., nature10498. 52(6):569–581, 1937. doi:10.1103/PhysRev.52.569. [101] M. von Smoluchowski. Zur kinetischen Theorie der [120] W. Feller. Die Grundlagen der Volterraschen Theorie Brownschen Molekularbewegung und der Suspensionen. des Kampfes ums Dasein in wahrscheinlichkeitstheor- Ann. Phys. (Berlin), 326(14):756–780, 1906. doi:10. etischer Behandlung. Acta Biotheor., 5(1):11–40, 1939. 1002/andp.19063261405. doi:10.1007/BF01602932. [102] O. Klein. Zur statistischen Theorie der Suspensionen und [121] M. Delbrück. Statistical fluctuations in autocatalytic Lösungen. Ark. Mat. Astr. Fys., 16(5):PhD thesis, reactions. J. Chem. Phys., 8(1):120–124, 1940. doi: 1921. 10.1063/1.1750549. [103] H. A. Kramers. Brownian motion in a field of [122] K. Singer. Application of the theory of stochastic force and the diffusion model of chemical reactions. processes to the study of irreproducible chemical Physica, 7(4):284–304, 1940. doi:10.1016/S0031- reactions and nucleation processes. J. R. Stat. Soc. B, 8914(40)90098-2. 15(1):92–106, 1953. [104] H. Risken. The Fokker-Planck Equation - Methods of [123] A. F. Bartholomay. Stochastic models for chemical Solution and Applications, volume 18 of Springer Series reactions: I. Theory of the unimolecular reaction in Synergetics. Springer, Berlin, Heidelberg, 2nd process. B. Math. Biophys., 20(3):175–190, 1958. doi: edition, 1996. doi:10.1007/978-3-642-61544-3. 10.1007/bf02478297. [105] M. Kimura. Some problems of stochastic-processes in [124] I. M. Krieger and P. J. Gans. First-order stochastic genetics. Ann. Math. Statist., 28(4):882–901, 1957. processes. J. Chem. Phys., 32(1):247–250, 1960. doi: doi:10.1214/aoms/1177706791. 10.1063/1.1700909. [106] H. P. de Vladar and N. H. Barton. The contribution [125] K. Ishida. Stochastic model for bimolecular reaction. J. of statistical physics to evolutionary biology. Trends Chem. Phys., 41(8):2472–2478, 1964. doi:10.1063/1. Ecol. Evol., 26(8):424–432, 2011. doi:10.1016/j.tree. 1726290. 2011.04.002. [126] D. A. McQuarrie. Kinetics of small systems. I. J. Chem. [107] R. P. Feynman. The principle of least action in quantum Phys., 38(2):433–436, 1963. doi:10.1063/1.1733676. mechanics. In L. M. Brown, editor, Feynman’s Thesis: [127] D. A. McQuarrie, C. J. Jachimowski, and M. E. Russell. A New Approach to Quantum Theory, pages 1–69. Kinetics of small systems. II. J. Chem. Phys., World Scientific Publishing Company, Singapore, 2005. 40(10):2914–2921, 1964. doi:10.1063/1.1724926. [108] P. E. Kloeden and E. Platen. Numerical Solution [128] D. A. McQuarrie. Stochastic approach to chemical of Stochastic Differential Equations, volume 23 of kinetics. J. Appl. Probab., 4(3):413–478, 1967. doi: Applications of Mathematics - Stochastic Modelling and 10.2307/3212214. Applied Probability. Springer, Berlin, Heidelberg, 1992. [129] G. Nicolis and I. Prigogine. Self-Organization in doi:10.1007/978-3-662-12616-5. Nonequilibrium Systems - From Dissipative Structures [109] S. N. Majumdar. Brownian functionals in physics and to Order Through Fluctuations, chapter 10. John Wiley computer science. Curr. Sci., 89(12):2076–2092, 2005. & Sons, New York, NY, 1977. [110] C. Gardiner. Stochastic Methods - A Handbook for the [130] B. P. Lee. Renormalization group calculation for the Natural and Social Sciences, volume 13 of Springer reaction kA → ∅. J. Phys. A: Math. Gen., 27(8):2633– Series in Synergetics. Springer, Berlin, Heidelberg, 4th 2652, 1994. doi:10.1088/0305-4470/27/8/004. edition, 2009. [131] B. P. Lee and J. Cardy. Renormalization group [111] D. A. Harville. Matrix Algebra From a Statistician’s study of the A + B → ∅ diffusion-limited reaction. J. Perspective, chapter 21.9. Springer, New York, NY, Stat. Phys., 80(5-6):971–1007, 1995. doi:10.1007/ 1997. doi:10.1007/b98818. BF02179861. [112] G. Volpe and J. Wehr. Effective drifts in dynamical [132] J. Cardy and U. C. Täuber. Theory of branching systems with multiplicative noise: a review of recent and annihilating random walks. Phys. Rev. Lett., progress. Rep. Prog. Phys., 79(5), 2016. doi:10.1088/ 77(23):4780–4783, 1996. doi:10.1103/PhysRevLett. 0034-4885/79/5/053901. 77.4780. [113] R. Lipowsky, S. Klumpp, and T. M. Nieuwenhuizen. [133] H.-K. Janssen and U. C. Täuber. The field theory Random walks of cytoskeletal motors in open and closed approach to percolation processes. Ann. Phys., Master equations and the theory of stochastic path integrals 70

315(1):147–192, 2005. doi:10.1016/j.aop.2004.09. 3. 011. [154] T. Reichenbach, M. Mobilia, and E. Frey. Coexistence [134] R. J. Glauber. Time-dependent statistics of the Ising versus extinction in the stochastic cyclic Lotka-Volterra model. J. Math. Phys., 4(2):294–307, 1963. doi: model. Phys. Rev. E, 74(5):051907, 2006. doi:10. 10.1063/1.1703954. 1103/PhysRevE.74.051907. [135] K. Kawasaki. Diffusion constants near the critical point [155] I. Volkov, J. R. Banavar, S. P. Hubbell, and A. Maritan. for time-dependent Ising models. I. Phys. Rev., Patterns of relative species abundance in rainforests 145(1):224–230, 1966. doi:10.1103/PhysRev.145.224. and coral reefs. Nature, 450(7166):45–49, 2007. doi: [136] K. Kawasaki. Diffusion constants near the critical point 10.1038/nature06197. for time-dependent Ising models. II. Phys. Rev., [156] T. Butler and D. Reynolds. Predator-prey quasi- 148(1):375–381, 1966. doi:10.1103/PhysRev.148.375. cycles from a path-integral formalism. Phys. Rev. [137] K. Kawasaki. Diffusion constants near the critical point E, 79(3):032901, 2009. doi:10.1103/PhysRevE.79. for time-dependent Ising models. III. Self-diffusion 032901. constant. Phys. Rev., 150(1):285–290, 1966. doi: [157] T. Butler and N. Goldenfeld. Robust ecological pattern 10.1103/physrev.150.285. formation induced by demographic noise. Phys. Rev. [138] M. B. Elowitz and S. Leibler. A synthetic oscillat- E, 80(3):030902(R), 2009. doi:10.1103/PhysRevE.80. ory network of transcriptional regulators. Nature, 030902. 403(6767):335–338, 2000. doi:10.1038/35002125. [158] J.-M. Park, E. Muñoz, and M. W. Deem. Quas- [139] M. Thattai and A. van Oudenaarden. Intrinsic noise ispecies theory for finite populations. Phys. Rev. in gene regulatory networks. Proc. Natl. Acad. Sci. E, 81(1):011902, 2010. doi:10.1103/PhysRevE.81. USA, 98(15):8614–8619, 2001. doi:10.1073/pnas. 011902. 151588598. [159] A. E. Noble, A. Hastings, and W. F. Fagan. Multivariate [140] C. V. Rao, D. M. Wolf, and A. P. Arkin. Control, ex- Moran process with Lotka-Volterra phenomenology. ploitation and tolerance of intracellular noise. Nature, Phys. Rev. Lett., 107(22):228101, 2011. doi:10.1103/ 420(6912):231–237, 2002. doi:10.1038/nature01258. PhysRevLett.107.228101. [141] G. Karlebach and R. Shamir. Modelling and analysis of [160] A. Dobrinevski and E. Frey. Extinction in neutrally gene regulatory networks. Nat. Rev. Mol. Cell. Biol., stable stochastic Lotka-Volterra models. Phys. Rev. 9(10):770–780, 2008. doi:10.1038/nrm2503. E, 85(5):051903, 2012. doi:10.1103/PhysRevE.85. [142] V. Shahrezaei and P. S. Swain. Analytical distributions 051903. for stochastic gene expression. Proc. Natl. Acad. Sci. [161] A. J. Black and A. J. McKane. Stochastic formulation USA, 105(45):17256–17261, 2008. doi:10.1073/pnas. of ecological models and their applications. Trends 0803850105. Ecol. Evol., 27(6):337–345, 2012. doi:10.1016/j.tree. [143] L. S. Tsimring. Noise in biology. Rep. Prog. Phys., 2012.01.014. 77(2):026601, 2014. doi:10.1088/0034-4885/77/2/ [162] U. C. Täuber. Population oscillations in spatial stochastic 026601. Lotka–Volterra models: a field-theoretic perturbational [144] N. T. J. Bailey. A simple stochastic epidemic. Biometrika, analysis. J. Phys. A: Math. Theor., 45(40):405002, 37(3/4):193–202, 1950. doi:10.2307/2332371. 2012. doi:10.1088/1751-8113/45/40/405002. [145] M. J. Keeling and J. V. Ross. On methods for studying [163] H.-Y. Shih and N. Goldenfeld. Path-integral calculation stochastic disease dynamics. J. R. Soc. Interface, for the emergence of rapid evolution from demographic 5(19):171–181, 2008. doi:10.1098/rsif.2007.1106. stochasticity. Phys. Rev. E, 90(5):050702(R), 2014. [146] A. J. Black and A. J. McKane. WKB calculation of doi:10.1103/PhysRevE.90.050702. an epidemic outbreak distribution. J. Stat. Mech. [164] J. Cremer, A. Melbinger, and E. Frey. Growth Theor. Exp., 2011(12):P12006, 2011. doi:10.1088/ dynamics and the evolution of cooperation in microbial 1742-5468/2011/12/P12006. populations. Sci. Rep., 2:281, 2012. doi:10.1038/ [147] K. Rock, S. Brand, J. Moir, and M. J. Keeling. Dynamics srep00281. of infectious diseases. Rep. Prog. Phys., 77(2):026602, [165] M. Reiter, S. Rulands, and E. Frey. Range expansion 2014. doi:10.1088/0034-4885/77/2/026602. of heterogeneous populations. Phys. Rev. Lett., [148] E. Clayton, D. P. Doupé, A. M. Klein, D. J. Winton, B. D. 112(14):148103, 2014. doi:10.1103/PhysRevLett.112. Simons, and P. H. Jones. A single type of progenitor cell 148103. maintains normal epidermis. Nature, 446(7132):185– [166] M. F. Weber, G. Poxleitner, E. Hebisch, E. Frey, and 189, 2007. doi:10.1038/nature05574. M. Opitz. Chemical warfare and survival strategies [149] T. Chou. Peeling and sliding in nucleosome repositioning. in bacterial range expansions. J. R. Soc. Interface, Phys. Rev. Lett., 99(5):058105, 2007. doi:10.1103/ 11(96):20140172, 2014. doi:10.1098/rsif.2014.0172. PhysRevLett.99.058105. [167] K. Wienand, M. Lechner, F. Becker, H. Jung, and E. Frey. [150] I. Volkov, J. R. Banavar, S. P. Hubbell, and A. Maritan. Non-selective evolution of growing populations. PLoS Neutral theory and relative species abundance in ONE, 10(8):e0134300, 2015. doi:10.1371/journal. ecology. Nature, 424(6952):1035–1037, 2003. doi: pone.0134300. 10.1038/nature01883. [168] T. Reichenbach, M. Mobilia, and E. Frey. Mobility [151] A. J. McKane and T. J. Newman. Stochastic models promotes and jeopardizes biodiversity in rock-paper- in population biology and their deterministic analogs. scissors games. Nature, 448(7157):1046–1049, 2007. Phys. Rev. E, 70(4):041902, 2004. doi:10.1103/ doi:10.1038/nature06095. PhysRevE.70.041902. [169] T. Reichenbach, M. Mobilia, and E. Frey. Noise and [152] A. J. McKane and T. J. Newman. Predator-prey correlations in a spatial population model with cyclic cycles from resonant amplification of demographic competition. Phys. Rev. Lett., 99(23):238105, 2007. stochasticity. Phys. Rev. Lett., 94(21):218102, 2005. doi:10.1103/PhysRevLett.99.238105. doi:10.1103/PhysRevLett.94.218102. [170] T. Reichenbach, M. Mobilia, and E. Frey. Self- [153] M. Mobilia, I. T. Georgiev, and U. C. Täuber. organization of mobile populations in cyclic compet- Phase transitions and spatio-temporal fluctuations in ition. J. Theor. Biol., 254(2):368–383, 2008. doi: stochastic lattice Lotka-Volterra models. J. Stat. Phys., 10.1016/j.jtbi.2008.05.014. 128(1):447–483, 2007. doi:10.1007/s10955-006-9146- [171] M. Mobilia and M. Assaf. Fixation in evolutionary Master equations and the theory of stochastic path integrals 71

games under non-vanishing selection. Europhys. Lett., statistical mechanics: from a paradigmatic model to 91(1):10002, 2010. doi:10.1209/0295-5075/91/10002. biological transport. Rep. Prog. Phys., 74(11):116601, [172] M. Assaf and M. Mobilia. Large fluctuations and 2011. doi:10.1088/0034-4885/74/11/116601. fixation in evolutionary games. J. Stat. Mech. Theor. [191] D. Vorberg, W. Wustmann, R. Ketzmerick, and A. Eck- Exp., 2010(09):P09009, 2010. doi:10.1088/1742- ardt. Generalized Bose-Einstein condensation into 5468/2010/09/P09009. multiple states in driven-dissipative systems. Phys. [173] E. Frey. Evolutionary game theory: Theoretical concepts Rev. Lett., 111(24):240405, 2013. doi:10.1103/ and applications to microbial communities. Physica A, PhysRevLett.111.240405. 389(20):4265–4298, 2010. doi:10.1016/j.physa.2010. [192] D. Vorberg, W. Wustmann, H. Schomerus, R. Ketzmerick, 02.047. and A. Eckardt. Nonequilibrium steady states of [174] A. Melbinger, J. Cremer, and E. Frey. Evolutionary ideal bosonic and fermionic quantum gases. Phys. game theory in growing populations. Phys. Rev. Lett., Rev. E, 92(6):062119, 2015. doi:10.1103/PhysRevE. 105(17):178101, 2010. doi:10.1103/PhysRevLett.105. 92.062119. 178101. [193] M. R. Evans and B. Waclaw. Condensation in stochastic [175] J. Cremer, A. Melbinger, and E. Frey. Evolutionary mass transport models: beyond the zero-range process. and population dynamics: A coupled approach. Phys. J. Phys. A: Math. Theor., 47(9):095001, 2014. doi: Rev. E, 84(5):051921, 2011. doi:10.1103/PhysRevE. 10.1088/1751-8113/47/9/095001. 84.051921. [194] S. Grosskinsky, F. Redig, and K. Vafayi. Condensation in [176] A. Traulsen, J. C. Claussen, and C. Hauert. Stochastic the inclusion process and related models. J. Stat. Phys., differential equations for evolutionary dynamics with 142(5):952–974, 2011. doi:10.1007/s10955-011-0151- demographic noise and mutations. Phys. Rev. 9. E, 85(4):041901, 2012. doi:10.1103/PhysRevE.85. [195] C. Giardinà, F. Redig, and K. Vafayi. Correlation 041901. inequalities for interacting particle systems with duality. [177] A. J. Black, A. Traulsen, and T. Galla. Mixing times J. Stat. Phys., 141(2):242–263, 2010. doi:10.1007/ in evolutionary game dynamics. Phys. Rev. Lett., s10955-010-0055-0. 109(2):028101, 2012. doi:10.1103/PhysRevLett.109. [196] C. Giardinà, J. Kurchan, and F. Redig. Duality and exact 028101. correlations for a model of heat conduction. J. Math. [178] J. Knebel, M. F. Weber, T. Krüger, and E. Frey. Phys., 48(3):033301, 2007. doi:10.1063/1.2711373. Evolutionary games of condensates in coupled birth- [197] F. Spitzer. Interaction of Markov processes. Adv. death processes. Nat. Commun., 6:6977, 2015. doi: Math., 5(2):246–290, 1970. doi:10.1016/0001- 10.1038/ncomms7977. 8708(70)90034-4. [179] A. Melbinger, J. Cremer, and E. Frey. The emergence of [198] H. Spohn. Large Scale Dynamics of Interacting Particles. cooperation from a single mutant during microbial life Texts and Monographs in Physics. Springer, Berlin, cycles. J. R. Soc. Interface, 12(108):20150171, 2015. Heidelberg, 1991. doi:10.1007/978-3-642-84371-6. doi:10.1098/rsif.2015.0171. [199] T. M. Liggett. Stochastic Interacting Systems: Con- [180] J. Krug and H. Spohn. Kinetic roughening of growing tact, Voter and Exclusion Processes, volume 324 surfaces. In C. Godrèche, editor, Solids Far from of Grundlehren der mathematischen Wissenschaften. Equilibrium, Collection Alea-Saclay: Monographs and Springer, Berlin, Heidelberg, 1999. doi:10.1007/978- Texts in Statistical Physics. Cambridge University 3-662-03990-8. Press, Cambridge, 1991. [200] G. M. Schütz. Exactly solvable models for many-body [181] W. Weidlich. Physics and social science - the approach systems far from equilibrium. In C. Domb and of synergetics. Phys. Rep., 204(1):1–163, 1991. doi: J. L. Lebowitz, editors, Phase Transitions and Critical 10.1016/0370-1573(91)90024-G. Phenomena, volume 19. Academic Press, San Diego, [182] W. Weidlich and M. Braun. The master equation CA, 2000. approach to nonlinear economics. J. Evol. Econ., [201] S. A. Janowsky and J. L. Lebowitz. Finite-size effects and 2(3):233–265, 1992. doi:10.1007/bf01202420. shock fluctuations in the asymmetric simple-exclusion [183] T. Lux. Herd behaviour, bubbles and crashes. Econ. J., process. Phys. Rev. A, 45(2):618–625, 1992. doi: 105(431):881–896, 1995. doi:10.2307/2235156. 10.1103/PhysRevA.45.618. [184] C. Castellano, S. Fortunato, and V. Loreto. Statistical [202] B. Derrida, E. Domany, and D. Mukamel. An exact physics of social dynamics. Rev. Mod. Phys., 81(2):591– solution of a one-dimensional asymmetric exclusion 646, 2009. doi:10.1103/RevModPhys.81.591. model with open boundaries. J. Stat. Phys., 69(3- [185] D. Gross, J. F. Shortle, J. M. Thompson, and C. M. 4):667–687, 1992. doi:10.1007/BF01050430. Harris. Fundamentals of Queueing Theory. John Wiley [203] B. Derrida, M. R. Evans, V. Hakim, and V. Pasquier. & Sons, Hoboken, NJ, 4th edition, 2008. Exact solution of a 1D asymmetric exclusion model [186] K. Nagel and M. Schreckenberg. A cellular automaton using a matrix formulation. J. Phys. A: Math. Gen., model for freeway traffic. J. Phys. I France, 2(12):2221– 26(7):1493–1517, 1993. doi:10.1088/0305-4470/26/7/ 2229, 1992. doi:10.1051/jp1:1992277. 011. [187] D. Helbing. Traffic and related self-driven many-particle [204] G. Schütz and E. Domany. Phase transitions in an systems. Rev. Mod. Phys., 73(4):1067–1141, 2001. doi: exactly soluble one-dimensional exclusion process. J. 10.1103/RevModPhys.73.1067. Stat. Phys., 72(1-2):277–296, 1993. doi:10.1007/ [188] T. Reichenbach, T. Franosch, and E. Frey. Exclusion BF01048050. processes with internal states. Phys. Rev. Lett., [205] G. M. Schütz. Exact solution of the master equation for 97(5):050603, 2006. doi:10.1103/PhysRevLett.97. the asymmetric exclusion process. J. Stat. Phys., 88(1- 050603. 2):427–445, 1997. doi:10.1007/BF02508478. [189] T. Kriecherbauer and J. Krug. A pedestrian’s view [206] K. Johansson. Shape fluctuations and random matrices. on interacting particle systems, KPZ universality Commun. Math. Phys, 209(2):437–476, 2000. doi: and random matrices. J. Phys. A: Math. Theor., 10.1007/s002200050027. 43(40):403001, 2010. doi:10.1088/1751-8113/43/40/ [207] V. B. Priezzhev. Exact nonstationary probabilities in the 403001. asymmetric exclusion process on a ring. Phys. Rev. [190] T. Chou, K. Mallick, and R. K. P. Zia. Non-equilibrium Lett., 91(5):050601, 2003. doi:10.1103/PhysRevLett. Master equations and the theory of stochastic path integrals 72

91.050601. formization and quasi-Monte Carlo. J. Chem. Phys., [208] V. B. Priezzhev. Non-stationary probabilities for the 128(15):154109, 2008. doi:10.1063/1.2897976. asymmetric exclusion process on a ring. Pramana J. [226] F. Didier, T. A. Henzinger, M. Mateescu, and V. Wolf. Phys., 64(6):915–925, 2005. doi:10.1007/BF02704153. Fast adaptive uniformization of the chemical master [209] C. A. Tracy and H. Widom. Integral formulas for the equation. In High Performance Computational Sys- asymmetric simple exclusion process. Commun. Math. tems Biology, HIBI ’09, pages 118–127, Trento, 14-16 Phys, 279(3):815–844, 2008. doi:10.1007/s00220- Oct. 2009. IEEE. doi:10.1109/HiBi.2009.23. 008-0443-3. [227] D. Gross and D. R. Miller. The randomization technique [210] C. A. Tracy and H. Widom. A Fredholm determinant as a modeling tool and solution procedure for transient representation in ASEP. J. Stat. Phys., 132(2):291– Markov processes. Oper. Res., 32(2):343–361, 1984. 300, 2008. doi:10.1007/s10955-008-9562-7. doi:10.1287/opre.32.2.343. [211] C. A. Tracy and H. Widom. Asymptotics in ASEP [228] A. Reibman and K. Trivedi. Numerical transient analysis with step initial condition. Commun. Math. Phys, of Markov-models. Comput. Opns Res., 15(1):19–36, 290(1):129–154, 2009a. doi:10.1007/s00220-009- 1988. doi:10.1016/0305-0548(88)90026-3. 0761-0. [229] A. P. A. van Moorsel and W. H. Sanders. Adapt- [212] C. A. Tracy and H. Widom. Total current fluctuations ive uniformization. Comm. Statist. Stochastic in the asymmetric simple exclusion process. J. Math. Models, 10(3):619–647, 1994. doi:10.1080/ Phys., 50(9):095204, 2009. doi:10.1063/1.3136630. 15326349408807313. [213] A. Melbinger, L. Reese, and E. Frey. Microtubule length [230] W. Feller. On the integro-differential equations of purely regulation by molecular motors. Phys. Rev. Lett., discontinuous Markoff processes. Trans. Amer. Math. 108(25):258104, 2012. doi:10.1103/PhysRevLett.108. Soc., 48:488–515, 1940. doi:10.2307/1990095. 258104. [231] J. L. Doob. Markoff chains–denumerable case. Trans. [214] Y. Ishihama, T. Schmidt, J. Rappsilber, M. Mann, Amer. Math. Soc., 58(3):455–473, 1945. doi:10.2307/ F. U. Hartl, M. J. Kerner, and D. Frishman. Protein 1990339. abundance profiling of the Escherichia coli cytosol. [232] J. L. Doob. Topics in the theory of Markoff chains. BMC Genomics, 9:102, 2008. doi:10.1186/1471- Trans. Amer. Math. Soc., 52:37–64, 1942. doi:10. 2164-9-102. 1090/S0002-9947-1942-0006633-7. [215] C. R. Doering, K. V. Sargsyan, and L. M. Sander. [233] D. G. Kendall. Stochastic processes and population Extinction times for birth-death processes: exact growth. J. R. Stat. Soc. B, 11(2):230–282, 1949. results, continuum asymptotics, and the failure of [234] D. G. Kendall. An artificial realization of a simple “birth- the Fokker-Planck approximation. Multiscale Model. and-death” process. J. R. Stat. Soc. B, 12(1):116–119, Simul., 3(2):283–299, 2005. doi:10.1137/030602800. 1950. [216] J. R. Norris. Markov Chains, chapter 2–3. Cambridge [235] M. S. Bartlett. Stochastic processes or the statistics of Series in Statistical and Probabilistic Mathematics. change. J. R. Stat. Soc. C, 2(1):44–64, 1953. doi: Cambridge University Press, Cambridge, 1998. doi: 10.2307/2985327. 10.2277/0521633966. [236] A. B. Bortz, M. H. Kalos, and J. L. Lebowitz. A new [217] C. Moler and C. Van Loan. Nineteen dubious ways to algorithm for Monte Carlo simulation of Ising spin compute the exponential of a matrix, twenty-five years systems. J. Comput. Phys., 17(1):10–18, 1975. doi: later. SIAM Rev., 45(1):3–49, 2003. doi:10.1137/ 10.1016/0021-9991(75)90060-1. S00361445024180. [237] U. Seifert. Stochastic thermodynamics, fluctuation [218] C. Moler and C. Van Loan. Nineteen dubious ways to theorems and molecular machines. Rep. Prog. Phys., compute the exponential of a matrix. SIAM Rev., 75(12):126001, 2012. doi:10.1088/0034-4885/75/12/ 20(4):801–836, 1978. doi:10.1137/1020098. 126001. [219] B. Munsky and M. Khammash. The finite state projection [238] N. Empacher. Die Wegintegrallösung der Master- algorithm for the solution of the chemical master gleichung. PhD thesis, University of Stuttgart, 1992. equation. J. Chem. Phys., 124(4):044104–1–044104–13, [239] D. Helbing. A contracted path integral solution of the 2006. doi:10.1063/1.2145882. discrete master equation. Phys. Lett. A, 195(2):128– [220] K. Burrage, M. Hegland, S. MacNamara, and R. B. Sidje. 134, 1994. doi:10.1016/0375-9601(94)90085-X. A Krylov-based finite state projection algorithm for [240] D. Helbing and R. Molini. Occurence probabilities of solving the chemical master equation arising in the stochastic paths. Phys. Lett. A, 212(3):130–137, 1996. discrete modelling of biological systems. In A. N. doi:10.1016/0375-9601(96)00010-2. Langville and W. J. Stewart, editors, Proc. of The A. [241] S. X. Sun. Path summation formulation of the master A. Markov 150th Anniversary Meeting, pages 21–37, equation. Phys. Rev. Lett., 96(21):210602, 2006. doi: Charleston, SC, 12-14 June 2006. 10.1103/PhysRevLett.96.210602. [221] W. Magnus. On the exponential solution of differential [242] B. Harland and S. X. Sun. Path ensembles and path equations for a linear operator. Comm. Pure sampling in nonequilibrium stochastic systems. J. Appl. Math., 7(4):649–673, 1954. doi:10.1002/cpa. Chem. Phys., 127(10):104103, 2007. doi:10.1063/1. 3160070404. 2775439. [222] S. Blanes, F. Casas, J. A. Oteo, and J. Ros. The [243] A. D. Jackson and S. Pigolotti. Statistics of tra- Magnus expansion and some of its applications. jectories in two-state master equations. Phys. Rev. Phys. Rep., 470(5-6):151–238, 2009. doi:10.1016/j. E, 79(2):021121, 2009. doi:10.1103/PhysRevE.79. physrep.2008.11.001. 021121. [223] A. Klenke. Probability Theory - A Comprehensive Course, [244] H. Jasiulewicz and W. Kordecki. Convolutions of chapter 17.3. Universitext. Springer, London, 2nd Erlang and of Pascal distributions with applications to edition, 2014. doi:10.1007/978-1-4471-5361-0. reliability. Demonstratio Math., 36(1):231–238, 2003. [224] A. Jensen. Markoff chains as an aid in the study of Markoff [245] M. Akkouchi. On the convolution of exponential processes. Scand. Actuar. J., 1953(Supp. 1):87–91, distributions. J. Chungcheong Math. Soc., 21(4), 2008. 1953. doi:10.1080/03461238.1953.10419459. [246] H. E. Daniels. Saddlepoint approximations in statistics. [225] A. Hellander. Efficient computation of transient solu- Ann. Math. Stat., 25(4):631–650, 1954. doi:10.1214/ tions of the chemical master equation based on uni- aoms/1177728652. Master equations and the theory of stochastic path integrals 73

[247] M. A. Gibson and J. Bruck. Efficient exact stochastic Phys., 121(21):10356, 2004. doi:10.1063/1.1810475. simulation of chemical systems with many species and [262] A. Chatterjee, D. G. Vlachos, and M. A. Katsoulakis. Bi- many channels. J. Phys. Chem. A, 104(9):1876–1889, nomial distribution based τ-leap accelerated stochastic 2000. doi:10.1021/jp993732q. simulation. J. Chem. Phys., 122(2):024112, 2005. doi: [248] A. Slepoy, A. P. Thompson, and S. J. Plimpton. 10.1063/1.1833357. A constant-time kinetic Monte Carlo algorithm for [263] Y. Cao, D. T. Gillespie, and L. R. Petzold. Avoiding simulation of large biochemical reaction networks. J. negative populations in explicit Poisson tau-leaping. J. Chem. Phys., 128(20):205101, 2008. doi:10.1063/1. Chem. Phys., 123(5):054104, 2005. doi:10.1063/1. 2919546. 1992473. [249] D. T. Gillespie, A. Hellander, and L. R. Petzold. Per- [264] Y. Cao, D. T. Gillespie, and L. R. Petzold. Efficient step spective: Stochastic algorithms for chemical kinetics. size selection for the tau-leaping simulation method. J. Chem. Phys., 138(17):170901, 2013. doi:10.1063/ J. Chem. Phys., 124(4):044109, 2006. doi:10.1063/ 1.4801941. 1.2159468. [250] M. Tomita, K. Hashimoto, K. Takahashi, T. S. Shimizu, [265] A. Auger, P. Chatelain, and P. Koumoutsakos. R-leaping: Y. Matsuzaki, F. Miyoshi, K. Saito, S. Tanida, Accelerating the stochastic simulation algorithm by K. Yugi, J. C. Venter, and C. A. Hutchison III. E- reaction leaps. J. Chem. Phys., 125(8):084103, 2006. CELL: software environment for whole-cell simulation. doi:10.1063/1.2218339. Bioinformatics, 15(1):72–84, 1999. URL: http://www. [266] Y. Cao, D. T. Gillespie, and L. R. Petzold. Adaptive e-cell.org/, doi:10.1093/bioinformatics/15.1.72. explicit-implicit tau-leaping method with automatic tau [251] D. Adalsteinsson, D. McMillen, and T. C. Elston. selection. J. Chem. Phys., 126(22):224101, 2007. doi: Biochemical network stochastic simulator (BioNetS): 10.1063/1.2745299. software for stochastic modeling of biochemical net- [267] D. F. Anderson. Incorporating postleap checks in tau- works. BMC Bioinform., 5(1):24, 2004. URL: http:// leaping. J. Chem. Phys., 128(5):054103, 2008. doi: www.utm.utoronto.ca/%7Emcmillen/Bionets/, doi:10. 10.1063/1.2218339. 1186/1471-2105-5-24. [268] T. Lu, D. Volfson, L Tsimring, and J. Hasty. Cellular [252] M. Ander, P. Beltrao, B. Di Ventura, J. Ferkinghoff- growth and division in the Gillespie algorithm. IEE Borg, M. Foglierini, A. Kaplan, C. Lemerle, I. Tomás- P. Syst. Biol., 1(1):121–128, 2004. doi:10.1049/sb: Oliveira, and L. Serrano. SmartCell, a framework 20045016. to simulate cellular processes that combines stochastic [269] D. F. Anderson. A modified next reaction method approximation with diffusion and localisation: analysis for simulating chemical systems with time depend- of simple networks. IEE P. Syst. Biol., 1(1):129–138, ent propensities and delays. J. Chem. Phys., 2004. URL: http://software.crg.es/smartcell/, 127(21):214107–1–214107–10, 2007. doi:10.1063/1. doi:10.1049/sb:20045017. 2799998. [253] J. Hattne, D. Fange, and J. Elf. Stochastic [270] E. Roberts, S. Be’er, C. Bohrer, R. Sharma, and M. Assaf. reaction-diffusion simulation with MesoRD. Dynamics of simple gene-network motifs subject to Bioinformatics, 21(12):2923–2924, 2005. URL: extrinsic fluctuations. Phys. Rev. E, 92(6):062717, http://mesord.sourceforge.net/, doi:10.1093/ 2015. doi:10.1103/PhysRevE.92.062717. bioinformatics/bti431. [271] W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. [254] K. R. Sanft, S. Wu, M. Roh, J. Fu, R. K. Lim, Flannery. Numerical Recipes - The Art of Scientific and L. R. Petzold. StochKit2: software for dis- Computing, chapter 17.5, pages 931–942. Cambridge crete stochastic simulation of biochemical systems University Press, Cambridge, 3rd edition, 2007. with events. Bioinformatics, 27(17):2457–2458, 2011. [272] J. E. Moyal. Stochastic processes and statistical physics. URL: http://stochkit.sourceforge.net/, doi:10. J. R. Stat. Soc. B, 11(2):150–210, 1949. 1093/bioinformatics/btr401. [273] N. G. van Kampen. A power series expansion of the [255] I. Hepburn, W. Chen, S. Wils, and E. De Schut- master equation. Can. J. Phys., 39(4):551–567, 1961. ter. STEPS: efficient simulation of stochastic re- doi:10.1139/p61-056. action–diffusion models in realistic morphologies. [274] D. Schnoerr, G. Sanguinetti, and R. Grima. Approxim- BMC Syst. Biol., 6:36, 2012. URL: http://steps. ation and inference methods for stochastic biochemical sourceforge.net/, doi:10.1186/1752-0509-6-36. kinetics - a tutorial review. J. Phys. A: Math. Theor., [256] B. Drawert, S. Engblom, and A. Hellander. URDME: 50(9):093001, 2017. doi:10.1088/1751-8121/aa54d9. a modular framework for stochastic simulation of [275] L. A. Goodman. Population growth of the sexes. reaction-transport processes in complex geometries. Biometrics, 9(2):212–225, 1953. doi:10.2307/3001852. BMC Syst. Biol., 6:76, 2012. URL: http://urdme. [276] P. Whittle. On the use of the normal approximation in github.io/urdme/, doi:10.1186/1752-0509-6-76. the treatment of stochastic processes. J. R. Stat. Soc. [257] L. Petzold, C. Krintz, and P. Lötstedt. StochSS: B, 19(2):268–281, 1957. Stochastic Simulation Service. URL: http://www. [277] M. J. Keeling. Multiplicative moments and measures of stochss.org/. persistence in ecology. J. Theor. Biol., 205(2):269–281, [258] D. T. Gillespie. Approximate accelerated stochastic simu- 2000. doi:10.1006/jtbi.2000.2066. lation of chemically reacting systems. J. Chem. Phys., [278] I. Nåsell. An extension of the moment closure method. 115(4):1716–1733, 2001. doi:10.1063/1.1378322. Theor. Popul. Biol., 64(2):233–239, 2003. doi:10. [259] D. T. Gillespie and L. R. Petzold. Improved leap- 1016/S0040-5809(03)00074-1. size selection for accelerated stochastic simulation. J. [279] C. H. Lee, K.-H. Kim, and P. Kim. A moment closure Chem. Phys., 119(16):8229, 2003. doi:10.1063/1. method for stochastic reaction networks. J. Chem. 1613254. Phys., 130(13):134107, 2009. doi:10.1063/1.3103264. [260] M. Rathinam, L. R. Petzold, Y. Cao, and D. T. Gillespie. [280] P. Smadbeck and Y. N. Kaznessis. A closure scheme Stiffness in stochastic chemically reacting systems: for chemical master equations. Proc. Natl. Acad. Sci. The implicit tau-leaping method. J. Chem. Phys., USA, 110(35):14261–14265, 2013. doi:10.1073/pnas. 119(24):12784, 2003. doi:10.1063/1.1627296. 1306481110. [261] T. Tian and K. Burrage. Binomial leap methods for [281] D. Schnoerr, G. Sanguinetti, and R. Grima. Validity con- simulating stochastic chemical kinetics. J. Chem. ditions for moment closure approximations in stochastic Master equations and the theory of stochastic path integrals 74

chemical kinetics. J. Chem. Phys., 141(8):084103, 2014. [301] U. C. Täuber, M. J. Howard, and H. Hinrichsen. doi:10.1063/1.4892838. Multicritical behavior in coupled directed percolation [282] E. Lakatos, A. Ale, P. D. W. Kirk, and M. P. H. Stumpf. processes. Phys. Rev. Lett., 80(10):2165–2168, 1998. Multivariate moment closure techniques for stochastic doi:10.1103/PhysRevLett.80.2165. kinetic models. J. Chem. Phys., 143(9):094107, 2015. [302] H. Hinrichsen and M. Howard. A model for anomalous doi:10.1063/1.4929837. directed percolation. Eur. Phys. J. B, 7(4):635–643, [283] D. Schnoerr, G. Sanguinetti, and R. Grima. Com- 1999. doi:10.1007/s100510050656. parison of different moment-closure approximations [303] Y. Y. Goldschmidt, H. Hinrichsen, M. Howard, and for stochastic chemical kinetics. J. Chem. Phys., U. C. Täuber. Nonequilibrium critical behavior in uni- 143(18):185101, 2015. doi:10.1063/1.4934990. directionally coupled stochastic processes. Phys. Rev. [284] F. Di Patti, S. Azaele, J. R. Banavar, and A. Maritan. E, 59(6):6381–6408, 1999. doi:10.1103/PhysRevE.59. System size expansion for systems with an absorbing 6381. state. Phys. Rev. E, 83(1):010102(R), 2011. doi: [304] M. Hnatich and J. Honkonen. Velocity-fluctuation- 10.1103/PhysRevE.83.010102. induced anomalous kinetics of the A + A → ∅ reaction. [285] P. A. M. Dirac. The Lagrangian in quantum mechanics. Phys. Rev. E, 61(4):3904–3911, 2000. doi:10.1103/ Phys. Z. Sowjetunion, 3:64–72, 1933. PhysRevE.61.3904. [286] R. P. Feynman. Space-time approach to non-relativistic [305] D. C. Vernon. Long range hops and the pair annihilation quantum mechanics. Rev. Mod. Phys., 20(2):367–387, reaction A + A → ∅: renormalization group and 1948. doi:10.1103/RevModPhys.20.367. simulation. Phys. Rev. E, 68(4):041103, 2003. doi: [287] R. P. Feynman and A. R. Hibbs. Quantum Mechanics 10.1103/PhysRevE.68.041103. and Path Integrals: Emended Edition. Dover Books on [306] A. A. Winkler and E. Frey. Validity of the law of Physics. Dover Publications, New York, NY, 2010. mass action in three-dimensional coagulation processes. [288] P. A. M. Dirac. The Principles of Quantum Mechanics, Phys. Rev. Lett., 108(10):108301, 2012. doi:10.1103/ chapter VI. Oxford University Press, Oxford, 4th PhysRevLett.108.108301. edition, 1958. [307] I. Homrighausen, A. A. Winkler, and E. Frey. Fluctuation [289] D. C. Mattis and M. L. Glasser. The uses of quantum field effects in the pair-annihilation process with Lévy theory in diffusion-limited reactions. Rev. Mod. Phys., dynamics. Phys. Rev. E, 88(1):012111, 2013. doi: 70(3):979–1001, 1998. doi:10.1103/RevModPhys.70. 10.1103/PhysRevE.88.012111. 979. [308] M. Droz and L. Sasvari. Renormalization-group approach [290] A. Altland and B. D. Simons. Condensed Matter Field to simple reaction-diffusion phenomena. Phys. Rev. E, Theory. Cambridge University Press, Cambridge, 2nd 48(4):R2343–R2346, 1993. doi:10.1103/PhysRevE.48. edition, 2010. doi:10.1017/CBO9780511789984. R2343. [291] J. C. Baez and J. Biamonte. Quantum Techniques for [309] B. P. Lee and J. Cardy. Scaling of reaction zones in the Stochastic Mechanics. arXiv:1209.3632, 2015. A + B → 0 diffusion-limited reaction. Phys. Rev. E, [292] M. E. Fisher. Renormalization group theory: Its basis 50(5):R3287–R3290, 1994. doi:10.1103/PhysRevE.50. and formulation in statistical physics. Rev. Mod. R3287. Phys., 70(2):653–681, 1998. doi:10.1103/RevModPhys. [310] M. J. Howard and G. T. Barkema. Shear flows and 70.653. segregation in the reaction A + B → ∅. Phys. Rev. [293] J. Cardy. Renormalisation group approach to reaction- E, 53(6):5949–5956, 1996. doi:10.1103/PhysRevE.53. diffusion problems. In J.-B. Zuber, editor, Mathemat- 5949. ical Beauty of Physics, volume 24 of Adv. Ser. Math. [311] Z. Konkoli, H. Johannesson, and B. P. Lee. Fluctuation Phys., pages 113–128, 1997. arXiv:cond-mat/9607163. effects in steric reaction-diffusion systems. Phys. Rev. [294] L. Peliti. Renormalisation of fluctuation effects in E, 59(4):R3787–R3790, 1999. doi:10.1103/PhysRevE. the A + A → A reaction. J. Phys. A: Math. Gen., 59.R3787. 19(6):L365–L367, 1986. doi:10.1088/0305-4470/19/ [312] R. Dickman and R. Vidigal. Path-integral representation 6/012. for a stochastic sandpile. J. Phys. A: Math. Gen., [295] M. Howard and J. Cardy. Fluctuation effects and 35(34):7269–7285, 2002. doi:10.1088/0305-4470/35/ multiscaling of the reaction-diffusion front for A + B → 34/303. ∅. J. Phys. A: Math. Gen., 28(13):3599–3621, 1995. [313] H. J. Hilhorst, O. Deloubrière, M. J. Washenberger, doi:10.1088/0305-4470/28/13/007. and U. C. Täuber. Segregation in diffusion-limited [296] M. Howard. Fluctuation kinetics in a multispecies multispecies pair annihilation. J. Phys. A: Math. Gen., reaction-diffusion system. J. Phys. A: Math. Gen., 37(28):7063–7093, 2004. doi:10.1088/0305-4470/37/ 29(13):3437–3460, 1996. doi:10.1088/0305-4470/29/ 28/001. 13/016. [314] H. K. Janssen, F. van Wijland, O. Deloubrière, and [297] K. Oerding. The A + B → ∅ annihilation reaction in a U. C. Täuber. Pair contact process with diffusion: quenched random velocity field. J. Phys. A: Math. Failure of master equation field theory. Phys. Rev. Gen., 29(2):7051–7065, 1996. doi:10.1088/0305-4470/ E, 70(5):056114, 2004. doi:10.1103/PhysRevE.70. 29/22/009. 056114. [298] P.-A. Rey and M. Droz. A renormalization group study [315] S. Whitelam, L. Berthier, and J. P. Garrahan. Renormal- of a class of reaction-diffusion models, with particles ization group study of a kinetically constrained model input. J. Phys. A: Math. Gen., 30(4):1101–1114, 1997. for strong glasses. Phys. Rev. E, 71(2):026128, 2005. doi:10.1088/0305-4470/30/4/013. doi:10.1103/PhysRevE.71.026128. [299] T. Sasamoto, S. Mori, and M. Wadati. Universal [316] M. Hnatič, J. Honkonen, and T. Lučivjanský. Two- properties of the mA + nB → ∅ diffusion-limited loop calculation of anomalous kinetics of the reaction reaction. Physica A, 247(1-4):357–378, 1997. doi: A + A → ∅ in randomly stirred fluid. Eur. Phys. J. B, 10.1016/s0378-4371(97)00387-7. 86(5):214, 2013. doi:10.1140/epjb/e2013-30982-9. [300] F. van Wijland, K. Oerding, and H. J. Hilhorst. [317] L. Canet, B. Delamotte, O. Deloubrière, and Wilson renormalization of a reaction-diffusion process. N. Wschebor. Nonperturbative renormalization- Physica A, 251(1-2):179–201, 1998. doi:10.1016/ group study of reaction-diffusion processes. S0378-4371(97)00603-1. Phys. Rev. Lett., 92(19):195703, 2004. doi: Master equations and the theory of stochastic path integrals 75

10.1103/PhysRevLett.92.195703. [335] F. W. J. Olver, D. W. Lozier, R. F. Boisvert, and [318] L. Canet, H. Chaté, and B. Delamotte. Quantitative C. W. Clark, editors. NIST Handbook of Mathematical phase diagrams of branching and annihilating random Functions. Cambridge University Press, New York, walks. Phys. Rev. Lett., 92(25):255703, 2004. doi: NY, 2010. Print companion to http://dlmf.nist. 10.1103/PhysRevLett.92.255703. gov/. [319] L. Canet, H. Chaté, B. Delamotte, I. Dornic, and [336] K. J. Wiese. Coherent-state path integral versus coarse- M. A. Muñoz. Nonperturbative fixed point in a grained effective stochastic equation of motion: from nonequilibrium phase transition. Phys. Rev. Lett., reaction diffusion to stochastic sandpiles. Phys. Rev. 95(10):100601, 2005. doi:10.1103/PhysRevLett.95. E, 93(4):042117, 2016. doi:10.1103/PhysRevE.93. 100601. 042117. [320] V. Elgart and A. Kamenev. Classification of phase [337] R. B. Stinchcombe, M. D. Grynberg, and M. Barma. transitions in reaction-diffusion models. Phys. Rev. Diffusive dynamics of deposition-evaporation systems, E, 74(4):041101, 2006. doi:10.1103/PhysRevE.74. jamming, and broken symmetries in related quantum- 041101. spin models. Phys. Rev. E, 47(6):4018–4036, 1993. [321] M. A. Buice and J. D. Cowan. Field-theoretic approach doi:10.1103/PhysRevE.47.4018. to fluctuation effects in neural networks. Phys. Rev. [338] M. D. Grynberg, T. J. Newman, and R. B. Stinchcombe. E, 75(5):051919, 2007. doi:10.1103/PhysRevE.75. Exact-solutions for stochastic adsorption-desorption 051919. models and catalytic surface processes. Phys. Rev. E, [322] M. A. Buice and J. D. Cowan. Statistical mechanics of 50(2):957–971, 1994. doi:10.1103/PhysRevE.50.957. the neocortex. Prog. Biophys. Mol. Biol., 99(2–3):53– [339] S. Goldstein. On diffusion by discontinuous movements, 86, 2009. doi:10.1016/j.pbiomolbio.2009.07.003. and on the telegraph equation. Q. J. Mech. Appl. [323] M. A. Buice and C. C. Chow. Beyond mean field theory: Math., 4(2):129–156, 1951. doi:10.1093/qjmam/4.2. statistical field theory for neural networks. J. Stat. 129. Mech. Theor. Exp., 2013(03):P03003, 2013. doi:10. [340] M. Kac. A stochastic model related to the telegrapher’s 1088/1742-5468/2013/03/P03003. equation. Rocky Mt. J. Math., 4(3):497–510, 1974. [324] E. Bettelheim, O. Agam, and N. M. Shnerb. “Quantum Reprinting of an article published in 1956. doi:10. phase transitions” in classical nonequilibrium processes. 1216/RMJ-1974-4-3-497. Physica E, 9(3):600–608, 2001. doi:10.1016/S1386- [341] F. A. Berezin. The Method of Second Quantization, 9477(00)00268-X. chapter 1.3. Academic Press, Orlando, FL, 1966. [325] U. C. Täuber. Stochastic population oscillations in [342] M. Combescure and D. Robert. Coherent States spatial predator-prey models. J. Phys.: Conf. Ser., and Applications in Mathematical Physics, chapter 319:012019, 2011. doi:10.1088/1742-6596/319/1/ 11.2. Theoretical and Mathematical Physics. Springer, 012019. Dordrecht, 2012. doi:10.1007/978-94-007-0196-0. [326] B. Zhang and P. G. Wolynes. Stem cell differentiation [343] M. A. Aliev. On the description of the Glauber dynamics as a many-body problem. Proc. Natl. Acad. Sci. of one dimensional disordered Ising chain. Physica A, USA, 111(28):10185–10190, 2014. doi:10.1073/pnas. 277(3-4):261–273, 2000. doi:10.1016/S0378-4371(99) 1408561111. 00442-2. [327] W. Feller. On the theory of stochastic processes, with [344] M. Mobilia, R. K. P. Zia, and B. Schmittmann. Complete particular reference to applications. In Proc. [First] solution of the kinetics in a far-from-equilibrium Ising Berkeley Symp. on Math. Statist. and Prob., pages 403– chain. J. Phys. A: Math. Gen., 37(32):L407–L413, 432, Berkeley, CA, 1949. University of California Press. 2004. doi:10.1088/0305-4470/37/32/L03. [328] A. M. Walczak, A. Mugler, and C. H. Wiggins. Analytic [345] F. C. Alcaraz, M. Droz, M. Henkel, and V. Rittenberg. methods for modeling stochastic regulatory networks. Reaction-diffusion processes, critical dynamics, and In X. Liu and M. D. Betterton, editors, Computational quantum chains. Ann. Phys., 230(2):250–302, 1994. Modeling of Signaling Networks, volume 880 of Methods doi:10.1006/aphy.1994.1026. in Molecular Biology, pages 273–322. Humana Press, [346] G. Schütz and S. Sandow. Non-abelian symmetries of New York, NY, 2012. doi:10.1007/978-1-61779-833- stochastic processes: derivation of correlation functions 7_13. for random-vertex models and disordered-interacting- [329] J. G. Skellam. The frequency distribution of the difference particle systems. Phys. Rev. E, 49(4):2726–2741, 1994. between two Poisson variates belonging to different doi:10.1103/PhysRevE.49.2726. populations. J. R. Stat. Soc., 109(3):296, 1946. doi: [347] G. M. Schütz. Reaction-diffusion processes of hard-core 10.2307/2981372. particles. J. Stat. Phys., 79(1-2):243–264, 1995. doi: [330] H. Takayasu and A. Yu. Tretyakov. Extinction, survival, 10.1007/BF02179389. and dynamical phase transition of branching annihilat- [348] M. Henkel, E. Orlandini, and J. Santos. Reaction-diffusion ing random walk. Phys. Rev. Lett., 68(20):3060–3063, processes from equivalent integrable quantum chains. 1992. doi:10.1103/PhysRevLett.68.3060. Ann. Phys., 259(2):163–231, 1997. doi:10.1006/aphy. [331] I. Jensen. Critical behavior of branching annihilating 1997.5712. random walks with an odd number of offsprings. Phys. [349] E. Carlon, M. Henkel, and U. Schollwöck. Density matrix Rev. E, 47(1):R1–R4, 1993. doi:10.1103/PhysRevE. renormalization group and reaction-diffusion processes. 47.R1. Eur. Phys. J. B, 12(1):99–114, 1999. doi:10.1007/ [332] L. E. Ballentine. Quantum Mechanics - A Modern s100510050983. Development, chapter 6.2. World Scientific Publishing [350] E. Carlon, M. Henkel, and U. Schollwöck. Critical Company, Singapore, 1998. properties of the reaction-diffusion model 2A → 3A, [333] M. E. Peskin and D. V. Schroeder. An Introduction To 2A → 0. Phys. Rev. E, 63(3):036101, 2001. doi: Quantum Field Theory. Westview Press, Boulder, CO, 10.1103/PhysRevE.63.036101. 1995. [351] S.-C. Park, D. Kim, and J.-M. Park. Path-integral [334] J. Ohkubo. One-parameter extension of the Doi–Peliti formulation of stochastic processes for the exclusive formalism and its relation with orthogonal polynomials. particle systems. Phys. Rev. E, 62(6):7642–7645, 2000. Phys. Rev. E, 86(4):042102, 2012. doi:10.1103/ doi:10.1103/PhysRevE.62.7642. PhysRevE.86.042102. [352] S.-C. Park and J.-M. Park. Generating function, path Master equations and the theory of stochastic path integrals 76

integral representation, and equivalence for stochastic [372] Y. Lan, P. G. Wolynes, and G. A. Papoian. A variational exclusive particle systems. Phys. Rev. E, 71(2):026113, approach to the stochastic aspects of cellular signal 2005. doi:10.1103/PhysRevE.71.026113. transduction. J. Chem. Phys., 125(12):124106, 2006. [353] J. Tailleur, J. Kurchan, and V. Lecomte. Mapping doi:10.1063/1.2353835. out-of-equilibrium into equilibrium in one-dimensional [373] J. Ohkubo. Approximation scheme for master equations: transport models. J. Phys. A: Math. Theor., Variational approach to multivariate case. J. Chem. 41(50):505001, 2008. doi:10.1088/1751-8113/41/50/ Phys., 129(4):044108, 2008. doi:10.1063/1.2957462. 505001. [374] P. D. Drummond, T. G. Vaughan, and A. J. Drummond. [354] R. Kubo, M. Matsuo, and K. Kitahara. Fluctuation and Extinction times in autocatalytic systems. J. Phys. relaxation of macrovariables. J. Stat. Phys., 9(1):51–96, Chem. A, 114(39):10481–10491, 2010. doi:10.1021/ 1973. doi:10.1007/BF01016797. jp104471e. [355] Hu Gang. Stationary solution of master equations in [375] P. D. Drummond, C. W. Gardiner, and D. F. Walls. the large-system-size limit. Phys. Rev. A, 36(12):5782– Quasiprobability methods for nonlinear chemical and 5790, 1987. doi:10.1103/PhysRevA.36.5782. optical systems. Phys. Rev. A, 24(2):914–926, 1981. [356] M. I. Dykman, E. Mori, J. Ross, and P. M. Hunt. Large doi:10.1103/PhysRevA.24.914. fluctuations and optimal paths in chemical kinetics. J. [376] D. Elderfield. Exact macroscopic dynamics in non- Chem. Phys., 100(8):5735–5750, 1994. doi:10.1063/1. equilibrium chemical-systems. J. Phys. A: Math. Gen., 467139. 18(11):2049–2059, 1985. doi:10.1088/0305-4470/18/ [357] D. A. Kessler and N. M. Shnerb. Extinction rates for 11/026. fluctuation-induced metastabilities: a real-space WKB [377] M. Droz and A. McKane. Equivalence between Poisson approach. J. Stat. Phys., 127(5):861–886, 2007. doi: representation and Fock space formalism for birth- 10.1007/s10955-007-9312-2. death processes. J. Phys. A: Math. Gen., 27(13):L467– [358] B. Meerson and P. V. Sasorov. Noise-driven unlimited L474, 1994. doi:10.1088/0305-4470/27/13/002. population growth. Phys. Rev. E, 78(6):060103(R), [378] P. D. Drummond. Gauge poisson representations for 2008. doi:10.1103/PhysRevE.78.060103. birth/death master equations. Eur. Phys. J. B, [359] C. Escudero and A. Kamenev. Switching rates of 38(4):617–634, 2004. doi:10.1140/epjb/e2004-00157- multistep reactions. Phys. Rev. E, 79(4):041149, 2009. 2. doi:10.1103/PhysRevE.79.041149. [379] O. Deloubrière, L. Frachebourg, H. J. Hilhorst, and [360] M. Assaf and B. Meerson. Extinction of metastable K. Kitahara. Imaginary noise and parity conservation stochastic populations. Phys. Rev. E, 81(2):021116, in the reaction A + A 0. Physica A, 308(1-4):135– 2010. doi:10.1103/PhysRevE.81.021116. 147, 2002. doi:10.1016/S0378-4371(02)00548-4. [361] O. Ovaskainen and B. Meerson. Stochastic models of [380] K. G. Petrosyan and C.-K. Hu. Nonequilibrium Lyapunov population extinction. Trends Ecol. Evol., 25(11):643– function and a fluctuation relation for stochastic 652, 2010. doi:10.1016/j.tree.2010.07.009. systems: Poisson-representation approach. Phys. Rev. [362] B. Meerson and P. V. Sasorov. Extinction rates of estab- E, 89(4):042132, 2014. doi:10.1103/PhysRevE.89. lished spatial populations. Phys. Rev. E, 83(1):011129, 042132. 2011. doi:10.1103/PhysRevE.83.011129. [381] J. Burnett and I. J. Ford. Coagulation kinetics [363] N. R. Smith and B. Meerson. Extinction of oscillating beyond mean field theory using an optimised Poisson populations. Phys. Rev. E, 93(3):032109, 2016. doi: representation. J. Chem. Phys., 142(19):194112, 2015. 10.1103/PhysRevE.93.032109. doi:10.1063/1.4921350. [364] M. Assaf and B. Meerson. WKB theory of large deviations [382] M. S. Swanson. Path Integrals and Quantum Processes, in stochastic populations. arXiv:1612.01470, 2016. chapter 5.1. Academic Press, San Diego, CA, 1992. [365] J. W. Turner and M. Malek-Mansour. On the absorbing [383] R. F. Pawula. Generalizations and extensions of the zero boundary problem in birth and death processes. Fokker-Planck-Kolmogorov equations. IEEE Trans. Physica A, 93(3-4):517–525, 1978. doi:10.1016/0378- Inform. Theory, 13(1):33–41, 1967. doi:10.1109/TIT. 4371(78)90172-3. 1967.1053955. [366] M. Assaf and B. Meerson. Spectral theory of metastability [384] R. F. Pawula. Approximation of the linear Boltzmann and extinction in a branching-annihilation reaction. equation by the Fokker-Planck equation. Phys. Rev., Phys. Rev. E, 75(3):031122, 2007. doi:10.1103/ 162(1):186–188, 1967. doi:10.1103/PhysRev.162.186. PhysRevE.75.031122. [385] P. Thomas, C. Fleck, R. Grima, and N. Popović. System [367] M. Assaf, B. Meerson, and P. V. Sasorov. Large fluctu- size expansion using Feynman rules and diagrams. J. ations in stochastic population dynamics: momentum- Phys. A: Math. Theor., 47(45):455007, 2014. doi: space calculations. J. Stat. Mech. Theor. Exp., 10.1088/1751-8113/47/45/455007. 2010(07):P07018, 2010. doi:10.1088/1742-5468/ [386] A. Andreanov, G. Biroli, J.-P. Bouchaud, and A. Lefèvre. 2010/07/P07018. Field theories and exact stochastic equations for inter- [368] L. C. Evans. Partial Differential Equations: Second Edi- acting particle systems. Phys. Rev. E, 74(3):030101(R), tion, volume 19 of Graduate Studies in Mathematics. 2006. doi:10.1103/PhysRevE.74.030101. American Mathematical Society, 2 edition, 2010. [387] A. Lefèvre and G. Biroli. Dynamics of interacting particle [369] I. B. Schwartz, E. Forgoston, S. Bianco, and L. B. Shaw. systems: stochastic process and field theory. J. Stat. Converging towards the optimal path to extinction. Mech. Theor. Exp., 2007(07):P07024, 2007. doi:10. J. R. Soc. Interface, 8(65):1699–1707, 2011. doi:10. 1088/1742-5468/2007/07/P07024. 1098/rsif.2011.0159. [388] K. Itakura, J. Ohkubo, and S.-i. Sasa. Two Langevin [370] M. Assaf, A. Kamenev, and B. Meerson. Population equations in the Doi-Peliti formalism. J. Phys. A: extinction risk in the aftermath of a catastrophic Math. Theor., 43(12):125001, 2010. doi:10.1088/ event. Phys. Rev. E, 79(1):011127, 2009. doi:10.1103/ 1751-8113/43/12/125001. PhysRevE.79.011127. [389] H. K. Janssen, U. C. Täuber, and E. Frey. Exact results [371] B. Meerson and O. Ovaskainen. Immigration-extinction for the Kardar-Parisi-Zhang equation with spatially dynamics of stochastic populations. Phys. Rev. correlated noise. Eur. Phys. J. B, 9(3):491–511, 1999. E, 88(1):012124, 2013. doi:10.1103/PhysRevE.88. doi:10.1007/s100510050790. 012124. [390] E. Frey, U. C. Täuber, and H. K. Janssen. Scaling Master equations and the theory of stochastic path integrals 77

regimes and critical dimensions in the Kardar-Parisi- model. Z. Phys. B, 47(4):365–374, 1982. doi:10.1007/ Zhang problem. Europhys. Lett., 47(1):14–20, 1999. BF01313803. doi:10.1209/epl/i1999-00343-4. [410] L. Canet, H. Chaté, and B. Delamotte. General framework [391] R. L. Stratonovich. On a method of calculating quantum of the non-perturbative renormalization group for non- distribution functions. Sov. Phys. Dokl., 2:416, 1957. equilibrium steady states. J. Phys. A: Math. Theor., [392] J. Hubbard. Calculation of partition functions. Phys. Rev. 44(49):495001, 2011. doi:10.1088/1751-8113/44/49/ Lett., 3(2):77–78, 1959. doi:10.1103/PhysRevLett.3. 495001. 77. [411] J. Zinn-Justin. Phase Transitions and Renormaliza- [393] W. Horsthemke and A. Bach. Onsager-Machlup Function tion Group. Oxford Graduate Texts. Oxford Univer- for one dimensional nonlinear diffusion processes. sity Press, Oxford, 2007. doi:10.1093/acprof:oso/ Z. Phys. B, 22(2):189–192, 1975. doi:10.1007/ 9780199227198.001.0001. BF01322364. [412] R. Dickman and R. Vidigal. Path integrals and [394] D. Dürr and A. Bach. The Onsager-Machlup function as perturbation theory for stochastic processes. Braz. Lagrangian for the most probable path of a diffusion J. Phys., 33(1):73–93, 2003. doi:10.1590/S0103- process. Commun. Math. Phys, 60(2):153–170, 1978. 97332003000100005. doi:10.1007/BF01609446. [413] G. U. Yule. A mathematical theory of evolution, based [395] H. Ito. Probabilistic construction of Lagrangean of on the conclusions of Dr. J. C. Willis, F.R.S. Phil. diffusion process and its application. Prog. Theor. Trans. R. Soc. B, 213(402-410):21–87, 1925. doi:10. Phys., 59(3):725–741, 1978. doi:10.1143/PTP.59.725. 1098/rstb.1925.0002. [396] Y. Takahashi and S. Watanabe. The probability [414] M. H. DeGroot and M. J. Schervish. Probability and functionals (Onsager-Machlup functions) of diffusion Statistics, chapter 5.5. Pearson, Boston, 4th edition, processes. In D. Williams, editor, Stochastic Integrals, 2012. volume 851 of Lecture Notes in Mathematics, pages [415] C. Wissel. Manifolds of equivalent path integral solutions 433–463. Springer, Berlin, Heidelberg, 1981. doi:10. of the fokker-planck equation. Z. Phys. B, 35(2):185– 1007/BFb0088735. 191, 1979. doi:10.1007/BF01321245. [397] J. Zinn-Justin. Quantum Field Theory and Critical [416] B. P. Lee. Critical Behavior in Non-Equilibrium Systems. Phenomena, chapter 4.6. Number 113 in International PhD thesis, University of California, Santa Barbara, Series of Monographs on Physics. Oxford University 1994. URL: http://www.eg.bucknell.edu/~bvollmay/ Press, Oxford, 4th edition, 2002. leethesis.pdf. [398] Y. Tang, R. Yuan, and P. Ao. Summing over trajectories of [417] T. Mansour and M. Schork. Commutation Relations, stochastic dynamics with multiplicative noise. J. Chem. Normal Ordering, and Stirling Numbers, page 410. Phys., 141(4):044125, 2014. doi:10.1063/1.4890968. Discrete Mathematics and Its Applications. Chapman [399] M. Chaichian and A. Demichev. Path Integrals in and Hall/CRC, Boca Raton, FL, 2015. Physics - Volume I: Stochastic Processes and Quantum [418] H. F. Trotter. On the product of semi-groups of operators. Mechanics, volume 1 of Series in Mathematical and Proc. Amer. Math. Soc., 10(4):545–551, 1959. doi: Computational Physics, chapter 1. IOP Publishing, 10.1090/S0002-9939-1959-0108732-6. Bristol, 2001. [419] J. Touchard. Sur les cycles des substitutions. Acta Math., [400] M. J. Howard and U. C. Täuber. ‘Real’ versus ‘imaginary’ 70(1):243–297, 1939. doi:10.1007/BF02547349. noise in diffusion-limited reactions. J. Phys. A: Math. [420] G. Peccati and M. S. Taqqu. Wiener Chaos: Moments, Gen., 30(22):7721–7731, 1997. doi:10.1088/0305- Cumulants and Diagrams - A survey with computer 4470/30/22/011. implementation, volume 1 of Bocconi & Springer [401] D. Hochberg, M.-P. Zorzano, and F. Morán. Complex Series, chapter 3.3. Springer, Milan, 2011. doi:10. noise in diffusion-limited reactions of replicating and 1007/978-88-470-1679-8. competing species. Phys. Rev. E, 73(6):066109, 2006. [421] K. Kang and S. Redner. Fluctuation effects in Smoluchow- doi:10.1103/PhysRevE.73.066109. ski reaction-kinetics. Phys. Rev. A, 30(5):2833–2836, [402] D. C. Torney and H. M. McConnell. Diffusion- 1984. doi:10.1103/PhysRevA.30.2833. limited reactions in one dimension. J. Phys. Chem., [422] F. Benitez and N. Wschebor. Branching-rate expansion 87(11):1941–1951, 1983. doi:10.1021/j100234a023. around annihilating random walks. Phys. Rev. [403] S. Redner. A Guide to First-Passage Processes. E, 86(1):010104, 2012. doi:10.1103/PhysRevE.86. Cambridge University Press, Cambridge, 2001. 010104. [404] I. Jensen. Critical exponents for branching annihilating [423] F. Benitez and N. Wschebor. Branching and annihilating random walks with an even number of offspring. random walks: Exact results at low branching rate. Phys. Rev. E, 50(5):3623–3633, 1994. doi:10.1103/ Phys. Rev. E, 87(5):052132, 2013. doi:10.1103/ PhysRevE.50.3623. PhysRevE.87.052132. [405] M. Moshe. Recent developments in Reggeon field theory. [424] P.-A. Rey and J. Cardy. Asymptotic form of the approach Phys. Rep., 37(3):255–345, 1978. doi:10.1016/0370- to equilibrium in reversible recombination reactions. J. 1573(78)90098-4. Phys. A: Math. Gen., 32(9):1585–1603, 1999. doi: [406] E. Frey, U. C. Täuber, and F. Schwabl. Crossover 10.1088/0305-4470/32/9/008. from self-similar to self-affine structures in percolation. [425] Z. Konkoli and H. Johannesson. Two-species reaction- Europhys. Lett., 26(6):413–418, 1994. doi:10.1209/ diffusion system with equal diffusion constants: An- 0295-5075/26/6/003. omalous density decay at large times. Phys. Rev. [407] E. Frey, U. C. Täuber, and F. Schwabl. Crossover E, 62(3):3276–3280, 2000. doi:10.1103/PhysRevE.62. from isotropic to directed percolation. Phys. Rev. 3276. E, 49(6):5058–5072, 1994. doi:10.1103/PhysRevE.49. [426] N. Bleistein and R. A. Handelsman. Asymptotic Expan- 5058. sions of Integrals, chapter 6.1. Dover Publications, New [408] H. K. Janssen. On the nonequilibrium phase transition in York, NY, 1986. reaction-diffusion systems with an absorbing stationary [427] A. Mugler, A. M. Walczak, and C. H. Wiggins. Spectral state. Z. Phys. B, 42(2):151–154, 1981. doi:10.1007/ solutions to stochastic models of gene expression with BF01319549. bursts and regulation. Phys. Rev. E, 80(4):041921, [409] P. Grassberger. On phase transitions in Schlögl’s second 2009. doi:10.1103/PhysRevE.80.041921. Master equations and the theory of stochastic path integrals 78

[428] J. Ohkubo. Duality in interacting particle systems and boson representation. J. Stat. Phys., 139(3):454–465, 2010. doi:10.1007/s10955-009-9910-2. [429] J. Ohkubo. Solving partial differential equation via stochastic process. In C. S. Calude, M. Hagiya, K. Morita, G. Rozenberg, and J. Timmis, editors, Unconventional Computation, volume 6079, pages 105– 114. Springer, Berlin, Heidelberg, 2010. doi:10.1007/ 978-3-642-13523-1_13. [430] J. Ohkubo. Extended duality relations between birth- death processes and partial differential equations. J. Phys. A: Math. Theor., 46(37):375004, 2013. doi: 10.1088/1751-8113/46/37/375004. [431] J. Ohkubo. Duality-based calculations for transition prob- abilities in birth-death processes. arXiv:1509.07193, 2015. [432] F. Benitez, C. Duclut, H. Chaté, B. Delamotte, I. Dornic, and M. A. Muñoz. Langevin equations for reaction- diffusion processes. Phys. Rev. Lett., 117(10):100601, 2016. doi:10.1103/PhysRevLett.117.100601.