The Master Equation

Johanne Hizanidis

November 2002

Contents

1 Stochastic Processes 1 1.1 Why and how do stochastic processes enter into physics? . . . 1 1.2 Brownian Motion: a stochastic process ...... 2

2 Markov processes 2 2.1 The conditional probability ...... 2 2.2 Markov property ...... 2

3 The Chapman-Kolmogorov (C-K) equation 3 3.1 The C-K equation for stationary and homogeneous processes . 3

4 The Master equation 4 4.1 Derivation of the Master equation from the C-K equation . . . 4 4.2 Detailed balance ...... 5

5 The mean-ﬁeld equation 5

6 One-step processes: examples 7 6.1 The Poisson process ...... 8 6.2 The decay process ...... 8 6.3 A Chemical reaction ...... 9

A Appendix 11

1 STOCHASTIC PROCESSES

1 Stochastic Processes

A stochastic process is the time evolution of a stochastic variable. So if Y is the stochastic variable, the stochastic process is Y (t). A stochastic variable is deﬁned by specifying the set of possible values called ’range’ or ’set of states’ and the probability distribution over this set. The set can be discrete (e.g. number of molecules of a component in a reacting mixture),continuous (e.g. the velocity of a Brownian particle) or multidimensional. In the latter case, the stochastic variable is a vector (e.g. the three velocity components of a Brownian particle). The ﬁgure below helps to get a more intuitive idea of a (discrete) stochastic

process. At successive times, the most probable values of Y have been drawn as heavy dots. We may select a most probable trajectory from such a picture. Nothing excludes the possible existence of two or more trajectories of equal probability.

1.1 Why and how do stochastic processes enter into physics? Why: The most classical case where stochastic processes enter is Statistical Mechanics. Statistical Mechanics studies systems of large numbers of particles. Therefore, precise calculations can not be made. Instead, average values are measured, through probability consideration. How: The concept of statistical mechanics is replacing the system by an ensemble, i.e. a collection of microstates of the sytem. This ensemble serves to visualize the probability distribution over the set of microstates.

1 2 MARKOV PROCESSES

1.2 Brownian Motion: a stochastic process A very classical example of a stochastic process is the Brownian motion, i.e. the motion of a heavy colloidal particle immersed in a fluid made up of light particles. The stochastic variable Y in this case may be the position or velocity of the Brownian particle. If Y were deterministic, we could find an expression for the evolution in time of Y , such to give Y at each t. But Y is a stochastic variable. Each t doesn’t have a specific value for Y , but a probability for Y ’s value.

2 Markov processes

In order to understand the Markov property, the conditional probability should be deﬁned.

2.1 The conditional probability

The conditional probability P1|1(y2, t2|y1, t1) is defined through the following relation: P2(y1, t1; y2, t2) = P1|1(y2, t2|y1, t1)P1(y1, t1) (1) which means that the joint probability of finding y1 at t1 and y2 at t2, equals the probability of finding y1 at t1 times the probability of finding y2 at t2, given y1 at t1. The conditional probability must satisfy the following prop- erties

1. P1|1 ≥ 0

2. P1|1(y2, t2|y1, t1)dy2 = 1 R 3. P1(y2, t2) = P1|1(y2, t2|y1, t1)P1(y1, t1)dy1 R Property 3 follows from equation (1), when integrated over y1. Integrat- ing the left side of (1) over y1 gives: P2(y1, t1; y2, t2)dy1 = P1(y2, t2), i.e. P1(y2, t2) is the marginal probability distributionR of P2 with respect to y2.

2.2 Markov property A Markov process is deﬁned by the following relation, which is called the Markov property:

P1|n−1(yn, tn|yn−1, tn−1; ...; y1, t1) = P1|1(yn, tn|yn−1, tn−1) (2) t1 < t2 < ... < tn

2 3 THE CHAPMAN-KOLMOGOROV (C-K) EQUATION

The Markov property merely expresses that, for a Markov process, the probability of a transition at time tn−1 from a value yn−1 to a value yn at time tn, depends only on the value of y at the time tn−1, and not on the previous history of the system. P1|1 is called the transition probability. For a Markov process the joint probabilities for n ≥ 3 are all expressed in terms of P1 and P1|1. For n = 3:

P3(y1, t1; y2, t2; y3, t3) = P2(y1, t1; y2, t2)P1|2(y3, t3|y1, t1; y2, t2)

= P1(y1, t1)P1|1(y2, t2|y1, t1)P1|1(y3, t3|y2, t2) (3)

3 The Chapman-Kolmogorov (C-K) equation

Taking relation (3), integrating it over y2 and dividing both sides by P1 gives us the Chapman-Kolmogorov equation.

P (y , t |y , t ) = P (y , t |y , t )P (y , t |y , t )dy ∗ (4) 1|1 3 3 1 1 Z 1|1 3 3 2 2 1|1 2 2 1 1 2

This equation states that a process starting at t1 with value y1 reaches y3 at t3 via any one of the possible values y2 at the intermediate time t2.(Equations with an asterisk are analytically proved in the Appendix)

3.1 The C-K equation for stationary and homogeneous processes First let’s deﬁne these two types of Markov processes:

• Stationary : A process Y is stationary if it is not aﬀected by a shift in time, i.e. Y (t) and Y (t + ) have the same probability distribution.

• Homogeneous : A homogeneous process is a nonstationary Markov ∗ process and is deﬁned by the probability P (y1) ≡ P1|1(y1|y0). For such processes the transition probability depends only on the time interval τ = t2 − t1.

For both stationary and homogeneous processes, a special notation is used for the transition probability and the C-K equation:

P1|1(y2, t2|y1, t1) = Tτ (y2|y1) (5) T 0 (y |y ) = T 0 (y |y )T (y |y )dy (6) τ+τ 3 1 Z τ 3 2 τ 2 1 2

3 4 THE MASTER EQUATION

4 The Master equation

4.1 Derivation of the Master equation from the C-K equation

We take the transition probability Tτ 0 and expand it in a Taylor series over zero, considering small τ 0: 0 02 Tτ 0 (y3|y2) = δ(y2 − y3) + τ W (y3|y2) + O(τ ) (7) The delta function expresses that the probability to stay at the same state after time zero equals one, whereas the probability to change state after time zero equals zero. W (y3|y2) is the time derivative of the transition probability at τ 0 = 0. Thus it is called transition probability per unit time. This expression must satisfy the normalization property. Therefore the in- tegral over y3 must equal one. In order for that to happen, the above form must be corrected in the following sense: 0 0 02 Tτ 0 (y3|y2) = (1 − α0τ )δ(y2 − y3) + τ W (y3|y2) + O(τ ) (8) 0 where the delta function has been corrected by the coefficient 1 − α0τ which corresponds to the probability for no transition to have taken place at all. Therefore: α (y ) = W (y |y )dy (9) 0 2 Z 3 2 3 Putting (8) into (6), dividing by τ 0 and going to the limit τ 0 → 0 gives us the differential form of the Chapman-Kolmogorov equation which is called the Master equation: ∂ T (y |y ) = [W (y |y )T (y |y ) − W (y |y )T (y |y )]dy ∗ (10) ∂τ τ 3 1 Z 3 2 τ 2 1 2 3 τ 3 1 2 It is useful to cast the equation in a more intuitive form. Noting that all transition probabilities are for a given value y1 at t1, we may write, suppressing redundant indices: ∂P (y, t) = [W (y|y0)P (y0, t) − W (y0|y)P (y, t)]dy0 (11) ∂t Z And if the range of Y is a discrete set of states with labels n, the equation reduces to: dpn(t) = [W 0 p 0 (t) − W 0 p (t)] (12) dt nn n n n n Xn0 This form of the Master equation makes the physical meaning more clear: the Master equation is a gain-loss equation for the probability of each state n. The first term is the gain due to transitions from other states n0, and the second term is the loss due to transitions into other states n0.

4 5 THE MEAN-FIELD EQUATION

4.2 Detailed balance In the steady state condition, the left side of the Master equation equals zero. Therefore the steady state condition property has the form:

Wnn0 pn0 = ( Wn0n)pn (13) Xn0 Xn0

This relation expresses the obvious fact that in steady state, the sum of all transitions per unit time into any state n must be balanced by the sum of all transitions fom n into other states n0. Detailed balance is the stronger assertion that for each pair n, n0 separately the transitions must balance:

Wnn0 pn0 = Wn0npn (14)

The following figures illustrates the difference between the steady state condition property and detailed balance. The length of the arrows are proportional to the transition rate. In the first figure the anticlockwise transition proceeds at twice the rate of the clockwise transition. S (steady state), holds in the first figure but D (detailed balance) does not. In the second figure however,D holds, therefore S holds as well.

Note that detailed balance is a necessary but not suﬃcient condition for ther- modymanic equilibrium. In terms of quantum mechanics, detailed balance in thermodynamic equilibrium follows from microscopsic reversibility. However, it may also hold in some cases for nonequlibrium. Therefore, thermodyamic equilibrium is the strongest condition (suﬃcient but not necessary).

5 The mean-ﬁeld equation

Let Y be a physical quantity with a Markov character. The Master equation determines its probability distribution at all t > 0. In ordinary macroscopic physics, however, one ignores fluctuations and treats Y as if it were a non- stochastic, single-valued quantity hY i. The evolution of hY i is described by a deterministic differential equation for hY i called the mean-field equation. As the Master equation determines the entire probability distribution, it must be possible to derive from it the mean-field equation as an approximation for the case that fluctuations are negligible.

5 5 THE MEAN-FIELD EQUATION

First one has the exact identity:

d ∂P (y, t) hY i(t) = y dy dt Z ∂t = y[W (y|y0)P (y0, t) − W (y0|y)P (y, t)]dydy0 ZZ = (y0 − y)W (y0|y)P (y, t)dydy0 ZZ = a (y)P (y, t)dy = ha (y)i ∗ (15) Z 1 1

Deﬁning the jump moments aν(y) by :

a (y) = (y0 − y)νW (y0|y)dy0, ν = 0, 1, 2, ... (16) ν Z

Therefore, the mean-ﬁeld equation for the time evolution hY i(t) if a1(y) is a linear function of y then is same as: d hY i = a (hY i) (17) dt 1

If, however, a1(hY i) is not a linear function of hY i, one has a diﬀerent form for a1(hY i) by expanding it into a Taylor series over hY i.

1 00 ha (Y )i = a (hY i) + h(Y − hY i)2ia (hY i) + ... ∗ (18) 1 1 2 1 The evolution of hY i in the course of time is therefore not determined by hY i itself, but is inﬂuenced by the ﬂuctuations around this average (variance 2 σ ). So, for nonlinear a1(hY i) we need an equation for the variance as well. Similar to what was done for hY i:

d hY 2i = (y02 − y2)W (y0|y)P (y)dydy0 dt ZZ = [(y0 − y)2 + 2y(y0 − y)]W (y0|y)P (y)dydy0 ZZ

= ha2(Y )i + 2hY a1(Y i

This is identical with: dσ2 = ha (Y )i + 2h(Y − hY i)a (Y ))i ∗ (19) dt 2 1

6 6 ONE-STEP PROCESSES: EXAMPLES

Replacing hY i(t) with y(t) we get the two equations that constitute the ﬁrst approximation beyond the mean-ﬁeld equation:

1 00 y˙ = a (y) + σ2a (y) (20) 1 2 1 ˙2 2 0 (σ ) = a2(y) + 2σ a1(y) (21)

6 One-step processes: examples

There is a very important family of Markov processes, called generation- recombination processes or birth-death processes, which in short we call one- step processes. These processes are continuous in time, their range consists of integers n, and only jumps between adjacent states are permitted. The following ﬁgure helps to visualize a one-step process. The Master equation

for such processes is:

p˙n = rn+1pn+1 + gn−1pn−1 − (rn + gn)pn (22) where rn is the probability per unit time for a jump from state n to state n − 1 and gn is the probability per unit time for a jump from n to n + 1. One step processes occur at:

• generation and recombination processes of charge carriers

• single-electron tunneling

• surface growth of atoms

• birth and death of individuals

One step processes can be subdivided based on the coeﬃcients rn and gn into the following categories:

• Linear, if the coeﬃcients are linear functions of n

• Nonlinear, if the coeﬃcients are nonlinear functions of n

• Random walks, if the coeﬃcients are constant

7 6 ONE-STEP PROCESSES: EXAMPLES

6.1 The Poisson process One example of a random walk is the Poisson process. The Poisson process calculates the probability of n events occurring at time t > 0. This event could be for example the tunneling of electrons through a single barrier (shot noise). The Poisson process is deﬁned by:

rn = 0, gn = q, pn(0) = δn,0 (23) To visualize the poisson process look at the ﬁgure below:

No recombination exists and q is a constant. The Kronecker delta merely expresses that the probability for no events to have occurred after time zero equals one, and the probability of more than one event occurring after time zero equals zero. The Master equation for the poisson process has the form: ∗ p˙n = q(pn−1 − pn) (24) which has the following solution: (qt)n p (t) = e−qt ∗ (25) n n! 6.2 The decay process An example of a linear one-step process is the decay process. Consider a piece of radioactive material. The number of active nuclei surviving at time t > 0, N(t), is a non-stationary Markov process. We want to ﬁnd the evolution of a collection of nuclei. Let P (n, t) be the probability that there are n surviving nuclei at time t. If γ is the decay probability per unit time for one nucleus, the transition probability from n0 to n in a short time ∆t is (according to (8) ): 0 0 0 2 P1|1(n, t + ∆t|n , t) = δn0,n(1 − γn ∆t) + δn0−1,nγn ∆t + O(∆t) (26) The second term is the decay probability for n0 nuclei. In a very short time interval ∆t we can expect not more than one of the n0 nuclei to decay. Hence the δn0−1,n in front of the decay probability. The ﬁrst term corresponds to the case that no transition takes place. The tranistion probability can be written also as: 0 0 P1|1(n, t + ∆t|n , t) = 0, n > n (27) = n0γ∆t, n = n0 − 1 (28) = O(∆t)2, n < n0 − 1 (29)

8 6 ONE-STEP PROCESSES: EXAMPLES

Therefore the transition probability per unit time is

0 Wnn0 = γn δn,n0−1 (30)

Inserting this form in (12) gives the Master equation for the decay process:

p˙n(t) = γ(n + 1)pn+1(t) − γnpn(t) (31)

We now apply a device for linear Master equations which consists in multi- plying both sides of (31) by n and summing over n. Relabeling indices n in the sum pn+1 , so to obtain pn throughout we get:

∞ ∞ ∞ 2 np˙n = γ n(n + 1)pn+1 − γ n pn Xn=0 Xn=0 Xn=0 ∞ ∞ 2 = γ (n − 1)npn − γ n pn Xn=0 Xn=0 ∞

= −γ npn Xn=0

Thus we have found the mean-ﬁeld equation for N(t): d hN(t)i = −γhN(t)i (32) dt

Solving the above equation for hN(0)i = n0, gives:

−γt hN(t)i = n0e (33)

6.3 A Chemical reaction An example of a nonlinear one-step process is a chemical reaction such as: k X 2X k' k, k0 are reaction constants, deﬁned in chemical kinetics. For a state n, the generation probability per unit time gn has the value kn. For the reverse reaction, one can assemble pairs of molecules of X, in n(n − 1) ways (this is a result of the combinations of n over 2, times 2 , since each pair is counted 0 twice). Thus rn = k n(n − 1).

9 6 ONE-STEP PROCESSES: EXAMPLES

The ﬁgure below helps to visualize this type of a one-step process:

Each reaction has a gain and a loss term. Therefore, in the Master equation we will have two gain terms and two loss terms, thus four terms. For each reaction separately the gain and loss terms are as follows:

• X(n) → 2X(n+1): For this reaction, we are interested in the gain and loss of state X. Thus we call X state n, and 2X state n + 1. Gain of X means recombination. Therefore the gain term for this reaction is 0 k n(n + 1)pn+1. Loss of X means generation. The loss terms is then −knpn

• X(n−1) ← 2X(n): For this reaction, we are interested in the gain and loss of state 2X. Thus we call 2X state n, and X state n − 1. Gain of 2X means generation. Therefore the gain term for this reaction is k(n − 1)pn+1. Loss of X means recombination. The loss terms is then 0 −k n(n − 1)pn The Master equation becomes:

0 0 p˙n = k n(n + 1)pn+1 + k(n − 1)pn−1 − knpn − k n(n − 1)pn (34) Applying what we did in the linear case, we obtain: ∞ ∞ ∞ ∞ 0 2 0 2 2 np˙n = k n(n − 1)pn−1 + k n (n + 1)pn+1 − [k n (n − 1) + kn ]pn Xn=0 Xn=0 Xn=0 Xn=0 ∞ ∞ ∞ 0 2 0 2 2 = k n(n + 1)pn + k n(n − 1) pn − [k n (n − 1) + kn ]pn Xn=0 Xn=0 Xn=0 ∞ ∞ 0 = k npn − k n(n − 1)pn Xn=0 Xn=0 This is not a closed equation due to the nonlinearity. So one needs a differential equation for hn(n − 1)i and so on (infinte hierarchy). To simplify we make the Mean-Field Approximation: hn(n − 1)i = hnihni, so to obtain the mean-field equation for the chemical reaction: d hni = khni − k0hni2 (35) dt

10 A APPENDIX

A Appendix

1. Proof of (4):

P3(y1, t1; y2, t2; y3, t3) = P1(y1, t1)P1|1(y2, t2|y1, t1)P1|1(y3, t3|y2, t2) P (y , t ; y , t ; y , t )dy = P (y , t )P (y , t |y , t )P (y , t |y , t )dy Z 3 1 1 2 2 3 3 2 Z 1 1 1 1|1 2 2 1 1 1|1 3 3 2 2 2 P (y , t ; y , t ) = P (y , t ) P (y , t |y , t )P (y , t |y , t )dy 2 1 1 3 3 1 1 1 Z 1|1 2 2 1 1 1|1 3 3 2 2 2 P (y , t |y , t )P (y , t ) = P (y , t ) P (y , t |y , t )P (y , t |y , t )dy 1|1 3 3 1 1 1 1 1 1 1 1 Z 1|1 2 2 1 1 1|1 3 3 2 2 2 P (y , t |y , t ) = P (y , t |y , t )P (y , t |y , t )dy 1|1 3 3 1 1 Z 1|1 2 2 1 1 1|1 3 3 2 2 2

2. Proof of (10):

Putting (8) in (6) gives:

T 0 (y |y ) = T 0 (y |y )T (y |y )dy τ+τ 3 1 Z τ 3 2 τ 2 1 2 = [(1 − α τ 0)δ(y − y ) + τ 0W (y |y )]T (y |y )dy Z 0 2 3 3 2 τ 2 1 2 = T (y |y )δ(y − y )dy − τ 0 α (y )δ(y − y )T (y |y )dy Z τ 2 1 2 3 2 Z 0 2 2 3 τ 2 1 2 + τ 0 W (y |y )T (y |y )dy Z 3 2 τ 2 1 2 (9) = T (y |y ) − τ 0 W (y |y )T (y |y )dy + τ 0 W (y |y )T (y |y )dy τ 3 1 Z 2 3 τ 3 1 2 Z 3 2 τ 2 1 2

T 0 (y |y ) − T (y |y ) τ+τ 3 1 τ 3 1 = [W (y |y )T (y |y ) − W (y |y )T (y |y )]dy τ 0 Z 3 2 τ 2 1 2 3 τ 3 1 2 ∂T (y |y ) τ 3 1 = [W (y |y )T (y |y ) − W (y |y )T (y |y )]dy ∂τ Z 3 2 τ 2 1 2 3 τ 3 1 2

3. Proof of (15): d hY i(t) = y[W (y|y0)P (y0, t) − W (y0|y)P (y, t)]dydy0 dt ZZ = yW (y|y0)P (y0, t)dydy0 − yW (y0|y)P (y, t)dydy0 ZZ ZZ

| y ↔{zy0 } 11 A APPENDIX

= y0W (y0|y)P (y, t)dydy0 − yW (y0|y)P (y, t)dydy0 ZZ ZZ

= (y0 − y)W (y0|y)P (y, t)dydy0 ZZ

= a (y)P (y, t)dy = ha (y)i Z 1 1

4. Proof of (18):

Taylor series of a1(Y ) over hY i:

1 a (Y ) = a (hY i) + a0 (hY i)(Y − hY i) + a00(hY i)(Y − hY i)2 + ... 1 1 1 2 1

The mean value of the above is:

1 ha (Y )i = a (hY i) + a0 (hY i)h(Y − hY i)i + a00(hY i) h(Y − hY i)2i + ... 1 1 1 2 1 σ2

| {z1 2 00} Considering that h(Y − hY i)i = 0 : ha1(Y )i = a1(hY i) + 2 σ a1(hY i)

5. Proof of (19):

= ha2(Y )i + 2hY a1(Y )i

12 A APPENDIX

Now:

dσ2(t) dhY 2i dhY i2 = − dt dt dt dhY 2i dhY i = − 2 hY i dt dt = ha2(Y )i + 2hY a1(Y )i − 2hY iha1(Y )i

= ha2(Y )i + 2h(Y − hY i)a1(Y )i 0 2 = ha2(Y )i + 2h(Y − hY i)a1(hY i) + a1(hY i)(Y − hY i) + ...i 0 2 = ha2(Y )i + 2hY − hY iia1(hY i) + 2a1(hY i)h(Y − hY i) i 2 0 = a2(hY i) + 2σ a1(hY i)

6. Intuitive Proof of (24):

For the poisson process (where events are independent), the probability of n events happening depends only on the time interval. Therefore we can write that the probability for one event happening in the time interval ∆t is P (n = 1, ∆t) = P (1, ∆t) = q∆t. Based on that we get:

P (n, t + ∆t) = P (n, t; 0, ∆t) + P (n − 1, t; 1, ∆t) = P (n, t)(1 − q∆t) + P (n − 1, t)q∆t

P (n, t + ∆t) − P (n, t) = q∆t(Pn−1 − Pn)

Thus: p˙n = q(pn−1 − pn)

7. Proof of (25):

We introduce the so-called generating function:

∞ n G(s, t) = s pn(t) Xn=0

where s is independent of time. Diﬀerentiating this function and sub-

13 A APPENDIX

stituting (12) in it gives:

dG(s, t) ∞ = snp˙ (t) dt n Xn=0 ∞ n = s [q(pn−1 − pn)] Xn=0 ∞ ∞ n−1 n = qs s pn−1(t) − q s pn(t) Xn=1 Xn=0 dG(s, t) = q(s − 1)G(s, t) dt

G(s, t) = G(s, 0)eq(s−1)t

∞ n But, G(s, 0) = s pn(0) = 1 (only the n=0 term survives, because Xn=0

we choose the initial condition pn(0) = δn,0 ). So:

G(s, t) = eq(s−1)t = eqste−qt ∞ (qst)n = e−qt n! Xn=0 ∞ n = s pn(t) Xn=0

(qt)n p (t) = e−qt n n!

14 REFERENCES

References

[1] N. G. van Kampen, Stochastic Processes in Physics and Chemistry (North-Holland, Amsterdam, 1981).

[2] C. W. Gardiner, Handbook of Stochastic Methods (Springer, Berlin, Hei- delberg, New York, 1985).

[3] A. Papoulis, Probability, Random Variables and Stochastic Processes (McGraw-Hill, Tokyo, 1965).

[4] R. Balescu, Equilibrium and nonequilibrium Statistical Mechanics (Wiley- Interscience, New York, 1975).

[5] P. Landsberg, Thermodynamics (Interscience Pubishers, New York, Lon- don, 1961).