Approach to equilibrium and non-equilibrium stationary distributions of interacting many-particle systems that are coupled to different heat baths

Roland R. Netz∗ Fachbereich Physik, Freie Universit¨atBerlin, 14195 Berlin, Germany (Dated: September 17, 2020) A Hamiltonian-based model of many harmonically interacting massive particles that are subject to linear friction and coupled to heat baths at different temperatures is used to study the dynamic approach to equilibrium and non-equilibrium stationary states. An equilibrium system is here defined as a system whose stationary distribution equals the Boltzmann distribution, the relation of this definition to the conditions of detailed balance and vanishing probability current is discussed both for underdamped as well as for overdamped systems. Based on the exactly calculated dynamic approach to the stationary distribution, the functional that governs this approach, which is called the free entropy Sfree(t), is constructed. For the stationary distribution Sfree(t) becomes maximal and its time derivative, the free entropy production S˙free(t), is minimal and vanishes. Thus, Sfree(t) characterizes equilibrium as well as non-equilibrium stationary distributions by their extremal and stability properties. For an equilibrium system, i.e. if all heat baths have the same temperature, the free entropy equals the negative free energy divided by temperature and thus corresponds to the Massieu function which was previously introduced in an alternative formulation of statistical mechanics. Using a systematic perturbative scheme for calculating velocity and position correlations in the overdamped massless limit, explicit results for few particles are presented: For two particles localization in position and momentum space is demonstrated in the non-equilibrium stationary state, indicative of a tendency to phase separate. For three elastically interacting particles heat flows from a particle coupled to a cold reservoir to a particle coupled to a warm reservoir if the third reservoir is sufficiently hot. This does not constitute a violation of the second law of thermodynamics, but rather demonstrates that a particle in such a non-equilibrium system is not characterized by an effective temperature which equals the temperature of the heat bath it is coupled to. Active particle models can be described in the same general framework, which thereby allows to characterize their entropy production not only in the stationary state but also in the approach to the stationary non- equilibrium state. Finally, the connection to non-equilibrium thermodynamics formulations that include the reservoir entropy production is discussed.

I. INTRODUCTION for the breakdown of equilibrium statistical mechanics is the occurrence of phase transitions induced by NEQ Systems that even in their stationary state are not in driving. As a result of external forces acting on parti- equilibrium have in the last decades received renewed at- cle systems, laning and phase separations have been ob- tention since standard concepts of thermodynamics and served [20–25]. Internal NEQ effects can generate parti- statistical mechanics do not work and because of their cle propulsion that is sustained by chemical or biochemi- experimental relevance [1–14]. Some of the motivation cal means. NEQ effects can also be induced by stochastic for studying such systems comes from experimental ob- forces due to the coupling of particles to different heat servations of molecular processes in living systems, which or chemical energy reservoirs. Such internal NEQ effects are fundamentally non-equilibrium (NEQ)[15, 16]. Apart have also been shown to lead to clustering and phase from biological applications, experimental advances allow separation [26–35]. Experimentally, symmetry-breaking for the construction of NEQ systems either from biolog- transitions in suspensions of swimming bacteria and fila- ical components that are driven out of equilibrium by ment systems driven by motor proteins have indeed been ATP consumption or by using macromolecular and col- demonstrated [36, 37]. Also in more coarse-grained mod- loidal assemblies that can be driven out of equilibrium els, defined by migration rules, collective ordering can by chemical fuels or by applying external forces[17, 18]. be obtained and describes the response of pedestrians Two prominent features of NEQ systems are depar- to spatial confinement or the flocking and swarming of tures from the canonical Boltzmann distribution and vio- animals [12, 38]. NEQ transformations also occur for arXiv:2009.07544v1 [cond-mat.stat-mech] 16 Sep 2020 lation of the fluctuation-dissipation theorem, which were polymers that are driven by externally applied torques recently demonstrated to go hand in hand for a harmon- [39, 40]. ically coupled particle system [19], the fundamental hall- The equilibrium fluctuation-dissipation theorem mark of NEQ systems is the violation of the detailed (FDT) describes the dynamical response of an equi- balance condition [5, 8]. A particularly striking example librium system to a small perturbation. Since it is a key concept for systems close to equilibrium, and since any violation of the FDT clearly signals that a system is off equilibrium, the derivation of generalized ∗ [email protected] fluctuation-dissipation relations that also hold for NEQ 2 systems is central and has been discussed in the context temperatures allows us to prepare the system in a unique of laser [41], chaotic [42], glassy [43], driven colloidal [44], NEQ stationary state and to calculate all contributions to sheared [45–47] and active systems [48]. Generalized the entropy production as the system approaches the sta- NEQ fluctuation-dissipation relations were derived tionary state. Instead of considering trajectories in phase [41, 49–54] and compared with experimental data for space, we base our theory on the time-dependent distri- glasses [55], colloids [56, 57], bundles of biological bution. The key point of our model is that we can exactly filaments [58, 59], living cells [60] and biological gels calculate the time derivative of the distribution entropy that are driven by motor proteins in the presence of and from that construct, by comparison with the inde- ATP[19, 61]. pendently calculated relaxation of the distribution, the Significant theoretical progress has been made in the time-dependent functional that governs the approach to characterization of NEQ systems by generalized NEQ equilibrium as well to NEQ stationary distributions. In fluctuation-dissipation relations, as discussed above, and analogy to the relation between the energy and the free by fluctuation relations that give bounds and exact re- energy, this functional is called the free entropy, Sfree(t), lations involving trajectory ensembles [62–64]. The field since it contains the distribution entropy of the system of stochastic thermodynamics has linked the statistics of and also accounts for the interactions within the system trajectories to entropy-production contributions [14, 65]. and the coupling to the reservoirs. Using the free entropy A good fraction of contemporary work on NEQ systems is Sfree(t), the total entropy Stot(t), wich includes the en- concerned with analyzing the solutions of governing dy- tropy of the interacting particle system S(t) as well as namic equations either by simulation techniques or, for the reservoir entropy Sres(t), can be decomposed as simple systems, by analytical methods. In essence, and ˙◦ as acknowledged by most workers in the field, NEQ the- Stot(t) = S(t) + Sres(t) = Sfree(t) + tSres (1) ory is far from the predictive power and understanding ˙◦ furnished by e.g. the usage of thermodynamic potentials up to unimportant constants, where Sres denotes the en- in the context of equilibrium scenarios. For an isolated tropy production due to heat transfer with all reservoirs system the entropy is maximized, but this provides little in the unique stationary state. While Sres(t) is difficult to help for the type of NEQ problems one is typically in- calculate for the stochastic models used for the descrip- terested in, because they are not isolated. For a system tion of heat reservoirs and in fact increases boundlessly, ˙◦ coupled to a single heat reservoir and in the absence of ex- Sfree(t) and Sres can be explicitly calculated. Similar to ternal driving forces, which gives rise to the equilibrium the free energy of statistical mechanics, different observ- canonical ensemble, the free energy is minimized. The ables can be derived from the free entropy by taking suit- free energy, when evaluated exactly or approximately, al- able derivatives, as is shown in Sect. IV A. lows to predict phase transitions, structures and all static For an equilibrium system, i.e. when all heat baths ˙◦ properties of an equilibrium system. In the search for a have the same temperature and Sres = 0, and in the similarly useful framework, the first theoretical studies stationary state, the free entropy is time-independent and on NEQ systems formulated general extremal principles equals the negative free energy divided by temperature, that express the system’s tendency to extremize its dissi- i.e. pation, i.e., its entropy production [2, 3]. These early ex- S• = −F •/T = S• − U •/T, (2) tremal principles were limited to linear and homogeneous free systems close to equilibrium and included interactions where filled circles in our paper denote the equilibrium only indirectly in terms of phenomenological coefficients. stationary state. In fact, the free entropy functional had Subsequently, the Master equation approach allowed to been used by Massieu already in 1869 [72, 73], a few years derive extremal principles for general NEQ stationary before Gibbs introduced his energy transforms. The ad- states in terms of a generalized entropy [5, 6, 8, 11]. More vantage of the free entropy functional has been pointed recently, Onsager’s variational principle[1] was revisited out by Planck [74] and Schrdinger [75], while the name and used for the study of various NEQ problems in soft was introduced more recently in the mathematical liter- condensed matter [66]. ature [76, 77]. The free entropy is central in the con- In this paper we start from a quadratic Hamiltonian text of our model, since the presence of different heat model with general interactions. By adding friction bath temperatures does not allow to define a unique NEQ terms and stochastic fields to the Hamilton equations, version of the free energy. In a number of previous pa- we arrive at the general linear Hamiltonian-based many- pers functionals were derived that in a NEQ stationary dimensional Langevin equation that, for suitably chosen state are extremal and thereby allow to study the relax- friction and stochastic parameters, describes a many- ation and the stability of NEQ systems and the relation body system of massive particles coupled to multiple of these functionals to the Kullback-Leibler entropy[78] heat baths with different and well-defined temperatures, was pointed out [5–8, 11, 79–83]. Our model differs from a model that corresponds to the non-equilibrium version those works since we introduce NEQ by the coupling to of the multidimensional Ornstein-Uhlenbeck process and multiple heat baths with different temperatures. While has in certain limits been studied in literature [4, 51, 67– the early approaches to NEQ thermodynamics were cen- 71]. The explicit presence of heat baths with different tered on the total entropy production S˙tot(t), i.e. the 3 time derivative of the total entropy including the reser- tions based on Boltzmann statistics, on the fluctuation- voirs [2, 3], in later developments the entropy produc- dissipation theorem and on the detailed-balance condi- tion due to heat transfer from the reservoirs, which in a tion are equivalent only for symmetric friction matrices ˙◦ NEQ stationary state is constant and given by Sres, has and that the detailed-balance condition is the strictest of been separated off and called the house-keeping entropy all three. [9, 10, 84]. One advantage of our model is that since the As a simple application of our model we present re- particles have finite masses, the heat fluxes between the sults for three interacting massive particles that are cou- particles and the heat reservoirs can be calculated from pled to temperature reservoirs at different temperatures, energy balance considerations including the kinetic en- for which we demonstrate that heat flows from a particle ergy and thus the stationary reservoir entropy production coupled to a cold reservoir to a particle coupled to a warm ˙◦ Sres can be derived straightforwardly. For this we intro- reservoir if the third reservoir is sufficiently hot. This of duce a systematic perturbation scheme using the particle course does not constitute a violation of the second law masses as expansion parameters to calculate NEQ mixed of thermodynamics. Rather, this NEQ entrainment effect position-velocity correlations. can be rationalized by the fact that the reservoirs are not We here show that for a system coupled to differ- coupled directly to each other but rather indirectly via ent temperature reservoirs, the free entropy functional the particles, and that the particles are not characterized Sfree(t) can be written down explicitly and is for the NEQ solely by the heat bath temperatures. This point can be stationary distribution maximal and constant in time. explained in more detail by considering just two particles The free entropy production S˙free(t) is positive except at that are coupled to different heat baths: We demonstrate the stationary NEQ state, for which it vanishes, i.e. that the concept of a NEQ effective temperature has only a rather limited value, since each covariance matrix entry ˙ Sfree(t) ≥ 0. (3) of two coupled NEQ particles would have to be attributed a different effective temperature. For two particles we This shows that the NEQ stationary state is stable with also demonstrate that NEQ effects give rise to localiza- respect to small perturbations and that in the NEQ sta- tion effects both in position and in momentum space, tionary state the total entropy production is given solely which is reminiscent of attractive interactions. This re- by the reservoirs. The time derivative of the free entropy flects the tendency of NEQ systems to phase separate in production, which would be the second time derivative both position and momentum space. Finally, we show of the free entropy, is not needed, unlike early NEQ ap- how active particle models can be described using our proaches [2, 3]. Our framework thus treats NEQ and general framework and discuss the connection between equilibrium systems on the same footing, as for an equi- the free entropy production and the total entropy pro- librium system the NEQ free entropy smoothly crosses duction that includes the reservoirs. over to the standard free energy divided by −T , see Eq. In the following sections we first treat general NEQ (2), and so the standard equilibrium and stability condi- systems and derive the necessary conditions to reach a tions of statistical mechanics are recovered. stationary state and a stationary equilibrium state, de- Interestingly, the dynamic approach to the equilibrium scribed by the Lyapunov and the Lyapunov-Boltzmann and to the NEQ stationary distributions obey the same equations, respectively. In Sect. IV B we start treating differential equation, as is shown in Sect. III C. In fact, the core model of this paper, where particles are cou- while equilibrium and NEQ stationary distributions ex- pled to heat reservoirs that are characterized by different hibit many fundamental differences, the relaxation times temperatures, and present various explicit examples. that characterize the approach to stationarity are inde- pendent of the heat bath temperatures and therefore do not allow to distinguish equilibrium from NEQ systems. II. MANY-PARTICLE HAMILTONIAN MODEL While this might be a simplification due to our neglect of non-linear interactions, we argue that non-linear systems A. From Hamilton to Langevin equations can typically be quadratically approximated around lo- cally stable states and thus our results should also apply to sufficiently well-behaved non-linear systems. To proceed, we consider N massive particles in one We also demonstrate that the definition of equilibrium dimension with positions xα and momenta pα that move we are using in this paper, which is based on the Boltz- according to the Hamilton equations mann distribution, is equivalent to the detailed-balance ∂H(~x,~p) condition only if the friction matrix is symmetric and if x˙ α(t) = , (4) the random fields couple separately to position and veloc- ∂pα ity degrees of freedom. For a simple system consisting of two coupled massive particles, we show that if the Boltz- ∂H(~x,~p) p˙ (t) = − , (5) mann distribution is realized, the fluctuation-dissipation α ∂x theorem is satisfied, even when the friction matrix is α asymmetric (and thus the condition of detailed balance where α = 1 ...N is an index that runs over all particles. is not satisfied). This shows that equilibrium defini- The case of M interacting particles in three dimensions 4 is described by N = 3M particle coordinates and is im- couple to different particle coordinates. For simplicity, plicitly included in our model. Using the antisymmetric we assume Gaussian white random fields with zero mean 0 0 matrix hFi(t)i = 0 and and variances hFi(t)Fj(t )i = 2δijδ(t−t ). Non-Markovian models with colored noise can be ob-  0 1 0 0  tained by integrating out degrees of freedom and need −1 0 0 0   not explicitly be considered [85]. In standard friction  0 0 0 1  U =   (6) models friction forces couple to the momentum degree of  0 0 −1 0    freedom and are proportional to particle velocities. In . .. more elaborate models that include hydrodynamic inter- actions, the friction force acting on a given particle de- and the state vector pends on the velocities of all particles. In the first part

T of this paper the matrices Γ and Φ are kept general and ~z(t) = (x1(t), p1(t), x2(t), p2(t) ...) (7) Γ can also be asymmetric (which will be shown to have the Hamilton equations can be written compactly as direct consequences for the detailed-balance condition in Sect. III D). In the second part of the paper, starting in ∂H(~z) Sect. IV B, we model heat baths with different temper- z˙i(t) = Uij , (8) atures and for this will assume Γ and Φ to be diagonal. ∂zj The general form of the linear Langevin Eq. (13) does where i = 1 ... 2N is an index that runs over all po- not directly reveal whether it describes an equilibrium or sition and momentum coordinates. Throughout this pa- a NEQ system[70], this point will be addressed further per, greek indices denote particles (running from 1 to N), below. roman indices denote coordinates (running from 1 to 2N) and indices that appear more than once are summed over except primed indices. We consider quadratic Hamilto- B. Stationary distribution nians of the form The algebraic solution of the Langevin Eq. (13) is H(~z) = ziHijzj/2 (9) Z t that are described by a general symmetric matrix H −tA 0 −(t−t0)A 0 zi(t) = eij zj(0) + dt eij ΦjkFk(t ), (14) which we assume to be positive definite, i.e., H(~z) > 0 for 0 general ~z. A possible linear term in the Hamiltonian can where zj(0) denotes the initial particle positions and mo- be absorbed into the definition of the state vector zj and need not be considered explicitly. For quadratic Hamil- menta at time zero. The average over the noise gives tonians the Hamilton equations are linear and given by −tA hzi(t)i = eij zj(0) (15) z˙ (t) = U H z (t). (10) i ij jk k which can be viewed as the solution of the noise-averaged More specific forms of the Hamiltonian matrix H, in par- version of the Langevin Eq. (13) ticular Newtonian Hamiltonians where momentum und position degrees of freedom are decoupled, will be dis- hz˙i(t)i = −Aikhzk(t)i. (16) cussed later, in the first part of this paper the discussion applies to general Hamiltonian models. If all eigenvalues of the matrix A have positive real com- By adding linear friction terms and random fields to ponents, a unique stationary distribution exists and is the Hamilton equations, which will be later shown to characterized by a vanishing mean hzi(t → ∞)i = 0. mimic the coupling to heat baths with in general differ- The covariance matrix of the deviations from the mean ent temperatures, we obtain the coupled linear Langevin ∆zi(t) = zi(t)−hzi(t)i follows from squaring the solution equations Eq. (14) and averaging over the noise, leading to[86] t Z 0 0 z˙i(t) = UijHjkzk(t) − Γikzk(t) + ΦikFk(t) (11) 0 −(t−t )A −(t−t )A Eij(t) ≡ h∆zi(t)∆zj(t)i = 2 dt eik ejl Ckl, 0 which by definition of the generally asymmetric coupling (17) matrix where the random correlation matrix is defined by A = −U H + Γ (12) ik ij jk ik Ckl = ΦkmΦlm = Clk (18) can be written more compactly as and is symmetric by construction. Since

z˙i(t) = −Aikzk(t) + ΦikFk(t). (13) AikEkj + AjkEki = AikEkj + AjkEik = Z t Here, Γ is the friction coefficient matrix and Φ is the ran- 0 d −(t−t0)A −(t−t0)A 2 dt 0 eik ejl Ckl, (19) dom strength matrix that describes how random fields 0 dt 5 the stationary covariance matrix, denoted by an open where the time-dependent normalization constant is circle and defined by given by q ◦ 2N Eij ≡ Eij(t → ∞), (20) N (t) = (2π) det E(t) (25) is unique and given by the Lyapunov equation and only exists if the covariance matrix E(t) is positive definite (note that by its definition E(t) is also symmet- ◦ ◦ 2Cij = AikEkj + AjkEki (21) ric). The proof is standard textbook material [86], in Ap- pendix B we present a derivation based on random-field if all eigenvalues of the matrix A have positive real parts, path integrals, which has the advantage that it can in which we will assume to be true throughout this paper. principle be generalized to non-Gaussian colored noise; there we furthermore demonstrate that the expression Eq. (24) is in fact the Green’s function of the general III. SYSTEMS THAT HAVE AN EQUILIBRIUM Langevin Eq. (13), i.e., the conditional probability dis- DISTRIBUTION tribution at time t for the case that the distribution is a delta function at time t = 0. A. Distributions, entropy and free energy With the Gaussian form Eq. (24), the integral in Eq. (23) can be performed and yields the time-dependent dis- In equilibrium, the normalized distribution in terms of tribution entropy as the state vector ~z is given by the Boltzmann distribution 2N S(t)/kB = N + ln N (t) = N + (1/2) ln[(2π) det E(t)]. ρ(~z) = e−βH(~z)/Z, (22) (26) The internal energy is given by where β = 1/(k T ) denotes the inverse thermal energy B U(t) = hH(~z(t))i = H hz (t)z (t)i/2 (27) and Z = R d~ze−βH(~z) is the partition function. We will ij i j discuss the connection of this definition of equilibrium to = Hijhzi(t)ihzj(t)i/2 + HijEij(t)/2. the conditions of detailed balance, vanishing probability With these results for the entropy and internal energy, current as well as the fluctuation-dissipation theorem in the free energy Sect. III D. Positive definiteness of the Hamiltonian ma- trix H guarantees that Z is finite (if the Hamiltonian is F(t) = U(t) − T S(t) (28) invariant with respect to one or few degrees of freedom follows as they can be separated off to make the reduced Hamilto- nian positive definite). For a quadratic Hamiltonian, the F(t) = Hijhzi(t)ihzj(t)i/2 + HijEij(t)/2 (29) 2N average state vector vanishes and all covariances can be − kBTN − (kBT/2) ln[(2π) det E(t)]. calculated from the Boltzmann distribution, which only The extremum of the free energy is determined by involves inversion of the Hamiltonian matrix. h~z(t)i = 0 and by For the later discussion of the NEQ scenario, it is in- structive to derive the equilibrium distribution also via ∂F(t) −1 = Hij/2 − kBTEji (t)/2 = 0, (30) the thermodynamic route. From the thermodynamic def- ∂Eij initions of the free energy F = −kBT ln Z and the en- tropy S = −∂F/∂T , the Shannon expression for the en- the solution of which is time-independent and defines the tropy directly follows as equilibrium distribution (denoted by a filled circle) as • −1 Z Eij = kBTHij . (31) S/k = − d~zρ(~z) ln ρ(~z), (23) B In deriving Eq. (30) we used the basic algebraic rela- −1 tion ∂ ln det E/∂Eij = Eji . The partial derivative de- the derivation is shown in Appendix A. Note that the notes the derivative with respect to one matrix compo- Shannon expression Eq. (23) can also be used to de- nent while keeping all other components fixed. The equi- scribe the distribution entropy for time-dependent distri- librium free energy follows by reinserting h~z(t)i = 0 and butions, i.e. for non-stationary and even NEQ situations, • the solution Eij into the free energy expression Eq. (29) since the expression makes no reference to the equilib- and is given by rium ensemble or to the presence of a heat bath. This • 2N will allow us to describe the time-dependent approach to F = −(kBT/2) ln[(2πkBT ) / det H]. (32) equilibrium as well as to stationary NEQ distributions. We next want to show that the extremum of the free For the linear Langevin Eq. (13), the time-dependent energy is in fact a minimum (for this we neglect the triv- probability distribution is Gaussian and can be written ial quadratic dependence of Eq. (29) on the mean state as vector h~z(t)i). We first realize that ρ(~z, t) = (24) 2 ∂ F(t) −1 −1 −1 −1
= kBTEik (t)Elj (t)/2, (33) N (t) exp −[zi − hzi(t)i]Eij (t)[zj − hzj(t)i]/2 , ∂Ekl∂Eij 6

−1 where we used the basic algebraic relation ∂Ekn /∂Eij = equation describes a NEQ system. Two obvious NEQ −1 −1 scenarios come to mind: i) A Newtonian Hamiltonian −Eki Ejn . Around the equilibrium distribution Eij = • Eij the free energy is to second order given by many-body system coupled to heat baths characterized by different temperatures (as will be discussed starting • F(t) − F ' (34) in Section IV B), and ii) a many-body system with off- −1 −1 Hki(Eij(t) − Hij kBT )Hjl(Elk(t) − Hlk kBT )/(2kBT ), diagonal friction terms that do not obey Eq. (37). Let us give a simple example, namely the harmonic which can be rewritten as oscillator with Hamiltonian H = Kx2/2+p2/(2m), which k T (δ − βH E (t))(δ − βH E ) is described by the Hamiltonian matrix F(t) − F • ' B il ik kl il lk ki . 2 K 0  (35) H = . (38) The latter form is quadratic and of the general form 0 1/m F(t) − F • ' k TB B /2 with B ≡ δ − βH E (t), B il li il il ik kl We choose a diagonal momentum friction model de- but this by itself does not guarantee that F(t) − F • is scribed by the friction matrix positive since Bil is not necessarily symmetric. In Ap- pendix C we show by diagonalization that the positivity 0 0  • Γ = , (39) of the expression Eq. (34) for F(t) − F follows from the 0 γ/m fact that H is symmetric and positive definite. The Gaussian distribution Eq. (24) in conjunction where the friction force is proportional to the velocity with Eq. (31) is equivalent to the Boltzmann distribution of the particle and only enters the momentum degree of Eq. (22), which we have thus rederived by minimizing freedom. The matrix product appearing on the right side the time-dependent free-energy functional Eq. (29). But of the Lyapunov-Boltzmann Eq. (37) is given by the free energy functional Eq. (29) is not only valid in equilibrium but also describes systems that approach the 0 0 ΓH−1 = (40) equilibrium distribution. This is an important insight, 0 γ as this functional framework will allow us to characterize the approach not only to equilibrium but also to station- and thus the equilibrium random correlation matrix fol- ary NEQ distributions. lows as 0 0  C• = . (41) 0 k T γ B. When does a Langevin equation describe an B equilibrium system? This agrees with the well-known result that the equilib- rium Langevin equation of a massive particle involves a In this section we will explore under which conditions random field that acts on the momentum degree of free- the Langevin Eq. (13) describes an equilibrium system, dom only and is proportional to the friction coefficient which will put stringent conditions on the random cor- γ, where the proportionality constant kBT defines the relation matrix C and on the friction matrix Γ. We in temperature of the reservoir. Conversely, any non-trivial this paper define a system to be in equilibrium if the deviation of the matrix C from Eq. (41), i.e. any devia- stationary state corresponds to the Boltzmann distribu- tion that cannot be captured by a modified temperature, tion, the relation to other definitions of equilibrium will indicates a NEQ system. For the simple example of a be discussed in Sect. III D. We implement this condition one-dimensional harmonic oscillator considered here, this ◦ by replacing the stationary covariance matrix Eij in the could for example be the presence of an additional entry Lyapunov Eq. (21) by the equilibrium covariance matrix in the symmetric matrix C, e.g. additional entries in the • Eij from Eq. (31), by which we obtain off-diagonals or in the upper-left diagonal.

• −1 −1 2Cij/(kBT ) = AikHkj + AjkHki . (36) C. Dynamic approach to the stationary Inserting the expression for the Langevin matrix A from distribution Eq. (12) and using the fact that the matrix U is an- tisymmetric, see Eq. (6), we arrive at the Lyapunov- From the time-dependent analogue of the Shannon en- Boltzmann equation tropy Eq. (23) • −1 −1 2Cij/(kBT ) = ΓikHkj + ΓjkHki . (37) Z S(t)/kB = − d~zρ(~z, t) ln ρ(~z, t), (42) If for arbitrary Hamiltonian matrix H the friction ma- trix Γ and the random force correlation matrix C obey we obtain by differentiation this equation, the Langevin equation given by Eq. (13) Z describes the dynamics of an equilibrium system. Con- ˙ versely, if C and Γ do not satisfy Eq. (37) the Langevin S(t)/kB = − d~zρ˙(~z, t)[ln ρ(~z, t) + 1]. (43) 7

The time derivative of the density distributionρ ˙(~z, t) is and finally yields determined by the Fokker-Planck equation ˙ −1 ∂ ln ρ(~z, t) −1 Ekl = hz˙i(t)iE ∆zj + {Ekl(t) − ∆zk∆zl} . ρ˙(~z, t) = [∇kAkmzm + ∇k∇mCkm] ρ(~z, t) (44) ∂t ij 2 (53) which follows via Kramers-Moyal expansion of the Comparison of Eqs. (50) and (53) term by term yields Langevin Eq. (13) [86]. With the Gaussian time de- ˙ −1 −1 −1 −1 −1 pendent distribution Eq. (24) we obtain from Eq. (44) Eij (t) = −2CkmEkj (t)Emi (t)+Ejk (t)Aki+Eik (t)Akj, the expression (54) where we have used Eq. (16). From the basic algebraic ρ˙(~z, t) = ρ(~z, t) × (45) ˙ ˙ −1 −1 relation Eil(t) = −Eij(t)Ejk (t)Ekl (t) we finally obtain h −1 −1 −1 −1 i Akk − AkmzmEkj ∆zj + Cij(Eik ∆zkEjl ∆zl − Eij ) . from Eq.(54) the temporal change of the covariance ma- trix as Inserting this into the expression Eq.(43) and calculat- ing all Gaussian expectation values, we obtain the final E˙ ij(t) = 2Cij − AikEkj(t) − AjkEki(t) (55) expression for the time derivative of the entropy as which, using Eq.(21), can be rewritten as −1 S˙(t)/kB = CkmE (t) − Akk, (46) km ˙ ◦ ◦ Eij(t) = −Aik[Ekj(t) − Ekj] − Ajk[Eki(t) − Eki]. (56) which holds for equilibrium as well as for NEQ systems. From the Lyapunov Eq. (21) we can derive the expression Note that this expression holds for equilibrium as well as for NEQ systems. As would be expected, the temporal ◦−1 CijEij = Akk, (47) change of the covariance matrix vanishes in the station- ◦ ary state, i.e. when Ekm(t) = Ekm. inserting this into Eq. (46) we obtain the alternative expression D. Conditions of detailed balance and vanishing ˙ −1 ◦−1 S(t)/kB = Ckm[Ekm(t) − Ekm ], (48) probability current which demonstrates that the entropy change vanishes in ◦ Our definition of equilibrium employs the Boltzmann the stationary state Ekm(t) = Ekm, as is expected. We distribution Eq. (22) and not the condition of detailed will later come back to Eq. (48) as it allows to write balance, which is often used as the defining property of down one of the constitutive dynamic equations for NEQ equilibrium [87]. The reason for using the Boltzmann systems. condition is that it is very easy to implement, while the We next calculate the time derivative of the covariance condition of detailed balance is for underdamped many- matrix. From the expression Eq. (45) we immediately particle systems rather involved. In fact, for overdamped read off that systems the detailed balance condition becomes equiv- ∂ ln ρ(~z, t)/∂t = (49) alent to the condition of vanishing probability current [5, 8], which in literature is also called the potential con- A − A z E−1∆z + C (E−1∆z E−1∆z − E−1), kk km m kj j ij ik k jl l ij dition. We will in this section formulate the condition for which can be rewritten as the probability current to vanish and then compare with the detailed balance condition, for which the derivation −1 −1 −1 ∂ ln ρ(~z, t)/∂t = ∆zi∆zj[CkmEkj Emi − AkiEkj ] (50) is presented in Appendix D. +A − C E−1 − A hz (t)iE−1∆z . The Fokker-Planck Eq. (44) can be interpreted as a kk km km km m kj j balance equation On the other hand, using the definition of the Gaussian distribution Eq.(24) we find ρ˙(~z, t) = −∇kJk(~z, t) (57) ∂ ln ρ(~z, t) ∂  1 with the probability current J (~z, t) being given as = − ln[(2π)2N det E(t)] (51) ∂t ∂t 2 Jk(~z, t) = − [Akmzm + ∇mCkm] ρ(~z, t). (58) 1  − [z − hz (t)i]E−1(t)[z − hz (t)i] , 2 i i ij j j For the Gaussian distribution Eq. (24) we obtain (for simplicity we set hzi(t)i = 0 here) for the current which can be rewritten as  −1 ∂ ln ρ(~z, t) Jk(~z, t) = − Akm − CkjEmj zmρ(~z, t). (59) = hz˙ (t)iE−1(t)∆z (52) ∂t i ij j The probability current vanishes, i.e. J (~z, t) = 0, for ˙ −1   k Ekl ∂ 1 −2N −1 1 −1 + −1 ln[(2π) det E (t)] − ∆ziEij (t)∆zj ∂Ekl 2 2 AkmEmj = Ckj. (60) 8

• From the equilibrium condition Ekj(t) = Ekj = E. Time-dependent free energy: extremal and −1 stability properties kBTHkj , Eq. (31), and the explicit form of the ma- trix A in Eq. (12), we obtain the vanishing probability current condition The time derivative of the free energy expression Eq. (28) reads

−1 ˙ ˙ ˙ ΓkmHmj − Ukj = Ckj/(kBT ), (61) F(t) = Hijhz˙i(t)ihzj(t)i + HijEij(t)/2 − kBT S(t) (62)

and using Eqs. (16), (46), (55) is explicitly given by which in the general case is not satisfied in the equilib- rium situation defined by the Lyapunov-Boltzmann Eq. F˙ (t) = −hzj(t)iHjiAikhzk(t)i + HijCij − HijAikEkj(t) (37), since C is a symmetric matrix, while U is antisym- −1 −1 −kBTCkmEkm(t) + kBTAkk. (63) metric and typically ΓkmHmj is an asymmetric matrix. We conclude that for an underdamped system, the prob- After some manipulation this expression can be rewritten ability generally does not vanish, this is trivially illus- as trated by the fact that a harmonic oscillator performs orbits in phase space. In fact, current mathematical F˙ (t) = −hzj(t)iHjiΓikhzk(t)i (64) work is devoted to separating phase space trajectories of −1 Elj(t) −1 underdamped systems into periodic and diffusive parts −Ckm[Hml − kBTE (t)] [Hjk − kBTE (t)]. ml k T jk [82, 83]. In Appendix E we show that in the overdamped B (i.e. massless) limit the probability current in equilib- The quadratic form of Eq. (64) directly demonstrates rium vanishes if the friction matrix Γ is symmetric. Since that for the equilibrium distribution, defined by Ekj(t) = we did not impose that Γ is symmetric so far, we see that • −1 Ekj = kBTHkj and hzk(t)i = 0, the free energy is sta- our definition of equilibrium, which is based on the sta- ˙ tionary distribution being equal to the Boltzmann distri- tionary and does not change in time, i.e. F(t) = 0. This bution, is for overdamped systems only equivalent to the is somewhat trivial since this just reflects that the equi- vanishing probability current condition for a symmetric librium distribution is a special case of a stationary dis- friction matrix. We conclude that the vanishing proba- tribution. More importantly, from the fact that H, C bility current condition is even in the overdamped limit and E are symmetric matrices that are positive-definite a more restrictive criterion than our Boltzmann distribu- or semi-positive-definite, we derive in Appendix C that tion criterion. the free energy does not increase in time, i.e. Furthermore, in Appendix F we show for the spe- F˙ (t) ≤ 0 (65) cial case of two coupled particles, that the equilib- rium fluctuation-dissipation theorem holds if our Boltz- in all generality, this means that the equilibrium distribu- mann definition for equilibrium is satisfied and in par- tion is stable with respect to perturbations. For this we ticular also works for an asymmetric friction matrix. have used that the first term of Eq. (64) is not negative This suggests that equilibrium definitions based on the since the product of two positive semi-definite matrices Boltzmann distribution and based on the fluctuation- is also positive semi-definite. Equation (65) corresponds dissipation theorem are equivalent and that the condi- to the second law of thermodynamics, here derived for tion of vanishing probability current is more restrictive many-body systems described by quadratic Hamiltonians and requires the symmetry of the friction matrix. and Langevin equations with friction terms and random fields. These results are graphically illustrated in Fig. 1: Finally, in Appendix D we derive the condition of de- Figure 1a) shows that the free energy F(t) is minimal tailed balance for our underdamped Hamiltonian model in equilibrium defined by E (t) = E• and hz (t)i = 0, and demonstrate that it is equivalent to the Boltzmann- kj kj k distribution based criterion for equilibrium only if the as demonstrated by Eq. (35). Figure 1b) shows that ˙ friction matrix is symmetric. Clearly, an asymmetric fric- the time derivative of the free energy F(t) is negative ex- tion matrix breaks the physical principle of equal actio cept in equilibrium where it vanishes, as follows from Eq. and reactio, so for physical models where friction is pro- (65). This indicates that a non-stationary distribution duced e.g. by hydrodynamic interactions, the friction flows monotonically towards the equilibrium stationary matrix should be symmetric and our definition of equi- distribution, as indicated by the red arrows in the top librium (based on the Boltzmann distribution) is fully graph. equivalent to the condition of detailed balance. More We finally derive a set of constitutive dynamic equa- abstract models with asymmetric friction matrices are tions, which we will in the next section generalize for conceivable, for such models the distribution is predicted NEQ systems. For an equilibrium system the Lyapunov- to be of the Boltzmann type if the Lyapunov-Boltzmann Boltzmann condition Eq. (37) is satisfied and the sta- ◦ Eq. (37) is satisfied, yet the condition of detailed balance tionary covariance matrix Ekj is given by the equilib- ◦ • −1 is violated. rium state, i.e. Ekj = Ekj = kBTHkj . Comparison of 9 the expression for the entropy production Eq. (48) and and the advantage of this functional was already recog- the derivative of the free energy Eq. (30) yields nized by Planck [74] and Schrdinger [75]. The free en- tropy concept is particularly useful in the current NEQ 2 ∂F(t) setting, since for the NEQ systems we will consider there S˙(t) = − Cij , (66) kBT ∂Eij is no unique temperature. In fact, the free entropy concept allows for straightfor- which is a simple relation between the entropy produc- ward generalization to the NEQ case. For this we replace ˙ •−1 ◦−1 tion S and the free energy derivative, the first constitu- Eij in Eq. (69) by Eij , after which we obtain the tive dynamic relation. The other dynamic relations are NEQ version of the free entropy obtained from the time derivative of the covariance ma- ◦−1 ◦−1 trix Eq. (56), S (t) S(t) E Eij(t) E hzi(t)ihzj(t)i free = − ij − ij . (70)   kB kB 2 2 ˙ 2 • ∂F(t) • ∂F(t) Eij(t) = − AikEkl Ejm + AjkEkl Emi , kBT ∂Elm ∂Elm Note that the entropy production in the stationary NEQ (67) state, which for suitably chosen friction and random and by comparing Eq. (16) with the derivative of the strength matrices can be described as being due to heat free energy Eq. (29) with respect to hzi(t)i, fluxes in and out of heat reservoirs, is, as illustrated in Eq. (1), not included in the free entropy, but will be dis- 1 • ∂F(t) cussed in Section IV B. Clearly, for an equilibrium sys- hz˙i(t)i = − AikEkl . (68) ◦−1 •−1 kBT ∂hzli tem, for which Eij = Eij = Hij/(kBT ), the general expression Eq. (70) reduces to the equilibrium expres- Equations (66), (67) and (68) are the constitutive dy- sion Eq. (69). Similar functionals, which in a NEQ namic equations for equilibrium systems that relate tem- stationary state are extremal, were derived previously poral changes of all relevant quantities to derivatives of [5–8, 11, 79–83], in Appendix G we show that the free the free energy with respect to the state variables, i.e. to entropy expression Eq.(70) is equivalent to the Kullback- generalized thermodynamic forces. Leibler entropy [6, 78]. Our model differs from previ- ous approaches in that the definition of separate friction and random strength matrices will allow us to describe IV. TRULY NON-EQUILIBRIUM SYSTEMS the coupling to multiple heat baths with different well- defined temperatures. A. Extremal and stability properties of the free Using the free entropy, the constitutive dynamic equa- entropy tions (66), (67) and (68) can be rewritten as

∂Sfree(t)/kB We remark that the word non-equilibrium (NEQ) typ- S˙(t) = 2Cij , (71) ically refers to two very different situations: For a NEQ ∂Eij system, i.e. when the Lyapunov-Boltzmann condition Eq. (37) is not satisfied, the stationary distribution is ◦ ∂Sfree(t)/kB E˙ ij(t) = 2AikE Ejm a NEQ stationary distribution, which is characterized kl ∂Elm by a non-vanishing positive entropy production. For ◦ ∂Sfree(t)/kB + 2AjkE Emi, (72) such a system an equilibrium distribution does not ex- kl ∂Elm ist. An equilibrium system is one where the Lyapunov-

Boltzmann condition Eq. (37) is satisfied, but even such ◦ ∂Sfree(t)/kB a system is off equilibrium as long as it has not settled hz˙i(t)i = AikEkl , (73) ∂hzli in its stationary equilibrium distribution. Obviously, it is not possible to use the constitutive where the temperature has obviously (and trivially) dis- equilibrium dynamic equations (66), (67) and (68) in the appeared. When Eq.(70) is used in conjunction with the NEQ case, because of the appearance of the equilibrium constitutive dynamic equations (71), (72) and (73), the temperature T . The temperature can be trivially elimi- exact dynamic evolution equations (derived for general nated by introducing a modified thermodynamic poten- NEQ systems) Eqs. (48), (56) and (16) are reproduced, tial, which is called the free entropy and which is, for this independently confirms the validity of the expression the equilibrium scenario using Eqs. (27), (28) and (31), Eq. (70). given by We next show that the free entropy has very similar properties for NEQ systems as the free energy has for •−1 •−1 S (t) F(t) S(t) E Eij(t) E hzi(t)ihzj(t)i equilibrium systems, namely S (t) is extremal and in free ≡ − = − ij − ij . free kB kBT kB 2 2 fact maximal in the stationary NEQ state and the sta- (69) tionary state is stable in the sense that S˙free(t) ≥ 0. For As mentioned before, the equilibrium free entropy had this we basically repeat the steps leading to the minimal been originally introduced by Massieu in 1869 [72, 73] condition of the free energy Eq. (35). The extremum of 10

S F free E E

Sfree

a) c) < <

F E E

F E Sfree E b) d)

E E

FIG. 1. Graphical illustration of extremal and stability properties of equilibrium and non-equilibrium (NEQ) systems. For an equilibrium system, the free energy in a) takes its minimal value F • given by Eq. (32) at the equilibrium value of the covariance matrix E•, determined by Eq. (31). The free energy production F˙ in b) is negative and vanishes at the equilibrium state, see Eq. (65). This means that the system is stable with respect to perturbations and flows towards the equilibrium state, as ◦ indicated by red arrows in a). For a NEQ system, the free entropy Sfree(t) in c) takes its maximal value Sfree at the stationary ◦ value of the covariance matrix E . The free entropy production S˙free(t) in d) is positive and vanishes in the stationary state. This means that the system is stable with respect to perturbations and flows towards the NEQ stationary state, as indicated by red arrows in c).

Sfree(t) is given by h~z(t)i = 0 and determined by The free entropy production follows from Eq. (70) by taking a time derivative as ∂Sfree(t)/kB −1 ◦−1 = Eji (t)/2 − Eji /2 = 0, (74) ◦−1 ˙ ∂Eij S˙ (t) S˙(t) E Eij(t) free = − ij −E◦−1hz˙ (t)ihz (t)i. (78) k k 2 ij i j the solution of which yields the time-independent station- B B ary distribution, i.e. E (t) = E◦ . With the stationary ij ij Using our previous results for S˙(t), Eq. (48), for E˙ (t), covariance matrix E◦ many observables can be calcu- ij ij Eq. (56), and for hz˙ (t)i, Eq. (16), we arrive after a few lated, for example the internal energy follows according i intermediate steps at to Eq. (27). The stationary free entropy is given by ˙ ◦−1 ◦ 2N ◦ Sfree(t)/kB = hzj(t)iEji Aikhzk(t)i (79) S /kB = (1/2) ln[(2π) det E ]. (75) free ◦−1 −1 ◦−1 −1 +Ckm[Eml − Eml (t)]Elj(t)[Ejk − Ejk (t)]. The second derivative of the free entropy is given by The quadratic form of this expression shows that in the 2 ◦ ∂ Sfree(t)/kB −1 −1 stationary state, defined by Ekj(t) = Ekj and hzk(t)i = = −Eik (t)Elj (t)/2. (76) ∂Ekl∂Eij 0, the free entropy production of the system vanishes. More importantly, from the fact that E◦, C and E ◦ Around the stationary state Eij = Eij the free entropy are symmetric matrices that are semi-positive-definite or thus is to second order given by positive-definite, it follows that the free entropy does not

◦ decrease in time, i.e. Sfree(t)/kB − Sfree/kB ' (77) ◦−1 ◦−1 ◦ ◦ ˙ Eik Elj (Eij(t) − EijkBT )(Ekl(t) − EklkBT )/2, Sfree(t) ≥ 0 (80) which is positive since E◦−1 is symmetric and positive in all generality, this means that the stationary NEQ definite (the general proof for this is given in Appendix distribution is stable with respect to perturbations, the C). proof is given in Appendix C. In writing Eq. (80) we have 11 used that the first term of Eq. (79) is not negative since with which the Hamiltonian can be written as the product of the two positive-definite matrices A and E◦ is also positive-definite, where positive definiteness H(~y) = yαHαy/2. (81) of the asymmetric matrix A is equivalent to demanding that all eigenvalues of the symmetric part of A are posi- Here we introduced the particle state vector tive, which is more restrictive than the condition that the y (t) = (x (t), p (t))T (82) eigenvalues of A have all positive real components, as re- α α α quired for the existence of a stationary state in Sect. II B. where xα(t) and pα(t) are the position and the momen- This result is here derived for interacting many body sys- tum of particle α. The N × N entries of the Hamiltonian tems described by quadratic Hamiltonians and is graphi- matrix Hα each consist of 2 × 2 matrices. These sub cally illustrated in Fig. 1. Figure 1c) shows that the free matrices are expanded in terms of the matrices entropy Sfree(t) is maximal at the stationary distribution ◦       defined by Ekj(t) = E and hzk(t)i = 0, which follows 0 1 1 0 0 0 kj u = , s = , r = . (83) from Eq. (77). Figure 1d) shows that the time deriva- −1 0 0 0 0 1 tive of the free entropy S˙free(t) is positive except at the stationary distribution where it vanishes, as follows from From the matrix products Eq. (79). This indicates that the system flows monoton-  0 0 0 1 ically towards the stationary state, as indicated by the us = ru = , ur = su = (84) red arrows in Fig. 1c). −1 0 0 0 The free entropy maximization principle applies to it is easily seen that all 2 × 2 matrices can be expanded NEQ and equilibrium systems alike. On the one hand in terms of the four matrices s, r, us and ur, which thus this allows to treat NEQ and equilibrium systems within form a convenient complete set. We will make in the a unified framework and thereby eliminates a disturb- following repeated use of the properties ing schism in the description of these systems. On the other hand, the usage of the free entropy, which does s2 = s, r2 = r, u2 = −1, rs = 0 = sr. (85) not include the stationary reservoir entropy production (which is related to the so-called house-keeping entropy The Langevin Eq. (13) can be written as [9, 10, 84] and increases linearly in time in a stationary NEQ state, see Eq. (1) and as discussed in the next sec- y˙α(t) = −Aαy(t) + ΦαF(t). (86) tion), brings a significant methodological advantage over early approaches to NEQ systems, according to which where Aα is given by stationary NEQ states are defined by an extremum of A = −U H + Γ (87) the total entropy production (including the reservoirs) α αγ γ α and stability criteria invoke the time derivative of the en- and Uαγ can be written as Uαγ = uδαγ . The Hamiltonian tropy production, i.e. the second time derivative of the matrix is for Newtonian systems given by total entropy [87]. The connection between the free en- tropy production (excluding the reservoirs) and the total Hα0 = shα0 + rδα0/mα0 (88) entropy production (including the reservoirs) is discussed in the Conclusions and also in Appendix H. where hα is a general N × N symmetric interaction ma- trix that only acts on positional degrees of freedom (and thus is multiplied by the matrix s) and the second term B. Stationary entropy production for particles is the kinetic energy which is diagonal in the momentum coupled to temperature reservoirs degrees of freedom (and thus is multiplied by the ma- 0 trix r) and mα0 is the mass of particle α . Note that the 0 The calculations so far were completely general and primed index α is not summed over. From the product no restrictions on the type of the Hamiltonian matrix properties of the 2 × 2 matrices the inverse Hamiltonian H, the friction matrix Γ and the random correlation ma- matrix follows as trix C were imposed. To gain insight into the entropy −1 −1 Hα0 = shα0 + rδα0mα0 . (89) production of the reservoirs, denoted by S˙res(t), we need to restrict the discussion to Newtonian systems, mean- To allow for a clear definition of reservoir temperatures, ing that in the Hamiltonian the spatial and momentum we will in the remainder treat momentum-diagonal fric- coordinates decouple and the kinetic energy is diagonal. tion, for which the friction matrix Γ is reduced to the This will allow to ascribe well-defined temperatures to momenta entries as different heat baths, which will then be used to calculate the reservoir entropy production from the individual heat Γα0 = rγα0/mα0 (90) fluxes between the system and the heat baths. To con- veniently use the symmetry of such Newtonian systems, and where the matrix γα0 is diagonal and given by γα0 = 0 we switch to particle indices, denoted by greek symbols, δα0γα0 , where γα0 is the friction coefficient of particle α . 12

2 ◦ The Lyapunov-Boltzmann condition Eq. (37) in terms of and dhvαi /dt = 0 and using that hα is symmetric, we particle indices reads obtain for the reservoir entropy production in the sta- tionary state • −1 −1 2Cα/(kBT ) = Γαγ Hγ + Γγ Hγα , (91) ˙◦ ◦ 1 ◦ Sres/kB = −βαhαhxvαi = hαhxvαi (β − βα). which, using Eqs. (89) and (90) and the matrix product 2 (98) properties Eq. (85), leads to This expression shows that one necessary condition for

• • • p a non-zero stationary reservoir entropy production is Cα0 = rδα0γα0 kBT, Φα0 = rφα0 = rδα0 γα0 kBT. that reservoirs have different temperatures. Other neces- (92) sary conditions are a non-vanishing stationary position- These expressions show that the random correlation and ◦ velocity coupling hxvαi , which for Newtonian Hamilto- random strength matrices are diagonal in the particle in- nian systems is only obtained off equilibrium, and a non- dices and proportional to r and thus only couple momen- vanishing interaction strength hα. To obtain explicit tum degrees to each other. The NEQ generalization of results for the stationary reservoir entropy production Eq. (92) for reservoirs with different temperatures reads we need to calculate the position-velocity correlations ◦ p hxvαi , which do not vanish even in the overdamped Cα0 = rγα0 δα0/βα0 , Φα0 = rδα0 γα0 /βα0 , (93) massless limit for a NEQ system. For this a systematic perturbative scheme is introduced in the next section. where βα0 = 1/(kBTα0 ) denote the different inverse ther- mal energies of reservoirs, each characterized by a tem- perature Tα0 . The expressions Eq. (93) are central to C. Perturbative solution of the Lyapunov equation our paper as they define the NEQ model we are using to derive all following results. The solution of the Lyapunov equation (21) for N par- To calculate an explicit expression for the reservoir en- ticles described by a state vector with 2N components tropy production we multiply the Langevin equation (86) consists of determining all 2N(2N + 1)/2 entries of the for the momentum component pα(t) by pα(t) and use symmetric covariance matrix E. The calculation is cum- Eqs. (87) and (88) to obtain bersome even for only N = 2 particles [19]. Here we 2 introduce a systematic expansion of the Lyapunov equa- p 0 (t)p ˙ 0 (t) = d(p 0 (t))/(2dt) = (94) α α α tion with the particles mass as the perturbation param- −pα0 (t)hα0x(t) − pα0 (t)γα0 pα0 (t)/mα0 + pα0 (t)Φα0α0 Fα0 (t), eter, which employs a projection of the matrix equations

0 onto the complete set of 2 × 2 matrices Eqs. (83) and where we again note that the primed index α is not (84) introduced in the previous section. This expansion summed over. From this expression the average heat- is subtle, since the leading-order result for the covariance ing rate of the system due to reservoir α0, i.e. the work matrix in the limit mα → 0 is not obtained by taking this performed by the random force per unit time, the last limit upfront in the Langevin equation. In fact, the over- term in Eq. (94), minus the friction work dissipated by 0 damped limit is commonly obtained by setting mα = 0 the particle α per unit time, the second-last term in Eq. in the Langevin equation [86], which correctly describes (94), turns out to be the long-time particle dynamics but obviously misses the 2 short-time ballistic particle dynamics and leads to di- Q˙ 0 (t) = Φ 0 0 hp 0 (t)F 0 (t)i/m 0 − γ 0 hp 0 (t)p 0 (t)i/m 0 α α α α α α α α α α vergent instantaneous particle velocities. Since we need 2 1 dhpα0 (t)i = + h 0 hp 0 (t)x (t)i/m 0 . (95) position-velocity correlations in order to estimate the en- 2m 0 dt α  α  α α tropy production according to Eq. (98), it is advisable Since the last line of Eq. (95) is nothing but the time to keep velocities to leading order as mα → 0. As turns derivative of the sum of the kinetic and potential ener- out, position-velocity correlations that result from the gies of particle α0, we see that the reservoir heating rate perturbation calculation stay finite even in the mα → 0 Q˙ α0 (t) balances the particle energy at each instance of limit. time (a clear consequence of the quadratic Hamiltonian To proceed, we expand the covariance matrix as approximation). For the entropy production of all reser- Eαβ = hxαxβis + hpαpβir + hxαpβiut + hpαxβius (99) voirs S˙res(t) we thus obtain from Eq. (95) the expression and insert the expressions for A, Eq. (87), H, Eq. (88), ˙ 2 Γ, Eq. (90), C, Eq. (93) and E, Eq. (99), into the Sres(t) ˙ βα dhpα(t)i βαhα ≡ −βαQα(t) = − − hpαxi. Lyapunov Eq. (21). The resulting expression splits into kB 2mα dt mα (96) an equation proportional to s for α 6= β We now replace momenta by velocities vα0 (t) = ◦ ◦ 0 = hxαvβi + hxβvαi , (100) pα0 (t)/mα0 . From the fact that in the stationary state an equation proportional to s for α = β d hx x i◦ = hx v i◦ + hv x i◦ = 0 (97) ◦ dt α  α  α  0 = hxαvαi , (101) 13 an equation proportional to r for α0 6= β0 and which are characterized by the two diagonal fric- ◦ tion coefficients γ1 and γ2 as defined in Eq. (90). The 0 = (γα0 mβ0 + γβ0 mα0 )hvα0 vβ0 i (102) particles are coupled to two heat reservoirs character- ◦ ◦ +mβ0 hα0γ hxγ vβ0 i + mα0 hβ0γ hxγ vα0 i , ized by inverse thermal energies β1 and β2 as defined in an equation proportional to r for α0 = β0 Eq. (93). This is a model system that has been consid- ered by different researchers [69, 88–90]. We here repro- ◦ ◦ duce the complete covariance matrix from our previous 1/βα0 = mα0 hvα0 vα0 i + mα0 hα0γ hxγ vβ0 i /γα0 , (103) calculation[19]. By straightforward solution of the set of an equation proportional to ur for α0 6= β0 linear equations (100)-(105) we obtain to leading order in the particle masses ◦ ◦ ◦ mβ0 hvα0 vβ0 i = γβ0 hxα0 vβ0 i + hβ0γ hxγ xα0 i , (104)   ◦ 1 h12 hx1x1i = h22d + ∆ γ2 , (107) and an equation proportional to ur for α0 = β0 β1 h22

◦ ◦ mα0 hvα0 vα0 i = hα0γ hxγ xα0 i , (105)   ◦ 1 h12 hx2x2i = h11d − ∆ γ1 , (108) where we have converted all momenta pα0 to velocities β2 h11 vα0 = pα0 /mα0 . Note that there are also two equa- tions proportional to us which however are equivalent   ◦ h12d 1 1 h11γ2 − h22γ1 to the equations proportional to ur. The limit mα0 → 0 hx1x2i = − + + ∆ , (109) must be taken with care. Equation (105) shows that 2 β1 β2 h12 ◦ −1 hvα0 vα0 i ∼ m , which reflects the equipartition theo- rem in the equilibrium case, while Eq. (102) suggests that 1  1 h m  ◦ ◦ ◦ 12 1 hvαvβi ∼ hxαvβi for α 6= β. Together with Eq. (104), hv1v1i = + ∆ , (110) ◦ ◦ 0 m1 β1 γ1 this suggests that hvαvβi ∼ hxαvβi ∼ m for α 6= β and thus that these terms do not necessarily vanish in   the mass-less limit mα → 0. This in turn means that ◦ 1 1 h12m2 hv2v2i = − ∆ , (111) the terms in the boxes in Eqs. (103) and (104) can be m2 β2 γ2 treated perturbatively in an expansion in powers of mα and can, to leading order in the particles masses, be ne- glected. Corrections to the leading-order results for the ◦ m2h11 − m1h22 hv1v2i = ∆ , (112) covariances can be systematically calculated by insert- γ1m2 + γ2m1 ing the leading-order results for the terms in the boxes and by solving the resulting equations term-by-term in hx v i◦ = −hv x i◦ = ∆, (113) powers of the particle masses. Such a calculation of next- 1 2 1 2 leading-order terms would also allow to assess the accu- 2 −1 where d = (h11h22 − h12) is the inverse determinant of racy of the leading-order results, but is rather involved the interaction matrix and the parameter because for NEQ systems mixed position-velocity cross-   correlations are present, as we will show explicitly for h12 1 1 ∆ = − (114) the simple case of two coupled particles in the next sec- γ1h22 + γ2h11 β2 β1 tion. As a main result, we conclude that while the mean- squared velocities of particles diverge in the overdamped is a measure of the departure from equilibrium. For limit, the position-velocity correlations between different ∆ = 0, i.e. in equilibrium, the covariance matrix ele- particles take non-zero and finite values for NEQ systems ments are given by the inverse of the Hamiltonian ma- in the overdamped limit. trix according to Eq. 31 and in particular the off-diagonal ◦ velocity coupling terms hv1v2i and the position-velocity ◦ ◦ correlations hx1v2i = −hv1x2i vanish. Off equilibrium, V. APPLICATIONS that means for ∆ 6= 0, these covariances are non zero and thus the symmetry of the covariance matrix changes A. Two particles coupled to different temperature abruptly. The expressions in the square brackets in Eqs. reservoirs: effective temperature concept and (107) - (111) could in principle be used to define inverse position/momentum localization effective temperatures for the covariance elements that are non-zero in equilibrium: inspection of the terms in We now present explicit results for two particles that the square brackets shows that they are all different. For are described by the Newtonian Hamiltonian as defined the covariances Eqs. (112) and (113) that are propor- generally in Eq. (88), tional to ∆, the effective inverse temperatures are also different and diverge as equilibrium is approached, i.e. as 2 2 2 2 H = h11x1/2 + h22x2/2 + h12x1x2 + m1v1/2 + m2v2/2 ∆ → 0. While for one or two of the covariances an effec- (106) tive temperature can be defined [91], consideration of the 14 entire covariance matrix shows that the ascription of an and for which momenta do not couple to positions, par- effective temperature to a particle is not possible. This ticle velocities are uncorrelated to each other in equilib- suggests that an effective temperature picture, where one rium. We thus conclude that the momentum localization assigns effective temperatures to particles, does not de- demonstrated in Eq. (116) is a NEQ phenomenon that scribe the particle statistics correctly and in particular has no equilibrium analogue for Newtonian Hamiltoni- does not characterize well the transition from equilib- ans. rium, ∆ = 0, to NEQ, ∆ 6= 0. To rescue the effective temperature picture one would have to ascribe different temperatures to each covariance matrix element, which B. Stationary entropy production for three coupled clearly is far from the usefulness of the temperature defi- particles: Heat flux from cold to warm reservoir nition at equilibrium. When basing the effective temper- ature definition on the fluctuation-dissipation relation, We will now present an explicit solution for a system as an additional effect a frequency dependence appears consisting of three coupled particles, as schematically vi- [19, 42, 43, 92], which is not reflected in the effective sualized in Fig. 2a). Since here we are only interested temperatures one would obtain based on the covariance in the stationary entropy production, Eq. (98), we only matrix elements. need to calculate the stationary velocity-position cross ◦ As an additional illustration of NEQ effects, we cal- terms hxvαi . We consider the simplified Hamiltonian culate the mean-squared difference between the particle h h m m m positions, which from Eqs. (107) - (109) is to order m0 H = 1 (x −x )2 + 3 (x −x )2 + 1 v2 + 2 v2 + 3 v2 2 1 2 2 2 3 2 1 2 2 2 3 in the particle mass given by (117)      where only particles 1 and 2 and particles 2 and 3 are 2 ◦ β1 + β2 β1 − β2 γ1 − γ2 h(x1 − x2) i = 1 − , coupled via harmonic bonds. We also assume the friction 2hβ1β2 β1 + β2 γ1 + γ2 coefficients to be all the same, i.e. γ1 = γ2 = γ3 ≡ γ. (115) 2 ◦ The solution strategy consists in eliminating hv 0 i by where we have considered two particles whose center of α inserting Eq. (105) into Eq. (103), which results in 3 mass is not confined, i.e. h = h = −h = h > 0 11 22 12 equations, and solving the six equations defined by Eq. (note that this limit must be taken with care since the (104) by using Eqs. (100) and (101). These are nine interaction matrix h is not invertible in this case). The ij equations for nine unknowns, it turns out that Eq. (102) first term is the equilibrium result which survives in the is not needed. The results are to order m0 in the particle limit β = β . Note that when the two friction coeffi- 1 2 mass given by cients and the two temperatures are different from each   other, NEQ effects modify the equilibrium result. In fact, ◦ 1 h1 h1 + 3h3 2h1 + 3h3 hx2v3i = + − , in the limits β1/β2  1 and γ1/γ2  1 (or β2/β1  1 6γ(h1 + h3) β1 β2 β3 and γ2/γ1  1) the distance between the particles tends (118) to zero, thus indicating positional co-localization of par-   ◦ 1 2h3 + 3h1 h3 h3 + 3h1 ticles, which in equilibrium one would only obtain from hx1v2i = − − , strong attractive interactions between particles. Interest- 6γ(h1 + h3) β1 β3 β2 2 (119) ingly, since βα = γα/bα, where bα denotes the strength of the Gaussian white noise that enters the Langevin equa-   ◦ 1 h3 + h2 h1 − h3 h1 + h2 tion, we see that co-localization is automatically obtained hx1v3i = + − , (120) γΘ β1 β2 β3 when the friction coefficients of particles γα are modified where we defined for convenience while keeping the random strengths bα fixed. This indi- 2 2 2 2 cates a tendency of particles to phase separate in position h2 − h3 h2 − h1 Θ ≡ h1 + 2h2 + h3 + + . (121) space in NEQ, which is indeed obtained in mixtures of h1 h3 particles that are coupled to different heat baths [31–34]. Inserting these results into the expression for the entropy A similar calculation for the mean-squared velocity dif- production Eq. (98) we obtain ference based on Eqs. (110) - (112) gives ˙◦ 1 ◦ 1 ◦ β + β  mh2 β − β  γ − γ  Sres/kB = h1hx2v1i (β1 − β2) + h3hx2v3i (β3 − β2) h(v −v )2i◦ = 1 2 1 − 12 1 2 1 2 2 2 1 2 2 mβ1β2 hγ1γ2 β1 + β2 γ1 + γ2  s s ! (116) 1 β1 β3 = h1h3 − (122) to order m0 and where we used the simplifications m = 12γ(h1 + h3) β3 β1 m = m and h = h = h. Also here we see that in 1 2 11 22 s s !2 the limit β1/β2  1 and γ1/γ2  1 (or β2/β1  1 and β β +h (3h + h ) 1 − 2 γ2/γ1  1) the momentum difference between the par- 1 1 3 β2 β1 ticles goes down. This is indicative of co-localization in momentum space, meaning that different particles tend s s !2  β3 β2 to move with the same velocity. For a Newtonian Hamil- +h3(3h3 + h1) −  . tonian Eq. (88) which is diagonal in momentum space β2 β3 15

Obviously, the entropy production is positive and finite which for equal elastic coupling strengths h1 = h3 sim- in the zero mass limit if the reservoir temperatures are plifies to different and if the coupling strengths h1 and h3 are finite T T and positive. 3 > 4/5 + 1 . (129) The limiting case of h3 = 0 is insightful: in this case, T2 5T2 particle 3 becomes decoupled and we are basically left with only two coupled particles. The expression Eq. As expected, the conditions Eqs. (126) and (128) are (122) simplifies to simultaneously satisfied, i.e. heat flows into reservoir 1 and out of reservoir 3, for s s !2 ˙◦ h1 β1 β2 Sres/kB = − (123) T1 < T2 < T3, (130) 4γ β2 β1 i.e., per unit time the heat amount |Q˙ ◦|, flows from par- which describes the stationary entropy production of two 3 ticle 3 (which is coupled to the hot heat reservoir) to reservoirs of inverse temperatures β and β that act on 1 2 particle 2 (which is coupled to the warm reservoir), and two particles that are subject to friction with coefficients ˙ ◦ γ and which are coupled by a harmonic spring of strength the heat amount |Q1| flows from particle 2 to particle 1 (which is coupled to the cold reservoir). h1. Obviously, also this entropy production is never neg- ative and finite if the reservoir temperatures are different More interestingly, the conditions Eqs. (126) and (128) can also be simultaneously satisfied for the scenario and if the coupling strength h1 is finite. The case of three particles that are coupled to heat T < T < T . (131) reservoirs at three different temperatures allows to 2 1 3 demonstrate as an interesting NEQ effect the pumping In this case there is a small range of temperatures where of heat against a temperature gradient. Before we go heat flows from particle 3 (coupled to the hot heat reser- into the analysis, we note that this does not constitute a voir) to particle 2 (coupled to the cold reservoir), which is violation of the second law of thermodynamics but rather expected, but at the same time a heat amount |Q˙ ◦| flows is a NEQ entrainment effect. To proceed, we calculate 1 from particle 2 (coupled to the cold reservoir) to particle the stationary heat flux from the heat reservoir at tem- 1 (coupled to the warm reservoir). A similar scenario is perature T to particle 1, which according to Eqs. (95) 1 provided for and (119) is given by ˙ ◦ ◦ ◦ Q1 = − h1hx2v1i = h1hx1v2i T1 < T3 < T2, (132) h i = h1 2h3+3h1 − h3 − h3+3h1 . (124) 6γ(h1+h3) β1 β3 β2 where for a small range of temperatures heat flows from particle 2 (coupled to the hot heat reservoir) to particle 1 The stationary heat flux from the heat reservoir at tem- (coupled to the cold reservoir), which is expected, but at perature T3 to particle 3 is according to Eqs. (95) and ˙ ◦ (118) given by the same time a heat amount |Q3| flows from particle 3 (coupled to the warm reservoir) to particle 2 (coupled to ˙ ◦ ◦ Q3 = − h3hx2v3i the hot reservoir). This means we find situations where h i heat flows against the temperature gradient of the heat = − h3 h1 + h1+3h3 − 2h1+3h3 . (125) 6γ(h1+h3) β1 β2 β3 reservoirs that are coupled to the particles. This situa- tion is indicated in Fig. 2b) in a phase diagram for equal Due to energy conservation Q˙ ◦ = −Q˙ ◦ − Q˙ ◦ holds. Note 2 1 3 elastic coupling h = h , in which case the simplified that all stationary heat fluxes Q˙ ◦, Q˙ ◦, Q˙ ◦ obviously van- 1 3 1 2 3 inequalities Eqs. (127) and (129) hold. In the phase dia- ish in equilibrium, i.e. when the temperatures of all heat gram the inequalities Eqs. (127) and (129) are indicated reservoirs are the same. The flux from heat reservoir 1 by straight lines, and the region where Eq. (130) holds according to Eq. (124) is negative, Q˙ ◦ < 0, i.e. heat 1 is indicated in green. The regions where the inequalities flows into reservoir 1, for Eqs. (127) and (129) and in addition Eq. (131) or Eq. T3 T1 (132) hold are indicated in red. Our discussion uses the > −1 − 3h1/h3 + (2 + 3h1/h3) , (126) ˙ ◦ T2 T2 fact that any heat Q1 that is transferred to or from reser- voir 1 is transferred between particles 1 and 2 via elastic which for equal elastic coupling strengths h1 = h3 sim- ˙ ◦ plifies to interactions, likewise, any heat Q3 that is transferred to or from reservoir 3 is transferred between particles 2 and T T 3 > −4 + 5 1 . (127) 3. In other words, we do not only know the stationary T2 T2 heat fluxes from the reservoirs but also the stationary energy fluxes between the particles, which is due to the The flux from heat reservoir 3 according to Eq. (125) is ˙ ◦ simple linear topology of the elastic particle interactions positive, Q3 > 0, i.e. heat flows out of reservoir 3, for as indicated in Fig. 2a). T h + 3h h T As mentioned before, the finding of a heat flux against 3 > 1 3 + 1 1 , (128) T2 2h1 + 3h3 2h1 + 3h3 T2 the reservoir temperature gradient does not violate the 16 second law of thermodynamics. Firstly, the heat flux where θ(t) denotes the Theta function with the properties from the particle coupled to the cold heat bath to the par- θ(t) = 1 for t > 0 and θ(t) = 0 for t < 0, which makes ticle coupled to the warm heat bath is accompanied by an the memory function single-sided. The noise FR(t) in Eq. even larger heat flux from the particle coupled to the hot (135) is given by heat bath to the particle coupled to the warm heat bath, t h Z 0 our result for the total entropy production Eq. (122) p 1 0 −(t−t )h1/γ 0 FR(t) = γ/β1F1(t) + √ dt e F2(t ) is strictly positive. Secondly, heat is not transferred di- β2γ −∞ rectly between the heat reservoirs but only between par- (137) ticles that are coupled to heat reservoirs. In this connec- and consists of the noise acting directly on the first parti- tion it is crucial to remember (according to our previous cle and a term due to noise acting on the second particle. discussion centered around the covariance elements Eqs. The latter term consists of a convolution integral because (107) - (113)) that particles are not characterized by the this noise is transmitted via the elastic bond of strength temperature of the heat reservoir they are coupled to. h1. Defining the auto-correlation function of the random Therefore, since the particles do not have well-defined ef- noise as CFF(t) = hFR(0)FR(t)i, we obtain [19] fective temperatures, the second law of thermodynamics −|t|h1/γ is not violated. One could be tempted to define effec- β1CFF(t) = 2γδ(t) + h1(β1/β2)e . (138) tive temperatures based on the heat fluxes, but such a Comparing Eq. (136) and Eq. (138), we see that definition would be based solely on one entry of the co- β C (t) = Γ(|t|), a consequence of the standard variance matrix, namely the off-diagonal position velocity 1 FF ◦ fluctuation-dissipation theorem [86], only holds for β1 = coupling hxαvβi , and not work for other applications. β2, i.e., if the two reservoir temperatures are the same. If β1 6= β2, the memory kernel Γ(t) and the random force correlation function C (t) differ, which points to FDT C. Mapping on active particles FF violation and thus to the presence of an active NEQ pro- cess. The equation of motion Eq. (135) is similar to pre- Similar to recent calculations[70], we here show how viously studied active particle models that were shown active particle models can be described within the current to exhibit NEQ phase transitions [93, 94]. In fact, the framework of Markovian coupled particles. We want to active Ornstein-Uhlenbeck model is obtained by setting describe a single active particle, for this we reduce the the exponential in the memory function Eq. (136) to zero Hamiltonian Eq. (117) to two coupled massive particles while keeping the exponential term in the noise correlator and obtain Eq. (138). The entropy production of such active parti- cle models has been intensely studied and debated [95– h1 2 m1 2 m2 2 H = (x1 − x2) + v + v . (133) 98]. The entropy production of the present active parti- 2 2 1 2 2 cle model, defined by Eq. (135) in conjunction with Eqs. By assuming the friction coefficients to be the same, i.e. (136) and (138), is given exactly by the simple expression γ1 = γ2 ≡ γ, and choosing two different heat bath tem- Eq. (123); alternatively, it can be obtained directly from peratures β1 and β2, we obtain from Eqs. (86), (87), simulations by evaluating the position-velocity correla- (88), (90) the coupled set of linear Langevin equations tion functions in the general expression Eq. (98). The p advantage of the present active particle model, which fol- m1 v˙1(t) = −γv1(t) − h1[x1(t) − x2(t)] + γ/β1F1(t), lows from a Newtonian Hamiltonian, is that the extremal p and stability conditions in terms of the free entropy func- m2 v˙2(t) = −γv2(t) − h1[x2(t) − x1(t)] + γ/β2F2(t). tional derived in this work are valid and not only describe The Langevin equation for the second particle is straight- the stationary NEQ distribution itself but also the ap- forwardly solved in the mass-less limit m2 = 0 and gives proach to the stationary NEQ distribution. We hasten to add that not all active particle models can be mapped on t   Z 0 h 0 −(t−t )h1/γ 1 0 −1/2 0 our model, in particular models with non-Gaussian veloc- x2(t) = dt e x1(t ) + (β2γ) F2(t ) . −∞ γ ity distributions are not captured by our linear equations. (134) In future work interacting active particles similar to the Inserting this solution into the Langevin equation for the model derived here will be studied analytically as well as first particle, we obtain the generalized Langevin equa- in simulations, it will be interesting to see whether the tion NEQ position and momentum localization effects demon- strated in Sect. V A also show up in those more complex Z ∞ 0 0 0 systems. m1v˙1(t) = − dt Γ(t − t )v1(t ) + FR(t). (135) −∞

The memory function that appears in Eq. (135) is given VI. CONCLUSIONS by h i In this paper we consider the approach of Hamiltonian −th1/γ Γ(t) = θ(t) 2γδ(t) + h1e (136) many-body systems that are coupled to multiple heat 17

T1 T2 T3 a) h1 h3

T3/T2 T2< T1 < T3 b)

T1< T2 < T3 ∘ �̇ > 0 ∘ 1 �̇ < 0

T1< T3 < T2

∘ �̇ < 0 ∘ �̇ > 0 0 T1/T2 0 1

FIG. 2. a) Schematic representation of three particles that interact elastically and that are coupled to heat baths with different temperatures. b) Stationary heat flux diagram for symmetric elastic couplings h1 = h3, showing a colored region where heat ˙ ◦ ˙ ◦ flows from particle 3 to particle 2, i.e. Q3 > 0, and heat flows from particle 2 to particle 1, i.e. Q1 < 0. The green area denotes the temperature range T1 < T2 < T3 and the red areas denote T2 < T1 < T3 and T1 < T3 < T2. The green area is the expected temperature range where heat flows from particle 3 coupled to the hot reservoir at T3 to particle 2 coupled to the warm reservoir at T2 and from particle 2 coupled to the warm reservoir at T2 to particle 1 coupled to the cold reservoir at T1. In the red areas heat flows between the particles against the temperature gradients of the heat reservoirs: In the upper red area heat flows from particle 2 coupled to the cold reservoir at T2 to particle 1 coupled to the warm reservoir at T1, in the lower red area heat flows from particle 3 coupled to the warm reservoir at T3 to particle 2 coupled to the hot reservoir at T2. reservoirs with different temperatures to stationary NEQ expression distributions. Based on the exactly calculated approach of the covariance matrix E(t) to its stationary NEQ form ◦−1 ˙ ◦ S˙tot(t) S˙(t) Eij Eij(t) E , we construct the functional Sfree(t) that yields the = − − E◦−1hz˙ (t)ihz (t)i + S˙◦ , ◦ ij i j res stationary covariance matrix E at its extremum with kB kB 2 respect to variations of E(t). This function is called the (140) free entropy, and it consists of the system distribution which consists of the system distribution entropy pro- ˙ entropy S(t) and a term that accounts for interactions duction S(t), the stationary reservoir entropy production ˙◦ within the system, described by the Hamiltonian, and Sres, and two terms that account for interactions within the frictional and noise coupling to the heat reservoirs. the system as well as the frictional and noise coupling to the environment. NEQ effects make these coupling terms Since in the stationary state the free entropy produc- differ dramatically from their equilibrium counterparts, tion by construction vanishes, as explained in Section in which case E◦−1 becomes replaced by the Hamiltonian IV A, the difference between the free entropy production ij and the total entropy production is the reservoir entropy matrix Hij/(kBT ). In Appendix H we attempt to derive production in the stationary state, S˙◦ , which is constant Eq. (139) explicitly by calculating the time-dependent res ˙ in time. The total entropy production S˙ (t), which is reservoir entropy production Sres(t) from our explicit ex- tot pressions for the time-dependent heat fluxes between the the sum of the system entropy production S˙(t) and the ˙ heat reservoirs and the system. We demonstrate that the reservoir entropy production Sres(t), follows from Eq. (1) heat fluxes are by themselves not sufficient to derive the by differentiation as reservoir entropy production S˙res(t) if the reservoir tem- peratures are not the same, which implies that internal ˙ ˙ ˙ ˙ ˙◦ Stot(t) = S(t) + Sres(t) = Sfree(t) + Sres. (139) reservoir degrees of freedom contribute in a nonneglible manner to the reservoir entropy production for a NEQ system. The connection between the free entropy pro- Using the result in Eq. (78) we obtain the explicit duction S˙free(t) (excluding the reservoirs) and the total 18 entropy production S˙tot(t) (including the reservoirs) will tion program (grant agreement No. [835117]) and from be reconsidered in future work using microscopic models the Infosys Foundation. for the heat reservoirs. Appendix A: Derivation of Shannon entropy It is important to note that the reservoir entropy pro- ˙◦ duction in the stationary state, denoted as Sres and given We start from the canonical partition function explicitly in Eq. (98), does not depend on the time- Z dependent covariance matrix E(t) but only on the sta- −βH(~z) tionary covariance matrix E◦, so it is a constant with Z = d~ze (A1) respect to variations in E(t); thus, the total entropy pro- duction S˙tot(t) exhibits the identical extremal and sta- where β = 1/(kBT ) denotes the inverse thermal energy. bility properties as the free entropy production S˙free(t). Using the thermodynamic definition of the free energy It trivially follows from Eq. (140) that the total entropy F = −kBT ln Z we can write Stot(t) of a NEQ system increases indefinitely with time, Z −βH(~z) as shown explicitly in Eq. (1), and thus is not bounded, F = −kBT ln d~ze . (A2) whereas the free entropy Sfree(t) is a well-defined and fi- nite expression even for NEQ systems. Expression Eq. (140) will be useful for various applications whenever the From this we obtain, using the thermodynamic definition total entropy production of different NEQ states of a sys- S = −∂F/∂T , for the entropy tem need to be compared. The functional Eq. (140) Z −1 −1 −βH(~z) should also allow to construct approximate methods for S = kB ln Z + Z T d~zH(~z)e . (A3) the description of NEQ non-linear systems as well as for NEQ phase transitions. From the definition of the normalized equilibrium distri- One advantage of the current formulation is that in the bution Eq.(22), we obtain limit when all heat reservoir temperatures become equal and the NEQ system transforms into an equilibrium sys- βH(~z) = − ln ρ(~z) − ln Z, (A4) tem, the NEQ free entropy smoothly crosses over to the equilibrium free energy divided by −T , which displays which is inserted into Eq.(A3) to give the standard equilibrium extremal and stability proper- Z ties of a canonical system. We thus have derived a uni- −1 −βH(~z) fied framework to treat NEQ systems that are coupled S = −kBZ d~ze ln ρ(~z). (A5) to heat reservoirs at different temperatures on the same footing as equilibrium systems. We have in our work re- Using again the definition of the normalized equilibrium stricted ourselves to one specific class of NEQ models, distribution Eq.(22), we finally obtain the Shannon ex- namely where NEQ is produced by stochastic forces with pression for the entropy as vanishing mean. Forces with a non-vanishing mean are Z straightforward to include and will be treated in future S/k = − d~zρ(~z) ln ρ(~z). (A6) work. B Our approach rests on the harmonic approximation for the interaction Hamiltonian and for the friction and stochastic terms. It is not clear how to extend the present Appendix B: The time dependent probability derivation to non-linear systems, here variational and distribution is Gaussian perturbative methods will most likely be helpful. Clearly, a harmonic model can always be obtained from a more Here we show that the time-dependent distribution complex, non-linear model by a saddle-point expansion function is Gaussian and governed by the time-dependent in terms of suitably defined deviatory coordinates. Our inverse covariance matrix. The calculation holds for equi- model should thus also apply to non-linear systems as librium as well as for NEQ systems. Given a solution ~z(t) long as this saddle-point approximation is justified. of the Langevin Eq. (13), we construct the time depen- dent probability distribution as

ACKNOWLEDGMENTS ρ(z~0, t) = ~δ(z~0 − ~z(t)). (B1)

We acknowledge funding from the DFG via the SFB Using the Fourier representation of the delta function and 1114, from the European Research Council (ERC) under the time-dependent solution Eq. (14) for given initial the European Unions Horizon 2020 research and innova- value, we obtain 19

t Z d~ω   Z 0  ~0 ~ 0 0 −(t−t )A 0 ρF [z , t, F (·)] = 2N exp −ıωizi + ıωi hzi(t)i + dt eij ΦjkFk(t ) , (B2) (2π) 0 which is a functional of the random force trajectory F~ (t). The probability distribution is obtained by a path integral over all random force trajectories as Z 0 0 ρ(z~ , t) = DF~ (·)P [F~ (·)]ρF [z~ , t, F~ (·)]. (B3)

Here P (F~ (·)) is the path integral weight, which for diagonal white noise is given by

 1 Z  P [F~ (·))] = N −1 exp − dtdt0F (t)δ(t − t0)F (t0) (B4) F 4 k k

0 0 and leads to hFi(t)i = 0 and hFi(t)Fj(t )i = 2δijδ(t − t ), where the normalization factor is given by NF . The path integral over the random noise in Eq.(B3) can be performed and leads to the expression

Z d~ω ρ(z~0, t) = exp {−ıω z0 + ıω hz (t)i − ω E (t)ω } , (B5) (2π)2N i i i i i ij j where the covariance matrix Eij(t) is given by Eq.(17). Performing the integral over ~ω leads to

~0 −1  0 −1 0 ρ(z , t) = N (t) exp −[zi − hzi(t)i]Eij (t)[zj − hzj(t)i]/2 , (B6) with N given by Eq. (25), which is identical to the expression given in Eq. (24). Since the mean state vector hzi(t)i in Eq. (B6) depends according to Eq. (15) on the initial state vector zi(0), the time-dependent probability distribution we derived here is in fact the conditional distribution and thus corresponds to the Green’s function, this can be made explicit by rewriting Eq. (B6) as n o ~0 −1 0 −tA −1 0 −tA ρ(z , t|~z, 0) = N (t) exp −[zi − eik zk]Eij (t)[zj − ejl zl]/2 . (B7)

Appendix C: Semi-positive definiteness of matrix where in the last step we used that E is symmetric and E E product trace P is thus an orthogonal matrix. The matrix Di0j0 = E E δi0jdi0 is diagonal with diagonal elements di0 . We thus We start from the part of the expression Eq. (64) for obtain the free energy production rate of an equilibrium system ˙ E that involves the trace of a product of four matrices, kBT F = −CkmMmlPliDijPnjMnk. (C4) E F˙ = −C [H −k TE−1] lj [H −k TE−1], (C1) We next define N = M P and obtain, by using that km ml B ml k T jk B jk mi ml li B M is symmetric, where we have for simplicity omitted all time dependen- ˙ E cies. An analogous expression also appears in the free kBT F = −CkmNmiDijNkj. (C5) entropy production in Eq. (79). The matrix trace ex- pressions that appear in the free energy Eq. (34) and We next diagonalize the matrix C by the similarity trans- the free entropy Eq. (77) are special cases of the more formation general matrix product trace we consider here. −1 C C−1 C C C−1 C C C,T By defining the matrix Mml = Hml − kBTEml , the CP P = P D P = P D P , (C6) expression can be written more compactly as C C where Di0j0 = δi0jdi0 and obtain kBT F˙ = −CkmMmlEljMjk (C2) where all matrices C,M,E are symmetric and assumed ˙ C C C E kBT F = −Pkl DloPmoNmiDijNkj. (C7) to be non-defective, E is positive definite, C is semi- positive definite and the definiteness of M (since it is C We now define Roi = PmoNmi, which is equivalent to the difference of two matrices) is not specified. We first R = N P C , and thus obtain diagonalize the matrix E by the similarity transformation lj kj kl E E−1 E E E−1 E E E,T ˙ C E EP P = P D P = P D P (C3) kBT F = −DloRoiDijRlj. (C8) 20

Now using that DC and DE are diagonal matrices we the covariance matrix elements according to the two sides obtain of the detailed-balance condition Eq. (D3) follow explic- itly as ˙ C E C E 2 kBT F = −dl Rlidi Rli = −dl di Rli ≤ 0, (C9) ls −tA −tA ◦ hδoδpi = Eop(t) + (δok − eok )(δpl − epl )Ekl (D6) where the inequality follows since the matrix elements of R are real and all eigenvalues of the matrices C and E and are not negative. rs hδoδpi = WoiEij(t)Wjp (D7) −tA −tA ◦ + (δok − Wolelm Wmk)(δpn − Wpqeqr Wrn)Ekn. Appendix D: Condition of detailed balance Since a Gaussian distribution is completely specified by its covariance matrix, detailed balance is satisfied if Detailed balance is satisfied if the probability for a hδ δ ils = hδ δ irs holds. Similar to previous treatments transition from a state vector ~z at time t to a state vector o p o p 0 [5, 99, 100], we expand the detailed-balance condition in ~z at time t+τ is the same as the probability for a transi- time. From Eq. (17) we obtain tion from ~z0 at time t to ~z at time t+τ, provided that all 2 3 velocities are reversed. Detailed balance is satisfied for Eij(t) = 2tCij − t (AikCkj + AjkCki) + O(t ). (D8) systems that are in equilibrium and is standardly used as the definition of equilibrium [87]. In this section we To second order in t we thus obtain will demonstrate that the condition of detailed balance ls 2 ◦ is equivalent to the condition based on the Boltzmann hδoδpi = 2tCop + t (AokEklApl − AokCkp − ApkCko) distribution, provided that the friction matrix is sym- (D9) metric and that the stochastic field correlations satisfy and certain symmetry relations. Our calculation is similar to rs hδoδpi = 2tWoiCijWjp (D10) classical derivations [5, 8, 99, 100]. 2 ◦ Using the joint probability distribution, the detailed + t Wol(AlmWmnEntWtsArs − AlkCkr − ArkCkl)Wrp. balance condition can be written as [5, 99, 100]

ρ(~z0, t; ~z, 0) = ρ(W~z, t; W~z0, 0) (D1) From enforcing the detailed-balance condition term by term in powers of t, we obtain the two conditions where W is the diagonal matrix that reverses all velocities and is given by Cop = WoiCijWjp (D11)

1 0 0 0  and 0 −1 0 0 ◦   AokE Apl − AokCkp − ApkCko (D12) 0 0 1 0  kl W =   . (D2) = W (A W E◦ W A − A C − A C )W , 0 0 0 −1  ol lm mn nt ts rs lk kr rk kl rp   . .. which are consistent with previous results [5, 99, 100]. To The detailed-balance condition can be brought into its evaluate the conditions Eqs. (D11) and (D12), it is useful standard form by using the conditional probability, re- to express all matrices as products of particle matrices sulting in and 2 × 2 submatrices as introduced in Sect. IV B. With this, the matrix W in Eq. (D2) can be written as ρ(~z0, t|~z, 0)ρ◦(~z) = ρ(W~z, t|W~z0, 0)ρ◦(~z0), (D3) Wαβ = δαβ(s − r), (D13) where where s and r are 2 × 2 matrices defined in Eq. (83) and ◦ ◦−1  ◦−1 ρ (~z) = ρ(~z, t → ∞) = N exp −ziEij zj/2 (D4) where greek indices number particles. To test the conse- quences of condition Eq. (D11), we expand the random is the stationary distribution and accordingly the time correlation matrix in the complete set of 2×2 submatrices argument is omitted. The stationary normalization con- according to stant follows from Eq. (25) as N ◦ = p(2π)2N det E◦. 0 s r ut us The conditional probability distribution ρ(~z , t|~z, 0), Cαβ = sCαβ + rCαβr + utCαβ + usCαβ. (D14) which is equivalent to the Green’s function of the stochas- tic problem defined by the Langevin or the Fokker-Planck Inserting Eqs. (D13) and (D14) into Eq. (D11), we ob- equation, is explicitly given in Eq. (B7) and is valid for tain by using the matrix product properties Eq. (85) that ut us equilibrium as well as for non-equilibrium systems. With Cαβ = 0 = Cαβ, i.e. the random correlation matrix can- the definition not contain components that couple momenta and po- sitions. We will later see that this condition is indeed ~z0 = ~z + ~δ, (D5) satisfied for a large class of Hamiltonian models. 21

To evaluate the consequences of the condition Eq. not satisfied, even if the system obeys the Boltzmann (D12), we need to specify the symmetry of the Hamil- distribution. If the friction matrix γαβ is symmetric, tonian model. As in Eq. (88), the Hamiltonian matrix the terms proportional to ru and ur disappear and the shall describe a Newtonian system where positions and detailed-balance condition is satisfied. We conclude that momenta are decoupled and furthermore momentum con- for a symmetric friction matrix the condition of detailed tributions are diagonal in the particles indices, balance and the definition of equilibrium we use in the main text, based on the Boltzmann distribution, are Hα0 = shα0 + rδα0/mα0 . (D15) equivalent. Since none of the effects we study in this paper depend on the presence of asymmetries in the fric- The friction matrix is copied from Eq.(90) and only acts tion matrix, it is permissible and convenient to use the on momentum degrees of freedom, Boltzmann definition for equilibrium, which we primarily do since it is much easier to implement. The presence of Γ 0 = rγ 0 /m 0 , (D16) α α  an asymmetric friction matrix means that the principle in the following we explicitly allow the matrix γα0 also of equal actio and reactio is broken on the level of the to be asymmetric. Using the inverse Hamiltonian from Langevin equation, this therefore constitutes a distinct Eq. (89) and Eq. (D16), the Lyapunov-Boltzmann con- NEQ scenario. dition Eq. (37) yields the equilibrium random correlation matrix Appendix E: Condition of vanishing probability 2Cα/(kBT ) = (γα + γα)r, (D17) current for overdamped particle dynamics which only acts on momentum degrees of freedom. We In Sect. III D we show that the probability current is conclude that the first detailed-balance condition Eq. generally nonzero for underdamped particles. Here we (D11) is automatically satisfied for a Newtonian system consider overdamped particle motion and show that the that is described by the Boltzmann distribution and thus probability current vanishes only for a symmetric fric- satisfies the Lyapunov-Boltzmann condition Eq. (37). tion matrix. To procced, we rewrite the equation of mo- We next probe condition Eq. (D12). According to tion expressed in terms of particle coordinates yα(t) = the definition used in the main text, in equilibrium (x (t), p (t))T , Eq. (86), as a second-order differential ◦ • −1 α α Ekl = Ekl = kBTHkl holds and thus the stationary equation as ◦ covariance matrix Ekl does not couple momentum and position coordinates, therefore condition Eq. (D12) sim- mα0 x¨α0 (t) = −γα0βx˙ β(t) − hα0βxβ(t) + φα0βFβ(t), (E1) plifies to where the Hamiltonian matrix hα0β only acts on positions A E• AT − A C − CT AT (D18) ok kl lp ok kp ok kp and the friction matrix γα0β only acts on momenta. φα0β • T T T is the N × N random coupling strength matrix. Setting = Wol(AlmEmsAsr − AlkCkr − ClkAkr)Wrp, all masses mα to zero, we obtain

−1 −1 where the superindex T denotes the transpose of the ma- x˙β(t) = −γαβ hβγ xγ (t) + γαβ φβγ Fγ (t), (E2) trix including the 2 × 2 submatrices. Equation (D18) holds if the left side is a matrix that does not couple which we can rewrite in a form similar to Eq. (13) as momenta and positions. From the expression for A, Eq. od od x˙ α(t) = −Aαβxβ(t) + ΦαβFβ(t), (E3) (87), and Uαγ = uδαγ we obtain where the overdamped versions of A and Φ are N ×N ma- Aα00 = −ushα00 − urδα00 /mα0 + rγα00 /m0 . (D19) od −1 od −1 trices given by Aαγ = γαβ hβγ and Φαγ = γαβ φβγ . The • −1 od od od Replacing Ekl by the explicit result for Hkl from Eq. overdamped version of Eq. (18) is Ckl = ΦkmΦlm. The (89), and using the results for C and A from Eqs. (D17) overdamped version of the Lyapunov-Boltzmann equa- and (D19) we finally obtain for the left hand side of Eq. tion Eq. (37) turns out to be

(D18) od −1 −1 2Cαβ/(kBT ) = γ + γ (E4) −1 αβ βα rhα0β0 + sδα0β0 mβ0 (D20) −1 and constitutes the relation between the random strength + r [(γ 0 − γ 0 )γ 0 + (γ 0 − γ 0 )γ 0 ] m /2 α α  β  β β  α   matrix and the friction matrix for overdamped systems. −1 −1 + ru(γα0β0 − γβ0α0 )mβ0 /2 + ur(γα0β0 − γβ0α0 )mα0 /2. The overdamped version of the condition of vanishing probability current follows from Eq. (60) as

od −1 For an asymmetric friction matrix γαβ, this expression Cαβ/(kBT ) = γαβ (E5) contains terms that couple positions and momenta, these od are the two terms proportional to ru and ur, and there- and, since Cαβ is symmetric by construction, can only be fore the second detailed-balance condition Eq. (D12) is satisfied if the friction matrix γαβ is symmetric. Thus, if 22

γαβ is not symmetric, the probability current is non-zero yet that exhibit non-zero probability currents in the sta- even if the distribution is given by the Boltzmann dis- tionary state. tribution. We conclude that the condition of vanishing probability current and the equilibrium definition we use in this paper, namely that the stationary distribution is Appendix F: Fluctuation-dissipation theorem given by the Boltzmann distribution, see Eqs. (22) and (31), are not equivalent even in the overdamped limit. Here we demonstrate for a specific two-particle model In fact, the condition of vanishing probability current with an asymmetric friction matrix, that the fluctuation- is more restrictive since it precludes asymmetric friction dissipation theorem can be satisfied even when the sys- matrices, which is not necessary to obtain Boltzmann dis- tem, as demonstrated in Appendix D, does not obey de- tributions. Thus, there is a class of overdamped particle tailed balance. To proceed, we consider a system of two models that exhibit stationary Boltzmann distributions massive particles that are coupled by a general friction matrix. Following the notation of Eq. (E1) we write

m1 x¨1(t) = −γ1x˙ 1(t) − γ12x˙ 2(t) + φ1F1(t) + φ12F2(t), (F1)

m2 x¨2(t) = −γ2x˙ 2(t) − γ21x˙ 1(t) + φ2F2(t) + φ21F1(t). (F2) Note that for ease of calculation we omit any positional couplings between the particles. Similar to the active-particle model in Sect. V C, where the two particle positions are coupled, the Langevin equation for the second particle can be solved and gives t   Z 0 γ φ φ 0 −(t−t )γ2/m2 21 0 2 0 21 0 x˙ 2(t) = dt e − x˙ 1(t ) + F2(t ) + F1(t ) . (F3) −∞ m2 m2 m2 Inserting this solution into the Langevin equation for the first particle, we obtain the generalized Langevin equation Z ∞ 0 0 0 m1x¨1(t) = − dt Γ(t − t )x ˙ 1(t ) + FR(t). (F4) −∞ The memory function that appears in Eq. (F4) is given by

 γ γ  12 21 −tγ2/m2 Γ(t) = θ(t) 2γ1δ(t) − e , (F5) m2 where θ(t) denotes the Theta function with the properties θ(t) = 1 for t > 0 and θ(t) = 0 for t < 0, which makes the memory function single-sided. The noise FR(t) in Eq. (F4) is given by t   Z 0 γ φ γ φ 0 −(t−t )γ2/m2 12 2 0 12 21 0 FR(t) = φ1F1(t) + φ12F2(t) − dt e F2(t ) + F1(t ) . (F6) −∞ m2 m2

The auto-correlation function of the random noise CFF(t) = hFR(0)FR(t)i follows as

2 2 2 2 2 −|t|γ2/m2 CFF(t) = 2(φ1 + φ12)δ(t) − [2γ12(φ2φ12 + φ1φ21) − γ12(φ2 + φ21)/γ2]e . (F7)

Defining the Langevin equation in analogy to Eq. (86) in terms of the particle momenta pα0 (t) = mα0 vα0 (t) as p˙α(t) = −Aαβpβ(t)+φαβFβ(t) where Aαβ0 = γαβ0 /mβ0 and Hαβ0 = δαβ0 /mβ0 , we obtain from Eq. (36) the Boltzmann- Lyapunov equation

2Cαβ = 2φkmφlm = kBT (γαβ + γβα), (F8) or, explicitly,

 2 2    φ1 + φ12 φ1φ21 + φ2φ12 2γ1 γ12 + γ21 2 2 2 = kBT . (F9) φ1φ21 + φ2φ12 φ2 + φ21 γ12 + γ21 2γ2

Provided Eq. (F9) holds, it is easy to see that Eqs. (F5) [86] and Eq. (F7) satisfy the fluctuation-dissipation theorem CFF(t) = kBT Γ(|t|), (F10) 23 even if the friction matrix γαβ is asymmetric. In con- Appendix H: Non-stationary entropy production of trast and as shown in Appendix D, the more restrictive heat reservoirs detailed-balance condition is only satisfied if the friction matrix γαβ is symmetric. This suggests that the defi- We start from the expression for the time-dependent nition of equilibrium we use in this paper, namely that entropy production of all reservoirs Eq. (96), which is the stationary distribution corresponds to the Boltzmann based on the result for the heat fluxes between the reser- distribution, coincides with the fluctuation-dissipation voirs and the system in Eq. (95). We recall that the theorem even for asymmetric γαβ. On the other hand, calculation of the reservoir heat fluxes is based on the the detailed-balance condition is more restrictive and can general Newtonian Hamiltonian as defined by Eq. (88) only be satisfied if the friction matrix is symmetric. with diagonal momentum friction as defined by Eq. (90) and the NEQ model defined by the diagonal noise corre- lation matrix Eq. (93). Appendix G: Equivalenz of free entropy and Using the result for the time derivative of the covari- Kullback-Leibler entropy ance matrix Eq. (55) together with the explicit expres- sion for the A matrix Eq. (87) and the expansion of the covariance matrix E Eq. (99), we obtain for the time The multidimensional expression for the Kullback- derivative of the squared particle momenta Leibler entropy reads [6, 78] 2 dhp 0 (t)i 2γα0 ◦ α 2 Z ρ (~z) = 2Cα0α0 − hpα0 (t)i − 2hα0hpα0 xi. (H1) S = − d~zρ(~z) ln , (G1) dt mα0 KL ρ(~z) Inserting this into the total reservoir entropy production where the Gaussian probability distribution ρ(~z) and the Eq. (96) we obtain Gaussian stationary probability distribution ρ◦(~z) are S˙ (t) β γ hp2 (t)i 1  given by res = α α α − , (H2) kB mα mα βα ρ(~z) = N −1 exp −z E−1z /2
, (G2) i ij j where the non-primed index α is summed over. Clearly, in an equilibrium stationary state, where the kinetic en- 2 ergy of each particle obeys hpα(t)i/(mα) = 1/βα and ◦ ◦−1 ◦−1
ρ (~z) = N exp −ziEij zj/2 (G3) βα = β, the entropy production vanishes. In a station- 2 ary state we can replace hpα(t)i by the expression Eq. with the normalization constants N = p(2π)2N det E (103) and thereby recover Eq. (98), so our calculation thus far is consistent. We will now test whether the sta- and N ◦ = p(2π)2N det E◦. Note that we dropped all tionary NEQ distribution constitutes an extremum of the time dependencies for simplicity. Using the inequality total entropy production, which according to Eq. (139) ρ◦(~z) ρ◦(~z) is given by − ln ≥ 1 − (G4) ρ(~z) ρ(~z) S˙tot(t) = S˙(t) + S˙res(t), (H3) and the fact that ρ(~z) and ρ◦(~z) are normalized, it im- where S˙(t) denotes the Shannon entropy production of mediately follows that the system distribution Eq. (46) and S˙res(t) is given by Eq. (H2). As an explicit calculation shows, the derivative SKL ≥ 0 (G5) of the reservoir entropy production Eq. (H2) with respect to the covariance matrix gives ◦ and that SKL = 0 for ρ(~z) = ρ (~z). A short calcula- ◦ ˙ tion shows that also δSKL/δρ(~z) = 0 for ρ(~z) = ρ (~z), ∂Sres(t) −1 ◦ = ΓikCkmΓmj, (H4) thus, the Kullback-Leibler is minimal for ρ(~z) = ρ (~z). ∂Eij Performing the Gaussian integrals in Eq. (G1), one im- mediately obtains which using Eqs. (90) and (93) is a diagonal matrix with 2 momentum entries βα0 γα0 /mα0 . For the system entropy ◦−1  2N  production we obtain from Eq. (46) SKL = Eij Eij/2 − N − ln (2π) det E /2  2N ◦ + ln (2π) det E /2. (G6) ∂S˙(t) = −E−1(t)C E−1(t) (H5) ∂E ik km mj Comparison of the Kullback-Leibler entropy Eq. (G6) ij with the non-equilibrium free entropy expression Eq. and thus for the total entropy (70) shows that the two expressions are identical except ˙ the sign and terms that do not depend on covariance ∂Stot(t) −1 −1 −1 = ΓikCkmΓmj − Eik (t)CkmEmj(t). (H6) matrix elements Eij. ∂Eij 24

Multiplying by Cli we obtain therefore for an equilibrium system the total entropy pro- duction as defined in Eq. (H3) is extremal in the equi- librium state. ∂S˙ (t) C tot = Γ Γ − C E−1(t)C−1 E−1(t), (H7) It is easy to verify that for the stationary NEQ dis- li ∂E lk kj li ik km mj ◦ ◦ ij tribution Eki with non zero matrix elements hpipji and ◦ hxipji for i 6= j, and Γ and C given by Eqs. (90) and (93), Eq. (H9) is not satisfied. This implies that the to- which vanishes when tal entropy production Eq. (H3), which is the sum of the non-stationary reservoir entropy production Eq. (96) or −1 −1 −1 ΓlkΓkj = CliEik (t)CkmEmj(t) (H8) (H2) (calculated from the heat fluxes between reservoirs and the system) and the system distribution entropy pro- duction Eq. (46), does not yield the exactly calculated holds. Taking the square root and multiplying by E we ◦ stationary distribution Eki at its extremum and thus dif- obtain fers from the total entropy production constructed from the sum of the free entropy and the stationary reservoir Γ E (t) = C (H9) entropy production according to Eq. (139). We con- lk ki li clude that non-stationary entropy contributions that pre- sumably stem from internal reservoir degrees of freedom as the equation which determines the extremum of the render the expressions Eqs. (96) and (H2) incomplete. total entropy production. It turns out that that Eq. (H9) These internal reservoir degrees could be included by us- • is satisfied by the equilibrium stationary distribution Eki, ing microscopic models for the heat reservoirs.

[1] L. Onsager, Reciprocal relations in irreversible pro- 126001 (2012). cesses. i, Phys. Rev. 37, 405 (1931). [15] M. C. Marchetti, J. F. Joanny, S. Ramaswamy, T. B. [2] I. Prigogine, Etude thermodynamic des phenomene ir- Liverpool, J. Prost, M. Rao, and R. A. Simha, Hydrody- reversibles (Desoer, Liege, 1947). namics of soft active matter, Rev. Mod. Phys. 85, 1143 [3] I. Prigogine and P. Mazur, Sur l’extension de la thermo- (2013). dynamique aux phenomenes irreversibles lies aux degres [16] F. S. Gnesotto, F. Mura, J. Gladrow, and C. P. Broed- de liberte internes, Physica XIX, 241 (1953). ersz, Broken detailed balance and non-equilibrium dy- [4] J. L. Lebowitz, Stationary nonequilibrium gibbsian en- namics in living systems: a review, Rep. Prog. Phys.. sembles, Phys. Rev. 114, 1192 (1959). 81, 066601 (2018). [5] R. Graham and H. Haken, Generalized thermodynamic [17] C. Bechinger, R. Di Leonardo, H. L¨owen, C. Reichhardt, potential for markoff systems in detailed balance and far G. Volpe, and G. Volpe, Active particles in complex from thermal equilibrium, Z. Physik 243, 289 (1971). and crowded environments, Rev. Mod. Phys. 88, 045006 [6] F. Schl¨ogl, On stability of steadystates, Z. Physik 243, (2016). 303 (1971). [18] P. Illien, R. Golestanian, and A. Sen, ’fuelled’ motion: [7] I. Procaccia and R. D. Levine, Potential work: A phoretic motility and collective behaviour of active col- statistical-mechanical approach for systems in disequi- loids, Chem. Soc. Rev. 46, 5508 (2017). librium, J. Chem. Phys. 65, 3357 (1976). [19] R. R. Netz, Fluctuation-dissipation relation and sta- [8] J. Schnakenberg, Network theory of microscopic and tionary distribution of an exactly solvable many-particle macroscopic behavior of master equation system, Rev. model for active biomatter far from equilibrium, J. Mod. Phys. 48, 571 (1976). Chem. Phys. 148, 185101 (2018). [9] Y. Oono and M. Paniconi, Steady state thermodynam- [20] S. Katz, J. L. Lebowitz, and H. Spohn, Phase transi- ics, Progress of Theoretical Physics Supplement 130, 29 tions in stationary nonequilibrium states of model lattice (1998). systems, Phys. Rev. B 28, 1655 (1983). [10] T. Hatano and S. I. Sasa, Steady-state thermodynamics [21] J. Krug, Boundary-induced phase transitions in driven of langevin systems, Phys. Rev. Lett. 86, 3463 (2001). diffusive systems, Phys. Rev. Lett. 67, 1882 (1991). [11] M. Esposito, U. Harbola, and S. Mukamel, Entropy [22] B. Schmittmann and R. K. P. Zia, Driven diffusive sys- fluctuation theorems in driven open systems: Applica- tems. an introduction and recent developments, Phys. tion to electron counting statistics, Phys. Rev. E 76, Rep. 301, 45 (1998). 031132 (2007). [23] J. Dzubiella, G. P. Hoffmann, and H. L¨owen, Lane for- [12] S. Ramaswamy, The mechanics and statics of active mation in colloidal mixtures driven by an external field, matter, Annu. Rev. Condens. Matter Phys. 1, 323 Phys. Rev. E 65, 021402 (2002). (2010). [24] R. R. Netz, Conduction and diffusion in two- [13] T. Chou, K. Mallick, and R. K. P. Zia, Non-equilibrium dimensional electrolytes, Europhys. Lett. 63, 616 statistical mechanics: from a paradigmatic model to bi- (2003). ological transport, Rep. Prog. Phys. 74, 116601 (2011). [25] N. Kumar, H. Soni, S. Ramaswamy, and A. K. Sood, [14] U. Seifert, Stochastic thermodynamics, fluctuation the- Flocking at a distance in active granular matter, Nature orems and molecular machines, Rep. Prog. Phys. 75, Communications 5, 4688 (2014). 25

[26] E. Bertin, M. Droz, and G. Gr´egoire, Boltzmann effective temperatures in an active fluid of self-propelled and hydrodynamic description for self-propelled parti- particles, Europhys. Lett. 111, 26001 (2007). cles, Phys. Rev. E 74, 022101 (2006). [47] M. Kr¨ugerand M. Fuchs, Fluctuation dissipation rela- [27] I. Theurkauff, C. Cottin-Bizonne, J. Palacci, C. Ybert, tions in stationary states of interacting brownian parti- and L. Bocquet, Dynamic clustering in active colloidal cles under shear, Phys. Rev. Lett. 102, 135701 (2009). suspensions with chemical signaling, Phys. Rev. Lett. [48] D. Levis and L. Berthier, Driven activation vs. thermal 108, 268303 (2012). activation, Europhys. Lett. 79, 60006 (2015). [28] R. Golestanian, Collective behavior of thermally active [49] T. Harada and S. I. Sasa, Equality connecting energy colloids, Phys. Rev. Lett. 108, 038303 (2012). dissipation with a violation of the fluctuation-response [29] T. Speck, J. Bialke, A. M. Menzel, and H. L¨owen, Ef- relation, Phys. Rev. Lett. 95, 130602 (2005). fective cahn-hilliard equation for the phase separation of [50] F. Zamponi, F. Bonetto, L. F. Cugliandolo, and J. Kur- active brownian particles, Phys. Rev. Lett. 112, 218304 chan, A fluctuation theorem for non-equilibrium relax- (2014). ational systems driven by external forces, J. Stat. Mech. [30] F. Ginot, I. Theurkauff, D. Levis, C. Ybert, L. Boc- , P09013 (2005). quet, L. Berthier, and C. Cottin-Bizonne, Nonequilib- [51] J. Prost, J.-F. Joanny, and J. M. R. Parrondo, Gen- rium equation of state in suspensions of active colloids, eralized fluctuation-dissipation theorem for steady-state Phys. Rev. X 5, 011004 (2015). systems, Phys. Rev. Lett. 103, 090601 (2009). [31] A. Y. Grosberg and J. F. Joanny, Nonequilibrium statis- [52] M. Baiesi, C. Maes, and B. Wynants, Fluctuations and tical mechanics of mixtures of particles in contact with response of nonequilibrium states, Phys. Rev. Lett. 103, different thermostats, Phys. Rev. E 92, 032118 (2015). 010602 (2009). [32] S. N. Weber, C. A. Weber, and E. Frey, Binary mixtures [53] U. Seifert and T. Speck, Fluctuation-dissipation theo- of particles with different diffusivities demix, Phys. Rev. rem in nonequilibrium steady states, Europhys. Lett. 89, Lett. 116, 058301 (2016). 10007 (2010). [33] H. Tanaka, A. A. Lee, and M. P. Brenner, Hot parti- [54] L. Willareth, I. M. Sokolov, Y. Roichman, and B. Lind- cles attract in a cold bath, Phys. Rev. Fluids 2, 043103 ner, Generalized fluctuation-dissipation theorem as a (2017). test of the markovianity of a system, Europhys. Lett. [34] J. Smrek and K. Kremer, Small activity differences 118, 20001 (2017). drive phase separation in active-passive polymer mix- [55] S. Jabbari-Farouji, D. Mizuno, D. Derks, G. H. Weg- tures, Phys. Rev. Lett. 118, 098002 (2017). dam, F. C. MacKintosh, C. F. Schmidt, and D. Bonn, [35] S. S. N. Chari, C. Dasgupta, and P. K. Maiti, Scalar Effective temperatures from the fluctuation-dissipation activity induced phase separation and liquid-solid tran- measurements in soft glassy materials, Europhysics Let- sition in lennard-jones system, Soft Matter 15, 7275 ters 84, 20006 (2008). (2019). [56] J. R. Gomez-Solano, A. Petrosyan, S. Ciliberto, [36] A. Sokolov, I. S. Aranson, J. O. Kessler, and R. E. R. Chetrite, and K. Gawedzki, Experimental verifica- Goldstein, Concentration dependence of the collective tion of a modified fluctuation-dissipation relation for a dynamics of swimming bacteria, Phys. Rev. Lett. 98, micron-sized particle in a nonequilibrium steady state, 158102 (2007). Phys. Rev. Lett. 103, 040601 (2009). [37] R. Suzuki and A. R. Bausch, The emergence and tran- [57] J. Mehl, V. Blickle, U. Seifert, and C. Bechinger, sient behaviour of collective motion in active filament Experimental accessibility of generalized fluctuation- systems, Nature Communications 8, 41 (2017). dissipation relations for nonequilibrium steady states, [38] D. Helbing, Traffic and related self-driven many-particle Phys. Rev. E 82, 032401 (2010). systems, Rev. Mod. Phys. 73, 1067 (2001). [58] P. Martin, A. J. Hudspeth, and F. J¨ulicher, Compari- [39] C. W. Wolgemuth, T. R. Powers, and R. E. Goldstein, son of a hair bundle’s spontaneous oscillations with its Twirling and whirling: Viscous dynamics of rotating response to mechanical stimulation reveals the underly- elastic filaments, Phys. Rev. Lett. 84, 1623 (2000). ing active process, Proc. Nat. Acad. Sci. USA 98, 14380 [40] H. Wada and R. R. Netz, Plectoneme creation reduces (2001). the rotational friction of a polymer, Europhys. Lett. 87, [59] L. Dinis, P. Martin, J. Barral, J. Prost, and J.- 38001 (2009). F. Joanny, Fluctuation-response theorem for the active [41] G. S. Agarwal, Fluctuation-dissipation theorems for sys- noisy oscillator of the hair-cell bundle, Phys. Rev. Lett. tems in non-thermal equilibrium and applications, Z. 109, 160602 (2012). Physik 252, 25 (1972). [60] P. Bohec, F. F. Gallet, C. Maes, S. Safaverdi, P. Visco, [42] P. C. Hohenberg and B. I. Shraiman, Chaotic behavior and F. van Wijland, Probing active forces via a of an extended system, Physica D 37, 109 (1989). fluctuation-dissipation relation: Application to living [43] L. F. Cugliandolo, J. Kurchan, and L. Peliti, Energy cells, Europhys. Lett. 102, 50005 (2013). flow, partial equilibration, and effective temperatures in [61] D. Mizuno, C. Tardin, C. F. Schmidt, and F. C. MacK- systems with slow dynamics, Phys. Rev. E 55, 3898 intosh, Nonequilibrium mechanics of active cytoskeletal (1997). networks, Science 315, 370 (2007). [44] T. Speck and U. Seifert, Restoring a fluctuation- [62] J. L. Lebowitz and H. Spohn, A gallavotti cohen-type dissipation theorem in a nonequilibrium steady state, symmetry in the large deviation functional for stochastic Europhys. Lett. 74, 391 (2006). dynamics, Journal of Statistical Physics 95, 333 (1999). [45] N. Xu and C. S. O’Hern, Effective temperature in ather- [63] G. E. Crooks, Entropy production fluctuation theorem mal systems sheared at fixed normal load, Phys. Rev. and the nonequilibrium work relation for free energy dif- Lett. 94, 055701 (2005). ferences, Phys. Rev. E 60, 2721 (1999). [46] P. Ilg and J. L. Barrat, From single-particle to collective [64] C. Jarzynski, Hamiltonian derivation of a detailed fluc- 26

tuation theorem, Journal of Statistical Physics 98, 77 process: Equilibrium supercurrent and nonequilibrium (2000). driven circulation with dissipation, Eur. Phys. J. Spe- [65] U. Seifert, Entropy production along a stochastic tra- cial Topics 224, 781 (2015). jectory and an integral fluctuation theorem, Phys. Rev. [84] T. Speck and U. Seifert, Integral fluctuation theorem for Lett. 95, 040602 (2005). the housekeeping heat, J. Phys. A: Amth. Gen. 38, L581 [66] M. Doi, J. Zhou, Y. Di, and X. Xu, Application of the (2005). onsager-machlup integral in solving dynamic equations [85] R. Zwanzig, Memory effects in irreversible thermody- in nonequilibrium systems, Phys. Rev. E 99, 063303 namics, Physical Review 124, 983 (1961). (2019). [86] H. Risken, The Fokker-Planck Equation (Springer, [67] Z. Rieder, J. L. Lebowitz, and E. Lieb, Properties of a Berlin, 1984). harmonic crystal in a stationary nonequilibrium state, [87] S. R. de Groot and P. Mazur, Non-Equilibrium Thermo- J. Math. Phys. 8, 1073 (1967). dynamics (North-Holland Pub. Co., Amsterdam, 1962). [68] R. Zwanzig, Non linear generalized langevin equations, [88] A. Crisanti, A. Puglisi, and D. Villamaina, Nonequi- J. Stat. Phys. 9, 215 (1973). librium and information: The role of cross correlations, [69] R. Filliger and P. Reimann, Brownian gyrator: A mini- Phys. Rev. E 85, 061127 (2012). mal heat engine on the nanoscale, Phys. Rev. Lett. 99, [89] V. Dotsenko, A. Maciolek, O. Vasilyev, and G. Oshanin, 230602 (2007). Two-temperature langevin dynamics in a parabolic po- [70] L. P. Dadhichi, A. Maitra, and S. Ramaswamy, Origins tential, Phys. Rev. E 87, 062130 (2013). and diagnostics of the nonequilibrium character of ac- [90] A. Berut, A. Imparato, A. Petrosyan, and S. Ciliberto, tive systems, Journal of Statistical Mechanics , 123201 Theoretical description of effective heat transfer between (2018). two viscously coupled beads, Phys. Rev. E 94, 052148 [71] J. Li, J. M. Horowitz, T. R. Gingrich, and N. Fakhri, (2016). Quantifying dissipation using fluctuating currents, Na- [91] A. Y. Grosberg and J. F. Joanny, Dissipation in a sys- ture Communications 15, 1666 (2019). tem driven by two different thermostats, Polymer Sci- [72] H. B. Callen, Thermodynamics and an Introduction to ence, Series C 60, 118 (2018). Thermostatistics (John Wiley Sons, New York, 1985). [92] A. Puglisi, A. Sarracino, and V. A., Temperature in [73] R. Balian, Francois massieu and the thermodynamic po- and out of equilibrium: A review of concepts, tools and tentials, C. R. Physique 18, 526 (2017). attempts, Phys. Rep. 709, 1 (2017). [74] M. Planck, Treatise on Thermodynamics (Dover Publi- [93] Y. Fily and M. C. Marchetti, Athermal phase separation cations, New York, 1945). of self-propelled particles with no alignment, Phys. Rev. [75] E. Schr¨odinger, Statistical Thermodynamics (Cam- Lett. 108, 235702 (2012). bridge University Press, Cambridge, 1964). [94] T. F. F. Farage, P. Krinninger, and J. M. Brader, Effec- [76] P. Biane and R. Speicher, Free diffusions, free entropy tive interactions in active brownian suspensions, Phys. and free fisher information, Ann. I. H. Poincare - PR 3 Rev. E 91, 042310 (2015). 7, 581 (2001). [95] E. Fodor, C. Nardini, M. E. Cates, J. Tailleur, P. Visco, [77] D. Voiculescu, Free entropy, Bull. London Math. Soc. and F. van Wijland, How far from equilibrium is active 34, 257 (2002). matter? Phys. Rev. Lett. 117, 038103 (2016). [78] S. Kullback and R. A. Leibler, On information and suf- [96] D. Mandal, K. Klymko, and M. R. DeWeese, Entropy ficiency, Annals of Math. Statistics 22, 79 (1951). production and fluctuation theorems for active matter, [79] C. Kwon, P. Ao, and D. J. Thouless, Structure of Phys. Rev. Lett. 119, 258001 (2017). stochastic dynamics near fixed points, Proc. Nat. Acad. [97] S. Shankar and M. C. Marchetti, Hidden entropy pro- Sci. USA 102, 13029 (2005). duction and work fluctuations in an ideal active gas, [80] H. Ge, Extended forms of the second law for general Phys. Rev. E 98, 020604(R) (2018). time-dependent stochastic processes, Phys. Rev. E 80, [98] L. Dabelow, S. Bo, and R. Eichhorn, Irreversibility in 021137 (2009). active matter systems: Fluctuation theorem and mutual [81] H. Ge and H. Qian, Physical origins of entropy pro- information, Phys. Rev. X 9, 021009 (2019). duction, free energy dissipation, and their mathematical [99] N. G. Van Kampen, Derivation of the phenomenological representations, Phys. Rev. E 81, 051133 (2010). equations from the master equation i, Physica 23, 707 [82] H. Qian, A decomposition of irreversible diffusion pro- (1957). cesses without detailed balance, J. Math. Phys. 54, [100] N. G. Van Kampen, Derivation of the phenomenological 053302 (2013). equations from the master equation ii, Physica 23, 816 [83] H. Qian, Thermodynamics of the general diffusion (1957).