Algebra i analiz St. Petersburg Math. J. Tom 24 (2012), 4 Vol. 24 (2013), No. 4, Pages 529–553 S 1061-0022(2013)01252-0 Article electronically published on May 24, 2013
MAYER’S TRANSFER OPERATOR APPROACH TO SELBERG’S ZETA FUNCTION
A. MOMENI AND A. B. VENKOV
Abstract. These notes are based on three lectures given by the second author at Copenhagen University (October 2009) and at Aarhus University, Denmark (Decem- ber 2009). Mostly, a survey of the results of Dieter Mayer on relationships between Selberg and Smale–Ruelle dynamical zeta functions is presented. In a special sit- uation, the dynamical zeta function is defined for a geodesic flow on a hyperbolic plane quotient by an arithmetic cofinite discrete group. More precisely, the flow is defined for the corresponding unit tangent bundle. It turns out that the Selberg zeta function for this group can be expressed in terms of a Fredholm determinant of a classical transfer operator of the flow. The transfer operator is defined in a certain space of holomorphic functions, and its matrix representation in a natural basis is given in terms of the Riemann zeta function and the Euler gamma function.
Contents §1. General theory 529 §2. Mayer’s transfer operator for PSL(2, Z) 531 2.1. Matrix representation of Mayer’s transfer operator, its eigenvectors and eigenvalues 534 2.2. Nuclear spaces, nuclear operators, and Grothendieck’s theory 535 §3. Integral representation of Mayer’s transfer operator 538 §4. Calculation of the trace 540 4.1. Calculation of the trace via the integral representation 543 §5. Ruelle’s zeta function and transfer operator 544 §6. Selberg’s zeta function and transfer operator 546 §7. Number theoretic approach to the relationship between Selberg’s zeta function and Mayer’s transfer operator 549 References 552
§1. General theory We quote Ruelle [14, 15] to introduce his general notion of a transfer operator and a dynamical zeta function for a given dynamical system. First, we give the definition of a weighted dynamical system. Let Λ be a set weighted by a function g :Λ−→ C. Assume that Λ describes a system, then the dynamics of the
2010 Mathematics Subject Classification. Primary 11M36, 11M41. Key words and phrases. Mayer’s transfer operator, Selberg’s zeta function. The authors would like to thank Dieter Mayer for several important remarks and we would like to say also that all possible mistakes in the text belong to us but not to Mayer’s theory we presented in this paper. This work was supported by DAAD, the International Center of TU Clausthal, and the Danish National Research Foundation Center of Excellence, Center for Quantum Geometry of Moduli Spaces(QGM).
c 2013 American Mathematical Society 529
License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 530 A. MOMENI AND A. B. VENKOV
system is given by a map F :Λ−→ Λ. The triplet D := (Λ,F,g) is called a weighted dynamical system or simply a dynamical system. The transfer operator method is applicable if the map F is not invertible, that is, for example, when its inverse is not unique. More precisely, the set of inverse branches of F must be finite or at least countable and discrete in a natural topology. For such a dynamical system, the action of the Ruelle transfer operator L on a function f :Λ−→ C is defined by (1.1) (Lf)(x)= g(y)f(y). y∈F −1{x} Let the set of transfer operators for all dynamical systems of the set Λ with respect to the product ◦ given by (L1 ◦L2)f = L1(L2f) be an algebra denoted by S.Atraceon this algebra is a linear functional Tr : S −→ C such that Tr(L1L2)=Tr(L2L1) for every L1 and L2 in S. For a given trace Tr, a determinant Det for the operators of the algebra can formally be defined by ∞ zm (1.2) Det(I − zL)=exp − Tr Lm . m m=1 On the other hand, a weighted dynamical system D =(Λ,F,g)isequippedwiththe so-called Ruelle dynamical zeta function defined by ∞ − zm m1 (1.3) ζ(z)=exp g(F kx) , m m=1 x∈Fix F m k=0 where Fix F m denotes the set of all fixed points of F m.ThesetFixF m is finite or countably infinite for all m>0. Like other zeta functions, the Ruelle dynamical zeta function has some sort of Euler product |P|−1 −1 |P | k (1.4) ζ(z)= 1 − z g(F xP ) , {P } k=0
where {P } denotes the set of periodic orbits of F with length |P | and xP is an arbitrary element of P . We shall assume that (1.2), (1.3), and (1.4) are absolutely convergent at least for z in a certain domain in C. In general, analytic properties of zeta functions give an important information about the corresponding systems in question. For example, a Tauberian theorem yields the classical prime number theorem from the positions of poles and zeros of the Riemann zeta function in the critical strip. In the same way, we are interested in the analytic properties of the dynamical zeta function to get more information about the corresponding dynamical system. An important method to study the analytic properties of dynamical zeta functions is the transfer operator method. In this method, the analytic properties of the zeta function are related to the spectral properties of a transfer operator through a relationship between the Fredholm determinant of the transfer operator and the dynamical zeta function. An interesting realization of the general program described above is the Mayer trans- fer operator acting on some Banach space of holomorphic functions on a disk [9]. This operator is assigned to the dynamical system related to the geodesic flow on the hyper- bolic plane modulo an arithmetical cofinite discrete group Γ. In this case the Fredholm determinant of the transfer operator is equal to the Selberg zeta function for the corre- sponding discrete group, which is one of the most important aspects of Mayer’s transfer operator theory. Indeed this identity provides us a new insight to the theory of quantum chaos. It turns out that the Mayer transfer operator, which is a purely classical object,
License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use MAYER’S TRANSFER OPERATOR 531
surprisingly contains all information we can obtain from the corresponding Schr¨odinger operator.
§2. Mayer’s transfer operator for PSL(2, Z) We start with introducing some notation and definitions. The hyperbolic plane H is the upper half-plane {x + iy ∈ C |y>0} equipped with the Poincar´emetricds2 = y−2(dx2 + dy2)andthemeasuredμ(z)=y−2 dx dy. Thus, geodesics on H are the semicircles with centra and the end points on the real axis. The group of all orientation preserving isometries of the hyperbolic plane H is identified with the group (2.1) PSL(2, R)=SL(2, R)/{±I} acting on H by linear fractional transformations defined by az + b ab z → gz = ,g= ,z∈ H. cz + d cd The modular group PSL(2, Z) is a discrete subgroup of PSL(2, R) defined by ab (2.2) PSL(2, Z)= ad − bc =1, a,b,c,d∈ Z /{±I}. cd This is a noncocompact Fuchsian group of the first kind. Let M =PSL(2, Z)\H be the quotient space of the hyperbolic plane H mod PSL(2, Z). This is a surface with one cusp and two conical singularities. Consider the continuous dynamics given by the geodesic flow ϕt on T1M, the unit tangent bundle of M.Froma physics point of view, the tangent bundle said to be unit if the geodesic flow describes the motion of a free particle on M with unit magnitude of velocity. As has already been mentioned, a transfer operator can be defined if the corresponding dynamical map has a finite or countable set of inverse branches, while the geodesic flow is continuous and determines an invertible map on T1M. Thus, we first discretize the geodesic flow by constructing a Poincar´e map of ϕt. It is known that, by a suitable choice of the Poincar´e section in T1M, the dynamics of ϕt reduces to the Poincar´e map given by (see [3])
P :[0, 1] × [0, 1] × Z2 → [0, 1] × [0, 1] × Z2,
(2.3) 1 − P (x1,x2,)= TGx1, 1 , , [ ]+x2 x1 where [x] denotes the integral part of x and
1 ∈ x mod 1 if x (0, 1], (2.4) TGx = 0ifx =0 is the Gauss map. Remark 2.1. The group PSL(2, Z) does not contain the reflection relative to the y-axis. Consequently, for every geodesic on M there exists a unique geodesic on M such that their representatives on the upper half-plane are located symmetrially relative to the y-axis.ThesameistruefortheorbitsonT1M. This fact is reflected by two possible values of the parameter . We are interested in the expanding part of the map P , which reflects the ergodic aspects of the geodesic flow ϕt,
Pex :[0, 1] × Z2 → [0, 1] × Z2, (2.5) Pex(x, )=(TGx, −).
License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 532 A. MOMENI AND A. B. VENKOV
It remains to select a suitable weight function. Mayer chose the following weight function: s 2s (2.6) g(x, )=(TG) (x)=x , where s is a complex parameter. In fact, in accordance with Sina˘ı’s paper [18], the ergodic properties of ϕt are related to this weight function. The Mayer transfer operator is a transfer operator for the weighted dynamical system
2s (2.7) D1 = [0, 1] × Z2,Pex,g(x, )=x , i.e., (2.8) Lsf(x, )= g(y)f(y). −1 y=Pex (x, ) We notice that for s = 1 the Mayer transfer operator reduces to the Perron–Frobenious operator for the Gauss map. The map Pex has an infinite number of discrete inverse branches given by
1 (2.9) P −1(x, )= , − ,n∈ N; ex x + n thus, the Mayer transfer operator is formally expressed as ∞ 1 2s 1 (2.10) Lrsf(x, )= f , − ,= ±1. x + n x + n n=1 Since the weighted dynamical system (2.7) and the group PSL(2, Z) are closely related, sometimesthisoperatoriscalledtheMayertransferoperatorforPSL(2, Z). Equivalently, we can write the Mayer operator (2.10) in a vector form: ∞ #– 1 2s #– 1 (2.11) Lrs f (x)= M.f x + n x + n n=1 with 01 #– f(x, 1) (2.12) M = , f (x)= . 10 f(x, −1) If we take the group (2.13) PGL(2, Z)=GL(2, Z)/{±1} instead of the group PSL(2, Z), where ab (2.14) GL(2, Z)= ad − bc = ±1, a,b,c,d∈ Z , cd then the orbits corresponding to the two possible values of will be identified, because PGL(2, Z) contains the reflection relative to the y-axis. Note that this reflection acts H →− on not by a linear fractional transformation but as the map z zs. Therefore, the dynamical system D in (2.7) reduced to the following dynamical system: 1 2s (2.15) D2 = [0, 1],TG,g(x)=x . The Mayer transfer operator for PGL(2, Z) is a transfer operator corresponding to the dynamical system D2, ∞ 1 2s 1 (2.16) L f(x)= f , s x + n x + n n=1 which is sometimes called the transfer operator for the billiard flow or for the Gauss map, because the dynamics of D2 is defined by the Gauss map TG. The analytic properties
License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use MAYER’S TRANSFER OPERATOR 533
of this operator extend easily to the original Mayer operator. Thus, from now on we consider this simplified version of the Mayer operator. Up to now we have seen how the ergodic aspects of the geodesic flow fix the form of the Mayer operator with no information about the function space on which Ls acts. At the next step we should decide about this space in such a way that the operator Ls become a nice operator from an analytical point of view, with well-defined trace and determinant. For this, first, we need two lemmas. Let the real x in (2.16) be extended to z in the complex domain (2.17) Dr = z ∈ C ||z − 1| (2.18) lim ψn(z)=0,z∈ Dr; n→∞ consequently, the smallest lower bound of the radius is r = 1. But the upper bound is at most r<2 because otherwise ψ1 is not contracting at z = −1. To determine the largest upper bound of r we note that the maps ψn are conformal, mapping circles into circles. We also note that the ψn’s map the disk Dr to disks with centers on the real axis. Thus, to investigate the contracting properties of ψn, it suffices to investigate the transformations of the end-points x− =1− r or x+ =1+r of the diameter of Dr lying on the real axis. Moreover, since (2.19) ψn+1(x) <ψn(x),n∈ N,x>−1, to find the upper bound of r it suffices to find the upper bound of r for ψ1.But (2.20) ψ1(x+) <ψ1(x−), 1 License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 534 A. MOMENI AND A. B. VENKOV ≤ ∞ ≥ ∞ ∞ need Sn Mn < for all n 1and n=1 Mn < ,whereMn is a positive number. 1 − −2σ These requirements are fulfilled for σ =Re(s) > 2 if we take Mn =(n r +1) f , where f is bounded because f ∈ B(Dr). This completes the proof. √ 1+ 5 L Corollary 2.4. Let r be in the interval 1, 2 . Then the operator rs for the group Z ⊕ 1 PSL(2, ) is defined on the Banach space B(Dr) B(Dr) for Re(s) > 2 and leaves this space invariant. In Subsection 2.2 we shall see that this choice of the function space makes Mayer’s operator a nuclear operator, which confirms that we are on the right track. 2.1. Matrix representation of Mayer’s transfer operator, its eigenvectors and eigenvalues. In the next subsection we shall see that the transfer operator is compact and the spectrum of this operator in the space B(Dr)isadiscretesetofeigenvaluesof finite multiplicity. Now we are going to derive a matrix representation for the transfer operator. Let f ∈ B(Dr) be an eigenfunction of Ls, (2.24) Lsf(z)=λf(z); the holomorphy of f on Dr allows the following expansion: ∞ k (2.25) f(z)= ck(z − 1) ,z∈ Dr; k=0 Inserting this in (2.24), we get ∞ ∞ ∞ 1 2s 1 k (2.26) c − 1 = λ c (z − 1)k, z + n k z + n k n=1 k=0 k=0 but 1 k k k 1 j (2.27) − 1 = (−1)k−j , z + n j z + n j=0 so that (2.26) can be written as ∞ ∞ ∞ k k 1 2s+j (2.28) c (−1)k−j = λ c (z − 1)k. k j z + n k k=0 j=0 n=1 k=0 On the other hand, we have the following Taylor expansion at z =1: ∞ 1 2s+j (−1)m Γ(2s + j + m) 1 2s+j+m (2.29) = (z − 1)m, z + n m! Γ(2s + j) 1+n m=0 which implies ∞ ∞ 1 2s+j (−1)m Γ(2s + j + m) (2.30) = ζ(2s + j + m) − 1 (z − 1)m, z + n m! Γ(2s + j) n=1 m=0 wherewehaveusedtheidentity ∞ 1 β (2.31) = ζ(β) − 1. 1+n n=1 Finally, inserting (2.30) in (2.28), we get ∞ ∞ ∞ m k (2.32) amk(s)ck(z − 1) = λ ck(z − 1) , k=0 m=0 k=0 License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use MAYER’S TRANSFER OPERATOR 535 where k k (−1)m+k−j Γ(2s + j + m) (2.33) a (s)= ζ(2s + j + m) − 1 , mk j m! Γ(2s + j) j=0 which leads finally to the following eigenvalue equation for the transfer operator in the matrix representation: ∞ (2.34) amk(s)ck = λcm. k=0 Based on these calculations, we can formulate the following lemma. { − k}∞ Lemma 2.5. In the natural basis (z 1) k=0, in accordance with (2.25), the eigen- ∈ { }∞ functions f B(Dr) have the representation given by the sequence ck k=0 satisfying (2.34). Moreover, the transfer operator in this basis is an infinite-dimensional matrix whose matrix entries are given by (2.33). Remark 2.6. Mayer [13] derived a simpler matrix representation of the transfer operator, given by (−1)m Γ(2s + k + m) (2.35) a (s)= ζ(2s + k + m). mk m! Γ(2s + k) In this representation, the basis of the space B(Dr)ischosenasfollows: { }∞ (2.36) ζ(2s + k, z +1) k=0, where ζ(s, w) denotes the Hurwitz zeta function. 2.2. Nuclear spaces, nuclear operators, and Grothendieck’s theory. Let B be an arbitrary Banach space; we denote its dual by B∗, which is the space of bounded functionals on B. A linear operator L on B is said to be nuclear of order q if it has the representation (see [11]) L ∗ (2.37) = λnfn(.)fn, n { } { ∗} ∗ ≤ ∗≤ where fn and fn are families in B and B (respectively) with fn 1and fn 1, {λn} is a sequence of complex numbers, and q =inf ≤ 1 |λn| < ∞ . n For certain classes of nuclear operators, a trace functional is available. A remarkable L ≤ 2 result of Grothendieck says that nuclear operators of order q 3 have a trace tr = ρi as the sum of eigenvalues ρi counted with multiplicities. Moreover, the Fredholm deter- minant of L is defined as (2.38) det(1 − zL)=exptrlog(1− zL)= (1 − ρiz), i or equivalently, ∞ zn (2.39) det(1 − zL)=exp − tr Ln . n n=1 Consequentely, if L = L(s) is a holomorphic function of a parameter s in a domain, then the corresponding determinant is also a holomorphic function of s in this domain (see [7, 11]). License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 536 A. MOMENI AND A. B. VENKOV Lemma 2.7. The Mayer transfer operator Ls for the group PGL(2, Z), given by ∞ 1 2s 1 (2.40) L f(z)= f , s z + n z + n n=1 acting on the Banach space B(Dr) defined as in Lemma 2.3, in the domain σ =Re(s) > 1 1 2 , is a nuclear operator of order q = 2σ . Proof. We are going to find a representation of Mayer’s transfer operator in the form of 3 (2.37). To avoid overloading, we give the proof for r = 2 . First, we insert the Taylor expansion of f at z = 1 in (2.40), ∞ ∞ f (m)(1) 1 m 1 2s (2.41) L f(z)= − 1 . s m! z + n z + n n=1 m=0 We introduce the family of functions 1 2σ 1 m 1 2s (2.42) fn,m(z):= − 1 , an z + n z + n where 1 1 (2.43) an := sup = 1 . ∈ − z Dr z + n n 2 Since 1 (2.44) sup − 1 ≤ 1, z∈Dr z + n we have (2.45) fn,m≤1. Next we define f (m)(1) (2.46) f ∗ (f):=rm . n,m m! Obviously, the Cauchy estimates yield ∗ ≤ (2.47) fn,m 1. The desired representation is given by ∞ ∞ L ∗ (2.48) sf(z)= λn,mfn,m(f)fn,m(z) n=1 m=0 with −m 2σ (2.49) λn,m := r (an) . We note that ∞ ∞ ∞ ∞ 1 2 σ (2.50) |λ | = r− m . n,m n − 1 n=1 m=0 m=0 n=1 2 The first sum is a geometrical series absolutely convergent for any >0, and the second 1 2σ − sum for any > 2σ is absolutely convergent to (2 1)ζ(2σ). Therefore, ∞ ∞ 1 1 (2.51) |λ | = (22σ − 1)ζ(2σ),> . n,m 1 − (r−1) 2σ n=1 m=0 License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use MAYER’S TRANSFER OPERATOR 537 1 This shows that for σ> , 2 ∞ ∞ 1 (2.52) inf |λ | < ∞ = , n,m 2σ n=1 m=0 which completes the proof. Mayer proved a stronger assertion about the nuclearity of the transfer operator in a more elegant way (see [10]). In fact he proved that the transfer operator is nuclear of order zero. This proof is based on some properties of the so-called nuclear spaces. It was Alexander Grothendieck who first introduced the class of nuclear spaces (see [7]). Roughly speaking, nuclear spaces are the maximal class of linear topological spaces with nice properties from an analytic point of view. For example, they admit a generalized kernel theorem of L. Schwartz. Definition 2.8. A locally convex topological vector space E is a nuclear space if and only if every continuous linear map of E into any Banach space is nuclear (see [16, p. 100]). The space H(D) of holomorphic functions on the open disk D with the open compact topology is an example of a nuclear space. The space H(D) satisfies stronger conditions than those of Deffinition 1. Indeed, every bounded linear map of the nuclear space H(D) to any Banach space is not only nuclear but nuclear of order zero [7]. For more details about the nuclear spaces, we refer the reader to [7, 5, 16]. Now, to prove the nuclearity of the√ transfer operator, first we extend it to the nuclear ⊃ ∈ 1+ 5 space H(Dr) B(Dr) with r 1, 2 ,thatis, Ls : H(Dr) → B(Dr), ∞ (2.53) 1 2s 1 L f(z)= f . s z + n z + n n=1 L → 1 Lemma 2.9. The operator√ s : H(Dr) B(Dr) for Re(s) > 2 is nuclear of order ∈ 1+ 5 zero, where r (1, 2 ). Proof. As has been mentioned before, every bounded linear map of H(Dr)toaBanach space is nuclear of order zero. Thus, it suffices to prove the boundedness of Ls. For this, we should show that there exists a neighborhood of zero V (0) ⊂ H(Dr)thatismapped to a bounded subset of B(D ). To begin with, we introduce the sequence of open disks r (2.54) Kn = w = ψn(z) z ∈ Dr whose radii rn and centers cn are given by r (2.55) r = n (n +1)2 − r2 and n +1 (2.56) c = , 0 . n (n +1)2 − r2 We choose the following neighborhood of zero V (0) ⊂ H(Dr): (2.57) V (0) = f ∈ H(Dr) sup |f(z)| (2.58) Kn ⊂ Ks 1,n∈ N. License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 538 A. MOMENI AND A. B. VENKOV For any f ∈ H(Dr)wehave ∞ 1 2σ (2.59) sup |Lsf(z)|≤ sup f(ψn(z)). ∈ 1 − r + n ∈ z Dr n=1 z Dr But (2.58) implies (2.60) sup f(ψn(z)) ≤ sup f(z)=M. z∈Dr z∈K1 Inserting this in (2.59) completes the proof. On the other hand, we have Ls = Ls ◦ ı where ı is the bounded injection given by ı : B(Dr) → H(Dr),ı(f)=f. Then, the nuclearity of Ls leads to that of Ls, because the product of a bounded operator with a nuclear operator is also nuclear with the same order. Thus, we arrive at the following corrolary. Corollary 2.10. The Mayer transfer operator for the groups PGL(2, Z) and PSL(2, Z) 1 in the domain Re(s) > 2 is nuclear of order zero. Note that in this corrolary we have deduced the nuclearity of the transfer operator for PSL(2, Z) from the nuclearity of the transfer operator for PGL(2, Z) and (2.11). Any nuclear operator is compact (see [16, p. 99]). Thus, we obtain the following corollary. Corollary 2.11. The Mayer transfer operator for each of the groups PSL(2, Z) and Z 1 PGL(2, ) in the domain Re(s) > 2 is compact. §3. Integral representation of Mayer’s transfer operator In this section we discuss a new model for the Mayer transfer operator in a Hilbert space. This is important for the investigation of the transfer operator, because in this model the Grothendieck theory of nuclear operators in Banach spaces reduces to the simpler theory of linear operators in Hilbert spaces. We follow Mayer [11] to derive an integral representation for the transfer operator in 1 J a Hilbert space for Re(s) > 2 .Let ν (u) denote the Bessel function (see [6]), ∞ u 2k+ν (−1)k (3.1) J (u)= . ν 2 k!Γ(k + ν +1) k=0 Then it is not difficult to check the following inequality: ∞ ∞ √ 2 dt dt 1 (3.2) J − (2 tt ) < ∞, Re(s) > . 2s 1 et − 1 et − 1 2 0 0 √ 1 J Thus, in the domain Re(s) > 2 , the Bessel function 2s−1(2 tt ) can be viewed as a Hilbert–Schmidt kernel with respect to the measure dt (3.3) dm(t)= . et − 1 Therefore, we can define a Hilbert–Schmidt integral operator K given by s ∞ √ + (3.4) Ksϕ(t)= J2s−1(2 tt )ϕ(t ) dm(t ),ϕ∈ L2(R ,dm), 0 License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use MAYER’S TRANSFER OPERATOR 539 + where L2(R ,dm) denotes the Hilbert space of square integrable functions on the positive real axis with respect to the measure dm given in (3.3). Obviously, the operator Ks is 1 bounded in this space for Re(s) > 2 . We are going to explain how this operator is related to the Mayer transfer operator. First, we consider the following integral transform: ∞ −zt s− 1 + (3.5) (Tsϕ)(z)= e t 2 ϕ(t) dm(t),ϕ∈ L2(R ,dm),z∈ C/(−∞, −1]. 0 This transform can be regarded as the composition of multiplication by the exponential function and the Mellin transform; since both of them are formally invertible, so is Ts. 1 H For a complex s with Re(s) > 2 , consider the space 1 of holomorphic functions in the + domain z ∈ C/(−∞, −1] that is the image of L2(R ,dm) under Ts, + (3.6) H1 = f(z)=(Tsϕ)(z) ϕ ∈ L2(R ,dm),z∈ C/(−∞, −1] . The operator L H →H (3.7) s : 1 1 given by L K −1 (3.8) s = Ts sTs K L is isomorphic to s and has the same spectrum. Shortly we shall see that s has the same form as Mayer’s operator. 1 H Lemma 3.1. For Re(s) > 2 , on the space 1 we have ∞ 1 2s 1 (3.9) L f(z)= f . s z + n z + n n=1 Proof. Let f(z)=(Tsϕ)(z). Then ∞ − − 1 L K zt s 2 K (3.10) sf(z)=Ts(z) sϕ = dm(t)e t ( sϕ)(t), 0 or ∞ s− 1 ∞ √ t 2 −zt (3.11) L f(z)= dt e J − 2 tt ϕ(t ) dm(t ). s t − 2s 1 0 e 1 0 Formula (3.1) implies ∞ √ − k s− 1 ( tt ) (3.12) J − (2 tt )=(tt ) 2 . 2s 1 k!Γ(k +2s) k=0 Inserting (3.12) in (3.11) and rearranging the terms, we get ∞ ∞ ∞ 2s−1 − k s− 1 t −zt ( tt ) (3.13) L f(z)= t 2 ϕ(t ) dm(t ) dt e , s et − 1 k!Γ(k +2s) 0 0 k=0 or ∞ ∞ − k ∞ 2s+k−1 −zt s− 1 ( t ) t e (3.14) L f(z)= t 2 ϕ(t ) dm(t ) dt . s k!Γ(k +2s) et − 1 0 k=0 0 But the Hurwitz zeta function has the integral presentation [6], 1 ∞ tw−1e−qt (3.15) ζ(w; q)= dt, Re(w) > 1. − −t Γ(w) 0 1 e License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 540 A. MOMENI AND A. B. VENKOV Thus, the integral over t in (3.14) can be replaced by the Hurwitz zeta function, ∞ ∞ − k s− 1 ( t ) (3.16) L f(z)= t 2 ϕ(t ) dm(t ) ζ(k +2s; z +1). s k! 0 k=0 On the other hand, for Re(w) > 1 the Hurwitz zeta function is defined by ∞ 1 w (3.17) ζ(w; q)= , q + n n=0 which leads to the formula ∞ ∞ ∞ − k 2s+k s− 1 ( t ) 1 (3.18) L f(z)= t 2 ϕ(t ) dm(t ) , s k! z + n 0 k=0 n=1 or ∞ ∞ ∞ 2s − k k 1 s− 1 ( t ) 1 (3.19) L f(z)= t 2 ϕ(t ) dm(t ) . s z + n k! z + n n=1 0 k=0 −t ( 1 ) Since the sum over k is the Taylor expansion of the function e z+n ,weget ∞ ∞ 2s 1 s− 1 −t ( 1 ) (3.20) L f(z)= t 2 e z+n ϕ(t ) dm(t ), s z + n n=1 0 or, by (3.5), ∞ 1 2s 1 (3.21) L f(z)= (T ϕ) . s z + n s z + n n=1 This means that ∞ 1 2s 1 (3.22) L f(z)= f . s z + n z + n n=1 L L As we saw in this lemma, the Mayer transfer operator s and the operator s are of the same form. Using this fact and considering the spaces on which these operators L L act, we can see that every eigenfunction of s is an eigenfunction of s. In [13], Mayer proved the converse; thus, we arrive at the following lemma. 1 L L Lemma 3.2. For Re(s) > 2 , the operators s and s have the same spectrum. 1 K Corollary 3.3. In the domain Re(s) > 2 , the integral operator s and the Mayer transfer operator Ls for PGL(2, Z) have the same spectrum counted with multiplicities. §4. Calculation of the trace As was explained in §2, the transfer operator is of trace class. In this part we are going to calculate the trace of the transfer operator and its powers. We illustrate two different approaches for the calculation of trace; the first is based on the contracting property of the map 1 (4.1) ψ (z)= , n z + n which allows us to apply the method of geometric trace, and the second employs the integral representation of the transfer operator. By using (4.1), the Mayer transfer operator for PGL(2, Z) can be written in the form ∞ 2s (4.2) Lsf(z)= ψn(z) f(ψn(z)). n=1 License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use MAYER’S TRANSFER OPERATOR 541 To calculate the trace of Ls, first we need to calculate the trace of the terms 2s (4.3) Ls,nf(z)=ψn(z) f(ψn(z)),n∈ N. L With the same arguments as in Subsection 2.1,√ we can show that the operator s,n is ∈ N ∈ 1+ 5 nuclear of order zero for all n .Forr 1, 2 ,themapψn(z)onDr has a unique fixed point z∗ given by n √ n n2 +4 (4.4) z∗ = − + , n 2 2 which is obtained by solving the equation 1 (4.5) = z. z + n The existence of a unique solution for (4.5) in Dr is crucial for the calculation of the trace of Ls,n. Before proceeding further we quote a lemma from [11] concerning the eigenvalues of a general composition operator. Lemma 4.1. Let D ⊂ C be an arbitrary domain, ψ a holomorphic map on D with auniquefixedpointz∗ ∈ D,andϕ an arbitrary function in the Banach space B(D). Then the spectrum of the composition operator Cf = ϕψ ◦ f on B(D) consists of simple ∗ ∗ n eigenvalues λn = ϕ(z )(ψ (z )) , n =0, 1,..., which converge to zero as n →∞. Remark 4.2. A contracting map ψ on a domain D is said to be a map of the domain D strictly inside itself if (4.6) inf ψ(z) − z≥δ>0. z∈D,z ∈C\D Such maps allways have a unique fixed point in D. In accordance with this lemma, the eigenvalues of Ls,n are given by ∗ 2s ∗ m − m ∗ 2s ∗ 2m ∈ N ∪{ } (4.7) λm(n)=(ψn(zn)) (ψn(zn)) =( 1) (zn) (zn) ,m 0 , which are all simple. Therefore, the trace of Ls,n is simply the sum of them, ∞ (z∗)2s (4.8) tr L = λ (n)= n . s,n m 1+(z∗)2 m=0 n L 1 L Then we obtain the trace of s for Re(s) > 2 by summing the traces of all s,n’s, ∞ (z∗)2s (4.9) tr L = n , s 1+(z∗)2 n=1 n 1 where Re(s) > 2 ensures absolute convergence. Next we calculate the trace of the powers of the transfer operator. First, we note that Ln ··· L L L (4.10) s = s,i1 s,i2 ... s,in , i1≥1 i2≥1 in≥1 where as before L 2s (4.11) s,ik f(z)=ψik (z) f(ψik (z)), L L L and the composition operator s,i1 s,i2 ... s,in has the form L L L s,i1 s,i2 ... s,in f(z) (4.12) 2s = ψi1 (z)[ψi2 ψi1 (z)] ...[ψin ψin−1 ...ψi1 (z)] f(ψin ...ψi1 (z)). For convenience, without fear of confusion we introduce the following notation: L L L L (4.13) s,n := s,i1 s,i2 ... s,in ,ψn := ψin ...ψi1 (z); License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 542 A. MOMENI AND A. B. VENKOV then formula (4.12) can be written as follows: n 2s (4.14) Ls,nf(z)= ψk(z) f(ψn(z)). k=1 On the other hand, the composition map ψn has a unique fixed point zn on Dr,given by a periodic continuous fraction, (4.15) zn = 0, in,in−1,...,i1 . The uniqueness of the fixed point zn enables us to apply Lemma 4.1 to the composition operator Ls,n in (4.14). Thus, we get the eigenvalues of Ls,n: n 2s dψn m (4.16) λm(zn)= ψk(zn) ,m∈ N. dz z=z k=1 n Using the chain rule and observing that dψ (4.17) ik =(−1)(ψ )2,i∈ N, dz ik k we can write the derivative of the composition function ψn(z)as d n 2 n 2 − n − n (4.18) ψn(z) =( 1) ψik ψk−1(zn) =( 1) ψk(zn) , dz z=z n k=1 k=1 where ψ0 := id and the last identity simply comes from the definition of ψk in (4.13). By inserting (4.18) in (4.16), the eigenvalues λm(zn) can be written as n 2s+2m nm (4.19) λm(zn)=(−1) ψk(zn) ,m∈ N. k=1 Moreover, we have n−k (4.20) ψk(zn)=T zn, where T is the Gauss map given in (2.4). Thus, the eigenvalues of Ls,n in (4.19) can be written as n−1 2s+2m nm k (4.21) λm(zn)=(−1) T zn ,m∈ N. k=0 Because of the simplicity of the eigenvalues, the trace of Ln is the sum of all λm(zn)’s, ∞ − 2s n 1 T kz L k=0 n (4.22) tr s,n = λm(zn)= − , − − n n 1 k 2 m=0 1 ( 1) k=0 T zn or with a more explicit notation for Ls,n and zn given in (4.13) and (4.15) respectively, n−1 k 2s T 0, i ,i − ,...,i L L L k=0 n n 1 1 (4.23) tr s,i1 s,i2 ... s,in = − . − − n n 1 k 2 1 ( 1) k=0 T 0, in,in−1,...,i1 Ln Finally, by (4.10), we obtain the trace of s by summing the contribution of all the L L L composition operators i1 i2 ... in : n−1 k 2s T 0, in,in−1,...,i1 (4.24) tr Ln = ··· k=0 . s n−1 2 − − n k − i1≥1 i2≥1 in≥1 1 ( 1) k=0 T 0, in,in 1,...,i1 License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use MAYER’S TRANSFER OPERATOR 543 We note that, in formula (4.24), the sum is over all fixed points 0, in,in−1,...,i1 of the map ψn, as mentioned in (4.15). But this set coincides with the set of fixed points of the Gauss map Fix T n.Thus,(4.24)canbewritteninamorecompactform − 2s n 1 T k(x) Ln k=0 (4.25) tr s = − . − − n n 1 k 2 x∈Fix T n 1 ( 1) k=0 T (x) 4.1. Calculation of the trace via the integral representation. In this subsection we calculate the trace of the integral operator ∞ √ + (4.26) Ksϕ(t)= J2s−1(2 tt )ϕ(t ) dm(t ),ϕ∈ L2(R ,dm), 0 1 for Re(s) > 2 . By the standard results of the theory of linear operators in Hilbert spaces, the trace of Ks is given by the integral ∞ (4.27) tr Ks = J2s−1(2t) dm(t), 0 or by inserting the measure (3.3), ∞ J − (2t) (4.28) tr K = 2s 1 dt. s t − 0 e 1 To calculate this integral, we use the identity ∞ 1 (4.29) = e−nt, et − 1 n=1 obtaining ∞ ∞ −nt (4.30) tr Ks = e J2s−1(2t) dt. n=1 0 The integral above can be calculated, √ ∞ 2s 2 −nt xn n n +4 J − − (4.31) e 2s 1(2t) dt = 2 ,xn = + . 0 1+xn 2 2 Therefore, ∞ x2s (4.32) tr K = n s 1+x2 n=1 n which coincides with (4.9), i.e., (4.33) tr Ks =trLs, as was expected in view of Corollary 3.3. To calculate the trace of powers of Ks,itisnot difficult to show that ∞ ∞ √ √ Kn J J J (4.34) tr s = dm(tn) ... dm(t1) 2s−1(2 t1t2) ... 2s−1(2 tn−1tn) 2s−1(2 tnt1). 0 0 Calculating this integral, we arrive at the expected result: Kn Ln (4.35) tr s =tr s . We do not know of any simple direct proof of (4.35) (see [11]; a similar calculation was done in [20]). License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 544 A. MOMENI AND A. B. VENKOV §5. Ruelle’s zeta function and transfer operator In this section we shall denote various zeta functions by other letters, not necessarily ζ. As was mentioned in the §1, the Ruelle zeta function for a given weighted dynamical system (Λ,F,g) is defined by ∞ − zn n1 (5.1) ζ (z)=exp g(F kx) . R n n=1 x∈Fix F n k=0 The Ruelle zeta function for the dynamical system D2 defined in (2.15) at the point z =1 reduces to the dynamical function ξ(s)givenby ∞ − 1 n1 (5.2) ξ(s)=exp (T kx)2s . n n=1 x∈Fix T n k=0 In the following lemma we see the close relationship between the Mayer transfer operator and the dynamical function ξ(s). Lemma 5.1. Let Ls be the Mayer transfer operator for PGL(2, Z) given in (2.40),and 1 let ξ(s) be the zeta function defined by (5.2).ForRe(s) > 2 , we have det(1 + L ) (5.3) s+1 = ξ(s), det(1 −Ls) where the determinant of the transfer operator is defined in the sense of Grothendieck by (2.39). 1 L L Proof. From Lemma 2.9 we know that for Re(s) > 2 , the transfer operators s and s+1 are both nuclear of order zero. Therefore, by (2.39) we have ∞ (−1)n (5.4) det(1 + L )=exp − tr Ln s+1 n s+1 n=1 and ∞ 1 (5.5) det(1 −L )=exp − tr Ln . s n s n=1 Dividing (5.4) by (5.5), we get ∞ det(1 + L ) 1 1 (5.6) s+1 =exp − (−1)n tr Ln +trLn , Re(s) > . det(1 −L ) n s+1 s 2 s n=1 On the other hand, inserting the traces of Ls and Ls+1 from (4.25), we obtain n−1 − − n Ln Ln k 2s (5.7) ( 1) tr s+1 +tr s = T (x) , x∈Fix T n k=0 which completes the proof. Now we consider the dynamical system D1 defined in (2.7), which is closely related to the geodesic flow on the upper half-plane mod PSL(2, Z). The Ruelle zeta function (5.1) for the dynamical system D1 at z = 1 reduces to the zeta function η(s)givenby ∞ − 1 n1 (5.8) η(s)=exp g(P k (x, )) . n ex ∈ n n=1 (x, ) Fix Pex k=0 From (2.5) we have (5.9) Pex(x, )=(Tx,−), License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use MAYER’S TRANSFER OPERATOR 545 where T is the Gauss map. Formula (5.9) shows that, obviously, the odd powers of Pex have no fixed points, and therefore, summation in (5.8) is restricted to the even integers ∈ N n , − 1 n1 (5.10) η(s)=exp g(P k (x, )) . n ex ∈ n n even (x, ) Fix Pex k=0 On the other hand, by iterating (5.9) we get k k − k (5.11) Pex(x, )=(T x, ( 1) )). Therefore, (2.6) and (5.11) imply k k 2s (5.12) g(Pex(x, )) = (T x) . Inserting (5.12) in (5.10), we have − 1 n1 (5.13) η(s)=exp (T kx)2s . n ∈ n n even (x, ) Fix Pex k=0 ∈ N ± n Finally, we note that for even n every pair of fixed points (x, 1) of the map Pex n n corresponds to the fixed point x of the map T . Consequently, summation over Fix Pex can be replaced by twice the sum over the set Fix T n defined in (1.3), that is, − 1 n1 (5.14) η(s)=exp 2 (T kx)2s . n n even x∈Fix T n k=0 Thenextlemmashowshowη(s) is related to the transfer operator. 1 Lemma 5.2. For Re(s) > 2 , we have det(1 −L2 ) (5.15) s+1 = η(s), −L2 det(1 s) where Ls denotes the transfer operator for PGL(2, Z), and the determinants are defined in the sense of Grothendieck. Proof. As in the previous lemma, in accordance with Grothendieck’s Fredholm determi- nant (2.39) for nuclear operators, we have 1 1 (5.16) det(1 −L2 )=exp − tr L2m , Re(s) > , s+1 m s+1 2 m and 1 1 (5.17) det(1 −L2)=exp − tr L2m , Re(s) > . s m s 2 m Then det(1 −L2 ) 1 1 (5.18) s+1 =exp − tr L2m +trL2m , Re(s) > , det(1 −L2) m s+1 s 2 s m or equivalently det(1 −L2 ) 1 1 (5.19) s+1 =exp 2 − tr Ln +trLn , Re(s) > . det(1 −L2) n s+1 s 2 s n even On the other hand, by (4.25) and a simple algebraic calculation, we get n−1 − Ln Ln k 2s (5.20) tr s+1 +tr s = T (x) ,neven. x∈Fix T n k=0 License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 546 A. MOMENI AND A. B. VENKOV Inserting this in (5.19), we arrive at the desired result. L Z 1 Corollary 5.3. Let rs denote the transfer operator for PSL(2, ).ForRe(s) > 2 ,we have det(1 − Lrs+1) (5.21) = η(s). det(1 − Lrs) Proof. The representation of Mayer’s transfer operator for PSL(2, Z) given in (2.11) immediately leads to − L −L L −L2 (5.22) det(1 rs)=det(1 s)det(1+ s) = det(1 s). Together with the previous lemma, this gives the desired result. §6. Selberg’s zeta function and transfer operator In this section we illustrate the relationship between the Selberg zeta function and the Mayer transfer operator, which is a most important aspect of Mayer’s theory. First, we recall the definition of the Selberg zeta function. Let Γ be a Fuchcian group of the first kind. The Selberg zeta function for Γ is defined in the domain Re(s) > 1byan absolutely convergent infinite product (see [17]) ∞ −k−s (6.1) ZΓ(s)= (1 −N(P ) ), k=0 {P }Γ where P runs over all primitive hyperbolic conjugacy classes of Γ and N (P ) > 1 denotes the norm of P . By definition, every hyperbolic element P of the group Γ is conjugated by an element from PSL(2, R)toa(2× 2)-matrix ρ 0 (6.2) 0 ρ−1 with ρ>1, which gives the norm of P as N (P )=ρ2. With the help of the Selberg trace formula, it is proved that (1) ZΓ(s) has analytic (meromorphic) continuation to the entire complex s-plane; (2) ZΓ(s) satisfies the functional equation ZΓ(1 − s)=Ψ(s)ZΓ(s) with some known function Ψ; (3) the nontrivial zeros of ZΓ(s) are related to eigenvalues and resonances of the automorphic Laplacian A(Γ) for the group Γ. It is well known that there is a one-to-one correspondence between the primitive hyperbolic conjugacy classes of Γ represented by P with norm N (P ) and the prime closed geodesics c on the Riemann surface H \ Γ(with obvious singularities in some cases) with length (c) such that (6.3) N (P )=e(c). This fact enables us to define the Selberg zeta function in the following equivalent form: ∞ (6.4) ZΓ(s)= (1 − exp(−(s + k)(c))), k=0 {c}Γ where {c}Γ runs over all prime closed geodesics on H \ Γ. License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use MAYER’S TRANSFER OPERATOR 547 In the sequel we shall need an Euler product for the dynamical function η(s) defined in (5.14), − 1 n1 2s (6.5) η(s)=exp T kx . n n even x∈Fix T n k=0 Following Ruelle [15], we rewrite η(s) as an Euler product. To begin with, we recall that for x ∈ [0, 1], the set (6.6) φ = x,Tx,...,Tkx,... is an orbit of the Gauss map T given in (2.4). The orbit φ is periodic if there exists an integer u ∈ N such that (6.7) T ux = x. The integer u ∈ N is called a period of the periodic orbit φ.Wesaythataperiodicorbit φ has primitive of minimal period m if m is the minimum of the set of all periods of φ. We denote the set of all primitive periodic orbits of T of minimal period m by Per(m). Now we are going to replace the sum over Fix T n in (6.5) by a sum over the primitive periodic orbits Per(m). For this, first we introduce the subset M Fix T m ⊂ Fix T m, containing the periodic continued fractions with minimum period m.Then ( n )m−1 n−1 2s m (6.8) T kx = (T kx)2s, x∈Fix T n k=0 m|n x∈M Fix T m k=0 or, observing that x ∈ M Fix T m is of period m, − − n n1 2s m1 m (6.9) T kx = (T kx)2s . x∈Fix T n k=0 m|n x∈M Fix T m k=0 Now by replacing the sum over M Fix T m by a sum over Per(m), we get − − n n1 2s m1 m k k 2s (6.10) T x = m (T xφ) , x∈Fix T n k=0 m|n φ∈Per(m) k=0 where xφ is an arbitrary point of the orbit φ ∈ Per(m) and the factor m comes from the fact that a periodic orbit φ containing a point x ∈ M Fix T m contains also the set of points (6.11) T kx ∈ M Fix T m k =0,...,m− 1 . Inserting (6.10) in (6.5), we see that m−1 n 1 m (6.12) η(s)=exp 2 m (T kx )2s , n φ n even m|n φ∈Per(m) k=0 or, by rearranging the sum, ∞ − 1 r1 q (6.13) η(s)=exp 2 (T kx )2s . q φ r even φ∈Per(r) q=0 k=0 Since ∞ 1 (6.14) − log(1 − w)= wq, q q=1 License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 548 A. MOMENI AND A. B. VENKOV formula (6.13) reduces to r−1 k 2s (6.15) η(s)=exp − 2 log 1 − (T xφ) , r even φ∈Per(r) k=0 or r−1 −2 k 2s (6.16) η(s)=exp log 1 − (T xφ) , r even φ∈Per(r) k=0 which leads finally to the desired Euler product, 1 (6.17) η(s)= − . − r 1 k 2s 2 r even φ∈Per(r) (1 k=0(T xφ) ) The Euler product for η(s) given above is crucial for the following lemma which is a bridge between the Mayer transfer operator and Selberg zeta function. Lemma 6.1. For Re(s) > 1, we have ∞ (6.18) Z(s)−1 = η(s + l), l=0 where Z(s) is the Selberg zeta function for the group PSL(2, Z) and η(s) is defined in (6.5) with the Euler product given in (6.17). Proof. First, we need to rewrite (6.17) as a product over the primitive priodic orbits of the map Pex defined in (2.5). Let Per(r) denote the set of primitive periodic orbits of minimal period r =2a with a ∈ N for the map Pex. In accordance with (5.11), an element φ ∈ Per( r) passing the point (x, ) is defined by the following set: (6.19) φ = (T kx, (−1)k) | k =0,...,r− 1,= ±1 , where x ∈ [0, 1] and = ±1. We note that all periodic orbits of Pex have even period r. Obviously, every two elements of Per( r) correspond to an element of Per(r). Noting that the terms in the product (6.17) do not depend on , we see that the power 2 in the denominator of (6.17) disappears if we replace Per(r)byPer( r), 1 (6.20) η(s)= r−1 , 1 − (T kx)2s r even φ∈Per( r) k=0 φ where xφ is an arbitrary point of φ. On the other hand, according to Series [2] and Adler and Flatto [1], there is a one-to-one correspondence between Per( r)andthesetof primitive periodic orbits ϑ on the unit tangent bundle T1M, M =PSL(2, Z) \ H, with theperiod(see[12]) r−1 − k (6.21) τ(ϑ)= 2ln T xφ. k=0 These facts recover the physical situation behind the abstract number-theoretic ap- pearance of the problem, leading to the following equivalent formula for the dynamical zeta function in (6.20): 1 (6.22) η(s)= − − , { } 1 exp( sτ(ϑ)) ϑ Γ { } \ H where ϑ Γ denotes the set of all primitive periodic orbits on T1M, M =Γ with Γ=PSL(2, Z). But there is also a one-to-one correspondence between the primitive License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use MAYER’S TRANSFER OPERATOR 549 periodic orbits on T1M and the closed geodesics c on M with length (c)=τ(ϑ). Thus, we have another equivalent formula for our zeta function 1 (6.23) η(s)= − − . { } 1 exp( β(c)) c Γ { } \H Z Here c Γ denotes the set of primitive closed geodesics on M =Γ with Γ = PSL(2, ), and (c) is the length of the closed geodesic c. Note that because of the unity of the tangent bundle, (c)=τ(ϑ). Finally, inserting (6.23) in (6.4) provides the desired result. This lemma immediately leads to the most important feature of the Mayer transfer operator theory, namely, the following is true. Theorem 6.2. For Re(s) > 1, we have (6.24) det(1 − Lrs)=Z(s) and −L2 (6.25) det(1 s)=Z(s), where Z(s) denotes the Selberg zeta function for the group PSL(2, Z), Lrs is the transfer operator also for PSL(2, Z),andLβ is the transfer operator for PGL(2, Z). Proof. It suffices to insert (5.21) and (5.15) in (6.18). Remark 6.3. The domain of validity of (6.24) and (6.25) extends immediately to all s ∈ C except possibly some small singular set, because the Selberg zeta function is a meromorphic function on the entire plane C. §7. Number theoretic approach to the relationship between Selberg’s zeta function and Mayer’s transfer operator In the previous section, based on the one-to-one correspondence between the prim- itive periodic orbits of Pex and primitive periodic orbits on the phase space T1M,we passed from the number theoretic appearance of the problem to its dynamical nature. In this physical realization of the problem, we can see the relationship of the Selberg zeta function with the Mayer transfer operator. Efrat [4] and later Lewis and Zagier [8] reproved Mayer’s result in a purely number theoretic approach. In this section we are going to illustrate the alternative approach of Lewis and Zagier. First, we introduce their notation. Let γ ∈ GL(2, Z)actonDr via a linear fractional transformation. The right action of the semigroup ab (7.1) Ξ = γ = ∈ GL(2, Z) γ(D ) ⊆ D cd r r on the space B(Dr)isgivenby ab − az + b (7.2) π f(z)=(cz + d) 2sf , s cd cz + d where Dr and B(Dr)arethesameasin§2. Then the Mayer transfer operator for Z 1 PGL(2, ) in the domain Re(s) > 2 can be represented by ∞ 01 (7.3) L = π . s s 1 n n=1 License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 550 A. MOMENI AND A. B. VENKOV 01 ∈ ∈ N We note that 1 n Ξ for all n . The set of the so-called reduced elements of SL(2, Z) is defined by ab (7.4) Red = ∈ SL(2, Z) 0 ≤ a ≤ b, c ≤ d . cd For a hyperbolic element γ of SL(2, Z), a positive integer k = k(γ) is defined to be the k ∈ Z largest integer such that γ = γ1 for some γ1 SL(2, ). Therefore, for a primitive hyperbolic element we have k = 1. Now we quote the heart of the proof of Lewis and Zagier, based on a classical reduction theory for quadratic forms, as a lemma whose proof one can find in [8]. Lemma 7.1. (1) Every reduced matrix γ ∈ Red has a unique decomposition of the form 01 01 (7.5) ... ,n1,...,n2l ≥ 1, 1 n1 1 n2l for a unique positive integer l = l(γ), called the length of γ. (2) There are 2l(γ)/k(γ) reduced representatives with the same length l(γ) in every hyperbolic conjugacy class of SL(2, Z) containing γ. Next, consider the Selberg zeta function for the group Γ = SL(2, Z)forRe(s) > 1, ∞ −m−s (7.6) ZΓ(s)= 1 −N(P ) . m=0 {P } Γ Taking the logarithm of both sides and using the Taylor expansion of log 1−N(P )−m−s , we get ∞ ∞ 1 (7.7) − log Z(s)= N (P )−k(m+s). k {P }Γ m=0 k=1 The absolute convergence of the product (7.6) implies that of the sum above. Thus, the interchange of the sums over m and k is allowed, ∞ ∞ 1 (7.8) − log Z(s)= N (P )−ks N (P )−km, k {P }Γ k=1 m=0 −1 but the sum over m is the Taylor expansion of 1 −N(P )−k ,sothat ∞ 1 N (P )−ks (7.9) − log Z(s)= . k 1 −N(P )−k {P }Γ k=1 k k Since N (P ) = N (P ), we can consider the double sum over k and {P }Γ, the primitive hyperbolic conjugacy classes, as a single sum over all, not only primitive hyperbolic conjugacy classes, denoted by {γ}Γ, 1 N (γ)−s (7.10) − log Z(s)= . k(γ) 1 −N(γ)−1 {γ}Γ The second part of Lemma 7.1 enables us to replace the sum over the hyperbolic conjugacy classes by the the sum over the set Red of reduced matrices: 1 N (γ)−s (7.11) − log Z(s)= . 2l(γ) 1 −N(γ)−1 γ∈Red Now we need the following lemma. License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use MAYER’S TRANSFER OPERATOR 551 Lemma 7.2. For γ ∈ Ξ, the trace of the operator πs(γ) acting on B(Dr) is given by N (γ)−s (7.12) tr(π (γ)) = . s 1 −N(γ)−1 Proof. Let ψγ denote the action of γ ∈ ΞonDsr, az + b ab (7.13) ψ (z)=γz := ,γ= . γ cz + d cd We also put (7.14) j(γ,z)=cz + d. Then the operator πs(γ)iswrittenintheform −2s (7.15) πs(γ)f(z)=j(γ,z) f(ψγ (z)). Since the definiton of Ξ in (7.1) shows that ψγ maps Dsr strictly inside itself, we can apply Lemma 4.1 to get the eigenvalues of πs(γ), which are all simple. These eigenvalues are given by m ∗ −2s dψγ (7.16) λm(γ)=j(γ,x ) , dz z=x∗ ∗ where x is a unique fixed point of ψγ in Dsr. The existence of a unique fixed point comes from Remark 4.2. On the other hand, we note that dψ 1 (7.17) γ = , dz j(γ,z)2 whence (7.16) reduces to ∗ −2s−2m (7.18) λm(γ)=j(γ,x ) . The sum of all λm(γ) gives the trace of πs(γ), ∞ j(γ,x∗)−2s (7.19) tr(π (γ)) = j(γ,x∗)−2s−2m = . s 1 − j(γ,x∗)−2 m=1 To complete the proof we must show that N (γ)−1 = j(γ,x∗)−2.Forthis,weobserve that (7.20) j(gγg−1,gx∗)=j(γ,x∗),g∈ SL(2, R), but there exists an element g ∈ SL(2, R) such that −1 ∗ − ρ 0 (7.21) gx =0,gγg1 = ∈ Ξ,ρ>1. 0 ρ Thus, (7.22) j(γ,x∗)=j(gγg−1,gx∗)=ρ = N (γ), where the last identity follows from the definition of the norm in §6. Inserting (7.22) in (7.19) completes the proof. We note that Red ⊂ Ξ; thus, we can insert (7.12) in (7.11), 1 (7.23) − log Z(s)=tr π (γ) 2l(γ) s γ∈Red but part 1) of Lemma 7.1 allows us to write ∞ ∞ 2l 1 01 (7.24) − log Z(s)=tr π . 2l s 1 n l=1 n=1 License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 552 A. MOMENI AND A. B. VENKOV That is, we have ∞ 1 (7.25) − log Z(s)= tr(L )2l, 2l s l=1 or ∞ 1 (7.26) Z(s)=exp − tr(L )2l . 2l s l=1 Finally, since the right-hand side of the equation above coincides with the Fredholm L2 determinant of s, we get the desired result, namely, −L2 (7.27) Z(s) = det(1 s). References [1]R.L.AdlerandL.Flatto,Cross section map for the geodesic flow on the modular surface,Con- ference in Modern Analysis and Probability (New Haven, 1982), Contemp. Math., vol. 26, Amer. Math. Soc., Providence, RI, 1984, pp. 9–24. MR0737384 (85j:58128) [2] R. Bowen and C. Series, Markov maps associated with Fuchsian groups, Inst. Hautes Etudes´ Sci. Publ. Math. No. 50 (1979), 153–170. MR0556585 (81b:58026) [3] C.-H. Chang and D. Mayer, Thermodynamic formalism and Selberg’s zeta function for modular groups, Regul. Chaotic Dyn. 5 (2000), 281–312. MR1789478 (2001k:37045) [4] I. Efrat, Dynamics of the continued fraction map and the spectral theory of SL(2, Z), Invent. Math. 114 (1993), 207–218. MR1235024 (94h:11052) [5]I.M.Gelfand and N. Ya. Vilenkin, Generalized functions.Vol.4.Some applications of harmonic analysis. Equipped Hilbert spaces, Fizmatgiz, Moscow, 1961; English transl., Acad. Press, New York–London, 1964. MR0146653 (26:4173); MR0435834 (55:8786d) [6] I. S. Gradshteyn and I. M. Ryzhik, Tables of integrals, sums, series and products, Fizmatgiz, Moscow, 1963; English transl., Acad. Press, New York–London, 1965. MR0161996 (28:5198); MR0197789 (33:5952) [7] A. Grothendieck, Produits tensoriels topologiques et espaces nucl´eaires,Mem.Amer.Math.Soc. No. 16 (1955), 140 pp. MR0075539 (17:763c) [8] J. Lewis and D. Zagier, Period functions and the Selberg zeta function for the modular group, The Mathematical Beauty of Physics (Saclay, 1996), Adv. Ser. Math. Phys., vol. 24, World Sci., Singapore, 1997, pp. 83-97. MR1490850 (99c:11108) [9] D. Mayer, The thermodynamic formalism approach to Selberg’s zeta function for PSL(2, Z), Bull. Amer. Math. Soc. (N.S.) 25 (1991), 55–60. MR1080004 (91j:58130) [10] , On a zeta function related to the continued fraction transformation, Bull. Soc. Math. France 104 (1976), 195–203. MR0418168 (54:6210) [11] , Continued fractions and related transformations, Ergodic Theory, Symbolic Dynamics, and Hyperbolic Spaces (Trieste, 1989), Oxford Univ. Press, New York, 1991, pp. 175–222. MR1130177 [12] , Thermodynamics formalism and quantum mechanics on the modular surface,FromPhase Transitions to Chaos, World Sci. Publ., River Edge, NJ, 1992, pp. 521–529. MR1172552 (93h:58121) [13] , On the thermodynamic formalism for the Gauss map, Comm. Math. Phys. 130 (1990), 311–333. MR1059321 (91g:58216) [14] D. Ruelle, Dynamical zeta functions and transfer operators, Notices Amer. Math. Soc. 49 (2002), 887–895. MR1920859 (2003d:37026) [15] , Dynamical zeta functions for piecewise monotone maps of the interval, CRM Monogr. Ser., vol. 4, Amer. Math. Soc., Providence, RI, 1994. MR1274046 (95m:58101) [16] H. H. Schaefer, Topological vector spaces, The Macmillan Co., New York, 1966. MR0193469 (33:1689) [17] A. Selberg, Harmonic analysis and discontinuous groups in weakly symmetric Riemannian spaces with applications to Dirichlet series, J. Indian Math. Soc. (N.S.) 20 (1956), 47–87. MR0088511 (19:531g) [18] Ya. G. Sina˘ı, Gibbs measures in ergodic theory, Uspekhi Mat. Nauk 27 (1972), no. 4, 21–64; English transl., Russian Math. Surveys 27 (1972), no. 4, 21–69. MR0399421 (53:3265) [19] A. B. Venkov, Spectral theory of automorphic functions,TrudyMat.Inst.Steklov.153 (1981), 172 pp.; English transl., Math. Appl. (Soviet Ser.), vol. 51, Kluwer Acad. Publ. Group, Dordrecht, 1990. MR0665585 (85j:11060a); MR1135112 (93a:11046) License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use MAYER’S TRANSFER OPERATOR 553 [20] , The automorphic scattering matrix for the Hecke group Γ[2 cos(π/q)], Internat. Conf. on Analytical Methods in Number Theory and Analysis (Moscow, 1981), Trudy Mat. Inst. Steklov. 163 (1984), 32–36; English transl. in Proc. Steklov Inst. Math. 1985, no. 4 (163). MR0769866 (86e:11041) Department of Statistical Physics and Nonlinear Dynamics, Institute of Theoretical Physics, Clausthal University of Technology, 38678, Clausthal-Zellerfeld, Germany E-mail address: [email protected] Institute for Mathematics and Centre of Quantum Geometry QGM, University of Aarhus, 8000, Aarhus C, Denmark E-mail address: [email protected] Received 22/SEP/2011 Originally published in English License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use