PRIME GEODESIC THEOREM IN THE 3-DIMENSIONAL HYPERBOLIC SPACE OLGA BALKANOVA, DIMITRIOS CHATZAKOS, GIACOMO CHERUBINI, DMITRY FROLENKOV, AND NIKO LAAKSONEN 3 Abstract. For Γ a cofinite Kleinian group acting on H , we study the Prime 3 Geodesic Theorem on M = ΓnH , which asks about the asymptotic behaviour of lengths of primitive closed geodesics (prime geodesics) on M. Let EΓ(X) be the error in the counting of prime geodesics with length at most log X. For the Picard manifold, Γ = PSL(2; Z[i]), we improve the classical bound of Sarnak, 5=3+ 13=8+ EΓ(X) = O(X ), to EΓ(X) = O(X ). In the process we obtain a mean subconvexity estimate for the Rankin{Selberg L-function attached to Maass{Hecke cusp forms. We also investigate the second moment of EΓ(X) for a general cofinite group Γ, and show that it is bounded by O(X16=5+). 1. Introduction Prime Geodesic Theorems describe the asymptotic behaviour of primitive closed geodesics on hyperbolic manifolds. The classical case is that of Riemann surfaces ΓnH2, where Γ ⊂ PSL(2; R) is a cofinite Fuchsian group. This problem was first studied by Huber [9, 10] and most importantly by Selberg (see e.g. [14, Theo- rem 10.5]) as a consequence of his trace formula. More specifically, let Γ(X) denote the analogue of the summatory von Mangoldt function, namely X Γ(X) = ΛΓ(N(P )); N(P )≤X where ΛΓ(N(P )) = log N(P0), if P is a power of the primitive hyperbolic conjugacy class P0, and zero otherwise. Here N(P ) denotes the norm of P , and the length of the closed geodesic corresponding to P is log N(P ). Selberg proved that, as Date: August 21, 2018. 2010 Mathematics Subject Classification. Primary 11F72; Secondary 11M36, 11L05. Key words and phrases. Prime Geodesic Theorem, Selberg Trace Formula, Kuznetsov Trace Formula, Kloosterman sums. The first author was partially supported by the Royal Swedish Academy of Sciences project no. MG2018-0002 and by the European Research Council under the European Community's Seventh Framework Programme (FP7/2007-2013) with ERC Grant Agreement nr. 615722 MOTMELSUM. arXiv:1712.00880v2 [math.NT] 20 Aug 2018 The second author would like to thank the School of Mathematics of University of Bristol and the Mathematics department of King's College London for their support and hospitality during the academic year 2016-17. During these visits, he received funding from an LMS 150th Anniversary Postdoctoral Mobility Grant 2016-17 and the European Unions Seventh Framework Programme (FP7/2007-2013) / ERC grant agreement no. 335141 Nodal. He is also currently supported by the Labex CEMPI (ANR-11-LABX-0007-01). The third author was supported by a Leibniz fellowship, and thanks the MFO for excellent working conditions, and by a \Ing. Giorgio Schirillo" postdoctoral grant by the \Istituto Nazionale di Alta Matematica". The last author would like to thank the Mathematics department at KTH in Stockholm, where he received funding from KAW 2013.0327. He would also like to thank the Department of Mathematics and Statistics at McGill University and the Centre de Recherches Math´ematiquesfor their hospitality. We also want to thank Yiannis Petridis, Maksym Radziwill and Kannan Soundararajan for their helpful comments. 1 2 BALKANOVA, CHATZAKOS, CHERUBINI, FROLENKOV, AND LAAKSONEN X ! 1, we have X Xsj Γ(X) = + EΓ(X); sj 1=2<sj ≤1 where the full main term is a finite sum that comes from the small eigenvalues of the hyperbolic Laplacian, λj = sj(1 − sj) < 1=4, and the error term EΓ(X) is bounded by O(X3=4). When Γ is arithmetic, further improvements on the bound for the error term were deduced by Iwaniec [13, Theorem 2], Luo and Sarnak [20, Theorem 1.4] (see also Koyama [16]), and Cai [3]. The crucial step in all of these works is proving a non-trivial bound on spectral exponential sums (see Section3). Recently, for the modular group, Soundararajan and Young [28, Theorem 1.1] proved the currently best known result, which gives 25=36+ EΓ(X) = O(X ): They do this by exploiting a connection to Dirichlet L-functions and using an inequality of Conrey and Iwaniec [5, Corollary 1.5] to estimate their value on the critical line. In this paper we study the Prime Geodesic Theorem in the three-dimensional 3 hyperbolic space H . Let Γ be a cofinite Kleinian group and let Γ(X) be the analogous summatory von Mangoldt function attached to Γ, which counts hy- perbolic (and loxodromic) conjugacy classes in the group. The small eigenvalues λj = sj(2 − sj) < 1 provide a finite number of terms that form the full main term of Γ(X), as X ! 1, namely X Xsj MΓ(X) := ; sj 1<sj ≤2 and we write EΓ(X) = Γ(X) − MΓ(X): For general cofinite groups, Sarnak [26, Theorem 5.1] proved the following non- trivial bound for the error term: 5=3+ (1.1) EΓ(X) = O(X ): For the Picard group, Γ = PSL(2; Z[i]), Koyama [17, Theorem 1.1] improved this 11=7+ bound conditionally to O(X ) by assuming a mean Lindel¨ofhypothesis for symmetric square L-functions attached to Maass forms on ΓnH3. Our main result is the following unconditional improvement of (1.1) in the arithmetic case Γ = PSL(2; Z[i]). Theorem 1.1. Let Γ = PSL(2; Z[i]). Then 13=8+ EΓ(X) = O(X ): The error term EΓ can be understood as a summation over the non-trivial zeros of the Selberg zeta function through an explicit formula (see (3.1)). This is analogous to the relationship between the explicit formula for the Riemann zeta function and the Prime Number Theorem, and it is the starting point in our proof of Theorem 1.1. The fundamental tool in our proof is the Kuznetsov formula (Theorem 2.3), which relates the problem to the study of Kloosterman sums for Gaussian integers. In two dimensions Cherubini and Guerreiro examined the second moment of the error term for general cofinite Fuchsian groups, and they improved the error term on average [4, Theorems 1.1, 1.4]. We initiate the study of the second moment of EΓ(X) in three dimensions and prove the following theorem by an application of the Selberg trace formula. PRIME GEODESIC THEOREM IN THE HYPERBOLIC 3-SPACE 3 Theorem 1.2. Let Γ be a cofinite Kleinian group. Let V , ∆ be sufficiently large positive real numbers with ∆ ≤ V . Then Z V +∆ 1 2 18=5 −2=5 2=5 jEΓ(x)j dx V ∆ (log V ) : ∆ V 8=5+ By taking ∆ = V in the above theorem, we see that EΓ(X) X on average. More precisely, we obtain the following result. Corollary 1.3. For every η > 8=5 there exists a set A ⊆ [2; 1) of finite logarithmic R −1 η measure (i.e. for which A x dx is finite), such that EΓ(X) = O(X ) for X ! 1, X2 = A. Remark 1.4. Using a mean-to-max argument, we recover Sarnak's bound (1.1) as a corollary of Theorem 1.2. Indeed, a bound of the type O(V β∆−γ ) for the second α moment of EΓ(X) in short intervals leads to the estimate EΓ(X) = O(X ), where β+γ α = 2+γ . Theorem 1.2 allows β = 18=5 + and γ = 2=5, leading to (1.1). Remark 1.5. It is interesting to speculate on what is expected for the error term EΓ(X). In analogy with two dimensions, it is tempting to say that one should 1+ expect a bound of size the square root of the main term, namely EΓ(X) X . However, there are reasons indicating that it is not possible to reach this bound, the most relevant being that the explicit formula (3.1) in its current form has a 3=2+ 1=2 natural barrier EΓ(X) X by taking T = X . We discuss this issue further in Remark 3.1, after the definition of the explicit formula and of the spectral exponential sum associated to it. Remark 1.6. By using a different method, the authors of [1] proved an estimate E(X) X3=2+θ=2, which explicitly depends on the subconvexity exponent θ for quadratic Dirichlet L-functions over Z[i]. Finally, we recall the connection between class numbers and the Prime Geodesic Theorem for the Picard group. Denote by D the set of discriminants of binary quadratic forms (over Z[i]), so that 2 D = fm 2 Z[i]: m ≡ y (mod4) for some y 2 Z[i] and m is not a perfect squareg; and let h(d) be the number of classes of primitive binary quadratic forms of dis- criminant d. Moreover, for d 2 D, consider the Pell-type equation t2 − du2 = 4. All the solutions are generated by a fundamental unit p t + du = 0 0 ; j j > 1: d 2 d Sarnak [26, Corollary 4.1] proved the identity X 2 2 2 h(d) log jdj = πΓ(x ) = #fP0 primitive hyperbolic, N(P0) ≤ x g; d2D jd|≤x which we can relate to Γ via summation by parts. Thus every result on Γ(x) translates immediately to a statement on the average size of the class numbers h(d) (see [26, Theorem 7.1]). Therefore, we obtain the following corollary as a consequence of Theorem 1.1 and Theorem 1.2. Corollary 1.7. For every > 0 we have X 4 13=4+ (1.2) h(d) = Li(X ) + O(X ): d2D jd|≤X 4 BALKANOVA, CHATZAKOS, CHERUBINI, FROLENKOV, AND LAAKSONEN On average over X, we obtain, for V ≥ ∆ 1, Z V +∆ 2 1 X 4 18=5+ −2=5 (1.3) h(d) − Li(X ) dX V ∆ : ∆ V d2D jd|≤X The rest of this paper is organized as follows: in Section2 we discuss the necessary background material and the main tools of our proofs, namely the Selberg and the Kuznetsov trace formulas, whereas in Sections3 and4 we prove Theorem 1.1 and Theorem 1.2, respectively.