A Kloosterman Sum in a Relative Trace Formula For

REPRESENTATION THEORY An Electronic Journal of the American Mathematical Society Volume 2, Pages 370–392 (September 16, 1998) S 1088-4165(98)00049-1

A KLOOSTERMAN SUM IN A RELATIVE

TRACE FORMULA FOR GL4

YANGBO YE

Abstract. We study a Kloosterman sum for GL4 and prove that it is equal to an exponential sum over a quadratic number ﬁeld. This identity has applications in a relative trace formula for GL4 which might be used to give a new proof of quadratic base change and characterize its image.

1. Introduction A main result of this article is the following identity of exponential sums. Theorem 1. Let τ be a non-zero square-free integer which is not equal to 1.Letb be a non-zero integer and c a positive odd integer such that (b, c)=(τ,c)=1.Then

e2πi x1+x2+¯x2x¯3+x2x3x¯4+bx¯1x4+bx¯1x3 /c 1 x c, ≤ i≤ (xXi,c)=1 for i=1,... ,4 (1) 2πi n+bn(x2 τy2)+2¯n(x +x ) /c = e 1− 1 1 2 . 1 n,x ,x ,y ,y c, ≤ 1 2 1 2≤ (n,c)=(x2 τy2X,c)=(x2 τy2,c)=1, 1− 1 2− 2 x2 τy2 x2 τy2 (mod c) 1− 1≡ 2− 2 Here we denote by x¯ the inverse of x modulo c. The sum on the left side of the above identity can be regarded as a generalization of the classical Kloosterman sum e2πi(x+bx¯)/c 1 x c, (x,c≤X)=1≤ where (b, c)=1andxx¯ 1(modc). We will see that it is indeed a Kloosterman ≡ sum for the group GL4. The expression on the right side of (1) is an exponential sum taken over certain algebraic integers of the quadratic number field Q(√τ). Other identities of this kind have been studied by several authors including Zagier [24], Katz [12], Jacquet and Ye [9], Duke and Iwaniec [3], Ye [21] and [23], and Mao and Rallis [13]. Some of these known identities have applications in automorphic forms and representation theory. To look at similar applications of the identity in Theorem 1 let us consider its p-adic version. Let F be a p-adic field of characteristic zero and L = F (√τ) an unramified quadratic extension field of F with τ F . Assume that 2 = τ =1.Denote ∈ | |F | |F Received by the editors April 9, 1997 and, in revised form, August 27, 1998. 1991 Mathematics Subject Classification. Primary 11L05; Secondary 11F70, 22E55.

c 1998 American Mathematical Society

370 A KLOOSTERMAN SUM FOR GL4 371 by A the group of diagonal matrices in GL4,byNthe group of upper triangular matrices with unit diagonal entries in GL4,andbyK(F) the maximal compact subgroup of GL4(F ) which consists of matrices with entries in RF and determinants in RF×.HereRFis the ring of integers in F and RF× is the group of invertible 1 1 elements in RF .Letw= 1 and a =diag(a1,a1,a2,a2) with a1,a2 F×. 1 ∈ Denote by Uw(F ) the subgroup of N(F ) consisting of matrices whose entries at (1, 2) and (3, 4) positions are both zero, and by Nw(F ) the subgroup of N(L) deﬁned by tnwn¯ = w. Let ψF be a non-trivial character of F of order zero; hence ψF is trivial on RF 1 but non-trivial on $F− RF ,where$F is a prime element in F . Deﬁne a character θF on N(F )byθF(n)=ψF( 1 i

(2) θF (un) du dn = θF (nn¯) dn.

u U (F )Z,n N(F) n N (FZ) N(L), ∈ w ∈ ∈ w \ tuwan K(F ) tnwan¯ K(L) ∈ ∈

The left side of (2) is a p-adic Kloosterman sum for GL4 in the integral form as in Friedberg [4] and Stevens [14]. To see the significance of the identities in Theorems 1 and 2, let us introduce a relative trace formula for GLn and look at its applications in representation theory. Let E be an algebraic number field and E0 = E(√τ) a quadratic extension of E with τ E×.DenotebyE and E0 the adele rings of E and E0, respectively, ∈ A A and by EA× and EA0× the idele groups of E and E0, respectively. Then EA is the restricted product of local fields Ev over all places v of E and EA0 is the restricted product of local fields Ew0 over all places w of E0.Forz= wzwin EA0 the Galois conjugation z z¯ is defined using the Galois conjugation on E0 over E.Denoteby N the global7→ norm map. Then the global norm-oneQ group E 1 is the kernel EA0 /EA A0 1 of N , and we have E = z/z¯ z E0× .Ifvis inert in E ,wedenoteby EA0 /EA A0 A 0 + { | ∈ } + Ev the group of elements of Ev× which are norms. Define EA as the group of + z = v zv EA× such that zv Ev for every inert place v. From the∈ exact sequence ∈ Q

NE /E 1 A0 A + 1 E 0 E 0× E 1 −→ A −→ A −−−−−−→ A −→ 1 we know that if an idele class character χ0 of E0 is trivial on EA0 , then there is an idele class character χ of E such that χ = χ N . This character χ is uniquely 0 EA0 /EA determined up to a multiplication by the idele◦ class character η of E attached to the quadratic extension ﬁeld E0. We note that E× and E0× are indeed GL1(E)andGL1(E0), respectively. Also, 1 E0 is actually the unitary group of one variable in E0 over E. This suggests a possible generalization of the above example to GLn. Let S(E) be the set of invertible Hermitian matrices in GL (E0). For any s n ∈ S(E)wedenotebyHs(E) the corresponding unitary group:

t H (E)= h GL (E0) hsh¯ = s . s { ∈ n | } 372 YANGBO YE

An automorphic irreducible cuspidal representation π0 of GLn(EA0 ) with central character ω0 is said to be Hs-distinguished if the periodic integral

µ(φ)= φ(h)dh

H (E)ZH (E ) s \ s A is a non-zero linear form on the space of π0. Then the proposition below is a generalization of our example to GLn.

Proposition 1. Let π0 be an Hs -distinguished representation of GLn(EA0 ) with central character ω0 for a unitary group Hs.Thenπ0is the quadratic base change of an automorphic irreducible cuspidal representation π of GLn(EA) with a central character ω. The central characters satisfy the condition ω0 = ω NE /E . ◦ A0 A

In the case of GL2 a representation π0 is Hs-distinguished if its Asai L-function has a pole at s = 1 (Asai [2]). For n>2 a similar situation is also true. There- fore, Proposition 1 characterizes quadratic base change by analytic behavior of L-functions. A proof of this proposition can be found in Harder, Langlands, and Rapoport [6] and Jacquet [7]. The converse of this proposition is expected to be true also but the proof appears much more diﬃcult. For GL2 the converse is proved in Harder, Langlands, and Rapoport [6], Ye [16] and [17], and Jacquet and Ye [8]. For GL3 it is proved in a series of papers by Jacquet and Ye ([7], [9], [10], [11], and [18]). The main technique used in [7] through [11] and [16] through [18] is a relative trace formula which is indeed an equality of two trace formulas. On one side is a Kuznetsov trace formula and on the other side is a so-called relative Kloosterman integral.

First let us look at the Kuznetsov trace formula for GLn.Letf= vfvbe a smooth function of compact support on GLn(EA). We want to assume that at Q+ any inert place v of E the local function fv is supported on the group GLn (Ev)= + g GLn(Ev) det g Ev .Letχbe an idele class character of E. Then we deﬁne the∈ kernel function ∈

1 (3) Kf (g,h)= f(zg− ξh)χ(z)d×z

Z+(E)ZZ+(E ) ξ GLn(E) \ A ∈X + where Z (E) is the set of matrices diag(z,...,z) in the center Z(E)ofGLn(E) + with z being in the subgroup E of E× consisting of norms. Let ψ = v ψv be a non-trivial additive character of EA trivial on E such that for almost all v the local Q character ψv has order zero, i.e., ψv is trivial on the ring of integer Rv of Ev but 1 non-trivial on $v− Rv,where$v is a prime element in Rv.DenotebyNthe group of upper triangular matrices with unit diagonal entries. We deﬁne a character θ on N by θ(n)=ψ( 1 i

+ S (EA). Now we deﬁne a kernel function

t KΦ(g)= Φ(z gξg¯ )χ(z) dz

Z+(E)ZZ+(E ) ξ S(E) \ A ∈X and deﬁne the relative Kloosterman integral

(5) KΦ(n)θ(nn¯) dn.

N(E )ZN(E ) 0 \ A0 The relative trace formula is then

t 1 Kf ( n1,n2)θ(n1− n2)dn1 dn2 N(E) ZN(E ) N(E) ZN(E ) \ A \ A

= KΦ(n)θ(nn¯) dn.

N(E )ZN(E ) 0 \ A0

Here the equality means that for a given smooth function f = v fv of compact support on GL(n, FA) there exists a smooth function Φ = Φv of compact sup- v Q port on S(FA) or a finite sum of these Φ, and vice versa, such that the above relative trace formula holds. There are restrictions on the way in whichQ one chooses these functions: (i) The matching of f and Φ should be made through matching of local functions fv and Φw. (ii) At an inert unramified non-Archimedean place v of E the characteristic function f0 of K(Ev) should be matched with the characteristic function Φ0 of K(E0 ) S(E ). w ∩ v (iii) At a non-Archimedean place v of E which splits into w1 and w2 the characteristic function of K(Ev) should be matched with Φ1 Φ2 via convolution where Φ is the characteristic function of K(E ) S(E ). ⊗ i w0 i v (iv) At an inert unramified non-Archimedean∩ place v of E, a compactly supported bi-K(Ev)-invariant function fv should be matched with a function Φw via the base change map of Hecke algebras (see Arthur and Clozel [1]). (v) At a non-Archimedean place v of E which splits into w1 and w2 acompactly supported bi-K(Ev)-invariant function fv should be matched with a function of the form Φ1 Φ2 via convolution. Here the convolution is used as the base change map of Hecke⊗ algebras in splitting cases. The matchings in (iv) and (v) are called the fundamental lemma of the relative trace formula while the matchings in (ii) and (iii) are called the fundamental lemma for unit elements of Hecke algebras. The splitting cases (iii) and (v) are easy. Proving the matching in (ii) and hence the the fundamental lemma for unit elements is the first step in establishing the relative trace formula. Once this has been done, one might be able to prove the matchings in (iv) using the techniques discussed in Ye [20]. One might then be able to deduce certain matchings in (i) from the fundamental lemma using the Shalika germ expansions introduced in Jacquet and Ye [10] and [11] and exponential sum expansions in Ye [19] and [20]. To apply the relative trace formula to base change problems one needs to study continuous spectrum of the relative trace following the work of Jacquet [7]. 374 YANGBO YE

The present work is a step toward a proof of the matchings in (ii) for GL4, i.e., the fundamental lemma for unit elements of Hecke algebras. More precisely, Theorem 2 proves the matchings in (ii) for GL4 for certain local orbital integrals which will be deﬁned below. By the Bruhat decomposition the group GLn(E) can be decomposed into the disjoint union of double cosets tN(E)wA(E)N(E), where w goes over the Weyl group W of the group A of diagonal matrices. Applying this decomposition to the sum in the kernel function Kf (g,h) in (3) we can express the Kuznetsov trace formula in (4) as a sum of global orbital integrals

I(waz, f)χ(z) dz

w a + Z X XZ (EA) where I(wa, f)= f(tuwan)θ(un) du dn.

u UwZ(EA), n∈ N(E ) ∈ A Here the sums are taken over w and a of the form

w1 a1 .. .. (6) w =  .  ,a= ., w a  r  r 1     .. where wi = . GLn (E), ai is in the center of GLn (E), and n1 + +nr = 1 ∈ i i ··· n.Forsucha wwe denote by Uw the unipotent subgroup of GLn consisting of I1 .∗∗.. matrices ∗ where Ii is the identity matrix in GLni . Note that in our Ir computation other Weyl matrices w yield zero orbital integrals. Similarly, using a double coset decomposition of GLn(E0) the relative Klooster- man integral in (5) can be written as a sum of global orbital integrals

J(waz, Φ)χ(z) dz

w a + Z X XZ (EA) where J(wa, Φ) = Φ(tnwan¯ )θ(nn¯) dn.

N (E Z) N(E ) w A \ A0 Here the sums are taken over the same w and a as above and Nw(EA)isthe t subgroup of N(EA0 ) deﬁned by nwn¯ = w. Consequently, the relative trace formula can be reduced to identities of global orbital integrals I(wa, f)=J(wa, Φ) for any w and a as in (6) with ai being in the center of GLni (EA). Express- ing I(wa, f)andJ(wa, Φ) as products of local orbital integrals Iv(wa, fv)and Jw(wa, Φw), we need to prove that

(7) Iv(wa, fv)=Jw(wa, Φw) A KLOOSTERMAN SUM FOR GL4 375

for all w and a of the form (6) but with ai being in the center of GLni (Ew), when v is inert in E0 with w lying above v.Inthecaseofvbeing non-Archimedean and unramiﬁed, proving (7) for fv being the characteristic function of K(Ev)andΦw being the characteristic function of K(Ew0 ) S(Ev) for all w is equivalent to proving the fundamental lemma for the unit elements∩ of Hecke algebras. Back to GL4, one needs to prove (7) for the following w:

1 1 1 1 w = ,w= , 1  1  21  1   1         1 1 1 1 w= ,w= ,w=I. 3 1 4 1 5 1   1         The case of w1 is trivial. The cases of w2 and w3 were proved in Ye [22]. In this article we will prove (7) for w4 with Ev = F , Ew0 = L,andψv =ψF.Thus,the only remaining unproved case for the fundamental lemma of unit elements of Hecke algebras is w5 = I. We want to point out that for the group GL2, the non-trivial case for the fun- 1 damental lemma is similar to our case of w2.ForGL3,thecasesof 1 and 1 1 1 are again similar to our cases of w2 and w3.Thecaseofw4does not appear 1 in GL2 and GL3. Exponential sums corresponding to w2 and w3 are hyper-Kloosterman sums which are studied by Katz [12], Friedberg [4], and Stevens [14]. The Klooster- man sum of the form on the left side of (2) for w4 in GL4 has also been studied by Friedberg [4] and Stevens [14]. What is new in the present paper is its new expression given on the left side of (1). Also new in this paper is certainly the identity of exponential sums in Theorem 1. This identity can be regarded as a lifting of the exponential sum on the left side of (1) to a quadratic number ﬁeld. Because of its connection with our relative trace formula, it might be a kind of manifestation of the underlying quadratic base change.

2. Proof of Theorem 1 We will use a local argument to prove the identity in Theorem 1. Let ψ =

ψR p< ψp be an additive character of QA which is trivial on Q such that its real ∞ 2πix component is given by ψR(x)=e− and each p-adic local character ψp has order Q 2πix equal to 0. Since ψ is trivial on Q, for any x Q we have e = ψp(x). ∈ p< Recall that we assumed that c is odd and (τ,c) = 1. For any prime divisor∞ p of c Q if τ =1,thenpsplits in E = Q(√τ); if τ = 1, then p is inert unramified in p p − E = Q(√τ). In the former case Qp E = E1p E2p is isomorphic to the direct ⊗Q ⊕ sum of two copies of Qp while in the latter case Qp E = Ep is an unramified ⊗Q quadratic extension field of Qp. Now we can write the identity in (1) in terms of local products 376 YANGBO YE

1 1 x2x3 bx4 bx3 ψp x1 + x2 + + + + c x2x3 x4 x1 x1 p c x R×/(1+cR ) Y| i∈ pX p for i=1,... ,4 m b x + x = ψ 1+ ψ tr 1 2 p p Ep/Qp c x1x¯1 ◦ cm p c, x ,x R× /(1+cR ), | 1 2 Ep Ep τY ∈ X p = 1 m R×/(1+cR ), − ∈ p p x1x¯1 x2x¯2(1+cRp) ∈ m b x1 + x2 + y1 + y2 ψp 1+ ψp . · c x1y1 cm p c, m,x ,x ,y ,y R×/(1+cR ), | 1 2 1 2 p p τY X∈ =1 x1y1 x2y2(1+cRp) p ∈ We can rewrite the sums on both sides of the above identity as local integrals. Theorem 1 is thus reduced to local identities in the following two lemmas. Lemma 1. Let F be a non-Archimedean local field of characteristic 0 with 2 =1. | |F Let L = F (√τ ) be an unramified quadratic extension field of F with τ RF×.Denote ∈ C by ψF a non-trivial character of F of order zero. For any b RF× and c $F RF× with C>0we have ∈ ∈ (8) 1 1 x2x3 bx4 bx3 ψF x1 + x2 + + + + dx1 dx2 dx3 dx4 c x2x3 x4 x1 x1 Z 4 (RF×)

C m b x1 + x2 = qF ψF 1+ ψF trL/F dm dx1 dx2. c x1x¯1 ◦ cm m ZR×, ∈ F x ,x R×, 1 2∈ L x x¯ x x¯ (1+$C R ) 1 1∈ 2 2 F F Lemma 2. Let F be a non-Archimedean local ﬁeld of characteristic 0 with 2 = | |F 1.DenotebyψF a non-trivial character of F of order 0. For any b RF× and C ∈ c $ R× with C>0we have ∈ F F 1 1 x2x3 bx4 bx3 ψF x1 + x2 + + + + dx1 dx2 dx3 dx4 c x2x3 x4 x1 x1 Z 4 (RF×)

C m b = qF ψF 1+ c x1y1 m,x ,x ,yZ ,y R×, 1 2 1 2∈ F x y x y (1+$C R ) 1 1∈ 2 2 F F x + x + y + y ψ 1 2 1 2 dm dx dx dy dy . · F cm 1 2 1 2 Lemma 2 is trivial; its proof is by changing variables. We will devote the rest of this section to Lemma 1.

Proof of Lemma 1. We first consider the integral on the left side of (8). We note that for any b ,b R an integral of the form 1 2 6∈ F b (9) ψ xb + 2 dx F 1 x Z RF× A KLOOSTERMAN SUM FOR GL4 377 is non-zero only if b1 b2RF×.Ifoneofb1and b2 is in RF but the other is not, the ∈ 1 integral in (9) is non-zero only if the latter is in $F− RF×. Applying these results to the integral with respect to x1 on the left side of (8) we get two non-vanishing cases: (i) C =1andx +x $ R ,and 3 4 ∈ F F (ii) C 1andx +x R×. ≥ 3 4 ∈ F In case (i) the integral can be computed directly. Namely, the integrand becomes x1 1 ψF ( + ) because the order of ψF is zero. The integral with respect to c cx2x3 1 x4 x3 + $F RF equals qF− and the integrals with respect to x1 and x2 are both ∈ 1 1 equal to qF− . With the integral with respect to x3 being 1 qF− we conclude that − 3 1 − case (i) yields qF− (1 qF− ). To compute case− (ii) we use a Mellin transform. Let χ be a multiplicative character of F .Ifχis ramified, we denote its conductor exponent by a(χ)which a is the smallest positive integer a such that χ is trivial on 1 + $F RF . We integrate 1 the expression on the left side of (8) against χ− (b) with respect to b R×: ∈ F 1 1 1 x2x3 bx4 bx3 χ− (b)ψF x1 + x2 + + + + db dx1 dx2 dx3 dx4. c x2x3 x4 x1 x1 Z 5 (RF×) x +x R× 3 4∈ F C Now we change variables successively from b to y0 = b(x3 + x4)/(cx1) $F− RF×, C C ∈ from x1 to y1 = x1/c $F− RF×, from x2 to y2 =1/(cx2x3) $F− RF×, from x4 to ∈ ∈ 2 x = x4/x3 RF× with x +1 RF×, and finally from x3 to y3 =(x+1)/(c xx3y2) C ∈ ∈ ∈ $F− RF×. Then the above integral becomes 4 x (10) q 4C χ 4(c) χ 1(y)ψ (y) dy χ 1 dx. F− − − F − (x +1)2 ZC Z $− R× x R×, F F ∈ F x+1 R× ∈ F If the character χ is unramified, the first integral in (10) vanishes unless C =1. 4 1 When C =1,wegetqF− (1 2qF− ). If χ is ramified, then the same integral vanishes unless a(χ)=C;inthiscase−

1 χ− (y)ψF(y)dy = ε(χ, ψ)

ZC $F− RF× where the local ε-factor is deﬁned as in Tate [15]. Together with our results for case (i) we conclude that the Mellin transform of the integral on the left side of (8) equals

3 1 2 qF− (1 qF− qF− )ifχis unramiﬁed and C =1; − − 4 2 4C 4 1 1 z q− χ (c) ε(χ, ψ ) χ− − F − F 4 (11)   RF ( 1+Z $F RF )  − ±  if χ is ramiﬁed and a(χ)=C; 0otherwise.   Here we rewrote the last integral in (10) by using a new variable z =(1 x)/(1+x) R ( 1+$ R ). − ∈ F − ± F F 378 YANGBO YE

Now we turn to the integral on the right side of (8). Again we integrate it against 1 χ− (b) with respect to b R× to get its Mellin transform ∈ F C 1 m b qF χ− (b)ψF 1+ c x1x¯1 b,mZR×, ∈ F x ,x R×, 1 2∈ L x x¯ x x¯ (1+$C R ) 1 1∈ 2 2 F F x + x ψ tr 1 2 db dm dx dx . · F ◦ L/F cm 1 2 C Changing variables successively from b to y0 = bm/(cx1x¯1) $F− RF×, from x1 C ∈ C to y1 = x1/(cm) $L− RL×, from x2 to y2 = x2/(cm) $L− RL× with y2y¯2 C ∈ C ∈ ∈ y1y¯1(1 + $F RF ), from m to y3 = m/c $F− RF×, and from y2 to ε RL× with εε¯ 1+$CR by y = y ε,weget ∈ ∈ ∈ F F 2 1 2 3C 1 4 1 qF− χ− (c) χ− (y)ψF(y)dy ZC $F− RF×

1 χ− N (y )ψ tr (y (1 + ε)) dε dy . · ◦ L/F 1 F ◦ L/F 1 1 ZC y $− R×, 1∈ L L ε R×, ∈ L εε¯ 1+$C R ∈ F F If χ is unramiﬁed, then the integral with respect to y above indicates that it is non-zero only when C = 1. In this non-zero case, we get

3 2 3 1 2 q− (q 1) dε dε = q− 1 q− q− . F F − − F − F − F ε 1+Z$LRL, Z ∈− ε RL×, εε¯ 1+$F RF ∈ ∈ 1+ε RL×, εε¯ 1+∈$ R ∈ F F If χ is ramified, then the integral with respect to y vanishes unless a(χ)=C.Since we assumed that the quadratic extension L is unramified and 2 F = 1, we know that the conductor exponent a(χ N )=Cwhen a(χ)=Cand| the| order of ψ tr ◦ L/F F ◦ L/F is again zero. Consequently, the integral with respect to y1 is non-zero only if C 1+ε RL× when a(χ)=C. Changing variables from y1 to x = y1(1 + ε) $L− RL× we get∈ the local ε-factor ε(χ N ,ψ tr ) multiplied by the integral∈ ◦ L/F F ◦ L/F 2C χ NL/F (1 + ε) dε = qF− χ(2 + ε +¯ε); ◦ C ε,1+ε R×/(1+$ R ), ε ZR×, L L L L ∈ XC ∈ εε¯ 1+$ RF 1+ε R×, F ∈ L ∈ εε¯ 1+$C R ∈ F F here we wrote the integral in terms of a finite sum. Now we can set ε = C 1 2 (1 + z√τ)/(1 z√τ) with z R /$ R .Thenχ(2 + ε +¯ε)=χ− (1 τz )/4 . − ∈ F F F − Using an integral again we get 2 C 1 1 τz χ N (1 + ε) dε = q− χ− − dz. ◦ L/F F 4 ε ZR×, RZF ∈ L 1+ε R×, ∈ L εε¯ 1+$C R ∈ F F A KLOOSTERMAN SUM FOR GL4 379

Therefore, the Mellin transform of the right side of (8) becomes

3 1 2 qF− (1 qF− qF− )ifχis unramified and C =1; − − 2 2 4C 4 1 1 τz qF− χ− (c) ε(χ, ψF ) ε(χ NL/F ,ψF trL/F ) χ− − dz (12)  ◦ ◦ 4  RZ  F  if χ is ramified and a(χ)=C; 0otherwise.   To compare the expressions in (11) and (12) we recall a well-known identity between local ε-factors (see, e.g., Gérardin and Labesse [5]) ε(χ, ψ )ε(χη, ψ )=ε(χ N ,ψ tr ) F F ◦ L/F F ◦ L/F where η is the quadratic multiplicative character of F attached to the extension field L.SinceLis assumed to be unramified over F ,wehaveη(c)=( 1)C for C C − any c $F RF×; hence ε(χη, ψ)=( 1) ε(χ, ψF )whena(χ)=C. Now we need a lemma.∈ −

Lemma 3. Let F be a non-Archimedean local field of characteristic 0 with 2 =1. | |F Let L = F (√τ ) be an unramified quadratic extension field of F with τ R×.Denote ∈ F by χ a ramified character of F × whose conductor exponent is a(χ)=C.Then

1 2 C 1 2 (13) χ− (1 z ) dz =( 1) χ− (1 τz )dz. − − − R ( 1+Z$ R ) RZF F−± F F Together, with the above remark, Lemma 3 implies that the corresponding expressions in (11) and (12) are equal. That is to say, the Mellin transforms of the two sides of (8) are the same for any multiplicative character χ. By Fourier’s inversion formula, we conclude that the two sides of (8) are equal. To complete the proof of Lemma 1, we still have to prove Lemma 3. When 1 1 C = 1, the left side of (13) equals q− (1+2 χ− (1 x)) where the sum is F − taken over all squares x =1inRF×/(1 + $F RF ), and the right side of (13) equals 1 1 6 P qF− (1+2 χ− (1 x)) where x goes over all non-squares in RF×/(1 + $F RF ). Then− (13) follows from− P 1 1 q− 1+2 χ− (1 x) F − x RF×/(1+$F RF ) ∈ is aX square, x=1 6 1 1 + q− 1+2 χ− (1 x) F − x RF×/(1+$F RF ) ∈is a non-squareX 1 1 =2qF− χ− (a)

a R×/(1+$ R ) ∈ F X F F =0 where a =1 x. When C>−1, the integrals on both side of (13) can be taken over z $[C/2]R . ∈ F F Indeed, if z $[C/2]R ,wecansetz=u(1 + v) and express an integral above as 6∈ F F a ﬁnite sum with respect to u of integrals with respect to v $[(C+1)/2]R .Since ∈ F F 380 YANGBO YE

[C/2] u $F RF , we can conclude that the integrals with respect to v vanish. This way6∈ the identity in (13) is reduced to

1 2 C 1 2 χ− (1 z ) dz =( 1) χ− (1 τz )dz. − − − [C/Z2] [C/Z2] $F RF $F RF

If C is even, then the integrands above are both equal to 1 and hence the equality. If C is odd, this equality can then be proved in the same way as what we did for the case of C =1. This completes the proof of Theorem 1.

3. The orbital integral IF (wa, f0) To prove Theorem 2 we have to compute the integrals on the two sides of (2). Recall that the integral on the left side of (2) is the local orbital integral

t IF (wa, f0)= f0(uwan)θF (un) du dn

u UZw(F), n∈ N(F) ∈

1 1 where f is the characteristic function of K(F )andw= 1 . Similarly, the 0 1 right side of (2) is the local orbital integral

t JF (wa, Φ0)= Φ0(nwan¯ )θF (nn¯) dn

N (FZ)N(L) w \ where Φ is the characteristic function of K(L) S(F )withthesamew. We will 0 ∩ compute IF (wa, f0) in this section and then JF (wa, Φ0)inthenextsectioninorder to show that they are equal.

w1 b1 Iy Let us denote w = w , a = , u = Uw(F), and n = 1 b2 I ∈ n1 x 1 N(F )in2 2 blocks, where bi is a scale matrix with diagonal entries n2− ∈ × equal toa and w = 1 . Then the matrix condition tuwan K(F )forthe i 1 1 ∈ integral deﬁning IF (wa, f0) becomes

w1b1n1 w1b1x (14) t t 1 K(F). yw1b1n1 yw1b1x + w1b2n− ∈ 2

We ﬁrst conclude from this matrix condition that a a R× and a R .If 1 2 ∈ F 1 ∈ F a,a R× we can see that I (wa, f ) = 1. Thus, from now on we assume that 1 2 ∈ F F 0 A A 1 m a $ R× and a $− R× with A>0. Then n = 1 with m 1 ∈ F F 2 ∈ F F 1 1 1 ∈ A t 1 $F− RF . We can also apply the automorphism g wG g− wG to the matrix 1 7→ 1 1 m2 condition in (14), where w = 1 . This way we can get n = with G 1 2 1 A t A m2 $F− RF . Back to (14) we know that x, w1 yw1n1 M2 2($F− RF ). Setting ∈ t ∈ × z = w1 yw1b1n1 M2 2(RF ) and changing from x to xb2 we can rewrite the last ∈ × A KLOOSTERMAN SUM FOR GL4 381 condition in (14) and get

8A 1 1 I (wa, f )=q θ Izn1− b1− F 0 F F I A m ,m Z$− R , 1 2∈ F F x,z M2 2(RF ), 1 ∈ 1 × A zn1− x n2− +M2 2($ RF ) ∈− × F n1 xb2 θF 1 dm1 dm2 dx dz. · n2− Denote x = x1 x2 and z = z1 z2 with x ,z R . There are three cases. x3 x4 z3 z4 i i ∈ F (i) m2 RF . Then the condition ∈ 1 1 A (15) zn1− x n2− +M2 2($FRF) ∈− × 1 implies that zn1− x K2 2(F), det(zx),det(x), det(z) R×, x, z, n1 K2 2(F ), ∈ × ∈ F ∈ × and m1 RF . ∈ M 1 (ii) m2 $F RF× with 0 >M> A. Then (15) implies that zn1− x GL2(F ), ∈ − M ∈ det(zx),det(x), det(z) RF×, x, z K2 2(F ), and m1 $F RF×. A ∈ ∈ × ∈ A (iii) m2 $F− RF×. Then from (15) we have m1 $F− RF×, x4,z1 RF×, x ,z $AR∈ ,andx ,x ,z ,z R . We will denote the∈ integrals corresponding∈ 3 3 ∈ F F 1 2 2 4 ∈ F to these three cases by I1, I2,andI3 so that IF (wa, f0)=I1+I2+I3. First we compute I1:

8A z3 I1 = qF ψF + a2x3 dm1 dm2 dx dz. a1 m ,mZ R , 1 2∈ F x,z K2 2(F ), 1 ∈ 1 × A zn1− x n2− +M2 2($ RF ) ∈− × F

By changing variables from x to n1x and integrating with respect to m1, m2, x2, x4, z1,andz2 successively we arrive at

5A z3 I1 = qF ψF + a2x3 dx1 dx3 dz3 dz4. a1 x ,x ,zZ,z R 1 3 3 4∈ F with x or x R× 1 3∈ F and z or z R×, 3 4∈ F x z +x z $AR 1 3 3 4∈ F F

A If x1 RF× we have z3 x3z4/x1 + $F RF .Ifx1 $FRF,thenx3 RF× and z ∈xz/x + $AR ∈−. By integrating z in the ﬁrst∈ case and z in the∈ second 4 ∈− 1 3 3 F F 3 4 case we can further compute I1 and conclude that

3 1 2 (16) I = q 1 q− + q− if A =1; 1 F − F F 3A 1 (17) = q 1 q− if A>1. F − F Next, let us turn to I2. After integrating with respect to m2 we get

7A z3 I2 = qF ψF + a2x3 + m1 m1x4z1 dm1 dx dz a1 − A

M m $ R×, 1 ∈ F F x ,x ,z ,z R×, 1 4 1 4 ∈ F x ,z R , 2 2 ∈ F M x ,z $− R , 3 3 ∈ F F x z m x z +x z 1+$AR , 1 1 − 1 3 1 3 2 ∈− F F x z m x z +x z 1+$AR , 2 3 − 1 4 3 4 4 ∈− F F x z m x z +x z $AR . 1 3 − 1 3 3 3 4 ∈ F F

We will consider two cases: A A (i) x3 $F RF (then z3 $F RF )and ∈ A ∈ (ii) x $ R (then z x R×). 3 6∈ F F 3 ∈ 3 F In case (i) the integrand simpliﬁes to ψF (m1 m1x4z1) and we have x4 A − A ∈ 1/(z m z )+$ R and z 1/(x m x )+$ R with x m x R× − 4 − 1 3 F F 1 ∈− 1− 1 3 F F 1 − 1 3 ∈ F and z4 m1z3 RF×. After integrating the integrals with respect to x4 and z1, changing− variables∈ from x to x = x m x and from z to z = z m z ,and 1 1 − 1 3 4 4 − 1 3 integrating with respect to x3 and z3,weget

m q3A ψ m 1 dm dx dz. F F 1 − xz 1 A

By integrating with respect to x1 we see that this integral vanishes unless M = 1. 3A 1 1 − Computing the case of M = 1, we get qF − (1 qF− ) for case (i). − X − In case (ii) we set x3,z3 $F RF× with M X

5A qF A

From the last condition attached to the integral we know that z3 x3z4/ (x mx)+$AR . Hence, the integral with respect to z vanishes∈− unless 1− 1 3 F F 3 A KLOOSTERMAN SUM FOR GL4 383

A +2M 0. Then we get for case (ii) ≥ 4A x3z4 m1 qF ψF + a2x3 + m1 dm1 dx dz −a1(x1 m1x3) − x1z4 A/2 M<0 ZM − ≤ m $ R×, − X 1∈ F F x1,z4 RF×, ∈M x $− R , 3∈ F F x m x R× 1− 1 3∈ F 4A x3z4 m1 qF ψF +a2x3 +m1 dm1 dx dz. − −a1(x1 m1x3) − x1z4 A/2 M<0 ZM − ≤ m $ R×, − X 1∈ F F x ,z R×, 1 4∈ F x $AR 3∈ F F The second integral above is the same as the integral in case (i) because the inte- 3A 1 1 grand is actually equal to ψ (m m /(x z )); we thus get the same q − (1 q− ). F 1 − 1 1 4 F − F For the ﬁrst integral we change variables from x3 to x = m1x3 RF with M ∈ x x R× and from z to z = z /m $− R× to get − 1 ∈ F 4 4 1 ∈ F F 4A xz a2x 1 qF ψF + +m1 dm1 dx dx1 dz. −a1(x x1) m1 − x1z A/2 M<0 ZM − − X≤ m1 $F RF×, x∈ R , ∈ F x1,x x1 RF×, − M∈ z $− R× ∈ F F Applying our results on (9) to the integral with respect to z, we conclude that A+2M A+2M x $F RF× if M< 1andx $F RF if M = 1. We claim that the integral∈ vanishes when M<− 1and∈A+2M>0. Indeed, we− change variables from − x to a1x and set b = a1a2. Integrating the integral with respect to b RF× against 1 ∈ χ− (b) we get the Mellin transform

3A 1 qF χ− (b) db A/2

The integral with respect to y1 vanishes if a(χ) = M .Ifa(χ)= M 2, the integral with respect to y equals zero because6 we− can set y as y(1− + c)≥ with [M/2] c $F− RF and integrate with respect to c. Since the Mellin transform of our integral∈ vanishes for any character χ when M< 1andA+2M>0, we prove the claim. − 384 YANGBO YE

Now we compute the integral when M = 1andA+2M>0, i.e., A>2. Since the integrand becomes −

xz a2x 1 ψF + + m1 , a1x1 m1 − x1z A+2M we can integrate with respect to x $F RF . The integral is non-zero only if 2 A ∈ z/(a x )+a /m $ − R . Consequently, we get 1 1 2 1 ∈ F F 3A+2 1 qF ψF m1 dm1 dx1 dz. − x1z 1 m $Z− R×, 1∈ F F x R×, 1∈ F z $F RF×, ∈ 2 A z/(a x )+a /m $ − R 1 1 2 1∈ F F 2 The last condition attached to the integral implies that z a1a2x1/m1 + $F RF . After integrating with respect to z we get ∈−

3A 1 qF ψF m1 1+ 2 dm1 dx1 a1a2x1 1 m $Z− R× , 1∈ F F x R× 1∈ F = q3A q dx dx F F 1 − 1 x ZR×, x ZR× 1∈ F 1∈ F x2 1/(a a )+$ R 1∈− 1 2 F F 3A 1 which equals qF (1 + qF− )if a1a2 is a square in RF×/(1 + $F RF ) and equals 3A 1 − qF ( 1+qF− )if a1a2 is not a square in RF×/(1 + $F RF ). For− the case of−A +2M = 0, with A being even, we have

xz a2x 1 ψF + + m1 dm1 dx dx1 dz. −a1(x x1) m1 − x1z ZA/2 − m1 $F− RF×, ∈x R , ∈ F x1,x x1 RF×, −A/2∈ z $ R× ∈ F F A/2 4 For the above integral, now we choose c $ R× and set b = a c /a R×.We ∈ F F 2 1 ∈ F then change variables from m1 to y1 = cm1 RF×, from z to y2 = cz/a1 RF×, 2 ∈ 2− ∈ from x to y = a (x x )/c R×, and from x to y = a x /c R×.After 4 1 − 1 ∈ F 1 3 1 1 ∈ F collecting all of these results on I2 we get

4A 1 1 y2y3 I2 = qF ψF y1 + y2 + + c y2y3 y4 Z 4 (RF×) by by + 4 + 3 dy dy dy dy if A 2iseven y y 1 2 3 4 (18) 1 1 ≥ 3A 1 q (q− +1) if a a is a square in R×/(1 + $ R ) F F − 1 2 F F F and A>2 + 3A 1 qF (q− 1) if a1a2 is not a square in R×/(1 + $F RF )  F − − F and A>2.   A KLOOSTERMAN SUM FOR GL4 385

Now we compute

I = q8A ψ (m m ) dm dm dx dz 3 F F 1 − 2 1 2 Z where the integral is taken over A m ,m $− R×, 1 2 ∈ F F x ,x ,z ,z R , 1 2 2 4 ∈ F x ,z $AR , 3 3 ∈ F F x ,z R×, 4 1 ∈ F x z m x z +x z 1+$AR , 1 1 − 1 3 1 3 2 ∈− F F x z m x z +x z 1+$AR , 2 3 − 1 4 3 4 4 ∈− F F x z m x z +x z m +$AR . 2 1 − 1 4 1 4 2 ∈ 2 F F A If we integrate with respect to m2 ﬁrst, we will get qF− and the above integrand becomes ψF (m1 + m1x4z1). Having integrated with respect to x1, x2, x3, z2, z3, and z4 we get

3A I3 = qF ψF m1(1 + x4z1) dm1 dx4 dz1 Z A m $− R× , 1∈ F F x ,z R× 4 1∈ F 2 1 (19) = q (1 q− )ifA=1; F − F (20) =0 ifA>1. Collecting our results in (16) through (20) we get the following expression of IF (wa, f0). Lemma 4. Under the assumption of Lemma 1 we have

IF (wa, f0)=1 ifA=0; 3 =qF if A =1; =2q3A if A 3isoddand a a is a square in F ; F ≥ − 1 2 4A 1 1 = qF ψF x1 + x2 + c x2x3 Z 4 (RF×) x2x3 bx4 bx3 + + + dx1 dx2 dx3 dx4 x4 x1 x1 if A 2iseven; 3A 1 ≥ +qF (1 qF− )ifA=2; 3A − +2qF if A 4isevenand a1a2 is a square in F ; = 0 otherwise≥ . −

4. The orbital integral JF (wa, Φ0)

1 n− Iu 1 mi We write any element in Nw(F ) N(L)asn= 1 where ni = \ n2 I 1 with m1,m2 F and u M2 2(L). Then the matrix condition attached to the ∈ ∈ × integral deﬁning JF (wa, Φ0) on the right side of (2) becomes t 1 1 t 1 1 n1− w1b1n1− n1− w1b1n1− u (21) t t 1 1 t t 1 1 t K(L). u¯ n− w1b1n− u¯ n− w1b1n− u + n2w1b2n2 ∈ 1 1 1 1 386 YANGBO YE

First, we conclude from (21) that a1 RF and a1a2 RF× if the integral is non-zero. ∈ ∈ A If a1 RF×,thenJF(wa, Φ0) = 1. We will assume from now on that a1 $F RF× ∈ A A ∈ and a $− R× with A>0. Then from (21) we get m ,m $− R following 2 ∈ F F 1 2 ∈ F F z1 z2 the arguments used in Section 3. Changing variables from u to z = z3 z4 = t 1 1 n1− w1b1n1− u M2 2(RL) and from mi to mi/2weget ∈ × 8A JF(wa, Φ0)=qF z1 ψF(m2 m1)ψF trL/F dm1 dm2 dz. · − ◦ a1 Z A m ,m $− R , 1 2∈ F F z M2 2(RL), ∈ × t m1 1 01 A z¯ z a1a2 +M2 2($LRL) 10 ∈− 1m2 × As in the last section there are three cases:

(i) m1,m2 RF, z K2 2(L); ∈ M ∈ × (ii) m1,m2 $F RF×, z K2 2(L) with A

5A y1 J1 = qF ψF trL/F dy1 dy3. ◦ a1 y ,y Z R , 1 3∈ L y or y R×, 1 3∈ L y y¯ +¯y y $AR 1 3 1 3∈ F F A If y R×, the integral with respect to y yields q− and we get 1 ∈ L 3 F 4A y1 qF ψF trL/F dy1 ◦ a1 y ZR× 1∈ L 2 which equals qF when A = 1 and vanishes when A>1. If y1 $LRL,then − 3 2∈ y3 RL×.WhenA= 1 the integrand equals 1 and we get qF (1 qF− ). When A>1 we∈ change variables from y to y = y y¯ $ R and get − 1 0 1 3 ∈ L L 5A y0 qF ψF trL/F dy0 dy3 ◦ a1y¯3 y $ R withZy +¯y $AR , 0∈ L L 0 0∈ F F y R× 3∈ L 3A 1 which equals q (1 q− ). Adding these results together, we ﬁnally have F − F 3 1 2 J1 = qF (1 qF− qF− )ifA=1; 3A − 1− =q (1 q− )ifA>1. F − F To compute J2, we ﬁrst integrate with respect to m2.Then

7A m1z2z¯2 z1 J2=qF ψF m1 ψF trL/F dm1 dz1 dz2 dz3 dz4 − − a1a2 ◦ a1 A

3A m1z2z¯2 qF ψF m1 dm1 dz2. − − a1a2 A1. ∈ In case (ii) we have z a a /(m z¯ +¯z )+$ZR . Integrating with respect 2 ∈− 1 2 1 1 3 L L to z4 and z2 we arrive at

5A qF 0< M Z

5A 2M a1a2 uz qF qF ψF m 1+ ψF trL/F dm du dz. zz¯ ◦ a1m(1 + u) 0< M Z A/2− and apply the Mellin transform to the integral: −

5A 2M 1 qF qF χ− (b) db A/2

A+2Z M u $ R×, ∈ L L (1+u)(1+¯u) 1+$A+M R ∈ F F Since A+2M>0wecanset1+u=(1+w)(1+v√τ)/(1 v√τ) with w $A+M R − ∈ L L A+2M M 2 2 and v $ R×/(1 + $− R ) .Thenχ(uu¯)=χ( 4τv /(1 τv )) and the ∈ F F F F − − [M/2] sum with respect to v vanishes because we can set v = v0(1+c) with c $F− RF and sum over c. Since the Mellin transform for any character χ is zero,∈ we prove our claim. Back to the case of A +2M = 0, we point out that then Z = M = A + M − and u R×. Since the integral with respect to m vanishes if u $ R when ∈ L ∈ L L M< 1, we can take the integral with respect to u over RL. − 1 Now let us turn to the case of M = 1. Since m(1 + a1a2/(zz¯)) $F− RF and A 1 − ∈ (1 + u)(1 +u ¯) 1+$ − R , we can again change variables from z to z(1 + u)and ∈ F F thus erase the factor (1+u) from the integrand in (22). We can also set z = z0(1+v) with v $LRL. Integrating with respect to v we know that Z must be equal to A 1 in∈ order to have a non-zero integral. Consequently, we get − 5A 2 a1a2 uz qF − ψF m 1+ ψF trL/F dm du dz. zz¯ ◦ a1m 1 m $Z− R×, ∈ F F z RL×, ∈A 2 u $L− RL×, ∈ A 1 (1+u)(1+¯u) 1+$ − R ∈ F F 3A 1 1 Recall that from case (i) we got qF − (1 + qF− ). This is indeed equal to the above A 1 integral with u being taken over $L− RL. Adding this expression to the above A KLOOSTERMAN SUM FOR GL4 389

A 2 A 1 integral we thus can set u $L− RL with (1 + u)(1 +u ¯) 1+$F− RF.IfA>2, we can further compute the∈ integral with respect to u ∈

uz ψF trL/F du ◦ a1m AZ 2 u $L− RL, ∈ A 1 (1+u)(1+¯u) 1+$ − R ∈ F F uz = ψF trL/F du ◦ a1m AZ 2 u $L − RL, ∈ A 1 u+¯u $ − R ∈ F F w√τz = ψF trL/F dv dw ◦ a1m AZ 2 w $F − RF , ∈ A 1 v $ − R ∈ F F with u = v + w√τ. Since the order of ψF is zero, the last integral above equals 3 2A qF− if y $F RF and vanishes if y RF×,wherez=x+y√τ. Back to (22), when A>2wehave∈ ∈

a a q3A+1 ψ m 1+ 1 2 dm dz F F zz¯ 1 m $Z− R×, ∈ F F z R× with y $ R ∈ L ∈ F F a a = q3A ψ m 1+ 1 2 dm dx F F x2 1 m $Z− R×, ∈ F F x R× ∈ F q3A(q 1 +1) if a a is a square; = F F− 1 2 3A 1 − q (q− 1) if a a is not a square. ( F F − − 1 2 To summarize our computation we have

a a J = q4A ψ m 1+ 1 2 2 F F zz¯ ZA/2 m $− R×, ∈ F F z RL×, u∈R , ∈ L (1+u)(1+¯u) 1+$A/2R ∈ F F uz ψF trL/F dm du dz if A 2iseven · ◦ a1m ≥ 3A 1 q (q− +1) if a a is a square in R×/(1 + $ R ) F F − 1 2 F F F and A 3 + 3A 1 ≥ qF (q− 1) if a1a2 is not a square in R×/(1 + $F RF )  F − − F and A 3.  ≥   A/2 After changing variables from m to n = cm RF× where c $F RF×, from z to 2 ∈ ∈ x = c z/a R×, and from u to x = x (1 + u) R×,wecanwritetheabove 1 − 1 ∈ L 2 − 1 ∈ L 390 YANGBO YE integral as

9A/2 n b J2 = qF ψF 1+ c x1x¯1 n ZR×, ∈ F x ,x R×, 1 2∈ L x x¯ x x¯ +$A/2R 1 1∈ 2 2 F F x + x ψ tr 1 2 dn dx dx if A 2iseven · F ◦ L/F cn 1 2 ≥ 3A 1 q (q− +1) if a a is a square in R×/(1 + $ R ) F F − 1 2 F F F and A 3 + 3A 1 ≥ qF (q− 1) if a1a2 is not a square in R×/(1 + $F RF )  F − − F and A 3 ≥   4 where as before b = a c /a R×. 2 1 ∈ F Finally, we compute J3:

8A J3 = qF z1 ψF(m2 m1)ψF trL/F dm1 dm2 dz · − ◦ a1 Z A m ,m $− R×, 1 2∈ F F z M2 2(RL), ∈ × t m1 1 01 A z¯ z a1a2 +M2 2($LRL) 10 ∈− 1m2 × where the matrix condition is equivalent to

m z z¯ +¯z z +z z¯ $AR , 1 1 1 1 3 1 3 ∈ F F m z¯z +¯z z +z z¯ a a +$AR 1 1 2 1 4 2 3 ∈− 1 2 L L m z z¯ +¯z z +z z¯ a a m +$AR . 1 2 2 2 4 2 4 ∈− 1 2 2 F F

The last condition above implies that z2 RL×; hence from the second condition we get z $AR . Then we can integrate with∈ respect to m , z , z ,andz to get 1 ∈ L L 2 4 3 1