Fourier Transform and the Global Gan–Gross–Prasad Conjecture for Unitary Groups

Fourier transform and the global Gan–Gross–Prasad conjecture for unitary groups

Wei Zhang∗†‡

Abstract We conﬁrm the global Gan–Gross–Prasad conjecture for unitary groups under some local restrictions for the automorphic representations. We also obtain some result towards the Flicker–Rallis conjecture characterizing the image of weak base change from any unitary group via distinction by the general linear subgroup.

Contents

1 Introduction to Main results2

2 Relative trace formulae of Jacquet–Rallis6 2.1 Orbital integrals...... 7 2.2 RTF on the general linear group...... 11 2.3 RTF on unitary groups...... 13 2.4 Comparison: fundamental lemma and transfer...... 15 2.5 Proof of Theorem 1.1:(ii) =⇒ (i)...... 20 2.6 Proof of Theorem 1.4...... 21 2.7 Proof of Theorem 1.1:(i) =⇒ (ii)...... 24

3 Reduction steps 24 3.1 Reduction to Lie algebra...... 25 3.2 Reduction to local transfer around zero...... 30 ∗Address: Department of Mathematics, Columbia University, 2990 Broadway New York, NY 10027 USA. †Email: [email protected] ‡The author is partially supported by NSF Grant DMS #1001631.

1 4 Smooth transfer for Lie algebra 45 4.1 A relative local trace formula...... 45 4.2 A Davenport–Hasse type relation...... 52 4.3 A property under base change...... 57 4.4 All Fourier transforms preserve transfer...... 59 4.5 Completion of the proof of Theorem 2.6...... 63

A Appendix: Spherical characters for a strongly tempered pair By Atsushi Ichino, Wei Zhang 65

1 Introduction to Main results

The study of periods and heights related to automorphic forms and Shimura varieties have recently received a lot of attention. One pioneering example is the work of Harder–Langlands– Rapoport ([24]) on the Tate conjecture for Hilbert-Blumenthal modular surfaces. Another example which motivates the current paper is the study of the Neron–Tate heights of Heegner points or CM points on the modular curve X0(N) by Gross and Zagier ([20]) in 1980s, on Shimura curves by S. Zhang in 1990’s, completed by Yuan–Zhang–Zhang ([54]) recently, and Kudla–Rapoport–Yang ([37]), Bruinier–Ono ([5]) in various prospectives. At almost the same time as the Gross–Zagier’s work, Waldspurger ([49]) discovered a formula that relates certain toric periods to some Rankin L-series, the same type L-function appeared in the Gross–Zagier formula. In 1990’s, Gross and Prasad formulated a conjectural generalization of Waldspurger’s work to higher rank orthogonal groups ([18],[19]) (later reﬁned by Ichino–Ikeda ([28])). Recently, Gan, Gross and Prasad have generalized the conjectures to classical groups ([11]) including unitary groups and symplectic groups. The conjectures are on the relation between period integrals and certain L-values. The main result of this paper is to conﬁrm their conjecture for unitary groups under some local restrictions. In the following we want to describe the main results of the paper in more details.

Gan–Gross–Prasad conjecture for unitary groups. Let E/F be a quadratic extension of number fields. Let W be a (non-degenerate) hermitian space of dimension n and denote by 0 U(W ) the corresponding unitary group as an algebraic group over F . Let Gn := ResE/F GLn be the restriction of scalar of GLn. We recall the local base change map when a place v is split 0 or the representation is unramified. If a place v of F is split in E/F , we may identify Gn(Fv) with GLn(Fv)×GLn(Fv) and identify U(W )(Fv) with a subgroup consisting of elements of the t −1 t form (g, g ), g ∈ GLn(Fv) and g is the transpose of g. Let p1, p2 be the two isomorphisms between U(W )(Fv) with GLn(Fv) induced by the two projections from GLn(Fv) × GLn(Fv) to GLn(Fv). If πv is an irreducible admissible representation of U(W )(Fv), we define the ∗ ∗ 0 ∗ local base change BC(πv) to be the representation p1πv ⊗ p2πv of G (Fv) where pi πv is a representation of GLn(Fv) obtained by the isomorphism pi. Note that for the split case, the local base change map is injective. When v is non-split and U(W ) is unramified at v, there

2 is a local base change map at least when πv is an unramified representation of U(W )(Fv), cf. [11]. Now let π be a cuspidal automorphic representation of U(W )(A), an automorphic 0 representation Π of Gn(A) is called the weak base change of π if Πv is the local base change of πv for all but finitely many places v where πv is unramified ([22]). We will then denote it by BC(π). If Π = BC(π) is cuspidal, by strong multiplicity one for GLn, it is unique. Throughout this article, we will assume the following hypothesis on the base change Hypothesis (∗): For all W and all dimension n, the weak base change exists and satisfies the following local–global compatibility at all split places v: the v-component of BC(π) is the local base change of πv. Remark 1. By a result of Harris–Labbese ([22, Theorem 2.2.2]), the hypothesis is valid if (1) π have supercuspidal components at at least two split places and (2) either n is odd or all archimedean places of F are complex. 0 Let W, W be two hermitian spaces. Then for almost all v, the unitary groups U(W )(Fv) 0 0 and U(W )(Fv) are isomorphic. We say that two automorphic representations π, π of U(W )(A) 0 0 and U(W )(A) respectively are nearly equivalent if πv ' πv for all places v of F . Conjecturally, all automorphic representations in a Vogan’s L-packet form precisely a single nearly equiva- 0 lence class. By the strong multiplicity one theorem for GLn, if π, π are nearly equivalent, their weak base changes are the same. We recall the notion of (global) distinction following Jacquet. Let G be a reductive group over F and H a subgroup. Let A0(G) be the space of cuspidal automorphic forms on G(A). We define a period integral

`H : A0(G) → C Z φ 7→ φ(h)dh ZG∩H(A)H(F )\H(A) whenever the integral makes sense. Similarly, if χ is a character of H(F )\H(A), we deﬁne Z `H,χ(φ) = φ(h)χ(h)dh. ZG∩H(A)H(F )\H(A)

For a cuspidal automorphic representation π (viewed as a subrepresentation of A0(G)), we say that it is (χ-, resp.) distinguished by H if the linear functional `H (`H,χ, resp.) is nonzero when restricted to π. Even if the multiplicity one fails for G, this definition still makes sense as our π comes with a fixed embedding into A0(G). To state the main theorem of this paper on the global Gan–Gross–Prasad conjecture, we let W ⊂ V be a fixed pair of (non-degenerate) hermitian spaces of dimension n and n + 1 respectively. The embedding W ⊂ V induces an embedding of unitary groups ι : U(V ) ⊂ U(W ). We will consider U(W ) as a subgroup of U(V ) × U(W ) by the diagonal embedding.

Theorem 1.1. Let π be a cuspidal automorphic representation of U(V ) × U(W ). Suppose that

3 (1) All v|∞ is split in E/F .

(2) For two distinct places v ∈ {v1, v2} (non-archimedean) split in E/F , πv is supercuspidal. Then the following are equivalent

(i) L(Π,R, 1/2) 6= 0 for the weak base change Π of π. Here L(Π, R, s) = L(Πn+1 × Πn, s) is the Rankin–Selberg L-function if we write Π = Πn ⊗ Πn+1. (ii) For some hermitian space W 0 ⊂ V 0 of dimension n and n + 1 respectively, and some automorphic representation π0 of U(V 0) × U(W 0) nearly equivalent to π, such that π0 is distinguished by U(W 0).

Note that we don’t assume that the representation π0 occurs with multiplicity one in the 0 0 0 space of cuspidal automorphic forms A0(U(V ) × U(W )). By π we do mean a subspace of 0 0 A0(U(V ) × U(W )). The theorem confirms the global conjecture of Gan–Gross–Prasad for unitary group under the local restrictions (1) and (2). The two conditions are due to some technical issue we now describe. Our approach is by a simple version of Jacquet–Rallis relative trace formulae. The first assumption is due to the fact that we only prove the existence of smooth transfer for a p-adic field (cf. Remark2). The second assumption is due to the fact that we use a “cuspidal” test function at a split place and use a test function with nice support at another split place (cf. Remark3). We remark that previously, it is known by [15],[16] that (ii) =⇒ (i) for both unitary and orthogonal groups. Remark 2. In the archimedean case we have some partial result for the existence of smooth transfer (Theorem 3.18). If we assume the local–global compatibility of weak base change at a non-split archimedean place, we may replace the first assumption by the following: if v|∞ is non-split, then W, V are positive definite (hence πv is finite dimensional) and

HomU(W )(Fv)(πv, C) 6= 0.

Remark 3. In the theorem, we may weaken the second condition and only require that πv1 is supercuspidal and πv2 is tempered. Remark 4. We recall some by-no-means complete history related to this conjecture. In the lower rank cases, a lot of works have been done on the global Gan–Gross–Prasad conjecture for orthogonal groups: the work of Waldspurger on SO(2)×SO(3) ([49]), the work of Garrett ([13]), Piatetski-Shapiro–Rallis, Garret–Harris, Harris–Kudla ([21]), Gross–Kudla ([17]), and Ichino ([27]) on the case of SO(3)×SO(4) or the so-called Jacquet’s conjecture. For the general case, Ginzburg–Jiang–Rallis’s work ([15],[16] etc.) proves one direction of the conjectures for both orthogonal and the unitary cases.

4 Remark 5. The original local Gross–Prasad conjecture ([18],[19], for the orthogonal case) has also been essentially resolved based on a recent series of work by Waldspurger and Moeglin ([52],[53] etc.). It is expected that similar technique should work for the local conjecture of Gan–Gross–Prasad in the unitary case ([11]). But in our paper we won’t need this. According to the local conjecture of Gan–Gross–Prasad for unitary groups and the multiplicity one of π0, such relevant ([11]) pair (W 0,V 0) and π0 in the theorem should be unique. Remark 6. Concerning the reﬁned version of the Gan–Gross–Prasad conjecture of Ichino– Ikeda type ([28]) (explicitly stated by N. Harris [23] in the unitary case), we hope that the approach of trace formula may eventually establish the unitary case. One prototype is the case of SO(2)×SO(3) by Chen and Jacquet ([6]) where they essentially reproved the Waldspurger formula using trace formula.

An application to non-vanishing of central L-values We have an application to the existence of non-vanishing twist of Rankin–Selberg L-function.

Theorem 1.2. Let E/F be a quadratic extension of number ﬁelds such that all archimedean places are split. Let σ be a cuspidal automorphic representation of GLn+1(AE). Assume that σ is a weak base change of an automorphic representation of some unitary group U(V ) which is locally supercuspidal at two split places of F . Then there exists a cuspidal automorphic representation τ of GLn(AE) such that the central value of the Rankin–Selberg L-function 1 L(σ × τ, ) 6= 0. 2

× × Flicker–Rallis conjecture. Let η = ηE/F be the quadratic character of F \A associated to the quadratic extension E/F by class ﬁeld theory. By abuse of notation, we will denote by η the quadratic character η ◦ det (det being the determinant map) of GLn(A).

Conjecture 1.3 (Flicker–Rallis, [9]). An automorphic cuspidal representation Π on ResE/F GLn(A) is a weak base change from a cuspidal automorphic π on some unitary group of n-varaibles if and only if it is distinguished (ηE/F -distinguished, resp.) by GLn,F if n is odd (even, resp.). Another result of the paper is to conﬁrm one direction of Flicker-Rallis conjecture under the same local restrictions as in the previous theorem. In fact, this result is used in the proof of our previous theorem on the global Gan–Gross–Prasad conjecture for unitary groups.

Theorem 1.4. Suppose that a cuspidal automorphic representation π of U(W )(A) satisﬁes: (1) All v|∞ is split in E/F .

(2) For two distinct places v ∈ {v1, v2} (non-archimedean) split in E/F , πv is supercuspidal.

Then the weak base change BC(π) is (η-, resp.) distinguished by GLn,F if n is odd (even, resp.).

5 Remark 7. If Π is distinguished by GLn,F , then Π is conjugate self-dual ([9]). Moreover, the partial Asai L-function has a pole at s = 1 if and only if Π is distinguished by GLn,F ([8], [10]). In [14], it is further proved that if the central character of Π is distinguished, then Π is conjugate self-dual if and only if Π is distinguished (resp., distinguished or η-distinguished) if n is odd (resp., even). We briefly describe the contents of each section. In section 2, we prove the main theorems based on the existence of transfer. In section 3 we reduce the existence of transfer to the existence of local transfer around zero of the “Lie algebras” (an infinitesimal version). In section 4, we show the existence of smooth transfer on Lie algebras for a p-adic field . Acknowledgements. The author would like to thank A. Aizenbud, D. Goldfeld, B. Gross, H. Jacquet, D. Jiang, Y. Liu, D. Prasad, Y. Sakellaridis, B. Sun, Y. Tian, H. Xue, X. Yuan, Z. Yun, S. Zhang, X. Zhu for their help during the preparation of the paper. He also thanks the Morningside Center of Mathematics of Chinese Academy of Sciences, the Mathematical Science Center of Tsinghua University for their hospitality and support where part of the paper was written.

2 Relative trace formulae of Jacquet–Rallis

Let F be a field of character zero. For a reductive group H acting on an affine variety X, we say that a point x ∈ X(F ) is • H-semisimple if Hx is Zariski closed in X (when F is a local field, equivalently, H(F )x is closed in X(F ) for the analytic topology, cf. [2]).

• H-regular if the stabilizer Hx of x has the minimal dimension. If no confusion, we will simply use the words “semisimple” and “regular”. And we say that x is regular semisimple if it is regular and semisimple. In this paper, we will be interested in the following two cases

• X = G is a reductive group and H = H1 ×H2 is a product of two (reductive) subgroups of G where H1 (H2, resp.) acts by left (right, resp.) multiplication. • X = V is a vector space (considered as an affine variety) with an action by a reductive group H. For later use, we also recall that the categorical quotient of X by H (cf.[2],[39]) consists of a pair (Y, π) where Y is an algebraic variety and π : X → Y is an H-morphism such that for any pair (Y 0, π0) with H-morphism π0 : X → Y 0, there exists a unique morphism φ : Y → Y 0 such that π0 = φ ◦ π If such a pair exists, then it is unique up to a canonical isomorphism. When X is affine (in all our cases), the categorical quotient always exists. Indeed we may construct as follows. Consider the affine variety X//H := Spec O(X)H

6 together with the obvious quotient morphism

H π = πX,G : X → Spec O(X) .

Then (X//H, π) is a categorical quotient of X by H. By abuse of notation, we will also let π denote the induced map X(F ) → (X//H)(F ) if no confusion arises. Now we fix some notations. Let F be a number field or a local field, and let E be a quadratic extension.

0 • The general linear case. We will consider the F -algebraic group G = ResE/F (GLn+1 × GLn) 0 and two subgroups: H1 is the diagonal embedding of ResE/F GLn (where GLn is em- 0 0 bedded into GLn+1 by g 7→ diag[g, 1]) and H2 = GLn+1,F × GLn,F embedded into G in the obvious way. In this paper for an F -algebraic group H, we will denote by ZH the center of H. We note that Z 0 ∩ Z 0 is trivial. G H1 • The unitary case. We will also consider a pair of hermitian spaces over the quadratic extension E of F : V and a codimension one subspace W . Suppose that W is of dimension n. Without loss of generality, we may always assume V = W ⊕ Eu where u has norm one: (u, u) = 1. In particular, the isometric class of V is determined by W . We have an obvious embedding of unitary groups U(W ) ,→ U(V ). Let G = GW = U(V ) × U(W ) and let ∆ : U(W ) → G be the diagonal embedding with image H (or HW to emphasize the dependence on W ) as a subgroup of G.

× × For a number field F , let η = ηE/F : F \A → {±1} be the quadratic character associated to E/F by class field theory. By abuse of notation we will also denote by η the character of 0 H2(A) defined by η(h) := η(det(h1)) (η(det(h2)), resp.) if h = (h1, h2) ∈ GLn−1(A) × GLn(A) 0 × × × and n odd (even, resp.). Fix a character η : E \AE → C (not necessarily quadratic) such 0 0 that its restriction η |A× = η. We similarly define the local analogue ηv, ηv.

2.1 Orbital integrals We first introduce the local orbital integrals appearing in the relative trace formulae of Jacquet–Rallis. We refer [57, sec. 2] on important properties of orbits (namely, double cosets). Later on in sec. 3 we will also recall some of them. We now let F be a local field of characteristic zero. And let E be a quadratic semisimple F -algebra. Namely, E is either a quadratic extension or F × F . 0 0 0 We start with the general linear case. If an element γ ∈ G (F ) is H1 ×H2-regular semisim- 0 ∞ 0 ple, for simplicity we will say that it is regular semisimple. For such a γ and f ∈ Cc (G (F )), we define its orbital integral as: Z Z 0 0 −1 (2.1) O(γ, f ) := f (h1 γh2)η(h2)dh1dh2. 0 0 H1(F ) H2(F )

7 The integral converges absolutely. This depends on the choice of Haar measure. But in this paper, the choice of measure is not crucial since we will only concern non-vanishing problem. In the following, we always pre-assume that we have made a choice of a Haar measure on each group. The integral is then absolutely convergent. Up to ±1 (due to the character η), the integral depends only on the orbit of γ. 0 0 We may simplify the orbital integral as follows. Identify H1\G with ResE/F GLn+1. Then we consider the morphism between F -varieties:

ν : ResE/F GLn+1 → Sn+1 g 7→ gg¯−1 where Sn is the subvariety of ResE/F GLn defined by the equation ss¯ = 1. By Hilbert Satz-90, this defines an isomorphism of two affine varieties

ResE/F GLn+1/GLn+1,F ' Sn+1, and in the level of F -points:

GLn+1(E)/GLn+1(F ) ' Sn+1(F ).

0 0 We may integrate f over H1(F ) to get a function on ResE/F GLn+1(F ): Z ¯0 0 f (x) := f ((x, 1)h1)dh1, x ∈ ResE/F GLn+1(F ). 0 H1(F )

0 Now assuming that n is odd, so the character η on H2 is indeed only nontrivial on the component GLn+1,F . Then we may introduce a function on Sn+1(F ) as follows: when ν(x) = s ∈ S (F ), we deﬁne n+1 Z f˜0(x) := f¯0(xg)η0(xg)dg. GLn+1(F ) ˜0 ∞ ∞ Then f ∈ Cc (Sn+1(F )) and all functions in Cc (Sn+1(F )) arise this way. Now it is easy to see that for γ = (γ1, γ2): Z 0 0 −1 ˜0 −1 −1 (2.2) O(γ, f ) = η (det(γ1γ2 )) f (h sh)η(h)dh, s = ν(γ1γ2 ). GLn(F ) If n is even, we simply deﬁne in the above Z f˜0(x) := f¯0(xg)dg, GLn+1(F ) and then Z 0 ˜0 −1 −1 (2.3) O(γ, f ) = fv(h sh)η(h)dh, s = ν(γ1γ2 ). GLn(F )

8 Up to a sign, the integral on the RHS depends only on the orbit of s under the conjugation by GLn(F ). 0 0 0 Then an element γ = (γ1, γ2) ∈ G (F ) is H1 × H2-regular semisimple if and only if −1 s = ν(γ1γ2 )) ∈ S(F ) is GLn,F -regular semisimple. Recall that essentially from [44, §6], an element s ∈ Sn+1(F ) is GLn,F -regular semisimple if and only if

i+j ∗ ∆(s) := det(es e )i,j=0,1,...,n 6= 0 where e = (0, ...0, 1) is a row vector and e∗ its transpose. 0 0 0 We will also need to consider the action of H = ZG0 H1 ×H2 on G . Though the categorical 0 0 0 quotient of G by ZG0 H1 × H2 exists, we are not sure how to explicitely write down a set of 0 0 generators of invariant regular functions nor how to determine when γ is ZG0 H1 × H2-regular 0 0 semisimple. But we may give an explicit Zariski open subset consisting of ZG0 H1 ×H2-regular semisimple elements. It suﬃces to work with the space Sn+1. Then we have the action of × 2 ZG0 × GLn,F on Sn+1: h ∈ GLn,F acts by the conjugation, z = (z1, z2) ∈ ZG0 ' (E ) acts by −1 Galois-conjugate conjugation by z2 z1:

−1 −1 z ◦ s = (z2 z1)s(z1 z2).

0 0 The two subgroups ZGLn+1,F ⊂ ZG and {((1, z2), z2) ∈ ZG × GLn,F |z2 ∈ ZGLn,F } clearly A b act trivially on S . We let Z denote their product. We may write s = where n+1 0 c d A ∈ Mn(E), d ∈ E, etc.. At least we have the following ZG0 × GLn,F -invariants on Sn+1:

NE/F (tr(A)), NE/F d.

We say that s is Z-regular semisimple if s is GLn-regular semisimple and the above two invariants are nonzero. When E is a ﬁeld, this is equivalent to:

tr(A) 6= 0, d 6= 0, ∆(s) 6= 0.

Otherwise we replace “6= 0” by “∈ E×” in these inequalities. The Z-regular semisimple locus, denoted by Z, clearly forms a Zariski open dense subset in Sn+1.

Lemma 2.1. If s is Z-regular semisimple, the stabilizer of s is precisely Z0 and its ZG0 × GLn,F -orbit is closed. In particular, a Z-regular semisimple element is ZG0 × GLn,F -regular semisimple.

Proof. Suppose (z, h) ◦ s = s. As tr(A) 6= 0, d 6= 0, up to modiﬁcation by elements in Z0, we may assume that z = 1. Then the ﬁrst assertion follows from the fact that the stabilizer of s is trivial for the GLn-action on Sn+1. When tr(A) 6= 0, d 6= 0, besides NE/F (tr(A)), NE/F d, the following are also ZG0 × GLn,F -invariant: tr ∧i A cAjb , , 1 < i ≤ n, 0 ≤ j ≤ n − 1. (tr(A))i (tr(A))j+1d

9 0 Then we claim that two Z-reglar semisimple s, s are in the same ZG0 × GLn,F -orbit if and only if they have the same invariants (listed above). One direction is obvious. We now assume that s, s0 are Z-reglar semisimple and have the same invariants. In particular, the values of 0 0 NE/F (tr(A)), NE/F d are the same. Replacing s by zs for a suitable z ∈ ZG0 , we may assume that s0 and s have the same tr(A) and d. Then s, s0 have the same value for tr ∧i A, 1 ≤ i ≤ n j 0 and cA b, 0 ≤ j ≤ n − 1. Then by [57, sec.2], s and s are conjugate by GLn,F since they are also GLn,F -regular semisimple. Therefore, the ZG0 × GLn,F -orbit of s consists of s ∈ Sn+1 such that for a ﬁxed tuple (α, β, αi, βj)

NE/F (tr(A)) = α, NE/F = β, and tr ∧i A cAjb α = , β = 1 < i ≤ n, 0 ≤ j ≤ n − 1. i (tr(A))i j (tr(A))j+1d The second set of conditions can be rewritten as

i i j j+1 tr ∧ A − αi(tr(A)) = 0, cA b − βj(tr(A)) d = 0, for 1 < i ≤ n, 0 ≤ j ≤ n − 1. This shows that the ZG0 × GLn,F -orbit of s is Zariski closed.

0 Let χ be a character of the center Z 0 (F ) that is trivial on Z 0 (F ). If an element G H2 γ ∈ G0(F ) is Z-regular semisimple, we deﬁne a χ0-orbital integral: Z Z Z 0 0 −1 −1 0 (2.4) Oχ0 (γ, f ) := f (h1 z γh2)χ (z)η(h2)dzdh1dh2. 0 0 H1(F ) Z 0 (F )\H2(F ) ZG0 (F ) H2 The integral is then absolutely convergent. We now move to the unitary case. Similarly, we will simply use “regular semisimple” for ∞ the action of H × H on G. For a regular semisimple δ ∈ G(F ) and f ∈ Cc (G(F )), we deﬁne its orbital integral Z O(δ, f) = f(x−1δy)dxdy. H(F )×H(F ) It converges absolutely. Similar to the linear case, we may simplify the orbital integral O(δ, f). Introduce a new function on U(V )(F ): Z (2.5) f¯(g) = f(g, 1)h)dh, g ∈ U(V )(F ). U(W )(F )

Then for δ = (δn+1, δn) ∈ G(F ), we may rewrite the previous integral as Z ¯ −1 −1 (2.6) O(δ, f) = f(y (δn+1δn )y)dy. H(F )

10 We thus have the action of U(W ) on U(V ) by conjugation. Then the element δ = (δn+1, δn) ∈ −1 G(F ) is H ×H-regular semisimple if and only if δn+1δn ∈ U(V )(F ) is U(W )-regular semisimple for the conjugation action. An element δ ∈ U(V )(F ) is U(W )-regular semisimple if and only if δiu, i = 0, 1, ..., n, form an E-basis of V . We also need to consider the action of the center ZG. We similarly deﬁne the notion of Z-regular semisimple. Then Lemma 2.1 easily extends to the unitary case. Let χ be a character of the center ZG(F ). If an element δ ∈ G(F ) is Z-regular semisimple, we deﬁne a χ-orbital integral: Z Z −1 (2.7) Oχ(δ, f) := f(x zδy)χ(z)dzdxdy. H(F )×H(F ) ZG(F ) The integral is then absolutely convergent.

2.2 RTF on the general linear group We will use “RTF” to stand for “relative trace formula”. Now we recall the construction of 0 Jacquet–Rallis’ RTF on the general linear side ([35]). Fix a Haar measure on ZG0 (A), Hi(A) 0 (i = 1, 2) etc. and the counting measure on ZG0 (F ) Hi(F )(i = 1, 2) etc.. 0 ∞ 0 For f ∈ Cc (G (A)), we deﬁne a kernel function

X 0 −1 Kf 0 (x, y) = f (x γy). γ∈G0(F )

0 0 For a character χ of ZG0 (F )\ZG0 (A), we deﬁne the χ -part of the kernel function Z X 0 −1 −1 Kf 0,χ(x, y) = f (x z γy)χ(z)dz. ZG0 (F )\ZG0 (A) γ∈G(F )

We then consider a distribution on G0(A): Z Z 0 I(f ) = Kf 0 (h1, h2)η(h2)dh1dh2. 0 0 0 0 H1(F )\H1(A) H2(F )\H2(A) 0 0 Similarly, for a character χ of Z 0 (F )\Z 0 ( ) that is trivial on Z 0 ( ), we define the χ -part G G A H2 A of the distribution Z Z 0 Iχ0 (f ) = Kf 0,χ(h1, h2)η(h2)dh1dh2. 0 0 0 0 H1(F )\H1(A) Z 0 (A)H2(F )\H2(A) H2 Note that in general these integrals may diverge. But for our purpose, it suffices to consider 0 0 ∞ 0 test functions f satisfying some local conditions. We say that a function f ∈ Cc (G (A)) is 0 0 0 nice with respect to χ if it is decomposable f = ⊗vfv and satisfies: 0 ∞ 0 • For at least one place v1, fv1 ∈ Cc (G (Fv1 )) is a essentially a matrix coefficient of a supercuspidal representation with respect to χ0 . This means that f˜0(g) = R f 0(gz)χ0 (z)dz v1 ZG(Fv) v1 is a matrix coefficient of a supercuspidal representation.

11 0 • For at least one place v2, fv2 is supported on the locus of Z-regular semisimple elements.

0 0 • f∞ is K∞-ﬁnite where K∞ is the maximal compact subgroup of G (F∞).

0 0 0 Lemma 2.2. Suppose that f = ⊗vfv is nice with respect to χ .

0 0 0 • As a function on H1(A) × H2(A), Kf 0 (h1, h2) is compactly supported modulo H1(F ) × 0 0 H2(F ). In particular, the integral I(f ) converges absolutely.

0 0 0 • As a function on H1(A) × H2(A), Kf 0,χ0 (h1, h2) is compactly supported modulo H1(F ) × 0 0 H (F )Z 0 ( ). In particular, the integral I 0 (f ) converges absolutely. 2 H2 A χ

Proof. The kernel function Kf 0 can be written as

X X 0 −1 −1 f (h1 γ1 γγ2h2), 0 0 0 0 γ∈H1(F )\G (F )/H2(F ) (γ1,γ2)∈H1(F )×H2(F ) where the outer sum is over regular semisimple γ. First we claim that the outer sum has only finite many non-zero terms. Let Ω be the support of f 0. Note that the invariants of G0(A) defines a continuous map from G0(A) to X(A) where X is the categorical quotient of G0 by 0 0 H1 × H2. So the image of Ω will be a compact set in X(A). On the other hand the image of −1 −1 h1 γ1 γγ2h2 is in the discrete set X(F ). Moreover for a fixed x ∈ X(F ) there is at most one 0 0 H1(F ) × H2(F ) double coset with given invariants. This shows the outer sum has only finite many non-zero terms. 0 0 0 It remains to show that for a fixed γ0 ∈ G (F ), the function on H1(A) × H2(A) defined 0 −1 0 by (h1, h2) 7→ f (h1 γ0h2) has compact support. Consider the continuous map H1(A) × 0 0 −1 H2(A) → G (A) given by (h1, h2) 7→ h1 γ0h2. When γ is regular semisimple, this defines an homeomorphism onto a closed subset of G0(A). This implies the desired compactness and completes the proof the first assertion. The second one is similarly proved using the Z-regular semi-simplicity. The last lemma allows us to decomposes the distribution into a finite sum of orbital integrals X I(f 0) = O(γ, f 0), {γ} 0 0 0 0 0 where the sum is over regular semisimple γ ∈ H1(F )\G (F )/H2(F ). If f = ⊗vfv is decomposable, we may decompose the orbital integral as a product of local orbital integrals:

0 Y 0 O(γ, f ) = O(γ, fv), v

0 where O(γ, fv) is deﬁned as in the previous subsection. Similarly, we have

0 X 0 Iχ0 (f ) = Oχ0 (γ, f ), {γ}

12 0 0 0 where the sum is over regular semisimple γ ∈ ZG0 (F )H1(F )\G (F )/H2(F ). For a cuspidal automorphic representation Π of G0(A) whose central character is trivial on Z 0 ( ), we deﬁne a spherical character (a relative version of character): H2 A Z ! Z ! 0 X 0 IΠ(f ) = Π(f )φ(x)dx φ(x)dx 0 0 0 0 H1(F )\H1(A) Z 0 (A)H2(F )\H2(A) φ∈B(Π) H2 where the sum is over an orthonormal basis B(Π) of Π. For nice f 0, we may choose the basis such that the sum is ﬁnite. We then have a simple RTF:

0 0 0 Theorem 2.3. If f is nice with respect to χ , then Iχ0 (f ) is equal to

X 0 X 0 Oχ0 (f ) = IΠ(f ), {γ} Π where the sum on the LHS runs over all regular semisimple

0 0 0 γ ∈ H1(F )\G (F )/ZG(F )H2(F ) and the sum on the RHS runs over all cuspidal automorphic representations Π with central character χ0.

0 Proof. It suﬃces to treat the spectral side. Since fv1 is essentially a matrix coeﬃcient of a super-cuspidal representation, an argument for the simple version of Arthur–Selberg trace 0 formula also applies to the current case and we obtain that the kernel Iχ0 (f ) only has only cuspidal part.. This yields an absolutely convergent sum

0 X 0 Iχ0 (f ) = IΠ(f ), Π where Π runs over automorphic cuspidal representations of G0(A) with central character χ0.

2.3 RTF on unitary groups

∞ We now recall the RTF of Jacquet–Rallis on the unitary case. For f ∈ Cc (G(A)) we consider a kernel function X −1 Kf (x, y) = f(x γy), γ∈G(F ) and a distribution Z Z J(f) := Kf (x, y)dxdy. H(F )\H(A) H(F )\H(A)

13 2 × For a fixed central character χ = (χn+1, χn): ZG(A) ' U(1)(A) → C we introduce the kernel Z Z −1 Kf,χ(x, y) = Kf (zx, y)χ(z)dz = Kf (x, z y)χ(z)dz, ZG(F )\ZG(A) ZG(F )\ZG(A) and a distribution Z Z Jχ(f) := Kf,χ(x, y)dxdy. H(F )\H(A) H(F )\H(A) Note that the center ZG is anisotropic and its intersection with H is trivial. The integrals J(f) and Jχ(f) are usually not convergent. Similar to the general linear case, we will consider a simplified trace formula and for our purpose we only discuss the ∞ convergence issue for the χ-part for a fixed χ. We say that such a function f ∈ Cc (G(A)) is nice with respect to χ if f = ⊗vfv satisfies

• For at least one place v1, fv1 is a essentially a matrix coeﬃcient of a supercuspidal ˜ R representation with respect to χv . This means that fv (g) = f(gz)χv (z)dz is a 1 1 ZG(Fv) 1 matrix coeﬃcient of a supercuspidal representation.

• For at least one place v2, fv2 is supported on the locus of Z-regular semisimple elements.

• f∞ is K∞-finite where K∞ is the maximal compact subgroup of G(F∞). For a cuspidal automorphic representation π, we define a spherical character as a distribution on G(A): X Z Z Jπ(f) = π(f)φ(x)dx φ(x)dx , φ∈B(π) HF \H(A) HF \H(A) where the sum is over an orthonormal basis B(π) of π. For nice f, we may choose the basis such that the sum is finite. For nice test functions, we have a simple RTF.

Theorem 2.4. If f is nice with respect to χ, then Jχ(f) is equal to X X Oχ(δ, f) = Jπ(f), δ π where the LHS runs over all regular simisimple orbits

δ ∈ H(F )\G(F )/ZG(F )H(F ), and the RHS runs over all cuspidal automorphic representations π with central character χ. Here in the RHS, by a π we mean a sub-representation of the space of cuspidal automorphic forms. So, a prior, two such representations may be isomorphic (as we don’t know yet the multiplicity one for such a π). Proof. The proof is the same as for the RTF on the linear side.

14 2.4 Comparison: fundamental lemma and transfer Some preparation. We first recall the matching of orbits without proof. The proof can be found [44] and [57]. Now the field F is either a number field or a local field of characteristic zero. Note that we may embed Sn+1(F ) to GLn+1(E). Also we may realize U(V ) as a subgroup of GLn+1. Even though such realization is not unique, the following notion is independent of the realization: we say that δ ∈ U(V )(F ) and s ∈ Sn+1(F ) match if s and δ (both considered as elements in GLn+1(E)) are conjugate by an element in GLn(E). Then it is proved in [57] that this defines a natural bijection between the set of regular semisimple orbits of Sn+1(F ) and the disjoint union of regular semisimple orbits of U(V ) where V = W ⊕ Eu (with (u, u) = 1) and W runs over all (isometric classes of) hermitian spaces over E. To state the matching of test functions, we need to introduce a “transfer factor”: it is a compatible family of functions {Ωv}v indexed by all places v of F , where Ωv is defined on the regular semisimple locus of Sn+1(Fv), and they satisfy:

• If s ∈ Sn+1(F ) is regular semisimple, then we have a product formula Y Ωv(s) = 1. v

−1 • For any h ∈ GLn(Fv) and s ∈ Sn(Fv), we have Ω(h sh) = η(h)Ωv(s).

0 × × × We may construct it as follows. We have ﬁxed a character η : E \AE → C (not 0 necessarily quadratic) such that its restriction η |A× = η. Then we may deﬁne

0 −[(n+1)/2] n (2.8) Ωv(s) := ηv(det(s) det(es, ..., es )).

It is easy to verify that the family {Ωv}v deﬁnes a transfer factor. We also extend this to a transfer factor on G0: it is a compatible family of functions (to 0 abuse notation) {Ωv}v on the regular semisimple locus of G (Fv), indexed by all places v of F , such that

0 Q • If γ ∈ G (F ) is regular semisimple, then we have a product formula v Ωv(γ) = 1.

0 • For any hi ∈ Hi(Fv) and s ∈ Sn(Fv), we have Ω(h1γh2) = η(h2)Ωv(γ). −1 We may construct it as follows: write γ = (γ1, γ2) and s = ν(γ1γ2 ), if n is odd, we set:

0 −1 0 −(n+1)/2 n (2.9) Ωv(γ) := η (det(γ1γ2 ))ηv(det(s) det(es, ..., es )), and if n is even, we set:

0 −n/2 n (2.10) Ωv(γ) := ηv(det(s) det(es, ..., es )).

0 ∞ For a place v of F , we consider f ∈ C(Sn+1(Fv)) and the tuple (fW )W , fW ∈ Cc (U(V )(Fv)) indexed by the set of all (isometric classes of) hermitian spaces W over Ev = E ⊗ Fv and we

15 0 set V = W ⊕ Evu. We say that f ∈ C(Sn(Fv)) and the tuple (fW )W are (smooth) transfer of each other if 0 Ωv(s)O(s, f ) = O(δ, fW ), whenever a regular semisimple s ∈ Sn+1(Fv) matches δ ∈ U(V )(Fv). ∞ 0 ∞ W Similarly we extend the deﬁnition to (smooth) transfer between Cc (G (Fv)) and Cc (G (Fv)) where GW = U(V ) × U(W ). It is then obvious that the the existence of the two transfers are equivalent. For a split place v, the existence of transfer is almost trivial. To see this, we may directly work with smooth transfer on the original groups. We may identify GLn(E⊗Fv) = GLn(Fv)× 0 0 0 GLn(Fv) and suppose the function fn = fn,1⊗fn,2. We identify the unitary group U(W )(Fv) ' ∞ 0 GLn(Fv) and let fn ∈ Cc (GLn(Fv)). Similarly we have fn+1 for GLn+1(Fv) etc.. Proposition 2.5. If v is split in E, then the smooth transfer exists. In fact we may take the 0 0,∨ 0,∨ 0 −1 convolution fi = fi,1 ? fi,2 where i = n, n + 1 and fi,2 (g) = fi,2(g )

0 0 0 Proof. In this case the quadratic character ηv is trivial. For f = fn+1 ⊗ fn, the orbital 0 integral O(γ, f ) can be computed in two steps: ﬁrst we integrate over H2(Fv) then over the rest. Deﬁne Z ¯0 0 0 0 0,∨ fi (x) = fi,1(xy)fi,2(y)dy = fi,1 ? fi,2 (x), i = n, n + 1. GLi(Fv)

0 Then obviously we have the orbital integral for γ = (γn+1, γn) ∈ G (Fv) and γi = (γi,1, γ,2) ∈ GLi(Fv) × GLi(Fv), i = n, n + 1: Z Z 0 ¯0 −1 ¯0 −1 O(γ, f ) = fn+1(xγn+1,1γn+1,2y)fn(xγn,1γn,2y)dxdy. GLn(Fv) GLn(Fv) Now the lemma follows easily. Now we use E/F to denote a local (genuine) quadratic ﬁeld extension.

Theorem 2.6. If E/F is non-archimedean, then the smooth transfer exists.

The proof will occupy section 3 and 4. 0 Let χ be a character of ZG(F ) and deﬁne the character χ of ZG0 (F ) to be the base change of χ.

0 Corollary 2.7. If f and fW match, then the χ-orbital integrals also match:

0 Oχ(δ, fW ) = Ω(γ)Oχ0 (γ, f ) if γ and δ match.

16 Proof. It suffices to verify that the orbital integrals are compatible with multiplication by central elements in the following sense: consider z ∈ E× × E× identified with the center of G0(F ) in the obvious way. We denote byz ¯ the Galois conjugate coordinate-wise. Then z/z¯ ∈ E1 × E1 which can be identified with the center of G(F ) in the obvious way. Assume 0 that δ and γ match. Then so do zγ and z/zδ¯ . We have by assumption that f and fW match:

0 Ω(zγ)O(zγ, f ) = O(z/zδ,¯ fW ) for all z. It is an easy computation to show that our deﬁnition of transfer factors satisfy

Ω(zγ) = Ω(γ).

We need wwo more theorems to prove the trace formula identity. The ﬁrst one is the fundamental lemma for units in the spherical Hecke algebras. Let E/F be an unramifeid quadratic extension (non-archimedean). There are precisely two isometric classes of hermitian space W : one with a self-dual lattice is denoted by W0 and the other W1. For W0 we consider the stabilizer K of a self-dual lattice. Theorem 2.8 (Yun,[55]). There is a constant c(n) depending only on n such that the fundamental lemma of Jacquet–Rallis holds for all quadratic extension E/F with residue character ∞ 0 larger than c(n); namely, the function 1K ∈ Cc (G (F )) and the pair fW0 = 1K , fW1 = 0 are transfer of each other.

The second result is a theorem of automorphic-Cebotarev-density type. It will allows to separate (cuspidal) spectrums without using the fundamental lemma for the full spherical Hecke algebras at non-split places. It is stronger than the strong multiplicity one theorem for GLn. Theorem 2.9 (Ramakrishnan). Let E/F be a quadratic extension. Two cuspidal automorphic representations Π1, Π2 of ResE/F GLn(A) are isomorphic if and only if Π1,v ' Π2,v for almost all places v of F that are split in E/F .

The proof can be found in [45].

The trace formula identity. We ﬁrst have the following corse form.

0 Proposition 2.10. Fix a character χ of ZG(F )\ZG(A) and let χ be its base change. Fix a split place v0 and a supercuspidal representation πv0 of G(Fv) with central character χv0 . Suppose that

0 • f and (fW )W are all nice functions and are smooth transfer of each other.

17 0 • Let Πv0 be the local base change of πv0 . Then fv0 is essentially a matrix coeﬃcient of

Πv0 and is related to fW,v0 as prescribed in Prop. 2.5 (In particular, fW,v0 is essentially

a matrix coeﬃcient of πv0 ).

0 0 Fix ⊗vπv where the product is over almost all split places v and each πv is irreducible unram- iﬁed. Then we have X 0 X X IΠ(f ) = JπW (fW ), Π W πW 0 where the sums run over all automorphic representations Π of G (A) and πW of G(A) with central characters χ0, χ respectively such that

0 • πW,v ' πv for almost all split v.

• πW,v0 is the ﬁxed supercuspidal representation πv0 .

• Π = BC(πW ) is a weak base change of πW , and Πv0 is the local base change of πv0 . In particular Π is cuspidal and the LHS contains at most one term.

Proof. We may assume that all test functions are decomposable. Let S be a finite set of 0 places such that for any v outside S, fv and fW,v are units of the spherical Hecke algebras (in particular, v is non-archimedean and unramified in E/F ) and for all Hermitian spaces W such S that fW 6= 0, they are the same outside S and are all unramified. So we may identify G(A ) for all such W appearing here. Now we enlarge S so that for all non-split v outside S, the fundamental lemma for unit holds. The fundamental lemma for the spherical Hecke algebra 0 0 holds at a split place. Therefore for any f ,S in the spherical Hecke algebra H(G0(AS)//K S) 0S Q S S S (K = v∈ /) is the usual maximal compact subgroup) and f ∈ H(G(A )//K ) such that 0 at a non-split v, fv, fv are the units, we have a trace formula identity

0 0,S X S I(fS ⊗ f ) = J(fW,S ⊗ f ). W Again all these test functions are nice so we may apply the simple trace formulae:

X 0 0,S X X S IΠ(fS ⊗ f ) = JπW (fW,S ⊗ f ). Π W πW

Here all Π, πW are cuspidal automorphic representations whose component at v0 are the given ones. Let λ S (λ S , resp.) be the linear functional of the spherical Hecke algebras Π πW 0 H(G0(AS)//K S)(H(G(AS)//KS), resp.). Then we observe that

0 0,S 0,S 0 IΠ(fS ⊗ f ) = λΠS (f )IΠ(fS ⊗ 1K0S )

S and similarly for JπW (fW,S ⊗ f ). Note that we are only allowed to take the unit elements in the spherical Hecke algebras. Therefore we can view both sides as linear functionals on the

18 0 spherical Hecke algebra H(G0(AS,split)//K S,split) where “split” indicate we only consider the product over all split places outside S. These linear functionals are linearly independent. In 0 particular, for the ﬁxed ⊗vπv , we may have an equality as claimed in the theorem. Since such Π’s are cuspidal, there exists at most one Π by the automorphic-Cebotarev-density theorem of Ramakrishnan.

Now we come to the trace formula identity which will allow us to deduce the main theorems in Introduction. Proposition 2.11. Let E/F be a quadratic extension such that all archimedean places v|∞ are split. Fix a hermitian space W0 and deﬁne V0, the group G etc. as before. Let π be a cuspidal automorphic representation of G such that for a split place v0, πv0 is supercuspidal. 0 0 Consider decomposable nice functions f and (fW )W satisfying the same conditions as in the previous proposition. Then we have a trace formula identity:

0 X X IΠ(f ) = JπW (fW ),

W πW where Π = BC(π) and the sum in the RHS runs over all W and all πW nearly equivalent to π.

0 Proof. Apply the previous proposition to πv = πv for almost all split v. Then in the sum of the RHS there, all πW have the same weak base change Π. Note that the local base change map are injective for split places and for unramiﬁed representations at non-split unramiﬁed places. This implies that all πW are in the same nearly equivalence class.

A non-vanishing result. Finally, to see that the second condition on the niceness of a test function does not lose generality in some sense, at least for supercuspidal representations, we will need some “regularity” result for the distribution Jπ. By the multiplicity one result [3] and [48], we may ﬁx an appropriate choice of generator `Hv ∈ HomHv (πv, C)(`Hv = 0 if the space is zero) and decompose Y `H = cπ `Hv , v where cπ is a constant depending on the cuspidal automorphic representation π (and its realization in A0(G)). This gives a decomposition of the spherical character as a product of local spherical characters 2 Y Jπ(f) = |cπ| Jπv (fv), v where the spherical character is deﬁned as X Jπv (fv) = `Hv (πv(fv)φv)`Hv (φv).

φv∈B(πv)

19 ∞ Note that Jπv is a distribution of positive type, namely, for all fv ∈ Cc (G(Fv)),

∗ ∗ −1 Jπv (fv ∗ fv ) ≥ 0, fv (g) := f(g ).

∗ We will also say that a function of the form fv ∗ fv is of positive type.

Proposition 2.12. Let πv be a tempered representation of G(Fv). Then there exists function ∞ fv ∈ Cc (G(Fv)) supported in the Z-regular semisimple locus such that

Jπv (fv) 6= 0.

The proof is given in the Appendix (Theorem A.2). Equivalently, the result can be stated as follows: the support of the spherical character Jπv (as a distribution on G(Fv)) is not contained in the complement of regular semisimple locus.

2.5 Proof of Theorem 1.1: (ii) =⇒ (i) Proposition 2.13. Let E/F be a quadratic extension such that all archimedean places v|∞ are split. Let W be a codimension one subspace of a hermitian space V of dimension n + 1. Let π be a cuspidal automorphic representation of U(V ) × U(W ) such that for a split places v1, πv1 is supercuspidal and for a split v2 6= v1, πv2 is tempered. Denote by Π its weak base change. If π is distinguished by the diagonal embedding of U(W ), then L(Π,R, 1/2) 6= 0 and 0 Π is η-distinguished by H2 = GLn+1,F × GLn,F . In particular, in Theorem 1.1, (ii) implies (i).

Proof. We apply Prop. 2.11. It suﬃces to show that there exist f 0 as in Prop. 2.11 such that

0 IΠ(f ) 6= 0.

0 We will ﬁrst choose an appropriate f := fW and then choose f to be a transfer of the tuple (fW , 0..., 0) where for all hermitan space other than W we choose the zero functions. We choose f 0 satisfying the conditions of Prop. 2.11 . Then the trace formula identity from Prop. 2.11 is reduced to 0 X IΠ(f ) = JπW (fW ).

πW

Note that for all πW , they have the same local component at v1, v2 by the Hypothesis on the local–global compatibility for weak base change at split places. We choose f = fW = ⊗vfv as follows. By the assumption on the distinction of π, we may assume that for some function g of positive type Jπ(f) > 0. We may assume that at v1, gv1 is essentially a matrix coeﬃcient of πv1 . This is clearly possible. Then we have

2 Y Jπ(g) = |cπ| Jπ(gv) > 0 v

20 and for all πW nearly equivalent to π:

JπW (g) ≥ 0.

Now we choose fv = gv for every place v other than v2. We choose fv2 to be supported in the Z-regular semisimple locus. By Prop. 2.12, we may choose an fv2 such that

J (f ) 6= 0. πv2 v2 For this choice of f, the trace identity is reduced to ! 0 X 2 (v2) I (f ) = J (f ) |c | J (v ) (f ) Π πv2 v2 πW 2 πW πW where the superscript means the away from v2-part. In the sum, every term is non-negative as we choose f (v2) of positive type. And at least one of these terms is non-zero. Therefore 0 we conclude that for this choice the RHS above is non-zero. This shows that IΠ(f ) 6= 0 and completes the proof.

2.6 Proof of Theorem 1.4 A key ingredient is the following Burger–Sarnark type principle following a terminology of Prasad [41].

Proposition 2.14. Let V be a hermitian space of dimension n and W a nondegenerate subspace of codimension one. Let π be a cuspidal automorphic representation of U(V )(A). Fix a ﬁnite (non-empty) set S of places and an irreducible representation σv of U(W )(Fv) for each v ∈ S such that

• If v ∈ S is archemedean, both W and V are positive deﬁnite at v.

0 • If v ∈ S is non-archimedean and split, σv is induced from a representation of ZvKv where Kv is a compact open subgroup and Zv is the center of U(W )(Fv).

0 • If v ∈ S is either archimedean or non-split, the contragredient of σv appears as a quotient of πv restricted to U(W )(Fv).

Then there exists a cuspidal automorphic representation σ of U(W )(A) such that

0 • σv = σv for all v ∈ S.

• the linear form `W on π ⊗ σ is non-zero. We ﬁrst show the following variant of [41, Lemma 1].

21 Lemma 2.15. Suppose that we are in the following situations

• F is a number ﬁeld.

• G is a reductive algebraic group deﬁned over F , and H is a reductive subgroup of G.

• S is a ﬁnite set of places of F such that if v ∈ S is archimedean, then G(Fv) is compact. Q Q Denote GS = v∈S G(Fv) and HS = v∈S H(Fv). • the center Z of H is assumed to be anisotropic over F .

Let π be a cuspidal automorphic representation of G(A). Let ⊗v∈Sµv be an irreducible representation of HS such that

(1) Assume that µv appears as a quotient of πv restricted to H(Fv) for all v ∈ S.

(2) for each non-archimedean v ∈ S, µv is supercuspidal representations of H(Fv), and it is an induced representation µ = IndHv ν from a representation µ of a subgroup Z K v ZvKv v v v v where Kv are certain open compact subgroups of Hv.

0 Q 0 0 Then there is an automorphic representation µ = v µv of H(A) and functions f1 ∈ π, f2 ∈ µ such that

(i) Z ¯ f1(h)f2(h)dh 6= 0, H(F )\H(A) and

(ii) If v ∈ S is archimedean µ0 = µ . If v ∈ S is non-archimedean, µ0 = IndHv ν0 is v v v ZvKv v 0 0 induced from νv where νv|Kv = νv|Kv .

Proof. The proof is a variant of [41, Lemma 1]. If v ∈ S is archimedean, let Kv = H(Fv) and νv = µv. It is compact by assumption. We consider the restriction of πv to Kv for each v ∈ S. By the assumption and Frobenius reciprocity, νv|Kv is a quotient representation of πv|K . Since Kv is compact, νv|K is also a sub-representation for v ∈ S. This means that v v Q we may ﬁnd a function f on G(A) whose KS = v∈S Kv translates span a space which is isomorphic to ⊗v∈Sνv|Kv as KS-modules. By the same argument as in [41, Lemma 1] (using weak approximation), we may assume that such f has non-zero restriction (denoted by f˜) to H(F )\H(A). Now note that Z(F )\Z(A) is compact. For a character χ of Z(F )\Z(A) we may deﬁne Z ˜ ˜ −1 f 7→ fχ(h) := f(zh)χ (z)dz, h ∈ H(F )\H(A). Z(F )\Z(A) ˜ ˜ As ZS and KS commute, each of f and fχ generates a space of functions on H(F )\H(A) which is isomorphic to ⊗v∈Sνv|Kv as KS-modules. There must exist some χ such that it is

22 non-zero. For such a χ, it is necessarily true that χv|Zv∩Kv = ωµv |Zv∩Kv where ωµv is the central character of ν . In particular, we may replace µ = IndHv ν by µ0 := IndHv ν0 where ν is v v ZvKv v v ZvKv v v 0 an irreducible representation of ZvKv with central character χv and νv|Kv = νv|Kv . Certainly 0 0 such µv is still supercuspidal if v ∈ S is non-archimedean and if v ∈ S is archimedean, µv = µv. ˜ Now we consider the space generated by fχ under ZSKS translations. This space is certainly Q 0 isomorphic to v∈S νv as ZSKS-modules. The rest of the proof is the same as in [41, Lemma ˜ 1], namely applying [41, Lemma 2] to the space of HS-translations of fχ which is isomorphic to ⊗ IndHv ν0 as H -modules. v∈S ZvKv v S We now prove Prop. 2.14. Proof. We apply the lemma above to the case H = U(W ),G = U(V ). Then the center is anisotropic. If v ∈ S is split non-archimedean, it is always true that µv appears as a quotient of πv (the local conjecture in [11] for the general linear group is known to hold for generic representations). The proposition then follows immediately.

Remark 8. Noting that any supercuspidal representation of GLn(Fv) for a non-archimedean v is induced from an irreducible representation of an open subgroup which is compact modulo center. But an open subgroup of GLn(F ) compact modulo center is not necessarily of the form KvZv. As pointed out by Prasad to the author, it may be possible to choose an arbitrary supercuspidal µv if one suitably extends the result in Lemma 2.15. Now we come to the proof of Theorem 1.4. Proof. We show this by induction on the dimension of W . If n := dimW is equal to one, then the theorem is obvious. Now assume that for all dimension at most n hermitian spaces, the statement holds. Let V be a n + 1-dimensional hermitian space and W is a codimension one subspace. And let π be a cuspidal automorphic representation such that πv1 is supercuspidal 0 for a split place v1. By [4], there exists a supercuspidal representation σv1 that veriﬁes the assumptions of Prop. 2.14. Then we apply Prop. 2.14 to S = {v1} to choose a cuspidal automorphic representation σ of U(W ) such that π ⊗ σ is distinguished by U(W ) (for the diagonal embedding into U(V ) × U(W )). Then by Prop. 2.13, the weak base change of π ⊗ σ 0 is η−distinguished by H2 = GLn+1,F ×GLn,F . By induction hypothesis, the weak base change of σ is (η-, resp.) distinguished by GLn,F if n is odd (even, resp.). Together we conclude that the weak base change of π is (η-, resp.) distinguished by GLn+1,F if n + 1 is odd (even, resp.). This completes the proof. Remark 9. In the most ideal situation, (namely, if we have the trace formula identity for all test functions f, not just the nice ones), then we may use the proof of Prop. 2.13 to show ﬁrst the existence of weak base change, then use the proof of Theorem 1.4 to show the distinction of the weak base change as predicted by the conjecture of Flicker–Rallis. But it seems impossible to characterize the image of the weak base change using the Jacquet–Rallis trace formulae.

23 2.7 Proof of Theorem 1.1: (i) =⇒ (ii) Now we ﬁnish the proof of the other direction of Theorem 1.1:(i) =⇒ (ii). It suﬃces to replace the condition (2) by that πv1 is supercuspidal at a split v1, πv2 is tempered for v2 6= v1 split. 0 By Theorem 1.4, the weak base change Π = BC(π) is η-distinguished by H2. By the assumption on nonvanishing of L(Π,R, 1/2), we know that Π is also distinguished by 0 ResE/F GLn. Therefore, IΠ is non-zero distribution on G (A). Therefore some some decom- 0 0 0 posable f = ⊗vfv, IΠ(f ) 6= 0. Note that the multiplicity one also holds in this case:

dim Hom 0 (Π , ) ≤ 1. Hi(Fv) v C

Similar to the decomposition of the distribution Jπ, we may ﬁx a decomposition Y IΠ = cΠ IΠv . v

0 0 0 In particular, cΠ 6= 0 and for the f above, IΠv (fv) 6= 0 for all v. We want to modify f at the two places v1, v2 to apply Prop. 2.11. 0 It is easy to see that we may replace fv1 by a function essentially a matrix coeﬃcient of

πv1 . Now note that the distribution at the split place v2 I (f 0 ) = J (f ) Πv2 v2 πv2 v2

0 if Πv2 is the local base change of πv2 and fv2 is the transfer of fv2 as prescribed by Prop. 2.5. By Prop. 2.12, J (f ) 6= 0 for some f supported in Z-regular semisimple locus. Therefore, πv2 v2 v2 we may choose f 0 supported in Z-regular semisimple locus such that I (f 0 ) 6= 0. v2 Πv2 v2 Now we replace f 0 , i = 1, 2 by the new choices. Then we let the tuple (f ) be a transfer vi W of f 0 satisfying the conditions in Prop. 2.11. By the trace formula identity of Prop. 2.11,

0 X 0 IΠ(f ) = JπW 0 (fW )

πW 0

0 where the sum in RHS runs over all W , and all πW 0 nearly equivalent to π. There must be 0 at least one term Jπ0 (fW 0 ) 6= 0 for some W . This completes the proof. Remark 10. The study of heights of some algebraic cycles on certain Shimura varieties con- stitutes a natural arithmetic version of the global Gan–Gross–Prasad conjecture ([11],[56]). The two themes—–periods and heights—–can be uniﬁed in an approach using relative trace formulae [57], at least for the unitary case in the current moment.

3 Reduction steps

In this and the next section, we will prove the existence of smooth transfer at a non- archimedean non-split place as well as a partial result at an archimedean non-split place.

24 In this section, we reduce the question to an analogue on “Lie algebras” (an infinitesimal version) and then to a local question around zero. Let F be a local field of characteristic zero. In√ this section, both archimedean and non-archimedean local fields are allowed. Let E = F [ τ] be a quadratic extension where τ ∈ F ×. We remind the reader that, even though our interest is in the genuine quadratic extension E/F , we may actually allow E to be split, namely, τ ∈ (F ×)2.

3.1 Reduction to Lie algebra We ﬁrst reduce the smooth transfer to an inﬁnitesimal version. 0 We consider the action of H := GLn,F on the tangent space of the symmetric space Sn+1 at the identity matrix 1n+1:

(3.1) Sn+1 := {x ∈ Mn+1(E)|x +x ¯ = 0.} which will be called the “Lie” algebra of Sn+1. When no confusion arises, we will drop the subscript and write it S for simplicity. Let W be a hermitian space of dimension n and let V = W ⊕ Eu with (u, u) = 1. We consider the restriction to H = HW = U(W ) of the adjoint action of U(V ) on U(V ) and its Lie algebra (as an F -vector space):

∗ (3.2) U(V ) = {x ∈ EndE(V )|x + x = 0} where x∗ is the adjoint with respect to the hermitian form on V :

(xa, b) = (a, x∗b), a, b ∈ V.

0 It will be more convenient to consider the action of H on the Lie algebra of GLn+1:

{x ∈ Mn(E)|x =x ¯}' gln+1.

Relative to the H-action, we have notions of regular semisimple elements. Similarly for the H0-action. Analogous to the group case, regular semisimple elements have trivial stabilizers. We also deﬁne an analogous matching of orbits as follows. We may identify EndE(V ) with Mn+1(E) by choosing a basis of V . Then for regular semisimple x ∈ S(F ) and y ∈ U(V )(F ), we say that they (and their orbits) match each other if x and y when considered as elements in Mn+1(E) are conjugate by an element in GLn(E). We will also say that x and y are transfer of each other and denote the relation by x ↔ y. Then, analogous to the case for groups, the notion of transfer deﬁnes a bijection between regular semisimple orbits

0 a S(F )rs/H (F ) ' U(V )(F )rs/H(F )(3.3) W

25 where the disjoint union runs over all isomorphism classes of n-dimensional hermitian space 0 0 W . We recall some results from [44], [57]. For the natural map πF : S(F ) → (S//H )(F ) and πW,F : U(V )(F ) → (U(V )//H)(F ), the ﬁber of a regular semisimple element consists of 0 0 precisely one orbit with trivial stabilizer. Moreover, πF is surjective. In particular, πF induces a bijection: 0 0 S(F )rs//H (F ) ' (S//H )(F )rs.

And πW,F induces a bijection between U(V )(F )rs/H(F ) and its image in (U(V )//H)(F ). A more intrinsic way is to establish an isomorphism between the categorical quotients between S//H0 and U(V )//H. To state this more precisely, let us consider the invariants on them. We may choose a set of invariants on Sn+1 tr ∧i x, e · xj ·t e, i = 1, 2, ..., n + 1, j = 1, 2, ..., n; and on U(V ) for V = W ⊕ Eu:

tr ∧i y, (yju, u), i = 1, 2, ..., n + 1, j = 1, 2, ..., n.

A b where x ∈ S and y ∈ U(V ). If we write S 3 x = ,A ∈ S , an equivalent set n+1 n+1 c d n of invariants on Sn+1 are tr ∧i A, c · Aj · b, d i = 1, 2, ..., n, j = 0, 1, 2, ..., n − 1.

Similarly for the unitary case. Denote by Q = A2n+1 the 2n + 1-dimensional affine space (in this and the next section we are always in the local situation and A will denote the affine line). Then the invariants above define a morphism

πS : Sn+1 → Q x 7→ (tr ∧i x, e · xj ·t e), i = 1, 2, ..., n + 1, j = 1, 2, ..., n.

To abuse notation, we will also denote by πS the morphism deﬁned by the second set of invariants above. Similarly we have morphism denoted by πU for the unitary case. We usually simply write π if there is no confusion.

Lemma 3.1. For each case V = S or U(V ), the pair (Q, πV ) defines a categorical quotient of V. Equivalently, the set of invariants above is a set of generators of the ring of invariant polynomials for Sn+1 and U(V ). Proof. As this is a geometric statement, it suffices to treat the case V = S or equivalently gln+1. We will use Igusa’s criterion ([29, Lemma 4], or [42, Theorem 4.13]): Let a reductive group H act on an irreducible affine variety V. Let Q be a normal irreducible affine variety, and π : V → Q be a morphism which is constant on the orbits of H such that

26 (1) Q − π(V) has codimension at least two;

−1 (2) There exists a non-empty open subset Q0 of Q such that the ﬁber π (q) of q ∈ Q0 contains exactly one closed orbit.

Then (Q, π) is a categorical quotient for the H action on V . For gln+1, the morphism π is clearly constant on the orbits of H. Now we deﬁne a section of π close to the classical companion matrices. Consider

 0 1 0 ... 0  0 0 1 ... 0   ...... 0 1 0 .   bn bn−1 ... b1 1 an an−1 ... a1 d

Then its invariants are

i i−1 j tr ∧ A = (−1) bi, c · A · b = aj+1, d i = 1, 2, ..., n, j = 0, 1, 2, ..., n − 1.

This gives us an explicit choice of section of π and it shows that π is surjective. This verifies (1). By [44], for all regular semisimple q ∈ Q, the fiber of q consists of at most one closed orbit. It follows by the explicit construction above that the fiber contains precisely one closed orbit. The regular semisimple elements form the complement of a principle divisor and hence we have verified condition (2). This completes the proof. By this result, we have a natural isomorphism between the categorical quotient S//H0 and U(V )//H. And this enables us to define the transfer of orbits. In the bijection (3.3) the appearance of disjoint union is due to the fact that the map between F -points induced by πS is surjective but the one by πU(V ) is not. By this lemma, more generally, we say that two semisimple elements x ∈ S(F ) and y ∈ U(V )(F ) match each other if they have the same invariants, or equivalently their image in the quotients correspond to each other under the isomorphism between the categorical quotients. Given a semisimple x ∈ S(F ) (not necessary regular), in general there may be more than one matching semisimple orbits in U(V )(F ). We need also to define a transfer factor in the level of Lie algebra.

0 Definition 3.2. Consider the action of H on X = gln+1 or S. A transfer factor is a smooth × h function ω : X(F )rs → C such that ω(x ) = η(h)ω(x). Obviously, two transfer factors ω, ω0 differer by a H0(F )-invariant smooth function ξ : × X(F )rs → C . If ξ extends to a smooth function on X(F ) (and with moderate grows towards infinity for a norm on X(F ) if F is archimedean), we say that ω, ω0 are equivalent and denote by ω ∼ ω0.

27 We have defined a transfer factor Ω earlier on the groups. We now√ define a transfer factor on the Lie algebra and we will only consider its equivalence class. If τx ∈ S(F ) is regular semisimple, we define √ ω( τx) := η(det(e, ex, ex2, ..., exn))(3.4) where e is the row vector (0, 0, ..., 0, 1) and (e, ex, ex2, ..., exn) is the matrix consisting of the row vectors eXi. Now we may similarly define the notion of transfer of test functions on S(F ) and U(V )(F ). 0 ∞ ∞ For an f ∈ Cc (S(F )), and a tuple (fW )W where fW ∈ Cc (U(V )(F )), they are called a (smooth) transfer of each other if for all matching regular semisimple S(F ) 3 x ↔ y ∈ u(V )(F ), V = W ⊕ Eu, we have

0 ω(x)O(x, f ) = O(y, fW ).

For n ∈ Z≥1, we rewrite the “smooth transfer conjecture” of Jacquet-Rallis for the symmetric space S and the unitary group U(V ): 0 ∞ ∞ Conjecture Sn+1: For any f ∈ Cc (Sn+1), its transfer (fW )W (fW ∈ Cc (U(V ))) exists. And the other direction also holds, namely, given any a tuple (fW )W , there exists its transfer f 0. The corresponding statement for Lie algebra can be stated as 0 ∞ ∞ Conjecture Sn+1: For any f ∈ Cc (Sn+1(F )), its transfer (fW )W (fW ∈ Cc (U(V )(F ))) exists. And the other direction also holds. Note that the statement depends on the choice of a transfer factor. But it is obvious that the truth of the conjecture does not depend on the choice of the transfer factor within an equivalence class. We now reduce the group version to the the Lie algebra version.

Theorem 3.3. Conjecture Sn+1 implies Conjecture Sn+1,. To prove this theorem, we need some preparation. Let us deﬁne a set for a scalar ν ∈ E

Dν = {x|det(ν − x) 6= 0}.

We will choose a basis of V to realize the unitary group U(V ) as a subgroup of GLn+1,E. Lemma 3.4. Let ξ ∈ E1. The map (“Cayley” transform)

αξ : Mn+1(E) − D1 → GLn+1(E) x 7→ −ξ(1 + x)(1 − x)−1. induces an H-equivariant isomorphism between Sn+1(F )−D1 and Sn+1(F )−Dξ. In particular, 1 if we choose a sequence of distinct ξ1, ξ2, ..., ξn+2 ∈ E , the images of Sn+1(F ) − D1 under αξi form a ﬁnite cover by open subset of Sn+1(F ). Similarly, αξ induces an U(W )-equivariant isomorphism between U(V )(F )−D1 and U(V )(F )− Dξ.

28 Proof. First we verify that the image of αξ lies in S = Sn+1. We need to show

α(x)¯α(x) = (1 + x)(1 − x)−1(1 +x ¯)(1 − x¯)−1 = 1.

This is equivalent to

(1 + x)(1 − x)−1(1 +x ¯) = 1 − x¯ ⇐⇒(1 − x)(1 + x)(1 − x)−1(1 +x ¯) = (1 − x)(1 − x¯).

Note that 1 − x and 1 + x commutes:

(1 + x)(1 +x ¯) = (1 − x)(1 − x¯).

This reduces to (if char F 6= 2) x +x ¯ = 0.

Note that if det(ξ − s) 6= 0, αξ has an inverse deﬁned by

s 7→ −(ξ + s)(ξ − s)−1.

This shows that the image of αξ is S − Dξ and αξ deﬁnes an isomorphism between two aﬃne varieties. The same argument proves the assertion for unitary groups.

Lemma 3.5. The transfer factors are compatible under the Cayley map αµ. Proof. It suﬃces to consider the case µ = 1 as the argument is the same for a general µ. Note that (1 + x) and (1 − x)±1 commute:

−1 0 −1 i n−1 Ω((1 + x)(1 − x) ) = η (det((1 + 2(1 − x) ) e)i=0 ).

Let T = 2(1 − x)−1 Then it is easy to see that the determinant is equal to

i n−1 i n−1 det((1 + T ) e)i=0 = det(T e)i=0 by elementary operations on a matrix. This is equal to

n n(n−1)/2 i n−1 n i n−1 2 (−1) det((1 − x) e)i=0 = 2 det(x e)i=0 .

Therefore we have proved that

−1 0 n i n−1 √ Ω((1 + x)(1 − x) ) = η (2 det(x e)i=0 ) = c · ω(x/ τ). for a non-zero constant c.

29 × Now for more ﬂexibility, we consider the statement following Pµ for µ ∈ F : ∞ Pµ: For f ∈ Cc (Sn+1 − Dµ), its transfer (fW ) exists and can be chosen such that fJ ∈ ∞ Cc (U(V ) − Dµ). And the other direction also holds. × Then it is clear that Pν holds for all µ ∈ F implies Conjecture Sn by partition of unity applying to the open cover of S and U(V ) for distinct µ0, ..., µn+1. To prove Theorem 3.3, it remains to show the following:

× Lemma 3.6. Conjecture Sn+1 implies Pµ for all µ ∈ F .

∞ Proof. Fix such a µ. Assume that (fW ) is a transfer of f ∈ Cc (S − Dµ). Let Y = supp(f) ⊆ 0 S(F ) and Z = π (Y ). It suﬃces to show that for each W there exists a function αW ∈ C∞(U(V )(F )) (this means smooth when F is archimedean; locally constant when F is non- archimedean) satisfying

1. αW is H(F )-invariant,

2. αW | −1 ≡ 1, πW (Z)

3. αW |Dµ ≡ 0.

∞ Then we may replace fW by fW αW which will still be in C (U(V )(F )) and has the same orbital integral as fW . 0 0 Now note that Z = π (Y ) ⊆ (S//H )(F ) is compact. And Dµ is the preimage under 0 ∞ πe of a hypersurface denoted by C in (S//H )(F ). Then we may ﬁnd a C function β on (S//H0)(F ) satisfying

1. β|Z ≡ 1.

2. β|C ≡ 0. When F is archimedean, one may construct such a β using bump function. When F is non- archimedean function, we may cover Z by an open compact subsets Zb which is disjoint from C. Then we take β to be the characteristic function of Zb. Then we may take αW to be the pull-back of β under πW,F . The other direction can be proved similarly.

3.2 Reduction to local transfer around zero

2n+1 0 From now on we will denote by Qn = A or simply Q the common base Sn+1//H ' U(V )//H as an aﬃne variety. The aim of this section is to reduce the existence of transfer to the existence of a local transfer near zero.

30 Localization. We ﬁx a transfer factor ω and let π0 : S(F ) → Q(F ) and π : U(V ) → Q be the induced maps on the rational points.

Deﬁnition 3.7. Let Φ be a function on Q(F )rs which vanishes outside a compact set of Q(F ). (1) For x ∈ Q(F ), we say that Φ is a local orbital integral for S around x ∈ Q(F ) if there ∞ exists a neighborhood U of x and a function f ∈ Cc (S(F )) such that for all y ∈ Urs, and z with π0(z) = y we have

Φ(y) = ω(z)O(z, f).

(2) Similarly we can deﬁne a local orbital integral for U(V ).

Note that if Φ is a local orbital integral for a transfer factor ω, it is also a local orbital integral for any other equivalent transfer factor ω0 ∼ ω. Then we have the following localization principle for orbital integrals.

Proposition 3.8. Let Φ be a function on Q(F )rs which vanishes outside a compact set ξ of Q(F ). If Φ is a local orbital integral for S at x for all x ∈ Q(F ), then it is an orbital integral, ∞ 0 namely there exists f ∈ Cc (S(F )) such that for all Y ∈ Q(F )rs, and z with π (z) = y we have Φ(y) = ω(z)O(z, f). Similar result holds for U(V ).

∞ Proof. By assumption, for each x ∈ ξ we have an open neighborhood Ux and fx ∈ Cc (S(F )).

By the compactness we may ﬁnd ﬁnitely many of them, say x1, ..., xm, such that Uxi cover ξ.

Then we apply “partition of unity” to the cover of Q(F ) by Uxi (i = 1, 2, ..., m) and Q(F ) − ξ to obtain smooth functions βi, β on Q(F ) such that supp(βi) ⊆ UXi and supp(β) ⊆ Q(F ) − ξ P Pm and β + i βi is the identify function on Q(F ). Since Φβ ≡ 0, we may write Φ = i=1 Φi 0 where Φi = Φβi is a function on Q(F )rs which vanishes outside Uxi . Then αi = βi ◦ π is a 0 ∞ smooth H (F )-invariant function on S(F ) and fxi αi ∈ Cc (S(F )). We claim that for every 0−1 y ∈ Q(F )rs and z ∈ π (y), we have ω(z)O(z, fxi αi) = Φi(y). In deed, the left hand side is equal to ω(z)O(z, fx )β(y). If Y is outside Ux , then both sides vanish. If y ∈ Ux , then by i i Pi the choice of fxi , we have ω(z)O(z, fxi ) = Φ(y). By the claim, we may take f = i fxi αi to complete the proof.

∞ 0 For f ∈ Cc (S(F )), we deﬁne a “direct image” π∗,ω(f) as the function on Q(F )rs:

0 π∗,ω(f)(x) := ω(y)O(y, f)

0 −1 where x ∈ Q(F )rs, y ∈ (π ) (x). It clearly does not depend on the choice of y. Similarly, for ∞ fW ∈ Cc (U(V )), we deﬁne a function πW,∗(fW ) on Q(F )rs (extend by zero to those x ∈ Q(F )rs −1 such that πW (x) is empty). We will also write it as π∗,ω(fW ) with the trivial transfer factor ω = 1.

31 Deﬁnition 3.9. For x ∈ Q(F ), we say that the local transfer around x exists, if for all ∞ ∞ f ∈ Cc (S(F )), there exist (fW )W (fW ∈ Cc (U(V ))) such that in a neighborhood of x, the following equality holds 0 X π∗,ω(f) = π∗(fW ), W and conversely for any tuple (fW )W , we may ﬁnd f satisfying the equality.

Descent of orbital integrals We recall some results of [2]. Let V be a representation of a reductive group H. Let π : V → V//H be the categorical quotient. An open subset U ⊂ V(F ) is called saturated if it is the preimage of an open subset of (V//H)(F ). V Let x ∈ V(F ) be a semisimple element. Let NHx,x be the normal space of Hx at x. Then V V the stabilizer Hx acts naturally on the vector space NHx,x. We call (Hx,NHx,x) the sliced representation at x. An étale Luna slice (for short, a Luna slice) at x is by definition ([2]) a locally closed V smooth Hx-invariant subvariety Z 3 x together a strongly étalemorphism ι : Z → NHx,x such that the H-morphism φ : H ×Hx Z → V is strongly étale.Here, an H-morphism between two affine varieties φ : X → Y is called strongly étale if φ//H : X//H → Y//H is étaleand the induced diagram is Cartesian:

φ X / Y

φ//H X//H / Y//H.

When there is no confusion, we will simply say that Z is an étaleLuna slice. We then have the Luna’s e´tale slice theorem: Let a reductive group H acts on a smooth affine varieties X and let x ∈ X be semisimple. Then there exists a Luna slice at x. We will describe an explicit Luna slice in the next subsection for our case. We may indeed assume the morphism ι is essentially an open immersion in our case. As an application, we have an analogue of Harish-Chandra’s compactness lemma ([25, Lemma 25]). Lemma 3.10. Let x ∈ V(F ) be semisimple. Let Z be an étaleLuna slice at x. Then for any Hx(F )-invariant neighborhood ξ of x in Z(F ) whose image in (Z//Hx)(F ) is (relative) compact, and any compact subset Ξ of V(F ), the set

h {h ∈ Hx(F )\H(F ): ξ ∩ Ξ 6= ∅} is relatively compact in Hx(F )\H(F ). Proof. We consider thee ´tale Luna slice:

φ : H ×Hx Z → V.

32 Consider the composition:

H ×Hx Z ' V ×V//H Z//Hx ,→ V × Z//Hx. The composition is a closed immersion. Shrinking Z if necessary, we may take the F -points to get a closed embedding

i :(H ×Hx Z)(F ) ,→ V(F ) × (Z//Hx)(F ). We also have the projection

H ×Hx Z = (H × Z)//Hx → Hx\H. We denote

j :(H ×Hx Z)(F ) → (Hx\H)(F ).

0 Note that Hx(F )\H(F ) sits inside (Hx\H)(F ) as an open and closed subset. Let ξ be the image of ξ in (Z/Hx)(F ). Then we see that the set

h {h ∈ Hx(F )\H(F ): ξ ∩ Ξ 6= ∅} is contained in ji−1(Ξ ∩ ξ0) which is obviously compact.

We also need the analytic Luna slice theorem ([2, Theorem 2.7]): there exists (1) an open H(F )-invariant neighborhood U of H(F )x in V(F ) with an H-equivariant retract p : U → H(F )x;

−1 V (2)a Hx-equivariant embedding ψ : p (x) ,→ NHx,x(F ) with an open saturated image such that ψ(x) = 0.

−1 V Denote S = p (x) and N = NHx,x(F ). The quintuple (U, p, ψ, S, N) is then called an analytic Luna slice at x. V ψ −1 NHx,x(F ) o p (x) / U O p 0 o x / H(F )x From an ´etaleslice we may construct an analytic Luna slice from (cf. the proof of [2, Coro. A.2.4]). In our case, the existence of analytic Luna slice is self-evident once we describe the ´etaleLuna slices in next subsection. We recall some useful properties of analytic Luna slice ([2, Coro. 2.3.19]). If y ∈ p−1(x), and z := ψ(y). Then we have

33 (1)( H(F )x)z = H(F )y.

(2) N V(F ) = N N as H(F ) -spaces. H(F )y,y Hx(F )z,z y

(3) y is H-semisimple if and only if z is Hx-semisimple. As an application, we state the Harish-Chandra (semisimple-) descent for orbital integrals.

Proposition 3.11. Let x ∈ V(F ) be semisimple and let (U, p, ψ, S, N) be an analytic Luna V slice at x. Then there exists an neighborhood ξ ⊂ ψ(S) of 0 in NHx,x(F ) with the following properties

∞ ∞ V • For every f ∈ Cc (V(F )), we may associated to fx ∈ Cc (NHx,x(F )) such that for all

semisimple z ∈ ξ (with z = ψ(y)) such that η|Hy(F ) = 1, we have Z Z h h (3.5) f(y )η(h)dh = fx(z )η(h)dh. Hy(F )\H(F ) Hy(F )\Hx(F )

∞ V ∞ • And conversely, given fx ∈ Cc (NHx,x(F )), we may ﬁnd f ∈ Cc (V(F )) such that 3.5

holds for all semisimple z ∈ ξ such that η|Hy(F ) = 1. Proof. Let ξ0 be a relative compact neighborhood of x in S and let ξ = ψ(ξ0). By Harish- Chandra’s compactness lemma 3.10, we may ﬁnd a compact set C of Hx(F )\H(F ) that contains the set 0h {h ∈ Hx(F )\H(F ): ξ ∩ supp(f) 6= ∅}. In the non-archimedean case, we may assume that C is compact open. Choose any function ∞ α ∈ Cc (H(F )) such that the function Z Hx(F )\H(F ) 3 h 7→ α(gh)dg Hx(F ) is the characteristic function 1C in the non-archimedean case, and a bump function that takes value one on C and zero outside some larger compact subset C1 ⊃ C . We deﬁne a function on S by: Z h fx(y) := f(y )α(h)η(h)dh. H(F ) In the non-archimedean case, we may assume that S is a closed subset of V and in the archimedean case, we may assume that S contains a closed neighborhood of x in V whose V image in NHx,x(F ) is the pre image of a closed neighborhood in the categorical quotient. Then possibly using a bump function in the archimedean case to modify fx, we may assume that ∞ ∞ V fx ∈ Cc (S). The map f 7→ fx depends on ξ. We may also view fx ∈ Cc (NHx,x(F )) via the V the embedding ψ : S,→ NHx,x(F ).

34 Now the RHS of 3.5 is equal to Z Z f(yhg)α(g)η(g)dgη(h)dh Hy(F )\Hx(F ) H(F ) Z Z = f(yg)α(h−1g)η(g)dgdh Hy(F )\Hx(F ) H(F ) Z Z Z = f(yg)α(h−1pg)dpη(g)dgdh. Hy(F )\Hx(F ) Hy(F )\H(F ) Hy(F ) Interchange the order of the first two integrals and notice that when g ∈ C Z Z Z α(h−1pg)dpdh = α(h−1g)dh = 1. Hy(F )\Hx(F ) Hy(F ) Hx(F ) By the Harish-Chandra compactness lemma 3.10, the value of the above integral outside C does not matter when y ∈ ξ0. We thus obtain Z g f(y )1C (g)η(g)dg. Hy(F )\H(F ) This is equal to LHS when y ∈ ξ0 (or equivalently, ψ(y) = z ∈ ξ). V To show the converse, we note that ψ(S) is saturated in NHx,x(F ). Replacing fx by fx · 1S in the non-archimedean case, and by fx · αS for some bump function α in the archimedean ∞ cas, we may assume that supp(fx) ⊂ ψ(S). Then we choose a function β ∈ Cc (H(F )) such that Z (3.6) β(h)η(h)dh = 1. H(F ) Consider the surjective map H(F ) × S → U under which H(F ) × S is a Hx(F )-principal homogenous space over U (in the category of ∞ F -manifolds). It is obviously an submersion. We define f ∈ Cc (U) by integrating β ⊗ fx over the fiber Z h g −1 f(y ) := fx(ψ(y ))β(g h)dg, y ∈ S, h ∈ H(F ). Hx(F ) ∞ ∞ Then f ∈ Cc (U) can be also viewed as in Cc (V(F )). The LHS of 3.5 is then equal to Z Z g −1 fx(ψ(y ))β(g h)dgη(h)dh Hy(F )\H(F ) Hx(F ) Z Z Z g −1 −1 = fx(ψ(y ))β(g p h)dgη(h)dh Hy(F )\H(F ) Hy(F )\Hx(F ) Hy(F ) Z Z −1 g = β(g h)η(h)dh fx(ψ(y ))dg. Hy(F )\Hx(F ) H(F )

35 By (3.6), this is equal to Z g fx(ψ(y ))η(g)dg. Hy(F )\Hx(F ) This completes the proof.

Explicit étaleLuna slices Back to our case, we will first describe the sliced representations at a semisimple element of S or U(V ). Then we exhibit an étaleLuna slice at x. The key construction is already in [44]. But we need to show that their construction actually gives étaleLuna slices. The steps are close to the Harish-Chandra’s descent method (cf. [36]). We start with U(V ). It suffices to consider U(W ) × W . We may write an element in U(W ) × W as (X, w), X ∈ U(W ), w ∈ W . Denote by W2 the subspace of W generated by i X w, i ≥ 0. By [44, Theorem 17.2], for (X, w) to be semisimple, it is necessary that W2 is a non-degenerate subspace. In this case, we have an orthogonal decomposition W = W1 ⊕ W2. Then X stabilizes both subspace. So we may write X = diag[X1,X2] for Xi ∈ U(Wi). Then (X, w) is semisimple if and only if X2 is semisimple (in the usual sense) in U(W2). And it is also self-evident that (X2, w) defines a regular semisimple element relative to the action of

U(W2) on U(W2) × W2.Then the stabilizer of (X, w) is isomorphic to U(W1)X1 , the stabilizer of X1 under the action of U(W1). It is a product of the restriction of scaler of unitary group of lower dimension over an extension of F (including the general linear group). Let U(W1)X1 be the respective Lie algebra. Then U(W1)X1 acts on U(W1)X1 × W1. This representation of

U(W1)X1 is a product of representations of the same type (including the general linear case). The sliced representation at x is then isomorphic to the product of the above representation of U(W1)X1 on U(W1)X1 × W1 and the representation of the trivial group on the normal space at (X2, w) of U(W2)-orbit of (X2, w) in U(W2) × W2. A similar description for Sn also holds (cf. [44], [3]). And we have an equivalent version for the restriction of the adjoint action of GLn+1,F to GLn,F . We describe the general form a semisimple element in the Lie algebra gln+1,F . Let X u x = ,X ∈ gl , v d n,F

n n where u ∈ F , v ∈ Fn and we use F (Fn resp.) to denote the n-dimensional space of n column (row, resp.) vectors. There is an obvious pairing between F and Fn. Let U2 be the n n subspace of F spanned by u, Xu, ..., X u and similarly V2 the subspace of Fn spanned by n ⊥ ⊥ v, vX, ..., vX . And let V1 := U2 ⊂ Fn (U1 := V2 , resp.) be the orthogonal complement n of U2 (V2, resp.). Then for x to be semisimple, it is necessary that Fn (F , resp.) is the ⊥ ⊥ direct sum of U and V2 (V and U2, resp.). Assuming this, according to the decomposition 2 2 n X11 X12 F = U1 ⊕ U2, we may write X = . Then by [44, sec. 8], for such x to be 0 X22 semisimple, it is necessary and suﬃcient that X11 ∈ End(U1) is semisimple in the usual sense and X12 = 0.

36 Now we construct étaleLuna slices for semisimple elements. As we shall see as follows, a general case are basically a composition of two extreme cases: 1) the case w = 0 (namely, r = 0 minimal), 2) the case x = (X, w) is regular semisimple (namely, r = 0 maximal). The first case is essentially the same as the classical case (the Luna slice for the adjoint representation, cf. [36, sec. 14]). For the second case we have to resort to the existence theorem of Luna. The general case can be reduced to those two basic cases. We first describe a locally closed subvariety of x = (X, w) following [44, 18]. The case for S is similar following [44, sec. 7]. Let (X, w) be as above. Then we have an orthogonal decomposition W1 ⊕ W2. Denote r = dim W2 ≥ 1. We define a closed subvariety Ξ of r−1 U(W ) × W consisting of (Y, u) such that u, Y u, ..., Y u spans W2. In particular, x = (X, w) ∈ Ξ. Now in [44] Rallis and Schiffmann define an isomorphism of varieties

ι1 :Ξ → (U(W1) × W1) × (U(W2) × W2)rs whose inverse is defined as follows. According to the decomposition W1 ⊕ W2 we may write Y as Y Y 11 12 . Y21 Y22 0 r−1 −1 0 Then Y12 ∈ Hom(W2,W1). Define u = Y12Y u ∈ W1. Then ι1 maps (Y, u) to ((Y11, u ), (Y22, u)). i i It is then easy to see that Y u = Y22u ∈ W2 for i = 0, 1, ..., r − 1. Therefore (X22, u) is a regular semisimple element. It is not hard to see that this defines an isomorphism. Moreover it is equivariant under the action of U(W1) × U(W2): on the LHS the action is the restriction of that of U(W ) to the subgroup; on the RHS. Then the morphism ι induces a morphism between the categorical quotients

] ι1 :(U(W1) × W1)//U(W1) × (U(W2) × W2)rs//U(W2) → (U(W ) × W )//U(W ).

] Similarly, we have morphisms still denoted by ι1 and ι1 in the general linear case S. And we have an equivalent version for the restriction of the adjoint action of GLn+1,F to GLn,F . We describe the analogous construction for this. Let x ∈ gln+1,F be as above and denote by r r = dim U2 = dim V2. Without loss of generality, we may assume U2 = F embedded into n F by sending u to (0, ..., 0, u). Similarly for V2,U1,V1. Then we deﬁne a closed subvariety Y u0 Ξ consisting of y = such that u0, Y u0, ..., Y nu0 span U and v0, v0Y, ..., v0Y n span V . v0 d0 2 2 Then similarly we have an isomorphism ([44, sec. 7]):

n−r r ι1 :(gln−r,F × F × Fn−r) × (glr,F × F × Fr)rs × F → Ξ

−1 r−1 r−1 such that ι maps y to ((Y11,Y12Y22 u, vY22 Y21), (Y22, u, v), d). And it also induces a morphism

] n−r r ι1 :(gln−r × F × Fn−r)//GLn−r × (glr × F × Fr)rs//GLr × F → gln+1//GLn.

] Lemma 3.12. ι1 is ´etalefor both gln+1 (equivalently, S) and U(W ) × W .

37 Proof. We will the coordinates described earlier for these categorical quotients involved in the ] ] morphism ι1. By Jacobian criterion for étaleness, it suffices to show that the Jacobian of ι1 is non-zero everywhere. Indeed we will show the Jacobian is a non-zero constant. Therefore it suffices to compute the Jacbobian over the algebraic closure. In particular, it suffices to consider the equivalent question for the restriction of the adjoint representation of GLn+1,F ] to GLn,F . Note that the Jacobian is a regular function on the source of the morphism ι . It is then enough to show that it is a non-zero constant on a Zariski open subset. The categorical quotients gln+1,F //GLn,F is given by

Spec(F [α1, ..., αn+1, β1, β2, ..., βn]), where the regular invariant functions are deﬁned

i j ∗ αi = tr ∧ x, βj = eX e , x ∈ gln+1,F ,

∗ and e = en = (0, ..., 0, 1), e is the transpose of e. Another choice of invariants is as follows. When we write X u x = ,X ∈ gl . v d n,F then we may also identify the categorical quotient gln+1,F //GLn,F as

0 0 0 0 Spec(F [α1, ..., αn, d, β1, ..., βn]), where 0 i 0 j−1 αi = tr ∧ X, βj = vX u. In particular, the Jacobian of the isomorphism

0 0 0 0 ψ = ψn : Spec(F [α1, ..., αn, d, β1, ..., βn]) → Spec(F [α1, ..., αn+1, β1, β2, ..., βn]) is a nonzero constant

∂(α1, ..., αn+1, β1, β2, ..., βn) × (3.7) 0 0 0 0 = κn ∈ F . ∂(α1, ..., αn, d, β1, ..., βn)

To indicate the dependence on n we may write the invariants as αi(n) etc.. We choose an auxiliary open subvariety consisting of strongly regular elements in the sense of Jacquet–Rallis [35]. More precisely they are in the GLn,F -orbits to elements of the form   a1 1 c1 a2 1    × x = x(a1, ..., an+1, c1, ..., cn) :=  ......  , ai ∈ F, cj ∈ F .    cn−1 an 1  cn an+1

38 All such x’s with an+1 = 0 form a locally closed subvariety denoted by Θn of gln+1,F . Note that dimΘn = 2n. For d ∈ F , let δn(d) be the matrix with only non-zero entry d at the position (n + 1, n + 1). For (x, d) ∈ Θn × F , taking invariants αi, βj of x + δn(d) yields a morphism with Zariski dense image:

ξ = ξn :Θn × F → Spec(F [α1, ..., αn+1, β1, β2, ..., βn]).

Then we claim that the Jacobian of ξn is given by

∂(α1, ..., αn, d, β1, ..., βn) n(n−1)/2 2 n−1 (3.8) = (−1) κ1κ2...κnc2c3...cn . ∂(a1, ..., an + 1, c1..., cn) We prove this by induction on n. It is easy to very this for n = 1. Now for n > 1, we may 0 write ξn = ξn ◦ ψn. As d = an+1, we have ∂(α0 , ..., α0 , d, β0 , ..., β0 ) ∂(α0 , ..., α0 , β0 , ..., β0 ) 1 n 1 n = 1 n 1 n . ∂(a1, ..., an + 1, c1..., cn) ∂(a1, ..., an, c1..., cn)

0 0 Note that αi(n) = αi(n − 1) and for x ∈ Θn, β1(n)(x) = cn and

0 βj(n)(x) = cnβj−1(n − 1)(X), j ≥ 2, since (0, ..., 0, cn) = cnen−1. Here X is the n × n matrix by deleting the last row and the last column. This gives us

∂(α0 , ..., α0 , β0 , ..., β0 ) ∂(α0 , ..., α0 , β0 , ..., β0 ) 1 n 1 n = 1 n 1 n−1 ∂(a1, ..., an, c1..., cn) ∂(a1, ..., an, c1..., cn−1) which is equal to

n−1 n−1 ∂(α1(n − 1), ..., αn(n − 1), β1(n − 1), ..., βn−1(n − 1)) (−1) cn . ∂(a1, ..., an, c1..., cn−1)

0 By induction hypothesis, this gives the Jacobian of ξn:

n(n−1)/2 2 n−1 (−1) κ1κ2...κn−1c2c3...cn .

Together with the Jacobian of ψn (3.7), we have proved (3.8). ] Now let’s return to the morphism ι1. We have an obvious isomorphism

φ :Θn−r × Θr × F → Θn × F which sends the triple

(x(a1, .., an−r, 0, c1, ..., cn−r), x(an−r+1, .., an, 0, cn−r+1, ..., cn), d)

39 0 to x(a1, .., an+1, c1, ..., cn−r, cn−r+1, ..., cn) with

r 0 Y an+1 = d, cn−r = cn−r cn−r+i. i=1 We have the product

n−r r ξn−r,r :Θn−r × Θr × F → (gln−r × F × Fn−r)//GLn−r × (glr × F × Fr)rs//GLr × F.

Then it is to easy to see that the diagram commutes:

φ Θn−r × Θr × F / Θn × F

ξn−r,r ξn

n−r r (gln−r × F × Fn−r)//GLn−r × (glr × F × Fr)rs//GLr × F / gln+1//GLn

] Then the Jacobian of ι1 restricted to the image of ξn−r,r is equal to the ratio of the Jacobian of ξn over that of ξn−r,r and φ. By (3.8), we obtain this ratio is a non-zero constant times

c c3...cn−1 2 3 n = 1. 2 Qr n−r−1 2 r−1 Qr c2c3...(cn−r i=1 cn−r+i) · cn−r+2cn−r+3...cn · i=1 cn−r+i

] This shows that the Jacobian of ι1 is a non-zero constant on a Zariski dense subset and hence ] ] itself is a non-zero constant on the source of ι1. This shows that ι1 is ´etale.

We now reﬁne the morphism ι1. We return to the semisimple element x = (X, w) in the unitary case. Then X11 is semisimple in the usual sense. Therefore we may consider the Lie U U 0 U algebra (W1)X11 of the stabilizer U(W1)X11 and an open subvariety (W1)X11 of (W1)X11 consisting of those Y such that

det(ad(Y ); U(W1)/U(W1)X11 ) 6= 0, cf. [36, 14.5]. Let ι2 be the restriction of ι1 to

0 (U(W1)X11 × W1) × (U(W2) × W2)rs.

Then ι2 is U(W1)X11 -equivariant and it induces a morphism

] 0 ι2 :(U(W1)X11 × W1)//U(W1)X11 × (U(W2) × W2)rs//U(W2) → (U(W ) × W )//U(W ).

Note that the representation of U(W1)X11 on U(W1)X11 × W1 is a product of representations of the same type (including the linear case). Similar construction for gln+1 (equivalently, S).

] Lemma 3.13. The morphism ι2 is ´etalefor both gln+1 (equivalently, S) and U(W ).

40 Proof. We By the previous lemma, it suﬃces to show that the morphism

0 (U(W1)X11 × W1)//U(W1)X11 → (U(W1) × W1)//U(W1) is ´etale.It is not hard to show that the Jacobian of this morphism at the image of (Y, u) ∈ U 0 (W1)X11 × W1 is given by, up to a sign:

det(ad(Y ); U(W1)/U(W1)X11 ).

U 0 This is non-zero by the deﬁnition of (W1)X11 . To lighten notations, we denote

V = U(W ) × W, Vx = (U(W1)X11 × W1) × (U(W2) × W2),

H = U(W ),Hi = U(Wi), and Hx = U(W1)X11 which is isomorphic to the stabilizer of x. Similar notations apply to gln+1 (equivalently, S). Let 0 0 Vx = (U(W1)X11 × W1) × (U(W2) × W2)rs. 0 Then x ∈ Vx. Note that ι2 also induces a morphism

0 ι : H ×(Hx×H2) Vx → V by sending (h, x) to h · ι2(x). Lemma 3.14. The morphism ι is ´etale.

Proof. We show this in the unitary case. It suffices to show that the differential dι at (1, y) is an isomorphism. We first assume that X11 is a scaler. Then Hx = H1. Suppose y = (Y, u) ∈ Vx. Then it is not hard to see that for ∆Y = diag(∆Y11, ∆Y22), ∆u = (∆u1, ∆u2),

ι2(Y + ∆Y, u + ∆u) = ι(Y, u) + ∆ι2(Y, u) + higher terms, ∆Y11 φ∆u1 ∆ι2(Y, u) := , ... ∆Y22 where the part ... is determined by the hermitian condition, and φ∆u1 ∈ Hom(W2,W1) is the i r−1 homomorphism that sends Y22u2 to 0 for i = 0, 1, ..., r − 2 and Y22 u2 to ∆u1. Then the diﬀerential dι at (1, y) is given by

dι : U(W ) × V → V U(W1)X11 ×U(W2) x

(∆X, (∆Y, ∆u)) 7→ ([∆X, diag(Y11,Y22)] + ∆ι2(Y, u), ∆u2 + ∆X · u2).

41 Here the LHS means the quotient of (U(W ) × Vx)/U(W1)X11 × U(W2). By comparing the dimension, it suﬃces to show that dι is injective. Suppose that dι(∆X, (∆Y, ∆u)) = 0. By the action of (U(W ) × Vx)/U(W1)X11 × U(W2), we may assume that 0 φ ∆X = , φ ∈ Hom (W ,W ). ... 0 E 2 1

Then we need to show that φ = 0 and (∆Y, ∆u) = 0. From the diagonal blocks, It is easy to see that ∆Y = 0. Note that now we have ∆X · u2 = φu2 ∈ W1. Therefore ∆u2 + ∆X · u2 = 0 implies that both ∆u2 = 0 and φu2 = 0. Now we use the condition from the oﬀ-diagonal block to obtain

Y11φ − φY22 + φ∆u1 = 0 ∈ Hom(W2,W1). r−1 Since (Y22, u2) is regular semisimple, u2,Y22u2, ..., Y22 u2 form a basis of W2. We apply the i above homomorphism to Y22u2:

i+1 i i φY22 u2 = Y11φY22u2 + φ∆u1 Y22u2.

i+1 Since φu2 = 0, we may show that φY22 u2 = 0 recursively. This shows that φ = 0 ∈ Hom(W2,W1). This completes the proof when x11 is a scalar.

Now we consider a general semisimple X11, and Hx = U(W1)X11 . Then the assertion follows if we show that the following analogous morphism is ´etale

0 H1 ×Hx (U(W1)X11 × W1) → U(W1) × W1. Similar argument as in the above works and we omit the details.

Proposition 3.15. For both gln+1 (equivalently, S) and U(W ), the following diagram is cartisian 0 ι H ×(Hx×H2) Vx / V

π ι] 0 2 Vx//(Hx × H2) / V//H. Proof. Thanks to the previous lemmas, now the proof is similar to that of [36, Lemma 14.1]. We need to show the induced morphism

0 0 γ : H ×(Hx×H2) Vx → Vx//(Hx × H2) ×V//H V is indeed an isomorphism. It suﬃces to show that in an algebraic closure the induced map on the geometric points is bijective. From this we see that the question becomes equivalent for both gln+1 (equivalently, S) and U(W ). To simplify exposition, we consider the unitary case. Actually the bijectivity holds for any ﬁeld as we now show. To show the surjectivity, 0 0 0 0 we may write an element Vx//(Hx × H2) ×V//H V as ((Y , u ), (Y, u)) where (Y, u) ∈ Vx ] 0 0 is abused to denote its image in the quotient. Then ι2(Y , u ) = π(Y, u). This implies

42 that the u, Y u, ..., Y r−1u span a hermitian subspace that is isometric to that spanned by 0 0 0 0r−1 0 u2,Y22u2, ..., Y22 u2 which is non-degenerate. Therefore by Witt’s theorem there exists an 0i 0 i 0 0 h ∈ H such that h(Y22u2) = Y u for 0 ≤ i ≤ r−1. We may thus assume that u = u2,Y22 = Y22. The rest follows from [36, Lem. 14.1]. To show the injectivity, without loss of generality it suffices to show that if γ(1, y) = γ(h, z), then h ∈ Hx × H2 and y = hz. it suffices to show h ∈ Hx × H2 since the second assertion follows from this. Denote y = (Y, u) and z = (Z, w). Then h obviously preserves the subspace W2 and hence W1, too. It follows that h ∈ H1 ×H2. The rest follows from [36, Lem. 14.1]. We now may construct an ´staleLuna slice of a semisimple element x. We only do so in the unitary case. Choose an ´staleLuna slice Z2 of (X22, w) (which is H2-regular semisimple) for the action of H2 on U(W2) × W2. Then an étaleLuna slice of x can be chosen to be the U 0 image of (W1)X11 × W11 × Z2 under ι2. Similarly, we see that this also gives us a way to choose an analytic Luna slice once we fix a choice of analytic Luna slice for the H2-regular semisimple element (X22, w).

Smooth transfer for regular supported functions ∞ Lemma 3.16. Let V be either S or U(V ). Let f ∈ Cc (V(F )). Then the function π∗,ω(f) is smooth on (V//H)(F )rs and compact supported on (V//H)(F ). Proof. The smoothness follows from the first part of Prop. 3.11 and the fact that the stabilizer of a regular semisimple element is trivial. The support is contained in the continuous image of a compact set, hence compact. 0 ∞ ∞ Proposition 3.17. (1) If f ∈ Cc (Srs), then π∗,ω ∈ Cc (Q(F )rs). And conversely given ∞ 0 ∞ 0 φ ∈ Cc (Q(F )rs), there exists f ∈ Cc (Srs) such that π∗,ω(f ) = φ. ∞ ∞ ∞ (2) If fW ∈ Cc (U(V )rs), then π∗(fW ) ∈ Cc (Q(F )rs). And conversely given φ ∈ Cc (Q(F )rs), ∞ P there exists a tuple (fW ∈ Cc (U(V )rs))W such that W π∗(fW ) = φ. Proof. We only prove (1) and the proof of (2) is similar. By the previous lemma, it suffices to show the converse part. By the localization principle Prop. 3.8 (or rather its proof), it suffices to show that for every regular semisimple x ∈ Q(F ), φ is locally around x an orbital integral of a function with regular semisimple support. We now fix a regular semisimple x. Note that the stabilizer of x is trivial. When choosing of the analytic slice, we may require that S is contained in the regular semisimple locus. Then the result follows from the decent of orbital integral, the second part of Prop. 3.11. 0 ∞ Theorem 3.18. Given f ∈ Cc (Srs), there exists its smooth transfer (fW ) such that fW ∈ ∞ ∞ Cc (U(V )rs). Conversely, given a tuple (fW ∈ Cc (U(V )rs))W , there exists its smooth transfer 0 ∞ f ∈ Cc (Srs). In particular, this includes the existence of local transfer at a regular semisimple point z ∈ Q(F ).

43 Reduction to local transfer around 0 of sliced representations. Fix z ∈ Q(F ). 0 Within the fiber of x, there are one semisimple H -orbit in Sn+1 and finitely many semisimple H-orbits in U(V ). Note that there may be infinitely many non-semisimple orbits within the fiber. We have described the sliced representations of those semisimple elements and we have seen that those are still of the same type as S or U(V ) with lower dimension and possibly extending the base field F to a finite extension. So we may also speak of local transfer around zero of those sliced representations. To compare the meaning of transfer at z and zero of the sliced representations, we need to compare their transfer factors. X u We may define an equivalent choice of transfer factors as follows. For x = ∈ v d gln+1,F we define ν(x) = det(u, Xu, X2u, ..., Xn−1u). Then the transfer factor can be chosen as η(ν(x)) ∈ {±1}.

Lemma 3.19. We may choose an Hx-invariant neighborhood of x such that for any y in this neighborhood, ω(y) is equal to a non-zero constant times ω(ψ(y)). Proof. We only treat the two basic case: (1) r = 0, (2) r = n. The general case can be reduced to those two by the same strategy as in the proof above. When r = n, namely x is regular semisimple so that Hx is trivial, the assertion follows from that there is a neighborhood of x over which ω is a constant. When r = 0, this follows from the fact that

1/2 ν(y) = ±ν(ψ(y))det(ad(Y ), gln+1/gln+1,Y11 )

1/2 where det(ad(Y ), gln+1/gln+1,Y11 ) is a square root of det(ad(Y ), gln+1/gln+1,Y11 ) (for example it can be given by the determinant of ad(Y ) on the upper triangular blocks). Since det(ad(X), gln+1/gln+1,X11 ) 6= 0, we may shrink the neighborhood if necessary such that η(ν(y)) and η(ν(ψ(y))) differ only by a non-zero constant. Proposition 3.20 (Reduction to zero). Fix z ∈ Q(F ). If the local transfer around zero exists for the sliced representations, then the local transfer around z exists. Proof. By Prop. 3.11 , under our choice of the étaleLuna slice and hence an analytic Luna slice, the orbital integral of regular semisimple element near to a semisimple element can be written as an orbital integral of regular semisimple elements near zero of the sliced representation. By our construction above, the choice of étaleLuna slices on Sn+1 and U(V )’s can be made compatible. Moreover, the transfer factors are compatible with respect to the choice of analytic Luna slices by Lemma 3.19 above. This completes the proof. Remark 11. We see that the reduction steps are along the same line as those in the classical endoscopic transfer by Langlands–Shelstad. The only non-trivial point is the explicit construction of the étaleLuna slices which is a slightly more involved than the case of adjoint action of a reductive group on its Lie algebra.

44 4 Smooth transfer for Lie algebra

In this section we assume that F is non-archimedean of characteristic zero, namely a ﬁnite extension of Qp for some rational prime p.

4.1 A relative local trace formula To simplify notations we unify the linear side and the unitary side in this subsection. Let F be a ﬁeld and E be ane ´tale F -algebra of rank two. Namely, E is either a quadratic ﬁeld extension or F × F . Let W be a free E-module of rank n and B : W × W → E be a non-degenerate Hermitian form. We denote by H = HW the algebraic group U(W ):

H = U(W ) = {h ∈ AutE(W ):(hu, hv) = (u, v)} and U(W ) its Lie algebra:

U(W ) = {X ∈ EndE(W ):(Au, v) = −(u, Av)}.

Note that we allow E = F × F in which case we have H ' GLn. We will use x → x¯ to denote the (unique) non-trivial F -linear automorphism of E. In the case E = F × F , it is the permutation of the two coordinates. We consider the representation of H on an F -vector space: V = U(W ) × W. We usually denote by x = (X, w) an element of V for X ∈ U(W ), w ∈ W . We deﬁne

i j n−1 (4.1) ∆(x) = ∆(X, w) = det((X w, X w))i,j=0. Then x is regular semisimple if and only if ∆(x) 6= 0. In the case E = F × F , we may equivalently consider n gln × Fn × F , n where Fn (F , resp.) is the n-dimensional vector space of row (column, reps.) vectors. We consider the natural action of GLn by h · (X, u, v) = (h−1Xh, uh−1, hv). Similarly we deﬁne for an element x = (X, u, v)

i+j n−1 ∆(x) = ∆(X, u, v) = det(uX v)i,j=0 Then x is regular semisimple if and only if ∆(x) 6= 0. In either case, we denote by D(X) the discriminant of X ∈ U(W ) or gln, namely

Y 2 D(X) = (λi − λj) , i,j where λ1, λ2, ..., λn are the n eigenvalues of X.

45 Upper bound of orbital integrals. We ﬁrst estimate the orbital integral for a regular semisimple x ∈ V: Z (4.2) O(x, f) = f(xh)η(h)dh, H where η is any (unitary) character.

Lemma 4.1. Let Ω ⊂ Mn(F ) be a compact open set. Let T be a Cartan subgroup of GLn(F ) ∞ n and t ⊂ Mn(F ) be the corresponding Cartan subalgebra. Let ϕ ∈ Cc (Fn × F ). Then there exists a constant r > 0 depending only on n and a constant C depending only on n, ϕ, Ω with the following property: for all regular semisimple X ∈ t, and h ∈ such that h−1Xh ∈ Ω, we have Z ϕ(uth, h−1t−1v)dt ≤ Cmax{1, log|∆(X, u, v)|}r T Proof. If h−1Xh ∈ Ω, then for all i, j = 0, ..., n − 1, the following vectors are in a compact set depending only on the support of ϕ and Ω: h−1t−1Xiv, uXjth.

i j Write δ1 = (X v)i=0,...,n−1 ∈ Mn(F ) and δ2 = (uX )j=0,...,n−1 ∈ Mn(F ) so that ∆(X, u, v) = det(δ1δ2). Then the condition becomes

−1 −1 h t δ1, δ2th. are in a compact set Ω1 of Mn(F ) depending only on the support of ϕ and Ω. Qe Qe × We may identify twith Ei and T with Ei where Ei/F is a degree ni ﬁeld P i=1 i=1 extension such that ni = n. Let P be the parabolic of H = GLn−1 associated to the P partition ni = n with Levi decomposition P = MN and we may assume that T ⊂ M. Then T is elliptic in M. By Iwasawa decomposition H = NMK for K = GLn(OF ), we may write h = nmk, δ1 = n1m1k1, δ2 = k2m2n2. By enlarging Ω1 if necessary,we may assume that −1 −1 −1 −1 Ω1 = KΩ1K. Since h tδ1 = k m n tn1m1k1 ∈ Ω1, we may see that the Levi component −1 −1 (i) e m t m1 ∈ Ω2 for a compact set Ω2 of M2(F ). Similarly we have m2tm ∈ Ω2. Let t = (t )i=1 (i) × where t ∈ Ei similarly for m, m1, m2. Then we have some constant C1,C2 > 0 such that for all i = 1, 2, ..., e

(i) (i) −1 (i) (i) −1 C1|det(m1 (m ) )| ≤ |ti| ≤ C2|det(m2 m )| .

The volume of such ti is bounded above by

(i) (i) C4 − log|det(m1 m2 )| ≥ 0

46 Under this assumption, we have a constant C such that for all i = 1, 2, ..., e:

Proposition 4.2. There exists a constant C depending on f and integer r > 0 such that for all x ∈ V strongly regular:

|O(x, f)| ≤ Cmax{1, | log |∆(x)||r}max{1, |D(X)|−1/2}.

Proof. We only give the proof for the linear case. The unitary case is similar (perhaps easier). We choose a (ﬁnite) set of representatives of Cartan subalgebra t. We may assume that X ∈ t ∞ and f = φ ⊗ ϕ where φ ∈ Cc (Mn(F )) and ϕ as in the previous lemma. Then we have Now we have Z −1 |O(x, f)| ≤ C6max{1, | log |∆(x)||} |φ|(ht0h )dh. H/T By the bound of Harish-Chandra on the usual orbital integral, the above integral is bounded −1/2 by a constant times max{1, |D(X)| }. Since there are only ﬁnitely many ti, we may choose a uniform constant C to complete the proof.

Local integrability We want to show the orbital integral is a locally integrable function on V. The following result is probably well-known. But we could not ﬁnd a reference so we decide to include a proof here.

Lemma 4.3 (Igusa integral). Let P (x) ∈ F [x1, ..., xm] be a polynomial. Then there exists > 0 such that Z |P (x)|−dx < ∞. m OF If P is homogeneous, then there exists > 0 such that the function |P (x)|− is locally integrable everywhere.

47 Proof. The ﬁrst assertion implies the second one. If P is homogenous of degree k, assuming − m the ﬁrst assertion we want to show that |P | is locally integrable around any x0 ∈ F . In −n m deed, we may assume that x0 ∈ $ O for some n > 0. By homogeneity, we have Z Z |P (x)|−dx = |$|−mn+kn |P (x)|−dx < ∞. $−nOm Om

This shows the local integrality around x0. To show the ﬁrst assertion, we may assume that P ∈ OF [x] and F = Qp. Now following Igusa ([7]) we may deﬁne

n m n Nen := ]{x ∈ (Zp/p ) |f(x) ≡ 0 mod p }.

nm Let wn = Nen/p . Then wn+1 < wn and we want to prove that there exists > 0 such that

X n p (wn − wn+1) < ∞. n And deﬁne the Poincare series ∞ X n Pe(T ) = NenT . n=0

By the rationality of Pe(T ) proved ﬁrstly by Igusa ([7]), we may write

Y βi −ki,j Pe(T ) = Q(T ) (1 − αi,jp T ) i,j where , βi ∈ R, ki ∈ N>0,βi are distinct. αi,j are roots of unit, and Q(T ) is a polynomial ([7, n m nm Remark 3.3]). We must have all βi ≤ m since Nen ≤ |(Zp/p ) | = q for all n. If all βi < m, then we certainly can choose > 0 such that all βi < m − and hence

n(m−) Nen = O(p ).

nm Assume now that β0 = m and all other βi < m. Since |Nen| ≤ p we must have all k0,j = 1 (i.e., no multiplicity). Then for all < m − maxi6=0{βi} and aj ∈ C such that

nm X n n(m−) |Nen − p ajα0,j| = O(p ) j

0 P n 0 when n large. Let wn = j ajα0,j. Since αi,j are roots of unit, wn is periodic, say with period N. Then by 0 −n |wn − wn| = O(p ) we have 0 0 −n −n wn − wn+1 ≤ wn − wn+N ≤ |wn − wn+N | + O(p ) = O(p ). And this ﬁnishes the proof.

48 Remark 12. The same holds for archimedean local ﬁelds and they are called local zeta function.

Corollary 4.4. There exists > 0 such that the function

−1/2− m : x 7→ |D(X)| log |∆(x)| is locally integrable on V.

Proof. Let Ω be a compact subset of V. In Young’s inequality ap bq 1 1 ab ≤ + , a, b, p, q > 0, + = 1, p q p q we let p = 1 + 1 to obtain

|D(X)|−(1/2+)(1+1) | log |∆(x)||q m(x) ≤ + . 1 + 1 q We now need to use the Lie algebraic version of [25, Theorem 15], namely: there exists −1/2−2 2 > 0 such that the function X 7→ |∆(X)| is locally integrable on U(W ). This implies that for an appropriate choice of , 1, the ﬁrst term above is locally integrable on U(W ) × W . The Lemma above implies the second term is also locally integrable. This completes the proof.

In summary we have showed that

∞ Corollary 4.5. For any f ∈ Cc (V), we have • The absolute value of the orbital integral x 7→ |O(x, f)| is locally integrable on V.

• If X ∈ gln(F ) (U(W ) in the unitary case) is regular semisimple, w 7→ |O((X, w), f)| is locally integrable on W .

• If w ∈ W is regular semisimple, X 7→ |O((X, w), f)| is locally integrable on gln(F ) (U(W ) in the unitary case).

A relative local trace formula We now show a local trace formula for the representation of H on V. We would like to consider partial Fourier transform with respect to an invariant subspace V0 of V. We ﬁx an H-invariant bilinear form on V such that its restriction to any invariant subspace is non-degenerate (obviously such form exists). We deﬁne a partial Fourier transform: Z ∞ FV0 f(x) = f(y)ψ(hx, yi)dy, f ∈ Cc (V). V0

49 Choose the self-dual measure on V0. Then the fact that Fourier transform is an isometry on L2 space can be written as: Z Z

f1(x)f 2(x)dx = FV0 f1(x)FV0 f 2(x)dx. V0 V0

And it is clear for two orthogonal subspace V0, V1:

FV0+V1 = FV0 ◦ FV1 = FV1 ◦ FV0 .

n Returning to our case, V is either gln × Fn × F or U(W ) × W for a hermitian space W . In each case we have an abelian 2-group of Fourier transforms generated by the two

n partial transforms Fgln , FFn×F (Fu(W ), FW , resp.). We let fb denote the Fourier transform ˇ with respect to ψ and f the Fourier transform with respect to ψ−1.

n Theorem 4.6. Let V be either gln ×Fn ×F or U(W )×W and V0 an invariant-subspace with ⊥ an invariant complementary space V0 . We write x = (y, z) according to the decomposition ⊥ ⊥ ∞ V = V0 ⊕ V0 . Fix a regular semisimple z ∈ V0 . For f1, f2 ∈ Cc (V), deﬁne Z Z h T (f1, f2) = f1((y, z) )η(h)dh f2(y, z)dy, V0 H(F ) where η is trivial in the unitary case. Then we have ˇ T (f1, f2) = T (fb1, f2).

Proof. We consider the unitary case. The linear case is similar. The idea is the same as in Harish-Chandra’s work on the representability of the character of a supercuspidal representation. Take a sequence of increasing compact subsets Ωi of H such that H = ∪Ωi. Deﬁne Z h Oi(x, f) = f(x )dh. Ωi Then it is clear for a regular semisimple X:

O(x, f) = lim Oi(x, f). i→∞

∞ Note that we have for any f1, f2 ∈ Cc (V): Z Z h h ˇ f1((y, z) )f2(y, z)dy = fb1((y, z) )f2(y, z)dy. V0 V0 noting that the Fourier transform commutes with the H-translation. Therefore we have Z Z Z Z h h ˇ lim f1((y, z) )f2(y, z)dy dh = lim fb1((y, z) )f2(y, z)dy dh. i→∞ i→∞ Ωi V0 Ωi V0

50 As Ωi is compact, we may interchange the order of integration. Obviously we have |Oi((y, z), f1)| ≤ |Oi((y, z), |f1|)|. When z is regular semisimple, by Corollary 4.5 the function y 7→ |O((y, z), |f1|) is locally integrable. By Lebesgue dominated convergence we obtain Z Z Z h lim f1((y, z) )f2(y, z)dy dh = O((y, z), f1)f2(y, z)dy = T (f1, f2). i→∞ Ωi V0 V0 ˇ Similarly we get T (fb1, f2) from the RHS. This completes the proof.

A consequence. Now we consider the case n = 1. We deduce the representability of the Fourier transform of an orbital integral on W = M × M ∗ where M is a one dimensional F -vector space. And let H = GL(M) ' GL1,F act on W . Corollary 4.7. For any quadratic character η (possibly trivial), the Fourier transform of the orbital integral Z h Obw(f) = ω(w) fb(w )η(h)dh H (here ω(w) is a transfer factor) is represented by a locally constant H-invariant function on η ∞ the regular semisimple locus Wrs denoted by κ (w, ·); namely for any f ∈ Cc (W ) we have Z 0 0 η 0 0 Obw(f) = f(w )ω(w )κ (w, w )dw . W Similar result holds for the unitary case. Moreover, we may let F be a product of ﬁelds and we have similar results.

Proof. The proof is along the same line as the proof of Harish-Chandra of the representability of Fourier transform of orbital integrals ([26, Lemma 1.19, pp.12]) noting that the Howe’s ﬁniteness conjecture holds for the H-action on W ([43, Theorem 6.1]). In the next two subsections we will prove a more explicit result when η is nontrivial quadratic.

Obviously the kernel function κη(w, w0) can be viewed as a locally constant function on Wrs × Wrs invariant under H × H. Remark 13. Unfortunately, the Howe’s ﬁniteness conjecture (cf. [43, Theorem 6.1]) fails for n the H-action on gln × Fn × F when n ≥ 2. Therefore the proof of Harish-Chandra ([26]) does not say anything on the representability of the Fourier transform of H-orbital integral.

51 4.2 A Davenport–Hasse type relation We show a Davenport–Hasse type relation between two “Kloosterman sums”. It will be used to show that the Fourier transforms preserve transfer when n = 1. We first recall the definition and some basic properties of the Langlands’ constant. For a fixed non-trivial character ψ of F , and a field extension K/F of local fields, we define a character of K by ψK = ψ ◦ T rK/F . Let 1K be the trivial representation of the Weil group WK . Then the Langlands constant is defined to be

(IndK/F 1K , s, ψ) λK/F (ψ) := (1K , s, ψK ) which is a constant independent of s. In particular, it is given by

λK/F (ψ) := (IndK/F 1K , 1/2, ψ). as × The character of WF ' F :

ηK/F (ψ) := det(IndK/F 1K ) × is of order two. For a ∈ F we denote by ψa the character of F deﬁned by ψa(x) = ψ(ax). We then have λK/F (ψa) = ηK/F (a)λK/F (ψ). Moreover we have 2 λK/F (ψ) = ηK/F (−1).

In particular, λK/F (ψ) ∈ µ4 is a fourth root of unity. We ﬁrst show that the epsilon factor is in fact a sort of Gauss sum. Let ψ be a nontrivial additive character and dx be the self-dual measure on F . We choose a Haar measure on F × to be d×x. For a quasi-character χ of F ×, we may deﬁne its real exponent Re(χ) to be the unique real number r such that |χ(x)| = |x|r for all x ∈ F ×. We denoteχ ˆ = χ−1| · |. Lemma 4.8. The gamma factor as a meromorphic function of χ (namely, its value at χ| · |s is meromorphic for s ∈ C) is given by Z dx Z γ(χ, ψ) = ψ(x)ˆχ(x) = ψ(x)χ−1(x)dx. F × |x| F Here the RHS is interpreted as Z s dx ψ(x)ˆχ(x)|x| |s=0 |x|

52 Proof. By deﬁnition ζ(f,ˆ χˆ) γ(χ, ψ) = . ζ(f, χ) × dx Note that it does not depend on the choice of measure on F . We choose |x| . We choose n fn = 11+$ OF . Then we have

n n ˆ ˆ n ˆ fn(x) = ψ(x)1$ OF (x) = ψ(x)|$| 1OF (x$ ) and if n is larger than the conductor of χ:

n ζ(fn, χ) = |$| vol(OF ).

ˆ −m There is an integer m be such that 1OF = vol(OF ) · 1$ OF . Then we have

Z ˆ nˆ n dx ζ(fn, χˆ) = ψ(x)|$| 1OF (x$ )ˆχ(x) F × |x| Z n dx = vol(OF )|$| ψ(x)ˆχ(x) . −n−m $ OF |x| The RHS is interpreted in the sense of analytic continuation. Therefore we have for n large enough: Z dx γ(χ, ψ) = ψ(x)ˆχ(x) . −n−m $ OF |x|

We have some sort of Kloosterman sum. Let E be a quadratic extension and let E1 be the kernel of the norm map. Consider an analogue of Kloosterman sum: for a = AA¯ ∈ Norm(E×): Z × Φ(a) = ΦE/F (a) := ψE(Ax)d x. E1 1 ∞ The measure on E is chosen such that for all φ ∈ Cc (E): Z dX Z Z φ(X) = φ(At)dtd×a E× |X|E F × E1 where dX is the self-dual measure on E with respect to the character ψE and NA = a. Let η be the quadratic character of F × associated to E/F by local class ﬁeld theory. And deﬁne Z a × Ψ(a) = ΨE/F (a) := ψ(x + )η(x)d x. F × x The RHS is similarly interpreted as Z a ψ(x + )η(x)d×x 1/C<|x|

53 for large enough C (depending on a). Note that this integral becomes a constant when C is large enough. We also deﬁne some auxiliary functions: Z × ΨC (a) := ψ(a/y + y)η(y)d y. |a|/C<|y|

And we formally set Ψ∞ = Ψ. Obviously ΨC converges to Ψ∞ pointwisely as C → ∞. × Lemma 4.9. The function ΨC (possibly C = ∞) is a locally constant function on F . And there are constant β1, β2 independent of C such that

|ΨC (a)| ≤ β1| log |a|| + β2 for all a ∈ F ×. Proof. Let B be such that ψ(x) = 1 when |x| ≤ B. If |a| < B2, either |x| < B or |a/x| < B. So we may bound the integral by Z Z | ψ(a/x)η(x)d×x| + | ψ(x)η(x)d×x|. |x|a/B,|a|/C<|x|

The ﬁrst term is at most Z 1 d×x a/B<|x|

A key technical lemma is to estimate the asymptotic behavior of ΨC (a) and Φ(a) as |a| → ∞. Lemma 4.10. There is a constant A independent of C such that when |a| > A , there is a constant α independent of C such that

−1/4 |ΨC (a)| < α|a| and Φ(a) < α|a|−1/4.

54 Proof. The proof follows the strategy of that of [50, Prop. VIII.1]. We only prove the case for ΨC since the same proof with simple modiﬁcation also applies to Φ. Denote by v the valuation on F . We may choose a constant c such that whenever m ≥ c, the exponential map

m × $ OF → F X ti t 7→ et = i! i≥0

m converges and we have for t ∈ $ OF :

( et+e−t 2 v( 2 − 1) = v(t /2). (∗) et−e−t v( 2 ) = v(t).

m m Let Km be the image of $ OF . It is easy to see that Km = 1 + $ OF . We also choose an integer (the conductor of ψ) d such that ψ(x) = 1 if v(x) ≤ d and ψ(x) 6= 1 for some x with v(x) = d − 1. Now we choose ` ∈ Z such that (∗∗) ` > 4c − 2d + 10.

Now assume that v(a) < −`. To explain the idea and to warm up, we ﬁrst show that × when v(a) < −`,ΨC (a) = 0 unless a ∈ F . Suppose that a is not a square, then |a/x ± x| = max{|x|, |a/x|} ≥ |a|1/2. Z X −1 × ΨC (a) = ψ(ax + x)η(x)d x v(a/C) c be such that

n + min{i, v(a) − i} < d, 2n + min{i, v(a) − i} > d.

n Such n obvious exists due to (∗∗). Then we have a nontrivial character of $ OF : t 7→ ψ(ax−1e−t + xet))η(x) = ψ((ax−1 − x)t + (ax−1 + x)t2/2 + ...) = ψ((ax−1 − x)t)η(x).

Hence the integral over v(x) = i can be break into a sum of integrals over xKn where x runs i over $ OF /Kn. But obviously each term is of the form Z |x|−1 ψ(ax−1k−1 + xk)η(x)d×k Kn Z =|x|−1 ψ(ax−1e−t + xet))η(x) n $ OF =|x|−1η(x)ψ((ax−1 − x)t)dt = 0.

55 Now we may assume that a = b2 is a square. A change of variable yields: Z −1 × ΨC (a) = η(b) ψ(b(x + x))η(x)d x. |b/C|

For a ﬁxed x, we look for an integer n such that ( n + v(b) + v(x − x−1) < d, 2n + v(b) > d.

For example we may take n = 1 + [(d − v(b))/2] > c (due to (∗∗)). Then the second condition becomes:

(∗ ∗ ∗) v(x − x−1) < d − 1 − v(b) − [(d − v(b))/2] = [(d − 1 − v(b))/2].

−1 −1 −1 If this last inequality holds, the v(xk − x k ) = v(x − x ) if k ∈ Kn: this is obvious if v(x) 6= 0; if v(x) = 0, then

(x − x−1) − (xk − x−1k−1) = x(1 − k)(1 + x−2k−1) has valuation at least v(1 − k) ≥ n > v(x − x−1). In particular, for those x satisfying (∗ ∗ ∗), we may write the integral as a disjoint union of the form

X Z |x|−1η(x)|x|−1 ψ(b(x−1 − x)t)dt = 0. n x $ OF Therefore, we have proved that the only possible non-zero contribution comes from those −1 x violating (∗∗∗). In particular, |ΨC (a)| is bounded by the volume of x such that v(x−x ) ≥ [(d − 1 − v(b))/2]. It is easy to see that the volume is bounded by α|b|−1/2| = α|a|−1/4 for some constant α independent of C.

Remark 14. Just like [50, Prop. VIII.1], the same argument in the proof above actually yields a formula of Ψ(a) and Φ(a) for large a.

3 Lemma 4.11. When 4 < Re(χ) < 1, we have Z γ(χ, ψ)γ(χη, ψ) = Ψ(a)χ−1(a)da F

3 where the integral converges absolutely. Similarly, 4 < Re(χ) < 1 we have Z −1 γ(χE, ψE) = Φ(a)χ (a)da. F

56 Proof. We have for C large enough Z Z × × γ(χ, ψ)γ(χη, ψ) = ψ(x)ˆχ(x)d x ψ(y)χηc(y)d y. |x|

1/4 Note that |ΦC (a)| is bounded by β1| log |a|| + β2 when a is small and bounded by α|a| when a is large. The result now follows from Lebesgue’s dominance convergence theorem. Similarly the second result follows noting that |Φ(a)| is bounded by a constant and when a is small and bounded by α|a|1/4 when a is large. Theorem 4.12. We have Ψ(a) = Φ(a)λE/F (ψ).

Proof. Note that L(IndK/F 1K , s) = L(1K , s). Thus we have an equivalent form of Langlands constant: γ(IndK/F 1K , s, ψ) λK/F (ψ) := . γ(1K , s, ψK ) × 3 Then for all characters χ of F such that 4 < Re(χ) < 1, we have Z Z −1 −1 Ψ(a)χ (a)da = λE/F (ψ) Φ(a)χ (a)da. F F And both integrals converges absolutely. Now the theorem follows easily.

4.3 A property under base change For later use we need a property of the Langlands constant of a quadratic extension under base change. Qm If K = K1 × K2 × ... × Km, we define λK/F (ψ) as the product i=1 λKi/F (ψ). Similarly, Qm (IndK/F 1K , 1/2, ψ) is by definition i=1 (IndKi/F 1Ki , 1/2, ψ). Recall that for an arbitrary field extension K/F of degree d, we may define a discriminant × × 2 δK/F ∈ F /(F ) as follows. Choose a F -basis α1, ..., αd of K and let σ1, ..., σd be all F - embeddings of K into an algebraic closure of F . Then it is easy to see that

2 × det(σj(αi))1≤i,j≤d ∈ F .

57 × And if we change the F -basis, this only changes by a square in F . So we define δK/F to be × × 2 σj 2 the class in F /(F ) of det(αi ) . If K = F (α) and α has minimal polynomial f, then δK/F is the class of the discriminant of the polynomial f. In particular, in this case we can choose d(d−1)/2 × a representative of δK/F such that it lies in (−1) NK/F K . Finally, for a quadratic × character η of F , it makes sense to evaluate η(δK/F ). Finally it is evident how to extend the definition to a product of fields K = K1 × K2 × ... × Km. The major property we need is the following. Theorem 4.13. Let E be a quadratic extension and let F 0/F be a field extension of degree 0 0 d. Let E = E ⊗F F . Then we have d λE0/F 0 (ψF 0 ) = λE/F (ψF )ηE/F (δF 0/F ). × First of all we have the following simple observation. If we replace ψ by ψa, a ∈ F , the d RHS changes by a factor ηE/F (a) and the LHS changes by a factor ηE0/F 0 (a) = ηE/F (NF 0/F a) = d ηE/F (a ). Therefore it suffices to prove the identity for any one choice of ψ. Note that the theorem is equivalent to d (ηE0/F 0 , 1/2, ψF 0 ) = (ηE/F , 1/2, ψF ) · ηE/F (δF 0/F ). Lemma 4.14. If E/F is unramified, then the theorem holds.

Proof. For a p-adic local field F we let $F denote a fixed uniformizer. The level of a character −k ψ is by definition the maximal integer k such that ψ is trivial on $F OF . We choose ψ to have level zero. since ηE/F is unramified, we have

(ηE/F , 1/2, ψF ) = 1.

Now ψF 0 has level denoted by k which is equal to the valuation of the different DF 0/F , k namely DF 0/F = ($F 0 ) . Let e be the ramification index and f = d/e. 0 0× × Case I: f is even. Then E ⊂ F is a subfield. In particular, NF 0/F F ⊂ NE/F E . In this case, LHS is equal to 1 as ηE0/F 0 is trivial. As we may choose a representative of δF 0/F in d(d−1)/2 × × (−1) NK/F K ⊂ NE/F E , we see that ηE/F (δF 0/F ) = 1 k f Case II: f is odd. Then (ηE/F , 1/2, ψF ) = (−1) . Note that NF 0/F ($F 0 ) = ($F ) . So we kf have NF 0/F DF 0/F = ($F ) . Since f is even, kf and k have the same parity. As the valuation of NF 0/F DF 0/F has the same parity as the valuation of δF 0/F , we see that ηE/F (δF 0/F ) = (−1)kf = (−1)k as desired. Lemma 4.15. If E/F is archimedean, then the theorem holds. Proof. The case F = C is obvious. Now we assume that F = R and E = C. Then the only non-trivial case we need to consider is when F 0 = C. Then the LHS is equal to one. For the RHS, we have 2 λE/F (ψ) = ηE/F (−1) = −1. And ηE/F (DF 0/F ) = −1. This completes the proof.

58 We now treat the general case for E/F . We use a global argument. To do so we want to globalize the quadratic extension E/F . We use the following lemma from [12]. Lemma 4.16. Let E/F be a quadratic extension of non-archimedean local fields. Then there exists a totally real number field F with Fv0 as its completion at a place v0of F, and a quadratic totally imaginary extension E of F with corresponding completion E such that E is unramified over F at all finite places different from v0. Resulting from the fact that the global epsilon factor satisfies

(IndE/F 1E , s, ψF ) = (1E , s, ψE ), we have a product formula Y λEv/Fv (ψv) = 1, v where Ev is a product of ﬁeld extensions of Fv and v runs over all places of F. Also choose 0 0 0 an extension F of F such that v0 is inert and Fv0 ' F . Such choice obviously exists. Then it is clear that the theorem holds for all Ev/Fv at those v 6= v0. By the global identity and Q 0 v ηEv/Fv (δF /F ) = 1, we immediate get the desired identity at the place v0. This completes the proof of Theorem 4.13.

4.4 All Fourier transforms preserve transfer Now we need to consider simultaneously the linear and the unitary case. Let E/F be a fixed 0 n quadratic field extension. We set up some notations. We will use V to denote gln(F ) × F × Fn. There are two isomorphism classes of hermitian spaces which we will denote by W1,W2 respectively. Then we let Vi denote U(Wi) × Wi for i = 1, 2. 0 Note that we have an obvious way to match the partial Fourier transforms on V and Vi. Recall that all Haar measures to define Fourier transform are chosen to be self-dual.

Theorem 4.17. For any a ﬁxed Fourier transforms F, there exists a constant ν ∈ µ4 depend- ∞ ing only on n, ψ, E/F and the Fourier transform F with the following property: if f ∈ Cc (V) ∞ and the pair (fi ∈ Cc (Vi))i=1,2 are transfer of each other, so are νF(f) and (F(fi))i=1,2.

We now let Fa (Fb, Fc, resp.) be the Fourier transform with respect to the total space (the subspace U(W ), W , resp.). Consider the following three statements:

• An: For all E/F , ψ, and Wi of dimension n, there is a constant ν ∈ µ4 with the property: ∞ 0 ∞ if f ∈ Cc (V ) and fi ∈ Cc (Vi)(i = 1, 2) match, then so do νFa(f) and Fa(fi).

• Bn: For all E/F , ψ, and Wi of dimension n, there is a constant ν ∈ µ4 with the property: ∞ 0 ∞ if f ∈ Cc (V ) and fi ∈ Cc (Vi)(i = 1, 2) match, then so do νFb(f) and Fb(fi).

∞ 0 ∞ • Cn: For all E/F , ψ, and Wi of dimension n, if f ∈ Cc (V ) and fi ∈ Cc (Vi)(i = 1, 2) −n match, then so do λE/F (ψ) Fc(f) and Fc(fi).

59 An−1 ⇒ Bn. In this part, we will use fb to denote the Fourier transforms with respect to n gln and u(W ). We first consider the linear side. We let W = Fn × F (we use W as it is the counterpart in the general linear case of the Hermitian space W ) and consider it as an n F × F -module of rank n with the hermitian form (w, w) = uv if w = (u, v) ∈ Fn × F . ∞ 0 By the local trace formula, for w ∈ W with (w, w) 6= 0, we have for any f ∈ Cc (V ) and ∞ g ∈ Cc (gln(F )): Z Z η η (4.3) OX,w(f)ω(X, w)g(X)dX = OX,w(fb)ω(X, w)ˇg(X)dX. gln(F ) gln(F ) ⊥ Let (w) be the orthogonal complement of (F × F )w in W . Up to the H = GLn,F -action, we may choose w of the form (e, de0) where e = (0, ..., 0, 1) and d ∈ F × is the hermitian norm (w, w). Then the stabilizer of w in H can be identified with GLn−1,F (with the embedding η η η into GLn,F as before). If h ∈ GLn−1(F ), we have OX,w(f) = OXh,wh (f) = OXh,w(f). Then we may rewrite the integral in LHS of the local trace formula (Theorem 4.6) as Z Z Z η η h h OX,w(f)ω(X, w)g(X)dX = OX,w(f) ω(X , w)g(X )dh dq(X), gln(F ) Q(F ) GLn−1(F ) where Qn−1 = gln//GLn−1, q is the quotient morphism and the measure is a suitable one on ∞ Q(F ) such that for all g ∈ Cc (gln(F )): Z Z Z g(X)dX = g(Xh)dh dq(X). gln(F ) Q(F ) GLn−1(F ) h Note that ω(X , w) = η(h)ω(X, w) for h ∈ GLn−1(F ). Now it is easy to see that when we restrict the transfer factor ω(X, w) to X ∈ gln, it is a constant multiple of the transfer factor we have used to define the GLn−1,F -orbital integral. Moreover, this constant depends only on w and is denoted by cw. So we have Z Z h h h η ω(X , w)g(X )dh = ω(X, w) g(X )η(h)dh = cwOX (g). GLn−1(F ) GLn−1(F )

Then this depends only on the GLn−1(F )-orbit of X. Similar result holds for the RHS. The constant cw is then canceled. Replacing g by gb, we may rewrite the local trace formula as Z Z η η η η (4.4) OX,w(f)OX (gb)dq(X) = OX,w(fb)OX (g)dq(X). Q(F ) Q(F ) 0 Note that the Fourier transform here is Fb for GLn-action on V but it is the Fa for the GLn−1-action on gln. Now for the unitary case we also have a similar equality for i = 1, 2 (without the issue of transfer factors) Z Z OX,w(fi)OX (gbi)dq(X) = OX,w(fbi)OX (gi)dq(X). Q(F ) Q(F )

60 Here the stabilizer of w in U(W ) replaces GLn−1,F . Note that we have identiﬁed Q with Qi as before. Now suppose that f ↔ (fi). We want to show that for some constant ν:

η νO 0 0 (f) = O 0 0 (f )(4.5) X ,w b Xi ,wi bi

0 0 0 0 for any strongly regular semisimple (X , w ) ↔ (Xi , wi ). This would imply the equality for all matching regular semisimple elements by the local constancy of orbital integrals. We may choose g ↔ (gi) such that • Both are supported in the regular semisimple locus.

• There exists a small (open and compact) neighborhood ω of q(X0) ∈ Q(F ) with the η following property: (1) the functions on ω given by q(X) 7→ OX,w0 (fb) and q(Xi) 7→ η O 0 (f ) are constant; (2) the functions on Q(F ) given by q(X) 7→ O (g) and q(X ) 7→ Xi,wi bi X i OX (gi) are the characteristic function 1ω. This is clearly possible by Lemma 3.16. For such choice we have Z Z Oη (f)Oηi (g)dq(X) = Oη (f) dq(X) , i = 1, 2. X,w0 X b X0,w0 b Q i ω

Now by An−1, we have for some constant ν

νOη (g) = O (g ) X b Xi bi whenever X ↔ Xi. Now the desired equality (4.5) follows immediately for the same constant ν as in An−1. This shows that An−1 ⇒ Bn.

C1 ⇔ Cn It suﬃces to show C1 ⇒ Cn. Now we will use fb to denote the Fourier transform n with respect to Fn × F and W respectively. We want to show that if f and fi match, then for strongly regular (X, w) ↔ (Xi, wi), we have

n η (4.6) λE/F (ψ) OX,w(fb) = OXi,wi (fbi).

For X ∈ gln(F ) regular semisimple, let T be the centralizer of X in H and t its Lie algebra. Q × Pr Then T is isomorphic to j Fj for some ﬁeld extensions Fj/F of degree nj with j=1 bj = n. For Xi ∈ u(Wi) regular semisimple, let Ti be the centralizer of Xi in Hi and ti its Lie algebra. Let Ej = E ⊗F Fj. As X, Xi have the same characteristic polynomial, we know Q 1 that Ti is isomorphic to j ResFj /F Ej for the same tuple of ﬁeld extensions Fj/F . Here 1 × × E is the kernel (as an algebraic group) of norm homomorphism NE/F : E → F . Let 0 Q 0 0 0 F := i Fj,E := F ⊗F E. By [3, Sec.5,p.18], W is a rank one Hermitian space over E 0 0 0 with unitary group U(W, E /F ) ' T . We may identify F with the sub algebra F [X] ⊂ gln.

61 ∗ n For a more intrinsic exposition, we let M × M denote Fn × F and gln = End(M). We may describe the transfer factor as follows: u ∧ Xu ∧ X2u... ∧ Xn−1u ω(X, u, v) = η ,X ∈ End(M), u ∈ M, v ∈ M ∗, ω0

n where ω0 a ﬁxed generator of the line ∧F M. Obviously if we change the generator ω0, the transfer factor only changes by a constant in {±1}. Then under the action of F 0 = F [X], M is a free F 0-module of rank one. In this way, ∗ 0 M = HomF (M,F ) is canonically isomorphic to HomF 0 (M,F ). Indeed, we may deﬁne an 0 0 0 0 0 F -linear pairing (·, ·)F 0 : M × M → F such that for all λ ∈ F , x ∈ M, y ∈ M we have

trF 0/F (λx, y)F 0 = (λx, y)F .

1 0 Fixing a generator of ∧F 0 M ' M we deﬁne a transfer factor ω(w) ∈ {±1}. We also have 0 a compatibility ηE/F (NF 0/F x) = ηE0/F 0 (x) and NF 0/F x = det(x) when x ∈ F = F [X]. We then have an inversion formula as follows. Q Lemma 4.18. For a regular semisimple X ∈ gln(F ) with centralizer T ' j ResFj /F GL1 as η 0 above, let κb (w, w ) be the locally constant T × T -invariant function on Wrs × Wrs → C given by Corollary 4.7. Then we have Z η η η 0 0 ObX,w(f) = ηE/F (δF 0/F ) OX,w0 (f)κ (w, w )dq(w ), Q(F 0)

∗ where Q = (M × M )//ResF 0/F GL1.

∞ Proof. Without loss of generality, we may assume that f = φ ⊗ ϕ, φ ∈ Cc (gln(F )), ϕ ∈ ∞ ∗ Cc (M × M ). We now see that the orbital integral can be rewritten as Z η h η h OX,w(φ ⊗ ϕb) = ω(X, w)/ω(w) φ(X )η(h)Ow( ϕb)dh T \H Z h η h = ω(X, w)/ω(w) φ(X )η(h)Ow(cϕ)dh, T \H where we write hϕ(w) = ϕ(wh−1 ). By Corollary 4.7, we have Z η η η 0 0 (4.7) Ow(ϕb) = Ow0 (ϕ)κ (w, w )dq(w ). Q(F 0) Reversing the argument we obtain Z η 0 0 η η 0 0 OX,w(φ ⊗ ϕb) = ω(X, w)ω(w) ω(X, w )ω(w )OX,w0 (φ ⊗ ϕ)κ (w, w )dq(w ). Q(F 0)

62 We leave the reader to verify that

0 0 ω(X, w)ω(X, w ) = η(δX )ω(w)ω(w ) where δ(X) is the discriminant of the characteristic polynomial of X. In particular, η(δX ) = 0 0 η(δF 0/F ). Note that the product ω(X, w)ω(X, w )(ω(w)ω(w ), resp.) does not even depend n 1 on the choice of the generator of ∧F M (∧F 0 M = M, resp.).

Similarly we also have an inversion formula for the unitary case with a diﬀerent kernel 0 function κi(wi, wi). Finally we note that the kernel functions are given by the Kloosterman sums relative to E0/F 0. By Theorem 4.12, we have

η 0 0 κ (w, w ) = λE0/F 0 (ψF 0 )κi(wi, wi)

0 0 whenever w, w match wi, wi, i = 1, 2. Now the proof of C1 ⇒ Cn follows from the inversion formulae and the base change property (Theorem 4.13): n λE0/F 0 (ψF 0 ) = λE/F (ψF )η(δF 0/F ).

Remark 15. It is easy to see that for ﬁxed X,Xi, the statement C1 implies that up to a constant multiple the partial Fourier transform have matching orbital integrals for those elements with ﬁrst components X,Xi. The lengthy computation of the Davenport–Hasse relations is to show that this constant, a prior depending on X,Xi, is indeed independent of the choice of X,Xi.

Bn + Cn ⇒ An This is obvious since Fa = FbFc.

4.5 Completion of the proof of Theorem 2.6 The following homogeneity result enables us to deduce the existence of transfer from the compatibility with Fourier transform. In §3.1, we have reduced the existence of transfer on groups to the Lie algebra version, namely it suﬃces to show Conjecture Sn+1. Obviously, the statement is equivalent to the corresponding assertion on the following subspaces:

n sln × Fn × F and for Hermitian W su(W ) × W.

The reason is that the group GLn (U(W ), resp.) acts trivially on the orthogonal complement. n We let V be either sln(F ) × F × Fn or su(W ) × W and let η be the quadratic character associated to E/F in the former case and the trivial character in the latter. Theorem 4.19 (Aizenbud-Gourevitch). There is no distribution T on V with the following property

63 (1) T is (H, η)-invariant (hence so is Tˆ).

(2) T , TbV , TbW , Tbu(W ) are all supported in the nilpotent cone N . More precisely, it is proved in [1, Theorem 6.2.1] for the case η = 1. But the same proof goes through for the nontrivial quadratic η.

Corollary 4.20. Let C0 = ∩T Ker(T ) ∞ where T runs over all (H, η)-invariant distributions. Then we have Cc (V) is the sum of C0 ∞ ∞ and the image of all Fourier transforms of Cc (V − N ). Equivalently, any f ∈ Cc (V) can be written as V W u(W ) f = f0 + f1 + fb2 + fb3 + fb4 ∞ where f0 ∈ C0, fi ∈ Cc (V − N ), i = 1, 2, 3.

Proof. Let C be the subspace spanned by C0 and the image of all Fourier transforms of ∞ ∞ Cc (V−N ). If the quotient L = Cc (V)/C is not trivial, then there must exist a nontrivial linear functional on L. This induces a distribution T on V. As T is zero on C0, T is (H, η)-invariant. ∞ V As T is zero on Cc (V − N ), it is supported on N . Similarly, all Fourier transforms Tb etc. are all (H, η)-invariant and supported on N . This contradicts Theorem 4.19 above. Finally we may prove Theorem 2.6:

Proof. To abuse notations, we still use Q = A2n to denote the categorical quotient of V by H. By the localization principle Prop. 3.8, it is enough to prove the existence of local transfers at all z ∈ Q(F ). We will prove this by induction on n. If z ∈ Q(F ) is non-zero, the stabilizer of z is strictly smaller than H. By Prop. 3.20, the local transfer around z is implied by the local transfer around 0 for the sliced representations. This representations are (possibly a product) of the same type with smaller dimension (with possibly a base change of the base ﬁeld F ). Then by induction hypothesis, we may assume the existence of local transfers at all non-zero semisimple z ∈ Q(F ). Therefore, by induction hypothesis, we have the existence of smooth transfer for functions supported away from the nilpotent cone. Now by Corollary 4.20, it suﬃces to show the existence of smooth transfer for fbfor every one of the three Fourier transforms and f supported away from the nilpotent cone.. But this follows from Theorem 4.17. We thus complete the proof.

64 A Appendix: Spherical characters for a strongly tempered pair By Atsushi Ichino, Wei Zhang

Let F be a non-archimedean local field. We consider a pair (G, H) where G is a reductive group and H is a subgroup. We will also denote by G, H the sets of F -points. We will assume that H a spherical subgroup in the sense that X := G/H with the G-action from left is a spherical variety. Following [47, sec. 6], we say that the pair (G, H) is strongly tempered if for any tempered unitary representation π of G, and any u, v ∈ π, the associated matrix coefficient φu,v(g) := hπ(g)u, vi (h·, ·i is the hermitian G-invaraint inner product) satisfies 1 φu,v|H ∈ L (H). To check whether a pair (G, H) is strongly tempered, one uses the Harish-Chandra function Ξ. Let π0 be the normalized induction of the trivial representation of a Borel B of G and let v0 be the unique spherical vector such that hv0, v0i = 1. Then Ξ is the matrix coefficient:

Ξ(g) = hgv0, v0i. 1 We have Ξ(g) ≥ 0 for any g ∈ G. Then (G, H) is strongly tempered if Ξ|H ∈ L (H). The major examples are those appeared in the Gan–Gross–Prasad conjecture: • Let V be an orthogonal space of dimension n + 1 and W a codimension one subspace (both non-degenerate). Let SO(V ) and SO(W ) the corresponding special orthogonal groups. Let G = SO(W )×SO(V ) and H ⊂ G is the graph of the embedding SO(W ) ,→ SO(V ) induced by W,→ V .

• G = GLn × GLn+1, H is the graph of the embedding of GLn ,→ GLn+1 given by g 7→ diag(g, 1). • Let E/F be a quadratic extension of fields. Let V be a hermitian space of dimension n + 1 and W a codimension one subspace (both non-degenerate). Let U(V ) and U(W ) the corresponding unitary groups. Let G = U(W ) × U(V ) and H ⊂ G is the graph of the embedding U(W ) ,→ U(V ) induced by W,→ V . A proof of the fact that these pair (G, H) are strongly tempered can be found in [28] for the orthogonal case and [23] for the linear and unitary cases. Assume that (G, H) are strongly tempered. Then for any tempered representation π of G, following [28] we may define a matrix coefficient integration Z ν(u, v) := hπ(h)u, vidh. H Obviously ν ∈ HomH×H (π ⊗ π,e C). The integral is absolutely convergent by the strong temperedness. The following is a conjecture of Ichino–Ikeda in their refinement of the Gan–Gross–Prasad conjecture.

65 Theorem A.1 (Sakellaridis–Venkatesh,[47]). Assume that (G, H) is strongly tempered and X = G/H is wavefront. Let π be an irreducible representation. Assume that π is a discrete G series representation or π = IndP (σ) for a discrete series representation σ of the Levi of a parabolic subgroup P . Then HomH (π, C) 6= 0 if and only if ν does not vanish identically. The result is also proved in the orthogonal case by Waldspurger. Let π be a representation as in the Theorem above. We further assume that

dim HomH (π, C) ≤ 1. All the examples we list earlier satisﬁes this condition. Let ` ∈ HomH (π, C). Then we deﬁne a spherical character associated to ` to be a distribution on G such that

X ∞ θπ,`(f) := `(π(f)v)`(v), f ∈ Cc (G), v∈B(π) where B(π) is an orthonormal basis of π. The distribution is bi-H-invariant for the left and right translation by H. Obviously the distribution θπ,` is non-zero if and only if the linear functional ` is non-zero. This notes is to prove the following:

Theorem A.2. Assume that ` 6= 0. Fix any open dense subset Gr of G. Then the restriction ∞ of the distribution θπ,` to Gr is non-zero. Equivalently, there exists f ∈ Cc (Gr) such that θπ,`(f) 6= 0.

By Theorem of Sakellaridis–Venkatesh above, there exist v0 ∈ π such that Z `(v) = hπ(h)v, v0idh, v ∈ π. H

∞ Lemma A.3. For all f ∈ Cc (G), we have Z Z Z θπ,`(f) = f(g)Φ(h2gh1)dg dh1dh2, H H G where Φ(g) = hπ(h)v0, v0i. Proof. The proof is analogous to that of [40, Thm. 6.1]. We may rewrite θ as X θπ,`(f) = `(π(f)v)`(v) v∈B(π) Z Z X ∗ −1 −1 = hv, π(f )π(h )v0idh hπ(h )v0, vidh. v∈B(π) H H

66 P By ϕ = v∈B(π)hϕ, viv for any ϕ ∈ π, we have Z Z X −1 ∗ −1 θπ,`(f) = hπ(h1 )v0, π(f )π(h2 )v0idh1dh2 v∈B(π) H H Z Z Z = f(g)hπ(h2gh1)v0, v0idg dh1dh2. H H G

We consider the orbital integral of the matrix coeﬃcient Φ as in Lemma A.3: Z Z O(g, Φ) = Φ(h1gh2)dh1dh2, H H as well as the orbital integral O(g, Ξ) of the Harish-Chandra function Ξ. 1 ∞ Lemma A.4. The function g 7→ O(g, Ξ) on G is locally L . Equivalently, for any f ∈ Cc (G), the following integral is absolutely convergent Z |f(g)O(g, Ξ)|dg < ∞. G Proof. Consider a special maximal compact open subgroup K of G such that we have the following relation for a suitable measure on K: Z (A.1) Ξ(gkg0)dk = Ξ(g)Ξ(g0). K Such K exists ([46]). Without loss of generality, we may assume that f is the characteristic function of KgK for some g ∈ G. We then have Z Z Z Z Z |f(g)O(g, Ξ)|dg = Ξ(h1k1gk2h2)dh1dh2dk1dk2. G K K H H By (A.1) this is equal to Z Z Z Ξ(h1k1g)dh1dk1 Ξ(h2)dh2 . K H H By (A.1) again, we obtain

Z Z Ξ(g) Ξ(h1)dh1 Ξ(h2)dh2 < ∞. H H By Fubini theorem, this shows Z |f(g)O(g, Ξ)|dg < ∞. G

67 Lemma A.5. Let Φ be the matrix coeﬃcient as in Lemma A.3. The function g 7→ O(g, Φ) 1 ∞ on G is locally L and for any f ∈ Cc (G), we have Z θπ,`(f) = f(g)O(g, Φ)dg. G

Proof. For any tempered representation π of G and a matrix coeﬃcient φu,v associated to u, v ∈ π, we have |φu,v(g)| ≤ cΞ(g), g ∈ G for some constant c > 0. This shows that O(·, Φ) ∈ L1(G). Choose an increasing sequences of open compact subsets of H:

Ω1 ⊂ Ω2 ⊂ ... ⊂ Ωn ⊂ ... ⊂ H, ∪nΩn = H.

Now we deﬁne a partial orbital integral Z Z On(g, Φ) := Φ(h1gh2)dh1dh2. Ωn Ωn

It is clearly that On(·, Φ) converges to O(·, Φ) point-wisely. They are bounded point-wisely

|On(g, Φ)| ≤ O(g, Ξ), g ∈ G.

∞ 1 Fix f ∈ Cc (G). By Lemma A.4, the function g 7→ f(g)O(g, Ξ) is in L (G). By Lebesgue’s dominated convergence theorem, Z Z lim f(g)On(g, Φ)dg = f(g)O(g, Φ)dg. n→∞ G G On the other hand Z Z Z Z f(g)On(g, Φ)dg = f(g) Φ(h1gh2)dh1dh2 dg G G Ωn Ωn Z Z Z = f(g)Φ(h1gh2)dgdh1dh2. Ωn Ωn G as Ωn is compact as well as the support of f. This shows that Z Z Z Z lim f(g)On(g, Φ)dg = f(g)Φ(h2gh1)dg dh1dh2. n→∞ G H H G

This is equal to θπ,`(f) by Lemma A.3.

68 ∞ Now we prove Theorem A.2. Since θπ,` is non-zero, there exists f ∈ Cc (G) such that

θπ,`(f) 6= 0.

Equivalently, by Lemma A.5, Z f(g)O(g, Φ)dg 6= 0. G ∞ As Gr is open and dense, we may choose fn ∈ Cc (Gr), such that point-wisely on Gr we have

lim fn = f. n→∞

Without loss of generality, we may assume that f ≥ 0 point-wisely and f −fn ≥ 0 point-wisely. Then we have |(f(g) − fn(g))O(g, Φ)| ≤ 2f(g)|O(g, Φ)| which is integrable on G. By Lebesgue’s dominated convergence theorem, we have Z lim (f(g) − fn(g))O(g, Φ)dg = 0. n→∞ G R Since G f(g)O(g, Φ)dg 6= 0., we have for n large enough, Z fn(g)O(g, Φ)dg 6= 0, G or equivalently, θπ,`(fn) 6= 0. ∞ As fn ∈ Cc (Gr), this completes the proof. Remark 16. When comparing with the property of the usual character as established by Harish-Chandra, some questions remain for the spherical characters in this notes. For example, is the function g 7→ O(g, Φ) continuous or even locally constant on an open dense subset? If π is super-cuspidal, one may prove that the function g 7→ O(g, Φ) is locally constant on some open dense subset in the unitary case listed in the beginning.

References

[1] Aizenbud, Avraham. A partial analog of the integrability theorem for distributions on p-adic spaces and applications, to appear in Israel Journal of Mathematics

[2] Aizenbud, Avraham; Gourevitch, Dmitry. Generalized Harish-Chandra descent, Gelfand pairs, and an Archimedean analog of Jacquet-Rallis’s theorem. With an appendix by the authors and Eitan Sayag. Duke Math. J. 149 (2009), no. 3, 509–567.

69 [3] Aizenbud, Gourevitch, Rallis, Schiﬀmann, Multiplicity one theorems, Ann. of Math. (2) 172 (2010), no. 2, 1407–1434.

[4] Bushnell, Colin J.; Kutzko, Philip C. The admissible dual of GL(N) via compact open subgroups. Annals of Mathematics Studies, 129. Princeton University Press, Princeton, NJ, 1993.

[5] J. H. Bruinier and K. Ono, Heegner divisors, L-functions and harmonic weak Maass forms, Annals of Mathematics, 172 (2010), pp.2135-2181

[6] Jacquet, Herv´e;Chen, Nan. Positivity of quadratic base change L-functions. Bull. Soc. Math. France 129 (2001), no. 1, 3390.

[7] Denef, J. The rationality of the Poincar series associated to the p-adic points on a variety. Invent. Math. 77 (1984), no. 1, 1–23.

[8] Flicker, Yuval Z. Twisted tensors and Euler products, Bulletin de la Societe Matematique de France 116 (1988), 295313.

[9] Flicker, Yuval Z.On distinguished representations. J. Reine Angew. Math. 418 (1991), 139–172.

[10] Y. Flicker and D. Zinoniev,On the poles of twisted L-functions, Proceedings of the Japan Academy of Science 71 (1995), 114116.

[11] Wee Teck Gan, Benedict H. Gross and Dipendra Prasad. Symplectic local root numbers, central critical L-values, and restriction problems in the representation theory of classical groups Preprint, http://www.math.harvard.edu/∼gross/preprints/

[12] Wee Teck Gan, Benedict H. Gross and Dipendra Prasad. Restric- tions of representations of classical groups: Examples Preprint, http://www.math.harvard.edu/∼gross/preprints/

[13] P. Garrett, Decomposition of Eisenstein series: Rankin triple products. Ann. of Math. (2) 125 (1987), no. 2, 209–235.

[14] Gelbart, Stephen; Jacquet, Herv; Rogawski, Jonathan. Generic representations for the unitary group in three variables. Israel J. Math. 126 (2001), 173237.

[15] Ginzburg, David; Jiang, Dihua; Rallis, Stephen On the nonvanishing of the central value of the Rankin-Selberg L-functions. J. Amer. Math. Soc. 17 (2004), no. 3, 679C722

[16] Ginzburg, David; Jiang, Dihua; Rallis, Stephen On the nonvanishing of the central value of the Rankin-Selberg L-functions. II. Automorphic representations, L-functions and applications: progress and prospects, 157C191

70 [17] B. Gross and S. Kudla, Heights and the central critical values of triple product L- functions. Compositio Math. 81 (1992), no. 2, 143C209.

[18] Gross, Benedict H.; Prasad, Dipendra On the decomposition of a representation of SOn when restricted to SOn−1. Canad. J. Math. 44 (1992), no. 5, 974–1002

[19] Gross, Benedict H.; Prasad, Dipendra On irreducible representations of SO2n+1 × SO2m. Canad. J. Math. 46 (1994), no. 5, 930–950. [20] B. Gross; D. Zagier: Heegner points and derivatives of L-series. Invent. Math. 84 (1986), no. 2, 225–320. [21] M. Harris and S. Kudla, On a conjecture of Jacquet. Contributions to automorphic forms, geometry, and number theory, 355–371, Johns Hopkins Univ. Press, Baltimore, MD, 2004. [22] Harris, Michael; Labesse, Jean-Pierre. Conditional base change for unitary groups. Asian J. Math. 8 (2004), no. 4, 653683. [23] Neal Harris, A refined Gross–Prasad conjecture for unitary groups, PhD Thesis, 2011, UCSD. [24] Harder, G.; Langlands, R. P.; Rapoport, M. Algebraische Zyklen auf Hilbert-Blumenthal- Flchen. J. Reine Angew. Math. 366 (1986), 53120. [25] Harish-Chandra, Harmonic analysis on reductive p-adic groups. Notes by G. van Dijk. Lecture Notes in Mathematics, Vol. 162. Springer-Verlag, Berlin-New York, 1970 [26] Harish-Chandra, Admissible invariant distributions on reductive p-adic groups. Lie the- ories and their applications (Proc. Ann. Sem. Canad. Math. Congr., Queen’s Univ., Kingston, Ont., 1977), pp. 281–347. Queen’s Papers in Pure Appl. Math., No. 48, Queen’s Univ., Kingston, Ont., 1978 [27] Ichino Atsushi. Trilinear forms and the central values of triple product L-functions. Duke Math. J [28] A. Ichino and T. Ikeda. On the periods of automorphic forms on special orthogonal groups and the Gross-Prasad conjecture, Geom. Funct. Anal. 19 (2010), no. 5, 1378–1425. [29] Igusa, Jun-ichi. Geometry of absolutely admissible representations. Number theory, algebraic geometry and commutative algebra, in honor of Yasuo Akizuki, pp. 373C452. Kinokuniya, Tokyo, 1973. [30] HervéJacquet, Sur un résultatde Waldspurger, Ann. Sci. Ec.´ Norm. Supér.,IV. 19 (1986), no. 2, 185-229 [31] Hervé,Jacquet Sur un rsultat de Waldspurger. II. Compositio Math. 63 (1987), no. 3, 315–389.

71 [32] Jacquet, Hervé Smooth transfer of Kloosterman integrals. Duke Math. J. 120 (2003), no. 1, 121152. [33] Jacquet, Hervé Kloosterman identities over a quadratic extension. Ann. of Math. (2) 160 (2004), no. 2, 755779. [34] Jacquet, H.; Piatetskii-Shapiro, I. I.; Shalika, J. A. Rankin- Selberg convolutions. Amer. J. Math. 105 (1983), no. 2, 367–464. [35] HervéJacquet, S. Rallis. On the Gross-Prasad conjecture for unitary groups. in On certain L-functions, 205264, Clay Math. Proc., 13, Amer. Math. Soc., Providence, RI, 2011. [36] Kottwitz, Robert E. Harmonic analysis on reductive p-adic groups and Lie algebras. Harmonic analysis, the trace formula, and Shimura varieties, 393522, Clay Math. Proc., 4, Amer. Math. Soc., Providence, RI, 2005. [37] S. S. Kudla, M. Rapoport, and T. Yang, Modular forms and special cycles on Shimura curves. Annals of Mathematics Studies, 161. Princeton University Press, Princeton, NJ, 2006. x+373 pp. [38] Langlands, R.; Shelstad, D. Descent for transfer factors. The Grothendieck Festschrift, Vol. II, 485563, Progr. Math., 87, Birkhuser Boston, Boston, MA, 1990. [39] Mumford, D.; Fogarty, J.; Kirwan, F. Geometric invariant theory. Third edition. Ergeb- nisse der Mathematik und ihrer Grenzgebiete (2) , 34. Springer-Verlag, Berlin, 1994. xiv+292 pp. ISBN: 3-540-56963-4 . [40] Murnaghan, Fiona. Spherical characters: the supercuspidal case. Group representations, ergodic theory, and mathematical physics: a tribute to George W. Mackey, 301–313, Contemp. Math., 449, Amer. Math. Soc., Providence, RI, 2008. [41] Prasad, Dipendra.Relating invariant linear form and local epsilon factors via global meth- ods. With an appendix by Hiroshi Saito. Duke Math. J. 138 (2007), no. 2, 233261. [42] Popov, V. L.;Vinberg,B.. Invariant theory. [43] Rader, Cary; Rallis, Steve. Spherical characters on p-adic symmetric spaces. Amer. J. Math. 118 (1996), no. 1, 91–178. [44] Steve Rallis, Grard Schiffmann. Multiplicity one Conjectures. arXiv:0705.2168 [45] Dinakar Ramakrishnan. A Mild Tchebotarev theorem for GL(n), preprint (2010). [46] Silberger, Allan J. Introduction to harmonic analysis on reductive p-adic groups. Based on lectures by Harish-Chandra at the Institute for Advanced Study, 19711973. Mathematical Notes, 23. Princeton University Press, Princeton, N.J.; University of Tokyo Press, Tokyo, 1979. iv+371 pp.

72 [47] Y.Sakellaridis, A. Venkatesh. Periods and harmonic analysis on spherical varieties, preprint

[48] Sun B., Zhu C.Multiplicity one theorems: the Archimedean case, Annals Math., to appear.

[49] J. Waldspurger: Sur les valeurs de certaines fonctions L automorphes en leur centre de sym´etrie. Compositio Math. 54 (1985), no. 2, 173–242.

[50] Waldspurger, J.-L. Une formule des traces locale pour les algbres de Lie p-adiques. J. Reine Angew. Math. 465 (1995), 4199

[51] Waldspurger, J.-L. Le lemme fondamental implique le transfert. Compositio Math. 105 (1997), no. 2, 153–236.

[52] J. Waldspurger: Une formule intgrale relie la conjecture locale de Gross-Prasad. arXiv:0902.1875.

[53] J. Waldspurger: Une formule intgrale relie la conjecture locale de Gross-Prasad, 2me partie: extension aux reprsentations tempres . arXiv:0904.0314.

[54] Xinyi Yuan, Shou-Wu Zhang, Wei Zhang. Gross–Zagier formula on Shimura curves, to appear in Ann. Math. Stud..

[55] Zhiwei Yun: The fundamental lemma of Jacquet–Rallis in positive characteristics, Duke Math. J. 156 (2011), No.2, 167-228.

[56] S. Zhang: Linear forms, algebraic cycles, and derivatives of L-series. preprint.

[57] Wei Zhang: On arithmetic fundamental lemmas. Invent. Math., DOI: 10.1007/s00222- 011-0348-1.

Wei Zhang: Department of Mathematics, Columbia University, 2990 Broadway New York, NY 10027 USA Email address: [email protected].