Endoscopy for Unitary Symmetric Spaces 11

Home , Split Lie algebra

arXiv:1910.09685v2 [math.RT] 6 Apr 2020 efruaetesot rnfrcnetr n tt h fu the state and conjecture transfer smooth the formulate we n rv h udmna em o h ymti ar( pair symmetric the for lemma fundamental tran the smooth prove of and existence the establish We non-archimedean. xlctcmuain nasbeun ae [ paper subsequent a In computation. explicit aeteebdigo ntr groups unitary of embedding the have emta pcsover spaces Hermitian ntr ru frank of group unitary rnfrfcosfrtesmercsaecniee vran over considered space symmetric the for factors transfer soitdt ymti ar ftefr ( form the of pairs symmetric to associated omle eid fatmrhcfrs udmna lemma fundamental forms, automorphic of periods formulae, em nfl eeaiyvaadffrn ehiu.Aogothe Among computation. this technique. for different groundwork a necessary via the generality full in lemma fields, pedxA omn nTorsors on Comment References A. Appendix u rsn i st eeo nntsmlter frelati of theory infinitesimal a develop to is aim present Our hsppriiitsapormwt h olo tblzn t stabilizing of goal the with program a initiates paper This hswr a atal upre ya M-iosTae Aw Travel AMS-Simons an by supported partially was work This phrases. and words Key 2010 Date Let motivation. and setup the describe us Let ( for lemma fundamental relative The space symmetric endoscopy 5. the Relative of algebra Lie 4. The 3. Endoscopy 2. Introduction 1. A pi ,2020. 8, April : ahmtc ujc Classification. Subject Mathematics cpcdt n rnfrfcosfrtesmercpi ( pair symmetric the for factors transfer and data scopic Abstract. h udmna em o h ymti ar( conjectu t pair transfer many symmetric the for smooth for transfer the lemma smooth formulate fundamental of the existence then the We establish lemma, field. local a E NOCP O NTR YMTI SPACES SYMMETRIC UNITARY FOR ENDOSCOPY and A F oiae ygoa plctos epooeater frel of theory a propose we applications, global by Motivated h soitdrnso dls Let adeles. of rings associated the n E hsi h rteapeo eaieter fendoscopy. of theory relative a of example first the is This . e od:Edsoy ymti n peia aite,r varieties, spherical and symmetric Endoscopy, words: Key h ietsum direct The . U ( W 1 ) × 1. PNE LESLIE SPENCER U Introduction rmr 17;Scnay1F5 11F85. 11F55, Secondary 11F70; Primary Contents ( W 2 ) 1 U W ֒ → U (2 1 ( U n V ⊕ Les19a E/F 4 ) U ( U , ) W W (4) U , . 2 1 ( U , n ( ⊕ sas emta pc n we and space Hermitian a also is W eaqartcetnino global of extension quadratic a be V ,w sals hsfundamental this establish we ], ) (2) 2 W 1 × r n yNFgatDMS-1902865. grant NSF by and ard ) U and 2 × × U oa field local y ) (2 dmna em when lemma ndamental s ucin,adprove and functions, est frfrmn etfunctions, test many for sfer . U ( U n U n (2)). ) ( W ) where )), eadfundamental and re erltv rc formula trace relative he U , (4) V hns hspprlays paper this things, r eedsoi aaand data endoscopic ve 2 2 ( )16 )) U , n etwo be ) × (2) tv endo- ative U F ( × U nparticular, In . n d )over )) ( U n dimensional 2)vaan via (2)) eoe a denotes ) ltv trace elative F 26 25 13 is 9 6 1 2 SPENCER LESLIE

Let π be an irreducible cuspidal automorphic representation of U(W1 ⊕W2)AF . Roughly, π is said to be distinguished by the subgroup U(W1) × U(W2) if the period integral

ϕ(h)dh (1) Z[U(W1)×U(W2)] is not equal to zero for some vector ϕ ∈ π. Here, [H]= H(F )\H(AF ) for any F -group H. The study of distinction of automorphic representations with respect to certain subgroups is a large and active area of automorphic representation theory, but this particular case has appeared in the recent literature in several distinct ways. 1.1. Global Motivation. Motivated by the study of the arithmetic of specials cycles in unitary Shimura varieties, Wei Zhang outlined in his 2018 IAS lecture [Zha18] a comparison of a relative trace formula on GL(W1 ⊕ W2) with one on U(W1 ⊕ W2). On the linear group, we may consider period integrals over the subgroups GL(W1)×GL(W2) 1 and U(W1 ⊕ W2) ; on the unitary side, one considers periods of the form (1). Indeed, Chao Li and Wei Zhang have recently [LZ19] established the arithmetic fundamental lemma associated to this comparison, with applications to the global Kudla-Rapoport conjecture and arithmetic Siegel-Weil formula. This comparison fits into the general framework for a relative theory of quadratic base change proposed in [GW14], which in turn may be understood as a method of proving cases of the emerging relative Langlands program. For this comparison to be effective for global applications, several results are needed. To begin, one needs the fundamental lemma and smooth transfer to establish the pre- liminary comparison. This is already problematic as the unitary relative trace formula is not stable: when we consider the action of U(W1) × U(W2) on the symmetric variety U(W1 ⊕ W2)/U(W1) × U(W2), invariant polynomials distinguish only geometric orbits. Analogously to the Arthur–Selberg trace formula, one must “stabilize” the orbital integrals arising in the unitary relative trace formula in order to affect a comparison between the two formulas. The present paper lays the local foundations for this stabilization procedure. Before describing our results, we note that one implication of the conjectured comparison was recently proved by Pollack, Wan, and Zydor [PWZ19] under certain local assumptions via a different technique. 1.2. Relative endoscopy. We now let F be a local field and let E/F be a quadratic field extension. As above, if W1 and W2 are d-dimensional Hermitian vector spaces, then W = W1 ⊕ W2 is a 2d-dimensional Hermitian space with a distinguished involutive linear map: σ(w1 + w2) = w1 − w2 for wi ∈ Wi. This induces an involution on the σ unitary group U(W ) with the fixed point subgroup U(W ) = U(W1) × U(W2). Letting u(W ) denote the Lie algebra of U(W ), the differential of σ induces a Z/2Z-grading

u(W )= u(W )0 ⊕ u(W )1, i where u(W )i is the (−1) -eigenspace of σ. Then the fixed-point group U(W1) × U(W2) acts via restriction of the adjoint action on u(W )1. The pair (U(W1)×U(W2), u(W )1) is called an infinitesimal symmetric pair since u(W )1 is the tangent space to the symmetric space Qd = U(W )/(U(W1) × U(W2)) at the distinguished U(W1) × U(W2)- fixed point. In this paper, we develop a theory of elliptic endoscopy for the pair (U(W1) × U(W2), u(W )1), postponing the theory for the symmetric space Qd to a later paper.

1In fact, one should take a diﬀerent form of the unitary group. See [GW14] for the construction. ENDOSCOPYFORUNITARYSYMMETRICSPACES 3

We do this both as the inﬁnitesimal theory is simpler to state and also because the stabilization of the relative trace formula is expected to ultimately reduce to studying this case. This is partially motivated by the analogous reduction by Waldspurger [Wal95] and [Wal97a] in the case of the Arthur-Selberg trace formula. Recall (from [Rog90, Chapter 3], for example) that for the unitary group U(W ), an elliptic endoscopic datum is a triple (U(Va) × U(Vb),s,η) where Vn denotes a ﬁxed n-dimensional split Hermitian space2,

s ∈ Uˆ(W )=GL2d(C) is a semi-simple element, and

η : GLa(C) × GLb(C) → GL2d(C) induces an isomorphism between GLa(C) × GLb(C) and the connected component of the centralizer of s. Here a + b = 2d. Motivated by the work of Sakellaridis-Venkatesh [SV17] and Knop-Schalke [KS17] on the dual groups of spherical varieties, we consider the dual group of the symmetric variety Qd, ˆ ϕd ˆ Qd = Sp2d(C) −→ GL2d(C)= U(W ). While a full theory of relative endoscopy is not yet clear, we may na¨ıvely expect that elliptic endoscopic spherical varieties of Qd ought to correspond to spherical varieties of endoscopic groups of U(W ) that themselves correspond to centralizers semi-simple ˆ elements the dual group Qd = Sp2d(C). In particular, for each elliptic endoscopic subgroup Sp2a(C) × Sp2b(C) ⊂ Sp2d(C), we might expect a diagram

Sp2d(C) GL2d(C)

Sp2a(C) × Sp2b(C) GL2a(C) × GL2b(C), where the bottom horizontal arrow is the dual group embedding associated to the symmetric space Qa × Qb of the group U(V2a) × U(V2b). Happily, this hope is true once we take certain pure inner forms into account. It is not yet clear to us how to place the above heuristic with dual groups on a firm (and generalizable) footing. Our method for establishing the requisite matching of regular semi-simple orbits and effective definition of transfer factors instead reduces to the endoscopic theory of U(Vd) acting on its Lie algebra. The dual group interpretation outlined above only becomes clear a postiori. Recall that W = W1 ⊕W2 is our 2d-dimensional Hermitian space and let ξ = (H,s,η) be an elliptic endoscopic datum for U(W1), where H = U(Va) × U(Vb) with d = a + b. Fix representatives {α} and {β} of the isomorphism classes of Hermitian forms on the underlying vector spaces of Va and Vb. Then for each pair (α, β), we have the Lie algebras

u(Va ⊕ Vα) and u(Vb ⊕ Vβ), (2) where Vα simply denotes the Hermitian space (Va, α), and similarly for Vβ. Each equipped with a natural involution σα and σβ and the associated symmetric pairs

(U(Va) × U(Vα), u(Va ⊕ Vα)1) and (U(Vb) × U(Vβ), u(Vb ⊕ Vβ)1) are lower rank analogues of our initial symmetric pair (U(W1) × U(W2), u(W )1).

2We say Hermitian space V is split if there exists an isotropic subspace of maximal possible dimension and a self-dual lattice Λ ⊂ V . This implies that the associated unitary group is quasi-split. 4 SPENCER LESLIE

Deﬁnition 1.1. We say the quintuple

(ξ, α, β) = (U(Va) × U(Vb),s,η,α,β) is a relative elliptic endoscopic datum and the direct sum of the (−1)-eigenspaces (2) is an endoscopic infinitesimal symmetric pair for (U(W1) × U(W2), u(W )1). With this definition, we show how to match regular semi-simple orbits and define the transfer of orbits with appropriate transfer factors in Section 4. The key point is that we may relate the action of U(W1) × U(W2) on u(W )1 to the adjoint action of U(W1) on its twisted Lie algebra of Hermitian operators

Herm(W1)= {x ∈ End(W1) : hxu, vi = hu, xvi}.

This is Proposition 3.2, realizing Herm(W1) as the categorical quotient u(W )1//U(W2). Equipped with the above definition, we define smooth transfer of smooth compactly- supported test function as follows. Fix a regular semi-simple element x ∈ u(W )1 and let (ξ, α, β) be a relative endoscopic datum. This datum determines a character κ we use to define the relative κ-orbital integral ROκ(x,f)= κ(inv(x,x′)) RO(x′,f), ′ xX∼stx ′ where x ranges over the U(W1) × U(W2)-orbits in u(W )1 in the same stable orbit as x and the relative orbital integrals are as in Definition 9. When κ = 1, we write SRO = ROκ and call this the relative stable orbital integral. There is a good notion of when x matches the pair (xa,xb) ∈ u(Va⊕Vα)1⊕u(Vb⊕Vβ)1, and for such matching elements (xa,xb) and x, we define the relative transfer factor

∆rel((xa,xb),x) in Section 4. ∞ ∞ Deﬁnition 1.2. We say f ∈ Cc (u(W )1) and fα,β ∈ Cc (u(Va ⊕ Vα)1 ⊕ u(Vb ⊕ Vβ)1) are smooth transfers (or say that they match) if the following conditions are satisﬁed:

(1) For any matching orbits x and (xa,xb), we have an identify κ SRO((xa,xb),fα,β)=∆rel((xa,xb),x) RO (x,f). (3)

(2) If there does not exist x matching (xa,xb), then

SRO((xa,xb),fα,β) = 0. With this definition, we state Conjecture 4.4, asserting that smooth transfers exist for all smooth compactly-supported functions on u(W )1. As a first check for our definition, we prove this conjecture for test functions supported in a certain open dense subset of u(W )1 (see Proposition 4.5). ∞ Proposition 1.3. Let f ∈ Cc (u(W )1) and assume supp(f) is contained in the non- iso singular locus u(W )1 (see (8) below). Let (U(Va)×U(Vb),s,η,α,β) be a relative elliptic endoscopic datum. Then there exists ∞ fα,β ∈ Cc (u(Va ⊕ Vα)1 ⊕ u(Vb ⊕ Vβ)1) such that f and fα,β are smooth transfers of each other. Our proof of this proposition relies on the good behavior of the categorical quotient

u(W )1 −→ u(W )1//U(W2) =∼ Herm(W1) over the non-singular locus. This enables a reduction to the analogous setting of the twisted Lie algebra. The general conjecture will require other techniques. ENDOSCOPYFORUNITARYSYMMETRICSPACES 5

When E/F is an unramified extension of non-archimedean local fields, we also formulate the fundamental lemma for the “unit element.” More specifically, suppose that Vd = W1 = W2 is split, and let Λd ⊂ Vd be a self-dual lattice. In this case,

u(W )1 = HomE(Vd,Vd) = End(Vd), and the choice of self-dual lattice Λd gives a natural compact open subring End(Λd) ⊂

End(Vd). Let 1End(Λd) denote the indicator function for this subring. This also induces an integral model U(Λd) of U(V ). Setting U(Λd) ⊂ U(V ) as the OF -points, this gives a hyperspecial maximal compact subgroup. Now suppose that ξ = (U(Va) × U(Vb),s,η) is an endoscopic datum for Herm(Vd). ∼ Under our assumptions, we have Vd = Va ⊕Vb (see Lemma 4.6). We may assume further that Λd = Λa ⊕ Λb for ﬁxed self-dual lattices Λa ⊂ Va and Λb ⊂ Vb. There are only four possible pairs (α, β), and we set (α0, β0) to be the split pair. We equip these groups with Haar measures normalized so that the given hyperspecial maximal subgroups have volume 1.

Theorem 1.4. The function 1End(Λd) matches 1End(Λa) ⊗ 1End(Λb) if (α, β) = (α0, β0) and matches 0 otherwise. This is proved in [Les19a] via a combination of new local results on orbital integrals and a new global comparison or relative trace formulas. In Section 5, we verify this statement directly for (U(4),U(2)×U(2)) by reducing to a family of transfer statements on the twisted Lie algebra and explicitly computing all orbital integrals involved. We expect that this fundamental lemma implies the smooth transfer conjecture 4.4. This is analogous to the work of Waldspurger [Wal97a] for the Arthur-Selberg trace formula and Chong Zhang [Zha15] for the Guo-Jacquet relative trace formula. This reduction is work-in-progress by the author. The outline of the paper is as follows. In Section 2, we recall the necessary notions and details from the theory of endoscopy, focusing on the case of the unitary Lie algebra. In Section 3, we define the symmetric space under consideration and define the orbital integral to be studied. In Section 4, we define our proposal for a theory of relative endoscopy data in this setting and state the associated transfer conjecture and fundamental lemma. We prove the existence of transfer for many functions in Proposition 4.5. In Section 5, we end by proving this fundamental lemma for the case of (U(4),U(2) × U(2)) by an explicit computation.

1.3. Acknowledgements. We wish to thank Wei Zhang for sharing personal computations relating to the stable comparison outlined in [Zha18] (in particular, for the computation in Proposition 5.2) which led us to the conjectured notion of endoscopic spaces and for general advice and encouragement. We also thank Jayce Getz for suggesting we consider the notion of relative endoscopy, as well as for many useful conversations and invaluable advice. Finally, we thank the anonymous referee for several helpful comments.

1.4. Notation. Throughout we assume that E/F is a quadratic extension of local fields. When non-archimedean, we assume F has either odd or zero characteristic. Let O ⊂ F denote the ring of integers and let OE ⊂ E be its integral closure in E. We denote by val : E× → Z the unique extension of the normalized valuation on F . Thus, if ̟ is a uniformizer of F , then val(̟) = 1. We fix an algebraic closure F and a separable closure F sep of F and let Γ = Gal(F sep/F ) × × denote the Galois group. Denote by ωE/F : F → C the quadratic character associated to the extension E/F via local class field theory. Let Nm = NmE/F denote the norm map and set U(1) = ker(Nm). 6 SPENCER LESLIE

We only consider smooth affine algebraic varieties over F . We use boldface notation for an algebraic variety Y and use Roman font Y = Y(F ) for its F -points. This space is naturally endowed with a locally-compact topology. When F is non-archimedean, this topology makes Y an l-space (see [BZ76]), and we will consider the Schwartz space ∞ Cc (Y ) of locally-constant, compactly-supported C-valued functions. When (W, h·, ·i) is a Hermitian space over E, we denote by U(W ) := U(W, h·, ·i) the associated unitary group. We set Vd to be a fixed split Hermitian space of dimension d, so that U(Vd) is a fixed quasi-split unitary group of rank d. We also fix representatives {τ} of the isomorphism classes of Hermitian form on the underlying vector space Vd, and denote by Vτ the associated pure inner form with U(Vτ ) the unitary group. Any unitary group U(W ) acts on its Lie algebra u(W ) as well as its twisted Lie algebra Herm(W )= {x ∈ End(W ) : hxu, vi = hu, xvi} by the adjoint action. For any δ ∈Herm(W ), we denote by Tδ ⊂ U(W ) the centralizer.

2. Endoscopy In this section, we recall the necessary facts from the theory of endoscopy for unitary Lie algebras. We follow [Xia18], to which we refer the reader interested in proofs. Let W be a d dimensional Hermitian space over E. As previously noted, we will work with the twisted Lie algebra Herm(W )= {x ∈ End(W ) : hxu, vi = hu, xvi}. Let δ ∈Herm(W ) be regular and semi-simple. The theory of rational canonical forms m implies that there is a decomposition F [δ] := F [X]/(χδ(x)) = i=1 Fi, where Fi/F is a ﬁeld extension and χδ(x) is the characteristic polynomial of δ. Setting Ei = E ⊗F Fi, we have Q E[δ]= Ei = Ei × Fi ⊕ Fi, Yi iY∈S1 iY∈S2 where S1 = {i : Fi + E} and S2 = {i : Fi ⊇ E}.

Lemma 2.1. Let δ ∈Herm(W ) be regular semi-simple, let Tδ denote the centralizer of δ in U(W ). Then ∼ × × Tδ = ZU(W )(F )E[δ] /F [δ] , where Z (F ) denotes the center of U(W ). Moreover, H1(F, T )= /2 and U(W ) δ S1 Z Z Q C(T /F ) := ker H1(F, T ) → H1 (F,U(W )) = ker Z/2Z → Z/2Z , δ δ ab   S Y1 1   where Hab denotes abelianized cohomology in the sense of [Bor98] and the map on cohomology is the summation of the factors. Proof. This is proved, for example, in [Rog90, 3.4]. ∼ × × Set TS1 = ZU(W )(F ) i∈S1 Ei /Fi for the unique maximal compact subgroup of Tδ. We choose the unique measure dt on Tδ giving this subgroup volume 1. As we are studying theQ geometric theory of endoscopy (that is, κ-orbital integrals), it is natural to use abelianized cohomology as in the lemma. If we deﬁne 1 1 D(Tδ/F ) := ker H (F, Tδ) → H (F,U(W )) , it is well known that the set of rational conjugacy classes Ost(δ) in the stable conjugacy class of y are in natural bijection with D(Tδ/F ). This is given by the map ∼ inv(δ, −) : Ost(δ) −→D(Tδ/F ), (4) ENDOSCOPYFORUNITARYSYMMETRICSPACES 7 where [δ′] 7→ inv(δ, δ′) := [σ ∈ Gal(E/F ) 7→ g−1σ(g)], for any g ∈ GL(W ) such that δ′ = Ad(g)(δ). There is always an injective map

D(Tδ/F ) −→ C(Tδ/F ), 1 1 induced by the (surjective) map H (F,U(W )) → Hab(F,U(W )). This latter map is ∼ an isomorphism when F is a non-archimedean local field. In particular, D(Tδ/F ) = C(Tδ/F ) in this case. When F = R, the pointed set D(Tδ/F ) need not even be a group and the difference between the two is important. Subtle notions such as K-groups have been introduced to, among other things, understand spectral implications of this distinction. See [Lab11] for more details. 2.1. Endoscopy for unitary Lie algebras. The general construction of elliptic endoscopic data and matching of stable orbits is standard, though rather involved. See [Rog90] for a review. Happily, our present case allows for a simplified presentation, enabling us to avoid several technicalities. An elliptic endoscopic datum for Herm(W ) is the same as a datum for the group U(W ), namely a triple (U(Va) × U(Vb),s,η), where a + b = d. Here s ∈ Uˆ(W ) a semi-simple element of the Langlands dual group of U(W ), and an embedding

( η : Uˆ(Va) × Uˆ(Vb) ֒→ Uˆ(W identifying Uˆ(Va) × Uˆ(Vb) with the neutral component of the centralizer of s in the L-group LU(W ). Fixing such a datum, we consider the endoscopic Lie algebra

Herm(Va) ⊕Herm(Vb).

Let δ ∈Herm(W ) and (δa, δb) ∈Herm(Va) ⊕Herm(Vb) be regular semi-simple. Denote Wa,b = Va ⊕ Vb. In the non-archimedean case, the isomorphism class of Wa,b is uniquely determined by those of Va and Vb [Jac62a, Theorem 3.1.1]. For F = R, we can and do choose the form on Wa,b to have signature (n,n) or (n + 1,n), depending on the parity of d. In particular, U(Wa,b) is always quasi-split. 2.2. Matching of orbits. We first recall the notion of Jacquet–Langlands transfer between two non-isomorphic Hermitian spaces W and W ′. If we identify the underlying vector spaces (but not necessarily the Hermitian structures) W =∼ En =∼ W ′, (5) we have embeddings ′ .(Herm(W ), Herm(W ) ֒→ gln(E Then δ ∈Herm(W ) and δ′ ∈ Herm(W ′) are said to be Jacquet–Langlands transfers if they are GLn(E)-conjugate in gln(E). This is well defined since the above embeddings ′ are determined up to GLn(E)-conjugacy. Note that if δ and δ are Jacquet–Langlands transfers, then δ′ = Ad(g)(δ) for some g ∈ GL(W ) and we obtain a well-defined cohomology class ′ −1 1 inv(δ, δ ) := [σ ∈ Gal(E/F ) 7→ g σ(g)] ∈ H (F, Tδ) extending the invariant map to D(Tδ/F ). ′ Definition 2.2. In the case that W = Wa,b, we say that y and (δa, δb) are transfers (or are said to match) if they are Jacquet–Langlands transfers in the above sense. 8 SPENCER LESLIE

For later purposes, note that we have an embedding

,(φa,b : Herm(Va) ⊕Herm(Vb) ֒→Herm(Wa,b ∼ well deﬁned up to conjugation by U(Wa,b). If W = Wa,b, we say that a matching pair y and (δa, δb) are a nice matching pair if we may choose φa,b so that

φa,b(δa, δb)= δ.

rss ∞ 2.2.1. Orbital integrals. For δ ∈ Herm(W ) and f ∈ Cc (Herm(W )), we deﬁne the orbital integral Orb(δ, f)= f(g−1δg)dg,˙ ZTδ\U(W ) where dg is a Haar measure on U(W ), dt is the unique normalized Haar measure on the torus Tδ, and dg˙ is the invariant measure such that dtdg˙ = dg. To an elliptic endoscopic datum (U(Va)×U(Vb),s,η) and regular semi-simple element δ ∈Herm(W ), there is a natural character × κ : C(Tδ/F ) → C , which may be computed as follows. For matching elements δ and (δa, δb), 1 1 H (F, Tδ)= Z/2Z = Z/2Z × Z/2Z = H (F, Tδa × Tδb ), (6) YS1 SY1(a) SY1(b) where the notation indicates which elements of S1 arise from the torus Tδa or Tδb . 1 × Lemma 2.3. [Xia18, Proposition 3.10] Consider the character κ˜ : H (F, Tδ) → C such that on each Z/2Z-factor arising from S1(a), κ˜ is the trivial map, while it is the unique non-trivial map on each Z/2Z-factor arising from S1(b). Then

κ =κ ˜|C(Tδ/F ). Using the invariant map ∼ ,( inv(δ, −) : Ost(δ) −→D(Tδ/F ) ֒→ C(Tδ/F ∞ we form the κ-orbital integral of f ∈ Cc (Herm(W1)) Orbκ(δ, f)= κ(inv(δ, δ′)) Orb(δ′,f). ′ δX∼stδ When κ = 1 is trivial, write SO = Orbκ .

2.2.2. Transfer factors. We now recall the transfer factor of Langlands-Sheldstad and Kottwitz. This is a function rss rss ∆ : [Herm(Va) ⊕Herm(Vb)] ×Herm(W ) → C. The two important properties are

(1) ∆((δa, δb), δ)=0if δ does not match (δa, δb), and (2) if δ is stably conjugate to δ′, then κ ′ κ ′ ∆((δa, δb), δ)Orb (δ, f) = ∆((δa, δb), δ )Orb (δ ,f). While the general deﬁnition, given in [LS87] for the group case and [Kot99] in the quasi- split Lie algebra setting, is subtle, our present setting enjoys the following simpliﬁed formulation. When δ ∈Herm(W ) and (δa, δb) ∈Herm(Va) ⊕Herm(Vb) do not match, we set

∆((δa, δb), δ) = 0. ENDOSCOPYFORUNITARYSYMMETRICSPACES 9

Now suppose that δ and (δa, δb) match. We deﬁne the relative discriminant

D(δ)= (xa − xb), x ,x Ya b where xa (resp. xb) ranges over the eigenvalues of δa (resp. δb) in F . Remark 2.4. This is precisely the quotient of the standard Weyl discriminants that occurs in the factor ∆IV in [LS87]. It is well known that the magnitude of D(δ) measures the diﬀerence in asymptotic behavior of orbital integrals on Herm(W ) and Herm(Va)⊕ Herm(Vb). ∼ Recall our notation Wa,b = Va ⊕ Vb and ﬁrst assume that W = Wa,b and that δ and (δa, δb) are a nice matching pair. In this case, the transfer factor is then given by

∆((δa, δb), δ) := ωE/F (D(δ))|D(δ)|F , (7) where ωE/F is the quadratic character associated to E/F . Now for any matching pair δ and (δa, δb), let ′ δ = φa,b(δa, δb) ∈Herm(Wa,b). As discussed in Section 2.1, δ and δ′ are Jacquet–Langlands transfers of each other and we set ′ ∆((δa, δb), δ)= κ(inv(δ, δ ))ωE/F (D(δ))|D(δ)|F , 1 × where κ : H (F, Tδ) → C is the character arising from the datum (U(Va) × U(Vb),s,η) and inv is the extension of the invariant map discussed in Section 2.1. Remark 2.5. For the interested reader, when F is non-archimedean and U(W ) is quasi- split, our transfer factor agrees with the Lie algebra transfer factor studied in [Kot99], multiplied by the discriminant factor ∆IV . This formulation simply chooses a diﬀerent distinguished conjugacy class in the stable orbit. See Remark 5.4. 2.3. Smooth transfer. A pair of functions ∞ ∞ f ∈ Cc (Herm(W )) and fa,b ∈ Cc (Herm(Va) ⊕Herm(Vb)) are said to be smooth transfers (or matching functions) if the following conditions are satisﬁed:

(1) for any matching elements regular semi-simple elements δ and (δa, δb), κ SO((δa, δb),fa,b) = ∆((δa, δb), δ)Orb (δ, f);

(2) if there does not exist y matching (δa, δb), then

SO((δa, δb),fa,b) = 0. The following theorem follows by combining [LN08], [Wal06], and [Wal97b]. ∞ Theorem 2.6. For any f ∈ Cc (Herm(W1)), there exists a smooth transfer fa,b ∈ ∞ Cc (Herm(Va) ⊕Herm(Vb)). 3. The Lie algebra of the symmetric space

Let E/F be a quadratic extension of local ﬁelds of odd residue characteristic. Let W1 and W2 be two Hermitian spaces of dimension d over E. Let u(W ) denote the Lie algebra of U(W ), where W = W1 ⊕ W2 is a 2d dimensional Hermitian space. The diﬀerential of the involution σ acts on u(W ) by the same action and induces a Z/2Z-grading

u(W )= u(W )0 ⊕ u(W )1, i where u(W )i is the (−1) -eigenspace of the map σ. 10 SPENCER LESLIE

Lemma 3.1. We have natural identiﬁcations

u(W )0 = u(W1) ⊕ u(W2), and u(W )1 = HomE(W2,W1).

Here U(W1) × U(W2) acts on u(W )1 by the restriction of the adjoint action. In terms −1 of W1 and W2, the action is given by (g, h) · ϕ = g ◦ ϕ ◦ h . Proof. If σ : W → W denotes the linear involution, the involution induced on u(W ) is x 7→ σ ◦ x ◦ σ. It is a simple exercise in the deﬁnitions that any element x ∈ u(W ) may be uniquely expressed as x x x = 11 12 , x∗ x 12 22 where xii ∈ u(Vi), x12 ∈ Hom(W2,W1) and if h·, ·ii denotes the Hermitian pairing on Vi, ∗ then x12 ∈ Hom(W1,W2) is the unique linear map satisfying ∗ hx12v, wi1 = hv,x12wi2 for all v ∈ W2 and w ∈ W1. It follows that x −x σ(x)= 11 12 , −x∗ x 12 22 and the lemma follows.

In particular, any element x ∈ u(W )1 may be uniquely written X x = δ(X)= , −X∗ where X ∈ HomE(W2,W1). For any such x, we denote by Hx ⊂ U(W1) × U(W2) the stabilizer of x. rss Define the regular semi-simple locus u(W )1 to be the set of δ ∈ u(W )1 whose orbit under U(W1)×U(W2) is closed and of maximal dimension. In our present case, we have rss rss u(W )1 = u(W )1 ∩ u(W ) , where u(W )rss is the classical regular semi-simple locus of the Lie algebra. This is due to the fact that the symmetric pair (U(W ),U(W1) × U(W2) is geometrically quasi-split. See [Les19b, Sec. 2] for more details on quasi-split symmetric spaces. Let iso ∼ u(W )1 = IsoE(W2,W1) (8) be the open subvariety of elements δ(X) where X : W2 → W1 is a linear isomorphism. iso We refer to u(W )1 as the non-singular locus. There are natural contraction maps ri : u(W )1 →Herm(Wi) given by −XX∗ : i = 1 ri(δ(X)) = ∗ (−X X : i = 2. n Define the map π : u(W )1 → A given by π(x) = (a1(x), . . . , an(x)), where i−1 ai(x) = the coefficient of t in det(tI − r1(x)).

Lemma 3.2. The map r := r1 intertwines the U(W1) action on u(W )1 and the adjoint action on Herm(W1). Moreover, the pair (Herm(W1), r) is a categorical quotient for the U(W2)-action on u(W )1. Proof. The equivariance statement is obvious. As the categorical quotient assertion is geometric, we may assume without loss that F = F . The action we consider is following action of GLd(F ) × GLd(F ) on Matd(F ) × Matd(F ): (g, h) · (X,Y ) = (gXh−1, hY g−1). ENDOSCOPY FOR UNITARY SYMMETRIC SPACES 11

The map r becomes the product map

Matd(F ) × Matd(F ) → Matd(F ) (X,Y ) 7→ XY. We make use of Igusa’s criterion [Zha14, Section 3]: let a reductive group H act on an irreducible affine variety X. Let Q be a normal irreducible variety, and let π : X → Q be a morphism that is constant on H orbits such that (1) Q − π(X) has codimension at least two, (2) there exists a nonempty open subset Q′ ⊂ Q such that the fiber π−1(q) of q ∈ Q′ contains exactly one orbit. Then (Q,π) is a categorical quotient of (H, X). Note that it is clear that r is surjective as X → (X,Id) provides a section, so that the first criterion is satisfied. For the second ′ criterion, we note that the open set Q = GLd(F ) works.

Note that a similar argument gives the following lemma for the quotient by both unitary actions.

d Lemma 3.3. The pair (A ,π) is a categorical quotient for the U(W1) × U(W2) action on u(W )1. Proof. As in the proof of the previous proposition, we may pass to the algebraic closure, at which point it is evident that the map π is surjective. The uniqueness of orbits over a non-empty subset follows from the associated statement in Proposition 3.2 and the theory of rational canonical forms.

rss iso Lemma 3.4. There is an inclusion u(W )1 ⊂ u(W )1 . ∼ Proof. We again pass to the algebraic closure F = F and assume that u(W ) = gl2d(F ). Just as before, we now consider the action of GLd(F ) × GLd(F ) on Matd(F ) × Matd(F ) by (g, h) · (X,Y ) = (gXh−1, hY g−1). As before, we are now considering the action (g, h) · (X,Y ) = (gXh−1, hY g−1). of GLd(F )×GLd(F ) on Matd(F )×Matd(F ). The invariant of this action is π(X,Y )(t)= det(tI − XY ) as in Lemma 3.3. Recalling that the inﬁnitesimal symmetric space Matd(F ) × Matd(F ) is quasi-split, the element (X,Y ) is regular semi-simple in Matd(F )×Matd(F )= gl2d(F )1 if and only if the element X Z = ∈ gl (F ) Y 2d is regular semi-simple. Letting χZ (t) = det(tI −Z) denote the characteristic polynomial, Z is regular semi-simple if and only if χZ has distinct roots. Now a simple exercise in linear algebra shows that 2 χZ(t)= π(X,Y )(t ). rss Thus, γ ∈ gl2d(F ) is possible only if 0 is not a root of π(X,Y ), implying the lemma.

3.1. Relative orbital integrals. We now introduce the primary objects of interest: the relative orbital integrals for the symmetric pair (U(W1) × U(W2), u(W )1). For any x ∈ u(W )1, we set −1 Hx := {(h, g) ∈ U(W1) × U(W2) : h xg = x}. 12 SPENCER LESLIE

∞ Deﬁnition 3.5. For f ∈ Cc (u(W )1), and x ∈ u(W )1 a relatively semi-simple element, we deﬁne the relative orbital integral of f by −1 ˙ ˙ RO(x,f) := x f(h1 xh2)dh1dh2, (9) Hx\U(W1)×U(W2) where dhi are Haar measures on U(Vi) and dh˙ 1dh˙ 2 is the invariant measure such that dtdh˙ 1dh˙ 2 = dh1dh2. Here dt the unique normalized measure on the torus Hx as in Section 2. As always, the value of RO(f,x) depends on the choice of the measures dhi. We note that since the orbit of x is closed, the integral is absolutely convergent. Let iso Herm(W1) := Herm(W1) ∩ GL(W1) be the open subset of non-singular Hermitian forms.

iso iso Lemma 3.6. The restriction r : u(W )1 →Herm(W1) is a U(W2)-torsor. Moreover, iso for x ∈ u(W )1 , we have an isomorphism ∼ Hx −→ Tr(x) given by (h1, h2) 7→ h1. Proof. For the first claim, we saw in the proof of Proposition 3.2 that the claim holds over the algebraic closure of F , which suffices. For the second claim, we construct an inverse. Let h ∈ Tr(x). Then hx also lies in the fiber over r(x). By the torsor property, ′ there exists a unique h ∈ U(W2) such that hx = xh′. Define the inverse morphism by h 7→ (h, h′). It is clear that this gives a section. Notation 3.7. We will always use lower-case Roman letters x,y to denote vectors in the infinitesimal symmetric space u(W )1 and the like, and will use lower-case Greek letters δ, γ to denote vectors in the Hermitian quotient Herm(W1), etc. 3.2. Application of the contraction map. By Lemma 3.6, the contraction map to iso the non-singular locus of u(W )1 is a U(W2)-torsor. Noting that iso u(W )1 ⊂ u(W )1 is GL(u(W )1)-stable, Corollary A.2 implies the decomposition iso iso Herm(W1) = u(W )1 /U(Vα), 1 [α]∈H (GF,U(W2)) where the subscript α indicates the appropriate pure inner twist. Proposition A.3 thus tells us that ∞ iso ∞ iso (r/U(W2))! : Cc (u(W )1 ) → Cc (Herm(W1) ) 1 [α]∈H (GF,U(W2)) is surjective. We may extend this to a non-surjective map

∞ ∞ iso (r/U(W2))! : Cc (u(W )1) → C (Herm(W1) ), 1 [α]∈H (GF,U(W2)) where the push-forward r!(f) will not be compactly supported if supp(f) is not contained iso in u(W )1 . It is always of relatively-compactly support in Herm(W1) [Zha14, Lemma 3.12]. ENDOSCOPY FOR UNITARY SYMMETRIC SPACES 13

∼ ∞ Lemma 3.8. Fix Haar measures on U(W1), U(W2), and Hx = Tr(x). For f ∈ Cc (u(W )1) and for x ∈ u(W )1 regular semi-simple, we have

RO(x,f) = Orb(r(x), r!(f)), where both sides are normalized with our choices of Haar measures. Proof. If x is a regular semi-simple element, then everything is absolutely convergent as the orbit of r(x) is closed and the the support of supp(r!(f)) is relatively compact in Herm(W1). Rearranging the integrals and applying Lemmas 3.4 and 3.6 implies that

−1 RO(x,f)= r!(f)(g r(x)g)dg.˙ ZTr(x)\U(W1)

rss The following lemma is needed to study relative κ-orbital integrals. Let x ∈ u(W )1 rss and set δ = r(x) ∈Herm(W1) . ∼ Lemma 3.9. Let φ : Hx −→ Tδ be the map from Lemma 3.6. Then φ induces an isomorphism between ∼ C(Hx/F ) −→ C(Tδ/F ), where 1 1 C(Hx/F ) = ker H (F,Hx) → Hab(F,U(W1) × U(W2)) and 1 1 C(Tδ/F ) = ker H (F, Tδ) → Hab(F,U(W1)) . Proof. Consider the commutative diagram 1 ιx 1 1 H (F,Hx) Hab(F,U(W1)) × Hab(F,U(W2)) ∗ ∗ φ p1 1 ιδ 1 H (F, Tδ) Hab(F,U(W1)). ∗ ∗ where φ and p1 are the maps induce on cohomology. If α ∈ C(Hx/F ), then ιδφ(α) = p1(ιx(α)) = 1. This allows us to extend the diagram to 1 ιx 1 1 1 C(Hx/F ) H (F,Hx) Hab(F,U(W1)) × Hab(F,U(W2)) ∗ ∗ φ p1 1 ιδ 1 1 C(Tδ/F ) H (F, Tδ) Hab(F,U(W1)), where the arrow C(Hx/F ) → C(Tδ/F ) is an injection. By Lemma 2.1, we need only |S1|−1 show that |C(Hx/F )| = 2 . But this follows since the image of ιx lies in 1 1 ker Hab(F,U(W1) × U(W2)) → Hab(F,U(W )) . Combined with the injection C(Hx/F ) → C(Tδ/F ), this forces |S1|−1 |S1|−1 2 ≤ |C(Hx/F ) ≤ 2 , and the lemma follows.

4. Relative endoscopy In this section, we ﬁll in the details of our notion of endoscopic symmetric spaces and study the conjectural transfer and the fundamental lemma in this context. We establish the existence of smooth transfer for many functions. 14 SPENCER LESLIE

4.1. Relative endoscopic data. Let ξ = (U(Va)×U(Vb),s,η) be an elliptic endoscopic datum for U(W1), where d = a + b. Fix representatives {α} and {β} of the isomorphism classes of Hermitian forms on the underlying vector spaces of Va and Vb. For each pair (α, β), we have the Lie algebras

u(Va ⊕ Vα) and u(Vb ⊕ Vβ), where Vα simply denotes the Hermitian space (Va, α), and similarly for Vβ. Each equipped with a natural involution σα and σβ inducing inﬁnitesimal symmetric pairs

(U(Va) × U(Vα), u(Va ⊕ Vα)1) and (U(Vb) × U(Vβ), u(Vb ⊕ Vβ)1) . As in the introduction, we call the quintuple

(ξ, α, β) = (U(Va) × U(Vb),s,η,α,β) is a relative elliptic endoscopic datum and the direct sum of these infinitesimal symmetric pairs is an endoscopic infinitesimal symmetric pair for (U(W1) × U(W2), u(W )1). Notation 4.1. For the next two subsections, we adopt the following notation: fix a relative endoscopic datum (ξ, α, β) and set α,β g = u(W ) and h = u(Va ⊕ Vα) ⊕ u(Vb ⊕ Vβ). α,β The endoscopic space h1 comes equipped with the U(Vα)×U(Vβ)-invariant contraction map α,β rα,β : h1 →Herm(Va) ⊕Herm(Vb). rss α,β rss Definition 4.2. We say that x ∈ g1 matches the pair (xa,xb) ∈ (h1 ) if

r(x) ∈Herm(W1) and rα,β(xa,xb) ∈Herm(Va) ⊕Herm(Vb) match in the sense of Deﬁnition 2.2.

For matching elements (xa,xb) and x, we deﬁne the relative transfer factor

∆rel((xa,xb),x) := ∆(rα,β(xa,xb), r(x)), (10) where the right-hand side is the deﬁnition given in Section 2.2.2.

Lemma 4.3. As (xa,xb) varies over a stable (U(Va) × U(Vα)) × (U(Vb) × U(Vβ))-orbit α,β in h1 , the element x varies over a stable U(W1) × U(W2)-orbits in g1. Proof. This follows from the corresponding case for unitary Lie algebras and the fact that the regular stabilizers in the quotients r : g1 →Herm(W1) are trivial. rss 4.2. Smooth transfer. Fix x ∈ g1 and let (ξ, α, β) be a relative endoscopic datum. The endoscopic triple ξ = (H,s,η) of U(W1) determines a character × κ : C(Tr(x)/F ) → C via the construction of Lemma 2.3. By Lemma 3.9, we may pull this character back along the isomorphism ∼ C(Hx/F ) −→ C(Tr(x)/F ), × to obtain a character which we also call κ : C(Hx/F ) → C . We now define the relative κ-orbital integral to be ROκ(x,f) := κ(inv(x,x′)) RO(x′,f), ′ xX∼x ′ where x ranges over the representatives of the U(W1) × U(W2)-orbits stably conjugate to x. By Lemma 3.9, inv(x,x′) = inv(r(x), r(x′)). We stated the definition of smooth transfer in the context of relative endoscopy in Definition 1.2. ENDOSCOPY FOR UNITARY SYMMETRIC SPACES 15

∞ Conjecture 4.4. For any relative endoscopic datum (ξ, α, β) and any f ∈ Cc (g1), ∞ α,β there exists fα,β ∈ Cc (h1 ). such that f and fα,β match. We remark that this conjecture does not depend on the choice of Haar measures used in deﬁning the orbital integrals; coherent choices are needed for the fundamental lemma in the next section. We do not prove this conjecture in general, but are able to establish it in many cases. ∞ iso Proposition 4.5. Let f ∈ Cc (g1) and assume supp(f) ⊂ g1 . Then there exists ∞ α,β fα,β ∈ Cc (h1 ) such that f and fα,β match. Proof. As we have seen, the restriction of the contraction map iso iso r : g1 →Herm(W1) is a U(W2)-torsor. In particular, it is a submersion onto its image, implying that if iso supp(f) ⊂ g1 , ∞ iso r!(f) ∈ Cc (Herm(W1) ) is also compactly supported. Setting δ = r(x), we may now apply Theorem 2.6 to ﬁnd a smooth compactly-supported function

fa,b : Herm(Va) ⊕Herm(Vb) → C matching r!(f). Since the open subset iso iso Herm(Va) ⊕Herm(Vb) ⊂Herm(Va) ⊕Herm(Vb) iso is determined by the non-vanishing of the determinant, it follows that δ ∈Herm(W1) iso iso if and only if (δa, δb) ∈Herm(Va) ⊕Herm(Vb) . In particular, κ SO((δa, δb),fa,b) = ∆((δa, δb), δ)Orb (δ, r!(f)) = 0 rss iso whenever δ ∈ Herm(W1) does not lie in Herm(W1) . We thus lose no orbital integral information by replacing fa,b with

iso iso fa,b · 1Herm(Va) ⊗ 1Herm(Vb) . iso iso In particular, we can assume supp( fa,b) ⊂Herm(Va) ⊕Herm(Vb) . Considering the U(Vα) × U(Vβ)-torsor α,β iso iso iso rα,β : (h1 ) −→ Herm(Va) ⊕Herm(Vb) , iso iso another application of Corollary A.2 implies that Herm(Va) ⊕Herm(Vb) decom- poses as a disjoint union α,β iso (h1 ) /U(Vǫ) × U(Vν). 1 1 [ǫ,ν]∈H (F,U(VGa))×H (F,U(Vb)) Proposition A.3 now implies that there exist functions ∞ ǫ,ν iso fǫ,ν ∈ Cc ((h1 ) ), 1 with {(ǫ,ν)} ranging over pure inner forms H (F,U(Va) × U(Vb)), such that

fa,b = (rǫ,ν)!(fǫ,ν). ǫ,ν X In this way, for rα,β(xa,xb) = (δa, δb) we ﬁnd that

SRO((xa,xb),fα,β) = SO((δa, δb),fa,b), proving the proposition. 16 SPENCER LESLIE

The main obstruction to proving Conjecture 4.4 is that while r!(f) is always of relatively compact support. More information is needed about the singularities of this map to extend the previous result. 4.3. The endoscopic fundamental lemma. We now assume that E/F is an unramified extension of non-archimedean local fields. Suppose that Vd = W1 = W2 is split. In the non-archimedean setting, this implies that there exists a self-dual lattice Λd ⊂ Vd, a choice of which we now fix. In this case,

u(W )1 = HomE(Vd,Vd) = End(Vd), and the choice of self-dual lattice Λd gives a natural compact open subring End(Λd) ⊂

Lemma 4.6. Suppose that ξ = (U(Va)×U(Vb),s,η) is an endoscopic datum for Herm(Vd). ∼ Under our assumptions, we have Vd = Va ⊕ Vb. Proof. First, we recall the notion of a determinant d(V ) of a Hermitian space V : for any basis {xλ} of V , set let det(hxλ,xµi) be the determinant of the resulting matrix representation of the Hermitian form. Change of basis multiplies this by a norm from × × × E , so we set d(V ) to be the resulting class in F /NmE/F (E ). The theorem of Jacobowitz [Jac62a, Theorem 3.1.1] tells us that two Hermitian spaces V and W are isomorphic if and only if dim(V ) = dim(W ) and d(V ) ∼ d(W ). In the unramiﬁed non-archimedean setting, it is well known that a Hermitian space is split if and only if d(V ) is a norm. For example, we can choose a basis such with respect to which the form is represented by the identity matrix. Since

d(Va ⊕ Vb)= d(Va) · d(Vb) is then also a norm, the lemma follows.

We ﬁx such an isomorphism by imposing Λd = Λa ⊕ Λb for ﬁxed self-dual lattices Λa ⊂ Va and Λb ⊂ Vb; this is determined up to U(Λd) × U(Λd)-conjugation. Note that there are only four possible pairs (α, β), and we set (α0, β0) to be the split pair. We equip these groups with Haar measures normalized so that the given hyperspecial maximal subgroups have volume 1.

Theorem 4.7. (Relative fundamental lemma) If (α, β) = (α0, β0), the functions 1End(Λd) and 1End(Λa) ⊗ 1End(Λb) are smooth transfers. Otherwise, 1End(Λd) matches 0. This theorem is the main result of [Les19a], and its proof is beyond the scope of this paper. In the next section, we give a proof of this statement in the case of (U(V4),U(W2) × U(W2)).

5. The relative fundamental lemma for (U(V4),U(V2) × U(V2)) We continue with the assumption that E/F is an unramiﬁed extension of non- archimedean local ﬁelds. Let U(V4) be the quasi-split unitary group of rank 4 and (U(V2) × U(V2), End(V2)) the associated symmetric space. In this case, the only non- trivial endoscopic space to consider is End(V1) ⊕ End(V1) =∼ E ⊕ E with the action of [U(V1) × U(V1)] × [U(V1) × U(V1)].

Theorem 5.1. For the endoscopic space End(V1)⊕End(V1) of End(V2), the fundamental lemma holds. ENDOSCOPY FOR UNITARY SYMMETRIC SPACES 17

Our proof is computational. Let Λ be our rank 2 self-dual lattice, and let 1End(Λ) be the associated indicator function. The idea is to compute the push forward

∗ r!(1End(Λ))(XX )= 1End(Λ)(Xh)dh. ZU(V2) Once we have done this, we compute the associated integrals on the twisted Lie algebra and verify the κ-orbital integrals agree with the stable relative orbital integrals on the endoscopic side. The proof will be completed by combining Proposition 5.6 and Proposition 5.8 below. 5.1. Computing the pushforward. For the sake of computation, we ﬁx an element × ζ such that E = F (ζ) where ζ ∈ OE and ζ = −ζ. Here the overline indicates the non-trivial Galois element. We also ﬁx the split Hermitian form ζ J = . −ζ The contraction morphism

r : End(V2) −→ Herm(V2) T is given by X 7→ XX∗, where X∗ = JX J −1. Set Φ := r!1End(Λ). Then Φ is supported on the subset of Herm(V2) with val(det)) ≥ 0. Note that the group GL(V2) acts on Herm(V2) via the twisted action g · δ = gδg∗. and that for any g ∈ GL(Λ) ⊂ GL(V2),

∗ Φ(gr(x)g )= 1End(Λ)(gxh)dh = 1End(Λ)(gxh)dh = Φ(r(x)). ZU(V2) ZU(V2) Thus, Φ is constant on GL(Λ)-orbits of Herm(V2). By [Jac62b], we may choose the forms ̟i ̟(i,j) = ζ ̟j as representatives of these orbits.

Proposition 5.2. Let Φn be the restriction of Φ to Hermval(det)=n. Then Φn = 0 if n is odd or n< 0. If n ≥ 0 is even, we compute that

n/2 k n/2 j k Φ = q 1 (k,n−k) = q 1 , n   GL(Λ)·̟ Herm(Λ)n,k Xk=0 Xj=0 Xk=0   k where Herm(Λ)n,k := Herm(V2) ∩ ̟ End(Λ)val(det)=n−2k. Proof. As noted above, if M ∈ Im(r), then val(det(M)) is even. This implies that Φn = 0 when n is odd, so we assume now that n is even. Also, Φn = 0 for n < 0. Finally, the equality of the two expressions for Φn is a simple exercise. We thus show the left-most expression. We need only to compute Φ(̟(i,j)). Since Φ(̟(i,j))=Φ(̟(j,i)), we are free to assume that i ≤ j. Noting that j − i = 2l is even, we choose a section of the invariant map r over ̟(i,j): ̟iζ 1 2 X = i l , (i,j) l ̟ + ζ ̟ − 2 ! ∗ (i,j) where j − i = 2l. Then r(X(i,j))= X(i,j)X(i,j) = ̟ . 18 SPENCER LESLIE

We have the maximal compact subgroup U(Λ) ⊂ U(V2). Our choice of Hermitian form implies that the group t 1 x B = : t ∈ E×,x ∈ F t−1 1 is the F -points of a Borel subgroup of U(V2). The Iwasawa decomposition implies that

(i,j) Φ(̟ )= 1End(Λ)(X(i,j)h)dh ZU(V2) = 1End(Λ)(X(i,j)b). b∈B/(XB∩U(Λ)) Now the product is of the form ̟iζ 2u+̟iζ 1 2 1 u t t 2t i+l −1 = i . ̟l − ̟ ζ 1 t t̟l ̟l 2u−̟ ζ 2 ! 2t ! Therefore, we need val(t) ≥ 0, and val(2u + ̟iζ) ≥ val(t), and val(2u − ̟iζ)+ l ≥ val(t). A set of representatives of the quotient B/B ∩ K is given by 1 u ̟k , 1 ̟−k 2k i with k ∈ Z≥0 and u ∈ F/̟ OF . Since u ∈ F and b = ̟ ζ ∈ Fζ, we have val(u + b)= val(u − b) = min{val(u), val(b)}. In particular, i ≥ k, so that i ≥ 0. For each 0 < k ≤ i, where val(t) = k, we are free to pick any coset with u ∈ ̟kO/̟2kO so that 2k − 1 ≥ val(u) ≥ k so that there are qk options for u, where q = #(O/̟). Including the identity coset (with k = 0), we obtain i Φ(̟(i,n−i))= qk. Xk=0 5.2. Regular semi-simple elements and stable conjugacy. Lemma 2.1 tells us that the only relatively regular semi-simple elements x ∈ End(V2) we need to consider are those such that Hx =∼ U(1) × U(1), since rational and stable conjugacy agree for the other regular semi-simple conjugacy classes. Hereafter, we use the notation ∼st to denote stable conjugacy.

Lemma 5.3. Assume δ ∈Herm(V2) has a stabilizer isomorphic to U(1) × U(1). There are values a ∈ F and µ, λ ∈ F × with µλ ∈ Nm(E×) \ (F ×)2 such that δ is rationally conjugate to a λζ . µζ a Proof. Note that if the centralizer of δ in U(V2) is U(1) × U(1), then the centralizer of × × δ in GL2(E) is isomorphic to E × E . Lemma 2.1 now implies that the eigenvalues of δ lie in F , so there exist a, b ∈ F such that a + b δ ∼ . st a − b Taking a as above, µ = 1 and λ = (b/ζ)2, and checking characteristic polynomials we see that δ is stably conjugate a λζ γ = . ζ a ENDOSCOPY FOR UNITARY SYMMETRIC SPACES 19

If δ ∼ γ, then we are done. Otherwise, consider a λ̟−1ζ γ′ = . ̟ζ a ′ ′ The previous argument implies that δ ∼st γ ∼st γ ; we claim that γ and γ are not rationally conjugate. This suffices to prove the lemma as there are only 2 rational classes in the stable conjugacy class of δ by Lemma 2.1. Let us consider the function a λζ val = val(µ). µζ a x y Note that val(γ) 6≡ val(γ′) (mod 2). If h = ∈ U(V ), z w 2 a + yw − xzλζ xxλζ − yy hγh−1 = . ww − zzλζ a − yw + xzλζ For hγh−1 = γ′, we would need both yw = xzλζ and yw = xzλζ. But this is impossible unless all terms are zero, in which case it is easy to check that val(hγh−1) ≡ val(γ) (mod 2). It follows that γ and γ′ are not rationally conjugate. Remark 5.4. The representative a b2ζ−1 γ = ζ a from the lemma lies in the rational points of a Kostant section of the Chevalley invariant map χ : Herm(V2) −→ t/WT where T ⊂ U(V2) is the diagonal torus, t = Lie(T ), and WT is the associated Weyl group. Our computation of the transfer factors below verifies that the transfer factor ∆ agrees with Kottwitz’s formulation in [Kot99] in this case. ∼ Thus for any regular element δ ∈ Herm(V2), such that Tδ = U(1) × U(1), we may choose representatives of the two rational classes in the stable conjugacy class of δ to be of the form a λ ζ δ = ± , ± µ ζ a ± where η(µ±)= ±1. 5.3. Orbital integrals. We begin with a simple lemma. a λζ Lemma 5.5. Let δ = be as above, and denote by X = X = δ − aI the µζ a µ,λ 2 off-diagonal matrix. Assume µλ ∈ Nm(E×) \ (F ×)2 and set val(µλ) = 2m. Then m qk : η(µ) = 1  k=0 Orb(Xµ,λ, 1End(Λ))=  Xm−1 .   qk : η(µ)= −1 k=0  X   20 SPENCER LESLIE

Proof. We ﬁrst consider the case that η(µ) = 1. Since E/F is unramiﬁed, this restriction implies that val(µ)= n is even. We may assume that n = 0, since this does not change the conjugacy class of δ. As above, the Iwasawa decomposition on U(V2) implies

−1 Orb(Xµ,λ, 1End(Λ))= 1End(Λ)(hXµ,λh )dh ZU(V2) −1 = 1End(Λ)(hXµ,λh ). h∈B∩XU(Λ)\B A set of representatives of the quotient B/B ∩ U(Λ) is given by ̟−k 1 u h = , ̟k 1 2k with k ∈ Z and u ∈ F/̟ OF . Thus, we need to count k and u such that

λ−µu2 −1 µu ̟2k hXµ,λh = ζ 2k µ̟ −µu ! is integral. This forces the inequalities val(u) ≥ 0, min{m, val(u)}≥ k ≥ 0. We have used the fact that µλ∈ / (F ×)2 in identifying val(λ − µu2) = 2min{m, val(u)}. From this the result follows easily in this case. Now if we assume that η(µ)= −1, then necessarily val(µ) is odd, and we are free (up to conjugation) to assume val(µ)= −1. The result now follows from a similar argument as above.

We now compute the orbital integrals Orb(γ, Φn). Considering only regular elements ∼ with centralizer Tδ = U(1) × U(1), Lemma 5.3 implies we need only consider elements of the form a λ ζ a + b δ = ± ∼ , ± µ ζ a st a − b ± where η(µ±)= ±1. Then {δ+, δ−} are representatives of the two conjugacy classes in the stable conjugacy class. The character κ constructed in Lemma 2.3 gives our character

κ(inv(δ+, δ±)) = η(µ±).

Proposition 5.6. Set n1 = val(b). Then

n2 κ q : if val(a + b), val(a − b) ≡ 0 (2), Orb (δ+, Φn)= (11) ( 0 : otherwise. Proof. We explicitly compute the individual orbital integrals and then take the appropriate weighted sums. There are three cases to consider. For convenience, we record the results of these computations here. Set val(a) = n1, val(b) = n2, and val(det(δ∗)) = n ≥ min{2n1, 2n2}, where ∗ = ±. (1) If n1 >n2, then n2 = n/2 ≥ 0 and we have

n2 n2 1+n2 n2 j 2 q : if ∗ =+, Orb(δ∗, Φn)= q − 2+n2 qn2 : if ∗ = −. k=0 j=k ( 2 X X ENDOSCOPY FOR UNITARY SYMMETRIC SPACES 21

(2) If n1 0 and we have

n1 n2 1+n1 n2 j 2 q : if ∗ =+, Orb(δ∗, Φn)= q − 2+n1 qn2 : if ∗ = −. k=0 j=k ( 2 X X (3) If n1 = n2, then val(det(δ∗)) = n ≥ 2n1. If it is odd, then Orb(δ∗, Φ) = 0. Otherwise,

n2 n2 1+n2 n2 j 2 q : if ∗ =+, Orb(δ∗, Φn)= q − 2+n2 qn2 : if ∗ = −. k=0 j=k ( 2 X X Remark 5.7. The ﬁnal case n1 = n2 contains the nearly singular case studied in [Pol15]. In that work, the author only considers elements x with centralizer E× × E× in U(4), which forces the eigenvalues of r(x) to be norms. These have even valuation and we compute n2 = 2 val(x − y) = 2Vm, in his notation. In cases (1) and (3), we obtain

n2 κ q : if n2 ≡ 0 (2), Orb (δ+, Φn) = Orb(δ+, Φn) − Orb(δ−, Φn)= ( 0 :if n2 ≡ 1 (2). This is clearly equivalent to the statement of the proposition in case (1); for case (3), note that at most one of the eigenvalues a ± b has valuation greater than n2. Without loss of generality, we may assume that val(a + b)= n2 and val(a − b)= k. Then 2 2 n2 + k = val(a − b )= n, and n being even implies n2 ≡ k (2). Finally, case (2) becomes

n2 κ q : if n1 ≡ 0 (2), Orb (δ+, Φn)= ( 0 :if n1 ≡ 1 (2), which immediately implies the claim. We now prove these formulas. First assume that n1 >n2. Then for any for h ∈ U(V2), for any 0 ≤ k ≤ n/2, −1 k −1 k hδ∗h ∈ ̟ End(Λ)val(det)=n−2k ⇐⇒ hXµ,λh ∈ ̟ End(Λ)val(det)=n−2k.

Indeed, since val(det(δ∗)) = 2n2 = n and similarly for Xµ,λ, the only requirement is that k the entries lie in ̟ O. This holds for aI2 by assumption, so that it holds for the entries −1 −1 of hδh if and only if it holds for the entries of hXµ,λh . Using this and our computation of Φn, we have n/2 k Orb(δ , Φ )= q Orb δ , 1 k ∗ n ∗ ̟ End(Λ)val(det)=n−2k Xk=0 n/2 k = q Orb X , 1 k µ,λ ̟ End(Λ)val(det)=n−2k Xk=0 n/2 k −k = q Orb ̟ Xµ,λ, 1End(Λ)val(det)=n−2k Xk=0 n/2 k −k = q Orb ̟ Xµ,λ, 1End(Λ) . Xk=0 22 SPENCER LESLIE

−k This last reduction follows since ̟ Xµ,λ has the correct determinant, so that the orbital integrals over the test functions 1End(Λ)val(det)=n−2k and 1End(Λ) agree. Thus, we are reduced to computing the orbital integral

−k Orb ̟ Xµ,λ, 1End(Λ) . Now let δ∗ = δ+. By Lemma 5.5,

(n−2k)/2 qj : k even, −k  j=0 Orb ̟ Xµ,λ, 1End(Λ) = (n−X2k)/2−1   qj : k odd, j=0  X  so that  n/2 n/2 1 + (n/2) Orb(δ , Φ )= qj − qn/2. + n 2 Xk=0 Xj=k The computation is similar for δ∗ = δ−, and we ﬁnd

n/2 n/2 2 + (n/2) Orb(δ , Φ )= qj − qn/2. − n 2 Xk=0 Xj=k In the case that n1 < n2, there is a similar reduction. Indeed, since the valuation is −1 k −1 k correct, hδ∗h ∈ ̟ End(Λ)val(det)=n−2k if and only if hXµ,λh ∈ ̟ End(Λ). Writing

−k −1 −k −k −1 h̟ δ∗h = ̟ aI2 + h̟ Xµ,λh .

Since k ≤ n1 < n2, it is clear that integrality of the left-hand side is equivalent to the −k −1 integrality of h̟ Xµ,λh . Therefore, we consider the orbital integral

n1 k −k Orb(δ∗, Φn)= q Orb ̟ Xµ,λ, 1End(Λ) , Xk=0 which is computed as above. A similar argument works in the case that n1 = n2, provided k ≤ n1. In general,

n1 k −k Orb(δ∗, Φn)= q Orb ̟ Xµ,λ, 1End(Λ) Xk=0 n/2 k + q Orb δ , 1 k . ∗ ̟ End(Λ)val(det)=n−2k k=Xn1+1 The ﬁrst set of integrals are computed as above. Consider now the case that n1 < k ≤ n/2. Considering the sum

−k −1 −k −k −1 h̟ δ∗h = ̟ aI2 + h̟ Xµ,λh , −k −k −1 −k −1 since ̟ a∈ / OE, we ﬁnd that h̟ δ∗h ∈ End(Λ) if and only if h̟ Xµ,λh ∈ −k −k −1 End(Λ) − ̟ aI2. In particular, the lack of integrality of h̟ Xµ,λh is precisely canceled by the central term. We show this such a cancellation is not possible. Indeed, writing h = kb for k ∈ GL(Λ) −k −1 −k −1 and b ∈ B, then h̟ δ∗h ∈ End(Λ) if and only if b̟ δ∗b ∈ End(Λ), so that we ENDOSCOPY FOR UNITARY SYMMETRIC SPACES 23 may reduce to elements in the Borel subgroup as before. As previously noted, we may assume that our representatives are of the form ̟−m 1 u h = , ̟m 1 with m ∈ Z and u ∈ F . Thus, we have 2 µuζ λζ−µu ζ bX b−1 = ̟2m . µ,λ µ̟2mζ −µuζ k k But uµ ∈ F so that it is not possible for a + uµζ ∈ ̟ OE when a∈ / ̟ O. It follows that the orbital integrals Orb δ , 1 k vanish for k>n = val(a). ∗ ̟ End(Λ)val(det)=n−2k 1 rss 5.4. The endoscopic side. Let δ+ ∈ Herm(V2) be as in the previous section. Up to stable conjugacy, a + b δ ∼ , + st a − b 2 and we send δ+ → (a + b, a − b) ∈ Herm(V1) ⊕Herm(V1) =∼ F . Recall that the split Hermitian form on V1 = Ev is given so that hv, vi∈ Nm(E). The relative orbital integrals for this action are trivial: in the case of a single copy of (U(V1)×U(V1), u(V1 ⊕V1)1), the contraction map r : End(V1) →Herm(V1) corresponds to the ﬁeld norm NmE/F : E → F . Moreover, the action of U(V1) × U(V1) on E is given by (g, g′) · e = geg′, so that the contraction map is invariant with respect to both copies of U(V1) and takes × the E × U(V1) action to (g, h) · ee = geeg = Nm(g)Nm(e). ∼ × For any smooth integrable function φ on u(V1 ⊕ V1)1 = E that is OE -invariant and any x ∈ E×,

RO(x, φ):= φ(gxg′)dgdg′ ZU(V1)×U(V1)

= r∗φ(Nm(g)Nm(x))dg = r∗φ(Nm(x)) = φ(x). ZU(W1) κ We introduce the function Φ : Herm(V1) ×Herm(V1) → C 1 : val(x) ≡ val(y) ≡ 0 (mod 2), Φκ(x,y)= ( 0 : otherwise.

Letting 1End(Λ1) ⊗1End(Λ1) denote the basic function for the endoscopic symmetric space, it is easy to check that κ Φ = r(α0,β0),! 1End(Λ1) ⊗ 1End(Λ1) .

Proposition 5.8. For δ+ as in Proposition 5.6, we have κ κ SO((a + b, a − b), Φ ) = ∆((a + b, a − b), δ+)Orb (δ+, Φ). Proof. Our previous remarks allow us to compute the left-hand side: 1 : val(x), val(y) ≡ 0 (mod 2), SO((x,y), Φκ)= ( 0 : otherwise. 24 SPENCER LESLIE

For the right-hand side, some care must be taken with the transfer factor. When the matching δ+ 7→ (a + b, a − b) is a nice matching in the sense of Section 2, the transfer factor (7) may be computed as

− val(b) n2 −n2 ∆((a + b, a − b), δ+) = (−q) = (−1) q , using the notation from Proposition 5.6. This matching is nice if and only if the restriction of the Hermitian form of V2 to each of the two eigenlines V2 = L1 ⊕ L2 of δ+ corresponds to a split Hermitian form. A simple computation shows that this is the case if and only if n2 = val(b) is even. When n2 is odd, then δ− is a nice match with (a + b, a − b) so that

n2 −n2 −n2 ∆((a + b, a − b), δ+)= −∆((a + b, a − b), δ−)= −(−1) q = q . κ By Proposition 5.6, we see that Orb (δ+, Φ) vanishes unless both eigenvalues a + b and a − b are norms. Comparing with (11), we obtain the desired identity. ENDOSCOPY FOR UNITARY SYMMETRIC SPACES 25

Appendix A. Comment on Torsors In this appendix, we record some elementary properties of functions on torsors. The proofs are standard exercises which we omit. Let G be an aﬃne algebraic group over a ﬁeld F . There is a well-known correspon- dence 1 1 sep sep {F -torsors of G}/ ∼ ←→ H (F, G) := Hcont(Gal(F /F ), G(F )), where G(F sep) is endowed with the discrete topology. Let f : X → Y be a G-torsor and let [α] ∈ H1(F, G). As explained in [Ser97, Chapt. 1, §5], we may twist X by α to obtain another G-torsor fα : Xα → Y. Setting Xα = Xα(F ) and Y = Y(F ), the following result is standard. Proposition A.1. Let f : X → Y be a G-torsor. Then

Y = fα(Xα). 1 [α]∈HG(F,G) 1 This is true even if Xα = ∅ for some α ∈ H (F, G). Suppose that X = V is a linear representation of G. In this case, Hilbert’s theorem 90 implies that we for any [α] ∈ H1(F, G), ∼ Vα = V and that these isomorphisms may be chosen compatibly so that each pure inner form Gα acts on the variety V. This is discussed in greater detail in [BGW15], for example. Corollary A.2. Suppose that V is a linear representation of G. Let f : X → Y is a G-torsor and assume that there exists G-equivariant embeddings X ⊂ V such that the image is GL(V)-stable and Y ⊂ V// G such that the diagram XV f q YV// G,

1 where q is the canonical quotient map, commutes. Then for any [α] ∈ H (F, G), Xα =∼ X and there exist twisted torsor maps fα : X → Y such that

Y = fα(X). 1 [α]∈HG(F,G) Now suppose that F is a local ﬁeld. Let G be a reductive group over F and f : X → Y is a G-torsor of F -varieties such that there exists embeddings producing a diagram of G-varieties as in the corollary. We introduce the notation f/G : [Y/G] → Y to denote the map of Hausdorﬀ spaces f f/G : [Y/G] := X −→α Y. 1 [α]∈HG(F,G) Proposition A.3. The map f/G is a submersion. In particular, it induces a surjective map ∞ ∞ (f/G)! : Cc (X) → Cc (Y ) 1 [α]∈HM(F,G) where for a function φ = α φα and for any y ∈ Y P (f/G)! φ(y)= φα(g · x)dgα, ZGα 26 SPENCER LESLIE

−1 where x ∈ (f/G) (y), and dgα is the Haar measure on Gα chosen in such a way that all measures dgβ are compatible via the inner twisting from G.

References [BGW15] Manjul Bhargava, Benedict H Gross, and Xiaoheng Wang. Arithmetic invariant theory ii: Pure inner forms and obstructions to the existence of orbits. In Representations of reductive groups, pages 139–171. Springer, 2015. [Bor98] Mikhail Borovoi. Abelian Galois cohomology of reductive groups. Mem. Amer. Math. Soc., 132(626):viii+50, 1998. [BZ76] I.N. Bernstein and A.V. Zelevinsky. Representations of the group GL(n, F ), where F is a local non-Archimedean field. Uspehi Mat. Nauk, 31(3(189)):5–70, 1976. [GW14] Jayce R. Getz and Eric Wambach. Twisted relative trace formulae with a view towards unitary groups. Amer. J. Math., 136(1):1–58, 2014. [Jac62a] Ronald Jacobowitz. Hermitian forms over local fields. Amer. J. Math., 84:441–465, 1962. [Jac62b] Ronald Jacobowitz. Hermitian forms over local fields. American Journal of Mathematics, 84(3):441–465, 1962. [Kot99] Robert Kottwitz. Transfer factors for Lie algebras. Representation Theory of the American Mathematical Society, 3(6):127–138, 1999. [KS17] Friedrich Knop and Barbara Schalke. The dual group of a spherical variety. Transactions of the Moscow Mathematical Society, 78:187–216, 2017. [Lab11] Jean-Pierre Labesse. Introduction to endoscopy: Snowbird lectures, revised version, May 2010 [revision of mr2454335]. In On the stabilization of the trace formula, volume 1 of Stab. Trace Formula Shimura Var. Arith. Appl., pages 49–91. Int. Press, Somerville, MA, 2011. [Les19a] Spencer Leslie. The endoscopic fundamental lemma for unitary Friedberg-Jacquet periods. arXiv preprint arXiv:1911.07907, 2019. [Les19b] Spencer Leslie. A partial analogue of the Grothendieck-Springer resolution for symmetric spaces. arXiv preprint arXiv:1904.09217, 2019. [LN08] Gérard Laumon and Bao Châu Ngô. Le lemme fondamental pour les groupes unitaires. Ann. of Math. (2), 168(2):477–573, 2008. [LS87] R. P. Langlands and D. Shelstad. On the definition of transfer factors. Math. Ann., 278(1- 4):219–271, 1987. [LZ19] Chao Li and Wei Zhang. Kudla-Rapoport cycles and derivatives of local densities. arXiv preprint arXiv:1908.01701, 2019. [Pol15] Jason KC Polák. Exposing relative endoscopy in unitary symmetric spaces. Research in the Mathematical Sciences, 2(1):9, 2015. [PWZ19] Aaron Pollack, Chen Wan, and Michal Zydor. On the residue method for period integrals. arXiv preprint arXiv:1903.02544, 2019. [Rog90] Jonathan D. Rogawski. Automorphic representations of unitary groups in three variables, volume 123 of Annals of Mathematics Studies. Princeton University Press, Princeton, NJ, 1990. [Ser97] Jean-Pierre Serre. Galois cohomology. Springer-Verlag, Berlin, 1997. Translated from the French by Patrick Ion and revised by the author. [SV17] Yiannis Sakellaridis and Akshay Venkatesh. Periods and harmonic analysis on spherical varieties. Astérisque, (396):viii+360, 2017. [Wal95] J.-L. Waldspurger. Une formule des traces locale pour les algèbres de Lie p-adiques. J. Reine Angew. Math., 465:41–99, 1995. [Wal97a] J.-L. Waldspurger. Le lemme fondamental implique le transfert. Compositio Math., 105(2):153–236, 1997. [Wal97b] J.-L. Waldspurger. Le lemme fondamental implique le transfert. Compositio Math., 105(2):153–236, 1997. [Wal06] J.-L. Waldspurger. Endoscopie et changement de caractéristique. J. Inst. Math. Jussieu, 5(3):423–525, 2006. [Xia18] Jingwei Xiao. Endoscopic transfer for unitary Lie algebras. arXiv preprint arXiv:1802.07624, 2018. [Zha14] Wei Zhang. Fourier transform and the global Gan-Gross-Prasad conjecture for unitary groups. Ann. of Math. (2), 180(3):971–1049, 2014. [Zha15] Chong Zhang. On the smooth transfer for Guo–Jacquet relative trace formulae. Compositio Mathematica, 151(10):1821–1877, 2015. ENDOSCOPY FOR UNITARY SYMMETRIC SPACES 27

[Zha18] Wei Zhang. Special cycles on simple Shimura varieties. https://www.youtube.com/watch?v=v1JGu1lffCQ, March 2018.

Department of Mathematics, Duke University, 120 Science Drive, Durham, NC, USA E-mail address: [email protected]