On a Correspondence Between Cuspidal

Home , Stephen Rallis

JOURNAL OF THE AMERICAN MATHEMATICAL SOCIETY Volume 12, Number 3, Pages 849–907 S 0894-0347(99)00300-8 Article electronically published on April 26, 1999

ON A CORRESPONDENCE BETWEEN CUSPIDAL

REPRESENTATIONS OF GL2n AND Sp2n

f DAVID GINZBURG, STEPHEN RALLIS, AND DAVID SOUDRY

Introduction

Let K be a number field and A its ring of adeles. Let η be an irreducible, automorphic, cuspidal representation of GLm(A). Assume that η is self-dual. By Langlands conjectures, we expect that η is the functorial lift of an irreducible, automorphic, cuspidal representation σ of G(A), where G is an appropriate classical group defined over K. We know by [J.S.1] that the partial (as well as the complete) L-function LS(η η, s) has a (simple) pole at s = 1. Thus, since LS(η η, s)= LS(η, Sym2,s)LS⊗(η, Λ2,s), either LS(η, Λ2,s)orLS(η, Sym2,s) has a pole⊗ at s =1. If m =2n, then one expects that in the first case G =SO2n+1, and in the second S 2 case G =SO2n.Ifm=2n+ 1, we know that L (η, Λ ,s)isentire(see[J.S.2]), S 2 and hence L (η, Sym ,s) has a pole at s = 1. Here, one expects that G =Sp2n. In [G.R.S.1], we started a program to prove the existence of σ as above. See also [G.R.S.2], [G.R.S.3]. We proposed to prove this existence by explicit construction. We constructed a space Vσ(η) of automorphic forms on G(A), invariant under right translations. The elements of Vσ(η) are Bessel coefficients, or Fourier-Jacobi coefficients, restricted to G(A), of residues at s = 1 of certain Eisenstein series on an appropriate (“larger”) group H(A), induced from η.ThespaceVσ(η)has an extra property. It contains all irreducible, automorphic, cuspidal, generic (with respect to a given nondegenerate character) representations σ of G(A) which lift weakly to η (i.e. the unramified parameters of σ, at almost all places, are deter- L mined by those of η through the L-embedding G, GLm(C)). This is a first step → towards the well-known conjecture that every tempered L-packet on G(A) contains a generic representation. Since we believe in the existence of functorial lifting from G to GLm, we expect that Vσ(η) is nontrivial. This remained as a conjecture in [G.R.S.1]. From now on we restrict ourselves to the following special case. Let m =2n, and assume that LS(η, Λ2,s) has a pole at s =1.Now,weaddtheas- sumption that LS(η, 1 ) = 0. (It can be checked that LS(η, 1 ) =0,iffL(η, 1 ) =0, 2 6 2 6 2 6 for a unitary η.) Fix a nontrivial character ψ of K A. One expects that the “back- \ ward” lift of η to SO2n+1(A) is lifted from the metaplectic group Sp2n(A), via the theta correspondence associated to ψ (see [F]). Indeed, in this case we constructed

in [G.R.S.1] a space Vσ(η), as above, of automorphic forms on Sp2nf(A). We proved in [G.R.S.1, Theorem 13] that the elements of Vσ(η) are cuspidal in the sense that f Received by the editors July 22, 1998 and, in revised form, March 1, 1999. 1991 Mathematics Subject Classiﬁcation. Primary 11F27, 11F70, 11F85. The ﬁrst and third authors’ research was supported by The Israel Science Foundation founded by the Israel Academy of Sciences and Humanities.

c 1999 American Mathematical Society

849

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 850 DAVID GINZBURG, STEPHEN RALLIS, AND DAVID SOUDRY

all their constant terms along radicals of parabolic subgroups are identically zero. The main (global) result of this paper is Main Theorem (global). Let η be an irreducible, automorphic, cuspidal, self-dual S 2 representation of GL2n(A). Assume that L (η, Λ ,s) has a pole at s =1and that S 1 L (η, 2 ) =0. Then the space of automorphic cusp forms Vσ(η) on Sp2n(A) is nontrivial.6 f The space Vσ(η) aﬀords an automorphic, cuspidal, genuine representation σ(η)

of Sp2n(A). The construction of σ(η) and the results of [G.R.S.1] show Theorem 1. The cuspidal representation σ(η) contains all irreducible, automor- f 1 phic, cuspidal, genuine representations σ of Sp2n(A), which have a nontrivial ψ− - 1 S Whittaker coeﬃcients (i.e. ψ− -generic) and for which Lψ(σ η, s) has a pole at 1 ⊗ s =1. Each irreducible summand of σ(η) is fψ− -generic. As explained in [G.R.S.4, Sec. 3.1], there is no canonical deﬁnition of the (stan-

dard) L-function for σν ην on Sp (Kν ) GL n(Kν ) for almost all places ν.It ⊗ 2n × 2 depends on a choice of a nontrivial character of Kν. It turns out that f (0.1) Lψ (σν ην ,s)=L(θψ (σν) ην,s) ν ⊗ ν ⊗

whenever all data is unramiﬁed. Here θψν (σν ) is the unramiﬁed representation of

SO2n+1(Kν ), obtained from σν , under the local theta correspondence θψν , with

respect to ψν , from Sp2n(Kν )toSO2n+1(Kν ). The r.h.s. of (0.1) is the standard L-function for SO n GL n. This also suggests that the notion of an L-group 2 +1 × 2 can be extended to fSp2n(F ). Its L-group should be the L-group of SO2n+1(F ), i.e. Sp2n(C). (See [Sa] where this is justiﬁed in terms of Hecke algebras.) Note again that the choicef of a nontrivial character of Kν enables us to associate an

unramified representation of Sp2n(Kν ) to a semisimple conjugacy class in Sp2n(C). Thus, the standard embedding Sp2n(C) GL2n(C) should yield a (weak) lift from ⊂ genuine automorphic representationsf σ of Sp2n(A) to automorphic representations π of GL2n(A), only after we fixed ψ. In light of (0.1), π should be the weak lift of θψ(σ), the representation of SO2n+1f(A), obtained from σ under the theta correspondence with respect to ψ.Notethatifσon Sp2n(A)hasaψ-weak lift to S η on GL2n(A), then, of course, L (σ η, s) has a pole at s = 1. (The converse is ψ ⊗ not trivial. In a sequel to this paper, we will show, usingf some of the new results of this paper, that σ(η) is exactly the direct sum of all irreducible, automorphic, 1 genuine, cuspidal and ψ− -generic representations σ of Sp2n(A) which, under the ψ-weak lift to GL2n(A), lift to η. This will show, using Theorem 1, that if σ is 1 S ψ− -generic on Sp2n(A), and L (σ η, s) has a pole at sf=1,thenηis the ψ-weak ψ ⊗ lift of σ.) The global theoryf sketched so far has a beautiful local counterpart. The con- structions and theorems are motivated by the global results. However, the passage is not trivial. This local theory occupies the majority of the contents of this paper. Let F be a local nonarchimedean field of characteristic zero. Fix a nontrivial character of F still denoted by ψ. We construct an explicit map, which associates

an irreducible, supercuspidal representation σ(τ) of the metaplectic group Sp2n(F ) to an irreducible self-dual, supercuspidal representation τ of GL2n(F ), such that L(τ,Λ2,s) has a pole at s = 0. (See [Sh1] for the deﬁnition of L(τ,Λ2,sf).) The

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use CUSPIDAL REPRESENTATIONS OF GL2n AND Sp2n 851

representation σ(τ) has a Whittaker model, with respect tof the standard nonde- 1 generate character determined by ψ− . The deﬁning property of σ(τ), and this is in essence the main local result of this work, is

Main Theorem (local). The representation σ(τ) is the unique irreducible rep- 1 resentation σ of Sp2n(F ), which is supercuspidal, ψ− -generic and such that the gamma factor γ(σ τ,s,ψ) has a pole at s =1. × f The gamma factor just mentioned is the local gamma factor which appears in the corresponding local theory of global integrals of Shimura type, which represent the

standard (partial) L-function for generic cusp forms on Sp2n(A) GL2n(A). (See [G.R.S.4]. In general, the local gamma factor γ(σ τ,s,ψ) may be× deﬁned similarly 1 ⊗ for any irreducible ψ− -generic representation of Sp2k(Ff) GLm(F ). We expect that it is equal, at least up to a simple exponential, to the× local gamma factor associated (say, by Shahidi, or as in [So]) to θψ(σ)f τ on SO k (F ) GLm(F ), ⊗ 2 +1 × where θψ(σ) is the local ψ-theta lift of σ to SO2k+1(F ).) Here we get the added property that σ(τ) is irreducible, which we do not have yet in the global case, but, of course, conjecture to be true. Due to our lack of

knowledge of the local representation theory of Sp2n(F ), we do not yet have a good deﬁnition of the local L-factor Lψ(σ η, s)andthelocalε-factor ε(σ τ,s,ψ), such ⊗ ⊗ that f

ε(σ τ,s,ψ)Lψ(σ τ,1 s) (0.2) γ(σ τ,s,ψ)= ⊗ ⊗ − . ⊗ Lψ(σ τ,s) ⊗ b However, the gamma factor on the l.h.s. of (0.2) is well defined (details will be given in the paper) and the condition that γ(σ τ,s,ψ) has a pole at s =1should ⊗ really reflect the condition that Lψ(σ τ,s) has a pole at s =0. We will now recall some details from⊗ [G.R.S.1], [G.R.S.4] in order to complete the introductory global picture and thenceb draw the outline of the local theory, noting the beautiful analogy that runs throughout. We start with the global integrals mentioned above. These are introduced in [G.R.S.4]. Let us briefly explain how they are constructed. Let π be an irreducible, automorphic, cuspidal representation

of Sp2k(A). We assume that π is genuine and has nontrivial Whittaker coeﬃcients, 1 with respect to a standard nondegenerate character, corresponding to ψ− .(In thef text we denote this Whittaker character by ψk.) Let η be an irreducible, automorphic, cuspidal representation of GLm(A). We assume that k

(k) (0.3) ϕ(g)Jk,ψ(ωψ (g,1)φ, Eη(g,s))dg Sp (K)ZSp ( ) 2k \ 2k A (k) where ϕ is a cusp form in the space of π, ωψ denotes the Weil representation k of Sp2k(A) attached to ψ (it acts on Schwartz-Bruhat functions φ S(A )) and ∈ Jk,ψ( , ) denotes a Fourier-Jacobi coeﬃcient. The integral (0.3) is Eulerian, and LS (π η,s) forf decomposable data, it represents ψ ⊗ where LS (π η, s)isthe LS(η,Λ2,2s)LS(η,s+ 1 ) ψ 2 ⊗ partial, standard L-function of π η, corresponding to ψ. ⊗

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 852 DAVID GINZBURG, STEPHEN RALLIS, AND DAVID SOUDRY

Let m =2n, and let LS(η, Λ2,s)haveapoleats= 1. Assume also that S 1 L (η, 2 ) =0.Inthiscase,Eη(g,s) has a (simple) pole at s =1,andthen, S 6 S 2 for Lψ(π η, s)tohaveapoleats= 1, we must have (using that L (η, Λ , 2s) S ⊗1 · L (η, s + 2 ) is holomorphic and nonzero at s =1) (k) Jk,ψ(ω (g,ε)φ, Ress Eη(g,s)) 0 ψ =1 6≡

for (g,ε) Sp2k(A). We prove in [G.R.S.1], ∈ Theorem 2. We have, for k

σ(η)=σn(η)ofSp2n(A), which acts by right translations in the space (where – denotes complex conjugation) f (n) V = (g,ε) Jn,ψ(ω (g,ε)φ, Ress Eη(g,s)) . σ(η) { 7→ ψ =1 } Once we have the main global theorem of this paper (i.e. Vσ(η) = 0), then Theorem 1, mentioned earlier, is proven in [G.R.S.1]. 6 We now present in more details the local (supercuspidal) counterpart of the global theory sketched above. For an irreducible, self-dual supercuspidal represen- 2 tation τ of GL2n(F ), such that L(τ,Λ ,s) has a pole at s = 0, we construct the

following representation of Sp2n(F ): (n) 1 (0.4) σn(τ)=J n JN ,χ (πτ ) ωψ . f H n+1 n− ⊗ Here πτ is the Langlands quotient of ρτ,1, the representation of Sp4n(F ), induced b 1/2 from the Siegel parabolic subgroup, and τ det . πτ is the analog of the ⊗| ·| residue at s = 1 of the Eisenstein series. JU (resp. JU,χ) denotes the Jacquet functor with respect to a unipotent group U and the trivial character (resp. the character χ of U). Nn+1 is the unipotent radical of the standard parabolic subgroup Qn+1, which preserves a ﬂag of isotropic subspaces of dimensions from 1 to n +1, and χn is the character of Nn+1, which equals ψ on each simple root group inside Nn+1, and is trivial on the other root groups. n is the Heisenberg group in 2n + 1 variables. It is embedded in the unipotent radicalH of the standard parabolic

subgroup of Sp2n+2(F ), which preserves a line, and we take the natural embedding of Sp2n+2(F )insideLevi(Qn+1). Moreover, replacing n by k, k<2n,ateachplace

in (0.4), we obtain a sequence of representations σk(τ)ofSp2k(F ). We prove

The tower property. For the ﬁrst k0 , such that σk (τ) =0, the representation b 0 f6 σk0 (τ) is supercuspidal, i.e. its Jacquet functors, with respect to unipotent radicals of parabolic subgroups, vanish. (In particular σ (τ) is semisimple.) k0 b b This is the exact analog of tower (2.3) in [G.R.S.1]. The tower property is valid b even if we just require that τ is supercuspidal and replace πτ by any constituent of ρτ,s. The nonvanishing of σk(τ) is guaranteed for k = n.

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use CUSPIDAL REPRESENTATIONS OF GL2n AND Sp2n 853

f Theorem A. Let θ be a generic representation of GL2n(F ). Then, for a subrepresentation π of ρ 1 , we have θ, 2 (n) J n (JN ,χ 1 (π) ωψ ) =0. H n+1 n− ⊗ 6 In particular (for τ as above),

σˆn(τ) =0. 6 The main property of the residue at s = 1 of the Eisenstein series (on Sp4n(A)) Eη(g,s) is that this residue has a nontrivial period along Sp2n(A) Sp2n(A) [G.R.S.1, Theorem 2]. We prove the local analog. ×

Theorem B. For τ as above, the Langlands quotient πτ admits nontrivial Sp2n(F ) Sp (F )-invariant functionals, and hence (by [G.R.S.1, Theorem 17]) × 2n σk(τ)=0, for k

The tower property and Theorem A now prove that σn(τ) is supercuspidal, and b hence semisimple. The irreducibility of σn(τ) follows from the following results:

(i) Each summand of σn(τ) is ψ-generic (Propositionb4.2). (2n) (ii) There is a certain unipotent subgroupb E n Sp (F ) and a character ψ 2 ⊂ 4n of E2n (Sec. 4.1), suchb that the dimension of the space of ψ-Whittaker func- (2n) tionals on (the space of) σn(τ) is equal to dim JE2n,ψ (πτ ). (iii) dim J (2n) (πτ )=dimJ (πτ)=1,where V2n is the standard maximal E2n,ψ V2n,ψ unipotent subgroup of Sp b (F ), and ψ is the character, which is trivial on the 4n e Siegel radical and is equal to the standard ψ-nondegenerate character on the GL2n-Levi part of V2n. e It is interesting that these arguments are local variants, adapted from the proof of the main global theorem (Vσ(η) =0). The representation σ(τ) which appears6 in the main local theorem is the contragredient of σn(τ). The paper is organized as follows. Section 1 is an extended introduction. We explain howb the local theory of the global integrals (0.3) gives rise to the representations σk(τ). We define representationsσ ˜k(τ)ofSp2k(F ), such that there is a surjective morphismσ ˆk(τ) σ˜ k ( τ ). The elements of the space V ofσ ˜k(τ) −→ σ˜k (τ) f are ψ-Whittakerb functions on Sp2k(F ), which appear as inner integrals to the local integrals which emerge from (0.3). We explain in Section 1.2 that for a supercusp- 1 idal, ψ− -generic representationf σ of Sp (F ), k<2n,γ(σ τ,s,ψ) has a pole at 2k × s = 1, if and only if σ is a summand ofσ ˜k(τ) (Corollary 1.2.3). The main theorem of Section 1 is Theorem 1.3, which is Theoremf A above. We discovered this theorem only recently, and sinceb it stands in such a generality (arbitrary θ...) we preferred it over our older proof outlined in [G.R.S.3, Theorem 9]. In Section 2, we prove the tower property of the representations σk(τ) k<2n. In Section 3, we prove Theorem B. For this, we need to relate two local{ exterior} square gamma factors of τ:the one defined by Shahidi (as a local coefficient)b and the one which emerges from the work of Jacquet-Shalika, by examining the local theory which corresponds to their Rankin-Selberg integrals which represent the (partial) exterior square L-function. In Section 4, we prove the irreducibility of σn(τ). We prove the results (ii) and (iii) above in a much larger generality. In Section 5 we prove the main global theorem (i.e. V = 0). The proof here does not depend on the material presented in the σ(η) 6 b

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 854 DAVID GINZBURG, STEPHEN RALLIS, AND DAVID SOUDRY

previous section, although the analogy is clear. We have chosen to place the global result in this section in order to maintain some sort of continuity in the paper. Still we made Section 5 self-contained for the convenience of the reader who is interested in the global result ﬁrst. We marked the precise reference for each notation which appeared previously. However, we did not refrain from pointing out the analogy with the local theory. Section 6 is an appendix, where we prove results that we

need on the local theory of L-functions for Sp2k GLm (Sections 6.1 and 6.2) and on exterior square gamma factors for τ (Section× 6.3). f General notation. We write the elements of the symplectic group Sp2k(F ), with 1 1 wk respect to the skew-symmetric matrix where wk =  . . wk . . −   1    If x is a k k matrix, such that wkx is symmetric, we sometimes denote  × I x I `(x)= k , `(x)= k . Ik xIk Of course, `(x)and`(x) lie in Sp (F ). If a GLk(F ), we sometimes denote 2k ∈ a m(a)= a∗ t 1 where a∗ = wk a− wk. (The dependence on k will always be made clear.) For a representation π, we denote a space of its realization by Vπ. Let U be a unipotent group, χ a character of U,andπa smooth representation of U acting on Vπ.WedenotebyJU,χ(Vπ )orJU,χ(π)thespace

Vπ Span π(u)v χ(u)v u U, v V − ∈ ∈ . n o which we also call the Jacquet module of π with respect to χ.Whenχ=1,we

abbreviate to JU (Vπ)orJU(π). We denote by jU,χ the projection jU,χ : Vπ −→ J U,χ(Vπ ). k We denote by eij the standard basis of the matrix space Mk(F ). Thus, { }i,j=1 eij is the k k matrix, which has 1 in the (i, j)-th coordinate and zero elsewhere. Again, the dependence× on k will be made clear. We denote by Indc compact induction. Finally, let us recall the notions of Whittaker coeﬃcients, Whittaker model and genericity. Let G be a reductive split group over K. Fix a maximal split torus and a maximal unipotent subgroup U. This corresponds to a choice of a basis of simple roots for the corresponding root system. For each simple root α,letxα denote the corresponding root coordinate inside U.Letχbe a character of U(A), trivial on U(K). We say that χ is generic if it is nontrivial on xα(r) r A ,for { | ∈ } each simple root α. Thus, there is a set aα K∗ α simple root , such that { ∈ | − } χ(xα(t)) = ψ(aαt), for each simple root α and t A (and χ(xα(t))=1foreach ∈ positive nonsimple root α). Let χ be a generic character of U(A). An automorphic representation σ of G(A), acting on a space Vσ of automorphic forms, is χ-generic if

1 ϕ(u)χ− (u)du 0 ,ϕVσ. 6≡ ∈ U(K)ZU( ) \ A

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use CUSPIDAL REPRESENTATIONS OF GL2n AND Sp2n 855

This Fourier coefficient is called a χ-Whittaker coefficient. Similarly,f over a local field, say F = Kν , a generic character of U(F ) is a character χ of U(F ), such that χ is nontrivial on xα(r) r F , for each simple root α. A smooth representation { | ∈ } σ of G(F ) acting in Vσ is χ-generic if HomU(F )(σ, χ) = 0. A nontrivial element ` of the last space is called a χ-Whittaker functional,6 and a χ-Whittaker model of σ (defined by `) is the space of functions on G(F ), g `(σ(g)v) v Vσ . { 7→ | ∈ } Note that σ is χ-generic iff J (Vσ) = 0. These notions adapt easily to Sp (F ) U(F ),χ 6 2k and Sp2k(A). Here we choose U to be the standard maximal unipotent subgroup. There is a canonical splitting of the two-fold cover over U(F )(resp. U(A)).f See, for example,f [M.V.W, p. 43]. In this case, ψ above defines a standard generic character by applying ψ to the sum of entries in the second diagonal of an element of U.We sometimes abbreviate to “ψ-generic”, “ψ-Whittaker”, etc.

1. Preliminaries, motivations and the main theorems

1.1. Gamma factors for Sp2k(F ) GLm(F )(k

for generic cusp forms on Sp2kf(A) GLm(A), where A is the adele ring of a number field K. We explained in [G.R.S.1],× [G.R.S.3] how they lead to a map from irre- S 2 ducible, automorphic cuspidalf representations η on GL2n(A), such that L (η, Λ ,s) has a pole at s =1andL(η, 1 ) = 0, to irreducible automorphic, cuspidal ψ-generic 2 6 representations of Sp2n(A), which should be the inverse to the functorial lift (once we fix a nontrivial character ψ of k A). \ We motivate andf explain the analogous local counterpart of the above map by means of the local theory of the Shimura type integrals studied in [G.R.S.4]. Let F be a local nonarchimedean field of characteristic zero. Fix a nontrivial character of F , denoted again by ψ.Letσ(resp. τ) be an irreducible representation

of Sp2k(F )(resp.GLm(F)). We assume that τ is generic and that σ is genuine and 1 generic with respect to the character ψk− of Vk, the standard maximal unipotent subgroupf of Sp2k(F ), where k f (1.1) ψk : v ψ vi,i . 7→ +1 i=1 X Consider the representation

Sp2m(F ) s 1/2 ρτ,s =Ind τ det − (normalized induction) Pm ⊗| ·|

induced from Pm, the Siegel parabolic subgroup of Sp2m(F ). We think of an element ϕτ,s in the space of ρτ,s as a complex function on Sp (F ) GLm(F ) such that for 2m × a GLm(F ) ∈

a s+ m ϕτ,s ∗ g,Im = det a 2 ϕτ,s(g,a) 0 a∗ | | !

and a ϕτ,s(g,a) lies in the Whittaker model of τ with respect to the character 7→ k 1 m 1 − − (1.2) ψm0 (z)=ψ zi,i+1 +2zk,k+1 + zi,i+1 . i=1 i k ! X =X+1

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 856 DAVID GINZBURG, STEPHEN RALLIS, AND DAVID SOUDRY

Consider the parabolic subgroups Qm,i = Dm,i n Nm,i of Sp2m(F ), where

a1 ..  .  am i  −  aj F ∗ Dm,i =  g  ∈ ,  1  g Sp2i(F ) (  a−  ∈ )  m i   − .   ..     a 1  1−    z ∗∗ Nm,i = I2i Sp2m(F ) z Zm i . (  ∗∈ ∈ − ) z∗

  Here Zm i is the standard maximal unipotent subgroup of GLm i(F ). Assume − − that k

m k 1 − − (1.3) χk(v)=ψ vi,i ,vNm,k . +1 ∈ +1 i=1 X χk is the restriction to Nm,k+1 of the standard generic character deﬁned by ψ. Consider the following subgroup of Nm,k:

Im k 1 − − 1 xz   (1.4) Hk = h = I2kx0 Sp (F ) . ∈2m (  1  )    Imk1  −−   Hk is naturally identified with Nm,k/Nm,k+1 and is isomorphic to the Heisenberg 2k wk group k on F equipped with the symplectic form defined by 2 , H wk − 1 . where wk = . . (k k). The isomorphism is given by jm,k(x; z)=h,as   × 1   in (1.4). Note that we use a slightly different isomorphism than the one in [G.R.S.4], the purpose being the use of the Whittaker model of σ with respect to ψk in (1.1) rather than the character ψk in [G.R.S.4, Sec. 1]. (k) Let ω be the Weil representation of k o Sp2k(F ) which corresponds to the ψ H e (k) k character (0; z) ψ(z) of the center of k. ω acts on S(F )—the space of 7→ H ψ k f Schwartz Bruhat functions on F .Fork= 0, we define Sp0(F )= I, 0 = 1 z {} H z F and ω(0) = ψ. The local integrals which emerge from [G.R.S.4] 01|∈ ψ f nare o

(k) (1.5) J(W, φ, ϕτ,s)= W (g)ω (h g)φ(ξ ) ψ · 0 Z Z Z g V Sp (F ) h k k v Nm,k0 +1 Nm,k+1 ∈ k\ 2k ∈Y \H ∈ \ fτ,s(γm,kvjm,k(hg))χk(v)dvdhdg. f ·

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use CUSPIDAL REPRESENTATIONS OF GL2n AND Sp2n 857

1 k f Here, W lies in the ψ− Whittaker model of σ, φ is in S(F ), ξ =(0 01), k 0 ··· 0 Ik 00 000Imk (1.6) γm,k =  −−, Imk00 0 −  00Ik 0     and jm,k(hg)=jm,k(h)jm,k(g), where, for g Sp (F ), ∈ 2k Im k − (1.7) jm,k(g)= g .   Im k −   z 0 u 0 Ik 0 u0 z Zm k Nm,k0 +1 =   Sp2m(F ) ∈ − , ( Ik 0 ∈ u with zero bottom row)  z∗    k k = (0,y;0) k y F , Y { ∈H | ∈ } fτ,s(h)=ϕτ,s(h;Im) . The integral (1.5) converges absolutely for s in a right half plane, and for a holomorphic section ϕτ,s, it has a meromorphic continuation to the whole complex plane, s being a rational function of q− ,whereqis the number of elements in the residue field of F . J satisfies the following properties: (1.8) 1 J(W, φ, ρτ,s(v)ϕτ,s)=χ− (v)J(W, φ, ϕτ,s) , for v Nm,k , k ∈ +1 (1.9) (k) J(σ(g,ε)W, ω (h (g,ε))φ, ρτ,s(jm,k(hg))ϕτ,s)=J(W, φ, ϕτ,s) ψ · for (g,ε) Sp (F ),h k; ∈ 2k ∈H Jsatisfies a functional equation. The second side of the functional equation can be f deduced from the global integrals, by applying an intertwining operator Ms in the Eisenstein series. This translates to the local case as follows. Take Ms to be the local intertwining operator, acting on the space of ρτ,s, which corresponds to the Ik Weyl element . Ms takes ρτ,s to ρτ,1 s. Define J(W, φ, Ms(ϕτ,s)) by Ik − substituting M(−ϕ )(r, d )inJ(W, φ, ϕ ), instead of ϕ (r, I ), where d = s τ,s m,k τ,s b τ,s e m m,k t1 0

. ti tm k . , such that = 1, for all i = m k,and − = 2. (This  .  ti+1 tm k+1 − 6 − − − 0 t  m is done in order to preserve the transformation rule in (1.2).) J(W, φ, Ms(ϕτ,s)) satisfies the equivariance properties (1.8), (1.9). An adaptation to the local case of the proof of [G.R.S.4, Theorem 5.1] shows that there exists a meromorphice function γ(σ τ,s,ψ), such that × γ(σ τ,s,ψ) (1.10) × J(W, φ, ϕ )=J(W, φ, M (ϕ )). γ(τ,Λ2,2s 1,ψ) τ,s s τ,s − s This amounts to proving that except for a finite numbere of values of q− , there is, up to scalars, a unique trilinear form in the variables (W, φ, ϕτ,s) which satisfies

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 858 DAVID GINZBURG, STEPHEN RALLIS, AND DAVID SOUDRY

(1.8), (1.9). We give the details in Sections 6.1, 6.2. In (1.10), γ(τ,Λ2,2s 1,ψ)is the Shahidi local coeﬃcient of τ, which corresponds to Λ2 [Sh1]. − Consider the inner integral of J(W, φ, ϕτ,s) (1.5) (with g = I): (1.11) (k) Jm,k(φ, ϕτ,s)= ωψ (h)φ(ξ0)fτ,s(γm,kvjm,k(h))χk(v)dvdh. h Z v N Z N ∈Yk\Hk ∈ m,k0 +1\ m,k+1 Note that (k) 1 (1.12) Jm,k(ω (u h)φ, ρτ,s(vjm,k(u h))ϕτ,s)=ψk(u)χ− (v)Jm,k(φ, ϕτ,s) ψ · · k for v Nm,k ,h k,u Vk. This shows that ∈ +1 ∈H ∈ (k) (1.13) Wφ,ϕτ,s (g,ε)=Jm,k(ωψ (g,ε)φ, ρτ,s(jm,k(g))ϕτ,s)

is a ψ-Whittaker function on Sp2k(F ), and more generally, Jm,k is an element of the (k) 1 dual to the Jacquet module JVk,ψk J k JN ,χ (ρτ,s) ωψ . It is possible f H m,k+1 k− ⊗ to adapt the proof of Theorem 5.1h in [G.R.S.4] and show that thei embedding in Vρ of the subspace of functions S( m,k; Pm,τs), supported inside the (open) orbit τ,s O m,k = Pm γm,kNm,kjm,k(Sp (F )), induces an isomorphism of Sp (F )-modules O · 2k 2k (1.14) (k) f (k) 1 1 J k JN ,χ (Vρτ,s ) ωψ = J k JN ,χ (S( m,k; Pm,τs)) ωψ . H m,k+1 k− ⊗ ∼ H m,k+1 k− O ⊗ Details are given in Section 6.1. From this, it is easy to conclude

Proposition. The integral (1.11), which deﬁnes Jm,k(φ, ϕτ,s), stabilizes for large compact open subgroups of k k Nm,k0 +1 Nm,k+1, and hence Jm,k(φ, ϕτ,s) is holomorphic for all s. Y \H × \

The reason for the proposition is that, due to (1.14), we may replace ϕτ,s in Jm,k(φ, ϕτ,s)byanelementϕτ,s0 S( m,k; P, τs), for which Jm,k(φ, ϕτ,s0 ) converges absolutely, since the corresponding∈ integrandO is compactly supported. Note that the proposition implies the meromorphic continuation of the integrals J(W, φ, ϕτ,s),sincewecanwrite

(k) (1.15) J(W, φ, ϕτ,s)= W (g)Jm,k ωψ (g)φ, ρτ,s(jm,k(g))ϕτ,s dg

Vk SpZ (F ) \ 2k and now we use the Iwasawa decomposition of Sp2k(F ) and the asymptotic expansion of W (g). 1.2. Existence of a pole at s =1,forγ(σ τ,s,ψ). We assume from now on that m =2n.Letτbe supercuspidal, and rewrite× the functional equation (1.10) as 2 (1.16) γ(σ τ,s,ψ)J(W, φ, ϕτ,s)=L(τ,Λ ,2(1 s))J(W, φ, M ∗(ϕτ,s)) × − s where ε(τ,Λ2,2sb 1,ψ) e M ∗ = − M . s L(τ,Λ2,2s 1) s − We use the local factors deﬁned by Shahidi and the fact that L(ˆτ,Λ2,1 z) γ(τ,Λ2,z,ψ)=ε(τ,Λ2,z,ψ) − . L(τ,Λ2,z)

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use CUSPIDAL REPRESENTATIONS OF GL2n AND Sp2n 859

1 f Shahidi showed that L(τ,Λ2,2s 1) Ms is holomorphic and nontrivial, and thence Ms∗ is holomorphic and nontrivial,− since ε(τ,Λ2,2s 1,ψ) is monomial. See [Sh1], [Sh2]. − Proposition. Assume that σ and τ are supercuspidal. Then J(W, φ, ϕτ,s) and

J(W, φ, Ms∗(ϕτ,s)) are holomorphic. Proof. Since σ is supercuspidal, W has compact support, modulo V .Nowuse e k (1.15) and the proposition in 1.1 to obtain the holomorphicity of J(W, φ, ϕτ,s). Since

Ms∗(ϕτ,s) is holomorphic, and J(W, φ, Ms∗(ϕτ,s)) has entirely the same structure, the same proof works in this case as well. e Redenote N2n,i = Ni, N20n,i = Ni0, j2n,k = jk, γ2n,k = γk, J2n,k = Jk, etc. Deﬁne (1.17) (k) Jk(φ, Ms∗(ϕτ,s)) = ωψ (h)φ(ξ0)Ms∗(ϕτ,s)(γkvjk(h),dk)χk(v)dvdh Z Z k k Nk0 +1 Nk+1 e Y \H \ (dk is d2n,k from Section 1.1). We write, for the record, (1.18) (k) J(W, φ, Ms∗(ϕτ,s)) = W (g)Jk(ωψ (g)φ, ρτ,1 s(jk(g))Ms∗(ϕτ,s))dg. − Z Vk Sp2k(F ) e \ e b Corollary 1. Assume that σ and τ are supercuspidal. Then the only possible poles of γ(σ τ,s,ψ) occur among those of L(τ,Λ2,2(1 s)). In particular, if τ is self-dual,× the only possible poles of γ(σ τ,s,ψ) occur− on the line Re(s)=1. × Proof. From the last proposition, the only polesb of the r.h.s. of (1.16) occur among 2 those of L(τ,Λ ,2(1 s)). In [G.R.S.4, Prop. 6.6], we showed that data (W, φ, ϕτ,s) − can be chosen, so that J(W, φ, ϕτ,s)=1,foralls. The functional equation (1.16) nowb implies the corollary. In case τ is self-dual, the only possible poles of L(τ,Λ2,2(1 s)) occur on the line Re(s) = 1. See [Sh1]. − Corollary 2. Assume that σ and τ are supercuspidal and that τ is self-dual. Then γ(σ τ,s,ψ) has a pole at s =1,ifandonlyifL(τ,Λ2,s) has a pole at s =0and × (k) (1.19) W (g)Jk(ω (g)φ, ρτ, (jk(g))M ∗(ϕτ, ))dg 0. ψ 0 1 1 6≡ Z Vk Sp2k(F ) \ e Assume that τ is self-dual and L(τ,Λ2,s) has a pole at s = 0. The condition

(1.19) suggests that we consider the following space of functions on Sp2k(F ): k (k) φ s(F ) (1.20) Vσ˜k (τ ) = Span (g,ε) Jk(ωψ (g,ε)φ, ρτ,0(jk(g))M1∗(ϕτ,1)) f∈ . 7→ ϕτ,1 Vρτ,1 n ∈ o

By (1.12) and (1.13), Vσ˜k (τ) consistse of Whittaker functions on Sp 2k(F ), with re-

spect to the standard character ψk of Vk ((1.1)). Vσ˜k (τ ) is invariant to right trans- f lations by Sp2k(F ), and hence aﬀords a smooth representationσ ˜k(τ)ofSp2k(F ). The condition (1.19) means that σ is paired intoσ ˜k(τ), which is equivalent, due to the supercuspidalityf of σ,to f Corollary 3. Under the above assumptions, γ(σ τ,s,ψ) has a pole at s =1,if × and only if σ is a summand of σ˜k(τ).

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 860 DAVID GINZBURG, STEPHEN RALLIS, AND DAVID SOUDRY

1.3. The question of nonvanishing of Vσ˜k (τ). In (1.20), the elements M1∗(ϕτ,1)

form an invariant subspace of Vρτ,0 . Let us consider, for a given irreducible generic representation θ of GL2n(F ), for a given subrepresentation (π(θ),Vπ(θ))ofρ 1 ,and θ, 2 for k<2n,

k (k) φ S(F ) (1.21) Vk,π(θ) = Span (g,ε) Jk ωψ (g,ε)φ, ρθ, 1 (jk(g))ϕ ∈ . 7→ 2 ϕ Vπ(θ) n ∈ o

This space aﬀords a representationσ ˜k,π(θ) (by right translations of Sp2k(F )). As in (1.3), this is a space of Whittaker functions, with respect to ψk. From (1.12), the map f

(k) (1.22) ϕ φ (g,ε) Jk(ωψ (g,ε)φ, ρθ, 1 (jk(g))ϕ) ⊗ 7→ 7→ 2 deﬁnes a surjective morphism of Sp2k(F )-modules

(k) (1.23) J J 1 (V ) ω V . k N ,χ−f π(θ) ψ k,π(θ) H k+1 k ⊗ −→ 1/2 Note the case θ = τ det − ,whereτis self-dual, supercuspidal and such 2 ⊗| ·| that L(τ,Λ ,s) has a pole at s =0.Hereρ 1 = ρτ,0. Shahidi proved that ρτ,0 θ, 2 has two constituents. One irreducible subrepresentation (nongeneric) πτ ,andone irreducible (generic) quotient ρτ . πτ is the image of M1 (or M1∗) applied to ρτ,1.It is the Langlands quotient of ρτ,1. Thus, in this case, the only nontrivial π(θ)isπτ. Hereσ ˜k(τ)=˜σk,π(θ).

Denote by σk,π(θ) the representation of Sp2k(F ) on the l.h.s. of (1.23), and in 1/2 2 case θ = τ det − , τ-self-dual supercuspidal, and with L(τ,Λ ,s)havinga ⊗| ·| pole at s =0,denoteb σk(τ)=σk,πτ . f Theorem. We have, in general, b b (1.24) σ˜ =0, n,π(θ) 6 and, in particular, (1.25) σ =0. n,π(θ) 6 n Proof. We will show that Jn(φ, ϕ) 0, as φ and ϕ vary in S(F )andVπ(θ), respectively. Explicating (1.11), web have6≡

In In (1.26) Jn(φ, ϕ)= φ(ξ0+un)ϕ  γn,1ψ(vn,1)d(u, v). uvIn Z !  u0 In     Here un is the n-th row of u.Sinceφis arbitrary, the nonvanishing of (1.26) in (φ, ϕ) is equivalent to

In In (1.27) ϕ  ,1ψ(vn,1)d(u, v) 0 , uvIn ! 6≡ u Z=0 n  0u0 In     as ϕ varies in Vπ(θ).

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use CUSPIDAL REPRESENTATIONS OF GL2n AND Sp2n 861

f y1 . Denote, for x =(x1,... ,xn 1), y = . and 1 i n 1, 2 j n, −   ≤ ≤ − ≤ ≤ yn 1  −  n 1   − `i(x)=` xs(ei,s + e2n+1 s,2n+1 i) , − − s=1 X n 1 j − ` (y)=` ys(es,j + e2n+1 j,2n+1 s) . − − s=1 X Let, for 0 i n 1, ≤ ≤ − In 1 − In (1.28) X(i, n)= x= +1  ( uvIn+1  0 u0 In 1  −    un+1 = un = =ui+1 =0 Sp4n(F ) ··· . ∈ vn,1 = vn+1,1 =0 )

Here, uj is the j-th row of u. The domain of integration in (1.27) is X(n 1,n). − Deﬁne, for x in X(i, n), as in (1.28)

ψi(x)=ψ(vn,2) , and consider

Ii(ϕ)= ϕ(x, 1)ψi(x)dx.

X(Zi,n)

The integral (1.27) is In 1(ϕ). We ﬁrst show that, for 1 i n 1, − ≤ ≤ − Ii(ϕ) 0 I i 1 ( ϕ ) 0, as ϕ varies in Vπ(θ) . 6≡ ⇐⇒ − 6≡ Indeed, we may assume that ϕ is a linear combination of vectors of the form

i+1 ξ f = ξ(y)ρθ, 1 ` (y) fdy ∗ 2 FZn n 1 where ξ S(F − )andf Vπ(θ). (Here we allow an abuse of notation when we use “ ”.)∈ We have ∈ ∗ i+1 Ii(ξ f)= ξ(y)f x ` (y), 1 ψi(x)dydx. ∗ · X(Zi,n) FZn Write x in X(i, n) in the form

(1.29) x = x ` (u )` (u ) ... ì(ui) 0 1 1 2 2 · · where x X(0,n). For i0 i,wehave 0 ∈ ≤ i+1 1 i+1 ì0(ui0)` (y)ì0 (ui0 )− = z(i0)` (y) where

z(i0)=m(I2n (ui y)e2n i,2n i +1) . − 0 · − − 0

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 862 DAVID GINZBURG, STEPHEN RALLIS, AND DAVID SOUDRY

It is easy to see that z(i0) normalizes X(i0 1,n), and preserves ψi0 1 and the i+1 − − measure dx on X(i0 1,n). Similarly, ` (y) normalizes X(0,n), and preserves ψ0 and the measure dx−on X(0,n). Finally, note that

f(z(i0)h, 1) = ψ(δi ,i(ui y)) . 0 0 · All this implies that (using obvious notation)

1 Ii(ξ f)= ξ(y)ψ− (ui y)f(x, 1)ψi(x)dydx ∗ · X(Zi,n) FZn

= ξ(ui)f(x, 1)ψi(x)dx = Ii 1(ξ f) . − ∗ X(Zi,n) b b Here we used the notation (1.29) for x. This proves our assertion. Now consider r1 . I0(ϕ). Let, for r = . and p =(p1,... ,pn 1),  .  − rn 1  −    n 1 − e(r)=` rs(es,n + en+1,2n+1 s) , − s=1 X n 1 − e(p)=` ps(en 1,s + e2n+1 s,n+2) . − − s=1 X Again, take ϕ to be a linear combination of vectors of the form

ξ f = ξ(p)ρθ, 1 (e(p))fdp ∗ 2 n 1 F Z−

n 1 where ξ S(F − )andf V .Denote ∈ ∈ π(θ)

In In S= v= Sp4n(F ) , ( vIn ∈ )  In     ψS(v)=ψ(vn,1) .

As before, we get

(1.30) I (ξ f)= ξ(r)f(a e(r),1)ψ (a)drda . 0 ∗ · S ZS FZn b Put

A(f)= f(a, 1)ψS(a)da . ZS

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use CUSPIDAL REPRESENTATIONS OF GL2n AND Sp2n 863

f Thus (in obvious notation) I0(ξ f)=A(ξ f). So far, we proved that Jn(φ, ϕ) 0 if and only if A(ϕ) 0. Denote,∗ for n i ∗ 2n 1, 6≡ 6≡ ≤ ≤ − 2n i 1 b −− Ki+1 = ki+1(t1,... ,t2n i)=` tj(ej,i+1 +e2n i,2nr1 j)+t2n ie2n i,i+1 , − − − − − j=1 n X o 2n i − Ri = ri(x1,... ,x2n i)=` xj (ei,j + e2n+1 j,2n+1 i) . − − − j=1 n X o Let, for n i 2n 1, ≤ ≤ − 2n i K = K`, ` i =Y+1 i ψ = ψS , Ki

(i) i A (ϕ)= f(k, 1)ψ (k)dk . KZi n (n) n i Note that K = S, A = A, ψ = ψS and ψ =1,fori n+ 1. Again, we may assume that ϕ has the form ≥

ξ f = ξ(r)ρθ, 1 (r)fdr ; ξ S(Ri),f Vπ(θ) . ∗ 2 ∈ ∈ RZi

It is easy to check that, for r = ri(x1,... ,x2n i) Ri and k = k0 ki+1(t1,... ,t2n i), i+1 1 − ∈ · − k0 K ,wehavethatkrk− V n,and ∈ ∈ 2 2n i 1 1 − f krk− k, 1) = ψ− (α tjxj f(k, 1) · · j=1 X (α =2ifi=n,andα=1ifi n+ 1). This implies ≥ (1.31)

(i) A (ξ f)= ξ(t1,... ,t2n i)ρθ, 1 (ki+1(t1,... ,t2n i))f(k0, 1)dtdk0 . ∗ − 2 − i+1 2n i KZ F Z− e Here n 1 − 1 ξ(t ,... ,tn)= ξ(rn(x ,... ,xn))ψ− 2 tjxj + xn(2tn 1) dx 1 1 − j=1 FZn X and forei n +1 ≥ 2n i − 1 ξ(t1,... ,t2n i)= ξ(ri(x1,... ,x2n i))ψ− tjxj dx . − − 2Zn i j=1 F − X e 2n i Clearly ξ varies over S(F − )asξvaries over S(Ri). Since the l.h.s of (1.31) has the form A(i+1)(ξ f), where this time e ∗ ξ fe= ξ(t1,... ,t2n i)ρθ, 1 (ki+1(t1,... ,t2n i))fdt, ∗ − 2 − 2n i F Z− e e

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 864 DAVID GINZBURG, STEPHEN RALLIS, AND DAVID SOUDRY

this shows that A(i)(ϕ) 0, if and only if A(i+1)(ϕ) 0, as ϕ varies in V , 6≡ 6≡ π(θ) for n i 2n 1, where A(2n)(ϕ)=ϕ(1, 1). Since A(2n)(ϕ) 0, on V ,we ≤ ≤ − 6≡ π(θ) conclude that Jn(φ, ϕ) 0, and the theorem is proved. 6≡ Here we record the following special case. Corollary. Let τ be an irreducible, self-dual, supercuspidal representation of 2 GL n(F ), such that L(τ,Λ ,s) has a pole at s =0.Thenσ˜n(τ)=0,and,in 2 6 particular, σn(τ) =0. 6 In Section 2 we will prove that σk(τ)=0,fork

tation of Sp4n(F ). Deﬁne for k<2n

(k) 1 σk,π = J k JN ,χ (π) ωψ . H k+1 k− ⊗ This is a representationb of Sp2k(F ). We will consider such representations (in this generality) in the course of this work. f 1.4. The representations σk(τ) and functoriality. Let τ be an irreducible, 2 self-dual, supercuspidal representation of GL2n(F ), such that L(τ,Λ ,s) has a pole at s = 0. By the principle of functoriality (adapted to metaplectic groups once we ﬁx a nontrivial character, say ψ,ofF), we expect that τ is the functorial lift of a

supercuspidal representation σ of Sp2n(F ). Although this notion is not well defined yet, we certainly expect that γ(σ τ,s,ψ) has a pole at s = 1, and if we impose × 1 that σ is generic (with respect tofψn− ), then σ should be uniquely determined by the pole condition. By Corollary 3 in Section 1.2, such a σ exists, if and only if σ is a summand ofσ ñ(τ), which by Corollary 1.2.3 is nontrivial. Thus, if we prove thatσ ñ(τ) is supercuspidal, then any summand ofσ ñ(τ) will provide an exampleb of σ as above.

1.5.b The main local theorem. The main local theorem of this paper is Theorem. Let τ be an irreducible, self-dual, supercuspidal representation of 2 GL2n(F ), such that L(τ,Λ ,s) has a pole at s =0. Then the representation σn(τ) is nontrivial, supercuspidal and irreducible. b We have already proved the nontriviality of σn(τ) (Theorem 1.3). The surjection (1.23) σn(τ) σ˜ n ( τ ) implies that −→ b σn(τ) = σ˜n(τ) b ∼ and we conclude, denoting by σ(τ) the contragredient of σ (τ), b n Corollary. Let τ be as above. There is a unique genuine, irreducible, supercuspidal b 1 representation σ of Sp2n(F ), which is generic with respect to ψn− and is such that γ(σ τ,s,ψ) has a pole at s =1. This is the representation σ(τ). × f

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use CUSPIDAL REPRESENTATIONS OF GL2n AND Sp2n 865

Remark. Although there is no precise theory of L-packets yetf available for meta-

plectic groups like Sp2n(F ), we consider that the last corollary establishes in principle the fact that σ(τ) is the unique ψ-generic representative of “the L-packet determined by τ andf ψ”. 1.6. Main steps of the proof. We will prove the following theorems, for τ as in Theorem 1.5.

Theorem 1. (The tower property): Assume that σj (τ)=0for all j

These two theorems imply that σn(τ) is supercuspidal. b We will get the irreducibility of σn(τ) by showing that all summands of σn(τ) are ψn-generic, and then show thatb up to scalars σn(τ) admits exactly one ψn- Whittaker functional. Note the followingb Corollary to Theorem 2, Corollaryb 1.2.1 and Corollary 1.2.3. b Corollary. Let σ be an irreducible, genuine, supercuspidal representation of 1 Sp2k(F ). Assume that k

E(g,ϕη,s)= ϕη,s(γg;I2n), γ P (K) Sp (K) ∈ 2n X\ 4n has a simple pole at s =1.Fork<2n, we deﬁned σk(η) to be the representation

by right translations of Sp2k(A) on the following space of automorphic functions: (k) (1.32) V = Span (g,ε) Jk(ω (g,ε)φ, Ress E(jk(g),ϕη,s)) σk (η) f { 7→ ψ =1 } where (1.33) (k) Jk(ωψ (g,ε)φ, Ress=1 E(jk(g),ϕη,s))

(k) = θ (h (g,ε)) Ress E(hvjk(g),ϕη,s)χk(v)dvdh ψ,φ · =1 (F )Z ( ) N (F )ZN ( ) Hk \Hk A k+1 \ k+1 A (k) (k) (ωψ is the corresponding global Weil representation and θψ,φ is the corresponding theta series). We proved in [G.R.S.1] that σk(η) = 0 for all k

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 866 DAVID GINZBURG, STEPHEN RALLIS, AND DAVID SOUDRY

Recall from [G.R.S.1] that σn(η) contains all irreducible, automorphic, cuspidal, 1 S ψ− -generic representations σ of Sp2n(A), such that Lψ(σ η, s) has a pole at s =1. × We denote σ(η)=σn(η). f

2. The tower property of the representations σk(τ) k< n { } 2 2.1. Statement of the tower property. Let 1 p k<2n, and consider the unipotent radical ≤ ≤ b

Ip xy Rp = I2(k p) x0 Sp2k(F )  −  ∈ ( Ip )   and the parabolic subgroup

m z∗∗ ∗∗  ∗∗∗ m GLp(F ) Qk p = I2(k p) Sp4n(F ) ∈ . − − ∗∗∈ z Z2n k (  z∗  ∈ − )  ∗  m∗     Regard σj(τ)asanSp (F) Nj-module. 2j · Theorem. We have a vector space isomorphism b f cQk p J (V ) Ind − V . Rp σˆk (τ ) = Nk p σk p(τ) ∼ − − This isomorphism is the local analog of formulab (2.44) in [G.R.S.1]. This also implies that if σj (τ) = 0 for all j

Let be a maximal nilpotent Lie subalgebra of Lie(Sp4n(F )). Let , , and beU Lie subalgebras of and let A, C, X, Y be the corresponding unipotentA C X Y U subgroups of Sp4n(F ). Let χ be a nontrivial character of C. We make the following assumptions: (i) C, X, Y A. (ii) X and Y⊂are abelian, normalize C and preserve χ. 1 1 (iii) The commutators x− y− xy lie in C, for all x X, y Y . In particular, Y normalizes D = CX and X normalizes B = CY∈. ∈ (iv) A = D o Y = B o X. (v) The set

1 1 x χ(x− y− xy) y Y { 7→ | ∈ } is the group of all characters of X. Moreover, writing x =expE,y =expS, for E ,S ,wehave ∈X ∈Y 1 1 χ(xyx− y− )=ψ((E,S))

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use CUSPIDAL REPRESENTATIONS OF GL2n AND Sp2n 867

where (, ) is a nondegenerate, bilinear pairing between f and . X Y BX = A = DY

X Y ,- (2.1) B=CY D = CX

Y X -, C Lemma. Assume (i)-(iv). Let π be a smooth representation of A.Extendχtriv- ially to characters χB of B and χD of D. Then we have an isomorphism of C- modules (2.2) J (π) = J (π) . B,χB ∼ D,χD Proof. Since X and Y normalize C and both preserve χ, X and Y act on JC,χ(π).

Consider JX (JC,χ(π)) = JD,χD (π). We have a natural surjection (over D)

(2.3) T : JC,χ(π) J D,χ (π) −→ D which induces a map of A-modules A (2.4) i : JC,χ(π) Ind JD,χ (π) −→ D D determined by (2.5) i(ξ)(y)=T(π(y)ξ),yY, ∈ where π is the representation of A in JC,χ(π). cA We will show that i is injective and Im(i) IndD JD,χD (π),andthentaking Jacquet modules in (2.4), with respect to Y ,weobtain⊆

cA JB,χ (π)=JY(JC,χ(π)) , JY Ind JD,χ (π) = JD,χ (π). B → D D ∼ D The last embedding is clearly surjective, and this will prove the lemma. We show the injectivity of i. Assume that ξ,inthespaceofJC,χ(π), is such that i(ξ)=0. From (2.3) and (2.5), this means that for each y Y , there is a compact open ∈ subgroup Ωy X, such that ⊂ (2.6) π(x y)ξdx =0 · ΩZ

for all compact open subgroups Ωy Ω X. 1 ⊆ ⊂ By assumption (iii), y− xy C, and hence we may rewrite (2.6) as ∈ 1 1 π(y) π(x)π(x− y− xy)ξdx =0,

ΩZ i.e.

1 1 (2.7) χ(x− y− xy)π(x)ξdx =0.

ΩZ 1 1 By assumption (v), x χ(x− y− xy) is an arbitrary character of X,asyvaries in Y , and hence (2.7) means7→ that ξ has zero image in all possible Jacquet modules of JC,χ(π) with respect to X (and an arbitrary character). This implies that ξ =0.

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 868 DAVID GINZBURG, STEPHEN RALLIS, AND DAVID SOUDRY

We now show that A cA (2.8) IndD JD,χD (π) = IndD JD,χD (π). Let f be a function in (the space of) the l.h.s. of (2.8). It is determined by its values on Y .Sincefis A-smooth, there is in X a small compact open subgroup R, such that f(y x)=f(y) y Y, x R. · ∀ ∈ ∈ We have 1 1 1 1 1 1 f (yx)=f(x (x− yx)) = f(x− yx)=f(x− yxy− y)=χ(x− yxy− )f(y) . · We may take R =exp where is a small neighborhood of zero. Then, for y =expS,S ,wegetO O ∈Y 1 f(exp S)=ψ− ((E,S))f(exp S) , E . ∀ ∈O This implies that the function y f(y) is compactly supported on Y . The lemma is proved. 7→

2.3. Proof of the tower property. It is convenient to ﬁrst perform conjugation by

Ip I2n k  −  βp = I2(k p) . −  I2n k  −   Ip    Let   1 V = βp jk(Rp k)Nk β− . · ·H +1 p Extend χk trivially to Rp and k, and put, for v V H ∈ 1 αk(v)=χk(βp− vβp) .

1 k Denote by πτ,ψ the representation of βpjk(Sp2k(F ) k)Nk+1βp− in Vπτ S(F ) deﬁned by ·H ⊗ 1 f1 πτ,ψ(βpuβ− )(ξ φ)=πτ(βpuβ− )ξ φ, u Nk , p ⊗ p ⊗ ∈ +1 1 1 (k) πτ,ψ(βpjk(r h)β− )(ξ φ)=πτ(βpjk(r h)β− )ξ ω (r h)φ, · p ⊗ · p ⊗ ψ · (2.9) r Sp (F ),h k. ∈ 2k ∈H Then we have a vector space isomorphism (induced by ξ φ πτ (βp)ξ φ) f ⊗ 7→ ⊗ 1 JRp (Vσ (τ)) = J (Vπτ,ψ ). k ∼ V,αk−

The elements of V have the formb

Ip 0 xby ez a c b0   (2.10) v = I2(k p) a0 x0 Sp4n(F ) − ∈  z∗ 0     e0 Ip    

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use CUSPIDAL REPRESENTATIONS OF GL2n AND Sp2n 869

f where z Z2n k. We have in the notation (2.10) ∈ − Ip xy (k) (2.11) πτ,ψ(v)(ξ φ)=πτ(v) ωψ I2(k p) x0 ⊗ ⊗  −  Ip

(e2n k,a2n k,(b0)2n k;c2n k,1))φ. · − − − − p Here e2n k is the last row of e,etc.Notethate2n k and (b0)2n k are in F and − 2(k p) − − a2n k is in F − . Finally − αk(v)=ψ(z12 + z23 + +z2n k 1,2n k). ··· − − − Let V˜ (1) be the subgroup of V which consists of elements of the form (2.10), with (1) e2n k = 0, and let be its “complement” in V , − L Ip 0 I2n k 1  ` −01−  (1) ˆ (2.12) = ` =  I2(k p)  Sp4n(F ) . L  −  ∈ (  1  )    0 I2n k 1   − −   `0 0 Ip    (1) (1)  (1) Note that the complement normalizes V˜ and preserves αk .Thus, L V˜ (1) L k p k p k 1 acts on JV˜ (1),α (Vπτ,ψ ). Write F as F F − and denote for φ S(F ) k− ⊕ ∈ kp φ (t)=φ(0,t) ,tF−. (2) ∈ Deﬁne k k p T : Vπ S(F ) V π S ( F − ) τ ⊗ −→ τ ⊗ by T (ξ φ)=ξ φ . ⊗ ⊗ (2) Then, for v V˜ (1), ∈ (k p) (2.13) T (πτ,ψ(v)(ξ φ)) = πτ (v)ξ ωψ − (ik p(v))φ(2) ⊗ ⊗ − (1) where ik p : V˜ k p is the homomorphism (in the notation (2.10)) − −→ H − ik p(v)=(a2n k;c2n k,1). − − − (k) (k p) For (2.13) we use the formulae for ωψ (and ωψ − ). Denote by πτ,ψ0 the represen- ˜ (1) k p tation of V on Vπ S(F − ), deﬁned by the r.h.s. of (2.13). Then T intertwines τ ⊗ πτ,ψ and πτ,ψ0 . V˜ (1)

˜ (1) Proposition 1. We have an isomorphism of V -modules

1 1 (2.14) J (Vπτ,ψ ) = J˜ (1) (Vπ0 ). V,αk− ∼ V ,αk− τ,ψ Proof. By passage to the quotients, T deﬁnes a surjection

1 1 1 T 0 : JV˜ (1),α (Vπ ,α ) J V˜ (1),α (Vπ0 ) k− τ,ψ k− −→ k− τ,ψ which, in turn, induces a V -map V i : J 1 (V ) Ind (J 1 (V )) V˜ (1),α− πτ,ψ V˜ (1) V˜ (1),α− π0 k −→ k τ,ψ

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 870 DAVID GINZBURG, STEPHEN RALLIS, AND DAVID SOUDRY

determined by (using notation 2.12)) ˆ ˆ (k) (2.15) i jV˜ (1),α 1 (ξ φ) (`)=jV˜(1),α 1 (πτ (`) [ωψ (`, 0, 0; 0)φ](2)) k− ⊗ k− ⊗ h i for `ˆ (1).Since ∈L (k) p k p ω (`, 0, 0; 0)φ(0,t)=φ(`, t) , for ` F ,t F − , ψ ∈ ∈ cV (2.15) implies that Im(i) Ind ˜ (1) (J ˜ (1) 1 (Vπ )). Let us show that i is in- V V ,αk− τ,ψ0 ⊆ k jective. For this, it is convenient to identify Vπτ,ψ with S(F ; Vπτ )—the space of V -valued Schwartz Bruhat functions on F k. Similarly, we identify V with πτ πτ,ψ0 k p S(F − ; Vπτ ). Let

ik : V R p k −→ H be the homomorphism (in the notation (2.10)) e Ip xy ik(v)= I2(k p) x0 (e2n k,a2n k,(b0)2n k;c2n k,1) .  −  · − − − − Ip e  k  Then the action of V on S(F ; Vπτ )isgivenby (k) (k) (2.16) πτ,ψ(v)(φ)=πτ(v) (ω (ik(v))φ)=ω (ik(v))(πτ (v) φ). ◦ ψ ψ ◦ (k) (The Weil representation ωψ acts on vector-valuede functionse by the same formulae ˜ (1) k p as for scalar-valued functions.) Similarly V acts on S(F − ; Vπτ )by (k p) (k p) (2.17) πτ,ψ0 (v)(φ)=πτ(v) (ωψ− (ik p(v)φ)) = ωψ − (ik p(v))(πτ (v) φ). ◦ − − ◦ k 1 Let φ S(F ; Vπτ ) be such that its class in JV˜ (1),α (Vπτ,ψ ) belongs to Ker(i). This ∈ k− (1) (`) ˆ ˆ 1 means that the function (on ) ` πτ (`) φ has zero image in JV˜ (1),α (Vπ0 ), L 7→ ◦ k− τ,ψ (`) (k) p where φ = ωψ (`, 0, 0; 0)φ .Thus,foreach` F, there is a compact open (2) ∈ (1) subgroup C` h V˜ , such thati ⊂ (k p) ˆ (`) (2.18) αk(v)πτ (v) ωψ − (ik p(v))(πτ (`) φ )dv =0 ◦ − ◦ CZ (1) for all compact open subgroups C` C V˜ . By (2.16)-(2.18) we get ⊆ ⊂ (k) k p (2.19) αk(v)πτ (v `ˆ) ω (ik(v `ˆ))φ(0,t) dv =0, t F − . · ψ · ∀ ∈ CZ h i (1) e Recall that `ˆ normalizes V˜ and preserves αk. Hence, changing variables v ˆ ˆ 1 7→ `v `− in (2.19) gives · (k) k p (2.20) αk(v)πτ (v) ω (ik(v))φ(`, t) dv =0, t F − , ψ ∀ ∈ CZ0 h i e ˆ 1 ˆ (1) for all compact open subgroups `− C`` C0 V . Using the formulae for (k) ⊆ ⊂ (k) ωψ (ik(v)), it is easy to see that the function ` ωψ (ik(v))φ(`, t) is uniformly smooth and has support contained in a compact7→ set,e independently of either v ∈ e e

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use CUSPIDAL REPRESENTATIONS OF GL2n AND Sp2n 871

˜ (1) k p f V or t F − . It then follows from (2.20) that there is a compact open subgroup ˜ (1)∈ C0 V , such that 0 ⊂ (k) χk(v)πτ (v) ω (ik(v))φdv =0 ◦ ψ CZ0 e ˜ (1) for all compact open subgroups C0 C0 V . This means that φ has zero image 0 ⊆ ⊂ 1 in J ˜ (1) (Vπτ,ψ ), and we proved that i is injective. Thus we have an injection of V ,αk− V -modules

cV i : J 1 (V ) , Ind (J 1 (V )), V˜ (1),α− πτ,ψ V˜ (1) V˜ (1),α− π0 k → k τ,ψ and hence, taking Jacquet modules with respect to (1), we get an embedding of V˜ (1)-modules L

cV J 1 (V ) , J (1) Ind (J 1 (V )) V,α− πτ,ψ V˜ (1) V˜ (1),α− π0 k → L k τ,ψ = J ˜ (1) 1 (Vπ ) . ∼ V ,αk− τ,ψ0 This last embedding is easily seen to be surjective, and this proves the proposition.

Lemma 2.2 will now serve as our apparatus which translates the Fourier expansion arguments of the proof of Theorem 8 in [G.R.S.1] to the local case. Redenote B(2) = V˜ (1).LetC(2) be the subgroup of B(2), which consists of elements of the form (2.10), such that e2n k 1 = e2n k =0.Let − − −

Ip 0 I2n k 2  y −01−  (2) Y = y =  I2(k p+1)  Sp4n(F ) ,  −  ∈ (  1  )    0 I2n k 2  b  − −   y0 0 Ip     Ip 0 x I2n k 1 0  − − 1  (2) X = xˇ =  I2(k p) Sp4n(F ) ,  − ∈ (  10x0 )    I2nk10  −−   Ip     D(2) = C(2) X(2) ,A(2) = D(2) Y (2), · ·

(2) 1 χ = αk− .

C(2)

We consider V as an A(2)-module by letting X(2) act through π only, i.e.x ˇ in πτ,ψ0 τ (2) k p X takes the class of ξ φ Vπ S(F − )totheclassofπτ(ˇx)ξ φ.Itiseasy ⊗ ∈ τ ⊗ ⊗

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 872 DAVID GINZBURG, STEPHEN RALLIS, AND DAVID SOUDRY

to check that the diamond A(2)

X(2) Y(2) ,- B(2) D(2)

Y (2) X(2) -, C(2)

satisﬁes assumptions (i)-(v) of Section 2.2, and hence by Lemma 2.2, we have

1 1 J (Vπτ,ψ ) = J˜ (1) (Vπ )=J (2) (2) (Vπ ) = J (2) (2) (Vπ ) . V,αk− ∼ V ,αk− τψ0 B ,χ τ,ψ0 ∼ D ,χ τ,ψ0 B(2) D(2) We repeat this process by taking B(3) = D(2), “deleting one row and adding one column”. In general, for 2 j 2n k,letC(j) be the group of elements of ≤ ≤ − Sp4n(F ) which have the form

Ip 0 u ∗∗∗∗ ez1 b  00z ∗∗∗∗ 2 ∗∗∗∗ (2.21) v =  I2(k p)   − ∗∗∗  z∗v0 u0  2 ∗   z∗ 0  1   e0 Ip     where z1 Z2n k j, z2 Zj and the ﬁrst two columns of u are zero. Let ∈ − − ∈

Ip 0 I2n k j  y −01−  (j) Y = y =  I2(k p+j 1)  Sp4n(F ) ,  − −  ∈ (  1  )    0 I2n k j  b  − −   y0 0 Ip     Ip 0 x I2n k j+1 0  − − 1  (j) X = xˇ =  I2(k p+j 2)  Sp4n(F ) ,  − −  ∈ (  10x0 )    I2nkj+1 0   −−   Ip     D(j) = C(j)X(j) ,B(j)=C(j)Y(j),A(j)=D(j)Y(j).

Finally let χ(j) be the character of C(j) deﬁned by (using the notation (2.21))

(j) 1 χ (v)=ψ− (z12 + z23 + +z2n k 1,2n k) ··· − − −

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use CUSPIDAL REPRESENTATIONS OF GL2n AND Sp2n 873

f z1 b where (z12,... ,z2n k 1,2n k) is the second diagonal of .Itiseasyto − − − 0 z 2 check that χ(j) and the diamond A(j)

X(j) Y(j) ,- B(j) D(j)

Y(j) X(j) -, C(j) satisfy (i)-(v) in Section 2.2. Note that (j) (j 1) B = D − and (j 1) (j) (2.22) χ −(j 1) = χ . D − C(j) (j) We let, for each j, X act on Vπ , through πτ only, and in this way Vπ becomes τ,ψ0 τ,ψ0 an A(j)-module for each j. We conclude from Lemma 2.2 that

J (j) (V ) = J (j) (V ) B(j),χ πτ,ψ0 D(j),χ πτ,ψ0 B(j) ∼ D(j) for all 2 j 2n k, and hence ≤ ≤ − (2.23) J 1(V ) = J (2n k) (V ) V,α− πτ,ψ (2n k) − πτ,ψ0 k ∼ D − ,χ (2n k) D − (this isomorphism is over the subgroup C of V which consists of elements of the form (2.2), such that e = 0). Note that

Ip u ∗∗∗ z z Z2n k (2n k)  ∗∗∗ ∈ − D − = I2(k p) Sp4n(F ) and the ﬁrst Nk p − ∗∗∈ ⊂ − (  z∗u0 column of u is zero)    Ip     and (2n k) 1 χ (2−n k) = χ− . D − k p (2n k) − D −

Consider the r.h.s. of (2.23) as an E-module, where

mx 1   E = I2(2n p 1) Sp4n(F ) . − − ∈ (  1 x0  )    m∗     E is isomorphic to the parabolic subgroup of GLp+1(F )oftype(p, 1) (the so- called mirabolic subgroup). By [B.Z.], the Jordan H¨older decomposition over E of

J (2n k) 1 (π0 ) is expressed through the various derivatives, which all, except D − ,χτ,ψ− τ,ψ one, clearly involve the Jacquet modules of πτ with respect to unipotent radicals

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 874 DAVID GINZBURG, STEPHEN RALLIS, AND DAVID SOUDRY

Ip ` − ∗∗ of the form I2(2n p+`) ,for0 `

Remark 1. The analysis done so far in this section is valid if we replace πτ by any smooth representation of Sp4n(F ). In particular (2.23) is true if we replace πτ by any smooth representation of Sp4n(F ). Now we use the supercuspidality of τ and the fact that πτ is the Langlands quotient of ρτ,1. It follows that the above derivatives vanish, since πτ is concentrated on the Siegel parabolic subgroup. (Note that for 0 `

JZ ,ψ 1 J (2n k) 1 (πτ,ψ0 ) = σk p(τ). p+1 − D − ,χk− p − − This together with (2.23) completes the proof of Theorem 2.1. b Remark 2. The argument above is valid if we replace πτ by ρτ,s or any constituent of ρτ,s,aslongasτis supercuspidal. To end this section, we would like to write the last two remarks (Remarks 1, 2) as two explicit propositions.

Let π be a smooth representation of Sp4n(F ). Consider, for 1 p k< 1 k ≤ ≤ 2n, the representation πψ of βpjk(Sp2k(F ) k)Nk+1βp− in Vπ S(F ), defined by H ⊗ (2n k) formulae (2.1), where we replace πτ by π.Letπψ∗ be the representation of D − k p in Vπ S(F − ) defined by f ⊗ (k p) π∗ (u)(ξ φ)=π(u)ξ ω − (u0)φ ψ ⊗ ⊗ ψ (2n k) where u u0 is the projection of D − on k p, defined by 7→ H − u0 =(u2n k+p,2n k+p+1,... ,u2n k+p,2n+k p+1) . − − − − Remark 1 says

Proposition 2. Let π be a smooth representation of Sp4n(F ).For1 p k<2n, (2n k) ≤ ≤ deﬁne V , D − and αk as before. Then we have a vector space isomorphism

1 1 J (Vπψ ) = J (2n k) (Vπ∗ ) . V,αk− ∼ D − ,χk− p ψ − The precise content of Remark 2 is

τ Proposition 3. Let τ be a supercuspidal representation of GL2n(F ).Letπ be a subquotient of ρ 1 . Then in the above notation, we have a vector space isomorphism τ, 2 cQk p J V Ind − V . Rp σk,πτ = Nk p σk p,πτ ∼ − − b b 3. Vanishing of the representations σk(τ) k

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use CUSPIDAL REPRESENTATIONS OF GL2n AND Sp2n 875

f 3.1. The case k =0. In this case N0 is the standard maximal unipotent subgroup of Sp4n(F )andχ0 is its standard nondegenerate character, which corresponds to ψ.Thusσ0(τ)=J 1 (πτ ). Since πτ is not generic (with respect to any N0,χ0− nondegenerate character of N0), we conclude that b σ0(τ)=0.

3.2. A reduction. Let C denote the center of the Heisenberg group k, and let b H (k) N = Nk jk(C). +1 · (k) This is a subgroup of Nk = Nk+1 jk( k). Let χ(k) be the character of N deﬁned by · H

(3.1) χ (u jk(0, 0; t)) = χk(u)ψ(t) . (k) · We have the following analog of Lemma 15 in [G.R.S.1].

Lemma. For a smooth representation π of Sp4n(F ), we have σk,π =0,ifandonly if b (3.2) J (k) 1 (π)=0. N ,χ(−k)

(k) 1 Proof. Assume that σk,π =0.Thus,J k JN ,χ (π) ωψ =0.Letbbe a 6 H k+1 k− ⊗ 6 nontrivial element of the dual of the last space. (We sometimes confuse a representation and its space.)b We regard b as an k-invariant bilinear form b(v, φ)on k H J 1 (Vπ) S(F ). For ﬁxed v, bv(φ)=b(v, φ) is a smooth distribution, and Nk+1,χk− ⊗ k hence there is a unique smooth function fv(z)onF , such that

b(v, φ)= φ(z)fv(z)dz .

FZk

The k-equivariance of b implies that H 1 t f (z)=ψ− (2zwk y + t)fv(z + u) . π(jk (u,0;0)jk(0,y;t))v · Here π denotes the action on the Jacquet module. Thus, fv, which is nontrivial, is fully determined by one value, as v varies. Choose fv(0). The linear form `(v)=fv(0) is nontrivial and satisﬁes

1 ` π(jk(0,y;t))v = ψ− (t)`(v).

We showed that b is uniquely determined by `, which is an element of the dual of

J ˜ (k) 1 (π), where N ,χ˜(−k)

(k) N˜ = Nk jk( k C) +1 Y · (k) and χ is the character of N˜ obtained by extending χ trivially to jk( k). (k) (k) Y We actually showed a vector space isomorphism Vσ∗ = J ˜ (k) 1 (Vπ)∗. In par- ˆk,π ∼ N ,χ˜(−k) e (k) ticular, σk,π = 0, if and only if JN˜ (k) ,χ 1 (π) =0.SinceN is a subgroup of 6 (−k) 6 (k) N˜ , it is now clear that if σ =0,thenJ 1 (π) = 0. Assume now that k,π N(k),χ− b 6 (k) 6 J ˜ (k) 1 (π)=0.IfJ (k) 1 (π) = 0, then, since k C is abelian, the (abelian) N ,χ˜k− N ,χ(−k) b 6 Y ·

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 876 DAVID GINZBURG, STEPHEN RALLIS, AND DAVID SOUDRY

group jk( k)actsonJN(k),χ 1 (Vπ), and hence there is a character χ0 of jk( k), Y k− Y 1 such that Jjk( k),χ0 (JN (k),χ (Vπ)) =0. χ0 has the form Y (−k) 6 k 1 χ0(jk(0,y;0))=ψ− ( xiyi) , i=1 X for x ,... ,xk F.Letϕbe a nontrivial linear functional on Vπ, such that 1 ∈ 1 (k) ϕ(π(v0v)ξ)=χ0(v0)χ− (v)ϕ(ξ), for v0 jk( k),v N . (k) ∈ Y ∈ 1 Deﬁne u = (xk,... ,x ,x )and − 2 2 1 ϕ0(ξ)=ϕ(π(jk(u, 0; 0))ξ) .

Then ϕ0 is a nontrivial linear functional on Vπ, such that 1 (k) ϕ0(π(v)ξ)=˜χ− (v)ϕ0(ξ), for v N˜ , (k) ∈

which implies that J 1 (π) = 0, a contradiction. N˜ (k),χ˜− (k) 6

3.3. Sp2n(F ) Sp2n(F )-invariant functionals. We recall (in Theorem 1) Theo- rem 17 in [G.R.S.1].× Let H be the image of the direct sum embedding of Sp (F ) 2n × Sp2n(F ) inside Sp4n(F ).

Theorem 1. Let π be an irreducible, representation of Sp4n(F ). Assume that the space of π admits nontrivial H-invariant functionals. Then

JN (k),χ 1 (π)=0, for 0 k

σ˜k,π =0, for 0 k

Theorem 2. The representation πτ admits nontrivial H-invariant functionals. b Proof. Recall again Shahidi’s local coeﬃcient L(τ,Λ2,1 z) γ(τ,Λ2,z,ψ)=ε(τ,Λ2,z,ψ) − . L(τ,Λ2,z) Our assumption is that τ is self-dual, supercuspidalb and that L(τ,Λ2,z) has a pole at z = 0. Jacquet and Shalika wrote a global integral which represents the (partial) exterior square L-function for automorphic cuspidal representations on GL2n(A) ([J.S.2]). Their global theory yields a corresponding local theory, which centers around a functional equation, the details of which we now explain. It has the form (3.3) γ(τ,Λ2,s,ψ) (W, φ, s)= (W, φ, 1 s) L L − where γ(τ,Λ2,s,ψ) is a rational function in q s. (W, φ, s) is the “local integral” − e b which depends on We, an element in the WhittakerL model of τ (which we take, e

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use CUSPIDAL REPRESENTATIONS OF GL2n AND Sp2n 877

f 1 for simplicity, with respect to the standard character, deﬁned by ψ− ), and on φ S(F n), which deﬁnes the following section for IndGLn(F ) α : Pn 1,1 s ∈ − s ns (3.4) fφ,s(g)= det g φ(t(0 01)g) t ωτ (t)d∗t. | | ··· | | FZ∗ 2 Here ωτ is the central character of τ(ωτ =1),Pn 1,1 is the parabolic subgroup of − GLn(F )oftype(n 1,1) and − s 1 m det m 2 α = − ω 1(a) ,mGL (F ),a F . s ∗ | n 1| τ− n 1 ∗ 0 a a − ∈ − ∈ | | Finally, (3.5)

(W, φ, s) L In X g = W µ ψ(trX)fφ,s(g)dXdg · In g ! C Z ZGL (F ) X b ZM (F ) n n\ n ∈ n\ n where bn is the space of n n upper triangular matrices, Cn is the center of GLn(F ) and µ is the Weyl element× deﬁned by

µ2i 1,i = µ2i,n+i =1,i=1,... ,n, − µ`,j =0 for (`, j) / (2i 1,i),(2i, n + i) i =1,... ,n . ∈{ − | } s The integral (3.5) converges absolutely for Re(s) 0 and is rational in q− .The r.h.s. of (3.3) has a similar structure

(3.6) (W, φ,ˆ 1 s) L − t 1 In X wn g− e= W µ t 1 In wn g− ! C Z ZGL (F ) X b ZM (F ) n n\ n ∈ n\ n ψ(trX)fφ,ˆ 1 s(g)dXdg . · − Hence φ is the Fourier transform of φ. The proof of (3.3) can be inferred from the corresponding Euler product expansion ofb the global integral of [J.S.2], as we do in Sections 6.1, 6.2. Exact details will appear in a forthcoming paper by J. Cogdell and I. Piatetski-Shapiro. Both γ(τ,Λ2,s,ψ)andγ(τ,Λ2,s,ψ) should be equal (at least up to a monomial). We show this in Section 6.3. In particular, γ(τ,Λ2,s,ψ) has a zero at s = 0 if and 2 only if γe(τ,Λ ,s,ψ) does. Consider then the functional equation (3.3) at s =0. Now, exactly the same proof as that of Proposition 1 in Section 7.1 of [J.S.2] shows that (W,e φ, 1 s) is holomorphic at s =0.Thisimpliesthat (W, φ, s) has a pole at s =L 0. Again,− a close look at the proof of Proposition 1 inL Section 7.1 of [J.S.2] showse thatb since (by our assumption of supercuspidality of τ) W has compact

In x g support modulo C2nZ2n, the function (x, g) W µ has 7→ · In g ! compact support in b Mn(F ) C Zn GLn(F ). This shows that the integral n\ × · \ (3.5) is absolutely convergent and holomorphic, in this case, as long as fφ,s is

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 878 DAVID GINZBURG, STEPHEN RALLIS, AND DAVID SOUDRY

holomorphic. Thus, if (W, φ, s) has a pole at s =0,fφ,s must have a pole at L s = 0. Note that (3.4) is just a Tate integral. We conclude that ωτ =1(!)and (3.7)

0 Ress (W, φ, s) 6≡ =0 L In X g = W µ ψ(trX)Ress=0 fφ,s(g)dXdg In g ! C Z ZGL (F ) b MZ (F ) n n\ n n\ n I X g = c φ(0) W µ n ψ(trX)dXdg · In g C Z ZGL (F ) b MZ (F ) n n\ n n\ n for some constant c. The last integral in (3.7) deﬁnes a Shalika functional on τ, i.e. a nontrivial linear functional λ on Vτ such that

In x g 1 λ τ v = ψ− (trX)λ(v) . In g For all x M n (F ), g GLn(F )andv Vτ. By Section 6.1 in [J.R.], it follows that ∈ ∈ ∈ Vτ admits a nontrivial GLn(F ) GLn(F )-invariant functional, where GLn(F ) × × g1 GLn(F )isembeddedby(g1,g2) . Choose such a functional ` (for 7→ g2 example, I0(v, 0) on p. 117 in [J.R.]). It deﬁnes an H-map

Sp2n(F ) Sp2n(F ) 1/2 (3.8) T` : ρτ,1 IndP P × δP P (normalized induction) −→ n× n n× n Sp4n(F ) 1/2 as follows. Think of the elements of ρτ,1 =Ind τ det as Vτ -valued P2n ⊗| ·| functions on Sp4n(F ). For a function f in the space of ρτ,1, deﬁne, for (g1,g2) Sp (F ) Sp (F ), ∈ 2n × 2n T`(f)(g1,g2)=`[f(g1,g2)] . We identify Sp (F ) Sp (F ) and its image H in Sp (F ). We have 2n × 2n 4n ax by n+1 T`(f) g1, g2 =det ab T`(f)(g1,g2) . 0a∗ 0b∗ | | ! The representation on the r.h.s. of (3.8) has the identity representation (of H)as its quotient. The composition of this quotient map and T` provides a nontrivial H-invariant form b on ρτ,1. Recall that ρτ,1 has two constituents: one irreducible subrepresentation Wτ , which is generic, and one irreducible quotient, which is πτ . The case k = 0 of Theorem 1 (in this section) shows that b(Wτ ) = 0, and hence b deﬁnes a nontrivial H-invariant form on the quotient Wτ ρτ,1 = πτ . We conclude from Theorems 3.3.1, 3.3.2 and Lemma 3.2\ that

σk(τ)=0 for 0 k

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use CUSPIDAL REPRESENTATIONS OF GL2n AND Sp2n 879

f 4. Irreducibility of σn(τ) In this section, we complete the proof of Theorem 1.5 (our main local theorem) and show the irreducibility of σn(τ). We assumeb that τ is irreducible, self-dual, and supercuspidal, such that L(τ,Λ2,s) has a pole at s =0. b 4.1. The group E2n. Let E2n denote the subgroup of Sp4n(F ), which consists of elements of the form:

I2 z1 I z  2 2 ∗∗ . . . .. ∗ . .      I2 zn 1   −   I2 y   ···  (4.1)  I2 zn0 1   − ∗  I2     ..   . ∗    z0   2   I2 z0   1   I2      Deﬁne

(2n) ψ (e)=ψ(tr(z1 + z2 + +zn 1)+y12 y21) . ··· − −

This is a character of E2n.

4.2. Main steps of the proof. Let U2n be the unipotent radical of (the Siegel parabolic subgroup) P2n.Wehave

n (4.2) JU (πτ) τ det 2n ' ⊗| ·|

as GL2n(F )-modules. In particular J (πτ ) is one-dimensional, where V2n is the V2n,ψ standard maximal unipotent subgroup of Sp (F )andψis trivial on U n and is e 4n 2 any nondegenerate character on m(Z2n). We will prove e Theorem 1. We have a vector space isomorphism

(4.3) JE ,ψ(2n) (Vπτ ) = J (Vπτ ) 2n ∼ V2n,ψ and in particular e

(2n) (4.4) dim JE2n,ψ (Vπτ )=1.

Theorem 2. We have a vector space isomorphism

1 (4.5) JE ,ψ(2n) (Vπτ ) = JVn,ψn (J ˜ (n) (Vπτ )). 2n ∼ N ,χ˜(−n)

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 880 DAVID GINZBURG, STEPHEN RALLIS, AND DAVID SOUDRY

Recall that Vn is the standard maximal unipotent subgroup of Sp2n(F )andψn is its standard nondegenerate character (1.1). Let us show how the last two theorems imply the irreducibility of σn(τ). We already know that σn(τ) is nontrivial and supercuspidal. Thus, σn(τ) is a direct

sum of irreducible (supercuspidal) representations of Sp2n(F ). (A supercuspidalb representation is semisimple.)b b f Proposition. Each summand σ of σn(τ) is ψn-generic, i.e. JV ,ψ (σ) =0. n n 6 Proof. We have an embedding b (n) Bil (σ, σ (τ)) Bil σ, J J 1 (ρ ) ω . F n F n N ,χ− τ,1 ψ Sp2n( ) −→ Sp2n( ) H n+1 n ⊗ Thus, the lastf space is nontrivial. Theoremf 6.2(c) in the Appendix implies that σ 1 b b b must be ψn− -generic, and hence σ is ψn-generic. To conclude the irreducibility, we have b

Theorem 3. The representation σn(τ) has a unique ψn-Whittaker model, i.e.

dim JVn,ψn (σn(τ)) = 1 . Proof. The proof of Lemma 3.2 canb be used to show that for a smooth representa-

tion π of Sp2n(F ) b ∗ ∗ 1 (4.6) JVk,ψk (Vσk,π ) = JVk,ψk (J ˜ (k) (Vπ)) . ∼ N ,χ˜(−k) h i h i Thus, for π = πτ and k = n,b

1 dim JVn,ψn (Vσn(τ))=dimJVn,ψn (J ˜ (n) (Vπτ )) = 1 , N ,χ˜(−n)

by (4.4) and (4.5). b 4.3. Proof of Theorem 4.2.1. We will prove the following more general theorem. Let H be the image of the direct sum embedding of Sp (F ) Sp (F ) inside 2n × 2n Sp4n(F ).

Theorem. Let π be an irreducible representation of Sp4n(F ). Assume that Vπ admits nontrivial H-invariant functionals. Then we have a vector space isomorphism

J (2n) (Vπ ) = J ˜(Vπ ) . E2n,ψ ∼ V2n,ψ Note that by Theorem 3.2 the last isomorphism is valid for π = πτ , and this will prove Theorem 4.2.1. We now start with the proof. As we have done in Section 2.3, the map ξ π(a)ξ deﬁnes an isomorphism (of vector spaces) 7→

(4.7) JE ,ψ(2n) (Vπ ) = J a (2n) (Vπ ) 2n ∼ (E2n) ,ψa a 1 (2n) (2n) 1 a where (E2n) = aE2na− and ψa (x)=ψ (a− xa), x (E2n) . We apply (4.7) twice. First, for ∈ b ..  .  b 1 1 (4.8) a =   , where b = − ,  b∗  11    .   ..     b   ∗  

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use CUSPIDAL REPRESENTATIONS OF GL2n AND Sp2n 881

a f we have (E n) = E n,andfore En (of the form (4.1)), 2 2 ∈ 2 (2n) ψa (e)=ψ(tr(z1 + +zn 1)+y1,1). ··· − Second, conjugate by the Weyl element ν deﬁned by

(4.9) νi,2i 1 =1, for i =1,...,2n, − ν n i, i = 1, for i =1,...,n, 2 + 2 − ν2n+i,2i =1, for i = n +1,...,2n,

otherwise,νi,j =0. 1 Up to signs, ν is µ− of Section 3.3 (with 4n replacing 2n). Denote 1 B = νE2nν− , 1 (2n) χ(νeν− )=ψa (e). Then

J (2n) (Vπ ) = JB,χ(Vπ) . E2n,ψ ∼

The subgroup B consists of those elements of Sp4n(F ) which have the form zx (4.10) v = yz0 where z,z0 Z2n and x and y are upper triangular and nilpotent. For v of the form (4.10),∈

(4.11) χ(v)=ψ(z12 + z23 + +zn,n+1 zn+1,n+2 z2n 1,2n). ··· − −···− − Our goal is to “fatten” x in (4.10), “using” y, by successive applications of Lemma 2.2 until we get from JB,χ to J . Let us introduce some notation. Let V2n,ψ

e I2n x = x M2n(F ) Sp4n(F ) . X { ∈ I2n ∈ } For a subspace S ,write`(S)= `(x) x S and `(S)= `(x) x S .Put ⊂X { | ∈ } { | ∈ } (4.12) = x x is nilpotent and upper triangular . X0 { ∈X | } An element in B can be written in the form (4.13) v = `(x)m(z)`(y)

where x, y and z Z n.Let ∈X0 ∈ 2 00 0 ∗ ··· ∗  .∗ ··· ∗ .. 1,2= x 0 x12(= x2n 1,2n)=0 = . Y { ∈X | − }    ∗∗  00    0   (1,2)   Let C be the subgroup of elements of the form (4.13) such that y 1,2. Thus ∈Y (1,2) C = `( )m(Z n)`( , ) . X0 2 Y1 2 Let Y (1,2) = `( 1,2) Y

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 882 DAVID GINZBURG, STEPHEN RALLIS, AND DAVID SOUDRY

where 0 t 0 0 ··· 1,2 . . . = F (e12 + e2n 1,2n)= . .. . tF. Y · −  |∈} n 0 t   00   Denote   X (1,1) = `( 1,1) X where t 0 0 0 ··· 1,1 . . . = F (e11 + e2n,2n)= . .. . t F . X · ( | ∈ )  0    0 t   Let   χ(1,2) = χ ,B(1,2) = B,D(1,2) = C(1,2)X(1,1) ,A(1,2) = D(1,2)Y (1,2) . C(1,2)

It is easy to check that χ(1,2) and the diamond

A(1,2)

X(1,1) Y(1,2) ,- B(1,2) D(1,2)

Y (1,2) X(1,1) -, C(1,2) satisfy assumptions (i)-(v) of Section 2.2. We conclude that

(4.14) JB(1,2),χ(1,2) (Vπ) = JD(1,2) ,χ(1,2) (Vπ) . B(1,2) ∼ D(1,2) Put 1,1 , = . X1 1 X0 ⊕X Then (1,2) D = `( , )m(Z n)`( , ). X1 1 2 Y1 2 (1,2) (1,2) χ is the character of D which is trivial on `( , ) `( , )andisχon D(1,2) X1 1 · Y1 2 m(Z n). Let 1 i

i,j = x 0 xr,` =0 for r, ` j 1, and xr,j =0 for r i , Y ( ∈X | ≤ − ≥ )

i,j (4.16) = F (ei,j + e2n j+1,2n i+1) . Y − −

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use CUSPIDAL REPRESENTATIONS OF GL2n AND Sp2n 883

i,j f (Note that if i + j =2n+1,then = Fei,j.) Deﬁne for 1 s r 2n Y ≤ ≤ ≤ r,s (4.17) = F (er,s + e2n+1 s,2n+1 r) X − − and for 1 s r n ≤ ≤ ≤ r `,q r,q (4.18) r,s = ( ) ( ). X X0 ⊕ X ⊕ X q ` r 1 q=s ≤M≤ − M For example, for n =4

∗∗∗∗∗∗∗∗ 0∗∗∗∗∗∗∗∗ ∗∗∗∗∗∗∗ 0000  3,2 = ∗∗∗∗. X 00000   ∗∗∗ 00000   ∗∗∗ 00000   ∗∗∗ 000000   ∗∗ Let, for 1 i

X (i,j) Y(i,j) ,- B(i,j) D(i,j)

(i,j) (j1,i) Y X− -, C(i,j) satisfy assumptions (i)-(v) of Section 2.2, and hence

(4.22) J (i,j) (V ) = J (i,j) (V ) B(i,j) ,χ π ∼ D(i,j) ,χ π B(i,j) D(i,j) for all 1 i

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 884 DAVID GINZBURG, STEPHEN RALLIS, AND DAVID SOUDRY

We conclude that

(4.23) J (1,2) (V ) = J (1,n+1) (V ). B ,χ π ∼ D(1,n+1),χ π D(1,n+1) Note that

(1,n+1) D = `( ,n)m(Z n)`( ,n ). X1 2 Y1 +1 In case n =4,

00000 ∗∗∗∗∗∗∗∗ 00000∗∗∗ ∗∗∗∗∗∗∗∗ 00000∗∗∗ ∗∗∗∗∗∗∗∗ ∗∗∗   00000000 1,n = ∗∗∗∗∗∗∗∗ , 1,n+1 =  . X 0000  Y 00000000  ∗∗∗∗   0000  00000000  ∗∗∗∗   0000  00000000  ∗∗∗∗   0000  00000000  ∗∗∗∗       Note that, so far in this proof, we did not use any particular property of Vπ. Deﬁne for n +1 r 2nand 1 s 2n +1 r ≤ ≤ ≤ ≤ − 2n+1 r `,q − r,q (4.24) r,s = ,n ( χ ) . X X1 ⊕ ⊕ X n+1 ` r 1 q=s 1 q M≤2n≤+1− ` M ≤ ≤ − In case n =4,

∗∗∗∗∗∗∗∗ ∗∗∗∗∗∗∗∗ ∗∗∗∗∗∗∗∗   6,2 =∗∗∗∗∗∗∗∗. X   ∗∗∗∗∗∗∗∗ 0   ∗∗∗∗∗∗∗ 00   ∗∗∗∗∗∗ 000   ∗∗∗∗∗   Consider the action of X(n+1,n) on the r.h.s of (4.23). Note that X(n+1,n) is the

center of n 1 (embedded in Sp4n(F )). We have, for any nontrivial character ξ of X(n+1,n),H −

(4.25) JX(n+1,n),ξ(JD(1,n+1),χ(1,n+1) (Vπ)) = 0. D(1,n+1)

(1,n+1) (n+1,n) (n 1) (1,n+1) Indeed, D X N − (see Section 3.2) and χn 1 = χ (1,n+1) ξ D (n 1) ⊃ − · N − is a character of N (n 1) of the same type of χ (in the sense of the remark − (n 1) following Theorem 3.3.1). Hence there is a surjection− e

(4.26) J (n 1) (V ) J (1,n+1) (V ). N − ,χn 1 π X ( n +1,n)D(1,n+1),ξ χ π − −→ · D(1,n+1) But the Jacquet modulee on the l.h.s. of (4.26) is zero by Section 3, and hence (4.25) (n+1,n) follows. We conclude that X acts trivially on JD(1,n+1),χ(1,n+1) (Vπ). Deﬁne D(1,n+1)

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use CUSPIDAL REPRESENTATIONS OF GL2n AND Sp2n 885

(n 1,n+2) (1,n+1) (n+1,n) (1,n+1) (nf 1,n+2) B − = D X and extend χD(1,n+1) to B − by making it (n+1,n) (n 1,n+2) trivial on X . Denote the resulting character by χ (−n 1,n+2) .Thus,wehave B −

(4.27) J (n 1,n+2) (V ) J (1,n+1) (V ). (n 1,n+2) − π = (1,n+1) π B − ,χ (n 1,n+2) ∼ D ,χ (1,n+1) B − D n 1 i,n+2 Now, we can continue as before, “replacing the n 1 coordinates of − − i=1 Y into n+1,1”. Deﬁne as before, for 1 i n 1, j n +2, X ≤ ≤ − ≥ L (i,j) (i,j) (i,j) (i,j) C = `( j 1,i+1)m(Z2n)`( i,j ),B = C Y , X − Y (i,j) (i,j) (j 1,i) (i,j) (i,j) (i,j) (4.28) D = C X − ,A = D Y .

(i,n+2) (i,n+2) Let χ be the character of C which is trivial on `( n ,i )`( i,n ) X +1 +1 Y +2 and is χ on m(Z2n). We are again at the situation of Lemma 2.2, and we conclude from (4.27), after n 1 successive applications, that −

(4.29) JD(1,n+1),χ(1,n+1) (Vπ) = JD(1,n+2),χ(1,n+2) (Vπ) . D(1,n+1) ∼ D(1,n+2) Next, we repeat the previous argument, using the assumption on π, Theorem 3.3.1 (k = n 2) and the remark which follows, and Lemma 3.2 to show that (n+2,n 1) − X − acts trivially on the r.h.s. of (4.29), and then proceed as before, using C(i,n+3),B(i,n+3),D(i,n+3),A(i,n+3) for i n 2, and so on. Note that in these stages of the proof, we use as above the assumption≤ − on π, Theorem 3.3.1 and Lemma 3.2, for 0 k n 1. We get ≤ ≤ − (4.30) J (1,n+2) (V ) = =J (1,2n) (V ). D(1,n+2),χ π ∼ ∼ D(1,2n) ,χ π D(1,n+2) ··· D(1,2n)

(1,2n) (1,2n) Note that D = `( 2n 1,1)m(Z2n)andχ (1,2n) is the character which is trivial − D X (2n,2n) (1,2n) on `( 2n 1,1)andχon m(Z2n). Note that X D = V2n, the standard X − (2n,2n) maximal unipotent subgroup of Sp4n(F ). As before, X acts trivially on JD(1,2n),χ(1,2n) (Vπ ), since π is nongeneric with respect to any nondegenerate char- D(1,2n) acter of V2n. Again, this follows from Theorem 3.3.1 and the remark which follows. We conclude that

J (1,2n) (Vπ ) = J (Vπ ) . D(1,2n) ,χ V2n,ψ D(1,2n) ∼ This completes the proof of the theorem. e 4.4. Proof of Theorem 4.2.2. This theorem is general. We prove

Theorem. Let π be a smooth representation of Sp4n(F ). Then we have a vector space isomorphism

1 JE ,ψ(2n) (Vπ ) = JVn,ψn J ˜ (n) (Vπ) . 2n ∼ N ,χ˜(−n) 1 We start with JVn,ψn J ˜ (n) (Vπ) . Again, we will ﬁrst use a conjugation by N ,χ˜(−n) ω = m(ω), where ω is the following Weyl element of GL2n(F ):

(4.31) ω2i,i =1,i=1,...,n, e e ω2i 1,i+n =1,i=1,...,n, e − ωi,j =0, otherwise. e e

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 886 DAVID GINZBURG, STEPHEN RALLIS, AND DAVID SOUDRY

I Note that ω = µ n ,whereµis the Weyl element used in Section 3.3. · In Denote e (n) 1 (4.32) ω jn(Vn)N˜ ω− = B. · · The elements of B have the form TX (4.33) b = 0T∗ where T has the following description:

1 t t , n 12 ··· 1 2 t21 1 t2,2n  ···  t31 t32 1 t3,2n (4.34) T =  . .  .  . .    t2n 1,1 t2n 1,2 1 t2n 1,2n  − − −   t2n,1 t2n,2 t2n,2n 1 1   −    tj+1,j . Put, for i, j 2n 1, tj = . , ti =(ti,i ,... ,ti, n). Then, for j =1,... ,n, ≤ −  .  +1 2 t  2n,j    0∗   ∗ 0 0   . (4.35) t2j 1 =. and t2j = . − .   0     ∗   0   0     (in case j = n, t2n 1 = (0)), and for i n − ≤ t2i 1 =(0 0 0) − ∗ ∗···∗ 2n 2i and t2i is arbitrary in F − .DenotebyT(n) the subgroup of such matrices T . For example, in case n =4 10 0 0 0 1∗ ∗ ∗ ∗0010∗∗∗∗∗∗0 0 ∗ ∗  0 1  T =∗ ∗ ∗∗∗∗. 000010 0  ∗   0 0 1  ∗ ∗ ∗ ∗∗ 00000010   00000001   Let χ be the following character of B (in the notation (4.33)):

(4.36) χ(b)=ψ((t13 + t24)+(t35 + t47)

+ +(t2n 3,2n 1 +t2n 2n,2n)+(x2n 1,2 x2n,1)). ··· − − − − −

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use CUSPIDAL REPRESENTATIONS OF GL2n AND Sp2n 887

Note that f

1 1 χ(b)=(ψn χ˜− )(ω− bω) . · (n) Thus,

(4.37) JVn,ψn J ˜ (n) 1 (Vπ) = JB,χ(Vπ). N ,χ˜(−n) ∼ We will use Lemma 2.2, as we did before, to “ﬁll in the zeroes of t2i 1, from right (2n) − to left, using t2i 1” and thus “obtain E2n from B,andψ from χ”. Let −

(i,1) (4.38) Y = m(I n + ye i, ) ,i=1,... ,n 1, { 2 2 1 } − (1,j) X = m(I n + xe , j ) ,j=2,3,... ,n, { 2 1 2 } (n 1,1) B − = B, n (i,1) (n 1,1) (1,j) B = b B − bj, =0, j> i X ,i n2, { ∈ | 1 ∀ 2 }· ≤− j=i+2 Y (i,1) (i,1) C = b B b i, =0 , { ∈ | 2 1 } D(1,i+1) = C(i,1)X(1,i+1),A(1,i+1) = D(1,i+1)Y (i,1) .

(i,1) Deﬁne a character χ by the same formula (4.36), except replace trs by crs and (i,1) (i,1) x2n 1,2 x2n,1 by c2n 1,2n+2 c2n,2n+1,forc C .Extendχ trivially to − − (1,i+1)− (i,1) − ∈ (i,1) (i,1) D and to B , denoting the extensions by χD(1,i+1) and χB(i,1) .Notethat (1,i+1) (i 1,1) (i,1) (i 1,1) (i,1) D = B − and χ (1,i+1) = χ − . The character χ and the D (i 1,1) C − diamond

A(1,i+1)

X(1,i+1) Y(i,1) ,- B(i,1) D(1,i+1)

Y (i,1) X(1,i+1) -, C(i,1)

satisfy assumptions (i)-(v) of Section 2.2, and we conclude that

J (i,1) (V ) = J (i,1) (V ) = J (i 1,1) (V ) (i,1) π (1,i+1) π (i 1,1) − π B ,χ (i,1) ∼ D ,χ (i,1) ∼ B − ,χ (i 1,1) B D B −

for i = n 1,n 2,... ,2. Thus − −

J (V ) = J (1,1) (V ) . B,χ π ∼ D(1,2) ,χ π D(1,2)

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 888 DAVID GINZBURG, STEPHEN RALLIS, AND DAVID SOUDRY

To visualize the above subgroups, it is enough to visualize their GL2n-Levi part (since they all lie in P2n, and contain the full unipotent radical of P2n). 1 2j 01  .  . 10 ↓ 0 Y (i,1) :  ,X(1,j) : ··· ∗ ··· .  ..  1 2i  .    −→ ∗  .  .  ..  .       1  01         (i,1) B : the ﬁrst column in its GL2n part has the form

∗ 0   ∗ 0     2 i ∗←− 0   . .   0     2i+3

and the ﬁrst row has the form (1 0 0 0↓ ), the rest of the GL2n part is like that of T in (4.35). ∗ ∗··· ∗∗···∗ (i,1) (i,1) C : like B , only that b2i,1 =0. (1,i+1) (i,1) D : like C , only that the ﬁrst row of its GL2n-part has the form: 10 0 0 ∗ ∗··· ∗∗···∗ 2i↑+1

(1,2) I2 Note that the GL2n-part of D looks like ∗ ,whereT0 T(n 1). In 0 T 0 ∈ − general, for 1 r, s n let ≤ ≤ (r,s) (r,s) (4.39) Y = m(I2n + ye2r,2s 1) ,X = m(I2n + xe2r 1,2s) . { − } { − } Let, for 1 j i n 1, ≤ ≤ ≤ − n (4.40) B(i,j) = B˜(i,j) X(j,s) , s=i+2 Y where

I 2 ∗∗ .. ˜(i,j) TX  .  T0 T(nj+1) B = Sp4n(F ) T = , ∈ − , 0T T`, j =0, `>2i ( ∗ ∈ I2 2 1 )  ∗ − ∀  T  0  

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use CUSPIDAL REPRESENTATIONS OF GL2n AND Sp2n 889

(i,j) (i,j) (j,i+1) f(i,j) (j,i+1) (4.41) C = b B b2i,2j 1 =0 ,D = C X , { ∈ | − } A(j,i+1) = D(j,i+1)Y (i,j) .

(j,n+1) (i,j) (i,j) (We let X = I4n , T (2) = I2n .) Let χ be the character of C { } {(i,j)} deﬁned by (4.36), where, for c C , we replace tr,s by cr,s and x2n 1,2 ∈ (j,i+1) (i 1,j) − − x2n,1 by c2n 1,2n+1 c2n,2n+1.NotethatD = B − ,fori j+1, (j,j+1)− (n −1,j+1) ≥ and D = B − . As before, we have the usual compatibility relations among the χ(i,j) and their (trivial) extensions to D(j,i+1),B(i,j).Onechecksthat χ(i,j),A(j,i+1),B(i,j),C(i,j),D(j,i+1), Y (i,j),X(j,i+1) satisfy assumptions (i)-(v) of Section 2.2. All in all, we conclude that

J (V ) = J (1,1) (V ) = =J (j,j) (V ) B,χ π ∼ D(1,2),χ π ∼ ∼ D(j,j+1),χ π D(1,2) ··· D(j,j+1)

= =J (n 1,n 1) (V ) . (n 1,n) − − π ∼ ∼ D − ,χ (n 1,n) ··· D − n ,n (n 1,n 1) n ( 1 ) − − (2 ) Note that D − = E2n and χD(n 1,n) = ψ . This completes the proof of the theorem. − We record the following corollary to (4.6) and the theorems in Sections 4.3 and 4.4. Recall that H denotes the image of the direct sum embedding of Sp (F ) 2n × Sp2n(F ) inside Sp4n(F ).

Corollary. Let π be an irreducible representation of Sp4n(F ). Assume that Vπ admits a nontrivial H-invariant functional. Then we have a vector space isomorphism

∗ ∗ JV ,ψ (Vσ ) = J (Vπ) . n n n,π ∼ V2n,ψ h i h i 5. Theb global casee In this section we place ourselves in the global set-up of Section 1.7, and we prove Theorem 1.7, which states that σn(η) =0,thespaceofσn(η)beinggiven by (1.33), for k = n. (The reader will note6 that the proof is analogous to that of Theorems 4.2.1 and 4.2.2, the repeated use of Lemma 2.2 being replaced by corresponding Fourier expansions. Of course, nothing of these is used here. We keep the notation of Section 4, except we use it for the global set-up; however we give a precise reference for each notation.) Denote (in the notation of (1.28))

(g,ϕ)=Ress E(g,ϕη,s) E =1 where we put, for short, ϕ = ϕη,1. As is clear from the proof of [G.R.S.1, Lemma 15], σn(τ) = 0, if and only if 6 (5.1) I(ϕ)= (v, ϕ)χ (v)dv 0. E (n) 6≡ ˜(n)Z˜(n) NK N \ A e Recall that zx 10∗∗∗∗yt  ∗ ˜ (n) In 0y0 N = v = ∗ Sp4n z Zn 1  In 0 ∈ ∈ −  ∗ n  1x0 o    z∗    

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 890 DAVID GINZBURG, STEPHEN RALLIS, AND DAVID SOUDRY

and, for v N˜ (n)(A), ∈ n 2 − χ˜(n)(v)=ψ zi,i+1 + xn 1 + t) − i=1 X (Zk denotes the standard maximal unipotent subgroup of GLk). We also proved in [G.R.S.1, Chapter 3] that for k

(2n) 1 (5.3) R(ϕ)= (ν v, ϕ)ψ (v)dv = (v ν ,ϕ)χ− (v)dv E 0 E · 0 E (K)ZE ( ) B ZB 2n \ 2n A K \ A where B and χ (global versions) are deﬁned in (4.10) and (4.11). Put, for 1 i

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use CUSPIDAL REPRESENTATIONS OF GL2n AND Sp2n 891

(j 1,i) f(j 1,i) Now, let us write the Fourier expansion along − (K) − (A), deﬁned in (4.17), of x (`(x)h, ϕ): X \X 7→ E

(5.6) (h, ϕ)= (`(t(ej 1,i + e2n+1 i,2n+2 j))h, ϕ)ψ(αt)dt . E E − − − α KKZ X∈ \A Substitute (5.6) in (5.4) and decompose (i,j) (i,j) (i,j) (i,j) (i,j) (i,j) B (K) B (A)=C (K) C (A) Y (K) Y (A) \ \ · \ with db = dcdy. These groups are deﬁned in (4.21). (Recall that Y (i,j) normalizes C(i,j) and preserves χ on C(i,j).) We get (5.7)

Rij (ϕ)=

Z (i,j) Z (i,j) (i,j) Z (i,j) Yi,j∗ (A) Y (K) Y ( ) C (K) C ( ) \ A \ A 1 (`(t(ej 1,i + e2n+1 i,2n+2 j))cy y∗ν0,ϕ)ψ(αt)χ− (c)dcdydy∗ . · E − − − · α K X∈ Now, use that (`(α(ei,j + e2n+1 j,2n+1 i))h)= (h), for α K, and conjugate E − − E ∈ `i,j (α)=`(α(ei,j + e2n+1 j,2n+1 i)) all the way to the right in (5.7). (Here, we use in full the global analog of− assumptions− (i)-(v) of Section 2.2.) We get

(5.8) Ri,j (ϕ)=

Z (i,j) Z (i,j) Yi,j∗ (A) Y (K) Y ( ) \ A 1 (v `i,j (α)yy∗ν ,ψ)χ− (v)dvdydy∗ · E · 0 α K ∈ D(i,j) (KZ) D(i,j) ) X \ A 1 = (by∗ν ,ϕ)χ− (b)dbdy∗ E 0 Z (i 1,j) Z (i 1,j) Yi∗ 1,j (A) B − (K) B − (A) − \ where, as in Section 4.3, B(0,j) = B(j,j+1). D(i,j) is deﬁned in (4.21). Here to shorten the notation, we keep using χ(v) as the character of B(i,j) which is χ(m(z)) on m(z) and trivial on the other factors. (Note the clear analogy with Section 4.3.) We conclude from (5.8) that

Ri,j (ϕ)=Ri 1,j (ϕ)if2i

R ,j(ϕ)=Rj,j (ϕ)if1jn, 1 +1 ≤≤ and hence

R(ϕ)=R1,n+1(ϕ) .

(n,1) Now perform (“in R ,n (ϕ)”) a Fourier expansion, as before, along (K) 1 +1 X \ (n,1)(A) to get (as in (5.7), (5.8)) X 1 (5.9) R(ϕ)= (by∗ν ,ϕ)χ− (b)dbdy∗. E 0 Z (1,n+1) Z (1,n+1) Yn∗ 1,n+2(A) D (K) D (A) − \

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 892 DAVID GINZBURG, STEPHEN RALLIS, AND DAVID SOUDRY

(1,n+1) Recall that an element of D has the form b = `(x ,n)m(z)`(y), where x ,n 1 1 ∈ χ1,n, z Z2n and y 1,n+1.(χ(b)=χ(m(z)), deﬁned in (4.11).) Note that, so far, we did∈ not use any∈Y property of (h, ϕ), except for being an automorphic func- E tion. Now write the Fourier expansion of (h, ϕ)alongX(n+1,n)(K) X(n+1,n)(A) (deﬁned in (4.21)): E \

(5.10) (h, ϕ)= `(ten ,n)h, ϕ ψ(αt)dt. E E +1 α K ∈ KZ X \A The contribution of all nontrivial characters in (5.10) (i.e. α = 0) to (5.9) is zero, 6 since, for α K∗, ∈ 1 `(ten ,n)by∗ν ,ϕ χ− (b)ψ(αt)dtdb E +1 0 KZA D(1,n+1)(K)ZD(1,n+1)( ) \ \ A contains an inner integral of the form

1 (bg, ϕ)χ(−n 1),α(b)db E − N (n 1)(K)ZN (n 1)( ) − \ − A (1,n+1) for g D (A)Y0∗,n+1(A)ν0, which is identically zero, by (5.2). Thus, in (5.9), we get∈

R(ϕ)=Rn 1,n+2(ϕ) − 1 (5.11) = (by∗ν ,ϕ)χ− (b)dbdy∗ E 0 Z (n 1,n+2) Z (n 1,n+2) Yn∗ 1,n+2(A) B − (K) B − (A) − \ (n 1,n+2) (1,n+1) (n+1,n) where as in (4.28) B − = D X ,andχ, as usual, extends trivially to X(n+1,n)(A). Now we continue as in (5.6) (“replacing the n 1 coordinates of n 1 i,n+2 − − into χn , ”). Assume (by induction) that R(ϕ)=Ri,n (ϕ), for i=1 Y +1 1 +2 i n 1. Write the Fourier expansion of (h, ϕ)alongX(n+1,i)(K) X(n+1,i)(A) forL≤i =−n 1,n 2,...: E \ − −

(5.12) (h, ϕ)= `(t(en+1,i + e2n+1 i,n))h, ϕ ψ(αt)dt E E − α K ∈ KZ X \A and substitute in (5.4) in Ri,n+2(ϕ). The steps (5.7), (5.8) are valid here as well and we get that Ri,n+2(ϕ)=Ri 1,n+2(ϕ), for 2 i n 1. Thus − ≤ ≤ − R(ϕ)=R1,n+2(ϕ) . n , n , As in (5.9), using the Fourier expansion (of (h, ϕ)) along X( +1 1)(K) X( +1 1)(A), we get E \

1 (5.13) R(ϕ)= (by∗ν ,ϕ)χ− (b)dbdy∗ E 0 Z (1,n+2) Z (1,n+2) Yn∗ 2,n+3(A) D (K) D (A) − \ (1,n+2) where, as usual, for b = `(x1,n+1)m(z)`(y) D (A)(x1,n+1 χ1,n+1(A), ∈ ∈ z Z2n(A), y 1,n+2(A)), ∈ ∈Y χ(b)=χ(m(z)). Now repeat the argument, which started at (5.10). Write the Fourier expansion of (n+2,n 1) (n+2,n 1) (h, ϕ)alongX − (K) X − (A). The contribution of each nontrivial E \

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use CUSPIDAL REPRESENTATIONS OF GL2n AND Sp2n 893

character in this expansion to (5.13) is zero, since it containsf an inner integral of the form

1 (bg, ϕ)χ(−n 2),α(b)db E − N (n 2)(K)ZN (n 2)( ) − \ − A which is identically zero, for α K∗, by (5.2). Thus, in (5.13) we get ∈ R(ϕ)=Rn 2,n+3(ϕ) − 1 (5.14) = (by∗ν ,ϕ)χ− (b)dbdy∗. E 0 Z (n 2,n+3) Z (n 2,n+3) n∗ 2,n+3(A) B − (K) B − (A) Y − \ Now we use Fourier expansions of (h, ϕ)alongX(n+2,i)(K) X(n+2,i)(A)fori= n 2,n 3,... , repeat steps (5.7),E (5.8) which are valid here\ and get R(ϕ)= − − Rn 2,n+3(ϕ)=Rn 3,n+3(ϕ)=Rn 4,n+3(ϕ)= = R1,n+3(ϕ). We continue in this− manner until we− get, for j n −+2, ··· ≥ R(ϕ)=R1,j(ϕ).

(j 1,1) (j 1,1) As in (5.9), using the Fourier expansion along X − (K) X − (A)weget \ 1 (5.15) R(ϕ)= (by∗ν ,ϕ)χ− (b)dbdy∗. E 0 Z (1,j) Z (1,j) Y2∗n j,j+1(A) D (K) D (A) − \ As before, using (5.2), the contribution to (5.15) of nontrivial characters in the (j,2n+1 j) (j,2n+1 j) Fourier expansion of (h, ϕ)alongX − (K) X − (A) is zero. Thus, E \ as in (5.11) (and (5.13)) R(ϕ)=R2n j,j+1(ϕ). Now, we use Fourier expansions − of (h, ϕ)alongX(j,i)(K) X (j,i)(A), for i =2n j, 2n j 1,... , repeat steps E \ − − − (5.7), (5.8), which are still valid, and get R(ϕ)=R2n j,j+1(ϕ)=R2n j 1,j+1(ϕ) − − − = =R ,j (ϕ) and as in (5.15), we get ··· 1 +1 1 R(ϕ)= (by∗ν ,ϕ)χ− (b)dbdy∗ . E 0 Z (1,j+1) Z (1,j+1) Y2∗n j 1,j+2(A) D (K) D (A) − − \ At the ﬁnal step, we get

1 (5.16) R(ϕ)= (by∗ν ,ϕ)χ− (b)dbdy∗ E 0 Z (1,2n) Z (1,2n) Y0∗,2n+1(A) D (K) D ( ) \ A (1,2n) where Y ∗ = `( )( is deﬁned in (4.12)). Note that D is the standard 0,2n+1 X0 X0 maximal unipotent subgroup of Sp4n and that z χ ∗ = χ(m(z)) , for z Z2n(A). 0 z∗ ∈ Thus, the inner integral of (5.16) reads as

1 (5.17) M(ϕ)(m(z)`(x)ν0)χ− (m)(z))dz

Z (K)ZZ ( ) 2n \ 2n A where

M(ϕ)(h)=Ress=1 M(s)ϕη,s(h),

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 894 DAVID GINZBURG, STEPHEN RALLIS, AND DAVID SOUDRY

the residue at s = 1 of the intertwining operator on ρη,s. Note that (5.17) is Eulerian, by uniqueness of the Whittaker model for η. Let us denote the r.h.s. of (5.17) by M(ϕ)W,χ(`(x)ν0). We record this as Theorem 1. Let η be an irreducible, self-dual, automorphic, cuspidal representa- S 2 S 1 tion of GL2n(A), such that L (η, Λ ,s) has a pole at s =1,andL(η, 2 ) =0. Then 6 (2n) (5.18) Ress=1 E(v, ϕη,s)ψ (v)dv = M(ϕ)W,χ(`(x)ν0 )dx .

E (K)ZE ( ) Z( ) 2n \ 2n A X0 A Here

χ(z)=ψ(z12 + z23 + +zn,n+1 zn+1,n+2 z2n 1,2n) ,zZ2n(A). ··· − −···− − ∈

The next step is to relate the l.h.s. of (5.18) with I(ϕ) in (5.1) (as we did in Theorem 4.2.2). Consider the ψn-Whittaker coeﬃcient of I(ϕ)alongVn,

1 S(ϕ)= I(jn(u) ϕ)ψ− (u)du · n V (K)ZV ( ) n \ n A 1 = (v jn(u),ϕ)χ (v) ψ− (u)dvdu . E · (n) · n Vn(K)ZVn(A) N˜ (n)ZN˜ (n) \ K \ A e As in (5.3), this time using conjugation by ω deﬁned in (4.31), we get

1 (5.19) S(ϕ)= (v ω,ϕ)χ− (v)dv E · B ZB K\ A where now B is defined by (4.32)-(4.35) and χ by (4.36). Introduce for 1 j i n 1 ≤ ≤ ≤ − 1 (5.20) Si,j (ϕ)= (vy∗ω,ϕ)χ− (v)dvdy∗ E i,j Z (i,j) Z (i,j) Yi,j∗ (A) B (K) B ( ) \ A (i,j) (i,j) where B is defined in (4.40) and χi,j is the character of B (A) defined as in (4.36), i.e.

(5.21) χi,j (b)=ψ((b13 + b24)+(b35 + b47)

+ +(b2n 3,2n 1 +b2n 2,2n)+(b2n 1,2n+2 b2n,2n+1)) . ··· − − − − − Here

T T (n) Yij∗ = m(T ) ∈ . ( Trs =0, if r2j 1, or s =2j 1andr>2i) − −

T(n) is deﬁned in (4.34), (4.35). Note that Sn 1,1(ϕ)=S(ϕ). Now write the − Fourier expansion of x (xh, ϕ)alongX(j,i+1)(K) X(j,i+1)(A) (deﬁned in (4.39)): 7→ E \

(5.22) (h, ϕ)= (m(I2n+te2j 1,2i+2)h, ϕ)ψ(αt)dt. E E − α K ∈ KZ X \A

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use CUSPIDAL REPRESENTATIONS OF GL2n AND Sp2n 895

We are exactly at the same situation as in (5.6), and we continuef in the same way as in (5.7), (5.8), using C(i,j), Y (i,j) deﬁned in (4.39), (4.41), to get

(5.23) S(ϕ)=Sn 1,n(ϕ). − (n 1,n) (2n) Since B − = E2n and χn 1,n = ψ , we get from (5.23) − (2n) (5.24) S(ϕ)= (vy∗ω,ϕ)ψ (v)dvdy∗. E Z Z Yn∗ 1,n(A) E2n(K) E2n(A) − \ Note that T T (n) (5.25) Yn∗ 1,n = m(T ) ∈ . − ( T is lower triangular)

Note that (5.24) is valid for any automorphic form on Sp4n(A) (we did not use any property of ). We record this in E

Theorem 2. For any automorphic form ξ on Sp4n(A), we have

1 ξ(v jn(u))χ (v)ψ− (u)dvdu · (n) n (n) (n) Vn(K)ZVn(A) ˜ Z ˜ \ NK N \ A e (2n) (5.26) = ξ(vy∗ω)ψ (v)dvdy∗ . Z Z Yn∗ 1,n(A) E2n(K) E2n(A) − \ We conclude from (5.18) and (5.26) Corollary. Under the assumption of Theorem 5.1, we have

1 Ress=1 E(vjn(u),ϕη,s)χ(n)(v)ψn− (u)dvdu

(n) (n) Vn(K)ZVn(A) ˜ Z ˜ \ NK N \ A e (5.27) = M(ϕ)W,χ (`(x)ν0y∗ω)dxdy∗ Z Z Yn∗ 1,n(A) 0(A) − X

where χ is deﬁned in Theorem 5.1, 0 in (4.12) and Yn∗ 1,n in (5.18). X − To conclude that σn(η) = 0, which is equivalent to I(ϕ) 0 (in (5.1)), it remains to prove the following two6 lemmas. 6≡

Lemma 1. For any automorphic representation π of Sp4n(A) (5.28) (2n) ξ(vy∗)ψ (v)dvdy∗ 0, as ξ varies in Vπ, 6≡ Z Z Yn∗ 1,n(A) E2n(K) E2n(A) − \ if and only if

(2n) ξ(v)ψ (v)dv 0, as ξ varies in Vπ . 6≡ E (F )ZE ( ) 2n \ 2n A

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 896 DAVID GINZBURG, STEPHEN RALLIS, AND DAVID SOUDRY

Lemma 2. Let L be an Sp4n(A)-invariant space of smooth functions on ZK X A\ Sp4n(A) where X is the unipotent radical of the Siegel parabolic subgroup and Z = m(Z2n). Assume that the representation of Sp4n(A) on L is of moderate growth, and the elements of L satisfy f(m(z)g)=χ(m(z))f(g) ,

for z Z2n(A).Then ∈ (5.29) f(`(x))dx 0 , as f varies in L 6≡ Z( ) X0 A (notation of (5.27)).

Proof of Lemma 1. Note that Yn∗ 1,n is abelian. Write it as the product Yn∗ 1,n = n 1 − − − Ki,where i=1

Q i

Ki = ki(t1,... ,ti)=m(I2n+ tje2i,2j 1) . { − } j=1 X Deﬁne s 1 − Rs = rs(t1,... ,ts 1)=m(I2n+ tie2i 1,2s) . { − − } i=1 X i i Denote the inner dv-integral of (5.28) by ϕξ(y∗). For 1 i n 1, K = K` ≤ ≤ − `=1 put Q

ai(ξ)= ϕξ(y)dy.

KiZ(A)

Note that (5.28) is an 1(ξ). By [D.M.] (applied at the inﬁnite places) we can write −

ξ = φα(x ,... ,xi)π(ri (x ,... ,xi)) (ξα)d(x ,... ,xi) 1 +1 1 · 1 α Zi XA i where φα S(A )andξα Vπ.Weget ∈ ∈

ai(ξ)= φα(x ,... ,xi)ϕξ (y ri (x ,... ,xi))dyd(x ,... ,xi) . 1 α · +1 1 1 α Zi iZ XA K (A) i i 1 Write K = K − Ki and y = y0ki(t ,... ,ti) accordingly. Note that · 1 1 yri (x ,... ,xi)y− E n +1 1 ∈ 2 and n (2n) 1 ψ (yri (x xi)y− )=ψ( xiti) . +1 1 ··· i=1 X

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use CUSPIDAL REPRESENTATIONS OF GL2n AND Sp2n 897

We get f

ai (ξ )= φα(t1,... ,ti)ϕξα(y0ki(t1,... ,ti))dy0d(t1,... ,ti)

α i Z1 Zi XK − (A) A b =ai 1(ξ0) −

where ξ0 = φα(t1,... ,ti)π(ki(t1,... ,ti)) (ξα)d(t1,... ,ti) ξ0 varies over all α i · · A of Vπ as weP varyR φα and ξα.Thusai(ξ) 0 if and only if ai 1(ξ) 0, where b 6≡ − 6≡ a0(ξ)=ϕξ(1). Thus, an 1(ξ) 0 if and only if ϕξ(1) 0. − 6≡ 6≡ Proof of Lemma 2. (The proof here is similar to part of the proof of Theorem 1.3.) Denote

(5.30) Ki+1 = i ki+1(t1,... ,ti)=` tj(ej,i+1 + e2n i,2n+1 j) , for i n, j=1 − − ≤ n 2n i 1 o  P −−  ki+1(t1,... ,t2n i)=` tj(ej,i+1 + e2n i,2n+1 j)+t2n ie2n i,i+1 ,  − j=1 − − − − n P for n +1 i 2no1.  ≤ ≤ −  Then 2n `( )= K` . X0 ` Y=2 Put 2n i K = K` . ` i =Y+1 Deﬁne (5.31) s rs(x1,... ,xs)=` xj(es,j + e2n+1 j,2n+1 s) , for s n, j=1 − − ≤ n P o Rs =  rs(x1,... ,x2n s)  −  2n s n − =` xj(es,j + e2n+1 j,2n+1 s) , for n +1 s 2n. − − ≤ ≤  j=1  P o Put, for 1 i 2n 1andf L,  ≤ ≤ − ∈

ai(f)= f(y)dy .

KiZ(A)

Note that the integral (5.29) is a1(f). Write, as in the previous lemma,

f = φα(r)r fαdk · α Z X Ri(A) i for φα S(Ri(A)), fα L.Writey K(A) in the form y = y0 k,where i∈+1 ∈ ∈ · y0 K (A)andk Ki+1(A), of the form (5.30). Write r Ri(A) in the form ∈ ∈ ∈

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 898 DAVID GINZBURG, STEPHEN RALLIS, AND DAVID SOUDRY

1 (5.31). Then yry− V2n(A)and ∈ 1 fα((yry− ) y)=ψ( xjtj)fα(y) · ± ( according to whether i>nor i n). We areX now at exactly the same situation ± ≤ of the last lemma, and we conclude that ai(f) 0 if and only if ai (f) 0, for 6≡ +1 6≡ i =1,... ,2n 1, where a n(f)=f(1). This proves the lemma, and completes the − 2 proof of the (global) nonvanishing of σn(τ).

6. Appendix 6.1. Proof of the isomorphism (1.14). We use the notation of Section 1.1. Let τ be an irreducible, generic representation of GLm(F ). Let σ be an irre-

ducible, generic, genuine representation of Sp2k(F ). We assume that k

γm,i,w = γm,i m(w) ,k i m, · ≤≤ where

Im i − − γm,i = I i  2  Im i − and  

w WGLm i WGLi k WGLm k ; ∈ − × − \ −

here WGLr denotes the Weyl group of GLr.Denote (m,k) = Pmγm,i,wjm,k(Sp (F ))Nm,k. Oi,w 2k Note that (m,k) is the unique open orbit. For E a union of orbits (m,k),let Ok,1 Oi,w S(E,Pm,τs) denote the space of smooth functions ϕ on E,withvaluesinthe Whittaker model W (τ,ψ0 ) (see (1.2)), such that, for g Sp (F ), a, r GLm(F ), m ∈ 2m ∈

a s+ m ϕ ∗ g,r = det a 2 ϕ(g,ra) , 0 a∗ | | !

and such that the support of ϕ is compact modulo Pm. We may arrange the orbits in a sequence

(m,k) (m,k) = Pmjm,k(Sp (F )) = Ω , Ω ,... ,ΩL = , Om,1 2k 1 2 Ok,1 i such that Fi = j=1 Ωj is closed in Sp2m(F ). We then have an exact sequence e r (6.2) 0 SS (Ωi ,Pm,τs) S ( F i ,Pm,τs) S ( F i ,Pm,τs) 0 . −→ +1 −→ +1 −→ −→

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use CUSPIDAL REPRESENTATIONS OF GL2n AND Sp2n 899

In (6.2), the map e is the natural embedding, and the map fr is the restriction to Fi. Applying the Jacquet functor J 1 to (6.2), we get the exact sequence Nm,k+1,χk−

0 JN ,χ 1 (S(Ωi+1,Pm,τs)) JN ,χ 1 (S(Fi+1,Pm,τs)) → m,k+1 k− → m,k+1 k− (6.3) JN ,χ 1 (S(Fi,Pm,τs)) 0 . → m,k+1 k− → (k) k Note that tensoring with ωψ (acting on S(F )) preserves exactness. Applying S(F k) to (6.3) and then J gives the exact sequence ⊗ Hk k 1 0 J k JN ,χ (S(Ωi+1,Pm,τs)) S(F ) → H m,k+1 k− ⊗ k 1 J k JN ,χ (S(Fi+1,Pmτs)) S(F ) → H m,k+1 k− ⊗ k 1 J k JN ,χ (S(Fi,Pm,τs)) S(F ) 0 . → H m,k+1 k− ⊗ → k 1 This reduces the study of J k JN ,χ (Vρτ,s S(F )) to the study of the H m,k+1 k− ⊗ (m,k) k 1 various J k JN ,χ (S( i,w ,Pm,τs) S(F )) , k i m.Wehave H m,k+1 k− O ⊗ ≤ ≤

Sp2k(F ) Nm,k (m,k) c · 1/2 γm,i,w S( ,Pm,τs) Ind (δ τs) Oi,w ' Ri,w Pm as Sp (F ) Nm,k-modules. The induction is not normalized. Here 2k · 1 Ri,w = γ− Pmγm,i,w Sp (F ) Nm,k, m,i,w ∩ 2k · 1/2 γm,i,w 1/2 1 (δ τs) (r)=δ τs(γm,i,wrγ− ) ,rRi,w, Pm Pm m,i,w ∈ 1/2 ax m s+ 2 (δP τs) =det a τ(a) . m 0a∗ | |

Sp2k (F ) Nm,k c · 1/2 γm,i,w We claim that J 1 Ind (δ τ ) =0,fori k+1and N ,χ− Ri,w P s m,k+1 k m ! ≥ w=Im. Indeed, by Lemma 4.3 in [G.R.S.4] there is a simple root subgroup x(t) 6 1 inside Nm,k+1, such that γm,i,w− x(t)γm,i,w lies in the unipotent radical of Pm.This 1/2 γm,i,w shows that x(t) Ri,w Nm,k+1,(δ τs) (x(t)) = id, while χk(x(t)) = ψ(t). ∈ ∩ Pm Put γ0 = γm,i, . We now show that, for i k +1, m,i 1 ≥ Sp2k(F ) Nm,k c · 1/2 γ0 (k) 1 m,i (6.4) J k JN ,χ IndR (δP τs) ωψ =0. H m,k+1 k− i,1 m ⊗ We may realize the inner space in the last Jacquet module as

Sp2k(F ) Nm,k 1/2 k c · γm,i0 S F ; J 1 IndR (δ τs) Nm,k+1,χk− i,1 Pm

Sp2k(F ) Nm,k 1/2 c · γm,i0 –thespaceofJ 1 IndR (δ τs) -vector valued Schwartz- Nm,k+1,χk− i,1 Pm Bruhat functions on F k.Wehave

z1 00cy0 zab xy0  2  rdb0 c0 z1 Zm i Ri,1 = Sp2m(F ) ∈ − .  r∗ a0 0∈ z2 Zi k (   ∈ − )  z∗ 0  2   z  1  

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 900 DAVID GINZBURG, STEPHEN RALLIS, AND DAVID SOUDRY

Sp (F ) N c 2k · m,k 1/2 γ Thus the functions f in Ind (δ τ ) m,i0 are determined by the values Ri,1 Pm s

Im i et00 r − Iik00 0 0  −  Ik 00 0 f jm,k(g) ,gSp2k(F ),  Ik 0 t0 ∈   !  Iik e0  −   Ini  − and  

z1 00cb0 z2ab xb0   z1∗ rdb0 c0 s+ m f h=det r 2 τ y0 z2 a f(h) .  r∗ a0 0 | | −   ! c0 0 r  z∗ 0 −  2     z∗  1 Let  

z1 00 z1 Zmi,z2 Zi k Bi,k = uz2 a ∈ − ∈ − (   last row of a is zero ) v 0Ik

and consider the characterψ of B deﬁned by ψ 1(z )ψ(z ). Then it is easy to i,k i,k − 1 2 see that there is an embedding

Sp (F ) N c 2k · m,k 1/2 Sp (F ) m γm,i0 2k s 2 +i JN ,χ 1 IndR (δP τs) , IndP JB ,ψ 1 (τ) det − . m,k+1 k− i,1 m → k i,k i,k− ·| ·| We regard J 1 (τ)asaGLk(F)-module, where GLk(F ) is embedded in GLm(F ) Bik ,ψi,k− Im k by g − . (The inductions are not normalized.) The embedding p is 7→ g Sp (F ) N c 2k · m,k 1/2 γ given through its pull-back to Ind (δ τ ) m,i,i0 by Ri,1 Pm s

p(f )(g)= ψ(em i,1)jB − i,k Z In i et00 r − Iik00 0 0  −  Ik 00 0 f jm,k(g) d(e, t, r) . ·  Ik 0 t0   !!  Iik e0  −   Imi  −  Sp2k (F ) Nm,k 1/2 k c · γm,i0 The action of the center of k on S F ; JN ,χ 1 IndR (δP τs) H m,k+1 k− i,1 m is by

((0, 0; t) ϕ)(x)=ψ(t)jm,k(0, 0; t) (ϕ(x)). · · It is easy to see that

p(jm,k(0, 0; t) ξ)=p(ξ) · Sp (F ) N 1/2 2k m,k γm,i0 for t F, ξ JN ,χ 1 IndR · (δP τs) .Sincepis injective, we ∈ ∈ m,k+1 k− i,1 m ﬁnd that

jm,k(0, 0; t) ξ = ξ, ·

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use CUSPIDAL REPRESENTATIONS OF GL2n AND Sp2n 901

and so f (0, 0; t) ϕ = ψ(t)ϕ · for Sp2k(F ) Nm,k 1/2 k c · γm,i0 t F, ϕ S F ;JN ,χ 1 IndR (δP τs) ∈ ∈ m,k+1 k− i,1 m Sp2k(F ) Nm,k 1/2 (k) c · γm,i0 JN ,χ 1 IndR (δP τs) ωψ ; ' m,k+1 k− i,1 m ⊗ (m,k) this implies (6.4). Denote m,k = . We showed that the natural embedding O Ok,1 (k) 1 1 of J k JN ,χ (S( m,k,Pmτs)) ωψ in J k JN ,χ (Vρτ,s ) ωψ is H m,k+1 k− O ⊗ H m,k+1 k− ⊗ surjective. This is the isomorphism (1.14).

6.2. The local functional equation for Sp2k(F ) GLm(F )(k

Sp2k(F )-invariant pairing between σ and (k) (k) 1 1 J k JN ,χ (Vρτ,s ) ωψ = J k JN ,χ (S( m,k,Pm,τs)) ωψ . f H m,k+1 k− ⊗ ∼ H m,k+1 k− O ⊗ This implies the functional equation (1.10). It is convenient to choose the representative γm,k (see (1.6)) for the orbit m,k (rather than γm,k, ). Denote the stabilizer O 1 by Rk.Wehave

Sp2k (F )Nm,k c 1/2 γm,k S( m,k,Pm,τs) Ind (δ τs) O ' Rk Pm (the induction is not normalized). Note that z 0 c 0 Ik 0 c0 Rk =   Sp2m(F ) z Zm k jm,k(Pk), ( Ik 0 ∈ ∈ − ) ·

 z∗    Im k  − 1/2 a m a γm,k s+ 2 (δP τs)  ∗  = det a τ , m a∗ | | Im k −  Im k  −  z 0 c 0 

1/2 γm,k Ik 0 c0 Ik c0 (δP τs)   = τ − ,zZmk. m Ik 0 z∗ ∈ −  z∗   Let  

Ik y z Zm k, Bk = ∈ − ( z the ﬁrst column of y is zero)

Ik y and consider the character η of B deﬁned by η = ψ 1(z). As before, k k k z − we have (with unnormalized induction notation) (6.5)

Sp2k (F ) Nm,k Sp2k(F ) k m c · 1/2 γm,k c ·H s +k 1 2 JN ,χ IndR (δP τs) ) = IndP jBk,η (τ) det − m,k+1 k− k m ∼ k·Yk k ·| ·|

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 902 DAVID GINZBURG, STEPHEN RALLIS, AND DAVID SOUDRY

a where ∗ (0,y;0)in Pk k acts on jBk,ηk (τ) through a∗ · ·Y I y 0 a k 0 τ −10 . Im k   − Imk1! −− The isomorphism p of (6.5) is given through its pull backp to

Sp2k(F ) Nm,k 1/2 Indc · (δ τ )γm,k Rk Pm s by

p(f ) jm,k((u, 0; t) g) · Im k 1 0 x 0 e1 e2 − − 1 u 0 te0  1 Ik00 0 = jBk,ηk f jm,k(g) d(x, e).  Ik u0 x0  Z   !!  10    Imk1  −− So far, we showed that  

Sp (F ) m c 2k ·Hk s +k J 1 (ρ ) Ind j (τ) det 2 N ,χ− τ,s = P Bk,ηk − m,k+1 k ∼ k·Yk ·| ·| (as Sp (F ) k-modules). 2k H Extend σ trivially to k. Denote the extension by σ .Wehave H 1 Sp (F ) m c 2k ·Hk s +k (k) (6.6) Bil σ, J (Ind (j (τ) det − 2 ) ω ) Sp (F ) k Pk k Bk ,ηk ψ 2k H Y ·| ·| ⊗ cSp2k(F ) k m (k) f ·H s 2 +k Bil σ1, IndP jBk ,ηk (τ) det − ω Sp2k(F ) k gk k ψ ' Y ·| ·| ⊗ Pk k H ·Y m f s +k (k) Bil σ1,jB ,η (τ) det − 2 ω . e Pk k k k ψ ' ·Y ·| ·| ⊗ Pk k ·Y e Our objective is then to show that the space (6.6) e is at most one-dimensional, s outside a ﬁnite set of values of q− . Regard an element A of the space (6.6) as a k trilinear form on Vσ Vτ S(F ), satisfying × × I y e ax a k 0 (6.7) A σ ,ε ξ, τ −10 v, h a∗ i z   Imk1! −−   ax ωψ ,ε (0,y;0) φ h 0a∗ i ! ! s+ m k = ψ(z) det a − 2 − A(ξ,v,φ) , | | for z Zm k. Consider the exact sequence ∈ − k i k j 0 S ( F 0 ) S ( F ) C 0 −→ \{ } −→ −→ −→ where i is the natural embedding and j(φ)=φ(0). If A(ξ,v,φ) vanishes on the image of i, for all ξ and v, then, since

ax 1/2 ωψ ,ε (0,y,0) φ(0) = εγψ,det a det a φ(0) , h 0a∗ i | | !

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use CUSPIDAL REPRESENTATIONS OF GL2n AND Sp2n 903

f A deﬁnes a pairing A on Vσ Vτ such that ×

ax ab m 1 1 s+ 2− k (6.8) A σ ,ε ξ,τ v=εγψ,− det a det a − ψ(z)A(ξ,v) h a∗ i z | | ! !

for z Zm k. Note∈ that− if σ is supercuspidal, then, by (6.8), A must be zero. In general, (6.8)

shows that, if A is nonzero, then σ pairs into the representation of Sp2k(F ) induced m 1 s − +k from Pk and γψ, det − 2 times the derivative (in the sense of [B.Z.]) of det ·| ·| Ik b f τ obtained by the Jacquet module of τ, with respect to z Zm k and e { z | ∈ − } Ik b 1 s the character ψ− (z). This implies that q lies in a certain ﬁnite set z 7→ 0 Sσ,τ (which depends on σ and τ). Outside this ﬁnite set, A must be zero, and k then A is fully determined by its restriction to Vσ Vτ S(F 0 ). Note that (k) × ×k \{ } the representation of Pk k, acting through ωψ on S(F 0 ), is isomorphic to P ·Y \{ } c kYk 1/2 Ind ˜0 det γψ,det cψ (nonnormalized induction), where Pfk 1 k − ·Y | ·| e ·

0 ax b Pk 1 = Pka= ∗,b GLk 1(F ) , − 0a∗ ∈ | 01 ∈ − ( ) Ik x cψ (0,y;0) =ψ(xk1 +2y1). Ik · s 0 Thus (6.7) and Frobenius reciprocity imply that for q− S , A deﬁnes a bilinear 6∈ σ,τ form A on Vσ Vτ , satisfying × ax a I e e A σ ,ε ξ,τ k v h 0a∗ i In k z − ! !

e m 1 1 s+ − +k 1 Ik e (6.9) = εγ− det a − 2 ψ− (xk )ψ0 A(ξ,v) ψ,det a| | 1 m z e ax 0 for Pk1. (See (1.2).) Note that A factors through the product Jσ Jτ 0a∗ ∈ − × Iek x of Jacquet modules of σ, with respect to and ψ(xk1), and of τ, with Ik ( ) Ik e 1 ˜ 0 respect to z Zm k and ψm0− . We regard Jσ and Jτ as Mk 1-modules, z | ∈ − − ( ) where

0 b Mk 1 = ∗ b GLk 1(F ) . − ( 01 ∈ − )

Now we continue exactly as in [So], p. 52 (using the analog of Proposition 8.2 in

[G.PS.], for Jσ, as explained in Proposition 11.2 of [G.PS.]). We conclude that s outside a ﬁnite set Sσ,τ of values of q− , all the various derivatives of Jσ and Jτ 0 which correspond to nonminimal standard parabolic subgroups of Mk 1 contribute −

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 904 DAVID GINZBURG, STEPHEN RALLIS, AND DAVID SOUDRY

zero to the space of bilinear forms A0 on Jσ Jτ , satisfying × m 1 1 s+ − +k (6.10) A0(σ a, ε ξ,τ(a)v)=εγ− det a − 2 A0(ξ,v) h i γ,det a| | 0 for a Mk 1, ξ Jσ, v Jτ . ∈ − ∈ ∈ Thus, outside Sσ,τ , the only possible contribution to (6.10) comes from the derivatives of Jσ and Jτ which correspond to the Whittaker models with respect 1 to ψk− and ψm0 , respectively. This contribution is of dimension one. The proof of the functional equation (1.10) is now complete. Note again the special case where σ is supercuspidal. We have already remarked 0 that Sσ,τ = φ. Moreover, Sσ,τ = φ, as well, since clearly any derivative of Jσ with 0 respect to a nonminimal parabolic subgroup in Mk 1 involves a Jacquet module − with respect to a unipotent radical of Sp2k(F ), and hence equals zero. Thus the only possible contribution to (6.10) results from the derivatives of Jσ and Jτ which 1 correspond to the Whittaker models with respect to ψk− and ψm0 . Summing up,

Theorem. Let σ and τ be irreducible representations of Sp2k(F ) and GLm(F ), respectively (k

f (k) d (σ, τ, s)=dimBil σ, J J 1 (ρ ) ω . ψ F k N ,χ− τ,s ψ Sp2k( ) H m,k+1 k ⊗ Then dψ(σ, τ ) is a ﬁnite number.f Moreover s (a) There is a ﬁnite set Sσ,τ C, such that for q− Sσ,τ ⊂ 6∈ dψ (σ, τ, s) 1 . ≤ 1 s (b) If σ is ψ− -generic and τ is generic, then, for q− Sσ,τ , k 6∈ dψ (σ, τ, s)=1.

(c) If σ is supercuspidal and τ is generic, then dψ(σ, τ, s) 1, for all s C,and 1 ≤ ∈ dψ(σ, τ, s)=1if and only if σ is ψk− -generic. 6.3. A result on exterior square gamma factors. In this section, we prove a result needed in the proof of Theorem 3.3.2. Here we let τ be an irreducible, su- 2 2 percuspidal representation of GL2n(F ). Recall that γ(τ,Λ ,s,ψ)andγ(τ,Λ ,s,ψ) denote the corresponding exterior square gamma factors following Shahidi [Sh1] and Jacquet-Shalika [J.S.2], respectively. e Proposition. There is an exponential c(s)=aαs+β,suchthat γ(τ,Λ2,s,ψ)=c(s)γ(τ,Λ2,s,ψ).

In particular, γ(τ,Λ2,s,ψ) =0if and only if γ(τ,Λ2,s,ψ) =0. s=0 e s=0

Proof. Embed τ inside an irreducible, automorphic, cuspidal representation π of e GL2n(A), where A is the adele ring of a number field k, such that at the place ν0, k = F ,andifπ= π,thenπ =τ.Wemaytakeπ to be unramified, at ν0 ∼ ν ν0 ∼ ν all finite places ν = ν0. Also, there is a nontrivial character ψ = ψν of k A, 6 N \ such that ψν0 = ψ. (See [Sh2], Section 4.) The global functional equation for the corresponding theory of the exterior square L-function amountse to N LS(π, Λ2, 1 s) (6.11) γ(π , Λ2,s,ψ )γ(τ,Λ2,s,ψ)= − . ν ν LS(π, Λ2,s) ν S ∈Y∞ b

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use CUSPIDAL REPRESENTATIONS OF GL2n AND Sp2n 905

f Here S is the set of archimedean places of k,andS=S ν0 . A similar equation∞ holds for γ as well. The global integral of [J.S.2] is ∞∪{ }

I X g I(ϕ, φ, s)= e ϕ n In g ! C ( )GL (ZF) GL ( ) X M (kZ) M ( ) n A n \ n A ∈ n \ n A ψ(trX)E(g,fφ,s)dxdg · n where ϕ is a cusp form in the space of π, φ S(A )andE(g,fφ,s) is the Eisenstein ∈ e series which corresponds to the section fφ,s (deﬁned by (3.4), except F ∗ is replaced by A∗ and ωτ by ωπ). The global functional equation for I(ϕ, φ, s) is obtained from the corresponding functional equation of the Eisenstein series (see [J.S.1], p. 546)

t 1 E(g,fφ,s)=E(g− ,fφ,ˆ1 s) − t where φ is the Fourier transform of φ (with respecte to ψ(x y)) and fφ,ˆ 1 s is 1 · − obtained from (3.4) by replacing s 1 s, ωτ ωπ− ,F∗ A∗.Thus, b 7→ − 7→ 7→ e (6.12) I(ϕ, φ, s)=I(ϕ, φ, 1 s) − where e b I(ϕ, φ, 1 s) − t 1 In X wng− e b= ϕ t 1 In wng− ! C ( )GL (ZF) GL ( ) M (F )ZM ( ) n A n \ n A n \ n A

ψ(trX)E(g,fφ,1 s)dxdg. · − Writing the Euler product expansion for decomposable data at each side of (6.12), e eb we get S 2 (W ,φ ,s) (Wν ,φν ,s)L (π, Λ ,s) L ∞ ∞ L 0 0 S 2 (6.13) = ˜(W ,φ ,1 s)˜ Wν ,φν ,1 s L (φ, Λ , 1 s) L ∞ ∞ − L 0 0 − − ˜ 1 wherewewritetheψ− -Whittakerb function of bϕ with respectb to ψ as Wν,the 1 product of local ψν− -Whittaker functions, and we let W = Wν . (Similar ∞ ν S Q ∈ ∞e notation for φ.) The factors and ˜ in (6.13) are deﬁned as inQ (3.5) and (3.6), L L and can be seen to extend to meromorphic functions in C. From (6.13) and (3.3), we conclude that there is a meromorphic function γ(π , Λ2,s,ψ ) such that ∞ ∞ (6.14) γ(π , Λ2,s,ψ ) (W ,φ ,s)= ˜(W ,φ ,1 s) ∞ ∞ L ∞ ∞ Le ∞ ∞ − and e LS(π, Λ2,b1 s) (6.15) γ(π , Λs,s,ψ )γ(τ,Λ2,s,ψ)= − ∞ ∞ LS(π, Λ2,s) b where we denotee π = ν S eπν . Of course, we replace W in (6.14) by any linear combination∞ of elements∈ ∞ of the form W . Here we∞ remark that the N ν S ∞ local functional equation can be obtained at∈Q each∞ archimedean place separately. The proof is an appropriate adaptation of the global Euler product expansion. Details will appear in a forthcoming paper of J. Cogdell and I. Piatetski-Shapiro.

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 906 DAVID GINZBURG, STEPHEN RALLIS, AND DAVID SOUDRY

2 Thus, we can deﬁne local gamma factors γ(πν , Λ ,s,ψν) at each archimedean place as well, but we won’t need this here. From (6.11) and (6.15), e (6.16) γ(π , Λ2,s,ψ )γ(τ,Λ2,s,ψ)=γ(π ,Λ2,s,ψ )γ(τ,Λ2,s,ψ) ∞ ∞ ∞ ∞ 2 2 wherewedenoteγ(π ,Λ,s,ψ )= γ(πν,Λ ,s,ψν). We know that the local ∞ ∞ ν S e e ε∈Q(s)∞L(1 s) Shahidi gamma factor has the form L(s)− ,whereε(s) is an exponential and L(s) is a product of one-dimensional local L-functions (see [Sh2]). Thus, at the r s 1 place ν0, L(s)= (1 aiq− )− , and at an archimedean place L(s)is,uptoan i=1 − m m Q 1 exponential, of the form Γ( 2 (s + si)) or Γ(s + si). i=1 i=1 2 s The function γ(τ,Λ ,s,ψQ ) is rational in qQ− . This is clear from the functional equation (3.3). The following two properties can be proved: 1 1 (A) Given s0 C, there exist W in W (π ,ψ− ), the ψ− -Whittaker model of e ∞ ∞ ∞ ∞ ∈ n π ,andφ S kν , such that (W ,φ ,s) is nonzero at s = s0.(Wedo ∞ ∞ ∈ ν S L ∞ ∞ not exclude a pole.) ∈Q∞ (B) There exist “Euler factors” G(π ,s)andG(π ,s) of the form ∞ ∞ r 1 1 P (s) Γ( (s + s ))Γ( (ns + s )) 0 2 i 2 b 0 i=1 Y or r

P0(s) Γ(s + si)Γ(ns + s0), i=1 Y (W ,φ ,s) ˜(W ,φ ,s) where P0(s) is a polynomial, such that L ∞ ∞ = gW ,φ (s)and L ∞ ∞ = G(π ,s) ∞ ∞ G(ˆπ ,s) ∞ 1 ∞n gW ,φ (s) are holomorphic for all W W (π ,ψ− )andφ S( kν). ∞ ∞ ∞ ∈ ∞ ∞ ∞ ∈ ν S Rewrite (6.14), using (B), as ∈Q∞ e G(π ,1 s)g (1 s) 2 W ,φ (6.17) γ(π , Λ ,s,ψ )= ∞ − ∞ ∞ − . ∞ ∞ G(π ,s)gW ,φ (s) b b ∞ e ∞ ∞ Let e γ(τ,Λ2,s,ψ) R(q s)= . − γ(τ,Λ2,s,ψ) s This is a rational function of q− . By (6.16) 2 e s 2 (6.18) γ(π , Λ ,s,ψ )=R(q− )γ(π ,Λ ,s,ψ ). ∞ ∞ ∞ ∞ Thus, from (6.17) e g (1 s) s 2 G(π ,1 s) W ,φ (6.19) R(q− )γ(π , Λ ,s,ψ )= ∞ − ∞ ∞ − . ∞ ∞ G(π ,s) · gW ,φ (s) ∞ ∞b ∞ s e s P(q− ) b Write R(q− )= s , a quotient of disjoint polynomials P (x)andQ(x)in [x]. Q(q− ) C s Let x a, a = 0, be a factor of either P (x)orQ(x). Since q− = a has inﬁnitely many− solutions,6 which all lie on a line parallel to the imaginary axis, we can pick a solution s0, far enough from the real axis so that it is not a pole or a zero of

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use CUSPIDAL REPRESENTATIONS OF GL2n AND Sp2n 907

2 ε(s)L(1 s) f γ(π , Λ ,s,ψ ) (recall that it has the form − ), and also s0 is not a pole ∞ ∞ L(s) G(ˆπ ,1 s) or a zero of the quotient of Euler factors G(∞π ,s−) . From (6.19), s0 is a zero, or ∞ g˜W ,φˆ (1 s) apoleof ∞ ∞ − for all W and φ (according to whether x a is a factor gW ,φ (s) ∞ ∞ ∞ ∞ − of P (x)orofQ(x)). Since gW ,φ (1 s)andgW φ (s) are holomorphic, we ∞ ∞ − ∞ ∞ conclude that s0 is a zero of gW ,φ (1 s)(resp.ofgW ,φ (s)), for all W ,φ . ∞ b∞ ∞ ∞ ∞ ∞ −m0 This contradicts (A). Thus Re(x)=αX ,form0 Z, and then (6.18) implies 2 m s b 2 ∈ that γ(π , Λ ,s,ψ )=αq−e 0 γ(π , Λ ,s,ψ ). This completes the proof of the proposition.∞ ∞ ∞ ∞ e References

[B.Z.] I.N. Bernstein and A.V. Zelevinsky, Representations of the group GL(n, F ), where F is a local non-archimedean field, Russian Math. Surveys, No. 3 p. 1-68 (1976). MR 54:12988 [D.M.] J. Dixmier and P. Malliavin, Factorizations de fonctions et de vecteurs indéfiniment différentiables, Bull. Sci. Math., II. Sér. No. 102, p. 471-542 (1978). MR 80f:22005

[F] M. Furusawa, On the theta lift from SO2n+1 to Spn, J. reine angew. Math. 466, p. 87-110 (1995). MR 96g:11052 [G.PS.] S. Gelbart and I. Piatetski-Shapiro, L-functions forfG GLn, in Springer Lecture Notes in Math. No. 1254, p. 53-136 (1987). MR 89k:11038 × [G.R.S.1] D. Ginzburg, S. Rallis and D. Soudry, On explicit lifts of cusp forms from GLm to classical groups, preprint (1997). [G.R.S.2] D. Ginzburg, S. Rallis and D. Soudry, A new construction of the inverse Shimura correspondence, IMRN, No. 7, p. 349-357 (1997). MR 98a:11056 [G.R.S.3] D. Ginzburg, S. Rallis and D. Soudry, Self-dual automorphic GLn-modules and construction of a backward lifting from GLn to classical groups, IMRN, No. 14, p. 687-701 (1997). MR 98e:22013 [G.R.S.4] D. Ginzburg, S. Rallis and D. Soudry, L-functions for symplectic groups, to appear in Bull. de la SMF. [J.R.] H. Jacquet and S. Rallis, Uniqueness of linear periods, Compositio Math., No. 102, p. 65-123 (1997). MR 97k:22025 [J.S.1] H. Jacquet and J. Shalika, On Euler products and the classiﬁcation of automorphic representations I, American J. Math, Vol. 103, No. 3, p. 499-558 (1981). MR 82m:10050a [J.S.2] H. Jacquet and J. Shalika, Exterior square L-functions, in Automorphic Forms, Shimura Varieties and L-functions, Vol. II edited by L. Clozel and J.S. Milne, Perspectives in Math., Academic Press, p. 143-226 (1990). MR 91g:11050 [M.V.W] C. Moeglin, M.-F. Vigneras, J.-L. Waldspurger, Correspondences de Howe sur un corps p-adique, L.N.M. 1291, Springer-Verlag (1987). MR 91f:11040 [Sh1] F. Shahidi, Twisted endescopy and irreducibility of induced representations for p-adic groups, Duke Math. J. No. 66, p. 1-41 (1992). MR 93b:22034 [Sh2] F. Shahidi, A proof of Langlands conjecture on Plancherel measures: complementary series for p-adic groups, Annals of Math. No. 132 p. 273-330 (1990). MR 91m:11095 [Sa] G. Savin, Local Shimura correspondence, Math. Ann. 280, p. 185-190 (1988). MR 89h:22018 [So] D. Soudry, Rankin-Selberg convolutions for SO2`+1: Local theory, Memoirs of AMS No. 500, p. 1-100 (1993). MR 94b:11043

(D. Ginzburg and D. Soudry) School of Mathematical Sciences, Sackler Faculty of Exact Sciences, Tel Aviv University, Tel Aviv 69978, Israel E-mail address: [email protected] E-mail address: [email protected]

(S. Rallis) Department of Mathematics, The Ohio State University, Columbus, Ohio 43210 E-mail address: [email protected]

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use