ON REPRESENTATIONS DISTINGUISHED BY UNITARY GROUPS BROOKE FEIGON, EREZ LAPID AND OMER OFFEN To the memory of Jonathan Rogawski Abstract. Let E=F be a quadratic extension of number fields. We study periods and reg- ularized periods of cusp forms and Eisenstein series on GLn(AE ) over a unitary group of a Hermitian form with respect to E=F . We provide factorization for these periods into locally defined functionals, express these factors in terms of suitably defined local periods and charac- terize global distinction. We also study in detail the analogous local question and analyze the space of invariant linear forms under a unitary group. Contents Index 2 Introduction 3 Part 1. Local Theory 7 1. Notation and preliminaries 7 2. Bessel distributions 15 3. Local identities of Bessel distributions 20 4. Open local periods 25 5. Closed periods 31 6. Applications of the geometric lemma 37 7. Local Bessel identities { the split case 43 8. Local Bessel identities { the unramified case 47 9. Dependence on additive character 50 Appendix A. Inner product for induced representations of GLn, by Erez Lapid and Omer Offen 54 Appendix B. Distinguished representations in the Archimedean case, by Avraham Aizenbud and Erez Lapid 60 Part 2. Global Theory 63 10. Factorization of unitary periods { cuspidal representations 63 11. Factorization of unitary periods { Eisenstein series 67 Appendix C. The relative trace formula, by Erez Lapid and Omer Offen 74 Part 3. Local Applications 76 12. Local Bessel identities and functional equations { the general case 76 13. Multiplicity of unitary periods 82 Appendix D. Upper semicontinuity of multiplicity, by Erez Lapid and Omer Offen 88 References 89 Date: May 4, 2012. The second and third named authors were partially supported by grants from the Israel Science Foundation. 1 Index ∗ w :τ (aM ) , 32 C , 73 ∗ ~ 0 0 (aM )+, 9 w (π ), 22 AG, 64 w~(π), 21 CM (w : σ; λ; ), 13 Fw, 38 0 D ! D , 21 !V , 9 E(g; '; λ), 68 rl(σ; λ), 69 G◦[x], 27 f^(π), 14 H, 8 Rpi(G), 13 I(D; λ), 15 I, ≺, 38 G IM (T ), 11 θ, 31 G G λ 0 , 21 IM (σ), IM (σ; λ), 11 J(' : x; σ; λ), 70 cs(σ; λ), 13, 68 JM (' : x; α, λ), 26 δQ, 8 0 0 L(s; π1 × π2), "(s; π1 × π2; ), γ(s; π1 × π2; ), 13 B(f : σ ; λ), 19 L(x), R(y), 9 Bπ0 , 18, 65 M(w; σ; λ), M(w; W(σ); λ), 12 θ~, 12 0 N(w; σ; λ), N(w; W(σ); λ), 13 n(σ ; λ), Jσ0 (x; α, λ), 29 P , M , U , 7 kgk, 16 0 0 0 − − − Pκ, Mκ, Uκ, 7 P , M, U , 8 Qx, 7 κ−, 8 − X[x], G0[x], 20 σ−, λ , 12 [·; ·]π, 54 BIM;x, BIM , 24 A, 3 Px, 3 | AM,σ, 68 π · χ, π[s], π , 10 M ∗ _ ∆0 , 8 π , π , 10 ∗ EG(X; π ), 17 F(' : x; β; λ), 36 Z(' : x; β; λ), 32 0, , 7 π 0 δg , 11 0, 0, 11 LQ(σ; λ), 14 R(G), 10 x Φ ! f 0, f ! f 0, 20 ρ, 9 Φ α, 17 σ[Z], 14 ΦM , 28 σ1 × · · · × σt, 11 0 0 0 0 0 0 supp (π), 14 Πcusp,Πusqr,Πsqr,Πtemp,Πgen,Π , 11 c τ;ti τ;an S(Y ), 9 Π ,Π , 22 t 0 0 t 0 0−1 τ ∗ 0 ( U0×U0; 0× 0 ) Πcusp,Πusqr,Πsqr,Πtemp,Πgen, Π, Π , 11 S (G ) , 19 −1 0 ∗ (U0; ) Πbc-gen, 22 S (X) 0 , 21 τ π0 Πreg, 84 α , BIx, BI, 23 0 Πunr, 14 B~(σ ; λ), 31 W , 8 1Γ, 1Q, 9 W M (L), 8 µ ' ξ,Γ (ν; µ), 11 W [M], M(w), 38 L;M 2 ' , 11 Ξ, 69 M ('; σ; λ), 12, 68 a∗ , a , a∗, a , 8 W Q Q 0 0 π, Wπ, 64 ai, 21 W W (π), 11 α ◦ T , 17 W(π0), 65 A(σ; λ), 68 W(π), 18 A , 64 π w(x), 7 B(π), b ,[B](π), [b] , 21 π π W(' : σ; λ), 12, 68 bc, 3, 21 f ∗ Φ, 9 ; , 15 E C rL;M , rM , 11 `;`^ BD (f), 15 wκ, 8 ~ DL(%; λ), 37 wσ, 12 % M β , 36 w0 , 8 X◦, 25 x • g, 7 G0, G, X, 3 G0±, 19 π0 XL, 31 rl0 , 65 !π, 10 η, 3 D· χ, 15 D◦, 15 C, 64 2 ON REPRESENTATIONS DISTINGUISHED BY UNITARY GROUPS 3 Introduction Let E=F be a quadratic extension of number fields with Galois involution τ and let A = AF be the ring of adeles of F: Denote by η : F ∗nA∗ ! {±1g the quadratic character associated to E=F 0 by class field theory. Let G = GLn =F and let G be the restriction of scalars from E to F of GLn =E. Let X = fg 2 G : tgτ = gg be the F -subvariety of Hermitian matrices in G: Here tg is the transpose of a matrix g: For x 2 X(F ) denote by Gx = fg 2 G : tgτ xg = xg the unitary group (defined over F ) determined by x: The unitary period integral Px(φ) of a cusp form φ of G(A)(= GLn(AE)) is defined by Z Px(φ) = φ(h) dh: x x G (F )nG (A) A cuspidal automorphic representation π of G(A) is called Gx-distinguished if there exists a cusp form φ in the space of π such that Px(φ) 6= 0: The study of unitary period integrals has seen much progress in recent years. In particular, Jacquet showed that an irreducible cuspidal automorphic representation π of G(A) is in the image of quadratic base change bc if and only if π is Gx- distinguished for some x 2 X(F ) [Jac05a]. In fact, it suffices to take the quasi-split unitary group [Jac10]. In order to obtain these results, Jacquet, in the course of many years, invented and developed the relative trace formula on G(A) and together with his collaborators compared it to a Kuznetsov trace formula on G0(A) (see [Ye88, Ye89, JY90, Jac92, JY92, Ye93, Ye94, Jac95, Ye95, JY96, Jac98, Ye98, JY99, Jac01, Jac02, Jac03a, Jac03b, Jac04b, Jac05a, Jac05b, Lap06, Jac10]). This comparison of trace formulas gives additional information about the period integrals. To describe it, we first recall that by Arthur-Clozel, an irreducible cuspidal automorphic representa- tion π of G(A) is in the image of quadratic base change if and only if π is Galois invariant ([AC89]). In this case, π is the image of exactly two irreducible cuspidal representations of G0(A) of the form π0, π0 ⊗ η. We say that E=F splits at infinity if every Archimedean place of F splits in E: When n = 3; x = e and E=F splits at infinity, Jacquet showed in [Jac01, Theorem 1] that the period integral Px is factorizable on π and that its local components are governed by certain identities 0 0 between Bessel distributions associated to the local components πv of π and πv of π. The restric- tion to the case n = 3 was made since at the time the necessary relative trace identity was only available in that case. Based on recent developments (mainly [Jac03b, Jac04b, Jac05a, Lap06]), the method and result of Jacquet can be generalized to any n and x (Theorem 10.2). We also remove the assumption that E=F splits at infinity thanks to a recent result by Aizenbud and Gourevitch ([AG]) extending smooth transfer to the Archimedean case. As pointed out in [Jac01], the factorization of the global unitary period is far from formal since there is no multiplicity one for the local invariant functionals. For instance, for a quadratic extension of p-adic fields, any irreducible unramified principal series representation admits 2n−1 linearly independent invariant functionals with respect to a given unitary group. The failure of multiplicity one reflects the fact that locally, quadratic base change is not one-to-one, even up to a twist by η. Along with the factorization we provide for any x 2 X(F ) a criterion for an irreducible, Galois-invariant, cuspidal automorphic representation π = bc(π0) of G(A) to be Gx-distinguished. Namely, π is Gx-distinguished except for purely local obstructions depending on the (finitely many) 0 x 0 local components of π (rather than π) at the places where G is not quasi-split. (We recall that πv is generally not determined by πv, even up to a quadratic twist.) In the non-Archimedean case (for 0 0 which non-quasi-split unitary groups exist only for n even) the obstruction is that πv ' πv ⊗ ηv. The Archimedean obstruction is a little more complicated but it is still given explicitly in terms 0 0 0 0 0 0 0 of πv. More precisely, let w~ (πv) be the largest k such that we can write πv = δ × σ where δ is 0 0 a representation of GL2k(Fv) such that δ ' δ ⊗ η and × denotes parabolic induction. Then we have the following characterization. 4 BROOKE FEIGON, EREZ LAPID AND OMER OFFEN Theorem 0.1. Let x be a Hermitian form of rank n with respect to E=F and let π be an irreducible x cuspidal representation of GLn(AE). Then π is G -distinguished if and only if there exists a 0 0 cuspidal representation π of GLn(AF ) such that π = bc(π ) (i.e., π is Galois invariant) and 0 0 w~ (πv) is not bigger than the Witt index of x with respect to Ev=Fv for all inert places v of F over which Gx is not quasi-split.