On the Self-Dual Representations of a P-Adic Group Dipendra Prasad

IMRN International Mathematics Research Notices 1999, No. 8

On the Self-dual Representations of a p-adic Group

Dipendra Prasad

In an earlier paper [P1], we studied self-dual complex representations of a finite group of Lie type. In this paper, we make an analogous study in the p-adic case. We begin by recalling the main result of that paper. Let G(F) be the group of F rational points of a connected reductive algebraic group G over a finite field F. Fix a Borel subgroup B of G, defined over F, which always exists by Lang’s theorem. Let B = TU be a Levi decomposition of the Borel subgroup B. Suppose that there is an element t0 ∈ T(F) that operates by 1 on all the simple roots in U with 2 respect to the maximal split torus in T. It can be seen that t0 belongs to the center of G. 2 We proved in [P1] that t0 operates by 1 on a self-dual irreducible complex representation , which has a Whittaker model, if and only if carries a symmetric G-invariant bilinear form. If carries a symmetric G-invariant bilinear form, then we call an orthogonal representation. If is an irreducible admissible self-dual representation of a p-adic group G, then there exists a nondegenerate G-invariant bilinear form B: → C. This form is unique up to scalars by a simple application of Schur’s lemma, and it is either symmetric or skew-symmetric. The aim of this paper is to provide for a criterion to decide which of the two possibilities holds in the context of p-adic groups similar to our work in the finite field case in [P1]. This partly answers a question raised by Serre (see [P1]). We are able to say nothing about representations not admitting a Whittaker model. Our method, however, works in some cases even when there is no element in T which operates on each simple root by 1. This is the case, for instance, in some cases of SLn and Spn, where we work instead with the similitude group where such an element exists; then we deduce information on SLn and Spn.

Received 29 December 1998. Revision received 21 January 1999. Communicated by Joseph Bernstein. 444 Dipendra Prasad

We also provide some complements to [P1], which were motivated by a similar result for algebraic representations of reductive groups (see [St, Lemma 78]). The criterion in [P1] and [St] therefore give two elements in the center of G such that the action of one of them determines whether a self-dual algebraic representation of G(F) is orthogonal, whereas the other element determines whether a self-dual complex representation of G(F) is orthogonal. We prove that these two elements in the center of G are actually the same. This also gives us an opportunity to give a proof of the result for algebraic representations of G(F) following the methods of [P1]. Finally, we extend another observation of the author from the context of ﬁnite groups to p-adic groups, which gives a method of checking whether a representation is self-dual.

1 Orthogonality criterion for algebraic representions

We recall that the theorem characterising self-dual representations of ﬁnite groups of Lie type in [P1] was proved using the following elementary result whose proof we sketch for the sake of completeness.

Lemma 1. Let H be a subgroup of a ﬁnite group G. Let s be an element of G which normalises H, and whose square belongs to the center of G. Let : H → C bea1- dimensional representation of H which is taken to its inverse by the inner conjugation action of s on H. Let be an irreducible representation of G in which the character appears with multiplicity 1. Then, if is self-dual, it is orthogonal if and only if the element s2 belonging to the center of G operates by 1 on .

Proof. Let v be a vector in on which H operates via . Clearly,Hoperates via the character 1 on the vector sv, and the space spanned by v and sv is a nondegenerate subspace for the unique G-invariant bilinear form on . It has dimension 1 or 2. The inner product of v with sv must be nonzero, but since the inner product is G-invariant, and in particular,s-invariant, the conclusion of the lemma follows.

Remark. In the context of G = G(F), the lemma is used with H = U(F) and a nondegenerate character on U. The character appears with multiplicity 1 by a theorem of

Gelfand and Graev for G = GLn(F), which was generalised by Steinberg to all reductive groups.

Lemma 2. Let G be a reductive algebraic group over an algebraically closed ﬁeld k.

Then there exists an element z0 in the center of G of order 2, which operates on an irreducible, self-dual algebraic representation of G by 1 if and only if the representation is orthogonal. On the Self-dual Representations of a p-adic Group 445

Proof. We offer two proofs: one is in the spirit of the proof of the earlier lemma, and the other is more classical. Let B = TU be a Levi decomposition of a Borel subgroup B in G. The choice of the Borel subgroup deﬁnes an ordering on the set of roots of G with respect to T. Let w0 be an element in G representing the element in the Weyl group, which takes all the positive roots to negative roots. By the theory of highest weights, irreducible algebraic representations of G are in bijective correspondence with dominant integral weights. It follows that an irreducible representation corresponding to dominant weight is self-dual if and only if =w0(). We apply Lemma 1, or rather its analogue for algebraic representations of algebraic groups, to the highest weight of T in . Let v0 be a vector in on which T operates via . The vector v0 is unique up to scalars. The square of the Weyl group element w0, which takes all the positive roots to negative roots, is the identity element in the Weyl 2 group of G; and in fact, we can choose a representative for w0 in G such that w0 belongs to the center of G. (The author thanks Hung Yean Loke for conﬁrming this for the group

E6.) If w0 =1, then it is easy to see that for any choice of a representative x in G for 2 w0 ∈ N(T)/T, where N(T) is the normaliser of T, x is independent of x, and belongs to the 2 = center of G.Weletw0 t0, where t0 is an element in the center of G.

Clearly w0() = is also a weight of . Therefore, by the complete reducibility of as a T module, the subspace generated by v0 and w0(v0) is a nondegenerate subspace of for the unique G-invariant bilinear form on .Wehave

h i=h 2 i=h i v0,w0(v0) w0(v0),w0(v0) w0(v0),t0v0 .

It follows that t0 operates by 1 on an irreducible self-dual representation if and only if is an orthogonal representation. We now turn to the second proof, which is the standard proof given, for example, in Bourbaki.

Let u0 be a regular unipotent element in U (i.e., an element in U whose component → in each U, simple , is nontrivial). Let j: SL2 G be a principal SL2 in G that takes 11 the element to u and the diagonal torus in SL to T. (The mapping j need not be 01 0 2 injective.)

An irreducible representation of G with highest weight , when restricted to SL2 via the mapping j, is a sum of certain irreducible representations of SL2. It can be seen that the representation of SL2 with highest weight the restriction of to the diagonal torus in SL2 appears with multiplicity 1. Therefore, if is self-dual, then it is orthogonal or symplectic depending exactly on whether this representation of SL2 is orthogonal or symplectic. 446 Dipendra Prasad 0 Let k = i , where i is a fourth root of 1. The inner conjugation action of k on 0 i the unique positive simple root of SL2 is by 1. This implies that the inner conjugation action of k on every simple root of G is by 1. This proves that j(k2) belongs to the center of G, and its action on a self-dual irreducible representation of G is by 1, if and only if the representation is orthogonal. Since the two proofs must deﬁne the same element in the center of the group, we obtain the following corollary. (It needs to be noted that, under the assumption of

Corollary 1 below, since w0 =1, all the algebraic representations of G are self-dual. In particular, for any element of order 2 in the center of G, there is a faithful self-dual representation of G nontrivial on that element.)

Corollary 1. Let G be a simple algebraic group with center Z for which w0 =1. For any 2 choice of w0 as an element in G, w0 is an element in the center of G and is independent of the representative in G for wo. Let T be a maximal torus in G, and let t0 be an element 2 in T which operates on all the simple roots of G by 1. Then t0 is also in the center of 2 2 2 G, and it is a well-deﬁned element in Z/Z . The two elements w0 and t0, thought of as elements in Z/Z2, are equal.

The following corollary is clear from the second proof of the lemma above.

Corollary 2. The element in the center of G that determines whether a self-dual algebraic representation of G is orthogonal, is the same as the element in the center of G that determines when a self-dual generic complex representation of G(F) is orthogonal. Both the elements in the center are considered up to squares in the center.

2 Orthogonality criterion for p-adic groups

Let G = G(k) be the group of k-rational points of a quasi-split reductive group over a local ﬁeld k. Let B = TU be a Levi decomposition of a Borel subgroup of G. Fix a nondegenerate character : U(k) → C . By a theorem of Gelfand and Kazhdan for GLn and Shalika [Sh] for general quasi-split reductive groups, there exists at most one dimensional space of linear forms `: → C on any irreducible admissible representation of G(k) that transforms via when restricted to U(k). Assume that there is an element t0 ∈ T(k) such that the inner conjugation action of t0 is by 1 on all the simple roots in U. Therefore, the inner 1 conjugation action of t0 on U takes to . Since the group U(k) is noncompact, one cannot use Lemma 1 in this context. However, a very nice idea of Rodier enables us to work with a compact approximation of (U, ), which we brieﬂy describe now. We refer the reader to Rodier’s article [R] for details. On the Self-dual Representations of a p-adic Group 447

Let U denote the unipotent radical of the Borel subgroup B of G which is opposite to B. Fix an integral structure on G so that one can speak about G(pn), where p is the maximal ideal in the maximal compact subring O in k generated by an element ∈ p.IfG is a quasi-split group over k which splits over an unramiﬁed extension of k, then G has a natural integral structure whose reduction modulo p is reductive. However, for our purposes, all we need is for the groups G(pn) to have the Iwahori factorisation,

G(pn) = U(pn)T(pn)U(pn).

We assume that the character on U(k) has been so normalised that it is trivial on O- rational points of every simple root, but not on 1O-rational points of any simple root. n Deﬁne a character n on G(p )by

2n n(u tu) = n(u) = ( u), where we leave the task of making the meaning of 2nu precise to the reader. Rodier proves the following result in [R]. Actually, Rodier works only with split groups, but his work extends easily to quasi-split groups.

Proposition 1. Let be an irreducible admissible representation of G(k). Then,

dim HomG(pn)(, n) dim HomG(pn+1)(, n+1) dim HomU(, ).

Moreover, dim HomG(pn)(, n) = dim HomU(, ) for all n large enough. Therefore, if has a Whittaker model, then for some n, dim HomG(pn)(, n) = 1.

Now the analogue of Lemma 1 can be applied to the subgroup H = G(pn) with the character n on it to deduce the following proposition. (We note that since the element n n n n t0 ∈ T(k) operates by 1 on all the simple roots of G, it preserves G(p ) = U (p )T(p )U(p ) 1 and takes the character n to n .)

Proposition 2. Let G be a quasi-split reductive algebraic group over a p-adic ﬁeld k with B = TU a Levi decomposition of a Borel subgroup of G. Assume that T(k) has an element = 2 t0 which operates by 1 on all the simple roots in U. Then z0 t0 belongs to the center of G and operates on an irreducible, self-dual generic representation of G by1ifand only if the representation is orthogonal.

3 Examples

In this section, we list some groups for which Proposition 2 applies to give a criterion for orthogonality of self-dual generic representations. As observed in [P1] for the case of ﬁnite ﬁelds, here is what the proposition says for the split groups. 448 Dipendra Prasad

(1) GLn: In this case, all the self-dual generic representations are orthogonal.

(2) SLn:Ifn 6 2 mod 4, then all the self-dual generic representations are orthogonal; if n 2 mod 4, and the local ﬁeld k has a 4th root of 1, then a self-dual generic representation is orthogonal if and only if 1 operates by 1. (3) Sp(2n): In this case, a self-dual generic representation is orthogonal if and only if 1 operates by 1. (4) SO(n): In this case, a self-dual generic representation is always orthogonal.

(5) Simply connected exceptional groups: In all cases except for E7, all self-dual generic representation is orthogonal. In the case of E7, a self-dual generic representation is orthogonal if and only if the center, which is Z/2, operates trivially.

Remark. In the case of Sp(2n), Proposition 2 does not directly apply, as there may not be an element in the group which operates on all simple roots by 1. However, as in [P1], one can give a proof by going to the symplectic similitude group, and again using Rodier’s method of approximating (U, ) by compact groups.

Remark. In the case of GLn(Fq), we were able to prove that any, not necessarily generic, self-dual representation of GLn(Fq) is orthogonal. We have not been able to do this here in the p-adic case, though we believe it should be true. We also remark that there is another proof about the orthogonality of self-dual representations of GLn in the p-adic in [PRam] that depended on the theory of new forms due to Jacquet, Piatetski-Shapiro, and Shalika. But the theory of new forms is also available only for generic representations.

Remark. In [P1], we constructed an example of a self-dual representation of SL(6, Fq) with q 3 mod 4 for which 1 acts trivially, even though the representation is symplectic. We also constructed an example of a self-dual representation of SL(6, Fq) with q 3 mod 4 for which 1 acts nontrivially, even though the representation is orthogonal. This construction remains valid for any p-adic ﬁeld of residue ﬁeld cardinality q 3 mod 4, and it works for any SL(4m + 2).

4 A criterion for self-dual representations

The following lemma was observed in [P2].

Lemma 3. If G is a ﬁnite group, and H is a subgroup of G such that g → g1 takes every double coset of G with respect to H to itself, then every irreducible representation of G with an H ﬁxed vector is self-dual, and is in fact an orthogonal representation. On the Self-dual Representations of a p-adic Group 449

Proof. Let be an irreducible representation of G with an H ﬁxed vector v ∈ . Fix a G-invariant Hermitian form on , and consider the matrix coefﬁcient

f(g) =hgv, vi.

1 Clearly,f(h1gh2) = f(g) for all h1,h2 in H. Therefore, since g → g takes every double coset of G with respect to H to itself, it follows that f(g) = f(g1). However,f(g1)isa matrix coefﬁcient of , and by orthogonality relations, matrix coefﬁcients of distinct representations are orthogonal. Therefore,= . Since g → g1 takes every double coset of G with respect to H to itself, it follows that (G, H) is a Gelfand pair, and the space of H-invariant vectors is a 1-dimensional nondegenerate subspace of , and hence , must be an orthogonal representation.

We extend this lemma to p-adic groups as follows. We have not been able to prove the last part of the previous lemma, which deals with orthogonal representations.

Lemma 4. Let H be a closed subgroup of a p-adic group G. Assume that any distribution on G that is H-bi-invariant, is invariant under the involution g → g1. Then every irreducible admissible representation of G with an H-invariant linear form on both and is self-dual.

Proof. Let

`: → C

`0: → C be nonzero H-invariant linear forms on and its dual. Let f be a locally constant, compactly supported function on G. The space of such functions operates on in the standard way and operates also on the space of all linear forms on . In particular, it makes sense to talk about (f)`. It is easy to see that (f)` in fact belongs to the smooth dual of , and therefore it makes sense to deﬁne the distribution B on G by

B(f) = `0((f)`).

The distribution B is easily seen to be H-bi-invariant. The distribution B is called the relative character of G with respect to H. It was introduced by H. Jacquet. Reversing the roles of and , one can deﬁne another distribution B0 on G by

B0(f) = `((f)`0). 450 Dipendra Prasad

If for a function f on G, we deﬁne f∨ to be the function

f∨(g) = f(g1), it can be seen that

B0(f) = B(f∨).

However, by hypothesis, an H-bi-invariant distribution on G is invariant under the involution g → g1, and therefore,B(f) = B(f∨). So,

B0(f) = B(f).

Now we appeal to the following proposition, which says that the relative character of a representation of G characterises the representation to deduce that = .

Proposition 3. Let 1 and 2 be two irreducible admissible representations of a p-adic group G. Assume that for a subgroup H of G, all the representations 1,1,2,2 have

H-invariant linear forms. Then if a relative character of 1 is equal to a relative character of 2, then 1 is isomorphic to 2.

Proof. We give a proof of this result, as it does not exist in the literature. (The author thanks J. Hakim for supplying this proof.) We prove that 1 and 2 are isomorphic by using a variant of the theorem that two irreducible representations are isomorphic if and only if a matrix coefficient of 1 is also a matrix coefficient of 2. The variant we use 0 S is to look at the distribution v1(fv1), where f belongs to the space (G) of locally constant, 0 compactly supported functions on G, and v1 and v1 are nonzero vectors in 1 and 1; the vector fv1 is the result of the natural action of f ∈ S(G)onv1 ∈ 1. We call this distribution a generalised matrix coefficient of 1. The variant we use is that if a generalised matrix coefficient of 1 (which is a distribution on G) is the same as for 2, then 1 and 2 are 0 isomorphic. We outline the simple proof of this. For this, let F 0 (g) = v (gv1)bethe v1,v1 1 corresponding matrix coefficient on G. Clearly, the distribution we called a generalised matrix coefficient is given by Z → F 0 dg, ∈ S(G). v1,v1

Now it is clear that if a generalised matrix coefﬁcient of 1 is the same as one for 2, then

1 and 2 share a matrix coefﬁcient, and hence they are isomorphic. On the Self-dual Representations of a p-adic Group 451

Let

`1: 1 → C 0 → C `1: 1

`2: 2 → C 0 → C `2: 2 be nonzero H-invariant linear forms. We assume that

0 = 0 `1(1(f)`1) `2(2(f)`2) for all f in the space S(G) of locally constant, compactly supported functions on G. Observe that for any f ∈ S(G),v0 = (f )` ∈ and for any f ∈ S(G),v = (f )`0 ∈ . Therefore, 1 1 1 1 1 1 2 1 1 2 1 1 0 0 0 it makes sense to talk about 1(f)v1 (v1). We choose f1 and f1 so that v1 and v1 are nonzero. We have

0 = 0 1(f)v1 (v1) 1(f)1(f1)`1 (1(f2)`1) = ∨ 0 1(f2 f f1)`1 (`1).

0 = ∈ = 0 ∈ Similarly, deﬁne v2 2(f1)`2 2 and v2 2(f2)`2 2.Wehaveasbefore, 0 = ∨ 0 2(f)v2 (v2) 2(f2 f f1)`2 (`2).

Therefore, for all f ∈ S(G), 0 = 0 1(f)v1 (v1) 2(f)v2 (v2),

0 where v1 and v1 are nonzero vectors in 1 and 1. From the remark at the beginning of the proof of this proposition,1 and 2 are isomorphic.

Corollary 3. An irreducible admissible representation of GL2(D), where D is a division algebra over a local ﬁeld k that has a nonzero invariant form for the subgroup H = D D sitting diagonally in GL2(D), must be self-dual and orthogonal.

Proof. It was proved in [P2] that a distribution on G which is H bi-invariant is invariant under the involution g → g1. It therefore sufﬁces to check that whenever a representation of GL2(D) has an H-invariant linear form, its contragredient also has an H-invariant linear form. This is a simple consequence of the Kirillov theory developed in [PR].

We now prove that a self-dual representation of GL2(D) containing an invariant form for the subgroup H = D D must be orthogonal. For this, we use the result proved 452 Dipendra Prasad in [PR] that if has a D D invariant form, then it also has a Shalika form which is unique up to scalars. We recall that is said to have a Shalika form `: → C if ` is left invariant by the diagonal matrices of the form (x, x),x∈ D , and transforms under the 1 upper triangular unipotent matrices X by (tr(X)), where is a nontrivial character 01 on k. The method of Rodier, as also observed in [PR], applies to GL2(D), and gives a compact approximation to the pair (U, tr), and therefore for some compact group Un, there is a character n that appears in with multiplicity 1. This implies orthogonality of by Lemma 1.

We end the paper with the following question.

Question. Let be an irreducible admissible orthogonal representation of GL2(D), where D is a division algebra over a local ﬁeld k. Does have an invariant form for the subgroup H = D D sitting diagonally in GL2(D), and therefore a Shalika model?

Acknowledgments

Dipendra Prasad thanks the Tata Institute of Fundamental Research, Bombay for providing the wonderful atmosphere where this work was completed. He would also like to thank Jeff Hakim for providing the proof of Proposition 3.

References

[P1] D. Prasad, On the self-dual representations of a ﬁnite group of Lie type, J. Algebra 210 (1998), 298–310. [P2] , Some remarks on representations of division algebras and Galois groups of local ﬁelds, J. Number Theory 74 (1999), 73–97.

[PR] D. Prasad and A. Raghuram, Kirillov theory for GL2(D) where D is a division algebra over a non-Archimedean local field, preprint, 1999. [PRam] D. Prasad and D. Ramakrishnan, Lifting orthogonal representations to spin groups and local root numbers, Proc. Indian Acad. (Math. Sci.) 105 (1995), 259–267. [R] F. Rodier, “Modèle de Whittaker et charactères de représentations” in Non-commutative Harmonic Analysis (Actes Colloq., Marseille-Luminy, 1974), Lecture Notes in Math. 466, Springer-Verlag, Berlin, 1975, 151–171.

[Sh] J. Shalika, The multiplicity one theorem for GLn, Ann. of Math. 100 (1974), 171–193. [St] R. Steinberg, Lectures on Chevalley Groups, notes prepared by John Faulkner and Robert Wilson, Yale University, New Haven, CT, 1968.

The Mehta Research Institute of Mathematics & Mathematical Physics, Chhatnag Road, Jhusi, Allahabad 211019, India; [email protected]