Proc. Nadl. Acad. Sci. USA Vol. 91, pp. 6026-6029, June 1994 Minimal representations, geometric quantization, and unitarity (Joseph ideal/nilpotent orbit/syMplec manfd/half-form/hypergeometrlc fuctio) RANEE BRYLINSKItI AND BERTRAM KOSTANT§ tPennsylvania State University, University Park, PA 16802; *Harvard University, Cambridge, MA 02138; and §Department of Mathematics, Institute of Technology, Cambridge, MA 02139 Contributed by Bertram Kostant, March 14, 1994

ABSTRACT In the framework of geometric quantization in H from the associated nilpotent orbit in g appeared first in we exlicitly construct, in a uniform fashion, a unitary minimal ref. 8, and there Vogan worked out the existence, unitarity, representation irT of every simply-connected real G. and K-type decomposition for several cases including some such that the maximal compact subgroup ofG. has finite center of the cases in the table. In. ref. 9 Vogan proved that, as a and G. admits some minimal representation. We obtain alge- virtual K-module, H must equal a multiple of the space of braic and analytic results about frf. We give several results on sections of a K-homogeneous half-form line bundle on Y the algebraic and symplectic geometry ofthe minimal nipotent minus some finite-dimensional K-module. orbits and then "quantize" these results to obtain the corre- Let 0 C o* be the minimal (nonzero) coadjoint orbit so that sponding representations. We assume (Lie Go)C is simple. o corresponds under the Killing form to Ornin. Then 0 is a complex symplectic manifold with respect to the Kirillov- Let Go be a simply-connected simple real Lie group and let Kostant-Souriau symplectic form. We have g* = f* e p* in K0 be a maximal compact group. Let g be the complexifica- the obvious way. A necessary condition for Go to admit a tion of %o = Lie Go with g = f + p a complexified Cartan minimal representation is that the set Y = o n p* is decomposition corresponding to go. Let G be a simply- nonempty-we assume this condition from now on. The set connected complex Lie group with g and let K be Yis then a smooth K-homogeneous Langian submanifold the subgroup corresponding to I. We assume throughout this of 0; in fact, Y is the cone of highest weight vectors in the note that the pair (G0, K0) satisfies the following equivalent simple K-module p*. In particular dim Y = Y2dim 0 and Y = conditions: (i) the associated symmetric space GO/Ko is K*C where E p*. The K-action on Y defines a Hamiltonian non-Hermitian, (ii) K0 has finite center, and (iii) p is irreduc- K-action on the cotangent bundle T* Y (equipped with its ible as a f-module. We will call an infinitesimally irreducible canonical symplectic form) with moment map IK: T* Ye f*. representation ire, of Go on a complex Hilbert space minimal We find that although 0 admits no K-invariant polariza- if its annihilator Ann ir, in the enveloping algebra U(g) is tion, in fact T* Y and a (ramified) 2-fold covering of 0 share equal to the Joseph ideal J (1)-i.e., 7rO, is "attached" to the a common smooth symplectic K-invariant open dense set M. minimal nilpotent orbit and the primitive ideal Ann ir, is Consequently the Hamiltonian G-action on 0 defines an completely prime. We assume throughout g is simple. infinitesimal Hamiltonian g-action on M-i.e., an infinitesi- In this note we extend the methods of ref. 2 to obtain, in mal action by vector fields which admits a moment map pu: auniform way, analogous results on the construction, unitary M-- g*. In this way, the infinitesimal f-action on T* Yis being structure, and harmonic analysis ofa minimal representation enlarged, over M (but not over all of T* Y), to an infinitesimal 'rr ofGo for each case where 'rw exists. In addition, we explain g-action. We may thus regard M as "Hamiltonian (g, K)- some of our results on the symplectic geometry of the M is intrinsic to the of minimal nilpotent orbit Oiam C g and show how these results space." Our construction of geometry underlie our explicit construction of the corresponding min- T* Y-one characterization is that M is equal to the union of imal representation. In our models the Harish-Chandra mod- all K-orbits on T* Y of principal orbit type. ule H for 1r, is the space IMY, N1/2) of algebraic sections of To state our results in more detail we introduce the the (K-homogeneous) half-form line bundle N112 on a (K- following. Let R(X) be the algebra of regular functions on an homogeneous) Lagrangian Y of Ongn. (H is spherical if and algebraic variety X. Then we have an inclusion R(T*Y) C only if N1/2 is K-equivariantly trivial-this occurred for the R(M) ofPoisson algebras where the symplectic form on T* Y three cases treated in ref. 2.) Furthermore, the elements of g defines a symplectic form on M. Each 4 E R(M) defines a act by explicit pseudodifferential operators on r(Y, N1/2). Hamiltonian vector field f# on M. This places our results clearly in the scheme of geometric As Y is K-homogeneous, each x E f defines a vector field quantization. Finally, we give explicit formulas in terms of nx on Y. Let #K(X) E R(T* Y) be the symbol of 'f and let OK hypergeometric functions for the unitary structure of iro and : S(f) -- R(T* Y) be the corresponding algebra homomor- also for a matrix coefficient of 7rT, restricted to an SL(2, R) phism, where S(f) = @,i:oS'(f) is the graded symmetric root subgroup. algebra of f. Then qK is the comorphism to AXK. The pairs (g, f) occurring here fall naturally into 10 cases- Let A E R(T* Y) be the symbol of the Euler vector field E see Table 1 below. The existence of Hilbert spaces carrying on Y. (The Euler vector field on a vector space V, or on any these unitary minimal representations, along with the K-type cone inside V, maps each linear function to itself.) Let a be decomposition of H, had already been established by meth- the unique vector field on T* Ywhich coincides with the Euler ods adapted to the individual cases, with the exception of vector field on every fiber of the cotangent fibration ty: T* Y case x if q 2 6: for cases i-iii see ref. 3; for iv-vii see ref. 4; -+ Y. Then we have a K-invariant algebra bigrading R(M) = for viii see ref. 5; for ix see refs. 6, 3 (ifp = q), and 7; and for EJpqezRpsq(M), where Rp,q(M) is the intersection of the x see ref. 5 if q = 4. The notion of determining the K-types p-eigenspace of {A with the q-eigenspace of a. Then OK(Sq(f)) C R(oq)(M), while Rp(Y) C R(p,o)(M), where Rp(Y) is the The publication costs ofthis article were defrayed in part by page charge p-eigenspace ofE and R(Y) C R(T* Y) is the inclusion defined payment. This article must therefore be hereby marked "advertisement" by ty. The Poisson bracket satisfies {R(p,q)(M), R(p',q')(M)} C in accordance with 18 U.S.C. §1734 solely to indicate this fact. R(p+p',q+q'-1)(M). Consequently, R,(M) = @p+q=1Rp,q(M), 6026 Downloaded by guest on October 1, 2021 Mathematics: Brylinski and Kostant Proc. NatI. Acad. Sci. USA 91 (1994) 6027

the 1-eigenspace of the vector field CA + a, is an (infinite- From now on we exclude two cases: (i) g = ?t(n, C), n - dimensional) Lie subalgebra of R(M) containing MK-. 3, since there the Joseph ideal is not defined, and (ii) g = ?w(p Let A' be the group of symplectic diffeomorphisms of T* Y + q, C) with p + q odd and p, q .4 since there (Howe and) defined by the group sa of K-invariant automorphisms of Y. Vogan (8) has shown that no minimal representation exists Then al' A C*, as .s is the one parameter group giving and, indeed, no K-homogeneous half-form bundle exists. We the flow of E. note that the requirement that 0 meets p* has already THEOREM 1. Let M C T*Y be the complement ofthe divisor excluded the following five cases of non-Hermitian symmet- (A = 0) so that M is a Zariski open dense K-invariant subset. ric pairs (g, f): (KI(2n, C), ?op(2n, C)) where n 2 2, (Bo(p + Then, up to the action of d', there is a unique finite- 1, C), Bo(p, C) + 5o(l, C)) where p 2 3, (4p(2p + 2q, C), dimensional subspace r C R,(M) such that r contains 4'0dt), ?p(2p, C) + 4p(2q, C))wherep, q . 1, (F4, !5p(9, C)), and (E6, r is closed under Poisson bracket, and r is isomorphic as a F4). This list follows from the classification in ref. 10 (see Lie algebra to g. The image ofthe moment map tt: M -g* Table 1 in ref. 11), as we show that 0 fails to meet p* if and corresponding to r lies in 0, andfurthermore, p(M) C 0 is only if K has a Zariski dense orbit on 0. a Zariski open dense K-invariant subset. Finally, pu defines a Now Y admits a unique (up to isomorphism) line bundle 2-to-1 cover M -* A(M) and the corresponding lifting of the N1/2 such that N112 ® N112 = N, where N is the top exterior Euler vector field on p(M) is equal to kA+,. power of the cotangent bundle on Y and, furthermore, N112 In fact, A2 = Fo°., where F E R(O), and then p(M) C 0 is has a unique K-homogeneous structure. N112 is then the the complement ofthe divisor (F = 0). Moreover, A and Fare "half-form" line bundle on Y. The K-action on N1/2 defines primitive K-invariants in that R(T* Y)K = C[A] and R(O) = a K-module structure on the space C[F]. Next we explicitly describe (a choice of) the Lie algebra r H = F(Y, N1/2) in terms of the 4K(x) (x E f), A and the functions on Y. Let Z(V) be the K-orbit of highest vectors in an ofalgebraic sections ofN1/2 and also a corresponding algebra weight homomorphism irK: U(f) -- End H. irreducible K-module V. Let v E Z(P). Then the Weyl group We have some natural differential operators on H whose translates ofv form a basis by weight vectors ofp. Associated symbols are the functions occurring in 1. Indeed, each to v we have the isotropy subgroup Kv, the isotropy subal- functionfv, v E p, defines a multiplication operator. Corre- gebra fv, the nilradical ftv of fv, and also the subset 'iv C p sponding to a vector field r on Y we have the order 1 defined by w E ',v if and only if w E p and ([v, w], v, w) is differential operator X, where T denotes the Lie derivative. an S-triple (i.e., we have (h, v] = 2v and [h, w] = -2w, where Ifx E I, then OK(x) is the symbol of T,,, and also TKW(x) = h = [v, w]). The subalgebra f is abelian and so the symmetric Finally, A is the symbol of E' = 5E. algebra S(O ) identifies with the universal enveloping algebra If L is a K-module then let (End L)K be the algebra of Uffv). Furthermore, [fv, If, v]] = Cv, and hence we have a K-finite endomorphisms of L. bilinear pairingj: S4(fV,) X S4(f) C determined uniquely by The "quantization" of Theorem 2 is j(X4, b4) = 24c4, where x E fv, b E I, and [x, [b, v]] = cv. THEOREM 3. If v E Z(p), then the operators mx(P,) and f, The action of f on p defines a homomorphism T: U(f) on H have the same image and,furthermore, they commute. End p. Consequently the quotient wrk(P,)/f, defines a global alge- PROPOSITION 1. Let v E Z(p). Then there exists a unique braic differential operator on N1/2. The linear span ofthese Kv-invariantpolynomial Pv E S4(f V) such that (i) (RPv)(Cw) = operators wK(Pv)/fv, v E Z(P), is a simple K-submodule of Cvfor all w E %, and (ii) j(P, b4) = 1 ifb E f and (7b4)(v) (End H)K equivalent to p. The operator E' on H is diago- E %V. nalizable with positive spectrum. Let In the proposition, condition i determines Pv up to a nonzero scalar and then ii is a normalization condition. ir: gD--End H [2] There is a natural isomorphism f: p R,(Y), v fv, defined by the inclusion Y C p*. be the linear map such that lr(x) = ITK(X) if x E f and THEOREM 2. The rational functions Oc(Pv)/fv on T*Y, where v E Z(p), are infact regular (i.e., have no poles), and =wK(PEv)1 - T,,, where furthermore these functions span a simple K-submodule of W~)= f, Tv =E(E 1)f [3] R(T*Y) equivalent to p. Let r C R,(M) be as in Theorem 1 and let : g -->t be a Lie algebra isomorphism extending OK. ifv E Z(p). Then r is a Lie algebra homomorphism. Hence Then, afterpossibly modifying 4 by the action ofA', we have ir is a representation of o by global algebraic pseudodffer- ential operators on N"/2. The operators Tv, v E Z(p), (re- +(v) = - gv, where gv = 1 K [1] spectively, fv, v E p) commute and generate a K-stable fv subalgebra of(End H)K isomorphic to R(Y). The proof of Theorem 3 again (cf. ref. 2) relies on an if v E Z(p). The functions fv, v E p, generate the (Poisson application of the generalized Capelli identity established in commutative) K-stable subalgebra R(Y) ofR(T*Y). But also ref. 12. thefunctions gv, v E Z(P), Poisson commute and generate a Let Hs, s E Z, be the s-eigenspace ofE' on H. Concerning K-stable subalgebra ofR(M) isomorphic to R(Y). the of ir we prove Notice that 4K(I) C R(o0,)(M), while fv E R(l,o)(M) and gv THEOREM 4. The representation 2 constructed in Theorem E R(-1,2)(M). The action of d fixes EK(E) and transformsfv 3 is irreducible. The annihilator of the algebra homomor- - gv into cf, - c-lgv, where c E C* s '. The formulas in phism ;i: OIL(g) -- End H corresponding to iris the Joseph 1 give the unique expression for 4+(v) in terms of the (local) ideal J. Furthermore, H admits a unique (up to scaling) coordinate system on M formed by A together with functions &0-invariantpositive definite Hermitian innerproduct B. The on M corresponding to a basis of the Heisenberg Lie algebra (g,K)-module H is then the associated Harish-Chandra mod- given by the nilradical of gv. ule of a unitary minimal representation ir. of Go on a We next "quantize" the functionsfv and gv in 1-i.e., we (complex) Hilbert space We. convert these functions into pseudodifferential operators on The spectrum ofE' on H is equal to {r, r 1.+ ..}, where the space of global sections of a (K-homogeneous) "half- r is thepositive integer or half-integer given in Table I below. form" line bundle N1/2 on Y. The eigenspaces ofE' on H are simple inequivalent K-mod- Downloaded by guest on October 1, 2021 6028 Mathematics: Brylinski and Kostant Proc. Nad. Acad. Sci. USA 91 (1994) Table 1. Cases of minimal representations Case g I V r i E6 !90(8, C) 6 C 5/2 ii E7 K(8, C) 7 C 4 iii 4o(16, C) 8 C 7 iv F4 ip(6, C) + Q[(2, C) 4 C @ SKC2) 2 v E6 ?R(6, C) + ?f(2, C) 4 C S2(C2) 3 vi E7 ?o(12, C) + QR2, C) 4 C S4(C2) 5 vii E8 E7 + BR(2, C) 4 C S(C2) 9 viii G2 QR2, C) + Q(2, C) 2 S2(C2) @ C 1 ix Bo(p + q, C) 0o(p, C) + ?o(q, C) p S(q-P)/2(Cp) S C Y2(q - 2) 3 p I q, 8 < p + q is even x 4o(3 + q, C) Kf(2, C) + 4o(q, C) 3 Sq-3(C2) @ C ½(q - 2) 4 s q, q is even

ules and so in particular H is a multiplicity-free K-module. where a and b are as in Theorem 5. We have Hp+r Vp*+,,(pV 2 0), so that H = Ep2oVp#+,v where A feature of our results is that we can construct the V# and V, is the minimal K-type. representations iT in any model of H so long as we are given As noted above, the K-type decompositions were already both the K-module structure and the R(Y)-module structure known. The operator E' and its minimal eigenvalue r are on H. In particular, the half-forms can be completely sup- completely new, however. pressed in the model. We illustrate this by two examples. In Table 1,1 is the rank ofthe symmetric pair (g, I). In case These cases are particularly simple since (unlike the cases viii, p = S3(C2) 8) C2. We may group cases ix and x together i-iii explained in ref. 2) here the polynomials P, (see Prop- by setting p = 3 in x. osition 1) factor into a product of 4 linear terms. Next, we explicitly describe the Hilbert space inner prod- 1: = = C)4. As uct. First we choose (as we can) nonzero vectors z E and Example Let g& 4o(4, 4). Then f Qf(2, Z(p) K-modules we have H = R(Y) = A model of so E Z(Hr) such that ((z, zf, z, Z) is an S-triple and so is @Ena0*S(C2)04. Kz-invariant. Here z z-P Z denotes the complex conjugation H is given in the following way. Let S be the polynomial ring map on g defined by go. In particular, then so and z are highest in 8 variables xp,, where p E {1, . . . , 4} and i E {1, 2}. Then weight vectors for the K-action with respect to a common H is the subalgebra of S generated by the 16 products choice of positive system. If H is spherical then so is, up to X1,iX2jX3,kX4,l, where i, j, k, I E {1, 2} so that scaling, the constant function 1. We normalize B by requiring that B(so, so) = 1. Since B is H = @ naoCn[xl,, x1,2}-Cn[x2,l, X2,2}Cn¶X3,1, x3,2] K-invariant it follows that Hp and Hq are B-orthogonal ifp # q and, furthermore, that each restriction BIHp is determined {Cn[x4,1, x4,2] C S, up to a scalar. Hence it suffices to know B on the highest where v] is the space of degree n polynomials in u and weight vectors 'r(z))(so) E Hn+r, n - 0. It turns out that these Cn[u, values are given by the coefficients of a Gaussian hypergeo- v. Notice then that H is the space of invariants in S under a metric function. scaling action of C* x C* x C*. Let (3 be the differential Define m to be the eigenvalue of 'n([z, z]) on so. operator on S given by THEOREM 5. We have lr(z)n(so) = fnso where n - 0. The structure on XC is normalized G.-invariant Hilbert space such X - + - + 1. that X1,1 ax11'j X1,2 aX1,2 fn Applying Theorem 3, we find that the following 28 pseudo- BB(so -\xn = 2F((a)b; r + 1; x) = an(b)" xn' nO n! n!/ft n2-on!(r differential operators on S preserve Hand, as operators on H, +1)n they form a basis of a complex Lie algebra g isomorphic to where a and b are given asfollows: (1) in cases i-viii in Table ?o(8, C). 1-i.e., when g is ofexceptional type-we have a = 1 + d/2 and b = 1 + d, where d is defined by 1 + 3d/2 = r + m; and a a a a (2) in cases ix-x, we have a = (q - 2)/2 and b = (q - p + Xpjl '~Xp,2 s~Xpjl Xp,2 2)/2. In every case we have a + b = r + 1 + m. aXp,2 axpl axpl - xp,2- The values of d in (1) turn out to be i, 1; ii, 2; iii, 4; iv, 1; where p E{1, 2, 3, 4}, v, 2; vi, 4; vii, 8; and viii, Y3. Generalizing the notion of spherical function, we have the (_.)i+j+k+l a4 matrix coefficients corresponding to the minimal K-type in H. We cannot as yet explicitly determine these matrix coeffi- XlwiX2jX3,kX4,1x1,x243,k4,l-AP((+p 1))ali8XJ8~k84laX1X.aX2j~'aX3,k'aX4,1' cients. But we can determine their restriction to the root subgroup L, = SL(2, R) of G, corresponding to the span of where {i, i'} = {J, j'} = {k, k'} = {l, l'} = {1, 2}. h = z + z, z, and z. As an example we consider the matrix coefficient co given by co(g) = B(g-so, so), g E Go. (Note that The normalized pre-Hilbert space structure B on H is such co is just the spherical function in case H is spherical.) The that function 4. given by q6(t) = co(exp th'), t E R, determines the value of c0 on L0. B(xpj/n!, xpJ/n!) = 1/(n + 1), THEOREM 6. The function 4,: R -- R has values where p E {1, . , 4}and i E {1, 2} (in agreement with ref. 0(t) = 2F,(a, b; r + 1; - sinh2 t) 6). Example 2: Let g& be of type G2. Let S be the polynomial ring in 4 variables ul, u2, x1, x2 and let S' R(Y) be the Downloaded by guest on October 1, 2021 Mathematics: Brylinski and Kostant Proc. Nati. Acad. Sci. USA 91 (1994) 6029 subalgebra generated by the 8 products uj'xjand u~ui xj, where Full details and proofs of the results announced here and {i, i'} = {1, 2}andj E {1, 2}. A model ofHis the S'-submodule in ref. 2 will appear elsewhere. H = (D2oC3"+[uj, u2WC"[xl, x2] C S. It is a pleasure to thank for helpful discussions concerninghalf-forms and cases vin andx in Table 1. Part ofthis work Let (3 be the differential operator on S given by was carried out while R.B. was visiting Harvard University, and she thanks the Harvard mathematics department for its hospitality. This a a research was supported in part by an Alfred P. Sloan Foundation (3=X1-+X2-+ 1. Fellowship to R.B. and National Science Foundation Grant DMS- ax2 ax2 9307460 to B.K. Theorem 3 implies that the following 14 pseudodifferential 1. Joseph, A. (1976) Ann. Sci. Ecole. Norm. Sup. 9, 1-30. operators on S preserve H and, as operators on H, they form 2. Brylinski, R. & Kostant, B. (1994) Proc. Natd. Acad. Sci. USA a basis ofa Lie algebra isomorphic to the complex simple Lie 91, 2469-2472. algebra of type G2. 3. Kazhdan, D. & Savin, G. (1990) in The Smallest Representa- tions ofSimply-Laced Groups, Israel Mathematics Conference Proceedings, Piatetski-Shapiro Festschrift, eds. Gelbart, S., U2 - Ul - -U2 - Howe, R. & Sarnak, P. (Weizmann Science, Jerusalem) Vol. 2 -1aU2 au, au, au2 pp. 209-223. 4. Gross, B. &Wallach, N. inLie Theory andGeometry:In Honor a a a a oflB. Kostant, eds. Brylinski, J.-L., Brylinski, R., Guillemin, V. X1 -, X2 -, XI - - X2- & Kac, V. (Birkhaeuser, Boston), in press.' aX2 ax, ax, aX2 5. Vogan, D. A. (1994) Invent. Math. 116, 677-791. 6. Kostant, B. (1990) in Operator Algebras, Unitary Representa- a4 tions, Enveloping Algebras, and Invariant Theory, eds. UiX-27x(3(3..1)+i+ 1) audx where {i, i'} = {j, }l= {1, 2} Connes, A., Duflo, M., Joseph, A. & Rentschler, R. (Birkhae- iaXJ user, Boston), pp. 85-124. 7. Binegar, B. & Zierau, R. (1991) Commun. Math. Phys. 138, (-l)'+j a4 245-258. 8. Vogan, D. A. (1981) in Non-commutative Harmonic Analysis '' 'j 278(,B + 1) auiauiaxjs where {i, il} = ti, j'} = {1, 2}. and Lie Groups, Springer Lecture Notes, eds. Carmona, J. & Verge, M. (Springer, Berlin), Vol. 880, pp. 506-535. The normalized pre-Hilbert space structure B on H is such 9. Vogan, D. A. (1991) in Harmonic Analysis on Reductive that Groups, eds. Barker, W. & Sally, P. (Birkhaeuser, Boston), pp. 315-388. (3n + 3)! 10. Brylinski, R. & Kostant, B. (1994) J. Am. Math. Soc. 7, B(IXixi/n!,B(U~n2nI' u'.,n+2xln!)u?" 2x'/n!) - 33n3!n!(n + 1)!(n + 1)! 269-298. 11. Brylinski, R. & Kostant, B. (1992) Bull. Am. Math. Soc. 26, 269-275. where i, j E {1, 2}. 12. Kostant, B. & Sahi, S. (1991) Adv. Math. 87, 71-92. Downloaded by guest on October 1, 2021