Proc. Nadl. Acad. Sci. USA Vol. 91, pp. 6026-6029, June 1994 Mathematics Minimal representations, geometric quantization, and unitarity (Joseph ideal/nilpotent orbit/syMplec manfd/half-form/hypergeometrlc fuctio) RANEE BRYLINSKItI AND BERTRAM KOSTANT§ tPennsylvania State University, University Park, PA 16802; *Harvard University, Cambridge, MA 02138; and §Department of Mathematics, Massachusetts Institute of Technology, Cambridge, MA 02139 Contributed by Bertram Kostant, March 14, 1994 ABSTRACT In the framework of geometric quantization in H from the associated nilpotent orbit in g appeared first in we exlicitly construct, in a uniform fashion, a unitary minimal ref. 8, and there Vogan worked out the existence, unitarity, representation irT of every simply-connected real Lie group G. and K-type decomposition for several cases including some such that the maximal compact subgroup ofG. has finite center of the cases in the table. In. ref. 9 Vogan proved that, as a and G. admits some minimal representation. We obtain alge- virtual K-module, H must equal a multiple of the space of braic and analytic results about frf. We give several results on sections of a K-homogeneous half-form line bundle on Y the algebraic and symplectic geometry ofthe minimal nipotent minus some finite-dimensional K-module. orbits and then "quantize" these results to obtain the corre- Let 0 C o* be the minimal (nonzero) coadjoint orbit so that sponding representations. We assume (Lie Go)C is simple. o corresponds under the Killing form to Ornin. Then 0 is a complex symplectic manifold with respect to the Kirillov- Let Go be a simply-connected simple real Lie group and let Kostant-Souriau symplectic form. We have g* = f* e p* in K0 be a maximal compact group. Let g be the complexifica- the obvious way. A necessary condition for Go to admit a tion of %o = Lie Go with g = f + p a complexified Cartan minimal representation is that the set Y = o n p* is decomposition corresponding to go. Let G be a simply- nonempty-we assume this condition from now on. The set connected complex Lie group with Lie algebra g and let K be Yis then a smooth K-homogeneous Langian submanifold the subgroup corresponding to I. We assume throughout this of 0; in fact, Y is the cone of highest weight vectors in the note that the pair (G0, K0) satisfies the following equivalent simple K-module p*. In particular dim Y = Y2dim 0 and Y = conditions: (i) the associated symmetric space GO/Ko is K*C where E p*. The K-action on Y defines a Hamiltonian non-Hermitian, (ii) K0 has finite center, and (iii) p is irreduc- K-action on the cotangent bundle T* Y (equipped with its ible as a f-module. We will call an infinitesimally irreducible canonical symplectic form) with moment map IK: T* Ye f*. representation ire, of Go on a complex Hilbert space minimal We find that although 0 admits no K-invariant polariza- if its annihilator Ann ir, in the enveloping algebra U(g) is tion, in fact T* Y and a (ramified) 2-fold covering of 0 share equal to the Joseph ideal J (1)-i.e., 7rO, is "attached" to the a common smooth symplectic K-invariant open dense set M. minimal nilpotent orbit and the primitive ideal Ann ir, is Consequently the Hamiltonian G-action on 0 defines an completely prime. We assume throughout g is simple. infinitesimal Hamiltonian g-action on M-i.e., an infinitesi- In this note we extend the methods of ref. 2 to obtain, in mal action by vector fields which admits a moment map pu: auniform way, analogous results on the construction, unitary M-- g*. In this way, the infinitesimal f-action on T* Yis being structure, and harmonic analysis ofa minimal representation enlarged, over M (but not over all of T* Y), to an infinitesimal 'rr ofGo for each case where 'rw exists. In addition, we explain g-action. We may thus regard M as "Hamiltonian (g, K)- some of our results on the symplectic geometry of the M is intrinsic to the of minimal nilpotent orbit Oiam C g and show how these results space." Our construction of geometry underlie our explicit construction of the corresponding min- T* Y-one characterization is that M is equal to the union of imal representation. In our models the Harish-Chandra mod- all K-orbits on T* Y of principal orbit type. ule H for 1r, is the space IMY, N1/2) of algebraic sections of To state our results in more detail we introduce the the (K-homogeneous) half-form line bundle N112 on a (K- following. Let R(X) be the algebra of regular functions on an homogeneous) Lagrangian Y of Ongn. (H is spherical if and algebraic variety X. Then we have an inclusion R(T*Y) C only if N1/2 is K-equivariantly trivial-this occurred for the R(M) ofPoisson algebras where the symplectic form on T* Y three cases treated in ref. 2.) Furthermore, the elements of g defines a symplectic form on M. Each 4 E R(M) defines a act by explicit pseudodifferential operators on r(Y, N1/2). Hamiltonian vector field f# on M. This places our results clearly in the scheme of geometric As Y is K-homogeneous, each x E f defines a vector field quantization. Finally, we give explicit formulas in terms of nx on Y. Let #K(X) E R(T* Y) be the symbol of 'f and let OK hypergeometric functions for the unitary structure of iro and : S(f) -- R(T* Y) be the corresponding algebra homomor- also for a matrix coefficient of 7rT, restricted to an SL(2, R) phism, where S(f) = @,i:oS'(f) is the graded symmetric root subgroup. algebra of f. Then qK is the comorphism to AXK. The pairs (g, f) occurring here fall naturally into 10 cases- Let A E R(T* Y) be the symbol of the Euler vector field E see Table 1 below. The existence of Hilbert spaces carrying on Y. (The Euler vector field on a vector space V, or on any these unitary minimal representations, along with the K-type cone inside V, maps each linear function to itself.) Let a be decomposition of H, had already been established by meth- the unique vector field on T* Ywhich coincides with the Euler ods adapted to the individual cases, with the exception of vector field on every fiber of the cotangent fibration ty: T* Y case x if q 2 6: for cases i-iii see ref. 3; for iv-vii see ref. 4; -+ Y. Then we have a K-invariant algebra bigrading R(M) = for viii see ref. 5; for ix see refs. 6, 3 (ifp = q), and 7; and for EJpqezRpsq(M), where Rp,q(M) is the intersection of the x see ref. 5 if q = 4. The notion of determining the K-types p-eigenspace of {A with the q-eigenspace of a. Then OK(Sq(f)) C R(oq)(M), while Rp(Y) C R(p,o)(M), where Rp(Y) is the The publication costs ofthis article were defrayed in part by page charge p-eigenspace ofE and R(Y) C R(T* Y) is the inclusion defined payment. This article must therefore be hereby marked "advertisement" by ty. The Poisson bracket satisfies {R(p,q)(M), R(p',q')(M)} C in accordance with 18 U.S.C. §1734 solely to indicate this fact. R(p+p',q+q'-1)(M). Consequently, R,(M) = @p+q=1Rp,q(M), 6026 Downloaded by guest on October 1, 2021 Mathematics: Brylinski and Kostant Proc. NatI. Acad. Sci. USA 91 (1994) 6027 the 1-eigenspace of the vector field CA + a, is an (infinite- From now on we exclude two cases: (i) g = ?t(n, C), n - dimensional) Lie subalgebra of R(M) containing MK-. 3, since there the Joseph ideal is not defined, and (ii) g = ?w(p Let A' be the group of symplectic diffeomorphisms of T* Y + q, C) with p + q odd and p, q .4 since there (Howe and) defined by the group sa of K-invariant automorphisms of Y. Vogan (8) has shown that no minimal representation exists Then al' A C*, as .s is the one parameter group giving and, indeed, no K-homogeneous half-form bundle exists. We the flow of E. note that the requirement that 0 meets p* has already THEOREM 1. Let M C T*Y be the complement ofthe divisor excluded the following five cases of non-Hermitian symmet- (A = 0) so that M is a Zariski open dense K-invariant subset. ric pairs (g, f): (KI(2n, C), ?op(2n, C)) where n 2 2, (Bo(p + Then, up to the action of d', there is a unique finite- 1, C), Bo(p, C) + 5o(l, C)) where p 2 3, (4p(2p + 2q, C), dimensional subspace r C R,(M) such that r contains 4'0dt), ?p(2p, C) + 4p(2q, C))wherep, q . 1, (F4, !5p(9, C)), and (E6, r is closed under Poisson bracket, and r is isomorphic as a F4). This list follows from the classification in ref. 10 (see Lie algebra to g. The image ofthe moment map tt: M -g* Table 1 in ref. 11), as we show that 0 fails to meet p* if and corresponding to r lies in 0, andfurthermore, p(M) C 0 is only if K has a Zariski dense orbit on 0. a Zariski open dense K-invariant subset.