Restrictions of Unitary Representations of Real Reductive Groups

Toshiyuki Kobayashi Research Institute for Mathematical Sciences, Kyoto University, Kyoto, 606-8502, Japan [email protected]

1 Reductive Lie groups

Classical groups such as the general linear group GL(n, R) and the Lorentz group O(p, q) are reductive Lie groups. This section tries to give an elementary introduction to the structures of reductive Lie groups based on examples, which will be used throughout this chapter. All the materials of this section and further details may be found in standard textbooks or lecture notes such as [23, 36, 99, 104].

1.1 Smallest objects

The “smallest objects” of representations are irreducible representations. All unitary representations are built up of irreducible unitary representations by means of direct integrals (see §3.1.2). The “smallest objects” for Lie groups are those without nontrivial connected normal subgroups; they consist of simple Lie groups such as SL(n, R), and one- dimensional abelian Lie groups such as R and S1. Reductive Lie groups are locally isomorphic to these Lie groups or their direct products. Loosely, a theorem of Duﬂo [10] asserts that all irreducible unitary representations of a real algebraic group are built from those of reductive Lie groups. Throughout this chapter, our main concern will be with irreducible decompositions of unitary representations of reductive Lie groups. In Section 1 we summarize necessary notation and basic facts on reductive Lie groups in as elementary a way as possible.

Keywords and phrases: unitary representations, branching laws, semisimple Lie group, discrete spectrum, homogeneous space 2000 MSC: primary 22E4; secondary 43A85, 11F67, 53C50, 53D20 140 T. Kobayashi 1.2 Generallinear group GL(N, R)

The general linear group GL(N, R) is a typical example of reductive Lie groups. First, we set up notation for G := GL(N, R). We consider the map

− θ : G → G, g → tg 1.

Clearly, θ is an involutive automorphism in the sense that θ satisﬁes: θ ◦ θ = id, θ is an automorphism of the Lie group G.

The set of the ﬁxed points of θ

θ K := G ={g ∈ G : θg = g} − ={g ∈ GL(N, R) : tg 1 = g} is nothing but the orthogonal group O(N), which is compact because O(N) is a 2 bounded closed set in the Euclidean space in M(N, R) RN in light of the following inclusion:

N N ( ) ⊂{ = ( ) ∈ ( , R) 2 = }. O N g gij M N : gij N i=1 j=1

Furthermore, there is no larger compact subgroupofGL(N, R) which contains O(N).Thus, K = O(N) is a maximal compact subgroup of G = GL(N, R). Conversely, any maximal compact subgroupofG is conjugate to K by an element of G. The involution θ (or its conjugation) is called a Cartaninvolution of GL(N, R).

1.3 Cartan decomposition

The Lie algebras of G and K will be denoted by g and k, respectively. Then, for G = GL(N, R) and K = O(N), we have the following direct sum decomposition:

g = gl(N, R) := the set of N × N real matrices % k = o(N) :={X ∈ gl(N, R) : X =−tX} ⊕ p = Symm(N, R) :={X ∈ gl(N, R) : X = tX}.

The decomposition g = k + p is called the Cartan decomposition of the Lie algebra g corresponding to the Cartan involution θ. This decomposition lifts to the Lie group G in the sense that we have the diffeomorphism Restrictions of Unitary Representations of Real Reductive Groups 141 ∼ p × K → G,(X, k) → eX k. (1.3.1)

The map (1.3.1) is bijective, as (X, k) is recovered from g ∈ GL(N, R) by the formula 1 − X = log(gtg), k = e X g. 2 Here, log is the inverse of the bijection

exp : Symm(N, R) → Symm+(N, R) :={X ∈ Symm(N, R) : X 0}.

It requires a small computation of the Jacobian to see the map (1.3.1) is a Cω- diffeomorphism (see [23, Chapter II, Theorem 1.7]). The decomposition (1.3.1) is known as the polar decomposition of GL(N, R) in linear algebra, and is a special case of a Cartan decomposition of a reductive Lie group in Lie theory (see (1.4.2) below).

1.4 Reductive Lie groups

A connected Lie group G is called reductive if its Lie algebra g is reductive, namely, it is isomorphic to the direct sum of simple Lie algebras and an abelian Lie algebra. In the literature on representation theory, however, different authors have introduced and/or adopted several variations of the category of reductive Lie groups, in particular, with respect to discreteness, linearity and covering. In this chapter we adopt the following definition of “reductive Lie group”. Some advantages here are: • The structural theory of reductive Lie group can be explained in an elementary way, parallel to that of GL(n, R). • It is easy to verify that (typical) classical groups are indeed real reductive Lie groups (see §1.5). Definition 1.4. We say a Lie group G is linearreductive if G can be realized as a closed subgroupofGL(N, R) satisfying the following two conditions: i) θG = G. ii) G has at most finitely many connected components. In this chapter we say G is a reductive Lie group if it is a finite covering of a linear reductive Lie group. Then the condition (i) of Definition 1.4 implies that its Lie algebra g is reductive. If G is linear reductive, then by using its realization in GL(N, R) we define

K := G ∩ O(N).

Then K is a maximal compact subgroupofG. The Lie algebra k of K is given by g ∩ o(N). We deﬁne p := g ∩ Symm(N, R). Similar to the case of GL(N, R), we have the following Cartan decompositions: 142 T. Kobayashi

g = k + p (direct sum decomposition) (1.4.1) ∼ p × K → G (diffeomorphism) (1.4.2) by taking the restriction of the corresponding decompositions for GL(N, R) (see (1.3.1)) to g and G, respectively.

1.5 Examples of reductive Lie groups

Reductive Lie groups in the above definition include all compact Lie groups (especially finite groups) and classical groups such as the general linear group GL(N, R), GL(N, C), the special linear group SL(N, R), SL(N, C), the generalized Lorentz group (the indefinite orthogonal group) O(p, q), the symplectic group Sp(N, R), Sp(N, C), and some others such as U(p, q), Sp(p, q), U ∗(2n), O∗(2n), etc. In this subsection, we review some of these classical groups, and explain how to prove that they are reductive Lie groups in the sense of Definition 1.4. For this purpose, the following theorem is useful. Theorem 1.5.1. Let G be a linear algebraic subgroupofGL(N, R).IfθG = G, thenGis a reductive Lie group. Here we recall that a linear algebraic group (over R)isasubgroup G ⊂ GL(N, R) which is an algebraic subset in M(N, R), that is, the set of zeros of an ideal of polynomial functions with coefficients in R. Sketch of proof. In order to prove that G has at most finitely many connected components, it is enough to verify the following two assertions:

• K = G ∩ O(N) is compact. ∼ • The map p × K → G,(X, k) → eX k is a homeomorphism (Cartan decomposition). The ﬁrst assertion is obvious because G is closed. The second assertion is deduced from the bijection (1.3.1) for GL(N, R). For this, the nontrivial part is a proof of the implication “eX k ∈ G ⇒ X ∈ p”. Let us prove this. We note that if eX k ∈ G, then e2X = (eX k)(θ(eX k))−1 ∈ G, and therefore e2nX ∈ G for all n ∈ Z.ToseeX ∈ p, we want to show e2tX ∈ G for all t ∈ R. This follows from Chevalley’s lemma: let X ∈ Symm(N, R);ifesX satisﬁes a polynomial equation of entries for any s ∈ Z, then so does esX for any s ∈ R. As special cases of Theorem 1.5.1, we pin down Propositions 1.5.2, 1.5.3 and 1.5.6. Proposition 1.5.2. SL(N, R) :={g ∈ GL(N, R) : det g = 1} is a reductive Lie group.

Proof. Obvious from Theorem 1.5.1. Restrictions of Unitary Representations of Real Reductive Groups 143

Proposition 1.5.3. Let A be an N × N matrix such thatA2 = cI (c = 0).Then

G(A) :={g ∈ GL(N, R) : tg Ag = A} is a reductive Lie group.

Proof. First, it follows from the deﬁnition that G(A) is a linear algebraic subgroup of GL(N, R). Second, let us prove θ(G(A)) = G(A).Iftg Ag = A, then g−1 = A−1tg A. Since −1 = 1 A c A,wehave 1 I = gg−1 = g( A)tg(cA−1) = g Atg A−1. c Hence g Atg = A, that is θg ∈ G(A). Then the proposition follows from Theo- rem 1.5.1. I O Example 1.5.4. If A = p (p + q = n), then O −Iq

G(A) = O(p, q)(indeﬁnite orthogonal group).

In this case

K = O(p, q) ∩ O(p + q) O(p) × O(q). OB p ={ : B ∈ M(p, q; R)}. tBO OI Example 1.5.5. If A = n (N = 2n), then −In O

G(A) = Sp(n, R) (real symplectic group).

In this case,

K = Sp(n, R) ∩ O(2n) U(n). AB ∼ k ={ : A =−tA, B = tB} → u(n), −BA AB → A + iB. −BA AB ∼ p ={ : A = tA, B = tB} → Symm(n, C), B −A AB → A + iB. B −A 144 T. Kobayashi

Next let F = R, C or H (quaternionic number ﬁeld). We regard Fn as a right F×-module. Let × M(n, F) := the ring of endomorphisms of Fn, commuting with F -actions ∪ GL(n, F) := the group of all invertibles in M(n, F). Proposition 1.5.6. GL(n, F)(F = R, C, H) is a reductive Lie group.

Sketch of proof. Use C R2 as R-modules and H C2 as right C-modules, and then realize GL(n, C) in GL(2n, R) and GL(n, H) in GL(2n, C). For example, GL(n, H) can be realized in GL(2n, C) as an algebraic subgroup: ∗ U (2n) ={g ∈ GL(2n, C) : g¯ J = Jg}, 0 −I where J = n . It can also be realized in GL(4n, R) as an algebraic subgroup In 0 which is stable under the Cartan involution θ : g → tg−1. Classical reductive Lie groups are obtained by Propositions 1.5.2, 1.5.3 and 1.5.6, and by taking their intersections and their ﬁnite coverings. For examples, in appro- priate realizations, the following intersections U(p, q) = SO(2p, 2q) ∩ GL(p + q, C) Sp(p, q) = SO(4p, 4q) ∩ GL(p + q, H) ∗ ∗ SU (2n) = U (2n) ∩ SL(2n, C) ∗ SO (2n) = GL(n, H) ∩ SO(2n, C) are all reductive (linear) Lie groups.

1.6 Inclusionsofgroups and restrictions of representations As the constructions in the previous section indicate, these classical Lie groups enjoy natural inclusive relations such as

··· ⊂ GL(n, R) ⊂ Sp(n, R) ⊂ Sp(n, C) ⊂ Sp(2n, R) ⊂ GL(4n, R) ⊂ ··· ∪∪∪ ∪ ∪ ··· ⊂ O(p, q) ⊂ U(p, q) ⊂ Sp(p, q) ⊂ U(2p, 2q) ⊂ O(4p, 4q) ⊂ ··· ∩∩∩ ∩ ∩ ··· ⊂ GL(n, R) ⊂ GL(n, C) ⊂ GL(n, H) ⊂ GL(2n, C) ⊂ GL(4n, R) ⊂ ··· ∩∩∩ ∩ ∩ ...... where p + q = n. Our objective in this chapter will be to restrict unitary representations of a group to its subgroups. This may be regarded as a representation-theoretic counterpart of inclusive relations between Lie groups as above. Restrictions of Unitary Representations of Real Reductive Groups 145 2 Unitary representations and admissible representations

To deal with inﬁnite-dimensional representations, we need a good category to work with. This section introduces some standard notation such as continuous representations followed by a more specialized category such as discrete decomposable representations and admissible representations.

2.1 Continuous representations Let H be a topological vector space over C. We shall write: End(H) := the ring of continuous endomorphisms of H, GL(H) := the group of all invertibles in End(H). Deﬁnition 2.1.1. Let G be a Lie group and π : G → GL(H) a group homomorphism. We say (π, H) is a continuousrepresentation if the following map G × H → H,(g,v) → π(g)v (2.1.1) is continuous. If H isaFrechet´ space (a Banach space, a Hilbert space, etc.), the continuity condition in this deﬁnition is equivalent to strong continuity — that g → π(g)v is continuous from G to H for each v ∈ H. A continuous representation (π, H) is irreducible if there is no invariant closed subspace of H but for obvious ones {0} and H.

2.2 Example Example 2.2.1. Let G = R and H = L2(R).Fora ∈ G and f ∈ H, we deﬁne T (a) : L2(R) → L2(R), f (x) → f (x − a). Then it is easy to see that the map (2.1.1) is continuous. The resulting continuous representation (T, L2(R)) is called the regularrepresentation of R. We note that T : G → GL(H) is not continuousifGL(H) is equipped with operator norm %% because √ lim %T (a) − T (a0)%= 2 = 0. (2.2.1) a→a0 This observation explains that the continuity of (2.1.1) is a proper one to deﬁne the notion of continuous representations.

2.3 Unitary representations Definition 2.3.1. A unitary representation is a continuous representation π defined on a Hilbert space H such that π(g) is a unitary operator for all g ∈ G. The set of equivalent classes of irreducible unitary representaitons of G is called the unitary dual of G, and will be denoted by G. 146 T. Kobayashi 2.4 Admissible restrictions 2.4.1 (Unitarily) discretely decomposable representations Let G be a Lie group and let π be a unitary representation of G on a (separable) Hilbert space. Definition 2.4.1. We say the unitary representation π is (unitarily) discretelydecomposable, or simply, discretelydecomposable if π is unitarily equivalent to a discrete sum of irreducible unitary representations of G: ⊕ π|G nπ (σ )σ. (2.4.1) σ ∈ G ⊕ Here, nπ : G →{0, 1, 2,...,∞}, and nπ (σ )σ denotes the Hilbert completion of an algebraic direct sum of irreducible representations

(σ! ⊕ σ ⊕···⊕"# σ$). σ∈G nπ (σ ) If we have the unitary equivalence (2.4.1), then nπ (σ ) is given by

nπ (σ ) = dim HomG (σ, π|G ), the dimension of the space of continuous G-homomorphisms. The point of Deﬁnition 2.4.1 is that there is no continuous spectrum in the decomposition (see Theorem 3.1.2 for a general nature of irreducible decompositions).

2.4.2 Admissible representations Let π be a unitary representation of G. Deﬁnition 2.4.2. We say π is G-admissible if it is (analytically) discretely decomposable and if nπ (σ ) < ∞ for any σ ∈ G . Example. 1) Any ﬁnite-dimensional unitary representation is admissible. 2) The regular representation of a compact group K on L2(K ) is also admissible by the Peter–Weyl theorem. We shall give some more examples where admissible representations arise naturally in various contexts.

2.4.3 Gelfand–Piateski-Shapiro’stheorem Let be a discrete subgroupofaunimodular Lie group (for example, any reductive Lie groupisunimodular). Then we can induce a G-invariant measure on the coset space G/ from the Haar measure on G, and deﬁne a unitary representation of G on the Hilbert space L2(G/), consisting of square integrable functions on G/. Theorem 2.4.3 (Gelfand–Piateski-Shapiro). If G/ is compact, then the representation on L2(G/)is G-admissible. Proof. See [105, Proposition 4.3.1.8] for a proof duetoLanglands. Restrictions of Unitary Representations of Real Reductive Groups 147

2.4.4 Admissible restrictions

Our main objective in this chapter is to study restrictions of a unitary representation π of G with respect to its subgroup G. Deﬁnition 2.4.4. We say the restriction π|G is G -admissible if it is G -admissible (Deﬁnition 2.4.2). Our main concern in later sections will be with the case where G is noncompact. We end up with some basic results on admissible restrictions for later purposes.

2.4.5 Chain rule of admissible restrictions

Theorem 2.4.5. Suppose G ⊃ G1 ⊃ G2 are (reductive) Lie groups, and π is auni- π| tary representation of G.Ifthe restriction G2 is G2-admissible, then the restriction π| G1 is G1-admissible.

Sketch of proof. Use Zorn’s lemma. The proof parallels Langlands’ proof of Gelfand– Piateski-Shapiro. See [41, Theorem 1.2] for details. Here is an immediate consequence of Theorem 2.4.5. Corollary. Let π be aunitary representation of G, andK a compact subgroupof G. Assumethat dim HomK (σ, π|K )<∞ for any σ ∈ K .Then the restriction π|G is G-admissible for any G containing K . This corollary is a key to a criterion of admissible restrictions, which we shall return in later sections (see Theorem 6.3.4).

2.4.6 Harish-Chandra’s admissibility theorem

Here is a special, but very important example of admissible restrictions: Theorem 2.4.6 (Harish-Chandra [21]). Let G be a reductive Lie group withamax- imalcompact subgroupK.Forany π ∈ G,the restriction π|K is K-admissible. Since K is compact, the point of Theorem 2.4.6 is the ﬁniteness of K-multiplicities: dim HomK (σ, π|K )<∞ for any σ ∈ K . (2.4.6)

Remark. In a traditional terminology, a continuous (not necessarily unitary) representation π of G is called admissible if (2.4.6) holds. For advanced readers, we give a ﬂavor of the proof of Harish-Chandra’s admissibility theorem without going into details. Sketch of proof of Theorem 2.4.6. See [104], Theorem 3.4.10; [105], Theorem 4.5.2.11. 148 T. Kobayashi

Step 1. Any principal series representation is K -admissible. This is an easy consequence of Frobenius reciprocity because the K -structure of a principal series is given as the induced representation of K from an irreducible representation of a subgroup of K . Step 2. Let π be any irreducible unitary representation of G. The center Z(g) of the enveloping algebra U(g) acts on π ∞ as scalars. Here, π ∞ denotes the representation on smooth vectors. Step 2 is duetoSegal and Mautner. It may be regardedasa generalization of Schur’s lemma. Step 3. Any π can be realized in a subquotient of some principal series representation (Harish-Chandra, Lepowsky, Rader), or more strongly,inasubrepresentation of some principal series representation (Casselman’s embedding theorem). Together with Step 1, Theorem 2.4.6 follows. Casselman’s proof is to use the theory of the Jacquet module and the n-homologies of representations. There is also a more analytic proof for Casselman’s embedding theorem without using n-homologies: This proof consists of three steps: (i) to compactify G ([79]), (ii) to realize π ∞ in C∞(G) via matrix coefﬁcients, (iii) to take the boundary values into principal series representations (see [80], [104] and references therein).

2.4.7 Further readings

For further details, see Chapter 4 [105] for general facts on continuous representations in Section 2.1; see [35, Chapter 8] and [104, Chapter 3] (and also an exposition [102]) for Harish-Chandra’s admissibility theorem 2.4.6 and the idea of n- homologies. For more information about G-admissible restrictions with noncompact G, see exposition [47, 50].

3 SL(2, R) and Branching Laws

Branching laws of unitary representations are related to many different areas of mathematics —spectral theory of partial differential operators, harmonic analysis, combi- natorics, differential geometry, complex analysis, ···. The aim of this subsection is to give a ﬂavor of various aspects of branching laws throughanumber of examples arising from SL(2, R).

3.1 Branching Problems

3.1.1 Direct integral of Hilbert spaces

We recall brieﬂy how to generalize the concept of the discrete direct sum of Hilbert spaces into the direct integral of Hilbert spaces, and explain the general theory of irreducible decomposition of unitary representations that may contain a continuous spectrum. Since the aim here is just to provide a wider perspective regarding discrete Restrictions of Unitary Representations of Real Reductive Groups 149 decomposable restrictions which will be developed in later chapters, we shall try to minimize the exposition. See [34] for further details on §3.1. Let H be a (separable) Hilbert space, and (, µ) a measure space. We construct a Hilbert space, denoted by ⊕ Hdµ(λ), consisting of those H-valued functions s : → H with the following two properties:

i) For any v ∈ H, (s(λ), v) is measurable with respect to µ. 2 ii) %s(λ)%H is square integrable with respect to µ.

⊕ The inner product on Hdµ(λ) is given by (s, s ) := (s(λ), s (λ))Hdµ(λ). ⊕ 2 Example. If H = C, then Hdµ(λ) L ().

More generally, if a “measurable” family of Hilbert spaces Hλ parameterized by λ ∈ is given, then one can also deﬁne a direct integral of Hilbert spaces ⊕ Hλdµ(λ) in a similar manner. Furthermore, if (πλ, Hλ) is a “measurable” family of unitary representations of a Lie group G for each λ, then the map

(s(λ))λ → (πλ(g)s(λ))λ ⊕ deﬁnes a unitary operator on Hλdµ(λ). The resultingunitary representation is called the direct integralofunitary representaions (πλ, Hλ) and will be denoted by ⊕ ⊕ πλdµ(λ), Hλdµ(λ) .

3.1.2 Irreducible decomposition

The following theorem holds more generally for a group G of type I in the sense of von Neumann algebras. Theorem 3.1.2. Everyunitary representation π of a reductive Lie group G ona (separable) Hilbert space is unitarily equivalenttoa direct integral of irreducible unitary representationsofG:

⊕ π nπ (σ )σ dµ(σ). (3.1.2) G

Here, dµ is a Borel measure on the unitary dual G , nπ : G → N∪{∞} is ameasur- able function, and nπ (σ )σ is amultiple of the irreducible unitary representation σ . 150 T. Kobayashi

3.1.3 Examples Example 3.1.3. 1) (Decomposition of L2(R)). Let G = R. For a parameter ξ ∈ R, we deﬁne a one-dimensional unitary representation of the abelian Lie group R by × ixξ χξ : R → C , x → e . Then we have a bijection R R,χξ ↔ ξ. The regular representation T of R on L2(R) is decomposed into irreducibles of R by the Fourier transform ⊕ T χξ dξ. R 2) (Decomposition of L2(S1)) Let G = S1 R/2πZ.BytheFourier series expansion, we have a discrete sum of Hilbert spaces: ⊕ L2(S1) C einθ . n∈Z This is regarded as the irreducible decomposition of the regular representation of the compact abelian Lie group S1 on L2(S1). Here we have identiﬁed S1 with Z by einθ ↔ n. We note that there is no discrete spectrum in (1), while there is no continuous spectrum in (2). In particular, L2(S1) is S1-admissible, as already mentioned in §2.4.2 in connection with the Peter–Weyl theorem.

3.1.4 Branching problems Let π be an irreducible unitary representation of a group G, and let G be its subgroup. By a branching law, we mean the irreducible decomposition of π when restricted to the subgroup G. Branchingproblems ask to ﬁnd branching laws as explicitly as possible.

3.2 Unitary dualofSL(2, R) Irreducible unitary representations of G := SL(2, R) were classiﬁed by Barg- mann [2] in 1947. In this section, we recall an important family of them.

3.2.1 SL(2, R)-action on P1C

The Riemann sphere P1C = C ∪ {∞} splits into three orbits H+, H− and S1 under az + b the linear fractional transformation of SL(2, R), z → . See Figure 3.2.1 top cz + d of next page. We shall associate a family of irreducible unitary representations of SL(2, R) to these orbits in §3.2.2 and §3.2.3. Restrictions of Unitary Representations of Real Reductive Groups 151

FIGURE 3.2.1.

3.2.2 Unitary principal series representations

First, we consider the closed orbit

S1 R ∪ {∞}. a b For λ ∈ C such that Re λ = 1, and for g−1 = ∈ G, we deﬁne cd + 2 2 −λ ax b πλ(g) : L (R) → L (R), f → (πλ(g) f )(x) :=|cx + d| f . cx + d

Then the following holds:

Proposition 3.2.2. For any λ ∈ C such that Re λ = 1, (πλ, L2(R)) is an irreducible unitary representation of G = SL(2, R). Exercise. Prove Proposition 3.2.2. Hint 1) It is straightforward to see

πλ(g1g2) = πλ(g1)πλ(g2)(g1, g2 ∈ G), %π ( ) % =% % ( ∈ ). λ g f L2(R) f L2(R) g G

2) Let K = SO(2). For irreducibility, it is enough to verify the following two claims: a) Any closed G-invariant subspace W of L2(R) contains a nonzero K - invariant vector. − λ b) Any K -invariant vector in L2(R) is a scalar multiple of |1 + x2| 2 .

3.2.3 Holomorphic discrete series representations

Next we construct a family of representations attached to the open orbit H+ (see Figure 3.2.1). Let O(H+) be the space of holomorphic functions on the upper half plane H+. For an integer n ≥ 2, we deﬁne + = O(H ) ∩ 2(H , n−2 ). Vn : + L + y dx dy 152 T. Kobayashi

+ Then it turns out that Vn is a nonzero closed subspace of the Hilbert space − L2(H+, yn 2 dx dy), from which the Hilbert structure is induced. a b For g−1 = ∈ G, we deﬁne cd + + + − az + b π (g) : V → V , f (z) → (cz + d) n f . n n n cz + d

Then the following proposition holds: (π +, +)( ≥ ) Proposition 3.2.3 (Holomorphic discrete series). Each n Vn n 2 is an irreducible unitary representation of G. Let us discuss this example in the following exercises: π +( )( ∈ ) + Exercise. Verify that n g g G is a unitary operator on Vn by a direct computation. We shall return to the irreducibility of (π +, V +) in §3.3.4. n n cos θ − sin θ − Exercise. Let kθ := , and f (z) := (z + i) n. Prove the following sin θ cos θ n formula: π +( ) = inθ . n kθ fn e fn π + χ ,χ , As we shall see in Proposition 3.3.3 (1), the K -types occurring in n are n n+2 χ ,... ∈ + n+4 . Hence, the vector fn Vn is called a minimal K -type vector of the (π +, +) representation n Vn . In §7.1, we shall construct a family of unitary representations attached to elliptic (π +, +) coadjoint orbits. The above construction of n Vn is a simplest example in which the Dolbeault cohomology groupturns up in the degree 0 because the elliptic coadjoint orbit is biholomorphic to a Stein manifold H+ in this case (see Theorem 7.1.4 for a general feature). (π +, +) Similar to n Vn , we can construct another family of irreducible unitary rep- π − = , , ,... resentations n (n 2 3 4 )ofG on the Hilbert space − = O(H ) ∩ 2(H , | |n−2 ). Vn : − L − y dx dy (π −, −) The representation n Vn is called the anti-holomorphic discrete series representation.

3.2.4 Restrictionand proof for irreducibility

Given a representation π of G, how can one find finer properties of π such as irreducibility, Jordan–Holder¨ series, etc.? A naive (and sometimes powerful) approach is to take the restriction of π toasuitable subgroup H. For instance, the method of (g, K )-modules (see Section 4) is based on “discretely decomposable” restrictions to Restrictions of Unitary Representations of Real Reductive Groups 153 a maximal compact subgroup K .Together with “transition coefficients” that describe the actions of p on g-modules, the method of (g, K )-modules provides an elementary π π ± and alternative proof of irreducibility of both λ and n , simultaneously (e.g. [95, Chapter 2]; see also Howe and Tan [26] for some generalization to the Lorentz group O(p, q)). On the other hand, the restriction to noncompact subgroups is sometimes effec- tive in studying representations of G. We shall return to this point in Section 8. For a better understanding, let us observe a number of branching laws of unitary representations with respect to both compact and noncompact subgroups in the case SL(2, R).

3.3 Branching lawsofSL(2, R)

3.3.1 Subgroups of SL(2, R)

Let us consider three subgroups of G = SL(2, R): ⎧ ⎪ ={ θ ∈ R/ πZ} 1, ⎪K : kθ : 2 S ⎪ ) ⎪ ⎨⎪ 1 b N := : b ∈ R R, ⊃ = G H : ⎪ 01 ⎪ ) ⎪ s ⎪ e 0 ⎩⎪A := : s ∈ R R. 0 e−s

For ξ ∈ R, we deﬁne a one-dimensional unitary representation of R by

× ixξ χξ : R → C , x → e .

1 For n ∈ Z, χn is well deﬁned as a unitary representation of S R/2πZ.

3.3.2 Branching laws G ↓ K, A, N

Proposition 3.3.2. Fix λ ∈ C such that Re λ = 1.Then thebranching lawsofa principal series representation πλ of G = SL(2, R) to thesubgroups K, N andA are given by the following: * * ⊕ πλ χ 1) K 2n. n∈Z * ⊕ * 2) πλ χξ dξ. N R * ⊕ * 3) πλ 2χξ dξ. A R Here we have used the identiﬁcations K Z, and N A R. 154 T. Kobayashi

Sketch of proof. In all three cases, the proof reduces to (Euclidean) harmonic analysis. Here are some more details. 1) We deﬁne a unitary map (up to scalar) Tλ by * ψ *−λ + ψ , 2 2 1 * * Tλ : L (R) → L (S ), f → *cos * f tan . (3.3.2) 2 2

Then Tλ respects the K -actions as follows:

Tλ(πλ(kϕ) f )(θ) = (Tλ f )(θ + 2ϕ).

Thus, the branching law (1) follows from the (discrete) decomposition of L2(S1) by the Fourier series expansion:

⊕ L2(S1) C einθ , n∈Z as given in Example 3.1.3 (2). * 1 b * 2) Since (πλ f )(x) = f (x − b), the restriction πλ is nothing but the 01 N regular representation of R on L2(R). Hence the branching law (2) is given by the Fourier transform as in Example 3.1.3. 3) We claim that the multiplicity in the continuous spectrumisuniformly two. To see this, we consider the decomposition

2 2 2 2 2 L (R) L (R+) ⊕ L (R−) −−−−→ L (R) ⊕ L (R) T++T− where T+ (likewise, T−) is deﬁned by

λ 2 2 t t T+ : L (R+) → L (R), f (x) → e 2 f (e ). * * (πλ , 2(R+)) Then the unitary representation A L of A is unitarily equivalent to the regular representation of A R on L2(R) because es 0 T+(πλ f )(t) = (T+ f )(t − 2s). 0 e−s * * πλ 2(R) Hence, the branching law A follows from the Plancherel formula for L as was given in Example 3.1.3 (1).

3.3.3 Restriction of holomorphic discrete series

Without a proof, we present branching laws (abstract Plancherel formulas) of holo- π + = , , ,... = ( , R) morphic discrete series representations n (n 2 3 4 )ofG SL 2 with respect to its subgroups K, N and A: Restrictions of Unitary Representations of Real Reductive Groups 155

Proposition 3.3.3. Fix n = 2, 3, 4,... . * ∞ +* ⊕ π χ + 1) n K n 2k. k=0 * ⊕ +* π χξ ξ 2) n N d . R+ * ⊕ π +* χ ξ 3) n ξ d . A R It is remarkable that the multiplicity is free in all of the above three cases in Proposition 3.3.3 (compare with the multiplicity 2 results in Proposition 3.3.2 (3)). This multiplicity-free result holds in more general branching laws of unitary highest weight representations (see [42]).

π + ( = , , ,...) 3.3.4 Irreducibility of n n 2 3 4 π + ( , ) One of the traditional approaches to show the irreducibility of n is to use g K - modules together with explicit transition coefﬁcients of Lie algebra actions; (e.g., [2], [26], Chapter 2 [95]). Aside from this, we shall explain another approach based on the restriction to noncompact subgroups N and A. This is a small example that restrictions to noncompact subgroups are also usefulinstudying representations of the whole group. (See Subsection 8.1 for this line of research.) + Suppose that W is a G-invariant closed subspace in Vn . In view of the branching law of the restriction to the subgroup N (Proposition 3.3.3 (2)), the representation (π +, ) n W of N is unitarily equivalent to the direct integral ⊕ χξ dξ E for some measurable set E in R+,asaunitary representation of N R.Furthermore, since W is invariant also by the subgroup A, E must be stable under the dilation, namely, E is either the ∅ or R+ (up to a measure zero set). Hence, the invariant { } + (π +, +) subspace W is either 0 or Vn . This shows that n Vn is already irreducible as a representation of the subgroup AN.

3.4 ⊗-product representationsofSL(2, R) 3.4.1 Tensor product representations

Tensor product representations are a special case of restrictions and their decompositions are a special case of branching laws. In fact, suppose π and π are unitary representations of G. Then the outer tensor product π π is a unitary representation of the direct product group G × G. Its restriction to the diagonally embedded subgroup G: 156 T. Kobayashi

G → G × G, g → (g, g) gives rise to the tensor product representation π ⊗ π of the group G. Let us consider the irreducible decomposition of the tensor product representations, as a special case of branching laws.

3.4.2 πλ ⊗ πλ (principal series)

Proposition 3.4.2. Let Re λ = Re λ = 1.Then,thetensor product representation of two(unitary) principal series representations πλ and πλ decomposes into irreducibles of G = SL(2, R) as follows:

⊕ ∞ ⊕ + − πλ ⊗ πλ 2πν dν + (π + π ). Re ν=1 2n 2n Im ν≥0 n=1 A distinguishing feature here is that both continuous and discrete spectrum occur. Discrete spectrum occurs with multiplicity free, while continuous spectrum with multiplicity two. Sketch of proof. Unlike the branching laws in §3.3, we shall use noncommutative harmonic analysis. That is, the decomposition of the tensor product representation reduces to the Plancherel formula for the hyperboloid. To be more precise, we divide the proof into four steps:

Step 1. The outer tensor product πλ πλ is realized on

L2(R)⊗L2(R) L2(R2), or equivalently, on L2(T2) via the intertwining operator Tλ ⊗ Tλ (see (3.3.2)). Step 2. Consider the hyperboloid of one sheet:

X :={(x, y, z) ∈ R3 : x2 + y2 − z2 = 1}.

We identify X with the set of matrices: zx+ y {B = : Trace B = 0, det B = 1}. x − y −z

Then, G = SL(2, R) acts on X by

X → X, B → gBg−1.

Step 3. Embed the G-space X into an open dense subset of the (G ×G)-space T2 so that it is equivariant with respect to the diagonal homomorphism G → G × G. (This is a conformal embedding with respect to natural pseudo-Riemannian metrics.) Restrictions of Unitary Representations of Real Reductive Groups 157

Step 4. The pullback ι∗ followed by a certain twisting (depending on λ and λ) sends L2(T2) onto L2(X). The Laplace–Beltrami operator on X (a wave operator) commutes with the action of G (not by G × G), and its spectral decomposition gives rise to the Plancherel formula for L2(X),orequivalently the branching law πλ ⊗πλ . For advanced readers who are already familiar with the basic theory of representations of semisimple Lie groups, it would be easier to understand some of the above steps by the (abstract) Mackey theory. For example, the above embedding ι : X → T2 may be written as

X G/MA G/(P ∩ P)→ (G × G)/(P × P), where P and P are opposite parabolic subgroups of G such that P ∩ P = MA := a 0 : a ∈ R× . 0 a−1 A remaining part is to prove that the irreducible decomposition is essentially independent of λ and λ (see [45]). See, for example, [14, 81] for the Plancherel formula for the hyperboloid O(p, q)/O(p − 1, q) (the above case is essentially the same when (p, q) = (2, 1)), and chapters of van den Ban, Delorme and Schlichtkrull in [3] for the more general case (reductive symmetric spaces).

π + ⊗ π + 3.4.3 m n (holomorphic discrete series) Proposition 3.4.3. Let m, n ≥ 2.Then

∞ ⊕ π + ⊗ π + π + m n m+n+2 j j=0

A distinguishing feature here is that there is no continuous spectrum (i.e. “discretely decomposable restriction”), even though it is a branching law with respect to a noncompact subgroup (i.e., diagonally embedded G). π + ⊗ π + Sketch of proof. Realize m n as holomorphic functions of two variables on H+ × H+. Take their restrictions to the diagonally embedded submanifold 158 T. Kobayashi

ι : H+ → H+ × H+. π + Then the “bottom” representation m+n (see irreducible summands in Proposition π + ( = , ,...) 3.4.3) arises. Other representations m+n+2 j j 1 2 are obtained by taking normal derivatives with respect to the embedding ι.

4 (g, K )-modules and inﬁnitesimal discrete decomposition

4.1 Category of (g, K )-modules

Section 4 starts with a brief summary of basic results on (g, K )-modules. Advanced readers can skip Subsection 4.1, and go directly to Subsection 4.2 where the concept of inﬁnitesimal discrete decomposition is introduced. This concept is an algebraic analog of the property “having no continuous spectrum”, and is powerful in the algebraic study of admissible restrictions of unitary representations.

4.1.1 K -ﬁnite vectors

Let G be a reductive Lie group with a maximal compact subgroup K . Suppose (π, H) isa(K -)admissible representation of G onaFrechet´ space. (See §2.4.6 and the remark there for the deﬁnition.) We deﬁne a subset of H by

HK :={v ∈ H : dimC"K · v# < ∞}.

Here "K ·v# denotes the complex vector space spanned by {π(k)v : k ∈ K }. Elements in HK are called K -ﬁnite vectors. Then it turns out that HK is a dense subspace of H, and decomposes into an algebraic direct sum of irreducible K -modules:

HK HomK (σ, π|K ) ⊗ Vσ (algebraic direct sum). (σ,Vσ )∈K

4.1.2 Underlying (g, K )-modules

Retain the setting as before. Suppose (π, H) is a continuous representation of G on aFrechet´ space H.Ifπ|K is K -admissible, then the limit

π(etX)v − v dπ(X)v := lim t→0 t exists for v ∈ HK and X ∈ g, and dπ(X)v is again an element of HK .Thus, g and K act on HK .

Deﬁnition 4.1.2. With the action of g and K , HK is called the underlying (g, K )- module of (π, H). Restrictions of Unitary Representations of Real Reductive Groups 159

To axiomize “abstract (g, K )-modules”, we pin down the following three properties of HK (here, we omit writing π or dπ): − k · X · k 1 · v = Ad(k)X · v(X ∈ g, k ∈ K ), (4.1.2)(a) dimC"K · v# < ∞, (4.1.2)(b) * d * etX · v − v Xv = * (X ∈ k). (4.1.2)(c) dt t=0 t (Of course, (4.1.2)(c) holds for X ∈ g in the above setting.)

4.1.3 (g, K )-modules

Now we forget (continuous) representations of a group G and consider only the action of g and K . Deﬁnition (Lepowsky). We say W is a (g, K )-module if W is a g-module and at the same time a K -module satisfying the axioms (4.1.2)(a), (b) and (c). The point here is that no topology is speciﬁed. Nevertheless, many of the fundamental properties of a continuous representation are preserved when passing to the underlying (g, K )-module. For example, irreducibility (or more generally, Jordan– Holder¨ series) of a continuous representation is reduced to that of (g, K )-modules, as the following theorem indicates: Theorem 4.1.3 (Harish-Chandra [21]). Let (π, H) be a (K-)admissible continuous representation of G onaFrec´ het space H.Then there is a lattice isomorphism between { closed G-invariantsubspaces of H} and {g-invariantsubspaces of HK }. The correspondence is given by

V −→ VK := V ∩ HK ,

VK −→ VK (closure in H).

In particular, (π, H) is irreducible if andonly if its underlying (g, K )-module is irreducible.

4.1.4 Inﬁnitesimallyunitary representations

Unitary representations form an important class of continuous representations. Uni- tarity can be also studied algebraically by using (g, K )-modules. Let us recall some known basic results in this direction. A unitary representation of G gives rise to a representation of g by essentially skew-adjoint operators (Segal and Mautner; see Vergne [92] for references). Con- versely, let us start with a representation of g having this property: 160 T. Kobayashi

Definition. A (g, K )-module W is infinitesimallyunitary if it admits a positive definite invariant Hermitian form. Here by “invariant” we mean the following two conditions: for any u,v ∈ W,

(Xu,v)+ (u, Xv) = 0 (X ∈ g), (k · u, k · v) = (u,v) (k ∈ K ).

The point of the following theorem is that analytic objects (unitary representations of G) can be studied by algebraic objects (their underlying (g, K )-modules). Theorem 4.1.4. 1) Any irreducible inﬁnitesimallyunitary (g, K )-module is the underlying (g, K )-modules of some irreducible unitary representation of G onaHilbert space. 2) Two irreducible unitary representationsofG on Hilbert spaces are unitarily equivalentifandonly if their underlying (g, K )-modules are isomorphic as (g, K )- modules.

4.1.5 Scope

Our interest throughout this chapter focuses on the restriction of unitary representations. The above mentioned theorems suggest that we deal with restrictions of unitary representations using the algebraic methods of (g, K )-modules. We shall see that this idea works quite successfully for admissible restrictions. Along this line, we shall introduce the concept of inﬁnitesimally discretely decomposable restrictions in §4.2.

4.1.6 For further reading

See [95, Chapter 1], [104, Chapter 3], and [105, Chapter 4] for §4.1.

4.2 Inﬁnitesimal discrete decomposition

The aim of Section 4 is to establish an algebraic formulation of the condition “having no continuous spectrum”. We are ready to explain this notion by means of (g, K )- modules with some background of motivations.

4.2.1 Wiener subspace

We begin with an observation in the opposite extremal case — “having no discrete spectrum”. Observation 4.2.1. There is no discrete spectruminthePlancherel formula for L2(R).Equivalently, there is no nonzero closed R-irreducible subspace in L2(R). Restrictions of Unitary Representations of Real Reductive Groups 161

An easy proof for this is given simply by observing ξ eix ∈/ L2(R) for any ξ ∈ R. A less easy proof is to deduce from the following claim: Claim. For any nonzero closed R-invariant subspace W in L2(R), there exists an inﬁnite decreasing sequence {W j } of closed R-invariant subspaces:

W W1 W2 ··· . Exercise. Prove that there is no discrete spectruminthePlancherel formula for L2(R) by using the above claim. Sketch of the proof of Claim. Let W be a closed R-invariant subspace in L2(R) (a Wiener subspace). Then one can ﬁnd a measurable subset E of R such that W = F(L2(E)), that is, the imageoftheFourier transform F of square integrable functions supported on E. Then take a decreasing sequence of measurable sets

E E1 E2 ··· 2 and put W j := F(L (E j )). This is what we wanted.

4.2.2 Discretely decomposable modules

Let us consider an opposite extremal case, namely, the property “having no continuous spectrum”. We shall introduce a notion of this nature for representations of Lie algebras as follows. Deﬁnition 4.2.2. Let g be a Lie algebra and let X be a g-module. We say X is discretelydecomposable as a g -module if there is an increasing sequence {X j } of g-modules satisfying both (1) and (2): ∞ (1) X = X j . j=0 (2) X j is of ﬁnite length as a g -module for any j.

Here we note that X j is not necessarily ﬁnite dimensional.

4.2.3 Inﬁnitesimally discretely decomposable representations

Let usturn to representations of Lie groups. Suppose G ⊃ G is a pair of reductive Lie groups with maximal compact subgroups K ⊃ K , respectively. Let (π, H) bea(K -)admissible representation of G.

Definition 4.2.3. The restriction π|G is infinitesimallydiscretelydecomposable if the underlying (g, K )-module (πK , HK ) is discretely decomposable as a g -module (Definition 4.2.2). 162 T. Kobayashi

4.2.4 Examples

1) It is always the case√ if G is compact, especially if G ={e}. 2) Let πλ (λ ∈ 1 + −1R) be a principal* series representation of G = SL(2, R) * (see §3.3.2). Then, the restriction* πλ is inﬁnitesimally discretely decompos- * K πλ able, while the restriction A is not inﬁnitesimally discretely decomposable. See also explicit branching laws given in Proposition 3.3.2.

4.2.5 Unitary case

So far, we have not assumed the unitarity of π. The terminology “discretely decomposable” ﬁts well if π is unitary, as seen in the following theorem. Theorem 4.2.5. Let G ⊃ G be a pair of reductive Lie groups withmaximalcompact subgroups K ⊃ K , respectively. Suppose (π, H) ∈ G.Then the following three conditionson the triple (G, G,π)are equivalent:

i) The restriction π|G is inﬁnitesimally discretely decomposable. ii) The underlying (g, K )-module (πK , HK ) decomposes into an algebraic direct sum of irreducible (g, K )-modules:

πK nπ (Y )Y(algebraic direct sum), Y

where Y runsover all irreducible (g, K )-modules and we have deﬁned

nπ (Y ) := dim Homg,K (Y, HK ) ∈ N ∪ {∞}. (4.2.5) iii) There exists an irreducible (g , K )-module Y such that nπ (Y ) = 0.

Sketch of proof. (i) ⇒ (iii) : Obvious. (i) ⇒ (ii) : Use the assumption that π is unitary. (iii) ⇒ (i) : Use the assumption that π is irreducible. (ii) ⇒ (i): If X is decomposed into the algebraic direct sum of irreducible -∞ -j (g , K )-modules, say Yi , then we put X j := Yi ( j = 0, 1, 2,...). i=0 i=0 We end this section with two important theorems without proof (their proof is not very difﬁcult). Restrictions of Unitary Representations of Real Reductive Groups 163

4.2.6 Inﬁnitesimal discrete decomposability ⇒ discrete decomposability

For π ∈ G and σ ∈ G, we deﬁne

mπ (σ ) := dim HomG (σ, π|G ), the dimension of continuous G-intertwining operators. Then, in general, the following inequality holds:

nπ (σK ) ≤ mπ (σ ).

This inequality becomes an equality if the restriction π|G is infinitesimally discretely decomposable. More precisely, we have: Theorem 4.2.6. In the setting of Theorem 4.2.5, if oneof(therefore, any of) the three equivalentconditionsissatisfied, then the restriction π|G is discretely decomposable (Definition 2.4.1), that is, we have an equivalence of unitary representations of G:

⊕ π|G mπ (σ )σ (discrete Hilbert sum). σ∈ G

Furthermore, the above multiplicitymπ (σ ) coincides with the algebraic multiplicity nπ (σK ) given in (4.2.5). It remains an open problem (see [48, Conjecture D]) whether discrete decomposability implies inﬁnitesimal discrete decomposability.

4.2.7 K -admissibility ⇒ inﬁnitesimally discrete decomposability

Theorem 4.2.7. Retain the setting of Theorem 4.2.5. If π is K -admissible, then the restriction π|G is inﬁnitesimally discretely decomposable.

4.2.8 For further reading

Materials of §4.2 are taken from the author’s papers [44] and [48]. See also [47] and [49] for related topics.

5Algebraic theory of discretely decomposable restrictions

The goal of this section is to introduce associated varieties to the study of inﬁnitesi- mally discretely decomposable restrictions. 164 T. Kobayashi 5.1 Associated varieties

Associated varieties give a coarse approximation of modules of Lie algebras. This subsection summarizes quickly some known results on associated varieties (see Vo- gan’s treatise [98] for more details).

5.1.1 Graded modules

Let V be a ﬁnite-dimensional vector space over C.Weuse the following notation:

V ∗ : the dual vector space of V. -∞ S(V ) = Sk(V ) : the symmetric algebra of V. k=0 -k j Sk(V ) := S (V ). j=0 -∞ Let M = Mk be a ﬁnitely generated S(V )-module. We say M is a graded S(V )- k=0 module if

i S (V )M j ⊂ Mi+ j for any i, j.

The annihilator AnnS(V )(M) is an ideal of S(V ) deﬁned by

AnnS(V )(M) :={f ∈ S(V ) : f · u = 0 for any u ∈ M}.

Then, AnnS(V )(M) is a homogeneous ideal, namely, ∞ k AnnS(V )(M) = (AnnS(V )(M) ∩ S (V )), k=0 and thus, ( ) ={λ ∈ ∗ (λ) = ∈ ( )} SuppS(V ) M : V : f 0 for any f AnnS(V ) M is a closed cone in V ∗. Hence, we have deﬁned a functor

{graded S(V )-modules} ; {closed cones in V ∗} → ( ). M SuppS(V ) M

5.1.2 Associated varieties of g-modules

Let gC be a Lie algebra over C. For each integer n ≥ 0, we deﬁne a subspace Un(gC) of the enveloping algebra U(gC) of the Lie algebra gC by

Un(gC) := C-span{Y1 ···Yk ∈ U(gC) : Y1, ··· , Yk ∈ gC, k ≤ n}. Restrictions of Unitary Representations of Real Reductive Groups 165

It follows from the deﬁnition that ∞ U(gC) = Un(gC), n=0 C = U0(gC) ⊂ U1(gC) ⊂ U2(gC) ⊂··· .

Then the graded ring ∞ gr U(gC) := Uk(gC)/Uk−1(gC) k=0 is isomorphic to the symmetric algebra ∞ k S(gC) = S (gC) k=0 by the Poincare–Birkhoff–Witt´ theorem. The point here is that the Lie algebra structure on gC is forgotten in the graded ringgr U(gC) S(gC). Suppose X is a finitely generated gC-module. We fix its generators v1,... ,vm ∈ X, and define a filtration {X j } of X by

m X j := U j (gC)vi . i=1 -∞ Then the graded module gr X := X j / X j−1 becomes naturally a finitely gen- j=0 erated graded module of gr U(gC) S(gC).Thus, as in §5.1.1, we can define its support by V ( ) = ( ). gC X : SuppS(gC ) gr X V ( ) It is known that the variety gC X is independent of the choice of generators v1,... ,vm of X. V ( ) Definition 5.1.2. We say gC X is the associatedvariety of a gC-module X. Its complex dimension is called the Gelfand–Kirillovdimension, denote by Dim(X). V ( ) ∗ By definition, the associated variety gC X is a closed cone in gC.Forareduc- ∗ V ( ) tive Lie algebra, we shall identify gC with gC, and thusregard gC X asasubset of gC.

5.1.3 Associated varieties of G-representations

So far we have considered a representation of a Lie algebra. Now we consider the case where X comes from a continuous representation of a reductive Lie group G as the underlying (g, K )-module. The scheme is as follows: 166 T. Kobayashi

(π, H) : an admissible representation of G of ﬁnite length, ⇓ (πK , HK ) : its underlying (g, K )-module, ⇓ HK :regarded as a U(gC)-module, ⇓ gr HK : its graded module (an S(gC)-module), ⇓ V (H ) ∗ gC K : the associated variety (a subset of gC).

5.1.4 Nilpotentcone

Let g be the Lie algebra of a reductive Lie group G. We define the nilpotent cone by N gC :={H ∈ gC :ad(H) is a nilpotent endomorphism}. N ( ) Then gC is an Ad GC -invariant closed cone in gC . Let gC = kC + pC be the complexification of a Cartan decomposition g = k + p. We take a connected complex Lie group KC with Lie algebra kC. (π, H) ∈ V Theorem 5.1.4 ([98]). If G,then its associated variety gC (HK ) is a KC- invariant closed subset of N pC := NgC ∩ pC. Theorem 5.1.4 holds in a more general setting where π isa(K -)admissible (nonunitary) representation of G of finite length. Sketch of proof. Here are key ingredients: ( ) ( ) H ⇒ V ( ) ⊂ N . 1) The center Z gC of U gC acts on K as scalars gC X gC H ⇒ V ( ) 2) K acts on K gC X is KC-invariant. H ⇒ V ( ) ⊂ . 3) K is locally K -finite gC X pC

In [60], Kostant and Rallis studied the KC-action on the nilpotent cone NpC and N proved that the number of KC-orbits on pC is ﬁnite ([60, Theorem 2]; see also [8, 89]). Therefore, there are only ﬁnitely many possibilities of associated varieties V gC (HK ) for admissible representations (π, H) of G.Asanillustrative example, we shall give an explicit combinatorial description of all KC-orbits on NpC in §5.2 for the G = U(2, 2) case.

5.2 Restrictions and associated varieties

This subsection presents the behavior of associated varieties with respect to inﬁnites- imally discretely decomposable restrictions. Restrictions of Unitary Representations of Real Reductive Groups 167

5.2.1 Associated varieties of irreducible summands

Let g be a reductive Lie algebra and g its subalgebra which is reductive in g. This is the case if g ⊂ g ⊂ gl(n, R) are both stable under the Cartan involution X →−tX. We write ∗ → ( )∗ prg→g : gC gC for the natural projection dual to gC → gC.Suppose X is an irreducible g-module, and Y is an irreducible g-module. Then, we can deﬁne their associated varieties in ∗ ∗ gC and (gC) , respectively. Let us compare them in the following diagram: V ( ) ⊂ ∗ gC X ⏐gC ⏐ prg→g ∗ V (Y ) ⊂ ( ) gC gC

Theorem 5.2.1. If Homg (Y, X) ={0},then

(V (X)) ⊂ V (Y ). prg→g gC gC (5.2.1)

Sketch of proof. V (X) V (Y ) In order to compare two associated varieties gC and gC ,we shall take a double filtration as follows: First take an ad(g)-invariant complementary subspace q of g in g: g = g ⊕ q. Choose a finite-dimensional vector space F ⊂ Y , and we define a subspace of X by ⎧ ⎫ ⎨ A1, ··· , A p ∈ g (p ≤ i), ⎬ = C ··· ··· v , ··· , ∈ ( ≤ ), . Xij : -span ⎩A1 A p B1 Bq : B1 Bq g q j ⎭ v ∈ F

Then ) - Xi, j is a filtration of X, + ≤ 1 i j 2N N , , X0 j j is a filtration of Y V ( ) through which we can define and compare the associated varieties gC X and V (Y ) gC . This gives rise to (5.2.1). Remark 5.1. The above approach based on double filtration is taken from [44, Theo- rem 3.1]; Jantzen [30, page 119] gave an alternative proof to Theorem 5.2.1. 168 T. Kobayashi

5.2.2 G: compact case

Let us apply Theorem 5.2.1 to a very special case, namely, G = K = K (a maximal compact subgroup). Obviously, there exists a (ﬁnite-dimensional) irreducible representation Y of K ( , ) ={ } V ( ) ={ } such that Homk Y X 0 . Since Y is ﬁnite dimensional, therefore kC Y 0 . Then Theorem 5.2.1 means that (V ( )) ={ }, prg→k gC X 0 or equivalently, V ( ) ⊂ gC X pC (5.2.2) ∗ if we identify gC with gC. This inclusion (5.2.2) is nothing but the one given in Theorem 5.1.4. Therefore, Theorem 5.2.1 may be regarded as a generalization of Theorem 5.1.4.

5.2.3 Criterion for inﬁnitesimally discretely decomposable restrictions

For a noncompact G, Theorem 5.2.1 leads ustoauseful criterion for inﬁnitesimal discrete decomposability by means of associated varieties. Corollary 5.2.3. ([44, Corollary 3.4]). Let (π, H) ∈ G, and G ⊃ G be a pair of reductive Lie groups. If the restriction π|G is inﬁnitesimally discretely decomposable, then (V (H )) ⊂ N . prg→g gC K gC Here N is the nilpotentconeof . gC gC Proof. The corollary readily follows from Theorem 5.2.1 and Theorem 5.1.4.

The converse statement also holds if (G, G) is a reductive symmetric pair. That (V (H )) N is, if prg→g gC K is contained in the nilpotent cone gC , then the restriction π|G is inﬁnitesimally discretely decomposable. This result was formulated and proved in [44] in the case where πK is of the form Aq(λ) (e.g. discrete series representations). A general case was conjectured in [48, Conjecture B], and Huang and Vogan have announced an afﬁrmative solution to it. See Remark 8.1 after Theorem 8.2.2 for further illuminating examples of associated varieties regarding the theta correspondence for reductive dual pairs.

5.3 Examples

In this subsection, we examine Corollary 5.2.3 for the pair of reductive Lie groups (G, Gi )(i = 1, 2, 3): Restrictions of Unitary Representations of Real Reductive Groups 169

The goal of this subsection is to classify all KC-orbits in the nilpotent variety N pC corresponding to the conditions in Corollary 5.2.3 for inﬁnitesimally discretely decomposable restrictions with respect to the subgroups Gi (i = 1, 2, 3). (The case i = 3 is trivial.)

5.3.1 Strategy We divide into two steps: N § Step 1. Classify KC-orbits on pC ( 5.3.2). C O N (O) ⊂ N Step 2. List all K -orbits on pC such that prg→gi piC (§5.3.4, §5.3.5, §5.3.6).

5.3.2 KC-orbits on NpC = ( , ) N For G U 2 2 , let us write down explicitly all KC-orbits on gC by using elementary linear algebra. Relation to representation theory will be discussed in §5.3.3. We realize in matrices the group G = U(2, 2) as t {g ∈ GL(4, C) : g I2,2g = I2,2}, where I2,2 := diag(1, 1, −1, −1). We shall identify g1 O KC GL(2, C) × GL(2, C), ↔ (g1, g2), Og2 OA pC M(2, C) ⊕ M(2, C), ↔ (A, B). BO

Then the adjoint action of KC on pC is given by ( , ) → ( −1, −1) A B g1 Ag2 g2 Bg1 (5.3.2) for (g1, g2) ∈ GL(2, C) × GL(2, C). Furthermore, the nilpotent cone NpC = NgC ∩ pC is given by {(A, B) ∈ M(2, C) ⊕ M(2, C) : Both AB and BAare nilpotent matrices}. N ≤ , , , ≤ We deﬁne a subset (possibly an empty set) of pC for 0 i j k l 2by O ij :={(A, B) ∈ NpC : rank A = i, rank B = j, k l rank AB = k, rank BA = l}.

Since rank A, rank B, rank AB, and rank BAare invariants of the KC-action (5.3.2), all Oij are KC-invariant sets. Furthermore, the following proposition holds: k l 170 T. Kobayashi

Proposition 5.3.2. 1) Each Oij is a single KC-orbit if it is not empty. N k l 2) The nilpotentcone pC splits into 10 KC-orbits. These orbits together with their dimensions are listed in the following diagram:

FIGURE 5.3.2.

O Here | stands for the closure relation O ⊃ O. O There is also known a combinatorial description of nilpotent orbits of classical reductive Lie groups. For example, nilpotent orbits in u(p, q) are parameterized by signed Youngdiagrams ([8, Theorem 9.3.3]). For the convenience of readers, let us illustrate brieﬂy it by the above example. Asigned Young diagram of signature (p, q) isaYoung diagram in which every box is labeled with a + or − sign with the following two properties: 1) The number of boxes labeled + is p, while that of boxes labeled − is q. 2) Signs alternate across rows (they do not need to alternate down columns). Two such signed diagrams are regarded as equivalent if and only if one can be obtained by interchanging rows of equal length. For a real reductive Lie group G, the number of KC-orbits on NpC is ﬁnite ([60]). Furthermore, there is a bijection (the Kostant–Sekiguchi correspondence [89]) { N }↔{ N }. KC-orbits on pC G-orbits on g N Therefore, KC-orbits on pC are also parameterized by signed Young diagrams. For example, Figure 5.3.2 may be written as Restrictions of Unitary Representations of Real Reductive Groups 171

FIGURE 5.3.3.

5.3.3 Interpretationsfrom representation theory

We have given a combinatorial description of all KC-orbits on NpC . The goal of this subsection is to provide (without proof) a list of the orbits which are realized as the associated varieties of speciﬁc representations of G = U(2, 2).

1) Oij is the associated variety of some highest (or lowest) weight module of G if k l and only if i = 0or j = 0, namely, Oij corresponds to one of the circled points k l (Fig. 5.3.4 below). 2) Oijis the associated variety of some (Harish-Chandra) discrete series representa- k l tions of G (i.e. irreducible unitary representations that can be realized in L2(G)) if and only if Oij corresponds to one of the circled points (Fig. 5.3.5 below). k l To see this, we recall the work of Beilinson and Bernstein which realizes irreducible (g, K )-modules with regular infinitesimal characters by using D-modules on the flag variety of GC GL(4, C). The D-module which corresponds to a discrete series representation is supported on a closed KC-orbit W, and its associated variety is given by the image of the moment map of the conormal bundle of W ([4]). A combinatorial description of this map for some classical groups may be found in [109]. For the case of G = U(2, 2), compare [41, Example 3.7] for the list of all KC-orbits on the flag variety of GC with [44, §7] (a more detailed explanation of Figure 5.3.2) for those of the image of the moment map of the conormal bundles. 172 T. Kobayashi

FIGURE 5.3.4.

FIGURE 5.3.5.

5.3.4 Case (G, G1) = (U(2, 2), Sp(1, 1)) (essentially, (SO(4, 2), SO(4, 1)))

The goal of this subsection is to classify the KC-orbits Oij on NpC such that (O ) k l = ( , ) prg→g ij is contained in the nilpotent cone of g1C, in the case G1 Sp 1 1 1 k l Spin(4, 1). Let g1 = k1 +p1 be a Cartan decomposition of g1, and we shall identify p1C with M( , C) gC → g C pC M( , C) ⊕ 2 . The projection prg→g1 : 1 when restricted to 2 M(2, C) is given by M( , C) ⊕ M( , C) → M( , C), prg→g1 : 2 2 2 (A, B) → A + τ B, a b −db where τ := . cd c −a Then we have Restrictions of Unitary Representations of Real Reductive Groups 173

(O ) ⊂ N ≤ , , , ≤ Proposition 5.3.4. prg→g ij p1C if andonly if 0 i j k l 1, namely, 1 k l Oij corresponds to oneofthe circled points (Fig. 5.3.6 below): k l

FIGURE 5.3.6.

5.3.5 Case (G, G2) = (U(2, 2), U(2, 1) × U(1))

We write down the result without proof: O N (O ) ⊂ N Proposition 5.3.5. Let ij be a KC-orbit on pC .Then, prg→g ij p2C if k l 2 k l andonly if l = 0, namely, Oij corresponds to oneofthe circled points (Fig. 5.3.7): k l

FIGURE 5.3.7. 174 T. Kobayashi

5.3.6 Case (G, G3) = (U(2, 2), U(2) × U(2))

Since G3 is compact, we have obviously (O ) ⊂ N O N O Proposition 5.3.6. prg→g ij p3C for any KC-orbit ij in pC , namely ij 3 k l k l k l is oneofthe circled points below (Fig. 5.3.8):

FIGURE 5.3.8.

It would be an interesting exercise to compare the above circled points with those in §5.3.3 (meaning from representation theory).

6 Admissible restriction and microlocal analysis

Section 6 tries to explain how an idea coming from microlocal analysis leads to a criterion for admissibility of restrictions of unitary representations. The main idea is to look at the singularity spectrum (or the wave front set) of the hyperfunction (or distribution) character of an irreducible representation. Its ﬂavor is explained in §6.1. Basic concepts such as asymptotic K -support are introduced in §6.2, and they will play a crucial role in the main results of §6.3. In §6.4, we give a sketch of the idea of an alternative (new) proof of Theorem 6.3.3 using symplectic geometry.

6.1 Hyperfunction characters

This subsection explains very briefly reasons why we need distributions or hyperfunctions in defining characters of infinite-dimensional representations and how Restrictions of Unitary Representations of Real Reductive Groups 175 they work. The exposition of Schwartz’s distribution characters (§6.1.1- 6.1.3) is in- fluenced by Atiyah’s article [1], and the exposition for Sato’s hyperfunction characters (§6.1.4–6.1.6) is intended to explain the idea of our main applications to restrict- ingunitary representations in the spirit of the papers [33, 43] of Kashiwara–Vergne and the author.

6.1.1 Finite group

Let G be a ﬁnite group. Consider the (left) regular representation π of G on

L2(G) Cg. g∈G

In light of a matrix expression of π(g) with respect to the basis on the right-hand side, the character χπ of π is given by the following form: #G (g = e), Trace π(g) = 0 (g = e).

6.1.2 Dirac’s delta function

How does the character formula look like as the order #G of G goes to inﬁnity? The character will not be a usual function if #G =∞. We shall take G = S1 as an example of an inﬁnite group below, and consider the character of the regular representation π. Then we shall interpret Trace π as the Dirac delta function both from Schwartz’s distributions and Sato’s hyperfunctions.

6.1.3 Schwartz’s distributions

Let G = S1 R/2πZ. Let π be the regular representation π of S1 on L2(S1). From the viewpoint of Schwartz’s distributions, the character χπ = Trace π is essentially Dirac’s delta function: Lemma 6.1.3. Trace π(θ)dθ = 2πδ(θ)

Sketch of proof. Take a test function f ∈ C∞(S1), and we develop f into the Fourier series: 1 inθ f (θ) = an( f )e , 2π n∈Z ( ) = (θ) −inθ θ where an f : S1 f e d . We do not go into technical details here (such as the proof of the fact that (θ)π(θ) θ 2( 1) S1 f d is a trace class operator on L S ), but present only a formal computation 176 T. Kobayashi " f, Trace π(θ)dθ#= f (θ) Trace π(θ)dθ S1 = f (θ)e−imθ dθ S1 m∈Z = am( f ) m∈Z = 2π f (0) ="f, 2πδ(θ)#.

This is what we wanted to verify.

6.1.4 Sato’s hyperfunctions

Let us give another interpretation of the character of the regular representation of G = S1.Weregard S1 as the unit circle in C. From the viewpoint of Sato’shy- perfunctions [32, 83], the character χπ is given as boundary values of holomorphic functions: Lemma 6.1.4. 1 −1 Trace π(θ) = lim + lim . z→eiθ 1 − z z→eiθ 1 − z |z|↑1 |z|↓1

Sketch of proof (heuristic argument). Formally, we may write θ Trace π(θ) = ein = zn, n∈Z n∈Z where we put z = eiθ . Let us “compute” this infinite sum as follows: zn = zn + zn (6.1.4) n∈Z n≥0 n<0 − = 1 + 1 . 1 − z 1 − z Since the first term converges in |z| < 1 and the second one in |z| > 1, there is no intersection of domains where the above formula makes sense in a usual way. However, it can be justified as boundary values of holomorphic functions, in the theory of hyperfunctions. This is Lemma 6.1.4.

6.1.5 Distributionsorhyperfunctions

The distribution character in Lemma 6.1.3 is essentially the same as the hyperfunction character in Lemma 6.1.4. Cauchy’s integral formula bridges them. Restrictions of Unitary Representations of Real Reductive Groups 177

To see this, we take an arbitrary real analytic test function F(z) ∈ A(S1). Then F(z) extends holomorphically in the complex neighborhood of S1, say, 1 − 2ε< |z| < 1 + 2ε for some ε>0. Then, 3 3 1 1 1 −1 F(z)dz + F(z)dz 2πi | |= −ε 1 − z 2πi | |= +ε 1 − z z 31 z 1 1 F(z) = dz = F(1) (6.1.5) 2πi γ z − 1 where γ is a contoursurrounding z = 1. We deﬁne a function f on R/2πZ by

f (θ) = eiθ F(eiθ ).

We note F(1) = f (0).Ifz = eiθ , then F(z)dz = if(θ)dθ. Hence, the formula (6.15) amounts to 1 2π 1 −1 ( lim + lim ) f (θ)dθ = f (0). (6.1.6) θ θ 2π 0 z→ei 1 − z z→ei 1 − z |z|↑1 |z|↓1

Since any smooth test function on S1 can be approximated by real analytic functions, the formula (6.16) holds for any f ∈ C∞(S1). Hence we have shown the relation between Lemma 6.1.3 and Lemma 6.1.4.

6.1.6 Strategy

We recall that our main goal in this chapter is the restriction π|G in the setting where • π is an irreducible unitary representation of G, • G is a reductive subgroupofG.

We are particularly interested in the (non-)existence of continuous spectrumin the irreducible decomposition of the restriction π|G . For this, our strategyissummarized as follows:

1) Restriction of a representation π toasubgroup G. ⇑ 2) Restriction of its character Trace π toasubgroup. ⇑ 3) Restriction of a holomorphic function to a complex submanifold.

For (2), we shall ﬁnd that the interpretation of a character as a hyperfunction ﬁts well. Then (3) makes sense if the domain of holomorphy is large enough. In turn, the domain of holomorphy will be estimated by the invariants of the representation π, namely the asymptotic K -support ASK (π), which we are going to explain in the next section. 178 T. Kobayashi 6.2 Asymptotic K -support

Associated to a representation π of a compact Lie group K , the asymptotic K - support ASK (π) is defined as a closed cone in a positive Weyl chamber. Here, we are mostly interested in the case where π is infinite dimensional (therefore, not irreducible as a K -module). The goal of this subsection is to introduce the definition of the asymptotic K -support and to explain how it works for the criterion of admissibility of the restriction of a unitary representation of a (noncompact) reductive Lie group G based on the ideas explained in §6.1.

6.2.1 Asymptotic cone

Let S beasubset of a real vector space RN . The asymptotic cone S∞ is an important notion in microlocal analysis (e.g. [32], Deﬁnition 2.4.3), which is a closed cone deﬁned by

N S∞ :={y ∈ R : there exists a sequence (yn,εn) ∈ S × R>0 such that lim εn yn = y and lim εn = 0}. n→∞ n→∞ Example 6.2.1. We illustrate the correspondence

S ⇒ S∞ with two-dimensional examples (see Fig. 6.2.1).

Figure 6.2.1 Restrictions of Unitary Representations of Real Reductive Groups 179

6.2.2 Cartan–Weyl highest weighttheory We review quickly a well-known fact about the ﬁnite-dimensional representations of compact Lie groups and ﬁx notation as follows. Let K be a connected compact Lie group, and take a maximal torus T . We write + k and t for their√ Lie algebras, respectively. Fix a positive root system (k, t), and ∗ write C+ √(⊂ −1t ) for the corresponding closed Weyl chamber. We regard T as a lattice of −1t∗ and put + := T ∩ C+.

We note that the asymptotic cone +∞ is equal to C+. For λ ∈ +, we shall denote by τλ an irreducible (ﬁnite-dimensional) representation of K whose highest weight is λ. Then Cartan–Weyl highest weight theory (for a connected compact Lie group) establishes a bijection between K and +: √ ∗ K Λ+ ⊂ C+ ⊂ −1t ∈ ∈

τλ ↔ λ

6.2.3 Asymptotic K -support For a representation π of K , we deﬁne the K -support of π by (π) ={λ ∈ (τ ,π) = }. SuppK : + : HomK λ 0 Later, π will be a representation of a noncompact reductive Lie group G when restricted to its maximal compact subgroup K .Thus, we have in mind the case where π is inﬁnite dimensional (and therefore, not irreducible as a K -module). Its asymptotic cone (π) = (π)∞ ASK : SuppK is called the asymptotic K -support of π. We note that it is a closed cone contained (π) ⊂ = ∞ in the closed Weyl chamber C+, because SuppK + and C+ + . The asymptotic K -support ASK (π) was introduced by Kashiwara and Vergne [33] and has the following property (see [43, Proposition 2.7] for a rigorous statement):

Thesmaller the asymptotic K -support ASK (π),thelarger thedomain of KC in which thecharacter Trace(π) extends holomorphically.

For further properties of the asymptotic K -support ASK (π), see [33, 43, 53]. In particular, we mention: Theorem 6.2.3. [53] Suppose π ∈ G,ormore generally, π is a (K-)admissible representation of G of finite length.(π is not necessarilyunitary.) Then the asymptotic K-support ASK (π) is a finite union of polytopes, namely,there exists a finite set {αij}⊂C+ such that l (π) = R {α ,α ,... ,α }. ASK ≥0- span k1 k2 kmk k=1 180 T. Kobayashi

6.2.4 Examples fromSL(2, R)

Let G = SL(2, R). We recall the notation in Section 3; in particular, we identify K Z (π) (π) π with . Here is a list of SuppK and ASK for some typical representations of G. π (π) (π) SuppK ASK (1) 1 {0} {0} (2) πλ 2Z R ( ) π + { , + , + ,...} R 3 n n n 2 n 4 ≥0 ( ) π − {− , − − , − − ,...} R 4 n n n 2 n 4 ≤0

Here 1 denotes the trivial one-dimensional representation, πλ a principal series rep- π ± ( ≥ ) resentation, and n n 2 a holomorphic or anti-holomorphic representation of G. Now let us recall §6.1.4 for a hyperfunction character. In the case (2), the charac- (π) = ikθ 2( 1) ter Trace k∈2Z e has a similar nature to the character of L S computed in §6.1. In the cases (3) and (4), the asymptotic K -support is smaller, and the charac- (π) = ±ikθ ters Trace k∈n+2N e converges in a larger complex domain, as we have seen in a similar property for each of the summands in (6.1.4).

6.3 Criterion for admissible restriction

6.3.1 The closed coneCK (K )

Let K be a connected compact Lie group, and K its closed subgroup. Dual to the inclusion k ⊂ k of Lie algebras, we write ∗ → ( )∗ prk→k : k k ( )⊥ for the projection, and k for the kernel of prk→k . We fix a positive definite and Ad(K )-invariant bilinear form on k, and regard t∗ asasubspace of k∗. √Associated to the pair (K, K ) of compact Lie groups, we define a closed cone in −1t∗ by √ ∗ ⊥ C(K ) ≡ CK (K ) := C+ ∩ −1Ad (K )(k ) . (6.3.1)

6.3.2 Symmetric pair

For a compact symmetric pair (K, K ), the closed cone CK (K ) takes a very simple form. Let us describe it explicitly. Suppose that (K, K ) is a symmetric pair deﬁned by an involutive automorphism σ of K .Asusual, the differential of σ will be denoted by the same letter. Taking a conjugation by K if necessary, we may and do assume that t and +(k, t) are chosen so that Restrictions of Unitary Representations of Real Reductive Groups 181

−σ −σ −σ 1) t := t ∩ k is a maximal abelian subspace of k , + −σ + ( , ) ={λ| −σ λ ∈ ( , )}\{ } 2) k t : t : k t 0 is a positive system of the restricted root system (k, t−σ ). √ −σ ∗ −σ ∗ We write (t )+ (⊂ −1(t ) ) for the corresponding dominant Weyl chamber. Then we have (see [23]). −σ ∗ Proposition 6.3.2. CK (K ) = (t )+. One can also give an alternative proof of Proposition 6.3.2 by using Theo- rem 6.4.3 below and a Cartan–Helgason theorem [105], §3.3.1.

6.3.3 Criterion for K -admissibility

We are ready to explain the main results of Section 6, namely, Theorem 6.3.3 and Theorem 6.3.4 regarding a criterion for admissible restrictions. Theorem 6.3.3 ([43, 53]). Let K ⊃ K be a pair of compact Lie groups, andXa K-module. Then the following twoconditions are equivalent: 1) XisK-admissible. 2) CK (K ) ∩ ASK (π) ={0}.

Sketch of proof. Let us explain an idea of the proof (2) ⇒ (1). The K -character Trace(π|K ) of X is a distribution (or a hyperfunction) on K . The condition (2) implies that its wave front set (or its singularity spectrum) is transversal to the submanifold K . Then the restriction of this distribution (or hyperfunction) to K is well deﬁned, and coincides with the K -character of X (see [43], Theorem 2.8 for details).

6.3.4 Sufﬁcientcondition for G-admissibility

Now, let us return our original problem, namely, the restriction to a noncompact reductive subgroup. Theorem 6.3.4. ([43]) Let π ∈ G, and G ⊃ G be a pair of reductive Lie groups. Take maximalcompact subgroups K ⊃ K , respectively.If

CK (K ) ∩ ASK (π) ={0},

then the restriction π|G is G -admissible, that is, π|G splits discretely: ⊕ π|G nπ (τ)τ τ∈G into irreducible representationsofG withnπ (τ) < ∞ for any τ ∈ G . 182 T. Kobayashi

6.3.5 Remark The assumption of Theorem 6.3.4 is obviously fulﬁlled if CK (K ) ={0} or if ASK (π) ={0}. What do these extremal cases mean? Here is the answer: 1) CK (K ) ={0}⇔K = K . In this case, Theorem 6.3.4 is nothing but Harish-Chandra’s admissibility theorem, as we explained in Section 2 (see Theorem 2.4.6).

2) ASK (π) ={0}⇔dim π<∞. In this case, the conclusion of Theorem 6.3.4 is nothing but the complete reducibility of a ﬁnite-dimensional unitary representation. Thus, the second case is more or less trivial. In this connection, one might be tempted to ask when ASK (π) becomes the second smallest, namely, of the form R+v generated by a single element v? It follows from the following result of Vogan [94] that this is always the case if π is a minimal representation of G (in the sense that its annihilator is the Joseph ideal). Theorem 6.3.5 (Vogan). Let π be aminimal representation of G.Then there exists aweight ν such that πK τmv+ν, m∈N asK-modules, where v is the highest weightofp, and τmv+ν is the irreducible representation of K withhighest weight mv + ν.In particular,

ASK (π) = R+v.

This reﬂects the fact that there are fairly rich examples of a discretely decomposable restriction of the minimal representation π of a reductive group G with respect to a noncompact reductive subgroup G (e.g. Howe [24] for G = Mp(n, R), Kobayashi–Ørsted [57] for G = O(p, q), and Gross–Wallach [19] for some exceptional Lie groups G).

6.4 Application of symplectic geometry 6.4.1 Hamiltonian action Let (M,ω)be a symplectic manifold on which a compact Lie group K acts as symplectic automorphisms by τ : K × M → M. We write k → X(M), X → X M for the vector ﬁeld induced from the action. The action τ is called Hamiltonian if there exists a map ∗ : M → k X X such that d = ι(X M )ω for all X ∈ k, where we put (m) ="X, (m)#.We say is the momentummap.Forasubset Y of M, The momentum set (Y ) is deﬁned by √ (Y ) = −1 (Y ) ∩ C+. Restrictions of Unitary Representations of Real Reductive Groups 183

6.4.2 Afﬁne varieties

Let V be a complex Hermitian vector space. Assume that a compact Lie group K acts on V as a unitary representation. Then the action of K is also symplectic if we equip V with the symplectic form ωV (u,v) =−Im(u,v), as we saw the inclusive relation U(n) ⊂ Sp(n, R) in Section 1 (see Example 1.5.5). Then, V is Hamiltonian with the momentum map √ ∗ −1 : V → k , (v)(X) = (Xv,v), (X ∈ k). 2 We extend the K -action to a complex linear representation of KC.Suppose V is a KC-stable closed irreducible afﬁne variety of V .Wehavenaturally a representa- C V (C V ) tion of K on the space of regular functions [ ]. Then SuppK [ ] is a monoid, namely, there exists a ﬁnite number of elements in +, say λ1,...,λk,such that (C V ) = Z {λ ,...,λ }. SuppK [ ] ≥0-span 1 k (6.4.2) The momentum set (V) is a “classical” analog of the set of highest weights (C) SuppK . That is, the following theorem holds: (V) = R (C V ) Proposition 6.4.2. [88] ≥0 SuppK [ ] . Together with (6.4.2), we have (V) = R {λ ,...,λ }= ( (C V )). ≥0-span 1 k ASK SuppK [ ]

6.4.3 Cotangentbundle

Let M = T ∗(K/K ), the cotangent bundle of the homogeneous space K/K . We define an equivalence relation on the direct product K × k⊥ by (k,λ) ∼ (kh, Ad∗(h)−1λ) for some h ∈ H, and write [k,λ] for its equivalence class. Then the ⊥ set of equivalence classes, denoted by K ×K k , becomes a K -equivariant homo- geneousbundle over K/K , and is identified with T ∗(K/K ). Then the symplectic manifold T ∗(K/K ) is naturally a Hamiltonian with the momentum map ∗ ∗ ∗ : T (K/K ) → k , [k,λ] → Ad (k)λ. ∗ The momentum set (T (K/K )) equals to CK (K ), as follows from the definition (6.3.1). The closed cone CK (K ) can be realized as the asymptotic K -support of a certain induced representation of K . That is, we can prove: Theorem 6.4.3. [53] Let τ be an arbitrary finite-dimensional representation of K . Then, ( ) = ( K (τ)). CK K ASK IndK In particular, 2 CK (K ) = ASK (L (K/K )). (6.4.3) 184 T. Kobayashi

( K (τ)) Here we note that the asymptotic K -support ASK IndK does not change whatever we take as the class of functions (algebraic, square integrable, hyperfunc- ... K (τ) tions, ) in the deﬁnition of the induced representation IndK .

7 Discretely decomposable restriction of Aq(λ)

In the philosophy of the Kostant–Kirillov orbit method, there is an important family of irreducible unitary representations of a reductive Lie group G, “attached to” elliptic coadjoint orbits. Geometrically, they are realized in dense subspaces of Dolbeault cohomology groups of certain equivariant holomorphic line bundles. This is a vast generalization duetoLanglands, Schmid, and others of the Borel–Weil–Bott theorem: from compact to noncompact; from ﬁnite dimensional to inﬁnite dimensional. Algebraically, they are also expressed as Zuckerman’s derived functor modules Aq(λ) by using so-called cohomological parabolic induction. Among all of them in importantance is the unitarizability theorem of these modules under certain positivity conditions on the parameter, as proved by Vogan and Wallach. Analytically, some of them can be realized in L2-spaces on homogeneous spaces. For example, Harish-Chandra’s discrete series representations for group manifolds and Flensted-Jensen’s for symmetric spaces are such cases. Subsection 7.1 collects some of basic results about these representations from these three — geometric, algebraic, and analytic — aspects. Subsection 7.2 speculates about their restrictions to noncompact subgroups, and examines the results of Sections 5 and 6 for these modules.

7.1 Elliptic orbits and geometric quantization

7.1.1 Elliptic orbits

Consider the adjoint action of a Lie group G on its Lie algebra g.ForX ∈ g,we deﬁne the adjoint orbit OX by

OX := Ad(G)X G/G X .

Here, G X is the isotropy subgroupatX, given by {g ∈ G :Ad(g)X = X}.

Deﬁnition 7.1.1. An element X ∈ g is elliptic if ad(X) ∈ EndC(gC) is diagonal- izable and if all eigenvalues are purely imaginary. Then, OX is called an elliptic orbit. Example. If G is a compact Lie group, then any adjoint orbit is elliptic. Let g = k + p be a Cartan decomposition of the Lie algebra g of G.Weﬁxa maximal abelian subspace t of k. Restrictions of Unitary Representations of Real Reductive Groups 185

FIGURE 7.1. (Co)adjoint orbits of G = SL(2, R)

Any elliptic element is conjugate to an element in k under the adjoint action of G.Furthermore, any element of k is conjugate to an element in t under the adjoint action of K . Hence, for X ∈ g, we have the following equivalence:

OX is an elliptic orbit ⇔ OX ∩ k = φ ⇔ OX ∩ t = φ.

From now on, without loss of generality, we can and do take X ∈ t when we deal with an elliptic orbit.

7.1.2 Complex structure onanelliptic orbit

Every elliptic orbit OX carries a G-invariant complex structure. There are several choices of G-invariant complex structures on OX . We shall choose one in the following manner. (See, for√ example, [56] for further details.) First, we note that −1ad(X) ∈ End(gC) is a semisimple√ transformation with all eigenvalues real. Then the eigenspace decomposition of −1ad(X) leads to the Gelfand–Naimark decomposition

− + gC = u + (gX )C + u . (7.1.2) √ Here u+ (respectively, u−) is the direct sumofeigenspaces of −1ad(X) with positive (respectively, negative) eigenvalues. We deﬁne a parabolic subalgebra of gC by 186 T. Kobayashi

+ q := (gX )C + u .

For simplicity, suppose G is a connected reductive Lie group contained in a (connected) complex Lie group GC with Lie algebra gC. Let Q be the parabolic subgroup of GC with Lie algebra q. We note that Q is a connected complex subgroupofGC. Then the key ingredients here are

G ∩ Q = G X , g + q = gC.

Hence the natural inclusion G ⊂ GC induces an open embedding of OX into the generalized ﬂag variety GC/Q:

OX G/G X → GC/Q. open

Hence we can deﬁne a complex structure on the adjoint orbit OX from the one on GC/Q. Obviously, the action of G on OX is biholomorphic. For a further structure on OX , we note that OX contains another (smaller) generalized ﬂag variety OK = ( ) / , X : Ad K X K K X of which the complex dimension will be denoted by S. In summary, we have OK ⊂ O → / , X X GC Q closed open and in particular,

Proposition 7.1.2. Any elliptic orbit OX carries aG-invariantcomplex structure through an open embedding into the generalized ﬂag varietyGC/Q. Furthermore, O OK X contains a compact complex submanifold X .

7.1.3 Elliptic coadjoint orbit

For a reductive Lie group G, adjoint orbits on g and coadjoint orbits on g∗ can be identiﬁed via a nondegenerate G-invariant bilinear form. For example, such a bilinear form is given by g × g → R,(X, Y ) → Trace(XY) if g is realized√ in gl(N, R) such that t g√= g. ∗ Let λ ∈ −1g . We write Xλ ∈ −1g for the corresponding element via the isomorphism √ √ ∗ −1g −1g. √ We write X :=− −1Xλ ∈ g. Then isotropy subgroups of the adjoint action and the coadjoint action coincide — G X = Gλ. Restrictions of Unitary Representations of Real Reductive Groups 187

Assume that X is an elliptic element. With analogous notation as in §7.1.2, Proposition 7.1.2 states that the coadjoint orbit ∗ Oλ := Ad (G) · λ G/Gλ = G/G X Ad(G) · X =: OX carries a G-invariant complex structure. The Lie algebra√ of Gλ is given by gλ :={X ∈ g : λ([X, Y ]) = 0 for all Y ∈ g}. ∗ We deﬁne ρλ ∈ −1gλ by + + ρλ(Y ) := Trace(ad(Y ) : u → u ), for Y ∈ gλ.Wesayλ is integral if the Lie algebra homomorphism

λ + ρλ : gλ → C lifts to a character of Gλ. For simplicity, we shall write Cλ+ρλ for the lifted character. Then

Lλ := G ×Gλ Cλ+ρλ → Oλ is a G-equivariant holomorphic line bundle over the coadjoint orbit Oλ.

7.1.4 Geometric quantization a` laSchmid–Wong

This subsection completes the following scheme of “geometric quantization” of an elliptic coadjoint orbit Oλ. √ λ ∈ −1g∗ an elliptic and integral element > ↓ Lλ → Oλ a G-equivariant holomorphic line bundle > ↓ ∗(O , L ) (λ) H∂¯ λ λ a representation of G We have already explained the ﬁrst step. Here is a summary of the second step: √ Theorem 7.1.4. Let λ ∈ −1g∗ be elliptic andintegral. j (O , L ) 1) (topology)The Dolbeault cohomology groupH∂¯ λ λ carries a Frec´ het topology,onwhichGacts continuously. j (O , L ) = = 2) (vanishing theorem)H∂¯ λ λ 0 if j S. H S(O , L ) 3) (unitarizability)There is a dense subspace in H∂¯ λ λ withwhichaG- invariant Hilbert structure can be equipped. 4) (irreducibility)Ifλ is “sufﬁciently regular”,then the unitary representation of G on H is irreducible and nonzero. We shall denote by (λ) the unitary representation constructed in Theorem 7.1.4 (3). Here are some comments on and further introductions to Theorem 7.1.4. 188 T. Kobayashi

1) The nontrivial part of the statement (1) is that the rangeofthe∂¯-operator is closed with respect to the Frechet´ topology on the space of (0, q)-forms. The difficulty arises from the fact that Oλ G/Gλ is noncompact. This closed range problem was solved affirmatively by Schmid in the case Gλ compact, and by H. Wong for general Gλ early in 1990s [107]. 2) This vanishing result is an analog of Cartan’s Theorem for Stein manifolds. We note that Oλ is Stein if and only if S = 0. In this case, Oλ is biholomorphic to a Hermitian symmetric space of noncompact type, and the statement (2) asserts the vanishing of all cohomologies in higher degrees. The resulting representations in the 0th degree in this special case are highest weight representations. An opposite extremal case is when G is compact (see the next subsection §7.1.5). In this case, our choice of the complex structure on OX implies that the dual of Lλ is ample, and the statement (2) asserts that all the cohomologies vanish except for the top degree. 3) The unitarizability was conjectured by Zuckerman, and proved under a certain positivity condition on the parameter λ by Vogan [96] and Wallach independently [103] in the 1980s. See also a proof in the works of Knapp–Vogan [37] or Wallach [104]. In our formulation which is suitable for the orbit method, this positivity condition is automatically satisfied. 4) Vogan introduced the condition “good range” and a slightly weaker one “fair range”. The statement (4) of Theorem 7.1.4 holds if λ is in the good range ([96]). Special cases of Theorem 7.1.4 contain many interesting representations as we shall see in §7.1.5 ∼§7.1.7.

7.1.5 Borel–Weil–Bott theorem

If G is a compact Lie group, then any coadjoint orbit Oλ is elliptic and becomes a compact complex manifold (a generalized flag variety). Then by a theorem of j (O , L ) Kodaira–Serre, the Dolbeault cohomology groups H∂¯ λ λ are finite dimensional. The representations constructed in Theorem 7.1.4 are always irreducible, and ex- haust all irreducible (finite dimensional, unitary) representations of G. This is known as the Borel–Weil–Bott construction.

7.1.6 Discrete series representations

Suppose (X,µ)is a G-space with G-invariant measure µ. Then on the Hilbert space L2(X, dµ) of square integrable functions, there is a natural unitary representation of G by translations. Deﬁnition 7.1.6. An irreducible unitary representation π of G is called a discrete series representation for L2(X, dµ) (or simply, for X)ifπ can be realized in a G-invariant closed subspace of L2(X, dµ),orequivalently, if

2 dim HomG (π, L (X)) = 0, Restrictions of Unitary Representations of Real Reductive Groups 189 where HomG denotes the space of continuous G-intertwining operators. We shall write Disc(X) for the subset of G consisting of all discrete series representations for L2(X, dµ). It may happen that Disc(X) = φ.

7.1.7 Harish-Chandra’s discrete series representations

Let G be a real reductive linear Lie group. If (X,µ) = (G, Haar measure) with left G-action, then Disc(G) was classiﬁed by Harish-Chandra. In the context of Theo- rem 7.1.4, Disc(G) is described as follows: Theorem 7.1.7. Let G be a realreductivelinear Lie group.

Disc(G) ={(λ) : λ is integral and elliptic, Gλ is a compact torus.}

This theorem presents a geometric construction of discrete series representations. Such a construction was conjectured by Langlands in the 1960s via L2-cohomology and proved by Schmid [85]. We can see easily that there exists λ such that Gλ is a compact torus if and only if rank G = rank K . In this case, there are countably many integral and elliptic λ such that Gλ is a compact torus. In particular, the above formulation of Theorem 7.1.7 includes a Harish-Chandra celebrated criterion:

Disc(G) = φ ⇔ rank G = rank K. (7.1.7)

7.1.8 Discrete series representations for symmetric spaces

Suppose G/H is a reductive symmetric space. Here are some typical examples.

SL(p + q, R)/SO(p, q), GL(p + q, R)/(GL(p, R) × GL(q, R)), GL(n, C)/GL(n, R).

Without loss of generality, we may assume that H is stable under a ﬁxed Cartan involution θ of G. Then we may formulate the results of Flensted-Jensen, Matsuki– Oshima and Vogan on discrete series representations for reductive symmetric spaces as follows: Theorem 7.1.8. Let G/Hbea reductivesymmetric space. Then ⎧ ⎫ ⎨ λ is elliptic, andsatisﬁes a certain ⎬ ( / ) = (λ) λ| ≡ . Disc G H ⎩ : integralcondition, h 0, ⎭ Gλ/(Gλ ∩ H) is a compact torus

We note that the original construction of discrete series representations for G/H did not use Dolbeault cohomology groups but used the Poisson transform of hyperfunctions (or distributions) on real ﬂag varieties. It follows from the duality theorem 190 T. Kobayashi due to Hecht–Miliciˇ c–Schmid–Wolf´ [22] that these discrete series representations are isomorphic to some (λ). The above formulation of the description of discrete series representations is taken from the author’s exposition ([45, Example 2.9]). As in the case of Harish-Chandra’s discrete series representations, we can see easily that such λ exists if and only if rank G/H = rank K/H ∩ K . Hence Theo- rem 7.1.8 contains a criterion for the existence of discrete series representations for reductive symmetric spaces:

Disc(G/H) = φ ⇔ rank G/H = rank K/H ∩ K.

This generalizes (7.1.7) (since the group case can be regarded as a symmetric space (G × G)/ diag(G)), and was proved by Flensted-Jensen, Matsuki and Oshima (see [16, 45, 84] for further reading).

7.2 Restriction of (λ) attached to elliptic orbits

7.2.1 Asymptotic K -support, associated variety

Throughout this section, (λ) will be a unitary representation of G attached to an integral elliptic coadjoint orbit Oλ.Weassume that (λ) is nonzero. This is the case if λ is in the good range ([96, 97]). √ We may and do assume Xλ ∈ −1t (see (7.1.1)). Let gC = kC + pC be the complexiﬁcation of a Cartan decomposition, and

− + gC = uλ + (gλ)C + uλ

√the Gelfand–Naimark√ decomposition (7.1.2) corresponding to the action of −1ad(− −1Xλ) = ad(Xλ). We deﬁne the set of t-weights (positive noncompact roots) by √ + + ∗ λ (p) := (uλ ∩ pC, t) ⊂ −1t . (7.2.1)

Here is an explicit estimate on the asymptotic K -support, and the associated variety of (λ). Theorem 7.2.1. Suppose that (λ) = 0 in the above setting. + 1) ASK ((λ)) ⊂ R≥0 -span λ (p). V ((λ)) = ∗( )( − ∩ ) 2) gC Ad KC uλ pC . Proof. See [43, §3] for the Statement (1). See [4] or [97] for the Statement (2) for a regular λ; and [44] for a general case.

7.2.2 Restriction to a symmetric subgroup

We recall the notation and convention in §6.3.2, where (G, G) is a reductive symmetric pair deﬁned by an involutive automorphism σ of G. In particular,wehave Restrictions of Unitary Representations of Real Reductive Groups 191

−σ ∗ CK (K ) = (t )+.

Then the following theorem tells us explicitly when the restriction (λ)|G is infinitesimally discretely decomposable. We note that the approach by associated varieties and wave front sets leads us to the same criterion for admissible restrictions. Theorem 7.2.2 ([44, Theorem 4.2]). Let (λ) be anonzero unitary representation of Gattached to an integral elliptic coadjoint orbit, and (G, G) a reductivesymmetric pair defined byaninvolutive automorphism σ of G.Then the following four conditionson (G,σ,λ)are equivalent: i) (λ)|K is K -admissible. ii) (λ)|G is infinitesimally discretely decomposable. + −σ ∗ iii) R≥0 -span λ (p) ∩ (t )+ ={0}. ( (KC)( − ∩ C)) ⊂ N . iv) prg→g Ad u p pC

Sketch of proof. (iii)⇒(i) This follows from the criterion of K -admissibility given in Theo- rem 6.3.3 together with Theorem 7.2.1. (i)⇒(ii) See Theorem 4.2.7. (ii)⇒(iv) This follows from the criterion for inﬁnitesimally discretely decomposable restrictions by means of associated varieties. Use Corollary 5.2.3 and Theorem 7.2.1. (iv)⇒(iii) This part can be proved only by techniques of Lie algebras (without representation theory).

7.3 U(2, 2) ↓ Sp(1, 1)

This subsection examines Theorem 7.2.2 by an example

(G, G ) = (U(2, 2), Sp(1, 1)).

More precisely, we shall take a discrete series representation of U(2, 2) (with Gelfand–Kirillov dimension 5), and explain how to verify the criteria (iii) (root data) and (iv) (associated varieties) in Theorem 7.2.2.

7.3.1 Non-holomorphic discrete series representations for U(2, 2)

Let t be a maximal√ abelian subspace of k u(2) + u(2), and take a standard basis ∗ {e1, e2, e3, e4} of −1t such that

(pC, t) ={±(ei − e j ) :1≤ i ≤ 2, 3 ≤ j ≤ 4}.

We shall ﬁx once and for all √ ∗ λ = λ1e1 + λ2e2 + λ3e3 + λ4e4 ∈ −1t 192 T. Kobayashi such that

λ1 >λ3 >λ4 >λ2,λj ∈ Z (1 ≤ j ≤ 4). (7.3.1) Such λ is integral and elliptic, and the resultingunitary representation (λ) is nonzero irreducible. We note that Gλ is isomorphic to a compact torus T4. Then (λ) is a Harish-Chandra’s discrete series representation by Theorem 7.1.7. Its Gelfand– Kirillov dimension is 5, as we shall see in (7.3.3) that its associated variety is ﬁve dimensional. Furthermore, (λ) is not a holomorphic discrete series representation which has Gelfand–Kirillov dimension 4 for G = U(2, 2).

7.3.2 Criterion for K -admissibility for U(2, 2) ↓ Sp(1, 1)

Retain the setting as in §7.3.1. In light of (7.3.1), the set of noncompact positive roots + λ (p) (see (7.2.1) for deﬁnition) is given by + λ (p) ={e1 − e3, e1 − e4, e3 − e2, e4 − e2}. √ Hence, via the identiﬁcation −1t∗ R4,wehave ⎧⎛ ⎞ ⎫ ⎪ + ⎪ ⎨⎪ a b ⎬⎪ ⎜ − − ⎟ R +( ) = ⎜ c d ⎟ ; , , , ≥ . ≥0 -span λ p ⎪⎝ − + ⎠ a b c d 0⎪ (7.3.2) ⎩⎪ a c ⎭⎪ −b + d

On the other hand, for (K, K ) ≡ (K, K σ ) = (U(2) × U(2), Sp(1) × Sp(1)),we have ⎛ ⎞ ⎛ ⎞ 1 0 ⎜ ⎟ ⎜ ⎟ −σ ∗ 1 0 (t ) = R ⎜ ⎟ + R ⎜ ⎟ . + ⎝0⎠ ⎝1⎠ 0 1

+ −σ ∗ Thus, R≥0 -span λ (p) ∩ (t )+ ={0} because the condition ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ a + b 1 0 ⎜− − ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ c d⎟ ∈ R ⎜1⎟ + R ⎜0⎟ ⎝−a + c⎠ ⎝0⎠ ⎝1⎠ −b + d 0 1 leads to a + b =−c − d, which occurs only if a = b = c = d under the assumption a, b, c, d ≥ 0 (see (7.3.2)). Thus, the condition (iii) of Theorem 7.2.2 holds. Therefore the restriction (λ)|K is K -admissible.

7.3.3 Associated variety of (λ)

Retain the setting of the example as above. Although we have already known that all of the equivalent conditions in Theorem 7.2.2 hold, it is illuminating to verify them Restrictions of Unitary Representations of Real Reductive Groups 193

V ((λ)) directly. So let us compute the associated variety gC and verify the condition (iv) of Theorem 7.2.2 using the computation given in Proposition 5.3.4. In light of − (uλ ∩ pC, t) ={−e1 + e3, −e1 + e4, e2 − e3, e2 − e4}, + (=−λ (p)) we have − 00 z 0 u ∩ pC , : x, y, z,w ∈ C λ x y w 0 via the identification pC M(2, C) ⊕ M(2, C) (see (5.3.2) in §5). Hence we have V ((λ)) = ( )( − ∩ ) = O gC Ad KC u pC 11 (7.3.3) 10 by Theorem 7.2.1, where we recall that O11 is a five-dimensional manifold defined 10 by

O11 ={(A, B) : rank A = rank B = rank AB = 1, rank BA = 0}. 10

FIGURE 7.3.

In Figure 7.3 above, the closure of O11 consists of 5 KC-orbits, which are de- 10 scribed by circled points on the left. In view of the classiﬁcation of KC-orbits O on N (O) ⊂ N pC such that prg→g pC by Proposition 5.3.4, one can observe that all the circled points on the left are also those on the right. Hence we conclude that

(O ) ⊂ N . prg→g 11 p 10 C ( (KC)( − ∩ C)) ⊂ N Therefore prg→g Ad u p pC , that is, the condition (iv) of Theo- rem 7.2.2 has been veriﬁed directly. 194 T. Kobayashi 8 Applications of branching problems So far we have explained a basic theory of restrictions of unitary representations of reductive Lie groups, with emphasis on discrete spectrum for noncompact subgroups. In Section 8, let us discuss what restrictions can do for representation theory. Furthermore, we shall discuss brieﬂy some new interactions of unitary representation theory with other branches of mathematics through restrictions of representations to subgroups. For further details on applications, we refer to [47, Sections 3 and 4], and [50, Sections 4, 5 and 6] and references therein.

8.1 Understanding representations via restrictions 8.1.1 Analysis and Synthesis An idea of “analysis through decomposition” is to try to decompose the object into the smallest units, and to try to understand how it can be built up from the smallest units. This method may be pursued until one reaches the smallest units. Then, what can we do with the analysis on the smallest units? Only a complete changeof viewpoint allows ustogofurther. For example, molecules may be regarded as the “smallest units”;but they could be decomposed into atoms which may be regarded as the “smallest” in another sense, and then they consist of electrons, protons and neu- trons, and then... Each step of these decompositions requires a different viewpoint. An irreducible representation π of a group G is the “smallest unit” as representations of G.Nowasubgroup G could provide a “different viewpoint”. In fact, π would no longer then be the “smallest unit” as representations of G. This leads us to a method to study the irreducible representations of G by taking the restriction to G, still in the spirit of “analysis through decomposition”.

8.1.2 Cartan–Weyl highest weighttheory,revisited Let G be a connected compact Lie group, and take G := T to be a maximal toral subgroup. Let π be an irreducible representation of G. Obviously π is T -admissible. We write the branching law of the restriction π| as T π|T nπ (λ)Cλ, (8.1.2) λ where Cλ is a one-dimensional representation of T , and nπ (λ) is its multiplicity. Of course, the whole branching law (that is, weight decomposition) (8.1.2) determines the representation π because ﬁnite-dimensional representations are determined by their characters and because characters are determined on their restrictions to the maximal torus T .Much more strongly, Cartan–Weyl highest weight theory asserts that a single element λ (the “largest” piece in the decomposition (8.1.2)) is sufﬁcient to determine π. This may be regarded as an example of the spirit: to understand representations through their restrictions to subgroups. Restrictions of Unitary Representations of Real Reductive Groups 195

8.1.3 Vogan’s minimalK-type theory

Let G be a reductive linear Lie group, and take G := K to be a maximal compact subgroupofG. Let π be an irreducible unitary representation of G. Then π is K -admissible (Harish-Chandra’s admissibility theorem, see Theorem 2.4.6). As we have already mentioned in Section 4, Harish-Chandra’s admissibility laid the foundation of the theory of (g, K )-modules, an algebraic approach to inﬁnite-dimensional representations of reductive Lie group G. Let us write the branching law as ⊕ π|K nπ (τ)τ. (8.1.3) τ∈K Then Vogan’s minimal K -type theory shows that a single element (or a small number of elements) τ (the “smallest ones” in the branching law (8.1.3)) gives critical information for the classiﬁcation of irreducible (g, K )-modules ([93, 95]) and also for understanding the unitary dual G ([82, 100]). This may be regarded as another example of the same spirit of understanding representations through their restrictions to subgroups.

8.1.4 Restrictionstononcompact groups

In contrast to the restriction to compact subgroups as we discussed in §8.1.2 and §8.1.3, not much is known about the restriction of irreducible unitary representations to noncompact subgroups. I try now to indicate a few examples where the restriction to noncompact subgroups G provides a successfulcluetounderstanding representations π of G. This idea works better, particularly, in the case where π belongsto“singular” (or “small”) representations, which are the most mysterious part of the unitary dual G in the cur- rent status of this ﬁeld. 1) (Parabolic restriction) Some of “small” representations of G remain irreducible when restricted to parabolic subgroups. (See, for example, [59, 106].) Con- versely, Torasso [90] constructed minimal representations by makinguse of their restriction to maximal parabolic subgroups. 2) (Nonvanishing condition for Aq(λ)) In the philosophy of the orbit method, it is perhaps a natural problem to classify all elliptic and integral orbits Ad(G)λ such that the correspondingunitary representation (λ) = 0 (or equivalently, the underlying (g, K )-module Aq(λ) = 0, see §7.1.4). This is always the case if λ is in the good range in the sense of Vogan [96]. For a general O-stable parabolic subalgebra q, the set λ (outside the good range) such that Aq(λ) = 0 may be very complicated. Such a set was found explicitly in [39, Chapter 4] in the special setting where there exists a noncompact reductive subgroup G such that the restriction (λ)|G is G -admissible (see Theorem 7.2.2). Some other (combinatorial) approaches to this problem may be seen in [39, Chapter 5] and [91], but the general case remains open. 196 T. Kobayashi

3) (Jordan–Holder¨ series) Even in the case where π is neither unitary nor irreducible, the restriction to subgroups sometimes gives usagood tool to study the representation. Very recently, Lee and Loke [62] determined explicitly the Jordan– Holder¨ series of certain degenerate principal series representations and classiﬁed which irreducible subquotients are unitarizable. Previously known cases in this direction were limited to the special setting in which degenerate principal series representations have a multiplicity free K -type decomposition as was in the Howe and Tan paper [26]. Lee and Loke’s idea was to generalize this method by replacing a maximal compact subgroup K by a certain noncompact subgroup G such that the restriction to G splits discretely with multiplicity free. These three examples may be also regarded as concrete cases where one can have a better understanding of inﬁnite-dimensional representations through their restrictions to noncompact subgroups.

8.2 Construction of representationsofsubgroups

Suppose we are given a representation π of G. Then the knowledge of (a part of) the branching law of the restriction π|G may be regarded as a construction of irreducible representations of G. One of the advantages of admissible restrictions is that there is no continuous spectrum in the branching law so that each irreducible summand could be explicitly captured. In this way admissible restrictions may also serve as a method to study representations of subgroups.

8.2.1 Finite-dimensional representations

Consider the natural representation of G = GL(n, C) on V = Cn. Then the m-th tensor power T m(V ) = V ⊗···⊗V becomes an irreducible representation of the direct product group G = G ×···×G. The restriction from G to the diagonally embedded subgroupofG is no longer irreducible, and its branching law gives rise to irreducible representations of G as irreducible summands. When restricted to the diagonally embedded subgroup G GL(n, C), it decomposes into irreducible representations of G. (A precise description is given by the Schur–Weyl duality m which treats T (V ) as a representation of the direct product group GL(n, C)×Sm.) Conversely, any polynomial representation of G can be obtained in this way for some m ∈ N. This may be regarded as a construction of irreducible representations of the subgroup GL(n, C) via branching laws.

8.2.2 Highest weight modules

The above idea can be extended to construct some irreducible inﬁnite-dimensional representations, called highest weight representations. For example, let us consider π = ( , R) the Weil representation of the metaplectic group G1 Mp n . Then, the m-th π ⊗···⊗π × ( ) tensor power becomes a representation of G1 O m , which decom- × ( ) poses discretely into multiplicity free irreducible representations of G1 O m , and Restrictions of Unitary Representations of Real Reductive Groups 197

in particular, gives rise to irreducible highest weight representations of G1. More generally, we have the following theorem which is part of Howe’s theory regarding reductive dual pairs: Theorem 8.2.2 (Theta correspondence [24]). Let π be the Weil representation of the metaplectic group G = Mp(n, R), and G = G G forms a dualpair with 1 2 G compact. Then the restriction π|G decomposes discretely into irreducible mul- 2 = π| tiplicity free representationsofG G1G2.In particular, the restriction G is G-admissible. Furthermore, each irreducible summandisahighest weight representation of G. A typical example of the setting of Theorem 8.2.2 is given locally as ( , , ) = ( ( , R), ( , R), ( )), g g1 g2 sp nm sp n o m (sp((p + q)m, R), u(p, q), u(m)) ∗ (sp(2pm, R), o (2p), sp(m)).

The classiﬁcation of irreducible highest weight representations was accom- plished in the early 1980s by Enright–Howe–Wallach and Jakobsen, independently (see [12, 28]). Quite a large part of irreducible highest weight representations were constructed as irreducible summands of the restriction of the Weil representation in the setting of Theorem 8.2.2 by Howe, Kashiwara, Vergne and others in the 1970s. Remark 8.1 Theorem 8.2.2 ﬁts into our framework of Sections 4 –7. Very recently, Nishiyama–Ochiai–Taniguchi [75] and Enright–Willenbring [13] have made a detailed study of irreducible summands occurring in the restriction π|G in the setting ( , ) of Theorem 8.2.2 under the assumption that G G is in the stable range with G2 ≤ R smaller, that is, m -rank G1. Then, as was pointed out in Enright–Willenbring [13, Theorem 6], the Gelfand–Kirillov dimension of each irreducible summand Y is dependent only on the dual pair and is independent of Y . They proved this result based on case-by-case argument. We note that this result follows directly from a general theory ([44, Theorem 3.7]) of discretely decomposable restrictions. Fur- thermore, one can show by using the results in [13, 44] that the associated variety V (Y ) (V (π)) gC coincides with the projection prg→g gC . (See Theorem 5.2.1 for the inclusive relation in the general case). t

8.2.3 Small representations

The idea of constructing representations of subgroups as irreducible summands works also for non-highest weight representations. As one can observe from the criterion for the admissibility of restrictions (see Theorem 6.3.4), the restriction π|G tends to be discretely decomposable if its asymptotic K -support ASK (π) is small. In particular, if π is a minimal representation, then ASK (π) is one dimensional (Remark 6.3.5). Thus, there is a good possibility that ASK (π) is discretely decomposable if π is “small”. 198 T. Kobayashi

For example, if G = O(p, q) (p, q ≥ 2; p + q ≥ 8 even) and π is the minimal representation of G, then the restriction to its natural subgroup G = O(p, q) × O(q) is G-admissible for any q, q such that q +q = q (see [57, Theorem 4.2]). In this case, branching laws give rise to unitary representations “attached to” minimal elliptic orbits (recall the terminology from §7.1). It is also G-admissible if G = U(p, q), p = 2p and q = 2q. The idea of constructing representations via branching laws was also used in a paper by Gross and Wallach [19], where they constructed interesting“small”unitary representations of exceptional Lie groups G by taking the restrictions of another small representation of G. Discretely decomposable branching laws for noncompact G are used also in the theory of automorphic forms for exceptional groups by J.-S. Li [64]. Note also that Neretin [74] recently constructed some of irreducible unitary representations of O(p, q) with K -ﬁxed vector (namely, so-called spherical unitary representations) as discrete spectrum of the restriction of irreducible representations of U(p, q). In his case, the restriction contains both continuous and discrete spectrum.

8.3 Branching problems In general, it is a hard problem to ﬁnd branching laws of unitary representations. Except for highest weight modules (e.g. the Weil representation, holomorphic discrete series representations) or principal series representations, not much has been studied systematically on the branching laws of irreducible unitary representations of reductive Lie groups with respect to noncompact subgroups until recently. From the viewpoint of ﬁnding explicit branching laws, an advantage of admissible restrictions is that we may employ algebraic techniques because there is no continuous spectrum. Recently, a number of explicit branching laws have been found (e.g. [19, 20, 38, 40, 41, 57, 64, 65, 66, 108]) in the context of admissible restrictions to noncompact reductive subgroups.

8.4 Global analysis Let G/H be the homogeneous space of a Lie group G by a closed subgroup H. The idea of noncommutative harmonic analysis is to try to understand functions on G/H by means of representations of the group G. For example, we refer in this Lie Theory series to the volume [3] and chapters by van den Ban, Delorme, and Schlichtkrull on this subject where G/H is a reductive symmetric space. Our main concern here is with non-symmetric spaces for which very little is known.

8.4.1 Global analysis and restriction of representations

Our approach to this problem is different from traditional approaches: The main machinery here is the restriction of representations. Restrictions of Unitary Representations of Real Reductive Groups 199

1) Embed G/H into a larger homogeneous space G/H. 2) Realize a representation π of G onasubspace V → C∞(G/H). 3) Restrict the space V of functions to the submanifold G/H, and understand it by the restriction π|G of representations. One may also consider variants of this idea by replacing C∞ by L2, holomorphic functions, sections of vector bundles, or cohomologies. Also, one may consider the restriction to submanifolds after taking normal derivatives (e.g. [72, 29]). Our optimistic idea here (which is used in [38, 41], for example) is that the knowledge of any two of the three would be usefulinunderstanding the remaining one.

Restriction from G to G

Analysis on G/H Analysis on G/H

In particular, we shall consider the setting where G/H is the space on which we wish to develop harmonic analysis, and where G/H is the space on which we have already a good understanding of harmonic analysis (e.g.agroup manifold or a symmetric space).

8.4.2 Discrete series and admissible representations

As we explained in §7.1.7 and §7.1.8, a necessary and sufﬁcient condition for the existence of discrete series representations is known for a reductive group and also for a reductive symmetric space. However, in the generality that H ⊂ G is a pair of reductive subgroups, it is still an open problem to determine which homogeneous space G/H admits discrete series representations. Let us apply the strategyin§8.4.1 to a non-symmetric homogeneous space G/H. We start with a discrete series π for G/H, and consider the branching law of the restriction π|G . We divide the status of an embedding G/H → G/H into three cases, from the best to more general settings. 1) The case G/H G/H. If subgroups G, H ⊂ G satisfy G H = G then we have a natural diffeomorphism G/H G/H. For example,

U(p, q)/U(p − 1, q) SO0(2p, 2q)/SO0(2p − 1, 2q), G2(R)/SL(3, R) SO0(4, 3)/SO0(3, 3), G2(R)/SU(2, 1) SO0(4, 3)/SO0(4, 2). (see [41, Example 5.2] for a list of such examples). In this case, any discrete spectrum of the branching law contribute to discrete series representations for G/H, and conversely, all discrete series representations for G/H are obtained in this way. 200 T. Kobayashi

2) (Generic orbit) Suppose G/H is a principal orbit in G/H in the sense of Richardson, namely, there is a G-open subset U in G/H such that any G-orbit in U is isomorphic to G/H. Then, any discrete spectrum of the branching law π|G contributes to discrete series representations for G/H [46, §8]. See [38, 41, 63] for concrete examples. 3) (General case; e.g.sigular orbits) For a general embedding G/H → G/H the above strategy may not work; the restriction of L2-functions does not always yield L2-functions on submanifolds. A remedy for this is to impose the G-admissibility of the restriction of π, which justiﬁes again the above strategy ([46]). For instance, let us consider the action of a group G on G itself. If we consider the left action, then the action is transitive and we cannot get new results from the above strategy. However, the action is nontrivial if we consider it from both the left and the right such as G → G, x → gxσ(g)−1 for some groupautomorphism σ of G (e.g. σ is the identity, a Cartan involution, etc.). For example, if G = Sp(2n, R) and take an involutive automorphism σ such that Gσ Sp(n, C), then the following homogeneous manifolds

G/H = Sp(2n, R)/(Sp(n0, C) × GL(n1, C) ×···×GL(nk, C)) occurasG-orbits on G for any partition (n0,...,nk) of n. (These orbits arise as “general case”, namely, in the case (3).) Then, we can prove that there always exist discrete series for the above homogeneous spaces for any partition (n0,...,nk) of n by using the criterion of admissible restrictions (Theorem 6.3.4). This is new except for the case n1 =···=nk = 0, where G/H = Sp(2n, R)/Sp(n, C) is a symmetric space. We refer to [46] for further details. There are also further results, for example, by Neretin, Olshanski–Neretin, and Ørsted–Vargas that interact the restriction of representations and harmonic analysis on homogeneous manifolds [73, 74, 78].

8.5 Discrete groups and restriction of unitary representations 8.5.1 Matsushima–Murakami’s formula Let G be a reductive linear Lie group, and a cocompact discrete subgroup without torsion. By Gelfand–Piateski-Shapiro’s theorem (see Theorem 2.4.3), the right regular representation on L2(\G) is G-admissible, so that it decomposes discretely: 2 L (\G) n(π)π π∈G with n(π) < ∞. Since acts on the Riemannian symmetric space G/K properly discontinuously and freely, the quotient space X = \G/K becomes a compact smooth manifold. Its cohomology group can be described by means of the multiplicities n(π) by a theorem of Matsushima–Murakami (see a treatise of Borel–Wallach [6]): Restrictions of Unitary Representations of Real Reductive Groups 201

Theorem 8.5.1. Let X = \G/Kbeas above. Then

∗ n(π) ∗ H (X; C) C ⊗ H (g, K ; πK ). π∈G ∗ Here H (g, K ; πK ) denotes the (g, K )-cohomologyofthe(g, K )-module πK n(π) ∗ ∗ ([6, 95]). We say C ⊗ H (g, K ; πK ) is the π-component of H (X; C), and ∗ denote by H (X)π . ∗ Furthermore, H (g, K ; πK ) is nonzero except for a ﬁnite number of π, for which the (g, K )-cohomologies are explicitly computed by Vogan and Zuckerman ([101]). (More precisely, such π is exactly the representations that we explained in §7.1.3, namely, π is isomorphic to certain (λ) with λ = ρλ.)

8.5.2 Vanishing theorem for modular varieties

Matsushima–Murakami’s formula interacts the topology of a compact manifold X = \G/K with unitary representations of G. Its object is a single manifold X. Let us consider the topology of morphisms, that is our next object is a pair of manifolds Y, X. For this, we consider the following setting:

⊂ G ⊃ K , ∩∩ ∩ ⊂ G ⊃ K, such that G ⊂ G are a pair of reductive linear Lie groups, K := K ∩G is a maximal compact subgroup, and := ∩ G is a cocompact in G. Then Y := \G/K is also a compact manifold. We have a natural map

ι : Y → X.

Then, the modular variety ι(Y ) is totally geodesic in X. We write [Y ] ∈ Hm(Y ; Z) for the fundamental class deﬁned by Y , where we put m = dim Y . Then, the cycle ι∗[Y ] in the homology group Hm(X; Z) is called the modularsymbol. Theorem 8.5.2 (avanishing theorem for modularsymbols). If ASK(π)∩CK(K ) ={0} (see Theorem 6.3.4) andifπ = 1 (the trivialone-dimensional representation), then the modularsymbol ι∗[Y ] is annihilated by the π-componentHm(X)π in the m perfect pairing Hm(X; C) × H (X; C) → C. The discreteness of irreducible decomposition plays a crucial role both in Matsu- shima–Murakami’s formula and in the above-mentioned vanishing theorem for modular varieties. In the former, L2(\G) is G-admissible (Gelfand–Piateski-Shapiro), while the restriction π|G is G -admissible (see Theorem 6.3.4) in the latter. 202 T. Kobayashi

8.5.3 Clifford–Klein problem

A Clifford–Klein form of a homogeneous space G/H is the quotient manifold \G/H where is a discrete subgroupofG acting properly discontinuously and freely on G/H. Any Riemannian symmetric space G/K admits a compact Clifford– Klein form (Borel [5]). On the other hand, there is no compact Clifford–Klein form of O(n, 1)/O(n−1), namely, any complete Lorentz manifold with constant sectional curvature is noncompact (Calabi–Markus phenomenon [7]). It remains an unsolved problem to classify homogeneous spaces G/H which admit compact Clifford–Klein forms even for the special case where G/H is a symmetric space such as SL(n, R)/SO(p, n − p). Recently, Margulis revealed a new connection of this problem with restrictions of unitary representations. He found an obstruction for the existence of compact Clifford–Klein forms for G/H. His approach is to consider the unitary representation of G on the Hilbert space L2(\G) from the right ( is a discrete subgroupofG), and to take the restriction to the subgroup H. The key techniqueistostudy the asymptotic behavior of matrix coefﬁcients of these unitary representations (see a paper of Margulis[70] and also of Oh [76]). We refer to [49, 71] and references therein for an overall exposition and open questions related to this problem.

Acknowledgement. This exposition is based on courses given by the author at the European School on Group Theory in Odense, in August 2000. I am very grateful to the organizers, Bent Ørsted and Henrik Schlichtkrull for their warm hospitality during my visit. I have had also the good fortune to give courses relavant to this material, at the Summer School at Yonsei University, organized by W. Schmid and J.-H. Yang in 1999, at the graduate course at Harvard University in 2001, and also at the University of Tokyo in 2002. I am very grateful to many colleagues at these institutions for the supportive atmosphere. The comments, questions, and valuable advice of all readers have been a great help in writingup this work. It is my pleasure to acknowledge my deep indebtness to the secretariat of RIMS, for indispensable help in preparing the LATEX manuscript.

References

[1] M. F. Atiyah, The Harish-Chandra character, London Math. Soc. Lecture Note Series, Vol. 34, 1979, 176–181. [2] V. Bargmann, Irreducible unitary representations of the Lorentz group. Ann.ofMath. (2) 48 (1947), 568–640. [3] E. van den Ban, P. Delorme, and H. Schlichtkrull. Lie Theory:Harmonic Analysis on Symmetric Spaces, Progress in Mathematics, Vol. 230, (eds. J.-P. Anker and B. Ørsted), Birkha¨user, Boston, 2004. [4] W. Borho and J. L. Brylinski, Differential operators on homogeneous spaces I, Invent. Math., 69 (1982), 437–476; Part III, ibid. 80 (1985), 1–68. Restrictions of Unitary Representations of Real Reductive Groups 203

[5] A. Borel, Compact Clifford–Klein forms of symmetric spaces, Topology 2 (1963), 111–122. [6] A. Borel and N. Wallach, ContinuousCohomology, Discrete Subgroups, and Represen- tationsofReductive Groups, Second edition, Mathematical Surveys and Monographs, Vol. 67, American Mathematical Society, Providence, RI, 2000. [7] E. Calabi and L. Markus, Relativistic space forms, Ann.ofMath., 75 (1962), 63–76. [8] D. H. Collingwood and W. M. McGovern, Nilpotent orbits in semisimple Lie algebras, Van Nostrand Reinhold Mathematics Series, Van Nostrand Reinhold Co., New York, 1993. [9] P. Delorme, Formule de Plancherel pour les espaces symetriq´ ues red´ uctifs, Annals of Math., 147 (1998), 412–452. [10] M. Duflo, Theorie´ de Mackey pour les groupes de Lie algebriq´ ues, Acta Math., 149 (1982), 153–213. [11] M. Duflo and J. Vargas, Proper map and multiplicities; in preparation. [12] T. Enright, R. Howe and N. Wallach, A classification of unitary highest weight modules, Representation theory of reductive groups (Park City,Utah, 1982), 97–143, Prog. Math., Vol. 40, Birkhaüser, Boston, 1983. [13] T. Enright and J. Willenbring, Hilbert series, Howe duality and branching for classical groups, Ann.ofMath. (2), 159 (2004), 337–375. [14] J. Faraut, Distributions spheriq´ ues sur les espaces hyperboliques, J. Math. Pures Appl., (9) 58 (1979), 369–444. [15] M. Flensted-Jensen, Discrete series for semisimple symmetric spaces, Annals of Math., 111 (1980), 253–311. [16] M. Flensted-Jensen, Analysis on Non-Riemannian Symmetric Spaces, CBMS Regional Conference Series in Mathematics, Vol. 61, Amer. Math. Soc., 1986. [17] I. M. Gelfand and N. Ya. Vilenkin, Generalized Functions. Vol. 4: Applicationsofhar- monic analysis, (English translation), Academic Press, New York, London, 1964. [18] I. M. Gelfand, M. I. Graev and I. Piateski-Shapiro, Representation TheoryandAuto- morphic Functions,Saunders Math. Books, 1969. [19] B. H. Gross and N. R. Wallach, A distinguished family of unitary representations for the exceptional groups of real rank = 4, Lie Theoryand Geometry,Inhonor of Bertram Kostant, Progress in Math., Vol. 123, 1994, Birkhaüser, 289–304. [20] B. Gross and N. Wallach, Restriction of small discrete series representations to symmetric subgroups, Proc. Sympos. Pure Math., 68 (2000), A.M.S., 255–272. [21] Harish-Chandra, Representations of semi-simple Lie groups, I, III, Trans. A.M.S., 75 (1953), 185–243; 76 (1954), 234–253; IV, Amer.J.Math., 77 (1955), 743–777. [22] H. Hecht, D. Miliciˇ c,´ W. Schmid, and J. A. Wolf, Localization and standard modules for real semisimple Lie groups. I. The duality theorem, Invent. Math. 90 (1987), 297–332. [23] S. Helgason, Differential Geometry, Lie Groups and Symmetric Spaces, Pure Appl. Math. Vol. 80, (1978). [24] R. Howe, θ-series and invariant theory, Proc. Symp. Pure Math., 33 (1979), Amer. Math. Soc., 275–285. [25] R. Howe, Transcending classical invariant theory, J. Amer. Math. Soc., 2 (1989), 535– 552. [26] R. Howe and E.-C. Tan, Homogeneousfunctions on light cones: the infinitesimal structure of some degenerate principal series representations, Bull. Amer. Math. Soc., 28 (1993), 1–74. [27] J-S. Huang, P. Pandziˇ c´ and G. Savin, New dual pair correspondences, DukeMath.J., 82 (1996), 447–471. 204 T. Kobayashi

[28] H. P. Jakobsen, Hermitian symmetric spaces and their unitary highest weight modules, J. Funct. Anal. 52 (1983), 385–412. [29] H. P. Jakobsen and M. Vergne, Restrictions and expansions of holomorphic representations, J. Funct. Anal., 34, (1979), 29–53. [30] J. C. Jantzen, Nilpotent orbits in representation theory. Lie Theory, Prog. Math., Vol. 228, 2003, Birkhaüser, Boston, 1–211. [31] A. Joseph, Annihilators and associated varieties of unitary highest weight modules, Ann. Scient. Ecole´ Normale Superie´ ure, (1992), 1–45. [32] M. Kashiwara, T. Kawai and T. Kimura, FoundationsofAlgebraic Analysis, Princeton Math. Series, Vol. 37, Princeton Univ. Press, 1986. [33] M. Kashiwara and M. Vergne, K-types andsingular spectrum, Lect. Notes in Math., Vol. 728, 1979, Springer-Verlag, 177–200. [34] A. A. Kirillov, Elements of theTheory of Representations,Grundlehren der Mathema- tischen Wissenschaften, Band 220, Springer-Verlag, Berlin-New York, 1976. xi + 315 pp. [35] A. W. Knapp, Representation Theory of Semisimple Groups, An overview based on examples, Princeton Mathematical Series, Vol. 36, Princeton University Press, Princeton, NJ, 1986. [36] A. W. Knapp, Lie Groups Beyond an Introduction, Second edition, Progress in Mathe- matics, Vol. 140, Birkhaüser, Boston, 2002. [37] A. W. Knapp and D. Vogan, Jr., CohomologicalInductionandUnitary Representations, Princeton U.P., 1995. [38] T. Kobayashi, Unitary representations realized in L2-sections of vector bundles over semisimple symmetric spaces, Proceedings atthe27-28th Symp. of FunctionalAnalysis andRealAnalysis (1989), Math. Soc. Japan, 39–54. [39] T. Kobayashi, SingularUnitary Representations and Discrete Series for Indefinite Stiefel Manifolds U(p, q; F)/U(p−m, q; F), Mem. Amer. Math. Soc., Vol.462, 1992. [40] T. Kobayashi, The restriction of Aq(λ) to reductive subgroups, Proc. Japan Acad., 69 (1993), 262–267. [41] T. Kobayashi, Discrete decomposability of the restriction of Aq(λ) with respect to reductive subgroups and its application, Invent. Math., 117 (1994), 181–205. [42] T. Kobayashi, Multiplicity free branching laws for unitary highest weight modules, Proceedingsofthe Symposium on Representation Theoryheld at Saga,Kyushu 1997, ed. K. Mimachi, 1997, 9–17. [43] T. Kobayashi, Discrete decomposability of the restriction of Aq(λ), II. —micro-local analysis and asymptotic K -support, Annals of Math., 147 (1998), 709–729. [44] T. Kobayashi, Discrete decomposability of the restriction of Aq(λ), III. —restriction of Harish-Chandra modules and associated varieties, Invent. Math., 131 (1998), 229–256. [45] T. Kobayashi, Harmonic analysis on homogeneous manifolds of reductive type and unitary representation theory, Sugaku, 46 (1994), Math. Soc. Japan (in Japanese), 124–143; Translations, Series II, Selected Papers on Harmonic Analysis, Groups, andInvariants (K. Nomizu, ed.), Vol. 183 (1998), A.M.S., 1–31. [46] T. Kobayashi, Discrete series representations for the orbit spaces arising from two in- volutions of real reductive Lie groups, J. Funct. Anal., 152 (1998), 100–135. [47] T. Kobayashi, Theory of discrete decomposable branching laws of unitary representations of semisimple Lie groups and some applications, Sugaku, 51 (1999), Math. Soc. Japan (in Japanese), 337–356; English translation, to appear in Sugaku Exposi- tion,Transl. Series, A.M.S. Restrictions of Unitary Representations of Real Reductive Groups 205

[48] T. Kobayashi, Discretely decomposable restrictions of unitary representations of reductive Lie groups — examples and conjectures, Advanced Study inPure Math., 26 (2000), 98–126. [49] T. Kobayashi, Discontinuous groups for non-Riemannian homogeneousspaces, Math- ematics unlimited—2001 andbeyond, Springer, Berlin, 2001, 723–747. [50] T. Kobayashi, Unitary representations and branching laws, Proceedingsofthe I.C.M. 2002 at Beijing, 2 (2002), 615–627. [51] T. Kobayashi, Conformal geometry and analysis on minimal representations, Lecture Notes of the Winter School 2002 on geometry and physics, Czech Republic, Rendiconti del Circolo Matematico di Palermo, Serie II,Suppl. 71 (2003), 15–40. [52] T. Kobayashi, Introduction to actions of discrete groups on pseudo-Riemannian homogeneous manifolds, The 2000 Twente Conference on Lie Groups (Enschede). Acta Appl. Math., 73 (2002), 115–131. [53] T. Kobayashi, Admissible restrictions of unitary representations of reductive Lie groups – discrete decomposability and symplectic geometry, in preparation. [54] T. Kobayashi, Geometry of multiplicity-free representations of GL(n), visible actions on flag varieties, and tri-unity, Acta Appl. Math., 81 (2004), 129–146. [55] T. Kobayashi and T. Oda, Vanishing theorem of modular symbols on locally symmetric spaces, Comment. Math. Helvetici, 73 (1998), 45–70. [56] T. Kobayashi and K. Ono, Note on Hirzebruch’s proportionality principle, J. Fac. Sci. Univ.Tokyo 37 (1990), 71–87. [57] T. Kobayashi and B. Ørsted, Analysis on minimal representations of O(p, q), Part II. Branching Laws, Adv.Math., 180 (2003), 513–550. [58] T. Kobayashi and B. Ørsted, Analysis on minimal representations of O(p, q), Part III. − , − Ultrahyperbolic equations on Rp 1 q 1, Adv.Math., 180 (2003), 551–595. [59] T. Kobayashi and T. Oshima, Lie Groups and Lie Algebras, Iwanami Shoten, (in Japanese), (2001), pp. 610; English translation (to appear). [60] B. Kostant and S. Rallis, Orbits and representations associated with symmetric spaces. Amer.J.Math. 93 (1971), 753–809. [61] R. Langlands, On theclassification of irreducible representationsofreal algebraic groups, mimeographed notes, 1973. [62] S-T. Lee and H-Y. Loke, Degenerate principal series of U(p, q) and Spin(p, q), Com- positio Math., 132 (2002), 311–348. [63] J-S. Li, On the discrete series of generalized Stiefel manifolds, Trans. Amer. Math. Soc., Vol. 340, 1993, 753–766. [64] J-S. Li, Two reductive dual pairs in groups of type E, Manuscripta Math., 91 (1996), 163–177. [65] H-Y. Loke, Restrictions of quaternionic representations, J. Funct. Anal., 172 (2000), 377–403. [66] H-Y. Loke, Howe quotients of unitary characters and unitary lowest weight modules, preprint. [67] G. W. Mackey, Induced representations of locally compact groups I, Annals of Math., 55 (1952), 101–139. [68] D. Manchon, Distributions as` upport compact et representations´ unitaires. J. Lie The- ory, 9 (1999), 403–424. [69] D. Manchon, Front d’onde et propagation des singularites´ pour un vecteur-distribution, Colloq. Math., 81 (1999), 161–191. [70] G. Margulis, Existence of compact quotients of homogeneous spaces, measurably proper actions, and decay of matrix coefficients, Bull. Soc. Math.France, 125 (1997), 1–10. 206 T. Kobayashi

[71] G. Margulis, Problems and conjectures in rigidity theory, Mathematics: Frontiers and Perspectives, 161–174, Amer. Math. Soc., Providence, RI, 2000. [72] S. Martens, The characters of the holomorphic discrete series, Proc. Nat. Acad. Sci. USA, 72, (1975), 3275–3276. [73] Y. A. Neretin and G. I. Ol’shanskii, Boundary values of holomorphic functions, special unitary representations of O(p, q), and their limits, ZapiskiNauchnykh Seminarov POMI, 223 (1995), 9–91. [74] Y. Neretin, Plancherel formula for Berezin deformation of L2 on Riemannian symmetric space, J. Funct. Anal. 189 (2002), 336–408. [75] K. Nishiyama, H. Ochiai, and K. Taniguchi, Bernstein degree and associated cycles of Harish-Chandra modules — Hermitian symmetric case —, Asterisq´ ue 273 (2001), 13–80. [76] H. Oh, Tempered subgroups and representations with minimal decay of matrix coefficients, Bull. Soc. Math.France, 126 (1998), 355–380. [77] G. Olafsson´ and B. Ørsted, The holomorphic discrete series of an affine symmetric space, I, J. Funct. Anal., 81 (1988), 126–159. [78] B. Ørsted and J. Vargas, Restriction of square integrable representations: discrete spectrum, preprint. [79] T. Oshima, A realization of Riemannian symmetric spaces, J. Math. Soc. Japan, 30 (1978), 117–132. [80] T. Oshima, A realization of semisimple symmetric spaces and construction of boundary value maps, Adv. Stud. Pure. Math., 14 (1988), 603–650. [81] W. Rossmann, Analysis on real hyperbolic spaces, J. Funct. Anal. 30 (1978), 448–477. [82] S. A. Salamanca-Riba and D. A. Vogan Jr., On the classification of unitary representations of reductive Lie groups. Ann.ofMath. 148 (1998), 1067–1133. [83] M. Sato, Theory of hyperfunctions. I, J. Fac. Sci. Univ.Tokyo, Sect. I, 8 (1959), 139– 193. [84] H. Schlichtkrull, Hyperfunctions andHarmonic Analysis on Symmetric Spaces, Progress in Mathematics, Vol. 49, Birkhaüser, Boston, 1984. [85] W. Schmid, On a conjecture of Langlands, Ann.Math. 93 (1971), 1–42. [86] W. Schmid and K. Vilonen, Characteristic cycles and wave front cycles of representations of reductive Lie groups, Ann.ofMath., 151 (2000), 1071–1118. [87] Y. Shalom, Rigidity, unitary representations of semisimple groups, and fundamental groups of manifolds with rank one transformation group, Ann.ofMath., 152 (2000), 113–182. [88] R. Sjamaar, Convexity properties of the moment mapping, re-examined, Adv.Math., 138 (1998), 46–91. [89] J. Sekiguchi, Remarks on real nilpotent orbits of a symmetric pair, J. Math. Soc. Japan, 39 (1987), 127–138. [90] P. Torasso, Methode´ des orbites de Kirillov–Duflo et representations´ minimales des groupes simples sur un corps local de caracteristiq´ uenulle, DukeMath.J., 90 (1997), 261–377. [91] P. E. Trapa, Annihilators and associated varieties of Aq(λ) modules for U(p, q), Com- positio Math., 129 (2001), 1–45. [92] M. Vergne, Some comments on the work of I. E. Segal on group representations. Special issue dedicated to the memory of I. E. Segal. J. Funct. Anal., 190 (2002), 29–37. [93] D. A. Vogan, Jr., The algebraic structure of the representation of semisimple Lie groups. I, Ann.ofMath., 109 (1979), 1–60. [94] D. A. Vogan, Jr., Singular unitary representations, Lecture Notes in Mathematics, Vol. 880, 1980, Springer, 506–535. Restrictions of Unitary Representations of Real Reductive Groups 207

[95] D. A. Vogan, Jr., RepresentationsofReal Reductive Lie Groups, Progress in Math., Vol. 15, Birkhaüser, 1981. [96] D. A. Vogan, Jr., Unitarizability of certain series of representations, Annals of Math., 120 (1984), 141–187. [97] D. A. Vogan, Jr., Irreducibility of discrete series representations for semisimple symmetric spaces, Advanced Studies inPure Math. 14 (1988), 191–221. [98] D. A. Vogan, Jr., Associated varieties and unipotent representations, Harmonic Anal- ysis on Reductive Lie Groups, Progress in Math., Vol. 101, Birkhaüser, 1991, 315–388. [99] D. A. Vogan, Jr., The orbit method and unitary representations for reductive Lie groups, Algebraic and analytic methods in representation theory (Sonderborg, 1994), (eds. B. Ørsted and H. Schlichtkrull) 243–339, Perspect. Math., Vol. 17, Academic Press, San Diego, 1996. [100] D. A. Vogan, Jr., ALanglands classification for unitary representations, Analysis on Homogeneous Spaces and Representation Theory of Lie Groups, Okayama–Kyoto 1997, (eds. T. Kobayashi et al.) Adv. Stud. Pure Math., Vol. 26, 2000, 299–324. [101] D. A. Vogan, Jr. and G. J. Zuckerman, Unitary representations with nonzero cohomology, Compositio Math., 53 (1984), 51–90. [102] N. R. Wallach, Representations of semi-simple Lie groups and Lie algebras, Lie the- ories and their applications, Proc. Ann. Sem.Canad. Math.Congr., Queen’sUniv., Kingston,Ont., 1977, 154–246, Academic Press, New York, 1978. [103] N. R. Wallach, On the unitarizability of derived functor modules, Invent. Math., 78 (1984), 131–141. [104] N. R. Wallach, Real Reductive Groups I, II, Pure and Appl. Math., Vol. 132, Academic Press, 1988, 1992. [105] G. Warner, Harmonic Analysis onSemisimple Lie Groups I, II, Springer Verlag, Berlin, 1972. [106] J. A. Wolf, Representations that remain irreducible on parabolic subgroups, Dif- ferential geometrical methods in mathematical physics Proc. Conf., Aix-en- Provence/Salamanca,1979, 129–144, Lecture Notes in Math., Vol. 836, Springer, Berlin, 1980. [107] H. Wong, Dolbeault cohomological realization of Zuckerman modules associated with finite rank representations, J. Funct. Anal., 129 (1995), 428–454. [108] J. Xie, Restriction of discrete series of SU(2, 1) to S(U(1) × U(1, 1)), J. Funct. Anal., 122 (1994), 478–518, Ph.D dissertation, Rutgers University. [109] A. Yamamoto, Orbits in the flag variety and images of the moment map for classical groups. I, Represent. Theory , 1 (1997), 329–404.